L4: Specification-driven
You define what to build instead of building it: write the spec, set the tests, judge the results. Leave for twelve hours and come back to a green suite.
The path.
- The harness as an artifact In Agentic Engineering: Core Ideas "Agentic coding" is an overloaded term.
- Evals-as-tests In Verification & Testing for Agents If generation is cheap and verification is the bottleneck, then verification infrastructure is your leverage.
- LLMOps & guardrails Shipping LLM features without evals and observability is flying blind: outputs are non-deterministic, quality is subjective, and regressions are silent.
Check yourself.
One question per step. Take it cold to find where to start, or after reading to see what stuck. Nobody's grading you.
Best so far:
Question 1 of 3
-
Your agent keeps failing on a task. Where does the handbook say to look first?
Most agent failures are harness gaps, not model gaps. A decent model with a great harness beats a great model with a bad one.
Covered in step 1 · The harness as an artifact -
Your test suite is green. Why run evals on agent behavior too?
A prompt, model, or harness change can silently degrade agent behavior while every existing test stays green; evals-as-tests catch those regressions.
Covered in step 2 · Evals-as-tests -
How should prompts be managed once a feature is in production?
Version prompts like code: store them outside the app, track versions, run evals per version, and roll back. The anti-pattern is prompts hard-coded and edited in place with no history or eval gate.
Covered in step 3 · LLMOps & guardrails
-
L0: Manual
The ground floor: AI as a search engine and occasional tab-complete. No real productivity change.
Everyone starts here. There's nothing to unlock yet.
-
L1: Discrete task offloading
You hand the AI small, self-contained jobs: write the unit tests, draft the docstring, generate one well-scoped function. The real work is still yours.
Chat assistants · Prompting 101 · IDE assistants
Explore path -
L2: Active pairing
The AI writes alongside you all day, handling the boring parts while you steer and review everything as it lands. Shapiro estimates about 90% of AI-native developers live here.
CLI agents · AGENTS.md & CLAUDE.md · What an agent actually is
Explore path -
L3: Human-in-the-loop management
You mostly stop typing code. Agents run several tasks at once and your day becomes reviewing their work. Most people never go past this level.
Own the outer loop · Agentic code review · Role separation · MCP & tools
Explore path -
L4: Specification-driven
You define what to build instead of building it: write the spec, set the tests, judge the results. Leave for twelve hours and come back to a green suite.
The harness as an artifact · Evals-as-tests · LLMOps & guardrails
You're on this rung
-
L5: The dark factory
Agents building the software with humans out of the loop. Shapiro reports it working only for teams under five people, and calls it likely our future.
Swarms & fleets · The factory model · Human factors
Explore path