Verification & Testing for Agents

If generation is cheap and verification is the bottleneck, then verification infrastructure is your leverage. As Osmani puts it, until it catches up with generation, "human review isn't optional overhead, it's the safety system." This chapter is about building that system so it scales with output instead of collapsing under it.

The stakes here are measurable. GitClear's analysis of 211 million changed lines (2020-2024) reads like the verification gap's fossil record. Code cloning rose from 8.3% to 12.3% of changed lines, refactoring sank from ~25% of changed lines to under 10%, and copy/pasted code exceeded moved code for the first time in the dataset's history. That is more duplication and less consolidation, exactly the decay you'd predict. Simon Willison's "Vibe Engineering" states the constructive inverse, that "if your project has a robust, comprehensive and stable test suite agentic coding tools can fly with it." The same infrastructure that catches the decay also sets how much autonomy you can safely grant.

Agentic code review: evidence over vibes

The first move is to stop treating review as a human reading every line and start treating it as a pipeline that produces evidence a human then judges. That means automated checks wired into the agent's lifecycle (tests, type checks, linters, and increasingly a dedicated reviewer agent that reads the diff against the original spec and flags divergence) running as hooks on lifecycle events (pre-commit, post-edit) so the checks are systematic rather than remembered. The outer-loop discipline applies. You want an accountability contract (what was checked, what the evidence showed, why you shipped), not a gut-feel thumbs-up. And the reviewer must be separate from the implementer. A self-reviewing agent inherits the same self-grading bias that makes "are you done?" unreliable. Kent Beck, TDD's creator, supplies the adversarial caveat from his "augmented coding" experiments. Tests become a "superpower" with agents because agents "can (and do!) introduce regressions," and yet he has "trouble stopping AI agents from deleting tests in order to make them 'pass!'" (Pragmatic Engineer interview). The guardrail is also a target. Protect the verification pipeline from the implementer, with tests owned by the checker role or locked against edits, or an agent optimizing for green will optimize the greenness checker away.

Evals-as-tests

For agent behavior itself (as opposed to the code it writes), the emerging practice is evals-as-tests, encoding the behaviors you require as automated evaluations you run like a test suite, so you catch regressions when you change a prompt, swap a model, or edit the harness. This matters because the usual test suite checks the product, not the process that produced it, and a harness change can silently degrade quality with every existing test still green. The factory model's insistence that tests encode "what should happen, not what currently does" applies doubly here, because your evals are the executable specification of acceptable agent behavior, and they're only as honest as the cases you thought to write.

The practical literature here is unusually good. Hamel Husain's "Your AI Product Needs Evals", the essay much of the discipline traces back to, argues that "unsuccessful products almost always share a common root cause: a failure to create robust evaluation systems." It organizes evals into three levels (cheap assertions run constantly, human-and-model grading of real transcripts, A/B tests) and insists the unglamorous core practice is reading your own data, because "you must remove all friction from the process of looking at data." Anthropic's "Demystifying Evals for AI Agents" adds the agent-specific mechanics. Start with 20-50 realistic tasks sourced from observed failures, mix code-based, model-based, and human graders, and grade outcomes rather than trajectories, since checking for "a sequence of tool calls in the right order" proves "too rigid and results in overly brittle tests."

Who validates the validators?

Reviewer agents and LLM judges push the verification problem up a level rather than solving it. Shreya Shankar and colleagues' aptly titled "Who Validates the Validators?" is the peer-reviewed form of the self-grading worry. It reports that "LLM-generated evaluators simply inherit all the problems of the LLMs they evaluate, requiring further human validation." Its key empirical finding, criteria drift, is the useful surprise, that "users need criteria to grade outputs, but grading outputs helps users define criteria." You cannot fully specify "good" before contact with real outputs, so a static rubric handed to a judge model decays in production. The operational consequence, echoed in Anthropic's guidance that judges "should be closely calibrated with human experts," is that the human calibration loop above the judge is a permanent fixture rather than scaffolding. Automate the volume, and keep sampling the judgments.

DTU: safe, high-volume testing against the world

The hardest verification problem is testing against services you don't control, the Oktas, Jiras, Slacks, and Google Workspaces your agent touches. You can't hammer live third-party APIs at agent volumes without hitting rate limits, tripping abuse detection, running up costs, or corrupting real data, and you especially can't safely test the dangerous failure modes. StrongDM's Digital Twin Universe (DTU) (writeup) answers this with "behavioral clones of the third-party services our software depends on," offline replicas that mirror those APIs and their observable behavior. You build the double from the API contract and known edge cases, then validate it against the real dependency until the behavioral differences stop showing up. The payoff is testing "at volumes and rates far exceeding production limits", deterministic, replayable, and able to exercise failure modes that would be dangerous or impossible against live services. StrongDM's own account of why this is newly possible is worth noting. Engineers always wanted production-grade test replicas but never proposed them because the effort seemed prohibitive. Agents made building and maintaining the clones cheap enough that what was "unthinkable six months ago" is now "routine." That is the deeper pattern of this whole chapter. The same drop in generation cost that created the verification crisis also makes it affordable to build the verification infrastructure that resolves it. Whether you reach for a full DTU or a humbler set of high-fidelity mocks, the principle holds. As agent output volume explodes, quality only stays up if you can verify at the same volume, and that means investing in test doubles, sandboxes, and evals as deliberately as you invest in the agents themselves.