Can I trust what agents produce?
Verification pipelines, a separate reviewer, and the ops that catch regressions. The discipline that decides how much autonomy you can afford.
The path.
- Verification & Testing for Agents If generation is cheap and verification is the bottleneck, then verification infrastructure is your leverage.
- Maker / checker split In Orchestration Patterns Once one agent works, the obvious move is to run several. This is where the biggest gains and the biggest self-inflicted wounds both live.
- Evals & observability Shipping LLM features without evals and observability is flying blind: outputs are non-deterministic, quality is subjective, and regressions are silent.
- Know your own failure modes The techniques above make you faster. This chapter is about what they can quietly cost you, and how not to pay it.
Check yourself.
One question per step. Take it cold to find where to start, or after reading to see what stuck. Nobody's grading you.
Best so far:
Question 1 of 4
-
After a harness or prompt change, every code test still passes. Can you trust that agent quality held?
A harness change can silently degrade agent behavior with every existing test still green, which is why the chapter pushes evals-as-tests: encode required behaviors as evaluations you run like a test suite.
Covered in step 1 · Verification & Testing for Agents -
Why does a separate checker agent beat asking the implementer "are you done?"
Models systematically overestimate their own completeness, so a verifier with fresh context and an adversarial brief catches what the author's self-grading bias misses. Checkers over-report, though, so a human still judges the judge.
Covered in step 2 · Maker / checker split -
You adopt a strong LLM as judge to score outputs at scale. What does the chapter insist you still do?
LLM-as-judge scales far past human grading but carries position, verbosity, and self-preference biases, so the judge must be validated against human labels periodically.
Covered in step 3 · Evals & observability -
According to the research on cognitive surrender, why is over-trusting AI output so hard to catch in yourself?
In the cited Wharton trials, people accepted the AI's wrong answers 73% of the time while AI availability pushed their confidence up. You borrow the model's confidence without forming an independent judgment.
Covered in step 4 · Know your own failure modes