A backend engineer spent one week doing real Rails codebase work exclusively through LLM agents — writing code, reviewing merge requests, writing specs — then analyzed the session logs. Key stats: 2,200 session files across four tools (Claude Code, Codex, Cursor, Amp), 350 hand-typed prompts, ~22,000 tokens typed producing ~1,500 tokens of output per token typed, and 54 wall-clock hours with sessions live. Code review consumed ~40% of Claude spend. The per-hunk reviewer produced 59 candidate findings but only 17 (29%) were real after refutation. Mutation testing was the only method with 100% precision across 4 findings. Key lessons: LLM agent failures are correlated not independent, so multi-agent unanimity is a reason to reread the prompt rather than proceed; negative answers are more dangerous than false positives because they close lines of inquiry; bake-offs are more valuable for exposing divergence than for picking winners; and standing rules only hold if turned into mechanical checks rather than context-dependent instructions.
Table of contents
How I measured itThe numbersThe workWhere the models were wrongWhat it was worthWhat I take away from thisInstead of a conclusionQuestions this post answers
What precision rate does LLM-based code review actually achieve compared to mutation testing?
In a week of real backend work, a per-hunk LLM reviewer produced 59 candidate findings across three runs, of which only 17 (29%) were confirmed real after a separate refutation step. Mutation testing, by contrast, found 4 findings and all 4 were real (100%). The difference is structural: mutation testing runs an experiment on live code, while LLM review produces an opinion without changing anything. Engineers weighing LLM review against traditional verification methods track these kinds of precision comparisons on daily.dev.
Why do multiple LLM agents agreeing on a finding not increase confidence the way redundant systems do?
LLM agent failures are not independent — they share training data, share priors, and are typically given the same input. Four agents agreeing on a wrong answer looks identical to four agents making the same mistake because the prompt was flawed. In practice, when all agents agree, the right response is to reread the prompt, not to proceed with confidence. Every real defect found in one week came from a method doing something the other agents were not doing. Developers building multi-agent review pipelines find relevant workflow patterns and failure analyses on daily.dev.
How do LLM agents handle standing rules and instructions as context windows fill up?
Standing rules written into instruction files tend to hold early in a session and break late, because they stop competing for attention once the context window is full. The exception is rules that have been turned into mechanical checks — a banned-word rule enforced via a grep command was followed consistently across sessions, while a review-comment format saved to an instructions file was missing the next day in two separate sessions. Any rule that depends on the model remembering will degrade; rules implemented as search commands do not. Backend engineers managing LLM agent workflows share context-window gotchas like this on daily.dev.