--- title: How I evaluate a coding agent before letting it near our... description: Public benchmarks like SWE-bench can be misleading when evaluating coding agents for real-world use. A practical evaluation framework is proposed using your... canonical: https://daily.dev/posts/how-i-evaluate-a-coding-agent-before-letting-it-near-our-codebase-dk65yam5j twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How I evaluate a coding agent before letting it near our codebase | daily.dev og:description: Public benchmarks like SWE-bench can be misleading when evaluating coding agents for real-world use. A practical evaluation framework is proposed using your... og:url: https://daily.dev/posts/how-i-evaluate-a-coding-agent-before-letting-it-near-our-codebase-dk65yam5j og:image: https://api.daily.dev/og/posts/Dk65Yam5J.png og:image:alt: How I evaluate a coding agent before letting it near our codebase og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How I evaluate a coding agent before letting it near our codebase **[kartik-nvjk](https://daily.dev/sources/tvtveimdkrcqwudullfmy)** · [@kartiknvj](https://daily.dev/kartiknvj) · 4 min read · 4 upvotes · 0 comments ## Summary Public benchmarks like SWE-bench can be misleading when evaluating coding agents for real-world use. A practical evaluation framework is proposed using your own merged pull requests: revert the repo to pre-merge state, give the agent the original issue, and score the resulting diff against five criteria — review readiness, correct tool ordering, codebase coherence, plan adherence, and appropriate backtracking. The test set is refreshed weekly with the latest 50 merged PRs, scaled to 200 when comparing close candidates. Key benchmark pitfalls include training data contamination, artificially clean repo surfaces, idiom drift with internal conventions, and a lower review bar than real teams apply. ## Content When we were picking a coding agent, I almost chose one off its SWE-bench score. It looked great on the public leaderboard, around 70 percent. Then I ran it against our own codebase and it solved closer to 30 percent of the tickets I gave it. Both numbers were real. They just measured different things. Public benchmarks are built to be neutral. Our codebase is the opposite. SWE-bench runs on a fixed set of open-source Python issues. Our repo has framework conventions the benchmark has never seen, internal libraries that are not on any package index, and a review bar that would reject half the diffs the public set happily accepts. A few things kept biting me: - **Contamination.** The public issues predate most model cutoffs, so an agent may have seen the accepted fix during training. That inflates the score and tells you nothing about new work. - **A cleaner surface than real life.** The benchmark hands the agent a tidy repo, a test command, and a clear issue. My engineers hand it a half-loaded editor, a partial diff, and a vague Slack thread. - **Idiom drift.** Our custom wrappers, decorators, and fixture style are things no public benchmark can model, so agents that look idiomatic still produce diffs our reviewers bounce on sight. - **The review-bar gap.** The benchmark passes a patch if it makes the failing test pass. We pass a patch if it does that _and_ does not break unrelated tests, does not reformat the file, does not add stray imports, and is small enough to review in about ninety seconds. So I stopped evaluating agents on anyone else's tickets and started using ours. The best test set I had was the work we had already merged. I take our last 50 merged pull requests, revert the repo to the commit right before each one, hand the agent the original issue or commit message, and let it work. Then I compare its diff against the diff we actually merged and shipped. The score is not whether the text matches. It is whether that diff would have passed our code review. Once the agent produces a patch, I score it on five things, because each one catches a failure the leaderboard never shows: - **Would the diff pass review?** Not character-for-character equal to what we merged, but behaviorally equivalent and clean enough that a reviewer would approve it. I dock it for unrelated reformatting, for failing a test our version passed, and for being twice the size it needed to be. - **Did it use its tools in the right order?** A coding agent reads files, edits them, runs tests, and commits. I check that it read a file before editing it, edited the file it said it would, ran the tests after the edit instead of committing blind, and committed only what it actually changed. - **Did it keep the whole codebase coherent?** The failure reviewers hate most is renaming a helper in one module and leaving broken call sites in three others. I follow the imports out from every file it changed and check that every place needing an update actually got one, config included. - **Did the plan survive the loop?** The current agents write a plan before they touch anything. I check that the plan matched what was asked, and that the final diff still matched the plan. Agents drift halfway through, quietly contradict the request, and still pass the tests. - **Did it know when to undo?** When a test failed after an edit, did the agent revert and try another way, or pile more code on top until the suite was in pieces? An agent averaging more than two edits per failure without a single revert is one that ships regressions. A few things made this actually work. I refresh the set every week, adding the newest merged PR and dropping the oldest, so it keeps looking like what we ship today. Fifty PRs is enough to separate good agents from bad ones, and when I need to choose between two that both look strong, I grow it toward 200. I also run a quick security pass on every diff, since agents invent fake credentials and reach for risky shell patterns more often than you would expect. None of this is exotic. It is just the difference between asking "is this agent good" and asking "is this agent good at our work." The leaderboard answers the first question. Only your own merged PRs answer the second, and the second is the one you are actually paying for. If you have brought a coding agent into your team, what convinced you it was ready: a benchmark score, a trial run on real tickets, or something else? ## Similar posts on daily.dev - [Many SWE-bench-Passing PRs Would Not Be Merged into Main](https://daily.dev/posts/many-swe-bench-passing-prs-would-not-be-merged-into-main-wkgbbkgid) · Hacker News · 0 upvotes · 0 comments - [Separating signal from noise in coding evaluations](https://daily.dev/posts/separating-signal-from-noise-in-coding-evaluations-swoud4dtk) · Hacker News · 0 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#security](https://daily.dev/tags/security), [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents), [#code-review](https://daily.dev/tags/code-review) [View this post on daily.dev](https://daily.dev/posts/how-i-evaluate-a-coding-agent-before-letting-it-near-our-codebase-dk65yam5j) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/how-i-evaluate-a-coding-agent-before-letting-it-near-our-codebase-dk65yam5j","headline":"How I evaluate a coding agent before letting it near our codebase","text":"Public benchmarks like SWE-bench can be misleading when evaluating coding agents for real-world use. A practical evaluation framework is proposed using your own merged pull requests: revert the repo to pre-merge state, give the agent the original issue, and score the resulting diff against five criteria — review readiness, correct tool ordering, codebase coherence, plan adherence, and appropriate backtracking. The test set is refreshed weekly with the latest 50 merged PRs, scaled to 200 when comparing close candidates. Key benchmark pitfalls include training data contamination, artificially clean repo surfaces, idiom drift with internal conventions, and a lower review bar than real teams apply.","url":"https://daily.dev/posts/how-i-evaluate-a-coding-agent-before-letting-it-near-our-codebase-dk65yam5j","datePublished":"2026-07-15T15:31:22.617Z","dateModified":"2026-09-13T20:59:27.857Z","author":{"@type":"Person","name":"kartik-nvjk","url":"https://daily.dev/kartiknvj","image":"https://media.daily.dev/image/upload/s--3gGgsVCw--/f_auto/v1781456774/avatars/avatar_TvTVeiMdkRCqWUDullFmy?_a=BAMAMiWQ0","description":"AI Developer | Making AI reliable, trustworthy & accessible to everyone | community contributor","interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"EndorseAction"},"userInteractionCount":2810}},"image":"https://media.daily.dev/image/upload/s--McvrzjF_--/f_auto/v1784129482/posts/Dk65Yam5J?_a=BAMAMicg0","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":4},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/tvtveimdkrcqwudullfmy","name":"kartik-nvjk"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"kartik-nvjk","item":"https://daily.dev/sources/tvtveimdkrcqwudullfmy"},{"@type":"ListItem","position":3,"name":"How I evaluate a coding agent before letting it near our codebase"}]} ```