--- title: Coding agents can be evaluated. We just have to evaluate... description: Coding agents are non-deterministic and complex, but that doesn't make them unevaluable. A coding agent is a full system — model, harness, tools, context, and... canonical: https://daily.dev/posts/coding-agents-can-be-evaluated-we-just-have-to-evaluate-the-work--ulsexxnux twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Coding agents can be evaluated. We just have to evaluate the work. | daily.dev og:description: Coding agents are non-deterministic and complex, but that doesn't make them unevaluable. A coding agent is a full system — model, harness, tools, context, and... og:url: https://daily.dev/posts/coding-agents-can-be-evaluated-we-just-have-to-evaluate-the-work--ulsexxnux og:image: https://api.daily.dev/og/posts/uLsExxNuX.png og:image:alt: Coding agents can be evaluated. We just have to evaluate the work. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Coding agents can be evaluated. We just have to evaluate the work. **[The New Stack](https://daily.dev/sources/newstack)** · 6 min read · 1 upvotes · 0 comments ## Summary Coding agents are non-deterministic and complex, but that doesn't make them unevaluable. A coding agent is a full system — model, harness, tools, context, and environment — and must be evaluated as such. Rather than exact-match grading, evaluations should use executable contracts: does it build, do tests pass, are APIs compatible, do migrations work, are security checks clean? Beyond pass rates, a useful eval covers six layers: outcome, change quality, trajectory, human intervention, economics, and production impact. Non-determinism is a statistical problem solved by running tasks multiple times and reporting success distributions. Even open-ended tasks with incomplete requirements can be evaluated using simulated product-owner interactions and hidden behavioral contracts. Benchmarks like ICAE-Bench and Dialogue SWE-Bench are already moving in this direction. The standard isn't perfection — it's whether the evaluation produces better decisions than demos and provider claims alone. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How should I evaluate an AI coding agent instead of just checking if it passes tests? Evaluate the whole system across multiple layers: outcome (did it satisfy the task), change quality (would you accept the implementation), trajectory (how it got there), human intervention required, economic cost, and production impact after merge. Executable contracts like build success, hidden test passage, API compatibility, and security checks provide harder-to-bluff signals than a passing test alone, since an agent can weaken assertions or hard-code values to fake success. _daily.dev surfaces evaluation frameworks like this for teams deciding how to trust coding agents in production._ ### Why does a coding agent solve the same task successfully one time and fail the next? Non-determinism in coding agents is a statistical problem, not proof that evaluation is impossible. Because an agent may solve a task on one run and fail on another, a single run is insufficient evidence; the fix is running the same task multiple times under controlled starting conditions and reporting the success distribution, cost variance, completion time, and frequency of serious failure modes with confidence intervals. _understanding agent variance helps developers judge reliability claims before betting workflows on daily.dev's feed of AI engineering practices._ ### How can an AI coding agent be evaluated on ambiguous, open-ended tasks like 'modernize this service'? Give the agent access to a controlled user or product-owner simulator backed by a private set of hidden requirements, then score whether it identifies ambiguity, asks useful questions, incorporates answers, and avoids inventing requirements, before checking the finished system against the hidden behavioral contract. Research benchmarks such as ICAE-Bench and Dialogue SWE-Bench already isolate this interactive, dialogue-driven capability from raw coding performance. _teams weighing agent autonomy on fuzzy tasks can track emerging evaluation research like this through daily.dev._ ## Similar posts on daily.dev - [The Anatomy of Harness Engineering: How to Evaluate, Iterate, and Guard AI Coding Agents](https://daily.dev/posts/the-anatomy-of-harness-engineering-how-to-evaluate-iterate-and-guard-ai-coding-agents-m4zyf0sa6) · Google Developers · 1 upvotes · 1 comments - [Coding agents are only as good as the signals you feed them](https://daily.dev/posts/coding-agents-are-only-as-good-as-the-signals-you-feed-them-9euzsy5sw) · The New Stack · 1 upvotes · 0 comments - [Making AI work through eval hygiene](https://daily.dev/posts/making-ai-work-through-eval-hygiene-9yhow3tph) · InfoWorld · 0 upvotes · 0 comments --- Tags: [#testing](https://daily.dev/tags/testing), [#ai-agents](https://daily.dev/tags/ai-agents) [View this post on daily.dev](https://daily.dev/posts/coding-agents-can-be-evaluated-we-just-have-to-evaluate-the-work--ulsexxnux) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Coding agents can be evaluated. We just have to evaluate the work.","url":"https://daily.dev/posts/coding-agents-can-be-evaluated-we-just-have-to-evaluate-the-work--ulsexxnux","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/coding-agents-can-be-evaluated-we-just-have-to-evaluate-the-work--ulsexxnux"},"datePublished":"2026-08-09T15:02:50.670Z","dateModified":"2026-09-13T20:57:15.737Z","description":"Coding agents are non-deterministic and complex, but that doesn't make them unevaluable. A coding agent is a full system — model, harness, tools, context, and...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/293ec74abed843f6d4038ea39772f019?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/293ec74abed843f6d4038ea39772f019?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"The New Stack","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"The New Stack","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/newstack","url":"https://daily.dev/sources/newstack"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/coding-agents-can-be-evaluated-we-just-have-to-evaluate-the-work--ulsexxnux","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"testing,ai-agents","timeRequired":"PT6M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"The New Stack","item":"https://daily.dev/sources/newstack"},{"@type":"ListItem","position":3,"name":"Coding agents can be evaluated. We just have to evaluate the work."}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/coding-agents-can-be-evaluated-we-just-have-to-evaluate-the-work--ulsexxnux#faq","mainEntity":[{"@type":"Question","name":"How should I evaluate an AI coding agent instead of just checking if it passes tests?","acceptedAnswer":{"@type":"Answer","text":"Evaluate the whole system across multiple layers: outcome (did it satisfy the task), change quality (would you accept the implementation), trajectory (how it got there), human intervention required, economic cost, and production impact after merge. Executable contracts like build success, hidden test passage, API compatibility, and security checks provide harder-to-bluff signals than a passing test alone, since an agent can weaken assertions or hard-code values to fake success. daily.dev surfaces evaluation frameworks like this for teams deciding how to trust coding agents in production."}},{"@type":"Question","name":"Why does a coding agent solve the same task successfully one time and fail the next?","acceptedAnswer":{"@type":"Answer","text":"Non-determinism in coding agents is a statistical problem, not proof that evaluation is impossible. Because an agent may solve a task on one run and fail on another, a single run is insufficient evidence; the fix is running the same task multiple times under controlled starting conditions and reporting the success distribution, cost variance, completion time, and frequency of serious failure modes with confidence intervals. understanding agent variance helps developers judge reliability claims before betting workflows on daily.dev's feed of AI engineering practices."}},{"@type":"Question","name":"How can an AI coding agent be evaluated on ambiguous, open-ended tasks like 'modernize this service'?","acceptedAnswer":{"@type":"Answer","text":"Give the agent access to a controlled user or product-owner simulator backed by a private set of hidden requirements, then score whether it identifies ambiguity, asks useful questions, incorporates answers, and avoids inventing requirements, before checking the finished system against the hidden behavioral contract. Research benchmarks such as ICAE-Bench and Dialogue SWE-Bench already isolate this interactive, dialogue-driven capability from raw coding performance. teams weighing agent autonomy on fuzzy tasks can track emerging evaluation research like this through daily.dev."}}]} ```