--- title: Why SWE-bench Verified no longer measures frontier... description: OpenAI has stopped reporting SWE-bench Verified scores and recommends other model developers do the same, citing two critical issues. First, an audit of 138... canonical: https://daily.dev/posts/why-swe-bench-verified-no-longer-measures-frontier-coding-capabilities-yitnqurmf twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Why SWE-bench Verified no longer measures frontier coding capabilities | daily.dev og:description: OpenAI has stopped reporting SWE-bench Verified scores and recommends other model developers do the same, citing two critical issues. First, an audit of 138... og:url: https://daily.dev/posts/why-swe-bench-verified-no-longer-measures-frontier-coding-capabilities-yitnqurmf og:image: https://api.daily.dev/og/posts/YItnquRmf.png og:image:alt: Why SWE-bench Verified no longer measures frontier coding capabilities og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Why SWE-bench Verified no longer measures frontier coding capabilities **[Hacker News](https://daily.dev/sources/hn)** · 10 min read · 5 upvotes · 0 comments ## Summary OpenAI has stopped reporting SWE-bench Verified scores and recommends other model developers do the same, citing two critical issues. First, an audit of 138 problems that OpenAI o3 consistently failed found that 59.4% contain flawed test cases — either too narrow (enforcing specific implementation details) or too wide (testing functionality not described in the problem). Second, contamination analysis shows all tested frontier models (GPT-5.2, Claude Opus 4.5, Gemini 3 Flash) can reproduce exact gold patches and verbatim problem details, indicating training data exposure. Models that have seen benchmark solutions during training score higher not because they are more capable, but because they have extra information needed to pass underspecified tests. OpenAI recommends switching to SWE-bench Pro, which shows significantly less contamination, and calls on the industry to invest in privately authored, uncontaminated benchmarks. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Separating signal from noise in coding evaluations](https://daily.dev/posts/separating-signal-from-noise-in-coding-evaluations-swoud4dtk) · Hacker News · 0 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#ai-coding](https://daily.dev/tags/ai-coding), [#llm](https://daily.dev/tags/llm), [#openai](https://daily.dev/tags/openai) [View this post on daily.dev](https://daily.dev/posts/why-swe-bench-verified-no-longer-measures-frontier-coding-capabilities-yitnqurmf) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Why SWE-bench Verified no longer measures frontier coding capabilities","url":"https://daily.dev/posts/why-swe-bench-verified-no-longer-measures-frontier-coding-capabilities-yitnqurmf","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/why-swe-bench-verified-no-longer-measures-frontier-coding-capabilities-yitnqurmf"},"datePublished":"2026-04-26T16:53:48.082Z","dateModified":"2026-10-14T02:28:22.183Z","description":"OpenAI has stopped reporting SWE-bench Verified scores and recommends other model developers do the same, citing two critical issues. First, an audit of 138...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/3840ae73bbdab20d8fee955265d199e6?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/3840ae73bbdab20d8fee955265d199e6?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/why-swe-bench-verified-no-longer-measures-frontier-coding-capabilities-yitnqurmf","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":5},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,ai-coding,llm,openai","timeRequired":"PT10M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Why SWE-bench Verified no longer measures frontier coding capabilities"}]} ```