--- title: Separating signal from noise in coding evaluations description: OpenAI audited SWE-Bench Pro, a widely used coding benchmark, and found that approximately 30% of its 731 public tasks are broken. The audit used a pipeline... canonical: https://daily.dev/posts/separating-signal-from-noise-in-coding-evaluations-swoud4dtk twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Separating signal from noise in coding evaluations | daily.dev og:description: OpenAI audited SWE-Bench Pro, a widely used coding benchmark, and found that approximately 30% of its 731 public tasks are broken. The audit used a pipeline... og:url: https://daily.dev/posts/separating-signal-from-noise-in-coding-evaluations-swoud4dtk og:image: https://api.daily.dev/og/posts/SWouD4dTk.png og:image:alt: Separating signal from noise in coding evaluations og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Separating signal from noise in coding evaluations **[Hacker News](https://daily.dev/sources/hn)** · 6 min read · 0 upvotes · 0 comments ## Summary OpenAI audited SWE-Bench Pro, a widely used coding benchmark, and found that approximately 30% of its 731 public tasks are broken. The audit used a pipeline combining Codex-based investigator agents and reviews by five experienced software engineers per flagged task. Four main failure categories were identified: overly strict tests that enforce specific implementation details, underspecified prompts that omit requirements, low-coverage tests that allow incomplete fixes to pass, and misleading prompts that contradict test expectations. Human reviewers were more conservative than agents, finding more broken tasks and more overlapping issues per task. OpenAI retracts its earlier recommendation to adopt SWE-Bench Pro and calls on the evaluation community to build new benchmarks designed specifically for testing model capabilities rather than repurposing open-source pull request history. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Why SWE-bench Verified no longer measures frontier coding capabilities](https://daily.dev/posts/why-swe-bench-verified-no-longer-measures-frontier-coding-capabilities-yitnqurmf) · Hacker News · 5 upvotes · 0 comments - [The Benchmark Meaning Gap](https://daily.dev/posts/the-benchmark-meaning-gap-tyqdoxw6z) · JetBrains · 0 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#openai](https://daily.dev/tags/openai), [#ai-coding](https://daily.dev/tags/ai-coding) [View this post on daily.dev](https://daily.dev/posts/separating-signal-from-noise-in-coding-evaluations-swoud4dtk) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Separating signal from noise in coding evaluations","url":"https://daily.dev/posts/separating-signal-from-noise-in-coding-evaluations-swoud4dtk","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/separating-signal-from-noise-in-coding-evaluations-swoud4dtk"},"datePublished":"2026-07-09T07:50:02.045Z","dateModified":"2026-07-09T07:52:19.099Z","description":"OpenAI audited SWE-Bench Pro, a widely used coding benchmark, and found that approximately 30% of its 731 public tasks are broken. The audit used a pipeline...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/3305a976166122e4629984aa56f74bab?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/3305a976166122e4629984aa56f74bab?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/separating-signal-from-noise-in-coding-evaluations-swoud4dtk","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,openai,ai-coding","timeRequired":"PT6M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Separating signal from noise in coding evaluations"}]} ```