--- title: Many SWE-bench-Passing PRs Would Not Be Merged into Main description: METR researchers had 4 active maintainers from 3 SWE-bench Verified repositories review 296 AI-generated pull requests to assess whether benchmark scores... canonical: https://daily.dev/posts/many-swe-bench-passing-prs-would-not-be-merged-into-main-wkgbbkgid twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Many SWE-bench-Passing PRs Would Not Be Merged into Main | daily.dev og:description: METR researchers had 4 active maintainers from 3 SWE-bench Verified repositories review 296 AI-generated pull requests to assess whether benchmark scores... og:url: https://daily.dev/posts/many-swe-bench-passing-prs-would-not-be-merged-into-main-wkgbbkgid og:image: https://api.daily.dev/og/posts/wKgBbkgiD.png og:image:alt: Many SWE-bench-Passing PRs Would Not Be Merged into Main og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Many SWE-bench-Passing PRs Would Not Be Merged into Main **[Hacker News](https://daily.dev/sources/hn)** · 18 min read · 0 upvotes · 0 comments ## Summary METR researchers had 4 active maintainers from 3 SWE-bench Verified repositories review 296 AI-generated pull requests to assess whether benchmark scores translate to real-world usefulness. The key finding: maintainer merge rates are on average 24 percentage points lower than SWE-bench automated grader scores, meaning roughly half of test-passing PRs would not be merged. After normalizing for noise in maintainer decisions using a golden baseline of human-written PRs, AI agents score about 50% of what the benchmark suggests. Rejection reasons include code quality issues, breaking unrelated code, and core functionality failures. The study also finds the rate of improvement in maintainer-assessed quality may be slower than benchmark scores suggest, though this finding is weaker. Importantly, the authors note this is not a fundamental capability ceiling — agents were not given the chance to iterate based on feedback as human developers would. The conclusion is that SWE-bench scores should be treated as one signal among many, not a direct proxy for real-world developer productivity. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Separating signal from noise in coding evaluations](https://daily.dev/posts/separating-signal-from-noise-in-coding-evaluations-swoud4dtk) · Hacker News · 0 upvotes · 0 comments - [Open source maintainers are drowning in AI-generated pull requests. Enterprise teams are next.](https://daily.dev/posts/open-source-maintainers-are-drowning-in-ai-generated-pull-requests-enterprise-teams-are-next--zwod8qmqp) · The New Stack · 0 upvotes · 0 comments - [Most coding agent benchmarks skip large-scale refactoring. Not this one.](https://daily.dev/posts/most-coding-agent-benchmarks-skip-large-scale-refactoring-not-this-one--p55v6sihn) · The New Stack · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#code-review](https://daily.dev/tags/code-review) [View this post on daily.dev](https://daily.dev/posts/many-swe-bench-passing-prs-would-not-be-merged-into-main-wkgbbkgid) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Many SWE-bench-Passing PRs Would Not Be Merged into Main","url":"https://daily.dev/posts/many-swe-bench-passing-prs-would-not-be-merged-into-main-wkgbbkgid","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/many-swe-bench-passing-prs-would-not-be-merged-into-main-wkgbbkgid"},"datePublished":"2026-03-12T00:04:07.625Z","dateModified":"2026-03-12T00:04:39.422Z","description":"METR researchers had 4 active maintainers from 3 SWE-bench Verified repositories review 296 AI-generated pull requests to assess whether benchmark scores...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/a163994e794f7b964bb7b6971ab4e361?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/a163994e794f7b964bb7b6971ab4e361?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/many-swe-bench-passing-prs-would-not-be-merged-into-main-wkgbbkgid","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,code-review","timeRequired":"PT18M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"Many SWE-bench-Passing PRs Would Not Be Merged into Main"}]} ```