--- title: Most coding agent benchmarks skip large-scale... description: A new multilingual benchmark, SWE-Bench ProMax, tests AI coding agents on large-scale refactoring across 170 real-commit instances spanning seven languages,... canonical: https://daily.dev/posts/most-coding-agent-benchmarks-skip-large-scale-refactoring-not-this-one--p55v6sihn twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Most coding agent benchmarks skip large-scale refactoring. Not this one. | daily.dev og:description: A new multilingual benchmark, SWE-Bench ProMax, tests AI coding agents on large-scale refactoring across 170 real-commit instances spanning seven languages,... og:url: https://daily.dev/posts/most-coding-agent-benchmarks-skip-large-scale-refactoring-not-this-one--p55v6sihn og:image: https://api.daily.dev/og/posts/p55v6sIHN.png og:image:alt: Most coding agent benchmarks skip large-scale refactoring. Not this one. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Most coding agent benchmarks skip large-scale refactoring. Not this one. **[The New Stack](https://daily.dev/sources/newstack)** · 6 min read · 0 upvotes · 0 comments ## Summary A new multilingual benchmark, SWE-Bench ProMax, tests AI coding agents on large-scale refactoring across 170 real-commit instances spanning seven languages, finding the best model resolves only 41.2% of tasks. Researchers built it to fix quality issues plaguing existing benchmarks like SWE-bench Verified, where nearly 60% of unsolved instances had flawed tests. Experts from SUSE and ActiveState argue current LLMs lack true structural understanding of large codebases, struggle with attention limits on large contexts, and fail to handle time-dependent issues like race conditions. Proposed fixes include combining LLMs with retrieval-augmented generation, LSP and code-graph tooling, human validation, and deterministic tooling rather than raw multi-file diffs. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How well do AI coding agents perform on large-scale code refactoring tasks? The best-performing model achieves only a 41.2% resolve rate on SWE-Bench ProMax, a new benchmark specifically designed to test large-scale refactoring. The benchmark contains 170 instances drawn from real commits across seven languages: Python, Java, TypeScript, Go, C, C++, and Rust, with issue descriptions and test suites manually curated to remove quality flaws found in prior benchmarks. _Track how coding agents actually perform on hard refactoring work by following benchmark coverage on daily.dev._ ### Why do AI coding agents struggle with large-scale refactoring compared to smaller coding tasks? Large language models lack true structural understanding of code, relying instead on token proximity that doesn't guarantee grasp of a codebase's actual dependencies and data flows. Large or complex codebases that max out the context window can overwhelm attention mechanisms, causing models to miss important details, while time-dependent issues like race conditions, lost idempotency, and retry handling further trip up solutions that only pass superficial tests. _Developers weighing AI agents for refactoring can follow this debate on model limitations via daily.dev._ ### Why are many AI coding benchmarks considered unreliable indicators of real-world agent performance? Many benchmarks run quickly on small, deterministic contexts that exclude hard tasks like large-scale refactoring, and frontier models often already know the answers because solutions have leaked into training data. An audit found nearly 60% of unsolved SWE-bench Verified instances contained flawed tests, meaning failures may reflect broken test suites rather than genuine model shortcomings. _Anyone choosing which benchmark to trust for evaluating coding agents can track this scrutiny on daily.dev._ ## Similar posts on daily.dev - [13 Models and 4 Agents on SWE Tasks: Go, Java, Python, Rust, TS](https://daily.dev/posts/13-models-and-4-agents-on-swe-tasks-go-java-python-rust-ts-ithomlqxu) · Hacker News · 2 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents) [View this post on daily.dev](https://daily.dev/posts/most-coding-agent-benchmarks-skip-large-scale-refactoring-not-this-one--p55v6sihn) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Most coding agent benchmarks skip large-scale refactoring. Not this one.","url":"https://daily.dev/posts/most-coding-agent-benchmarks-skip-large-scale-refactoring-not-this-one--p55v6sihn","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/most-coding-agent-benchmarks-skip-large-scale-refactoring-not-this-one--p55v6sihn"},"datePublished":"2026-08-21T11:55:26.953Z","dateModified":"2026-08-21T11:55:54.655Z","description":"A new multilingual benchmark, SWE-Bench ProMax, tests AI coding agents on large-scale refactoring across 170 real-commit instances spanning seven languages,...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5a64614eb3e725152225e3afe1374ed8?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5a64614eb3e725152225e3afe1374ed8?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"The New Stack","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"The New Stack","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/newstack","url":"https://daily.dev/sources/newstack"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/most-coding-agent-benchmarks-skip-large-scale-refactoring-not-this-one--p55v6sihn","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,llm,ai-agents","timeRequired":"PT6M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"The New Stack","item":"https://daily.dev/sources/newstack"},{"@type":"ListItem","position":3,"name":"Most coding agent benchmarks skip large-scale refactoring. Not this one."}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/most-coding-agent-benchmarks-skip-large-scale-refactoring-not-this-one--p55v6sihn#faq","mainEntity":[{"@type":"Question","name":"How well do AI coding agents perform on large-scale code refactoring tasks?","acceptedAnswer":{"@type":"Answer","text":"The best-performing model achieves only a 41.2% resolve rate on SWE-Bench ProMax, a new benchmark specifically designed to test large-scale refactoring. The benchmark contains 170 instances drawn from real commits across seven languages: Python, Java, TypeScript, Go, C, C++, and Rust, with issue descriptions and test suites manually curated to remove quality flaws found in prior benchmarks. Track how coding agents actually perform on hard refactoring work by following benchmark coverage on daily.dev."}},{"@type":"Question","name":"Why do AI coding agents struggle with large-scale refactoring compared to smaller coding tasks?","acceptedAnswer":{"@type":"Answer","text":"Large language models lack true structural understanding of code, relying instead on token proximity that doesn't guarantee grasp of a codebase's actual dependencies and data flows. Large or complex codebases that max out the context window can overwhelm attention mechanisms, causing models to miss important details, while time-dependent issues like race conditions, lost idempotency, and retry handling further trip up solutions that only pass superficial tests. Developers weighing AI agents for refactoring can follow this debate on model limitations via daily.dev."}},{"@type":"Question","name":"Why are many AI coding benchmarks considered unreliable indicators of real-world agent performance?","acceptedAnswer":{"@type":"Answer","text":"Many benchmarks run quickly on small, deterministic contexts that exclude hard tasks like large-scale refactoring, and frontier models often already know the answers because solutions have leaked into training data. An audit found nearly 60% of unsolved SWE-bench Verified instances contained flawed tests, meaning failures may reflect broken test suites rather than genuine model shortcomings. Anyone choosing which benchmark to trust for evaluating coding agents can track this scrutiny on daily.dev."}}]} ```