--- title: What We Learned by Reproducing 2,200 papers from ICML description: A Hugging Face-organized hackathon had over 1,200 participants use coding agents (Claude Code, Codex, Cursor, and others) to attempt reproductions of ICML 2026... canonical: https://daily.dev/posts/what-we-learned-by-reproducing-2-200-papers-from-icml-buot4npxf twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: What We Learned by Reproducing 2,200 papers from ICML | daily.dev og:description: A Hugging Face-organized hackathon had over 1,200 participants use coding agents (Claude Code, Codex, Cursor, and others) to attempt reproductions of ICML 2026... og:url: https://daily.dev/posts/what-we-learned-by-reproducing-2-200-papers-from-icml-buot4npxf og:image: https://api.daily.dev/og/posts/BUoT4nPxf.png og:image:alt: What We Learned by Reproducing 2,200 papers from ICML og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # What We Learned by Reproducing 2,200 papers from ICML **[Hugging Face](https://daily.dev/sources/huggingface)** · 9 min read · 0 upvotes · 0 comments ## Summary A Hugging Face-organized hackathon had over 1,200 participants use coding agents (Claude Code, Codex, Cursor, and others) to attempt reproductions of ICML 2026 papers, claim by claim. Across 19 days, participants produced 6,816 public reproduction logbooks covering 2,226 papers (about a third of the conference), launching 2,962 cloud compute jobs. An automated judge model issued per-claim verdicts of verified, falsified, toy, or inconclusive. Results: 51% of examined papers had at least one claim independently verified, while 23% had at least one claim falsified or contested, including 242 papers where independent teams reached opposite verdicts on the same claims. Detailed case studies show agents catching subtle theoretical errors, code-versus-theory mismatches, and evaluation flaws like padding-token dilution. The piece argues humans remain essential for steering agents, catching scale-dependent errors, and making perceptual judgments agents cannot fully automate, likening the human role to a PI managing capable but fallible researchers. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What percentage of ICML 2026 papers had claims that were falsified when independently reproduced using AI coding agents? About 23% of examined papers (496 out of roughly 2,226) had at least one claim falsified or contested, including 49 papers where every claim was falsified and 242 papers where independent reproduction teams reached opposite verdicts on the same claims. In total, 35,908 claims were judged, with 3,978 individual claims confirmed through real experiments. _Track how AI-assisted reproducibility findings reshape trust in published ML research on daily.dev._ ### How did an AI agent-based reproduction catch a flaw in the paper Towards Optimal Robustness in Learning-Augmented Paging? A participant's reproduction measured the paper's claimed additive robustness term and found it actually grows like 0.38 ln k rather than the claimed O(1), locating the exact step in the proof that breaks. A follow-up re-implementation extended the test to k = 1,024 and confirmed the growth at roughly nine sigma, showing the true robustness is Hk + Θ(log k) instead of Hk + O(1). _Developers evaluating AI-agent research tooling can follow real audit outcomes like this one on daily.dev._ ### Can AI coding agents fully replace human reviewers when reproducing machine learning research papers? No, agents alone hit real limits: they got stuck in local loops, misread scale-dependent behavior such as verdicts on the paging paper that stopped checking before log-k growth appeared, and sometimes built falsifications on top of unit mismatches. The most reliable results came from workflows where a human steered the agent, questioned assumptions, or made perceptual judgments, similar to a PI guiding grad students. _Anyone weighing how much to trust agentic research workflows can follow these findings on daily.dev._ ## Similar posts on daily.dev - [Major AI conference flooded with peer reviews written fully by AI](https://daily.dev/posts/major-ai-conference-flooded-with-peer-reviews-written-fully-by-ai-ufcjjr3xg) · Hacker News · 1 upvotes · 0 comments - [Humans May Soon be Unable to Supervise AI without Asking Another AI](https://daily.dev/posts/humans-may-soon-be-unable-to-supervise-ai-without-asking-another-ai-gxbefwjqx) · Medium · 0 upvotes · 0 comments --- Tags: [#machine-learning](https://daily.dev/tags/machine-learning), [#ai-agents](https://daily.dev/tags/ai-agents), [#scientific-research](https://daily.dev/tags/scientific-research) [View this post on daily.dev](https://daily.dev/posts/what-we-learned-by-reproducing-2-200-papers-from-icml-buot4npxf) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"What We Learned by Reproducing 2,200 papers from ICML","url":"https://daily.dev/posts/what-we-learned-by-reproducing-2-200-papers-from-icml-buot4npxf","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/what-we-learned-by-reproducing-2-200-papers-from-icml-buot4npxf"},"datePublished":"2026-08-13T17:01:33.506Z","dateModified":"2026-08-13T17:02:01.193Z","description":"A Hugging Face-organized hackathon had over 1,200 participants use coding agents (Claude Code, Codex, Cursor, and others) to attempt reproductions of ICML 2026...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/de94d83b65289c261abd4762d00a9a30?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/de94d83b65289c261abd4762d00a9a30?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hugging Face","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hugging Face","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/f1f55c67d81a4330acf5b90b26b0c8e1","url":"https://daily.dev/sources/huggingface"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/what-we-learned-by-reproducing-2-200-papers-from-icml-buot4npxf","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"machine-learning,ai-agents,scientific-research","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hugging Face","item":"https://daily.dev/sources/huggingface"},{"@type":"ListItem","position":3,"name":"What We Learned by Reproducing 2,200 papers from ICML"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/what-we-learned-by-reproducing-2-200-papers-from-icml-buot4npxf#faq","mainEntity":[{"@type":"Question","name":"What percentage of ICML 2026 papers had claims that were falsified when independently reproduced using AI coding agents?","acceptedAnswer":{"@type":"Answer","text":"About 23% of examined papers (496 out of roughly 2,226) had at least one claim falsified or contested, including 49 papers where every claim was falsified and 242 papers where independent reproduction teams reached opposite verdicts on the same claims. In total, 35,908 claims were judged, with 3,978 individual claims confirmed through real experiments. Track how AI-assisted reproducibility findings reshape trust in published ML research on daily.dev."}},{"@type":"Question","name":"How did an AI agent-based reproduction catch a flaw in the paper Towards Optimal Robustness in Learning-Augmented Paging?","acceptedAnswer":{"@type":"Answer","text":"A participant's reproduction measured the paper's claimed additive robustness term and found it actually grows like 0.38 ln k rather than the claimed O(1), locating the exact step in the proof that breaks. A follow-up re-implementation extended the test to k = 1,024 and confirmed the growth at roughly nine sigma, showing the true robustness is Hk + Θ(log k) instead of Hk + O(1). Developers evaluating AI-agent research tooling can follow real audit outcomes like this one on daily.dev."}},{"@type":"Question","name":"Can AI coding agents fully replace human reviewers when reproducing machine learning research papers?","acceptedAnswer":{"@type":"Answer","text":"No, agents alone hit real limits: they got stuck in local loops, misread scale-dependent behavior such as verdicts on the paging paper that stopped checking before log-k growth appeared, and sometimes built falsifications on top of unit mismatches. The most reliable results came from workflows where a human steered the agent, questioned assumptions, or made perceptual judgments, similar to a PI guiding grad students. Anyone weighing how much to trust agentic research workflows can follow these findings on daily.dev."}}]} ```