--- title: A 27B Model Learned to Direct Research Instead of Just... description: A newly discussed paper introduces Replica, a scalable task space for training research agents by deleting a result figure from a published paper and requiring... canonical: https://daily.dev/posts/a-27b-model-learned-to-direct-research-instead-of-just-coding-it-i1bup4itl twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: A 27B Model Learned to Direct Research Instead of Just Coding It | daily.dev og:description: A newly discussed paper introduces Replica, a scalable task space for training research agents by deleting a result figure from a published paper and requiring... og:url: https://daily.dev/posts/a-27b-model-learned-to-direct-research-instead-of-just-coding-it-i1bup4itl og:image: https://api.daily.dev/og/posts/I1bUp4ITL.png og:image:alt: A 27B Model Learned to Direct Research Instead of Just Coding It og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # A 27B Model Learned to Direct Research Instead of Just Coding It **[Collections](https://daily.dev/sources/collections)** · 3 min read · 2 upvotes · 0 comments ## Summary A newly discussed paper introduces Replica, a scalable task space for training research agents by deleting a result figure from a published paper and requiring an agent to reproduce it through actual experimentation. The resulting 27B parameter agent, Faraday, outsources code writing to GPT-5.5 while focusing its own training on directing research: choosing experiments, simplifying scope, and judging whether results look credible. On 68 unseen AI-for-science tasks, Faraday scored 0.791 versus 0.748 for Claude Opus 4.8 and 0.729 for GPT-5.5, winning outright on 60% of tasks. Rollout analysis suggests Faraday more often tested the actual mechanism in a paper rather than shortcutting to a plausible-looking number. The authors argue long-horizon scientific reasoning can be trained directly into model weights without heavy scaffolding, suggesting a smaller specialized 'director' model paired with a larger 'coder' model could outperform one large generalist model, at least for paper replication tasks. ## Content A 27-billion-parameter agent called Faraday just beat Claude Opus 4.8 and GPT-5.5 on paper replication tasks. It didn't do it by being a better coder. It did it by being a better *director* of coding. That distinction is the whole story here, and I think it's worth sitting with for a second. Faraday actually uses GPT-5.5 under the hood to write and run code. It just doesn't trust GPT-5.5 to decide what research to do. Faraday handles the hypothesis-driven thinking - what to implement, what to simplify, which experiments matter - and delegates the grunt work of implementation to a stronger coding model. The small model calls the big one as a tool, not the other way around. ## How you train something to "think like a researcher" The task setup, called Replica, is almost annoyingly simple once you hear it: take a published paper, strip out one of its results figures, and tell the agent to recreate that result by actually running the experiment. What makes this hard is everything papers leave out. Real papers don't specify every hyperparameter, every preprocessing step, every judgment call the original authors made. So replicating a result forces the same kind of hypothesis-driven exploration that open-ended research requires, and it surfaces exactly the details that got glossed over in the writeup. An automated rubric judge grades each attempt, and according to the authors it runs low-noise and lines up well with human judgments of replication quality. The team trained Faraday on 242 of these replication tasks, then tested it on 68 unseen AI-for-science tasks it had never seen during training. ## The numbers, and why they're more interesting than they look Faraday scored 0.791. Claude Opus 4.8 scored 0.748. GPT-5.5 scored 0.729. Faraday beat both models outright on 60% of the tasks. But the score gap isn't really the interesting part. What stood out in the rollout analysis is that Faraday was more likely to actually test the underlying mechanism the paper claimed, rather than taking a shortcut to an answer that merely looked correct. That's a meaningfully different failure mode than "got a lower score" - it suggests Faraday was doing something closer to actual science instead of pattern-matching toward plausible-looking plots. The researchers frame this as evidence that long-horizon scientific reasoning can live in a model's weights, rather than needing to be bolted on through increasingly elaborate agent harnesses and scaffolding. ## Why this might matter more than the benchmark suggests The implication I keep coming back to: you might not need one giant model that's good at everything. You can train a smaller model to specialize in the research-direction part of the job - forming hypotheses, deciding what's worth testing, judging whether a result is believable - and let it lean on larger, more general coding models for execution. That's a division of labor that mirrors how research teams actually work. Nobody expects the PI to also be the fastest programmer in the room. If this pattern holds up outside of paper replication, it's a real argument against the assumption that scientific AI progress just means scaling one model bigger. Paper: [Training AI Scientists to Replicate Research](https://arxiv.org/abs/2608.13331) ## Questions this post answers ### What is the Faraday research agent and how does it outperform larger models like GPT-5.5? Faraday is a 27B parameter agent trained to direct scientific research rather than write code itself; it delegates coding to GPT-5.5 while deciding what experiments to run, when to simplify, and whether results look believable. On 68 unseen AI-for-science tasks it scored 0.791, beating Claude Opus 4.8 (0.748) and GPT-5.5 (0.729), winning outright on 60% of tasks. _daily.dev surfaces emerging agent architectures like this for engineers tracking how AI research tools evolve._ ### What is the Replica task space used for training AI research agents? Replica is a scalable task space that trains research agents by taking a published paper, deleting one of its results figures, and requiring the agent to recreate that result by actually running the experiment. This forces the agent to fill in gaps papers leave out, such as what to implement, what to simplify, and which experiments matter, closely mirroring real replication work. An automated rubric judge grades attempts and tracks well with human judgment. _follow daily.dev for developments in benchmarks that test genuine reasoning over pattern matching in AI systems._ ## Community take How the wider developer community reacted, aggregated from 4 discussions and 19 comments across x (as of 2026-08-15). **TL;DR:** Reactions are intrigued by the idea of a small model directing a larger one, but several commenters question how much the replication benchmark actually proves and whether the evaluation setup can be trusted. **Sentiment:** 30% positive · 55% mixed · 15% skeptical **The case for** - The division of labor between a director model and an implementer model is seen as a plausible and economically sensible scaling path. - Some find the paper's ability to surface details original authors missed genuinely research-like. **The pushback** - The rubric/judge used to grade attempts may matter more than the 27B headline number and could cap generalization. - Replicating a paper is seen as a bounded task since the answer and methodology already exist, unlike open-ended research with no known outcome. - One commenter notes a separate attempt at unpublished replication saw function-level recovery drop sharply, raising doubts about how well this generalizes. - It's questioned whether the benchmark holds out papers published before the model's training cutoff. - The paper itself reportedly lacks code/artifacts, making it ironic that a paper about replication isn't easily replicable. - Unclear how the grading handles cases where original paper results don't actually replicate. **By community** - x (mixed): Replies mix genuine interest in the director/implementer split with pointed methodological questions about the benchmark's rigor and generalizability. **Hottest debate:** Whether replicating known, published results is a meaningful test of research judgment versus a much easier bounded task than real open-ended research. **Open questions** - Does the Replica benchmark hold out papers published after the models' training cutoffs? - How does the automated rubric/judge handle cases where the original paper's results don't actually replicate? - What is the cost of running these agents at scale? - Why doesn't the paper include code or artifacts to allow its own claims to be replicated? **Highlights** > @rohanpaul_ai the automated rubric grading each attempt is doing more work than the 27B number, that's what actually caps how far this generalizes > — [cdiamond on x](https://x.com/cdiamond/status/2088434789380407521) > @omarsar0 replicating a paper is bounded in a way open research isn't. you already know a result exists and roughly what experiment produced it, so this measures following an established methodology carefully. deciding which experiment to run next with no known answer waiting is a > — [vsaietta on x](https://x.com/vsaietta/status/2088314251819548705) > @omarsar0 Replication as a task space is smart, because the answer key is the paper and the paper has usually already been read. We rebuilt one where the findings were never public; function-level recovery stopped at 29.7%. Does Replica hold out papers past the model cutoff? > — [ipezyGJ on x](https://x.com/ipezyGJ/status/2088322440397050258) > @omarsar0 Kinda absurd that a paper about replicating papers doesn't include code/artifacts so it can be easily replicated itself 😞 > — [combin8or on x](https://x.com/combin8or/status/2088328154091802833) > @omarsar0 how does the rubric judge handle papers where the original results dont replicate? > — [bountyAIhunter on x](https://x.com/bountyAIhunter/status/2088326812266823965) **Source threads** - [x](https://x.com/rohanpaul_ai/status/2088405028784779504) · 0 points · 8 comments - [x](https://x.com/omarsar0/status/2088309745740591429) · 0 points · 11 comments - [x](https://x.com/arankomatsuzaki/status/2088414262557593628) · 0 points · 0 comments - [x](https://x.com/omarsar0/status/2088491445544161701) · 0 points · 0 comments ## Similar posts on daily.dev - [Inherent, founded by DeepMind alumni, says its AI ‘teammate’ just outperformed Anthropic and OpenAI at replicating research](https://daily.dev/posts/inherent-founded-by-deepmind-alumni-says-its-ai-teammate-just-outperformed-anthropic-and-openai--m1b6c4qaa) · TechCrunch · 0 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents), [#claude](https://daily.dev/tags/claude), [#gpt](https://daily.dev/tags/gpt) [View this post on daily.dev](https://daily.dev/posts/a-27b-model-learned-to-direct-research-instead-of-just-coding-it-i1bup4itl) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"A 27B Model Learned to Direct Research Instead of Just Coding It","url":"https://daily.dev/posts/a-27b-model-learned-to-direct-research-instead-of-just-coding-it-i1bup4itl","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/a-27b-model-learned-to-direct-research-instead-of-just-coding-it-i1bup4itl"},"datePublished":"2026-08-14T23:19:33.556Z","dateModified":"2026-08-15T19:58:56.638Z","description":"A newly discussed paper introduces Replica, a scalable task space for training research agents by deleting a result figure from a published paper and requiring...","image":"https://pbs.twimg.com/media/HPuBLuVaoAAMhYY.png","thumbnailUrl":"https://pbs.twimg.com/media/HPuBLuVaoAAMhYY.png","isAccessibleForFree":true,"articleSection":"Collections","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Collections","logo":"https://media.daily.dev/image/upload/s--fk_6ycEi--/f_auto,q_auto/v1780996001/logos/collections?_a=BAMAMiWQ0","url":"https://daily.dev/sources/collections"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/a-27b-model-learned-to-direct-research-instead-of-just-coding-it-i1bup4itl","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,llm,ai-agents,claude,gpt","timeRequired":"PT3M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Collections","item":"https://daily.dev/sources/collections"},{"@type":"ListItem","position":3,"name":"A 27B Model Learned to Direct Research Instead of Just Coding It"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/a-27b-model-learned-to-direct-research-instead-of-just-coding-it-i1bup4itl#faq","mainEntity":[{"@type":"Question","name":"What is the Faraday research agent and how does it outperform larger models like GPT-5.5?","acceptedAnswer":{"@type":"Answer","text":"Faraday is a 27B parameter agent trained to direct scientific research rather than write code itself; it delegates coding to GPT-5.5 while deciding what experiments to run, when to simplify, and whether results look believable. On 68 unseen AI-for-science tasks it scored 0.791, beating Claude Opus 4.8 (0.748) and GPT-5.5 (0.729), winning outright on 60% of tasks. daily.dev surfaces emerging agent architectures like this for engineers tracking how AI research tools evolve."}},{"@type":"Question","name":"What is the Replica task space used for training AI research agents?","acceptedAnswer":{"@type":"Answer","text":"Replica is a scalable task space that trains research agents by taking a published paper, deleting one of its results figures, and requiring the agent to recreate that result by actually running the experiment. This forces the agent to fill in gaps papers leave out, such as what to implement, what to simplify, and which experiments matter, closely mirroring real replication work. An automated rubric judge grades attempts and tracks well with human judgment. follow daily.dev for developments in benchmarks that test genuine reasoning over pattern matching in AI systems."}}]} ```