--- title: Agents on Rails: the first benchmark report | daily.dev description: The Rails Foundation ran eight LLM agents against 21 atomic Rails tasks (bug reports, security findings, feature requests filed against the Writebook app),... canonical: https://daily.dev/posts/agents-on-rails-the-first-benchmark-report-1bsnzp6hz twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Agents on Rails: the first benchmark report | daily.dev og:description: The Rails Foundation ran eight LLM agents against 21 atomic Rails tasks (bug reports, security findings, feature requests filed against the Writebook app),... og:url: https://daily.dev/posts/agents-on-rails-the-first-benchmark-report-1bsnzp6hz og:image: https://api.daily.dev/og/posts/1bsnzp6HZ.png og:image:alt: Agents on Rails: the first benchmark report og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Agents on Rails: the first benchmark report **[RUBYLAND](https://daily.dev/sources/rubyla)** · 4 min read · 2 upvotes · 2 comments ## Summary The Rails Foundation ran eight LLM agents against 21 atomic Rails tasks (bug reports, security findings, feature requests filed against the Writebook app), three runs each. Claude Fable 5 would top the leaderboard at ~95% but failed every run of a pen-test-worded task, so it drops behind other models on the published ranking. Price barely predicts score past a point: Luna (91 cents for 63 runs) beats pricier models, while Sol and Opus buy modest gains at 36x and 132x the cost. Bumping Luna's reasoning effort to xhigh closed most of the frontier gap for $2.34 per corpus pass. Separately, models mostly write their own code instead of using Rails' built-in APIs even when available, and recall of the right API correlates with higher solve rates. The benchmark corpus, run harness (lemans), and full results are being open-sourced under the Rails GitHub. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Which AI model performed best on the Rails agent benchmark, and why isn't it ranked first? Claude Fable 5 scored the highest at roughly 95% on the Rails Foundation's 21-task agent benchmark, but it refused every one of the three runs on a task worded like a security or pen-test report, which knocked it out of the top ranked position despite its otherwise leading accuracy. This shows model refusals can matter as much as raw capability when picking an agent for security-adjacent work. _Teams picking a coding agent for security-sensitive tasks can track benchmark nuances like this via daily.dev._ ### Does increasing reasoning effort improve a cheap LLM's coding agent performance on Rails tasks? Yes, raising Luna's reasoning effort from default to high pushed its score from about 73% to around 86% on a 21-task Rails benchmark, replicated across four sweeps, costing $1.36 for a full corpus pass. Pushing to xhigh effort raised it further to 89% for $2.34, closing most of the gap to frontier models without changing the underlying model. _daily.dev helps developers tuning agent cost versus reasoning effort keep up with results like these._ ### Do AI coding agents use existing framework APIs or write custom code when solving Rails tasks? Mostly they write custom code instead of reaching for the relevant built-in Rails API, even when every task in a benchmark corpus turns on one specific API. Recall of the correct API ranged from 8% (DeepSeek) to 35% (Fable) across models, and runs that did recall the API solved the task 92% of the time versus 87% for hand-rolled solutions, while ignoring a recalled API entirely dropped success to 64%. _daily.dev surfaces findings like this for teams weighing how much they can trust agent-generated Rails code._ ## Community discussion Top comments from developers on daily.dev. **@rizzdev** · 0 upvotes > Finding the API then shipping around it is the fail **@agustinbarrientos** · 0 upvotes > Can the next report put those three pen-test refusals next to the scores? ## Similar posts on daily.dev - [Agents on Rails: Stage 2. Can a model ship a feature?](https://daily.dev/posts/agents-on-rails-stage-2-can-a-model-ship-a-feature--yfvchzqdl) · Rails · 2 upvotes · 1 comments - [Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase](https://daily.dev/posts/benchmarking-coding-agents-on-databricks-multi-million-line-codebase-qjzvv6o07) · databricks · 1 upvotes · 0 comments - [One Week of Building and Reviewing Code With LLM Agents](https://daily.dev/posts/one-week-of-building-and-reviewing-code-with-llm-agents-0bpygawo1) · RUBYLAND · 0 upvotes · 0 comments - [ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks — by Artificial Analysis and IBM](https://daily.dev/posts/itbench-aa-frontier-models-score-below-50-on-the-first-benchmark-for-agentic-enterprise-it-tasks--inm3wceim) · Hugging Face · 1 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#ai-agents](https://daily.dev/tags/ai-agents), [#ruby](https://daily.dev/tags/ruby), [#rails](https://daily.dev/tags/rails), [#claude](https://daily.dev/tags/claude) [View this post on daily.dev](https://daily.dev/posts/agents-on-rails-the-first-benchmark-report-1bsnzp6hz) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Agents on Rails: the first benchmark report","url":"https://daily.dev/posts/agents-on-rails-the-first-benchmark-report-1bsnzp6hz","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/agents-on-rails-the-first-benchmark-report-1bsnzp6hz"},"datePublished":"2026-08-13T16:57:25.356Z","dateModified":"2026-08-13T16:58:09.494Z","description":"The Rails Foundation ran eight LLM agents against 21 atomic Rails tasks (bug reports, security findings, feature requests filed against the Writebook app),...","image":"https://rubyonrails.org/assets/images/opengraph.png","thumbnailUrl":"https://rubyonrails.org/assets/images/opengraph.png","isAccessibleForFree":true,"articleSection":"RUBYLAND","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"RUBYLAND","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/rubyla","url":"https://daily.dev/sources/rubyla"},"commentCount":2,"discussionUrl":"https://daily.dev/posts/agents-on-rails-the-first-benchmark-report-1bsnzp6hz","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":2}],"keywords":"ai,ai-agents,ruby,rails,claude","timeRequired":"PT4M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"RUBYLAND","item":"https://daily.dev/sources/rubyla"},{"@type":"ListItem","position":3,"name":"Agents on Rails: the first benchmark report"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/agents-on-rails-the-first-benchmark-report-1bsnzp6hz","comment":[{"@type":"Comment","text":"Finding the API then shipping around it is the fail","datePublished":"2026-08-14T11:11:49.203Z","url":"https://daily.dev/posts/1bsnzp6HZ#c-xDmdEIOGD","author":{"@type":"Person","name":"Andrew","url":"https://daily.dev/rizzdev","image":"https://media.daily.dev/image/upload/s--35vRfXAA--/f_auto/v1785413727/avatars/avatar_fQ7ttCKtPuu6tDvcgQXv5?_a=BAMAMicg0"}},{"@type":"Comment","text":"Can the next report put those three pen-test refusals next to the scores?","datePublished":"2026-08-15T16:36:34.465Z","url":"https://daily.dev/posts/1bsnzp6HZ#c-97uOOgO77","author":{"@type":"Person","name":"Agustin Barrientos","url":"https://daily.dev/agustinbarrientos","image":"https://media.daily.dev/image/upload/s--5ayxQnqn--/f_auto/v1788281802/avatars/avatar_wQYYVe5Tbj0NJ7C7qPoa8?_a=BAMAMicg0"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/agents-on-rails-the-first-benchmark-report-1bsnzp6hz#faq","mainEntity":[{"@type":"Question","name":"Which AI model performed best on the Rails agent benchmark, and why isn't it ranked first?","acceptedAnswer":{"@type":"Answer","text":"Claude Fable 5 scored the highest at roughly 95% on the Rails Foundation's 21-task agent benchmark, but it refused every one of the three runs on a task worded like a security or pen-test report, which knocked it out of the top ranked position despite its otherwise leading accuracy. This shows model refusals can matter as much as raw capability when picking an agent for security-adjacent work. Teams picking a coding agent for security-sensitive tasks can track benchmark nuances like this via daily.dev."}},{"@type":"Question","name":"Does increasing reasoning effort improve a cheap LLM's coding agent performance on Rails tasks?","acceptedAnswer":{"@type":"Answer","text":"Yes, raising Luna's reasoning effort from default to high pushed its score from about 73% to around 86% on a 21-task Rails benchmark, replicated across four sweeps, costing $1.36 for a full corpus pass. Pushing to xhigh effort raised it further to 89% for $2.34, closing most of the gap to frontier models without changing the underlying model. daily.dev helps developers tuning agent cost versus reasoning effort keep up with results like these."}},{"@type":"Question","name":"Do AI coding agents use existing framework APIs or write custom code when solving Rails tasks?","acceptedAnswer":{"@type":"Answer","text":"Mostly they write custom code instead of reaching for the relevant built-in Rails API, even when every task in a benchmark corpus turns on one specific API. Recall of the correct API ranged from 8% (DeepSeek) to 35% (Fable) across models, and runs that did recall the API solved the task 92% of the time versus 87% for hand-rolled solutions, while ignoring a recalled API entirely dropped success to 64%. daily.dev surfaces findings like this for teams weighing how much they can trust agent-generated Rails code."}}]} ```