--- title: Agents on Rails: The LLM Benchmark Project | daily.dev description: The Rails Foundation launched 'Agents on Rails,' an ongoing benchmark initiative measuring how well leading agentic coding tools perform on Ruby on Rails... canonical: https://daily.dev/posts/agents-on-rails-the-llm-benchmark-project-oyyfzyygr twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Agents on Rails: The LLM Benchmark Project | daily.dev og:description: The Rails Foundation launched 'Agents on Rails,' an ongoing benchmark initiative measuring how well leading agentic coding tools perform on Ruby on Rails... og:url: https://daily.dev/posts/agents-on-rails-the-llm-benchmark-project-oyyfzyygr og:image: https://api.daily.dev/og/posts/oyyFzyyGr.png og:image:alt: Agents on Rails: The LLM Benchmark Project og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Agents on Rails: The LLM Benchmark Project **[RUBYLAND](https://daily.dev/sources/rubyla)** · 5 min read · 0 upvotes · 0 comments ## Summary The Rails Foundation launched 'Agents on Rails,' an ongoing benchmark initiative measuring how well leading agentic coding tools perform on Ruby on Rails codebases. Commissioned from Evil Martians, Stage 1 tests 8 frontier and open-weight models on atomic, self-contained tasks, evaluating accuracy, speed, token spend, cost, and whether models use current Rails APIs. Results and methodology are published on a new leaderboard and AI page. Stage 2, coming soon, will test more realistic multi-step work like building apps from scratch. The benchmark tasks, methodology, and eventually the harness (lemans) are being open-sourced, with the benchmark set to re-run as new models are released. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is the Agents on Rails benchmark and what does it measure? Agents on Rails is an ongoing initiative from the Rails Foundation, built with Evil Martians, that benchmarks agentic coding tools on Ruby on Rails codebases. Stage 1 tests 8 frontier and open-weight models on small, self-contained tasks, measuring accuracy, speed, token spend, cost, and whether models use current Rails APIs. Results appear on a public leaderboard. _Teams choosing an AI coding agent for Rails work can track benchmark updates like this on daily.dev._ ### Is the Rails AI agent benchmark harness open source? The tasks and methodology behind the Rails agent benchmark are already public in a GitHub repository, with raw benchmark runs uploaded shortly after release. The Ruby test harness used to run the evaluations, called lemans and built by Evil Martians, will also be open-sourced for the community soon. _Developers tracking open-source AI benchmarking tools can follow releases like lemans on daily.dev._ ### What will Stage 2 of the Rails AI agent benchmark test? Stage 2 of the Agents on Rails benchmark moves beyond isolated atomic tasks to more realistic, complex work, including longer multi-step tasks, adding features to existing apps, and building a Rails app from scratch, more closely mimicking real production development workflows. _Engineers evaluating coding agents for real-world Rails projects can follow benchmark stages like this on daily.dev._ ## Similar posts on daily.dev - [Agents on Rails: the first benchmark report](https://daily.dev/posts/agents-on-rails-the-first-benchmark-report-1bsnzp6hz) · RUBYLAND · 2 upvotes · 2 comments - [Agents on Rails: lemans goes open source](https://daily.dev/posts/agents-on-rails-lemans-goes-open-source-5oh4876px) · Rails · 1 upvotes · 0 comments - [Home](https://daily.dev/posts/home-l9pbnpdgr) · Ruby Flow · 0 upvotes · 0 comments - [Agents on Rails: Stage 2. Can a model ship a feature?](https://daily.dev/posts/agents-on-rails-stage-2-can-a-model-ship-a-feature--yfvchzqdl) · Rails · 2 upvotes · 1 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#open-source](https://daily.dev/tags/open-source), [#ai-agents](https://daily.dev/tags/ai-agents), [#ruby](https://daily.dev/tags/ruby), [#rails](https://daily.dev/tags/rails) [View this post on daily.dev](https://daily.dev/posts/agents-on-rails-the-llm-benchmark-project-oyyfzyygr) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Agents on Rails: The LLM Benchmark Project","url":"https://daily.dev/posts/agents-on-rails-the-llm-benchmark-project-oyyfzyygr","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/agents-on-rails-the-llm-benchmark-project-oyyfzyygr"},"datePublished":"2026-08-13T16:57:25.385Z","dateModified":"2026-08-17T19:37:23.895Z","description":"The Rails Foundation launched 'Agents on Rails,' an ongoing benchmark initiative measuring how well leading agentic coding tools perform on Ruby on Rails...","image":"https://rubyonrails.org/assets/images/opengraph.png","thumbnailUrl":"https://rubyonrails.org/assets/images/opengraph.png","isAccessibleForFree":true,"articleSection":"RUBYLAND","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"RUBYLAND","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/rubyla","url":"https://daily.dev/sources/rubyla"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/agents-on-rails-the-llm-benchmark-project-oyyfzyygr","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,open-source,ai-agents,ruby,rails","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"RUBYLAND","item":"https://daily.dev/sources/rubyla"},{"@type":"ListItem","position":3,"name":"Agents on Rails: The LLM Benchmark Project"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/agents-on-rails-the-llm-benchmark-project-oyyfzyygr#faq","mainEntity":[{"@type":"Question","name":"What is the Agents on Rails benchmark and what does it measure?","acceptedAnswer":{"@type":"Answer","text":"Agents on Rails is an ongoing initiative from the Rails Foundation, built with Evil Martians, that benchmarks agentic coding tools on Ruby on Rails codebases. Stage 1 tests 8 frontier and open-weight models on small, self-contained tasks, measuring accuracy, speed, token spend, cost, and whether models use current Rails APIs. Results appear on a public leaderboard. Teams choosing an AI coding agent for Rails work can track benchmark updates like this on daily.dev."}},{"@type":"Question","name":"Is the Rails AI agent benchmark harness open source?","acceptedAnswer":{"@type":"Answer","text":"The tasks and methodology behind the Rails agent benchmark are already public in a GitHub repository, with raw benchmark runs uploaded shortly after release. The Ruby test harness used to run the evaluations, called lemans and built by Evil Martians, will also be open-sourced for the community soon. Developers tracking open-source AI benchmarking tools can follow releases like lemans on daily.dev."}},{"@type":"Question","name":"What will Stage 2 of the Rails AI agent benchmark test?","acceptedAnswer":{"@type":"Answer","text":"Stage 2 of the Agents on Rails benchmark moves beyond isolated atomic tasks to more realistic, complex work, including longer multi-step tasks, adding features to existing apps, and building a Rails app from scratch, more closely mimicking real production development workflows. Engineers evaluating coding agents for real-world Rails projects can follow benchmark stages like this on daily.dev."}}]} ```