--- title: "Agents on Rails: The LLM Benchmark Project" url: https://daily.dev/posts/agents-on-rails-the-llm-benchmark-project-zhwtv2f3p source_url: https://rubyonrails.org/2026/8/13/llm-benchmarking-project type: article source: "Rails" published: 2026-08-13T16:25:35.724Z updated: 2026-08-17T19:37:23.895Z tags: ["ai", "ai-agents", "ruby", "rails"] reading_time: 5 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Agents on Rails: The LLM Benchmark Project **[Rails](https://daily.dev/sources/rails)** · 5 min read · 0 upvotes · 0 comments ## Summary The Rails Foundation launched Agents on Rails, a new ongoing benchmark project measuring how well leading AI coding agents (both frontier and open-weight models) perform on Ruby on Rails codebases. Commissioned by Evil Martians, Stage 1 tests atomic, self-contained tasks across 8 models, evaluating accuracy, speed, token spend, cost, and whether models use current Rails APIs. Results and methodology are published on a new leaderboard and AI page, with raw data and the benchmark harness (called lemans) to be open-sourced. Stage 2 will test more realistic, multi-step development work and is coming soon. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is the Agents on Rails benchmark project and what does it measure? Agents on Rails is an ongoing initiative from the Rails Foundation, built by Evil Martians, that benchmarks how well AI coding agents perform on Ruby on Rails codebases. Stage 1 tests 8 frontier and open-weight models on small, self-contained tasks, measuring accuracy, speed, token spend, cost, and whether models use current Rails APIs. Results appear on a public leaderboard with accompanying findings reports. _Rails teams choosing between AI coding models can track these benchmark results as they evolve on daily.dev._ ### What is lemans in the context of Ruby AI benchmarking? Lemans is the Ruby test harness built by Evil Martians to run the Agents on Rails benchmark evaluations against AI coding agents. It executes the atomic coding tasks used in Stage 1 testing and will soon be open-sourced alongside the raw benchmark runs and full methodology already published in the rails/ai-evals GitHub repository. _Developers building their own AI coding evaluations can follow open-source tooling like lemans via daily.dev._ ## Similar posts on daily.dev - [Agents on Rails: the first benchmark report](https://daily.dev/posts/agents-on-rails-the-first-benchmark-report-1bsnzp6hz) · RUBYLAND · 2 upvotes · 2 comments - [Agents on Rails: lemans goes open source](https://daily.dev/posts/agents-on-rails-lemans-goes-open-source-5oh4876px) · Rails · 1 upvotes · 0 comments - [Home](https://daily.dev/posts/home-l9pbnpdgr) · Ruby Flow · 0 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#ai-agents](https://daily.dev/tags/ai-agents), [#ruby](https://daily.dev/tags/ruby), [#rails](https://daily.dev/tags/rails) [View this post on daily.dev](https://daily.dev/posts/agents-on-rails-the-llm-benchmark-project-zhwtv2f3p)