The Rails Foundation launched Agents on Rails, a new ongoing benchmark project measuring how well leading AI coding agents (both frontier and open-weight models) perform on Ruby on Rails codebases. Commissioned by Evil Martians, Stage 1 tests atomic, self-contained tasks across 8 models, evaluating accuracy, speed, token spend, cost, and whether models use current Rails APIs. Results and methodology are published on a new leaderboard and AI page, with raw data and the benchmark harness (called lemans) to be open-sourced. Stage 2 will test more realistic, multi-step development work and is coming soon.
Questions this post answers
What is the Agents on Rails benchmark project and what does it measure?
Agents on Rails is an ongoing initiative from the Rails Foundation, built by Evil Martians, that benchmarks how well AI coding agents perform on Ruby on Rails codebases. Stage 1 tests 8 frontier and open-weight models on small, self-contained tasks, measuring accuracy, speed, token spend, cost, and whether models use current Rails APIs. Results appear on a public leaderboard with accompanying findings reports. Rails teams choosing between AI coding models can track these benchmark results as they evolve on daily.dev.
What is lemans in the context of Ruby AI benchmarking?
Lemans is the Ruby test harness built by Evil Martians to run the Agents on Rails benchmark evaluations against AI coding agents. It executes the atomic coding tasks used in Stage 1 testing and will soon be open-sourced alongside the raw benchmark runs and full methodology already published in the rails/ai-evals GitHub repository. Developers building their own AI coding evaluations can follow open-source tooling like lemans via daily.dev.