--- title: Agents on Rails: lemans goes open source | daily.dev description: Rails has open-sourced lemans, the Ruby-based evaluation harness behind its Agents on Rails benchmark, along with a companion coding agent called miniswen (a... canonical: https://daily.dev/posts/agents-on-rails-lemans-goes-open-source-5oh4876px twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Agents on Rails: lemans goes open source | daily.dev og:description: Rails has open-sourced lemans, the Ruby-based evaluation harness behind its Agents on Rails benchmark, along with a companion coding agent called miniswen (a... og:url: https://daily.dev/posts/agents-on-rails-lemans-goes-open-source-5oh4876px og:image: https://api.daily.dev/og/posts/5oH4876px.png og:image:alt: Agents on Rails: lemans goes open source og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Agents on Rails: lemans goes open source **[Rails](https://daily.dev/sources/rails)** · 8 min read · 1 upvotes · 0 comments ## Summary Rails has open-sourced lemans, the Ruby-based evaluation harness behind its Agents on Rails benchmark, along with a companion coding agent called miniswen (a Ruby port of mini-SWE-agent). The post walks through installing and running lemans against Daytona or Docker sandboxes, reporting results, and its anti-cheating guardrails (restoring graded test/bin files from snapshots, locking network access during evaluation, and redacting provider credentials from trajectory logs). It also shares scores from four newly benchmarked models: Sonnet 5 (44/63, the weakest Anthropic result so far), Terra (49/63, now the fastest model at 182s median), an open-weight Qwen 3.8-27B (48/63 but slow and poor Rails API recall), and an unnamed stealth model 'ox-alpha' (52/63). ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is lemans and how do I use it to benchmark AI coding agents on Rails tasks? Lemans is a Ruby-native open source harness for evaluating LLM coding agents, released by the Rails Foundation as the tool behind the Agents on Rails leaderboard. Install it via `gem install lemans`, clone the ai-evals repo, set an LLM provider API key and a Daytona sandbox key, then run `lemans run --model / --task `. Results are stored per trial and summarized with `lemans report`. _Developers benchmarking coding agents against Rails codebases can follow ongoing coverage of tools like lemans on daily.dev._ ### How does the open-weight Qwen 3.8-27B model perform compared to cloud LLMs on Rails coding tasks? Qwen 3.8-27B scores 48 out of 63 on the Agents on Rails benchmark, placing it in the middle of the pack, but it takes a median of 27 minutes per run, 1.7x slower than the next-slowest model, and has the worst Rails API recall of any tested model at 7.9%. It requires raising evaluation timeouts from 30 to 60 minutes. _Teams weighing local open-weight models against cloud LLM subscriptions can track speed and accuracy trade-offs like these on daily.dev._ ### How does Anthropic's Sonnet 5 model score compared to Opus 4.8 on Rails coding benchmarks? Sonnet 5 scores 44 out of 63, the weakest Anthropic result recorded, versus Opus 4.8's stronger showing, stretching the Anthropic model family's range from 92% down to 70%. Sonnet 5 actually reaches for the correct Rails API more often than Opus 4.8 (25.4% versus 15.9% recall) yet still finishes six runs behind at twice the time. _Developers choosing between Anthropic model tiers for coding agents can compare benchmark trade-offs like these on daily.dev._ ## Similar posts on daily.dev - [Agents on Rails: the first benchmark report](https://daily.dev/posts/agents-on-rails-the-first-benchmark-report-1bsnzp6hz) · RUBYLAND · 2 upvotes · 2 comments - [MiniMax M2.7: The self-refactoring Agent architecture](https://daily.dev/posts/minimax-m2-7-the-self-refactoring-agent-architecture-srovepd7w) · Daily Dose of Data Science \| Avi Chawla \| Substack · 0 upvotes · 0 comments - [Home](https://daily.dev/posts/home-l9pbnpdgr) · Ruby Flow · 0 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#open-source](https://daily.dev/tags/open-source), [#ai-agents](https://daily.dev/tags/ai-agents), [#ruby](https://daily.dev/tags/ruby), [#rails](https://daily.dev/tags/rails) [View this post on daily.dev](https://daily.dev/posts/agents-on-rails-lemans-goes-open-source-5oh4876px) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Agents on Rails: lemans goes open source","url":"https://daily.dev/posts/agents-on-rails-lemans-goes-open-source-5oh4876px","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/agents-on-rails-lemans-goes-open-source-5oh4876px"},"datePublished":"2026-08-24T18:09:16.263Z","dateModified":"2026-09-13T21:24:57.910Z","description":"Rails has open-sourced lemans, the Ruby-based evaluation harness behind its Agents on Rails benchmark, along with a companion coding agent called miniswen (a...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/f3219107ad4885c4e776650b7cdd8928?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/f3219107ad4885c4e776650b7cdd8928?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Rails","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Rails","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/d44501fa43a2441a9e27a08367cdfa52","url":"https://daily.dev/sources/rails"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/agents-on-rails-lemans-goes-open-source-5oh4876px","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,open-source,ai-agents,ruby,rails","timeRequired":"PT8M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Rails","item":"https://daily.dev/sources/rails"},{"@type":"ListItem","position":3,"name":"Agents on Rails: lemans goes open source"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/agents-on-rails-lemans-goes-open-source-5oh4876px#faq","mainEntity":[{"@type":"Question","name":"What is lemans and how do I use it to benchmark AI coding agents on Rails tasks?","acceptedAnswer":{"@type":"Answer","text":"Lemans is a Ruby-native open source harness for evaluating LLM coding agents, released by the Rails Foundation as the tool behind the Agents on Rails leaderboard. Install it via `gem install lemans`, clone the ai-evals repo, set an LLM provider API key and a Daytona sandbox key, then run `lemans run --model / --task `. Results are stored per trial and summarized with `lemans report`. Developers benchmarking coding agents against Rails codebases can follow ongoing coverage of tools like lemans on daily.dev."}},{"@type":"Question","name":"How does the open-weight Qwen 3.8-27B model perform compared to cloud LLMs on Rails coding tasks?","acceptedAnswer":{"@type":"Answer","text":"Qwen 3.8-27B scores 48 out of 63 on the Agents on Rails benchmark, placing it in the middle of the pack, but it takes a median of 27 minutes per run, 1.7x slower than the next-slowest model, and has the worst Rails API recall of any tested model at 7.9%. It requires raising evaluation timeouts from 30 to 60 minutes. Teams weighing local open-weight models against cloud LLM subscriptions can track speed and accuracy trade-offs like these on daily.dev."}},{"@type":"Question","name":"How does Anthropic's Sonnet 5 model score compared to Opus 4.8 on Rails coding benchmarks?","acceptedAnswer":{"@type":"Answer","text":"Sonnet 5 scores 44 out of 63, the weakest Anthropic result recorded, versus Opus 4.8's stronger showing, stretching the Anthropic model family's range from 92% down to 70%. Sonnet 5 actually reaches for the correct Rails API more often than Opus 4.8 (25.4% versus 15.9% recall) yet still finishes six runs behind at twice the time. Developers choosing between Anthropic model tiers for coding agents can compare benchmark trade-offs like these on daily.dev."}}]} ```