--- title: "Agents on Rails: lemans goes open source" url: https://daily.dev/posts/agents-on-rails-lemans-goes-open-source-5oh4876px source_url: https://rubyonrails.org/2026/8/24/agents-on-rails-lemans type: article source: "Rails" published: 2026-08-24T18:09:16.263Z updated: 2026-08-24T18:09:41.942Z tags: ["ai", "open-source", "ai-agents", "ruby", "rails"] reading_time: 8 upvotes: 1 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Agents on Rails: lemans goes open source **[Rails](https://daily.dev/sources/rails)** · 8 min read · 1 upvotes · 0 comments ## Summary Rails has open-sourced lemans, the Ruby-based evaluation harness behind its Agents on Rails benchmark, along with a companion coding agent called miniswen (a Ruby port of mini-SWE-agent). The post walks through installing and running lemans against Daytona or Docker sandboxes, reporting results, and its anti-cheating guardrails (restoring graded test/bin files from snapshots, locking network access during evaluation, and redacting provider credentials from trajectory logs). It also shares scores from four newly benchmarked models: Sonnet 5 (44/63, the weakest Anthropic result so far), Terra (49/63, now the fastest model at 182s median), an open-weight Qwen 3.8-27B (48/63 but slow and poor Rails API recall), and an unnamed stealth model 'ox-alpha' (52/63). ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Is the lemans benchmarking harness used for Agents on Rails open source? Yes, lemans, the Ruby harness behind the Agents on Rails benchmark scores, has been open sourced by the Rails team. It is installable as a gem, supports Daytona and Docker sandbox backends, integrates with any LLM provider through RubyLLM, and includes a companion coding agent called miniswen, a Ruby port of mini-SWE-agent. _Track new Ruby AI tooling releases like this on daily.dev as the agentic coding ecosystem evolves._ ### How well does the Qwen 3.8-27B open-weight model perform on the Agents on Rails coding benchmark? Qwen 3.8-27B scores 48 out of 63 on the Agents on Rails benchmark, landing in the middle of the pack despite being small enough to run locally. However it takes a median of 27 minutes per run, 1.7x slower than the next-slowest model, and has the worst Rails API recall in the field at 7.9%. _Compare local versus cloud LLM tradeoffs for coding tasks by following benchmark updates on daily.dev._ ### How does Anthropic's Sonnet 5 model compare to Opus 4.8 on Rails coding tasks? Sonnet 5 scores 44 out of 63, the weakest Anthropic result recorded on the Agents on Rails benchmark, versus Opus 4.8's stronger showing that stretches the family's range from 70% to 92%. Sonnet 5 actually reaches for the correct Rails API more often than Opus 4.8 (25.4% versus 15.9% recall) but still finishes six runs behind at twice the runtime. _Weigh model tradeoffs like these before picking an AI coding assistant, tracked on daily.dev._ ## Similar posts on daily.dev - [Agents on Rails: the first benchmark report](https://daily.dev/posts/agents-on-rails-the-first-benchmark-report-1bsnzp6hz) · RUBYLAND · 2 upvotes · 2 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#open-source](https://daily.dev/tags/open-source), [#ai-agents](https://daily.dev/tags/ai-agents), [#ruby](https://daily.dev/tags/ruby), [#rails](https://daily.dev/tags/rails) [View this post on daily.dev](https://daily.dev/posts/agents-on-rails-lemans-goes-open-source-5oh4876px)