Ora built a platform to benchmark AI agents like Claude Code, ChatGPT, Gemini, Hermes, OpenClaw, and Vercel's eve framework against live customer websites, measuring where agents fail during signup, integration, and payment flows. Everything runs on Vercel, including separate agent runtimes for each harness since none share infrastructure expectations. When Vercel launched eve, Ora benchmarked it under the same conditions as competitors, finding eve took 7% fewer steps, achieved 2x native success rate, and found 9% more valid endpoints compared to Claude Code running the same models. A prompt-caching fix after benchmark feedback cut eve's cost by roughly 15%. Ora subsequently rebuilt its own agent infrastructure on eve, citing eve's Next.js-like configuration and a sandbox override feature that let Ora's tracing instrumentation work inside eve's execution environment. The 16-person engineering team ships hundreds of commits daily, largely through coding agents managing Vercel infrastructure directly.
Table of contents
Benchmarking every major agent side by sideOra benchmarked eve, then built on itHow 16 engineers ship hundreds of commits a dayWhat's nextQuestions this post answers
How does Vercel's eve agent framework compare to Claude Code in real-world benchmarks?
In head-to-head testing across hundreds of real journeys using the same underlying models (Claude Fable 5 and Haiku 4.5), eve completed tasks with 7% fewer steps, achieved twice the native success rate on the target site instead of falling back to web search, and found 9% more valid, callable endpoints than Claude Code. daily.dev surfaces comparisons like this for teams choosing between agent frameworks.
What is a sandbox override in an AI agent framework like eve, and why would I need one?
A sandbox override lets developers swap out an agent framework's default isolated execution environment for a custom one. eve ships with its own sandbox for safe execution by default, but Ora needed agents to run inside its own instrumented tracing environment instead, and the override let it swap that in without building anything new. Developers wiring custom tracing into agent frameworks track these integration details on daily.dev.
What percentage of websites are ready for AI agents to sign up, integrate, and pay for products?
Roughly 99% of the web is not agent-ready, according to Ora's own measurements from running agents against live customer sites to complete signup, integration, and payment tasks. This estimate held both at the company's founding and after continued testing across many customer domains. Teams preparing sites for agentic traffic follow findings like this on daily.dev.