--- title: Enterprise AI benchmarks are broken | daily.dev description: DevRev has open-sourced an Enterprise AI Agent Benchmark that tests AI agents on messy, permission-controlled data at scale rather than exotic reasoning tasks.... canonical: https://daily.dev/posts/enterprise-ai-benchmarks-are-broken-os7hvrdov twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Enterprise AI benchmarks are broken | daily.dev og:description: DevRev has open-sourced an Enterprise AI Agent Benchmark that tests AI agents on messy, permission-controlled data at scale rather than exotic reasoning tasks.... og:url: https://daily.dev/posts/enterprise-ai-benchmarks-are-broken-os7hvrdov og:image: https://api.daily.dev/og/posts/Os7hVrdOv.png og:image:alt: Enterprise AI benchmarks are broken og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Enterprise AI benchmarks are broken **[The New Stack](https://daily.dev/sources/newstack)** · 9 min read · 0 upvotes · 0 comments ## Summary DevRev has open-sourced an Enterprise AI Agent Benchmark that tests AI agents on messy, permission-controlled data at scale rather than exotic reasoning tasks. The benchmark uses a scale-invariant dataset modeled on a mid-size software company at four sizes (1x to 64x), scoring agents on precision, efficiency, and safety. DevRev's own agent, Computer, outperformed Claude Code by 22–35 accuracy points and used 3.4x fewer tokens on identical tasks using the same Opus model family — attributing the gap to data-retrieval architecture rather than model capability. The benchmark follows an L1–L4 autonomy framework borrowed from self-driving levels, with only L1 and L2 currently released. Critics may note that DevRev designed the benchmark around the exact problem its own architecture solves, though the dataset, harness, and traces are fully open-sourced for independent verification. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Center for Responsible, Decentralized Intelligence at Berkeley](https://daily.dev/posts/center-for-responsible-decentralized-intelligence-at-berkeley-lkuvgfu7q) · Hacker News · 0 upvotes · 0 comments - [The Open Agent Leaderboard](https://daily.dev/posts/the-open-agent-leaderboard-vs2qmkwoy) · Hugging Face · 0 upvotes · 0 comments - [Introducing o11y-bench: an open benchmark for AI agents running observability workflows](https://daily.dev/posts/introducing-o11y-bench-an-open-benchmark-for-ai-agents-running-observability-workflows-psucsa908) · Grafana Labs · 20 upvotes · 0 comments - [Claude did best on a new benchmark for agents that build agents. It still passed fewer than a quarter of the tests.](https://daily.dev/posts/claude-did-best-on-a-new-benchmark-for-agents-that-build-agents-it-still-passed-fewer-than-a-quarte-rrx7ttpl4) · The New Stack · 2 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#ai-agents](https://daily.dev/tags/ai-agents), [#mcp](https://daily.dev/tags/mcp) [View this post on daily.dev](https://daily.dev/posts/enterprise-ai-benchmarks-are-broken-os7hvrdov) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Enterprise AI benchmarks are broken","url":"https://daily.dev/posts/enterprise-ai-benchmarks-are-broken-os7hvrdov","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/enterprise-ai-benchmarks-are-broken-os7hvrdov"},"datePublished":"2026-07-09T17:56:52.674Z","dateModified":"2026-07-09T18:56:23.879Z","description":"DevRev has open-sourced an Enterprise AI Agent Benchmark that tests AI agents on messy, permission-controlled data at scale rather than exotic reasoning tasks....","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/69f02d631b34ea25dba147ebaf705168?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/69f02d631b34ea25dba147ebaf705168?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"The New Stack","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"The New Stack","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/newstack","url":"https://daily.dev/sources/newstack"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/enterprise-ai-benchmarks-are-broken-os7hvrdov","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,ai-agents,mcp","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"The New Stack","item":"https://daily.dev/sources/newstack"},{"@type":"ListItem","position":3,"name":"Enterprise AI benchmarks are broken"}]} ```