ORCA-bench is a new benchmark that tests AI agents on production-fidelity oncall root cause analysis using a live OpenTelemetry Astronomy Shop (19 microservices, 13 languages, ~50 GB of telemetry over 6 days). The key finding: frontier agents (Opus 4.7, Sonnet 4.6, GPT-5.5, GLM-5, DeepSeek-V4-Pro) achieve only 25.3% RCA accuracy on medium tasks and 10.0% on hard tasks. Crucially, 'hard' doesn't mean deeper reasoning — it means the incident report is vague ('users are reporting site issues') rather than naming the symptom. Removing pre-scoping costs 19–50 percentage points of accuracy. A secondary finding: agents lock onto the loudest symptom and stop searching, a failure class distinct from hallucination. Removing source-code access drops accuracy 9–16 points, yet agents spend only 16–20% of commands reading code versus 70–72% on telemetry queries. The author argues most agent benchmarks measure the easy rung by handing models pre-scoped problems, and proposes five additions to agent evals: report specificity as an axis, counting simultaneous true answers, scoring 'true but irrelevant' separately from hallucination, including control tasks where nothing is wrong, and tracking which evidence source the agent queried versus which moved the score.
Table of contents
Getting distracted is a different failure from being wrongThe evidence they need is the evidence they skipWhat I would measure insteadQuestions this post answers
What RCA accuracy do frontier LLM agents achieve on hard oncall tasks in the ORCA-bench benchmark?
Frontier agents achieve only 10.0% RCA accuracy on hard oncall tasks in ORCA-bench, and 25.3% on medium tasks. Hard tasks are defined by vague incident reports ('users are reporting site issues') rather than deeper reasoning requirements. The same telemetry and evidence is available across all difficulty levels — only the pre-scoping in the report changes, costing 19–50 percentage points of accuracy depending on the model. Engineers evaluating AI agents for production ops track findings like these on daily.dev before committing to a toolchain.
How does removing source code access affect AI agent root cause analysis accuracy?
Removing source code access drops RCA accuracy by 9 to 16 percentage points across every model tested in ORCA-bench. Despite this, agents spend only 16–20% of their commands reading source code versus 70–72% on telemetry queries. Incident-time accuracy is barely affected by removing source code (82.6% to 79.9% for Opus 4.7), confirming the ablation degrades only what it should. Teams building oncall agents can follow the evidence on where agents underinvest their query budget on daily.dev.
What is the difference between a hallucinating AI agent and one that gives a true but irrelevant answer in incident diagnosis?
A hallucinating agent states something false, which can be caught by any rubric that knows what is false. An agent giving a true-but-irrelevant answer finds real evidence, constructs a plausible diagnosis, and stops looking — missing the actual incident. Catching this second failure requires a rubric that knows the correct answer, not just what is wrong. ORCA-bench measures hallucination rates ranging from 7% (DeepSeek-V4-Pro) to 40% (GLM-5), but the true-but-irrelevant failure largely goes unmeasured. Developers designing agent eval rubrics for production systems find the sharpest debates on daily.dev.