--- title: Evaluate agent performance | daily.dev description: Google's Data Cloud Frontier AI team introduces Discovery Bench, a meta-benchmarking framework that uses information-theoretic surprisal to generate calibrated... canonical: https://daily.dev/posts/evaluate-agent-performance-znjiurbqt twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Evaluate agent performance | daily.dev og:description: Google's Data Cloud Frontier AI team introduces Discovery Bench, a meta-benchmarking framework that uses information-theoretic surprisal to generate calibrated... og:url: https://daily.dev/posts/evaluate-agent-performance-znjiurbqt og:image: https://api.daily.dev/og/posts/ZNjiuRbqt.png og:image:alt: Evaluate agent performance og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Evaluate agent performance **[Google Cloud](https://daily.dev/sources/gcp)** · 11 min read · 1 upvotes · 0 comments ## Summary Google's Data Cloud Frontier AI team introduces Discovery Bench, a meta-benchmarking framework that uses information-theoretic surprisal to generate calibrated 'easy' and 'hard' query variations for evaluating data discovery agents. Rather than a single pass/fail score, the framework sweeps ambiguity levels to map exactly where agent performance degrades. Key findings include: agents can score F1=1.00 at neutral phrasing but 0.00 at high ambiguity (a 'cliff' invisible to static benchmarks), and there is often a non-monotonic 'sweet spot' where medium specificity outperforms maximum specificity. The team also discovered that a widely-used benchmark (kramabench-astronomy) contained broken ground truth, and that LLM-generated difficulty sweeps diverged wildly from TF-IDF-grounded ones — underscoring the need to evaluate the evaluators themselves. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is Discovery Bench and how does it evaluate AI data retrieval agents differently from a standard benchmark? Discovery Bench is a meta-benchmark from Google's Data Cloud frontier AI team that generates easy and hard variations of each evaluation case using information-theoretic surprisal, rather than relying on a single fixed-phrasing test. It measures a query's surprisal, the uncertainty about the correct dataset given the query's wording, then sweeps ambiguity levels to reveal exact points where an agent's retrieval accuracy collapses, something a pass/fail score cannot show. _Anyone building agent evaluation pipelines can follow ongoing framework research like this through daily.dev._ ### How much can removing a single distinguishing keyword hurt an AI retrieval agent's accuracy? In one tested case, an F1-optimized retrieval agent running on Gemini 3.1 Pro scored a perfect F1 of 1.00 at neutral query phrasing but dropped to F1 of 0.00 once the distinguishing token 'TLE' was removed from the query, causing it to lose the correct table entirely. Across the full ambiguity sweep on the KramaBench satellite dataset, F1 ranged from 0.34 at high ambiguity up to 0.81 at medium ambiguity. _Developers tuning retrieval-agent robustness can track findings like this on daily.dev to spot failure modes early._ ### Why did LLM-generated and TF-IDF-grounded ambiguity sweeps produce different agent evaluation scores? A pure-LLM-generated ambiguity sweep scored an agent at F1 ≈ 0.34 at high ambiguity, while a sweep grounded in TF-IDF surprisal scored the same agent at F1 ≈ 0.85 under the same conditions, a huge discrepancy. The TF-IDF-grounded sweep is considered more reliable because surprisal gives it a quantitative footing that a free-running LLM's term choices lack, showing that the benchmark generator itself needs evaluation. _Teams choosing evaluation tooling for AI agents can compare approaches like this through daily.dev._ ## Similar posts on daily.dev - [Measuring What Matters with Jules](https://daily.dev/posts/measuring-what-matters-with-jules-bwyrmpwn7) · Google Developers · 1 upvotes · 0 comments - [The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents](https://daily.dev/posts/the-agentic-analytics-benchmark-measuring-model-accuracy-and-efficiency-in-analytical-agents-81a3conc5) · ClickHouse · 0 upvotes · 0 comments - [Center for Responsible, Decentralized Intelligence at Berkeley](https://daily.dev/posts/center-for-responsible-decentralized-intelligence-at-berkeley-lkuvgfu7q) · Hacker News · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents) [View this post on daily.dev](https://daily.dev/posts/evaluate-agent-performance-znjiurbqt) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Evaluate agent performance","url":"https://daily.dev/posts/evaluate-agent-performance-znjiurbqt","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/evaluate-agent-performance-znjiurbqt"},"datePublished":"2026-07-10T16:12:29.696Z","dateModified":"2026-09-14T08:01:02.752Z","description":"Google's Data Cloud Frontier AI team introduces Discovery Bench, a meta-benchmarking framework that uses information-theoretic surprisal to generate calibrated...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/9df795f90d3290a29c3b5ccad1c3e312?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/9df795f90d3290a29c3b5ccad1c3e312?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Google Cloud","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Google Cloud","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/gcp","url":"https://daily.dev/sources/gcp"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/evaluate-agent-performance-znjiurbqt","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,ai-agents","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Google Cloud","item":"https://daily.dev/sources/gcp"},{"@type":"ListItem","position":3,"name":"Evaluate agent performance"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/evaluate-agent-performance-znjiurbqt#faq","mainEntity":[{"@type":"Question","name":"What is Discovery Bench and how does it evaluate AI data retrieval agents differently from a standard benchmark?","acceptedAnswer":{"@type":"Answer","text":"Discovery Bench is a meta-benchmark from Google's Data Cloud frontier AI team that generates easy and hard variations of each evaluation case using information-theoretic surprisal, rather than relying on a single fixed-phrasing test. It measures a query's surprisal, the uncertainty about the correct dataset given the query's wording, then sweeps ambiguity levels to reveal exact points where an agent's retrieval accuracy collapses, something a pass/fail score cannot show. Anyone building agent evaluation pipelines can follow ongoing framework research like this through daily.dev."}},{"@type":"Question","name":"How much can removing a single distinguishing keyword hurt an AI retrieval agent's accuracy?","acceptedAnswer":{"@type":"Answer","text":"In one tested case, an F1-optimized retrieval agent running on Gemini 3.1 Pro scored a perfect F1 of 1.00 at neutral query phrasing but dropped to F1 of 0.00 once the distinguishing token 'TLE' was removed from the query, causing it to lose the correct table entirely. Across the full ambiguity sweep on the KramaBench satellite dataset, F1 ranged from 0.34 at high ambiguity up to 0.81 at medium ambiguity. Developers tuning retrieval-agent robustness can track findings like this on daily.dev to spot failure modes early."}},{"@type":"Question","name":"Why did LLM-generated and TF-IDF-grounded ambiguity sweeps produce different agent evaluation scores?","acceptedAnswer":{"@type":"Answer","text":"A pure-LLM-generated ambiguity sweep scored an agent at F1 ≈ 0.34 at high ambiguity, while a sweep grounded in TF-IDF surprisal scored the same agent at F1 ≈ 0.85 under the same conditions, a huge discrepancy. The TF-IDF-grounded sweep is considered more reliable because surprisal gives it a quantitative footing that a free-running LLM's term choices lack, showing that the benchmark generator itself needs evaluation. Teams choosing evaluation tooling for AI agents can compare approaches like this through daily.dev."}}]} ```