--- title: The Agentic Analytics Benchmark: Measuring model... description: ClickHouse built and open-sourced data-agent-mnist, a benchmark harness for evaluating LLM-powered analytics agents against real production data warehouse... canonical: https://daily.dev/posts/the-agentic-analytics-benchmark-measuring-model-accuracy-and-efficiency-in-analytical-agents-81a3conc5 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents | daily.dev og:description: ClickHouse built and open-sourced data-agent-mnist, a benchmark harness for evaluating LLM-powered analytics agents against real production data warehouse... og:url: https://daily.dev/posts/the-agentic-analytics-benchmark-measuring-model-accuracy-and-efficiency-in-analytical-agents-81a3conc5 og:image: https://api.daily.dev/og/posts/81A3CONc5.png og:image:alt: The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents **[ClickHouse](https://daily.dev/sources/clickhouse)** · 20 min read · 0 upvotes · 0 comments ## Summary ClickHouse built and open-sourced data-agent-mnist, a benchmark harness for evaluating LLM-powered analytics agents against real production data warehouse traffic rather than static text-to-SQL datasets. They tested 28 models across proprietary and open-weight tiers on 201 real analytics questions, scoring correctness via a provider-diverse ground-truth committee and an LLM-as-jury panel that excludes a candidate's own model family. Claude Fable 5.1 leads on correctness at 76.6%, but DeepSeek V4 Flash achieves 65.7% at 52x lower cost ($1 vs $52 for the full run) and faster speed. Failure analysis shows 'wrong plan' (not SQL syntax or joins) is the dominant failure mode across nearly all models, suggesting split-role architectures (strong model for planning, cheap model for execution) as a promising future direction. Contamination testing confirms the benchmark's private dataset avoids memorization issues that plague older public benchmarks like Spider. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Which model scored highest on ClickHouse's agentic analytics benchmark and how much does it cost to run? Claude Fable 5.1 topped the pass rate at 76.6% across 201 real analytics questions, but running the full benchmark with it costs $52, compared to just $1 with DeepSeek V4 Flash, which scores 65.7% (11 percentage points lower). DeepSeek V4 Pro placed second overall, only 2 points behind the leader, at over 75% lower cost. _Weigh accuracy against cost tradeoffs like this before picking a model for your analytics agent, a topic developers track on daily.dev._ ### What is the most common way LLM agents fail at agentic analytics tasks? Wrong planning (labeled failure mode FM2) accounts for 53 to 82% of failures across every model with a meaningful failure count, far outweighing wrong data selection, implementation errors, or runtime errors. Modern models rarely produce SQL that fails to execute, showing that syntax and execution are no longer the bottleneck; planning and schema discovery are. _Understanding where agents actually break helps teams building analytics agents architect around the real bottleneck, a theme developers follow on daily.dev._ ### How do you build an unbiased benchmark for LLM agents without a validated ground-truth answer? Three models from three different providers (Claude Opus 4.8, GPT-5.5, Gemini 2.5 Pro) independently solve each question, and if at least two agree, that result becomes ground truth; questions with three different answers are dropped. Scoring then uses a three-seat jury panel, one seat per major provider, excluding any seat matching the candidate's own model family, with majority vote deciding. _Anyone designing evaluation pipelines for AI agents can compare methodologies like this via daily.dev._ ## Similar posts on daily.dev - [Choosing the Right Model for Your Inference Use Case](https://daily.dev/posts/choosing-the-right-model-for-your-inference-use-case-7usyrqlyu) · DigitalOcean Community · 2 upvotes · 0 comments - [The Open Agent Leaderboard](https://daily.dev/posts/the-open-agent-leaderboard-vs2qmkwoy) · Hugging Face · 0 upvotes · 0 comments - [Evaluate agent performance](https://daily.dev/posts/evaluate-agent-performance-znjiurbqt) · Google Cloud · 1 upvotes · 0 comments - [Introducing o11y-bench: an open benchmark for AI agents running observability workflows](https://daily.dev/posts/introducing-o11y-bench-an-open-benchmark-for-ai-agents-running-observability-workflows-psucsa908) · Grafana Labs · 20 upvotes · 0 comments - [ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks — by Artificial Analysis and IBM](https://daily.dev/posts/itbench-aa-frontier-models-score-below-50-on-the-first-benchmark-for-agentic-enterprise-it-tasks--inm3wceim) · Hugging Face · 1 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#ai-agents](https://daily.dev/tags/ai-agents), [#clickhouse](https://daily.dev/tags/clickhouse), [#data-warehouse](https://daily.dev/tags/data-warehouse) [View this post on daily.dev](https://daily.dev/posts/the-agentic-analytics-benchmark-measuring-model-accuracy-and-efficiency-in-analytical-agents-81a3conc5) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents","url":"https://daily.dev/posts/the-agentic-analytics-benchmark-measuring-model-accuracy-and-efficiency-in-analytical-agents-81a3conc5","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/the-agentic-analytics-benchmark-measuring-model-accuracy-and-efficiency-in-analytical-agents-81a3conc5"},"datePublished":"2026-09-01T18:13:24.033Z","dateModified":"2026-09-14T06:08:41.436Z","description":"ClickHouse built and open-sourced data-agent-mnist, a benchmark harness for evaluating LLM-powered analytics agents against real production data warehouse...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/0aec63025a98c86e6091adfcb0b5f150?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/0aec63025a98c86e6091adfcb0b5f150?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"ClickHouse","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"ClickHouse","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/clickhouse","url":"https://daily.dev/sources/clickhouse"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/the-agentic-analytics-benchmark-measuring-model-accuracy-and-efficiency-in-analytical-agents-81a3conc5","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,ai-agents,clickhouse,data-warehouse","timeRequired":"PT20M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"ClickHouse","item":"https://daily.dev/sources/clickhouse"},{"@type":"ListItem","position":3,"name":"The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/the-agentic-analytics-benchmark-measuring-model-accuracy-and-efficiency-in-analytical-agents-81a3conc5#faq","mainEntity":[{"@type":"Question","name":"Which model scored highest on ClickHouse's agentic analytics benchmark and how much does it cost to run?","acceptedAnswer":{"@type":"Answer","text":"Claude Fable 5.1 topped the pass rate at 76.6% across 201 real analytics questions, but running the full benchmark with it costs $52, compared to just $1 with DeepSeek V4 Flash, which scores 65.7% (11 percentage points lower). DeepSeek V4 Pro placed second overall, only 2 points behind the leader, at over 75% lower cost. Weigh accuracy against cost tradeoffs like this before picking a model for your analytics agent, a topic developers track on daily.dev."}},{"@type":"Question","name":"What is the most common way LLM agents fail at agentic analytics tasks?","acceptedAnswer":{"@type":"Answer","text":"Wrong planning (labeled failure mode FM2) accounts for 53 to 82% of failures across every model with a meaningful failure count, far outweighing wrong data selection, implementation errors, or runtime errors. Modern models rarely produce SQL that fails to execute, showing that syntax and execution are no longer the bottleneck; planning and schema discovery are. Understanding where agents actually break helps teams building analytics agents architect around the real bottleneck, a theme developers follow on daily.dev."}},{"@type":"Question","name":"How do you build an unbiased benchmark for LLM agents without a validated ground-truth answer?","acceptedAnswer":{"@type":"Answer","text":"Three models from three different providers (Claude Opus 4.8, GPT-5.5, Gemini 2.5 Pro) independently solve each question, and if at least two agree, that result becomes ground truth; questions with three different answers are dropped. Scoring then uses a three-seat jury panel, one seat per major provider, excluding any seat matching the candidate's own model family, with majority vote deciding. Anyone designing evaluation pipelines for AI agents can compare methodologies like this via daily.dev."}}]} ```