--- title: The Agentic Analytics Benchmark: ranking tables and... description: TypeSafe's Jev classifier labels failed analytics-agent runs and ranks warehouse tables for questions in about a second per call. On a four-model development... canonical: https://daily.dev/posts/the-agentic-analytics-benchmark-ranking-tables-and-labeling-failures-with-jev-rbhbpkq6t twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: The Agentic Analytics Benchmark: ranking tables and labeling failures with Jev | daily.dev og:description: TypeSafe's Jev classifier labels failed analytics-agent runs and ranks warehouse tables for questions in about a second per call. On a four-model development... og:url: https://daily.dev/posts/the-agentic-analytics-benchmark-ranking-tables-and-labeling-failures-with-jev-rbhbpkq6t og:image: https://api.daily.dev/og/posts/RbHBpkq6t.png og:image:alt: The Agentic Analytics Benchmark: ranking tables and labeling failures with Jev og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # The Agentic Analytics Benchmark: ranking tables and labeling failures with Jev **[ClickHouse](https://daily.dev/sources/clickhouse)** · 18 min read · 2 upvotes · 0 comments ## Summary TypeSafe's Jev classifier labels failed analytics-agent runs and ranks warehouse tables for questions in about a second per call. On a four-model development run of 277 failures, wrong data selection was the largest failure bucket, mostly entity resolution errors like wrong identifiers rather than wrong tables, and Jev flagged 18 answers the grader likely scored incorrectly. Its table ranking hit nDCG@10 of 0.951 and recall@10 of 0.991 in 0.4 seconds with no fine-tuning. But feeding that ranking back to the agent as an in-loop hint left accuracy flat across 20 models, and a turn-count saving seen on the development set vanished on the held-out 201-question board. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Does giving an AI agent a hint about which database table to use improve text-to-SQL accuracy? No, feeding a table-ranking hint back to the agent mid-run left accuracy flat across 20 models. Tested on a 136-question development set and a held-out 201-question board at a 60-turn budget, the pooled net effect was +12 recoveries minus regressions with a confidence interval spanning zero. Frontier models already find the right tables; their failures come from misusing the data once found, not from retrieval. _Anyone weighing agentic SQL tools against this evidence can track benchmark follow-ups on daily.dev._ ### How accurate can an LLM-based classifier be at ranking which database tables are relevant to a natural language question? A classifier called Jev reached a mean nDCG@10 of 0.951 and recall@10 of 0.991 on an 18-table warehouse, with 84% of questions having every needed table in the top five, all in about 0.4 seconds per question and with no fine-tuning or gold data at run time. On wider public catalogs of 75 to 710 tables it still reached 87% recall@10 in about 3.4 seconds. _Teams scoping retrieval components for analytics agents can compare numbers like these on daily.dev._ ### What are the most common ways AI agents fail at generating SQL queries from natural language questions? The largest failure bucket is wrong data selection, mostly entity resolution errors such as using the wrong identifier for an organization (29 cases) or never resolving a named entity before querying (20 cases), outweighing wrong table (28) or wrong column (10) choices. This breakdown came from labeling 277 failures across 544 attempts from four models: Claude Sonnet 4.6, GPT-4.1, Qwen3-Coder 30B, and Gemini 2.5 Flash. _Developers debugging agent SQL failures can follow benchmark breakdowns like this on daily.dev._ ## Similar posts on daily.dev - [The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents](https://daily.dev/posts/the-agentic-analytics-benchmark-measuring-model-accuracy-and-efficiency-in-analytical-agents-81a3conc5) · ClickHouse · 0 upvotes · 0 comments - [We need to talk about JEV \(in data pipelines\)](https://daily.dev/posts/we-need-to-talk-about-jev-in-data-pipelines--nr914jn81) · Medium · 44 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#llm](https://daily.dev/tags/llm), [#rag](https://daily.dev/tags/rag), [#agentic-ai](https://daily.dev/tags/agentic-ai), [#clickhouse](https://daily.dev/tags/clickhouse) [View this post on daily.dev](https://daily.dev/posts/the-agentic-analytics-benchmark-ranking-tables-and-labeling-failures-with-jev-rbhbpkq6t) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"The Agentic Analytics Benchmark: ranking tables and labeling failures with Jev","url":"https://daily.dev/posts/the-agentic-analytics-benchmark-ranking-tables-and-labeling-failures-with-jev-rbhbpkq6t","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/the-agentic-analytics-benchmark-ranking-tables-and-labeling-failures-with-jev-rbhbpkq6t"},"datePublished":"2026-10-27T17:01:33.496Z","dateModified":"2026-10-27T17:06:34.089Z","description":"TypeSafe's Jev classifier labels failed analytics-agent runs and ranks warehouse tables for questions in about a second per call. On a four-model development...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/a7baf48085c36522187596f77f236a3a?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/a7baf48085c36522187596f77f236a3a?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"ClickHouse","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"ClickHouse","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/clickhouse","url":"https://daily.dev/sources/clickhouse"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/the-agentic-analytics-benchmark-ranking-tables-and-labeling-failures-with-jev-rbhbpkq6t","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,llm,rag,agentic-ai,clickhouse","timeRequired":"PT18M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"ClickHouse","item":"https://daily.dev/sources/clickhouse"},{"@type":"ListItem","position":3,"name":"The Agentic Analytics Benchmark: ranking tables and labeling failures with Jev"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/the-agentic-analytics-benchmark-ranking-tables-and-labeling-failures-with-jev-rbhbpkq6t#faq","mainEntity":[{"@type":"Question","name":"Does giving an AI agent a hint about which database table to use improve text-to-SQL accuracy?","acceptedAnswer":{"@type":"Answer","text":"No, feeding a table-ranking hint back to the agent mid-run left accuracy flat across 20 models. Tested on a 136-question development set and a held-out 201-question board at a 60-turn budget, the pooled net effect was +12 recoveries minus regressions with a confidence interval spanning zero. Frontier models already find the right tables; their failures come from misusing the data once found, not from retrieval. Anyone weighing agentic SQL tools against this evidence can track benchmark follow-ups on daily.dev."}},{"@type":"Question","name":"How accurate can an LLM-based classifier be at ranking which database tables are relevant to a natural language question?","acceptedAnswer":{"@type":"Answer","text":"A classifier called Jev reached a mean nDCG@10 of 0.951 and recall@10 of 0.991 on an 18-table warehouse, with 84% of questions having every needed table in the top five, all in about 0.4 seconds per question and with no fine-tuning or gold data at run time. On wider public catalogs of 75 to 710 tables it still reached 87% recall@10 in about 3.4 seconds. Teams scoping retrieval components for analytics agents can compare numbers like these on daily.dev."}},{"@type":"Question","name":"What are the most common ways AI agents fail at generating SQL queries from natural language questions?","acceptedAnswer":{"@type":"Answer","text":"The largest failure bucket is wrong data selection, mostly entity resolution errors such as using the wrong identifier for an organization (29 cases) or never resolving a named entity before querying (20 cases), outweighing wrong table (28) or wrong column (10) choices. This breakdown came from labeling 277 failures across 544 attempts from four models: Claude Sonnet 4.6, GPT-4.1, Qwen3-Coder 30B, and Gemini 2.5 Flash. Developers debugging agent SQL failures can follow benchmark breakdowns like this on daily.dev."}}]} ```