<!-- mobian-agent-page publisher="dailydev" canonical="https://daily.dev/posts/agentic-sql-benchmarks-sdyyxkqqj" -->

---
title: Agentic SQL Benchmarks | daily.dev
description: Agentic SQL benchmarks have evolved from simple text-to-SQL matching to complex, multi-step agentic tasks, but they remain deeply flawed. Spider (2018) was...
canonical: https://daily.dev/posts/agentic-sql-benchmarks-sdyyxkqqj
twitter:card: summary_large_image
twitter:site: @dailydotdev
og:type: website
og:site_name: daily.dev
og:title: Agentic SQL Benchmarks | daily.dev
og:description: Agentic SQL benchmarks have evolved from simple text-to-SQL matching to complex, multi-step agentic tasks, but they remain deeply flawed. Spider (2018) was...
og:url: https://daily.dev/posts/agentic-sql-benchmarks-sdyyxkqqj
og:image: https://api.daily.dev/og/posts/sDYYxKqqj.png
og:image:alt: Agentic SQL Benchmarks
og:image:width: 1200
og:image:height: 630
og:locale: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Agentic SQL Benchmarks

**[DoltHub Blog](https://daily.dev/sources/dolthub-blog)** · 14 min read · 1 upvotes · 0 comments

## Summary

Agentic SQL benchmarks have evolved from simple text-to-SQL matching to complex, multi-step agentic tasks, but they remain deeply flawed. Spider (2018) was solved within years, with scores rising from 12.4% to over 80%. Spider 2.0, BIRD-Interact, LiveSQLBench, DataAgentBench, and ELT-Bench test harder agentic workflows, yet most saturate fast: DataAgentBench jumped from 57% to 99% in seven months. A 2026 paper found labeling error rates of 52.8% and 62.8% in BIRD Mini-Dev and Spider 2.0-Snow. Claude Sonnet 4.5 scores 23.8% alone on LiveSQLBench but 35.2% wrapped in the OpenHands framework, which makes it difficult to judge a model on its own merits.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://dolthub.com/blog/2026-11-19-agentic-sql-benchmarks>

## Questions this post answers

### Why do SQL benchmark leaderboards for LLM agents become meaningless so quickly?

Benchmarks saturate fast because agents improve on the specific benchmark rather than the underlying skill. DataAgentBench rose from 57% to 99% in just seven months, and Spider 2.0-Snow's top score reached 96.7%. Once scores approach the ceiling, differences between models become statistically meaningless and no longer reflect real-world SQL task performance.

_Anyone picking an AI coding agent for database work can track benchmark shakeups like this on daily.dev._

### How much does the agent harness matter versus the underlying LLM on SQL benchmarks?

The harness can matter as much as the model itself. Claude Sonnet 4.5 scores 23.8% alone on LiveSQLBench but jumps to 35.2% when wrapped in the OpenHands Agent Framework. Top leaderboard spots are usually held by branded agent products like 'Tianqiong Data Agent + GLM 5.2' rather than a bare model, making it hard to judge a model's standalone capability.

_Developers comparing AI coding agents for SQL work can follow harness-versus-model findings like this on daily.dev._

### How reliable are the labels in text-to-SQL benchmarks like BIRD and Spider 2.0?

They contain substantial labeling errors. A 2026 paper found annotation error rates of 52.8% in BIRD Mini-Dev and 62.8% in Spider 2.0-Snow. After correcting labels and rescoring 16 agents, leaderboard rankings shuffled by up to nine places, undermining confidence in published model comparisons on these benchmarks.

_Teams relying on SQL benchmark leaderboards to choose AI tools can track label-accuracy issues like this on daily.dev._

## Similar posts on daily.dev

- [The Agentic Analytics Benchmark: Measuring model accuracy and efficiency in analytical agents](https://daily.dev/posts/the-agentic-analytics-benchmark-measuring-model-accuracy-and-efficiency-in-analytical-agents-81a3conc5) · ClickHouse · 0 upvotes · 0 comments
- [\#1 on Spider 2.0–DBT Benchmark – How Databao Agent Did It](https://daily.dev/posts/1-on-spider-2-0-dbt-benchmark-how-databao-agent-did-it-ttdfli3h1) · JetBrains · 1 upvotes · 0 comments
- [If You Think You Can Do Real-World Text-to-SQL – Communications of the ACM](https://daily.dev/posts/if-you-think-you-can-do-real-world-text-to-sql-communications-of-the-acm-ewtjyorai) · Hacker News · 0 upvotes · 0 comments
- [How agentic software development will change databases](https://daily.dev/posts/how-agentic-software-development-will-change-databases-pmw8vpzgt) · databricks · 100 upvotes · 0 comments

---

Tags: [#ai](https://daily.dev/tags/ai), [#ai-agents](https://daily.dev/tags/ai-agents), [#llm](https://daily.dev/tags/llm), [#backend](https://daily.dev/tags/backend), [#sql](https://daily.dev/tags/sql)

[View this post on daily.dev](https://daily.dev/posts/agentic-sql-benchmarks-sdyyxkqqj)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]}
{"@context":"https://schema.org","@type":"TechArticle","headline":"Agentic SQL Benchmarks","url":"https://daily.dev/posts/agentic-sql-benchmarks-sdyyxkqqj","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/agentic-sql-benchmarks-sdyyxkqqj"},"datePublished":"2026-11-20T01:44:51.333Z","dateModified":"2026-11-20T01:45:23.256Z","description":"Agentic SQL benchmarks have evolved from simple text-to-SQL matching to complex, multi-step agentic tasks, but they remain deeply flawed. Spider (2018) was...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5df3aad9f7a830b0502830c2e3912678?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5df3aad9f7a830b0502830c2e3912678?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"DoltHub Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"DoltHub Blog","logo":"https://media.daily.dev/image/upload/s--LXDlvjS4--/f_auto,q_auto/v1780213587/logos/dolthub-blog?_a=BAMAMiWQ0","url":"https://daily.dev/sources/dolthub-blog"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/agentic-sql-benchmarks-sdyyxkqqj","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,ai-agents,llm,backend,sql","timeRequired":"PT14M"}
{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"DoltHub Blog","item":"https://daily.dev/sources/dolthub-blog"},{"@type":"ListItem","position":3,"name":"Agentic SQL Benchmarks"}]}
{"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/agentic-sql-benchmarks-sdyyxkqqj#faq","mainEntity":[{"@type":"Question","name":"Why do SQL benchmark leaderboards for LLM agents become meaningless so quickly?","acceptedAnswer":{"@type":"Answer","text":"Benchmarks saturate fast because agents improve on the specific benchmark rather than the underlying skill. DataAgentBench rose from 57% to 99% in just seven months, and Spider 2.0-Snow's top score reached 96.7%. Once scores approach the ceiling, differences between models become statistically meaningless and no longer reflect real-world SQL task performance. Anyone picking an AI coding agent for database work can track benchmark shakeups like this on daily.dev."}},{"@type":"Question","name":"How much does the agent harness matter versus the underlying LLM on SQL benchmarks?","acceptedAnswer":{"@type":"Answer","text":"The harness can matter as much as the model itself. Claude Sonnet 4.5 scores 23.8% alone on LiveSQLBench but jumps to 35.2% when wrapped in the OpenHands Agent Framework. Top leaderboard spots are usually held by branded agent products like 'Tianqiong Data Agent + GLM 5.2' rather than a bare model, making it hard to judge a model's standalone capability. Developers comparing AI coding agents for SQL work can follow harness-versus-model findings like this on daily.dev."}},{"@type":"Question","name":"How reliable are the labels in text-to-SQL benchmarks like BIRD and Spider 2.0?","acceptedAnswer":{"@type":"Answer","text":"They contain substantial labeling errors. A 2026 paper found annotation error rates of 52.8% in BIRD Mini-Dev and 62.8% in Spider 2.0-Snow. After correcting labels and rescoring 16 agents, leaderboard rankings shuffled by up to nine places, undermining confidence in published model comparisons on these benchmarks. Teams relying on SQL benchmark leaderboards to choose AI tools can track label-accuracy issues like this on daily.dev."}}]}
```

