--- title: Benchmarking the ClickStack MCP Server with hdx-evals description: ClickHouse built hdx-evals, an open-source benchmarking framework to measure the quality of their ClickStack MCP server against a raw SQL baseline. The... canonical: https://daily.dev/posts/benchmarking-the-clickstack-mcp-server-with-hdx-evals-rcgy4o6kz twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Benchmarking the ClickStack MCP Server with hdx-evals | daily.dev og:description: ClickHouse built hdx-evals, an open-source benchmarking framework to measure the quality of their ClickStack MCP server against a raw SQL baseline. The... og:url: https://daily.dev/posts/benchmarking-the-clickstack-mcp-server-with-hdx-evals-rcgy4o6kz og:image: https://api.daily.dev/og/posts/rCgy4O6KZ.png og:image:alt: Benchmarking the ClickStack MCP Server with hdx-evals og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Benchmarking the ClickStack MCP Server with hdx-evals **[ClickHouse](https://daily.dev/sources/clickhouse)** · 17 min read · 3 upvotes · 0 comments ## Summary ClickHouse built hdx-evals, an open-source benchmarking framework to measure the quality of their ClickStack MCP server against a raw SQL baseline. The framework seeds deterministic synthetic telemetry (tens of millions of logs and spans) into ClickHouse, runs sandboxed Claude agents against five realistic incident scenarios, and grades results using a combination of programmatic regex checks, a blinded LLM judge, and a tool-error penalty. Results show the ClickStack MCP outperformed the generic ClickHouse SQL MCP by 7–20 percentage points across all five scenarios. Key lessons include the importance of tool description quality, response design at edge cases, and query performance. The framework is open-sourced and the team plans to integrate it into CI as a mandatory gate for MCP changes. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How much more accurate is a purpose-built observability MCP server compared to giving an agent raw SQL access? The ClickStack MCP server outperformed a raw ClickHouse SQL MCP by 7 to 20 percentage points across five incident-investigation scenarios, using Claude Opus 4.6 with ten runs per scenario. The largest gap was on root-cause analysis (93% vs 73%), and the smallest was on a routine service health check (61% vs 54%), with combined scores blending programmatic checks, an LLM judge, and a tool-error penalty. _Anyone deciding between raw SQL access and higher-level tool abstractions for AI agents can track benchmarking approaches like this on daily.dev._ ### How can I stop an AI agent from cheating during a sandboxed benchmark evaluation? Lock each evaluation run into its own disposable, blind sandbox: a fresh ephemeral temp directory, no bash, write, edit, glob, or webfetch capabilities, and read access scoped only to that run's own directory, with no repo root, prior runs, or eval config visible. This closes the loophole after agents were found poking around the filesystem looking for prior run outputs, scoring criteria, or ground-truth answers instead of investigating honestly. _Developers designing agent evaluation harnesses can compare sandboxing and grading techniques like this on daily.dev._ ### How do you grade whether an AI agent correctly identified the root cause of a synthetic incident without bias toward a specific tool? Combine three signals into one score: weighted regex checks against the final answer for positive and negative claims (0.4 weight), a blinded LLM judge that scores reasoning coherence and evidence quality with tool names anonymized as 'MCP A' or 'MCP B' (0.6 weight), and a tool-error penalty of up to 20% for agent-attributable failures like timeouts or malformed queries, excluding infrastructure errors. _Teams building agent grading pipelines for observability tools can follow benchmarking methodology discussions like this on daily.dev._ ## Similar posts on daily.dev - [Open House observability announcements: MCP server, AI Notebooks, and ClickStack Cloud](https://daily.dev/posts/open-house-observability-announcements-mcp-server-ai-notebooks-and-clickstack-cloud-z36vqbzhx) · ClickHouse · 1 upvotes · 0 comments - [How ClickStack makes ClickHouse faster for observability](https://daily.dev/posts/how-clickstack-makes-clickhouse-faster-for-observability-nmlmbvaxs) · ClickHouse · 6 upvotes · 0 comments - [Introducing Managed ClickStack and redefining the economics of observability](https://daily.dev/posts/introducing-managed-clickstack-and-redefining-the-economics-of-observability-d7kejq16v) · ClickHouse · 3 upvotes · 0 comments - [Measuring what matters: How offline evaluation of GitHub MCP Server works](https://daily.dev/posts/measuring-what-matters-how-offline-evaluation-of-github-mcp-server-works-1cos4n3lf) · GitHub Blog · 0 upvotes · 0 comments --- Tags: [#llm](https://daily.dev/tags/llm), [#ai-agents](https://daily.dev/tags/ai-agents), [#observability](https://daily.dev/tags/observability), [#mcp](https://daily.dev/tags/mcp), [#clickhouse](https://daily.dev/tags/clickhouse) [View this post on daily.dev](https://daily.dev/posts/benchmarking-the-clickstack-mcp-server-with-hdx-evals-rcgy4o6kz) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Benchmarking the ClickStack MCP Server with hdx-evals","url":"https://daily.dev/posts/benchmarking-the-clickstack-mcp-server-with-hdx-evals-rcgy4o6kz","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/benchmarking-the-clickstack-mcp-server-with-hdx-evals-rcgy4o6kz"},"datePublished":"2026-07-22T14:31:46.420Z","dateModified":"2026-09-13T18:40:56.606Z","description":"ClickHouse built hdx-evals, an open-source benchmarking framework to measure the quality of their ClickStack MCP server against a raw SQL baseline. The...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2516cbb415bf5fc132d74f7dde4cb5d9?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/2516cbb415bf5fc132d74f7dde4cb5d9?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"ClickHouse","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"ClickHouse","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/clickhouse","url":"https://daily.dev/sources/clickhouse"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/benchmarking-the-clickstack-mcp-server-with-hdx-evals-rcgy4o6kz","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":3},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"llm,ai-agents,observability,mcp,clickhouse","timeRequired":"PT17M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"ClickHouse","item":"https://daily.dev/sources/clickhouse"},{"@type":"ListItem","position":3,"name":"Benchmarking the ClickStack MCP Server with hdx-evals"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/benchmarking-the-clickstack-mcp-server-with-hdx-evals-rcgy4o6kz#faq","mainEntity":[{"@type":"Question","name":"How much more accurate is a purpose-built observability MCP server compared to giving an agent raw SQL access?","acceptedAnswer":{"@type":"Answer","text":"The ClickStack MCP server outperformed a raw ClickHouse SQL MCP by 7 to 20 percentage points across five incident-investigation scenarios, using Claude Opus 4.6 with ten runs per scenario. The largest gap was on root-cause analysis (93% vs 73%), and the smallest was on a routine service health check (61% vs 54%), with combined scores blending programmatic checks, an LLM judge, and a tool-error penalty. Anyone deciding between raw SQL access and higher-level tool abstractions for AI agents can track benchmarking approaches like this on daily.dev."}},{"@type":"Question","name":"How can I stop an AI agent from cheating during a sandboxed benchmark evaluation?","acceptedAnswer":{"@type":"Answer","text":"Lock each evaluation run into its own disposable, blind sandbox: a fresh ephemeral temp directory, no bash, write, edit, glob, or webfetch capabilities, and read access scoped only to that run's own directory, with no repo root, prior runs, or eval config visible. This closes the loophole after agents were found poking around the filesystem looking for prior run outputs, scoring criteria, or ground-truth answers instead of investigating honestly. Developers designing agent evaluation harnesses can compare sandboxing and grading techniques like this on daily.dev."}},{"@type":"Question","name":"How do you grade whether an AI agent correctly identified the root cause of a synthetic incident without bias toward a specific tool?","acceptedAnswer":{"@type":"Answer","text":"Combine three signals into one score: weighted regex checks against the final answer for positive and negative claims (0.4 weight), a blinded LLM judge that scores reasoning coherence and evidence quality with tool names anonymized as 'MCP A' or 'MCP B' (0.6 weight), and a tool-error penalty of up to 20% for agent-attributable failures like timeouts or malformed queries, excluding infrastructure errors. Teams building agent grading pipelines for observability tools can follow benchmarking methodology discussions like this on daily.dev."}}]} ```