--- title: From Million-Token Agents to Eval-Ready Data | daily.dev description: A case study describes how a foundation model company built an Agent Trace data pipeline on Databend Cloud to handle terabyte-per-hour ingestion of execution... canonical: https://daily.dev/posts/from-million-token-agents-to-eval-ready-data-3cnglnplw twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: From Million-Token Agents to Eval-Ready Data | daily.dev og:description: A case study describes how a foundation model company built an Agent Trace data pipeline on Databend Cloud to handle terabyte-per-hour ingestion of execution... og:url: https://daily.dev/posts/from-million-token-agents-to-eval-ready-data-3cnglnplw og:image: https://api.daily.dev/og/posts/3CNgLNPLW.png og:image:alt: From Million-Token Agents to Eval-Ready Data og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # From Million-Token Agents to Eval-Ready Data **[Databend Blog](https://daily.dev/sources/databend)** · 6 min read · 0 upvotes · 0 comments ## Summary A case study describes how a foundation model company built an Agent Trace data pipeline on Databend Cloud to handle terabyte-per-hour ingestion of execution traces from a trillion-parameter reasoning model. The architecture separates raw Kafka ingestion (kept as flexible VARIANT/JSON) from incrementally processed, query-optimized trace tables using Databend's Stream and Task primitives. Independent compute warehouses isolate ingestion, transformation, and analytics workloads, while masking policies govern access to sensitive fields. The company reportedly moved the pipeline to production within days, shortening the feedback loop between agent execution, debugging, and Eval-driven model iteration. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [From Kafka to Databend Cloud: Engineering a Trillion-Scale Agent Trace Ingestion Pipeline](https://daily.dev/posts/from-kafka-to-databend-cloud-engineering-a-trillion-scale-agent-trace-ingestion-pipeline-4qtnulddc) · Databend Blog · 1 upvotes · 0 comments - [Data Engineering in the AI Era - Taming Fragmented Agent Trace Pipelines with a Unified Lakehouse](https://daily.dev/posts/data-engineering-in-the-ai-era---taming-fragmented-agent-trace-pipelines-with-a-unified-lakehouse-62xenewdp) · Databend Blog · 0 upvotes · 0 comments - [Real-Time AI Pipelines: Building a Kafka \+ Kubernetes \+ Databricks Stack for Agentic AI Systems](https://daily.dev/posts/real-time-ai-pipelines-building-a-kafka-kubernetes-databricks-stack-for-agentic-ai-systems-qoytpduhq) · Medium · 0 upvotes · 0 comments - [How LangChain Built an Agent-First Data Stack](https://daily.dev/posts/how-langchain-built-an-agent-first-data-stack-8ofcbfd88) · LangChain · 0 upvotes · 0 comments - [How I built agent-based security reviews on Databricks](https://daily.dev/posts/how-i-built-agent-based-security-reviews-on-databricks-rpfx5muvy) · databricks · 3 upvotes · 0 comments --- Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#backend](https://daily.dev/tags/backend), [#kafka](https://daily.dev/tags/kafka), [#llm-observability](https://daily.dev/tags/llm-observability) [View this post on daily.dev](https://daily.dev/posts/from-million-token-agents-to-eval-ready-data-3cnglnplw) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"From Million-Token Agents to Eval-Ready Data","url":"https://daily.dev/posts/from-million-token-agents-to-eval-ready-data-3cnglnplw","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/from-million-token-agents-to-eval-ready-data-3cnglnplw"},"datePublished":"2026-08-16T08:54:01.016Z","dateModified":"2026-08-16T08:59:12.279Z","description":"A case study describes how a foundation model company built an Agent Trace data pipeline on Databend Cloud to handle terabyte-per-hour ingestion of execution...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4d913d61c0aef5ef5d6a033b0c87cea1?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4d913d61c0aef5ef5d6a033b0c87cea1?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Databend Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Databend Blog","logo":"https://media.daily.dev/image/upload/s--Vkqa36Ma--/f_auto,q_auto/v1786870423/logos/databend?_a=BAMAMicg0","url":"https://daily.dev/sources/databend"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/from-million-token-agents-to-eval-ready-data-3cnglnplw","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-agents,backend,kafka,llm-observability","timeRequired":"PT6M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Databend Blog","item":"https://daily.dev/sources/databend"},{"@type":"ListItem","position":3,"name":"From Million-Token Agents to Eval-Ready Data"}]} ```