A case study describes how a foundation model company built an Agent Trace data pipeline on Databend Cloud to handle terabyte-per-hour ingestion of execution traces from a trillion-parameter reasoning model. The architecture separates raw Kafka ingestion (kept as flexible VARIANT/JSON) from incrementally processed, query-optimized trace tables using Databend's Stream and Task primitives. Independent compute warehouses isolate ingestion, transformation, and analytics workloads, while masking policies govern access to sensitive fields. The company reportedly moved the pipeline to production within days, shortening the feedback loop between agent execution, debugging, and Eval-driven model iteration.
Table of contents
When a single Agent run becomes a data-engineering workloadWhy Agent Traces need a data pipeline—not only observabilityArchitecture: from Kafka events to Eval-ready Trace tablesRunning ingestion, transformation, and analysis in productionWhy the team chose Databend CloudFrom Agent history to the next model iteration2 Impressions