An engineering deep-dive describes how a large AI model company built a Kafka-to-Databend Cloud pipeline to ingest trillion-scale Agent Trace data using the open-source bend-ingest-kafka tool. It covers extending Kafka partition parallelism into Databend writes with multiple Workers, using Raw Mode to capture full JSON payloads without upfront schema parsing, batching writes into compressed NDJSON files to avoid per-message inserts, delaying Kafka offset commits until data is durably written via COPY INTO, and using Databend Streams with MERGE INTO for incremental deduplication and transformation downstream. An end-to-end test with 100,000 messages validated the multi-file COPY behavior, achieving about 15,208 rows per second on a single Worker.
Table of contents
When Agent Trace Becomes the Data Layer for EvalsWhy Agent Trace Ingestion Is HardExtending Kafka Parallelism into Databend WritesRaw Mode: Capture the Full Event First, Model LaterBatch File Writes: Do Not Turn Every Message into a Warehouse InsertOffset Commit: Wait Until Data Is Really in the Raw TableRemoving Duplicate Effects with Stream and MERGE INTOVerifying Multi-File COPY with an End-to-End TestTrillion Scale Does Not Come from One Large MachineConclusionQuestions this post answers
How do you avoid losing data when committing Kafka offsets in a high-throughput ingestion pipeline?
Commit Kafka offsets only after the corresponding batch of files is successfully written into the destination table via COPY INTO, not right after uploading files to a staging area. Each worker keeps uploaded stage files and their Kafka batches in memory, retries COPY with the same files on failure, and only advances offsets once the whole group lands, giving at-least-once delivery with deduplication handled later via MERGE INTO. daily.dev surfaces engineering writeups like this for teams hardening their own Kafka ingestion pipelines.
Why should raw JSON payloads be stored before parsing them in a Kafka ingestion pipeline for AI agent traces?
Storing the raw JSON payload plus Kafka metadata (topic, partition, offset) in a raw table lets ingestion stay decoupled from frequently changing trace schemas, since a new model or agent framework version can add fields like reasoning tokens or cache-hit flags without requiring a consumer redeploy. This lowers CPU use during ingestion and lets teams reprocess raw data later when parsing rules change. Developers wrestling with schema drift in event pipelines can find similar patterns on daily.dev.
How much throughput can a single Kafka-to-warehouse ingestion worker achieve with batched multi-file COPY INTO writes?
In an end-to-end test using bend-ingest-kafka with 100,000 messages and a batch size of 1,000 records per file, one worker generated 100 files, executed only 20 COPY INTO operations (one per 5 files), and completed the full cycle of consumption, compression, upload, and offset commit in about 6.58 seconds, or roughly 15,208 rows per second. daily.dev helps engineers comparing batching strategies for warehouse ingestion track real numbers like this.