--- title: From Kafka to Databend Cloud: Engineering a... description: An engineering deep-dive describes how a large AI model company built a Kafka-to-Databend Cloud pipeline to ingest trillion-scale Agent Trace data using the... canonical: https://daily.dev/posts/from-kafka-to-databend-cloud-engineering-a-trillion-scale-agent-trace-ingestion-pipeline-4qtnulddc twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: From Kafka to Databend Cloud: Engineering a Trillion-Scale Agent Trace Ingestion Pipeline | daily.dev og:description: An engineering deep-dive describes how a large AI model company built a Kafka-to-Databend Cloud pipeline to ingest trillion-scale Agent Trace data using the... og:url: https://daily.dev/posts/from-kafka-to-databend-cloud-engineering-a-trillion-scale-agent-trace-ingestion-pipeline-4qtnulddc og:image: https://api.daily.dev/og/posts/4qTnuldDC.png og:image:alt: From Kafka to Databend Cloud: Engineering a Trillion-Scale Agent Trace Ingestion Pipeline og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # From Kafka to Databend Cloud: Engineering a Trillion-Scale Agent Trace Ingestion Pipeline **[Databend Blog](https://daily.dev/sources/databend)** · 10 min read · 1 upvotes · 0 comments ## Summary An engineering deep-dive describes how a large AI model company built a Kafka-to-Databend Cloud pipeline to ingest trillion-scale Agent Trace data using the open-source bend-ingest-kafka tool. It covers extending Kafka partition parallelism into Databend writes with multiple Workers, using Raw Mode to capture full JSON payloads without upfront schema parsing, batching writes into compressed NDJSON files to avoid per-message inserts, delaying Kafka offset commits until data is durably written via COPY INTO, and using Databend Streams with MERGE INTO for incremental deduplication and transformation downstream. An end-to-end test with 100,000 messages validated the multi-file COPY behavior, achieving about 15,208 rows per second on a single Worker. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How does bend-ingest-kafka avoid losing Kafka messages during a crash before data is written? It delays committing Kafka offsets until the corresponding batch of files has been successfully written into the Databend Raw Table via COPY INTO, not merely uploaded to Stage. Each worker keeps uploaded Stage files and their Kafka batch in memory, retries COPY with the same files on failure, and only commits offsets once the write succeeds, giving at-least-once delivery with downstream deduplication. _Engineers hardening Kafka pipelines against data loss can browse implementation patterns like this on daily.dev._ ### What is Raw Mode in bend-ingest-kafka and why use it for Agent Trace data? Raw Mode writes each Kafka message's full JSON payload plus topic, partition, and offset metadata into a Raw Table without parsing it into a fixed schema. This keeps the ingestion consumer schema-agnostic so upstream trace format changes, like new reasoning-token or tool-call fields, do not require redeploying consumers; parsing and modeling happen later using Databend Streams and MERGE INTO. _Teams designing schema-flexible ingestion for fast-changing event data can find similar patterns on daily.dev._ ### How does bend-ingest-kafka decide when to trigger a COPY INTO batch load? It uses OR-based thresholds: a COPY INTO runs immediately once 128 pending files accumulate, or after 5 seconds pass since the first uploaded file in the group, whichever comes first. In a 100,000-message end-to-end test with 1,000 messages per batch producing 100 files and one COPY per 5 files, this required only 20 COPY executions, reaching about 15,208 rows per second on one worker. _Data engineers tuning batch-load thresholds for warehouse ingestion can compare approaches like this on daily.dev._ ## Similar posts on daily.dev - [Data Engineering in the AI Era - Taming Fragmented Agent Trace Pipelines with a Unified Lakehouse](https://daily.dev/posts/data-engineering-in-the-ai-era---taming-fragmented-agent-trace-pipelines-with-a-unified-lakehouse-62xenewdp) · Databend Blog · 0 upvotes · 0 comments - [Real-Time AI Pipelines: Building a Kafka \+ Kubernetes \+ Databricks Stack for Agentic AI Systems](https://daily.dev/posts/real-time-ai-pipelines-building-a-kafka-kubernetes-databricks-stack-for-agentic-ai-systems-qoytpduhq) · Medium · 0 upvotes · 0 comments - [Why Batch ELT Breaks at Enterprise Scale](https://daily.dev/posts/why-batch-elt-breaks-at-enterprise-scale-2uoypymjg) · Confluent Blog · 0 upvotes · 0 comments - [Bringing you the Data Agent Kit](https://daily.dev/posts/bringing-you-the-data-agent-kit-2nwdfzwf3) · Google Cloud · 0 upvotes · 0 comments - [Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned](https://daily.dev/posts/building-service-topology-at-scale-architecture-challenges-and-lessons-learned-qptwmjvf7) · Netflix TechBlog · 26 upvotes · 0 comments --- Tags: [#data-engineering](https://daily.dev/tags/data-engineering), [#kafka](https://daily.dev/tags/kafka), [#data-warehouse](https://daily.dev/tags/data-warehouse) [View this post on daily.dev](https://daily.dev/posts/from-kafka-to-databend-cloud-engineering-a-trillion-scale-agent-trace-ingestion-pipeline-4qtnulddc) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"From Kafka to Databend Cloud: Engineering a Trillion-Scale Agent Trace Ingestion Pipeline","url":"https://daily.dev/posts/from-kafka-to-databend-cloud-engineering-a-trillion-scale-agent-trace-ingestion-pipeline-4qtnulddc","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/from-kafka-to-databend-cloud-engineering-a-trillion-scale-agent-trace-ingestion-pipeline-4qtnulddc"},"datePublished":"2026-08-16T08:54:01.284Z","dateModified":"2026-09-14T06:41:57.219Z","description":"An engineering deep-dive describes how a large AI model company built a Kafka-to-Databend Cloud pipeline to ingest trillion-scale Agent Trace data using the...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d5fb2ac0fc3d307886ee1d5d98eba572?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d5fb2ac0fc3d307886ee1d5d98eba572?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Databend Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Databend Blog","logo":"https://media.daily.dev/image/upload/s--Vkqa36Ma--/f_auto,q_auto/v1786870423/logos/databend?_a=BAMAMicg0","url":"https://daily.dev/sources/databend"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/from-kafka-to-databend-cloud-engineering-a-trillion-scale-agent-trace-ingestion-pipeline-4qtnulddc","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-engineering,kafka,data-warehouse","timeRequired":"PT10M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Databend Blog","item":"https://daily.dev/sources/databend"},{"@type":"ListItem","position":3,"name":"From Kafka to Databend Cloud: Engineering a Trillion-Scale Agent Trace Ingestion Pipeline"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/from-kafka-to-databend-cloud-engineering-a-trillion-scale-agent-trace-ingestion-pipeline-4qtnulddc#faq","mainEntity":[{"@type":"Question","name":"How does bend-ingest-kafka avoid losing Kafka messages during a crash before data is written?","acceptedAnswer":{"@type":"Answer","text":"It delays committing Kafka offsets until the corresponding batch of files has been successfully written into the Databend Raw Table via COPY INTO, not merely uploaded to Stage. Each worker keeps uploaded Stage files and their Kafka batch in memory, retries COPY with the same files on failure, and only commits offsets once the write succeeds, giving at-least-once delivery with downstream deduplication. Engineers hardening Kafka pipelines against data loss can browse implementation patterns like this on daily.dev."}},{"@type":"Question","name":"What is Raw Mode in bend-ingest-kafka and why use it for Agent Trace data?","acceptedAnswer":{"@type":"Answer","text":"Raw Mode writes each Kafka message's full JSON payload plus topic, partition, and offset metadata into a Raw Table without parsing it into a fixed schema. This keeps the ingestion consumer schema-agnostic so upstream trace format changes, like new reasoning-token or tool-call fields, do not require redeploying consumers; parsing and modeling happen later using Databend Streams and MERGE INTO. Teams designing schema-flexible ingestion for fast-changing event data can find similar patterns on daily.dev."}},{"@type":"Question","name":"How does bend-ingest-kafka decide when to trigger a COPY INTO batch load?","acceptedAnswer":{"@type":"Answer","text":"It uses OR-based thresholds: a COPY INTO runs immediately once 128 pending files accumulate, or after 5 seconds pass since the first uploaded file in the group, whichever comes first. In a 100,000-message end-to-end test with 1,000 messages per batch producing 100 files and one COPY per 5 files, this required only 20 COPY executions, reaching about 15,208 rows per second on one worker. Data engineers tuning batch-load thresholds for warehouse ingestion can compare approaches like this on daily.dev."}}]} ```