--- title: How we built a production Kafka connector for ClickHouse® description: Tinybird built a custom Kafka to ClickHouse connector to overcome limitations of ClickHouse's built-in Kafka engine, including partial Schema Registry support... canonical: https://daily.dev/posts/how-we-built-a-production-kafka-connector-for-clickhouse--awxluas12 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How we built a production Kafka connector for ClickHouse® | daily.dev og:description: Tinybird built a custom Kafka to ClickHouse connector to overcome limitations of ClickHouse's built-in Kafka engine, including partial Schema Registry support... og:url: https://daily.dev/posts/how-we-built-a-production-kafka-connector-for-clickhouse--awxluas12 og:image: https://api.daily.dev/og/posts/AWxLuAS12.png og:image:alt: How we built a production Kafka connector for ClickHouse® og:image:width: 1200 og:image:height: 630 og:locale: en --- [Tinybird](https://daily.dev/sources/tinybird) [Read post](https://api.daily.dev/r/AWxLuAS12) # [How we built a production Kafka connector for ClickHouse®](https://api.daily.dev/r/AWxLuAS12 "Go to post") Tinybird built a custom Kafka to ClickHouse connector to overcome limitations of ClickHouse's built-in Kafka engine, including partial Schema Registry support and scaling challenges. The Python-based service uses a controller/worker pattern for horizontal scaling and fault isolation, with rendezvous hashing for topic assignment and circuit breaker patterns for reliability. Key optimizations include RowBinary encoding via a custom C++ library (30% faster ingestion), an ingestion proxy to prevent TOO\_MANY\_PARTS errors, and native libraries (librdkafka, orjson, fastavro) for performance-critical operations. The connector processes billions of events daily with KEDA autoscaling, comprehensive observability through ClickHouse ops logs and Prometheus metrics, and enterprise features like Schema Registry integration, automatic metadata injection, and quarantine tables for failed messages. [#python](/tags/python "Check all #python posts")[#kubernetes](/tags/kubernetes "Check all #kubernetes posts")[#kafka](/tags/kafka "Check all #kafka posts")[#data-engineering](/tags/data-engineering "Check all #data-engineering posts")[#clickhouse](/tags/clickhouse "Check all #clickhouse posts") Jan 21 • 10m read time • From [tinybird.co](https://api.daily.dev/r/AWxLuAS12 "tinybird.co") [](https://api.daily.dev/r/AWxLuAS12 "Go to post") Table of contents [Why build our own Kafka connector?](https://api.daily.dev/r/AWxLuAS12 "Why build our own Kafka connector?")[ Architecture of a scalable Kafka connector](https://api.daily.dev/r/AWxLuAS12 "Architecture of a scalable Kafka connector")[ Five design decisions that make it production ready](https://api.daily.dev/r/AWxLuAS12 "Five design decisions that make it production ready")[ How we scale and monitor billions of events](https://api.daily.dev/r/AWxLuAS12 "How we scale and monitor billions of events")[ Getting Started](https://api.daily.dev/r/AWxLuAS12 "Getting Started")[ What we learned](https://api.daily.dev/r/AWxLuAS12 "What we learned") Comment Bookmark Copy Share your thoughts Post Share this post [](https://daily.dev/sources/tinybird) [Tinybird](https://daily.dev/sources/tinybird "https://daily.dev/sources/tinybird") Tinybird is a real-time data analytics platform for building fast and scalable data pipelines, offer... Read more 95 Followers • 1.5K Upvotes #### Would you recommend this post? Copy link Slack WhatsApp Facebook X New Squad Copy linkShare with your friends ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How we built a production Kafka connector for ClickHouse®","url":"https://daily.dev/posts/how-we-built-a-production-kafka-connector-for-clickhouse--awxluas12","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-we-built-a-production-kafka-connector-for-clickhouse--awxluas12"},"datePublished":"2026-01-21T23:13:14.169Z","dateModified":"2026-01-21T23:13:41.127Z","description":"Tinybird built a custom Kafka to ClickHouse connector to overcome limitations of ClickHouse's built-in Kafka engine, including partial Schema Registry support...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/0716b5768c4ca9f424430c9fbd548391?_a=AQAEulh","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/0716b5768c4ca9f424430c9fbd548391?_a=AQAEulh","isAccessibleForFree":true,"articleSection":"Tinybird","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Tinybird","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/Tinybird-Avatar","url":"https://daily.dev/sources/tinybird"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-we-built-a-production-kafka-connector-for-clickhouse--awxluas12","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"python,kubernetes,kafka,data-engineering,clickhouse","timeRequired":"PT10M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Tinybird","item":"https://daily.dev/sources/tinybird"},{"@type":"ListItem","position":3,"name":"How we built a production Kafka connector for ClickHouse®"}]} ```