--- title: Our billing pipeline was suddenly slow. The culprit was... description: Cloudflare's billing pipeline started missing daily deadlines after a ClickHouse table partitioning key change from (day) to (namespace, day) to enable... canonical: https://daily.dev/posts/our-billing-pipeline-was-suddenly-slow-the-culprit-was-a-hidden-bottleneck-in-clickhouse-t6saatu35 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Our billing pipeline was suddenly slow. The culprit was a hidden bottleneck in ClickHouse | daily.dev og:description: Cloudflare's billing pipeline started missing daily deadlines after a ClickHouse table partitioning key change from (day) to (namespace, day) to enable... og:url: https://daily.dev/posts/our-billing-pipeline-was-suddenly-slow-the-culprit-was-a-hidden-bottleneck-in-clickhouse-t6saatu35 og:image: https://api.daily.dev/og/posts/t6sAatu35.png og:image:alt: Our billing pipeline was suddenly slow. The culprit was a hidden bottleneck in ClickHouse og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Our billing pipeline was suddenly slow. The culprit was a hidden bottleneck in ClickHouse **[Cloudflare](https://daily.dev/sources/cloudflare)** · 11 min read · 7 upvotes · 0 comments ## Summary Cloudflare's billing pipeline started missing daily deadlines after a ClickHouse table partitioning key change from (day) to (namespace, day) to enable per-tenant retention. Standard metrics showed nothing wrong — I/O, memory, and parts read per query all looked normal. Using ClickHouse's trace_log and flame graphs, engineers discovered that over half of query duration was spent waiting on a single exclusive mutex (MergeTreeData) that every query planner thread had to acquire to copy the full parts list. Three sequential fixes were applied: (1) switching from an exclusive to a shared lock, eliminating the contention immediately; (2) deferring the full vector copy by maintaining a shared cache, reducing copy overhead; (3) replacing a linear scan over all parts with a binary search on the namespace partition key, cutting query durations by 50% and breaking the correlation between part count and latency. The fixes were contributed upstream and merged into ClickHouse 25.11. At peak, the cluster reached 160k parts per replica, but query durations remain stable. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Cloudflare Identifies Query Planning Bottleneck in ClickHouse](https://daily.dev/posts/cloudflare-identifies-query-planning-bottleneck-in-clickhouse-2yvq0qcqd) · InfoQ · 0 upvotes · 0 comments - [The 'Too Many Parts' Problem in ClickHouse](https://daily.dev/posts/the-too-many-parts-problem-in-clickhouse-ipyhjvduq) · BigData Boutique blog · 5 upvotes · 0 comments - [The Migrating Bottleneck](https://daily.dev/posts/the-migrating-bottleneck-ep4fzxzi1) · Webframp · 0 upvotes · 0 comments - [High Cardinality in ClickHouse at Scale: What Actually Breaks](https://daily.dev/posts/high-cardinality-in-clickhouse-at-scale-what-actually-breaks-pc6axtguq) · Last9 · 0 upvotes · 0 comments --- Tags: [#clickhouse](https://daily.dev/tags/clickhouse) [View this post on daily.dev](https://daily.dev/posts/our-billing-pipeline-was-suddenly-slow-the-culprit-was-a-hidden-bottleneck-in-clickhouse-t6saatu35) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Our billing pipeline was suddenly slow. The culprit was a hidden bottleneck in ClickHouse","url":"https://daily.dev/posts/our-billing-pipeline-was-suddenly-slow-the-culprit-was-a-hidden-bottleneck-in-clickhouse-t6saatu35","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/our-billing-pipeline-was-suddenly-slow-the-culprit-was-a-hidden-bottleneck-in-clickhouse-t6saatu35"},"datePublished":"2026-05-14T13:49:40.294Z","dateModified":"2026-05-14T13:50:03.762Z","description":"Cloudflare's billing pipeline started missing daily deadlines after a ClickHouse table partitioning key change from (day) to (namespace, day) to enable...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/9d9da48bfb7dfece739c83b2a125324a?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/9d9da48bfb7dfece739c83b2a125324a?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Cloudflare","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Cloudflare","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/38522e1d11354cd6b7af66f9d4316735","url":"https://daily.dev/sources/cloudflare"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/our-billing-pipeline-was-suddenly-slow-the-culprit-was-a-hidden-bottleneck-in-clickhouse-t6saatu35","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":7},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"clickhouse","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Cloudflare","item":"https://daily.dev/sources/cloudflare"},{"@type":"ListItem","position":3,"name":"Our billing pipeline was suddenly slow. The culprit was a hidden bottleneck in ClickHouse"}]} ```