--- title: Rewinding the exponential: How we keep hardware usage down description: Cloudflare's Data team details five years of engineering work to keep hardware usage from scaling linearly with exponential traffic growth, now at roughly two... canonical: https://daily.dev/posts/rewinding-the-exponential-how-we-keep-hardware-usage-down-ibfirkejh twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Rewinding the exponential: How we keep hardware usage down | daily.dev og:description: Cloudflare's Data team details five years of engineering work to keep hardware usage from scaling linearly with exponential traffic growth, now at roughly two... og:url: https://daily.dev/posts/rewinding-the-exponential-how-we-keep-hardware-usage-down-ibfirkejh og:image: https://api.daily.dev/og/posts/IBFirkeJh.png og:image:alt: Rewinding the exponential: How we keep hardware usage down og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Rewinding the exponential: How we keep hardware usage down **[Cloudflare](https://daily.dev/sources/cloudflare)** · 12 min read · 2 upvotes · 0 comments ## Summary Cloudflare's Data team details five years of engineering work to keep hardware usage from scaling linearly with exponential traffic growth, now at roughly two billion events per second. Key changes include breaking up a monolithic data schema in 2022 (35-40% bandwidth reduction), switching compression from LZ4 to zstd in 2024 (~40% reduction), rearchitecting pipeline replication to avoid double-writing across regions in spring 2026 (36 GB/s saved), decentralizing sampling in summer 2026 (25 GB/s saved across top datasets), and introducing multi-stage aggregation in fall 2026 (dropping HTTP traffic data from 9+ GB/s to under 2 GB/s). Combined, these changes eliminated roughly 70 GB/s of compressed traffic, equivalent to deferring two and a half years of hardware expansion. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How does Cloudflare's double sampling technique reduce stored data while keeping estimates accurate? Double sampling applies sampling twice in the pipeline, once in Logfwdr and once in centralized receivers, selecting a sample size equal to the square root of the total record count at each stage. This causes high-volume keys to shrink faster than low-volume ones: a key with 100x more data than another stores only about 4x the rows instead of 10x under single sampling, while estimates are reconstructed by scaling against the sample rate. _Teams balancing storage costs against data fidelity can follow pipeline engineering deep dives like this on daily.dev._ ### What is the benefit of switching data pipeline compression from LZ4 to zstd? Switching from LZ4 to zstd compression cut compressed traffic throughput by nearly 40% with little noticeable increase in CPU usage after a gradual fleet-wide rollout. Since uncompressed data volume stayed the same, the same size network pipe could support more than 1.5x the traffic, equivalent to roughly 10 months of deferred growth-driven hardware expansion. _Engineers evaluating compression algorithm trade-offs for high-throughput systems can track real-world results like this on daily.dev._ ### How did Cloudflare reduce cross-region replication traffic in its analytics pipeline without losing disaster resilience? Instead of double-writing every event to two centralized regions at the source, Cloudflare sends each event once to whichever centralized receiver is available, then replicates downstream from its buffering layer, Buftee, after sampling and compression have already shrunk the data. This cutover eliminated about 43 GB/s of compressed transit traffic while adding only roughly 7 GB/s of cross-data-center replication, a net reduction of 36 GB/s. _Architects designing resilient multi-region data pipelines can follow real-world trade-offs like these on daily.dev._ ## Similar posts on daily.dev - [500 Tbps of capacity: 16 years of scaling our global network](https://daily.dev/posts/500-tbps-of-capacity-16-years-of-scaling-our-global-network-epwhxsyeu) · Cloudflare · 0 upvotes · 0 comments - [Cloudflare Year in Review: AI Bots Crawl Aggressively, Post-Quantum Encryption Hits 50%, Go Doubles](https://daily.dev/posts/cloudflare-year-in-review-ai-bots-crawl-aggressively-post-quantum-encryption-hits-50-go-doubles-ibc0aje5i) · InfoQ · 0 upvotes · 0 comments - [How to Scale an Integration Pipeline Without Breaking Correctness](https://daily.dev/posts/how-to-scale-an-integration-pipeline-without-breaking-correctness-ghschovtz) · Towards Data Science · 1 upvotes · 0 comments - [Ensuring reliable OpenTelemetry ingestion at scale](https://daily.dev/posts/ensuring-reliable-opentelemetry-ingestion-at-scale-r8929yipq) · ClickHouse · 3 upvotes · 0 comments --- Tags: [#backend](https://daily.dev/tags/backend), [#distributed-systems](https://daily.dev/tags/distributed-systems), [#cloudflare](https://daily.dev/tags/cloudflare), [#clickhouse](https://daily.dev/tags/clickhouse), [#compression](https://daily.dev/tags/compression) [View this post on daily.dev](https://daily.dev/posts/rewinding-the-exponential-how-we-keep-hardware-usage-down-ibfirkejh) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Rewinding the exponential: How we keep hardware usage down","url":"https://daily.dev/posts/rewinding-the-exponential-how-we-keep-hardware-usage-down-ibfirkejh","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/rewinding-the-exponential-how-we-keep-hardware-usage-down-ibfirkejh"},"datePublished":"2026-10-13T15:52:27.774Z","dateModified":"2026-10-13T16:21:11.036Z","description":"Cloudflare's Data team details five years of engineering work to keep hardware usage from scaling linearly with exponential traffic growth, now at roughly two...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4eadb6506c3035af0ff94a96057dabd9?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/4eadb6506c3035af0ff94a96057dabd9?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Cloudflare","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Cloudflare","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/38522e1d11354cd6b7af66f9d4316735","url":"https://daily.dev/sources/cloudflare"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/rewinding-the-exponential-how-we-keep-hardware-usage-down-ibfirkejh","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"backend,distributed-systems,cloudflare,clickhouse,compression","timeRequired":"PT12M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Cloudflare","item":"https://daily.dev/sources/cloudflare"},{"@type":"ListItem","position":3,"name":"Rewinding the exponential: How we keep hardware usage down"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/rewinding-the-exponential-how-we-keep-hardware-usage-down-ibfirkejh#faq","mainEntity":[{"@type":"Question","name":"How does Cloudflare's double sampling technique reduce stored data while keeping estimates accurate?","acceptedAnswer":{"@type":"Answer","text":"Double sampling applies sampling twice in the pipeline, once in Logfwdr and once in centralized receivers, selecting a sample size equal to the square root of the total record count at each stage. This causes high-volume keys to shrink faster than low-volume ones: a key with 100x more data than another stores only about 4x the rows instead of 10x under single sampling, while estimates are reconstructed by scaling against the sample rate. Teams balancing storage costs against data fidelity can follow pipeline engineering deep dives like this on daily.dev."}},{"@type":"Question","name":"What is the benefit of switching data pipeline compression from LZ4 to zstd?","acceptedAnswer":{"@type":"Answer","text":"Switching from LZ4 to zstd compression cut compressed traffic throughput by nearly 40% with little noticeable increase in CPU usage after a gradual fleet-wide rollout. Since uncompressed data volume stayed the same, the same size network pipe could support more than 1.5x the traffic, equivalent to roughly 10 months of deferred growth-driven hardware expansion. Engineers evaluating compression algorithm trade-offs for high-throughput systems can track real-world results like this on daily.dev."}},{"@type":"Question","name":"How did Cloudflare reduce cross-region replication traffic in its analytics pipeline without losing disaster resilience?","acceptedAnswer":{"@type":"Answer","text":"Instead of double-writing every event to two centralized regions at the source, Cloudflare sends each event once to whichever centralized receiver is available, then replicates downstream from its buffering layer, Buftee, after sampling and compression have already shrunk the data. This cutover eliminated about 43 GB/s of compressed transit traffic while adding only roughly 7 GB/s of cross-data-center replication, a net reduction of 36 GB/s. Architects designing resilient multi-region data pipelines can follow real-world trade-offs like these on daily.dev."}}]} ```