--- title: Saving another 100TB of RAM with math (and Rust) | daily.dev description: Cloudflare engineers reduced RAM usage in their Pingora Backend Router (PBR) load-balancing service by over 100TB globally through changes to their open-source... canonical: https://daily.dev/posts/saving-another-100tb-of-ram-with-math-and-rust--vnv8imyss twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Saving another 100TB of RAM with math (and Rust) | daily.dev og:description: Cloudflare engineers reduced RAM usage in their Pingora Backend Router (PBR) load-balancing service by over 100TB globally through changes to their open-source... og:url: https://daily.dev/posts/saving-another-100tb-of-ram-with-math-and-rust--vnv8imyss og:image: https://api.daily.dev/og/posts/VNv8IMYss.png og:image:alt: Saving another 100TB of RAM with math (and Rust) og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Saving another 100TB of RAM with math (and Rust) **[Cloudflare](https://daily.dev/sources/cloudflare)** · 13 min read · 28 upvotes · 6 comments ## Summary Cloudflare engineers reduced RAM usage in their Pingora Backend Router (PBR) load-balancing service by over 100TB globally through changes to their open-source pingora-ketama consistent hashing library. They explain how consistent hashing distributes cacheable requests across servers, why more hashes per server improve load balance (via expected value and standard deviation), and how hash collisions on 32-bit integers eventually offset those gains. Two concrete optimizations delivered the savings: repacking a Rust struct to avoid alignment padding (a 25% memory reduction) and reducing the number of hashes per server by 90% without meaningful accuracy loss, informed by statistical analysis of the coefficient of variation and collision probability. The team migrated the new hash ring gradually, running old and new rings side-by-side per request to avoid mass cache invalidation, and the changes are now available as an opt-in feature in the pingora-ketama crate. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can you reduce Rust struct memory usage caused by alignment padding without using #[repr(packed)]? Store the fields as a raw byte array and access them via getter methods instead of using repr(packed), which avoids the unsafe implications of packed representations while compiling to the same machine code. For example, replacing a struct with a u32 hash and u32 index (8 bytes due to alignment) with a byte array holding a u32 hash and u16 index (6 bytes) cut memory usage for Cloudflare's consistent hashing structures by 25%. _daily.dev surfaces practical Rust memory-layout techniques like this for engineers optimizing large-scale systems._ ### How many hash points per server does Pingora use for consistent hashing? Pingora hardcodes 160 hash points per server, matching NGINX's default baseline for its upstream hash module. With 160 points per server across a 100-server example, the coefficient of variation in load distribution drops from about 99% (with one hash per server) to about 8%, meaning workload is distributed far more evenly. _Engineers tuning load balancer configs can track details like this on daily.dev before touching consistent hashing settings._ ### Why does increasing the number of hashes per server eventually stop improving consistent hashing accuracy? Because hashes are stored as 32-bit integers, collisions become increasingly likely as the hash count grows, following the birthday paradox, and each collision randomly drops a contribution meant to balance load. For data centers with 2048 servers, simulated error rates start rising once the hash count per server reaches roughly 10,000 to 100,000, which let Cloudflare cut hash counts per server by 90% without meaningful accuracy loss. _daily.dev helps engineers dig into distributed-systems tradeoffs like hash collisions before scaling infrastructure._ ## Community take How the wider developer community reacted, aggregated from 2 discussions and 138 comments across lobsters, hackernews (as of 2026-09-19). **TL;DR:** The technical writeup is broadly praised for its clarity, math-driven approach, and being human-written rather than AI-generated, though a sizable side conversation veers into a broader debate about software optimization culture, AI's effect on code quality, and job market implications; a few commenters raise alternative hashing algorithms and question whether the savings are as impressive as presented. **Sentiment:** 45% positive · 35% mixed · 20% skeptical **The case for** - Praised as a well-written, human-authored technical piece with genuine calculus/statistics content rather than AI slop. - Seen as a refreshing example of a large company still investing in careful optimization at scale. - Sparked useful technical side-discussion (e.g. jump hash vs ketama tradeoffs) that helped some readers better understand consistent hashing. **The pushback** - Some argue the actual improvement amounts to only a ~1% gain and question whether it's worth the attention given. - Skepticism that the original implementation was just poorly designed (packing integers better) rather than a deep insight. - Several suggest better alternatives exist (rendezvous/hierarchical rendezvous hashing, tournament hashing with sha256/wyhash) that could save even more memory. - One commenter felt the motivation section didn't clearly explain why consistent hashing is needed in the first place. - Tangential debate about whether Cloudflare's own infra choices (e.g. V8/JavaScript) undercut the optimization narrative. **By community** - lobsters (mixed): No substantive comments were provided, so no discernible community take. - hackernews (mixed): Technical appreciation for the writeup and math is mixed with skepticism about the actual significance of the savings and a long tangent about software optimization culture and AI-driven coding. **Hottest debate:** Whether the broader trend of software bloat and reduced optimization is due to shifting priorities toward shipping speed, AI-assisted coding, or is simply an inevitable consequence of cheap compute. **Open questions** - How does this hashing scheme compare in practice to alternatives like rendezvous hashing or hierarchical rendezvous hashing? - Is the packed hash-ring approach's memory savings still bounded by the 65k-entry limit mentioned, and how would that be addressed at greater scale? - Why exactly is consistent hashing preferred over simpler modulo-based sharding in this system? **Highlights** > I’m not sure why folks are finding this so revolutionary. There are teams of scientists at big techs with PhDs working on all kinds of optimization across compute fleets. This seems cute but its exposition of math is more along the lines of “look at how cool I am that I could do a bit of calculus”, and that makes me question the technical depth at CF. I found the motivation pretty lackluster: nowhere does it actually explain why you use consistent hashing (dividing the item space natively/regularly would actually cause much more than 1/n items to move, which is unintuitive) and how you actually use it. That said, it got me to spend a few minutes studying this and got me to understand the key bit I was missing. > — [christina97 on hackernews](https://news.ycombinator.com/item?id=49767027) > I would get rid of consistent hashing and ketama for a better system which works save an additional 600TiB. You use the first N bits of your key hash to pick the server partition so it’s a reasonable number (eg 128 servers per partition). Then use high quality precomputed hashes (first 64 bits of sha256) for the server name as N in H(K + N). Use wymum from wyhash as the H so that you do o(n) integer multiplications while retaining a result that’s still a good hash statistically. Now you’re using a tournament hash, the small N means O(N) vs O(N log N) doesn’t matter, and also this O(N) is also going to be much less CPU than computing 160 hashes per key as they do now, so much less latency added per request. > — [vlovich123 on hackernews · 1 comments](https://news.ycombinator.com/item?id=49762990) > Anyone have an idea how it behaves differently from google's jump hash algorithm? The cool thing about google's one is it's so short I can include it in a HN comment: int32_t JumpConsistentHash(uint64_t key, int32_t num_buckets) { int64_t b = 1, j = 0; while (j < num_buckets) { b = j; key = key * 2862933555777941757ULL + 1; j = (b + 1) * (double(1LL << 31) / double((key >> 33) + 1)); } return b; } https://arxiv.org/pdf/1406.2294 > — [cyberpunk on hackernews · 2 comments](https://news.ycombinator.com/item?id=49760090) > I have used Google's jump hash. As I recall, one of the main differences is that jump hash doesn't have a mechanism to remove targets, eg, a server dies and you don't want to route requests to it. Traditional consistent hashing can do that. I guess if you had 4 servers, server #4 dies, then you can go back to 3 servers by just changing num_buckets from 4 to 3. But if server 1 dies, you can't. Jump hash does allow adding more targets and preserves the property that most request targets stay the same when adding a new target, so if you had 3 targets and add a fourth, ~8% of the requests that would have been sent to targets 1-3 are sent to target 4, evenly chosen from servers 1-3. > — [prirun on hackernews](https://news.ycombinator.com/item?id=49760802) > TLDR: the existing implementation was poorly designed. They packed 2 integers better and saved memory. > — [goodpoint on hackernews](https://news.ycombinator.com/item?id=49764973) **Source threads** - [lobsters](https://lobste.rs/s/meo3ou/saving_another_100tb_ram_with_math_rust) · 1 points · 0 comments - [hackernews](https://news.ycombinator.com/item?id=49758580) · 218 points · 138 comments ## Community discussion Top comments from developers on daily.dev. **@pdfopsdev** · 1 upvotes > a (u32, usize) pair is 16 bytes on 64-bit with 4 of them padding, and nobody looks until it's multiplied by a whole fleet. a const assert on size_of for the hot structs is cheap insurance so a field change can't quietly undo the win. **@byteoutlaw** · 0 upvotes > Ahhhh - They we're the reason for the rammaggedon! ;) **@starfallcodes** · 0 upvotes > hmmmm i dont trust **@kartiknvj** · 0 upvotes > The detail about 1% improvements magnifying at scale is something I think about with agent evals too. A small change in how you score one axis can shift the whole ranking when you run hundreds of trials. The Rust angle is interesting: lower memory pressure means more room for context. Are you using Rust for any of your evaluation pipeline components? --- Tags: [#rust](https://daily.dev/tags/rust), [#cloudflare](https://daily.dev/tags/cloudflare) [View this post on daily.dev](https://daily.dev/posts/saving-another-100tb-of-ram-with-math-and-rust--vnv8imyss) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Saving another 100TB of RAM with math (and Rust)","url":"https://daily.dev/posts/saving-another-100tb-of-ram-with-math-and-rust--vnv8imyss","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/saving-another-100tb-of-ram-with-math-and-rust--vnv8imyss"},"datePublished":"2026-09-18T17:26:37.198Z","dateModified":"2026-09-19T19:47:00.747Z","description":"Cloudflare engineers reduced RAM usage in their Pingora Backend Router (PBR) load-balancing service by over 100TB globally through changes to their open-source...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/97e0fa12fbd2fe84838d03e16f3863fc?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/97e0fa12fbd2fe84838d03e16f3863fc?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Cloudflare","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Cloudflare","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/38522e1d11354cd6b7af66f9d4316735","url":"https://daily.dev/sources/cloudflare"},"commentCount":6,"discussionUrl":"https://daily.dev/posts/saving-another-100tb-of-ram-with-math-and-rust--vnv8imyss","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":28},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":6}],"keywords":"rust,cloudflare","timeRequired":"PT13M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Cloudflare","item":"https://daily.dev/sources/cloudflare"},{"@type":"ListItem","position":3,"name":"Saving another 100TB of RAM with math (and Rust)"}]} {"@context":"https://schema.org","@type":"WebPage","@id":"https://daily.dev/posts/saving-another-100tb-of-ram-with-math-and-rust--vnv8imyss","comment":[{"@type":"Comment","text":"a (u32, usize) pair is 16 bytes on 64-bit with 4 of them padding, and nobody looks until it’s multiplied by a whole fleet. a const assert on size_of for the hot structs is cheap insurance so a field change can’t quietly undo the win.","datePublished":"2026-09-18T18:03:21.127Z","url":"https://daily.dev/posts/VNv8IMYss#c-N4v1PhJb1","author":{"@type":"Person","name":"PDFops","url":"https://daily.dev/pdfopsdev","image":"https://media.daily.dev/image/upload/s---8isRBKc--/f_auto/v1782922291/avatars/avatar_orjMeK8QKaaVZwGq7ScPz?_a=BAMAMicg0"},"interactionStatistic":{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1}},{"@type":"Comment","text":"Ahhhh - They we’re the reason for the rammaggedon! ;)","datePublished":"2026-09-18T18:51:04.029Z","url":"https://daily.dev/posts/VNv8IMYss#c-31EjKPQFX","author":{"@type":"Person","name":"Jacob B. Bonde","url":"https://daily.dev/byteoutlaw","image":"https://media.daily.dev/image/upload/s--veTChHK7--/f_auto/v1733220070/avatars/avatar_5cQvIZKr5tDFDotVDjulg"}},{"@type":"Comment","text":"hmmmm i dont trust","datePublished":"2026-09-19T06:25:45.601Z","url":"https://daily.dev/posts/VNv8IMYss#c-RQ7tOtfmv","author":{"@type":"Person","name":"Starfall :D","url":"https://daily.dev/starfallcodes","image":"https://media.daily.dev/image/upload/s--4xoTqFbk--/f_auto/v1786806197/avatars/avatar_ecPDW57nUVRidLxwsxruz?_a=BAMAMicg0"}},{"@type":"Comment","text":"The detail about 1% improvements magnifying at scale is something I think about with agent evals too. A small change in how you score one axis can shift the whole ranking when you run hundreds of trials. The Rust angle is interesting: lower memory pressure means more room for context. Are you using Rust for any of your evaluation pipeline components?","datePublished":"2026-09-18T20:52:23.327Z","url":"https://daily.dev/posts/VNv8IMYss#c-lu6BULHn5","author":{"@type":"Person","name":"kartik-nvjk","url":"https://daily.dev/kartiknvj","image":"https://media.daily.dev/image/upload/s--3gGgsVCw--/f_auto/v1781456774/avatars/avatar_TvTVeiMdkRCqWUDullFmy?_a=BAMAMiWQ0"}}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/saving-another-100tb-of-ram-with-math-and-rust--vnv8imyss#faq","mainEntity":[{"@type":"Question","name":"How can you reduce Rust struct memory usage caused by alignment padding without using #[repr(packed)]?","acceptedAnswer":{"@type":"Answer","text":"Store the fields as a raw byte array and access them via getter methods instead of using repr(packed), which avoids the unsafe implications of packed representations while compiling to the same machine code. For example, replacing a struct with a u32 hash and u32 index (8 bytes due to alignment) with a byte array holding a u32 hash and u16 index (6 bytes) cut memory usage for Cloudflare's consistent hashing structures by 25%. daily.dev surfaces practical Rust memory-layout techniques like this for engineers optimizing large-scale systems."}},{"@type":"Question","name":"How many hash points per server does Pingora use for consistent hashing?","acceptedAnswer":{"@type":"Answer","text":"Pingora hardcodes 160 hash points per server, matching NGINX's default baseline for its upstream hash module. With 160 points per server across a 100-server example, the coefficient of variation in load distribution drops from about 99% (with one hash per server) to about 8%, meaning workload is distributed far more evenly. Engineers tuning load balancer configs can track details like this on daily.dev before touching consistent hashing settings."}},{"@type":"Question","name":"Why does increasing the number of hashes per server eventually stop improving consistent hashing accuracy?","acceptedAnswer":{"@type":"Answer","text":"Because hashes are stored as 32-bit integers, collisions become increasingly likely as the hash count grows, following the birthday paradox, and each collision randomly drops a contribution meant to balance load. For data centers with 2048 servers, simulated error rates start rising once the hash count per server reaches roughly 10,000 to 100,000, which let Cloudflare cut hash counts per server by 90% without meaningful accuracy loss. daily.dev helps engineers dig into distributed-systems tradeoffs like hash collisions before scaling infrastructure."}}]} ```