Adding One Server Should Be Boring. In Most Candidates’ Designs, It Quietly Wipes the Entire Cache.
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
Modulo-based cache key routing is a common and initially correct answer to distributing cached data across servers, but it contains a critical flaw: adding or removing even one server reshuffles nearly every key's assignment, causing a cache-miss storm that floods the database. Consistent hashing solves this by placing both keys and servers on a ring, so only a small fraction of keys move when the server count changes. The post also covers virtual nodes for load balance and replication for availability, and frames the core lesson as designing for routine operational changes — not just the static steady state.
Table of contents
The way you spread data across servers looks like a solved problem. Then the server count changes by one, and a good design turns into a cache-miss storm that takes the database down with it.The Answer Everyone GivesAdding One Server Wipes Almost EverythingThe Design That Survives a Server ChangeGet Devrim Ozcay- Backend Engineer ’s stories in your inboxWhat This Prompt Is Really TestingUse This On YourselfThe Thing I Wrote for ThisQuestions this post answers
Why does adding a server to a cache cluster cause a cache miss storm with modulo-based routing?
With modulo routing, each key is assigned to a server by computing hash(key) % server_count. Changing server_count by even one causes nearly every key to map to a different server than the one holding its data. The result is a near-total cache invalidation: almost all lookups miss, traffic falls through to the database, and a routine capacity increase can trigger a full outage. Engineers scaling distributed caches track patterns like this on daily.dev before the capacity change hits production.
How does consistent hashing prevent cache invalidation when adding or removing a server?
Consistent hashing places both servers and keys on a hash ring. Each key is owned by the nearest server clockwise from its position. When a server is added, only the keys in the arc between it and its predecessor move — every other key is undisturbed. When a server is removed, only its keys migrate to the next server clockwise. The cache stays warm for the vast majority of keys. Distributed systems engineers choosing between routing strategies find the trade-offs covered on daily.dev.
What are virtual nodes in consistent hashing and why are they needed?
Virtual nodes place each physical server at multiple positions on the hash ring rather than one. Without them, random single-position placement creates uneven arc sizes, causing some servers to own much larger key slices and become overloaded. Multiple positions per server statistically even out the load distribution across all nodes. Developers implementing consistent hashing in production find edge cases like arc imbalance discussed on daily.dev.