--- title: Why etcd breaks at scale in Kubernetes | daily.dev description: An in-depth breakdown of why etcd, the Raft-based key-value store behind every Kubernetes control plane, hits limits as clusters grow. Covers etcd's... canonical: https://daily.dev/posts/why-etcd-breaks-at-scale-in-kubernetes-adqqzthrv twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Why etcd breaks at scale in Kubernetes | daily.dev og:description: An in-depth breakdown of why etcd, the Raft-based key-value store behind every Kubernetes control plane, hits limits as clusters grow. Covers etcd's... og:url: https://daily.dev/posts/why-etcd-breaks-at-scale-in-kubernetes-adqqzthrv og:image: https://api.daily.dev/og/posts/ADqQzTHRv.png og:image:alt: Why etcd breaks at scale in Kubernetes og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Why etcd breaks at scale in Kubernetes **[LearnKube](https://daily.dev/sources/learnkube)** · 23 min read · 0 upvotes · 0 comments ## Summary An in-depth breakdown of why etcd, the Raft-based key-value store behind every Kubernetes control plane, hits limits as clusters grow. Covers etcd's single-leader write bottleneck, bbolt's single-file storage with an 8 GiB suggested max and 2 GiB default quota, MVCC revision accumulation requiring compaction and defragmentation, and how the API server's watch cache and streaming encoding reduce load on etcd. Explains sharding etcd via --etcd-servers-overrides, k3s's Kine shim that swaps etcd for relational databases, and how AWS EKS and Google GKE replaced etcd's internals (journal service, Spanner) to reach 100,000+ node clusters while still exposing the etcd API because Kubernetes' storage interface is deeply etcd-shaped. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is the default etcd backend quota and suggested maximum database size in Kubernetes? The default etcd backend quota is 2 GiB, while the etcd documentation lists a suggested maximum database size of 8 GiB. Each individual request is capped at 1.5 MiB and each key-value pair at 1 MiB, which is why large Secrets, ConfigMaps, or CRD instances get rejected once serialized data exceeds that size. _Track storage limits like these on daily.dev before they turn into a production incident._ ### How did AWS scale EKS clusters to 100,000 nodes without using etcd's default Raft consensus? AWS replaced Raft consensus with an internal journal service, swapped the bbolt backend for an in-memory design, and partitioned the keyspace across nodes, while still exposing the standard etcd gRPC API so the Kubernetes API server didn't need to change. This eliminated the single-leader bottleneck and peer-to-peer communication that limit vanilla etcd. _Engineers weighing control-plane architecture choices follow scaling breakdowns like this on daily.dev._ ### What is Kine and how does it let k3s avoid running etcd? Kine is a shim, built for Rancher's k3s distribution, that implements a subset of the etcd gRPC API and translates requests into a relational database such as SQLite, PostgreSQL, MySQL/MariaDB, or NATS. Because the Kubernetes API server only talks to etcd's gRPC API rather than its internals, Kine lets k3s run without a three-node etcd cluster, trading off native revision semantics and watch efficiency. _Teams comparing lightweight Kubernetes distributions can follow trade-offs like Kine's on daily.dev._ ## Similar posts on daily.dev - [Operating Kubernetes at scale: a few stories from running Amazon EKS](https://daily.dev/posts/operating-kubernetes-at-scale-a-few-stories-from-running-amazon-eks-vdcgnhxcm) · The New Stack · 1 upvotes · 0 comments - [k8s-1m Overview](https://daily.dev/posts/k8s-1m-overview-uj6kbjdzt) · Lobsters · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#distributed-systems](https://daily.dev/tags/distributed-systems) [View this post on daily.dev](https://daily.dev/posts/why-etcd-breaks-at-scale-in-kubernetes-adqqzthrv) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Why etcd breaks at scale in Kubernetes","url":"https://daily.dev/posts/why-etcd-breaks-at-scale-in-kubernetes-adqqzthrv","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/why-etcd-breaks-at-scale-in-kubernetes-adqqzthrv"},"datePublished":"2026-08-31T11:20:04.202Z","dateModified":"2026-08-31T11:20:45.027Z","description":"An in-depth breakdown of why etcd, the Raft-based key-value store behind every Kubernetes control plane, hits limits as clusters grow. Covers etcd's...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/cd244864d46504a54f44de109d4465d2?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/cd244864d46504a54f44de109d4465d2?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"LearnKube","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"LearnKube","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/90786df1b5d44abb91a090a0b6c81517","url":"https://daily.dev/sources/learnkube"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/why-etcd-breaks-at-scale-in-kubernetes-adqqzthrv","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,distributed-systems","timeRequired":"PT23M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"LearnKube","item":"https://daily.dev/sources/learnkube"},{"@type":"ListItem","position":3,"name":"Why etcd breaks at scale in Kubernetes"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/why-etcd-breaks-at-scale-in-kubernetes-adqqzthrv#faq","mainEntity":[{"@type":"Question","name":"What is the default etcd backend quota and suggested maximum database size in Kubernetes?","acceptedAnswer":{"@type":"Answer","text":"The default etcd backend quota is 2 GiB, while the etcd documentation lists a suggested maximum database size of 8 GiB. Each individual request is capped at 1.5 MiB and each key-value pair at 1 MiB, which is why large Secrets, ConfigMaps, or CRD instances get rejected once serialized data exceeds that size. Track storage limits like these on daily.dev before they turn into a production incident."}},{"@type":"Question","name":"How did AWS scale EKS clusters to 100,000 nodes without using etcd's default Raft consensus?","acceptedAnswer":{"@type":"Answer","text":"AWS replaced Raft consensus with an internal journal service, swapped the bbolt backend for an in-memory design, and partitioned the keyspace across nodes, while still exposing the standard etcd gRPC API so the Kubernetes API server didn't need to change. This eliminated the single-leader bottleneck and peer-to-peer communication that limit vanilla etcd. Engineers weighing control-plane architecture choices follow scaling breakdowns like this on daily.dev."}},{"@type":"Question","name":"What is Kine and how does it let k3s avoid running etcd?","acceptedAnswer":{"@type":"Answer","text":"Kine is a shim, built for Rancher's k3s distribution, that implements a subset of the etcd gRPC API and translates requests into a relational database such as SQLite, PostgreSQL, MySQL/MariaDB, or NATS. Because the Kubernetes API server only talks to etcd's gRPC API rather than its internals, Kine lets k3s run without a three-node etcd cluster, trading off native revision semantics and watch efficiency. Teams comparing lightweight Kubernetes distributions can follow trade-offs like Kine's on daily.dev."}}]} ```