Amazon EKS engineers share how they operate Kubernetes at fleet scale across hundreds of thousands of clusters. Key architectural changes include replacing etcd's Raft consensus with a purpose-built durable journal (eliminating quorum-loss failures and moving etcd to in-memory storage), collocating etcd with the API server for lower latency, and partitioning etcd into resource-specific shards for failure isolation. The post covers upstream Kubernetes fixes for watch-cache locking contention, HPA mutex bottlenecks, and scheduler inefficiencies. It also introduces EKS Provisioned Control Plane, which lets customers reserve control plane capacity in explicit tiers (XL–8XL) with SLA guarantees. Edge deployments on AWS Outposts use the same stack with self-healing etcd and offline-tolerant observability. Three operational lessons are shared: work queue depth as a better health signal than liveness probes, maintenance ordering to avoid blocking writes during etcd defragmentation, and keeping liveness and readiness signals strictly separate.
