--- title: What running Kubernetes across millions of clusters... description: AWS engineers share lessons from operating Kubernetes at scale across millions of EKS clusters, focusing on how gray-zone failures — where a zone is degraded... canonical: https://daily.dev/posts/what-running-kubernetes-across-millions-of-clusters-taught-aws-about-zone-failures-396oc7g9a twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: What running Kubernetes across millions of clusters taught AWS about zone failures | daily.dev og:description: AWS engineers share lessons from operating Kubernetes at scale across millions of EKS clusters, focusing on how gray-zone failures — where a zone is degraded... og:url: https://daily.dev/posts/what-running-kubernetes-across-millions-of-clusters-taught-aws-about-zone-failures-396oc7g9a og:image: https://api.daily.dev/og/posts/396oc7g9a.png og:image:alt: What running Kubernetes across millions of clusters taught AWS about zone failures og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # What running Kubernetes across millions of clusters taught AWS about zone failures **[The New Stack](https://daily.dev/sources/newstack)** · 12 min read · 0 upvotes · 0 comments ## Summary AWS engineers share lessons from operating Kubernetes at scale across millions of EKS clusters, focusing on how gray-zone failures — where a zone is degraded but not fully down — can cascade into regional outages. The post explains the static stability principle: during a zonal impairment, automated systems should stop reacting, preserve existing capacity, and route around the bad zone rather than triggering replacements. It covers the EKS control plane's automated zonal shift mechanism (DNS weight shifts, Auto Scaling group suspension, leader election handoff), and the data plane's ARC zonal shift feature that cordons impaired nodes and removes their endpoints without evicting pods. Key lessons include: healthy zones must fence off bad ones, cross-zone dependencies can extend blast radius beyond the failing zone, and detection/shift infrastructure must rely on low-level primitives (S3, plain EC2) rather than higher-order services. The feature is free, enabled with a single CLI command, and supports both manual and automated (autoshift) modes with weekly practice runs. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is ARC zonal shift for Amazon EKS and how do I enable it? ARC zonal shift is a feature built on Amazon Application Recovery Controller that lets you move workload traffic away from an impaired Availability Zone in an EKS cluster without terminating nodes or evicting pods. Enable it with a single setting: aws eks update-cluster-config --name my-cluster --zonal-shift-config enabled=true, or through the console, eksctl, CloudFormation, or Terraform, with no downtime. _Teams hardening EKS clusters against zone failures can track AWS resilience guidance like this on daily.dev._ ### Why doesn't Amazon EKS zonal shift evict pods from the impaired Availability Zone? EKS avoids eviction because on large clusters, evicting all pods from a failed zone at once causes a surge of capacity requests to the remaining zones, often resulting in Insufficient Capacity errors, and workloads stay unevenly spread even after recovery. Instead, zonal shift cordons nodes and removes endpoints from EndpointSlices while leaving pods in place, so capacity returns immediately once the zone recovers. _Anyone weighing eviction versus capacity preservation strategies for Kubernetes can follow this kind of design tradeoff on daily.dev._ ### How does AWS test that Amazon EKS clusters actually survive a zonal failure? AWS uses AWS Fault Injection Service (FIS) to inject real packet loss and zonal isolation across a cluster rather than manipulating iptables or eBPF rules on individual hosts, which only exercises anticipated failure paths. Running these tests at regular cadence in its largest regions, such as IAD, surfaced a dependency-throttling bug on a control-plane call that would have blocked recovery during a real outage. _Engineers validating cluster resilience before an outage forces the discovery can follow testing practices like this on daily.dev._ ## Similar posts on daily.dev - [Karpenter now supports Amazon Application Recovery Controller zonal shift](https://daily.dev/posts/karpenter-now-supports-amazon-application-recovery-controller-zonal-shift-zbmgrol0r) · AWS · 0 upvotes · 0 comments - [When a Cloud Region Fails: Rethinking High Availability in a Geopolitically Unstable World](https://daily.dev/posts/when-a-cloud-region-fails-rethinking-high-availability-in-a-geopolitically-unstable-world-1ndvhoqp0) · InfoQ · 0 upvotes · 0 comments - [Operating Kubernetes at scale: a few stories from running Amazon EKS](https://daily.dev/posts/operating-kubernetes-at-scale-a-few-stories-from-running-amazon-eks-vdcgnhxcm) · The New Stack · 1 upvotes · 0 comments - [What happens when the cloud blows up?](https://daily.dev/posts/what-happens-when-the-cloud-blows-up--hppqgjaf9) · InfoWorld · 26 upvotes · 1 comments --- Tags: [#aws](https://daily.dev/tags/aws), [#kubernetes](https://daily.dev/tags/kubernetes), [#distributed-systems](https://daily.dev/tags/distributed-systems) [View this post on daily.dev](https://daily.dev/posts/what-running-kubernetes-across-millions-of-clusters-taught-aws-about-zone-failures-396oc7g9a) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"What running Kubernetes across millions of clusters taught AWS about zone failures","url":"https://daily.dev/posts/what-running-kubernetes-across-millions-of-clusters-taught-aws-about-zone-failures-396oc7g9a","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/what-running-kubernetes-across-millions-of-clusters-taught-aws-about-zone-failures-396oc7g9a"},"datePublished":"2026-07-10T15:04:05.601Z","dateModified":"2026-09-14T08:18:16.179Z","description":"AWS engineers share lessons from operating Kubernetes at scale across millions of EKS clusters, focusing on how gray-zone failures — where a zone is degraded...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/496af17199650dc7e420f183293f611b?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/496af17199650dc7e420f183293f611b?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"The New Stack","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"The New Stack","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/newstack","url":"https://daily.dev/sources/newstack"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/what-running-kubernetes-across-millions-of-clusters-taught-aws-about-zone-failures-396oc7g9a","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"aws,kubernetes,distributed-systems","timeRequired":"PT12M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"The New Stack","item":"https://daily.dev/sources/newstack"},{"@type":"ListItem","position":3,"name":"What running Kubernetes across millions of clusters taught AWS about zone failures"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/what-running-kubernetes-across-millions-of-clusters-taught-aws-about-zone-failures-396oc7g9a#faq","mainEntity":[{"@type":"Question","name":"What is ARC zonal shift for Amazon EKS and how do I enable it?","acceptedAnswer":{"@type":"Answer","text":"ARC zonal shift is a feature built on Amazon Application Recovery Controller that lets you move workload traffic away from an impaired Availability Zone in an EKS cluster without terminating nodes or evicting pods. Enable it with a single setting: aws eks update-cluster-config --name my-cluster --zonal-shift-config enabled=true, or through the console, eksctl, CloudFormation, or Terraform, with no downtime. Teams hardening EKS clusters against zone failures can track AWS resilience guidance like this on daily.dev."}},{"@type":"Question","name":"Why doesn't Amazon EKS zonal shift evict pods from the impaired Availability Zone?","acceptedAnswer":{"@type":"Answer","text":"EKS avoids eviction because on large clusters, evicting all pods from a failed zone at once causes a surge of capacity requests to the remaining zones, often resulting in Insufficient Capacity errors, and workloads stay unevenly spread even after recovery. Instead, zonal shift cordons nodes and removes endpoints from EndpointSlices while leaving pods in place, so capacity returns immediately once the zone recovers. Anyone weighing eviction versus capacity preservation strategies for Kubernetes can follow this kind of design tradeoff on daily.dev."}},{"@type":"Question","name":"How does AWS test that Amazon EKS clusters actually survive a zonal failure?","acceptedAnswer":{"@type":"Answer","text":"AWS uses AWS Fault Injection Service (FIS) to inject real packet loss and zonal isolation across a cluster rather than manipulating iptables or eBPF rules on individual hosts, which only exercises anticipated failure paths. Running these tests at regular cadence in its largest regions, such as IAD, surfaced a dependency-throttling bug on a control-plane call that would have blocked recovery during a real outage. Engineers validating cluster resilience before an outage forces the discovery can follow testing practices like this on daily.dev."}}]} ```