--- title: Thoughts on the Buildkite Aug 25 incident | daily.dev description: An analysis of Buildkite's August 25 production incident, in which an application deploy consumed all available Kubernetes cluster headroom while background... canonical: https://daily.dev/posts/thoughts-on-the-buildkite-aug-25-incident-czbo1iygo twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Thoughts on the Buildkite Aug 25 incident | daily.dev og:description: An analysis of Buildkite's August 25 production incident, in which an application deploy consumed all available Kubernetes cluster headroom while background... og:url: https://daily.dev/posts/thoughts-on-the-buildkite-aug-25-incident-czbo1iygo og:image: https://api.daily.dev/og/posts/czbo1IYGO.png og:image:alt: Thoughts on the Buildkite Aug 25 incident og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Thoughts on the Buildkite Aug 25 incident **[Surfing Complexity](https://daily.dev/sources/surfingcomplexity)** · 13 min read · 2 upvotes · 0 comments ## Summary An analysis of Buildkite's August 25 production incident, in which an application deploy consumed all available Kubernetes cluster headroom while background workers were simultaneously trying to scale out. The resulting delay in cluster autoscaling caused runaway replica requests (exacerbated by an unusually high maxReplicas setting), and when new nodes finally came online, the sudden surge of new pods and network endpoints overwhelmed CoreDNS, which was not itself configured to autoscale. CoreDNS failure cascaded into API, job dispatch, and notification failures for all customers, recovered after pausing deploys and scaling CoreDNS directly. The piece walks through Kubernetes autoscaling mechanics, explains how interacting control loops (HPA and cluster autoscaler) created an unexpected failure mode, and lists the many contributing factors identified in Buildkite's own write-up. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why did CoreDNS become unavailable during the Buildkite August incident? An application deploy consumed all available Kubernetes node headroom while background worker pods tried to scale out simultaneously. Delayed EKS cluster autoscaling caused runaway replica requests due to a maxReplica limit set too high, and when new nodes finally came online, the resulting surge in pods and network endpoint churn overwhelmed the fixed-size CoreDNS service, which had no autoscaling of its own. _Teams debugging autoscaling-triggered DNS outages can track similar postmortems and root causes on daily.dev._ ### How did Buildkite recover from the CoreDNS outage that caused customer-facing failures? Responders paused further application deployments, deployed more CoreDNS service replicas, raised the memory available to each CoreDNS replica, and expanded the node pool running them. The new CoreDNS pods came into service by 23:12 UTC, after which DNS errors fell rapidly and customer-facing services recovered fully by 23:16 UTC. _Engineers designing incident response playbooks for DNS or service-discovery failures can follow related writeups on daily.dev._ ### Why can setting a high maxReplica count on a Kubernetes worker pool cause problems instead of preventing them? A maxReplica count set higher than needed, intended to avoid running out of capacity, allowed runaway autoscaling to continue unchecked when metrics stayed unchanged during a cluster capacity shortage. This let a worker pool request far more pods than necessary once new nodes came online, contributing to the surge that overwhelmed CoreDNS. _Anyone tuning Kubernetes autoscaler limits can compare tradeoffs and real incident lessons on daily.dev._ ## Similar posts on daily.dev - [Post-incident review for 20th October 2025](https://daily.dev/posts/post-incident-review-for-20th-october-2025-5ec8gudkw) · Buildkite · 0 upvotes · 0 comments - [GitHub has another tough day](https://daily.dev/posts/github-has-another-tough-day-dvefjjxpq) · Surfing Complexity · 18 upvotes · 4 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes) [View this post on daily.dev](https://daily.dev/posts/thoughts-on-the-buildkite-aug-25-incident-czbo1iygo) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Thoughts on the Buildkite Aug 25 incident","url":"https://daily.dev/posts/thoughts-on-the-buildkite-aug-25-incident-czbo1iygo","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/thoughts-on-the-buildkite-aug-25-incident-czbo1iygo"},"datePublished":"2026-09-02T03:42:18.103Z","dateModified":"2026-09-14T09:15:10.237Z","description":"An analysis of Buildkite's August 25 production incident, in which an application deploy consumed all available Kubernetes cluster headroom while background...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/295014887911dc4a788d83e32ff00f4c?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/295014887911dc4a788d83e32ff00f4c?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Surfing Complexity","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Surfing Complexity","logo":"https://media.daily.dev/image/upload/s--sRZ8ppTj--/f_auto,q_auto/v1774514117/logos/surfingcomplexity?_a=BAMAMiWQ0","url":"https://daily.dev/sources/surfingcomplexity"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/thoughts-on-the-buildkite-aug-25-incident-czbo1iygo","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":2},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes","timeRequired":"PT13M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Surfing Complexity","item":"https://daily.dev/sources/surfingcomplexity"},{"@type":"ListItem","position":3,"name":"Thoughts on the Buildkite Aug 25 incident"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/thoughts-on-the-buildkite-aug-25-incident-czbo1iygo#faq","mainEntity":[{"@type":"Question","name":"Why did CoreDNS become unavailable during the Buildkite August incident?","acceptedAnswer":{"@type":"Answer","text":"An application deploy consumed all available Kubernetes node headroom while background worker pods tried to scale out simultaneously. Delayed EKS cluster autoscaling caused runaway replica requests due to a maxReplica limit set too high, and when new nodes finally came online, the resulting surge in pods and network endpoint churn overwhelmed the fixed-size CoreDNS service, which had no autoscaling of its own. Teams debugging autoscaling-triggered DNS outages can track similar postmortems and root causes on daily.dev."}},{"@type":"Question","name":"How did Buildkite recover from the CoreDNS outage that caused customer-facing failures?","acceptedAnswer":{"@type":"Answer","text":"Responders paused further application deployments, deployed more CoreDNS service replicas, raised the memory available to each CoreDNS replica, and expanded the node pool running them. The new CoreDNS pods came into service by 23:12 UTC, after which DNS errors fell rapidly and customer-facing services recovered fully by 23:16 UTC. Engineers designing incident response playbooks for DNS or service-discovery failures can follow related writeups on daily.dev."}},{"@type":"Question","name":"Why can setting a high maxReplica count on a Kubernetes worker pool cause problems instead of preventing them?","acceptedAnswer":{"@type":"Answer","text":"A maxReplica count set higher than needed, intended to avoid running out of capacity, allowed runaway autoscaling to continue unchecked when metrics stayed unchanged during a cluster capacity shortage. This let a worker pool request far more pods than necessary once new nodes came online, contributing to the surge that overwhelmed CoreDNS. Anyone tuning Kubernetes autoscaler limits can compare tradeoffs and real incident lessons on daily.dev."}}]} ```