--- title: Quick thoughts on GitHub’s Sept 23 incident | daily.dev description: An analysis of GitHub's September 23 incident, where a failed planned maintenance operation on a primary database triggered an automated failover, after which... canonical: https://daily.dev/posts/quick-thoughts-on-github-s-sept-23-incident-522cyveou twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Quick thoughts on GitHub’s Sept 23 incident | daily.dev og:description: An analysis of GitHub's September 23 incident, where a failed planned maintenance operation on a primary database triggered an automated failover, after which... og:url: https://daily.dev/posts/quick-thoughts-on-github-s-sept-23-incident-522cyveou og:image: https://api.daily.dev/og/posts/522cyVEOu.png og:image:alt: Quick thoughts on GitHub’s Sept 23 incident og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Quick thoughts on GitHub’s Sept 23 incident **[Surfing Complexity](https://daily.dev/sources/surfingcomplexity)** · 4 min read · 15 upvotes · 0 comments ## Summary An analysis of GitHub's September 23 incident, where a failed planned maintenance operation on a primary database triggered an automated failover, after which replicas in the affected Azure Central US region couldn't resume replication correctly. This left the primary database handling all traffic alone, causing API errors, delayed label updates, and stale search results in Projects for over 20 hours. The writeup examines how the incident, despite not being load-triggered, still ended up as a saturation failure due to lost replica capacity, and connects it to broader patterns of how planned maintenance, failover mechanisms, and capacity constraints interact to cause outages. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What caused the GitHub outage on September 23? A failed planned maintenance operation on a primary database triggered an automated emergency failover, after which several replicas in GitHub's Azure Central US region were unable to resume replication correctly. This reduced database capacity, causing elevated 500 and 404 errors, failures installing GitHub Apps and creating organizations, and delayed Projects processing. API errors were mitigated by 10:58 UTC, with full recovery by 04:55 UTC the next day. _Engineers tracking GitHub reliability trends can follow incident breakdowns like this on daily.dev._ ### Why did losing database replicas cause outage even though the incident wasn't triggered by a traffic spike? When replicas failed to resume replication after an emergency failover, the primary database had to absorb all read traffic that was previously distributed across the replica cluster, effectively creating a saturation failure from a capacity loss rather than a demand increase. This shows how reliability subsystems meant to reduce downtime can themselves become a source of overload during recovery. _Developers designing database failover strategies can dig into cases like this one on daily.dev._ ## Similar posts on daily.dev - [Saturation at GitHub: the saga continues](https://daily.dev/posts/saturation-at-github-the-saga-continues-2uymwgoad) · Surfing Complexity · 6 upvotes · 0 comments - [Addressing GitHub’s recent availability issues](https://daily.dev/posts/addressing-github-s-recent-availability-issues-gzw2ygubb) · GitHub Blog · 3 upvotes · 0 comments - [GitHub has another tough day](https://daily.dev/posts/github-has-another-tough-day-dvefjjxpq) · Surfing Complexity · 18 upvotes · 4 comments --- Tags: [#github](https://daily.dev/tags/github), [#database](https://daily.dev/tags/database), [#distributed-systems](https://daily.dev/tags/distributed-systems) [View this post on daily.dev](https://daily.dev/posts/quick-thoughts-on-github-s-sept-23-incident-522cyveou) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Quick thoughts on GitHub’s Sept 23 incident","url":"https://daily.dev/posts/quick-thoughts-on-github-s-sept-23-incident-522cyveou","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/quick-thoughts-on-github-s-sept-23-incident-522cyveou"},"datePublished":"2026-10-04T00:11:05.058Z","dateModified":"2026-10-04T00:11:24.857Z","description":"An analysis of GitHub's September 23 incident, where a failed planned maintenance operation on a primary database triggered an automated failover, after which...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/62530ed728ad29f493a016cc414981fd?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/62530ed728ad29f493a016cc414981fd?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Surfing Complexity","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Surfing Complexity","logo":"https://media.daily.dev/image/upload/s--sRZ8ppTj--/f_auto,q_auto/v1774514117/logos/surfingcomplexity?_a=BAMAMiWQ0","url":"https://daily.dev/sources/surfingcomplexity"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/quick-thoughts-on-github-s-sept-23-incident-522cyveou","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":15},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"github,database,distributed-systems","timeRequired":"PT4M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Surfing Complexity","item":"https://daily.dev/sources/surfingcomplexity"},{"@type":"ListItem","position":3,"name":"Quick thoughts on GitHub’s Sept 23 incident"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/quick-thoughts-on-github-s-sept-23-incident-522cyveou#faq","mainEntity":[{"@type":"Question","name":"What caused the GitHub outage on September 23?","acceptedAnswer":{"@type":"Answer","text":"A failed planned maintenance operation on a primary database triggered an automated emergency failover, after which several replicas in GitHub's Azure Central US region were unable to resume replication correctly. This reduced database capacity, causing elevated 500 and 404 errors, failures installing GitHub Apps and creating organizations, and delayed Projects processing. API errors were mitigated by 10:58 UTC, with full recovery by 04:55 UTC the next day. Engineers tracking GitHub reliability trends can follow incident breakdowns like this on daily.dev."}},{"@type":"Question","name":"Why did losing database replicas cause outage even though the incident wasn't triggered by a traffic spike?","acceptedAnswer":{"@type":"Answer","text":"When replicas failed to resume replication after an emergency failover, the primary database had to absorb all read traffic that was previously distributed across the replica cluster, effectively creating a saturation failure from a capacity loss rather than a demand increase. This shows how reliability subsystems meant to reduce downtime can themselves become a source of overload during recovery. Developers designing database failover strategies can dig into cases like this one on daily.dev."}}]} ```