--- title: How GitHub Learned How Hard Distributed Systems Are «... description: A brief post-mortem summary of the October 2018 GitHub outage, where a ~1-minute primary data center disconnect triggered an untested failover to a secondary... canonical: https://daily.dev/posts/how-github-learned-how-hard-distributed-systems-are-ipspace-net-blog-wy4nc42wu twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How GitHub Learned How Hard Distributed Systems Are « ipSpace.net blog | daily.dev og:description: A brief post-mortem summary of the October 2018 GitHub outage, where a ~1-minute primary data center disconnect triggered an untested failover to a secondary... og:url: https://daily.dev/posts/how-github-learned-how-hard-distributed-systems-are-ipspace-net-blog-wy4nc42wu og:image: https://api.daily.dev/og/posts/wy4nC42wU.png og:image:alt: How GitHub Learned How Hard Distributed Systems Are « ipSpace.net blog og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How GitHub Learned How Hard Distributed Systems Are « ipSpace.net blog **[ipSpace.net Blog](https://daily.dev/sources/ipspace)** · 1 min read · 0 upvotes · 0 comments ## Summary A brief post-mortem summary of the October 2018 GitHub outage, where a ~1-minute primary data center disconnect triggered an untested failover to a secondary data center. Batch jobs continued writing to the primary during the failover, causing the two database replicas to fall out of sync. The cleanup took over 24 hours. The key lessons: distributed systems are inherently hard to make highly available, and disaster recovery plans are meaningless without realistic, thorough testing. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What caused the GitHub outage in October 2018 and how long did it take to recover? A roughly 1-minute disconnect of the primary data center triggered a failover to the secondary data center that had never been tested in production. During the failover, batch jobs continued modifying data in the primary data center, causing the two database replicas to diverge. Reconciling the out-of-sync replicas took over 24 hours to resolve. _Engineers building resilient backend systems track real-world outage post-mortems like this one on daily.dev._ ## Similar posts on daily.dev - [Saturation at GitHub: the saga continues](https://daily.dev/posts/saturation-at-github-the-saga-continues-2uymwgoad) · Surfing Complexity · 6 upvotes · 0 comments - [GitHub availability report: February 2026](https://daily.dev/posts/github-availability-report-february-2026-9u55mdfgo) · GitHub Blog · 0 upvotes · 0 comments - [GitHub has another tough day](https://daily.dev/posts/github-has-another-tough-day-dvefjjxpq) · Surfing Complexity · 18 upvotes · 4 comments --- Tags: [#database](https://daily.dev/tags/database), [#distributed-systems](https://daily.dev/tags/distributed-systems) [View this post on daily.dev](https://daily.dev/posts/how-github-learned-how-hard-distributed-systems-are-ipspace-net-blog-wy4nc42wu) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How GitHub Learned How Hard Distributed Systems Are « ipSpace.net blog","url":"https://daily.dev/posts/how-github-learned-how-hard-distributed-systems-are-ipspace-net-blog-wy4nc42wu","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-github-learned-how-hard-distributed-systems-are-ipspace-net-blog-wy4nc42wu"},"datePublished":"2026-08-11T08:55:57.539Z","dateModified":"2026-08-11T09:22:35.881Z","description":"A brief post-mortem summary of the October 2018 GitHub outage, where a ~1-minute primary data center disconnect triggered an untested failover to a secondary...","image":"https://media.daily.dev/image/upload/s--OHB84bZF--/f_auto/v1722860399/public/Placeholder%2010","thumbnailUrl":"https://media.daily.dev/image/upload/s--OHB84bZF--/f_auto/v1722860399/public/Placeholder%2010","isAccessibleForFree":true,"articleSection":"ipSpace.net Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"ipSpace.net Blog","logo":"https://media.daily.dev/image/upload/s--xS7E82MT--/f_auto,q_auto/v1786438303/logos/ipspace?_a=BAMAMicg0","url":"https://daily.dev/sources/ipspace"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-github-learned-how-hard-distributed-systems-are-ipspace-net-blog-wy4nc42wu","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"database,distributed-systems","timeRequired":"PT1M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"ipSpace.net Blog","item":"https://daily.dev/sources/ipspace"},{"@type":"ListItem","position":3,"name":"How GitHub Learned How Hard Distributed Systems Are « ipSpace.net blog"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-github-learned-how-hard-distributed-systems-are-ipspace-net-blog-wy4nc42wu#faq","mainEntity":[{"@type":"Question","name":"What caused the GitHub outage in October 2018 and how long did it take to recover?","acceptedAnswer":{"@type":"Answer","text":"A roughly 1-minute disconnect of the primary data center triggered a failover to the secondary data center that had never been tested in production. During the failover, batch jobs continued modifying data in the primary data center, causing the two database replicas to diverge. Reconciling the out-of-sync replicas took over 24 hours to resolve. Engineers building resilient backend systems track real-world outage post-mortems like this one on daily.dev."}}]} ```