An analysis of GitHub's latest major outage, which lasted nearly eight hours and stemmed from network saturation on Central US load balancers during a traffic peak. The cascade began with an Istio sidecar pod hitting concurrency limits due to a misconfigured autoscaling policy, which propagated to HAProxy nodes exhausting flow limits and breaking the gateway auth path. Optimistic retry logic worsened the overload, and a previously unknown VS Code retry bug amplified traffic to the Copilot Token Service by roughly 10x (from 7-9K RPS to 70-100K RPS). Responders mitigated by shifting traffic regions, pausing HAProxy nodes, changing retry logic via PR, and blocking token requests before gradually restoring traffic. The piece critiques the write-up for omitting diagnostic details and architectural diagrams, and argues that GitHub's follow-up actions focus too narrowly on preventing recurrence of this specific failure rather than building general capability to handle novel incidents.
Table of contents
Saturation strikes yet againDiagnostic details: missing in actionRetries made things worseThe Copilot Token Service sees 10X trafficMitigating the incident: multiple strategies“Never again” means never preparing for a novel incidentQuestions this post answers
What caused the eight-hour GitHub outage involving Copilot authentication failures?
Network saturation on Central US load balancers during a traffic peak triggered a cascading failure. An Istio sidecar pod hit concurrency limits because an autoscaling policy watched host service limits but not sidecar limits, which cascaded until four HAProxy nodes exhausted flow limits, degrading the gateway auth path and causing widespread authentication failures across GitHub and Copilot. daily.dev surfaces incident writeups like this for engineers building resilience into their own auth and gateway paths.
Why did a VS Code bug amplify Copilot Token Service traffic during an outage?
A latent retry bug in Visual Studio Code triggered when delayed replies came from a single internal endpoint, amplifying traffic by roughly 10x and delaying recovery of the Copilot Token Service. Traffic to that service jumped from a normal 7-9K requests per second to 70-100K RPS, and since the client couldn't be patched mid-incident, GitHub mitigated it server-side by blocking token requests at the load balancer. engineers debugging retry storms in production can track how others diagnosed and mitigated similar cascades via daily.dev.
How did GitHub mitigate the load balancer saturation incident without waiting for a client-side fix?
Responders shifted traffic from the Central US region to Northern Virginia, paused HAProxy on the four saturated nodes, changed gateway retry logic through a pull request, blocked inbound Copilot Token Service requests at the load balancer with 403 responses, and then gradually ramped the blocked traffic back up once the system stabilized. daily.dev helps SREs compare mitigation playbooks for saturation and retry-storm incidents across companies.