Analyzing GitHub's recent outage, the piece examines a misconfigured autoscaling policy that monitored host service metrics but not Istio sidecar concurrency limits, causing saturation. The author uses this as a springboard to discuss autoscaling fundamentals (CPU-based vs. request-based vs. thread-pool saturation metrics, citing Slack's 2021 outage as a parallel) and then critiques the tendency to fixate on a single defective component after an incident. Drawing on David Woods' 'component substitution fallacy,' the argument is that systems are always full of latent defects yet usually don't fail, so resilience analysis should focus on interactions between factors (traffic changes, retries, HAProxy saturation, authentication load) rather than blaming one broken part.
Table of contents
Share this:Questions this post answers
What caused the recent GitHub outage involving Istio sidecars?
An Istio sidecar pod reached its concurrency limits and failed to autoscale correctly because the autoscaling policy monitored the host service's load metrics but did not account for load on the sidecar itself. This mismatch meant the system didn't add capacity in time, contributing to saturation during the incident, alongside factors like changing traffic patterns, retry logic, and HAProxy node saturation. Teams debugging autoscaling incidents can follow real-world postmortems like this on daily.dev.
Why can a service become saturated even when CPU utilization is low?
A service can saturate under low CPU when using thread-per-request with a thread pool: if downstream request latency increases, threads block waiting on I/O rather than consuming CPU, so all pool threads get tied up despite low CPU readings. This happened to Slack in its January 2021 outage, prompting them to add autoscaling rules based on thread count rather than relying solely on CPU metrics. daily.dev helps engineers building resilient services track lessons from real production incidents like this one.
What is the component substitution fallacy in incident analysis?
It is the mistaken belief, described by David Woods, that improving system reliability just means finding and fixing the specific defective component blamed for an outage. Systems always contain latent component defects yet are not constantly failing, meaning defects alone rarely cause outages; interactions between multiple factors, such as traffic shifts, retry logic, and infrastructure saturation, deserve first-class analysis. Engineers refining their incident review process can find frameworks like this discussed on daily.dev.