A collection of practical observations about software incidents: most resolve on their own, most human interventions make things worse, and the first instinct should be to do nothing and observe. Effective incident resolution usually involves simple actions like disabling a feature flag, and success depends more on system knowledge than technical brilliance. The post also covers the political dynamics of incident response — resolving incidents earns goodwill from leadership, but being the go-to incident resolver is not a durable career strategy because executives can't distinguish heroic effort from obvious fixes.
Questions this post answers
Should I immediately try to fix things when I join an incident call or wait and observe first?
The first action in an incident should be nothing. Most incidents resolve on their own — well over half, by one experienced engineer's estimate — and hasty interventions frequently make things worse. Classic examples include clearing a queue that contained critical billing jobs, or forcing redeploys that stress the system more than the original problem. Waiting, observing, and understanding the situation before acting is the safer default. Engineers navigating on-call decisions track hard-won incident wisdom like this on daily.dev.
What kinds of actions actually resolve software incidents in practice?
Incident-resolving actions are almost always simple: disabling a problematic feature flag, reverting a recent code change, or wrapping something in a cache. The complexity is not in the fix itself but in knowing which lever to pull — which is why an engineer familiar with the codebase outperforms a group of strong engineers who aren't. The typical pattern is five minutes writing the patch and an hour waiting for review, CI, and deploy. Developers building on-call intuition for their systems find real-world incident patterns discussed on daily.dev.
75K Impressions2 Comments