--- title: Our Runbook Had 43 Steps. Step 17 Assumed the Old... description: An incident response failed at step seventeen of a 43-step runbook because it instructed responders to drain traffic through a load balancer that had been... canonical: https://daily.dev/posts/our-runbook-had-43-steps-step-17-assumed-the-old-cluster-still-existed--ycgkp1tid twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Our Runbook Had 43 Steps. Step 17 Assumed the Old Cluster Still Existed. | daily.dev og:description: An incident response failed at step seventeen of a 43-step runbook because it instructed responders to drain traffic through a load balancer that had been... og:url: https://daily.dev/posts/our-runbook-had-43-steps-step-17-assumed-the-old-cluster-still-existed--ycgkp1tid og:image: https://api.daily.dev/og/posts/YcGKP1tID.png og:image:alt: Our Runbook Had 43 Steps. Step 17 Assumed the Old Cluster Still Existed. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Our Runbook Had 43 Steps. Step 17 Assumed the Old Cluster Still Existed. **[Medium](https://daily.dev/sources/medium_js)** · 4 min read · 0 upvotes · 0 comments ## Summary An incident response failed at step seventeen of a 43-step runbook because it instructed responders to drain traffic through a load balancer that had been decommissioned five months earlier during an architecture migration. The piece argues that runbooks decay silently since documents keep rendering correctly even after the underlying infrastructure changes, unlike code which fails loudly. The team's fix was replacing the bloated 43-step document with fourteen verified actions plus decision points, extracting executable checks (resource existence, command resolution, permissions, rehearsable traffic movement) that get validated on a schedule and tested in game days, rather than trusting memory or prose. The core lesson: documentation is operational infrastructure with its own dependencies and failure modes, and every architecture change should ask which recovery assumption it invalidated. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### why does an incident runbook fail even though it passed review and looks up to date A runbook can fail because it suffers from semantic drift: the underlying infrastructure changed (cluster names, traffic management, permissions, failover processes) but the recovery document's steps still describe the old system. Documents keep rendering correctly with no visible sign of decay, unlike code which fails loudly when a dependency disappears, so reviewers can read the whole page without ever executing its assumptions. _daily.dev helps ops teams keep tabs on infrastructure changes that quietly invalidate recovery runbooks._ ### how do you keep a runbook from going stale after a production architecture migration Extract the runbook's factual assumptions into executable checks instead of leaving them as prose: verify required resources exist, commands resolve against current tooling, dashboards and ownership records are reachable, operator permissions are sufficient, traffic movement can be rehearsed against a non-production target, and recovery artifacts remain available. Run these checks on a schedule and validate the full sequence in game days. _engineers refining incident processes track patterns like runbook validation on daily.dev._ ### how many steps should an incident runbook have Fewer than you'd think: one team cut a 43-step runbook down to fourteen verified actions plus several decision points after discovering the long version was detailed enough to discourage rehearsal and fragile enough to break after ordinary infrastructure changes. The shorter version states the goal, expected evidence, and stopping condition for each action so responders can notice when reality diverges instead of blindly executing stale commands. _daily.dev surfaces production engineering lessons for teams rethinking bloated incident documentation._ ## Similar posts on daily.dev - [Disaster Recovery is a Process, Not a Tool \(Part 2\)](https://daily.dev/posts/disaster-recovery-is-a-process-not-a-tool-part-2--vft09yg96) · Planet PostgreSQL · 3 upvotes · 0 comments - [Beyond the Runbook: How to Scale SRE Operations for Cloud-Native Infrastructure](https://daily.dev/posts/beyond-the-runbook-how-to-scale-sre-operations-for-cloud-native-infrastructure-gmmvjkfsg) · Cloud Native Now · 0 upvotes · 0 comments - [How We Turned Data Engineering Runbooks Into Reliable AI Skills](https://daily.dev/posts/how-we-turned-data-engineering-runbooks-into-reliable-ai-skills-t0at3fj35) · Halodoc · 2 upvotes · 0 comments --- [View this post on daily.dev](https://daily.dev/posts/our-runbook-had-43-steps-step-17-assumed-the-old-cluster-still-existed--ycgkp1tid) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Our Runbook Had 43 Steps. Step 17 Assumed the Old Cluster Still Existed.","url":"https://daily.dev/posts/our-runbook-had-43-steps-step-17-assumed-the-old-cluster-still-existed--ycgkp1tid","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/our-runbook-had-43-steps-step-17-assumed-the-old-cluster-still-existed--ycgkp1tid"},"datePublished":"2026-09-02T03:47:55.255Z","dateModified":"2026-09-02T03:50:05.516Z","description":"An incident response failed at step seventeen of a 43-step runbook because it instructed responders to drain traffic through a load balancer that had been...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5bb4c29dd1088d31b9f28a9dcab0f00c?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5bb4c29dd1088d31b9f28a9dcab0f00c?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/our-runbook-had-43-steps-step-17-assumed-the-old-cluster-still-existed--ycgkp1tid","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"","timeRequired":"PT4M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"Our Runbook Had 43 Steps. Step 17 Assumed the Old Cluster Still Existed."}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/our-runbook-had-43-steps-step-17-assumed-the-old-cluster-still-existed--ycgkp1tid#faq","mainEntity":[{"@type":"Question","name":"why does an incident runbook fail even though it passed review and looks up to date","acceptedAnswer":{"@type":"Answer","text":"A runbook can fail because it suffers from semantic drift: the underlying infrastructure changed (cluster names, traffic management, permissions, failover processes) but the recovery document's steps still describe the old system. Documents keep rendering correctly with no visible sign of decay, unlike code which fails loudly when a dependency disappears, so reviewers can read the whole page without ever executing its assumptions. daily.dev helps ops teams keep tabs on infrastructure changes that quietly invalidate recovery runbooks."}},{"@type":"Question","name":"how do you keep a runbook from going stale after a production architecture migration","acceptedAnswer":{"@type":"Answer","text":"Extract the runbook's factual assumptions into executable checks instead of leaving them as prose: verify required resources exist, commands resolve against current tooling, dashboards and ownership records are reachable, operator permissions are sufficient, traffic movement can be rehearsed against a non-production target, and recovery artifacts remain available. Run these checks on a schedule and validate the full sequence in game days. engineers refining incident processes track patterns like runbook validation on daily.dev."}},{"@type":"Question","name":"how many steps should an incident runbook have","acceptedAnswer":{"@type":"Answer","text":"Fewer than you'd think: one team cut a 43-step runbook down to fourteen verified actions plus several decision points after discovering the long version was detailed enough to discourage rehearsal and fragile enough to break after ordinary infrastructure changes. The shorter version states the goal, expected evidence, and stopping condition for each action so responders can notice when reality diverges instead of blindly executing stale commands. daily.dev surfaces production engineering lessons for teams rethinking bloated incident documentation."}}]} ```