--- title: We Rolled Back in Four Minutes. The Feature Flag Kept... description: A postmortem-style essay recounts a production incident where a Kubernetes rollback restored old code in four minutes, but the feature flag stayed on and... canonical: https://daily.dev/posts/we-rolled-back-in-four-minutes-the-feature-flag-kept-the-broken-path-alive--vl7onztmu twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: We Rolled Back in Four Minutes. The Feature Flag Kept the Broken Path Alive. | daily.dev og:description: A postmortem-style essay recounts a production incident where a Kubernetes rollback restored old code in four minutes, but the feature flag stayed on and... og:url: https://daily.dev/posts/we-rolled-back-in-four-minutes-the-feature-flag-kept-the-broken-path-alive--vl7onztmu og:image: https://api.daily.dev/og/posts/vl7ONZtMu.png og:image:alt: We Rolled Back in Four Minutes. The Feature Flag Kept the Broken Path Alive. og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # We Rolled Back in Four Minutes. The Feature Flag Kept the Broken Path Alive. **[Medium](https://daily.dev/sources/medium_js)** · 4 min read · 0 upvotes · 0 comments ## Summary A postmortem-style essay recounts a production incident where a Kubernetes rollback restored old code in four minutes, but the feature flag stayed on and continued routing customers into a workflow the old code couldn't handle. The piece argues that deployments are only one of several independently changing layers (flags, schema, queues, caches, clients), and that a real rollback plan must account for state that has already moved forward, not just the application binary. It closes with a promoted link to a paid 'Production Engineering Library' resource. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Why did my application still behave incorrectly after rolling back a Kubernetes deployment during an incident? Rolling back the application binary does not rewind other layers that changed behavior, such as feature flag configuration, database state, queued messages, and client caches. In one incident, a Kubernetes rollback completed in four minutes but customers kept hitting a broken workflow because the feature flag was still enabled and had already written state the old code could not process correctly. _Teams debugging rollback gaps like this compare production incident writeups on daily.dev._ ### Why are feature flags risky during a production rollback? Feature flags can act as a second, uncoordinated deployment system: they change production behavior instantly with a single click, but often lack the review, ownership, and rollback plan attached to code releases. During an incident, one person may roll back the application while assuming the flag will follow automatically, when the two systems are actually independent. _Engineers designing safer release processes track feature flag pitfalls like these on daily.dev._ ### What should a production rollback plan account for besides restoring the old application version? A complete rollback plan should separate three actions: stopping new exposure, restoring compatible code, and reconciling state created during the exposure window. It should also document which flags changed with the release, who owns disabling them, whether disabling repairs existing state, what queued work survives rollback, and which clients have cached the new behavior. _Teams writing incident runbooks like this reference production rollback strategies on daily.dev._ ## Similar posts on daily.dev - [Why Blue-Green Deployments Fail at Scale in Kubernetes — and What Works Instead](https://daily.dev/posts/why-blue-green-deployments-fail-at-scale-in-kubernetes-and-what-works-instead-slwbapfq7) · Cloud Native Now · 23 upvotes · 1 comments - [Automating Global Rollback for 1.5 Trillion Requests in 10 Minutes](https://daily.dev/posts/automating-global-rollback-for-1-5-trillion-requests-in-10-minutes-eiiupn5j1) · Salesforce Engineering · 30 upvotes · 1 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#distributed-systems](https://daily.dev/tags/distributed-systems) [View this post on daily.dev](https://daily.dev/posts/we-rolled-back-in-four-minutes-the-feature-flag-kept-the-broken-path-alive--vl7onztmu) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"We Rolled Back in Four Minutes. The Feature Flag Kept the Broken Path Alive.","url":"https://daily.dev/posts/we-rolled-back-in-four-minutes-the-feature-flag-kept-the-broken-path-alive--vl7onztmu","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/we-rolled-back-in-four-minutes-the-feature-flag-kept-the-broken-path-alive--vl7onztmu"},"datePublished":"2026-09-02T03:45:36.136Z","dateModified":"2026-09-02T03:49:48.590Z","description":"A postmortem-style essay recounts a production incident where a Kubernetes rollback restored old code in four minutes, but the feature flag stayed on and...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/8874f2f759c82df6149e9b8ea2bb20eb?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/8874f2f759c82df6149e9b8ea2bb20eb?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/we-rolled-back-in-four-minutes-the-feature-flag-kept-the-broken-path-alive--vl7onztmu","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,distributed-systems","timeRequired":"PT4M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"We Rolled Back in Four Minutes. The Feature Flag Kept the Broken Path Alive."}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/we-rolled-back-in-four-minutes-the-feature-flag-kept-the-broken-path-alive--vl7onztmu#faq","mainEntity":[{"@type":"Question","name":"Why did my application still behave incorrectly after rolling back a Kubernetes deployment during an incident?","acceptedAnswer":{"@type":"Answer","text":"Rolling back the application binary does not rewind other layers that changed behavior, such as feature flag configuration, database state, queued messages, and client caches. In one incident, a Kubernetes rollback completed in four minutes but customers kept hitting a broken workflow because the feature flag was still enabled and had already written state the old code could not process correctly. Teams debugging rollback gaps like this compare production incident writeups on daily.dev."}},{"@type":"Question","name":"Why are feature flags risky during a production rollback?","acceptedAnswer":{"@type":"Answer","text":"Feature flags can act as a second, uncoordinated deployment system: they change production behavior instantly with a single click, but often lack the review, ownership, and rollback plan attached to code releases. During an incident, one person may roll back the application while assuming the flag will follow automatically, when the two systems are actually independent. Engineers designing safer release processes track feature flag pitfalls like these on daily.dev."}},{"@type":"Question","name":"What should a production rollback plan account for besides restoring the old application version?","acceptedAnswer":{"@type":"Answer","text":"A complete rollback plan should separate three actions: stopping new exposure, restoring compatible code, and reconciling state created during the exposure window. It should also document which flags changed with the release, who owns disabling them, whether disabling repairs existing state, what queued work survives rollback, and which clients have cached the new behavior. Teams writing incident runbooks like this reference production rollback strategies on daily.dev."}}]} ```