--- title: Recovering a status page from a half-finished schema... description: A status page went down for 38 minutes after a Helm upgrade triggered a Postgres schema migration that was OOMKilled mid-run, leaving the database in a dirty... canonical: https://daily.dev/posts/recovering-a-status-page-from-a-half-finished-schema-migration-azumdrs2q twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Recovering a status page from a half-finished schema migration | daily.dev og:description: A status page went down for 38 minutes after a Helm upgrade triggered a Postgres schema migration that was OOMKilled mid-run, leaving the database in a dirty... og:url: https://daily.dev/posts/recovering-a-status-page-from-a-half-finished-schema-migration-azumdrs2q og:image: https://api.daily.dev/og/posts/AzumdRs2Q.png og:image:alt: Recovering a status page from a half-finished schema migration og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Recovering a status page from a half-finished schema migration **[Medium](https://daily.dev/sources/medium_js)** · 11 min read · 1 upvotes · 0 comments ## Summary A status page went down for 38 minutes after a Helm upgrade triggered a Postgres schema migration that was OOMKilled mid-run, leaving the database in a dirty state (version 23 marked applied but no DDL committed). Both the new and rolled-back binaries refused to start. Rather than restoring from a 6-hour-old backup and losing production uptime data, the team manually completed the three DDL statements from migration 0023 inside a single transaction with IF NOT EXISTS guards, then cleared the dirty flag in schema_migrations. The page was restored in 51 minutes with all data intact. Post-incident, they added a pre-upgrade Helm hook that snapshots the schema to object storage and increased migration job memory limits to prevent future OOMKills. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What does it mean when Postgres migrations report 'database schema version 23 is dirty, refusing to start'? It means a migration started running and never reported success, typically because the process was killed mid-run (e.g. OOMKilled) after the migration tool wrote its version row but before, or between, the actual DDL statements committed. The bookkeeping table says the migration is applied, but the real table structure may only be partially updated, so the application refuses to start until the mismatch is resolved. _Teams debugging dirty migration flags can find grounded incident walkthroughs like this one on daily.dev._ ### Should I restore from backup or manually finish a partial database migration after a failed deploy? Finishing the migration by hand is often the better choice when the failed migration is small, structural (not a bulk data rewrite), and the backup would lose valuable recent data. In one recovery, a team skipped a 6-hour-stale backup restore and instead reran three missing DDL statements with IF NOT EXISTS guards inside a single transaction, then cleared the dirty flag, completing recovery in 51 minutes with no data loss. _Weighing rollback versus manual recovery for a migration failure is easier with real incident writeups on daily.dev._ ### How can I prevent a Kubernetes migration job from being OOMKilled during a database schema migration? Set migration job resource requests and limits based on the largest migration observed in staging plus roughly a 50% buffer, since a too-low memory limit (such as 256Mi against an actual need of 380Mi) can kill the job mid-transaction and leave the schema in a dirty, partially-migrated state. Removing the liveness probe from migration jobs also lets a slow migration finish or fail on its own terms instead of being killed by Kubernetes. _Engineers hardening migration jobs against OOM kills can track these kinds of Kubernetes lessons on daily.dev._ ## Similar posts on daily.dev - [Zero-Downtime Postgres Migrations: A 2026 Developer Guide](https://daily.dev/posts/zero-downtime-postgres-migrations-a-2026-developer-guide-drbnzykbh) · Alex CloudStar · 0 upvotes · 0 comments - [Inside a PostgreSQL Checkpointer Bug: A Production Postmortem](https://daily.dev/posts/inside-a-postgresql-checkpointer-bug-a-production-postmortem-48qml9fr8) · Planet PostgreSQL · 17 upvotes · 0 comments - [Inside branch-based restores in Lakebase Postgres](https://daily.dev/posts/inside-branch-based-restores-in-lakebase-postgres-udhwx0m8z) · Neon · 1 upvotes · 1 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#database](https://daily.dev/tags/database), [#postgresql](https://daily.dev/tags/postgresql), [#helm](https://daily.dev/tags/helm) [View this post on daily.dev](https://daily.dev/posts/recovering-a-status-page-from-a-half-finished-schema-migration-azumdrs2q) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Recovering a status page from a half-finished schema migration","url":"https://daily.dev/posts/recovering-a-status-page-from-a-half-finished-schema-migration-azumdrs2q","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/recovering-a-status-page-from-a-half-finished-schema-migration-azumdrs2q"},"datePublished":"2026-07-16T19:32:58.586Z","dateModified":"2026-09-14T09:04:07.667Z","description":"A status page went down for 38 minutes after a Helm upgrade triggered a Postgres schema migration that was OOMKilled mid-run, leaving the database in a dirty...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/69debaa1e14cdbf3b3f260e11c942cdc?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/69debaa1e14cdbf3b3f260e11c942cdc?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Medium","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Medium","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/medium","url":"https://daily.dev/sources/medium_js"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/recovering-a-status-page-from-a-half-finished-schema-migration-azumdrs2q","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"kubernetes,database,postgresql,helm","timeRequired":"PT11M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Medium","item":"https://daily.dev/sources/medium_js"},{"@type":"ListItem","position":3,"name":"Recovering a status page from a half-finished schema migration"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/recovering-a-status-page-from-a-half-finished-schema-migration-azumdrs2q#faq","mainEntity":[{"@type":"Question","name":"What does it mean when Postgres migrations report 'database schema version 23 is dirty, refusing to start'?","acceptedAnswer":{"@type":"Answer","text":"It means a migration started running and never reported success, typically because the process was killed mid-run (e.g. OOMKilled) after the migration tool wrote its version row but before, or between, the actual DDL statements committed. The bookkeeping table says the migration is applied, but the real table structure may only be partially updated, so the application refuses to start until the mismatch is resolved. Teams debugging dirty migration flags can find grounded incident walkthroughs like this one on daily.dev."}},{"@type":"Question","name":"Should I restore from backup or manually finish a partial database migration after a failed deploy?","acceptedAnswer":{"@type":"Answer","text":"Finishing the migration by hand is often the better choice when the failed migration is small, structural (not a bulk data rewrite), and the backup would lose valuable recent data. In one recovery, a team skipped a 6-hour-stale backup restore and instead reran three missing DDL statements with IF NOT EXISTS guards inside a single transaction, then cleared the dirty flag, completing recovery in 51 minutes with no data loss. Weighing rollback versus manual recovery for a migration failure is easier with real incident writeups on daily.dev."}},{"@type":"Question","name":"How can I prevent a Kubernetes migration job from being OOMKilled during a database schema migration?","acceptedAnswer":{"@type":"Answer","text":"Set migration job resource requests and limits based on the largest migration observed in staging plus roughly a 50% buffer, since a too-low memory limit (such as 256Mi against an actual need of 380Mi) can kill the job mid-transaction and leave the schema in a dirty, partially-migrated state. Removing the liveness probe from migration jobs also lets a slow migration finish or fail on its own terms instead of being killed by Kubernetes. Engineers hardening migration jobs against OOM kills can track these kinds of Kubernetes lessons on daily.dev."}}]} ```