--- title: When failover isn’t safe: Building high-availability... description: Datadog engineers discovered during a gameday simulation that their Kubernetes-based PostgreSQL clusters couldn't safely fail over under zonal network failures... canonical: https://daily.dev/posts/when-failover-isn-t-safe-building-high-availability-postgresql-on-kubernetes-nljmhojvb twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: When failover isn’t safe: Building high-availability PostgreSQL on Kubernetes | daily.dev og:description: Datadog engineers discovered during a gameday simulation that their Kubernetes-based PostgreSQL clusters couldn't safely fail over under zonal network failures... og:url: https://daily.dev/posts/when-failover-isn-t-safe-building-high-availability-postgresql-on-kubernetes-nljmhojvb og:image: https://api.daily.dev/og/posts/nlJmhOJvB.png og:image:alt: When failover isn’t safe: Building high-availability PostgreSQL on Kubernetes og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # When failover isn’t safe: Building high-availability PostgreSQL on Kubernetes **[Datadog](https://daily.dev/sources/datadog)** · 18 min read · 4 upvotes · 1 comments ## Summary Datadog engineers discovered during a gameday simulation that their Kubernetes-based PostgreSQL clusters couldn't safely fail over under zonal network failures — replicas lagged too far behind the primary, leaving no safe promotion candidate. The team redesigned their setup using synchronous replication for standby nodes in the leader pool, coordinated by Patroni and ZooKeeper, while keeping read replicas on asynchronous replication. They benchmarked all synchronous_commit modes (remote_apply, on, remote_write, local), finding latency increases of 32–53% and throughput reductions of 23–34%, ultimately deploying remote_apply. The post covers five failure scenarios including loss of standbys, ZooKeeper unavailability, and leader crashes during commit, plus monitoring signals like SyncRep wait events and patroni_sync_standby metrics. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Similar posts on daily.dev - [Patroni and logical replication for PostgreSQL: Preventing data loss during failovers](https://daily.dev/posts/patroni-and-logical-replication-for-postgresql-preventing-data-loss-during-failovers-egazamrgt) · Palark Blog · 4 upvotes · 0 comments - [Managed Postgres, Examined: Azure Database for PostgreSQL Flexible Server](https://daily.dev/posts/managed-postgres-examined-azure-database-for-postgresql-flexible-server-suqnkdoou) · Planet PostgreSQL · 0 upvotes · 0 comments --- Tags: [#kubernetes](https://daily.dev/tags/kubernetes), [#postgresql](https://daily.dev/tags/postgresql) [View this post on daily.dev](https://daily.dev/posts/when-failover-isn-t-safe-building-high-availability-postgresql-on-kubernetes-nljmhojvb) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"When failover isn’t safe: Building high-availability PostgreSQL on Kubernetes","url":"https://daily.dev/posts/when-failover-isn-t-safe-building-high-availability-postgresql-on-kubernetes-nljmhojvb","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/when-failover-isn-t-safe-building-high-availability-postgresql-on-kubernetes-nljmhojvb"},"datePublished":"2026-06-05T11:58:13.117Z","dateModified":"2026-06-07T17:45:17.432Z","description":"Datadog engineers discovered during a gameday simulation that their Kubernetes-based PostgreSQL clusters couldn't safely fail over under zonal network failures...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/6bfca80114514fd5a27e138ba19901f6?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/6bfca80114514fd5a27e138ba19901f6?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Datadog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Datadog","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/8f9d7e3e4bd64bd7b9db9b2ea83e0c6a","url":"https://daily.dev/sources/datadog"},"commentCount":1,"discussionUrl":"https://daily.dev/posts/when-failover-isn-t-safe-building-high-availability-postgresql-on-kubernetes-nljmhojvb","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":4},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":1}],"keywords":"kubernetes,postgresql","timeRequired":"PT18M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Datadog","item":"https://daily.dev/sources/datadog"},{"@type":"ListItem","position":3,"name":"When failover isn’t safe: Building high-availability PostgreSQL on Kubernetes"}]} ```