A comprehensive step-by-step guide to setting up multi-region PostgreSQL disaster recovery and failback using Crunchy PostgreSQL Operator (PGO) on Kubernetes with MinIO as the WAL archive backend. Covers the full lifecycle: deploying an NGINX reverse proxy to satisfy pgBackRest's HTTPS requirement for MinIO, configuring primary and standby clusters across two regions, simulating a regional failure, promoting the standby to primary (with Patroni timeline switching), and the complex failback process. The failback section details how to avoid operator deadlocks and S3 archive poisoning caused by timeline conflicts, including cleaning up stale PVCs, purging the MinIO bucket, reinitializing the pgBackRest stanza, and redeploying the original primary as a standby on the new timeline.
Table of contents
Phase 1: The Secure TLS Gateway (NGINX Reverse Proxy)Generate the Self-Signed CertificateDeploy the NGINX Proxy ConfigurationPhase 2: Initial Multi-Region SetupPhase 3: Verifying Replication FlowPhase 4: Disaster Recovery (Active Failover)Reversing Roles & Failback (The Hard Part)Step 4: Re-apply the East Cluster Manifest as a StandbyStep 5: Verify the Completed DR LoopSummary & Key TakeawaysQuestions this post answers
How do I configure pgBackRest to use MinIO over HTTPS when MinIO doesn't have native TLS enabled?
Deploy an NGINX reverse proxy in the same namespace as MinIO that terminates TLS on port 443 using a self-signed certificate and forwards plain HTTP to MinIO on port 9000. Set `client_max_body_size 0`, `proxy_buffering off`, and `proxy_request_buffering off` in the NGINX config to handle large backup streams. In the pgBackRest config, set `repo1-s3-verify-tls: "n"` to skip certificate validation while still using HTTPS. Engineers wiring up pgBackRest to self-hosted object storage track solutions like this on daily.dev.
Why does my Crunchy PGO standby cluster fail to rejoin after a failover and how do I fix the timeline conflict?
After failover, the old primary may have written Timeline 3 metadata to the S3 archive while the promoted standby operates on Timeline 2. PGO refuses to delete unhealthy old StatefulSets, and the standby cannot replicate from a master on Timeline 2 when Timeline 3 exists in the archive. Fix it by deleting the PostgresCluster, all pods, PVCs, ConfigMaps, and endpoints in the old region, purging the MinIO bucket, running `pgbackrest stanza-create` and a full backup from the new primary, then redeploying the old region with `standby.enabled: true`. Teams running PostgreSQL failover in production find runbooks like this through daily.dev before an incident forces the search.
How do I promote a Crunchy PGO standby cluster to primary during a disaster recovery event?
Patch the standby's PostgresCluster manifest to set `spec.standby.enabled` to `false`. Patroni detects the change, breaks out of continuous archive-recovery mode, and initiates a timeline switch — moving from Timeline 1 to Timeline 2 — making the cluster a writable primary. First shut down the original primary by patching `spec.shutdown: true` to prevent split-brain. PostgreSQL operators handling DR runbooks share and refine procedures like this on daily.dev.