--- title: Nubank migrates 135M-customer payments to Aurora,... description: Nubank moved NuPay off self-managed Postgres to Aurora PostgreSQL, cutting worst-case query time from 13 minutes to seconds and replication lag from 4 minutes... canonical: https://daily.dev/posts/nubank-migrates-135m-customer-payments-to-aurora-postgres-skip-locked-hits-its-limits-i3gt4iebi twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Nubank migrates 135M-customer payments to Aurora, Postgres SKIP LOCKED hits its limits | daily.dev og:description: Nubank moved NuPay off self-managed Postgres to Aurora PostgreSQL, cutting worst-case query time from 13 minutes to seconds and replication lag from 4 minutes... og:url: https://daily.dev/posts/nubank-migrates-135m-customer-payments-to-aurora-postgres-skip-locked-hits-its-limits-i3gt4iebi og:image: https://api.daily.dev/og/posts/I3gt4iebi.png og:image:alt: Nubank migrates 135M-customer payments to Aurora, Postgres SKIP LOCKED hits its limits og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Nubank migrates 135M-customer payments to Aurora, Postgres SKIP LOCKED hits its limits **[Databases Digest](https://daily.dev/sources/databases_digest)** · 5 min read · 1 upvotes · 0 comments ## Summary Nubank moved NuPay off self-managed Postgres to Aurora PostgreSQL, cutting worst-case query time from 13 minutes to seconds and replication lag from 4 minutes to a few seconds. A hands-on account of running a Postgres job queue with SKIP LOCKED shows it works fine at steady low rates but collapses under spike load — heap churn from status transitions outpaces autovacuum at 400 jobs/sec. Supabase Pipelines enters public alpha with Postgres CDC to BigQuery, schema change detection, and usage-based pricing. AlloyDB's columnar engine now accelerates pgvector HNSW search by up to 4-5x with no application changes required. ## Content **TLDR:** Nubank moved NuPay off self-managed Postgres to Aurora PostgreSQL, cutting worst-case query time from 13 minutes to seconds and replication lag from 4 minutes to a few seconds. A hands-on account of running a Postgres job queue with SKIP LOCKED shows it works fine at steady low rates but collapses under spike load — heap churn from status transitions outpaces autovacuum at 400 jobs/sec. Supabase Pipelines enters public alpha with Postgres CDC to BigQuery, schema change detection, and usage-based pricing. AlloyDB's columnar engine now accelerates pgvector HNSW search by up to 4-5x with no application changes required. --- ## Nubank migrates NuPay payments from self-managed Postgres to Aurora PostgreSQL Nubank, serving 135 million customers, hit a wall with self-managed Postgres: queries taking over 13 minutes, replication lag exceeding 4 minutes at peak, and storage expansions requiring downtime. Aurora's log-based writes and shared storage architecture delivered up to 1,900x query improvement in specific cases and 15-20% average improvement across production workloads. Replication lag dropped to a few seconds. The migration used AWS DMS with 10 parallel tasks and CDC, with the actual cutover completing in just over an hour against a 5-hour maintenance window. Operational wins included maintenance windows shrinking from 11 hours to 1-1.5 hours and a 25% cost reduction by consolidating from three instances to two. [Read more](https://daily.dev/posts/uHKWMXH9s) ## Postgres SKIP LOCKED job queue works until it doesn't — 400 jobs/sec exposes the limits Three weeks of promising results with a Postgres-backed job queue using SELECT FOR UPDATE SKIP LOCKED: low claim latency, zero duplicate sends, $180/month saved over Redis. Week four hit a marketing spike at ~400 jobs/sec, producing a 650k-row backlog, autovacuum in continuous catch-up mode, primary CPU at 88%, and checkout API latency climbing. The late-night incident involved 4.2M dead tuples and connection pool exhaustion before an emergency rollback. The resolution was a hybrid: transactional order-linked jobs stay in Postgres at a sustained ~12/sec, while burst email and webhook fan-out moved back to Redis Streams. The core lesson is that heap churn from status transitions outpaces autovacuum under spike conditions, and sharing a jobs table with an OLTP primary creates correlated failure risk. [Read more](https://daily.dev/posts/SNI9qhS1s) ## Supabase Pipelines enters public alpha: Postgres CDC to BigQuery with schema change support Supabase Pipelines now replicates Postgres data to BigQuery in near real-time using logical replication, with the underlying engine being an open-source Rust-based CDC pipeline. The headline addition in public alpha is automatic schema change detection — adding, removing, or renaming columns and changing nullability or defaults no longer requires manual intervention. Initial sync is now parallelized across and within tables. Pricing is usage-based: $0.053/hour per active pipeline, $0.60/GB for initial copy, $3/GB for ongoing replication. ClickHouse, Snowflake, and DuckLake are next on the destination list. [Read more](https://daily.dev/posts/8Mkx7whU4) ## AlloyDB columnar engine delivers 4-5x faster pgvector HNSW search, no code changes AlloyDB's Columnar Engine now pins HNSW indexes directly into a specialized in-memory columnar format, bypassing the standard PostgreSQL buffer manager overhead — page pinning, lock acquisition, LRU management. Benchmarks on the GloVe 100 Angular dataset (1M+ records) show 4x-4.9x higher queries per second versus standard Postgres HNSW. At a fixed QPS of ~350, enabling the columnar engine also improves recall from 0.78 to over 0.94. Enabling requires setting two flags and calling a single SQL function. Currently in preview on AlloyDB. [Read more](https://daily.dev/posts/TKk8HPCfr) --- ## Also notable - **PostgreSQL file_extend_method GUC: when to change it from the default:** PostgreSQL 16+ defaults to posix_fallocate for file extension, which is faster and reduces fragmentation, but silently disables compression on Btrfs and triggered spurious 'no space left' errors on some XFS kernel versions — change it to write_zeros only if you're on one of those affected setups. [Read more](https://daily.dev/posts/2xiGzVfM6) - **pglayers: compose Postgres extensions via Docker COPY instead of compiling in Dockerfiles:** pglayers publishes each extension (pgvector, PostGIS, pg_cron, TimescaleDB) as a minimal FROM-scratch image containing only .so files and SQL scripts, relying on Postgres 18's new extension_control_path GUC to avoid file collisions — the tradeoff is ABI compatibility requirements against Debian Trixie/glibc 2.38 and supply-chain trust for pre-built binaries. [Read more](https://daily.dev/posts/DlBXESZJt) - **Weaviate query profiling available since v1.36.9, GA in v1.38:** Weaviate's per-query profiling flag returns per-stage, per-shard timing breakdowns inline on the response — covering hydration, filter cardinality, HNSW layer traversal, and compression rescore latency — without requiring environment variable changes or node restarts. [Read more](https://daily.dev/posts/EOSyw6Lmm) - **OAPE PostgreSQL DBA-1 certification exam debuts at PGConf.EU 2026 in Valencia:** The Open Alliance for PostgreSQL Education, a vendor-neutral non-profit, will hold its first community-owned PostgreSQL DBA-1 exam at PGConf.EU 2026 in Valencia — the first independent alternative to vendor-backed Postgres certifications. [Read more](https://daily.dev/posts/ecnFst4hw) - **Airbyte to dltHub migration: 99% cost reduction claim backed by compute-based pricing model:** dltHub claims 99.83% pipeline success rate across 50M production runs and compute-based pricing that can cut costs by 99%+ versus Airbyte's row/GB metering, with migration estimated at roughly one connector per day for senior engineers. [Read more](https://daily.dev/posts/TY2k0Oy70) ## Similar posts on daily.dev - [Migrating mission-critical payments at Nubank to Amazon Aurora PostgreSQL](https://daily.dev/posts/migrating-mission-critical-payments-at-nubank-to-amazon-aurora-postgresql-uhkwmxh9s) · AWS Database Blog · 5 upvotes · 0 comments - [I Tried the “Just Use Postgres” Queue for Six Weeks. SKIP LOCKED Paid Until It Didn’t](https://daily.dev/posts/i-tried-the-just-use-postgres-queue-for-six-weeks-skip-locked-paid-until-it-didn-t-sni9qhs1s) · Medium · 2 upvotes · 0 comments --- Tags: [#aws](https://daily.dev/tags/aws), [#database](https://daily.dev/tags/database), [#postgresql](https://daily.dev/tags/postgresql) [View this post on daily.dev](https://daily.dev/posts/nubank-migrates-135m-customer-payments-to-aurora-postgres-skip-locked-hits-its-limits-i3gt4iebi) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/nubank-migrates-135m-customer-payments-to-aurora-postgres-skip-locked-hits-its-limits-i3gt4iebi","headline":"Nubank migrates 135M-customer payments to Aurora, Postgres SKIP LOCKED hits its limits","text":"Nubank moved NuPay off self-managed Postgres to Aurora PostgreSQL, cutting worst-case query time from 13 minutes to seconds and replication lag from 4 minutes to a few seconds. A hands-on account of running a Postgres job queue with SKIP LOCKED shows it works fine at steady low rates but collapses under spike load — heap churn from status transitions outpaces autovacuum at 400 jobs/sec. Supabase Pipelines enters public alpha with Postgres CDC to BigQuery, schema change detection, and usage-based pricing. AlloyDB's columnar engine now accelerates pgvector HNSW search by up to 4-5x with no application changes required.","url":"https://daily.dev/posts/nubank-migrates-135m-customer-payments-to-aurora-postgres-skip-locked-hits-its-limits-i3gt4iebi","datePublished":"2026-07-22T04:17:59.191Z","dateModified":"2026-09-13T21:06:57.024Z","author":{"@type":"Organization","name":"Databases Digest","logo":"https://media.daily.dev/image/upload/s--ir1hdV8b--/f_auto,q_auto/v1773839400/logos/databases_digest?_a=BAMAMiiu0","url":"https://daily.dev/sources/databases_digest"},"interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/databases_digest","name":"Databases Digest"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Databases Digest","item":"https://daily.dev/sources/databases_digest"},{"@type":"ListItem","position":3,"name":"Nubank migrates 135M-customer payments to Aurora, Postgres SKIP LOCKED hits its limits"}]} ```