--- title: Postgres autovacuum incident costs $19K, Spark 4.2 ships... description: A production postmortem details how 847 million dead tuples and default autovacuum settings caused a 38-minute checkout outage on Black Friday, with $19,200 in... canonical: https://daily.dev/posts/postgres-autovacuum-incident-costs-19k-spark-4-2-ships-native-vector-search-hkyukkmw8 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Postgres autovacuum incident costs $19K, Spark 4.2 ships native vector search | daily.dev og:description: A production postmortem details how 847 million dead tuples and default autovacuum settings caused a 38-minute checkout outage on Black Friday, with $19,200 in... og:url: https://daily.dev/posts/postgres-autovacuum-incident-costs-19k-spark-4-2-ships-native-vector-search-hkyukkmw8 og:image: https://api.daily.dev/og/posts/HKYUkkMW8.png og:image:alt: Postgres autovacuum incident costs $19K, Spark 4.2 ships native vector search og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Postgres autovacuum incident costs $19K, Spark 4.2 ships native vector search **[Databases Digest](https://daily.dev/sources/databases_digest)** · 4 min read · 0 upvotes · 0 comments ## Summary A production postmortem details how 847 million dead tuples and default autovacuum settings caused a 38-minute checkout outage on Black Friday, with $19,200 in abandoned carts. Apache Spark 4.2 lands in Databricks Runtime 19 Beta with native vector similarity search, Auto CDC, and direct interop with Polars and DuckDB. Netflix's RocksDB SST-based pipeline migration cut deployment time from days to 30 minutes and infrastructure costs by 70%. A Redis Sentinel failover with 40 seconds of replica lag left 9,200 cart sessions reading stale data for an hour. ## Content **TLDR:** A production postmortem details how 847 million dead tuples and default autovacuum settings caused a 38-minute checkout outage on Black Friday, with $19,200 in abandoned carts. Apache Spark 4.2 lands in Databricks Runtime 19 Beta with native vector similarity search, Auto CDC, and direct interop with Polars and DuckDB. Netflix's RocksDB SST-based pipeline migration cut deployment time from days to 30 minutes and infrastructure costs by 70%. A Redis Sentinel failover with 40 seconds of replica lag left 9,200 cart sessions reading stale data for an hour. --- ## Postgres autovacuum 11 days behind triggers 38-minute checkout outage, $19,200 in lost carts Default autovacuum scale factors let an order_items table accumulate 847 million dead tuples and 186 GB of heap bloat over 11 days. A routine bulk UPDATE during Black Friday prep hit that table and locked checkout for 38 minutes — 1,847 failed checkouts, roughly $19,200 in abandoned carts. Root causes: scale_factor too coarse for high-churn tables, cost throttling letting workers fall behind, and long-running analytics queries blocking vacuum. The fix required per-table tuning (scale_factor down to 0.02), Grafana alerts on dead tuple ratios and last_autovacuum age, and a manual VACUUM that ran over 4 hours to bring the dead ratio from 412:1 down to 0.03:1. If you're running high-churn tables on default autovacuum settings, this postmortem is worth reading before your own Black Friday. [Read more](https://daily.dev/posts/e5bUzaGBR) ## Apache Spark 4.2 ships native vector search, Auto CDC, and zero-copy Polars/DuckDB interop Spark 4.2 is out in Databricks Runtime 19 Beta with 1,900+ commits from 260 contributors. The headline is a native NEAREST BY SQL operator for vector similarity search — if you're running a separate vector database purely for semantic search over embeddings, this is worth evaluating. Arrow-optimized UDFs are now on by default, and the Arrow C Data Interface means Spark can exchange data with Polars and DuckDB without serialization overhead. Auto CDC in Spark Declarative Pipelines adds a declarative Python API for SCD Type 1 ingestion, which removes a fair amount of plumbing that teams have historically had to wire up themselves. [Read more](https://daily.dev/posts/xWZHYAayK) ## Netflix migrates 50 TB via RocksDB SST files, cuts deployment time from days to 30 minutes and costs by 70% Netflix redesigned its data ingestion pipeline using RocksDB SST files to move 50 TB without disrupting production. Deployment time dropped from days to 30 minutes and infrastructure costs fell 70%. The SST file approach sidesteps the write path entirely during bulk loads, which is the key reason the numbers are this dramatic. Worth understanding if you're dealing with large-scale data migrations where write amplification is the bottleneck. [Read more](https://daily.dev/posts/rt5Ihg3mF) ## Redis Sentinel failover with 40-second replica lag leaves 9,200 cart sessions stale for an hour With the primary still alive but slow (94% disk I/O wait from an AOF rewrite), the team promoted a replica carrying 40 seconds of lag. The result: 9,200 cart sessions read stale data for an hour, $23,400 in failed checkouts, and $3,800 in chargebacks. The postmortem covers the exact replication offset checks that should have run before promotion, why infra metrics going green masked ongoing product damage, and the fixes shipped afterward — lag-based failover gates, cart key versioning, read-your-writes for checkout, and quarterly failover drills with data verification. [Read more](https://daily.dev/posts/sxSANdObS) --- ## Also notable - **pgrust Rust rewrite of Postgres passes 100% of regression tests, follow-on claims 50% faster on transactional workloads:** pgrust, built using Claude Opus at a cost of $100K, has passed all Postgres regression tests; a follow-up called PG Rust Fast ($200K invested) claims 50% faster than Postgres on transactional workloads and comparable to ClickHouse on analytical ones. [Read more](https://daily.dev/posts/H7lGxRmlo) - **apitap 0.6.0 benchmarks 10M Postgres rows into ClickHouse in 1.59 seconds:** apitap 0.6.0 adds MySQL-to-MySQL transfer via LOAD DATA LOCAL INFILE with parallel range-partitioned pipes, and a reproducible open-source benchmark shows 10 million Postgres rows into ClickHouse in 1.59 seconds, though MySQL as a source takes 168 seconds for the same row count. [Read more](https://daily.dev/posts/KgGqPpUvf) - **PostgreSQL extra_float_digits: the old advice to set it to 3 is obsolete since PG12:** Since PostgreSQL 12, the default value of 1 uses a shortest-decimal algorithm that is both human-readable and exactly round-trips, making the old recommendation to set extra_float_digits=3 a compatibility habit rather than a requirement. [Read more](https://daily.dev/posts/xFMHD6Yck) - **RustFS vs. Garage: Rust-based S3-compatible alternatives to MinIO after its licensing shift:** RustFS targets high-performance data lake workloads (Iceberg, DuckDB) with Erasure Coding and strong consistency via Conditional PUT; Garage uses 3-way replication and CRDT-based eventual consistency for cold storage on cheap hardware, but lacks Conditional PUT support needed for data lake compatibility. [Read more](https://daily.dev/posts/EK5fJWlkH) ## Similar posts on daily.dev - [Postgres Autovacuum Fell Behind for 11 Days. Then One UPDATE Locked Checkout](https://daily.dev/posts/postgres-autovacuum-fell-behind-for-11-days-then-one-update-locked-checkout-e5buzagbr) · Medium · 15 upvotes · 0 comments --- Tags: [#backend](https://daily.dev/tags/backend), [#postgresql](https://daily.dev/tags/postgresql), [#redis](https://daily.dev/tags/redis), [#apache-spark](https://daily.dev/tags/apache-spark) [View this post on daily.dev](https://daily.dev/posts/postgres-autovacuum-incident-costs-19k-spark-4-2-ships-native-vector-search-hkyukkmw8) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"DiscussionForumPosting","mainEntityOfPage":"https://daily.dev/posts/postgres-autovacuum-incident-costs-19k-spark-4-2-ships-native-vector-search-hkyukkmw8","headline":"Postgres autovacuum incident costs $19K, Spark 4.2 ships native vector search","text":"A production postmortem details how 847 million dead tuples and default autovacuum settings caused a 38-minute checkout outage on Black Friday, with $19,200 in abandoned carts. Apache Spark 4.2 lands in Databricks Runtime 19 Beta with native vector similarity search, Auto CDC, and direct interop with Polars and DuckDB. Netflix's RocksDB SST-based pipeline migration cut deployment time from days to 30 minutes and infrastructure costs by 70%. A Redis Sentinel failover with 40 seconds of replica lag left 9,200 cart sessions reading stale data for an hour.","url":"https://daily.dev/posts/postgres-autovacuum-incident-costs-19k-spark-4-2-ships-native-vector-search-hkyukkmw8","datePublished":"2026-07-20T04:20:41.419Z","dateModified":"2026-07-20T04:20:59.662Z","author":{"@type":"Organization","name":"Databases Digest","logo":"https://media.daily.dev/image/upload/s--ir1hdV8b--/f_auto,q_auto/v1773839400/logos/databases_digest?_a=BAMAMiiu0","url":"https://daily.dev/sources/databases_digest"},"interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"isPartOf":{"@type":"WebPage","url":"https://daily.dev/sources/databases_digest","name":"Databases Digest"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Databases Digest","item":"https://daily.dev/sources/databases_digest"},{"@type":"ListItem","position":3,"name":"Postgres autovacuum incident costs $19K, Spark 4.2 ships native vector search"}]} ```