--- title: How We Pushed CDC into Postgres | daily.dev description: Snowflake's engineering team describes how they built push-based change data capture (CDC) for Postgres-to-Snowflake replication via a new Postgres extension... canonical: https://daily.dev/posts/how-we-pushed-cdc-into-postgres-gk0os0oyc twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: How We Pushed CDC into Postgres | daily.dev og:description: Snowflake's engineering team describes how they built push-based change data capture (CDC) for Postgres-to-Snowflake replication via a new Postgres extension... og:url: https://daily.dev/posts/how-we-pushed-cdc-into-postgres-gk0os0oyc og:image: https://api.daily.dev/og/posts/GK0OS0oyc.png og:image:alt: How We Pushed CDC into Postgres og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # How We Pushed CDC into Postgres **[Hacker News](https://daily.dev/sources/hn)** · 10 min read · 1 upvotes · 0 comments ## Summary Snowflake's engineering team describes how they built push-based change data capture (CDC) for Postgres-to-Snowflake replication via a new Postgres extension called snowflake_cdc. Instead of the traditional pull-based logical decoding approach, changes are pushed directly from Postgres into Apache Iceberg tables in object storage, then applied transactionally in Snowflake. Key innovations include: careful timeline management across write/decode/capture/apply stages, transactional boundaries preserved on both sides using the pg_lake extension, avoidance of expensive upserts by using exact insert/delete streams, and 'live views' that combine unapplied change logs with base tables for sub-minute lag without frequent apply cycles. The result is a self-managing replication system requiring no external connectors or complex failure handling. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is Snowflake's Data Mirroring feature for Postgres and how does it work? Data Mirroring is a public preview feature in Snowflake Postgres that replicates Postgres tables into Snowflake with low cost, low lag, and transactional consistency. A Postgres extension called snowflake_cdc pushes batches of changes into per-table change logs and a meta log stored as Apache Iceberg tables using background base workers, and Snowflake applies those batches transactionally and serverlessly, without external replication connectors. _Teams weighing Postgres to Snowflake replication options can track architecture deep dives like this on daily.dev._ ### Why does push-based CDC avoid the problems of traditional pull-based Postgres logical replication? Push-based CDC avoids problems because the client consuming changes in pull-based logical decoding knows nothing about Postgres's internal state, such as schema changes, snapshot alignment, or whether Postgres or the network is down. By using a Postgres extension to push changes directly into object storage as Iceberg tables, the system that knows Postgres's state controls coordination, decoupling producer and consumer and eliminating a whole class of infrastructure failure modes. _Developers debating replication architectures can follow deep dives like this on daily.dev to inform the decision._ ### Why is the upsert approach to database replication expensive and how does transactional replication avoid it? Upsert-based replication is expensive because inserts must be matched against existing rows in the target table, which is costly with columnar storage, it can create inconsistent intermediate states, and it is hard to combine recent changes with target tables efficiently. Transactional replication controlled from Postgres instead produces a stream of deletions and insertions applied exactly once, so insert-heavy workloads are appended rather than upserted, making them extremely fast and cost-efficient. _Engineers optimizing replication pipelines can dig into architecture trade-offs like these on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 4 discussions and 22 comments across hackernews (as of 2026-09-14). **TL;DR:** Commenters find the push-based CDC approach technically interesting and appreciate the engineering detail, but raise concerns about the closed-source nature of the core CDC extension, SaaS lock-in, and note that comparable solutions already exist in the ecosystem. **Sentiment:** 45% positive · 40% mixed · 15% skeptical **The case for** - The push-based approach avoids expensive upserts by using exact insert/delete streams, which commenters recognize as a meaningful architectural improvement. - Offloading the apply work to Snowflake rather than Postgres keeps the capturing side cheap and lightweight. - The approach adds real competition to existing ETL/replication vendors like Fivetran. - ClickHouse's PeerDB acquisition is cited as a comparable success story, suggesting the market validates this direction. **The pushback** - The core CDC extension (pg_lake_repl) is closed-source and not publicly accessible, undermining the open-source framing of pg_lake. - SaaS-only database technology makes some developers deeply uncomfortable, especially smaller teams sensitive to storage and transfer costs. - Similar end-to-end replication capabilities have existed for years (Aurora zero-ETL, GoldenGate, Debezium+Delta Lake), though often with reliability issues. - Aurora zero-ETL, the closest batteries-included competitor, has been reported as unreliable in production with schema changes and unexplained outages. **By community** - hackernews (mixed): Technically engaged discussion with genuine interest in the architecture, tempered by frustration over the closed CDC repo, SaaS lock-in concerns, and comparisons to existing solutions. **Hottest debate:** Whether the CDC extension is truly open or effectively proprietary, given that the critical replication component lives in a private, inaccessible repository. **Open questions** - Will Snowflake open-source the pg_lake_repl CDC extension, or will it remain closed? - Can a Postgres-native provider (e.g., Supabase, Cloudflare) offer vendor-neutral Iceberg mirroring so users can plug in their own query engine? - How does this approach handle schema changes, which have historically been a fragile point for CDC systems? **Highlights** > The challenge is converting primary key updates/deletes to row offsets in a columnar table. That requires maintaining an expensive mapping or doing expensive scans, and is not something you want Postgres itself to do. It's also this bursty, memory-intensive workload that you'd rather not have a lot of dedicated infrastructure for. At Snowflake we use Snowflake to do the apply work. Hence end-to-end mirroring has more pieces than just Postgres, but the capturing of changes is cheap enough to do in Postgres directly. (Author) > — [mslot on hackernews · 1 comments](https://news.ycombinator.com/item?id=49242898) > Apologies, I could have introduced that better. The article links https://github.com/Snowflake-Labs/pg_lake but if you go looking for CDC, it's not there, and all you end up with is links to the private/closed repo that I just linked, in various corners and spots. The point is that all the CDC stuff is in the repo we don't get access to, that isn't open source: https://github.com/snowflake-eng/sfpg-extension-pg_lake_repl... > — [jauntywundrkind on hackernews](https://news.ycombinator.com/item?id=49242325) > Aurora zero etl was a nightmare for us. Almost any schema changes require a VACUUM FULL for it to continue functioning. On a few occasions, it just stopped running without an obvious explanation, requiring slow and lengthy back and forth threads with AWS support. If it worked as advertised, it would be great, but I can't recommend it for any serious production system. > — [bastawhiz on hackernews · 1 comments](https://news.ycombinator.com/item?id=49238811) > It's interesting to watch how different companies that offer both Postgres and warehousing solution under 1 roof approach the same problem: - ClickHouse focuses on traditional CDC (ClickPipes) and just make it blazingly fast - Databricks leans on their unified storage architecture (LTAP) to avoid copying data (though you can argue there is still a copy in the cache) - Snowflake uses a data mirroring CDC as extension so it runs directly on Postgres I'm still waiting for a Postgres provider to just let me mirror data directly to Iceberg, so I can plug in my own stateless query engine. > — [hasyimibhar on hackernews · 4 comments](https://news.ycombinator.com/item?id=49242146) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=49238050) · 108 points · 22 comments - [hackernews](https://news.ycombinator.com/item?id=49085546) · 4 points · 0 comments - [hackernews](https://news.ycombinator.com/item?id=49037876) · 4 points · 0 comments - [hackernews](https://news.ycombinator.com/item?id=49046793) · 3 points · 0 comments ## Similar posts on daily.dev - [What If You Could Skip the CDC Pipeline Entirely?](https://daily.dev/posts/what-if-you-could-skip-the-cdc-pipeline-entirely--wvg3iaboq) · Snowflake Community · 1 upvotes · 0 comments - [pg-cdc Frustratingly simple Postgres change data capture to AWS S3](https://daily.dev/posts/pg-cdc-frustratingly-simple-postgres-change-data-capture-to-aws-s3-l9lx1hrg2) · PostgreSQL · 8 upvotes · 0 comments --- Tags: [#postgresql](https://daily.dev/tags/postgresql), [#data-engineering](https://daily.dev/tags/data-engineering), [#snowflake](https://daily.dev/tags/snowflake), [#apache-iceberg](https://daily.dev/tags/apache-iceberg), [#change-data-capture](https://daily.dev/tags/change-data-capture) [View this post on daily.dev](https://daily.dev/posts/how-we-pushed-cdc-into-postgres-gk0os0oyc) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"How We Pushed CDC into Postgres","url":"https://daily.dev/posts/how-we-pushed-cdc-into-postgres-gk0os0oyc","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/how-we-pushed-cdc-into-postgres-gk0os0oyc"},"datePublished":"2026-08-10T04:05:46.004Z","dateModified":"2026-09-14T09:31:48.820Z","description":"Snowflake's engineering team describes how they built push-based change data capture (CDC) for Postgres-to-Snowflake replication via a new Postgres extension...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/de5818e8131824fce42f41cb2cd6ccee?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/de5818e8131824fce42f41cb2cd6ccee?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Hacker News","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Hacker News","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/hn","url":"https://daily.dev/sources/hn"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/how-we-pushed-cdc-into-postgres-gk0os0oyc","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"postgresql,data-engineering,snowflake,apache-iceberg,change-data-capture","timeRequired":"PT10M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Hacker News","item":"https://daily.dev/sources/hn"},{"@type":"ListItem","position":3,"name":"How We Pushed CDC into Postgres"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/how-we-pushed-cdc-into-postgres-gk0os0oyc#faq","mainEntity":[{"@type":"Question","name":"What is Snowflake's Data Mirroring feature for Postgres and how does it work?","acceptedAnswer":{"@type":"Answer","text":"Data Mirroring is a public preview feature in Snowflake Postgres that replicates Postgres tables into Snowflake with low cost, low lag, and transactional consistency. A Postgres extension called snowflake_cdc pushes batches of changes into per-table change logs and a meta log stored as Apache Iceberg tables using background base workers, and Snowflake applies those batches transactionally and serverlessly, without external replication connectors. Teams weighing Postgres to Snowflake replication options can track architecture deep dives like this on daily.dev."}},{"@type":"Question","name":"Why does push-based CDC avoid the problems of traditional pull-based Postgres logical replication?","acceptedAnswer":{"@type":"Answer","text":"Push-based CDC avoids problems because the client consuming changes in pull-based logical decoding knows nothing about Postgres's internal state, such as schema changes, snapshot alignment, or whether Postgres or the network is down. By using a Postgres extension to push changes directly into object storage as Iceberg tables, the system that knows Postgres's state controls coordination, decoupling producer and consumer and eliminating a whole class of infrastructure failure modes. Developers debating replication architectures can follow deep dives like this on daily.dev to inform the decision."}},{"@type":"Question","name":"Why is the upsert approach to database replication expensive and how does transactional replication avoid it?","acceptedAnswer":{"@type":"Answer","text":"Upsert-based replication is expensive because inserts must be matched against existing rows in the target table, which is costly with columnar storage, it can create inconsistent intermediate states, and it is hard to combine recent changes with target tables efficiently. Transactional replication controlled from Postgres instead produces a stream of deletions and insertions applied exactly once, so insert-heavy workloads are appended rather than upserted, making them extremely fast and cost-efficient. Engineers optimizing replication pipelines can dig into architecture trade-offs like these on daily.dev."}}]} ```