--- title: Unlocking real-time analytics: Streaming Aurora DSQL... description: A walkthrough demonstrates streaming Amazon Aurora DSQL change data capture (CDC) events into Apache Iceberg tables stored on Amazon S3 via Amazon Data... canonical: https://daily.dev/posts/unlocking-real-time-analytics-streaming-aurora-dsql-changes-into-apache-iceberg-ntedgboh3 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Unlocking real-time analytics: Streaming Aurora DSQL changes into Apache Iceberg | daily.dev og:description: A walkthrough demonstrates streaming Amazon Aurora DSQL change data capture (CDC) events into Apache Iceberg tables stored on Amazon S3 via Amazon Data... og:url: https://daily.dev/posts/unlocking-real-time-analytics-streaming-aurora-dsql-changes-into-apache-iceberg-ntedgboh3 og:image: https://api.daily.dev/og/posts/NteDgboH3.png og:image:alt: Unlocking real-time analytics: Streaming Aurora DSQL changes into Apache Iceberg og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Unlocking real-time analytics: Streaming Aurora DSQL changes into Apache Iceberg **[AWS Database Blog](https://daily.dev/sources/aws-database-blog)** · 6 min read · 0 upvotes · 0 comments ## Summary A walkthrough demonstrates streaming Amazon Aurora DSQL change data capture (CDC) events into Apache Iceberg tables stored on Amazon S3 via Amazon Data Firehose, queryable through Amazon Athena. The design uses two Iceberg tables: an append-only cdc_events audit trail ordered by transaction commit timestamp, and a current_state table that merges rows by primary key with tombstone flags for deletes. It covers handling out-of-order CDC delivery, reconstructing a strictly correct current state from cdc_events, deployment via a CloudFormation stack, and exploring results with a Streamlit dashboard. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I handle DELETE events when streaming Aurora DSQL CDC into an Iceberg current-state table? Delete events are recorded as tombstone rows rather than hard deletes. An AWS Lambda transformation function sets an _is_deleted flag to true on the row, and because the DSQL delete record only carries primary key columns, all non-key columns in the tombstone row become NULL. Consumers filter with WHERE _is_deleted = false to get the live current state. _Teams designing CDC pipelines can track patterns like this on daily.dev before hitting the same ordering pitfalls._ ### Why would a current_state table built from Aurora DSQL CDC events show a deleted row reappearing? Because Amazon Data Firehose applies upserts in the order it processes records, out-of-order delivery can let a late UPDATE overwrite a newer one or un-tombstone a previously deleted row. The tombstone prevents the delete from being lost in the audit trail, but does not stop a stale event from reviving the row in current_state; a strictly correct state must be reconstructed from cdc_events using the latest event per id by commit timestamp. _daily.dev helps engineers building CDC pipelines stay ahead of ordering and consistency gotchas like this._ ### How can I get a strictly correct current state from an unordered CDC stream with no end-of-transaction markers? Reconstruct it from an append-only audit table by taking the latest event per row id ordered by the transaction commit timestamp, since Aurora DSQL guarantees a total order of transactions via that timestamp. This differs from a merge-by-primary-key table, which applies upserts in arrival order and can be corrupted by out-of-order delivery. _daily.dev keeps developers building real-time analytics pipelines current on techniques like transaction ordering._ ## Similar posts on daily.dev - [Amazon Aurora PostgreSQL now supports direct querying of Apache Iceberg and Parquet data in your data lake](https://daily.dev/posts/amazon-aurora-postgresql-now-supports-direct-querying-of-apache-iceberg-and-parquet-data-in-your-dat-lh81d12gd) · AWS · 1 upvotes · 0 comments - [Implementing real-time change data capture with Debezium for Amazon Aurora PostgreSQL and Amazon RDS for PostgreSQL](https://daily.dev/posts/implementing-real-time-change-data-capture-with-debezium-for-amazon-aurora-postgresql-and-amazon-rds-wwimwgweo) · AWS Database Blog · 0 upvotes · 0 comments - [pg-cdc Frustratingly simple Postgres change data capture to AWS S3](https://daily.dev/posts/pg-cdc-frustratingly-simple-postgres-change-data-capture-to-aws-s3-l9lx1hrg2) · PostgreSQL · 8 upvotes · 0 comments - [The Battle for Compute: Unlocked by Iceberg V3 and Snowflake Interoperability](https://daily.dev/posts/the-battle-for-compute-unlocked-by-iceberg-v3-and-snowflake-interoperability-o4erztdpv) · Snowflake Community · 0 upvotes · 0 comments - [Bootstrap a Data Lakehouse in an Afternoon](https://daily.dev/posts/bootstrap-a-data-lakehouse-in-an-afternoon-co4kw9dyi) · Towards Data Science · 0 upvotes · 0 comments --- Tags: [#aws](https://daily.dev/tags/aws), [#apache-iceberg](https://daily.dev/tags/apache-iceberg), [#data-streaming](https://daily.dev/tags/data-streaming), [#change-data-capture](https://daily.dev/tags/change-data-capture) [View this post on daily.dev](https://daily.dev/posts/unlocking-real-time-analytics-streaming-aurora-dsql-changes-into-apache-iceberg-ntedgboh3) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Unlocking real-time analytics: Streaming Aurora DSQL changes into Apache Iceberg","url":"https://daily.dev/posts/unlocking-real-time-analytics-streaming-aurora-dsql-changes-into-apache-iceberg-ntedgboh3","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/unlocking-real-time-analytics-streaming-aurora-dsql-changes-into-apache-iceberg-ntedgboh3"},"datePublished":"2026-08-17T23:38:09.422Z","dateModified":"2026-08-17T23:38:33.382Z","description":"A walkthrough demonstrates streaming Amazon Aurora DSQL change data capture (CDC) events into Apache Iceberg tables stored on Amazon S3 via Amazon Data...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/797f5fc3b823d132ab0c058a34caabd5?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/797f5fc3b823d132ab0c058a34caabd5?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"AWS Database Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"AWS Database Blog","logo":"https://media.daily.dev/image/upload/s--En-avd2a--/f_auto,q_auto/v1780213446/logos/aws-database-blog?_a=BAMAMiWQ0","url":"https://daily.dev/sources/aws-database-blog"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/unlocking-real-time-analytics-streaming-aurora-dsql-changes-into-apache-iceberg-ntedgboh3","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"aws,apache-iceberg,data-streaming,change-data-capture","timeRequired":"PT6M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"AWS Database Blog","item":"https://daily.dev/sources/aws-database-blog"},{"@type":"ListItem","position":3,"name":"Unlocking real-time analytics: Streaming Aurora DSQL changes into Apache Iceberg"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/unlocking-real-time-analytics-streaming-aurora-dsql-changes-into-apache-iceberg-ntedgboh3#faq","mainEntity":[{"@type":"Question","name":"How do I handle DELETE events when streaming Aurora DSQL CDC into an Iceberg current-state table?","acceptedAnswer":{"@type":"Answer","text":"Delete events are recorded as tombstone rows rather than hard deletes. An AWS Lambda transformation function sets an _is_deleted flag to true on the row, and because the DSQL delete record only carries primary key columns, all non-key columns in the tombstone row become NULL. Consumers filter with WHERE _is_deleted = false to get the live current state. Teams designing CDC pipelines can track patterns like this on daily.dev before hitting the same ordering pitfalls."}},{"@type":"Question","name":"Why would a current_state table built from Aurora DSQL CDC events show a deleted row reappearing?","acceptedAnswer":{"@type":"Answer","text":"Because Amazon Data Firehose applies upserts in the order it processes records, out-of-order delivery can let a late UPDATE overwrite a newer one or un-tombstone a previously deleted row. The tombstone prevents the delete from being lost in the audit trail, but does not stop a stale event from reviving the row in current_state; a strictly correct state must be reconstructed from cdc_events using the latest event per id by commit timestamp. daily.dev helps engineers building CDC pipelines stay ahead of ordering and consistency gotchas like this."}},{"@type":"Question","name":"How can I get a strictly correct current state from an unordered CDC stream with no end-of-transaction markers?","acceptedAnswer":{"@type":"Answer","text":"Reconstruct it from an append-only audit table by taking the latest event per row id ordered by the transaction commit timestamp, since Aurora DSQL guarantees a total order of transactions via that timestamp. This differs from a merge-by-primary-key table, which applies upserts in arrival order and can be corrupted by out-of-order delivery. daily.dev keeps developers building real-time analytics pipelines current on techniques like transaction ordering."}}]} ```