--- title: Taking AUTO CDC to the next level: Solving the hardest... description: AUTO CDC in Apache Spark Declarative Pipelines now supports three advanced capabilities: Bitemporal CDC, Partial Updates, and open-source contributions to... canonical: https://daily.dev/posts/taking-auto-cdc-to-the-next-level-solving-the-hardest-real-world-use-cases-h2x6t6wdg twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Taking AUTO CDC to the next level: Solving the hardest real-world use cases | daily.dev og:description: AUTO CDC in Apache Spark Declarative Pipelines now supports three advanced capabilities: Bitemporal CDC, Partial Updates, and open-source contributions to... og:url: https://daily.dev/posts/taking-auto-cdc-to-the-next-level-solving-the-hardest-real-world-use-cases-h2x6t6wdg og:image: https://api.daily.dev/og/posts/h2x6T6Wdg.png og:image:alt: Taking AUTO CDC to the next level: Solving the hardest real-world use cases og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Taking AUTO CDC to the next level: Solving the hardest real-world use cases **[databricks](https://daily.dev/sources/databricks)** · 9 min read · 0 upvotes · 0 comments ## Summary AUTO CDC in Apache Spark Declarative Pipelines now supports three advanced capabilities: Bitemporal CDC, Partial Updates, and open-source contributions to Apache Spark 4.2. Bitemporal CDC tracks two independent timelines — business time and system time — using four system-managed columns, enabling point-in-time reconstruction for compliance use cases like SEC Rule 17a-4 and FINRA recordkeeping. Partial Updates allow CDC sources that emit only changed fields (with NULLs for unchanged columns) to update only specified columns, preventing accidental overwrites. The Python API for AUTO CDC Type 1 has been contributed to Apache Spark 4.2 as open source, with SQL syntax and SCD Type 2 support planned for future releases. The post includes code examples in both SQL and PySpark, MLflow integration patterns for ML reproducibility using bitemporal history, and links to documentation. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is bitemporal AUTO CDC in Spark Declarative Pipelines and how does it differ from standard SCD Type 2? Bitemporal AUTO CDC tracks two independent timelines instead of one: business time (when a fact changed in the real world) and system time (when the system learned about it). Each target table gets four system-managed columns, __START_AT/__END_AT for business time and __SYSTEM_START_AT/__SYSTEM_END_AT for system time, enabling point-in-time reconstruction along either axis, unlike standard SCD Type 2 which only tracks one timeline. _Teams building audit-ready CDC pipelines can follow feature rollouts like bitemporal AUTO CDC on daily.dev._ ### How does AutoCDC Partial Updates prevent NULL values from overwriting existing column data during CDC merges? Partial Updates lets update events modify only a subset of columns, treating NULLs in selected columns as 'do not update' rather than as overwrites. For example, if a target row is (1, 'A', 20) and an incoming update is (1, NULL, 30), enabling Partial Updates results in (1, 'A', 30) instead of erasing the existing name value, which is common with CDC sources that only emit changed fields. _Engineers hitting NULL-overwrite bugs in CDC pipelines can track fixes like partial updates on daily.dev._ ### Is AUTO CDC being contributed to open-source Apache Spark and which version will include it? Yes, the Python API for AUTO CDC Type 1 is being contributed to open-source Apache Spark, submitted through the standard SPIP and pull request process (SPARK-56249) rather than a single code drop, and is slated to ship in the next Apache Spark release after 4.2. It works with both Delta Lake and Apache Iceberg since it builds on Spark's streaming and table abstractions rather than a storage format. _Developers evaluating open-source CDC tooling can track Apache Spark's AUTO CDC contribution on daily.dev._ ## Similar posts on daily.dev - [Stop Hand-Coding Change Data Capture Pipelines](https://daily.dev/posts/stop-hand-coding-change-data-capture-pipelines-wnvuu7hry) · databricks · 0 upvotes · 0 comments - [Apache Flink CDC 3.6.0 Release Announcement](https://daily.dev/posts/apache-flink-cdc-3-6-0-release-announcement-siricutsr) · Flink · 1 upvotes · 0 comments - [How Spyne simplified their CDC pipeline with ClickPipes and ClickHouse Cloud](https://daily.dev/posts/how-spyne-simplified-their-cdc-pipeline-with-clickpipes-and-clickhouse-cloud-xtczu2jzw) · ClickHouse · 0 upvotes · 0 comments - [How We Pushed CDC into Postgres](https://daily.dev/posts/how-we-pushed-cdc-into-postgres-gk0os0oyc) · Hacker News · 1 upvotes · 0 comments --- Tags: [#apache-spark](https://daily.dev/tags/apache-spark), [#databricks](https://daily.dev/tags/databricks), [#change-data-capture](https://daily.dev/tags/change-data-capture) [View this post on daily.dev](https://daily.dev/posts/taking-auto-cdc-to-the-next-level-solving-the-hardest-real-world-use-cases-h2x6t6wdg) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Taking AUTO CDC to the next level: Solving the hardest real-world use cases","url":"https://daily.dev/posts/taking-auto-cdc-to-the-next-level-solving-the-hardest-real-world-use-cases-h2x6t6wdg","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/taking-auto-cdc-to-the-next-level-solving-the-hardest-real-world-use-cases-h2x6t6wdg"},"datePublished":"2026-08-11T22:48:26.071Z","dateModified":"2026-09-13T21:46:08.369Z","description":"AUTO CDC in Apache Spark Declarative Pipelines now supports three advanced capabilities: Bitemporal CDC, Partial Updates, and open-source contributions to...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c9041f75b32e35762d93743664f58769?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/c9041f75b32e35762d93743664f58769?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"databricks","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"databricks","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/fa7aa720f2db4d1eba826814730482c8","url":"https://daily.dev/sources/databricks"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/taking-auto-cdc-to-the-next-level-solving-the-hardest-real-world-use-cases-h2x6t6wdg","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"apache-spark,databricks,change-data-capture","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"databricks","item":"https://daily.dev/sources/databricks"},{"@type":"ListItem","position":3,"name":"Taking AUTO CDC to the next level: Solving the hardest real-world use cases"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/taking-auto-cdc-to-the-next-level-solving-the-hardest-real-world-use-cases-h2x6t6wdg#faq","mainEntity":[{"@type":"Question","name":"What is bitemporal AUTO CDC in Spark Declarative Pipelines and how does it differ from standard SCD Type 2?","acceptedAnswer":{"@type":"Answer","text":"Bitemporal AUTO CDC tracks two independent timelines instead of one: business time (when a fact changed in the real world) and system time (when the system learned about it). Each target table gets four system-managed columns, __START_AT/__END_AT for business time and __SYSTEM_START_AT/__SYSTEM_END_AT for system time, enabling point-in-time reconstruction along either axis, unlike standard SCD Type 2 which only tracks one timeline. Teams building audit-ready CDC pipelines can follow feature rollouts like bitemporal AUTO CDC on daily.dev."}},{"@type":"Question","name":"How does AutoCDC Partial Updates prevent NULL values from overwriting existing column data during CDC merges?","acceptedAnswer":{"@type":"Answer","text":"Partial Updates lets update events modify only a subset of columns, treating NULLs in selected columns as 'do not update' rather than as overwrites. For example, if a target row is (1, 'A', 20) and an incoming update is (1, NULL, 30), enabling Partial Updates results in (1, 'A', 30) instead of erasing the existing name value, which is common with CDC sources that only emit changed fields. Engineers hitting NULL-overwrite bugs in CDC pipelines can track fixes like partial updates on daily.dev."}},{"@type":"Question","name":"Is AUTO CDC being contributed to open-source Apache Spark and which version will include it?","acceptedAnswer":{"@type":"Answer","text":"Yes, the Python API for AUTO CDC Type 1 is being contributed to open-source Apache Spark, submitted through the standard SPIP and pull request process (SPARK-56249) rather than a single code drop, and is slated to ship in the next Apache Spark release after 4.2. It works with both Delta Lake and Apache Iceberg since it builds on Spark's streaming and table abstractions rather than a storage format. Developers evaluating open-source CDC tooling can track Apache Spark's AUTO CDC contribution on daily.dev."}}]} ```