--- title: Delta Lake Merge | daily.dev description: A detailed walkthrough of the Delta Lake MERGE command, covering INSERT, UPDATE, and DELETE operations within a single transaction using the PySpark API. It... canonical: https://daily.dev/posts/delta-lake-merge-b1pkolxis twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Delta Lake Merge | daily.dev og:description: A detailed walkthrough of the Delta Lake MERGE command, covering INSERT, UPDATE, and DELETE operations within a single transaction using the PySpark API. It... og:url: https://daily.dev/posts/delta-lake-merge-b1pkolxis og:image: https://api.daily.dev/og/posts/B1PKolxIS.png og:image:alt: Delta Lake Merge og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Delta Lake Merge **[Delta Lake Blog](https://daily.dev/sources/delta-lake)** · 14 min read · 0 upvotes · 0 comments ## Summary A detailed walkthrough of the Delta Lake MERGE command, covering INSERT, UPDATE, and DELETE operations within a single transaction using the PySpark API. It progresses from simple whenNotMatchedInsert examples through whenMatchedUpdate and whenMatchedDelete, contrasts merge with manually emulating the same logic on legacy Hive-style Parquet tables, and shows how to handle change data capture scenarios with differing schemas and partial (missing-value) updates. Includes example code and links to official documentation and helper libraries. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I use whenNotMatchedInsert with Delta Lake merge in PySpark? Call merge() on the target Delta table with a join condition against the source DataFrame, then chain whenNotMatchedInsert with a dictionary mapping target fields to source expressions to insert only rows that don't already exist. A shorthand, whenNotMatchedInsertAll, inserts all source fields automatically when source and target schemas match, avoiding the need to list each field manually. _Developers building CDC pipelines can track Delta Lake merge patterns and tips on daily.dev._ ### How do I merge partial change data with missing columns into a Delta Lake table? Use the whenMatchedUpdate clause with an explicit dictionary that checks whether each source value is present, falling back to the existing target value when it is null. Passing arbitrary expressions instead of plain field names lets a single merge query handle rows where only some columns, like name or age, were sent by an upstream change data capture system. _Engineers handling change data capture edge cases can find more Delta Lake guidance on daily.dev._ ### Why is Delta Lake merge better than manually merging Parquet tables? Delta Lake merge avoids rewriting the entire table and prevents the error-prone manual work required with Hive-style Parquet tables, where you must prefix columns, perform a full outer join, write per-column conditional expressions, and rewrite the whole dataset to a new location because you cannot overwrite a table while reading it. This manual approach is slow and hard to verify, especially at terabyte scale. _Anyone weighing Delta Lake against plain Parquet workflows can compare approaches on daily.dev._ --- Tags: [#data-engineering](https://daily.dev/tags/data-engineering), [#apache-spark](https://daily.dev/tags/apache-spark), [#etl](https://daily.dev/tags/etl), [#pyspark](https://daily.dev/tags/pyspark) [View this post on daily.dev](https://daily.dev/posts/delta-lake-merge-b1pkolxis) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Delta Lake Merge","url":"https://daily.dev/posts/delta-lake-merge-b1pkolxis","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/delta-lake-merge-b1pkolxis"},"datePublished":"2026-08-16T08:53:36.537Z","dateModified":"2026-08-16T08:57:19.244Z","description":"A detailed walkthrough of the Delta Lake MERGE command, covering INSERT, UPDATE, and DELETE operations within a single transaction using the PySpark API. It...","image":"https://media.daily.dev/image/upload/s--P4t4XyoV--/f_auto/v1722860399/public/Placeholder%2001","thumbnailUrl":"https://media.daily.dev/image/upload/s--P4t4XyoV--/f_auto/v1722860399/public/Placeholder%2001","isAccessibleForFree":true,"articleSection":"Delta Lake Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Delta Lake Blog","logo":"https://media.daily.dev/image/upload/s--7f2q9Asf--/f_auto,q_auto/v1786870393/logos/delta-lake?_a=BAMAMicg0","url":"https://daily.dev/sources/delta-lake"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/delta-lake-merge-b1pkolxis","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-engineering,apache-spark,etl,pyspark","timeRequired":"PT14M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Delta Lake Blog","item":"https://daily.dev/sources/delta-lake"},{"@type":"ListItem","position":3,"name":"Delta Lake Merge"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/delta-lake-merge-b1pkolxis#faq","mainEntity":[{"@type":"Question","name":"How do I use whenNotMatchedInsert with Delta Lake merge in PySpark?","acceptedAnswer":{"@type":"Answer","text":"Call merge() on the target Delta table with a join condition against the source DataFrame, then chain whenNotMatchedInsert with a dictionary mapping target fields to source expressions to insert only rows that don't already exist. A shorthand, whenNotMatchedInsertAll, inserts all source fields automatically when source and target schemas match, avoiding the need to list each field manually. Developers building CDC pipelines can track Delta Lake merge patterns and tips on daily.dev."}},{"@type":"Question","name":"How do I merge partial change data with missing columns into a Delta Lake table?","acceptedAnswer":{"@type":"Answer","text":"Use the whenMatchedUpdate clause with an explicit dictionary that checks whether each source value is present, falling back to the existing target value when it is null. Passing arbitrary expressions instead of plain field names lets a single merge query handle rows where only some columns, like name or age, were sent by an upstream change data capture system. Engineers handling change data capture edge cases can find more Delta Lake guidance on daily.dev."}},{"@type":"Question","name":"Why is Delta Lake merge better than manually merging Parquet tables?","acceptedAnswer":{"@type":"Answer","text":"Delta Lake merge avoids rewriting the entire table and prevents the error-prone manual work required with Hive-style Parquet tables, where you must prefix columns, perform a full outer join, write per-column conditional expressions, and rewrite the whole dataset to a new location because you cannot overwrite a table while reading it. This manual approach is slow and hard to verify, especially at terabyte scale. Anyone weighing Delta Lake against plain Parquet workflows can compare approaches on daily.dev."}}]} ```