--- title: Delta Lake Small File Compaction with OPTIMIZE | daily.dev description: Explains how Delta Lake's OPTIMIZE command compacts small data files into larger ones to reduce I/O overhead and metadata bloat that slow down queries. Covers... canonical: https://daily.dev/posts/delta-lake-small-file-compaction-with-optimize-ge4tjyxuk twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Delta Lake Small File Compaction with OPTIMIZE | daily.dev og:description: Explains how Delta Lake's OPTIMIZE command compacts small data files into larger ones to reduce I/O overhead and metadata bloat that slow down queries. Covers... og:url: https://daily.dev/posts/delta-lake-small-file-compaction-with-optimize-ge4tjyxuk og:image: https://api.daily.dev/og/posts/GE4tJYXuK.png og:image:alt: Delta Lake Small File Compaction with OPTIMIZE og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Delta Lake Small File Compaction with OPTIMIZE **[Delta Lake Blog](https://daily.dev/sources/delta-lake)** · 10 min read · 0 upvotes · 0 comments ## Summary Explains how Delta Lake's OPTIMIZE command compacts small data files into larger ones to reduce I/O overhead and metadata bloat that slow down queries. Covers a hands-on example with OPTIMIZE, predicate filtering to compact only recent partitions, the default 1GB target file size (configurable via spark.databricks.delta.optimize.maxFileSize), manual compaction workarounds for pre-1.2 versions, causes of small files (user error, over-partitioning, frequent incremental updates), the latency-vs-small-files tradeoff in ETL design, and why Delta Lake's ACID transactions and dataChange flag make compaction safer than on plain data lakes. Also contrasts OPTIMIZE with Z ORDER. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What is the default target file size when running OPTIMIZE on a Delta Lake table? Delta Lake targets 1 GB files by default when OPTIMIZE runs, a size chosen from years of customer usage showing it performs well on common compute instances. You can override it with the spark.databricks.delta.optimize.maxFileSize property if you have a specific reason to use a different size, but the default is recommended in most cases. _Anyone tuning Delta Lake compaction settings can track best practices for OPTIMIZE on daily.dev._ ### Which Delta Lake version introduced the OPTIMIZE command for compacting small files? OPTIMIZE was added to Delta Lake in version 1.2, with Python bindings following in version 2.0. Before that, small files had to be compacted manually by rewriting the entire dataset with a specified number of output files, an approach that is inefficient and error-prone compared to using OPTIMIZE. _Teams planning a Delta Lake upgrade can follow version-specific feature changes like this on daily.dev._ ### What does the dataChange flag mean in a Delta Lake transaction log entry after running OPTIMIZE? The dataChange flag is set to false for all files added and removed during an OPTIMIZE compaction, signaling that no new data was ingested, only existing data rearranged into larger files. This lets downstream jobs watching the table distinguish compacted files from genuinely new data and avoid unnecessary reprocessing. _Engineers building incremental pipelines on Delta Lake can dig into flags like this on daily.dev._ ## Similar posts on daily.dev - [Apache Iceberg Compaction Best Practices](https://daily.dev/posts/apache-iceberg-compaction-best-practices-60tdqenf7) · ITNEXT · 7 upvotes · 0 comments - [Optimizing our Datalake's Raw Data Layer by Smart Storage](https://daily.dev/posts/optimizing-our-datalake-s-raw-data-layer-by-smart-storage-wlnbqtoz5) · Halodoc · 2 upvotes · 0 comments - [Delta Lake Explained: Boost Data Reliability in Cloud Storage](https://daily.dev/posts/delta-lake-explained-boost-data-reliability-in-cloud-storage-oechbmvm4) · databricks · 0 upvotes · 0 comments - [Iceberg Lake for Data Analytics: Optimization Guide](https://daily.dev/posts/iceberg-lake-for-data-analytics-optimization-guide-kmewy4cgp) · ITNEXT · 1 upvotes · 0 comments - [Introducing Target File Size and Partition Writes for Snowflake Managed and Externally Managed Iceberg](https://daily.dev/posts/introducing-target-file-size-and-partition-writes-for-snowflake-managed-and-externally-managed-icebe-ogfzjrr8q) · Snowflake Community · 1 upvotes · 0 comments --- Tags: [#data-engineering](https://daily.dev/tags/data-engineering), [#apache-spark](https://daily.dev/tags/apache-spark), [#etl](https://daily.dev/tags/etl) [View this post on daily.dev](https://daily.dev/posts/delta-lake-small-file-compaction-with-optimize-ge4tjyxuk) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Delta Lake Small File Compaction with OPTIMIZE","url":"https://daily.dev/posts/delta-lake-small-file-compaction-with-optimize-ge4tjyxuk","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/delta-lake-small-file-compaction-with-optimize-ge4tjyxuk"},"datePublished":"2026-08-16T08:53:39.549Z","dateModified":"2026-08-16T08:58:06.915Z","description":"Explains how Delta Lake's OPTIMIZE command compacts small data files into larger ones to reduce I/O overhead and metadata bloat that slow down queries. Covers...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/012ce8c379ce98a647e20d105524b220?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/012ce8c379ce98a647e20d105524b220?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Delta Lake Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Delta Lake Blog","logo":"https://media.daily.dev/image/upload/s--7f2q9Asf--/f_auto,q_auto/v1786870393/logos/delta-lake?_a=BAMAMicg0","url":"https://daily.dev/sources/delta-lake"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/delta-lake-small-file-compaction-with-optimize-ge4tjyxuk","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-engineering,apache-spark,etl","timeRequired":"PT10M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Delta Lake Blog","item":"https://daily.dev/sources/delta-lake"},{"@type":"ListItem","position":3,"name":"Delta Lake Small File Compaction with OPTIMIZE"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/delta-lake-small-file-compaction-with-optimize-ge4tjyxuk#faq","mainEntity":[{"@type":"Question","name":"What is the default target file size when running OPTIMIZE on a Delta Lake table?","acceptedAnswer":{"@type":"Answer","text":"Delta Lake targets 1 GB files by default when OPTIMIZE runs, a size chosen from years of customer usage showing it performs well on common compute instances. You can override it with the spark.databricks.delta.optimize.maxFileSize property if you have a specific reason to use a different size, but the default is recommended in most cases. Anyone tuning Delta Lake compaction settings can track best practices for OPTIMIZE on daily.dev."}},{"@type":"Question","name":"Which Delta Lake version introduced the OPTIMIZE command for compacting small files?","acceptedAnswer":{"@type":"Answer","text":"OPTIMIZE was added to Delta Lake in version 1.2, with Python bindings following in version 2.0. Before that, small files had to be compacted manually by rewriting the entire dataset with a specified number of output files, an approach that is inefficient and error-prone compared to using OPTIMIZE. Teams planning a Delta Lake upgrade can follow version-specific feature changes like this on daily.dev."}},{"@type":"Question","name":"What does the dataChange flag mean in a Delta Lake transaction log entry after running OPTIMIZE?","acceptedAnswer":{"@type":"Answer","text":"The dataChange flag is set to false for all files added and removed during an OPTIMIZE compaction, signaling that no new data was ingested, only existing data rearranged into larger files. This lets downstream jobs watching the table distinguish compacted files from genuinely new data and avoid unnecessary reprocessing. Engineers building incremental pipelines on Delta Lake can dig into flags like this on daily.dev."}}]} ```