--- title: Maintaining Apache Iceberg Tables: Compaction,... description: A deep operational guide to maintaining Apache Iceberg tables covers the five ways tables degrade in production (file fragmentation, snapshot accumulation,... canonical: https://daily.dev/posts/maintaining-apache-iceberg-tables-compaction-snapshots-metadata-and-orphan-files-vk7yjrfyp twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Maintaining Apache Iceberg Tables: Compaction, Snapshots, Metadata and Orphan Files | daily.dev og:description: A deep operational guide to maintaining Apache Iceberg tables covers the five ways tables degrade in production (file fragmentation, snapshot accumulation,... og:url: https://daily.dev/posts/maintaining-apache-iceberg-tables-compaction-snapshots-metadata-and-orphan-files-vk7yjrfyp og:image: https://api.daily.dev/og/posts/Vk7yjrfYP.png og:image:alt: Maintaining Apache Iceberg Tables: Compaction, Snapshots, Metadata and Orphan Files og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Maintaining Apache Iceberg Tables: Compaction, Snapshots, Metadata and Orphan Files **[ITNEXT](https://daily.dev/sources/itnext)** · 33 min read · 0 upvotes · 0 comments ## Summary A deep operational guide to maintaining Apache Iceberg tables covers the five ways tables degrade in production (file fragmentation, snapshot accumulation, manifest fragmentation, orphan files, and delete file accumulation) and the four maintenance procedures Iceberg provides to address them: expire_snapshots, remove_orphan_files, rewrite_data_files, and rewrite_manifests. It explains why these must run in strict sequence (expire → orphans → compact → manifests), details bin-pack vs sort compaction strategies with concrete performance numbers, covers partial-progress mode and OCC conflict handling with concurrent writers, and contrasts manual Airflow/cron orchestration against an autonomous control-plane approach (the vendor's own LakeOps product) for managing maintenance at scale. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### In what order should I run Iceberg's expire_snapshots, remove_orphan_files, rewrite_data_files, and rewrite_manifests procedures? Run them in this order: expire_snapshots, then remove_orphan_files, then rewrite_data_files, then rewrite_manifests. Expiration dereferences files first, shrinking the scope for orphan cleanup; orphan cleanup then removes both accumulated and freshly dereferenced files; compaction runs on live data before manifests are rewritten, since compacting after manifest rewriting would immediately re-fragment the index. _Track sequencing pitfalls like this before they cost a wasted compaction run, with daily.dev._ ### Why does sorting an Iceberg table by a filtered column speed up queries so much? Sorting concentrates each column's values into a narrow range per file, so Parquet row-group min/max statistics let the query engine skip most files entirely for selective predicates. Unsorted tables typically scan close to 100% of files, while sorted tables scan only 5-15%, yielding an 8-12x speedup and often turning 30-second dashboard queries into sub-3-second ones. _Compare compaction strategies like sort vs bin-pack before committing storage to daily.dev readers researching Iceberg performance._ ### How much storage do orphan files typically waste in an Apache Iceberg data lake? Orphan file sweeps on mature Iceberg lakes routinely reclaim 20-40% of billable storage, since failed writes, OOMed compaction jobs, and snapshot expirations leave unreferenced files on object storage indefinitely. At $0.023/GB/month on S3 Standard, a 200 TB lake with 30% orphan accumulation wastes roughly $1,380 per month, or $16,560 per year. _Catch silent storage waste like Iceberg orphan files by following cost-optimization guidance on daily.dev._ ## Similar posts on daily.dev - [Apache Iceberg Compaction Best Practices](https://daily.dev/posts/apache-iceberg-compaction-best-practices-60tdqenf7) · ITNEXT · 7 upvotes · 0 comments - [Apache Iceberg Snapshot Expiration Explained](https://daily.dev/posts/apache-iceberg-snapshot-expiration-explained-uao54sov3) · Dremio Blog · 0 upvotes · 0 comments - [Debugging Apache Iceberg: A Production Guide](https://daily.dev/posts/debugging-apache-iceberg-a-production-guide-zyux3vjzn) · ITNEXT · 0 upvotes · 0 comments - [Apache Iceberg Architecture Deep Dive](https://daily.dev/posts/apache-iceberg-architecture-deep-dive-1znwuwiy4) · BigData Boutique blog · 5 upvotes · 0 comments - [Apache Iceberg Orphan File Cleanup Explained](https://daily.dev/posts/apache-iceberg-orphan-file-cleanup-explained-mr39sevwv) · Dremio Blog · 0 upvotes · 0 comments --- Tags: [#backend](https://daily.dev/tags/backend), [#data-engineering](https://daily.dev/tags/data-engineering), [#apache-spark](https://daily.dev/tags/apache-spark), [#apache-iceberg](https://daily.dev/tags/apache-iceberg) [View this post on daily.dev](https://daily.dev/posts/maintaining-apache-iceberg-tables-compaction-snapshots-metadata-and-orphan-files-vk7yjrfyp) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Maintaining Apache Iceberg Tables: Compaction, Snapshots, Metadata and Orphan Files","url":"https://daily.dev/posts/maintaining-apache-iceberg-tables-compaction-snapshots-metadata-and-orphan-files-vk7yjrfyp","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/maintaining-apache-iceberg-tables-compaction-snapshots-metadata-and-orphan-files-vk7yjrfyp"},"datePublished":"2026-08-23T21:59:37.139Z","dateModified":"2026-09-14T08:25:34.037Z","description":"A deep operational guide to maintaining Apache Iceberg tables covers the five ways tables degrade in production (file fragmentation, snapshot accumulation,...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/8e4cb763d85166dcf9b5e0a3694ed54a?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/8e4cb763d85166dcf9b5e0a3694ed54a?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"ITNEXT","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"ITNEXT","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/itnext2","url":"https://daily.dev/sources/itnext"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/maintaining-apache-iceberg-tables-compaction-snapshots-metadata-and-orphan-files-vk7yjrfyp","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"backend,data-engineering,apache-spark,apache-iceberg","timeRequired":"PT33M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"ITNEXT","item":"https://daily.dev/sources/itnext"},{"@type":"ListItem","position":3,"name":"Maintaining Apache Iceberg Tables: Compaction, Snapshots, Metadata and Orphan Files"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/maintaining-apache-iceberg-tables-compaction-snapshots-metadata-and-orphan-files-vk7yjrfyp#faq","mainEntity":[{"@type":"Question","name":"In what order should I run Iceberg's expire_snapshots, remove_orphan_files, rewrite_data_files, and rewrite_manifests procedures?","acceptedAnswer":{"@type":"Answer","text":"Run them in this order: expire_snapshots, then remove_orphan_files, then rewrite_data_files, then rewrite_manifests. Expiration dereferences files first, shrinking the scope for orphan cleanup; orphan cleanup then removes both accumulated and freshly dereferenced files; compaction runs on live data before manifests are rewritten, since compacting after manifest rewriting would immediately re-fragment the index. Track sequencing pitfalls like this before they cost a wasted compaction run, with daily.dev."}},{"@type":"Question","name":"Why does sorting an Iceberg table by a filtered column speed up queries so much?","acceptedAnswer":{"@type":"Answer","text":"Sorting concentrates each column's values into a narrow range per file, so Parquet row-group min/max statistics let the query engine skip most files entirely for selective predicates. Unsorted tables typically scan close to 100% of files, while sorted tables scan only 5-15%, yielding an 8-12x speedup and often turning 30-second dashboard queries into sub-3-second ones. Compare compaction strategies like sort vs bin-pack before committing storage to daily.dev readers researching Iceberg performance."}},{"@type":"Question","name":"How much storage do orphan files typically waste in an Apache Iceberg data lake?","acceptedAnswer":{"@type":"Answer","text":"Orphan file sweeps on mature Iceberg lakes routinely reclaim 20-40% of billable storage, since failed writes, OOMed compaction jobs, and snapshot expirations leave unreferenced files on object storage indefinitely. At $0.023/GB/month on S3 Standard, a 200 TB lake with 30% orphan accumulation wastes roughly $1,380 per month, or $16,560 per year. Catch silent storage waste like Iceberg orphan files by following cost-optimization guidance on daily.dev."}}]} ```