A deep operational guide to maintaining Apache Iceberg tables covers the five ways tables degrade in production (file fragmentation, snapshot accumulation, manifest fragmentation, orphan files, and delete file accumulation) and the four maintenance procedures Iceberg provides to address them: expire_snapshots, remove_orphan_files, rewrite_data_files, and rewrite_manifests. It explains why these must run in strict sequence (expire → orphans → compact → manifests), details bin-pack vs sort compaction strategies with concrete performance numbers, covers partial-progress mode and OCC conflict handling with concurrent writers, and contrasts manual Airflow/cron orchestration against an autonomous control-plane approach (the vendor's own LakeOps product) for managing maintenance at scale.
Table of contents
File fragmentationSnapshot accumulationManifest fragmentationOrphan file accumulationDelete file accumulationThe four operations at a glanceTwo approaches to maintaining your tablesQuestions this post answers
In what order should I run Iceberg's expire_snapshots, remove_orphan_files, rewrite_data_files, and rewrite_manifests procedures?
Run them in this sequence: expire_snapshots first, remove_orphan_files second, rewrite_data_files (compaction) third, and rewrite_manifests last. Expiration dereferences files so orphan cleanup can catch them; compaction should only touch live data; manifests must be rewritten after compaction or the consolidated index gets immediately fragmented again by compaction's output. Violating this order doesn't error but produces silently suboptimal results. daily.dev surfaces practical breakdowns like this for teams designing reliable Iceberg maintenance pipelines.
Why does sort compaction improve Iceberg query performance so much more than bin-pack compaction?
Sort compaction physically orders rows by a filtered column so each file's min/max statistics form a tight, non-overlapping range, letting engines skip 95%+ of files for point lookups via row-group pruning. Bin-pack only merges small files without reordering data, so min/max statistics still span the entire domain and skip nothing. Sorted tables typically scan 5-15% of data versus 100%, an 8-12x speedup, turning 30-second dashboard queries into sub-3-second ones. Developers weighing compaction strategies for Iceberg lakes can find this kind of deep comparison on daily.dev.
How much storage cost do orphan files waste in an Apache Iceberg data lake?
Orphan sweeps on mature Iceberg lakes routinely reclaim 20-40% of billable storage, since failed writes, OOMed compaction jobs, and expired snapshots leave data files on storage that no live snapshot references. At $0.023/GB/month on S3 Standard, a 200 TB lake with 30% orphan accumulation wastes about $1,380/month, or $16,560/year, until an explicit cleanup runs. daily.dev helps engineers tracking lakehouse cost-optimization tactics like Iceberg orphan cleanup stay informed.