Part 10 of an Apache Iceberg masterclass walks through the four core table maintenance operations: compaction (bin-pack and sort-order), snapshot expiry, orphan file cleanup, and manifest rewriting. It explains why each matters, gives sample OPTIMIZE TABLE syntax, and outlines retention policy tradeoffs. It also compares three maintenance strategies—manual scheduled jobs, semi-automated monitoring-triggered jobs, and fully automated platforms like Dremio's autonomous optimization or AWS S3 Tables—along with a recommended maintenance schedule and common pitfalls like running compaction during peak hours or expiring snapshots too aggressively.
Table of contents
Table of ContentsThe Four Maintenance OperationsThree Approaches to MaintenanceRecommended Maintenance ScheduleCommon Maintenance PitfallsTry Dremio Cloud free for 30 daysQuestions this post answers
How often should I run compaction on Apache Iceberg tables?
Run compaction daily for heavily-written Iceberg tables and weekly for lighter ones, triggering it when average file size drops below 64 MB. Compaction merges small files into 128-512 MB files using BIN_PACK, or sort-order compaction if queries filter on specific columns. Schedule it during off-peak hours since it competes with analytical queries for I/O bandwidth. daily.dev surfaces practical guidance for teams tuning lakehouse maintenance schedules like Iceberg compaction.
What safety delay should I use for Apache Iceberg orphan file cleanup?
Use a safety delay of at least 3 days before deleting orphan files in Apache Iceberg. Orphan cleanup scans storage and removes files not referenced by any snapshot after expiry; running it too aggressively risks deleting files from long-running write operations that haven't finished yet. Schedule this as a weekly job after snapshot expiry runs. Developers automating data lake hygiene can track workflow tips like Iceberg orphan cleanup on daily.dev.
How long should I retain Apache Iceberg snapshots for time travel?
Retention depends on the use case: 7 days for debugging recent issues, 30 days for monthly reporting compliance, 90+ days for regulatory audits, or 3-5 days if optimizing purely for storage cost. Longer retention means more metadata and storage from old data files, while shorter retention limits time travel but reduces cost. Always retain at least as long as the longest-running query to avoid failures. daily.dev helps engineers weighing storage cost against time travel needs stay on top of lakehouse practices.