--- title: Maintaining Apache Iceberg Tables: Compaction, Expiry,... description: Part 10 of an Apache Iceberg masterclass walks through the four core table maintenance operations: compaction (bin-pack and sort-order), snapshot expiry,... canonical: https://daily.dev/posts/maintaining-apache-iceberg-tables-compaction-expiry-and-cleanup-6sgmxcysn twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Maintaining Apache Iceberg Tables: Compaction, Expiry, and Cleanup | daily.dev og:description: Part 10 of an Apache Iceberg masterclass walks through the four core table maintenance operations: compaction (bin-pack and sort-order), snapshot expiry,... og:url: https://daily.dev/posts/maintaining-apache-iceberg-tables-compaction-expiry-and-cleanup-6sgmxcysn og:image: https://api.daily.dev/og/posts/6sgmXCysn.png og:image:alt: Maintaining Apache Iceberg Tables: Compaction, Expiry, and Cleanup og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Maintaining Apache Iceberg Tables: Compaction, Expiry, and Cleanup **[Dremio Blog](https://daily.dev/sources/dremio)** · 5 min read · 0 upvotes · 0 comments ## Summary Part 10 of an Apache Iceberg masterclass walks through the four core table maintenance operations: compaction (bin-pack and sort-order), snapshot expiry, orphan file cleanup, and manifest rewriting. It explains why each matters, gives sample OPTIMIZE TABLE syntax, and outlines retention policy tradeoffs. It also compares three maintenance strategies—manual scheduled jobs, semi-automated monitoring-triggered jobs, and fully automated platforms like Dremio's autonomous optimization or AWS S3 Tables—along with a recommended maintenance schedule and common pitfalls like running compaction during peak hours or expiring snapshots too aggressively. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How often should I run compaction on Apache Iceberg tables? Run compaction daily for heavily-written Iceberg tables and weekly for lighter ones, triggering it when average file size drops below 64 MB. Compaction merges small files into 128-512 MB files using BIN_PACK, or sort-order compaction if queries filter on specific columns. Schedule it during off-peak hours since it competes with analytical queries for I/O bandwidth. _daily.dev surfaces practical guidance for teams tuning lakehouse maintenance schedules like Iceberg compaction._ ### What safety delay should I use for Apache Iceberg orphan file cleanup? Use a safety delay of at least 3 days before deleting orphan files in Apache Iceberg. Orphan cleanup scans storage and removes files not referenced by any snapshot after expiry; running it too aggressively risks deleting files from long-running write operations that haven't finished yet. Schedule this as a weekly job after snapshot expiry runs. _Developers automating data lake hygiene can track workflow tips like Iceberg orphan cleanup on daily.dev._ ### How long should I retain Apache Iceberg snapshots for time travel? Retention depends on the use case: 7 days for debugging recent issues, 30 days for monthly reporting compliance, 90+ days for regulatory audits, or 3-5 days if optimizing purely for storage cost. Longer retention means more metadata and storage from old data files, while shorter retention limits time travel but reduces cost. Always retain at least as long as the longest-running query to avoid failures. _daily.dev helps engineers weighing storage cost against time travel needs stay on top of lakehouse practices._ ## Similar posts on daily.dev - [Maintaining Apache Iceberg Tables: Compaction, Snapshots, Metadata and Orphan Files](https://daily.dev/posts/maintaining-apache-iceberg-tables-compaction-snapshots-metadata-and-orphan-files-vk7yjrfyp) · ITNEXT · 0 upvotes · 0 comments - [Apache Iceberg Compaction Best Practices](https://daily.dev/posts/apache-iceberg-compaction-best-practices-60tdqenf7) · ITNEXT · 7 upvotes · 0 comments - [Apache Iceberg Snapshot Expiration Explained](https://daily.dev/posts/apache-iceberg-snapshot-expiration-explained-uao54sov3) · Dremio Blog · 0 upvotes · 0 comments - [Apache Iceberg Orphan File Cleanup Explained](https://daily.dev/posts/apache-iceberg-orphan-file-cleanup-explained-mr39sevwv) · Dremio Blog · 0 upvotes · 0 comments - [Debugging Apache Iceberg: A Production Guide](https://daily.dev/posts/debugging-apache-iceberg-a-production-guide-zyux3vjzn) · ITNEXT · 0 upvotes · 0 comments --- Tags: [#backend](https://daily.dev/tags/backend), [#data-engineering](https://daily.dev/tags/data-engineering), [#apache-iceberg](https://daily.dev/tags/apache-iceberg) [View this post on daily.dev](https://daily.dev/posts/maintaining-apache-iceberg-tables-compaction-expiry-and-cleanup-6sgmxcysn) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Maintaining Apache Iceberg Tables: Compaction, Expiry, and Cleanup","url":"https://daily.dev/posts/maintaining-apache-iceberg-tables-compaction-expiry-and-cleanup-6sgmxcysn","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/maintaining-apache-iceberg-tables-compaction-expiry-and-cleanup-6sgmxcysn"},"datePublished":"2026-08-16T08:53:14.590Z","dateModified":"2026-08-16T08:54:26.883Z","description":"Part 10 of an Apache Iceberg masterclass walks through the four core table maintenance operations: compaction (bin-pack and sort-order), snapshot expiry,...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/ed220a35fd6502eaa444deee3561cf2a?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/ed220a35fd6502eaa444deee3561cf2a?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Dremio Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Dremio Blog","logo":"https://media.daily.dev/image/upload/s--MGKANojy--/f_auto,q_auto/v1786870389/logos/dremio?_a=BAMAMicg0","url":"https://daily.dev/sources/dremio"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/maintaining-apache-iceberg-tables-compaction-expiry-and-cleanup-6sgmxcysn","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"backend,data-engineering,apache-iceberg","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Dremio Blog","item":"https://daily.dev/sources/dremio"},{"@type":"ListItem","position":3,"name":"Maintaining Apache Iceberg Tables: Compaction, Expiry, and Cleanup"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/maintaining-apache-iceberg-tables-compaction-expiry-and-cleanup-6sgmxcysn#faq","mainEntity":[{"@type":"Question","name":"How often should I run compaction on Apache Iceberg tables?","acceptedAnswer":{"@type":"Answer","text":"Run compaction daily for heavily-written Iceberg tables and weekly for lighter ones, triggering it when average file size drops below 64 MB. Compaction merges small files into 128-512 MB files using BIN_PACK, or sort-order compaction if queries filter on specific columns. Schedule it during off-peak hours since it competes with analytical queries for I/O bandwidth. daily.dev surfaces practical guidance for teams tuning lakehouse maintenance schedules like Iceberg compaction."}},{"@type":"Question","name":"What safety delay should I use for Apache Iceberg orphan file cleanup?","acceptedAnswer":{"@type":"Answer","text":"Use a safety delay of at least 3 days before deleting orphan files in Apache Iceberg. Orphan cleanup scans storage and removes files not referenced by any snapshot after expiry; running it too aggressively risks deleting files from long-running write operations that haven't finished yet. Schedule this as a weekly job after snapshot expiry runs. Developers automating data lake hygiene can track workflow tips like Iceberg orphan cleanup on daily.dev."}},{"@type":"Question","name":"How long should I retain Apache Iceberg snapshots for time travel?","acceptedAnswer":{"@type":"Answer","text":"Retention depends on the use case: 7 days for debugging recent issues, 30 days for monthly reporting compliance, 90+ days for regulatory audits, or 3-5 days if optimizing purely for storage cost. Longer retention means more metadata and storage from old data files, while shorter retention limits time travel but reduces cost. Always retain at least as long as the longest-running query to avoid failures. daily.dev helps engineers weighing storage cost against time travel needs stay on top of lakehouse practices."}}]} ```