Iceberg tables degrade over time through five mechanisms: small file accumulation, orphan files, metadata bloat, sort order decay, and partition skew. Each is explained with concrete impact numbers, causes, and diagnostic SQL queries against Iceberg metadata tables (table_files, table_snapshot). A worked example shows a table going from 100 files on day 1 to 18,000+ files and 10-second query planning by day 180 without compaction. The piece recommends periodic compaction, snapshot expiry, and orphan file cleanup, and closes by promoting Dremio and S3 Tables for automated maintenance.
Table of contents
Table of ContentsFive Types of DegradationReal-World Degradation TimelineHow to Diagnose Table HealthThe Maintenance ImperativeTry Dremio Cloud free for 30 daysQuestions this post answers
Why is my Apache Iceberg table getting slower over time even though the data size hasn't changed much?
Iceberg tables degrade because every append, update, and delete adds files and metadata without automatic cleanup. Common causes include small file accumulation from frequent commits (each file becomes a manifest entry the planner must evaluate), orphan files from expired snapshots or failed writes, metadata.json bloat past 1,000+ snapshots, sort order decay, and partition skew. Regular compaction and snapshot expiry fix these. Track Iceberg table maintenance techniques like this on daily.dev as your lakehouse scales.
What is the target file size to avoid the small file problem in Apache Iceberg?
Files should average 128-512 MB, with anything below 32 MB considered a small file problem. A table with 10,000 small files creates 10,000 manifest entries the query planner must evaluate versus just 40 entries for the same data stored in properly-sized 256 MB files, so planning time increases roughly linearly with file count. daily.dev surfaces practical benchmarks like this for engineers tuning data lake performance.
How many snapshots should an Apache Iceberg table keep before performance suffers?
Snapshot count should stay under 1,000; beyond that, older snapshots should be expired. At 10,000+ snapshots, the metadata.json file can exceed 100 MB, adding seconds to every query's planning phase before any data is even read, so configuring snapshot expiry policies is necessary to keep metadata lean. Developers debugging query planning slowdowns can follow guidance like this via daily.dev.