Almost Time To Start Coding!
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
A developer vents about frustrations with Apache Iceberg and similar open table formats (Delta Lake, Hudi), explaining their core architecture: parquet files on object storage, immutable metadata files, and a catalog system. Key pain points include write amplification from immutable metadata on every commit, expensive delete operations requiring separate delete files and compaction jobs, and the emerging anti-pattern of federating Iceberg tables with a separate OLTP database (losing unified catalog management). DuckLake is highlighted as a promising improvement that uses a PostgreSQL database for metadata instead of immutable files. The author outlines a side project idea to unify transactional databases, Kafka streams, and table formats under a single catalog with time travel support and multi-engine readability.