A developer shares progress on IceStream, a system for efficient streaming deletes into Apache Iceberg tables. After receiving feedback criticizing the use of Cassandra as an external index store, the author explores replacing it with Apache Paimon — an open table format with LSM-tree-based primary key tables stored entirely on object store. Paimon's fixed-bucket partitioning and SS-table structure enable efficient nested loop index joins via Flink, eliminating the need for a separate database system. This approach better aligns with the compute-storage separation philosophy of modern data lake architectures, reducing operational overhead while maintaining efficient delete handling.
•13m watch time
4 Impressions