A personal project walkthrough by an ex-Google SWE designing IceStream, a system to enable efficient streaming reads and writes on Apache Iceberg tables. The core problem: Iceberg lacks native support for streaming reads and primary keys, leading to small file proliferation and slow equality delete scans. The solution intercepts Iceberg's two main extension points — the FileIO interface and the REST catalog — to track Kafka offsets per data file in Postgres, maintain a row-level deletion index in Dolt (a Git-branching MySQL-compatible DB), and dynamically convert equality deletes to positional deletes at read time. On the read path, Kafka data is injected as a virtual manifest entry so readers can access unflushed data without waiting for a full compaction cycle. The design aims to be engine-agnostic, open source, and compatible with existing Flink, Spark, and Trino pipelines.

36m watch time
1 Impression