Branching Databases + Iceberg for Streaming Writes!! | Personal Projects With Ex-Google SWE, Ep 21
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
A deep technical walkthrough of redesigning Apache Iceberg streaming write architecture to align with Iceberg's intended commit semantics. The core problem: current implementation does heavy computation at catalog commit time (computing Kafka offsets, populating deletion indexes, converting equality deletes to positional deletes), making commits arbitrarily slow. The proposed solution moves this work to file write time by overriding Iceberg's FileIO interface, using DuckDB to read Parquet/ORC/Avro files as they're written, and storing per-file metadata in a Dolt database (a MySQL-compatible, git-semantics database). Dolt's branching model is used to handle ordering ambiguity between data files and positional delete files within the same commit. Equality delete conversion remains a challenge, with three options discussed: synchronous conversion at catalog intercept time, async background conversion, or lazy conversion at read time.