--- title: "Hacking The Iceberg Protocol: IceStream | Personal Projects With Ex-Google SWE, Ep 22" url: https://daily.dev/posts/hacking-the-iceberg-protocol-icestream-personal-projects-with-ex-google-swe-ep-22-03nz5eznk source_url: https://www.youtube.com/watch?v=ZN-Cda1NBqo type: video:youtube source: "Jordan has no life" published: 2026-05-31T07:49:25.220Z updated: 2026-05-31T09:26:33.481Z tags: ["data-engineering", "kafka", "apache-iceberg"] reading_time: 36 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Hacking The Iceberg Protocol: IceStream | Personal Projects With Ex-Google SWE, Ep 22 **[Jordan has no life](https://daily.dev/sources/jordan-has-no-life)** · 36 min read · 0 upvotes · 0 comments ## Summary A personal project walkthrough by an ex-Google SWE designing IceStream, a system to enable efficient streaming reads and writes on Apache Iceberg tables. The core problem: Iceberg lacks native support for streaming reads and primary keys, leading to small file proliferation and slow equality delete scans. The solution intercepts Iceberg's two main extension points — the FileIO interface and the REST catalog — to track Kafka offsets per data file in Postgres, maintain a row-level deletion index in Dolt (a Git-branching MySQL-compatible DB), and dynamically convert equality deletes to positional deletes at read time. On the read path, Kafka data is injected as a virtual manifest entry so readers can access unflushed data without waiting for a full compaction cycle. The design aims to be engine-agnostic, open source, and compatible with existing Flink, Spark, and Trino pipelines. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: --- Tags: [#data-engineering](https://daily.dev/tags/data-engineering), [#kafka](https://daily.dev/tags/kafka), [#apache-iceberg](https://daily.dev/tags/apache-iceberg) [View this post on daily.dev](https://daily.dev/posts/hacking-the-iceberg-protocol-icestream-personal-projects-with-ex-google-swe-ep-22-03nz5eznk)