A developer shares a pivot in their personal DataHarness project after realizing the pitfalls of creating yet another competing big data table standard. Instead of building a new format, the new direction focuses on extending Apache Iceberg as a metadata layer to incorporate disparate data sources like Kafka and PostgreSQL, enabling efficient union reads across query engines like Spark, Trino, and Flink. Key technical challenges discussed include adapting the Iceberg FileIO interface for non-file data sources, implementing positional deletes at scale for streaming use cases, and managing distributed row indexing during compaction — problems that MoonCake solved only partially by limiting itself to a single PostgreSQL node.
•11m watch time
1 Impression