--- title: "Scrapping (Re-Designing) The DataHarness | | Personal Projects With Ex-Google SWE, Ep 17" url: https://daily.dev/posts/scrapping-re-designing-the-dataharness-personal-projects-with-ex-google-swe-ep-17-esye4dm8h source_url: https://www.youtube.com/watch?v=A-TDJJRX8Rg type: video:youtube source: "Jordan has no life" published: 2026-05-31T07:49:26.552Z updated: 2026-05-31T09:17:20.567Z tags: ["data-engineering", "apache-kafka", "apache-iceberg", "change-data-capture"] reading_time: 11 upvotes: 0 comments: 0 language: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Scrapping (Re-Designing) The DataHarness | | Personal Projects With Ex-Google SWE, Ep 17 **[Jordan has no life](https://daily.dev/sources/jordan-has-no-life)** · 11 min read · 0 upvotes · 0 comments ## Summary A developer shares a pivot in their personal DataHarness project after realizing the pitfalls of creating yet another competing big data table standard. Instead of building a new format, the new direction focuses on extending Apache Iceberg as a metadata layer to incorporate disparate data sources like Kafka and PostgreSQL, enabling efficient union reads across query engines like Spark, Trino, and Flink. Key technical challenges discussed include adapting the Iceberg FileIO interface for non-file data sources, implementing positional deletes at scale for streaming use cases, and managing distributed row indexing during compaction — problems that MoonCake solved only partially by limiting itself to a single PostgreSQL node. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: --- Tags: [#data-engineering](https://daily.dev/tags/data-engineering), [#apache-kafka](https://daily.dev/tags/apache-kafka), [#apache-iceberg](https://daily.dev/tags/apache-iceberg), [#change-data-capture](https://daily.dev/tags/change-data-capture) [View this post on daily.dev](https://daily.dev/posts/scrapping-re-designing-the-dataharness-personal-projects-with-ex-google-swe-ep-17-esye4dm8h)