---
title: "Building the Data Harness Catalog for Apache Spark | Personal Projects With Ex-Google SWE, Ep 10"
url: https://daily.dev/posts/building-the-data-harness-catalog-for-apache-spark-personal-projects-with-ex-google-swe-ep-10-3nufybrdz
source_url: https://www.youtube.com/watch?v=Z0oTEeVjz_A
type: video:youtube
source: "Jordan has no life"
published: 2026-05-31T07:49:24.015Z
updated: 2026-05-31T09:23:15.972Z
tags: ["infrastructure", "kafka", "apache-spark", "data-lake", "apache-iceberg"]
reading_time: 16
upvotes: 0
comments: 0
language: en
---

> ## Documentation Index
> Fetch the complete documentation index at: https://daily.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Building the Data Harness Catalog for Apache Spark | Personal Projects With Ex-Google SWE, Ep 10

**[Jordan has no life](https://daily.dev/sources/jordan-has-no-life)** · 16 min read · 0 upvotes · 0 comments

## Summary

A developer documents building a custom Apache Spark catalog plugin called the 'Data Harness Catalog' that federates queries across multiple heterogeneous data sources — Kafka, Apache Iceberg tables, and YugabyteDB — returning a unified, consistent snapshot view. The approach uses Spark's catalog interface and extension/analyzer hooks to intercept queries and replace a logical table reference with a union of DataFrames from each source. Key technical details include: using Spark's JDBC sessionInitStatement to set YugabyteDB's read timestamp for snapshot reads, deserializing Avro-encoded Kafka data using Spark's built-in fromAvro function while handling Confluent Schema Registry byte offsets, and reading Iceberg tables as-of a specific timestamp. All timestamps are coordinated through the Data Harness service to ensure cross-source consistency. The author also explains the strategic shift from Trino to Spark, citing Spark's broader connector ecosystem and fewer required open-source contributions to make the project functional.

## Full article

daily.dev links to this article rather than hosting it. Read it at the original source: <https://www.youtube.com/watch?v=Z0oTEeVjz_A>

---

Tags: [#infrastructure](https://daily.dev/tags/infrastructure), [#kafka](https://daily.dev/tags/kafka), [#apache-spark](https://daily.dev/tags/apache-spark), [#data-lake](https://daily.dev/tags/data-lake), [#apache-iceberg](https://daily.dev/tags/apache-iceberg)

[View this post on daily.dev](https://daily.dev/posts/building-the-data-harness-catalog-for-apache-spark-personal-projects-with-ex-google-swe-ep-10-3nufybrdz)