A developer documents building a custom Apache Spark catalog plugin called the 'Data Harness Catalog' that federates queries across multiple heterogeneous data sources — Kafka, Apache Iceberg tables, and YugabyteDB — returning a unified, consistent snapshot view. The approach uses Spark's catalog interface and extension/analyzer hooks to intercept queries and replace a logical table reference with a union of DataFrames from each source. Key technical details include: using Spark's JDBC sessionInitStatement to set YugabyteDB's read timestamp for snapshot reads, deserializing Avro-encoded Kafka data using Spark's built-in fromAvro function while handling Confluent Schema Registry byte offsets, and reading Iceberg tables as-of a specific timestamp. All timestamps are coordinated through the Data Harness service to ensure cross-source consistency. The author also explains the strategic shift from Trino to Spark, citing Spark's broader connector ecosystem and fewer required open-source contributions to make the project functional.

16m watch time
1 Impression