A former Google SWE explores Icebase, a project that stores Apache Iceberg table metadata in PostgreSQL instead of object storage (S3/HDFS). The core problem: every Iceberg write generates multiple small metadata files (metadata JSON, manifest list, manifest files) on object store, incurring costly S3 PUT/GET operations. The solution uses Iceberg's FileIO interface to intercept metadata writes and redirect them to a database (PostgreSQL via ORM), while data files continue going to object store. The author discovered a similar project by a Cisco staff engineer, forked it, added two key improvements: a delegate FileIO for data files and reflection-based client instantiation for compatibility with Spark and Trino. The package was published to Maven.

15m watch time
1 Impression