Grab's data engineering team shares their journey migrating a petabyte-scale data lake from Hive Parquet to Apache Iceberg. The post covers the four key pain points that drove the migration (catalog latency, small file problem, operational toil, and catalog-storage sync issues), the performance gains achieved (10x query speedup, 95% S3 API cost reduction, ~50% compute savings), and the tooling built along the way. A central piece is the UnifiedSparkCatalog — an open-sourced Spark catalog that transparently routes queries to the correct format-specific catalog (Iceberg, Delta, Hudi, or Hive) without requiring users to know or specify the underlying table format. The post also covers lessons learned around Hive lock contention, timestamp compatibility between Spark 3.4 and Iceberg, and S3 storage tier cost spikes during migration.
