--- title: Scaling Grab's Data Lake: Our journey to Apache Iceberg... description: Grab's data engineering team shares their journey migrating a petabyte-scale data lake from Hive Parquet to Apache Iceberg. The post covers the four key pain... canonical: https://daily.dev/posts/scaling-grab-s-data-lake-our-journey-to-apache-iceberg-adoption-ibvihkenm twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption | daily.dev og:description: Grab's data engineering team shares their journey migrating a petabyte-scale data lake from Hive Parquet to Apache Iceberg. The post covers the four key pain... og:url: https://daily.dev/posts/scaling-grab-s-data-lake-our-journey-to-apache-iceberg-adoption-ibvihkenm og:image: https://api.daily.dev/og/posts/ibvihkenM.png og:image:alt: Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption **[Grab Tech Blog](https://daily.dev/sources/grab)** · 8 min read · 6 upvotes · 0 comments ## Summary Grab's data engineering team shares their journey migrating a petabyte-scale data lake from Hive Parquet to Apache Iceberg. The post covers the four key pain points that drove the migration (catalog latency, small file problem, operational toil, and catalog-storage sync issues), the performance gains achieved (10x query speedup, 95% S3 API cost reduction, ~50% compute savings), and the tooling built along the way. A central piece is the UnifiedSparkCatalog — an open-sourced Spark catalog that transparently routes queries to the correct format-specific catalog (Iceberg, Delta, Hudi, or Hive) without requiring users to know or specify the underlying table format. The post also covers lessons learned around Hive lock contention, timestamp compatibility between Spark 3.4 and Iceberg, and S3 storage tier cost spikes during migration. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How much can migrating a Hive Parquet table to Apache Iceberg reduce S3 API costs? Daily S3 API costs dropped by up to 95% for a heavily queried operations table after migrating from Hive Parquet to Apache Iceberg, with no changes to the queries themselves. The savings came mainly from larger file sizes and eliminating expensive S3 object listing during query planning, which had previously driven up both latency and API request costs. _daily.dev surfaces engineering writeups like this for teams weighing an Iceberg migration and its cost tradeoffs._ ### How do you support Iceberg, Delta, and Hive tables in the same Spark environment without users specifying catalog prefixes? A unified Spark catalog implementation called UnifiedSparkCatalog can register itself as the default spark_catalog, detect each table's format from Hive Metastore metadata using a TableTypeDetector utility, and delegate operations to the matching format-specific catalog (Iceberg's SparkCatalog, Delta's DeltaCatalog, etc.) transparently, with fallback to the base session catalog if no match is found. Grab open-sourced this tool on GitHub. _developers juggling mixed table formats in Spark can track tools like this on daily.dev before building their own._ ### Why would Spark's TIMESTAMP_NTZ type conflict with Apache Iceberg tables? Spark 3.4 introduced TIMESTAMP_NTZ (no time zone), while Iceberg defaults to TIMESTAMP_LTZ (local time zone), which caused compatibility issues with legacy Hive views during a large-scale Iceberg migration. The mismatch was resolved through a custom migration workflow and targeted patches to the Trino deployment to ensure consistent timestamp casting across engines. _daily.dev helps engineers debugging cross-engine timestamp mismatches find real migration fixes, not just docs._ ## Similar posts on daily.dev - [Does Apache Iceberg Actually Save You Money?](https://daily.dev/posts/does-apache-iceberg-actually-save-you-money--o9dk8skic) · BigData Boutique blog · 0 upvotes · 0 comments - [Iceberg Lake for Data Analytics: Optimization Guide](https://daily.dev/posts/iceberg-lake-for-data-analytics-optimization-guide-kmewy4cgp) · ITNEXT · 1 upvotes · 0 comments - [Migrating to Apache Iceberg: Strategies for Every Source System](https://daily.dev/posts/migrating-to-apache-iceberg-strategies-for-every-source-system-nk4me2gyi) · Dremio Blog · 0 upvotes · 0 comments - [Why Apache Iceberg Is Not “Just Another Table Format”](https://daily.dev/posts/why-apache-iceberg-is-not-just-another-table-format--jeknvsrjf) · Snowflake Community · 37 upvotes · 0 comments --- Tags: [#data-engineering](https://daily.dev/tags/data-engineering), [#apache-spark](https://daily.dev/tags/apache-spark), [#data-lake](https://daily.dev/tags/data-lake), [#apache-iceberg](https://daily.dev/tags/apache-iceberg) [View this post on daily.dev](https://daily.dev/posts/scaling-grab-s-data-lake-our-journey-to-apache-iceberg-adoption-ibvihkenm) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption","url":"https://daily.dev/posts/scaling-grab-s-data-lake-our-journey-to-apache-iceberg-adoption-ibvihkenm","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/scaling-grab-s-data-lake-our-journey-to-apache-iceberg-adoption-ibvihkenm"},"datePublished":"2026-07-10T00:40:41.028Z","dateModified":"2026-09-13T20:16:10.624Z","description":"Grab's data engineering team shares their journey migrating a petabyte-scale data lake from Hive Parquet to Apache Iceberg. The post covers the four key pain...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5918717ac7dc54d93e00639ec83b67d8?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/5918717ac7dc54d93e00639ec83b67d8?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Grab Tech Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Grab Tech Blog","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/c8b8850354ee4494a05777b62e42cc62","url":"https://daily.dev/sources/grab"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/scaling-grab-s-data-lake-our-journey-to-apache-iceberg-adoption-ibvihkenm","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":6},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-engineering,apache-spark,data-lake,apache-iceberg","timeRequired":"PT8M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Grab Tech Blog","item":"https://daily.dev/sources/grab"},{"@type":"ListItem","position":3,"name":"Scaling Grab's Data Lake: Our journey to Apache Iceberg adoption"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/scaling-grab-s-data-lake-our-journey-to-apache-iceberg-adoption-ibvihkenm#faq","mainEntity":[{"@type":"Question","name":"How much can migrating a Hive Parquet table to Apache Iceberg reduce S3 API costs?","acceptedAnswer":{"@type":"Answer","text":"Daily S3 API costs dropped by up to 95% for a heavily queried operations table after migrating from Hive Parquet to Apache Iceberg, with no changes to the queries themselves. The savings came mainly from larger file sizes and eliminating expensive S3 object listing during query planning, which had previously driven up both latency and API request costs. daily.dev surfaces engineering writeups like this for teams weighing an Iceberg migration and its cost tradeoffs."}},{"@type":"Question","name":"How do you support Iceberg, Delta, and Hive tables in the same Spark environment without users specifying catalog prefixes?","acceptedAnswer":{"@type":"Answer","text":"A unified Spark catalog implementation called UnifiedSparkCatalog can register itself as the default spark_catalog, detect each table's format from Hive Metastore metadata using a TableTypeDetector utility, and delegate operations to the matching format-specific catalog (Iceberg's SparkCatalog, Delta's DeltaCatalog, etc.) transparently, with fallback to the base session catalog if no match is found. Grab open-sourced this tool on GitHub. developers juggling mixed table formats in Spark can track tools like this on daily.dev before building their own."}},{"@type":"Question","name":"Why would Spark's TIMESTAMP_NTZ type conflict with Apache Iceberg tables?","acceptedAnswer":{"@type":"Answer","text":"Spark 3.4 introduced TIMESTAMP_NTZ (no time zone), while Iceberg defaults to TIMESTAMP_LTZ (local time zone), which caused compatibility issues with legacy Hive views during a large-scale Iceberg migration. The mismatch was resolved through a custom migration workflow and targeted patches to the Trino deployment to ensure consistent timestamp casting across engines. daily.dev helps engineers debugging cross-engine timestamp mismatches find real migration fixes, not just docs."}}]} ```