--- title: Querying Databricks Iceberg tables with Google Cloud’s... description: Google Cloud's borderless Lakehouse lets BigQuery query Databricks-managed Apache Iceberg tables directly, without copying data. BigQuery reads the same... canonical: https://daily.dev/posts/querying-databricks-iceberg-tables-with-google-cloud-s-lakehouse-dt6ilfxxl twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Querying Databricks Iceberg tables with Google Cloud’s Lakehouse | daily.dev og:description: Google Cloud's borderless Lakehouse lets BigQuery query Databricks-managed Apache Iceberg tables directly, without copying data. BigQuery reads the same... og:url: https://daily.dev/posts/querying-databricks-iceberg-tables-with-google-cloud-s-lakehouse-dt6ilfxxl og:image: https://api.daily.dev/og/posts/DT6ilFXXl.png og:image:alt: Querying Databricks Iceberg tables with Google Cloud’s Lakehouse og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Querying Databricks Iceberg tables with Google Cloud’s Lakehouse **[Google Cloud](https://daily.dev/sources/gcp)** · 7 min read · 0 upvotes · 0 comments ## Summary Google Cloud's borderless Lakehouse lets BigQuery query Databricks-managed Apache Iceberg tables directly, without copying data. BigQuery reads the same Parquet files Databricks writes in GCS, and metadata refreshes as often as every 5 minutes. Databricks Unity Catalog functions as the central metadata layer, with credential vending issuing short-lived tokens for GCS access. BigQuery uses Iceberg manifests for partition pruning, and Lakehouse caching (in preview) speeds up repeat queries on cross-cloud tables like those in Amazon S3. Regular Databricks OPTIMIZE and VACUUM maintenance helps keep files compacted and removes orphaned files, with a seven-day deletedFileRetentionDuration threshold before VACUUM runs. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How can I query Databricks-managed Iceberg tables directly from BigQuery without copying the data? Google Cloud's Lakehouse lets BigQuery read the same Parquet files Databricks writes in GCS directly, via a federated connection to Databricks Unity Catalog. Lakehouse contacts Unity Catalog for current table state and uses credential vending to issue short-lived tokens granting BigQuery temporary read access, with metadata refreshing as often as every 5 minutes. _daily.dev surfaces practical lakehouse federation setups for teams weighing zero-copy architectures over data pipelines._ ### Why doesn't BigQuery need a metadata cache when querying Iceberg tables? Iceberg manifests already identify which data files to read, so BigQuery plans each query by reading the Iceberg manifest files directly, with no metadata caching needed. The manifests maintain column-level statistical bounds, letting BigQuery skip irrelevant manifest records and Parquet files, reducing bytes scanned and runtime through partition pruning. _Engineers tuning Iceberg query performance track details like this on daily.dev before optimizing their own pipelines._ ### How do I compact small Parquet files in a Databricks Iceberg table to improve query performance? Run the OPTIMIZE command on Databricks, such as 'OPTIMIZE customer_events;', or for Iceberg tables managed outside Databricks, call 'rewrite_data_files' with a binpack strategy and target file size, for example 268435456 bytes (256MB) with a minimum of 67108864 bytes (64MB). This merges scattered files into larger blocks, typically 256MB to 512MB, improving BigQuery scan throughput. _Teams maintaining lakehouse storage hygiene follow compaction guidance like this via daily.dev._ ## Similar posts on daily.dev - [Borderless Lakehouse cross-cloud caching and connections](https://daily.dev/posts/borderless-lakehouse-cross-cloud-caching-and-connections-xctvphcsx) · Google Cloud · 0 upvotes · 0 comments - [New public datasets available in Google Cloud Lakehouse](https://daily.dev/posts/new-public-datasets-available-in-google-cloud-lakehouse-hxwrg2nwn) · Google Open Source Blog · 0 upvotes · 0 comments - [Couchbase Analytics \+ Apache Iceberg on Google Cloud: Query Your Lakehouse Without ETL](https://daily.dev/posts/couchbase-analytics-apache-iceberg-on-google-cloud-query-your-lakehouse-without-etl-l5aai8dvk) · Couchbase · 0 upvotes · 0 comments - [Google Cloud Introduces Cross-Engine Iceberg Support in BigQuery](https://daily.dev/posts/google-cloud-introduces-cross-engine-iceberg-support-in-bigquery-iuk5rijpt) · InfoQ · 0 upvotes · 1 comments - [Cross-cloud conversational analytics with borderless Lakehouse](https://daily.dev/posts/cross-cloud-conversational-analytics-with-borderless-lakehouse-g6rjhqjqv) · Google Cloud · 0 upvotes · 0 comments --- Tags: [#gcp](https://daily.dev/tags/gcp), [#data-engineering](https://daily.dev/tags/data-engineering), [#databricks](https://daily.dev/tags/databricks), [#google-bigquery](https://daily.dev/tags/google-bigquery), [#apache-iceberg](https://daily.dev/tags/apache-iceberg) [View this post on daily.dev](https://daily.dev/posts/querying-databricks-iceberg-tables-with-google-cloud-s-lakehouse-dt6ilfxxl) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Querying Databricks Iceberg tables with Google Cloud’s Lakehouse","url":"https://daily.dev/posts/querying-databricks-iceberg-tables-with-google-cloud-s-lakehouse-dt6ilfxxl","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/querying-databricks-iceberg-tables-with-google-cloud-s-lakehouse-dt6ilfxxl"},"datePublished":"2026-10-29T16:08:17.918Z","dateModified":"2026-10-29T16:08:49.207Z","description":"Google Cloud's borderless Lakehouse lets BigQuery query Databricks-managed Apache Iceberg tables directly, without copying data. BigQuery reads the same...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/9df795f90d3290a29c3b5ccad1c3e312?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/9df795f90d3290a29c3b5ccad1c3e312?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Google Cloud","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Google Cloud","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/gcp","url":"https://daily.dev/sources/gcp"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/querying-databricks-iceberg-tables-with-google-cloud-s-lakehouse-dt6ilfxxl","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"gcp,data-engineering,databricks,google-bigquery,apache-iceberg","timeRequired":"PT7M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Google Cloud","item":"https://daily.dev/sources/gcp"},{"@type":"ListItem","position":3,"name":"Querying Databricks Iceberg tables with Google Cloud’s Lakehouse"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/querying-databricks-iceberg-tables-with-google-cloud-s-lakehouse-dt6ilfxxl#faq","mainEntity":[{"@type":"Question","name":"How can I query Databricks-managed Iceberg tables directly from BigQuery without copying the data?","acceptedAnswer":{"@type":"Answer","text":"Google Cloud's Lakehouse lets BigQuery read the same Parquet files Databricks writes in GCS directly, via a federated connection to Databricks Unity Catalog. Lakehouse contacts Unity Catalog for current table state and uses credential vending to issue short-lived tokens granting BigQuery temporary read access, with metadata refreshing as often as every 5 minutes. daily.dev surfaces practical lakehouse federation setups for teams weighing zero-copy architectures over data pipelines."}},{"@type":"Question","name":"Why doesn't BigQuery need a metadata cache when querying Iceberg tables?","acceptedAnswer":{"@type":"Answer","text":"Iceberg manifests already identify which data files to read, so BigQuery plans each query by reading the Iceberg manifest files directly, with no metadata caching needed. The manifests maintain column-level statistical bounds, letting BigQuery skip irrelevant manifest records and Parquet files, reducing bytes scanned and runtime through partition pruning. Engineers tuning Iceberg query performance track details like this on daily.dev before optimizing their own pipelines."}},{"@type":"Question","name":"How do I compact small Parquet files in a Databricks Iceberg table to improve query performance?","acceptedAnswer":{"@type":"Answer","text":"Run the OPTIMIZE command on Databricks, such as 'OPTIMIZE customer_events;', or for Iceberg tables managed outside Databricks, call 'rewrite_data_files' with a binpack strategy and target file size, for example 268435456 bytes (256MB) with a minimum of 67108864 bytes (64MB). This merges scattered files into larger blocks, typically 256MB to 512MB, improving BigQuery scan throughput. Teams maintaining lakehouse storage hygiene follow compaction guidance like this via daily.dev."}}]} ```