--- title: Using Apache Iceberg with Python and MPP Query Engines description: Part twelve of an Apache Iceberg masterclass explains how to access Iceberg tables from Python and from MPP query engines. On the Python side, PyIceberg reads... canonical: https://daily.dev/posts/using-apache-iceberg-with-python-and-mpp-query-engines-hfwu3ng14 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Using Apache Iceberg with Python and MPP Query Engines | daily.dev og:description: Part twelve of an Apache Iceberg masterclass explains how to access Iceberg tables from Python and from MPP query engines. On the Python side, PyIceberg reads... og:url: https://daily.dev/posts/using-apache-iceberg-with-python-and-mpp-query-engines-hfwu3ng14 og:image: https://api.daily.dev/og/posts/hfwu3NG14.png og:image:alt: Using Apache Iceberg with Python and MPP Query Engines og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Using Apache Iceberg with Python and MPP Query Engines **[Dremio Blog](https://daily.dev/sources/dremio)** · 6 min read · 0 upvotes · 0 comments ## Summary Part twelve of an Apache Iceberg masterclass explains how to access Iceberg tables from Python and from MPP query engines. On the Python side, PyIceberg reads metadata directly and pushes filters down to manifests, DuckDB offers in-process SQL analysis with its Iceberg extension, and Polars provides lazy, parallel dataframe scans via scan_iceberg. PyIceberg also supports writes through Apache Arrow tables but is single-machine only. For production scale, MPP engines like Dremio (via Arrow Flight and Columnar Cloud Cache), Spark (the most feature-complete for reads/writes/maintenance), and Trino (low-latency ad-hoc queries) are recommended, alongside other engines like Athena, Snowflake, StarRocks, and Doris. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I read an Apache Iceberg table into a pandas DataFrame using PyIceberg? Load a catalog with PyIceberg's load_catalog function pointing to a REST catalog, then call catalog.load_table() to get the table and table.scan(row_filter=...) followed by .to_pandas() to materialize a pandas DataFrame. The row_filter is pushed down to manifest evaluation so only relevant data files are scanned, making it efficient for pulling subsets of large tables for analysis or ML training. _daily.dev surfaces practical Iceberg and PyIceberg workflows for engineers building data pipelines._ ### Can DuckDB query Apache Iceberg tables directly without Spark? Yes, DuckDB reads Iceberg tables through its Iceberg extension by installing and loading it, then calling iceberg_scan() with a table path inside a standard SQL query. It supports partition pruning and column statistics for file skipping, runs entirely in-process with no server to manage, and works well for local analysis, CI/CD data validation, and notebooks. _engineers comparing lightweight Iceberg query options can track tools like DuckDB on daily.dev._ ### Why is Dremio's Arrow Flight connection faster than JDBC for querying Iceberg data from Python? Arrow Flight sends query results in columnar Apache Arrow format directly to the client, avoiding the serialization overhead that JDBC and ODBC connectors incur, which makes it 10-100x faster for large result sets. Because data stays in Arrow format end-to-end from Iceberg Parquet through Dremio to the client pandas DataFrame, there are no format conversion bottlenecks. _teams weighing MPP engines for lakehouse analytics can follow performance comparisons on daily.dev._ ## Similar posts on daily.dev - [Apache Polaris with PyIceberg: A Guide](https://daily.dev/posts/apache-polaris-with-pyiceberg-a-guide-ykkijp45z) · Dremio Blog · 0 upvotes · 0 comments - [Amazon Aurora PostgreSQL now supports direct querying of Apache Iceberg and Parquet data in your data lake](https://daily.dev/posts/amazon-aurora-postgresql-now-supports-direct-querying-of-apache-iceberg-and-parquet-data-in-your-dat-lh81d12gd) · AWS · 1 upvotes · 0 comments - [Iceberg Lake for Data Analytics: Optimization Guide](https://daily.dev/posts/iceberg-lake-for-data-analytics-optimization-guide-kmewy4cgp) · ITNEXT · 1 upvotes · 0 comments - [New public datasets available in Google Cloud Lakehouse](https://daily.dev/posts/new-public-datasets-available-in-google-cloud-lakehouse-hxwrg2nwn) · Google Open Source Blog · 0 upvotes · 0 comments --- Tags: [#data-engineering](https://daily.dev/tags/data-engineering), [#duckdb](https://daily.dev/tags/duckdb), [#apache-iceberg](https://daily.dev/tags/apache-iceberg) [View this post on daily.dev](https://daily.dev/posts/using-apache-iceberg-with-python-and-mpp-query-engines-hfwu3ng14) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Using Apache Iceberg with Python and MPP Query Engines","url":"https://daily.dev/posts/using-apache-iceberg-with-python-and-mpp-query-engines-hfwu3ng14","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/using-apache-iceberg-with-python-and-mpp-query-engines-hfwu3ng14"},"datePublished":"2026-08-16T08:53:13.253Z","dateModified":"2026-08-16T08:53:54.641Z","description":"Part twelve of an Apache Iceberg masterclass explains how to access Iceberg tables from Python and from MPP query engines. On the Python side, PyIceberg reads...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/962aaf7c4b073b5a3779359cae35381d?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/962aaf7c4b073b5a3779359cae35381d?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Dremio Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Dremio Blog","logo":"https://media.daily.dev/image/upload/s--MGKANojy--/f_auto,q_auto/v1786870389/logos/dremio?_a=BAMAMicg0","url":"https://daily.dev/sources/dremio"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/using-apache-iceberg-with-python-and-mpp-query-engines-hfwu3ng14","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-engineering,duckdb,apache-iceberg","timeRequired":"PT6M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Dremio Blog","item":"https://daily.dev/sources/dremio"},{"@type":"ListItem","position":3,"name":"Using Apache Iceberg with Python and MPP Query Engines"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/using-apache-iceberg-with-python-and-mpp-query-engines-hfwu3ng14#faq","mainEntity":[{"@type":"Question","name":"How do I read an Apache Iceberg table into a pandas DataFrame using PyIceberg?","acceptedAnswer":{"@type":"Answer","text":"Load a catalog with PyIceberg's load_catalog function pointing to a REST catalog, then call catalog.load_table() to get the table and table.scan(row_filter=...) followed by .to_pandas() to materialize a pandas DataFrame. The row_filter is pushed down to manifest evaluation so only relevant data files are scanned, making it efficient for pulling subsets of large tables for analysis or ML training. daily.dev surfaces practical Iceberg and PyIceberg workflows for engineers building data pipelines."}},{"@type":"Question","name":"Can DuckDB query Apache Iceberg tables directly without Spark?","acceptedAnswer":{"@type":"Answer","text":"Yes, DuckDB reads Iceberg tables through its Iceberg extension by installing and loading it, then calling iceberg_scan() with a table path inside a standard SQL query. It supports partition pruning and column statistics for file skipping, runs entirely in-process with no server to manage, and works well for local analysis, CI/CD data validation, and notebooks. engineers comparing lightweight Iceberg query options can track tools like DuckDB on daily.dev."}},{"@type":"Question","name":"Why is Dremio's Arrow Flight connection faster than JDBC for querying Iceberg data from Python?","acceptedAnswer":{"@type":"Answer","text":"Arrow Flight sends query results in columnar Apache Arrow format directly to the client, avoiding the serialization overhead that JDBC and ODBC connectors incur, which makes it 10-100x faster for large result sets. Because data stays in Arrow format end-to-end from Iceberg Parquet through Dremio to the client pandas DataFrame, there are no format conversion bottlenecks. teams weighing MPP engines for lakehouse analytics can follow performance comparisons on daily.dev."}}]} ```