Part twelve of an Apache Iceberg masterclass explains how to access Iceberg tables from Python and from MPP query engines. On the Python side, PyIceberg reads metadata directly and pushes filters down to manifests, DuckDB offers in-process SQL analysis with its Iceberg extension, and Polars provides lazy, parallel dataframe scans via scan_iceberg. PyIceberg also supports writes through Apache Arrow tables but is single-machine only. For production scale, MPP engines like Dremio (via Arrow Flight and Columnar Cloud Cache), Spark (the most feature-complete for reads/writes/maintenance), and Trino (low-latency ad-hoc queries) are recommended, alongside other engines like Athena, Snowflake, StarRocks, and Doris.
Table of contents
Table of ContentsThe Python Ecosystem for IcebergMPP Query EnginesTry Dremio Cloud free for 30 daysQuestions this post answers
How do I read an Apache Iceberg table into a pandas DataFrame using PyIceberg?
Load a catalog with PyIceberg's load_catalog function pointing to a REST catalog, then call catalog.load_table() to get the table and table.scan(row_filter=...) followed by .to_pandas() to materialize a pandas DataFrame. The row_filter is pushed down to manifest evaluation so only relevant data files are scanned, making it efficient for pulling subsets of large tables for analysis or ML training. daily.dev surfaces practical Iceberg and PyIceberg workflows for engineers building data pipelines.
Can DuckDB query Apache Iceberg tables directly without Spark?
Yes, DuckDB reads Iceberg tables through its Iceberg extension by installing and loading it, then calling iceberg_scan() with a table path inside a standard SQL query. It supports partition pruning and column statistics for file skipping, runs entirely in-process with no server to manage, and works well for local analysis, CI/CD data validation, and notebooks. engineers comparing lightweight Iceberg query options can track tools like DuckDB on daily.dev.
Why is Dremio's Arrow Flight connection faster than JDBC for querying Iceberg data from Python?
Arrow Flight sends query results in columnar Apache Arrow format directly to the client, avoiding the serialization overhead that JDBC and ODBC connectors incur, which makes it 10-100x faster for large result sets. Because data stays in Arrow format end-to-end from Iceberg Parquet through Dremio to the client pandas DataFrame, there are no format conversion bottlenecks. teams weighing MPP engines for lakehouse analytics can follow performance comparisons on daily.dev.