Hotdata built and open-sourced a DuckLake integration for Apache DataFusion, adding a lakehouse format that manages snapshots and catalog metadata for Parquet files in object storage, with support for multiple database catalog backends. The team explains why they chose DuckLake over building a custom metadata layer or using Apache Iceberg, citing its relational metadata catalog as a good fit for provisioning millions of ephemeral, isolated databases for AI agents. They also added support for logical catalogs (a single metadata store hosting multiple independent DuckLake catalogs) and benchmarked DuckLake-backed tables against direct Parquet access using TPC-H queries across three dataset sizes, finding no measurable performance penalty—DuckLake was actually marginally faster due to lower query planning overhead. The implementation runs in production and has been donated to the Apache DataFusion Contrib organization.
Table of contents
What Is Apache DataFusion?What Is DuckLake?Why We Chose DuckLakeMeasuring Metadata OverheadSummaryQuestions this post answers
How does DuckLake's metadata catalog affect query performance compared to querying Parquet files directly in DataFusion?
Benchmarks using all 22 TPC-H queries across three dataset sizes (SF0.2, SF1, SF10) showed DuckLake-backed tables performed as fast or slightly faster than direct Parquet access, with ratios of 0.98x, 0.99x, and 0.94x respectively. Both approaches use DataFusion's Parquet reader, so the improvement comes from lower query planning overhead since DuckLake already knows which files belong to a snapshot, avoiding metadata discovery. Teams evaluating lakehouse formats for query performance can follow real benchmark results like this on daily.dev.
Why would a company choose DuckLake over Apache Iceberg for a lakehouse metadata catalog?
DuckLake's relational metadata catalog was easier to understand and implement than Iceberg's file-based metadata approach, resolving metadata lookups through a database rather than multiple object-store requests. This mattered for high-volume, latency-sensitive queries and for provisioning millions of ephemeral, isolated databases without duplicating metadata infrastructure, a workload pattern that doesn't fit traditional long-lived table architectures. Engineers weighing DuckLake against Iceberg can track comparisons like this on daily.dev before committing to a format.
What is DuckLake and how does its metadata storage differ from other lakehouse table formats?
DuckLake is a lakehouse specification that stores table metadata in a transactional database rather than as files in object storage, while still using Parquet files for the actual data. This avoids traversing chains of metadata files to determine which Parquet files to scan during a query. It was created by the DuckDB team but is engine-agnostic, allowing integrations like the one built for Apache DataFusion. Developers exploring new lakehouse formats can keep up with projects like DuckLake on daily.dev.