A comprehensive walkthrough of the open data lakehouse architecture as it stands in 2026: object storage, Apache Parquet, Apache Iceberg as the dominant table format, REST catalogs (Polaris, Glue, Gravitino, Nessie, Lakekeeper), multi-engine compute (Spark, Trino, Flink, DuckDB, Athena, Snowflake, StarRocks), and the operational gap around compaction, snapshot management, and observability that no single open-source component owns. Cites adoption figures (Iceberg used by 38% of large enterprises evaluating it, up from 12% in 2024) and describes Iceberg V3 features shipping in 2026 (deletion vectors, row lineage, Variant type) alongside Delta Lake's UniForm compatibility layer. Throughout, the piece promotes LakeOps, an autonomous control plane for compaction, maintenance, routing, observability, and AI/MCP access, positioning it as the missing operational layer for production lakehouses.
Table of contents
How we got here: warehouse → lake → lakehouseWhat is an open data lakehouse?The state of the open data lakehouse in 2026The five core componentsComponent 1: Object storageComponent 2: Open file format (Apache Parquet)The in-memory complement: Apache ArrowComponent 3: Open table format (Apache Iceberg)What Iceberg providesHow it works, concretelyThe operational realityComponent 4: Catalog and governanceThe REST Catalog standardCatalog optionsWhy the catalog matters so muchComponent 5: Compute enginesThe routing challengeThe operational layer: what nobody tells youQuestions this post answers
Why is Apache Iceberg considered the dominant table format instead of Delta Lake?
Apache Iceberg has broader engine support, a richer feature set, and the largest tooling ecosystem, and with Iceberg V3 it gained deletion vectors, row lineage, and the Variant data type, closing gaps that previously gave Delta Lake an edge. Databricks, Delta Lake's creator, shipped full Iceberg V3 support and added UniForm so Delta tables can be exposed as Iceberg-compatible, effectively ending the format competition in Iceberg's favor. Compare table format tradeoffs on daily.dev before locking in an architecture decision.
What causes Apache Iceberg tables to degrade in performance over time and how bad can it get?
Iceberg tables degrade when small files, snapshots, and delete files accumulate without active maintenance: a streaming job with 10-minute commits can create about 4,300 files per month, and above 20,000 files query planning alone can take 15-30 seconds versus under one second for properly compacted files. High-frequency ingestion can also produce 10,000+ snapshots per day, bloating metadata JSON past 400 MB and slowing catalog operations. Track table-health gotchas like this before they turn into production incidents, via daily.dev.
What is the difference between binpack and sort compaction strategies in Apache Iceberg?
Binpack compaction merges small files into larger ones without reordering data, reducing file count and metadata overhead but leaving row order unchanged. Sort compaction merges files and also physically reorders data by chosen columns, so Parquet min/max statistics let engines skip 90%+ of files during reads; it costs more to run but the read-side savings compound across every query and engine, and intelligent query-aware sorting has produced 2x to 12x query improvement over basic compaction. Weigh compaction strategy tradeoffs for Iceberg tables with daily.dev's developer feed.