Part 8 of a 15-part Apache Iceberg masterclass explores embedded catalogs, where the storage layer itself manages Iceberg metadata instead of relying on a separate catalog service. It covers AWS S3 Tables, launched in late 2024 as a dedicated S3 bucket type with built-in compaction, snapshot expiry, and orphan cleanup, and MinIO AI Stor, an on-premises alternative with S3 compatibility and GPU-accelerated analytics. Both trade flexibility for operational simplicity but come with vendor lock-in, limited monitoring, and cross-region latency concerns. The piece also discusses hybrid approaches combining embedded and standalone catalogs like Dremio Open Catalog for unified multi-source analytics.
Table of contents
Table of ContentsThe Embedded Catalog ModelAWS S3 TablesMinIO AI StorWhen Embedded Catalogs Make SenseThe Hybrid ApproachOperational Planning for Embedded CatalogsTry Dremio Cloud free for 30 daysQuestions this post answers
What is AWS S3 Tables and how does it embed the Iceberg catalog?
AWS S3 Tables is a dedicated S3 bucket type, launched in late 2024, designed specifically for Apache Iceberg tables. It manages the Iceberg catalog internally, handling metadata files, atomic commits, compaction, and snapshot management within the storage layer, so engines interact with a single endpoint for both catalog operations and data access via a REST-catalog-compatible interface. Track how storage-embedded catalogs like S3 Tables reshape lakehouse architecture decisions on daily.dev.
What are the downsides of using an embedded catalog like S3 Tables instead of a standalone Iceberg catalog?
Embedded catalogs create vendor dependency since tables are tied to the storage vendor's lifecycle, and migrating away requires converting every table to a different catalog. They also offer limited monitoring of internal maintenance like compaction schedules, add cross-region latency if compute runs elsewhere, and in S3 Tables' case, limit governance to what AWS IAM provides without column or row-level controls. Compare catalog trade-offs before locking a lakehouse architecture in, informed by discussions on daily.dev.
How does MinIO AI Stor implement an embedded Iceberg catalog for on-premises deployments?
MinIO AI Stor embeds Iceberg catalog functionality directly into its S3-compatible storage layer, managing table metadata, file tracking, and maintenance internally as data is written. Because it implements the S3 API, engines like Spark, Trino, and Dremio can interact with it with minimal configuration changes, and it also integrates GPU-aware processing and vector storage for AI workloads alongside Iceberg tables. Follow on-premises lakehouse tooling like MinIO AI Stor as options evolve, curated on daily.dev.