An exploration of using DuckDB inside AWS Lambda functions to ingest CSV data directly into Unity Catalog-governed Delta Lake tables, enabling concurrent writes without Spark clusters. The post argues that catalog-mediated commits (Catalog Commits) are the missing piece for safe multi-engine lakehouse writes, and walks through a working Dockerfile and Lambda handler that attaches to Unity Catalog via DuckDB's unity_catalog extension. The honest punchline: the full concurrent ingestion capability doesn't work yet, but the architecture is compelling and worth watching.
156 Impressions