Delta Lake 0.6.0 adds automatic schema evolution during merge operations, improved merge performance through automatic repartitioning and optimizations for merges without insert clauses, operation metrics in DESCRIBE HISTORY, and support for reading Delta tables from any file system with a Hadoop FileSystem implementation (writes still require a LogStore implementation).
Questions this post answers
Does Delta Lake 0.6.0 support automatic schema evolution during merge operations?
Yes, Delta Lake 0.6.0 supports automatic schema evolution in merge operations, letting the table schema evolve simultaneously with an upsert instead of requiring you to detect and apply schema changes beforehand. This is useful when upserting change data whose schema changes over time. Track Delta Lake merge and schema features like this one through daily.dev as your pipelines evolve.
How can I speed up slow merge operations on partitioned Delta Lake tables?
Delta Lake 0.6.0 introduces automatic repartitioning during merge into partitioned tables, which addresses slowdowns caused by merges generating too many small files. Enabling automatic repartition by the partition columns before writing improves merge performance in these cases. Developers tuning Delta Lake merge performance can follow updates like this on daily.dev.
Can Delta Lake read tables from any file system, not just specific storage backends?
Yes, as of version 0.6.0, Delta Lake can read tables from any storage system that has a Hadoop FileSystem implementation. Writing to Delta tables, however, still requires configuring a LogStore implementation to guarantee consistency on that storage system. Anyone choosing storage backends for Delta Lake can follow capability changes like this via daily.dev.