ClickHouse replaced its native Delta Lake implementation with the Rust Delta Kernel (delta-kernel-rs), offloading protocol handling to a shared abstraction layer. This enabled faster delivery of features like ACID writes, schema evolution, time travel with Change Data Feed, partition pruning, and statistics-based data skipping. The post details the significant engineering challenges of embedding a Rust library into a C++ codebase with strict build constraints: vendoring all crates, handling OpenSSL static linking, fixing cross-compilation issues, and dealing with sanitizer symbol failures. The team also contributed upstream improvements including dynamic logging configuration and asynchronous metadata processing for better scalability. Remaining gaps include inability to create empty Delta tables and plans for CDC-oriented workflows in ClickPipes.
