Databricks is bringing declarative ETL patterns to the Lakehouse SQL Editor, letting SQL analysts define APPEND, AUTO CDC, and REPLACE WHERE flows directly in queries instead of relying solely on the dedicated Lakeflow Pipelines Editor. APPEND flows handle incremental ingestion, AUTO CDC simplifies change-data-capture handling for SCD Type 1/2 without hand-written merge logic, and REPLACE WHERE enables targeted batch recomputation, which Databricks claims runs 3.4x faster and 2.5x cheaper than traditional REPLACE WHERE when powered by its Enzyme incrementalization engine. Genie Code can help generate and refine these declarative flows, and users can mix declarative and procedural SQL, moving to the Pipelines Editor as projects scale.
Table of contents
Simplifying recurring ETL patterns in LakehouseThe first declarative primitives available in LakehouseTl;dr: Why this matters for SQL practitionersFrom declarative primitives to full declarative pipelinesUse Genie Code to get started fasterQuestions this post answers
What is the AUTO CDC feature in Databricks Lakehouse SQL Editor?
AUTO CDC is a declarative change-data-capture feature now available directly in the Databricks Lakehouse SQL Editor, letting SQL users define keys, sequencing, delete handling, and SCD Type 1 or Type 2 storage with a few lines of code instead of hand-writing complex MERGE INTO logic to handle out-of-order CDC data. daily.dev helps data engineers track how CDC tooling like Databricks AUTO CDC keeps evolving.
How much faster is Enzyme-powered REPLACE WHERE compared to traditional REPLACE WHERE in Databricks?
Enzyme-powered REPLACE WHERE ran 3.4x faster and 2.5x cheaper than traditional REPLACE WHERE in Databricks Lakehouse benchmark testing. Enzyme is Databricks' automatic incrementalization engine that identifies and processes only the data changed within a specified predicate, avoiding full table recomputation for targeted batch refreshes like backfills or schema evolution. engineers weighing incremental recompute strategies follow benchmark claims like this on daily.dev.
Can I use declarative ETL flows directly in the Databricks SQL Editor instead of the Lakeflow Pipelines Editor?
Yes, Databricks now supports APPEND, AUTO CDC, and REPLACE WHERE declarative flows directly within SQL queries in the Lakehouse SQL Editor, without needing the dedicated Lakeflow Pipelines Editor. Databricks handles scheduling, incremental processing, and orchestration automatically, while users can still move to the Pipelines Editor for larger, multi-file, team-based projects. daily.dev keeps SQL practitioners current on where declarative ETL tooling is heading next.