Spark is Dead. Long Live DuckDB.

Apache Spark has peaked and is increasingly dominated by Databricks, which has turned it into a costly, complex enterprise product far removed from its open-source roots. DuckDB is emerging as a simpler, faster, and truly open-source alternative for the vast majority of data pipelines that don't actually need cluster compute. The author argues that most data workloads can be handled in-process with DuckDB — in Lambdas, Python scripts, Airflow jobs — and that the industry's addiction to the Lakehouse model and Databricks pricing will slowly give way to this leaner approach, even if the transition takes decades.