--- title: Introducing Apache Spark 4.2 | daily.dev description: Apache Spark 4.2 is now available in Databricks Runtime 19 Beta, delivering major new capabilities across four areas. A native semantic layer via metric views... canonical: https://daily.dev/posts/introducing-apache-spark-4-2-mv67nz1si twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Introducing Apache Spark 4.2 | daily.dev og:description: Apache Spark 4.2 is now available in Databricks Runtime 19 Beta, delivering major new capabilities across four areas. A native semantic layer via metric views... og:url: https://daily.dev/posts/introducing-apache-spark-4-2-mv67nz1si og:image: https://api.daily.dev/og/posts/Mv67Nz1Si.png og:image:alt: Introducing Apache Spark 4.2 og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Introducing Apache Spark 4.2 **[databricks](https://daily.dev/sources/databricks)** · 9 min read · 5 upvotes · 0 comments ## Summary Apache Spark 4.2 is now available in Databricks Runtime 19 Beta, delivering major new capabilities across four areas. A native semantic layer via metric views lets teams define governed business metrics once and query them consistently across BI, SQL, and AI tools. Python and Arrow integration improves with Arrow-optimized UDFs enabled by default, Pandas 3 support, Arrow C Data Interface interoperability with Polars and DuckDB, and Real-Time Mode extended to PySpark for millisecond-latency stateless streaming. New SQL primitives include vector similarity search with NEAREST BY, built-in GEOMETRY/GEOGRAPHY types with ST_* functions, SQL cursors, QUALIFY for window filtering, and time-series functions. First-class Auto CDC support in Spark Declarative Pipelines simplifies SCD Type 1 ingestion with a declarative Python API. Data Source V2 gains a standard CHANGES clause for querying change streams, transaction API foundations, and MERGE INTO performance improvements. Platform updates include a modernized Web UI with dark mode, Kubernetes heterogeneous executor management, and JDK 25 support, with over 1,900 commits from 260+ contributors. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### What new features does Apache Spark 4.2 add for AI and retrieval workloads? Apache Spark 4.2 adds native SQL primitives for vector similarity search including vector distance, similarity, and normalization functions, plus a NEAREST BY top-K ranking join for distance-based matching. It also introduces metric views, a governed semantic layer so agents and dashboards compute metrics like revenue consistently, and improves Spark Connect for embedding Spark as a remote execution service in AI applications. _Engineers building AI retrieval pipelines can track Spark's evolving vector and semantic features on daily.dev._ ### How does Auto CDC in Spark Declarative Pipelines simplify SCD Type 1 processing? Auto CDC, introduced in Spark 4.2's Spark Declarative Pipelines, lets users declaratively configure how change data capture events update a target table instead of writing manual merge logic. It handles deletions and out-of-order events automatically via a Python API, targeting use cases like customer profiles, product catalogs, and account records that need the latest record version maintained reliably. _Data engineers evaluating CDC ingestion patterns can follow Spark pipeline updates like this on daily.dev._ ### Does Apache Spark 4.2 support Real-Time Mode streaming in PySpark? Yes, Spark 4.2 extends Real-Time Mode (RTM) in Structured Streaming to PySpark, allowing stateless streaming queries without Python UDFs to run with millisecond end-to-end latency. Stateful support for RTM and Python UDF support in RTM are planned for a future Spark 4.x release and are tracked under SPARK-54699 and SPARK-57237. _Teams building low-latency streaming apps in Python can keep up with RTM's rollout on daily.dev._ ## Similar posts on daily.dev - [Announcing General Availability of Python Data Source API](https://daily.dev/posts/announcing-general-availability-of-python-data-source-api-kz7bygbgh) · databricks · 6 upvotes · 0 comments - [Apache Flink 2.2.0: Advancing Real-Time Data \+ AI and Empowering Stream Processing for the AI Era](https://daily.dev/posts/apache-flink-2-2-0-advancing-real-time-data-ai-and-empowering-stream-processing-for-the-ai-era-vyfow5bk8) · Flink · 5 upvotes · 0 comments - [Migrating from Apache Spark 3 to Spark 4](https://daily.dev/posts/migrating-from-apache-spark-3-to-spark-4-0py2476df) · Ubuntu · 0 upvotes · 0 comments --- Tags: [#data-engineering](https://daily.dev/tags/data-engineering), [#apache-spark](https://daily.dev/tags/apache-spark), [#change-data-capture](https://daily.dev/tags/change-data-capture), [#pyspark](https://daily.dev/tags/pyspark) [View this post on daily.dev](https://daily.dev/posts/introducing-apache-spark-4-2-mv67nz1si) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Introducing Apache Spark 4.2","url":"https://daily.dev/posts/introducing-apache-spark-4-2-mv67nz1si","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/introducing-apache-spark-4-2-mv67nz1si"},"datePublished":"2026-07-16T06:54:32.114Z","dateModified":"2026-09-14T09:09:18.591Z","description":"Apache Spark 4.2 is now available in Databricks Runtime 19 Beta, delivering major new capabilities across four areas. A native semantic layer via metric views...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/e8ffb39ec367ce369b9b4ccff635f2f0?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/e8ffb39ec367ce369b9b4ccff635f2f0?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"databricks","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"databricks","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/fa7aa720f2db4d1eba826814730482c8","url":"https://daily.dev/sources/databricks"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/introducing-apache-spark-4-2-mv67nz1si","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":5},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-engineering,apache-spark,change-data-capture,pyspark","timeRequired":"PT9M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"databricks","item":"https://daily.dev/sources/databricks"},{"@type":"ListItem","position":3,"name":"Introducing Apache Spark 4.2"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/introducing-apache-spark-4-2-mv67nz1si#faq","mainEntity":[{"@type":"Question","name":"What new features does Apache Spark 4.2 add for AI and retrieval workloads?","acceptedAnswer":{"@type":"Answer","text":"Apache Spark 4.2 adds native SQL primitives for vector similarity search including vector distance, similarity, and normalization functions, plus a NEAREST BY top-K ranking join for distance-based matching. It also introduces metric views, a governed semantic layer so agents and dashboards compute metrics like revenue consistently, and improves Spark Connect for embedding Spark as a remote execution service in AI applications. Engineers building AI retrieval pipelines can track Spark's evolving vector and semantic features on daily.dev."}},{"@type":"Question","name":"How does Auto CDC in Spark Declarative Pipelines simplify SCD Type 1 processing?","acceptedAnswer":{"@type":"Answer","text":"Auto CDC, introduced in Spark 4.2's Spark Declarative Pipelines, lets users declaratively configure how change data capture events update a target table instead of writing manual merge logic. It handles deletions and out-of-order events automatically via a Python API, targeting use cases like customer profiles, product catalogs, and account records that need the latest record version maintained reliably. Data engineers evaluating CDC ingestion patterns can follow Spark pipeline updates like this on daily.dev."}},{"@type":"Question","name":"Does Apache Spark 4.2 support Real-Time Mode streaming in PySpark?","acceptedAnswer":{"@type":"Answer","text":"Yes, Spark 4.2 extends Real-Time Mode (RTM) in Structured Streaming to PySpark, allowing stateless streaming queries without Python UDFs to run with millisecond end-to-end latency. Stateful support for RTM and Python UDF support in RTM are planned for a future Spark 4.x release and are tracked under SPARK-54699 and SPARK-57237. Teams building low-latency streaming apps in Python can keep up with RTM's rollout on daily.dev."}}]} ```