--- title: Announcing General Availability of Real-Time Mode for... description: Real-Time Mode (RTM) for Spark Declarative Pipelines (SDP) on Databricks is now generally available. It streams data continuously, runs all query stages... canonical: https://daily.dev/posts/announcing-general-availability-of-real-time-mode-for-spark-declarative-pipelines-on-databricks-fuxqnk0k6 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Announcing General Availability of Real-Time Mode for Spark Declarative Pipelines on Databricks | daily.dev og:description: Real-Time Mode (RTM) for Spark Declarative Pipelines (SDP) on Databricks is now generally available. It streams data continuously, runs all query stages... og:url: https://daily.dev/posts/announcing-general-availability-of-real-time-mode-for-spark-declarative-pipelines-on-databricks-fuxqnk0k6 og:image: https://api.daily.dev/og/posts/FUxQNK0k6.png og:image:alt: Announcing General Availability of Real-Time Mode for Spark Declarative Pipelines on Databricks og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Announcing General Availability of Real-Time Mode for Spark Declarative Pipelines on Databricks **[databricks](https://daily.dev/sources/databricks)** · 6 min read · 0 upvotes · 0 comments ## Summary Real-Time Mode (RTM) for Spark Declarative Pipelines (SDP) on Databricks is now generally available. It streams data continuously, runs all query stages simultaneously, and achieves end-to-end latency in the millisecond range. Databricks benchmarked RTM against Apache Flink on real customer feature computation workloads and found it up to 92% faster. To enable it, set the pipeline mode to 'continuous' and spark.databricks.streaming.realTimeMode.enabled to true, then define an update flow with pipelines.trigger set to 'RealTime'. Pipelines can write to Lakebase, Redis, or other OLTP stores via a ForEach sink, and the Databricks Feature Store uses this architecture to deliver end-to-end p99 latency of 200ms from Kafka to the online feature store. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How do I enable real-time mode in Spark Declarative Pipelines on Databricks? Set the pipeline mode to 'continuous' and set the Spark config spark.databricks.streaming.realTimeMode.enabled to true. Then define an update flow using the @dp.update_flow decorator with pipelines.trigger set to "RealTime" and a target pointing to a sink created with dp.create_sink(). _Teams wiring up low-latency pipelines can track Spark Declarative Pipelines updates like this on daily.dev._ ### How does Databricks Real-Time Mode performance compare to Apache Flink? Databricks benchmarked Real-Time Mode against Apache Flink on actual customer workloads performing feature computation and found RTM up to 92% faster than Flink. This lets teams drop a separate specialized streaming engine and run operational, low-latency workloads directly on Spark Declarative Pipelines instead. _Anyone comparing streaming engines for a migration can follow benchmarks like this on daily.dev._ ### Can a Spark Declarative Pipeline write directly to an OLTP store like Lakebase or Redis in real time? Yes, a real-time flow can write to Lakebase, Redis, or any other OLTP store using a ForEach sink. Define a writer that implements open, process and close, register it as a foreach_sink, and wire it to an update_flow; features then land in the operational store within milliseconds of being produced. _Engineers building real-time feature stores can keep up with sink options like these on daily.dev._ ## Similar posts on daily.dev - [Announcing General Availability of Real-Time Mode for Apache Spark Structured Streaming on Databricks](https://daily.dev/posts/announcing-general-availability-of-real-time-mode-for-apache-spark-structured-streaming-on-databrick-vdvvbvesy) · databricks · 4 upvotes · 0 comments - [Real-Time Mode: Ultra-low latency streaming on Spark APIs without a second engine](https://daily.dev/posts/real-time-mode-ultra-low-latency-streaming-on-spark-apis-without-a-second-engine-vyvg1l0h9) · databricks · 7 upvotes · 0 comments - [Breaking the Microbatch Barrier: The Architecture of Apache Spark Real-Time Mode](https://daily.dev/posts/breaking-the-microbatch-barrier-the-architecture-of-apache-spark-real-time-mode-hzjiecldg) · databricks · 0 upvotes · 0 comments - [Delivering Real-Time Personalization with Databricks and Redis](https://daily.dev/posts/delivering-real-time-personalization-with-databricks-and-redis-dzhnwe0pt) · Redis · 0 upvotes · 0 comments - [Apache Spark Real-Time Mode for Gaming: A Better Way to Do Real-Time Sessionization](https://daily.dev/posts/apache-spark-real-time-mode-for-gaming-a-better-way-to-do-real-time-sessionization-pxo2d4oxr) · databricks · 0 upvotes · 0 comments --- Tags: [#data-engineering](https://daily.dev/tags/data-engineering), [#apache-spark](https://daily.dev/tags/apache-spark), [#databricks](https://daily.dev/tags/databricks), [#apache-flink](https://daily.dev/tags/apache-flink) [View this post on daily.dev](https://daily.dev/posts/announcing-general-availability-of-real-time-mode-for-spark-declarative-pipelines-on-databricks-fuxqnk0k6) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Announcing General Availability of Real-Time Mode for Spark Declarative Pipelines on Databricks","url":"https://daily.dev/posts/announcing-general-availability-of-real-time-mode-for-spark-declarative-pipelines-on-databricks-fuxqnk0k6","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/announcing-general-availability-of-real-time-mode-for-spark-declarative-pipelines-on-databricks-fuxqnk0k6"},"datePublished":"2026-10-29T22:47:52.786Z","dateModified":"2026-10-29T22:51:45.373Z","description":"Real-Time Mode (RTM) for Spark Declarative Pipelines (SDP) on Databricks is now generally available. It streams data continuously, runs all query stages...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d3d71da304590cca23412f1be837fdab?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/d3d71da304590cca23412f1be837fdab?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"databricks","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"databricks","logo":"https://media.daily.dev/image/upload/t_logo,f_auto/v1/logos/fa7aa720f2db4d1eba826814730482c8","url":"https://daily.dev/sources/databricks"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/announcing-general-availability-of-real-time-mode-for-spark-declarative-pipelines-on-databricks-fuxqnk0k6","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"data-engineering,apache-spark,databricks,apache-flink","timeRequired":"PT6M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"databricks","item":"https://daily.dev/sources/databricks"},{"@type":"ListItem","position":3,"name":"Announcing General Availability of Real-Time Mode for Spark Declarative Pipelines on Databricks"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/announcing-general-availability-of-real-time-mode-for-spark-declarative-pipelines-on-databricks-fuxqnk0k6#faq","mainEntity":[{"@type":"Question","name":"How do I enable real-time mode in Spark Declarative Pipelines on Databricks?","acceptedAnswer":{"@type":"Answer","text":"Set the pipeline mode to 'continuous' and set the Spark config spark.databricks.streaming.realTimeMode.enabled to true. Then define an update flow using the @dp.update_flow decorator with pipelines.trigger set to \"RealTime\" and a target pointing to a sink created with dp.create_sink(). Teams wiring up low-latency pipelines can track Spark Declarative Pipelines updates like this on daily.dev."}},{"@type":"Question","name":"How does Databricks Real-Time Mode performance compare to Apache Flink?","acceptedAnswer":{"@type":"Answer","text":"Databricks benchmarked Real-Time Mode against Apache Flink on actual customer workloads performing feature computation and found RTM up to 92% faster than Flink. This lets teams drop a separate specialized streaming engine and run operational, low-latency workloads directly on Spark Declarative Pipelines instead. Anyone comparing streaming engines for a migration can follow benchmarks like this on daily.dev."}},{"@type":"Question","name":"Can a Spark Declarative Pipeline write directly to an OLTP store like Lakebase or Redis in real time?","acceptedAnswer":{"@type":"Answer","text":"Yes, a real-time flow can write to Lakebase, Redis, or any other OLTP store using a ForEach sink. Define a writer that implements open, process and close, register it as a foreach_sink, and wire it to an update_flow; features then land in the operational store within milliseconds of being produced. Engineers building real-time feature stores can keep up with sink options like these on daily.dev."}}]} ```