Announcing General Availability of Real-Time Mode for Spark Declarative Pipelines on Databricks

Databricks announced that Real-Time Mode in Spark Declarative Pipelines is now generally available. The company says the mode brings millisecond streaming to the same framework customers already use for ETL, including fraud scoring, live alerts, and features written into operational stores.

• RTM streams data continuously, runs all query stages at once, and moves records between stages as soon as they are produced.
• SDP already manages dependency resolution, incremental processing, orchestration, failure recovery, and data quality checks.
• Serverless RTM skips cluster sizing and bills only cores a query uses, with lower-cost packing and a native Lakebase sink listed as next steps.

• up to 100x faster — than traditional micro-batch, claimed when RTM launched last year
• up to 92% faster — than Flink on customer feature-computation workloads in Databricks benchmarks
• 200ms — Feature Store end-to-end p99 latency from Kafka to the online store
• last 10 minutes — example rolling window for a user transaction-amount feature
• sub-second latency — claimed for SDP serverless without complex infrastructure tuning

RTM was introduced last year as an evolution of Spark Structured Streaming.

Speed claims come from Databricks benchmarks on selected customer workloads, with dataset details in a linked repository.