Databricks–Redis Real-Time Personalization
Published September 8, 2026
Problem
Application teams need recommendations that reflect what a customer is doing within the current session, rather than the last batch refresh.
My contribution
I co-authored the guide with Anant Pingle at Databricks, explaining the integration between continuous stream processing and Redis serving through a runnable reference demo.
Key decisions
- Separate continuous computation in Databricks Real-Time Mode from low-latency application reads in Redis, instead of querying analytics systems on the request path.
- Use Spark’s ForeachWriter for the real-time sink because the Redis Spark Connector did not support Real-Time Mode when the guide was published; foreachBatch would reintroduce micro-batching.
- Use co-located Sorted Sets and Hashes, bounded per-user state, TTLs, and acknowledged pipelined writes. The demo’s simple scoring is not a production ML ranking model.
Evidence
Published September 8, 2026, with Anant Pingle (Databricks). The reference demo used Azure Event Hubs’ Kafka interface and Azure Managed Redis, 10,000 active users, and a 10-worker cluster (160 vCPU). It sustained 100,000 events per second at 157 ms p99 end-to-end latency with no backlog. These are demo benchmark results, not customer production results or a performance guarantee.
Technical detail
I co-authored this technical article with Anant Pingle at Databricks to show how streaming data becomes personalized application experiences. Databricks Real-Time Mode processes customer activity as it arrives, while Redis serves the latest recommendations to the application.
The article walks through an e-commerce example, from clickstream ingestion and session scoring to Redis data modeling and application reads, with a runnable demo and performance results.
Challenges & Solutions
Keeping recommendations current within a browsing session through continuous event processing.
Connecting Databricks Real-Time Mode to Redis through Spark’s ForeachWriter.
Serving ranked products efficiently with Redis Sorted Sets, Hashes, pipelined operations, and expiration.
Outcomes
Published a joint Redis–Databricks walkthrough with implementation code and architecture guidance.
Documented a test sustaining 100,000 events per second with 157 ms p99 end-to-end latency using Azure Managed Redis.
Explained how the pattern extends to fraud scoring, online ML features, and inventory updates.
Tech Stack
Explore
Philip Laussermair
Global Azure Managed Redis Lead Architect at Redis.