Lambda Architecture - How to implement the Merge Layer / Query Layer
What you are asking about is, in my view, the main problem of implementing a lambda architecture.  Here are some suggestions on how to solve it.

The combination of Spark and Spark Streaming largely supersedes the original lambda architecture (which usually involved Hadoop and Storm).  Read here an example of how to use a SparkContext and a separate StreamingContext to produce different RDDs, one for batch processed results and another for real-time results.

Once you have replicated that in your system, you still have to think about how to query both kind of RDDs.  The trivial case would be to just union both of them:

scala&gt; rdd1.union(rdd2).collect


Or maybe you can create a new DStream, similar to stateStream in the linked example, where some keys are kept for real-time results, and others for batch results.