Lambda Architecture - How to implement the Merge Layer / Query Layer I am reading up about lambda architecture.

It makes sense. we have queue based data ingestion. we have an in-memory store for data which is very new and we have HDFS for old data.

So we have our entire data set. in our system. very good.

but the architecture diagram shows that the merge layer is able to query both the batch layer and the speed layer in one shot.

How to do that?

Your batch layer is probably a map reduce job or a HIVE query. The speed layer query is probably a scala program which is execution on the spark.

Now how will you merge these?

Is there any guidance.
What you are asking about is, in my view, the main problem of implementing a lambda architecture.  Here are some suggestions on how to solve it.

The combination of Spark and Spark Streaming largely supersedes the original lambda architecture (which usually involved Hadoop and Storm).  Read here an example of how to use a SparkContext and a separate StreamingContext to produce different RDDs, one for batch processed results and another for real-time results.

Once you have replicated that in your system, you still have to think about how to query both kind of RDDs.  The trivial case would be to just union both of them:

scala&gt; rdd1.union(rdd2).collect


Or maybe you can create a new DStream, similar to stateStream in the linked example, where some keys are kept for real-time results, and others for batch results.