title: SPARK - Joining two data streams - maintenance of cache
body: <p>It is evident that the out of box join capability in spark streaming does not warrent a lot of real life use cases. The reason being it joins only the data contained in the micro batch RDDs.</p>
  
  <p>Use case is to join data from two kafka streams and enrich each object in stream1 with it's corresponding object in stream2 in spark and save it to HBase.</p>
  
  <p>Implementation would </p>
  
  <ul>
  <li><p>maintain a dataset in memory from objects from stream2, adding or replacing objects as and when they are recieved</p></li>
  <li><p>for every element in stream1, access the cache to find a matching object from stream2, save to HBase if match is found or put it back on the kafka stream if not.</p></li>
  </ul>
  
  <p>This question is on exploration of Spark streaming and it's API to find a way to implement the above mentioned.</p>