title: Apache spark streaming - cache dataset for joining
body: <p>I'm considering using Apache Spark streaming for some real-time work but I'm  not sure how to cache a dataset for use in a join/lookup.</p>
  
  <p>The main input will be json records coming from Kafka that contain an Id, I want to translate that id into a name using a lookup dataset. The lookup dataset resides in Mongo Db but I want to be able to cache it inside the spark process as the dataset changes very rarely (once every couple of hours) so I don't want to hit mongo for every input record or reload all the records in every spark batch but I need to be able to update the data held in spark periodically (e.g. every 2 hours).</p>
  
  <p>What is the best way to do this?</p>
  
  <p>Thanks.</p>