title: How to define a global read\write variables in Spark
body: <p>Spark has <code>broadcast</code> variables, which are read only, and <code>accumulator</code> variables, which can be updates by the nodes, but not read. Is there way - or a workaround - to define a variable which is both updatable and can be read?</p>
  
  <p>One requirement for such a read\write global variable would be to implement a cache. As files are loaded and processed as rdd's, calculations are performed. The results of these calculations - happening in several nodes running in parallel - need to be placed into a map, which has as it's key some of the attributes of the entity being processed. As subsequent entities within the rdd's are processed, the cache is queried.</p>
  
  <p>Scala does have <code>ScalaCache</code>, which is a facade for cache implementations such as <code>Google Guava</code>. But how would such a cache be included and accessed within a Spark application?</p>
  
  <p>The cache could be defined as a variable in the driver application which creates the <code>SparkContext</code>. But then there would be two issues:</p>
  
  <ul>
  <li>Performance would presumably be bad because of the network overhead
  between the nodes and the driver application.</li>
  <li>To my understanding, each rdd will be passed a copy of the variable
  (cache in this case) when the variable is first accessed by the
  function passed to the rdd. Each rdd would have it's own copy, not access to a shared global variable .</li>
  </ul>
  
  <p>What is the best way to implement and store such a cache?</p>
  
  <p>Thanks</p>