title: How can I update a broadcast variable in spark streaming?
body: <p>I have, I believe, a relatively common use case for spark streaming:</p>
  
  <p><strong>I have a stream of objects that I would like to filter based on some reference data</strong></p>
  
  <p>Initially, I thought that this would be a very simple thing to achieve using a <em>Broadcast Variable</em>:</p>
  
  <pre><code>public void startSparkEngine {
      Broadcast&lt;ReferenceData&gt; refdataBroadcast
        = sparkContext.broadcast(getRefData());
  
      final JavaDStream&lt;MyObject&gt; filteredStream = objectStream.filter(obj -&gt; {
          final ReferenceData refData = refdataBroadcast.getValue();
          return obj.getField().equals(refData.getField());
      }
  
      filteredStream.foreachRDD(rdd -&gt; {
          rdd.foreach(obj -&gt; {
              // Final processing of filtered objects
          });
          return null;
      });
  }
  </code></pre>
  
  <p>However, albeit infrequently, <strong>my reference data will change periodically</strong></p>
  
  <p>I was under the impression that I could modify and <em>re-broadcast</em> my variable on the driver and it would be propagated to each of the workers, however the <code>Broadcast</code> object is not <code>Serializable</code> and needs to be <code>final</code>.</p>
  
  <p>What alternatives do I have? The three solutions I can think of are:</p>
  
  <ol>
  <li><p>Move the reference data lookup into a <code>forEachPartition</code> or <code>forEachRdd</code> so that it resides entirely on the workers. However the reference data lives beind a REST API so I would also need to somehow store a timer / counter to stop the remote being accessed for every element in the stream.</p></li>
  <li><p>Restart the Spark Context every time the refdata changes, with a new Broadcast Variable.</p></li>
  <li><p>Convert the Reference Data to an <em>RDD</em>, then <code>join</code> the streams in such a way that I am now streaming <code>Pair&lt;MyObject, RefData&gt;</code>, though this will ship the reference data with every object.</p></li>
  </ol>