title: Why is HashPartioner needed in the StatefulNetworkWordCount in spark streaming example?
body: <p>My Question is regarding the StatefulNetworkWordCount example :</p>
  
  <p><a href="https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/StatefulNetworkWordCount.scala" rel="nofollow">https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/StatefulNetworkWordCount.scala</a></p>
  
  <p>Q1) The stateDstream RDD is maintained by the driver or the worker node or does each worker node has its own local copy of the complete state rdd?</p>
  
  <p>Q2) Why do we need a HashPartitioner in the following line :</p>
  
  <p><code>val stateDstream = wordDstream.updateStateByKey[Int](newUpdateFunc,
        new HashPartitioner (ssc.sparkContext.defaultParallelism), true, initialRDD)</code></p>
  
  <p>What is happening behind the scenes here ?</p>