title: Processing RDDs in a DStream in parallel
body: <p>I came across the following code which processes messages in Spark Streaming:</p>
  
  <pre><code>val listRDD = ssc.socketTextStream(host, port)
  listRDD.foreachRDD(rdd =&gt; {
    rdd.foreachPartition(partition =&gt; {
      // Should I start a separate thread for each RDD and/or Partition?
      partition.foreach(message =&gt; {
        Processor.processMessage(message)
      })
    })
  })
  </code></pre>
  
  <p>This is working for me but I am not sure if this is the best way. I understand that a DStream consists of "one to many" RDDs, but this code processes RDDs sequentially one after the other, right? Isn't there a better way - a method or function - that I can use so that all the RDDs in the DStream get processed in parallel? Should I start a separate thread for each RDD and/or Partition? Have I misunderstood how this code works under Spark?</p>
  
  <p>Somehow I think this code is not taking advantage of the parallelism in Spark. </p>