title: Spark Streaming Union Stream - parallelization
body: <p>This may be a basic question, but I am having some trouble understanding this.</p>
  
  <p>I am currently using the Microsoft Azure Event Hubs Streaming in my Spark/Scala application which is similar to Kafka.</p>
  
  <p>If I created a Unionized stream, I believe this unions multiple DStream objects abstracted to look like a single DStream, will the multiple RDDs in the stream be processed in parallel, or will each RDD be processed individually?</p>
  
  <p>To try and explain this more, here is a quick example:</p>
  
  <pre><code>sparkConf.set(SparkArgumentKeys.MaxCores, (partitionCount * 2).toString)
  
  val ssc = new StreamingContext(sparkConf, streamDuration)
  
  val stream = EventHubsUtils.createUnionStream(ssc, hubParams, storageLevel)
  stream.checkpoint(streamDuration)
  
  val strings = stream.map(f =&gt; new String(f))
  strings.foreachRDD(rdd =&gt; {
    rdd.map(f =&gt; f.split(' '))
  })
  </code></pre>
  
  <p>partitionCount is the number of partitions in the azure event hub.</p>
  
  <ol>
  <li>Does the initial "stream.map" perform on each RDD in parallel?</li>
  <li>Does "string.foreachRDD" process a single RDD at a time, or does it process all the RDDs in some parallel manner?</li>
  </ol>