title: Understanding Spark Structured Streaming Parallelism
body: <p>I'm a newbie in the Spark world and struggling with some concepts.  </p>
  
  <p>How does parallelism happen when using Spark Structured Streaming sourcing from Kafka ?</p>
  
  <p>Let's consider the following snippet code:</p>
  
  <pre class="lang-java prettyprint-override"><code>SparkSession spark = SparkSession
            .builder()
            .appName("myApp")
            .getOrCreate();   
  
  Dataset&lt;VideoEventData&gt; ds = spark
    .readStream()
    .format("kafka")
    ...
  
  gDataset = ds.groupByKey(...)
  
  pDataset = gDataset.mapGroupsWithState(
        ...
        /* process each key - values */
        loop values
          if value is valid - save key/value result in the HDFS
        ... 
  )
  
  StreamingQuery query = pDataset.writeStream()
            .outputMode("update")
            .format("console")
            .start();
  
  //await
  query.awaitTermination();
  </code></pre>
  
  <p>I've read that the parallelism is related with the number of data partitions, and the number of partitions for a Dataset is based on the <code>spark.sql.shuffle.partitions</code> parameter.</p>
  
  <ol>
  <li><p>For every batch (pull from the Kafka), will the pulled items be divided among the number of <code>spark.sql.shuffle.partitions</code>? For example, <code>spark.sql.shuffle.partitions=5</code> and <code>Batch1=100</code> rows, will we end up with 5 partitions with 20 rows each ?</p></li>
  <li><p>Considering the snippet code provided, do we still leverage in the Spark parallelism due to the <code>groupByKey</code> followed by a <code>mapGroups/mapGroupsWithState</code> functions ?</p></li>
  </ol>
  
  <p><strong>UPDATE:</strong></p>
  
  <p>Inside the <code>gDataset.mapGroupsWithState</code> is where I process each key/values and store the result in the HDFS. So, the output sink is being used only to output some stats in the console.</p>