title: Spark (streaming) RDD foreachPartitionAsync functionality/working
body: <p>I will come to actual question but please bear with my use-case first. I have the following use-case, say I got <code>rddStud</code> from somewhere:</p>
  
  <p><code>val rddStud: RDD[(String,Student)] = ???</code></p>
  
  <p>Where 'String' - some random string and 'Student' - <code>case class Student(name: String, id: String, arrivalTime: Long, classId: String)</code></p>
  
  <p>I am using <em>Student</em> only as an example - actual business logic has much different complicated class with many fields.</p>
  
  <p><strong>What I want to achieve is</strong> - students with same <code>id</code> must be processed in ascending order of their <code>arrivalTime</code>.</p>
  
  <p>For this here's what I am doing:</p>
  
  <pre><code>//Get RDD from Student.id -&gt; Student
  val studMapRdd: RDD[(String,Student)] = rddStud.map(tuple =&gt; { 
    val student = tuple._2
    (student.id,student)
  })
  
  //Make sure all students with same student.id are in same partition.
  //I can potentially use groupByKey/combineByKey.... etc, but I don't see much performance difference    
  val studPartitionRdd: RDD[(String,Student)] = studMapRdd.partitionBy(new HashPartitioner(studMapRdd.getNumPartitions))
  
  val studSortedRdd: RDD[(String,Student)] = studPartitionRdd.sortBy({ case(studentId,student} =&gt; 
      student.arrivalTime
   }, ascending = true)
  
  studSortedRdd.foreachPartition(itr =&gt;{
      itr.foreach{ case (studentId, student) =&gt; { 
         val studentName = student.name
         val time = student.arrivalTime 
         //send for additional processing studentName and time combination
      }
  })
  </code></pre>
  
  <p>My questions are:</p>
  
  <ol>
  <li>If I use foreachPartitionAsync - will it process all partitions parallely, but the elements in each partition in order? If not, what's the difference between foreachPartitionAsync and foreachAsync then?</li>
  <li>Does the approach of sorting after repartitioning seem reasonable? Or if you could suggest any optimizations in above logic?</li>
  </ol>
  
  <p>Much appreciated.</p>