title: when to use mapParitions and mapPartitionsWithIndex?
body: <p>The PySpark documentation describes two functions:</p>
  
  <blockquote>
  <pre><code>mapPartitions(f, preservesPartitioning=False)
  
     Return a new RDD by applying a function to each partition of this RDD.
  
     &gt;&gt;&gt; rdd = sc.parallelize([1, 2, 3, 4], 2)
     &gt;&gt;&gt; def f(iterator): yield sum(iterator)
     &gt;&gt;&gt; rdd.mapPartitions(f).collect()
     [3, 7]
  </code></pre>
  </blockquote>
  
  <p>And ...</p>
  
  <blockquote>
  <pre><code>mapPartitionsWithIndex(f, preservesPartitioning=False)
  
     Return a new RDD by applying a function to each partition of this RDD, 
     while tracking the index of the original partition.
  
     &gt;&gt;&gt; rdd = sc.parallelize([1, 2, 3, 4], 4)
     &gt;&gt;&gt; def f(splitIndex, iterator): yield splitIndex
     &gt;&gt;&gt; rdd.mapPartitionsWithIndex(f).sum()
     6
  </code></pre>
  </blockquote>
  
  <p>What use cases do these functions attempt to solve?  I can't see why they would be required.</p>