title: Pyspark dataframe repartitioning puts all data in one partition
body: <p>I have a dataframe with schema as follows:</p>
  
  <pre><code>root
   |-- category_id: string (nullable = true)
   |-- article_title: string (nullable = true)
  </code></pre>
  
  <p>And data that looks likes this:</p>
  
  <pre><code>+-----------+--------------------+
  |category_id|     articletitle   |
  +-----------+--------------------+
  |       1000|HP EliteOne 800 G...|
  |       1000|ASUS  EB1501P ATM...|
  |       1000|HP EliteOne 800 G...|
  |          1|ASUS R557LA-XO119...|
  |          1|HP EliteOne 800 G...|
  +-----------+--------------------+
  </code></pre>
  
  <p>There are just two distinct <code>category_id</code> 1000 and 1. </p>
  
  <p>I want to do a repartition by <code>category_id</code> and <code>mapPartition</code> on each of the partitions. </p>
  
  <pre><code>p_df = df.repartition(2, "category_id")
  p_df.rdd.mapPartitionsWithIndex(some_func)
  </code></pre>
  
  <p>But data is not getting partitioned correctly, the expected result is that each mappartition will have data only for one <code>category_id</code>. But actual result is that one partition gets 0 records while the other gets all the records.</p>
  
  <p>Why is this happening and how to fix this?</p>
  
  <p>There is already a <a href="https://stackoverflow.com/questions/31424396/how-does-hashpartitioner-work">question</a> on how spark partitioner works. My question is different as the answers contain only explanation on how the partitioner works, but my question is about why this happens (which is answered already) and how to fix it.</p>