title: Spark repartitioning by column with dynamic number of partitions per column
body: <p>How can a DataFrame be partitioned based on the count of the number of items in a column.  Suppose we have a DataFrame with 100 people (columns are <code>first_name</code> and <code>country</code>) and we'd like to create a partition for every 10 people in a country.</p>
  
  <p>If our dataset contains 80 people from China, 15 people from France, and 5 people from Cuba, then we'll want 8 partitions for China, 2 partitions for France, and 1 partition for Cuba.</p>
  
  <p>Here is code that will not work:</p>
  
  <ul>
  <li><code>df.repartition($"country")</code>: This will create 1 partition for China, one partition for France, and one partition for Cuba</li>
  <li><code>df.repartition(8, $"country", rand)</code>: This will create up to 8 partitions for each country, so it should create 8 partitions for China, but the France &amp; Cuba partitions are unknown.  France could be in 8 partitions and Cuba could be in up to 5 partitions.  See <a href="https://stackoverflow.com/questions/53037124/partitioning-a-large-skewed-dataset-in-s3-with-sparks-partitionby-method">this answer</a> for more details.</li>
  </ul>
  
  <p>Here's the <code>repartition()</code> documentation:</p>
  
  <p><a href="https://i.stack.imgur.com/LZXaj.png" rel="noreferrer"><img src="https://i.stack.imgur.com/LZXaj.png" alt="repartition documentation"></a></p>
  
  <p>When I look at the <code>repartition()</code> method, I don't even see a method that takes three arguments, so looks like some of this behavior isn't documented.</p>
  
  <p>Is there any way to dynamically set the number of partitions for each column?  It would make creating partitioned data sets way easier.</p>