title: pyspark: Efficiently have partitionBy write to same number of total partitions as original table
body: <p>I had a question that is related to pyspark's <code>repartitionBy()</code> function which I originally posted in a comment on <a href="https://stackoverflow.com/questions/40416357/spark-sql-difference-between-df-repartition-and-dataframewriter-partitionby/">this question</a>. I was asked to post it as a separate question, so here it is:</p>
  
  <p>I understand that <code>df.partitionBy(COL)</code> will write all the rows with each value of <code>COL</code> to their own folder, and that each folder will (assuming the rows were previously distributed across all the partitions by some other key) have roughly the same number of files as were previously in the entire table. I find this behavior annoying. If I have a large table with 500 partitions, and I use <code>partitionBy(COL)</code> on some attribute columns, I now have for example 100 folders which <em>each</em> contain 500 (now very small) files.</p>
  
  <p>What I would like is the <code>partitionBy(COL)</code> behavior, but with roughly the same file size and number of files as I had originally.</p>
  
  <p>As demonstration, the previous question shares a toy example where you have a table with 10 partitions and do <code>partitionBy(dayOfWeek)</code> and now you have 70 files because there are 10 in each folder. I would want ~10 files, one for each day, and maybe 2 or 3 for days that have more data.</p>
  
  <p>Can this be easily accomplished? Something like <code>df.write().repartition(COL).partitionBy(COL)</code> seems like it might work, but I worry that (in the case of a very large table which is about to be partitioned into many folders) having to first combine it to some small number of partitions <em>before</em> doing the <code>partitionBy(COL)</code> seems like a bad idea.</p>
  
  <p>Any suggestions are greatly appreciated!</p>