title: Optimizing parquet files in pyspark
body: <p>I have a large computational job in pyspark that can output to a parquet format reasonably quickly, but generates thousands of ~3Mb files across N partitions (where N is known). I know that I can repartition the data across those partitions before writing, but the shuffle job from ~2000 nodes takes seemingly forever. Given this, I have a two-part question:</p>
  <ol>
  <li><p>Will it be better to repartition and then write to parquet in a single step, or write to parquet then reingest and rewrite to parquet again, combining the small files? Essentially, this performs a parquet optimization step at the end. Note that I'm not worried about storage space.</p>
  </li>
  <li><p>If I take the route of reingesting and rewriting, is the following the best way of doing this operation?</p>
  </li>
  </ol>
  <pre class="lang-py prettyprint-override"><code>df.repartition(N, 'partition_name').write.partitionBy('partition_name').saveAsTable(...)
  </code></pre>
  <p>It seems that the repartition command introduces another slow shuffle operation, which really shouldn't be necessary, given that the data is already partitioned.</p>