title: Spark parquet part number not sequential?
body: <p>I am using spark 3.0 to transform and sort data. Something which is worrying me is that I partition the data into 250 buckets, sort and then save. What I see in the output directory is typically 160 files with the file name starting with part-0000-** but the the number sequence is not consistent:</p>
  
  <pre><code>part-00000-f667f314-69f2-40d0-ba8a-b5b934650158-c000.gz.parquet
  part-00002-f667f314-69f2-40d0-ba8a-b5b934650158-c000.gz.parquet
  part-00005-f667f314-69f2-40d0-ba8a-b5b934650158-c000.gz.parquet
  </code></pre>
  
  <p>So what happened to parts 1, 3 and 4? This occurs throughout the files. Am I loosing data or is spark merging partitions for some reason? Code is like this:</p>
  
  <pre><code> df = df.withColumn("bucket", F.abs(F.col("Id")) % F.lit(250))
   df = df.repartition(250,"bucket")
   df = df.sortWithinPartitions("id")
   df.write.option("compression", "gzip").parquet(outputPath)
  </code></pre>