title: Why Spark DataFrame is creating wrong number of partitions?
body: <p>I have a spark dataframe with 2 columns - <code>col1</code> and <code>col2</code>.</p>
  
  <pre><code>scala&gt; val df = List((1, "a")).toDF("col1", "col2")
  df: org.apache.spark.sql.DataFrame = [col1: int, col2: string]
  </code></pre>
  
  <p>When I write <code>df</code> on disk in <code>parquet</code> format, to write all the data in number of files equal to the number of unique values in <code>col1</code> I do a <code>repartition</code> using <code>col1</code>, like this:</p>
  
  <pre><code>scala&gt; df.repartition(col("col1")).write.partitionBy("col1").parquet("file")
  </code></pre>
  
  <p>Above code produces only one file in filesystem. But, the number of shuffle operations becomes 200.</p>
  
  <p><a href="https://i.stack.imgur.com/WKGRn.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/WKGRn.png" alt="enter image description here"></a></p>
  
  <p>I am not able to understand one thing here that if <code>col1</code> contains only one value, i.e., <code>1</code> then why it is creating 200 partitions in <code>repartition</code> ?</p>