title: Using dropDuplicates in dataframe causes changes in the partition number
body: <p>I have a large dataframe which I created with 800 partitions.</p>
  
  
  
  <pre class="lang-py prettyprint-override"><code>df.rdd.getNumPartitions()
  800
  </code></pre>
  
  <p>When I use dropDuplicates on the dataframe, it changes the partitions to default 200</p>
  
  <pre class="lang-py prettyprint-override"><code>df = df.dropDuplicates()
  df.rdd.getNumPartitions()
  200
  </code></pre>
  
  <p>This behaviour causes problem for me, as it will lead to out of memory.</p>
  
  <p>Do you have any suggestion on fixing this problem? I tried setting   spark.sql.shuffle.partition to 800 but it doesn't work. Thanks</p>