title: spark.sql.shuffle.partitions of 200 default partitions conundrum
body: <p>In many posts there is the statement - as shown below in some form or another - due to some question on shuffling, partitioning, due to JOIN, AGGR, whatever, etc.:</p>
  <p><em>... In general whenever you do a spark sql aggregation or join which shuffles data this is the number of <strong>resulting</strong> partitions = 200.
  This is set by spark.sql.shuffle.partitions. ...</em></p>
  <p>So, my question is:</p>
  <ul>
  <li>Do we mean that if we have set partitioning at 765 for a DF, for example,</li>
  <li>That the processing occurs against 765 partitions, but that the output is coalesced / re-partitioned standardly to 200 - referring here to word <strong>resulting</strong>?</li>
  <li>Or does it do the processing using 200 partitions after coalescing / re-partitioning to 200 partitions before JOINing, AGGR?</li>
  </ul>
  <p>I ask as I never see a clear viewpoint.</p>
  <p>I did the following test:</p>
  <pre><code>// genned a DS of some 20M short rows
  df0.count
  val ds1 = df0.repartition(765)
  ds1.count
  val ds2 = df0.repartition(765)
  ds2.count
  
  sqlContext.setConf(&quot;spark.sql.shuffle.partitions&quot;, &quot;765&quot;)
  // The above not included on 1st run, the above included on 2nd run.
  
  ds1.rdd.partitions.size
  ds2.rdd.partitions.size
  
  val joined = ds1.join(ds2, ds1(&quot;time_asc&quot;) === ds2(&quot;time_asc&quot;), &quot;outer&quot;) 
  joined.rdd.partitions.size
  joined.count
  joined.rdd.partitions.size
  </code></pre>
  <p>On the 1st test - <strong>not defining</strong> <em>sqlContext.setConf(&quot;spark.sql.shuffle.partitions&quot;, &quot;765&quot;)</em>, the processing and num partitions resulted was 200. Even though SO post 45704156 states it may not apply to DFs - this is a DS.</p>
  <p>On the 2nd test - <strong>defining</strong> <em>sqlContext.setConf(&quot;spark.sql.shuffle.partitions&quot;, &quot;765&quot;)</em>, the processing and num partitions resulted was 765. Even though SO post 45704156 states it may not apply to DFs - this is a DS.</p>