title: Spark and common partitionner : will two separate referentials partitioned by the same key only load their common partition at join time?
body: <p>French cities data (accounting, enterprises, jobs...) can be conveniently partitioned by department (<code>29=Finistère</code>, <code>33=Gironde</code>, etc...), and in my datasets this field is named <code>codeDepartement</code>.</p>
  <p>I have many referentials of entreprises, accounting, taxation and so on for these cities, they are sorted by <code>codeCommune</code> within these departments partitions :<br />
  <code>enterprises = enterprises.repartition(col(&quot;codeDepartement&quot;)).sortWithinPartition(&quot;codeCommune&quot;)</code></p>
  <hr />
  <p>Then, comes the time where I want to join them for analysis :</p>
  <ul>
  <li><p>I have <code>local[12]</code> configuration for master.</p>
  </li>
  <li><p>I want to study the cities of <em>Finistère</em>. It implies <em>Apache Spark</em> will load the <code>cities\cities_part_29</code> of the parquet file to create the internal RDD <code>cities_rdd_29</code>, or something like this, and I assume it does.</p>
  </li>
  </ul>
  <hr />
  <p>But when the join steps begin, even if join conditions are of the kind :<br />
  <code>enterprises.col(&quot;codeCommune&quot;) = cities.col(&quot;codeCommune&quot;)</code><br />
  <strong>is it enough</strong> to ensure that <em>Spark</em> will <em>only create</em> the internal enterprise RDD <code>enterprises_rdd_29</code> from the <code>enterprises\enterprises_part_29</code> and won't attempt to search among other ones ?</p>
  <ol>
  <li><p>Or isn't it enough and I should add for the join condition : <code>enterprises.col(&quot;codeDepartement&quot;) = cities.col(&quot;codeDepartement&quot;)</code> ?</p>
  </li>
  <li><p>Or, isn't it still enough, and I should have a more <strong>explicit common partitionner</strong> that will link these two datasets,<br />
  and then, how such a partitionner is created for a <em>Dataset</em> ?</p>
  </li>
  <li><p>I'm misleading and I'm missing something else ?</p>
  </li>
  </ol>