title: Spark: understanding the DAG and forcing transformations
body: <p>Hello stackoverflow community.</p>
  
  <p>I ask your help in understanding if my thoughts are correct or I'm missing some points in my Spark job.</p>
  
  <p>I currently have two rdds that I want to subtract.
  Both the rdds are built as different transformations on the same father RDD.</p>
  
  <p>First of all, the father RDD is cached after it is obtained:</p>
  
  <pre><code>val fatherRdd = grandFather.repartition(n).mapPartitions(mapping).cache
  </code></pre>
  
  <p>Then the two rdds are transformed.
  One is (pseudocode):</p>
  
  <pre><code>son1= rddFather.filter(filtering_logic).map(take_only_key).distinct
  </code></pre>
  
  <p>The other one is :</p>
  
  <pre><code>son2= rddFather.filter(filtering_logic2).map(take_only_key).distinct
  </code></pre>
  
  <p>The two sons are then subtracted to obtain only the keys in son1:</p>
  
  <pre><code>son1.subtract(son2)
  </code></pre>
  
  <p>I would expect the squence of the transformations to be the following:</p>
  
  <ol>
  <li>mapPartitions  </li>
  <li>repartition</li>
  <li>caching</li>
  </ol>
  
  <p>Then, starting from cached data, map filter map distinct on both rdds and then subtracting.</p>
  
  <p>This is not happening, what I see are the two <code>distinct</code> operations running in parallel, apparently not exploiting the benefits of caching (there are no skipped tasks), and taking almost the same computation time.
  Below the image of the dag taken from spark ui.
  <a href="https://i.stack.imgur.com/uIGQm.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/uIGQm.png" alt="enter image description here"></a></p>
  
  <p>Do you have any suggestions for me?</p>