title: Understanding Spark DAG Execution
body: <p>I'd like to understand Spark DAG model according to the <a href="http://spark.apache.org/docs/latest/programming-guide.html" rel="nofollow noreferrer">official Spark docs</a>. All transformations in Spark are lazy and by default, each transformed RDD may be recomputed each time you run an action on it. So I wrote a small program like below:</p>
  <pre><code>scala&gt; val lines = sc.textFile(&quot;C:\\Spark\\README.md&quot;)
  lines: org.apache.spark.rdd.RDD[String] = C:\Spark\README.md MapPartitionsRDD[1] at textFile at &lt;console&gt;:24
  
  scala&gt; val breakLInes = lines.flatMap(line=&gt;line.split(&quot; &quot;))
  breakLInes: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[2] at flatMap at &lt;console&gt;:26
  
  scala&gt; val createTuple = breakLInes.map(line=&gt;(line,1))
  createTuple: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[3] at map at &lt;console&gt;:28
  
  scala&gt; val wordCount = createTuple.reduceByKey
  reduceByKey   reduceByKeyLocally
  
  scala&gt; val wordCount = createTuple.reduceByKey(_+_)
  wordCount: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[4] at reduceByKey at &lt;console&gt;:30
  
  scala&gt; wordCount.first
  res0: (String, Int) = (package,1)
  </code></pre>
  <p>Now moving to spark UI below is my DAG Visualization for the first action:</p>
  <p><a href="https://i.stack.imgur.com/HPA3m.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HPA3m.png" alt="enter image description here" /></a></p>
  <p>Again executed:</p>
  <pre><code>scala&gt; wordCount.first
  res0: (String, Int) = (package,1)
  </code></pre>
  <p>Now moving to spark UI below is my DAG Visualization for the second action:</p>
  <p><a href="https://i.stack.imgur.com/5hTpX.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5hTpX.png" alt="enter image description here" /></a></p>
  <p>By default, each transformed RDD may be recomputed each time you run an action on it. Then why it is skipping stage 2 it should have again calculated stage 2 as no caching has been done?</p>