title: Spark + Scala transformations, immutability & memory consumption overheads
body: <p>I have gone through some videos in Youtube regarding <a href="https://www.youtube.com/watch?v=65aV15uDKgA" rel="nofollow">Spark</a> architecture.</p>
  
  <p>Even though Lazy evaluation, Resilience of data creation in case of failures, good functional programming concepts are reasons for success of Resilenace Distributed Datasets, one worrying factor is memory overhead due to multiple <a href="http://spark.apache.org/docs/latest/programming-guide.html#transformations" rel="nofollow">transformations</a> resulting into memory overheads due data immutability.</p>
  
  <p>If I understand the concept correctly, Every transformations is creating new data sets and hence the memory requirements will gone by those many times.  If I use 10 transformations in my code, 10 sets of data sets will be created and my memory consumption will increase by 10 folds.</p>
  
  <p>e.g.</p>
  
  <pre><code>val textFile = sc.textFile("hdfs://...")
  val counts = textFile.flatMap(line =&gt; line.split(" "))
                   .map(word =&gt; (word, 1))
                   .reduceByKey(_ + _)
  counts.saveAsTextFile("hdfs://...")
  </code></pre>
  
  <p>Above example has three transformations : <strong><em><code>flatMap, map and reduceByKey</code></em></strong>. Does it implies I need 3X memory of data for X size of data?</p>
  
  <p><strong><em>Is my understanding correct? Is caching RDD is only solution to address this issue?</em></strong> </p>
  
  <p><em>Once I start caching, it may spill over to disk due to large size and performance would be impacted due to disk IO operations. In that case, performance of Hadoop and Spark are comparable?</em></p>
  
  <p><strong><em>EDIT:</em></strong></p>
  
  <p>From the answer and comments, I have understood lazy initialization and pipeline process. My assumption of 3 X memory where X is initial RDD size is not accurate. </p>
  
  <p><strong><em>But is it possible to cache 1 X RDD in memory and update it over the pipleline? How does cache () works?</em></strong></p>