title: How to resolve Apache Spark StackOverflowError after multiple unions
body: <p>I have a Spark Scala program which uses a REST API to get data batch by batch, and once all the data is retrieved I operate on them.</p>
  
  <p><strong>Current Program:</strong></p>
  
  <ul>
  <li><p>For each batch, create RDD and merge it with the previous RDD
  created using the previous API call <code>rdd.union(currentRdd)</code>. </p></li>
  <li><p>Operate on final RDD</p></li>
  </ul>
  
  <p>A simple program to reproduce the issue: </p>
  
  <pre><code>    def main(args: Array[String]) = {
       val conf = new SparkConf().setAppName("Union test").setMaster("local[1]")
       val sc = new SparkContext(conf)
       val limit = 1000;
       var rdd = sc.emptyRDD[Int]
       for (x &lt;- 1 to limit) {
         val currentRdd = sc.parallelize(x to x + 3)
         rdd = rdd.union(currentRdd)
       }
       println(rdd.sum())
     }
  </code></pre>
  
  <p><strong>Problem:</strong>
  - When number of batches are high the program throws a <code>StackOverflowError</code> : <code>Exception in thread "main" java.lang.StackOverflowError
      at org.apache.spark.rdd.UnionRDD$$anonfun$1.apply</code></p>
  
  <p>I assume, that when the number of batches increases the RDD dependency graph becomes really complex and throwing the error.</p>
  
  <p>What is the best way to resolve this problem? </p>