title: Spark - how to handle with lazy evaluation in case of iterative (or recursive) function calls
body: <p>I have a recursive function that needs to compare the results of the current call to the previous call to figure out whether it has reached a convergence. My function does not contain any <code>action</code> - it only contains <code>map</code>, <code>flatMap</code>, and <code>reduceByKey</code>.  Since Spark does not evaluate transformations (until an action is called), my next iteration does not get the proper values to compare for convergence.</p>
  
  <p>Here is a skeleton of the function -</p>
  
  <pre><code>def func1(sc: SparkContext, nodes:RDD[List[Long]], didConverge: Boolean, changeCount: Int) RDD[(Long] = {
  
     if (didConverge)
        nodes
     else { 
         val currChangeCount = sc.accumulator(0, "xyz")         
         val newNodes = performSomeOps(nodes, currChangeCount) // does a few map/flatMap/reduceByKey operations
         if (currChangeCount.value == changeCount)  {
            func1(sc, newNodes, true, currChangeCount.value)
         } else {
            func1(sc, newNode, false, currChangeCount.value)
         }
     }
  }
  </code></pre>
  
  <p><code>performSomeOps</code> only contains <code>map</code>, <code>flatMap</code>, and <code>reduceByKey</code> transformations. Since it does not have any action, the code in <code>performSomeOps</code> does not execute.  So my <code>currChangeCount</code> does not get the actual count.  What that implies, the condition to check for the convergence (<code>currChangeCount.value == changeCount</code>) is going to be invalid.  One way to overcome is to force an action within each iteration by calling a <code>count</code> but that is an unnecessary overhead. </p>
  
  <p>I am wondering what I can do to force an action w/o much overhead or is there another way to address this problem?</p>