title: How to force DataFrame evaluation in Spark
body: <p>Sometimes (e.g. for testing and bechmarking) I want force the execution of the transformations defined on a DataFrame. AFAIK calling an action like <code>count</code> does not ensure that all <code>Columns</code> are actually computed, <code>show</code> may only compute a subset of all <code>Rows</code> (see examples below)</p>
  
  <p>My solution is to write the <code>DataFrame</code> to HDFS using <code>df.write.saveAsTable</code>, but this "clutters" my system with tables I don't want to keep any further.</p>
  
  <p>So what is the best way to trigger the evaluation of a <code>DataFrame</code>?</p>
  
  <p>Edit:</p>
  
  <p>Note that there is also a recent discussion on the spark developer list : <a href="http://apache-spark-developers-list.1001551.n3.nabble.com/Will-count-always-trigger-an-evaluation-of-each-row-td21018.html" rel="noreferrer">http://apache-spark-developers-list.1001551.n3.nabble.com/Will-count-always-trigger-an-evaluation-of-each-row-td21018.html</a></p>
  
  <p>I made a small example which shows that <code>count</code> on <code>DataFrame</code> does not evaluate everything (tested using Spark 1.6.3 and spark-master = <code>local[2]</code>):</p>
  
  <pre><code>val df = sc.parallelize(Seq(1)).toDF("id")
  val myUDF = udf((i:Int) =&gt; {throw new RuntimeException;i})
  
  df.withColumn("test",myUDF($"id")).count // runs fine
  df.withColumn("test",myUDF($"id")).show() // gives Exception
  </code></pre>
  
  <p>Using the same logic, here an example that <code>show</code> does not evaluate all rows:</p>
  
  <pre><code>val df = sc.parallelize(1 to 10).toDF("id")
  val myUDF = udf((i:Int) =&gt; {if(i==10) throw new RuntimeException;i})
  
  df.withColumn("test",myUDF($"id")).show(5) // runs fine
  df.withColumn("test",myUDF($"id")).show(10) // gives Exception
  </code></pre>
  
  <p>Edit 2 : For Eliasah: The Exception says this:</p>
  
  <pre><code>org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 6.0 failed 1 times, most recent failure: Lost task 0.0 in stage 6.0 (TID 6, localhost): java.lang.RuntimeException
      at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1.apply$mcII$sp(&lt;console&gt;:68)
      at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1.apply(&lt;console&gt;:68)
      at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anonfun$1.apply(&lt;console&gt;:68)
      at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
      at org.apache.spark.sql.execution.Project$$anonfun$1$$anonfun$apply$1.apply(basicOperators.scala:51)
      at org.apache.spark.sql.execution.Project$$anonfun$1$$anonfun$apply$1.apply(basicOperators.scala:49)
      at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
  .
  .
  .
  .
  
  Driver stacktrace:
      at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
      at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
      at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
      at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
      at scala.Option.foreach(Option.scala:236)
      at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
      at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
      at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
      at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
      at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
      at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)
      at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:212)
      at org.apache.spark.sql.execution.Limit.executeCollect(basicOperators.scala:165)
      at org.apache.spark.sql.execution.SparkPlan.executeCollectPublic(SparkPlan.scala:174)
      at org.apache.spark.sql.DataFrame$$anonfun$org$apache$spark$sql$DataFrame$$execute$1$1.apply(DataFrame.scala:1500)
      at org.apache.spark.sql.DataFrame$$anonfun$org$apache$spark$sql$DataFrame$$execute$1$1.apply(DataFrame.scala:1500)
      at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:56)
      at org.apache.spark.sql.DataFrame.withNewExecutionId(DataFrame.scala:2087)
      at org.apache.spark.sql.DataFrame.org$apache$spark$sql$DataFrame$$execute$1(DataFrame.scala:1499)
      at org.apache.spark.sql.DataFrame.org$apache$spark$sql$DataFrame$$collect(DataFrame.scala:1506)
      at org.apache.spark.sql.DataFrame$$anonfun$head$1.apply(DataFrame.scala:1376)
      at org.apache.spark.sql.DataFrame$$anonfun$head$1.apply(DataFrame.scala:1375)
      at org.apache.spark.sql.DataFrame.withCallback(DataFrame.scala:2100)
      at org.apache.spark.sql.DataFrame.head(DataFrame.scala:1375)
      at org.apache.spark.sql.DataFrame.take(DataFrame.scala:1457)
      at org.apache.spark.sql.DataFrame.showString(DataFrame.scala:170)
      at org.apache.spark.sql.DataFrame.show(DataFrame.scala:350)
      at org.apache.spark.sql.DataFrame.show(DataFrame.scala:311)
      at org.apache.spark.sql.DataFrame.show(DataFrame.scala:319)
      at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.&lt;init&gt;(&lt;console&gt;:74)
  .
  .
  .
  .
  </code></pre>