title: Spark. Data caching?
body: <p>I'm testing below script in spark-shell - single partition scan of partitioned table.</p>
  
  <pre><code>val s = System.nanoTime
  
  var q =
  s"""
  select * from partitioned_table where part_column = 'part_column_value' 
  """
  
  spark.sql(q).show
  
  println("Elapsed: " + (System.nanoTime-s) / 1e9 + " seconds")
  </code></pre>
  
  <p>First execution takes around 30 seconds while all subsequent executions take around 2 seconds.</p>
  
  <p>If we have a look at runtime statistics - there are two additional jobs before first execution</p>
  
  <p><a href="https://i.stack.imgur.com/KG3Ny.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/KG3Ny.png" alt="enter image description here"></a></p>
  
  <p>Looks like job with 1212 stages scans all the partitions in a table (total num of partitions 1199, total num of HDFS files for this table - 1384).</p>
  
  <p>I did not find a way to discover what exactly scala/java or SQL code is running for Job 0 but I suspect it's for caching.</p>
  
  <p>Each time I exit spark-shell and start it again - I see this two additional jobs before first executions.
  Of course, similar observations are true for other queries.</p>
  
  <p>Questions</p>
  
  <ol>
  <li>Is it possible to prove or negate hypothesis about caching?</li>
  <li>If it's for caching - how to disable cache and how to clean it up?</li>
  </ol>
  
  <p><strong>Update. Details about job.</strong></p>
  
  <p><a href="https://i.stack.imgur.com/Qfke3.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Qfke3.png" alt="enter image description here"></a></p>