title: Spark SQL partition pruning for a cached table
body: <p>Is partition pruning enabled for cached TempTables in apache spark? If so, how do I configure it?</p>
  
  <p>My data is a bunch of sensor readings in different installations, one row contains installationName, tag, timestamp and value.</p>
  
  <p>I have written the data in parquet format using the following commands:</p>
  
  <pre><code>rdd.toDF("installationName", "tag", "timestamp", "value")
    .repartition($"installationName", $"tag")
    .write.partitionBy("installationName","tag").mode("overwrite").parquet(config.output)
  </code></pre>
  
  <p>I read this data using the following command into an SQL table using Spark HiveContext:</p>
  
  <pre><code>val parquet = hc.read.parquet("/path_to_table/tablename")
  parquet.registerTempTable("tablename")
  </code></pre>
  
  <p>Now if I run a SQL query on this table, it does partition pruning as expected: </p>
  
  <pre><code>hc.sql("select * from tablename where installationName = 'XXX' and tag = 'YYY'")
  </code></pre>
  
  <p>And the query takes around 8 seconds. But if I cache the table in memory, and then perform the same query, it always takes around 50 seconds:</p>
  
  <pre><code>hc.sql("CACHE TABLE tablename")
  hc.sql("select * from tablename where installationName = 'XXX' and tag = 'YYY'")
  </code></pre>
  
  <p>I am currently using Spark 1.6.1.</p>