title: How to load specific Hive partition in DataFrame Spark 1.6?
body: <p>Spark 1.6 onwards as per the official <a href="http://spark.apache.org/docs/latest/sql-programming-guide.html#partition-discovery" rel="nofollow noreferrer">doc</a> we cannot add specific hive partitions to DataFrame</p>
  
  <p>Till Spark 1.5 the following used to work and the dataframe would have entity column and the data, as shown below:</p>
  
  <pre><code>DataFrame df = hiveContext.read().format("orc").load("path/to/table/entity=xyz")
  </code></pre>
  
  <p>However, this would not work in Spark 1.6. </p>
  
  <p>If I give base path like the following it does not contain entity column which I want in DataFrame, as shown below - </p>
  
  <pre><code>DataFrame df = hiveContext.read().format("orc").load("path/to/table/") 
  </code></pre>
  
  <p>How do I load specific hive partition in a dataframe? What was the driver behind removing this feature?</p>
  
  <p>I believe it was efficient. Is there an alternative to achieve that in Spark 1.6?</p>
  
  <p>As per my understanding, Spark 1.6 loads all partitions and if I filter for specific partitions it is not efficient, it hits memory and throws GC(Garbage Collection) errors because of thousands of partitions get loaded into memory and not the specific partition.</p>