title: PySpark - read recursive Hive table
body: <p>I have a Hive table that has multiple sub-directories in HDFS, something like:</p>
  
  <pre><code>/hdfs_dir/my_table_dir/my_table_sub_dir1
  /hdfs_dir/my_table_dir/my_table_sub_dir2
  ...
  </code></pre>
  
  <p>Normally I set the following parameters before I run a Hive script:</p>
  
  <pre><code>set hive.input.dir.recursive=true;
  set hive.mapred.supports.subdirectories=true;
  set hive.supports.subdirectories=true;
  set mapred.input.dir.recursive=true;
  
  select * from my_db.my_table;
  </code></pre>
  
  <p>I'm trying to do the same using PySpark,</p>
  
  <pre><code>conf = (SparkConf().setAppName("My App")
          ...
          .set("hive.input.dir.recursive", "true")
          .set("hive.mapred.supports.subdirectories", "true")
          .set("hive.supports.subdirectories", "true")
          .set("mapred.input.dir.recursive", "true"))
  
  sc = SparkContext(conf = conf)
  
  sqlContext = HiveContext(sc)
  
  my_table = sqlContext.sql("select * from my_db.my_table")
  </code></pre>
  
  <p>and end up with an error like:</p>
  
  <pre><code>java.io.IOException: Not a file: hdfs://hdfs_dir/my_table_dir/my_table_sub_dir1
  </code></pre>
  
  <p>What's the correct way to read a Hive table with sub-directories in Spark?</p>