title: pyspark, how to read Hive tables with SQLContext?
body: <p>I am new to the Hadoop ecosystem and I am still confused with few things. I am using Spark 1.6.0 (Hive 1.1.0-cdh5.8.0, Hadoop 2.6.0-cdh5.8.0)</p>
  
  <p>I have some Hive table that exist and I can do some SQL queries using HUE web interface with Hive (map reduce) and Impala (mpp).</p>
  
  <p>I am now using pySpark (I think behind this is pyspark-shell) and I wanted to understand and test HiveContext and SQLContext. There are many thready that discussed the differences between the two and for various version of Spark.</p>
  
  <p>With Hive context, I have no issue to query the Hive tables:</p>
  
  <pre><code>from pyspark.sql import HiveContext
  mysqlContext = HiveContext(sc) 
  FromHive = mysqlContext.sql("select * from table.mytable")
  FromHive.count()
  320
  </code></pre>
  
  <p>So far so good. Since SQLContext is subset of HiveContext, I was thinking that a basic SQL select should work:</p>
  
  <pre><code>from pyspark.sql import SQLContext
  sqlSparkContext = SQLContext(sc) 
  FromSQL = mysqlContext.sql("select * from table.mytable")
  FromSQL.count()
  
  Py4JJavaError: An error occurred while calling o81.sql.
  : org.apache.spark.sql.AnalysisException: Table not found: `table`.`mytable`;
  </code></pre>
  
  <p>I added the hive-site.xml to pyspark-shell. When running </p>
  
  <pre><code>sc._conf.getAll(
  </code></pre>
  
  <p>I see:</p>
  
  <pre><code>('spark.yarn.dist.files', '/etc/hive/conf/hive-site.xml'),
  </code></pre>
  
  <p>My questions are:</p>
  
  <ul>
  <li>Can I acess Hive table with SQLContext for simple queries (I know
  HiveContext is more powerfull but for me this is just to understand
  things)</li>
  <li>If this is possible what is missing ? I couldn't find any info apart
  from the hive-site.xml that I tried but doesn't seems to work</li>
  </ul>
  
  <p>Thanks a lot </p>
  
  <p>Cheers </p>
  
  <p>Fabien</p>