title: how to query Hive from Spark on CDH 5.4.4
body: <p>I'm runnning CDH 5.4.4 (which bundles Spark 1.3.0) and would like to read a Hive table into a Spark dataframe.</p>
  
  <p>Looking at the documentation, it suggests that we can do the following:</p>
  
  <pre><code>from pyspark.sql import HiveContext
  sqlContext = HiveContext(sc)
  results = sqlContext.sql("SHOW TABLES").collect()
  </code></pre>
  
  <p>... providing that Spark has been built with <code>-Phive</code> and <code>-Phive-thriftserver</code> flags set.</p>
  
  <p>I'm not sure whether Cloudera's build has those flags set.</p>
  
  <p>When I run the snippet, it returns the following error:</p>
  
  <pre><code>15/07/10 16:54:10 WARN HiveMetaStore: Retrying creating default database after error: Error creating transactional connection factory
    javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
  </code></pre>
  
  <p>I have two questions:</p>
  
  <ol>
  <li>does Cloudera's Spark build have the <code>hive</code> and <code>hive-thriftserver</code> flags set?</li>
  <li>what do I need to do to query Hive from Spark?</li>
  </ol>
  
  <p><strong>Update</strong></p>
  
  <p>This almost works:</p>
  
  <p>I created a symlink from <code>$SPARK_HOME/conf/</code> to <code>hive-site.xml</code>, i.e.</p>
  
  <pre><code>ln -s /etc/hive/conf.cloudera.hive/hive-site.xml $SPARK_HOME/conf/hive-site.xml
  </code></pre>
  
  <p>I then restarted the Spark service and was able to access Hive. Unfortunately, the symlink didn't survive a reboot.</p>