title: Unable to execute Hive queries using spark-submit
body: <p>I am not able run hive queries using spark-submit command. But, the same is getting executed in spark-shell. I am using AWS EMR as the cluster. </p>
  
  <p><strong>Below is my code written in eclipse scala IDE</strong></p>
  
  <pre><code> object HiveTest {
  
    def main(args: Array[String]): Unit =
      {
  
        val sparkConf = new SparkConf()
        sparkConf.setAppName("WordCountTest")
  
        val sc = new SparkContext(sparkConf)
  
  
        val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
        import sqlContext.implicits._
  
    sqlContext.sql("select * from stream_table");
  
      }
  }
  </code></pre>
  
  <p><strong>pom.xml</strong></p>
  
  <pre><code>&lt;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"&gt;
      &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;
  
      &lt;groupId&gt;spark&lt;/groupId&gt;
      &lt;artifactId&gt;word-count&lt;/artifactId&gt;
      &lt;version&gt;0.1-SNAPSHOT&lt;/version&gt;
      &lt;packaging&gt;jar&lt;/packaging&gt;
  
      &lt;name&gt;word-count&lt;/name&gt;
      &lt;url&gt;http://maven.apache.org&lt;/url&gt;
  
            &lt;build&gt;
          &lt;plugins&gt;
           &lt;plugin&gt;
      &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
      &lt;artifactId&gt;maven-assembly-plugin&lt;/artifactId&gt;
      &lt;executions&gt;
          &lt;execution&gt;
              &lt;phase&gt;package&lt;/phase&gt;
              &lt;goals&gt;
                  &lt;goal&gt;single&lt;/goal&gt;
              &lt;/goals&gt;
              &lt;configuration&gt;
                  &lt;archive&gt;
                  &lt;manifest&gt;
                      &lt;mainClass&gt;
                          HiveTest
                      &lt;/mainClass&gt;
                  &lt;/manifest&gt;
                  &lt;/archive&gt;
                  &lt;descriptorRefs&gt;
                      &lt;descriptorRef&gt;jar-with-dependencies&lt;/descriptorRef&gt;
                  &lt;/descriptorRefs&gt;
              &lt;/configuration&gt;
          &lt;/execution&gt;
      &lt;/executions&gt;
  &lt;/plugin&gt;
          &lt;/plugins&gt;
      &lt;/build&gt;
      &lt;properties&gt;
  
        &lt;encoding&gt;UTF-8&lt;/encoding&gt;
        &lt;scala.version&gt;2.11.8&lt;/scala.version&gt;
        &lt;scala.tools.version&gt;2.11&lt;/scala.tools.version&gt;
        &lt;spark.version&gt;2.0.0&lt;/spark.version&gt;
  
    &lt;/properties&gt;
  
  
      &lt;dependencies&gt;
      &lt;!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming --&gt;
  &lt;dependency&gt;
      &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
      &lt;artifactId&gt;spark-streaming_${scala.tools.version}&lt;/artifactId&gt;
      &lt;version&gt;${spark.version}&lt;/version&gt;
      &lt;scope&gt;provided&lt;/scope&gt;
  &lt;/dependency&gt;
  
  &lt;!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka --&gt;
  &lt;!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka --&gt;
  &lt;dependency&gt;
      &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
      &lt;artifactId&gt;spark-streaming-kafka_2.11&lt;/artifactId&gt;
      &lt;version&gt;1.6.3&lt;/version&gt;
  &lt;/dependency&gt;
  
  
  &lt;!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase --&gt;
  &lt;dependency&gt;
      &lt;groupId&gt;org.apache.hbase&lt;/groupId&gt;
      &lt;artifactId&gt;hbase&lt;/artifactId&gt;
      &lt;version&gt;0.90.0&lt;/version&gt;
  &lt;/dependency&gt;
  
        &lt;!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core --&gt;
  &lt;dependency&gt;
      &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
      &lt;artifactId&gt;spark-core_${scala.tools.version}&lt;/artifactId&gt;
      &lt;version&gt;${spark.version}&lt;/version&gt;
  &lt;/dependency&gt;
  
  
  &lt;!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql --&gt;
  &lt;dependency&gt;
      &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
      &lt;artifactId&gt;spark-sql_${scala.tools.version}&lt;/artifactId&gt;
      &lt;version&gt;${spark.version}&lt;/version&gt;
  &lt;/dependency&gt;
  
  &lt;!-- https://mvnrepository.com/artifact/org.apache.spark/spark-hive --&gt;
  &lt;dependency&gt;
      &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
      &lt;artifactId&gt;spark-hive_2.11&lt;/artifactId&gt;
      &lt;version&gt;${spark.version}&lt;/version&gt;
  &lt;/dependency&gt;
  
  
          &lt;dependency&gt;
              &lt;groupId&gt;org.scala-lang&lt;/groupId&gt;
              &lt;artifactId&gt;scala-library&lt;/artifactId&gt;
              &lt;version&gt;${scala.version}&lt;/version&gt;
          &lt;/dependency&gt;
  
  
          &lt;dependency&gt;
              &lt;groupId&gt;junit&lt;/groupId&gt;
              &lt;artifactId&gt;junit&lt;/artifactId&gt;
              &lt;version&gt;3.8.1&lt;/version&gt;
              &lt;scope&gt;test&lt;/scope&gt;
          &lt;/dependency&gt;
      &lt;/dependencies&gt;
  &lt;/project&gt;
  </code></pre>
  
  <p><strong>spark-submit command</strong> </p>
  
  <blockquote>
    <p>spark-submit --master local[2] --class HiveTest
    ./word-count-0.1-SNAPSHOT-jar-with-dependencies.jar</p>
  </blockquote>
  
  <p><strong>Error</strong></p>
  
  <pre><code>[hadoop@ip-10-134-23-168 jars]$ spark-submit --master local[2] --class HiveTest ./word-count-0.1-SNAPSHOT-jar-with-dependencies.jar
  18/02/12 10:58:45 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
  18/02/12 10:58:49 WARN Hive: Failed to access metastore. This class should not accessed in runtime.
  org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
          at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1236)
          at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174)
          at org.apache.hadoop.hive.ql.metadata.Hive.&lt;clinit&gt;(Hive.java:166)
          at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
          at org.apache.spark.sql.hive.client.HiveClientImpl.&lt;init&gt;(HiveClientImpl.scala:192)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  </code></pre>