title: Spark with Hive, Unable to instantiate SparkSession with Hive support because Hive classes are not found
body: <p>The spark app is to load data from Hive:</p>
  
  <pre><code>    SparkSession spark = SparkSession.builder()
          .appName(topics)
          .config("hive.metastore.uris", "thrift://device1:9083")
          .enableHiveSupport()
          .getOrCreate();
  </code></pre>
  
  <p>I start the spark by:</p>
  
  <pre><code>spark-submit --master local[*] --class zhihu.SparkConsumer target/original-kafka-consumer-0.1-SNAPSHOT.jar  --jars spark-hive_2.11-2.4.4.jar
  </code></pre>
  
  <p>maven pom.xml</p>
  
  <pre><code>&lt;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"&gt;
    &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;
    &lt;groupId&gt;org.zhihu&lt;/groupId&gt;
    &lt;artifactId&gt;kafka-consumer&lt;/artifactId&gt;
    &lt;packaging&gt;jar&lt;/packaging&gt;
    &lt;version&gt;0.1-SNAPSHOT&lt;/version&gt;
    &lt;name&gt;kafkadev&lt;/name&gt;
    &lt;url&gt;http://maven.apache.org&lt;/url&gt;
    &lt;repositories&gt;
      &lt;repository&gt;
        &lt;!-- Proper URL for Cloudera maven artifactory --&gt;
        &lt;id&gt;cloudera&lt;/id&gt;
        &lt;url&gt;https://repository.cloudera.com/artifactory/cloudera-repos/&lt;/url&gt;
      &lt;/repository&gt;
  
  &lt;/repositories&gt;
  &lt;dependencies&gt;
  
  &lt;!-- https://mvnrepository.com/artifact/org.apache.logging.log4j/log4j-core --&gt;
  &lt;!-- https://mvnrepository.com/artifact/org.apache.logging.log4j/log4j-api --&gt;
  &lt;dependency&gt;
      &lt;groupId&gt;org.apache.logging.log4j&lt;/groupId&gt;
      &lt;artifactId&gt;log4j-api&lt;/artifactId&gt;
      &lt;version&gt;2.8.2&lt;/version&gt;
  &lt;/dependency&gt;
  &lt;dependency&gt; &lt;!-- Spark dependency --&gt;
      &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
      &lt;artifactId&gt;spark-sql_2.11&lt;/artifactId&gt;
      &lt;version&gt;2.4.4&lt;/version&gt;
      &lt;scope&gt;compile&lt;/scope&gt;
  &lt;/dependency&gt;
  
  &lt;dependency&gt; &lt;!-- Spark dependency --&gt;
      &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
      &lt;artifactId&gt;spark-hive_2.11&lt;/artifactId&gt;
      &lt;version&gt;2.4.4&lt;/version&gt;
      &lt;scope&gt;compile&lt;/scope&gt;
  &lt;/dependency&gt;
  
  &lt;dependency&gt;
      &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
      &lt;artifactId&gt;spark-streaming_2.11&lt;/artifactId&gt;
      &lt;version&gt;2.4.4&lt;/version&gt;
      &lt;scope&gt;compile&lt;/scope&gt;
  
  &lt;/dependency&gt;
  &lt;dependency&gt;
      &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
      &lt;artifactId&gt;spark-streaming-kafka-0-10_2.11&lt;/artifactId&gt;
      &lt;version&gt;2.4.4&lt;/version&gt;
        &lt;/dependency&gt;
  
        &lt;dependency&gt;
            &lt;groupId&gt;org.apache.kafka&lt;/groupId&gt;
            &lt;artifactId&gt;kafka-clients&lt;/artifactId&gt;
            &lt;version&gt;2.1.0&lt;/version&gt;
  
            &lt;exclusions&gt;
          &lt;exclusion&gt;
              &lt;groupId&gt;org.apache.logging.log4j&lt;/groupId&gt;
              &lt;artifactId&gt;log4j-core&lt;/artifactId&gt;
          &lt;/exclusion&gt;
          &lt;exclusion&gt;
              &lt;groupId&gt;org.apache.log4j&lt;/groupId&gt;
              &lt;artifactId&gt;log4j-core&lt;/artifactId&gt;
          &lt;/exclusion&gt;
          &lt;exclusion&gt;
              &lt;groupId&gt;log4j&lt;/groupId&gt;
              &lt;artifactId&gt;log4j&lt;/artifactId&gt;
          &lt;/exclusion&gt;
      &lt;/exclusions&gt;
  
        &lt;scope&gt;compile&lt;/scope&gt;
    &lt;/dependency&gt;
    &lt;!-- gson --&gt;
    &lt;dependency&gt;
        &lt;groupId&gt;com.google.code.gson&lt;/groupId&gt;
        &lt;artifactId&gt;gson&lt;/artifactId&gt;
        &lt;version&gt;2.8.2&lt;/version&gt;
    &lt;/dependency&gt;
  
    &lt;dependency&gt;
        &lt;groupId&gt;junit&lt;/groupId&gt;
        &lt;artifactId&gt;junit&lt;/artifactId&gt;
        &lt;version&gt;3.8.1&lt;/version&gt;
        &lt;scope&gt;test&lt;/scope&gt;
    &lt;/dependency&gt;
  
    &lt;dependency&gt;
        &lt;groupId&gt;org.apache.hive&lt;/groupId&gt;
        &lt;artifactId&gt;hive-metastore&lt;/artifactId&gt;
        &lt;version&gt;2.1.1-cdh6.2.0&lt;/version&gt;
    &lt;/dependency&gt;
  
    &lt;dependency&gt;
        &lt;groupId&gt;org.apache.hive&lt;/groupId&gt;
        &lt;artifactId&gt;hive-service&lt;/artifactId&gt;
        &lt;version&gt;2.1.1-cdh6.2.0&lt;/version&gt;
    &lt;/dependency&gt;
  
    &lt;!-- runtime Hive --&gt;
    &lt;dependency&gt;
        &lt;groupId&gt;org.apache.hive&lt;/groupId&gt;
        &lt;artifactId&gt;hive-common&lt;/artifactId&gt;
        &lt;version&gt;2.1.1-cdh6.2.0&lt;/version&gt;
        &lt;scope&gt;runtime&lt;/scope&gt;
    &lt;/dependency&gt;
  
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.hive&lt;/groupId&gt;
          &lt;artifactId&gt;hive-beeline&lt;/artifactId&gt;
          &lt;version&gt;2.1.1-cdh6.2.0&lt;/version&gt;
          &lt;scope&gt;runtime&lt;/scope&gt;
      &lt;/dependency&gt;
  
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.hive&lt;/groupId&gt;
          &lt;artifactId&gt;hive-jdbc&lt;/artifactId&gt;
          &lt;version&gt;2.1.1-cdh6.2.0&lt;/version&gt;
          &lt;scope&gt;runtime&lt;/scope&gt;
      &lt;/dependency&gt;
  
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.hive&lt;/groupId&gt;
          &lt;artifactId&gt;hive-shims&lt;/artifactId&gt;
          &lt;version&gt;2.1.1-cdh6.2.0&lt;/version&gt;
          &lt;scope&gt;runtime&lt;/scope&gt;
      &lt;/dependency&gt;
  
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.hive&lt;/groupId&gt;
          &lt;artifactId&gt;hive-exec&lt;/artifactId&gt;
          &lt;version&gt;2.1.1-cdh6.2.0&lt;/version&gt;
          &lt;scope&gt;runtime&lt;/scope&gt;
      &lt;/dependency&gt;
  
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.hive&lt;/groupId&gt;
          &lt;artifactId&gt;hive-serde&lt;/artifactId&gt;
          &lt;version&gt;2.1.1-cdh6.2.0&lt;/version&gt;
          &lt;scope&gt;runtime&lt;/scope&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
          &lt;groupId&gt;org.apache.hive&lt;/groupId&gt;
          &lt;artifactId&gt;hive-contrib&lt;/artifactId&gt;
          &lt;version&gt;2.1.1-cdh6.2.0&lt;/version&gt;
          &lt;scope&gt;runtime&lt;/scope&gt;
      &lt;/dependency&gt;
  
      &lt;/dependencies&gt;
      &lt;build&gt;
          &lt;plugins&gt;
              &lt;plugin&gt;
                  &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
                  &lt;artifactId&gt;maven-compiler-plugin&lt;/artifactId&gt;
                  &lt;version&gt;3.7.0&lt;/version&gt;
                  &lt;configuration&gt;
                      &lt;source&gt;1.8&lt;/source&gt;
                      &lt;target&gt;1.8&lt;/target&gt;
                  &lt;/configuration&gt;
              &lt;/plugin&gt;
  
              &lt;plugin&gt;
                  &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
                  &lt;artifactId&gt;maven-shade-plugin&lt;/artifactId&gt;
                  &lt;version&gt;3.2.1&lt;/version&gt;
                  &lt;executions&gt;
                      &lt;execution&gt;
                  &lt;phase&gt;package&lt;/phase&gt;
                  &lt;goals&gt;
                      &lt;goal&gt;shade&lt;/goal&gt;
                  &lt;/goals&gt;
                  &lt;configuration&gt;
  
                      &lt;filters&gt;
  
                          &lt;filter&gt;
                              &lt;artifact&gt;*:*&lt;/artifact&gt;
                              &lt;excludes&gt;
                          &lt;exclude&gt;**/Log4j2Plugins.dat&lt;/exclude&gt;
                      &lt;/excludes&gt;
                  &lt;/filter&gt;
  
                          &lt;filter&gt;
                              &lt;artifact&gt;*:*&lt;/artifact&gt;
                              &lt;excludes&gt;
                                  &lt;exclude&gt;META-INF/*.SF&lt;/exclude&gt;
                                  &lt;exclude&gt;META-INF/*.DSA&lt;/exclude&gt;
                                  &lt;exclude&gt;META-INF/*.RSA&lt;/exclude&gt;
                              &lt;/excludes&gt;
                          &lt;/filter&gt;
                      &lt;/filters&gt;
                      &lt;artifactSet&gt;
                      &lt;excludes&gt;
                          &lt;exclude&gt;classworlds:classworlds&lt;/exclude&gt;
                          &lt;exclude&gt;junit:junit&lt;/exclude&gt;
                          &lt;exclude&gt;jmock:*&lt;/exclude&gt;
                          &lt;exclude&gt;*:xml-apis&lt;/exclude&gt;
                          &lt;exclude&gt;org.apache.maven:lib:tests&lt;/exclude&gt;
                      &lt;/excludes&gt;
                  &lt;/artifactSet&gt;
              &lt;skip&gt;true&lt;/skip&gt;
          &lt;/configuration&gt;
            &lt;/execution&gt;
          &lt;/executions&gt;
      &lt;/plugin&gt;
  
      &lt;/plugins&gt;
    &lt;/build&gt;
  &lt;/project&gt;
  </code></pre>
  
  <p>it looks no problem, but it always raised:</p>
  
  <pre><code>20/05/07 12:03:17 INFO spark.SparkContext: Added JAR file:/data/projects/zhihu_scraper/consumers/target/original-kafka-consumer-0.1-SNAPSHOT.jar at spark://device2:42395/jars/original-kafka-consumer-0.1-SNAPSHOT.jar with timestamp 1588824197724
  20/05/07 12:03:17 INFO executor.Executor: Starting executor ID driver on host localhost
  20/05/07 12:03:17 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33849.
  20/05/07 12:03:17 INFO netty.NettyBlockTransferService: Server created on device2:33849
  20/05/07 12:03:17 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
  20/05/07 12:03:17 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, device2, 33849, None)
  20/05/07 12:03:17 INFO storage.BlockManagerMasterEndpoint: Registering block manager device2:33849 with 366.3 MB RAM, BlockManagerId(driver, device2, 33849, None)
  20/05/07 12:03:17 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, device2, 33849, None)
  20/05/07 12:03:17 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, device2, 33849, None)
  20/05/07 12:03:17 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@63e5e5b4{/metrics/json,null,AVAILABLE,@Spark}
  Exception in thread "main" java.lang.IllegalArgumentException: Unable to instantiate SparkSession with Hive support because Hive classes are not found.
      at org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:869)
      at zhihu.SparkConsumer.main(SparkConsumer.java:72)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
      at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:845)
      at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
      at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
      at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
      at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:920)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:929)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  20/05/07 12:03:18 INFO spark.SparkContext: Invoking stop() from shutdown hook
  </code></pre>
  
  <p>I have tried all answers in this post <a href="https://stackoverflow.com/questions/39444493/how-to-create-sparksession-with-hive-support-fails-with-hive-classes-are-not-f/61050495#61050495">How to create SparkSession with Hive support</a>. But, none of them work for me.</p>