title: ClassNotFoundException: org.apache.spark.SparkConf with spark on hive
body: <p>I am trying to use SPARK as hive execution engine, but getting the below error. Spark 1.5.0 is installed and I am working with Hive 1.1.0 version with Hadoop 2.7.0 version.</p>
  
  <p><code>hive_emp</code> table is created as ORC format table in hive.</p>
  
  <pre><code>hive (Koushik)&gt; insert into table hive_emp values (2,'Koushik',1);
  Query ID = hduser_20150921072727_feba8363-258d-4d0b-8976-662e404bca88
  Total jobs = 1
  Launching Job 1 out of 1
  In order to change the average load for a reducer (in bytes):
    set hive.exec.reducers.bytes.per.reducer=&lt;number&gt;
  In order to limit the maximum number of reducers:
    set hive.exec.reducers.max=&lt;number&gt;
  In order to set a constant number of reducers:
    set mapreduce.job.reduces=&lt;number&gt;
  java.lang.NoClassDefFoundError: org/apache/spark/SparkConf
      at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.generateSparkConf(HiveSparkClientFactory.java:140)
      at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:56)
      at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55)
      at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116)
      at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:113)
      at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:95)
      at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
      at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
      at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1638)
      at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1397)
      at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183)
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1039)
      at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
      at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
      at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
      at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754)
      at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
      at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:601)
      at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
  Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf
      at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
      at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
      ... 25 more
  FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. org/apache/spark/SparkConf
  </code></pre>
  
  <p>I also set the spark path and execution engine in hive shell.</p>
  
  <pre><code>hduser@ubuntu:~$ spark-shell
      Welcome to
        ____              __
       / __/__  ___ _____/ /__
      _\ \/ _ \/ _ `/ __/  '_/
     /___/ .__/\_,_/_/ /_/\_\   version 1.5.0
        /_/
  
  Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_21)
  Type in expressions to have them evaluated.
  Type :help for more information.
  Spark context available as sc.
  SQL context available as sqlContext.
  
  scala&gt; exit;
  warning: there were 1 deprecation warning(s); re-run with -deprecation for details
  hduser@ubuntu:~$ hive
  SLF4J: Class path contains multiple SLF4J bindings.
  SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/spark-assembly-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/usr/lib/hive/auxlib/spark-assembly-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  SLF4J: Class path contains multiple SLF4J bindings.
  SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/spark-assembly-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/usr/lib/hive/auxlib/spark-assembly-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  
  Logging initialized using configuration in file:/usr/lib/hive/conf/hive-log4j.properties
  hive (default)&gt; use Koushik;
  OK
  Time taken: 0.593 seconds
  hive (Koushik)&gt; set spark.home=/usr/local/src/spark;
  </code></pre>
  
  <p>I have also created a .hiverc as below</p>
  
  <pre><code>hduser@ubuntu:/usr/lib/hive/conf$ cat .hiverc
  SET hive.cli.print.header=true;
  set hive.cli.print.current.db=true;
  set hive.auto.convert.join=true;
  SET hbase.scan.cacheblock=0;
  SET hbase.scan.cache=10000;
  SET hbase.client.scanner.cache=10000;
  SET hive.execution.engine=spark;
  </code></pre>
  
  <p>DEBUG mode error details given below:</p>
  
  <pre><code>hduser@ubuntu:~$ hive
  SLF4J: Class path contains multiple SLF4J bindings.
  SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/spark-assembly-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/usr/lib/hive/auxlib/spark-assembly-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  SLF4J: Class path contains multiple SLF4J bindings.
  SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/spark-assembly-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: Found binding in [jar:file:/usr/lib/hive/auxlib/spark-assembly-1.5.0-hadoop2.6.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
  SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
  
  Logging initialized using configuration in file:/usr/lib/hive/conf/hive-log4j.properties
  hive (default)&gt; use Koushik;
  OK
  Time taken: 0.625 seconds
  hive (Koushik)&gt; set hive --hiveconf hive.root.logger=DEBUG
                &gt; ;
  hive (Koushik)&gt; set hive.execution.engine=spark;
  hive (Koushik)&gt; desc hive_emp;
  OK
  col_name    data_type   comment
  empid                   int                                         
  empnm                   varchar(50)                                 
  deptid                  int                                         
  Time taken: 0.173 seconds, Fetched: 3 row(s)
  hive (Koushik)&gt; select * from hive_emp;
  OK
  hive_emp.empid  hive_emp.empnm  hive_emp.deptid
  Time taken: 1.689 seconds
  hive (Koushik)&gt; insert into table hive_emp values (2,'Koushik',1);
  Query ID = hduser_20151015112525_c96a458b-34f8-42ac-ab11-52c32479a29a
  Total jobs = 1
  Launching Job 1 out of 1
  In order to change the average load for a reducer (in bytes):
    set hive.exec.reducers.bytes.per.reducer=&lt;number&gt;
  In order to limit the maximum number of reducers:
    set hive.exec.reducers.max=&lt;number&gt;
  In order to set a constant number of reducers:
    set mapreduce.job.reduces=&lt;number&gt;
  java.lang.NoSuchMethodError: org.apache.spark.scheduler.LiveListenerBus.addListener(Lorg/apache/spark/scheduler/SparkListener;)V
      at org.apache.hadoop.hive.ql.exec.spark.LocalHiveSparkClient.&lt;init&gt;(LocalHiveSparkClient.java:85)
      at org.apache.hadoop.hive.ql.exec.spark.LocalHiveSparkClient.getInstance(LocalHiveSparkClient.java:69)
      at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:56)
      at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:55)
      at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:116)
      at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:113)
      at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:95)
      at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
      at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
      at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1638)
      at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1397)
      at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183)
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1039)
      at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
      at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
      at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
      at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:754)
      at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
      at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:601)
      at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
  FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. org.apache.spark.scheduler.LiveListenerBus.addListener(Lorg/apache/spark/scheduler/SparkListener;)V
  hive (Koushik)&gt; 
  </code></pre>
  
  <p>I have executed the above insert twice and both the times it fails. Please find the hive.log which generated today.<a href="https://drive.google.com/file/d/0B_Ed4jUfln0SNkhwNnVUMG9neGs/view?usp=sharing">hive.log</a></p>