title: CDH 6.2 Hive cannot execute queries neither on Spark nor MapReduce
body: <p>I'm trying to run a simple <code>select count(*) from table</code> query on Hive, but it fails with the following error:</p>
  
  <p><code>FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session 5414a8a4-5252-4ccf-b63e-2ee563f7d772_0: java.lang.ClassNotFoundException: org.apache.spark.SparkConf</code></p>
  
  <p>This is happening since I've moved to CDH 6.2 and enabled Spark (version 2.4.0-cdh6.2.0) as the execution engine of Hive (version 2.1.1-cdh6.2.0). </p>
  
  <p>My guess is that Hive is not correctly configured to launch Spark. I've tried setting the <code>spark.home</code> property of the <code>hive-site.xml</code> to <code>/opt/cloudera/parcels/CDH/lib/spark/</code>, and setting the <code>SPARK_HOME</code> environment variable to the same value, but it made no difference.</p>
  
  <p>A similar issue was reported <a href="https://stackoverflow.com/questions/32698134/classnotfoundexception-org-apache-spark-sparkconf-with-spark-on-hive">here</a>, but the solution (i.e., to put the <code>spark-assembly.jar</code> file in Hive's <code>lib</code> directory) cannot be applied (as the file is no longer built in latest Spark's versions).</p>
  
  <p><a href="https://stackoverflow.com/questions/46953218/hive-on-spark-cdh-5-7-failed-to-create-spark-client">A previous question</a> addressed a similar but different issue, related to memory limits on YARN.</p>
  
  <p>Also, switching to MapReduce as the execution engine still fails, but with a different error:</p>
  
  <p><code>FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. org/apache/hadoop/hdfs/protocol/SystemErasureCodingPolicies</code></p>
  
  <p>Looking for the latest error on Google shows no result at all.</p>
  
  <hr>
  
  <p>UPDATE: I discovered that queries do work when connecting to Hive through other tools (e.g., Beeline, Hue, Spark) and independently of the underlying execution engine (i.e., MapReduce or Spark). Thus, the error may lie within the Hive CLI, which is currently deprecated.</p>
  
  <hr>
  
  <p>UPDATE 2: the same problem actually happened on Beeline and Hue with a CREATE TABLE query; I was able to execute it only with the Hive interpreter of Zeppelin</p>