title: Jar conflicts between apache spark and hadoop
body: <p>I try to setup and run a Spark cluster running on top of YARN and using HDFS.</p>
  
  <p>I first set up Hadoop for HDFS using hadoop-3.1.0.
  Then I configured YARN and started both.
  I was able to upload data to the HDFS and yarn also seems to work fine.</p>
  
  <p>Then I installed spark-2.3.0-bin-without-hadoop on my master only and tried to submit an application.
  Since it is spark without Hadoop I had to modify spark-env.sh, adding the following line like mentioned in the documentation:</p>
  
  <pre><code>export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath)
  </code></pre>
  
  <p>Using only this line I got the following exception:</p>
  
  <pre><code>Error: Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster
  </code></pre>
  
  <p>Which I guess means that he does not find the Spark-libraries. So I added the spark jars to the classpath:</p>
  
  <pre><code>export SPARK_DIST_CLASSPATH=$(/usr/local/hadoop/bin/hadoop classpath):/usr/local/spark/jars/*
  </code></pre>
  
  <p>But now I get the following Exception:</p>
  
  <pre><code>com.fasterxml.jackson.databind.JsonMappingException: Incompatible Jackson version: 2.7.8
  </code></pre>
  
  <p>As it turns out, Hadoop 3.1.0 provides Jackson 2.7.8 while Spark 2.3.0 provides Jackson 2.6.7. As I see it, both are now in the classpath resulting in a conflict.</p>
  
  <p>Since it seems I really need both the Hadoop and Spark libraries to submit anything, I do not know how to get around that problem.</p>