title: ClassNotFoundException: com.fasterxml.jackson.annotation.JsonMerge when starting Spark
body: <p>While there are many similar errors with Spark I am still at a loss how to fix the following. Note that I do not have any  <code>Jackson</code> maven entries: only the <code>Spark</code> that pulls in jackson as transitive dependencies:</p>
  
  <pre><code>19/07/14 17:42:49 ERROR MetricsSystem: Sink class org.apache.spark.metrics.sink.MetricsServlet cannot be instantiated
  19/07/14 17:42:49 ERROR SparkContext: Error initializing SparkContext.
  java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:200)
      at org.apache.spark.metrics.MetricsSystem$$anonfun$registerSinks$1.apply(MetricsSystem.scala:194)
      at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130)
      at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:130)
      at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:236)
      at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
      at scala.collection.mutable.HashMap.foreach(HashMap.scala:130)
      at org.apache.spark.metrics.MetricsSystem.registerSinks(MetricsSystem.scala:194)
      at org.apache.spark.metrics.MetricsSystem.start(MetricsSystem.scala:102)
      at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:514)
      at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
      at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
      at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)
      at scala.Option.getOrElse(Option.scala:121)
          ..
          at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:131)
      at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:28)
  Caused by: java.lang.NoClassDefFoundError: com/fasterxml/jackson/annotation/JsonMerge
      at com.fasterxml.jackson.databind.introspect.JacksonAnnotationIntrospector.&lt;clinit&gt;(JacksonAnnotationIntrospector.java:50)
      at com.fasterxml.jackson.databind.ObjectMapper.&lt;clinit&gt;(ObjectMapper.java:291)
      at org.apache.spark.metrics.sink.MetricsServlet.&lt;init&gt;(MetricsServlet.scala:48)
      ... 41 more
  Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.annotation.JsonMerge
      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      ... 44 more
  </code></pre>
  
  <p>The only reference to jackson <em>at all</em> is in an AWS but it is an <em>exclusion</em> : </p>
  
  <pre><code>    &lt;dependency&gt;
          &lt;groupId&gt;com.amazonaws&lt;/groupId&gt;
          &lt;artifactId&gt;aws-java-sdk&lt;/artifactId&gt;
          &lt;version&gt;1.7.4&lt;/version&gt;
         &lt;exclusions&gt;
              &lt;exclusion&gt;
                  &lt;groupId&gt;com.fasterxml.jackson.core&lt;/groupId&gt;
                  &lt;artifactId&gt;jackson-core&lt;/artifactId&gt;
              &lt;/exclusion&gt;
              &lt;exclusion&gt;
                  &lt;groupId&gt;com.fasterxml.jackson.core&lt;/groupId&gt;
                  &lt;artifactId&gt;jackson-databind&lt;/artifactId&gt;
              &lt;/exclusion&gt;
          &lt;/exclusions&gt;
      &lt;/dependency&gt;
  </code></pre>
  
  <p>How can this <code>jackson</code> issue (probably a versioning) be resolved? Note: I have tried with Spark 2.4.2 and then downgraded to 2.3.0: same errors. I can not downgrade more than that.</p>