title: Remove JAR from Spark default classpath in EMR
body: <p>I'm executing a spark-submit script in an EMR step that has my super JAR as the main class, like</p>
  
  <pre><code>  spark-submit \
      ....
      --class ${MY_CLASS} "${SUPER_JAR_S3_PATH}"
  </code></pre>
  
  <p>... etc</p>
  
  <p>but Spark is by default loading the jar file:/usr/lib/spark/jars/guice-3.0.jar which contains com.google.inject.internal.InjectorImpl, a class that's also in the Guice-4.x jar which is in my super JAR. This results in a <code>java.lang.IllegalAccessError</code> when my service is booting up.</p>
  
  <p>I've tried setting some Spark conf in the <code>spark-submit</code> to put my super jar in the classpath in hopes of it getting loaded first, before Spark loads guice-3.0.jar. It looks like:</p>
  
  <pre><code>--jars "${ASSEMBLY_JAR_S3_PATH}" \
   --driver-class-path "/etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:${SUPER_JAR_S3_PATH}" \
   --conf spark.executor.extraClassPath="/etc/hadoop/conf:/etc/hive/conf:/usr/lib/hadoop-lzo/lib/*:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:${SUPER_JAR_S3_PATH}" \
  </code></pre>
  
  <p>but this results in the same error.</p>
  
  <p>Is there a way to remove that <code>guice-3.0.jar</code> from the default spark classpath so my code can use the <code>InjectorImpl</code> that's packaged in the <code>Guice-4.x</code> JAR? I'm also running Spark in client mode so I can't use <code>spark.driver.userClassPathFirst</code> or <code>spark.executor.userClassPathFirst</code></p>