title: Spark Java error NoClassDefFoundError on Amazon Elastic MapReduce
body: <p>I am trying to implement and run a Spark application on Amazon's Elastic MapReduce (EMR). So far I have been able to deploy and run a cluster with a "Spark Installation" bootstrap action using the following link:</p>
  
  <blockquote>
    <p>s3://elasticmapreduce/samples/spark/0.8.1/install-spark-shark.sh</p>
  </blockquote>
  
  <p>The script can be accessed from <a href="http://elasticmapreduce.s3.amazonaws.com/samples/spark/0.8.1/install-spark-shark.sh" rel="nofollow">this</a> internet direction.</p>
  
  <p>In order to upload the .jar application to the cluster, I have created a step configuration as follows:</p>
  
  <pre><code> HadoopJarStepConfig customConfig = new HadoopJarStepConfig()
                   .withJar("s3://mybucket/SparkApp.jar")
                   .withMainClass("SparkApp.java");
  
   StepConfig customJarStep = new StepConfig()
                   .withName("Run custom jar")                                                                                    
                   .withActionOnFailure(ActionOnFailure.CONTINUE)
                   .withHadoopJarStep(customConfig);
  </code></pre>
  
  <p>Finally, the following code shows the actual Spark application extracted from the wordcount example provided by the Spark team (For the 0.8.1 version). As you may notice, the code imports different spark libraries to be able to run the application. The libraries are: </p>
  
  <blockquote>
    <p>spark-core_2.9.3-0.8.1-incubating.jar
     (and) 
    scala-library-2.9.3.jar</p>
  </blockquote>
  
  <pre><code>import scala.Tuple2;
  import org.apache.spark.api.java.JavaPairRDD;
  import org.apache.spark.api.java.JavaRDD;
  import org.apache.spark.api.java.JavaSparkContext;
  import org.apache.spark.api.java.function.FlatMapFunction;
  import org.apache.spark.api.java.function.Function2;
  import org.apache.spark.api.java.function.PairFunction;
  
  import java.util.Arrays;
  import java.util.List;
  
  public class SparkApp {
    public static void main(String[] args) throws Exception {
  
      JavaSparkContext ctx = new JavaSparkContext("local", "JavaWordCount",
          System.getenv("SPARK_HOME"), System.getenv("SPARK_EXAMPLES_JAR"));
      JavaRDD&lt;String&gt; lines = ctx.textFile("c3://murquiabucket/wordcount.txt", 1);
  
      JavaRDD&lt;String&gt; words = lines.flatMap(new FlatMapFunction&lt;String, String&gt;() {
        public Iterable&lt;String&gt; call(String s) {
          return Arrays.asList(s.split(" "));
        }
      });
  
      JavaPairRDD&lt;String, Integer&gt; ones = words.map(new PairFunction&lt;String, String, Integer&gt;() {
        public Tuple2&lt;String, Integer&gt; call(String s) {
          return new Tuple2&lt;String, Integer&gt;(s, 1);
        }
      });
  
      JavaPairRDD&lt;String, Integer&gt; counts = ones.reduceByKey(new Function2&lt;Integer, Integer, Integer&gt;() {
        public Integer call(Integer i1, Integer i2) {
          return i1 + i2;
        }
      });
  
      List&lt;Tuple2&lt;String, Integer&gt;&gt; output = counts.collect();
      for (Tuple2 tuple : output) {
        System.out.println(tuple._1 + ": " + tuple._2);
      }
      System.exit(0);
    }
  }
  </code></pre>
  
  <p>The problem comes when I try to execute the jar (I made a fat jar to embed the necessary libraries) in the EMR cluster. The application terminates unsuccessfully due to the following error:</p>
  
  <blockquote>
    <p>Exception in thread "main" java.lang.NoClassDefFoundError:
    org/apache/mesos/Scheduler    at
    java.lang.ClassLoader.defineClass1(Native Method) ...</p>
  </blockquote>
  
  <p>For what I understand, there is a issue with Mesos, which I am not able to understand. If this information helps, this is the information of the EMR cluster:</p>
  
  <ul>
  <li>AMI Version: 2.4.2</li>
  <li>Hadoop Distribution: Amazon 1.0.3</li>
  <li>Spark Version: 0.8.1</li>
  <li>Scala Version: 2.9.3</li>
  </ul>