title: Packaging and Running Scala Spark Project with Maven
body: <p>I am writing an application in Scala that uses <a href="http://spark.apache.org/" rel="nofollow noreferrer">Spark</a>. I am packaging the app using Maven and running into problems when constructing an <a href="https://stackoverflow.com/questions/11947037/what-is-an-uber-jar">"uber" or "fat" jar</a>. </p>
  
  <p>The problem I am facing is that running the application works fine inside of an IDE or if I provide a non-uber-jar'd version of the dependencies as the java class path, but it does not work if I give the uber jar as the class path, i.e.</p>
  
  <pre><code>java -Xmx2G -cp target/spark-example-0.1-SNAPSHOT-jar-with-dependencies.jar debug.spark_example.Example data.txt 
  </code></pre>
  
  <p>does not work. I get the following error message:</p>
  
  <pre><code>ERROR SparkContext: Error initializing SparkContext.
  com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'akka.version'
      at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:124)
      at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:145)
      at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:151)
      at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:159)
      at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:164)
      at com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:206)
      at akka.actor.ActorSystem$Settings.&lt;init&gt;(ActorSystem.scala:168)
      at akka.actor.ActorSystemImpl.&lt;init&gt;(ActorSystem.scala:504)
      at akka.actor.ActorSystem$.apply(ActorSystem.scala:141)
      at akka.actor.ActorSystem$.apply(ActorSystem.scala:118)
      at org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:122)
      at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:54)
      at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53)
      at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1991)
      at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160)
      at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1982)
      at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:56)
      at org.apache.spark.rpc.akka.AkkaRpcEnvFactory.create(AkkaRpcEnv.scala:245)
      at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:52)
      at org.apache.spark.SparkEnv$.create(SparkEnv.scala:247)
      at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:188)
      at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:267)
      at org.apache.spark.SparkContext.&lt;init&gt;(SparkContext.scala:424)
      at debug.spark_example.Example$.main(Example.scala:9)
      at debug.spark_example.Example.main(Example.scala)
  </code></pre>
  
  <p>I would really appreciate help understanding what I need to add to the pom.xml file and why I need to add it to get this to work. </p>
  
  <p>I have searched online and found the following resources, which I tried (see in the pom), but could not get to work:</p>
  
  <p>1) Spark User Mailing list: <a href="http://apache-spark-user-list.1001560.n3.nabble.com/Packaging-a-spark-job-using-maven-td5615.html" rel="nofollow noreferrer">http://apache-spark-user-list.1001560.n3.nabble.com/Packaging-a-spark-job-using-maven-td5615.html</a></p>
  
  <p>2) <a href="https://stackoverflow.com/questions/27099684/how-to-package-spark-scala-application">how to package spark scala application</a></p>
  
  <p>I have a simple example that demonstrates this problem, a simple 1 class project (src/main/scala/debug/spark_example/Example.scala):</p>
  
  <pre><code>package debug.spark_example
  
  import org.apache.spark.{SparkConf, SparkContext}
  
  object Example {
    def main(args: Array[String]): Unit = {
      val sc = new SparkContext(new SparkConf().setAppName("Test").setMaster("local[2]"))
      val lines = sc.textFile(args(0))
      val lineLengths = lines.map(s =&gt; s.length)
      val totalLength = lineLengths.reduce((a, b) =&gt; a + b)
      lineLengths.foreach(println)
       println(totalLength)
     }
   }
  </code></pre>
  
  <p>Here is the pom.xml file:</p>
  
  <pre><code>&lt;project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"&gt;
    &lt;modelVersion&gt;4.0.0&lt;/modelVersion&gt;
    &lt;groupId&gt;debug.spark-example&lt;/groupId&gt;
    &lt;artifactId&gt;spark-example&lt;/artifactId&gt;
    &lt;version&gt;0.1-SNAPSHOT&lt;/version&gt;
    &lt;inceptionYear&gt;2015&lt;/inceptionYear&gt;
    &lt;properties&gt;
      &lt;scala.majorVersion&gt;2.11&lt;/scala.majorVersion&gt;
      &lt;scala.minorVersion&gt;.2&lt;/scala.minorVersion&gt;
      &lt;spark.version&gt;1.4.1&lt;/spark.version&gt;
    &lt;/properties&gt;
  
  
    &lt;repositories&gt;
      &lt;repository&gt;
        &lt;id&gt;scala-tools.org&lt;/id&gt;
        &lt;name&gt;Scala-Tools Maven2 Repository&lt;/name&gt;
        &lt;url&gt;http://scala-tools.org/repo-releases&lt;/url&gt;
      &lt;/repository&gt;
    &lt;/repositories&gt;
  
    &lt;pluginRepositories&gt;
      &lt;pluginRepository&gt;
        &lt;id&gt;scala-tools.org&lt;/id&gt;
        &lt;name&gt;Scala-Tools Maven2 Repository&lt;/name&gt;
        &lt;url&gt;http://scala-tools.org/repo-releases&lt;/url&gt;
      &lt;/pluginRepository&gt;
    &lt;/pluginRepositories&gt;
  
    &lt;dependencies&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;org.scala-lang&lt;/groupId&gt;
        &lt;artifactId&gt;scala-library&lt;/artifactId&gt;
        &lt;version&gt;${scala.majorVersion}${scala.minorVersion}&lt;/version&gt;
      &lt;/dependency&gt;
      &lt;dependency&gt;
        &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
        &lt;artifactId&gt;spark-core_${scala.majorVersion}&lt;/artifactId&gt;
        &lt;version&gt;${spark.version}&lt;/version&gt;
      &lt;/dependency&gt;
    &lt;/dependencies&gt;
  
  
    &lt;build&gt;
        &lt;sourceDirectory&gt;src/main/scala&lt;/sourceDirectory&gt;
        &lt;plugins&gt;
          &lt;plugin&gt;
            &lt;groupId&gt;org.scala-tools&lt;/groupId&gt;
            &lt;artifactId&gt;maven-scala-plugin&lt;/artifactId&gt;
            &lt;executions&gt;
              &lt;execution&gt;
                &lt;goals&gt;
                  &lt;goal&gt;compile&lt;/goal&gt;
                  &lt;goal&gt;testCompile&lt;/goal&gt;
                &lt;/goals&gt;
              &lt;/execution&gt;
            &lt;/executions&gt;
          &lt;/plugin&gt;
          &lt;plugin&gt;
            &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
        &lt;artifactId&gt;maven-eclipse-plugin&lt;/artifactId&gt;
        &lt;configuration&gt;
          &lt;downloadSources&gt;true&lt;/downloadSources&gt;
          &lt;buildcommands&gt;
            &lt;buildcommand&gt;ch.epfl.lamp.sdt.core.scalabuilder&lt;/buildcommand&gt;
          &lt;/buildcommands&gt;
          &lt;additionalProjectnatures&gt;
            &lt;projectnature&gt;ch.epfl.lamp.sdt.core.scalanature&lt;/projectnature&gt;
          &lt;/additionalProjectnatures&gt;
          &lt;classpathContainers&gt;
            &lt;classpathContainer&gt;org.eclipse.jdt.launching.JRE_CONTAINER&lt;/classpathContainer&gt;
            &lt;classpathContainer&gt;ch.epfl.lamp.sdt.launching.SCALA_CONTAINER&lt;/classpathContainer&gt;
          &lt;/classpathContainers&gt;
        &lt;/configuration&gt;
      &lt;/plugin&gt;
      &lt;plugin&gt;
        &lt;artifactId&gt;maven-assembly-plugin&lt;/artifactId&gt;
        &lt;version&gt;2.4&lt;/version&gt;
        &lt;executions&gt;
          &lt;execution&gt;
            &lt;id&gt;make-assembly&lt;/id&gt;
            &lt;phase&gt;package&lt;/phase&gt;
            &lt;goals&gt;
              &lt;goal&gt;attached&lt;/goal&gt;
            &lt;/goals&gt;
          &lt;/execution&gt;
        &lt;/executions&gt;
        &lt;configuration&gt;
          &lt;tarLongFileMode&gt;gnu&lt;/tarLongFileMode&gt;
          &lt;descriptorRefs&gt;
            &lt;descriptorRef&gt;jar-with-dependencies&lt;/descriptorRef&gt;
          &lt;/descriptorRefs&gt;
        &lt;/configuration&gt;
      &lt;/plugin&gt;
      &lt;plugin&gt;
        &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
        &lt;artifactId&gt;maven-shade-plugin&lt;/artifactId&gt;
        &lt;version&gt;2.2&lt;/version&gt;
        &lt;executions&gt;
          &lt;execution&gt;
            &lt;phase&gt;package&lt;/phase&gt;
            &lt;goals&gt;
              &lt;goal&gt;shade&lt;/goal&gt;
            &lt;/goals&gt;
            &lt;configuration&gt;
              &lt;minimizeJar&gt;false&lt;/minimizeJar&gt;
              &lt;createDependencyReducedPom&gt;false&lt;/createDependencyReducedPom&gt;
              &lt;artifactSet&gt;
                &lt;includes&gt;
                  &lt;!-- Include here the dependencies you want to be packed in your fat jar --&gt;
                  &lt;include&gt;*:*&lt;/include&gt;
                &lt;/includes&gt;
              &lt;/artifactSet&gt;
              &lt;filters&gt;
                &lt;filter&gt;
                  &lt;artifact&gt;*:*&lt;/artifact&gt;
                  &lt;excludes&gt;
                    &lt;exclude&gt;META-INF/*.SF&lt;/exclude&gt;
                    &lt;exclude&gt;META-INF/*.DSA&lt;/exclude&gt;
                    &lt;exclude&gt;META-INF/*.RSA&lt;/exclude&gt;
                  &lt;/excludes&gt;
                &lt;/filter&gt;
              &lt;/filters&gt;
              &lt;transformers&gt;
                &lt;transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"&gt;
                  &lt;resource&gt;reference.conf&lt;/resource&gt;
                &lt;/transformer&gt;
              &lt;/transformers&gt;
            &lt;/configuration&gt;
          &lt;/execution&gt;
        &lt;/executions&gt;
      &lt;/plugin&gt;
      &lt;plugin&gt;
        &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
        &lt;artifactId&gt;maven-surefire-plugin&lt;/artifactId&gt;
        &lt;version&gt;2.7&lt;/version&gt;
        &lt;configuration&gt;
          &lt;skipTests&gt;true&lt;/skipTests&gt;
        &lt;/configuration&gt;
      &lt;/plugin&gt;
    &lt;/plugins&gt;
  &lt;/build&gt;
  &lt;reporting&gt;
    &lt;plugins&gt;
      &lt;plugin&gt;
        &lt;groupId&gt;org.scala-tools&lt;/groupId&gt;
        &lt;artifactId&gt;maven-scala-plugin&lt;/artifactId&gt;
      &lt;/plugin&gt;
    &lt;/plugins&gt;
  &lt;/reporting&gt;
  &lt;/project&gt;
  </code></pre>
  
  <p>Many thanks in advance for your help. </p>