title: No FileSystem for scheme: hdfs when building fat jar in Spark; works fine in Eclipse mars
body: <p>I get that error when I do a fat jar with my spark program and then I launch it using:</p>
  
  <pre><code>java -jar -myApp.jar [args]
  </code></pre>
  
  <p>When I run my app in Eclipse, it works perfectly well. Here is my POM:  </p>
  
  <pre><code> &lt;properties&gt;
      &lt;project.build.sourceEncoding&gt;UTF-8&lt;/project.build.sourceEncoding&gt;
     &lt;maven.compiler.source&gt;1.7&lt;/maven.compiler.source&gt;
      &lt;maven.compiler.target&gt;1.7&lt;/maven.compiler.target&gt;
    &lt;/properties&gt;
  
    &lt;dependencies&gt;
  &lt;dependency&gt;
      &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
      &lt;artifactId&gt;spark-streaming-kafka_2.10&lt;/artifactId&gt;
      &lt;version&gt;1.5.0&lt;/version&gt;
  &lt;/dependency&gt;
  &lt;dependency&gt;
      &lt;groupId&gt;org.apache.kafka&lt;/groupId&gt;
      &lt;artifactId&gt;kafka_2.10&lt;/artifactId&gt;
      &lt;version&gt;0.9.0.1&lt;/version&gt;
  &lt;/dependency&gt;
  &lt;dependency&gt;
      &lt;groupId&gt;org.apache.spark&lt;/groupId&gt;
      &lt;artifactId&gt;spark-streaming_2.10&lt;/artifactId&gt;
      &lt;version&gt;1.6.1&lt;/version&gt;
  &lt;/dependency&gt;
  &lt;dependency&gt;
      &lt;groupId&gt;com.google.code.gson&lt;/groupId&gt;
      &lt;artifactId&gt;gson&lt;/artifactId&gt;
      &lt;version&gt;2.6.2&lt;/version&gt;
  &lt;/dependency&gt;
  &lt;dependency&gt;
      &lt;groupId&gt;org.apache.hadoop&lt;/groupId&gt;
      &lt;artifactId&gt;hadoop-hdfs&lt;/artifactId&gt;
      &lt;version&gt;2.6.0&lt;/version&gt;
  &lt;/dependency&gt;
   &lt;/dependencies&gt;
   &lt;build&gt;
  &lt;plugins&gt;
           &lt;plugin&gt;
              &lt;groupId&gt;org.apache.maven.plugins&lt;/groupId&gt;
              &lt;artifactId&gt;maven-shade-plugin&lt;/artifactId&gt;
              &lt;version&gt;2.1&lt;/version&gt;
                  &lt;executions&gt;
                      &lt;execution&gt;
                      &lt;phase&gt;package&lt;/phase&gt;
                          &lt;goals&gt;
                              &lt;goal&gt;shade&lt;/goal&gt;
                          &lt;/goals&gt;
                          &lt;configuration&gt;
                              &lt;source&gt;1.7&lt;/source&gt;
                              &lt;target&gt;1.7&lt;/target&gt;
                              &lt;transformers&gt;
                                  &lt;transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"&gt;
                                      &lt;resource&gt;reference.conf&lt;/resource&gt;
                                  &lt;/transformer&gt;
                                  &lt;transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"&gt;
                                      &lt;mainClass&gt;gft.hbaseMedia.MainSparkCopyHDFS&lt;/mainClass&gt;
                                  &lt;/transformer&gt;
                              &lt;/transformers&gt;
                                  &lt;filters&gt;
                                      &lt;filter&gt;
                                          &lt;artifact&gt;*:*&lt;/artifact&gt;
                                          &lt;excludes&gt;  
                                              &lt;exclude&gt;META-INF/*.SF&lt;/exclude&gt;
                                              &lt;exclude&gt;META-INF/*.DSA&lt;/exclude&gt;
                                              &lt;exclude&gt;META-INF/*.RSA&lt;/exclude&gt;
                                          &lt;/excludes&gt;
                                      &lt;/filter&gt;
                                  &lt;/filters&gt;
                          &lt;/configuration&gt;
                      &lt;/execution&gt;
                  &lt;/executions&gt;
              &lt;/plugin&gt;
      &lt;/plugins&gt;
      &lt;resources&gt;
              &lt;resource&gt;
                  &lt;directory&gt;resources&lt;/directory&gt;
              &lt;/resource&gt;
          &lt;/resources&gt;
    &lt;/build&gt;
  &lt;/project&gt;
  </code></pre>
  
  <p>The piece of code that gives problems is:</p>
  
  <pre><code>Configuration conf = new Configuration();
  conf.set("fs.defaultFS", hdfsConfPath);
  FileSystem fs = FileSystem.get(conf);
  </code></pre>
  
  <p>I know it's Maven that's screwing up things somehow, but I don't know how to solve this issue!</p>