title: How to build shaded jars for a Spark application for distributed and local execution?
body: <p>I'm building a Spark application and I want to generate two separate shaded .jar files, one for each of these contexts:</p>
  
  <ol>
  <li><p>For master=local mode, I want a single .jar file that can be executed with <code>java -jar shaded-for-local-mode.jar</code>. This should include all of the dependencies, including the Spark and Hadoop dependencies in use.</p></li>
  <li><p>For distributed mode, I want a single .jar file that excludes the Spark and Hadoop libraries (<code>org.apache.hadoop:*</code>) so that they do not conflict with the runtime environment provided by <code>spark-submit</code> (<a href="https://spark.apache.org/docs/1.6.1/submitting-applications.html" rel="nofollow noreferrer">spark docs</a>), but I also want to include a dependency that is within the <code>org.apache.hadoop</code> group (<code>org.apache.hadoop:hadoop-aws</code>) because it isn't provided by the runtime environment.</p></li>
  </ol>
  
  <p><a href="https://stackoverflow.com/questions/17845423/maven-shade-plugin-to-produce-two-jars">This answer</a> explained how to create two jars by using two separate <code>&lt;execution&gt;</code> blocks, but I'm having trouble getting the excludes to work as I want them. </p>
  
  <p>Here's the relevant shade <code>&lt;execution&gt;</code>:</p>
  
  <pre><code>&lt;execution&gt;
    &lt;id&gt;shade-spark-submit&lt;/id&gt;
    &lt;phase&gt;package&lt;/phase&gt;
    &lt;goals&gt;
      &lt;goal&gt;shade&lt;/goal&gt;
    &lt;/goals&gt;
    &lt;configuration&gt;
      &lt;shadedClassifierName&gt;shaded-spark-submit&lt;/shadedClassifierName&gt;
      &lt;artifactSet&gt;
        &lt;excludes&gt;
          &lt;exclude&gt;org.apache.spark:*&lt;/exclude&gt;
          &lt;!-- We want hadoop-aws, an explicit dependency of this project, but not any of the other hadoop packages. --&gt;
          &lt;exclude&gt;org.apache.hadoop:hadoop-auth&lt;/exclude&gt;
          &lt;exclude&gt;org.apache.hadoop:hadoop-common&lt;/exclude&gt;
          &lt;exclude&gt;org.apache.hadoop:hadoop-annotations&lt;/exclude&gt;
        &lt;/excludes&gt;
      &lt;/artifactSet&gt;
      &lt;finalName&gt;${project.artifactId}-${project.version}-shade-spark-submit&lt;/finalName&gt;
    &lt;/configuration&gt;
  &lt;/execution&gt;
  </code></pre>
  
  <p>My concern is that the <code>&lt;exclude&gt;</code> statements are explicitly excluding specific packages that I know about, which is slightly different than my goal: I want to always exclude <code>org.apache.hadoop:*</code> (even ones I don't happen to know about) but <em>include</em> <code>org.apache.hadoop:hadoop-aws</code>. The documentation for shade does not fully describe how <code>&lt;include&gt;</code> and <code>&lt;exclude&gt;</code> tags are processed.</p>
  
  <p>Thanks!</p>