title: sbt-assembly: Merge Errors - Deduplicate
body: <p>I am getting these errors using <code>sbt assembly</code>.</p>
  
  <p>I am using Spark which seems to be at the root of this problem.</p>
  
  <pre><code>val Spark =  Seq(
    "org.apache.spark" %% "spark-core" % sparkVersion,
    "org.apache.spark" %% "spark-sql" % sparkVersion,
    "org.apache.spark" %% "spark-streaming" % sparkVersion
  )
  </code></pre>
  
  <p>Error:</p>
  
  <pre><code>[error] 12 errors were encountered during merge
  [trace] Stack trace suppressed: run last coreBackend/*:assembly for the full output.
  [trace] Stack trace suppressed: run last core/*:assembly for the full output.
  [trace] Stack trace suppressed: run last commons/*:assembly for the full output.
  [error] (coreBackend/*:assembly) deduplicate: different file contents found in the following:
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.osgi/org.osgi.core/org.osgi.core-4.3.1.jar:OSGI-OPT/bnd.bnd
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.osgi/org.osgi.compendium/org.osgi.compendium-4.3.1.jar:OSGI-OPT/bnd.bnd
  [error] deduplicate: different file contents found in the following:
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Absent.class
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Absent.class
  [error] deduplicate: different file contents found in the following:
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Function.class
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Function.class
  [error] deduplicate: different file contents found in the following:
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Optional$1$1.class
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional$1$1.class
  [error] deduplicate: different file contents found in the following:
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Optional$1.class
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional$1.class
  [error] deduplicate: different file contents found in the following:
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Optional.class
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional.class
  [error] deduplicate: different file contents found in the following:
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Present.class
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Present.class
  [error] deduplicate: different file contents found in the following:
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/bundles/com.google.guava/guava/guava-18.0.jar:com/google/common/base/Supplier.class
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Supplier.class
  [error] deduplicate: different file contents found in the following:
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-common/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-api/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class
  [error] deduplicate: different file contents found in the following:
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-common/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-api/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class
  [error] deduplicate: different file contents found in the following:
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-common/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.hadoop/hadoop-yarn-api/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class
  [error] deduplicate: different file contents found in the following:
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-core_2.11/spark-core_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-launcher_2.11/spark-launcher_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.spark-project.spark/unused/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-common_2.11/spark-network-common_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-network-shuffle_2.11/spark-network-shuffle_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-unsafe_2.11/spark-unsafe_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-sql_2.11/spark-sql_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-catalyst_2.11/spark-catalyst_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
  [error] /Volumes/COYOTE/Developer/tibra/lib_managed/jars/org.apache.spark/spark-streaming_2.11/spark-streaming_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
  [error] (core/*:assembly) deduplicate: different file contents found in the following:
  [error] /Users/bryan/.ivy2/cache/org.osgi/org.osgi.core/jars/org.osgi.core-4.3.1.jar:OSGI-OPT/bnd.bnd
  [error] /Users/bryan/.ivy2/cache/org.osgi/org.osgi.compendium/jars/org.osgi.compendium-4.3.1.jar:OSGI-OPT/bnd.bnd
  [error] deduplicate: different file contents found in the following:
  [error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Absent.class
  [error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Absent.class
  [error] deduplicate: different file contents found in the following:
  [error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Function.class
  [error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Function.class
  [error] deduplicate: different file contents found in the following:
  [error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Optional$1$1.class
  [error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional$1$1.class
  [error] deduplicate: different file contents found in the following:
  [error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Optional$1.class
  [error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional$1.class
  [error] deduplicate: different file contents found in the following:
  [error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Optional.class
  [error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Optional.class
  [error] deduplicate: different file contents found in the following:
  [error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Present.class
  [error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Present.class
  [error] deduplicate: different file contents found in the following:
  [error] /Users/bryan/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:com/google/common/base/Supplier.class
  [error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:com/google/common/base/Supplier.class
  [error] deduplicate: different file contents found in the following:
  [error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-common/jars/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class
  [error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/jars/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factories/package-info.class
  [error] deduplicate: different file contents found in the following:
  [error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-common/jars/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class
  [error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/jars/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/factory/providers/package-info.class
  [error] deduplicate: different file contents found in the following:
  [error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-common/jars/hadoop-yarn-common-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class
  [error] /Users/bryan/.ivy2/cache/org.apache.hadoop/hadoop-yarn-api/jars/hadoop-yarn-api-2.2.0.jar:org/apache/hadoop/yarn/util/package-info.class
  [error] deduplicate: different file contents found in the following:
  [error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-core_2.11/jars/spark-core_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
  [error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-launcher_2.11/jars/spark-launcher_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
  [error] /Users/bryan/.ivy2/cache/org.spark-project.spark/unused/jars/unused-1.0.0.jar:org/apache/spark/unused/UnusedStubClass.class
  [error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-common_2.11/jars/spark-network-common_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
  [error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-network-shuffle_2.11/jars/spark-network-shuffle_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
  [error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-unsafe_2.11/jars/spark-unsafe_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
  [error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-sql_2.11/jars/spark-sql_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
  [error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-catalyst_2.11/jars/spark-catalyst_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
  [error] /Users/bryan/.ivy2/cache/org.apache.spark/spark-streaming_2.11/jars/spark-streaming_2.11-1.5.1.jar:org/apache/spark/unused/UnusedStubClass.class
  [error] (commons/*:assembly) deduplicate: different file contents found in the following:
  [error] /Users/bryan/.ivy2/cache/org.osgi/org.osgi.core/jars/org.osgi.core-4.3.1.jar:OSGI-OPT/bnd.bnd
  [error] /Users/bryan/.ivy2/cache/org.osgi/org.osgi.compendium/jars/org.osgi.compendium-4.3.1.jar:OSGI-OPT/bnd.bnd
  </code></pre>
  
  <p>I tried all the recommend solutions here with no luck.</p>
  
  <p><a href="https://stackoverflow.com/questions/25144484/sbt-assembly-deduplication-found-error?rq=1">sbt-assembly: deduplication found error</a></p>
  
  <p><a href="https://stackoverflow.com/questions/30574262/deduplicating-commons-validator-sbt-assembly">deduplicating commons-validator - sbt assembly</a></p>
  
  <p><a href="https://stackoverflow.com/questions/30446984/spark-sbt-assembly-deduplicate-different-file-contents-found-in-the-followi">spark + sbt-assembly: &quot;deduplicate: different file contents found in the following&quot;</a></p>