title: Scala + SBT - How to configure reference.conf for a shaded Akka library
body: <h1>TL;DR</h1>
  <p>I am trying to shade a version of the <code>akka</code> library and bundle it with my application (to be able to run a <code>spray-can</code> server on the <code>CDH 5.7</code> version of <code>Spark 1.6</code>). The shading process <a href="https://github.com/sbt/sbt-assembly/issues/173" rel="nofollow noreferrer">messes up <code>akka</code>'s default configuration</a>, and after manually providing a separate version of <code>akka</code>'s <code>reference.conf</code> for the shaded <code>akka</code>, it still looks like the 2 versions get mixed up somehow.</p>
  <p>Is shading <code>akka</code> versions known to cause problems? What am I doing wrong?</p>
  <h1>Background</h1>
  <p>I have a <code>Scala</code>/<code>Spark</code> application currently running on <code>Spark 1.6.1</code> standalone. The application runs a <a href="http://spray.io/" rel="nofollow noreferrer"><code>spray-can</code></a> http server using <code>spray 1.3.3</code>, which requires <code>akka 2.3.9</code> (<code>Spark 1.6.1</code> standalone includes a compatible <code>akka 2.3.11</code>).</p>
  <p>I am trying to migrate the application to a new <a href="http://www.cloudera.com/documentation/enterprise/5-7-x.html" rel="nofollow noreferrer"><code>Cloudera</code></a>-based <code>Spark</code> cluster running the <code>CDH 5.7</code> version of <code>Spark 1.6</code>. The problem is that <code>Spark 1.6</code> in <code>CDH 5.7</code> is bundled with <code>akka 2.2.3</code> which is not sufficient for <code>spray 1.3.3</code> to function properly.</p>
  <h1>Attempted solution</h1>
  <p>Following the suggestion in <a href="https://community.cloudera.com/t5/Advanced-Analytics-Apache-Spark/Spark-1-6-1-on-CDH-5-7-0/td-p/45230" rel="nofollow noreferrer">this post</a>, I decided to shade <code>akka 2.3.9</code> and bundle it along with my application. Although this time I stumbled upon a new problem - <code>akka</code> has it's default configuration defined in a <code>reference.conf</code> file, which should be located on the application's classpath. Due to a <a href="https://github.com/sbt/sbt-assembly/issues/173" rel="nofollow noreferrer">known issue</a> in <code>sbt-assembly</code>'s shading feature, it seems that the shaded <code>akka</code> library would require a separate configuration.</p>
  <p>So, I ended up shading <code>akka</code> with the following shade rule:</p>
  <pre><code>ShadeRule.rename(&quot;akka.**&quot; -&gt; &quot;akka_2_3_9_shade.@1&quot;)
      .inLibrary(&quot;com.typesafe.akka&quot; % &quot;akka-actor_2.10&quot; % &quot;2.3.9&quot;)
      .inAll
  </code></pre>
  <p>and including an additional <code>reference.conf</code> file in my project, which is identical to <code>akka</code>'s original <code>reference.conf</code>, but with all occurances of &quot;akka&quot; replaced with &quot;akka_2_3_9_shade&quot;.</p>
  <p>Now, though, it seems that the <code>Spark</code>-provided <code>akka</code> gets mixed up somehow with the shaded <code>akka</code>, as I'm getting the following error:</p>
  <pre><code>Exception in thread &quot;main&quot; java.lang.IllegalArgumentException: Cannot instantiate MailboxType [akka.dispatch.UnboundedMailbox], defined in [akka.actor.default-mailbox], make sure it has a public constructor with [akka.actor.ActorSystem.Settings, com.typesafe.config.Config] parameters
      at akka_2_3_9_shade.dispatch.Mailboxes$$anonfun$1.applyOrElse(Mailboxes.scala:197)
      at akka_2_3_9_shade.dispatch.Mailboxes$$anonfun$1.applyOrElse(Mailboxes.scala:195)
      at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
      at scala.util.Failure$$anonfun$recover$1.apply(Try.scala:185)
      at scala.util.Try$.apply(Try.scala:161)
      at scala.util.Failure.recover(Try.scala:185)
      at akka_2_3_9_shade.dispatch.Mailboxes.lookupConfiguration(Mailboxes.scala:195)
      at akka_2_3_9_shade.dispatch.Mailboxes.lookup(Mailboxes.scala:78)
      at akka_2_3_9_shade.actor.LocalActorRefProvider.akka$actor$LocalActorRefProvider$$defaultMailbox$lzycompute(ActorRefProvider.scala:561)
      at akka_2_3_9_shade.actor.LocalActorRefProvider.akka$actor$LocalActorRefProvider$$defaultMailbox(ActorRefProvider.scala:561)
      at akka_2_3_9_shade.actor.LocalActorRefProvider$$anon$1.&lt;init&gt;(ActorRefProvider.scala:568)
      at akka_2_3_9_shade.actor.LocalActorRefProvider.rootGuardian$lzycompute(ActorRefProvider.scala:564)
      at akka_2_3_9_shade.actor.LocalActorRefProvider.rootGuardian(ActorRefProvider.scala:563)
      at akka_2_3_9_shade.actor.LocalActorRefProvider.init(ActorRefProvider.scala:618)
      at akka_2_3_9_shade.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:619)
      at akka_2_3_9_shade.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:616)
      at akka_2_3_9_shade.actor.ActorSystemImpl._start(ActorSystem.scala:616)
      at akka_2_3_9_shade.actor.ActorSystemImpl.start(ActorSystem.scala:633)
      at akka_2_3_9_shade.actor.ActorSystem$.apply(ActorSystem.scala:142)
      at akka_2_3_9_shade.actor.ActorSystem$.apply(ActorSystem.scala:109)
      at akka_2_3_9_shade.actor.ActorSystem$.apply(ActorSystem.scala:100)
      at MyApp.api.Boot$delayedInit$body.apply(Boot.scala:45)
      at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
      at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
      at scala.App$$anonfun$main$1.apply(App.scala:71)
      at scala.App$$anonfun$main$1.apply(App.scala:71)
      at scala.collection.immutable.List.foreach(List.scala:318)
      at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
      at scala.App$class.main(App.scala:71)
      at MyApp.api.Boot$.main(Boot.scala:28)
      at MyApp.api.Boot.main(Boot.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:601)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  Caused by: java.lang.ClassCastException: interface akka_2_3_9_shade.dispatch.MailboxType is not assignable from class akka.dispatch.UnboundedMailbox
      at akka_2_3_9_shade.actor.ReflectiveDynamicAccess$$anonfun$getClassFor$1.apply(DynamicAccess.scala:69)
      at akka_2_3_9_shade.actor.ReflectiveDynamicAccess$$anonfun$getClassFor$1.apply(DynamicAccess.scala:66)
      at scala.util.Try$.apply(Try.scala:161)
      at akka_2_3_9_shade.actor.ReflectiveDynamicAccess.getClassFor(DynamicAccess.scala:66)
      at akka_2_3_9_shade.actor.ReflectiveDynamicAccess.CreateInstanceFor(DynamicAccess.scala:84)
      ... 34 more
  </code></pre>
  <p>The relevant code from my application's <code>Boot.scala</code> file is the following:</p>
  <pre><code>[45]    implicit val system = ActorSystem()
  ...
  [48]    val service = system.actorOf(Props[MyAppApiActor], &quot;MyApp.Api&quot;)
  ...
  [52]    val port = config.getInt(&quot;MyApp.server.port&quot;)
  [53]    IO(Http) ? Http.Bind(service, interface = &quot;0.0.0.0&quot;, port = port)
  </code></pre>