title: Typesafe Config in Spark
body: <p>I've defined a default configuration in my Spark application which is tucked in <code>src/main/resources/reference.conf</code>. I use <code>ConfigFactory.load()</code> to obtain the configuration.</p>
  
  <p>When I run the application with <code>spark-submit</code> it picks up these defaults. However, when I only want to override a few of the configurations available in <code>reference.conf</code> and provide <code>application.conf</code>, it does not seem to pick up these overrides. From the documentation I thought that <code>application.conf</code> is merged with <code>reference.conf</code> when calling <code>load()</code>, so that it's not necessary to re-define everything in <code>application.conf</code>.</p>
  
  <p>My <code>reference.conf</code> looks like this:</p>
  
  <pre><code>hdfs {
    rootDir: "/foo"
    dataDir: "hdfs://"${hdfs.rootDir}"/bar"
  }
  
  db {
    driver: "com.mysql.jdbc.Driver"
    ...
  }
  
  ...
  </code></pre>
  
  <p>What I'd now like to do is have an <code>application.conf</code> with, say, only a custom <code>hdfs</code> section because the rest is the same.</p>
  
  <p>I run my Spark app by supplying <code>application.conf</code> in both the <code>--files</code> parameter, in <code>--driver-class-path</code>, and <code>--conf spark.executor.extraClassPath</code>. This may be overkill but it works when I create a copy of <code>reference.conf</code> and change a few of the fields.</p>
  
  <p>What am I missing?</p>