title: Set hadoop configuration values on spark-submit command line
body: <p>We want to set the <code>aws</code> parameters that from code would be done via the <code>SparkContext</code>:</p>
  
  <pre><code>sc.hadoopConfiguration.set("fs.s3a.access.key", vault.user)
  sc.hadoopConfiguration.set("fs.s3a.secret.key", vault.key)
  </code></pre>
  
  <p>However we have a custom Spark launcher framework that requires all the custom Spark configurations to be done via <code>--conf</code> parameters to the <code>spark-submit</code> command line.</p>
  
  <p>Is there a way to "notify" the SparkContext to set <code>--conf</code> values to the <code>hadoopConfiguration</code> and not to its general <code>SparkConf</code> ?  Looking for something along the lines of</p>
  
  <pre><code>spark-submit --conf hadoop.fs.s3a.access.key $vault.user --conf hadoop.fs.s3a.access.key $vault.key
  </code></pre>
  
  <p>or</p>
  
  <pre><code>spark-submit --conf hadoopConfiguration.fs.s3a.access.key $vault.user --conf hadoopConfiguration.fs.s3a.access.key $vault.key
  </code></pre>