title: How to pass configuration from driver to executors in Spark?
body: <p>For example.</p>
  <pre class="lang-scala prettyprint-override"><code>object App {
  
    var confValue: String = &quot;&quot;
  
    def main(args: Array[String]): Unit = {
      // set conf by cmd args
      confValue = args.head
      // do some context init
      val dataset: Dataset[Int] = ???
      dataset.foreach { row =&gt;
        // get conf from executor
        println(confValue)
      }
    }
  }
  </code></pre>
  <p>I want to get conf on executors, but actually it can't be done because <code>confValue</code> only has been modified on the driver</p>
  <p>I know I can pass <code>confValue</code> to executors by local variable like this.</p>
  <pre class="lang-scala prettyprint-override"><code>  def main(args: Array[String]): Unit = {
      // set conf by cmd args
      val confValue = args[0]
      // do some context init
      val dataset: Dataset[Int] = ???
      dataset.foreach { row =&gt;
        // get conf from executor
        println(confValue)
      }
    }
  </code></pre>
  <p>But my spark job is huge. It has so many functions. I can't pass confValue everywhere as a local variable. For example:</p>
  <pre class="lang-scala prettyprint-override"><code>  def main(args: Array[String]): Unit = {
      // set conf by cmd args
      val confValue = args[0]
      // do some context init
      val dataset: Dataset[Int] = ???
      dataset.foreach { row =&gt;
        doSomeLogic(row)
      }
    }
  
    private def doSomeLogic(row: Int): Unit = {
      // get conf from executor
      println(confValue)
    }
  </code></pre>
  <p>There is so many <code>doSomeLogic</code>. So I can't pass <code>confValue</code> to all of them.
  Is there some way to pass <code>confValue</code> to every executors automatically?</p>
  <hr />
  <p>updated 1</p>
  <p>My spark code may like below</p>
  <pre class="lang-scala prettyprint-override"><code>object App {
  
    /** env flag, will be inited by cmd args, and be used in executors */
    var env: String = &quot;&quot;
    val spark: SparkSession = ???
  
    import spark.implicits._
  
    def main(args: Array[String]): Unit = {
      // read env from args
      env = args.head
  
      var ds: Dataset[Int] = ???
      ds = doLogic1(ds)
      ds = doLogic2(ds)
      doLogic3(ds)
    }
  
    private def doLogic1(ds: Dataset[Int]): Dataset[Int] = {
      ds.map { row =&gt;
        // env will be used here
        ???
      }
    }
  
    private def doLogic2(ds: Dataset[Int]): Dataset[Int] = {
      ds.map { row =&gt;
        // env will be used here
        ???
      }
    }
  
    private def doLogic3(ds: Dataset[Int]): Dataset[Int] = {
      ds.map { row =&gt;
        // env will be used here
        ???
      }
    }
  }
  </code></pre>
  <p><code>env</code> will be inited in <code>main</code>, and will be used in some of <code>doLogicN</code> functions. My spark project is a large project with many <code>doLogicN</code> functions, so passing the <code>env</code> flag to every <code>doLogicN</code> function will change too many codes.</p>
  <p>What is the simplest way to pass the <code>env</code> flag to all <code>doLogicN</code> functions?</p>
  <p>The most difficult point is that the <code>env</code> will be used in executors. If it will only be used in drivers I can pass it to everywhere by global <code>env</code> variable. But it won't work well in executors because the global <code>env</code> variable hasn't been inited. It only be inited in the driver side.</p>