title: How to execute computation per executor in Spark
body: <p>In my computation, I</p>
  
  <ul>
  <li>first broadcast some data, say <code>bc</code>,</li>
  <li>then compute some big data shared by all executor/partition:<code>val shared = f(bc)</code></li>
  <li>then run the distributed computing, using shared data</li>
  </ul>
  
  <p>To avoid computing the <code>shared</code> data on all RDD items, I can use <code>.mapPartitions</code> but I have much more partitions than executors. So it run the computation of shared data more times than necessary.</p>
  
  <p>I found a simple method to compute the shared data only once per executor (which I understood is the JVM actually running the spark tasks): using <code>lazy val</code> on the broadcast data.</p>
  
  <pre><code>// class to be Broadcast
  case class BC(input: WhatEver){
    lazy val shared = f(input)
  }
  
  // in the spark code
  val sc = ... // init SparkContext
  val bc = sc.broadcast(BC(...))
  
  val initRdd = sc.parallelize(1 to 10000, numSlices = 10000)
  initRDD.map{ i =&gt;
    val shared = bc.value.shared
    ... // run computation using shared data
  }
  </code></pre>
  
  <p>I think this does what I want, but</p>
  
  <ol>
  <li>I am not sure, can someone guaranties it?</li>
  <li>I am not sure lazy val is the best way to manage concurrent access, especially with respect to the spark internal distribution system. Is there a better way?</li>
  <li>if computing <code>shared</code> fails, I think it will be recomputed for all RDD items with possible retries, instead of simply stopping the whole job with a unique error.</li>
  </ol>
  
  <p>So, is there a better way?</p>