title: Spark Shuffle - How workers know where to pull data from
body: <p>I am trying to understand how does Spark shuffle dependencies under the hood. Thus I have two questions:</p>
  
  <ol>
  <li><p>In Spark, how does an executor know from what other executors it has to pull data from? </p>
  
  <ul>
  <li>Does each executor, after finishing its map side task, update its status and location to some central entity ( may be driver) and reduce side executor first contact driver to get location of each executor to 
  pull from and then pull from those executors directly?</li>
  </ul></li>
  <li><p>In a job with shuffle dependency, does driver schedule joins (or other tasks on shuffle dependency) only after all map side tasks has finished? </p>
  
  <ul>
  <li>Does it mean that each task will notify driver about its status and driver will orchestrate other dependent tasks in timely manner.</li>
  </ul></li>
  </ol>