title: When and how does spark distribute partitions on executors
body: <p>How does spark assign a partition to an executor. </p>
  
  <p>When I ran the following lines in spark shell with 1 driver and 5 executors: </p>
  
  <pre><code>&gt; var data = sc.textFile("file") // auto generates 2 partitions
  &gt; data.count()                   // materialize partitions on two nodes
  &gt; data = data.repartition(10)    // repartition data.count()            
  &gt; data.count()                   // 10 partitions still on original 2 nodes
  </code></pre>
  
  <p>After repartitioning, the 10 partitions still lies on the original two nodes (out of 5). This seems very inefficient, because 5 tasks are repeatedly run on each of the node containing the partitions instead of evenly distributed across nodes. The inefficiency is most obvious for iterative tasks which repeat many times on the same rdds.</p>
  
  <p>So my question is, how does spark decide which node has which partition, and is there a way we can force data to be moved to other nodes?</p>