title: How does Spark keep track of the splits in randomSplit?
body: <p>This question explains how Spark's random split works, <a href="https://stackoverflow.com/questions/32933143/how-does-sparks-rdd-randomsplit-actually-split-the-rdd/32933366">How does Sparks RDD.randomSplit actually split the RDD</a>, but I don't understand how spark keeps track of what values went to one split so that those same values don't go to the second split. </p>
  
  <p>If we look at the implementation of randomSplit:</p>
  
  <pre><code>def randomSplit(weights: Array[Double], seed: Long): Array[DataFrame] = {
   // It is possible that the underlying dataframe doesn't guarantee the ordering of rows in its
   // constituent partitions each time a split is materialized which could result in
   // overlapping splits. To prevent this, we explicitly sort each input partition to make the
   // ordering deterministic.
  
   val sorted = Sort(logicalPlan.output.map(SortOrder(_, Ascending)), global = false, logicalPlan)
   val sum = weights.sum
   val normalizedCumWeights = weights.map(_ / sum).scanLeft(0.0d)(_ + _)
   normalizedCumWeights.sliding(2).map { x =&gt;
    new DataFrame(sqlContext, Sample(x(0), x(1), withReplacement = false, seed, sorted))
  }.toArray
  }
  </code></pre>
  
  <p>we can see that it creates two DataFrames that share the same sqlContext and with two different Sample(rs). </p>
  
  <p>How are these two DataFrame(s) communicating with each other so that a value that fell in the first one is not included in the second one?</p>
  
  <p>And is the data being fetched twice? (Assume the sqlContext is selecting from a DB, is the select being executed twice?).</p>