title: Mind blown: RDD.zip() method
body: <p>I just <a href="https://stackoverflow.com/q/29265616/850781">discovered</a> the <a href="http://spark.apache.org/docs/latest/api/java/org/apache/spark/rdd/RDD.html#zip%28org.apache.spark.rdd.RDD,%20scala.reflect.ClassTag%29" rel="noreferrer"><code>RDD.zip()</code></a> method and I cannot imagine what <strong><em>its <a href="http://en.wikipedia.org/wiki/Design_by_contract" rel="noreferrer">contract</a></em></strong> could possibly be.</p>
  
  <p>I understand what it <em>does</em>, of course. However, it has always been my understanding that </p>
  
  <ul>
  <li>the <em>order of elements</em> in an RDD is a meaningless concept</li>
  <li>the number of partitions and their sizes is an implementation detail only available to the user for performance tuning</li>
  </ul>
  
  <p>In other words, an RDD is a <em>(multi)set</em>, not a <em>sequence</em> (and, of course, in, e.g., Python one gets <code>AttributeError: 'set' object has no attribute 'zip'</code>)</p>
  
  <p>What is wrong with my understanding above?</p>
  
  <p>What was the rationale behind this method?</p>
  
  <p>Is it legal outside the trivial context like <code>a.map(f).zip(a)</code>?</p>
  
  <p>EDIT 1: </p>
  
  <ul>
  <li>Another crazy method is <a href="http://spark.apache.org/docs/latest/api/java/org/apache/spark/rdd/RDD.html#zipWithIndex%28%29" rel="noreferrer"><code>zipWithIndex()</code></a>, as well as well as the various <a href="http://spark.apache.org/docs/latest/api/java/org/apache/spark/rdd/RDD.html#zipPartitions%28org.apache.spark.rdd.RDD,%20boolean,%20scala.Function2,%20scala.reflect.ClassTag,%20scala.reflect.ClassTag%29" rel="noreferrer"><code>zipPartitions()</code></a> variants.</li>
  <li>Note that <a href="http://spark.apache.org/docs/latest/api/java/org/apache/spark/rdd/RDD.html#first%28%29" rel="noreferrer"><code>first()</code></a> and <a href="http://spark.apache.org/docs/latest/api/java/org/apache/spark/rdd/RDD.html#take%28int%29" rel="noreferrer"><code>take()</code></a> are <em>not</em> crazy because they are just (non-random) samples of the RDD. </li>
  <li><a href="http://spark.apache.org/docs/latest/api/java/org/apache/spark/rdd/RDD.html#collect%28%29" rel="noreferrer"><code>collect()</code></a> is also okay - it just converts a <code>set</code> to a <code>sequence</code> which is perfectly legit.</li>
  </ul>
  
  <p>EDIT 2: The <a href="https://stackoverflow.com/a/29270736/850781">reply</a> says:</p>
  
  <blockquote>
    <p>when you compute one RDD from another the order of elements in the new RDD may not correspond to that in the old one. </p>
  </blockquote>
  
  <p>This appears to imply that even the trivial <code>a.map(f).zip(a)</code> is <strong><em>not</em></strong> guaranteed to be equivalent to <code>a.map(x =&gt; (f(x),x))</code>. What is the situation when <code>zip()</code> results are <em>reproducible</em>?</p>