title: Apache Spark: Splitting Pair RDD into multiple RDDs by key to save values
body: <p>I am using Spark 1.0.1 to process a large amount of data. Each row contains an ID number, some with duplicate IDs. I want to save all the rows with the same ID number in the same location, but I am having trouble doing it efficiently. I create an RDD[(String, String)] of (ID number, data row) pairs:</p>
  
  <pre class="lang-scala prettyprint-override"><code>val mapRdd = rdd.map{ x=&gt; (x.split("\\t+")(1), x)} 
  </code></pre>
  
  <p>A way that works, but is not performant, is to collect the ID numbers, filter the RDD for each ID, and save the RDD of values with the same ID as a text file.</p>
  
  <pre class="lang-scala prettyprint-override"><code>val ids = rdd.keys.distinct.collect
  ids.foreach({ id =&gt;
      val dataRows = mapRdd.filter(_._1 == id).values
      dataRows.saveAsTextFile(id)
  })
  </code></pre>
  
  <p>I also tried a groupByKey or reduceByKey so that each tuple in the RDD contains a unique ID number as the key and a string of combined data rows separated by new lines for that ID number. I want to iterate through the RDD only once using foreach to save the data, but it can't give the values as an RDD</p>
  
  <pre class="lang-scala prettyprint-override"><code>groupedRdd.foreach({ tup =&gt;
    val data = sc.parallelize(List(tup._2)) //nested RDD does not work
    data.saveAsTextFile(tup._1)
  })
  </code></pre>
  
  <p>Essentially, I want to split an RDD into multiple RDDs by an ID number and save the values for that ID number into their own location.</p>