title: spark java how to process rdd by key
body: <p>I’m just getting started with both Hadoop and Spark.  I’ve run into what I assume is a simple problem but I can’t seem to determine the best way forward!  I'm using Spark 1.2.1 since this is what is currently supported by Hortonworks which we are using.
  I have data in Hive that I’m analysing with the mlib KMeans clustering algorithm implementation.</p>
  
  <p>My data is naturally grouped by key, and I am doing my clustering analysis by this key.
  For Kmeans, I require a JavaRDD of the data points by key.</p>
  
  <p>My issue is that the only way I’ve managed to create this JavaRDD of data points is to do a mapToPair and GroupByKey on my source data in a JavaSchemaRDD, then loop over the keys, though I don’t think this is the optimal approach. 
  I guess what I need is a way to process my data and create the JavaRDD of data points by key.</p>
  
  <p>Sample code snippet:</p>
  
  <pre><code>// get the source data to cluster out of the database:
  JavaSchemaRDD sqlData = sqlContext.sql("SELECT * FROM "+sourceDBName+"."+sourceTableName);
  
  // grouped by key:
  JavaPairRDD&lt;String, Iterable&lt;Row&gt;&gt; groupedByKey = sqlData.mapToPair(new PairByKey()).groupByKey();
  
  // Loop over all keys:
  for (String key : groupedByKey.keys().collect())
  {
      // Filter for this Key:
      JavaSchemaRDD keyData = sqlData.filter(new FilterByKey(key));
  
      // Duration vs Score for this TP:
      JavaRDD&lt;Vector&gt; lengthVsScore = keyData.map(new GetLengthVsScore());
      lengthVsScore.cache();
  
      // Cluster the data into two classes using KMeans
      KMeansModel clusters = KMeans.train(lengthVsScore.rdd(), 2, 20);
  }
  </code></pre>
  
  <p>The hadoop table is created using a CSV file having four columns, data type in square brackets:
  actor[STRING],key[STRING],x[INT],y[INT]
  'key' is the value value that I want to group by, x and y are the numerical values that we wish to cluster on.</p>
  
  <p>Is there a better way to do this?  Avoid the for loop and groupedByKey.keys().collect()?</p>
  
  <p>Thanks in advance</p>