title: Spark data processing with grouping
body: <p>I need to group a set of csv lines by a certain column and do some processing on each group. </p>
  
  <pre><code>    JavaRDD&lt;String&gt; lines = sc.textFile
                          ("somefile.csv");
                  JavaPairRDD&lt;String, String&gt; pairRDD = lines.mapToPair(new SomeParser());
                  List&lt;String&gt; keys = pairRDD.keys().distinct().collect();
                  for (String key : keys)
                  {
                  List&lt;String&gt; rows = pairRDD.lookup(key);
  
              noOfVisits = rows.size();
              country = COMMA.split(rows.get(0))[6];
              accessDuration = getAccessDuration(rows,timeFormat);
              Map&lt;String,Integer&gt; counts = getCounts(rows);
              whitepapers = counts.get("whitepapers");
              tutorials = counts.get("tutorials");
              workshops = counts.get("workshops");
              casestudies = counts.get("casestudies");
              productPages = counts.get("productpages");        
              }
  
      private static long dateParser(String dateString) throws ParseException {
          SimpleDateFormat format = new SimpleDateFormat("MMM dd yyyy HH:mma");
          Date date = format.parse(dateString);
          return date.getTime();
      }
  dateParser is called for each row. Then min and max for the group is calculated to get the access duration. Others are string matches.
  </code></pre>
  
  <p>pairRDD.lookup is extremely slow.. Is there a better way to do this with spark.</p>