title: Spark RDD.aggregate vs RDD.reduceByKey?
body: <p>I have and RDD[String] containing one word per line. The size is currently very small, 10-20k lines, but the goal is to scale this up to hundreds of millions of lines. The issue I have is that doing a map/reduceByKey operation is taking surprisingly long even for this small dataset. I run the following:</p>
  
  <pre><code>val wordcount = filtered.map(w =&gt; (w,1)).reduceByKey(_ + _)
  </code></pre>
  
  <p>and for 16780 lines it takes 12321 ms on a 2 GHz i7 8 GB RAM machine. I found that there is a method called aggregate that might be more memory efficient and hence faster:</p>
  
  <pre><code>aggregate[U: ClassTag](zeroValue: U)(seqOp: (U, T) =&gt; U, combOp: (U, U) =&gt; U): U
  </code></pre>
  
  <p>I can't quite figure out how to implement this in my case. I'm assuming it should be something like</p>
  
  <pre><code>aggregate(collection.immutable.Map)(??)
  </code></pre>
  
  <p>So my questions are</p>
  
  <p><strong>1) Does it make sense to use aggregate instead of reduceByKey</strong></p>
  
  <p><strong>2) If it does, how would it be implemented?</strong></p>