title: Custom Aggregator Function before Spark 1.5
body: <p>I am quite new to Spark, and I wonder how to do something that is quite simple to do with Cascading framework.</p>
  
  <p>Let's suppose I have the following dataset :</p>
  
  <pre><code>&lt;date&gt; &lt;cpt_id&gt; &lt;mesure_type&gt; &lt;value&gt;
  20160603093021556 cpt1 idx1 11
  20160603093021556 cpt1 idx2 22
  20160603093021556 cpt1 idx3 33
  20160603093021556 cpt1 idx4 44
  20160603113021556 cpt2 idx1 09
  20160603113021556 cpt2 idx2 45
  20160603113021556 cpt2 idx3 66
  20160603193021556 cpt1 idx1 13
  20160603193021556 cpt1 idx2 25
  20160603193021556 cpt1 idx3 33
  20160603193021556 cpt1 idx4 44
  </code></pre>
  
  <p>and I want to aggregate that to get the following results (kind of denormalisation) :</p>
  
  <pre><code>&lt;date&gt; &lt;cpt_id&gt; &lt;idx1&gt; &lt;idx2&gt; &lt;idx3&gt; &lt;idx4&gt;
  20160603093021556 cpt1 11 22 33 44
  20160603113021556 cpt2 09 45 66 null
  20160603193021556 cpt1 13 25 33 44
  </code></pre>
  
  <p>With Cascading I would use a GroupBy (with date and cpt-id as grouping keys) and an Every buffer, to generate denormalized Tuples.</p>
  
  <p>With Spark, it seems that an User Defined Aggregator Function would be needed, but it is only available since Spark 1.5 (and 1.3.1 is available on my Yarn cluster).</p>
  
  <p>I can't see how to do such a process with 1.3.1 APIs.</p>
  
  <p>Thanks for your help and suggestion </p>