title: How do I compare two CSV files in PIG, and output only the different columns?
body: <p>Between an older csv file and a newer one, I want to find what fields have changed on rows with the same key. For example, if the unique key is in field $2, and we have two files:</p>
  <blockquote>
  <pre><code>Old csv file: 
       FIELD1,FIELD2,ID,FIELD4 
       a,a,key1,a 
       b,b,key2,b
  
  New csv file:  
       FIELD1,FIELD2,ID,FIELD4
       a,a2,key1,a2 
       b,b,key2,b 
  </code></pre>
  </blockquote>
  <p>Desired output something like:</p>
  <blockquote>
  <pre><code>     {FIELD2:a2,ID:key1,FIELD4:a2}
  </code></pre>
  </blockquote>
  <p>or in other words, on the row with ID = key1, the 2nd and 4th fields changed, and these are the changed values.</p>
  <p>A pig script that outputs the whole row if any field has changed, is:</p>
  <pre><code>old = load '$old' using PigStorage('\n') as (line:chararray);
  new = load '$new' using PigStorage('\n') as (line:chararray);
  cg = cogroup old by line, new by line;
  new_only = foreach (filter cg by IsEmpty(old)) generate flatten(new);
  store new_only into '$changes';
  </code></pre>
  <p>My initial idea (I'm not sure how to complete it) is:</p>
  <pre><code>old = LOAD $old USING PigStorage('|');
  new = LOAD $new USING PigStorage('|');
  cogroup_data = COGROUP old by $2, new by $2 -- 3rd column is unique key
  diff_data = FOREACH cogroup_data GENERATE DIFF(old,new);
  -- ({(a,a,key2,a),(a,a2,key2,a2)})
  -- ? what goes here ?
  </code></pre>