title: Remove the duplicate row in apache pig
body: <p>I want to remove the duplicate rows in pig. There are a lot of ways, but I am not sure if which one is better.</p>
  
  <p>Here is the data set, the schema is (f0,f1,id,f3,f4):</p>
  
  <pre><code>1,2,3,2015-02-21,2015-02-20
  1,2,3,2015-02-22,2015-02-20
  1,2,3,2015-02-23,2015-02-20
  1,2,4,2015-02-24,2015-02-20
  1,2,5,2015-02-25,2015-02-20
  </code></pre>
  
  <p>If any of rows whose f0,f1 and id are equal, then they are considered to be the duplicate. And I want to output one of them where f3 is minimum.</p>
  
  <p>But I also want to output which ids have the duplicates.</p>
  
  <p>That is, I will store or dump two relations.</p>
  
  <p>one of both relations are:</p>
  
  <pre><code>1,2,3,2015-02-21,2015-02-20
  1,2,4,2015-02-24,2015-02-20
  1,2,5,2015-02-25,2015-02-20
  </code></pre>
  
  <p>The other one is the id which has the duplicate rows, the schema is (id,f4)</p>
  
  <pre><code>3,2015-02-20
  </code></pre>
  
  <p>That is, id=3 has the duplicate data.</p>
  
  <p>Here is my workaround</p>
  
  <pre><code>r1 = LOAD 'data' USING PigStorage(',');
  r2 =  group r1 by ($0,$1,$2);
  r3 = FOREACH r2 GENERATE COUNT(r1) as c, r1;
  SPLIT r3 into r4 if c &gt; 1, r5 if c == 1;
  r6 = FOREACH r5 GENERATE flatten(r1);
  
  dups_id = FOREACH r4 {
  GENERATE flatten(r1.$2),flatten(r1.$4);
  };
  
  r7= distinct dups_id
  
  dump r7
  
  no_dups = FOREACH r4 {
  sorted = ORDER r1 by $3 ASC;
  lim = limit sorted 1;
  GENERATE flatten(lim);
  };
  
  r8 = union no_dups,r6
  
  dump r8
  </code></pre>
  
  <p>I think that this is a little complicated, and I doubt the performance.</p>
  
  <p>Is there any other better idea can implement this use case?</p>