title: Filter out the duplicates in PIG
body: <p>I have a records like this.</p>
  
  <pre><code>12:-64:12033:24:0:0:1495532058:1384:0:0:0:102
  23:-64:8820:24:0:0:1495532126:2788:0:0:0:102
  23:-64:8826:24:0:0:1495532132:3064:0:0:0:102
  23:-64:8826:24:0:0:1495532132:3065:0:0:0:102
  </code></pre>
  
  <p>I want to filter the duplicate or identical rows in pig.</p>
  
  <p>Note: I dont want to delete or remove the duplicate rows. I need to filter the duplicate rows and store in to one variable.Any help will be appreciated.</p>
  
  <pre><code>out1 = GROUP out BY ($1,$7,$11);
  records3 = FOREACH out1 {
  top_record1 = LIMIT out 1;
  final_rec = DISTINCT top_record1;
  GENERATE FLATTEN(final_rec);
  };
  </code></pre>