title: Hadoop append data to hdfs file and ignore duplicate entries
body: <p>How can I append data to HDFS files and ignore duplicate values?</p>
  
  <p>I have a huge HDFS file (MainFile) and I have 2 other new files from different sources and I want to append data from this files to the MainFile.</p>
  
  <p>Main File and the other files has same structure.</p>