title: Removing duplicate lines from a large dataset
body: <p>Let's assume that I have a very large dataset that can not be fit into the memory, there are millions of records in the dataset and I want to remove duplicate rows (actually keeping one row from the  duplicates)</p>
  
  <p>What's the most efficient approach in terms of space and time complexity ? </p>
  
  <p>What I thought : </p>
  
  <p>1.Using bloom filter , I am not sure about how it's implemented , but I guess the side effect is having false-positives , in that case how can we find if it's REALLY a duplicate or not ? </p>
  
  <p>2.Using hash values , in this case if we have a small number of duplicate values, the number of unique hash values would be large and again we may have problem with memory , </p>