title: How to generate unique hash values?
body: <p>I have 10M records. Each record has about 100 fields. When I read a record, I want to determine whether the record is a duplicate of another record I have read earlier. The criteria for duplicate check is based on 20 fields and checks for exact equality. I can take the new record and compare against all the previous records for each of the 20 fields.</p>
  
  <p>Another approach I am considering is to hash the 20 fields into a single field and compare the hash with that of all the previous records. For this I need a hash function that will give me uniqueness. I am aware that there are functions like SHA512, SHA224, etc. Which hash function would be suitable for my use case? </p>
  
  <p>Regards,<br>
  Yash</p>