title: Creating a unique key based on file content in python
body: <p>I got many, many files to be uploaded to the server, and I just want a way to avoid duplicates.</p>
  
  <p>Thus, generating a unique and small key value from a big string seemed something that a checksum was intended to do, and <a href="https://stackoverflow.com/questions/460576/hash-code-and-checksum-whats-the-difference">hashing seemed like the evolution of that</a>.</p>
  
  <p>So I was going to use hash md5 to do this. But then I read <a href="http://bytes.com/topic/python/answers/803689-md5-hash-url-utf-unicode-converting-ascii" rel="nofollow noreferrer">somewhere</a> that "MD5 are not meant to be unique keys" and I thought that's really weird.</p>
  
  <p><strong>What's the right way of doing this?</strong></p>
  
  <p><em>edit:</em> by the way, I took <a href="https://stackoverflow.com/questions/1131220/get-md5-hash-of-a-files-without-open-it-in-python/1131255#1131255">two</a> <a href="http://www.python.org/doc//current/library/hashlib.html#module-hashlib" rel="nofollow noreferrer">sources</a> to get to the following, which is how I'm currently doing it and it's working just fine, with Python 2.5:</p>
  
  <pre><code>import hashlib
  
  def md5_from_file (fileName, block_size=2**14):
      md5 = hashlib.md5()
      f = open(fileName)
      while True:
          data = f.read(block_size)
          if not data:
              break
          md5.update(data)
      f.close()
      return md5.hexdigest()
  </code></pre>