title: Avoid duplicate content on Node.js server
body: <p>I have small image hosting and I realized there many duplicate content. I want to eliminate this problem in the future by using checksum or hash code where newly uploaded file will be hashed, compared with existing image hash database, deleted if it already exist and user will be presented with the existing image link. All in one instance<br/><br/></p>
  
  <p>My setup is barebones Node.js+<a href="https://github.com/blueimp/jQuery-File-Upload" rel="nofollow">jQuery File Upload</a>+2 directories(one for a forum upload, another one for direct web upload).<br/><br/></p>
  
  <p>What is the best(fast&amp;reliable) hash and database setup for me to do this given the possibilities there might be thousand or million files in each directory? I think MD5 or SHA1 is overkill and might take a lot of resources. I would like to know if there any simpler solution.<br/><br/></p>
  
  <p><b>Statistics :</b><br/>
  ~1,000 image uploaded everyday<br/>
  ~400 kb average image size<br/>
  ~35,000 image in the server<br/>
  ~30% duplicated content (tested using MD5)</p>