12. A method according to claim 10 , further comprising: comparing an original document to the remaining documents in the set to identify those documents that are duplicates and near duplicates of the original document.
12. A method according to claim 10 , further comprising: comparing an original document to the remaining documents in the set to identify those documents that are duplicates and near duplicates of the original document. 13. A method according to claim 12 , further comprising: breaking each of the segments of the documents into sliding windows of N words and hashing each of the sliding windows; comparing the hashed sliding windows for the original document with the hashed sliding windows for each of the remaining documents in the set; and determining the similarity for each remaining document based on a number of the hashed sliding windows shared by the original document and that remaining document and based on a total number of the hashed sliding windows of the original document and that remaining document.