5. A system according to claim 1 , further comprising: a near duplicate similarity module to determine a similarity of the near duplicate documents.
1. A computer-implemented system for identifying near duplicate documents, comprising: a set of documents; and a server comprising memory and a central processing unit, wherein the central processing unit executes the following: a tokenizer to divide each document into segments; a hasher to hash each of the segments; an assignment module to generate for each hashed segment in each document a tuple comprising a document identification for the document associated with the segment, a hash code, and a sequence order based on an order in which the segment occurs in that document; a segment comparison module to compare the hashed segments of the documents; a document designation module to designate as near duplicates, two or more of the documents each comprising a different number of hashed segments when a portion of the hashed segments for one of the designated documents matches hashed segments of the other designated documents and the matching hashed segments occur in a common sequential order; and an original document selection module to identify one of the near duplicate documents having one such matching segment with a lower sequence order than the other near duplicate documents and to select as an original document the identified near duplicate document having the matching segment with the lower sequence order than the other near duplicate documents.