10. A computer-implemented method for identifying near duplicate documents, comprising steps of: obtaining a set of documents; dividing each document into segments and hashing each of the segments; generating for each hashed segment in each document tuples, each tuple comprising a document identification for the document associated with the segment, a hash code, and a sequence order based on an order in which the segment occurs in that document; comparing the hashed segments of the documents; and designating as near duplicates, two or more of the documents each comprising a different number of hashed segments when a portion of the hashed segments for one of the designated documents matches hashed segments of the other designated documents and the matching hashed segments of the two or more documents occur in a common sequential order; identifying one of the near duplicate documents having one such matching segment with a lower sequence order than the other near duplicate documents; and selecting as an original document the identified near duplicate document having the matching segment with the lower sequence order than the other near duplicate documents, wherein the steps are performed by a programmed computer.
10. A computer-implemented method for identifying near duplicate documents, comprising steps of: obtaining a set of documents; dividing each document into segments and hashing each of the segments; generating for each hashed segment in each document tuples, each tuple comprising a document identification for the document associated with the segment, a hash code, and a sequence order based on an order in which the segment occurs in that document; comparing the hashed segments of the documents; and designating as near duplicates, two or more of the documents each comprising a different number of hashed segments when a portion of the hashed segments for one of the designated documents matches hashed segments of the other designated documents and the matching hashed segments of the two or more documents occur in a common sequential order; identifying one of the near duplicate documents having one such matching segment with a lower sequence order than the other near duplicate documents; and selecting as an original document the identified near duplicate document having the matching segment with the lower sequence order than the other near duplicate documents, wherein the steps are performed by a programmed computer. 17. A method according to claim 10 , further comprising: clustering the documents based on the hashed segments.