1. A method for identifying near and exact-duplicate documents in a document collection, the method comprising: for each document in the collection performing by a computer the steps of: reading textual content from the document; filtering the textual content based on user settings; determining N most frequent words from the filtered textual content of the document to generate a first most frequent, word- 1 , to an Nth most frequent word, word-N, sorted from highest to lowest frequency; performing a quorum search using the N most frequent words from the filtered textual content in the document with a threshold M, wherein the threshold M is used to retrieve documents from the document collection having a number M of the N most frequent words; sorting results from the quorum search based on relevancy, whereby based on the values of N and M near and exact-duplicate documents are identified in the document collection; and determining the relevancy by taking a number of hits for the quorum search in the document and dividing the number of hits by a size of the document in kilobytes of text in the document or a size in kilobytes for the entire document.
1. A method for identifying near and exact-duplicate documents in a document collection, the method comprising: for each document in the collection performing by a computer the steps of: reading textual content from the document; filtering the textual content based on user settings; determining N most frequent words from the filtered textual content of the document to generate a first most frequent, word- 1 , to an Nth most frequent word, word-N, sorted from highest to lowest frequency; performing a quorum search using the N most frequent words from the filtered textual content in the document with a threshold M, wherein the threshold M is used to retrieve documents from the document collection having a number M of the N most frequent words; sorting results from the quorum search based on relevancy, whereby based on the values of N and M near and exact-duplicate documents are identified in the document collection; and determining the relevancy by taking a number of hits for the quorum search in the document and dividing the number of hits by a size of the document in kilobytes of text in the document or a size in kilobytes for the entire document. 3. The method of claim 1 , further comprising reading user preferences for the values of N and M, and including content filtering settings for the filtering of the textual content, wherein the content filtering settings include filtering of numbers, keyfields, noise words, and optical character recognition errors.