8. A method for determining whether a first document describes substantially similar subject matter to the subject matter described by a second document in a document corpus, wherein each of the first and second documents is organized into a plurality of fields, and wherein at least some of the fields of both the first document and the second document include tokens representing subject matter described by their respective document, the method comprising: obtaining the first document; selecting a field of the first document and selecting a token from the selected field, such that the selected token has the highest inverse document frequency (IDF) within the selected field with regard to all tokens in the document corpus; determining the number of tokens in the first document with a higher IDF value than the IDF value of the selected token; determining whether the number of tokens with a higher IDF value than the IDF value of the selected token exceeds a threshold; temporarily removing those tokens from the first document that have a higher IDF than the selected token if the threshold is not exceeded; generating a score based in part upon at least one remaining token common to both the first and second documents; identifying whether the first document describes substantially similar subject matter to the subject matter described by the second document in the document corpus, wherein the identifying comprises evaluating the score with respect to a threshold value; and providing an indication as to whether the first document describes substantially similar subject matter to the subject matter described by the second document in the document corpus according to the identification.
8. A method for determining whether a first document describes substantially similar subject matter to the subject matter described by a second document in a document corpus, wherein each of the first and second documents is organized into a plurality of fields, and wherein at least some of the fields of both the first document and the second document include tokens representing subject matter described by their respective document, the method comprising: obtaining the first document; selecting a field of the first document and selecting a token from the selected field, such that the selected token has the highest inverse document frequency (IDF) within the selected field with regard to all tokens in the document corpus; determining the number of tokens in the first document with a higher IDF value than the IDF value of the selected token; determining whether the number of tokens with a higher IDF value than the IDF value of the selected token exceeds a threshold; temporarily removing those tokens from the first document that have a higher IDF than the selected token if the threshold is not exceeded; generating a score based in part upon at least one remaining token common to both the first and second documents; identifying whether the first document describes substantially similar subject matter to the subject matter described by the second document in the document corpus, wherein the identifying comprises evaluating the score with respect to a threshold value; and providing an indication as to whether the first document describes substantially similar subject matter to the subject matter described by the second document in the document corpus according to the identification. 9. The method of claim 8 , wherein selecting a field of the first document comprises selecting a field according to its likelihood to contain tokens with the high IDF values.