1. A method of detecting a duplicate document, the method comprising: selecting a first document and a second document from a set of documents; comparing, by operation of a processor adapted to manipulate data within a computer system, a document description of the first document with a document description of the second document, wherein the document description of the first document comprises a selected subset of sentences of the first document, the sentences being selected and ordered in the document description as a function of a number of related phrases in the selected sentences, wherein the document description of the second document comprises a selected subset of sentences of the second document, the sentences being selected and ordered in the document description as a function of a number of related phrases in the selected sentences, wherein a phrase g j is a related phrase of another phrase g k occurring in the set of documents when an information gain of g j with respect to g k exceeds a predetermined threshold, the information gain of g j with respect to g k being a function of both actual and expected co-occurrence rates of g j and g k in the set of documents; and responsive to the document description of the first document matching the document description of the second document, identifying the first document and the second document as duplicate documents in the set of documents.
1. A method of detecting a duplicate document, the method comprising: selecting a first document and a second document from a set of documents; comparing, by operation of a processor adapted to manipulate data within a computer system, a document description of the first document with a document description of the second document, wherein the document description of the first document comprises a selected subset of sentences of the first document, the sentences being selected and ordered in the document description as a function of a number of related phrases in the selected sentences, wherein the document description of the second document comprises a selected subset of sentences of the second document, the sentences being selected and ordered in the document description as a function of a number of related phrases in the selected sentences, wherein a phrase g j is a related phrase of another phrase g k occurring in the set of documents when an information gain of g j with respect to g k exceeds a predetermined threshold, the information gain of g j with respect to g k being a function of both actual and expected co-occurrence rates of g j and g k in the set of documents; and responsive to the document description of the first document matching the document description of the second document, identifying the first document and the second document as duplicate documents in the set of documents. 6. The method of claim 1 , wherein comparing the document description of the first document with a document description of the second document further comprises: for the first document, generating the document description of the first document by selecting sentences of the document, and ordering the selected sentences in the document description as a function of a number of related phrases in the selected sentences; and for the second document, generating the document description of the second document by selecting sentences of the document, and ordering the selected sentences in the document description as a function of a number of related phrases in the selected sentences.