6. A system for detecting and managing duplicate documents, comprising: one or more processing units for executing programs; a network interface for receiving documents; and memory storing one or more programs to be executed by the one or more processing units, the one or more programs comprising instructions that when executed by the one or more processing units cause the system to: receive a first document, the received document characterized by a document content identifier; select, from a plurality of previously received documents, a set of documents sharing the same document content identifier, the first document and the selected set having associated score information; wherein documents having the same document content identifier have the same content and documents having different document content identifiers have different content; update the selected set of documents with the first document, in accordance with the score information associated with the first document and the selected set of documents, to produce an updated set of documents; determine a representative document for the updated set of documents in accordance with the score information; index the received document when the received document is the representative document for the updated set of documents; and repeat the receiving, selecting, updating, determining and indexing operations with respect to a plurality of received documents, each of which shares a respective document content identifier with a respective set of documents, such that at least some of the received documents are determined to be representative documents and are indexed.
6. A system for detecting and managing duplicate documents, comprising: one or more processing units for executing programs; a network interface for receiving documents; and memory storing one or more programs to be executed by the one or more processing units, the one or more programs comprising instructions that when executed by the one or more processing units cause the system to: receive a first document, the received document characterized by a document content identifier; select, from a plurality of previously received documents, a set of documents sharing the same document content identifier, the first document and the selected set having associated score information; wherein documents having the same document content identifier have the same content and documents having different document content identifiers have different content; update the selected set of documents with the first document, in accordance with the score information associated with the first document and the selected set of documents, to produce an updated set of documents; determine a representative document for the updated set of documents in accordance with the score information; index the received document when the received document is the representative document for the updated set of documents; and repeat the receiving, selecting, updating, determining and indexing operations with respect to a plurality of received documents, each of which shares a respective document content identifier with a respective set of documents, such that at least some of the received documents are determined to be representative documents and are indexed. 7. The system of claim 6 , wherein the document content identifier is a fixed length fingerprint of document content of a document characterized by the document content identifier.