16. A computer-implemented method for mapping a document to candidate duplicate documents in a document corpus, the method comprising: creating bitsets in a memory of a computer that are associated with tokens identified in documents in the document corpus, wherein the bitsets comprise a number of binary digits equivalent to the number of documents in the document corpus, and wherein the binary digits correspond to documents in the document corpus; for the binary digits in the bitsets, determining if the token associated with a bitset is contained in the document in the document corpus corresponding to a binary digit, and setting the binary digit if the token associated with the bitset is contained in the document in the document corpus corresponding to the binary digit; and compressing the bitsets in the memory of the computer.
16. A computer-implemented method for mapping a document to candidate duplicate documents in a document corpus, the method comprising: creating bitsets in a memory of a computer that are associated with tokens identified in documents in the document corpus, wherein the bitsets comprise a number of binary digits equivalent to the number of documents in the document corpus, and wherein the binary digits correspond to documents in the document corpus; for the binary digits in the bitsets, determining if the token associated with a bitset is contained in the document in the document corpus corresponding to a binary digit, and setting the binary digit if the token associated with the bitset is contained in the document in the document corpus corresponding to the binary digit; and compressing the bitsets in the memory of the computer. 18. The computer-implemented method of claim 16 , wherein the document is tokenized using adaptive tokenization, and wherein the tokens for the documents in the document corpus are generated using adaptive tokenization.