1. A method of processing documents, comprising: at a computer system comprising one or more processors and memory storing one or more programs for execution by the one or more processors so as to perform the method, grouping a set of documents into a plurality of clusters, the set of documents comprising a sequence of tokens, wherein each cluster includes one or more documents in the set of documents; generating a compressed representation of the set of documents from which respective documents in the set can be reconstructed, including generating, for the plurality of clusters, a compressed sequence of tokens by eliding duplicate sequences of tokens within respective clusters; and generating an index of the compressed sequence of tokens, including indexing each respective token in the compressed sequence of tokens based on a respective token position of the respective token in the compressed sequence.
1. A method of processing documents, comprising: at a computer system comprising one or more processors and memory storing one or more programs for execution by the one or more processors so as to perform the method, grouping a set of documents into a plurality of clusters, the set of documents comprising a sequence of tokens, wherein each cluster includes one or more documents in the set of documents; generating a compressed representation of the set of documents from which respective documents in the set can be reconstructed, including generating, for the plurality of clusters, a compressed sequence of tokens by eliding duplicate sequences of tokens within respective clusters; and generating an index of the compressed sequence of tokens, including indexing each respective token in the compressed sequence of tokens based on a respective token position of the respective token in the compressed sequence. 4. The method of claim 1 , wherein generating the compressed sequence of tokens includes generating, for each cluster, document reconstruction data that comprises a sequence of add elements and copy elements, the add elements each indicating a string of tokens to be added to the compressed sequence of tokens and to a reconstruction of a corresponding document, and the copy elements each indicating a token position in the compressed sequence of tokens and a quantity of tokens to be copied from the compressed sequence of tokens, starting at the indicated token position, to the reconstruction of the corresponding document.