1. A multi-tier document compression method, comprising: identifying a set of unique tokens contained in a set of documents, the set of documents comprising a sequence of tokens wherein each token has a token position in the sequence of tokens and each of the unique tokens comprises document content in the set of documents; assigning a unique first token identifier from a set of first token identifiers to each unique token based at least in part on the frequency of occurrence of the unique token in the set of documents, wherein high-frequency tokens are assigned smaller valued first token identifiers than low-frequency tokens; selecting a range of token positions in the set of documents; assigning a second token identifier from a set of second token identifiers to each first token identifier that is assigned to a token within the selected range of token positions in the set of documents; storing the second token identifiers in a repository for subsequent retrieval, wherein a sequence of the second token identifiers in the repository represent document content in the selected range of token positions in the set of documents; and generating a mapping of the second token identifiers to all corresponding first token identifiers for the selected range of token positions.
1. A multi-tier document compression method, comprising: identifying a set of unique tokens contained in a set of documents, the set of documents comprising a sequence of tokens wherein each token has a token position in the sequence of tokens and each of the unique tokens comprises document content in the set of documents; assigning a unique first token identifier from a set of first token identifiers to each unique token based at least in part on the frequency of occurrence of the unique token in the set of documents, wherein high-frequency tokens are assigned smaller valued first token identifiers than low-frequency tokens; selecting a range of token positions in the set of documents; assigning a second token identifier from a set of second token identifiers to each first token identifier that is assigned to a token within the selected range of token positions in the set of documents; storing the second token identifiers in a repository for subsequent retrieval, wherein a sequence of the second token identifiers in the repository represent document content in the selected range of token positions in the set of documents; and generating a mapping of the second token identifiers to all corresponding first token identifiers for the selected range of token positions. 10. The method of claim 1 , further comprising: sorting the set of documents by one or more sorting criteria.