36. A non-transitory computer-readable medium having stored thereon instructions, which, when executed by a processor in a document compression system, causes the processor to perform a multi-tier document compression method including the operations of: identifying a set of unique tokens contained in a set of documents, the set of documents comprising a sequence of tokens wherein each of the unique tokens comprises document content in the set of documents; assigning a unique first token identifier from a set of first token identifiers to each unique token based at least in part on the frequency of occurrence of the unique token in the set of documents, wherein high-frequency tokens are assigned smaller valued first token identifiers than low-frequency tokens; selecting a range of token positions in the set of documents; assigning a second token identifier from a set of second token identifiers to each first token identifier that is assigned to a token having a token position within the selected range of token positions in the documents; and storing the second token identifiers in a repository for subsequent retrieval, wherein a sequence of the second token identifiers in the repository represent document content in the selected range of token positions in the set of documents; and generating a mapping of the second token identifiers to all corresponding first token identifiers for the selected range of token positions.
36. A non-transitory computer-readable medium having stored thereon instructions, which, when executed by a processor in a document compression system, causes the processor to perform a multi-tier document compression method including the operations of: identifying a set of unique tokens contained in a set of documents, the set of documents comprising a sequence of tokens wherein each of the unique tokens comprises document content in the set of documents; assigning a unique first token identifier from a set of first token identifiers to each unique token based at least in part on the frequency of occurrence of the unique token in the set of documents, wherein high-frequency tokens are assigned smaller valued first token identifiers than low-frequency tokens; selecting a range of token positions in the set of documents; assigning a second token identifier from a set of second token identifiers to each first token identifier that is assigned to a token having a token position within the selected range of token positions in the documents; and storing the second token identifiers in a repository for subsequent retrieval, wherein a sequence of the second token identifiers in the repository represent document content in the selected range of token positions in the set of documents; and generating a mapping of the second token identifiers to all corresponding first token identifiers for the selected range of token positions. 51. The computer-readable medium of claim 36 , further comprising: associating ranges of token positions with portions of the set of documents; and storing a mapping of positions to sets of second token identifiers, wherein each set of second token identifiers corresponds to a respective portion of the set of documents.