21. A multi-tier document compression system, comprising: a first lexicon generator configured for receiving a set of documents, the set of documents comprising a sequence of tokens, wherein each of the tokens comprises document content in the set of documents; and assigning a unique first token identifier from a set of first token identifiers to each unique token in the set of documents based at least in part on the frequency of occurrence of the unique token in the set of documents, wherein high-frequency tokens are assigned smaller valued first token identifiers than low-frequency tokens; a second lexicon generator coupled to the first lexicon generator and configured for assigning a second token identifier from a set of second token identifiers to each first token identifier that is assigned to a token within a portion of the set of documents, wherein the second lexicon generator generates a mapping of the second token identifiers to all corresponding first token identifiers for the portion of the set of documents; and a repository configured for storing a sequence of the second token identifiers, wherein a sequence of second token identifiers represent the tokens in the portion of the set of documents for subsequent retrieval.
21. A multi-tier document compression system, comprising: a first lexicon generator configured for receiving a set of documents, the set of documents comprising a sequence of tokens, wherein each of the tokens comprises document content in the set of documents; and assigning a unique first token identifier from a set of first token identifiers to each unique token in the set of documents based at least in part on the frequency of occurrence of the unique token in the set of documents, wherein high-frequency tokens are assigned smaller valued first token identifiers than low-frequency tokens; a second lexicon generator coupled to the first lexicon generator and configured for assigning a second token identifier from a set of second token identifiers to each first token identifier that is assigned to a token within a portion of the set of documents, wherein the second lexicon generator generates a mapping of the second token identifiers to all corresponding first token identifiers for the portion of the set of documents; and a repository configured for storing a sequence of the second token identifiers, wherein a sequence of second token identifiers represent the tokens in the portion of the set of documents for subsequent retrieval. 22. The system of claim 21 , wherein each first token identifier comprises an M bit integer value.