1. A computer-implemented method for compressing a list of words in an index in two phases: a decomposition phase and an encoding phase, the method comprising: during the decomposition phase, generating static decomposition statistics and parsing all words in the list of words to create a symbol list, wherein parsing all words in the list of words to create the symbol list further comprises: generating a list of bi-grams from each word in the list of words, wherein each bigram contains two characters extracted from the word, storing the bi-grams in the symbol list, counting occurrence of each bi-gram, storing the occurrence count as a static decomposition statistic for each bigram; associating the occurrence count with the bi-gram, determining whether the occurrence count is above a specified threshold, filtering the bi-gram when the occurrence count is below the specified threshold by dropping the bi-gram from the symbol list, creating a super-gram that represents the bi-gram when the occurrence count is above the specified threshold, replacing the bi-gram with the super-gram in the symbol list, treating the super-gram as a single character, forming new bi-grams using the super-gram, and calculating an occurrence count for the new bi-grams; identifying substrings in each word in the list of words by decomposing the word based on the symbol list; and during the encoding phase, encoding the identified substrings with the symbol list, wherein the symbol list is associated with weights that vary based on decomposition progress and the weights include the static decomposition statistics and at least one post-decomposition statistic.