1. A system comprising: a computer-readable storage medium including instructions; and data processing apparatus capable of executing the instructions to perform operations comprising: generating a compact language model, including: receiving a collection of n-grams from a corpus, each n-gram of the collection having a corresponding first probability of occurring in the corpus, and generating a trie representing the collection of n-grams including calculating a left word vector and a diversity count vector using the collection of n grams, the left word vector identifying each distinct left word for a given right context in the collection, the diversity count vector identifying a count of distinct left words for each right context in the collection, and using the language model to identify a second probability of a particular string of words occurring.