1. A method performed by one or more computers, comprising: receiving data indicating a candidate transcription for an utterance and a context for the utterance; accessing a language model that includes a respective score for each of a plurality of features, each feature corresponding to a word or phrase occurring in an associated context that includes one or more preceding words; determining that the language model does not include a score for a feature corresponding to the candidate transcription in the particular context; determining a score corresponding to the candidate transcription in the particular context, wherein the score is determined based on one or more scores included in the language model for one or more of the plurality of features that are associated with the particular context; determining, using the language model and the determined score, a probability score indicating a likelihood of occurrence of the candidate transcription in the particular context; selecting, based on the probability score, a transcription for the utterance from among a plurality of candidate transcriptions; and providing the selected transcription to a client device.