17. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: accessing audio data; accessing information that indicates a first context, the first context comprising a first physical environment or physical state of a device that records the audio data; accessing at least one term; accessing information that indicates a second context, the second context comprising a second physical environment or physical state associated with the accessed term; determining a similarity score that indicates a degree of similarity between the first physical environment or physical state and the second physical environment or physical state; adjusting a language model based on the accessed term and the determined similarity score to generate an adjusted language model, wherein the adjusted language model includes the accessed term and a weighting value assigned to the accessed term based on the similarity score; and performing speech recognition on the audio data using the adjusted language model to select one or more candidate transcriptions for a portion of the audio data.
17. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: accessing audio data; accessing information that indicates a first context, the first context comprising a first physical environment or physical state of a device that records the audio data; accessing at least one term; accessing information that indicates a second context, the second context comprising a second physical environment or physical state associated with the accessed term; determining a similarity score that indicates a degree of similarity between the first physical environment or physical state and the second physical environment or physical state; adjusting a language model based on the accessed term and the determined similarity score to generate an adjusted language model, wherein the adjusted language model includes the accessed term and a weighting value assigned to the accessed term based on the similarity score; and performing speech recognition on the audio data using the adjusted language model to select one or more candidate transcriptions for a portion of the audio data. 20. The non-transitory computer storage medium of claim 17 , wherein the operations comprise determining that the accessed term was entered by a user, wherein: the audio data encodes speech of the user; the first context includes the environment in which the speech occurred; and the second context includes the environment in which the accessed term was previously entered by the user.