1. A computer-implemented method, comprising: accessing audio data; accessing information that indicates a first context, the first context comprising a first physical environment or physical state of a device that records the audio data; accessing at least one term; accessing information that indicates a second context, the second context comprising a second physical environment or physical state associated with the accessed term; determining a similarity score that indicates a degree of similarity between the first physical environment or physical state and the second physical environment or physical state; adjusting a language model based on the accessed term and the determined similarity score to generate an adjusted language model, wherein the adjusted language model includes the accessed term and a weighting value assigned to the accessed term based on the similarity score; and performing speech recognition on the audio data using the adjusted language model to select one or more candidate transcriptions for a portion of the audio data.