1. A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: receiving speech data and data indicating a candidate transcription for the speech data; accessing a phonetic representation for the candidate transcription; extracting, from the phonetic representation, multiple test sequences for a particular phone in the phonetic representation, each of the multiple test sequences including a different set of contextual phones surrounding the particular phone; receiving data indicating that an acoustic model includes data corresponding to one or more of the multiple test sequences; selecting, from among the one or more test sequences for which the acoustic model includes data, the test sequence that includes the highest number of contextual phones; accessing data from the acoustic model corresponding to the selected test sequence; and generating a score for the candidate transcription based on the accessed data from the acoustic model that corresponds to the selected test sequence.