11. One or more non-transitory computer readable media comprising executable code that, when executed, cause one or more computing devices to perform a process, comprising: receiving an utterance; performing speech recognition on the utterance, wherein performing speech recognition on the utterance comprises: identifying a first model component and a first subword unit that are associated with a first frame; and identifying a second model component associated with a second frame; selecting, from a speaker model associated with a user, a first distribution of likelihoods associated with the first subword unit, wherein the first distribution of likelihoods comprises a likelihood that the first component corresponds to the first frame; determining that the user is a speaker of the utterance from the second model component, the second frame, and the likelihood that the first model component corresponds to the first frame; and generating a textual transcription of the utterance from the speaker model associated with the user.
17. The one or more non-transitory computer readable media of claim 11 , wherein determining that the user is the speaker of the utterance is further based on at least one of: a duration of a particular phoneme state in speech recognition processing; which pronunciation of a word is used in the utterance; where vocal stress is applied; a vocal-track-length-normalizing factor used in speech recognition processing; a channel means normalization used in speech recognition processing; a constrained maximum likelihood linear regression transform used in speech recognition processing; time of day; or recent patterns of system usage.