6. The computer-implemented method of claim 4 , wherein performing speech recognition on the utterance further comprises determining a sequence of the first model component for the first frame and the second model component for the second frame.
4. A computer-implemented method, comprising: as implemented by one or more computing devices configured to execute specific instructions, receiving an utterance; performing speech recognition on the utterance, wherein performing speech recognition on the utterance comprises: identifying a first model component and a first subword unit that are associated with a first frame; and identifying a second model component associated with a second frame; selecting, from a speaker model associated with a user, a first distribution of likelihoods associated with the first subword unit, wherein the first distribution of likelihoods comprises a likelihood that the first model component corresponds to the first frame; determining that the user is a speaker of the utterance from the likelihood that the first component corresponds to the first frame, the second model component, and the second frame, and the likelihood that the first model component corresponds to the first frame; and generating a textual transcription of the utterance from the speaker model associated with the user.