1. A system comprising: a non-transitory computer-readable memory storing executable instructions; and one or more processors in communication with the non-transitory computer-readable memory, wherein the one or more processors are programmed by the executable instructions to at least: receive, from a client device, audio data regarding an utterance of a speaker; obtain from the audio data a plurality of frames, the plurality of frames comprising a first frame and a second frame; perform speech recognition on at least a portion of the audio data to determine: a first subword unit of a plurality of subword units, wherein the first subword unit is associated with the first frame; a first Gaussian mixture model component of a plurality of Gaussian mixture model components associated with the first subword unit; a second subword unit of the plurality of subword units, wherein the second subword unit is associated with the second frame; and a second Gaussian mixture model component of the plurality of Gaussian mixture model components associated with the second subword unit; obtain a first speaker model associated with a first speaker and a second speaker model associated with a second speaker, wherein the first speaker model comprises distributions of likelihoods regarding likelihoods of the plurality of Gaussian mixture model components being associated with the plurality of subword units, wherein each of the distributions of likelihoods is associated with one of; select, from the distributions of likelihoods, a first distribution of likelihoods associated with the first subword unit; identify, in the first distribution of likelihoods, a first likelihood associated with the first Gaussian mixture model component, wherein the first likelihood represents a likelihood that the first frame corresponds to the first speaker; select, from the distributions of likelihoods, a second distribution of likelihoods associated with the second subword unit; identify, in the second distribution of likelihoods, a second likelihood associated with the second Gaussian mixture model component, wherein the second likelihood represents a likelihood that the second frame corresponds to the first speaker; determine a likelihood that the first frame corresponds to the second speaker from the second speaker model, the first subword unit, and the first Gaussian mixture model component; determine a likelihood that the second frame corresponds to the second speaker from the second speaker model, the second subword unit, and the second Gaussian mixture model component; determine a likelihood that the utterance corresponds to the first speaker from the first likelihood and the second likelihood; determine a likelihood that the utterance corresponds to the second speaker from: the likelihood that the first frame corresponds to the second speaker; and the likelihood that the second frame corresponds to the second speaker; identify the speaker of the utterance based at least on: the likelihood that the utterance corresponds to the first speaker; and the likelihood that the utterance corresponds to the second speaker; and generate a textual transcription of the utterance from a speaker model associated with the speaker of the utterance.