9. The method of claim 1 comprising: determining, by the first user device, one or more third speaker models, associated with the first user device, for other people who may be located in a physical area near a physical location of the first user device; and determining, by the first user device, that the utterance was likely spoken by the first user based at least on (i) the first score that indicates a likelihood that the utterance was spoken by the first user of the first user device, (ii) the second score that indicates a likelihood that the utterance was spoken by the second user that is associated with the second user device, and (iii) the third speaker models for other people who may be located in a physical area near a physical location of the first user device.
9. The method of claim 1 comprising: determining, by the first user device, one or more third speaker models, associated with the first user device, for other people who may be located in a physical area near a physical location of the first user device; and determining, by the first user device, that the utterance was likely spoken by the first user based at least on (i) the first score that indicates a likelihood that the utterance was spoken by the first user of the first user device, (ii) the second score that indicates a likelihood that the utterance was spoken by the second user that is associated with the second user device, and (iii) the third speaker models for other people who may be located in a physical area near a physical location of the first user device. 12. The method of claim 9 comprising: determining, by the first user device for a third user device, a frequency with which the third user device is located in a physical area near a physical location of the first user device; determining, by the first user device, whether the frequency satisfies a threshold frequency; and associating, by the first user device, a third speaker model specific to a third user of the third user device with the first user device in response to determining that the frequency satisfies the threshold frequency.