19. The computer-readable medium of claim 17 comprising: determining, by the first user device for a third user device, a frequency with which the third user device is located in a physical area near a physical location of the first user device; determining, by the first user device, whether the frequency satisfies a threshold frequency; and storing, by the first user device, a third speaker model specific to a third user of the third user device in the third speaker models in response to determining that the frequency satisfies the threshold frequency.
17. The computer-readable medium of claim 12 comprising: determining, by the first user device, one or more third speaker models, stored on the first user device, for other people who may be located in a physical area near a physical location of the first user device; and determining, by the first user device, that the utterance was spoken by the first user using the first speaker model that is specific to the first user of the first user device, the second speaker model that is specific to the second user associated with the second user device, and the third speaker models for other people who may be located in a physical area near a physical location of the first user device.