17. The method of claim 15 comprising: receiving the second speaker model for the second user from a server in response to determining that the subset of the third speaker models comprises the second speaker model for the second user.
15. A computer-implemented method comprising: receiving, by a first user device, an audio signal that is generated by the first user device and that encodes an utterance; obtaining, by the first user device, a first score that indicates a respective likelihood that the utterance was spoken by a first user of the first user device; determining, by the first user device for a second user of a corresponding second user device, that the second user device is co-located with the first user device; identifying, by the first user device, one or more third speaker models associated with the first user device; and determining, by the first user device, that a subset of the third speaker models comprises a second speaker model for the second user in response to (i) determining that the second user device is co-located with the first user device and (ii) receiving the audio signal that is generated by the first user device and that encodes the utterance; determining, by the first user device, that the utterance was spoken by the first user using the first score and the second speaker model; and performing an action that corresponds with a spoken command encoded in the audio signal in response to determining that the utterance was spoken by the first user.