10. A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: receiving, by a first user device, an audio signal encoding an utterance; obtaining, by the first user device, a first score that indicates a likelihood that the utterance was spoken by a first user of the first user device; obtaining, by the first user device for a second user of a corresponding second user device that is co-located with the first user device, a second score that indicates a respective likelihood that the utterance was spoken by the second user; determining, by the first user device, a combination of the first score and the second score; normalizing, by the first user device using the combination of the first score and the second score, the first score and the second score to reduce penalties to the first score and the second score caused by background noise in the audio signal; determining, by the first user device, that the utterance was spoken by the first user using the normalized first score and the normalized second score; analyzing the audio signal to identify a command included in the utterance in response to determining that the utterance was spoken by the first user; and performing an action that corresponds with the command.
11. The system of claim 10 comprising: determining that neither the first score nor the second score satisfy a threshold value and that a user who spoke the utterance cannot be determined, wherein normalizing the first score and the second score comprises normalizing the first score and the second score in response to determining that neither the first score nor the second score satisfy the threshold value; and determining that the normalized first score satisfies the threshold value, wherein determining that the utterance was spoken by the first user comprises determining that the utterance was spoken by the first user in response to determining that the normalized first score satisfies the threshold value.