1. A computer-implemented method comprising: receiving, for each of multiple utterances of a hotword, a data set including at least (i) a speaker verification confidence score associated with the utterance, and (ii) environmental context data associated with the utterance; selecting from among the data sets, a subset of the data sets that are associated with a particular environmental context; selecting a particular data set from among the subset of data sets based on one or more selection criteria; selecting, as a speaker verification threshold for the particular environmental context, the speaker verification confidence score included in the particular data set; and providing the speaker verification threshold for use in performing speaker verification of utterances that are associated with the particular environmental context.