1. A method for maintaining speaker recognition performance, comprising: training a plurality of models respectively corresponding to speaker recognition scores from a plurality of speakers over a plurality of sessions; using the plurality of models to conclude whether a speaker seeking access to an environment is a non-ideal target speaker or a non-ideal non-target speaker, wherein using the plurality of models to conclude comprises: calculating a first probability that the speaker seeking access is the non-ideal target speaker; calculating a second probability that the speaker seeking access is the non-ideal non-target speaker; and determining whether the first probability, the second probability or a sum of the first probability and the second probability is above a probability threshold; and restricting the speaker seeking access from accessing the environment upon determining that the first probability, second probability or the sum of the first probability and the second probability is above the probability threshold, wherein the training, using and determining steps are performed by a computer system comprising a memory and at least one processor coupled to the memory.