1. A method comprising: receiving, by a computing device, an acoustic input signal at a speech recognizer; identifying, by the computing device, a user that is speaking based on the acoustic input signal; determining, by the computing device, speaker-specific information previously stored for the user; determining, by the computing device, a set of classifications, wherein the set of classifications are determined based on the speaker-specific information; classifying, by the computing device, portions of the acoustic input signal into different classifications in the set of classifications; selecting, by the computing device, a classification in the set of classifications based on a criterion associated with the classification; determining, by the computing device, a set of responses based on the recognized acoustic input signal, the classification, and the speaker-specific information for the user; determining, by the computing device, if the response should be output; and outputting, by the computing device, the response if it is determined the response should be output, wherein classifying portions is performed in an always on mode, and wherein identifying the user that is speaking is performed after receiving a trigger phrase to activate the speech recognizer.