1. A method for training a deep neural network that comprises a low rank hidden input layer with m nodes and an adjoining hidden layer with o nodes, the low rank hidden input layer comprising a first matrix A and a second matrix B with dimensions i×m and m×o, respectively, to identify a key phrase, the method comprising: receiving, by a speech recognition system that includes the deep neural network trained to identify the key phrase, a feature vector comprising i values that represent features of an audio signal encoding an utterance; determining, by the speech recognition system, an output vector comprising o values using the m nodes in the low rank hidden input layer by combining the feature vector, the first matrix A in the low rank hidden input layer, and the second matrix B included in the low rank hidden input layer using a linear function, wherein m is a smaller number than o; determining, by the speech recognition system using a non-linear function for the adjoining hidden layer that has the o nodes, another vector using the output vector that comprises the o values; determining, by the speech recognition system, a confidence score that indicates whether the utterance includes the key phrase using the other vector; adjusting, by the speech recognition system, one or more weights for the low rank hidden input layer based on an accuracy of the confidence score; and providing, by the speech recognition system, the deep neural network with the adjusted one or more weights for use in processing audio.