13. A non-transitory computer-readable medium for training a deep neural network that comprises a low rank hidden input layer with m nodes and an adjoining hidden layer with o nodes, the low rank hidden input layer comprising a first matrix A and a second matrix B with dimensions i×m and m×o, respectively, to identify a key phrase, the computer-readable medium storing software comprising instructions executable by a speech recognition system that includes one or more computers which, upon such execution, cause the one or more computers to perform operations comprising: receiving, by the speech recognition system that includes the deep neural network trained to identify the key phrase, a feature vector comprising i values that represent features of an audio signal encoding an utterance; determining an output vector comprising o values using the m nodes in the low rank hidden input layer by combining the feature vector, the first matrix A in the low rank hidden input layer, and the second matrix B included in the low rank hidden input layer using a linear function, wherein m is a smaller number than o; determining, using a non-linear function for the adjoining hidden layer that has the o nodes, another vector using the output vector that comprises the o values; determining a confidence score that indicates whether the utterance includes the key phrase using the other vector; adjusting one or more weights for the low rank hidden input layer based on an accuracy of the confidence score; and providing, by the speech recognition system, the deep neural network with the adjusted one or more weights for use in processing audio.
13. A non-transitory computer-readable medium for training a deep neural network that comprises a low rank hidden input layer with m nodes and an adjoining hidden layer with o nodes, the low rank hidden input layer comprising a first matrix A and a second matrix B with dimensions i×m and m×o, respectively, to identify a key phrase, the computer-readable medium storing software comprising instructions executable by a speech recognition system that includes one or more computers which, upon such execution, cause the one or more computers to perform operations comprising: receiving, by the speech recognition system that includes the deep neural network trained to identify the key phrase, a feature vector comprising i values that represent features of an audio signal encoding an utterance; determining an output vector comprising o values using the m nodes in the low rank hidden input layer by combining the feature vector, the first matrix A in the low rank hidden input layer, and the second matrix B included in the low rank hidden input layer using a linear function, wherein m is a smaller number than o; determining, using a non-linear function for the adjoining hidden layer that has the o nodes, another vector using the output vector that comprises the o values; determining a confidence score that indicates whether the utterance includes the key phrase using the other vector; adjusting one or more weights for the low rank hidden input layer based on an accuracy of the confidence score; and providing, by the speech recognition system, the deep neural network with the adjusted one or more weights for use in processing audio. 14. The computer-readable medium of claim 13 wherein adjusting the one or more weights for the low rank hidden input layer based on the accuracy of the confidence score comprises adjusting a greater quantity of the weights in the low rank hidden input layer when the accuracy does not satisfy a threshold accuracy than a smaller quantity of the weights that would be adjusted when the accuracy satisfies the threshold accuracy.