4. A system comprising: one or more computers; and computer-readable medium coupled to the one or more computers and having instructions stored thereon, which, when executed by the one or more computers, cause the one or more computers to perform operations comprising: receiving a sequence representing an utterance, the sequence comprising a plurality of audio frames; processing each audio frame in the sequence using a warping neural network, wherein the warping neural network is configured to process each of the audio frames to generate one or more warping factors for each audio frame in the sequence in accordance with trained values of a set of parameters of the warping neural network, and wherein the warping neural network has been trained on training data that associates each of a plurality of training audio frames with one or more corresponding warping factors; applying, for each audio frame, the one or more warping factors generated by the warping neural network for the audio frame to the audio frame to generate a respective modified audio frame, wherein the applying comprises using at least one of the warping factors to scale a respective frequency of the audio frame to a new respective frequency in the respective modified audio frame; and decoding the modified audio frames using a decoding neural network, wherein the decoding neural network is configured to output a word sequence that is a transcription of the utterance.
4. A system comprising: one or more computers; and computer-readable medium coupled to the one or more computers and having instructions stored thereon, which, when executed by the one or more computers, cause the one or more computers to perform operations comprising: receiving a sequence representing an utterance, the sequence comprising a plurality of audio frames; processing each audio frame in the sequence using a warping neural network, wherein the warping neural network is configured to process each of the audio frames to generate one or more warping factors for each audio frame in the sequence in accordance with trained values of a set of parameters of the warping neural network, and wherein the warping neural network has been trained on training data that associates each of a plurality of training audio frames with one or more corresponding warping factors; applying, for each audio frame, the one or more warping factors generated by the warping neural network for the audio frame to the audio frame to generate a respective modified audio frame, wherein the applying comprises using at least one of the warping factors to scale a respective frequency of the audio frame to a new respective frequency in the respective modified audio frame; and decoding the modified audio frames using a decoding neural network, wherein the decoding neural network is configured to output a word sequence that is a transcription of the utterance. 6. The system of claim 4 , wherein the applying comprises: using a first warping factor in the one or more warping factors to scale frequency within a first frequency range of the audio frame; and using a second respective warping factor in the one or more warping factors to scale frequency within a second frequency range of the audio frame to generate the respective modified audio frame.