1. A method performed by one or more computers, the method comprising: receiving, by the one or more computers, audio data that describes an utterance; processing the audio data using a neural network that has been trained as an acoustic model, wherein the processing comprises: providing, as input to the neural network, input vectors having values describing the utterance, the values including values representing audio waveform features, wherein the audio waveform features are determined using a filterbank having parameters trained jointly with weights of the neural network, wherein the neural network has first memory blocks for time information and second memory blocks for frequency information, the first memory blocks being different from the second memory blocks; wherein the first memory blocks are time-LSTM blocks that each have a state, and wherein the second memory blocks are frequency-LSTM blocks that each have a state and a corresponding frequency step in a sequence of multiple frequency steps, wherein the states are determined for each of a sequence of multiple time steps; wherein, for each of at least some of the frequency-LSTM blocks, the frequency-LSTM block determines its state using the state of the time-LSTM block corresponding to the same frequency step at the previous time step; and wherein, for each of at least some of the time-LSTM blocks, the time-LSTM block determines its state using the state of the frequency-LSTM block corresponding to the same time step and the previous frequency step; receiving, as output of the neural network, one or more scores that each indicate a likelihood that a respective phonetic unit represents a portion of the utterance; determining, by the one or more computers, a transcription for the utterance based on the one or more scores; and providing, by the one or more computers, the determined transcription as output of an automated speech recognizer.