1. A method performed by one or more computers, the method comprising: receiving, by the one or more computers, audio data that describes an utterance; determining, by the one or more computers, a transcription for the utterance, comprising processing the audio data using a neural network that has been trained as an acoustic model, the neural network having first memory blocks for time information and second memory blocks for frequency information, the first memory blocks being different from the second memory blocks, wherein the first memory blocks are time-LSTM blocks that each have a state, and wherein the second memory blocks are frequency-LSTM blocks that each have a state and a corresponding frequency step in a sequence of multiple frequency steps, wherein the states are determined for each of a sequence of multiple time steps; wherein, for each of at least some of the frequency-LSTM blocks, the frequency-LSTM block determines its state using the state of the time-LSTM block corresponding to the same frequency step at the previous time step; and wherein, for each of at least some of the time-LSTM blocks, the time-LSTM block determines its state using the state of the frequency-LSTM block corresponding to the same time step and the previous frequency step; and providing, by the one or more computers, the transcription for the utterance as output of an automated speech recognizer.