1. A method comprising: receiving an acoustic sequence, the acoustic sequence representing an utterance, and the acoustic sequence comprising a respective acoustic feature representation at each of a plurality of time steps; processing the acoustic feature representation at an initial time step using an acoustic modeling neural network to generate an output for the initial time step; for each subsequent time step of the plurality of time steps: receiving the acoustic representation for the time step, receiving an output generated by the acoustic modeling neural network for a preceding time step, generating a modified input from the output generated by the acoustic modeling neural network for the preceding time step and the acoustic representation for the time step, and processing the modified input using the acoustic modeling neural network to generate an output for the time step; and generating, for use in a speech recognition system, a phoneme representation for the utterance from the outputs for each of the time steps.