18. The system of claim 17 , wherein the recurrent neural network comprises: a first long short-term memory layer that receives both first and second channels of audio; and a second long short-term memory layer corresponding to the first channel and a third long short-term memory layer corresponding to the second channel, the second long short-term memory layer and the third long short-term memory layer each receiving the output of the first long short-term memory layer and providing a set of filter parameters for the corresponding channel.
17. A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: receiving a first channel of audio data corresponding to an utterance and a second channel of audio data corresponding to the utterance; generating, using a trained recurrent neural network, (i) a first set of filter parameters for a first filter based on the first channel of audio data and the second channel of audio data and (ii) a second set of filter parameters for a second filter based on the first channel of audio data and the second channel of audio data; generating a single combined channel of audio data by combining (i) audio data of the first channel that has been filtered using the first filter and (ii) audio data of the second channel that has been filtered using the second filter; inputting the audio data for the single combined channel to a neural network trained as an acoustic model; and providing a transcription for the utterance that is determined based at least on output that the neural network provides in response to receiving the audio data for the single combined channel.