1. A system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: accessing, by the one or more computers, a neural network that has been trained, using speech in each of multiple languages, to be able to provide prosody information for each of the multiple languages; providing, by the one or more computers, input to the neural network that includes (i) a representation of a text in a first language and (ii) a language identifier for the first language; generating, by the one or more computers, audio data for a synthesized utterance of the text in the first language based on prosody information for the text that is output by the neural network in response to receiving the representation of the text and the language identifier for the first language; and providing, by the one or more computers, the audio data for the synthesized utterance of the text in the first language.
8. The system of claim 1 , wherein generating, by the one or more computers, audio data for the synthesized utterance of the text in the first language comprises generating the audio data using the prosody information for the text that is output by the neural network and audio coefficients representing synthesized speech characteristics.