1. A method of updating speech recognition neural networks, the method comprising: receiving a first audio signal comprising a first speech utterance; performing speech recognition on the first audio signal based at least in part on an acoustic model neural network to obtain a lattice of speech recognition results, wherein the lattice comprises a first path associated with a first score a second path associate with a second score; updating first weights of the acoustic model neural network substantially in real time, wherein updating the first weights comprises performing a first update using information associated with the first path and performing a second update using information associated with the second path; receiving a second audio signal comprising a second speech utterance; and performing speech recognition on the second audio signal based at least in part on the acoustic model neural network and the updated first weights.