1. A system comprising: a non-transitory computer-readable memory storing executable instructions; and one or more computer processors in communication with the computer-readable memory, wherein the one or more computer processors are programmed by the executable instructions to at least: obtain acoustic model training data comprising: an input feature vector for input into an acoustic model, wherein the acoustic model comprises a neural network, and wherein the input feature vector comprises measurements of energies in a portion of an input signal, the input signal representing audio of both an utterance and acoustic interference; a reference speech feature vector, the reference speech feature vector comprising measurements of energies in a portion of a reference speech signal, the reference speech signal representing audio of the utterance substantially isolated from audio of the acoustic interference; and a reference interference feature vector, the reference interference feature vector comprising measurements of energies in a portion of reference interference signal, the reference interference signal representing the audio of the acoustic interference substantially isolated from the audio of the utterance; generate, using the neural network and the input feature vector, a test output comprising: a main acoustic model output comprising, for individual subword units of a plurality of subword units, a corresponding probability that at least a portion of the input feature vector represents audio comprising the subword unit; a speech prediction feature vector comprising, for individual measurements of energies in the portion of reference speech signal, a corresponding prediction of the measurement of the energy in the portion of the reference speech signal, wherein the prediction of the measurement of the energy in the portion of the reference speech signal is separate from the main acoustic model output; and an interference prediction feature vector comprising, for individual measurements of energies in the portion of the reference interference signal, a corresponding prediction of the measurement of the energy in the portion of the reference interference signal, wherein the prediction of the measurement of the energy in the portion of the reference interference signal is separate from both the prediction of the measurement of the energy in the portion of the reference speech signal and the main acoustic model output; compute a gradient of a weighted sum of loss functions, the weighted sum of loss functions comprising a main loss function, a first side task loss function, and a second side task loss function, wherein the main loss function represents a difference between the main acoustic model output and expected acoustic model output for the input feature vector, wherein the first side task loss function represents a difference between the speech prediction feature vector and the reference speech feature vector, and wherein the second side task loss function represents a difference between the interference prediction feature vector and the reference interference feature vector; modify a value of a parameter of the neural network by an amount indicated by the gradient, wherein the gradient indicates a degree to which the weighted sum of loss functions changes with respect to a change in the parameter, and wherein the amount indicted by the gradient is based on a desired change in the weighted sum of loss functions; process, using the neural network with the value of the parameter that has been modified, a second input feature vector to generate a probability that at least a portion of the second input feature vector corresponds to a first subword unit of the plurality of subword units; and generate speech recognition results comprising the first subword unit based at least partly on the probability.