1. A method for processing input received by a computing device comprising one or more processors, the method comprising: decoding input from a first input modality to produce posterior probabilities for words along paths in a common word graph; recording a decoding front for each of a plurality of possible input modalities, each decoding front comprising a set of nodes in the common word graph that define an end of a last word along a path in the common word graph that was assigned a probability by decoding an input from the respective input modality; receiving input from a second input modality after recording the decoding front for the second input modality; using the nodes of the recorded decoding front for the second input modality to determine where in the common word graph to begin rescoring and pruning the common word graph based on the input from the second input modality; using one or more of the processors, rescoring and pruning the common word graph based on averaging posterior probabilities from decoding input from the first input modality and decoding input from the second input modality; and outputting a hypothesis for the input based on the common word graph.