1. A method of recognizing an utterance comprising: receiving an utterance comprising a first portion having a first mode and a second portion having a second mode; generating, at a multimodal recognition system comprising a processor, a first mode recognition lattice and a first finite-state transducer based on the first mode; and generating a recognition model comprising: relating the first portion of the utterance to the second portion of the utterance using the first finite-state transducer; and generating a second finite-state transducer, the second finite-state transducer comprising a gesture and speech recognition model finite-state transducer, based on the first mode recognition lattice and the first finite-state transducer.