1. A computer-implemented method in a dialog system, comprising: receiving inputs from a plurality of input modalities within an input period, wherein the plurality of input modalities provides inputs regarding a plurality of dialog tasks, wherein each dialog task of the plurality of dialog tasks is associated with a specific duration within the input period, and wherein input from each of the plurality of modalities is valid input for a first dialog task when the input occurred within the specific duration associated with the first dialog task, the valid inputs from each modality for the first dialog task determined further by: ignoring inputs that are shorter than a determined minimum duration, and ignoring inputs that are associated with a dialog task of the plurality of dialog tasks different from the first task, and wherein the valid inputs for the first dialog task each have a discernable meaning; generating an initial hypothesis regarding an overall meaning of the valid inputs for the first dialog task based on the discernable meaning of one of the valid inputs for the first dialog task; determining if there is an error associated with the initial hypothesis using a confidence measure; receiving one or more constraints for the initial hypothesis based on one or more other valid inputs for the first dialog task if the initial hypothesis has an error, wherein the constraints are based on the discernable meanings provided by the one or more other valid inputs; combining the one or more constraints with the initial hypothesis; deriving a new hypothesis regarding an overall meaning of the valid inputs for the first dialog task; and using the new hypothesis to interpret the valid inputs regarding the first dialog task, wherein the dialog system provides the interpretation to a response generator within the dialog system to generate a system output to complete the first dialog task.