Cognitive Principles in Robust Multimodal Interpretation
  Multimodal conversational interfaces provide a natural means for users to
communicate with computer systems through multiple modalities such as speech
and gesture. To build effective multimodal interfaces, automated interpretation
of user multimodal inputs is important. Inspired by the previous investigation
on cognitive status in multimodal human machine interaction, we have developed
a greedy algorithm for interpreting user referring expressions (i.e.,
multimodal reference resolution). This algorithm incorporates the cognitive
principles of Conversational Implicature and Givenness Hierarchy and applies
constraints from various sources (e.g., temporal, semantic, and contextual) to
resolve references. Our empirical results have shown the advantage of this
algorithm in efficiently resolving a variety of user references. Because of its
simplicity and generality, this approach has the potential to improve the
robustness of multimodal input interpretation.
