1. A computer program product for recognizing intended user actions, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising: receiving, by a microphone and camera, user utterances indicative of user commands and associated user gestures for the user utterances; parsing, by a hardware-based recognizer, sample utterances and the user utterances into verb parts and noun parts; recognizing, by a hardware-based recognizer, the user utterances and the associated user gestures based on the sample utterances and descriptions of associated supporting gestures for the sample utterances, said recognizing step comprising sequentially comparing each of the verb parts and each of the noun parts from the user utterances both individually and as pairs to the verb parts and the noun parts of the sample utterances; tracking words and word pairs used in conjunction with one or more recognized gestures, and determining a frequency of accepted and rejected system actions based on the tracking; and selectively performing a given one of the user commands responsive to a recognition result.