1. A method for evolving a gesture set in a gesture-based system, the method comprising: capturing data representative of a user in a physical space; processing the captured data to identify a first motion or pose for the user, the first motion or pose for the user being processed to determine a first confidence level that the first motion or pose is the same as a computer representation of a motion or pose for a particular gesture that invokes an input command to modify a visual output of a computer, wherein the first motion or pose is processed to generate values for parameters associated with the particular gesture in a gesture filter, said first confidence level indicating a likelihood that the values of the parameters associated with the first motion or pose correspond to the values of the parameters of the particular gesture so as to cause the first motion or pose to be recognized as the particular gesture by the gesture filter to, in turn, invoke the input command to modify the visual output of the computer; and modifying the computer representation of the motion or pose invoking the input command based on the first motion or pose by the user such that subsequently captured data of the user or another user performing the first motion or pose is processed to determine a second confidence level that the first motion or pose is the same as the modified computer representation of the motion or pose for the particular gesture, said second confidence level indicating a likelihood that the values of the parameters associated with the subsequently captured first motion or pose correspond to the values of the parameters of the particular gesture so as to cause the subsequently captured first motion or pose to be recognized as the particular gesture by the gesture filter to, in turn, invoke the input command to modify the visual output of the computer, the second confidence level being greater than the first confidence level.