1. A system for determining a location of a pointing device in three-dimensional space and a location to where the pointing device is directed, the system comprising: one or more infrared light emitting diodes (LEDs); a control system coupled to the one or more infrared LEDs, the control system capable of causing the one or more infrared LEDs to emit infrared light; one or more cameras, capable of capturing one or more frames at substantially the same time as the one or more infrared LEDs are caused to emit the infrared light, at least one of the one or more frames recording an existence of an emission of the infrared light; a processing system, capable of: determining the location of the pointing device in three-dimensional space; determining the location to where the pointing device is directed; computing a similarity between an input sequence of sensor values output by the pointing device and at least one stored prototype sequence, each stored prototype sequence representing a sequence of sensor values that is generated if the pointing device is used to perform a unique gesture; and determining if the computed similarity between the input sequence and any stored prototype sequence exceeds a prescribed similarity threshold; and a communication interface, capable of transmitting a message across a transmission medium to a receiving unit, the message including data that can be used to determine the location of the pointing device in three-dimensional space and the location to where the pointing device is directed, and data relating to gesture determination, which the receiving unit can use to manipulate an object displayed on a display screen.