1. A control apparatus for providing a user with a control over an object displayed in a display apparatus by a pointing device, the control apparatus comprising: an image capturing module for recording a video sequence and generating N image frames, N being a positive integer; a separating module for capturing a pointing device image related to the pointing device from each image frame in accordance with a characteristic of the pointing device; a positioning module for calculating a specific point of the pointing device image of each image frame to generate a first set of specific point information; a constructing module for generating a trajectory in accordance with a pre-defined criterion and the first set of specific point information; and a processing module for analyzing the trajectory and generating a control signal to control the object; wherein, when the constructing module determines that generation of the trajectory is finished, the constructing module triggers a counter to execute a counting process; wherein, only after the counting process is finished, the trajectory is then transmitted to the processing module by the constructing module.