1. A method for detecting people by a surveillance system, the method comprising: identifying a plurality of foreground contour shapes of objects within a scene and a respective track for each of the plurality of foreground contour shapes from at least two frames in a video file; determining a plurality of features of the foreground contour shapes within the scene and the tracks of the foreground contour shapes, the features including a set of salient moving directions, and directional aspect ratios; assigning a respective intra-frame local detector to each location in the scene visited by the foreground contour shapes; and classifying at least a portion of each of the foreground contour shapes by using a feature of the foreground contour shape and the respective track of each foreground contour shape and applying the assigned respective intra-frame detector to the foreground contour shapes of the visited scene locations to determine whether each foreground contour shape matches a person reference model.