1. A method for generating a specific object detector comprising: with a generic detection model trained to identify objects in an object category, identifying a plurality of seed objects, each seed object being identified in a frame of a video sequence; for a plurality of iterations, for each of the plurality of seed objects: with a current motion model, identifying an object region in a different frame of the video sequence where the seed object is predicted to be located, and with a current appearance model, labeling sub-images of the different frame as positive or negative, and identifying as hard negatives, sub-images outside the object region that the current appearance model labels as positive for the object category; and with a processor, jointly updating the current appearance models for the plurality of seed objects to optimize a loss function which includes a regularization term that is a function of a distance between the current appearance models; and forming the specific object detector, comprising aggregating the appearance models of at least one of the iterations.