1. A method for tracking a location of an object in a sequence of images, the method comprising the steps of: representing the object in a current image of the sequence of images with an object template which comprises an image patch window covering an object region, wherein the sequence of images comprises previous, current, and subsequent images; representing the current and previous object templates in a dictionary of object templates; representing a candidate region in the subsequent image with one or more of a plurality of candidate windows corresponding to regions in the subsequent image on and around the location of the object in the current image; representing the background of the current image with a dictionary of background templates comprised of a plurality of windows located around the object template; computing a distance between the candidate windows and the object templates, using a learned distance metric based on the characteristics of the object templates and background templates; finding a winner candidate window that has the minimum learned distance from at least one of the object templates and selecting the winner candidate window location as the location of the object in the subsequent image; and updating the dictionary of object templates by incorporating the winner candidate.