1. A method for object localization in an image comprising: for an input image, generating a task-dependent representation of the input image based on relevance scores for an object to be localized, comprising: for each of a plurality of patches of the image, generating a patch-based representation based on low level features extracted from the patch; with a classifier, outputting a relevance score for each patch, based on the respective patch-based representation; and generating the task-dependent representation based on the patch relevance scores; thereafter, identifying at least one similar image from a set of images, based on the task-dependent representation of the input image and task-dependent representations of images in the set of images; and identifying a location of the object in the input image based on an object location annotation for at least one of the at least one similar images identified in the set of images, wherein at least one of the generating of the task-dependent representation, identifying of the at least one similar image, and the identifying a location of the object in the input image is performed with a computer processor.