1. A computer-implemented method comprising: extracting a first set of features from visual content of a particular image; accessing, for one or more image pairs in a plurality of images, a visual similarity score representing a level of similarity between visual features of images in each of the one or more pairs; identifying, a first set of features from visual content for a particular image from the plurality of images and by one or more processors, a set of nearest neighbor images based on the visual similarity scores of the one or more image pairs that include the particular image and identifying one or more nearest neighbor images based on a distance measure between the first set of features and a second set of features of visual content of the one or more nearest neighbor images; accessing, for each of one or more nearest neighbor images in the set of nearest neighbor images, historical click data specifying a count of user interactions with the nearest neighbor image when the nearest neighbor image was presented as a search result responsive to a query, and wherein a presentation time of the nearest neighbor image after the user interaction exceeded a pre-determined threshold; and determining, for the particular image and by one or more processors, a score based on the historical click data as weighted by the visual similarity to the particular image.