1. A method comprising: identifying a set of training images previously used to train an image relevance model that generates relevance measures of images to query based on content feature values of the images, the query being a unique set of one or more query terms; determining that at least a given portion of the training images from the set of training images that were used to train the image relevance model each include at least a specified number of matching visual features, and in response to determining that at least a given portion of the training images each include at least the specified number of matching visual features: identifying, as a subset of the set of training images, training images having at least a threshold portion of the specified number of matching visual features; assigning, to each training image in the subset of the training images, higher weights that exceed a weight assigned to other images from the set of training images that are not in the subset of training images; and generating a re-trained image relevance model based on visual features of the set of training images and the higher weights assigned to the training images in the subset of training images that have at least the partial portion of the specified number of matching visual features.