1. A method comprising: ranking, based on a first measure of visual similarity of each unlabeled image to a first reference image, unlabeled images in a set of images that includes at least two unlabeled images; generating a weighted feature vector based on visual features of the unlabeled images and the ranking; identifying, from the set of images, a subset of labeled images that includes a second reference image and at least two labeled images; determining, by one or more computers and based on the visual features of the at least two labeled images and the weighted feature vector, a second measure of visual similarity between the second reference image and each of the at least two labeled images, the second measure of visual similarity being determined independent of similarity feedback data indicative of user specified visual similarity between the second reference image and each of the at least two labeled images; ranking, by one or more computers, the at least two labeled images based on the second measure of visual similarity associated with each of the labeled images; determining, by one or more computers, that the ranking of the labeled images does not match a second ranking of the labeled images based on the similarity feedback data; and adjusting, by one or more computers, the weighted feature vector in response to the determination that the ranking does not match the second ranking and based on the similarity feedback data.