1. A method performed by data processing apparatus, the method comprising: obtaining, for a set of images, image feature data specifying visual features of images in the set of images; selecting, from the set of images, subsets of unlabeled images that each include at least two unlabeled images; determining, for each subset of unlabeled images and based on the visual features of the at least two unlabeled images, a first measure of visual similarity between a first reference image and each of the at least two unlabeled images included in the subset, the first measure of visual similarity being independent of first similarity feedback data indicative of user specified visual similarity between the first reference image and each of the at least two unlabeled images; ranking the unlabeled images based on the first measure of visual similarity associated with each of the unlabeled images; generating a weighted feature vector based on the visual features of the unlabeled images and the ranking; identifying, from the set of images, a subset of labeled images that includes a second reference image and at least two labeled images; determining, by one or more computers and based on the visual features of the at least two labeled images and the weighted feature vector, a second measure of visual similarity between the second reference image and each of the at least two labeled images, the second measure of visual similarity being determined independent of second similarity feedback data indicative of user specified visual similarity between the second reference image and each of the at least two labeled images; ranking, by one or more computers, the at least two labeled images based on the second measure of visual similarity associated with each of the labeled images; determining, by one or more computers, that the ranking of the labeled images does not match a second ranking of the labeled images based on the second similarity feedback; and adjusting, by one or more computers, the weighted feature vector in response to the determination that the ranking does not match the second ranking and based on the second similarity feedback.