1. A computer-implemented method, comprising: obtaining one or more query labels associated with a query image; selecting, in a data processing system, candidate images matching the one or more query labels; generating, in the data processing system, visual similarity scores for the candidate images, each of the visual similarity scores representing a visual similarity of a respective candidate image to the query image, the visual similarity scores being generated by an image similarity model that has been trained based on user similarity feedback and feature distances between images referenced by the user similarity feedback, the user similarity feedback specifying a reference query image and a first of two training images that was identified, by users, as more visually similar than a second of the two training images to the reference query image, each of the feature distances indicative of a visual distance between corresponding image feature scores for the reference query image and the first of the two training images; generating, in the data processing system, relevance scores for the candidate images based at least in part on the visual similarity scores, each relevance score representing a relevance measure of a respective candidate image to the query image; generating a ranking of the candidate images; selecting a highest ranking subset of the candidate images to be referenced by image search results; determining that a particular candidate image in the highest ranking subset of the candidate images does not have at least a threshold number of nearest neighbors in common with the query image, a nearest neighbor for the particular candidate image being another image having a visual similarity score that satisfies a nearest neighbor threshold, the nearest neighbor threshold being a value indicative of a pre-specified visual distance from the particular candidate image; and in response to determining that the particular candidate image does not have at least the threshold number of nearest neighbors in common with the query image, omitting the particular candidate image from the highest ranking subset of the candidate images that are referenced by the image search results.