1. A computer-implemented method, comprising: accessing a plurality of images referenced by initial image search results responsive to a query image, each of the plurality of images having a visual similarity score that satisfies an initial similarity threshold, the visual similarity score being generated by an image similarity model that has been trained based on user similarity feedback from a plurality of users and feature distances between images referenced by the user similarity feedback, the user similarity feedback specifying a reference query image and a first of two training images that was identified, by the plurality of users, as more visually similar than a second of the two training images to the reference query image, each of the feature distances indicative of a visual distance between corresponding image feature scores for the reference query image and the first of the two training images; selecting, in a data processing system and as a result image for the query image, a first image from the plurality of images, the first image having a visual similarity score that satisfies a first similarity threshold, the first similarity threshold being more selective than the initial similarity threshold; determining that a second image from the plurality of images has a first degree of visual similarity relative to the query image; determining that the second image from the plurality of images has a second degree of visual similarity relative to the first image from the plurality of images; determining, based on the first degree of visual similarity and the second degree of visual similarity, that the second image satisfies a similarity condition specifying required degrees of visual similarity between the second image, the query image, and the first image; and selecting, in the data processing system and as another result image for the query image, the second image from the plurality of images in response to determining that the second image satisfies the similarity condition, wherein a result image is an image for which a reference to the image is identified as a final image search result for the query image.