1. A method performed by data processing apparatus, the method comprising: obtaining images from first image results responsive to a first query, wherein each of a plurality of the obtained images is associated with a score and user behavior data wherein the user behavior data represents interactions of users with the obtained image when the obtained image was presented as a search result for the first query; selecting a plurality of the obtained images as a plurality of selected images, each image of the plurality of selected images having respective user behavior data that satisfies a threshold; associating each image of the plurality of the selected images with one or more annotations based on analysis of content of the selected image, the associating comprising: providing the plurality of the selected images to each of a plurality of different computer image annotators that each visually analyze an image to identify particular visual features in the images; receiving, from the plurality of computer image annotators, the annotations derived from visual analysis of the images to identify particular visual features in the images; providing the first query and the annotations to a plurality of different machine learning system generated classifiers to associate the first query with one or more categories wherein a respective plurality of the annotations is provided as input to each of the classifiers, and wherein at least one of the categories specifies a presence of one of the particular visual features in an image; receiving a second query, wherein the second query is the same or similar to the first query; obtaining second images responsive to the second query, wherein each of the second images is associated with a respective first rank so that the second images are arranged according to a first order; modifying the respective first rank of one or more of the second images based on one or more of the categories associated with the first query so that the second images are re-ordered according to a second order.