1. A method performed by data processing apparatus, the method comprising: obtaining images from first image results responsive to a first query, wherein each of a plurality of the obtained images is associated with a score and user behavior data wherein the user behavior data represents interactions of users with the obtained image when the obtained image was presented as a search result for the first query; selecting a plurality of the obtained images each having respective user behavior data that satisfies a threshold; associating each of a plurality of the selected images with one or more annotations based on analysis of content of the selected image, the annotations derived from visual analysis of the images to identify particular visual features in the images, and each annotation specifying a presence of one of the particular visual features common among a sub-set of selected plurality of images; and providing the first query and the annotations to a plurality of different machine learning system generated classifiers to associate the first query with one or more categories wherein a respective plurality of the annotations is provided as input to each of the classifiers, and wherein at least one of the categories specifies a presence of one of the particular visual features in an image.