15. A non-transitory computer readable medium encoded with a computer program comprising instructions that when executed cause one or more data processing apparatus to perform operations comprising: accessing a reference text label that specifies a first topic of a reference image; determining that a user interacted with an image presented in image search results as responsive to a particular query and during a same user session also interacted with the reference image also presented in the image search results as also responsive to the particular query; generating a measure of relevance of the image to the first topic based on a number of user sessions in which image search results included both the image and the reference image as responsive to a query and in which a user interacted with both of the reference image and the image; and labeling the image as an image that is relevant to the topic based on the measure of relevance meeting a specified threshold.
15. A non-transitory computer readable medium encoded with a computer program comprising instructions that when executed cause one or more data processing apparatus to perform operations comprising: accessing a reference text label that specifies a first topic of a reference image; determining that a user interacted with an image presented in image search results as responsive to a particular query and during a same user session also interacted with the reference image also presented in the image search results as also responsive to the particular query; generating a measure of relevance of the image to the first topic based on a number of user sessions in which image search results included both the image and the reference image as responsive to a query and in which a user interacted with both of the reference image and the image; and labeling the image as an image that is relevant to the topic based on the measure of relevance meeting a specified threshold. 20. The computer readable medium of claim 15 , wherein the instructions cause the one or more data processing apparatus to perform operations further comprising: receiving image search results responsive to a query that is classified as belonging to a second topic that is different than the first topic; and filtering the image search results to remove references to images having labels that indicate that the images belong to the first topic.