1. A computer-implemented method, comprising: accessing a reference text label that specifies a first topic of a reference image; determining, in a data processing system, that a user interacted with an image presented in image search results as responsive to a particular query and during a same user session also interacted with the reference image also presented in the image search results as also responsive to the particular query; generating, in the data processing system, a measure of relevance of the image to the first topic based on a number of user sessions in which image search results included both the image and the reference image as responsive to a query and in which a user interacted with both of the reference image and the image; and labeling the image as an image that is relevant to the topic based on the measure of relevance meeting a specified threshold.