11. A system, comprising: a data processing apparatus; and a data store storing instructions that, when executed by the data processing apparatus, cause the data processing apparatus to perform operations comprising: identifying a plurality of image search results that are responsive to a search query; determining that a relevance score for a particular image search result from the plurality of image search results meets a specified relevance score threshold, wherein the relevance score is determined independent of visual similarity between a particular image referenced by the particular image search result and other images that are referenced by the plurality of image search results; determining that image similarity data for the particular image indicates that the particular image meets a threshold level of visual similarity to at least a threshold number of the other images that are referenced by the plurality of image search results; determining that the particular image search result is a co-relevant image search result for the search query, the determination being performed in response to determining that both the relevance score meets the specified relevance score threshold, and the image similarity data for the particular image indicates that the particular image meets the threshold level of visual similarity to at least the threshold number of the other images; and generating an image rank score for the particular image search result based on the relevance score and an amplification factor for image search results that are determined to be co-relevant, wherein the amplification factor is used to increase or decrease the image rank score.
11. A system, comprising: a data processing apparatus; and a data store storing instructions that, when executed by the data processing apparatus, cause the data processing apparatus to perform operations comprising: identifying a plurality of image search results that are responsive to a search query; determining that a relevance score for a particular image search result from the plurality of image search results meets a specified relevance score threshold, wherein the relevance score is determined independent of visual similarity between a particular image referenced by the particular image search result and other images that are referenced by the plurality of image search results; determining that image similarity data for the particular image indicates that the particular image meets a threshold level of visual similarity to at least a threshold number of the other images that are referenced by the plurality of image search results; determining that the particular image search result is a co-relevant image search result for the search query, the determination being performed in response to determining that both the relevance score meets the specified relevance score threshold, and the image similarity data for the particular image indicates that the particular image meets the threshold level of visual similarity to at least the threshold number of the other images; and generating an image rank score for the particular image search result based on the relevance score and an amplification factor for image search results that are determined to be co-relevant, wherein the amplification factor is used to increase or decrease the image rank score. 13. The system of claim 11 , wherein the operations further comprise: determining that a second image search result, from the plurality of image search results that are responsive to the search query, is a co-irrelevant result, the determination being based on both: a relevance score for the second image search result meeting a specified irrelevance score threshold; and image similarity data for a second image referenced by the second image search result indicating that the second image meets a lower threshold level of visual similarity to at least a second threshold number of the other images that are referenced by the plurality of image search results; and generating an image rank score for the second image search result based on the relevance score and an inverse amplification factor for image search results that are determined to be co-irrelevant.