1. A method comprising: receiving a search query comprising one or more query terms; identifying, based on the search query, a set of images that are responsive to the search query; identifying one or more image relevance models that output, based on visual features of images responsive to queries, relevance adjustment values for images identified as responsive to the search queries, each of the queries being a set of one or more query terms received by a search system as a query input, and each set of one or more query terms being different from each other set of one or more query terms; determining that an image relevance model for the received search query is not included in the one or more image relevance models; identifying, from the one or more image relevance models and in response to determining that the image relevance model for the received search query is not included in the one or more image relevance models, an image relevance model for a different query that has been identified as similar to the received search query, wherein the different query is different from the received search query; and determining, by one or more data processing apparatus, a degree of similarity between the search query and the different query based, at least in part, on a level of similarity between one or more words included in the search query and the different query; determining, for image search results responsive to the received search query that are evaluated using the image relevance model for the different query, a fractional adjustment value based on the degree of similarity between the different query and the received search query, wherein the fractional adjustment value for a higher degree of similarity is lower than the fractional adjustment value for a lower degree of similarity; obtaining, based on visual features of each image in the set of images responsive to the search query, an image relevance adjustment value for the image using the image relevance model for the different query; reducing, for each of the images in the set of images, the image relevance adjustment value based on the fractional adjustment value; determining, for each of one or more images in the set of images, a relevance score for the image relative to the received search query based, at least in part on the reduced image relevance adjustment value; and outputting, based on the relevance score for the image, image search results that identify the image as responsive to the search query.