1. A method performed by data processing apparatus, the method comprising: identifying responsive images for a search phrase that includes two or more terms; determining, by one or more processors, interaction rankings for each of the responsive images based on a number of user interactions with the responsive image; creating, by one or more processors, two or more sub-queries based on the search phrase, the sub-queries each being a proper subset of the two or more terms; for each sub-query from the two or more sub-queries: determining, by one or more processors, sub-query model rankings for the responsive images based on a sub-query model for the sub-query and visual features of the responsive images, the sub-query model being an image relevance model for the sub-query; and determining, by one or more processors, a search phrase score for the sub-query model, the search phrase score being based on a measure of similarity between positions of the responsive images in each of the interaction rankings and the sub-query model rankings; and selecting, based on the search phrase scores for the sub-queries, one of the sub-query models as a model for the search phrase, the selected sub-query model having a search phrase score that meets a threshold search phrase score.