1. A system for projecting images and queries into a similarity space, comprising: a data store that provides queries and images as training data and relevance of the images to the queries; a relevance extraction component that, for each query, identifies a query relevance for the most relevant words to the query; and for each image, identifies an image relevance that is specific to the query for the most relevant words of the query from the relevance of the most relevant words to the image; and a learn projection component that learns a projection of the query relevances to a similarity space and a projection of the image relevances to the similarity space so that in the similarity space queries and their relevant images are close together and queries and their non-relevant images are not close together.