1. A method performed by data processing apparatus, the method comprising: receiving, by a data processing apparatus, a query determined to be a question query and a corresponding answer generated in response to the question query, the answer determined to be responsive to the question query, wherein each of the question query and the answer has one or more terms; generating, by the data processing apparatus, a contextual query that includes an element relating to the question query and an element relating to the answer; submitting, by the data processing apparatus, the contextual query to a rich content search process and receiving data specifying a first set of rich content item responsive to the contextual query, the rich content items ranked according to an order that is indicative of the relevance of each rich content item to a query for which the set was generated, the submitting comprising submitting each of the question query, the answer, and the contextual query to the rich content item search process and receiving data specifying the first set of rich content items responsive to the contextual query, a second set of rich content items responsive to the question query, and a third set of rich content items responsive to the answer, wherein each set of rich content items differs from each other respective set of rich content items, and the rich content items in each set of rich content items are ranked according to an order that is indicative of the relevance of each rich content item to a query for which the set was generated; determining, by the data processing apparatus, first rich content items in the first set of rich content items that meet a context condition that is indicative of a rich content item providing contextual information of both elements of the question query and the answer, query, the determining first rich content items comprising determining from the data specifying the first, second and third sets of rich content items, first rich content items in the first set that that are not included in both of the second set and the third set and second rich content items in the first set that are included in at least one of the second set or third set; determining, by the data processing apparatus, second rich content items in the first set of rich content items that do not meet the context condition; and selecting rich content items from one or more of the first rich content items and the second rich content items, wherein the selected one or more rich content items are provided as one or more answer rich content items; wherein the rich content items are non-textual content.