1. A method of processing a query in a multi-stage query processing system, comprising: performing first stage processing of the query, including: retrieving from an index, a list of token positions for one or more query terms in the query, wherein the token positions in the list specify positions of individual tokens that match the query terms; based on the list of token positions, producing a first set of document identifiers, and generating a first set of relevancy scores for a first set of documents corresponding to the first set of document identifiers; performing a second stage processing of the query, including: generating a second set of relevancy scores for the documents in the first set of documents based on one or more of: the list of token positions, distances between query terms in the documents, attributes of tokens in the documents, and a context in which a query term is used in a document of the first set of documents, and selecting a subset of the first set of documents based on the second set of relevancy scores; automatically generating one or more new query expansion terms based on documents in the subset of the first set of documents; and generating a new query expression based on the query and the one or more new query expansion terms for processing by at least the first processing stage to produce at least a second set of document identifiers corresponding to a second set of documents.
1. A method of processing a query in a multi-stage query processing system, comprising: performing first stage processing of the query, including: retrieving from an index, a list of token positions for one or more query terms in the query, wherein the token positions in the list specify positions of individual tokens that match the query terms; based on the list of token positions, producing a first set of document identifiers, and generating a first set of relevancy scores for a first set of documents corresponding to the first set of document identifiers; performing a second stage processing of the query, including: generating a second set of relevancy scores for the documents in the first set of documents based on one or more of: the list of token positions, distances between query terms in the documents, attributes of tokens in the documents, and a context in which a query term is used in a document of the first set of documents, and selecting a subset of the first set of documents based on the second set of relevancy scores; automatically generating one or more new query expansion terms based on documents in the subset of the first set of documents; and generating a new query expression based on the query and the one or more new query expansion terms for processing by at least the first processing stage to produce at least a second set of document identifiers corresponding to a second set of documents. 5. The method of claim 1 , further comprising: during the second stage processing of the query, generating snippets for one or more documents in the subset of the first set of documents; and automatically generating one or more new query expansion terms based on the snippets.