23. A computer implemented method for indexing documents in a document collection, the method comprising: collecting possible phrases from documents in the document collection; classifying each possible phrase as either a good phrase or a bad phrase according to a frequency of occurrence of the possible phrase; determining, for a pair of good phrases g j and g k in the document collection, an information gain of g k with respect to g j as a function of an actual co-occurrence rate of g j and g k and an expected co-occurrence rate of g j and g k in the document collection; storing as valid phrases, on a computer-readable storage medium, only good phrases that predict the occurrence of at least one other good phrase in the document collection where a good phrase g j predicts the occurrence of another good phrase g k in the document collection when the determined information gain of g k in the presence of g j exceeds a first predetermined threshold; identifying, for a plurality of selectively retained valid phrases g x , a phrase g y as a related phrase of g x where the information gain of g y in the presence of g x exceeds a second predetermined threshold that is more restrictive than the first predetermined threshold; and indexing documents in the document collection with respect to the valid phrases and the related phrases.
23. A computer implemented method for indexing documents in a document collection, the method comprising: collecting possible phrases from documents in the document collection; classifying each possible phrase as either a good phrase or a bad phrase according to a frequency of occurrence of the possible phrase; determining, for a pair of good phrases g j and g k in the document collection, an information gain of g k with respect to g j as a function of an actual co-occurrence rate of g j and g k and an expected co-occurrence rate of g j and g k in the document collection; storing as valid phrases, on a computer-readable storage medium, only good phrases that predict the occurrence of at least one other good phrase in the document collection where a good phrase g j predicts the occurrence of another good phrase g k in the document collection when the determined information gain of g k in the presence of g j exceeds a first predetermined threshold; identifying, for a plurality of selectively retained valid phrases g x , a phrase g y as a related phrase of g x where the information gain of g y in the presence of g x exceeds a second predetermined threshold that is more restrictive than the first predetermined threshold; and indexing documents in the document collection with respect to the valid phrases and the related phrases. 25. The method of claim 23 , wherein collecting possible phrases comprises: maintaining for each possible phrase and each good phrase a frequency count of a number of documents containing the phrase; maintaining for each possible phrase and each good phrase a frequency count of a number of instances of the phrase; and maintaining for each possible phrases and each good phrase a frequency count of a number of distinguished instances of the phrase, a distinguished instance of the phrase comprising the phrase distinguished from neighboring content in the document by grammatical or format markers.