1. A computer-implemented method of improving a search engine's ability to output results for a given query by generating an enhanced token stream used to index a document stored in a knowledge base, the computer-implemented method being performed by one or more processors executing computer executable instructions for the computer-implemented method, and the computer-implemented method comprising: receiving at a knowledge discovery module a document; wherein when the received document is not in a plain text format, preprocessing the received document by removing article tags and other extraneous document data to format the received document in a plain text format; once the received document is formatted in a plain text format, inputting the received document to an analyzer module that processes the received document to generate an enhanced token stream for the received document, by performing the following: at a tokenizer, converting the plain text into smaller atomic units that represent tokens corresponding to words or phrases and wherein the tokens form a token stream; enhancing the token stream by applying a plurality of filters to the token stream, wherein the plurality of filters comprise: (i) one or more canonicalization filters that perform at least one of the following: expand one or more camel-case words into corresponding constituent words added to the token stream; identify for one or more tokens a value type which is added to the token stream; and generate and add to the token stream a lower case version of an identified token; (ii) a truncation filter that removes from the token stream one or more frequently occurring common words represented by one or more tokens; (iii) an expansion filter that performs the following: identification of words in the token stream that are related to each other and are identified as at least two distinct tokens in the token stream, wherein identifying words in the token stream that are related to each other comprises: determining if a token in the token stream exists as an entity in a named entity list; determining if a next token in the token stream exists as a connected entity in the named entity list when the token is determined to exist in the named entity list; repeating the steps until the next token does not exist as a connected entity in the named entity list; and combining the at least two distinct tokens into a single token in the token stream; (iv) a normalization filter that returns corresponding words for one or more tokens to a base form; indexing the document with the enhanced token stream in a knowledge database; and using the indexed document to provide a response to a given query.
3. The computer-implemented method of claim 1 wherein one of the one or more canonicalization filters comprises a lowercase filter that converts all uppercase letters in the tokens to lowercase letters.