18. The computer-implemented method of claim 16 , wherein the document is tokenized using adaptive tokenization, and wherein the tokens for the documents in the document corpus are generated using adaptive tokenization.
18. The computer-implemented method of claim 16 , wherein the document is tokenized using adaptive tokenization, and wherein the tokens for the documents in the document corpus are generated using adaptive tokenization. 19. The computer-implemented method of claim 18 , wherein adaptive tokenization comprises: selecting one or more tokenization techniques based upon attributes of a document; and applying the selected tokenization techniques to the attributes of the document to identify the tokens contained in the document.