10. The non-transitory computer-readable storage medium of claim 7 , wherein the documents in the document corpus comprise one or more attributes, and wherein tokenize the documents in the document corpus to identify tokens contained in the documents comprises: selecting one or more tokenization techniques based upon the attributes; and applying the selected tokenization techniques to the attributes of the documents to identify the tokens contained in the documents.
10. The non-transitory computer-readable storage medium of claim 7 , wherein the documents in the document corpus comprise one or more attributes, and wherein tokenize the documents in the document corpus to identify tokens contained in the documents comprises: selecting one or more tokenization techniques based upon the attributes; and applying the selected tokenization techniques to the attributes of the documents to identify the tokens contained in the documents. 13. The non-transitory computer-readable storage medium of claim 10 , wherein at least one of the one or more tokenization techniques is configured to include an attribute name in the tokens for an attribute.