1. A method for computerized processing of document data, comprising: receiving the document data; retrieving tokenization rules for the document data; and applying the tokenization rules to the document data to generate a plurality of tokens, each having one or more concordable characters from the document data.