6. A method for tokenizing text, the method comprising: receiving input text by a featurizer that transforms said input text into a plurality of token structures each comprising a token and attributes corresponding to said token, the token being a smallest meaningful unit of text; providing said plurality of token structures to a comparator that performs a comparison of each token structure of said plurality of token structures to at least one language model lexicon to determine if said at least one language model lexicon contains said token structure and, if so, adds said token structure to a candidate list associated with the token structure; providing the plurality of token structures to a classifier that determines, for each token structure of the plurality of token structures, if a classifier model from a list of available classifier models is suitable for the token structure and, if so, applying said token structure to said suitable classifier model to determine classification information and storing the determined classification information in the token structure; and providing the plurality of token structures to a finalizer that, for each token structure in the plurality of token structures: determines if said token structure includes classification information; converts the text associated with the respective token of the token structure according to the classification information if the token structure includes classification information and outputs a converted token structure with the converted text to an output token list; and selects a top candidate token structure from the candidate list associated with the token structure if the token structure does not include classification information and outputs said top candidate token structure to the output token list.
6. A method for tokenizing text, the method comprising: receiving input text by a featurizer that transforms said input text into a plurality of token structures each comprising a token and attributes corresponding to said token, the token being a smallest meaningful unit of text; providing said plurality of token structures to a comparator that performs a comparison of each token structure of said plurality of token structures to at least one language model lexicon to determine if said at least one language model lexicon contains said token structure and, if so, adds said token structure to a candidate list associated with the token structure; providing the plurality of token structures to a classifier that determines, for each token structure of the plurality of token structures, if a classifier model from a list of available classifier models is suitable for the token structure and, if so, applying said token structure to said suitable classifier model to determine classification information and storing the determined classification information in the token structure; and providing the plurality of token structures to a finalizer that, for each token structure in the plurality of token structures: determines if said token structure includes classification information; converts the text associated with the respective token of the token structure according to the classification information if the token structure includes classification information and outputs a converted token structure with the converted text to an output token list; and selects a top candidate token structure from the candidate list associated with the token structure if the token structure does not include classification information and outputs said top candidate token structure to the output token list. 9. The method according to claim 6 , where the token comprises a set of characters.