1. A method of processing text within a natural language understanding system, the method comprising: via a processor, applying a first tokenization technique to a sentence using a statistical tokenization model; via the processor, applying a second subsequent tokenization technique to the sentence using a named entity only when the first tokenization technique does not extract a needed token according to a class of the sentence; and via the processor, outputting a token determined according to at least one of the tokenization techniques.
1. A method of processing text within a natural language understanding system, the method comprising: via a processor, applying a first tokenization technique to a sentence using a statistical tokenization model; via the processor, applying a second subsequent tokenization technique to the sentence using a named entity only when the first tokenization technique does not extract a needed token according to a class of the sentence; and via the processor, outputting a token determined according to at least one of the tokenization techniques. 6. The method of claim 1 , further comprising: via the processor, determining whether at least one token is needed according to the class and a tokenization result of the first tokenization technique; and when no further tokens are needed, via the processor, discontinuing processing of the sentence and outputting at least one token determined for the sentence.