13. A computer program product comprising: a computer-readable storage comprising computer-usable program code stored thereon that processes text within a natural language understanding system, the computer-readable storage comprising: computer-usable program code that applies a first tokenization technique to a sentence using a statistical tokenization model; computer-usable program code that applies a second subsequent tokenization technique to the sentence using a named only entity only when the first tokenization technique does not extract a needed token according to a class of the sentence; and computer-usable program code that outputs a token determined according to at least one of the tokenization techniques.
13. A computer program product comprising: a computer-readable storage comprising computer-usable program code stored thereon that processes text within a natural language understanding system, the computer-readable storage comprising: computer-usable program code that applies a first tokenization technique to a sentence using a statistical tokenization model; computer-usable program code that applies a second subsequent tokenization technique to the sentence using a named only entity only when the first tokenization technique does not extract a needed token according to a class of the sentence; and computer-usable program code that outputs a token determined according to at least one of the tokenization techniques. 18. The computer program product of claim 13 , wherein the computer-readable storage further comprises computer-usable program code that applies a third tokenization technique to the sentence using a different statistical tokenization model when the second subsequent tokenization technique does not obtain the needed token according to the class of the sentence.