25. The computer program product of claim 19 , the operations further comprising: identifying lesser order n-grams, the lesser order n-grams being derived from the first n-gram; and identifying a respective scaled probability of each of the lesser order n-grams identifying a word, the scaled probability for a particular lesser order n-gram composed of a particular number of atomic units identifying a word being determined by adjusting an initial probability of the lesser order n-gram identifying a word based at least in part on the particular number of atomic units.
26. The computer program product of claim 25 , wherein segmenting the plurality of tokens into one or more words comprises: when a product of the respective scaled probabilities of the lesser order n-grams derived from the first n-gram identifying a word exceeds the scaled probability of the first n-gram identifying a word, segmenting the first n-gram such that each of the lesser order n-grams identifies a respective word.