6. The method of claim 1 , wherein calculating a score based on each word or phrase assigned to a tokenization path comprises: calculating a score based on a frequency associated with each word or phrase assigned to the tokenization path and a length associated with each word or phrase assigned to the tokenization path.
6. The method of claim 1 , wherein calculating a score based on each word or phrase assigned to a tokenization path comprises: calculating a score based on a frequency associated with each word or phrase assigned to the tokenization path and a length associated with each word or phrase assigned to the tokenization path. 7. The method of claim 6 , wherein calculating the score based on a frequency associated with each word or phrase assigned to the tokenization path and a length associated with each word or phrase assigned to the tokenization path comprises: calculating a value for each word or phrase assigned to the tokenization path in accordance with the formula: 
 log(frequency*10)+(length+frequency/10) 2 , wherein frequency represents a frequency associated with the word or phrase and length represents a length associated with the word or phrase, and summing the values calculated for each word or phrase assigned to the tokenization path.