4. The method according to claim 3 , wherein the comparing includes: assigning scores to the grams based on the training data set, including assigning a composite score for one gram that is calculated based on neighboring grams, such that the composite score for the 1-gram is calculated based on scores assigned to the first, second and third tokens.
5. The method according to claim 4 , wherein the composite score for the 1-gram is an average of the scores assigned to the first, second and third tokens.