1. A method of selecting data for an automated training process comprising: identifying a plurality of occurrences of a non-standard token in a text corpus stored in a memory; identifying a first plurality of tokens in the text corpus that are located proximate to at least one of the occurrences of the non-standard token; identifying a plurality of occurrences of a candidate standard token in the text corpus; identifying a second plurality of tokens in the text corpus that are located proximate to at least one of the occurrences of the candidate standard token; identifying a contextual similarity between the first plurality of tokens and the second plurality of tokens; generating a statistical model for correction of non-standard tokens with the non-standard token in association with the standard token for generation of a statistical model only in response to the identified contextual similarity being greater than a predetermined threshold; and storing the generated statistical model in the memory for use in identification of another standard token that corresponds to another non-standard token identified in text data that are not included in the text corpus.
1. A method of selecting data for an automated training process comprising: identifying a plurality of occurrences of a non-standard token in a text corpus stored in a memory; identifying a first plurality of tokens in the text corpus that are located proximate to at least one of the occurrences of the non-standard token; identifying a plurality of occurrences of a candidate standard token in the text corpus; identifying a second plurality of tokens in the text corpus that are located proximate to at least one of the occurrences of the candidate standard token; identifying a contextual similarity between the first plurality of tokens and the second plurality of tokens; generating a statistical model for correction of non-standard tokens with the non-standard token in association with the standard token for generation of a statistical model only in response to the identified contextual similarity being greater than a predetermined threshold; and storing the generated statistical model in the memory for use in identification of another standard token that corresponds to another non-standard token identified in text data that are not included in the text corpus. 5. The method of claim 1 , the generation of the statistical model further comprising: generating a Hidden Markov Model (HMM) with the non-standard token in association with the standard token only in response to the identified contextual similarity being greater than a predetermined threshold.