1. A method comprising: determining that tokens have been incorrectly separated due to an incorrect Natural Language Processing (NLP) operation on a corpus; and recombining, as a part of correcting the incorrect NLP operation, the incorrectly separated tokens, the recombining comprising: determining whether a token from an ordered set of tokens is present in a dictionary, the dictionary being related to the corpus from which the ordered set of tokens is extracted; determining whether a compounding threshold has been reached, wherein the compounding threshold limits a number of tokens that can be agglutinated to form a compound token; agglutinating using a processor and a memory, responsive to the token not being present in the dictionary, and responsive to the compounding threshold not having been reached, the token with a next adjacent token in the ordered set of tokens to form the compound token; determining whether the compound token is present in the dictionary; assigning a weight to the compound token responsive to the compound token being present in the dictionary; computing a confidence rating of the compound token, the confidence rating being a function of the weight; and using the compound token and the confidence rating in performing an NLP operation on the corpus.
1. A method comprising: determining that tokens have been incorrectly separated due to an incorrect Natural Language Processing (NLP) operation on a corpus; and recombining, as a part of correcting the incorrect NLP operation, the incorrectly separated tokens, the recombining comprising: determining whether a token from an ordered set of tokens is present in a dictionary, the dictionary being related to the corpus from which the ordered set of tokens is extracted; determining whether a compounding threshold has been reached, wherein the compounding threshold limits a number of tokens that can be agglutinated to form a compound token; agglutinating using a processor and a memory, responsive to the token not being present in the dictionary, and responsive to the compounding threshold not having been reached, the token with a next adjacent token in the ordered set of tokens to form the compound token; determining whether the compound token is present in the dictionary; assigning a weight to the compound token responsive to the compound token being present in the dictionary; computing a confidence rating of the compound token, the confidence rating being a function of the weight; and using the compound token and the confidence rating in performing an NLP operation on the corpus. 6. The method of claim 1 , further comprising: producing a set of compound tokens, the compound token being a member of the set of compound tokens; reducing the set of compound tokens by eliminating a subset of compound tokens from the set of compound tokens, the subset of compound tokens comprising those confidence rated compound tokens which have a contextual relevance below a level of contextual relevance in a portion of the corpus; and weighting a remaining subset of compound tokens using one of (i) a length of the compound token and (ii) a frequency of occurrence of the compound token in the portion of the corpus.