Background: I work for a major airline in the United States and have been given a pretty significant text mining and machine learning task. Since there are several unique terms, acronyms, and abbreviations in aviation, I have written a program that reads through the documents. The program is using 4 groups of terms; WORDS, STOPS, AIRPORTS, and SYNONYMS. WORDS, STOPS, and AIRPORTS are all sets where the first two were initialized from NLTK and the last was initialized with a manually built list of airports that my employer services. SYNONYMS is a nested dictionary 'trie' style structure where the branches are misspellings and the leaves are the terms that they represent. it was initialized as an empty dictionary. I have also created a command prompt UI that informs me when it finds a word that it doesn't know and allows me to correct the word or add it to one of the aforementioned groups. The documents are being tokenized by spaCy and fed into scikit-learn machine learning algorithms.Problem: I've started noticing that some of the words that I previously accepted in WORDS or SYNONYMS are showing up with alternate meanings and could have a drastic impact on how well the algorithm operates. For example the word act can be the verb act; or it can be an abbreviation for actuator or active; or it can be an acronym for auxiliary central tank or a couple other things. At present, I'm using the UI to manually correct or ignore these occurrences, which is time consuming, cumbersome, and prone to errors.Desired result: What I'm looking for is a method of determining whether 'lt' means left or light depending on context (lt wing vs. warning lt) with minimal manual entry. At a minimum I would like some high level ideas/discussion on how this may be accomplished in python. Optimally, I would like some example code, preferably using the same tools I already have; spaCy, NLTK, and sklearn but I am flexible and willing to learn.Notes: I've considered a couple methods of doing this but I haven't been able to come up with any ideas strong enough to try. For example, I've considered marking the terms with the pos inside the sets by concatenating it with the word before adding it like 'rat:noun'. The issue with this is that rat:noun could be an animal or it could be an abbreviation for ram air turbine, which is also a noun.
Determining usage/context of terms/acronyms/abbreviations in python
python;nlp;text mining;data cleaning