How to normalize technical skills of IT using machine learning? I have a huge collection of skills collected/scraped from various online sources. It was huge effort done by our team.
Now, the biggest challenge we are facing is trying to normalize the skills back to its root form to reduce the duplication.
Here's are few examples,
agile    (or) agile methodologies  (or) agile software
java 2.x (or) java 3.x (or) java

These kind of duplicates are very common is my datasets. So I'm looking for a way to normalize them.
I am not very sure of the way to solve this issue. Can anyone suggest me some very good ways we can solve this problem with some decent accuracy?
Thank you
In a first iteration, use a sentence encoder.You can find pre-trained model on tensorflowhub (https://www.tensorflow.org/hub/tutorials/semantic_similarity_with_tf_hub_universal_encoder), spacy (https://spacy.io/universe/project/spacy-universal-sentence-encoder) or huggingface (https://huggingface.co/sentence-transformers/bert-base-nli-mean-tokens) to name a few API.
Regex, Levenshtein distance, and other old school metrics are great, but in general you are never certain as to what you want to consider &quot;as similar&quot; besides the fact they words have a few common concepts (such as agile methodologies vs scrum vs kanban).
However, if you have a good pretrained encoder, this similarities will be captured.
In the second and next iteration, you can now use the encoder to check if similarities between skills are flagged properly, using the for example cosine distance. You can then automatically label the similarities and eventually correct the labels. Once you have sufficient label, you can now retrain the encoders on your own (smaller) dataset.