How to normalize technical skills of IT using machine learning?
In a first iteration, use a sentence encoder.You can find pre-trained model on tensorflowhub (https://www.tensorflow.org/hub/tutorials/semantic_similarity_with_tf_hub_universal_encoder), spacy (https://spacy.io/universe/project/spacy-universal-sentence-encoder) or huggingface (https://huggingface.co/sentence-transformers/bert-base-nli-mean-tokens) to name a few API.
Regex, Levenshtein distance, and other old school metrics are great, but in general you are never certain as to what you want to consider &quot;as similar&quot; besides the fact they words have a few common concepts (such as agile methodologies vs scrum vs kanban).
However, if you have a good pretrained encoder, this similarities will be captured.
In the second and next iteration, you can now use the encoder to check if similarities between skills are flagged properly, using the for example cosine distance. You can then automatically label the similarities and eventually correct the labels. Once you have sufficient label, you can now retrain the encoders on your own (smaller) dataset.