How can I identify bigrams and trigrams that represent concepts? I have many text documents and I want to identify concepts in these documents in an unsupervised manner. One of my problems is that the concepts can be bigrams, trigrams, or even longer.
So, for example, out of all the bigrams, how can I identify the ones that are more likely to represent a concept?
A concept could be &quot;machine learning&quot;.
Are you aware of any standard approaches to solve this problem?
Edit: The corpus I am working with consists of papers accessed from web of science. That is, they are all in some given domain niche. I want to extract words, bigrams, trigrams... that represent common concepts/buzzwords from these papers. These could be &quot;Automated machine learning&quot;, &quot;natural language processing&quot; et cetera. I need to be able to distinguish these from other common n-grams such as &quot;New York&quot;, &quot;Barack Obama&quot;,...
I know that I could do this using a NER approach but this would require hand-labelling. Are you aware of any unsupervised ways to approach this problem? Or even a semi-superised method with little labelled data?
One of the renowned methods is using the TF-IDF method. You can define a lower bound for the TF-IDF score of the bigram or trigram to identify it as a keyword. Here, you can find more information about it.