gensim LDA: How can i generate topics with different words for each topic?
LDA provides for each topic and each word a probability that the topic generates that word. You can try assigning words to topics by just taking the max over all topics of the probability to generate the word. In other words if topic A generates "monkey" with probability 0.01 and topic B generates the word monkey with probability 0.02 then you can assign the word monkey to topic B.
I'm using the LDA algorithm from the gensim package to find topics in a given text.
I've been asked that the resulting topics will include different words for each topic, E.G If topic A has the word 'monkey' in it then no other topic should include the word 'monkey' in its list.
My thoughts so far: run it multiple times and each time add the previous words to the stop words list.
Since:
A) I'm not even sure of algorithmically/logically it's the right thing to do.
B) I hope there's a built in way to do it that i'm not aware of.
C) This is a large database, and it takes about 20 minutes to run the LDA
each time (using the multi-core version).
Question: Is there a better way to do it?
Hope to get some help,
Thanks.