title: choose the best Coherence Score for LDA model
body: <p>I am using python Gensim package to build LDA model (<a href="https://www.machinelearningplus.com/nlp/topic-modeling-gensim-python/#:%7E:text=Topic%20Modeling%20with%20Gensim%20(Python)&amp;text=Topic%20Modeling%20is%20a%20technique,in%20the%20Python%27s%20Gensim%20package" rel="nofollow noreferrer">https://www.machinelearningplus.com/nlp/topic-modeling-gensim-python/#:~:text=Topic%20Modeling%20with%20Gensim%20(Python)&amp;text=Topic%20Modeling%20is%20a%20technique,in%20the%20Python's%20Gensim%20package</a>)</p>
  <p>To choose the best number of Topics in the LDA, I calculated the Coherence score for (1-20)topics and then visulaize it</p>
  <pre><code>    def compute_coherence_values(dictionary, corpus, texts, limit, start=1, step=1):
      coherence_values = []
      model_list = []
      for num_topics in range(start, limit, step):
          lda_model_coh = gensim.models.ldamodel.LdaModel(corpus=corpus, num_topics=num_topics, id2word=id2word, per_word_topics=True,update_every=1,
                                             chunksize=100,random_state=80,
                                             passes=10,
                                             alpha='auto')
          model_list.append(lda_model_coh)
          coherence_model_lda = CoherenceModel(model=lda_model_coh, texts=data_words_nostops, dictionary=id2word, coherence='c_v')
          coherence_values.append(coherence_model_lda.get_coherence())
  
      return model_list, coherence_values
  
  model_list, coherence_values = compute_coherence_values(dictionary=id2word, corpus=corpus, texts=data_words_nostops, start=1, limit=21, step=1)
  print(coherence_values)
  </code></pre>
  <p>the out put is :</p>
  <blockquote>
  <p>[0.6110807023750182, 0.623346262237542, 0.611190819343431,
  0.6150879617345366, 0.6661056841233617, 0.6460622418348893, 0.6684570240561849, 0.6603704258720786, 0.6781376351229919, 0.6686810583507139, 0.6704931154541898, 0.6209832171172912, 0.6223242456220992, 0.583528787158143, 0.5672411886488239, 0.5485767400671002, 0.5603438856884889, 0.538775236148759, 0.5424604528457801, 0.536498799229393]</p>
  </blockquote>
  <p><img src="https://i.stack.imgur.com/ltqrC.png" alt="coherence score / topic numbers" />][1]</p>
  <p>As the chart shows, The coherence score value is highest score at the value 10. But when i visualize it using the intertopic Distance maps i found the topics are crowded and the overlaps between the topics is huge. So i visualize it using 5 and 7 topics.
  <img src="https://i.stack.imgur.com/JBgZZ.jpg" alt="intertopic distance map" /><a href="https://i.stack.imgur.com/JBgZZ.jpg" rel="nofollow noreferrer">2</a>
  I am not sure which number of topics is better to choose.
  My question is, How to choose the best coherence value. a scintfic reference recommendation will be a good help for me.<br />
  [1]: <a href="https://i.stack.imgur.com/ltqrC.png" rel="nofollow noreferrer">https://i.stack.imgur.com/ltqrC.png</a></p>