title: How to find common phrases from a text document
body: <p>I have a text file with a lot of comments/sentences, and I want to somehow find the most common phrases repeated in the document itself. I tried fiddling with it a bit with NLTK and I found this thread: <a href="https://stackoverflow.com/questions/2452982/how-to-extract-common-significant-phrases-from-a-series-of-text-entries">How to extract common / significant phrases from a series of text entries</a></p>
  
  <p>However, after trying it, I get odd results like these: </p>
  
  <pre><code>&gt;&gt;&gt; finder.apply_freq_filter(3)
  &gt;&gt;&gt; finder.nbest(bigram_measures.pmi, 10)
  [('m', 'e'), ('t', 's')]
  </code></pre>
  
  <p>And in another file where the phrase "this is funny" is very common, I get an empty list [].</p>
  
  <p>How should I go about doing this? </p>
  
  <p>Here's my full code:</p>
  
  <pre><code>import nltk
  from nltk.collocations import *
  bigram_measures = nltk.collocations.BigramAssocMeasures()
  trigram_measures = nltk.collocations.TrigramAssocMeasures()
  
  # change this to read in your data
  finder = BigramCollocationFinder.from_words('MkXVM6ad9nI.txt')
  
  # only bigrams that appear 3+ times
  finder.apply_freq_filter(3)
  
  # return the 10 n-grams with the highest PMI
  print finder.nbest(bigram_measures.pmi, 10)
  </code></pre>