title: Identifying important words and phrases in text
body: <p>I have text stored in a python string.</p>
  
  <p><strong>What I Want</strong></p>
  
  <ol>
  <li>To identify key words in that text.</li>
  <li>to identify N-grams in that text (ideally more than just bi and tri grams).</li>
  </ol>
  
  <p>Keep in mind...</p>
  
  <ul>
  <li>The text might be small (e.g. tweet sized)</li>
  <li>The text might be middle (e.g. news article sized)</li>
  <li>The text might be large (e.g. book or chapter sized)</li>
  </ul>
  
  <p><strong>What I Have</strong></p>
  
  <p>I'm already using <a href="http://www.nltk.org/" rel="noreferrer">nltk</a> to break the corpus into tokens and remove stopwords:</p>
  
  <pre><code>    # split across any non-word character
      tokenizer = nltk.tokenize.RegexpTokenizer('[^\w\']+', gaps=True)
  
      # tokenize
      tokens = tokenizer.tokenize(text)
  
      # remove stopwords
      tokens = [w for w in tokens if not w in nltk.corpus.stopwords.words('english')]
  </code></pre>
  
  <p>I'm aware of the BigramCollocationFinder and TrigramCollectionFinder which does exaclty what I'm looking for for those two cases.</p>
  
  <p><strong>The Question</strong></p>
  
  <p>I need advice for n-grams of higher order, improving the kinds of results that come from BCF and TCF, and advice on the best way to identify the most unique individual key words.</p>
  
  <p>Many thanks!</p>