title: How can I calculate perplexity using nltk
body: <p>I try to do some process on a text. It's part of my code:</p>
  
  <pre><code>fp = open(train_file)
  raw = fp.read()
  sents = fp.readlines()
  words = nltk.tokenize.word_tokenize(raw)
  bigrams = ngrams(words,2, left_pad_symbol='&lt;s&gt;', right_pad_symbol=&lt;/s&gt;)
  fdist = nltk.FreqDist(words)
  </code></pre>
  
  <p>In the old versions of <code>nltk</code> I found this code on <a href="https://stackoverflow.com/questions/16509685/ngram-model-and-perplexity-in-nltk">StackOverflow</a> for <code>perplexity</code></p>
  
  <pre><code>estimator = lambda fdist, bins: LidstoneProbDist(fdist, 0.2) 
  lm = NgramModel(5, train, estimator=estimator)
  print("len(corpus) = %s, len(vocabulary) = %s, len(train) = %s, len(test) = %s" % ( len(corpus), len(vocabulary), len(train), len(test) ))
  print("perplexity(test) =", lm.perplexity(test))   
  </code></pre>
  
  <p>However, this code is no longer valid, and I didn't find any other package or function in <code>nltk</code> for this purpose. Should I implement it?</p>