title: Ngram model and perplexity in NLTK
body: <p>To put my question in context, I would like to train and test/compare several (neural) language models. In order to focus on the models rather than data preparation I chose to use the Brown corpus from nltk and train the Ngrams model provided with the nltk as a baseline (to compare other LM against).</p>
  
  <p>So my first question is actually about a behaviour of the Ngram model of nltk that I find suspicious.
  Since the code is rather short I pasted it here:</p>
  
  <pre><code>import nltk
  
  print "... build"
  brown = nltk.corpus.brown
  corpus = [word.lower() for word in brown.words()]
  
  # Train on 95% f the corpus and test on the rest
  spl = 95*len(corpus)/100
  train = corpus[:spl]
  test = corpus[spl:]
  
  # Remove rare words from the corpus
  fdist = nltk.FreqDist(w for w in train)
  vocabulary = set(map(lambda x: x[0], filter(lambda x: x[1] &gt;= 5, fdist.iteritems())))
  
  train = map(lambda x: x if x in vocabulary else "*unknown*", train)
  test = map(lambda x: x if x in vocabulary else "*unknown*", test)
  
  print "... train"
  from nltk.model import NgramModel
  from nltk.probability import LidstoneProbDist
  
  estimator = lambda fdist, bins: LidstoneProbDist(fdist, 0.2) 
  lm = NgramModel(5, train, estimator=estimator)
  
  print "len(corpus) = %s, len(vocabulary) = %s, len(train) = %s, len(test) = %s" % ( len(corpus), len(vocabulary), len(train), len(test) )
  print "perplexity(test) =", lm.perplexity(test)
  </code></pre>
  
  <p>What I find very suspicious is that I get the following results:</p>
  
  <pre><code>... build
  ... train
  len(corpus) = 1161192, len(vocabulary) = 13817, len(train) = 1103132, len(test) = 58060
  perplexity(test) = 4.60298447026
  </code></pre>
  
  <p>With a perplexity of 4.6 it seems Ngram modeling is very good on that corpus. If my interpretation is correct then the model should be able to guess the correct word in roughly 5 tries on average (although there are 13817 possibilities...). If you could share your experience on the value of this perplexity (I don't really believe it)? I did not find any complaints on the ngram model of nltk on the net ( but maybe I do it wrong). Do you know a good alternatives to NLTK for Ngram models and computing perplexity?</p>
  
  <p>Thanks!</p>