title: What is the total bigram count returned for NLTK BigramCollocationFinder?
body: <p>I am trying to reproduce some common nlp metrics with my own code, including Manning and Scheutze's t-test for collocational significance and chi-square test for collocational significance. </p>
  
  <p>I call nltk.bigrams() on the following list of 24 tokens:</p>
  
  <pre><code>tokens = ['she', 'knocked', 'on', 'his', 'door', 'she', 'knocked', 'at', 
  'the', 'door','100', 'women', 'knocked', 'on', "Donaldson's", 'door', 'a', 
  'man', 'knocked', 'on', 'the', 'metal', 'front', 'door']`
  </code></pre>
  
  <p>I get 23 bigrams:</p>
  
  <pre><code>[('she', 'knocked'), ('knocked', 'on'), ('on', 'his'), ('his', 'door'), ('door', 'she'), 
  ('she', 'knocked'), ('knocked', 'at'), ('at', 'the'), ('the', 'door'), ('door', '100'), 
  ('100', 'women'), ('women', 'knocked'), ('knocked', 'on'), ('on', "Donaldson's"), 
  ("Donaldson's", 'door'), ('door', 'a'), ('a', 'man'), ('man', 'knocked'),
  ('knocked', 'on'), ('on', 'the'), ('the', 'metal'), ('metal', 'front'), ('front',    
  'door')]`
  </code></pre>
  
  <p>If I want to determine the t statistic for <code>('she', 'knocked')</code>, I input:</p>
  
  <pre><code>#Total bigrams is 23
  t = (2/23 - 4/23)/(math.sqrt(2/23/23))`
  t = 1.16826337761`
  </code></pre>
  
  <p>However, when I try: </p>
  
  <pre><code>finder = BigramCollocationFinder.from_words(tokens)`
  student_t = finder.score_ngrams(bigram_measures.student_t)`
  student_t = (('she', 'knocked'), 1.178511301977579)`
  </code></pre>
  
  <p>When I turn the size of my bigram population to 24 (the length of the original list of tokens), I get the same answer as NLTK:</p>
  
  <pre><code>('she', 'knocked'): 1.17851130198
  </code></pre>
  
  <p>My question is really simple: what do I use for my population count for these hypothesis tests? The length of the tokenized list or the length of the bigram list? Or does the procedure count a terminal unit that does not output in the nltk.bigram() method?</p>