title: Understanding NLTK collocation scoring for bigrams and trigrams
body: <p><strong>Background:</strong></p>
  
  <p>I am trying to compare pairs of words to see which pair is "more likely to occur" in US English than another pair.  My plan is/was to use the collocation facilities in NLTK to score word pairs, with the higher scoring pair being the most likely.</p>
  
  <p><strong>Approach:</strong></p>
  
  <p>I coded the following in Python using NLTK (several steps and imports removed for brevity):</p>
  
  <pre><code>bgm    = nltk.collocations.BigramAssocMeasures()
  finder = BigramCollocationFinder.from_words(tokens)
  scored = finder.score_ngrams( bgm.likelihood_ratio  )
  print scored
  </code></pre>
  
  <p><strong>Results:</strong></p>
  
  <p>I then examined the results using 2 word pairs, one of which should be highly likely to co-occur, and one pair which should not ("roasted cashews" and "gasoline cashews"). I was surprised to see these word pairing score identically:</p>
  
  <pre><code>[(('roasted', 'cashews'), 5.545177444479562)]
  [(('gasoline', 'cashews'), 5.545177444479562)]
  </code></pre>
  
  <p>I would have expected 'roasted cashews' to score higher than 'gasoline cashews' in my test.</p>
  
  <p><strong>Questions:</strong></p>
  
  <ol>
  <li>Am I misunderstanding the use of collocations?</li>
  <li>Is my code incorrect?</li>
  <li>Is my assumption that the scores should be different wrong, and if so why?</li>
  </ol>
  
  <p>Thank you very much for any information or help!</p>