Python: How to compute the top X most frequently used words in an NLTK corpus?
Question: I'm unsure if I've understood correctly how the FreqDist functions works on
Python. As I am following a tutorial, I am led to believe that the following
code constructs a frequency distribution for a given list of words and
calculates the top x frequently used words. (In the example below let corpus
be an NLTK corpus and file to be a filename of a file in that corpus)

    
    
    words = corpus.words('file.txt')
    fd_words = nltk.FreqDist(word.lower() for word in words)
    fd_words.items()[:x]
    

However, when I go through the following commands on Python, it seems to
suggest otherwise:

    
    
    >>> from nltk import *
    >>> fdist = FreqDist(['hi','my','name','is','my','name'])
    >>> fdist
    FreqDist({'my': 2, 'name':2, 'is':1, 'hi':1}
    >>> fdist.items()
    [('is',1),('hi',1),('my',2),('name',2)]
    >>> fdist.items[:2]
    [('is',1),('hi',1)]
    

The fdist.items()[:x] method is in fact returning the x least common words?

Can someone tell me if I have done something wrong or if the mistake lies in
the tutorial I am following?


Answer: By default a `FreqDist` is not sorted. I think you are looking for
`most_common` method:

    
    
    from nltk import FreqDist
    fdist = FreqDist(['hi','my','name','is','my','name'])
    fdist.most_common(2)
    

Returns:

    
    
    [('my', 2), ('name', 2)]
    

