How to extract word frequency from document-term matrix?
Question: I am doing LDA analysis with Python. And I used the following code to create a
document-term matrix

    
    
    corpus = [dictionary.doc2bow(text) for text in texts].
    

Is there any easy ways to count the word frequency over the whole corpus.
Since I do have the dictionary which is a term-id list, I think I can match
the word frequency with term-id.


Answer: You can use `nltk` in order to count word frequency in string `texts`

    
    
    from nltk import FreqDist
    import nltk
    texts = 'hi there hello there'
    words = nltk.tokenize.word_tokenize(texts)
    fdist = FreqDist(words)
    

`fdist` will give you word frequency of given string `texts`.

However, you have a list of text. One way to count frequency is to use
`CountVectorizer` from `scikit-learn` for list of strings.

    
    
    import numpy as np
    from sklearn.feature_extraction.text import CountVectorizer
    texts = ['hi there', 'hello there', 'hello here you are']
    vectorizer = CountVectorizer()
    X = vectorizer.fit_transform(texts)
    freq = np.ravel(X.sum(axis=0)) # sum each columns to get total counts for each word
    

this `freq` will correspond to value in dictionary `vectorizer.vocabulary_`

    
    
    import operator
    # get vocabulary keys, sorted by value
    vocab = [v[0] for v in sorted(vectorizer.vocabulary_.items(), key=operator.itemgetter(1))]
    fdist = dict(zip(vocab, freq)) # return same format as nltk
    

