Check word frequency of imported file against vocabulary python
Question: I want to create bag of words representation of text file in form of vector
(.toarray()) . I am using code :

    
    
    from sklearn.feature_extraction.text import CountVectorizer
    
    vectorizer = CountVectorizer(input="file")
    f = open('D:\\test\\45.txt')
    bag_of_words = vectorizer.fit_transform([f])
    print(bag_of_words)
    

I want to use vocabulary of countvectorizer for comparison. I have text file
which I tokenized and want to use it as vocabulary. How to do it?


Answer: Given that the tokenization was done by insert whitespaces between the single
tokens creating an vocabulary from text is as simple as:

    
    
    f = open('foo.txt')
    text = f.read() # text is a string
    tokens = text.split() # breaks the string in single tokens
    vocab  = list(set(tokens)) # set() removes the doubles form the token list
    

