inverse document frequency(idfit is the measure of how unique word is to this document in the given set of documents for calculating idf and formulating distinctive feature vectorwe need to reduce the weights of commonly occurring words like the and weigh up the rare words building bag of words model in nltk in this sectionwe will define collection of strings by using countvectorizer to create vectors from these sentences let us import the necessary packagefrom sklearn feature_extraction text import countvectorizer now define the set of sentences sentences=['we are using the bag of word model''bag of word used for extracting the features 'model is vectorizer_count countvectorizer(features_text vectorizer fit_transform(sentencestodense(print(vectorizer vocabulary_the above program generates the output as shown below it shows that we have distinct words in the above two sentences{'we' 'are' 'using' 'the' 'bag' 'of' 'word' 'model' 'is' 'used' 'for' 'extracting' 'features' these are the feature vectors (text to numeric formwhich can be used for machine learning solving problems in this sectionwe will solve few related problems category prediction in set of documentsnot only the words but the category of the words is also importantin which category of text particular word falls for examplewe want to predict whether given sentence belongs to the category emailnewssportscomputeretc in the following examplewe are going to use tf-idf to formulate feature vector to find the category of documents we will use the data from newsgroup dataset of sklearn we need to import the necessary packages