converting raw text into bag-of-words we do not have to write custom code for counting words and representing those counts as vector scikit' countvectorizer does the job very efficiently it also has very convenient interface scikit' functions and classes are imported via the sklearn package as followsfrom sklearn feature_extraction text import countvectorizer vectorizer countvectorizer(min_df= the parameter min_df determines how countvectorizer treats words that are not used frequently (minimum document frequencyif it is set to an integerall words occurring less than that value will be dropped if it is fractionall words that occur less than that fraction of the overall dataset will be dropped the parameter max_df works in similar manner if we print the instancewe see what other parameters scikit provides together with their default valuesprint(vectorizercountvectorizer(analyzer=wordbinary=falsecharset=utf- charset_error=strictdtype=input=contentlowercase=truemax_df= max_features=nonemax_n=nonemin_df= min_n=nonengram_range=( )preprocessor=nonestop_words=nonestrip_accents=nonetoken_pattern=(? )\ \ \ +\btokenizer=nonevocabulary=nonewe see thatas expectedthe counting is done at word level (analyzer=wordand the words are determined by the regular expression pattern token_pattern it wouldfor exampletokenize "cross-validatedinto "crossand "validatedlet us ignore the other parameters for now content ["how to format my hard disk"hard disk format problems " vectorizer fit_transform(contentvectorizer get_feature_names([ 'disk' 'format' 'hard' 'how' 'my' 'problems' 'to'the vectorizer has detected seven words for which we can fetch the counts individuallyprint( toarray(transpose()array([[ ][ ][ ][ ][ ][ ][ ]]dtype=int