print("transformed data (dense):\ {}format(cv transform(bards_wordstoarray())out[ ]transformed data (dense)[[ [ ]for most applicationsthe minimum number of tokens should be oneas single words often capture lot of meaning adding bigrams helps in most cases adding longer sequences--up to -grams--might help toobut this will lead to an explosion of the number of features and might lead to overfittingas there will be many very specific features in principlethe number of bigrams could be the number of unigrams squared and the number of trigrams could be the number of unigrams to the power of threeleading to very large feature spaces in practicethe number of higher -grams that actually appear in the data is much smallerbecause of the structure of the (englishlanguagethough it is still large here is what using unigramsbigramsand trigrams on bards_words looks likein[ ]cv countvectorizer(ngram_range=( )fit(bards_wordsprint("vocabulary size{}format(len(cv vocabulary_))print("vocabulary:\ {}format(cv get_feature_names())out[ ]vocabulary size vocabulary['be''be fool''but''but the''but the wise''doth''doth think''doth think he''fool''fool doth''fool doth think''he''he is''he is wise''himself''himself to''himself to be''is''is wise''knows''knows himself''knows himself to''man''man knows''man knows himself''the''the fool''the fool doth''the wise''the wise man''think''think he''think he is''to''to be''to be fool''wise''wise man''wise man knows'let' try out the tfidfvectorizer on the imdb movie review data and find the best setting of -gram range using grid searchin[ ]pipe make_pipeline(tfidfvectorizer(min_df= )logisticregression()running the grid search takes long time because of the relatively large grid and the inclusion of trigrams param_grid {"logisticregression__c"[ ]"tfidfvectorizer__ngram_range"[( )( )( )]grid gridsearchcv(pipeparam_gridcv= grid fit(text_trainy_trainprint("best cross-validation score{ }format(grid best_score_)print("best parameters:\ {}format(grid best_params_)bag-of-words with more than one word ( -grams