extracts tokens using regular expression by defaultthe regular expression that is used is "\ \ \ +\bif you are not familiar with regular expressionsthis means it finds all sequences of characters that consist of at least two letters or numbers (\wand that are separated by word boundaries (\bit does not find single-letter wordsand it splits up contractions like "doesn'tor "bit ly"but it matches " teras single word the countvectorizer then converts all words to lowercase charactersso that "soon""soon"and "soonall correspond to the same token (and therefore featurethis simple mechanism works quite well in practicebut as we saw earlierwe get many uninformative features (like the numbersone way to cut back on these is to only use tokens that appear in at least two documents (or at least five documentsand so ona token that appears only in single document is unlikely to appear in the test set and is therefore not helpful we can set the minimum number of documents token needs to appear in with the min_df parameterin[ ]vect countvectorizer(min_df= fit(text_trainx_train vect transform(text_trainprint("x_train with min_df{}format(repr(x_train))out[ ]x_train with min_dfwith stored elements in compressed sparse row formatby requiring at least five appearances of each tokenwe can bring down the number of features to , as seen in the preceding output--only about third of the original features let' look at some tokens againin[ ]feature_names vect get_feature_names(print("first features:\ {}format(feature_names[: ])print("features to :\ {}format(feature_names[ : ])print("every th feature:\ {}format(feature_names[:: ])out[ ]first features[' '' '' '' '' '' '' '' '' '' '' '' '' '' '' '' '' th'' '' '' '' '' '' '' '' '' th'' '' '' '' '' '' th'' '' '' th'' '' '' th'' '' '' th'' '' '' th'' '' '' '' mm'' '' th'features to ['repentance''repercussions''repertoire''repetition''repetitions''repetitious''repetitive''rephrase''replace''replaced''replacement''replaces''replacing''replay''replayable''replayed''replaying''replays''replete''replica'representing text data as bag of words