to "meetin generallemmatization is much more involved process than stemmingbut it usually produces better results than stemming when used for normalizing tokens for machine learning while scikit-learn implements neither form of normalizationcountvectorizer allows specifying your own tokenizer to convert each document into list of tokens using the tokenizer parameter we can use the lemmatization from spacy to create callable that will take string and produce list of lemmasin[ ]technicalitywe want to use the regexp-based tokenizer that is used by countvectorizer and only use the lemmatization from spacy to this endwe replace en_nlp tokenizer (the spacy tokenizerwith the regexp-based tokenization import re regexp used in countvectorizer regexp re compile('(? )\\ \\ \\ +\\ 'load spacy language model and save old tokenizer en_nlp spacy load('en'old_tokenizer en_nlp tokenizer replace the tokenizer with the preceding regexp en_nlp tokenizer lambda stringold_tokenizer tokens_from_listregexp findall(string)create custom tokenizer using the spacy document processing pipeline (now using our own tokenizerdef custom_tokenizer(document)doc_spacy en_nlp(documententity=falseparse=falsereturn [token lemma_ for token in doc_spacydefine count vectorizer with the custom tokenizer lemma_vect countvectorizer(tokenizer=custom_tokenizermin_df= let' transform the data and inspect the vocabulary sizein[ ]transform text_train using countvectorizer with lemmatization x_train_lemma lemma_vect fit_transform(text_trainprint("x_train_lemma shape{}format(x_train_lemma shape)standard countvectorizer for reference vect countvectorizer(min_df= fit(text_trainx_train vect transform(text_trainprint("x_train shape{}format(x_train shape) working with text data