extending the vectorizer with nltk' stemmer we need to stem the posts before we feed them into countvectorizer the class provides several hooks with which we could customize the preprocessing and tokenization stages the preprocessor and tokenizer can be set in the constructor as parameters we do not want to place the stemmer into any of thembecause we would then have to do the tokenization and normalization by ourselves insteadwe overwrite the method build_analyzer as followsimport nltk stem english_stemmer nltk stem snowballstemmer('english'class stemmedcountvectorizer(countvectorizer)def build_analyzer(self)analyzer super(stemmedcountvectorizerselfbuild_ analyzer(return lambda doc(english_stemmer stem(wfor in analyzer(doc)vectorizer stemmedcountvectorizer(min_df= stop_ words='english'this will perform the following steps for each post lower casing the raw post in the preprocessing step (done in the parent class extracting all individual words in the tokenization step (done in the parent class converting each word into its stemmed version as resultwe now have one feature lessbecause "imagesand "imagingcollapsed to one the set of feature names looks like the following[ 'actual' 'capabl' 'contain' 'data' 'databas' 'imag' 'interest' 'learn' 'machin' 'perman' 'post' 'provid' 'safe' 'storag' 'store' 'stuff' 'toy'running our new stemmed vectorizer over our postswe see that collapsing "imagingand "imagesreveals that post is actually the most similar post to our new postas it contains the concept "imagtwice==post with dist= this is toy post about machine learning actuallyit contains not much interesting stuff ==post with dist= imaging databases provide storage capabilities ==post with dist= most imaging databases safe images permanently