this is only after tokenizationlower casingand stop word removal if we also subtract those words that will be later filtered out via min_df and max_dfwhich will be done later in fit_transformit gets even worselist(set(analyzer( [ ][ ])intersectionvectorizer get_feature_names())[ 'cs' 'faq' 'thank'list(set(analyzer( [ ][ ])intersectionvectorizer get_feature_names())[ 'bh' 'thank'furthermoremost of the words occur frequently in other posts as wellas we can check with the idf scores remember that the higher the tf-idfthe more discriminative term is for given post and as idf is multiplicative factor herea low value of it signals that it is not of great value in generalfor term in ['cs''faq''thank''bh''thank']print('idf(% )= '%(termvectorizer _tfidf idf_[vectorizer vocabulary_[term]]idf(cs)= idf(faq)= idf(thank)= idf(bh)= idf(thank)= soexcept for bhwhich is close to the maximum overall idf value of the terms don' have much discriminative power understandablyposts from different newsgroups will be clustered together for our goalhoweverthis is no big dealas we are only interested in cutting down the number of posts that we have to compare new post to after allthe particular newsgroup from where our training data came from is of no special interest tweaking the parameters so what about all the other parameterscan we tweak them all to get better resultssure we couldof coursetweak the number of clusters or play with the vectorizer' max_features parameter (you should try that!alsowe could play with different cluster center initializations there are also more exciting alternatives to kmeans itself there arefor exampleclustering approaches that also let you use different similarity measurements such as cosine similaritypearsonor jaccard an exciting field for you to play