==post with dist= imaging databases store data ==post with dist= imaging databases store data imaging databases store data imaging databases store data best post is with dist= stop words on steroids now that we have reasonable way to extract compact vector from noisy textual postlet us step back for while to think about what the feature values actually mean the feature values simply count occurrences of terms in post we silently assumed that higher values for term also mean that the term is of greater importance to the given post but what aboutfor instancethe word "subject"which naturally occurs in each and every single postalrightwe could tell countvectorizer to remove it as well by means of its max_df parameter we couldfor instanceset it to so that all words that occur in more than percent of all posts would be always ignored but what about words that appear in percent of all postshow low would we be willing to set max_dfthe problem is that however we set itthere will always be the problem that some terms are just more discriminative than others this can only be solved by counting term frequencies for every postand in additiondiscounting those that appear in many posts in other wordswe want high value for given term in given value if that term occurs often in that particular post and very rarely anywhere else this is exactly what term frequency inverse document frequency (tf-idfdoestf stands for the counting partwhile idf factors in the discounting naive implementation would look like the followingimport scipy as sp def tfidf(termdocdocset)tf float(doc count(term))/sum(doc count(wfor in docsetidf math log(float(len(docset))/(len([doc for doc in docset if term in doc]))return tf idf for the following document setdocsetconsisting of three documents that are already tokenizedwe can see how the terms are treated differentlyalthough all appear equally often per documentaabbabc [" "][" "" "" "][" "" "" " [aabbabcprint(tfidf(" "ad)