figure - bi-gram feature vectors for our corpus based on bag of -grams model figure - clearly shows our bi-gram feature vectors where each feature is bi-gram of two contiguous words and the values depict the frequency of that bi-gram in each document you can use the ngram_range parameter to extend the -gram range to get -grams of higher orders typically -grams until order three are sufficient for most tasks in machine learning and natural language processing tf-idf model there are some potential problems which might arise with the bag of words model when it is used on large corpora since the feature vectors are based on absolute term frequenciesthere might be some terms which occur frequently across all documents and these will tend to overshadow other terms in the feature set the tf-idf model tries to combat this issue by using scaling or normalizing factor in its computation tf-idf stands for term frequency-inverse document frequencywhich uses combination of two metrics in its computationnamelyterm frequency (tf and inverse document frequency (idf this technique was developed for ranking results for queries in search engines and now it is an indispensable model in the world of information retrieval and text analytics mathematicallywe can define tf-idf as tfidf tf idfwhich can be expanded further to be represented as follows ae tfidf ( , tf ( , idf ( , tf ( , log cc / df ( heretfidf (wdis the tf-idf score for word in document the term tf (wdrepresents the term frequency of the word in document dwhich can be obtained from the bag of words model the term idf (wdis the inverse document frequency for the term wwhich can be computed as the log transform of the total number of documents in the corpus divided by the document frequency of the word wwhich is basically the frequency of documents in the corpus where the word occurs the following code depicts tf-idf based feature engineering on our corpus in [ ]from sklearn feature_extraction text import tfidfvectorizer tv tfidfvectorizer(min_df= max_df= use_idf=truetv_matrix tv fit_transform(norm_corpustv_matrix tv_matrix toarray(vocab tv get_feature_names(pd dataframe(np round(tv_matrix )columns=vocab