for more details on these topicsyou can jump ahead to of this book or refer to the section "text normalization,page of text analytics with python (apressdipanjan sarkar which covers each of these techniques in detail we will be normalizing our text here by lowercasingremoving special characterstokenizingand removing stopwords the following code helps us achieve this in [ ]wpt nltk wordpuncttokenizer(stop_words nltk corpus stopwords words('english'def normalize_document(doc)lower case and remove special characters\whitespaces doc re sub( '[^ -za- - \ ]'''docre idoc doc lower(doc doc strip(tokenize document tokens wpt tokenize(docfilter stopwords out of document filtered_tokens [token for token in tokens if token not in stop_wordsre-create document from filtered tokens doc join(filtered_tokensreturn doc normalize_corpus np vectorize(normalize_documentthe np vectorizefunction helps us run the same function over all elements of numpy array instead of writing loop we will now use this function to pre-process our text corpus in [ ]norm_corpus normalize_corpus(corpusnorm_corpus out[ ]array(['sky blue beautiful''love blue beautiful sky''quick brown fox jumps lazy dog''brown fox quick blue dog lazy''sky blue sky beautiful today''dog lazy brown fox quick']dtype='< 'you can compare each text document with its original form in our initial dataframe you will see that each document is in the lowercasespecial symbols have been removed and stopwords (words which carry little meaning like articlespronounsetc have been removed we can now engineer features from this preprocessed corpus bag of words model this is perhaps one of the simplest yet effective schemes of vectorizing features from unstructured text the core principle of this model is to convert text documents into numeric vectors the dimension or size of each vector is where indicates all possible distinct words across the corpus of documents each document once transformed is numeric vector of size where the values or weights in the vector indicate the frequency of each word in that specific document the following code helps us vectorize the text corpus into numeric feature vectors in [ ]from sklearn feature_extraction text import countvectorizer cv countvectorizer(min_df= max_df=