in [ ]import pandas as pd import numpy as np import re import nltk let' now load some sample text documentsdo some basic pre-processingand learn about various feature engineering strategies to deal with text data the following code creates our sample text corpus ( collection of text documents)which we will use in this section in [ ]corpus ['the sky is blue and beautiful ''love this blue and beautiful sky!''the quick brown fox jumps over the lazy dog ''the brown fox is quick and the blue dog is lazy!''the sky is very blue and the sky is very beautiful today''the dog is lazy but the brown fox is quick!labels ['weather''weather''animals''animals''weather''animals'corpus np array(corpuscorpus_df pd dataframe({'document'corpus'category'labels}corpus_df corpus_df[['document''category']corpus_df out[ ]document category the sky is blue and beautiful weather love this blue and beautiful skyweather the quick brown fox jumps over the lazy dog animals the brown fox is quick and the blue dog is lazyanimals the sky is very blue and the sky is very beaut weather the dog is lazy but the brown fox is quickanimals we can see that we have total of six documentswhere three of them are relevant to weather and the other three talk about animals as depicted by the category class label text pre-processing before feature engineeringwe need to pre-processcleanand normalize the text like we mentioned before there are multiple pre-processing techniquessome of which are quite elaborate we will not be going into lot of details in this section but we will be covering lot of them in further detail in future when we work on text classification and sentiment analysis following are some of the popular pre-processing techniques text tokenization and lower casing removing special characters contraction expansion removing stopwords correcting spellings stemming lemmatization