stemming and lemmatizationword stems are usually the base form of possible words that can be created by attaching affixes like prefixes and suffixes to the stem to create new words this is known as inflection the reverse process of obtaining the base form of word is known as stemming simple example are the words watcheswatchingand watched they have the word root stem watch as the base form the nltk package offers wide range of stemmers like the porterstemmer and lancasterstemmer lemmatization is very similar to stemmingwhere we remove word affixes to get to the base form of word however the base form in this case is known as the root word but not the root stem the difference being that the root word is always lexicographically correct word (present in the dictionarybut the root stem may not be so we will be using lemmatization only in our normalization pipeline to retain lexicographically correct words the function lemmatize_texthelps us with this aspect removing stopwordswords which have little or no significance especially when constructing meaningful features from text are also known as stopwords or stop words these are usually words that end up having the maximum frequency if you do simple term or word frequency in document corpus words like aantheand so on are considered to be stopwords there is no universal stopword list but we use standard english language stopwords list from nltk you can also add your own domain specific stopwords if needed the function remove_stopwordshelps us remove stopwords and retain words having the most significance and context in corpus we use all these components and tie them together in the following function called normalize_ corpus)which can be used to take document corpus as input and return the same corpus with cleaned and normalized text documents def normalize_corpus(corpushtml_stripping=truecontraction_expansion=trueaccented_char_removal=truetext_lower_case=truetext_lemmatization=truespecial_char_removal=truestopword_removal=true)normalized_corpus [normalize each document in the corpus for doc in corpusstrip html if html_strippingdoc strip_html_tags(docremove accented characters if accented_char_removaldoc remove_accented_chars(docexpand contractions if contraction_expansiondoc expand_contractions(doclowercase the text if text_lower_casedoc doc lower(remove extra newlines doc re sub( '[\ |\ |\ \ ]+'',docinsert spaces between special characters to isolate them special_char_pattern re compile( '([(-)!}])'doc special_char_pattern sub(\\ "doc