we also use our custom developed text pre-processing and normalization modulewhich you will find in the files named contractions py and text_normalizer py utilities related to supervised model fittingpredictionand evaluation are present in model_evaluation_utils pyso make sure you have these modules in the same directory and the other python files and jupyter notebooks for this getting the data the dataset will be available along with the code files for this in the github repository for this book at movie_reviews csv containing , labeled imdb movie reviews you can also download the same data from easily load it in python using the read_csvutility function from pandas text pre-processing and normalization one of the key steps before diving into the process of feature engineering and modeling involves cleaningpre-processingand normalizing text to bring text components like phrases and words to some standard format this enables standardization across document corpuswhich helps build meaningful features and helps reduce noise that can be introduced due to many factors like irrelevant symbolsspecial charactersxml and html tagsand so on the file named text_normalizer py has all the necessary utilities we will be using for our text normalization needs you can also refer to the jupyter notebook named text normalization demo ipynb for more interactive experience the main components in our text normalization pipeline are described in this section cleaning textour text often contains unnecessary content like html tagswhich do not add much value when analyzing sentiment hence we need to make sure we remove them before extracting features the beautifulsoup library does an excellent job in providing necessary functions for this our strip_html_tagsfunction enables in cleaning and stripping out html code removing accented charactersin our datasetwe are dealing with reviews in the english language so we need to make sure that characters with any other formatespecially accented characters are converted and standardized into ascii characters simple example would be converting to our remove_accented_ charsfunction helps us in this respect expanding contractionsin the english languagecontractions are basically shortened versions of words or syllables these shortened versions of existing words or phrases are created by removing specific letters and sounds more than often vowels are removed from the words examples would bedo not to don' and would to ' contractions pose problem in text normalization because we have to deal with special characters like the apostrophe and we also have to convert each contraction to its expandedoriginal form our expand_contractionsfunction uses regular expressions and various contractions mapped in our contractions py module to expand all contractions in our text corpus removing special charactersanother important task in text cleaning and normalization is to remove special characters and symbols that often add to the extra noise in unstructured text simple regexes can be used to achieve this our function remove_special_charactershelps us remove special characters in our codewe have retained numbers but you can also remove numbers if you do not want them in your normalized corpus