lemmatize text if text_lemmatizationdoc lemmatize_text(docremove special characters if special_char_removaldoc remove_special_characters(docremove extra whitespace doc re sub(+''docremove stopwords if stopword_removaldoc remove_stopwords(docis_lower_case=text_lower_casenormalized_corpus append(docreturn normalized_corpus the following snippet depicts small demo of text normalization on sample document using our normalization module in [ ]from text_normalizer import normalize_corpus in [ ]document """hellohellocan you hear mei just heard about python< >!\ \ it' an amazing language which can be used for scriptingweb development,\ \ \ \ information retrievalnatural language processingmachine learning artificial intelligence!\ what are you waiting forgo and get started he' learningshe' learningthey've already\ \ got headstart""in [ ]document out[ ]"hellohellocan you hear mei just heard about python!\ \ \ it' an amazing language which can be used for scriptingweb development\ \ \ \ \ information retrievalnatural language processingmachine learning artificial intelligence!\ \ what are you waiting forgo and get started he' learningshe' learningthey've already\ \ \ got headstart!\ in [ ]normalize_corpus([document]text_lemmatization=falsestopword_removal=falsetext_lower_case=falseout[ ]['hello hello can you hear me just heard about python it is an amazing language which can be used for scripting web development information retrieval natural language processing machine learning artificial intelligence what are you waiting for go and get started he is learning she is learning they have already got headstart 'in [ ]normalize_corpus([document]out[ ]['hello hello hear hear python amazing language use scripting web development information retrieval natural language processing machine learning artificial intelligence wait go get start learn learn already get headstart'