text mining and text analytics notice the changes since the last data_processing(function our tokenizer is now regular expression tokenizer regular expressions are not part of this book and are often considered challenging to masterbut all this simple one does is cut the text into words for wordsany alphanumeric combination is allowed (\ )so there are no more special characters or punctuations we also applied the word stemmer and removed list of extra stop words andall the hapaxes are removed at the end because everything needs to be stemmed first let' run our data preparation again if we did the same exploratory analysis as beforewe' see it makes more senseand we have no more hapaxes print wordfreqs_cat hapaxes(print wordfreqs_cat hapaxes(let' take the top words of each category again (see figure figure top most frequent words in "data scienceand "game of thronesreddit posts after data preparation we can see in figure how the data quality has improved remarkably alsonotice how certain words are shortened because of the stemming we applied for instance"scienceand "scienceshave become "scienc;"coursesand "coursehave become "cours,and so on the resulting terms are not actual words but still interpretable if you insist on your terms remaining actual wordslemmatization would be the way to go