text mining and text analytics figure the first word vector of the "datasciencecategory after first data processing attempt this sure looks pollutedpunctuations are kept as separate terms and several words haven' even been split further data exploration should clarify few things for us step data exploration we now have all our terms separatedbut the sheer size of the data hinders us from getting good grip on whether it' clean enough for actual use by looking at single vectorwe already spot few problems thoughseveral words haven' been split correctly and the vector contains many single-character terms single character terms might be good topic differentiators in certain cases for examplean economic text will contain more $psand $signs than medical text but in most cases these onecharacter terms are useless firstlet' have look at the frequency distribution of our terms wordfreqs_cat nltk freqdist(data['datascience']['all_words']plt hist(wordfreqs_cat values()bins range( )plt show(wordfreqs_cat nltk freqdist(data['gameofthrones']['all_words']plt hist(wordfreqs_cat values()bins range( )plt show(by drawing histogram of the frequency distribution (figure we quickly notice that the bulk of our terms only occur in single document single-occurrence terms such as these are called hapaxesand model-wise they're useless because single occurrence of feature is never enough to build reliable model this is good news for uscutting these hapaxes out will significantly shrink our data without harming our eventual model let' look at few of these singleoccurrence terms print wordfreqs_cat hapaxes(print wordfreqs_cat hapaxes(terms we see in figure make senseand if we had more data they' likely occur more often print wordfreqs_cat hapaxes(print wordfreqs_cat hapaxes(