text mining and text analytics probability of fetus identified as female--ultrasound at weeks ultrasound female male figure decision tree with one variablethe doctor' conclusion from watching an ultrasound during pregnancy what is the probability of the fetus being femalefrom to that' pretty good discriminator decision tree follows this same principleas shown in figure if another gender test has more predictive powerit could become the root of the tree with the ultrasound test being in the branchesand this can go on until we run out of variables or observations we can run out of observationsbecause at every branch split we also split the input data this is big weakness of the decision treebecause at the leaf level of the tree robustness breaks down if too few observations are leftthe decision trees starts to overfit the data overfitting allows the model to mistake randomness for real correlations to counteract thisa decision tree is prunedits meaningless branches are left out of the final model now that we've looked at the most important new techniqueslet' dive into the case study case studyclassifying reddit posts while text mining has many applicationsin this case study we focus on document classification as pointed out earlier in this this is exactly what google does when it arranges your emails in categories or attempts to distinguish spam from regular emails it' also extensively used by contact centers that process incoming customer questions or complaintswritten complaints first pass through topic detection filter so they can be assigned to the correct people for handling document classification is also one of the mandatory features of social media monitoring systems the monitored tweetsforum or facebook postsnewspaper articlesand many other internet resources are assigned topic labels this way they can be reused in reports sentiment analysis is specific type of text classificationis the author of post negativepositiveor neutral on somethingthat "somethingcan be recognized with entity recognition in this case study we'll draw on posts from reddita website also known as the selfproclaimed "front page of the internet,and attempt to train model capable of distinguishing whether someone is talking about "data scienceor "game of thrones the end result can be presentation of our model or full-blown interactive application in we'll focus on application building for the end userso for now we'll stick to presenting our classification model to achieve our goal we'll need all the help and tools we can getand it happens python is once again ready to provide them