case studyclassifying reddit posts with the data cleaning process "completed(remarka text mining cleansing exercise can almost never be fully completed)all that remains is few data transformations to get the data in the bag of words format firstlet' label all our data and also create holdout sample of observations per categoryas shown in the following listing listing final data transformation and data splitting before modeling holdout sample is comprised of unlabeled data from the two subreddits observations from each data set the labels are kept in separate data set holdoutlength holdout sample will be used to determine the model' flaws by constructing confusion matrix labeled_data [(word,'datascience'for word in data['datascience']['wordmatrix'][holdoutlength:]labeled_data [(word,'gameofthrones'for word in data['gameofthrones']['wordmatrix'][holdoutlength:]labeled_data [labeled_data extend(labeled_data labeled_data extend(labeled_data we create single data set with every word vector tagged as being either 'datascienceor 'gameofthrones we keep part of the data aside for holdout sample holdout_data data['datascience']['wordmatrix'][:holdoutlengthholdout_data extend(data['gameofthrones']['wordmatrix'][:holdoutlength]holdout_data_labels ([('datascience'for in xrange(holdoutlength)[('gameofthrones'for in xrange(holdoutlength)]data['datascience']['all_words_dedup'list(ordereddict fromkeysdata['datascience']['all_words'])data['gameofthrones']['all_words_dedup'list(ordereddict fromkeysdata['gameofthrones']['all_words'])all_words [all_words extend(data['datascience']['all_words_dedup']all_words extend(data['gameofthrones']['all_words_dedup']all_words_dedup list(ordereddict fromkeys(all_words)data for model training and testing is first shuffled prepared_data [({word(word in [ ]for word in all_words_dedup} [ ]for in labeled_dataprepared_holdout_data [({word(word in [ ]for word in all_words_dedup}for in holdout_datarandom shuffle(prepared_datatrain_size int(len(prepared_data train prepared_data[:train_sizetest prepared_data[train_size: list of all unique terms is created to build the bag of words data we need for training or scoring model data is turned into binary bag of words format size of training data will be of total and remaining will be used for testing model performance