Dealing with training set of questionable quality Most of the material I have read in the past usually assumes that the training set is flawless. However that doesn't seem to be the case here with what I am given. The data that is meant to send into the training set is often questionable (I don't even know how to start separating good from bad ones). I have spent much more time trying to pre-process the files in order to increase the validity of the data, than actually building the prototype.

I have only dumped a part of the data that is claimed to be better. There are still data from other sources to complement the missing part in the first set of training data. Given the amount of data, it is practically impossible for me to spend too much time on them.

So the question is, how do people deal with really horribly, inconsistent data in real life (assuming I am working with addresses, where people often write the wrong postcode due to bad city planning or lack of clear instruction, misspell the name of neighbourhood, plus data got "autocorrected" by excel such that "1-2" becomes "1-Feb", etc. etc.)? Specifically, how do I ensure the quality of the train classifier given the training set (and cross validation set) has questionable quality?
Welcome to the real world of data science. Here, the data sets are not as clean as you thought while doing those courses/tutorials online.  Those are super polished and refined.  But, the real world data is not so.

The step where one does the cleaning and scrubbing is called the data pre-processing step.

So, some nice data cleaning techniques, in addition to @jknappen's excellent answer are:


Elimination of zero variance columns/predictors: These columns are not important, and they cause the model and the fit to crash and leak errors. So, eliminating them would make complete sense.
Correlated Predictors: Reducing the level of correlation between the predictors would be a very nice step in the pre-processing process.
Scaling: You must be knowing why scaling is important during pre-process. 
Predictor Transformations


A nice reference from Kaggle forums where the pre-processing and cleaning of data sets is discussed.