fundamentals of machine learning validation_score np average(validation_scoresmodel get_model(model train(datatest_score model evaluate(test_datatrains the final model on all nontest data available validation scoreaverage of the validation scores of the folds iterated -fold validation with shuffling this one is for situations in which you have relatively little data available and you need to evaluate your model as precisely as possible 've found it to be extremely helpful in kaggle competitions it consists of applying -fold validation multiple timesshuffling the data every time before splitting it ways the final score is the average of the scores obtained at each run of -fold validation note that you end up training and evaluating models (where is the number of iterations you use)which can very expensive things to keep in mind keep an eye out for the following when you're choosing an evaluation protocoldata representativeness--you want both your training set and test set to be representative of the data at hand for instanceif you're trying to classify images of digitsand you're starting from an array of samples where the samples are ordered by their classtaking the first of the array as your training set and the remaining as your test set will result in your training set containing only classes - whereas your test set contains only classes - this seems like ridiculous mistakebut it' surprisingly common for this reasonyou usually should randomly shuffle your data before splitting it into training and test sets the arrow of time--if you're trying to predict the future given the past (for exampletomorrow' weatherstock movementsand so on)you should not randomly shuffle your data before splitting itbecause doing so will create temporal leakyour model will effectively be trained on data from the future in such situationsyou should always make sure all data in your test set is posterior to the data in the training set redundancy in your data--if some data points in your data appear twice (fairly common with real-world data)then shuffling the data and splitting it into training set and validation set will result in redundancy between the training and validation sets in effectyou'll be testing on part of your training datawhich is the worst thing you can domake sure your training set and validation set are disjoint licensed to