evaluation holding out data and cross-validation the model discussed in the preceding section is simple modelit achieves percent accuracy on its training data howeverthis evaluation may be overly optimistic we used the data to define what the threshold would beand then we used the same data to evaluate the model of coursethe model will perform better than anything else we have tried on this dataset the logic is circular what we really want to do is estimate the ability of the model to generalize to new instances we should measure its performance in instances that the algorithm has not seen at training thereforewe are going to do more rigorous evaluation and use held-out data for thiswe are going to break up the data into two blockson one blockwe'll train the modeland on the other--the one we held out of training--we'll test it the output is as followstraining error was testing error was ( the result of the testing data is lower than that of the training error this may surprise an inexperienced machine learnerbut it is expected and typical to see whylook back at the plot that showed the decision boundary see if some of the examples close to the boundary were not there or if one of the ones in between the two lines was missing it is easy to imagine that the boundary would then move little bit to the right or to the left so as to put them on the "wrongside of the border the error on the training data is called training error and is always an overly optimistic estimate of how well your algorithm is doing we should always measure and report the testing errorthe error on collection of examples that were not used for training these concepts will become more and more important as the models become more complex in this examplethe difference between the two errors is not very large when using complex modelit is possible to get percent accuracy in training and do no better than random guessing on testingone possible problem with what we did previouslywhich was to hold off data from trainingis that we only used part of the data (in this casewe used half of itfor training on the other handif we use too little data for testingthe error estimation is performed on very small number of examples ideallywe would like to use all of the data for training and all of the data for testing as well we can achieve something quite similar by cross-validation one extreme (but sometimes usefulform of cross-validation is leave-one-out we will take an example out of the training datalearn model without this exampleand then see if the model classifies this example correctly