setting hyperparameters in smart way in the preceding examplewe set the penalty parameter to we could just as well have set it to (or halfor or millionnaturallythe results vary each time if we pick an overly large valuewe get underfitting in extreme casethe learning system will just return every coefficient equal to zero if we pick value that is too smallwe overfit and are very close to olswhich generalizes poorly how do we choose good valuethis is general problem in machine learningsetting parameters for our learning methods generic solution is to use crossvalidation we pick set of possible valuesand then use cross-validation to choose which one is best this performs more computation (ten times more if we use folds)but is always applicable and unbiased we must be carefulthough in order to obtain an estimate of generalizationwe have to use two levels of cross-validationone level is to estimate the generalizationwhile the second level is to get good parameters that iswe split the data infor example folds we start by holding out the first fold and will learn on the other nine nowwe split these again into folds in order to choose the parameters once we have set our parameterswe test on the first fold nowwe repeat this nine other times the preceding figure shows how you break up single training fold into subfolds we would need to repeat it for all the other folds in this casewe are looking at five outer folds and five inner foldsbut there is no reason to use the same number of outer and inner foldsyou can use any numbers you want as long as you keep them separate this leads to lot of computationbut it is necessary in order to do things correctly the problem is that if you use piece of data to make any decisions about your model (including which parameters to set)you have contaminated it and you can no longer use it to test the generalization ability of your model this is subtle point and it may not be immediately obvious in factit is still the case that many users of machine learning get this wrong and overestimate how well their systems are doing because they do not perform cross-validation correctly