training and test loss given an arbitrary prediction function git is typically not possible to compute its risk `(gin ( howeverusing the training sample we can approximate `(gvia the empirical (sample averagerisk loss(yi (xi ))( ` (gn = which we call the training loss the training loss is thus an unbiased estimator of the risk (the expected lossfor prediction function gbased on the training data to approximate the optimal prediction function (the minimizer of the risk `( )we first select suitable collection of approximating functions and then take our learner to be the function in that minimizes the training lossthat isggt argmin ` (ggg training loss ( for examplethe simplest and most useful is the set of linear functions of xthat isthe set of all functions bx for some real-valued vector we suppress the superscript when it is clear which function class is used note that minimizing the training loss over all possible functions (rather than over all gdoes not lead to meaningful optimization problemas any function for which (xi yi for all gives minimal training loss in particularfor squared-error lossthe training loss will be unfortunatelysuch functions have poor ability to predict new (that isindependent from pairs of data this poor generalization performance is called overfitting overfitting by choosing function that predicts the training data exactly (and isfor example otherwise)the squared-error training loss is zero minimizing the training loss is not the ultimate goalthe prediction accuracy of new pairs of data is measured by the generalization risk of the learner for fixed training set it is defined as `(ggt loss(yggt ( ))( where (xyis distributed according to (xyin the discrete case the generalization risk is therefore`(ggt , loss(yggt ( ) (xy(replace the sum with an integral for the continuous casethe situation is illustrated in figure where the distribution of (xyis indicated by the red dots the training set (points in the shaded regionsdetermines fixed prediction function shown as straight line three possible outcomes of (xyare shown (black dotsthe amount of loss for each point is shown as the length of the dashed lines the generalization risk is the average loss over all possible pairs (xy)weighted by the corresponding (xygeneralization risk