proper way to assess the prediction accuracy of the learner at the feature vectors xn is to draw new response values yi ( xi ) nthat are independent from the responses yn in the training dataand then estimate the in-sample risk of gt via loss(yi gt (xi ) = for fixed training set twe can compare the training loss of the learner with the in-sample risk their differenceopt `in (gt ` (gt )expected optimism is called the optimism (of the training loss)because it measures how much the training loss underestimates (is optimistic aboutthe unknown in-sample risk mathematicallyit is simpler to work with the expected optimisme[opt xn xn =ex opt where the expectation is taken over random training set conditional on xi xi for ease of notationwe have abbreviated the expected optimism to ex opt where ex denotes the expectation operator conditional on xi xi as in example the feature vectors are stored as the rows of an nx matrix it turns out that the expected optimism for various loss functions can be expressed in terms of the (conditionalcovariance between the observed and predicted response theorem expected optimism for the squared-error loss and - loss with - responsethe expected optimism is ex opt covx (gt (xi )yi = ( proofin what followsall expectations are taken conditional on xn xn let yi be the response for xi and let yi gt (xi be the predicted value note that the latter depends on yn alsolet yi be an independent copy of yi for the same xi as in ( in particularyi has the same distribution as yi and is statistically independent of all { }including yi and therefore is also independent of yi we have bi ) yi ex (yi yi (yi ex (yi yi ) = = xb ex [yi yi ex yi ex yi covx ( yi yi = = ex opt the proof for the - loss with - response is left as exercise in summarythe expected optimism indicates how muchon averagethe training loss deviates from the expected in-sample risk since the covariance of independent random variables is zerothe expected optimism is zero if the learner gt is statistically independent from the responses yn