risk and loss in unsupervised learning where htn (thn : ni= (xthi thn is the hessian of rtn (that some thn between thn and thtaking expectations on both sides of ( )we obtain thn )htn (thn )(thb thn (thertn ( thn (thb replacing htn (thn with (thfor large and using ( )we have (thb thn )htn (thn )(thb thn -tr (thh- (thn thereforeasymptotically as we have ( theorem has number of interesting consequences similar to section the training loss `tn (gtn rtn ( thn tends to underestimate the risk `( (th )because the training set tn is used to both train (that isestimate thand to estimate the risk the relation ( tells us that on average the training loss underestimates the true risk by tr( (thh- (th))/( adding equations ( and ( )yields the following asymptotic approximation to the expected generalization risk ( ( thn rtn ( thn tr (thh- (thn the first term on the right-hand side of ( can be estimated (without biasvia the training loss rtn ( thn as for the second termwe have already mentioned that when the true model gthen (thh(ththereforewhen is deemed to be sufficiently rich class of models parameterized by -dimensional vector thwe may approximate the second term as tr( (th) - (th))/ tr( )/ / this suggests the following heuristic approximation to the (expectedgeneralization riskp ( ( thn rtn ( thn multiplying both sides of ( by and substituting tr (th) - (thpwe obtain the approximationn (thn - ln (xi thn ( = akaike information criterion the right-hand side of ( is called the akaike information criterion (aicjust like ( )the aic approximation can be used to compare the difference in generalization risk of two or more learners we prefer the learner with the smallest (estimatedgeneralization risk suppose thatfor training set the training loss rt (thhas unique minimum point th which lies in the interior of th if rt (this differentiable function with respect to ththen we can find the optimal parameter th by solving rt (th (xi th th = { st (th