theorem expected in-sample risk for linear models let be the model matrix for linear modelof dimension if var[ (xx = does not depend on xthen the expected in-sample risk (with respect to the squared-error lossfor random learner gt is given by ex `in (gt ex ` (gt ` ( where `is the irreducible risk proofthe expected optimism isby definitionex [`in (gt ` (gt )whichfor the squared-error lossis equal to ` /nusing exactly the same reasoning as in example note that here ` equation ( is the basis of the following model comparison heuristicestimate the irreducible risk ` via vb using model with relatively high complexity then choose the linear model with the lowest value of ky xb bk vb ( we can also use the akaike information criterion (aicas heuristic for model comparison we discussed the aic in the unsupervised learning setting in section but the arguments used there can also be applied to the supervised caseunder the in-sample model for the data in particularlet (xywe wish to predict the joint density { =xi ( xi ) (zf (xy: = using prediction function ( thfrom family :{ ( th)th rq }where ( thg(xy th: { =xi gi ( thn = note that is the number of parameters (typically larger than for linear model with design matrixfollowing section the in-sample cross-entropy risk in this case is (th:-ex ln ( th)and to approximate the optimal parameter thwe minimize the corresponding training loss ln ( thrtn (th: = the optimal parameter thn for the training loss is thus found by minimizing xln ln ( thn =