similar to ( compare this with the same decomposition for the average prediction function gbag in ( as egbag (xegt ( )we see that any possible improvement in the generalization risk must be due to the expected variance term averaging and bagging are thus only useful for predictors with large expected variancerelative to the other two terms examples of such "unstablepredictors include decision treesneural networksand subset selection in linear regression [ on the other hand"stablepredictors are insensitive to small data changesan example being the -nearest neighbors method note that for independent training sets tb reduction of the variance by factor is achievedvar gbag (xb- var gt (xagainit depends on the squared bias and irreducible loss how significant this reduction is for the generalization risk remark (limitations of baggingit is important to remember that gbag is not exactly equal to gavg which in turn is not exactly gspecificallygbag is constructed from the bootstrap approximation of the sampling pdf as consequencefor stable predictorsit can happen that gbag will perform worse than gt in addition to the deterioration of the bagging performance for stable proceduresit can also happen that gt has already achieved near optimal predictive accuracy given the available training data in this casebagging will not introduce significant improvement the bagging process provides an opportunity to estimate the generalization risk of the bagged model without an additional test set specificallyrecall that we obtain the tbsets from single training set by sampling via algorithm and use them to train separate models it can be shown (see exercise thatfor large sample sizeson average about third (more preciselya fraction - of the original sample points are not included in bootstrapped set tbfor thereforethese samples can be used for the loss estimation these samples are called out-of-bag (oobobservations specificallyfor each sample from the original data setwe calculate the oob loss using predictors that were trained without this particular sample the estimation procedure is summarized in algorithm hastie et al [ observe thatunder certain conditionsthe oob loss is almost identical to the -fold cross-validation loss in additionthe oob loss can be used to determine the number of trees required specificallywe can train predictors until the oob loss stops changing namelydecision trees are added until the oob loss stabilizes algorithm out-of-bag loss estimation inputthe original data set {( )(xnn yn )}the bootstrapped data sets { tb }and the trained predictors gt gtb outputout-of-bag loss for the averaged model for to do ci /indices of predictors not depending on (xi yi for to do if (xi yi tbthen ci ci {bp yi |ci |- bci gtb(xi li loss yi yi pn loob = li return loob out-of-bag