limitations despite the fact that the decision trees are extremely interpretablethe predictive accuracy is generally inferior to other established statistical learning methods in additiondecision treesand in particular very deep trees that were not subject to pruningare heavily reliant on their training set small change in the training set can result in dramatic change of the resulting decision tree their inferior predictive accuracyhoweveris direct consequence of the bias-variance tradeoff specificallya decision tree model generally exhibits high variance to overcome the above limitationsseveral promising approaches such as baggingrandom forestand boosting are introduced below the bagging approach was initially introduced in the context of an ensemble of decision trees howeverboth the bagging and the boosting methods can be applied to improve the accuracy of general prediction functions bootstrap aggregation the major idea of the bootstrap aggregation or bagging method is to combine prediction functions learned from multiple data setswith view to improving overall prediction accuracy bagging is especially beneficial when dealing with predictors that tend to overfit the datasuch as in decision treeswhere the (unprunedtree structure is very sensitive to small changes in the training set [ to start withconsider an idealized setting for regression treewhere we have access to iid copies tb of training set thenwe can train separate regression models ( different decision treesusing these setsgiving learners gt gtb and take their averageb gt ( ( gavg (xb = by the law of large numbersas the average prediction function converges to the expected prediction function :egt the following result shows that using gas prediction function (if it were knownwould result in an expected squared-error generalization risk that is less than or equal to the expected generalization risk for general prediction function gt it thus suggests that taking an average of prediction functions may lead to better expected squared-error generalization risk theorem expected squared-error generalization risk let be random training set and let xy be random feature vector and response that are independent of then  gt (xe ( in this section tk means the -th training setnot training set of size bagging