oob_pred_arr np zeros (lenx_train )for in range (lenx_train )) x_train [ireshape ( - [for in range n_estimators )if(np isin(ibootstrap_ds_arr [ ]=false ) append (bfor pred in bag[ ]oob_pred_arr [ioob_pred_arr [ (pred predict ( )/len( )l_oob _score (y_train oob_pred_arr print (decisiontreeregressor ^ score ", _score (y_test yhat)"nbagging ^ score " _score (y_test yhatbag )"nbagging oob ^ score ",l_oob decisiontreeregressor ^ score bagging ^ score bagging oob ^ score the decision tree bagging improves the test-set score by about (from to moreoverthe oob score ( is very close to the true generalization risk ( of the bagged estimator the bagging procedure can be further enhanced by introducing random forestswhich is discussed next random forests in section we discussed the intuition behind the prediction averaging procedure specificallyfor some feature vector let zb gtb ( ) be iid prediction valuesobtained from independent training sets tb suppose that var zb for all then the variance of the average prediction value is equal to / howeverif bootstrapped data sets {tbare used insteadthe corresponding random variables {zb will be correlated in particularzb gtb(xfor are identically distributed (but not independentwith some positive pairwise correlation it then holds that (see exercise var ( % ( while the second term of ( goes to zero as the number of observation increasesthe first term remains constant this issue is particularly relevant for bagging with decision trees for exampleconsider situation in which there exists feature that provides very good split of the data such feature will be selected and split for every {gtb} = at the root level and we will consequently end up with highly correlated predictions in such situationprediction averaging will not introduce the desired improvement in the performance of the bagged predictor