max_features = random_state = rf fit(x_train y_train yhatrf rf predict x_test print ("rf ^ score " _score (y_test yhatrf )"\nrf oob ^ score "rf oob_score_ rf ^ score rf oob ^ score remark (the optimal number of subset features mthe default values for are bp/ and for regression and classification settingrespectively howeverthe standard practice is to treat as hyperparameter that requires tuningdepending on the specific problem at hand [ note that the procedure of bagging decision trees is special case of random forest construction (see exercise consequentlythe oob loss is readily available for random forests while the advantage of bagging in the sense of enhanced accuracy is clearwe should also consider its negative aspects andin particularthe loss of interpretability specifically random forest consists of many treesthus making the prediction process both hard to visualize and interpret for examplegiven random forestit is not easy to determine subset of features that are essential for accurate prediction the feature importance measure intends to address this issue the idea is as follows each internal node of decision tree induces certain decrease in the training losssee ( let us denote this decrease in the training loss by loss ( )where is not leaf node of in additionrecall that for splitting rules of the type { ( )each node is associated with feature that determines the split using the above definitionswe can define the feature importance of as it ( loss ( { is associated with } ( internal while ( is defined for single treeit can be readily extended to random forests specificallythe feature importance in that case will be averaged over all trees of the forestthat isfor forest consisting of trees { tb }the feature importance measure isb irf ( it ( ) = ( example (feature importancewe consider classification problem with features the data is specifically designed to contain only informative features out of in the code belowwe apply the random forest procedure and calculate the corresponding feature importance measureswhich are summarized in figure feature importance