we can improve overfitting further by sampling not only the instances but also the predictors with random forestswe take multiple random samples of instances from the training data like we do in baggingbutfor each split in each decision treewe make the split based not on all the predictors but rather random sample of the predictors the number of predictors we consider for each split is usually the square root of the total number of predictors by sampling the predictors in this waythe random forests algorithm creates trees that are even less correlated with each other (compared to the trees in bagging)reducing overfitting and improving the generalization error boosting another approachknown as boostingis used to create multiple trees like in bagging but to build the trees sequentiallyusing what the ai learned from the previous tree to improve results on the subsequent tree each tree is kept pretty shallowwith only few decision splitsand the learning occurs slowlytree by tree of all the tree-based methodsgradient boosting machines are among the best-performing and are commonly used to win machine learning competitions strengths tree-based methods are among the best-performing supervised-learning algorithms for prediction problems these methods are able to capture complex relationships in the data by learning many simple rulesone rule at time they are also capable of handling missing data and categorical features weaknesses tree-based methods are difficult to interpretespecially if many rules are needed to make good prediction performance also becomes an issue as the number of features increase applications gradient boosting and random forests are excellent for prediction problems support vector machines instead of building trees to separate datawe can use algorithms to create hyperplanes in space that separate the dataguided by the labels that we have the approach is known as support vector machines (svmssvms allow some violations to this separation--not all the points within an area in hyperspace need to have the same label- for more on gradient boosting in machine learning competitionsconsult ben gorman' blog post (bit ly/ qy unsupervised learning in the machine learning ecosystem