tree (right)the full tree is quite large and hard to visualize prediction on new data point is made by checking which region of the partition of the feature space the point lies inand then predicting the majority target (or the single target in the case of pure leavesin that region the region can be found by traversing the tree from the root and going left or rightdepending on whether the test is fulfilled or not it is also possible to use trees for regression tasksusing exactly the same technique to make predictionwe traverse the tree based on the tests in each node and find the leaf the new data point falls into the output for this data point is the mean target of the training points in this leaf controlling complexity of decision trees typicallybuilding tree as described here and continuing until all leaves are pure leads to models that are very complex and highly overfit to the training data the presence of pure leaves mean that tree is accurate on the training seteach data point in the training set is in leaf that has the correct majority class the overfitting can be seen on the left of figure - you can see the regions determined to belong to class in the middle of all the points belonging to class on the other handthere is small strip predicted as class around the point belonging to class to the very right this is not how one would imagine the decision boundary to lookand the decision boundary focuses lot on single outlier points that are far away from the other points in that class there are two common strategies to prevent overfittingstopping the creation of the tree early (also called pre-pruning)or building the tree but then removing or collapsing nodes that contain little information (also called post-pruning or just pruningpossible criteria for pre-pruning include limiting the maximum depth of the treelimiting the maximum number of leavesor requiring minimum number of points in node to keep splitting it supervised learning