7.9 Tree ensembles | Using multiple decision trees --[Machine Learning | Andrew Ng]
 One of the weaknesses of using a single decision tree is that that decision tree can be highly sensitive to small changes in the data. And one solution to make the algorithm less sensitive or more robust is to build not one decision tree but to build a lot of decision trees and we call that a tree ensemble. Let's take a look. With the example that we've been using, the best feature to split on at the root node turned out to be the ear shape resulting in these two subsets of the data and then building further subtrees on these two subsets of the data. But it turns out that if you were to take just one of the ten examples and change it to a different cat so that instead of having pointy ears, round face, whiskers absent, this new cat has floppy ears, round face, whiskers present, with just changing a single training example, the highest information gain feature to split on becomes the whiskers feature instead of the ear shape feature. As a result of that, the subsets of data you get in the left and right subtrees become totally different and as you continue to run the decision tree learning algorithm recursively, you build out totally different subtrees on the left and right. So the fact that changing just one training example causes the algorithm to come up with a different split at the root and therefore a totally different tree, that makes this algorithm just not that robust. That's why when you're using decision trees, you often get a much better result, that is you get more accurate predictions if you train not just a single decision tree, but a whole bunch of different decision trees. And this is what we call a tree ensemble, which just means a collection of multiple trees. We'll see in the next few videos how to construct this ensemble of trees, but if you had this ensemble of three trees, each one of these is maybe a plausible way to classify cat versus not cat. If you had a new test example that you wanted to classify, then what you would do is run all three of these trees on your new example and get them to vote on what is the final prediction. So this test example has pointy ears, a not round face shape and whiskers are present. And so the first tree would carry an inference like this and predict that it is a cat. The second tree's inference would follow this path through the tree and therefore predict that it's not a cat. And the third tree would follow this path and therefore predict that it is a cat. These three trees have made different predictions. And so what we'll do is actually get them to vote and the majority vote of the predictions among these three trees is cat. So the final prediction of this ensemble of trees is that this is a cat, which happens to be the correct prediction. So the reason we use an ensemble of trees is by having lots of decision trees and having them vote, it makes your overall algorithm less sensitive to what any single tree may be doing because it gets only one vote out of three or one vote out of many many different votes and it makes your overall algorithm more robust. But how do you come up with all of these different plausible but maybe slightly different decision trees in order to get them to vote? In the next video, we'll talk about a technique from statistics called sampling with replacement. And this will turn out to be a key technique that we'll use in the video after that in order to build this ensemble of trees. So let's go on to the next video to talk about sampling with replacement.
[{"start": 0.0, "end": 7.48, "text": " One of the weaknesses of using a single decision tree is that that decision tree can be highly"}, {"start": 7.48, "end": 11.36, "text": " sensitive to small changes in the data."}, {"start": 11.36, "end": 16.7, "text": " And one solution to make the algorithm less sensitive or more robust is to build not one"}, {"start": 16.7, "end": 22.400000000000002, "text": " decision tree but to build a lot of decision trees and we call that a tree ensemble."}, {"start": 22.400000000000002, "end": 23.52, "text": " Let's take a look."}, {"start": 23.52, "end": 28.68, "text": " With the example that we've been using, the best feature to split on at the root node"}, {"start": 28.68, "end": 34.2, "text": " turned out to be the ear shape resulting in these two subsets of the data and then building"}, {"start": 34.2, "end": 38.04, "text": " further subtrees on these two subsets of the data."}, {"start": 38.04, "end": 42.519999999999996, "text": " But it turns out that if you were to take just one of the ten examples and change it"}, {"start": 42.519999999999996, "end": 47.96, "text": " to a different cat so that instead of having pointy ears, round face, whiskers absent,"}, {"start": 47.96, "end": 55.2, "text": " this new cat has floppy ears, round face, whiskers present, with just changing a single"}, {"start": 55.2, "end": 60.74, "text": " training example, the highest information gain feature to split on becomes the whiskers"}, {"start": 60.74, "end": 64.8, "text": " feature instead of the ear shape feature."}, {"start": 64.8, "end": 70.52000000000001, "text": " As a result of that, the subsets of data you get in the left and right subtrees become"}, {"start": 70.52000000000001, "end": 77.28, "text": " totally different and as you continue to run the decision tree learning algorithm recursively,"}, {"start": 77.28, "end": 81.08, "text": " you build out totally different subtrees on the left and right."}, {"start": 81.08, "end": 87.3, "text": " So the fact that changing just one training example causes the algorithm to come up with"}, {"start": 87.3, "end": 92.52, "text": " a different split at the root and therefore a totally different tree, that makes this"}, {"start": 92.52, "end": 96.36, "text": " algorithm just not that robust."}, {"start": 96.36, "end": 100.92, "text": " That's why when you're using decision trees, you often get a much better result, that is"}, {"start": 100.92, "end": 106.64, "text": " you get more accurate predictions if you train not just a single decision tree, but a whole"}, {"start": 106.64, "end": 109.18, "text": " bunch of different decision trees."}, {"start": 109.18, "end": 113.80000000000001, "text": " And this is what we call a tree ensemble, which just means a collection of multiple"}, {"start": 113.80000000000001, "end": 114.80000000000001, "text": " trees."}, {"start": 114.80000000000001, "end": 121.84, "text": " We'll see in the next few videos how to construct this ensemble of trees, but if you had this"}, {"start": 121.84, "end": 128.06, "text": " ensemble of three trees, each one of these is maybe a plausible way to classify cat versus"}, {"start": 128.06, "end": 129.56, "text": " not cat."}, {"start": 129.56, "end": 135.4, "text": " If you had a new test example that you wanted to classify, then what you would do is run"}, {"start": 135.4, "end": 141.64000000000001, "text": " all three of these trees on your new example and get them to vote on what is the final"}, {"start": 141.64000000000001, "end": 142.68, "text": " prediction."}, {"start": 142.68, "end": 148.84, "text": " So this test example has pointy ears, a not round face shape and whiskers are present."}, {"start": 148.84, "end": 155.32, "text": " And so the first tree would carry an inference like this and predict that it is a cat."}, {"start": 155.32, "end": 161.16, "text": " The second tree's inference would follow this path through the tree and therefore predict"}, {"start": 161.16, "end": 164.32, "text": " that it's not a cat."}, {"start": 164.32, "end": 171.22, "text": " And the third tree would follow this path and therefore predict that it is a cat."}, {"start": 171.22, "end": 174.12, "text": " These three trees have made different predictions."}, {"start": 174.12, "end": 178.4, "text": " And so what we'll do is actually get them to vote and the majority vote of the predictions"}, {"start": 178.4, "end": 181.22, "text": " among these three trees is cat."}, {"start": 181.22, "end": 186.4, "text": " So the final prediction of this ensemble of trees is that this is a cat, which happens"}, {"start": 186.4, "end": 188.22, "text": " to be the correct prediction."}, {"start": 188.22, "end": 194.26, "text": " So the reason we use an ensemble of trees is by having lots of decision trees and having"}, {"start": 194.26, "end": 200.46, "text": " them vote, it makes your overall algorithm less sensitive to what any single tree may"}, {"start": 200.46, "end": 205.72, "text": " be doing because it gets only one vote out of three or one vote out of many many different"}, {"start": 205.72, "end": 209.66, "text": " votes and it makes your overall algorithm more robust."}, {"start": 209.66, "end": 214.98, "text": " But how do you come up with all of these different plausible but maybe slightly different decision"}, {"start": 214.98, "end": 217.64, "text": " trees in order to get them to vote?"}, {"start": 217.64, "end": 224.05999999999997, "text": " In the next video, we'll talk about a technique from statistics called sampling with replacement."}, {"start": 224.05999999999997, "end": 228.6, "text": " And this will turn out to be a key technique that we'll use in the video after that in"}, {"start": 228.6, "end": 231.51999999999998, "text": " order to build this ensemble of trees."}, {"start": 231.52, "end": 248.32000000000002, "text": " So let's go on to the next video to talk about sampling with replacement."}]