7.7 Decision Tree Learning | Continuous valued features --[Machine Learning | Andrew Ng]
 Let's look at how you can modify a decision tree to work with features that aren't just discrete value but continuous value, that is features that can be any number. Let's start with an example. I have modified the cat adoption center of datasets to add one more feature, which is the weight of the animal in pounds. On average, between cats and dogs, cats are a little bit lighter than dogs, although there are some cats that are heavier than some dogs. But so the weight of an animal is a useful feature for deciding if it is a cat or not. So how do you get a decision tree to use a feature like this? The decision tree learning algorithm will proceed similarly as before, except that rather than considering splitting just on ear shape, face shape, and whiskers, you have to consider splitting on ear shape, face shape, whiskers, or weight. If splitting on the weight feature gives better information gain than the other options, then you will split on the weight feature. But how do you decide how to split on the weight feature? Let's take a look. Here's a plot of the data at the root node. I've plotted on the horizontal axis, the weight of the animal, and the vertical axis is cat on top and not cat below. So the vertical axis indicates the label y being 1 or 0. The way we will split on the weight feature would be if we were to split the data based on whether or not the weight is less than or equal to some value, let's say 8 or some of the number, that will be the job of the learning algorithm to choose. What we should do when considering splitting on the weight feature is to consider many different values of this threshold and then to pick the one that is the best. By the best, I mean the one that results in the best information gain. So in particular, if you were considering splitting the examples based on whether the weight is less than or equal to 8, then you would be splitting this data set into two subsets, where the subset on the left has two cats and the subset on the right has three cats and five dogs. So if you were to calculate our usual information gain calculation, you'll be computing the entropy at the root node, entropy of 0.5 minus now two-tenths times entropy of the left split has two other two cats, so entropy of two other two plus the right split has eight out of 10 examples, and then entropy of the eight examples on the right, three are cats, entropy of three-eighths, and this turns out to be 0.24. So this would be information gain if you were to split on whether the weight is less than or equal to 8. But we should try other values as well. So what if you were to split on whether or not the weight is less than or equal to 9? And that corresponds to this new line over here, and the information gain calculation becomes h of 0.5 minus, so now we have four examples in the left split, all cats, so that's four out of 10 times entropy of four out of four plus six examples on the right, of which you have one cat, so that's six-tenths times h of one-sixth, which is equal to, turns out, 0.61. So the information gain here looks much better, it's 0.61 information gain, which is much higher than 0.24. Or we could try another value, say 13, and the calculation turns out to look like this, which is 0.40. In the more general case, we'll actually try not just three values, but multiple values along the x-axis. And one convention would be to sort all of the examples according to the weight or according to the value on this feature, and take all the values that are the midpoints between the sorted list of training examples as the values for consideration for this threshold over here. This way, if you have 10 training examples, you will test nine different possible values for this threshold, and then try to pick the one that gives you the highest information gain. And finally, if the information gain from splitting on a given value of this threshold is better than the information gain from splitting on any other feature, then you will decide to split that node at that feature. And in this example, an information gain of 0.61 turns out to be higher than that of any other feature. It turns out there are actually two thresholds. And so, assuming the algorithm chooses this feature to split on, you would end up splitting the data set according to whether or not the weight of the animal is less than or equal to nine pounds. And so you end up with two subsets of the data like this, and you can then build recursively additional decision trees using these two subsets of the data to build out the rest of the tree. So to summarize, to get the decision tree to work on continuous value features at every node, when considering splits, you would just consider different values to split on, carry out the usual information gain calculation, and decide to split on that continuous value feature if it gives the highest possible information gain. So that's how you get a decision tree to work with continuous value features. Try different thresholds, do the usual information gain calculation, and split on the continuous value feature with the selected threshold if it gives you the best possible information gain of all possible features to split on. And that's it for the required videos on the core decision tree algorithm. After this, there is an optional video that you can watch or not that generalizes the decision tree learning algorithm to regression trees. So far, we've only talked about using decision trees to make predictions that are classifications, predicting a discrete category such as cat or not cat. But what if you have a regression problem where you want to predict a number? In the next video, I'll talk about a generalization of decision trees to handle that.
[{"start": 0.0, "end": 3.96, "text": " Let's look at how you can modify a decision tree to work with"}, {"start": 3.96, "end": 7.5, "text": " features that aren't just discrete value but continuous value,"}, {"start": 7.5, "end": 10.16, "text": " that is features that can be any number."}, {"start": 10.16, "end": 12.120000000000001, "text": " Let's start with an example."}, {"start": 12.120000000000001, "end": 17.8, "text": " I have modified the cat adoption center of datasets to add one more feature,"}, {"start": 17.8, "end": 21.32, "text": " which is the weight of the animal in pounds."}, {"start": 21.32, "end": 24.92, "text": " On average, between cats and dogs,"}, {"start": 24.92, "end": 28.400000000000002, "text": " cats are a little bit lighter than dogs,"}, {"start": 28.4, "end": 31.119999999999997, "text": " although there are some cats that are heavier than some dogs."}, {"start": 31.119999999999997, "end": 37.72, "text": " But so the weight of an animal is a useful feature for deciding if it is a cat or not."}, {"start": 37.72, "end": 42.58, "text": " So how do you get a decision tree to use a feature like this?"}, {"start": 42.58, "end": 46.68, "text": " The decision tree learning algorithm will proceed similarly as before,"}, {"start": 46.68, "end": 50.56, "text": " except that rather than considering splitting just on ear shape,"}, {"start": 50.56, "end": 51.879999999999995, "text": " face shape, and whiskers,"}, {"start": 51.879999999999995, "end": 53.84, "text": " you have to consider splitting on ear shape,"}, {"start": 53.84, "end": 56.480000000000004, "text": " face shape, whiskers, or weight."}, {"start": 56.48, "end": 59.559999999999995, "text": " If splitting on the weight feature gives"}, {"start": 59.559999999999995, "end": 62.08, "text": " better information gain than the other options,"}, {"start": 62.08, "end": 64.39999999999999, "text": " then you will split on the weight feature."}, {"start": 64.39999999999999, "end": 70.36, "text": " But how do you decide how to split on the weight feature? Let's take a look."}, {"start": 70.36, "end": 74.19999999999999, "text": " Here's a plot of the data at the root node."}, {"start": 74.19999999999999, "end": 76.47999999999999, "text": " I've plotted on the horizontal axis,"}, {"start": 76.47999999999999, "end": 77.82, "text": " the weight of the animal,"}, {"start": 77.82, "end": 83.19999999999999, "text": " and the vertical axis is cat on top and not cat below."}, {"start": 83.2, "end": 88.12, "text": " So the vertical axis indicates the label y being 1 or 0."}, {"start": 88.12, "end": 93.4, "text": " The way we will split on the weight feature would be if we were to split the data"}, {"start": 93.4, "end": 97.88, "text": " based on whether or not the weight is less than or equal to some value,"}, {"start": 97.88, "end": 100.60000000000001, "text": " let's say 8 or some of the number,"}, {"start": 100.60000000000001, "end": 103.88, "text": " that will be the job of the learning algorithm to choose."}, {"start": 103.88, "end": 110.64, "text": " What we should do when considering splitting on the weight feature is to"}, {"start": 110.64, "end": 117.24, "text": " consider many different values of this threshold and then to pick the one that is the best."}, {"start": 117.24, "end": 122.44, "text": " By the best, I mean the one that results in the best information gain."}, {"start": 122.44, "end": 127.68, "text": " So in particular, if you were considering splitting the examples"}, {"start": 127.68, "end": 131.6, "text": " based on whether the weight is less than or equal to 8,"}, {"start": 131.6, "end": 135.8, "text": " then you would be splitting this data set into two subsets,"}, {"start": 135.8, "end": 144.16000000000003, "text": " where the subset on the left has two cats and the subset on the right has three cats and five dogs."}, {"start": 144.16000000000003, "end": 150.4, "text": " So if you were to calculate our usual information gain calculation,"}, {"start": 150.4, "end": 154.04000000000002, "text": " you'll be computing the entropy at the root node,"}, {"start": 154.04000000000002, "end": 164.24, "text": " entropy of 0.5 minus now two-tenths times entropy of the left split has two other two cats,"}, {"start": 164.24, "end": 172.36, "text": " so entropy of two other two plus the right split has eight out of 10 examples,"}, {"start": 172.36, "end": 176.24, "text": " and then entropy of the eight examples on the right,"}, {"start": 176.24, "end": 178.96, "text": " three are cats, entropy of three-eighths,"}, {"start": 178.96, "end": 182.36, "text": " and this turns out to be 0.24."}, {"start": 182.36, "end": 189.12, "text": " So this would be information gain if you were to split on whether the weight is less than or equal to 8."}, {"start": 189.12, "end": 191.92000000000002, "text": " But we should try other values as well."}, {"start": 191.92, "end": 198.23999999999998, "text": " So what if you were to split on whether or not the weight is less than or equal to 9?"}, {"start": 198.23999999999998, "end": 203.6, "text": " And that corresponds to this new line over here,"}, {"start": 203.6, "end": 210.04, "text": " and the information gain calculation becomes h of 0.5 minus,"}, {"start": 210.04, "end": 213.64, "text": " so now we have four examples in the left split, all cats,"}, {"start": 213.64, "end": 220.76, "text": " so that's four out of 10 times entropy of four out of four plus six examples on the right,"}, {"start": 220.76, "end": 226.92, "text": " of which you have one cat, so that's six-tenths times h of one-sixth,"}, {"start": 226.92, "end": 229.84, "text": " which is equal to, turns out, 0.61."}, {"start": 229.84, "end": 232.51999999999998, "text": " So the information gain here looks much better,"}, {"start": 232.51999999999998, "end": 238.84, "text": " it's 0.61 information gain, which is much higher than 0.24."}, {"start": 238.84, "end": 242.04, "text": " Or we could try another value, say 13,"}, {"start": 242.04, "end": 249.12, "text": " and the calculation turns out to look like this, which is 0.40."}, {"start": 249.12, "end": 253.12, "text": " In the more general case, we'll actually try not just three values,"}, {"start": 253.12, "end": 256.72, "text": " but multiple values along the x-axis."}, {"start": 256.72, "end": 261.56, "text": " And one convention would be to sort all of the examples"}, {"start": 261.56, "end": 265.88, "text": " according to the weight or according to the value on this feature,"}, {"start": 265.88, "end": 271.92, "text": " and take all the values that are the midpoints between the sorted list of training examples"}, {"start": 271.92, "end": 276.52, "text": " as the values for consideration for this threshold over here."}, {"start": 276.52, "end": 279.52, "text": " This way, if you have 10 training examples,"}, {"start": 279.52, "end": 283.56, "text": " you will test nine different possible values for this threshold,"}, {"start": 283.56, "end": 288.03999999999996, "text": " and then try to pick the one that gives you the highest information gain."}, {"start": 288.03999999999996, "end": 294.64, "text": " And finally, if the information gain from splitting on a given value of this threshold"}, {"start": 294.64, "end": 298.76, "text": " is better than the information gain from splitting on any other feature,"}, {"start": 298.76, "end": 303.44, "text": " then you will decide to split that node at that feature."}, {"start": 303.44, "end": 310.52, "text": " And in this example, an information gain of 0.61 turns out to be higher than that of any other feature."}, {"start": 310.52, "end": 313.76, "text": " It turns out there are actually two thresholds."}, {"start": 313.76, "end": 318.4, "text": " And so, assuming the algorithm chooses this feature to split on,"}, {"start": 318.4, "end": 323.96, "text": " you would end up splitting the data set according to whether or not"}, {"start": 323.96, "end": 328.4, "text": " the weight of the animal is less than or equal to nine pounds."}, {"start": 328.4, "end": 332.44, "text": " And so you end up with two subsets of the data like this,"}, {"start": 332.44, "end": 337.28, "text": " and you can then build recursively additional decision trees"}, {"start": 337.28, "end": 341.0, "text": " using these two subsets of the data to build out the rest of the tree."}, {"start": 341.0, "end": 347.36, "text": " So to summarize, to get the decision tree to work on continuous value features at every node,"}, {"start": 347.36, "end": 351.6, "text": " when considering splits, you would just consider different values to split on,"}, {"start": 351.6, "end": 354.44, "text": " carry out the usual information gain calculation,"}, {"start": 354.44, "end": 361.88, "text": " and decide to split on that continuous value feature if it gives the highest possible information gain."}, {"start": 361.88, "end": 366.76, "text": " So that's how you get a decision tree to work with continuous value features."}, {"start": 366.76, "end": 370.88, "text": " Try different thresholds, do the usual information gain calculation,"}, {"start": 370.88, "end": 374.36, "text": " and split on the continuous value feature with the selected threshold"}, {"start": 374.36, "end": 380.88, "text": " if it gives you the best possible information gain of all possible features to split on."}, {"start": 380.88, "end": 386.36, "text": " And that's it for the required videos on the core decision tree algorithm."}, {"start": 386.36, "end": 390.12, "text": " After this, there is an optional video that you can watch or not"}, {"start": 390.12, "end": 395.4, "text": " that generalizes the decision tree learning algorithm to regression trees."}, {"start": 395.4, "end": 400.76, "text": " So far, we've only talked about using decision trees to make predictions that are classifications,"}, {"start": 400.76, "end": 404.64, "text": " predicting a discrete category such as cat or not cat."}, {"start": 404.64, "end": 409.04, "text": " But what if you have a regression problem where you want to predict a number?"}, {"start": 409.04, "end": 420.56, "text": " In the next video, I'll talk about a generalization of decision trees to handle that."}]