2.5 Practical Tips for Linear Regression | Feature scaling part 1-- [Machine Learning | Andrew Ng]
 So, welcome back. Let's take a look at some techniques that will make gradient descent work much better. In this video, you see a technique called feature scaling that will enable gradient descent to run much faster. Let's start by taking a look at the relationship between the size of a feature, that is, how big are the numbers for that feature, and the size of its associated parameter. As a concrete example, let's predict the price of a house using two features, x1, the size of the house, and x2, the number of bedrooms. Let's say that x1 typically ranges from 300 to 2000 square feet, and x2, in the dataset, ranges from 0 to 5 bedrooms. So for this example, x1 takes on a relatively large range of values, and x2 takes on a relatively small range of values. Now let's take an example of a house that has a size of 2000 square feet, has 5 bedrooms, and a price of 500k or $500,000. For this one trading example, what do you think are reasonable values for the size of parameters w1 and w2? Well, let's look at one possible set of parameters. Say w1 is 50, and w2 is 0.1, and b is 50 for the purposes of discussion. So in this case, the estimated price in thousands of dollars is 100,000k here, plus 0.5k, plus 50k, which is slightly over $100 million. So that's clearly very far from the actual price of $500,000. And so this is not a very good set of parameter choices for w1 and w2. Now let's take a look at another possibility. Say w1 and w2 were the other way around. w1 is 0.1, and w2 is 50, and b is still also 50. In this choice of w1 and w2, w1 is relatively small and w2 is relatively large. 50 is much bigger than 0.1. So here the predicted price is 0.1 times 2000 plus 50 times 5 plus 50. The first term becomes 200k, the second term becomes 250k, and then plus 50. So this version of the model predicts a price of $500,000, which is a much more reasonable estimate and happens to be the same price as the true price of the house. So hopefully you might notice that when a possible range of values of a feature is large, like the size in square feet, which goes all the way up to 2000, it's more likely that a good model will learn to choose a relatively small parameter value like 0.1. Likewise, when the possible values of a feature are small, like the number of bedrooms, then a reasonable value for its parameters will be relatively large, like 50. So how does this relate to gradient descent? Well, let's take a look at a scatter plot of the features, where the size in square feet is the horizontal axis, x1, and the number of bedrooms, x2, is on the vertical axis. If you plot the training data, you notice that the horizontal axis is on a much larger scale or much larger range of values compared to the vertical axis. Next, let's look at how the cost function might look in a contour plot. You might see a contour plot where the horizontal axis has a much narrower range, say between 0 and 1, whereas the vertical axis takes on much larger values, say between 10 and 100. So the contours form ovals or ellipses, and they're shorter on one side and longer on the other. And this is because a very small change to w1 can have a very large impact on the estimated price and thus a very large impact on the cost j, because w1 tends to be multiplied by a very large number, the size in square feet. In contrast, it takes a much larger change in w2 in order to change the predictions much, and thus small changes to w2 don't change the cost function nearly as much. So where does this leave us? This is what might end up happening if you were to run gradient descent, if you were to use your training data as is. Because the contours are so tall and skinny, gradient descent may end up bouncing back and forth for a long time before it can finally find its way to the global minimum. In situations like this, a useful thing to do is to scale the features. This means performing some transformation of your training data so that x1, say, might now range from 0 to 1, and x2 might also range from 0 to 1. So the data points now look more like this, and you might notice that the scale of the plot on the bottom is now quite different than the one on top. The key point is that the rescaled x1 and x2 are both now taking comparable ranges of values to each other. And if you run gradient descent on a cost function defined on this rescaled x1 and x2 using this transformed data, then the contours will look more like this, more like circles and less tall and skinny, and gradient descent can find a much more direct path to the global minimum. So to recap, when you have different features that take on very different ranges of values, you can cause gradient descent to run slowly, but rescaling the different features so they all take on a comparable range of values can speed up gradient descent significantly. How do you actually do this? Let's take a look at that in the next video.
[{"start": 0.0, "end": 4.28, "text": " So, welcome back."}, {"start": 4.28, "end": 8.8, "text": " Let's take a look at some techniques that will make gradient descent work much better."}, {"start": 8.8, "end": 13.36, "text": " In this video, you see a technique called feature scaling that will enable gradient"}, {"start": 13.36, "end": 16.240000000000002, "text": " descent to run much faster."}, {"start": 16.240000000000002, "end": 20.96, "text": " Let's start by taking a look at the relationship between the size of a feature, that is, how"}, {"start": 20.96, "end": 27.1, "text": " big are the numbers for that feature, and the size of its associated parameter."}, {"start": 27.1, "end": 33.32, "text": " As a concrete example, let's predict the price of a house using two features, x1, the size"}, {"start": 33.32, "end": 37.52, "text": " of the house, and x2, the number of bedrooms."}, {"start": 37.52, "end": 45.28, "text": " Let's say that x1 typically ranges from 300 to 2000 square feet, and x2, in the dataset,"}, {"start": 45.28, "end": 48.44, "text": " ranges from 0 to 5 bedrooms."}, {"start": 48.44, "end": 55.72, "text": " So for this example, x1 takes on a relatively large range of values, and x2 takes on a relatively"}, {"start": 55.72, "end": 58.4, "text": " small range of values."}, {"start": 58.4, "end": 66.88, "text": " Now let's take an example of a house that has a size of 2000 square feet, has 5 bedrooms,"}, {"start": 66.88, "end": 72.03999999999999, "text": " and a price of 500k or $500,000."}, {"start": 72.03999999999999, "end": 76.4, "text": " For this one trading example, what do you think are reasonable values for the size of"}, {"start": 76.4, "end": 79.88, "text": " parameters w1 and w2?"}, {"start": 79.88, "end": 84.0, "text": " Well, let's look at one possible set of parameters."}, {"start": 84.0, "end": 94.32, "text": " Say w1 is 50, and w2 is 0.1, and b is 50 for the purposes of discussion."}, {"start": 94.32, "end": 104.6, "text": " So in this case, the estimated price in thousands of dollars is 100,000k here, plus 0.5k, plus"}, {"start": 104.6, "end": 110.52, "text": " 50k, which is slightly over $100 million."}, {"start": 110.52, "end": 117.96, "text": " So that's clearly very far from the actual price of $500,000."}, {"start": 117.96, "end": 123.75999999999999, "text": " And so this is not a very good set of parameter choices for w1 and w2."}, {"start": 123.75999999999999, "end": 127.08, "text": " Now let's take a look at another possibility."}, {"start": 127.08, "end": 131.28, "text": " Say w1 and w2 were the other way around."}, {"start": 131.28, "end": 137.48, "text": " w1 is 0.1, and w2 is 50, and b is still also 50."}, {"start": 137.48, "end": 145.48, "text": " In this choice of w1 and w2, w1 is relatively small and w2 is relatively large."}, {"start": 145.48, "end": 148.51999999999998, "text": " 50 is much bigger than 0.1."}, {"start": 148.51999999999998, "end": 158.39999999999998, "text": " So here the predicted price is 0.1 times 2000 plus 50 times 5 plus 50."}, {"start": 158.39999999999998, "end": 165.6, "text": " The first term becomes 200k, the second term becomes 250k, and then plus 50."}, {"start": 165.6, "end": 171.6, "text": " So this version of the model predicts a price of $500,000, which is a much more reasonable"}, {"start": 171.6, "end": 176.78, "text": " estimate and happens to be the same price as the true price of the house."}, {"start": 176.78, "end": 182.79999999999998, "text": " So hopefully you might notice that when a possible range of values of a feature is large,"}, {"start": 182.79999999999998, "end": 187.64, "text": " like the size in square feet, which goes all the way up to 2000, it's more likely that"}, {"start": 187.64, "end": 194.0, "text": " a good model will learn to choose a relatively small parameter value like 0.1."}, {"start": 194.0, "end": 200.92, "text": " Likewise, when the possible values of a feature are small, like the number of bedrooms, then"}, {"start": 200.92, "end": 206.68, "text": " a reasonable value for its parameters will be relatively large, like 50."}, {"start": 206.68, "end": 210.0, "text": " So how does this relate to gradient descent?"}, {"start": 210.0, "end": 216.88, "text": " Well, let's take a look at a scatter plot of the features, where the size in square"}, {"start": 216.88, "end": 223.96, "text": " feet is the horizontal axis, x1, and the number of bedrooms, x2, is on the vertical axis."}, {"start": 223.96, "end": 229.72, "text": " If you plot the training data, you notice that the horizontal axis is on a much larger"}, {"start": 229.72, "end": 234.72, "text": " scale or much larger range of values compared to the vertical axis."}, {"start": 234.72, "end": 241.24, "text": " Next, let's look at how the cost function might look in a contour plot."}, {"start": 241.24, "end": 248.16, "text": " You might see a contour plot where the horizontal axis has a much narrower range, say between"}, {"start": 248.16, "end": 256.2, "text": " 0 and 1, whereas the vertical axis takes on much larger values, say between 10 and 100."}, {"start": 256.2, "end": 262.4, "text": " So the contours form ovals or ellipses, and they're shorter on one side and longer on"}, {"start": 262.4, "end": 264.0, "text": " the other."}, {"start": 264.0, "end": 270.28, "text": " And this is because a very small change to w1 can have a very large impact on the estimated"}, {"start": 270.28, "end": 277.36, "text": " price and thus a very large impact on the cost j, because w1 tends to be multiplied"}, {"start": 277.36, "end": 281.76, "text": " by a very large number, the size in square feet."}, {"start": 281.76, "end": 288.40000000000003, "text": " In contrast, it takes a much larger change in w2 in order to change the predictions much,"}, {"start": 288.40000000000003, "end": 294.86, "text": " and thus small changes to w2 don't change the cost function nearly as much."}, {"start": 294.86, "end": 296.88, "text": " So where does this leave us?"}, {"start": 296.88, "end": 301.64, "text": " This is what might end up happening if you were to run gradient descent, if you were"}, {"start": 301.64, "end": 305.04, "text": " to use your training data as is."}, {"start": 305.04, "end": 311.16, "text": " Because the contours are so tall and skinny, gradient descent may end up bouncing back"}, {"start": 311.16, "end": 317.56, "text": " and forth for a long time before it can finally find its way to the global minimum."}, {"start": 317.56, "end": 322.74, "text": " In situations like this, a useful thing to do is to scale the features."}, {"start": 322.74, "end": 328.8, "text": " This means performing some transformation of your training data so that x1, say, might"}, {"start": 328.8, "end": 335.32, "text": " now range from 0 to 1, and x2 might also range from 0 to 1."}, {"start": 335.32, "end": 340.0, "text": " So the data points now look more like this, and you might notice that the scale of the"}, {"start": 340.0, "end": 345.24, "text": " plot on the bottom is now quite different than the one on top."}, {"start": 345.24, "end": 351.64, "text": " The key point is that the rescaled x1 and x2 are both now taking comparable ranges of"}, {"start": 351.64, "end": 354.44, "text": " values to each other."}, {"start": 354.44, "end": 361.24, "text": " And if you run gradient descent on a cost function defined on this rescaled x1 and x2"}, {"start": 361.24, "end": 367.22, "text": " using this transformed data, then the contours will look more like this, more like circles"}, {"start": 367.22, "end": 373.0, "text": " and less tall and skinny, and gradient descent can find a much more direct path to the global"}, {"start": 373.0, "end": 375.0, "text": " minimum."}, {"start": 375.0, "end": 380.15999999999997, "text": " So to recap, when you have different features that take on very different ranges of values,"}, {"start": 380.16, "end": 385.08000000000004, "text": " you can cause gradient descent to run slowly, but rescaling the different features so they"}, {"start": 385.08000000000004, "end": 390.84000000000003, "text": " all take on a comparable range of values can speed up gradient descent significantly."}, {"start": 390.84000000000003, "end": 392.28000000000003, "text": " How do you actually do this?"}, {"start": 392.28, "end": 410.67999999999995, "text": " Let's take a look at that in the next video."}]