3.10 Regularization to Reduce Overfitting | Regularized linear regression-- [ML | Andrew Ng]
 In this video, we'll figure out how to get gradient descent to work with regularized linear regression. Let's jump in. Here's the cost function we've come up with in the last video for regularized linear regression. The first part is the usual squared error cost function. And now you have this additional regularization term, where lambda is the regularization parameter. And you'd like to find parameters w and b that minimize the regularized cost function. Previously, we were using gradient descent for the original cost function, just the first term, before we added that second regularization term. And previously, we had the following gradient descent algorithm, which is that we repeatedly update the parameters wj and b for j equals 1 through n according to this formula, and b is also updated similarly. Again, alpha is a very small positive number called the learning rate. In fact, the updates for regularized linear regression look exactly the same, except that now the cost j is defined a bit differently. Previously, the derivative of j with respect to wj was given by this expression over here, and the derivative with respect to b was given by this expression over here. Now that we've added this additional regularization term, the only thing that changes is that the expression for the derivative with respect to wj ends up with one additional term. This plus lambda over m times wj. And in particular, for the new definition of the cost function j, these two expressions over here, these are the new derivatives of j with respect to wj, and the derivative of j with respect to b. Recall that we don't regularize b, so we're not trying to shrink b. That's why the update to b remains the same as before, whereas the update to w changes because the regularization term causes us to try to shrink wj. So let's take these definitions for the derivatives and put them back into the expression on the left to write out the gradient descent algorithm for regularized linear regression. So to implement gradient descent for regularized linear regression, this is what you would have your code do. Here's the update for wj for j equals one through n, and here's the update for b. And as usual, please remember to carry out simultaneous updates for all of these parameters. Now, in order for you to get this algorithm to work, this is all you need to know. But what I'd like to do in the remainder of this video is to go over some optional material to convey a slightly deeper intuition about what this formula is actually doing, as well as chat briefly about how these derivatives are derived. The rest of this video is completely optional. It's completely okay if you skip the rest of this video. But if you have a strong interest in math, then stick with me. It's always nice to hang out with you here. And through these equations, perhaps you can build a deeper intuition about what the math and what the derivatives are doing as well. So let's take a look. Let's look at the update rule for wj and rewrite it in another way. We're updating wj as one times wj minus alpha times lambda over m times wj. So I've moved the term from the end to the front here. And then minus alpha times one over m and then the rest of that term over there. So we just rearrange the terms a little bit. And if we simplify, then we're seeing that wj is updated as wj times one minus alpha times lambda over m minus alpha times this other term over here. And you might recognize this second term as the usual gradient descent update for unregularized linear regression. This is the update for linear regression before we had regularization. And this is the term we saw in week two of this course. And so the only change when you have regularization is that instead of wj being set to be equal to wj minus alpha times this term is now w times this number minus the usual update. So this is what we had in week one of this course. So what is this first term over here? Well alpha is a very small positive number, say 0.01. Lambda is usually a small number, say one or maybe 10. Let's say lambda is one for this example, and m is the training set size, say 50. And so when you multiply alpha lambda over m, say 0.01 times one divided by 50, this term ends up being a small positive number, say 0.0002. And thus, one minus alpha lambda over m is going to be a number just slightly less than one. So this is wj's 0.9998. And so the effect of this term is that on every single iteration of gradient descent, you're taking wj and multiplying it by 0.9998. That is by some numbers slightly less than one before carrying out the usual update. So what regularization is doing on every single iteration is you're multiplying w by a number slightly less than one, and that has the effect of shrinking the value of wj just a little bit. So this gives us another view on why regularization has the effect of shrinking the parameters wj a little bit on every iteration. And so that's how regularization works. If you're curious about how these derivative terms were computed, I have just one last optional slide that goes through just a little bit of the calculation of the derivative term. Again, this slide and the rest of this video are completely optional, meaning you won't need any of this to do the practice labs and the quizzes. So let's step through quickly the derivative calculation. The derivative of j with respect to wj looks like this. Recall that f of x for linear regression is defined as w dot x plus b or w dot product x plus b. And it turns out that by the rules of calculus, the derivatives look like this is one over two m times the sum i equals one through m of w dot x plus b minus y times two xj plus the derivative of the regularization term, which is lambda over two m times two wj. Notice that the second term does not have the summation term from j equals one through n anymore. The twos cancel out here and here and also here and here. And so it simplifies to this expression over here. And finally, remember that wx plus b is f of x. And so you can rewrite it as this expression down here. So this is why this expression is used to compute the gradient in regularized linear regression. So you now know how to implement regularized linear regression. Using this, you will reduce overfitting when you have a lot of features and relatively small training set. And this should let you get linear regression to work much better on many problems. In the next video, we'll take this regularization idea and apply it to logistic regression to avoid overfitting for logistic regression as well. Let's take a look at that in the next video.
[{"start": 0.0, "end": 6.140000000000001, "text": " In this video, we'll figure out how to get gradient descent to work with regularized"}, {"start": 6.140000000000001, "end": 7.140000000000001, "text": " linear regression."}, {"start": 7.140000000000001, "end": 8.14, "text": " Let's jump in."}, {"start": 8.14, "end": 14.66, "text": " Here's the cost function we've come up with in the last video for regularized linear regression."}, {"start": 14.66, "end": 18.84, "text": " The first part is the usual squared error cost function."}, {"start": 18.84, "end": 25.68, "text": " And now you have this additional regularization term, where lambda is the regularization parameter."}, {"start": 25.68, "end": 32.48, "text": " And you'd like to find parameters w and b that minimize the regularized cost function."}, {"start": 32.48, "end": 38.72, "text": " Previously, we were using gradient descent for the original cost function, just the first"}, {"start": 38.72, "end": 43.44, "text": " term, before we added that second regularization term."}, {"start": 43.44, "end": 49.28, "text": " And previously, we had the following gradient descent algorithm, which is that we repeatedly"}, {"start": 49.28, "end": 56.4, "text": " update the parameters wj and b for j equals 1 through n according to this formula, and"}, {"start": 56.4, "end": 58.84, "text": " b is also updated similarly."}, {"start": 58.84, "end": 64.28, "text": " Again, alpha is a very small positive number called the learning rate."}, {"start": 64.28, "end": 69.48, "text": " In fact, the updates for regularized linear regression look exactly the same, except that"}, {"start": 69.48, "end": 73.8, "text": " now the cost j is defined a bit differently."}, {"start": 73.8, "end": 81.39999999999999, "text": " Previously, the derivative of j with respect to wj was given by this expression over here,"}, {"start": 81.39999999999999, "end": 87.36, "text": " and the derivative with respect to b was given by this expression over here."}, {"start": 87.36, "end": 92.96, "text": " Now that we've added this additional regularization term, the only thing that changes is that"}, {"start": 92.96, "end": 99.24, "text": " the expression for the derivative with respect to wj ends up with one additional term."}, {"start": 99.24, "end": 104.36, "text": " This plus lambda over m times wj."}, {"start": 104.36, "end": 109.39999999999999, "text": " And in particular, for the new definition of the cost function j, these two expressions"}, {"start": 109.39999999999999, "end": 116.75999999999999, "text": " over here, these are the new derivatives of j with respect to wj, and the derivative of"}, {"start": 116.75999999999999, "end": 119.16, "text": " j with respect to b."}, {"start": 119.16, "end": 124.56, "text": " Recall that we don't regularize b, so we're not trying to shrink b."}, {"start": 124.56, "end": 130.24, "text": " That's why the update to b remains the same as before, whereas the update to w changes"}, {"start": 130.24, "end": 136.84, "text": " because the regularization term causes us to try to shrink wj."}, {"start": 136.84, "end": 142.12, "text": " So let's take these definitions for the derivatives and put them back into the expression on the"}, {"start": 142.12, "end": 149.16, "text": " left to write out the gradient descent algorithm for regularized linear regression."}, {"start": 149.16, "end": 155.2, "text": " So to implement gradient descent for regularized linear regression, this is what you would"}, {"start": 155.2, "end": 157.68, "text": " have your code do."}, {"start": 157.68, "end": 162.92, "text": " Here's the update for wj for j equals one through n, and here's the update for b."}, {"start": 162.92, "end": 169.84, "text": " And as usual, please remember to carry out simultaneous updates for all of these parameters."}, {"start": 169.84, "end": 175.92, "text": " Now, in order for you to get this algorithm to work, this is all you need to know."}, {"start": 175.92, "end": 180.67999999999998, "text": " But what I'd like to do in the remainder of this video is to go over some optional material"}, {"start": 180.67999999999998, "end": 186.07999999999998, "text": " to convey a slightly deeper intuition about what this formula is actually doing, as well"}, {"start": 186.07999999999998, "end": 190.23999999999998, "text": " as chat briefly about how these derivatives are derived."}, {"start": 190.23999999999998, "end": 192.39999999999998, "text": " The rest of this video is completely optional."}, {"start": 192.39999999999998, "end": 196.92, "text": " It's completely okay if you skip the rest of this video."}, {"start": 196.92, "end": 199.88, "text": " But if you have a strong interest in math, then stick with me."}, {"start": 199.88, "end": 202.32, "text": " It's always nice to hang out with you here."}, {"start": 202.32, "end": 207.4, "text": " And through these equations, perhaps you can build a deeper intuition about what the math"}, {"start": 207.4, "end": 209.84, "text": " and what the derivatives are doing as well."}, {"start": 209.84, "end": 212.51999999999998, "text": " So let's take a look."}, {"start": 212.51999999999998, "end": 217.76, "text": " Let's look at the update rule for wj and rewrite it in another way."}, {"start": 217.76, "end": 229.51999999999998, "text": " We're updating wj as one times wj minus alpha times lambda over m times wj."}, {"start": 229.52, "end": 233.44, "text": " So I've moved the term from the end to the front here."}, {"start": 233.44, "end": 241.52, "text": " And then minus alpha times one over m and then the rest of that term over there."}, {"start": 241.52, "end": 244.78, "text": " So we just rearrange the terms a little bit."}, {"start": 244.78, "end": 253.28, "text": " And if we simplify, then we're seeing that wj is updated as wj times one minus alpha"}, {"start": 253.28, "end": 260.16, "text": " times lambda over m minus alpha times this other term over here."}, {"start": 260.16, "end": 266.52, "text": " And you might recognize this second term as the usual gradient descent update for unregularized"}, {"start": 266.52, "end": 268.28, "text": " linear regression."}, {"start": 268.28, "end": 272.84, "text": " This is the update for linear regression before we had regularization."}, {"start": 272.84, "end": 277.56, "text": " And this is the term we saw in week two of this course."}, {"start": 277.56, "end": 283.24, "text": " And so the only change when you have regularization is that instead of wj being set to be equal"}, {"start": 283.24, "end": 293.52, "text": " to wj minus alpha times this term is now w times this number minus the usual update."}, {"start": 293.52, "end": 296.62, "text": " So this is what we had in week one of this course."}, {"start": 296.62, "end": 300.04, "text": " So what is this first term over here?"}, {"start": 300.04, "end": 304.24, "text": " Well alpha is a very small positive number, say 0.01."}, {"start": 304.24, "end": 311.44, "text": " Lambda is usually a small number, say one or maybe 10."}, {"start": 311.44, "end": 318.32, "text": " Let's say lambda is one for this example, and m is the training set size, say 50."}, {"start": 318.32, "end": 328.36, "text": " And so when you multiply alpha lambda over m, say 0.01 times one divided by 50, this"}, {"start": 328.36, "end": 335.04, "text": " term ends up being a small positive number, say 0.0002."}, {"start": 335.04, "end": 340.36, "text": " And thus, one minus alpha lambda over m is going to be a number just slightly less than"}, {"start": 340.36, "end": 341.36, "text": " one."}, {"start": 341.36, "end": 344.12, "text": " So this is wj's 0.9998."}, {"start": 344.12, "end": 349.6, "text": " And so the effect of this term is that on every single iteration of gradient descent,"}, {"start": 349.6, "end": 354.88, "text": " you're taking wj and multiplying it by 0.9998."}, {"start": 354.88, "end": 360.3, "text": " That is by some numbers slightly less than one before carrying out the usual update."}, {"start": 360.3, "end": 366.24, "text": " So what regularization is doing on every single iteration is you're multiplying w by a number"}, {"start": 366.24, "end": 372.24, "text": " slightly less than one, and that has the effect of shrinking the value of wj just a little"}, {"start": 372.24, "end": 373.48, "text": " bit."}, {"start": 373.48, "end": 379.08, "text": " So this gives us another view on why regularization has the effect of shrinking the parameters"}, {"start": 379.08, "end": 382.32, "text": " wj a little bit on every iteration."}, {"start": 382.32, "end": 385.84000000000003, "text": " And so that's how regularization works."}, {"start": 385.84000000000003, "end": 390.40000000000003, "text": " If you're curious about how these derivative terms were computed, I have just one last"}, {"start": 390.40000000000003, "end": 396.12, "text": " optional slide that goes through just a little bit of the calculation of the derivative term."}, {"start": 396.12, "end": 401.0, "text": " Again, this slide and the rest of this video are completely optional, meaning you won't"}, {"start": 401.0, "end": 405.68, "text": " need any of this to do the practice labs and the quizzes."}, {"start": 405.68, "end": 409.32, "text": " So let's step through quickly the derivative calculation."}, {"start": 409.32, "end": 426.84, "text": " The derivative of j with respect to wj looks like this."}, {"start": 426.84, "end": 434.76, "text": " Recall that f of x for linear regression is defined as w dot x plus b or w dot product"}, {"start": 434.76, "end": 436.64, "text": " x plus b."}, {"start": 436.64, "end": 443.36, "text": " And it turns out that by the rules of calculus, the derivatives look like this is one over"}, {"start": 443.36, "end": 456.84, "text": " two m times the sum i equals one through m of w dot x plus b minus y times two xj plus"}, {"start": 456.84, "end": 467.2, "text": " the derivative of the regularization term, which is lambda over two m times two wj."}, {"start": 467.2, "end": 472.35999999999996, "text": " Notice that the second term does not have the summation term from j equals one through"}, {"start": 472.35999999999996, "end": 473.96, "text": " n anymore."}, {"start": 473.96, "end": 480.28, "text": " The twos cancel out here and here and also here and here."}, {"start": 480.28, "end": 488.03999999999996, "text": " And so it simplifies to this expression over here."}, {"start": 488.03999999999996, "end": 493.0, "text": " And finally, remember that wx plus b is f of x."}, {"start": 493.0, "end": 498.0, "text": " And so you can rewrite it as this expression down here."}, {"start": 498.0, "end": 503.35999999999996, "text": " So this is why this expression is used to compute the gradient in regularized linear"}, {"start": 503.35999999999996, "end": 505.2, "text": " regression."}, {"start": 505.2, "end": 509.64, "text": " So you now know how to implement regularized linear regression."}, {"start": 509.64, "end": 513.64, "text": " Using this, you will reduce overfitting when you have a lot of features and relatively"}, {"start": 513.64, "end": 515.6, "text": " small training set."}, {"start": 515.6, "end": 520.96, "text": " And this should let you get linear regression to work much better on many problems."}, {"start": 520.96, "end": 526.52, "text": " In the next video, we'll take this regularization idea and apply it to logistic regression to"}, {"start": 526.52, "end": 529.4399999999999, "text": " avoid overfitting for logistic regression as well."}, {"start": 529.44, "end": 540.12, "text": " Let's take a look at that in the next video."}]