1.17 Machine Learning Overview | Gradient descent intuition  --[Machine Learning | Andrew Ng]
 Now, let's dive more deeply into gradient descent to gain better intuition about what it's doing and why it might make sense. Here's the gradient descent algorithm that you saw in the previous video. And as a reminder, this variable, this Greek symbol alpha is the learning rate. And the learning rate controls how big of a step you take when updating the model's parameters w and b. So this term here, this d over dw, this is a derivative term. And by convention in math, this d is written with this funny font here. And in case anyone watching this has a PhD in math or is an expert in multivariate calculus, they may be wondering, that's not the derivative, that's the partial derivative. And yes, they'd be right. But for the purposes of implementing a machine learning algorithm, I'm just going to call it derivative, and don't worry about these little distinctions. So what we're going to focus on now is get more intuition about what this learning rate and what this derivative are doing, and why when multiplied together like this, it results in updates to parameters w and b that make sense. In order to do this, let's use a slightly simpler example, where we work on minimizing just one parameter. So let's say that you have a cost function j of just one parameter w, where w is a number. This means that gradient descent now looks like this. w is updated to w minus the learning rate alpha times d over dw of j of w. And you're trying to minimize the cost by adjusting the parameter w. So this is like our previous example, where we had temporarily set b equal to zero. With one parameter w instead of two, you can look at two dimensional graphs of the cost function j instead of three dimensional graphs. Let's look at what gradient descent does on this function j of w. Here on the horizontal axis is parameter w, and on the vertical axis is the cost j of w. Now, let's initialize gradient descent with some starting value for w. Let's initialize it at this location. So imagine that you start off at this point right here on the function j. What gradient descent will do is it will update w to be w minus learning rate alpha times d over dw of j of w. Let's look at what this derivative term here means. A way to think about the derivative at this point on the line is to draw a tangent line, which is a straight line that touches this curve at that point. In math, the slope of this line is the derivative of the function j at this point. And to get the slope, you can draw a little triangle like this. And if you compute the height divided by the width of this triangle, that is the slope. So for example, the slope might be, you know, 2 over 1, for instance. And when the tangent line is pointing up into the right, the slope is positive, which means that this derivative is a positive number, so it's greater than zero. And so the updated w is going to be w minus the learning rate times some positive number. The learning rate is always a positive number. So if you take w minus a positive number, you end up with a new value for w that is smaller. So on the graph, we are moving to the left, we are decreasing the value of w. And you may notice that this is the right thing to do if your goal is to decrease the cost j, because when we move toward the left on this curve, the cost j decreases and you're getting closer to the minimum for j, which is over here. So so far, gradient descent seems to be doing the right thing. Now, let's look at another example. Let's take the same function j of w as above. And now let's say that you initialize gradient descent at a different location, say by choosing a starting value for w that's over here on the left. So that's this point of the function j. Now the derivative term, remember is d over dw of j of w. And when we look at the tangent line at this point over here, the slope of this line is the derivative of j at this point. But this tangent line is sloping down into the right. And so this line sloping down into the right has a negative slope. In other words, the derivative j at this point is a negative number. For instance, if you draw a triangle, then the height like this is negative two and the width is one. So the slope is negative two divided by one, which is negative two, which is a negative number. So when you update w, you get w minus the learning rate times a negative number. And so this means you subtract from w a negative number. But subtracting a negative number means adding a positive number. And so you end up increasing w, because subtracting a negative number is the same as adding a positive number to w. So this step of gradient descent causes w to increase, which means you're moving to the right of the graph, and your cos j has decreased down to here. And again, it looks like gradient descent is doing something reasonable, it's getting you closer to the minimum. So hopefully, these last two examples show some of the intuition behind what the derivative term is doing, and why this helps gradient descent change w to get you closer to the minimum. I hope this video gave you some sense for why the derivative term in gradient descent makes sense. One other key quantity in the gradient descent algorithm is the learning rate alpha. How do you choose alpha? What happens if it's too small? What happens if it's too big? In the next video, let's take a deeper look at the parameter alpha to help build intuitions about what it does, as well as how to make a good choice for a good value of alpha for your implementation of gradient descent.
[{"start": 0.0, "end": 7.88, "text": " Now, let's dive more deeply into gradient descent to gain better intuition about what"}, {"start": 7.88, "end": 10.84, "text": " it's doing and why it might make sense."}, {"start": 10.84, "end": 15.52, "text": " Here's the gradient descent algorithm that you saw in the previous video."}, {"start": 15.52, "end": 21.56, "text": " And as a reminder, this variable, this Greek symbol alpha is the learning rate."}, {"start": 21.56, "end": 25.84, "text": " And the learning rate controls how big of a step you take when updating the model's"}, {"start": 25.84, "end": 28.8, "text": " parameters w and b."}, {"start": 28.8, "end": 35.0, "text": " So this term here, this d over dw, this is a derivative term."}, {"start": 35.0, "end": 40.72, "text": " And by convention in math, this d is written with this funny font here."}, {"start": 40.72, "end": 45.519999999999996, "text": " And in case anyone watching this has a PhD in math or is an expert in multivariate calculus,"}, {"start": 45.519999999999996, "end": 49.400000000000006, "text": " they may be wondering, that's not the derivative, that's the partial derivative."}, {"start": 49.400000000000006, "end": 50.88, "text": " And yes, they'd be right."}, {"start": 50.88, "end": 55.040000000000006, "text": " But for the purposes of implementing a machine learning algorithm, I'm just going to call"}, {"start": 55.04, "end": 60.2, "text": " it derivative, and don't worry about these little distinctions."}, {"start": 60.2, "end": 66.16, "text": " So what we're going to focus on now is get more intuition about what this learning rate"}, {"start": 66.16, "end": 72.72, "text": " and what this derivative are doing, and why when multiplied together like this, it results"}, {"start": 72.72, "end": 77.74, "text": " in updates to parameters w and b that make sense."}, {"start": 77.74, "end": 83.8, "text": " In order to do this, let's use a slightly simpler example, where we work on minimizing"}, {"start": 83.8, "end": 86.56, "text": " just one parameter."}, {"start": 86.56, "end": 94.2, "text": " So let's say that you have a cost function j of just one parameter w, where w is a number."}, {"start": 94.2, "end": 97.88, "text": " This means that gradient descent now looks like this."}, {"start": 97.88, "end": 107.16, "text": " w is updated to w minus the learning rate alpha times d over dw of j of w."}, {"start": 107.16, "end": 112.74, "text": " And you're trying to minimize the cost by adjusting the parameter w."}, {"start": 112.74, "end": 119.44, "text": " So this is like our previous example, where we had temporarily set b equal to zero."}, {"start": 119.44, "end": 124.6, "text": " With one parameter w instead of two, you can look at two dimensional graphs of the cost"}, {"start": 124.6, "end": 128.4, "text": " function j instead of three dimensional graphs."}, {"start": 128.4, "end": 135.4, "text": " Let's look at what gradient descent does on this function j of w."}, {"start": 135.4, "end": 142.4, "text": " Here on the horizontal axis is parameter w, and on the vertical axis is the cost j of"}, {"start": 142.4, "end": 143.4, "text": " w."}, {"start": 143.4, "end": 148.04, "text": " Now, let's initialize gradient descent with some starting value for w."}, {"start": 148.04, "end": 151.04000000000002, "text": " Let's initialize it at this location."}, {"start": 151.04000000000002, "end": 156.56, "text": " So imagine that you start off at this point right here on the function j."}, {"start": 156.56, "end": 164.8, "text": " What gradient descent will do is it will update w to be w minus learning rate alpha times"}, {"start": 164.8, "end": 168.32, "text": " d over dw of j of w."}, {"start": 168.32, "end": 172.08, "text": " Let's look at what this derivative term here means."}, {"start": 172.08, "end": 179.8, "text": " A way to think about the derivative at this point on the line is to draw a tangent line,"}, {"start": 179.8, "end": 184.16000000000003, "text": " which is a straight line that touches this curve at that point."}, {"start": 184.16000000000003, "end": 190.64000000000001, "text": " In math, the slope of this line is the derivative of the function j at this point."}, {"start": 190.64000000000001, "end": 195.08, "text": " And to get the slope, you can draw a little triangle like this."}, {"start": 195.08, "end": 201.46, "text": " And if you compute the height divided by the width of this triangle, that is the slope."}, {"start": 201.46, "end": 207.56, "text": " So for example, the slope might be, you know, 2 over 1, for instance."}, {"start": 207.56, "end": 212.96, "text": " And when the tangent line is pointing up into the right, the slope is positive, which means"}, {"start": 212.96, "end": 219.48000000000002, "text": " that this derivative is a positive number, so it's greater than zero."}, {"start": 219.48000000000002, "end": 227.44, "text": " And so the updated w is going to be w minus the learning rate times some positive number."}, {"start": 227.44, "end": 230.54000000000002, "text": " The learning rate is always a positive number."}, {"start": 230.54, "end": 237.04, "text": " So if you take w minus a positive number, you end up with a new value for w that is"}, {"start": 237.04, "end": 238.7, "text": " smaller."}, {"start": 238.7, "end": 245.84, "text": " So on the graph, we are moving to the left, we are decreasing the value of w."}, {"start": 245.84, "end": 249.79999999999998, "text": " And you may notice that this is the right thing to do if your goal is to decrease the"}, {"start": 249.79999999999998, "end": 256.03999999999996, "text": " cost j, because when we move toward the left on this curve, the cost j decreases and you're"}, {"start": 256.04, "end": 261.04, "text": " getting closer to the minimum for j, which is over here."}, {"start": 261.04, "end": 265.32, "text": " So so far, gradient descent seems to be doing the right thing."}, {"start": 265.32, "end": 268.96000000000004, "text": " Now, let's look at another example."}, {"start": 268.96000000000004, "end": 271.90000000000003, "text": " Let's take the same function j of w as above."}, {"start": 271.90000000000003, "end": 277.72, "text": " And now let's say that you initialize gradient descent at a different location, say by choosing"}, {"start": 277.72, "end": 282.48, "text": " a starting value for w that's over here on the left."}, {"start": 282.48, "end": 285.76, "text": " So that's this point of the function j."}, {"start": 285.76, "end": 293.76, "text": " Now the derivative term, remember is d over dw of j of w."}, {"start": 293.76, "end": 298.44, "text": " And when we look at the tangent line at this point over here, the slope of this line is"}, {"start": 298.44, "end": 301.03999999999996, "text": " the derivative of j at this point."}, {"start": 301.03999999999996, "end": 305.09999999999997, "text": " But this tangent line is sloping down into the right."}, {"start": 305.09999999999997, "end": 309.82, "text": " And so this line sloping down into the right has a negative slope."}, {"start": 309.82, "end": 314.52, "text": " In other words, the derivative j at this point is a negative number."}, {"start": 314.52, "end": 320.24, "text": " For instance, if you draw a triangle, then the height like this is negative two and the"}, {"start": 320.24, "end": 322.03999999999996, "text": " width is one."}, {"start": 322.03999999999996, "end": 327.84, "text": " So the slope is negative two divided by one, which is negative two, which is a negative"}, {"start": 327.84, "end": 329.64, "text": " number."}, {"start": 329.64, "end": 336.47999999999996, "text": " So when you update w, you get w minus the learning rate times a negative number."}, {"start": 336.47999999999996, "end": 341.68, "text": " And so this means you subtract from w a negative number."}, {"start": 341.68, "end": 347.44, "text": " But subtracting a negative number means adding a positive number."}, {"start": 347.44, "end": 354.68, "text": " And so you end up increasing w, because subtracting a negative number is the same as adding a"}, {"start": 354.68, "end": 357.9, "text": " positive number to w."}, {"start": 357.9, "end": 363.68, "text": " So this step of gradient descent causes w to increase, which means you're moving to"}, {"start": 363.68, "end": 369.24, "text": " the right of the graph, and your cos j has decreased down to here."}, {"start": 369.24, "end": 374.56, "text": " And again, it looks like gradient descent is doing something reasonable, it's getting"}, {"start": 374.56, "end": 377.32, "text": " you closer to the minimum."}, {"start": 377.32, "end": 383.68, "text": " So hopefully, these last two examples show some of the intuition behind what the derivative"}, {"start": 383.68, "end": 390.32, "text": " term is doing, and why this helps gradient descent change w to get you closer to the"}, {"start": 390.32, "end": 391.32, "text": " minimum."}, {"start": 391.32, "end": 397.04, "text": " I hope this video gave you some sense for why the derivative term in gradient descent"}, {"start": 397.04, "end": 398.54, "text": " makes sense."}, {"start": 398.54, "end": 403.52000000000004, "text": " One other key quantity in the gradient descent algorithm is the learning rate alpha."}, {"start": 403.52000000000004, "end": 404.68, "text": " How do you choose alpha?"}, {"start": 404.68, "end": 406.16, "text": " What happens if it's too small?"}, {"start": 406.16, "end": 407.88, "text": " What happens if it's too big?"}, {"start": 407.88, "end": 413.22, "text": " In the next video, let's take a deeper look at the parameter alpha to help build intuitions"}, {"start": 413.22, "end": 418.28000000000003, "text": " about what it does, as well as how to make a good choice for a good value of alpha for"}, {"start": 418.28, "end": 428.96, "text": " your implementation of gradient descent."}]