1.18 Machine Learning Overview | Learning rate  --[Machine Learning | Andrew Ng]
 The choice of the learning rate alpha will have a huge impact on the efficiency of your implementation of gradient descent, and if alpha, the learning rate, is chosen poorly, gradient descent may not even work at all. In this video, let's take a deeper look at the learning rate. This will also help you choose better learning rates for your implementations of gradient descent. So, here again is the gradient descent rule. W is updated to be W minus the learning rate alpha times the derivative term. To learn more about what the learning rate alpha is doing, let's see what could happen if the learning rate alpha is either too small, or if it is too large. For the case where the learning rate is too small, here's a graph where the horizontal axis is W and the vertical axis is the cos j, and here's a graph of the function j of W. Let's start gradient descent at this point here. If the learning rate is too small, then what happens is that you multiply your derivative term by some really, really, really small number. So, you're going to be multiplying by a number alpha that's really small, like 0.0000001, and so you end up taking a very small baby step like that. Then from this point, you're going to take another, you know, tiny, tiny little baby step, but because the learning rate is so small, the second step is also just minus Q. The outcome of this process is that you do end up decreasing the cos j, but incredibly slowly. So, here's another step, and another step, another tiny step, until you finally approach the minimum. But, as you may notice, you're going to need a lot of steps to get to the minimum. So, to summarize, if the learning rate is too small, then gradient descent will work, but it will be slow. It will take a very long time, because it's going to take these tiny, tiny baby steps, and it's going to need a lot of steps before it gets anywhere close to the minimum. Now, let's look at a different case. What happens if the learning rate is too large? Here's another graph of the cos function, and let's say we start gradient descent with W at this value here. So, it's actually already pretty close to the minimum. So, the derivative points to the right, but if the learning rate is too large, then you update W via a giant step to be all the way over here, and that's this point here on the function j. So, you move from this point on the left all the way to this point on the right, and now the cost has actually gotten worse. It has increased because it has started out at this value here, and after one step, it actually increased to this value here. Now, the derivative at this new point says to decrease W, but when the learning rate is too big, then you may take a huge step going from here all the way out here. So, now you've gotten to this point here, and again, if the learning rate is too big, then you take another huge step at the next iteration and way overshoot the minimum again. So, now you're at this point on the right, and one more time, you do another update and end up all the way here, and so you're now at this point here. So, as you may notice, you're actually getting further and further away from the minimum. So, if the learning rate is too large, then gradient descent may overshoot and may never reach the minimum. And another way to say that is that gradient descent may fail to converge and may even diverge. So, here's another question you may be wondering. What if your parameter W is already at this point here, so that your cost J is already at a local minimum? What do you think one step of gradient descent will do if you've already reached a minimum? So, this is a tricky one. When I was first learning this stuff, it actually took me a long time to figure it out, but let's work through this together. Let's suppose you have some cost function J, and the one you see here isn't a square error cost function, and this cost function has two local minima corresponding to the two values that you see here. Now, let's suppose that after some number of steps of gradient descent, your parameter W is over here, say equal to 5, and so this is the current value of W. This means that you're at this point on the cost function J, and that happens to be a local minimum. Turns out, if you draw a tangent to the function at this point, the slope of this line is zero, and thus, the derivative term here is equal to zero for the current value of W, and so your gradient descent update becomes W is updated to W minus the learning rate times zero, where here, that's because the derivative term is equal to zero, and this is the same as saying let's set W to be equal to W. So, this means that if you're already at a local minimum, gradient descent leaves W unchanged because it just updates the new value of W to be the exact same old value of W. So, concretely, let's say if the current value of W is 5, and alpha is 0.1, after one iteration, you update W as W minus alpha times zero, and it is still equal to 5. So, if your parameters have already brought you to a local minimum, then further gradient descent steps to absolutely nothing. It doesn't change the parameters, which is what you want, because it keeps the solution at that local minimum. This also explains why gradient descent can reach a local minimum even with a fixed learning rate alpha. Here's what I mean. To illustrate this, let's look at another example. Here's the cost function J of W that we want to minimize. Let's initialize gradient descent up here at this point. If we take one update step, maybe it'll take us to that point. And because this derivative is pretty large, gradient descent takes a relatively big step, right? Now, we're at this second point, where we take another step. And you may notice that the slope is not as steep as it was at the first point, so the derivative isn't as large. And so, the next update step will not be as large as that first step. Now, we're at this third point here, and the derivative is smaller than it was at the previous step and we'll take an even smaller step. As we approach the minimum, the derivative gets closer and closer to zero. So as we run gradient descent, eventually we're taking very small steps until you finally reach a local minimum. So just to recap, as we get nearer a local minimum, gradient descent will automatically take smaller steps. And that's because as we approach the local minimum, the derivative automatically gets smaller and that means the update steps also automatically get smaller, even if the learning rate alpha is kept at some fixed value. So that's the gradient descent algorithm. You can use it to try to minimize any cost function j, not just the mean squared error cost function that we're using for linear regression. In the next video, we're going to take the function j and set that back to be exactly the linear regression model's cost function, the mean squared error cost function that we had come up with earlier. And putting together gradient descent with this cost function that will give you your first learning algorithm, the linear regression algorithm.
[{"start": 0.0, "end": 7.12, "text": " The choice of the learning rate alpha will have a huge impact on the efficiency of your"}, {"start": 7.12, "end": 13.280000000000001, "text": " implementation of gradient descent, and if alpha, the learning rate, is chosen poorly,"}, {"start": 13.280000000000001, "end": 15.94, "text": " gradient descent may not even work at all."}, {"start": 15.94, "end": 19.8, "text": " In this video, let's take a deeper look at the learning rate."}, {"start": 19.8, "end": 25.04, "text": " This will also help you choose better learning rates for your implementations of gradient"}, {"start": 25.04, "end": 26.04, "text": " descent."}, {"start": 26.04, "end": 30.04, "text": " So, here again is the gradient descent rule."}, {"start": 30.04, "end": 35.92, "text": " W is updated to be W minus the learning rate alpha times the derivative term."}, {"start": 35.92, "end": 39.96, "text": " To learn more about what the learning rate alpha is doing, let's see what could happen"}, {"start": 39.96, "end": 46.36, "text": " if the learning rate alpha is either too small, or if it is too large."}, {"start": 46.36, "end": 51.2, "text": " For the case where the learning rate is too small, here's a graph where the horizontal"}, {"start": 51.2, "end": 59.800000000000004, "text": " axis is W and the vertical axis is the cos j, and here's a graph of the function j of"}, {"start": 59.800000000000004, "end": 65.28, "text": " W. Let's start gradient descent at this point here."}, {"start": 65.28, "end": 71.32000000000001, "text": " If the learning rate is too small, then what happens is that you multiply your derivative"}, {"start": 71.32000000000001, "end": 74.64, "text": " term by some really, really, really small number."}, {"start": 74.64, "end": 83.52, "text": " So, you're going to be multiplying by a number alpha that's really small, like 0.0000001,"}, {"start": 83.52, "end": 88.56, "text": " and so you end up taking a very small baby step like that."}, {"start": 88.56, "end": 92.76, "text": " Then from this point, you're going to take another, you know, tiny, tiny little baby"}, {"start": 92.76, "end": 99.52, "text": " step, but because the learning rate is so small, the second step is also just minus"}, {"start": 99.52, "end": 100.52, "text": " Q."}, {"start": 100.52, "end": 106.24, "text": " The outcome of this process is that you do end up decreasing the cos j, but incredibly"}, {"start": 106.24, "end": 107.24, "text": " slowly."}, {"start": 107.24, "end": 113.28, "text": " So, here's another step, and another step, another tiny step, until you finally approach"}, {"start": 113.28, "end": 114.28, "text": " the minimum."}, {"start": 114.28, "end": 119.08, "text": " But, as you may notice, you're going to need a lot of steps to get to the minimum."}, {"start": 119.08, "end": 126.36, "text": " So, to summarize, if the learning rate is too small, then gradient descent will work,"}, {"start": 126.36, "end": 127.92, "text": " but it will be slow."}, {"start": 127.92, "end": 132.92000000000002, "text": " It will take a very long time, because it's going to take these tiny, tiny baby steps,"}, {"start": 132.92000000000002, "end": 136.92000000000002, "text": " and it's going to need a lot of steps before it gets anywhere close to the minimum."}, {"start": 136.92000000000002, "end": 141.0, "text": " Now, let's look at a different case."}, {"start": 141.0, "end": 144.44, "text": " What happens if the learning rate is too large?"}, {"start": 144.44, "end": 148.88, "text": " Here's another graph of the cos function, and let's say we start gradient descent with"}, {"start": 148.88, "end": 152.04, "text": " W at this value here."}, {"start": 152.04, "end": 156.68, "text": " So, it's actually already pretty close to the minimum."}, {"start": 156.68, "end": 165.20000000000002, "text": " So, the derivative points to the right, but if the learning rate is too large, then you"}, {"start": 165.20000000000002, "end": 174.56, "text": " update W via a giant step to be all the way over here, and that's this point here on the"}, {"start": 174.56, "end": 175.56, "text": " function j."}, {"start": 175.56, "end": 182.28, "text": " So, you move from this point on the left all the way to this point on the right, and now"}, {"start": 182.28, "end": 184.4, "text": " the cost has actually gotten worse."}, {"start": 184.4, "end": 190.24, "text": " It has increased because it has started out at this value here, and after one step, it"}, {"start": 190.24, "end": 193.28, "text": " actually increased to this value here."}, {"start": 193.28, "end": 200.88, "text": " Now, the derivative at this new point says to decrease W, but when the learning rate"}, {"start": 200.88, "end": 207.24, "text": " is too big, then you may take a huge step going from here all the way out here."}, {"start": 207.24, "end": 213.64000000000001, "text": " So, now you've gotten to this point here, and again, if the learning rate is too big,"}, {"start": 213.64, "end": 218.6, "text": " then you take another huge step at the next iteration and way overshoot the minimum again."}, {"start": 218.6, "end": 224.32, "text": " So, now you're at this point on the right, and one more time, you do another update and"}, {"start": 224.32, "end": 230.39999999999998, "text": " end up all the way here, and so you're now at this point here."}, {"start": 230.39999999999998, "end": 236.64, "text": " So, as you may notice, you're actually getting further and further away from the minimum."}, {"start": 236.64, "end": 243.83999999999997, "text": " So, if the learning rate is too large, then gradient descent may overshoot and may never"}, {"start": 243.83999999999997, "end": 246.07999999999998, "text": " reach the minimum."}, {"start": 246.07999999999998, "end": 252.83999999999997, "text": " And another way to say that is that gradient descent may fail to converge and may even"}, {"start": 252.83999999999997, "end": 253.83999999999997, "text": " diverge."}, {"start": 253.83999999999997, "end": 259.32, "text": " So, here's another question you may be wondering."}, {"start": 259.32, "end": 266.64, "text": " What if your parameter W is already at this point here, so that your cost J is already"}, {"start": 266.64, "end": 269.59999999999997, "text": " at a local minimum?"}, {"start": 269.59999999999997, "end": 275.71999999999997, "text": " What do you think one step of gradient descent will do if you've already reached a minimum?"}, {"start": 275.71999999999997, "end": 279.12, "text": " So, this is a tricky one."}, {"start": 279.12, "end": 283.92, "text": " When I was first learning this stuff, it actually took me a long time to figure it out, but"}, {"start": 283.92, "end": 286.12, "text": " let's work through this together."}, {"start": 286.12, "end": 293.08, "text": " Let's suppose you have some cost function J, and the one you see here isn't a square"}, {"start": 293.08, "end": 299.12, "text": " error cost function, and this cost function has two local minima corresponding to the"}, {"start": 299.12, "end": 302.52, "text": " two values that you see here."}, {"start": 302.52, "end": 308.6, "text": " Now, let's suppose that after some number of steps of gradient descent, your parameter"}, {"start": 308.6, "end": 317.36, "text": " W is over here, say equal to 5, and so this is the current value of W. This means that"}, {"start": 317.36, "end": 323.92, "text": " you're at this point on the cost function J, and that happens to be a local minimum."}, {"start": 323.92, "end": 330.84000000000003, "text": " Turns out, if you draw a tangent to the function at this point, the slope of this line is zero,"}, {"start": 330.84000000000003, "end": 337.8, "text": " and thus, the derivative term here is equal to zero for the current value of W, and so"}, {"start": 337.8, "end": 344.72, "text": " your gradient descent update becomes W is updated to W minus the learning rate times"}, {"start": 344.72, "end": 351.68, "text": " zero, where here, that's because the derivative term is equal to zero, and this is the same"}, {"start": 351.68, "end": 360.32, "text": " as saying let's set W to be equal to W. So, this means that if you're already at a local"}, {"start": 360.32, "end": 366.52, "text": " minimum, gradient descent leaves W unchanged because it just updates the new value of W"}, {"start": 366.52, "end": 375.79999999999995, "text": " to be the exact same old value of W. So, concretely, let's say if the current value of W is 5,"}, {"start": 375.79999999999995, "end": 385.59999999999997, "text": " and alpha is 0.1, after one iteration, you update W as W minus alpha times zero, and"}, {"start": 385.59999999999997, "end": 388.4, "text": " it is still equal to 5."}, {"start": 388.4, "end": 395.12, "text": " So, if your parameters have already brought you to a local minimum, then further gradient"}, {"start": 395.12, "end": 397.8, "text": " descent steps to absolutely nothing."}, {"start": 397.8, "end": 401.84000000000003, "text": " It doesn't change the parameters, which is what you want, because it keeps the solution"}, {"start": 401.84000000000003, "end": 404.34000000000003, "text": " at that local minimum."}, {"start": 404.34000000000003, "end": 409.92, "text": " This also explains why gradient descent can reach a local minimum even with a fixed learning"}, {"start": 409.92, "end": 410.92, "text": " rate alpha."}, {"start": 410.92, "end": 413.32, "text": " Here's what I mean."}, {"start": 413.32, "end": 417.44, "text": " To illustrate this, let's look at another example."}, {"start": 417.44, "end": 422.64, "text": " Here's the cost function J of W that we want to minimize."}, {"start": 422.64, "end": 427.52, "text": " Let's initialize gradient descent up here at this point."}, {"start": 427.52, "end": 434.24, "text": " If we take one update step, maybe it'll take us to that point."}, {"start": 434.24, "end": 440.68, "text": " And because this derivative is pretty large, gradient descent takes a relatively big step,"}, {"start": 440.68, "end": 441.68, "text": " right?"}, {"start": 441.68, "end": 446.91999999999996, "text": " Now, we're at this second point, where we take another step."}, {"start": 446.91999999999996, "end": 451.76, "text": " And you may notice that the slope is not as steep as it was at the first point, so the"}, {"start": 451.76, "end": 454.03999999999996, "text": " derivative isn't as large."}, {"start": 454.03999999999996, "end": 459.64, "text": " And so, the next update step will not be as large as that first step."}, {"start": 459.64, "end": 466.88, "text": " Now, we're at this third point here, and the derivative is smaller than it was at the previous"}, {"start": 466.88, "end": 471.12, "text": " step and we'll take an even smaller step."}, {"start": 471.12, "end": 476.88, "text": " As we approach the minimum, the derivative gets closer and closer to zero."}, {"start": 476.88, "end": 483.2, "text": " So as we run gradient descent, eventually we're taking very small steps until you finally"}, {"start": 483.2, "end": 485.28, "text": " reach a local minimum."}, {"start": 485.28, "end": 491.48, "text": " So just to recap, as we get nearer a local minimum, gradient descent will automatically"}, {"start": 491.48, "end": 493.56, "text": " take smaller steps."}, {"start": 493.56, "end": 498.52, "text": " And that's because as we approach the local minimum, the derivative automatically gets"}, {"start": 498.52, "end": 505.44, "text": " smaller and that means the update steps also automatically get smaller, even if the learning"}, {"start": 505.44, "end": 508.88, "text": " rate alpha is kept at some fixed value."}, {"start": 508.88, "end": 511.9, "text": " So that's the gradient descent algorithm."}, {"start": 511.9, "end": 517.24, "text": " You can use it to try to minimize any cost function j, not just the mean squared error"}, {"start": 517.24, "end": 521.72, "text": " cost function that we're using for linear regression."}, {"start": 521.72, "end": 527.64, "text": " In the next video, we're going to take the function j and set that back to be exactly"}, {"start": 527.64, "end": 532.36, "text": " the linear regression model's cost function, the mean squared error cost function that"}, {"start": 532.36, "end": 534.76, "text": " we had come up with earlier."}, {"start": 534.76, "end": 539.16, "text": " And putting together gradient descent with this cost function that will give you your"}, {"start": 539.16, "end": 566.16, "text": " first learning algorithm, the linear regression algorithm."}]