2.8 Practical Tips for Linear Regression | | Choosing the learning rate-[MachineLearning|Andrew Ng]
 Your learning algorithm will run much better with an appropriate choice of learning rate. If it's too small, it will run very slowly, and if it's too large, it may not even converge. Let's take a look at how you can choose a good learning rate for your model. Concretely, if you plot the cost for a number of iterations and notice that the cost sometimes goes up and sometimes goes down, you should take that as a clear sign that gradient descent is not working properly. This could mean that there's a bug in the code, or sometimes it could mean that your learning rate is too large. So here's an illustration of what might be happening. Here the vertical axis is a cost function J, and the horizontal axis represents a parameter like maybe W1, and if the learning rate is too big, then if you start off here, your update step may overshoot the minimum and end up here, and in the next update step here, you're again overshooting, so you end up here, and so on. And that's why the cost can sometimes go up instead of decreasing. To fix this, you can use a smaller learning rate. So then your updates may start here and go down a little bit and down a bit, and will hopefully consistently decrease until it reaches the global minimum. Sometimes you may see that the cost consistently increases after each iteration, like this curve here. This is also likely due to a learning rate that is too large, and it could be addressed by choosing a smaller learning rate. But learning rates like this could also be a sign of a possible bug in the code. For example, if I wrote my code so that W1 gets updated as W1 plus alpha times this derivative term, this could result in the cost consistently increasing at each iteration. And this is because adding the derivative term moves your cost j further from the global minimum instead of close up. So remember, you want to use a minus sign, so the code should be updated W1, updated by W1 minus alpha times the derivative term. One debugging tip for a correct implementation of gradient descent is that with a small enough learning rate, the cost function should decrease on every single iteration. So if gradient descent isn't working, one thing I will often do, and I hope you find this tip useful too, one thing I'll often do is just set alpha to be a very, very small number and see if that causes the cost to decrease on every iteration. If even with alpha set to a very small number, j doesn't decrease on every single iteration, but instead sometimes increases, then that usually means there's a bug somewhere in the code. Note that setting alpha to be really, really small is meant here as a debugging step, and a very, very small value of alpha is not going to be the most efficient choice for actually training your learning algorithm. One important trade off is that if your learning rate is too small, then gradient descent can take a lot of iterations to converge. So when I am running gradient descent, I will usually try a range of values for the learning rate alpha. So I might start by trying a learning rate of 0.001, and I might also try a learning rate that's 10 times as large, say 0.01 and 0.1 and so on. And for each choice of alpha, you might run gradient descent just for a handful of iterations and plot the cost function j as a function of the number of iterations. And after trying a few different values, you might then pick the value of alpha that seems to decrease the learning rate rapidly, but also consistently. In fact, what I actually do is try a range of values like this. After trying 0.001, I'll then increase the learning rate threefold to 0.003. And after that, I'll try 0.01, which is again about three times as large as 0.003. So these are roughly trying out gradient descent with each value of alpha being roughly three times bigger than the previous value. So what I'll do is try a range of values until I found a value that's too small. And then also make sure I found a value that's too large. And I'll slowly try to pick the largest possible learning rate or just something slightly smaller than the largest reasonable value that I found. And when I do that, it usually gives me a good learning rate for my model. So I hope this technique too will be useful for you to choose a good learning rate for your implementation of gradient descent. In the upcoming optional lab, you can also take a look at how feature scaling is done in code and also see how different choices of the learning rate alpha can lead to either better or worse training of your model. I hope you have fun playing with the value of alpha and seeing the outcomes of different choices of alpha. So please take a look and run the code in the optional lab to gain a deeper intuition about feature scaling as well as the learning rate alpha. Training learning rates is an important part of training many learning algorithms. And I hope that this video gives you intuition about different choices and how to pick a good value for alpha. Now there are a couple more ideas they can use to make multiple linear regression much more powerful. And that is choosing custom features, which will also allow you to fit curves, not just a straight line to your data. Let's take a look at that in the next video.
[{"start": 0.0, "end": 6.76, "text": " Your learning algorithm will run much better with an appropriate choice of learning rate."}, {"start": 6.76, "end": 12.64, "text": " If it's too small, it will run very slowly, and if it's too large, it may not even converge."}, {"start": 12.64, "end": 16.72, "text": " Let's take a look at how you can choose a good learning rate for your model."}, {"start": 16.72, "end": 23.68, "text": " Concretely, if you plot the cost for a number of iterations and notice that the cost sometimes"}, {"start": 23.68, "end": 29.64, "text": " goes up and sometimes goes down, you should take that as a clear sign that gradient descent"}, {"start": 29.64, "end": 31.76, "text": " is not working properly."}, {"start": 31.76, "end": 36.04, "text": " This could mean that there's a bug in the code, or sometimes it could mean that your"}, {"start": 36.04, "end": 38.44, "text": " learning rate is too large."}, {"start": 38.44, "end": 42.44, "text": " So here's an illustration of what might be happening."}, {"start": 42.44, "end": 50.34, "text": " Here the vertical axis is a cost function J, and the horizontal axis represents a parameter"}, {"start": 50.34, "end": 57.88, "text": " like maybe W1, and if the learning rate is too big, then if you start off here, your"}, {"start": 57.88, "end": 63.92, "text": " update step may overshoot the minimum and end up here, and in the next update step here,"}, {"start": 63.92, "end": 68.4, "text": " you're again overshooting, so you end up here, and so on."}, {"start": 68.4, "end": 73.04, "text": " And that's why the cost can sometimes go up instead of decreasing."}, {"start": 73.04, "end": 76.08, "text": " To fix this, you can use a smaller learning rate."}, {"start": 76.08, "end": 81.6, "text": " So then your updates may start here and go down a little bit and down a bit, and will"}, {"start": 81.6, "end": 86.92, "text": " hopefully consistently decrease until it reaches the global minimum."}, {"start": 86.92, "end": 92.52, "text": " Sometimes you may see that the cost consistently increases after each iteration, like this"}, {"start": 92.52, "end": 93.96000000000001, "text": " curve here."}, {"start": 93.96000000000001, "end": 98.6, "text": " This is also likely due to a learning rate that is too large, and it could be addressed"}, {"start": 98.6, "end": 102.04, "text": " by choosing a smaller learning rate."}, {"start": 102.04, "end": 107.16, "text": " But learning rates like this could also be a sign of a possible bug in the code."}, {"start": 107.16, "end": 115.84, "text": " For example, if I wrote my code so that W1 gets updated as W1 plus alpha times this derivative"}, {"start": 115.84, "end": 122.08, "text": " term, this could result in the cost consistently increasing at each iteration."}, {"start": 122.08, "end": 127.72, "text": " And this is because adding the derivative term moves your cost j further from the global"}, {"start": 127.72, "end": 130.12, "text": " minimum instead of close up."}, {"start": 130.12, "end": 136.64000000000001, "text": " So remember, you want to use a minus sign, so the code should be updated W1, updated"}, {"start": 136.64000000000001, "end": 142.22, "text": " by W1 minus alpha times the derivative term."}, {"start": 142.22, "end": 147.72, "text": " One debugging tip for a correct implementation of gradient descent is that with a small enough"}, {"start": 147.72, "end": 153.8, "text": " learning rate, the cost function should decrease on every single iteration."}, {"start": 153.8, "end": 158.68, "text": " So if gradient descent isn't working, one thing I will often do, and I hope you find"}, {"start": 158.68, "end": 164.64, "text": " this tip useful too, one thing I'll often do is just set alpha to be a very, very small"}, {"start": 164.64, "end": 171.76, "text": " number and see if that causes the cost to decrease on every iteration."}, {"start": 171.76, "end": 178.32, "text": " If even with alpha set to a very small number, j doesn't decrease on every single iteration,"}, {"start": 178.32, "end": 182.67999999999998, "text": " but instead sometimes increases, then that usually means there's a bug somewhere in the"}, {"start": 182.67999999999998, "end": 184.84, "text": " code."}, {"start": 184.84, "end": 190.64, "text": " Note that setting alpha to be really, really small is meant here as a debugging step, and"}, {"start": 190.64, "end": 195.12, "text": " a very, very small value of alpha is not going to be the most efficient choice for actually"}, {"start": 195.12, "end": 197.6, "text": " training your learning algorithm."}, {"start": 197.6, "end": 202.88, "text": " One important trade off is that if your learning rate is too small, then gradient descent can"}, {"start": 202.88, "end": 206.12, "text": " take a lot of iterations to converge."}, {"start": 206.12, "end": 211.76, "text": " So when I am running gradient descent, I will usually try a range of values for the learning"}, {"start": 211.76, "end": 212.76, "text": " rate alpha."}, {"start": 212.76, "end": 219.0, "text": " So I might start by trying a learning rate of 0.001, and I might also try a learning"}, {"start": 219.0, "end": 225.35999999999999, "text": " rate that's 10 times as large, say 0.01 and 0.1 and so on."}, {"start": 225.36, "end": 231.20000000000002, "text": " And for each choice of alpha, you might run gradient descent just for a handful of iterations"}, {"start": 231.20000000000002, "end": 237.16000000000003, "text": " and plot the cost function j as a function of the number of iterations."}, {"start": 237.16000000000003, "end": 242.48000000000002, "text": " And after trying a few different values, you might then pick the value of alpha that seems"}, {"start": 242.48000000000002, "end": 247.60000000000002, "text": " to decrease the learning rate rapidly, but also consistently."}, {"start": 247.60000000000002, "end": 252.68, "text": " In fact, what I actually do is try a range of values like this."}, {"start": 252.68, "end": 260.12, "text": " After trying 0.001, I'll then increase the learning rate threefold to 0.003."}, {"start": 260.12, "end": 268.26, "text": " And after that, I'll try 0.01, which is again about three times as large as 0.003."}, {"start": 268.26, "end": 273.3, "text": " So these are roughly trying out gradient descent with each value of alpha being roughly three"}, {"start": 273.3, "end": 277.24, "text": " times bigger than the previous value."}, {"start": 277.24, "end": 281.84000000000003, "text": " So what I'll do is try a range of values until I found a value that's too small."}, {"start": 281.84, "end": 286.08, "text": " And then also make sure I found a value that's too large."}, {"start": 286.08, "end": 292.56, "text": " And I'll slowly try to pick the largest possible learning rate or just something slightly smaller"}, {"start": 292.56, "end": 295.96, "text": " than the largest reasonable value that I found."}, {"start": 295.96, "end": 300.65999999999997, "text": " And when I do that, it usually gives me a good learning rate for my model."}, {"start": 300.65999999999997, "end": 305.76, "text": " So I hope this technique too will be useful for you to choose a good learning rate for"}, {"start": 305.76, "end": 309.91999999999996, "text": " your implementation of gradient descent."}, {"start": 309.92, "end": 315.08000000000004, "text": " In the upcoming optional lab, you can also take a look at how feature scaling is done"}, {"start": 315.08000000000004, "end": 320.76, "text": " in code and also see how different choices of the learning rate alpha can lead to either"}, {"start": 320.76, "end": 324.16, "text": " better or worse training of your model."}, {"start": 324.16, "end": 328.82, "text": " I hope you have fun playing with the value of alpha and seeing the outcomes of different"}, {"start": 328.82, "end": 330.70000000000005, "text": " choices of alpha."}, {"start": 330.70000000000005, "end": 335.52000000000004, "text": " So please take a look and run the code in the optional lab to gain a deeper intuition"}, {"start": 335.52000000000004, "end": 339.88, "text": " about feature scaling as well as the learning rate alpha."}, {"start": 339.88, "end": 344.04, "text": " Training learning rates is an important part of training many learning algorithms."}, {"start": 344.04, "end": 348.08, "text": " And I hope that this video gives you intuition about different choices and how to pick a"}, {"start": 348.08, "end": 350.92, "text": " good value for alpha."}, {"start": 350.92, "end": 355.44, "text": " Now there are a couple more ideas they can use to make multiple linear regression much"}, {"start": 355.44, "end": 356.92, "text": " more powerful."}, {"start": 356.92, "end": 362.24, "text": " And that is choosing custom features, which will also allow you to fit curves, not just"}, {"start": 362.24, "end": 364.2, "text": " a straight line to your data."}, {"start": 364.2, "end": 370.64, "text": " Let's take a look at that in the next video."}]