5.11 Additional Neural Network Concepts | Advanced Optimization --[Machine Learning | Andrew Ng]
 Gradient descent is an optimization algorithm that is widely used in machine learning and was the foundation of many algorithms like linear regression and logistic regression and early implementations of neural networks. But it turns out that there are now some other optimization algorithms for minimizing the cost function that are even better than gradient descent. In this video, we'll take a look at an algorithm that can help you train your neural network much faster than gradient descent. Recall that this is the expression for one step of gradient descent. A parameter wj is updated as wj minus the learning rate alpha times this partial derivative term. How can we make this work even better? In this example, I plotted the cost function j using a contour plot comprising these ellipses. And so the minimum of this cost function is at the center of these ellipses down here. Now if you were to start gradient descent down here, one step of gradient descent, if alpha is small, may take you a little bit in that direction, then another step, then another step, then another step, then another step. And you notice that every single step of gradient descent is pretty much going in the same direction. And if you see this to be the case, you might wonder, well, why don't we make alpha bigger? Can we have an algorithm to automatically increase alpha to just make it take bigger steps and get to the minimum faster? There's an algorithm called the Adam algorithm that can do that. If it sees that the learning rate is too small, and we are just taking tiny little steps in a similar direction over and over, we should just make the learning rate alpha bigger. In contrast, here again is the same cost function. If we were starting here and had a relatively big learning rate alpha, then maybe one step of gradient descent takes us here, and the second step takes us here, third step, and the fourth step, and the fifth step, and the sixth step. And if you see gradient descent doing this, it's oscillating back and forth, you'd be tempted to say, well, why don't we make the learning rate smaller? And the Adam algorithm can also do that automatically. And with a smaller learning rate, you can then take a more smooth path toward the minimum of the cost function. So depending on how gradient descent is proceeding, sometimes you wish you had a bigger learning rate alpha, and sometimes you wish you had a smaller learning rate alpha. So the Adam algorithm can adjust the learning rate automatically. Adam stands for Adaptive Moment Estimation, or ADAM. And don't worry too much about what this name means, it's just what the authors had called this algorithm. But interestingly, the Adam algorithm doesn't use a single global learning rate alpha, it uses a different learning rates for every single parameter of your model. So if you have parameters w1 through w10, as well as b, then it actually has 11 learning rate parameters, alpha 1, alpha 2, all the way through alpha 10, for w1 through w10, as well as I'll call it alpha 11 for the parameter b. And the intuition behind the Adam algorithm is if a parameter wj or b seems to keep on moving in roughly the same direction, this is what we saw on the first example on the previous slide. But if it seems to keep on moving in roughly the same direction, let's increase the learning rate for that parameter. Let's go faster in that direction. Conversely, if a parameter keeps oscillating back and forth, this is what you saw in the second example on the previous slide, then let's not have it keep on oscillating or bouncing back and forth, let's reduce alpha j for that parameter a little bit. The details of how Adam does this are a bit complicated and beyond the scope of this course. But if you take some more advanced deep learning classes later, you may learn more about the details of this Adam algorithm. But in code, this is how you would implement it. The model is exactly the same as before. And the way you compile the model is very similar to what we had before, except that we now add one extra argument to the compile function, which is that we specify that the optimizer you want to use is tf.keris.optimizers.theAdamOptimizer. So the Adam optimization algorithm does need some default initial learning rate alpha. And in this example, I've set that initial learning rate to be 10 to the negative 3. But when you're using the Adam algorithm in practice, it's worth trying a few values for this initial, this default global learning rate. Try some larger and some smaller values to see what gives you the fastest learning performance. Compared to the original gradient descent algorithm that you had learned in the previous course though, the Adam algorithm, because it can adapt the learning rate a bit automatically, it is more robust to the exact choice of learning rate that you pick. Though it is still worth tuning this parameter a little bit to see if you can get somewhat faster learning. So that's it for the Adam optimization algorithm. It typically works much faster than gradient descent, and it's become a de facto standard in how practitioners train their neural networks. So if you're trying to decide what learning algorithm to use, what optimization algorithm to use to train your neural network, a safe choice would be to just use the Adam optimization algorithm. And most practitioners today will use Adam rather than the original gradient descent algorithm. And with this, I hope that your learning algorithms will be able to learn much more quickly. Now, in the next couple of videos, I'd like to touch on some more advanced concepts for neural networks. And in particular, in the next video, let's take a look at some alternative layer types.
[{"start": 0.0, "end": 8.8, "text": " Gradient descent is an optimization algorithm that is widely used in machine learning and"}, {"start": 8.8, "end": 14.56, "text": " was the foundation of many algorithms like linear regression and logistic regression"}, {"start": 14.56, "end": 18.56, "text": " and early implementations of neural networks."}, {"start": 18.56, "end": 23.42, "text": " But it turns out that there are now some other optimization algorithms for minimizing the"}, {"start": 23.42, "end": 27.42, "text": " cost function that are even better than gradient descent."}, {"start": 27.42, "end": 32.400000000000006, "text": " In this video, we'll take a look at an algorithm that can help you train your neural network"}, {"start": 32.400000000000006, "end": 34.760000000000005, "text": " much faster than gradient descent."}, {"start": 34.760000000000005, "end": 40.120000000000005, "text": " Recall that this is the expression for one step of gradient descent."}, {"start": 40.120000000000005, "end": 47.24, "text": " A parameter wj is updated as wj minus the learning rate alpha times this partial derivative"}, {"start": 47.24, "end": 48.64, "text": " term."}, {"start": 48.64, "end": 51.56, "text": " How can we make this work even better?"}, {"start": 51.56, "end": 58.760000000000005, "text": " In this example, I plotted the cost function j using a contour plot comprising these ellipses."}, {"start": 58.760000000000005, "end": 65.2, "text": " And so the minimum of this cost function is at the center of these ellipses down here."}, {"start": 65.2, "end": 72.0, "text": " Now if you were to start gradient descent down here, one step of gradient descent, if"}, {"start": 72.0, "end": 76.28, "text": " alpha is small, may take you a little bit in that direction, then another step, then"}, {"start": 76.28, "end": 79.72, "text": " another step, then another step, then another step."}, {"start": 79.72, "end": 85.24, "text": " And you notice that every single step of gradient descent is pretty much going in the same direction."}, {"start": 85.24, "end": 91.08, "text": " And if you see this to be the case, you might wonder, well, why don't we make alpha bigger?"}, {"start": 91.08, "end": 95.6, "text": " Can we have an algorithm to automatically increase alpha to just make it take bigger"}, {"start": 95.6, "end": 99.08, "text": " steps and get to the minimum faster?"}, {"start": 99.08, "end": 104.52, "text": " There's an algorithm called the Adam algorithm that can do that."}, {"start": 104.52, "end": 110.11999999999999, "text": " If it sees that the learning rate is too small, and we are just taking tiny little steps in"}, {"start": 110.11999999999999, "end": 116.32, "text": " a similar direction over and over, we should just make the learning rate alpha bigger."}, {"start": 116.32, "end": 120.96, "text": " In contrast, here again is the same cost function."}, {"start": 120.96, "end": 126.28, "text": " If we were starting here and had a relatively big learning rate alpha, then maybe one step"}, {"start": 126.28, "end": 130.6, "text": " of gradient descent takes us here, and the second step takes us here, third step, and"}, {"start": 130.6, "end": 134.12, "text": " the fourth step, and the fifth step, and the sixth step."}, {"start": 134.12, "end": 138.76, "text": " And if you see gradient descent doing this, it's oscillating back and forth, you'd be"}, {"start": 138.76, "end": 142.4, "text": " tempted to say, well, why don't we make the learning rate smaller?"}, {"start": 142.4, "end": 145.88, "text": " And the Adam algorithm can also do that automatically."}, {"start": 145.88, "end": 151.42000000000002, "text": " And with a smaller learning rate, you can then take a more smooth path toward the minimum"}, {"start": 151.42000000000002, "end": 153.32, "text": " of the cost function."}, {"start": 153.32, "end": 158.44, "text": " So depending on how gradient descent is proceeding, sometimes you wish you had a bigger learning"}, {"start": 158.44, "end": 164.16, "text": " rate alpha, and sometimes you wish you had a smaller learning rate alpha."}, {"start": 164.16, "end": 168.04, "text": " So the Adam algorithm can adjust the learning rate automatically."}, {"start": 168.04, "end": 174.48, "text": " Adam stands for Adaptive Moment Estimation, or ADAM."}, {"start": 174.48, "end": 178.84, "text": " And don't worry too much about what this name means, it's just what the authors had called"}, {"start": 178.84, "end": 180.64, "text": " this algorithm."}, {"start": 180.64, "end": 186.12, "text": " But interestingly, the Adam algorithm doesn't use a single global learning rate alpha, it"}, {"start": 186.12, "end": 190.96, "text": " uses a different learning rates for every single parameter of your model."}, {"start": 190.96, "end": 197.16, "text": " So if you have parameters w1 through w10, as well as b, then it actually has 11 learning"}, {"start": 197.16, "end": 204.0, "text": " rate parameters, alpha 1, alpha 2, all the way through alpha 10, for w1 through w10,"}, {"start": 204.0, "end": 209.04000000000002, "text": " as well as I'll call it alpha 11 for the parameter b."}, {"start": 209.04000000000002, "end": 216.08, "text": " And the intuition behind the Adam algorithm is if a parameter wj or b seems to keep on"}, {"start": 216.08, "end": 221.84, "text": " moving in roughly the same direction, this is what we saw on the first example on the"}, {"start": 221.84, "end": 222.84, "text": " previous slide."}, {"start": 222.84, "end": 227.16000000000003, "text": " But if it seems to keep on moving in roughly the same direction, let's increase the learning"}, {"start": 227.16000000000003, "end": 228.36, "text": " rate for that parameter."}, {"start": 228.36, "end": 230.68, "text": " Let's go faster in that direction."}, {"start": 230.68, "end": 236.12, "text": " Conversely, if a parameter keeps oscillating back and forth, this is what you saw in the"}, {"start": 236.12, "end": 241.84, "text": " second example on the previous slide, then let's not have it keep on oscillating or bouncing"}, {"start": 241.84, "end": 248.28, "text": " back and forth, let's reduce alpha j for that parameter a little bit."}, {"start": 248.28, "end": 254.12, "text": " The details of how Adam does this are a bit complicated and beyond the scope of this course."}, {"start": 254.12, "end": 258.16, "text": " But if you take some more advanced deep learning classes later, you may learn more about the"}, {"start": 258.16, "end": 260.96, "text": " details of this Adam algorithm."}, {"start": 260.96, "end": 264.08, "text": " But in code, this is how you would implement it."}, {"start": 264.08, "end": 267.46, "text": " The model is exactly the same as before."}, {"start": 267.46, "end": 272.91999999999996, "text": " And the way you compile the model is very similar to what we had before, except that"}, {"start": 272.91999999999996, "end": 279.84, "text": " we now add one extra argument to the compile function, which is that we specify that the"}, {"start": 279.84, "end": 287.0, "text": " optimizer you want to use is tf.keris.optimizers.theAdamOptimizer."}, {"start": 287.0, "end": 293.12, "text": " So the Adam optimization algorithm does need some default initial learning rate alpha."}, {"start": 293.12, "end": 300.36, "text": " And in this example, I've set that initial learning rate to be 10 to the negative 3."}, {"start": 300.36, "end": 304.68, "text": " But when you're using the Adam algorithm in practice, it's worth trying a few values for"}, {"start": 304.68, "end": 308.0, "text": " this initial, this default global learning rate."}, {"start": 308.0, "end": 313.92, "text": " Try some larger and some smaller values to see what gives you the fastest learning performance."}, {"start": 313.92, "end": 319.1, "text": " Compared to the original gradient descent algorithm that you had learned in the previous"}, {"start": 319.1, "end": 325.6, "text": " course though, the Adam algorithm, because it can adapt the learning rate a bit automatically,"}, {"start": 325.6, "end": 330.04, "text": " it is more robust to the exact choice of learning rate that you pick."}, {"start": 330.04, "end": 334.24, "text": " Though it is still worth tuning this parameter a little bit to see if you can get somewhat"}, {"start": 334.24, "end": 335.92, "text": " faster learning."}, {"start": 335.92, "end": 339.18, "text": " So that's it for the Adam optimization algorithm."}, {"start": 339.18, "end": 345.32000000000005, "text": " It typically works much faster than gradient descent, and it's become a de facto standard"}, {"start": 345.32000000000005, "end": 348.74, "text": " in how practitioners train their neural networks."}, {"start": 348.74, "end": 352.84000000000003, "text": " So if you're trying to decide what learning algorithm to use, what optimization algorithm"}, {"start": 352.84000000000003, "end": 358.48, "text": " to use to train your neural network, a safe choice would be to just use the Adam optimization"}, {"start": 358.48, "end": 359.98, "text": " algorithm."}, {"start": 359.98, "end": 364.04, "text": " And most practitioners today will use Adam rather than the original gradient descent"}, {"start": 364.04, "end": 365.54, "text": " algorithm."}, {"start": 365.54, "end": 371.52, "text": " And with this, I hope that your learning algorithms will be able to learn much more quickly."}, {"start": 371.52, "end": 378.0, "text": " Now, in the next couple of videos, I'd like to touch on some more advanced concepts for"}, {"start": 378.0, "end": 379.0, "text": " neural networks."}, {"start": 379.0, "end": 408.92, "text": " And in particular, in the next video, let's take a look at some alternative layer types."}]