10.14 Continuous State Spaces | Algorithm refinement ϵ greedy policy -[Machine Learning | Andrew Ng]
 In the learning algorithm that we developed, even while you're still learning how to approximate QFSA, you need to take some actions in the lunar lander. So how do you pick those actions while you're still learning? The most common way to do so is to use something called an epsilon greedy policy. Let's take a look at how that works. Here's the algorithm that you saw earlier. One of the steps in the algorithm is to take actions in the lunar lander. So when the learning algorithm is still running, we don't really know what's the best action to take in every state. If we did, we'd already be done learning. But even while we're still learning and don't have a very good estimate of QFSA yet, how do we take actions in this step of the learning algorithm? Let's look at some options. When you're in some state S, we might not want to take actions totally at random because that will often be a bad action. So one natural option would be to pick, whenever we're in state S, pick an action A that maximizes QFSA. So we may say, even if QFSA is not a great estimate of the Q function, let's just do our best and use our current guess of QFSA and pick the action A that maximizes it. It turns out this may work okay, but isn't the best option. Instead, here's what is commonly done. Here's option two, which is most of the time, let's say with probability 0.95, pick the action that maximizes QFSA. So most of the time, we'll try to pick a good action using our current guess of QFSA. But a small fraction of the time, let's say 5% of the time, we'll pick an action A randomly. Why do we want to occasionally pick an action randomly? Well here's why. Suppose there's some strange reason that QFSA was initialized randomly so that the learning algorithm thinks that firing the main thrusters is never a good idea, maybe the neural network parameters were initialized so that Q of S main is always very low. If that's the case, then the neural network, because it's trying to pick the action A that maximizes QFSA, it will never ever try firing the main thruster. And because it never ever tries firing the main thruster, it will never learn that firing the main thruster is actually sometimes a good idea. And so because of the random initialization, if the neural network somehow initially gets stuck in its mind that something's a bad idea just by chance, then option one means that it will never try out those actions and discover that maybe it's actually a good idea to take that action, like firing the main thruster sometimes. So under option two, on every step, we have some small probability of trying out different actions so that the neural network can learn to overcome its own possible preconceptions about what might be a bad idea that turns out not to be the case. This idea of picking actions randomly is sometimes called an exploration step because we're going to try out something that may not be the best idea, but we're going to just try out some action in some circumstance to explore and learn more about an action in a circumstance where we may not have had as much experience before. Taking an action that maximizes QFSA, sometimes this is called a greedy action because we're trying to actually maximize our return by picking this. Or in the reinforcement learning literature, sometimes you also hear this as an exploitation step. I know that exploitation is not a good thing. Nobody should ever exploit anyone else, but historically this was the term user-enforced learning to say, let's exploit everything we've learned to do the best we can. So in the reinforcement learning literature, sometimes you hear people talk about the exploration versus exploitation trade-off, which refers to how often do you take actions randomly or take actions that may not be the best in order to learn more versus trying to maximize your return by say taking the action that maximizes QFSA. This approach that is option two has a name is called an epsilon greedy policy where here epsilon is 0.05, is the probability of picking an action randomly. And this is the most common way to make your reinforcement learning algorithm explore a little bit, even while occasionally or maybe most of the time taking greedy actions. Oh, and by the way, a lot of people have commented that the name epsilon greedy policy is confusing because you're actually being greedy 95% of the time, not 5% of the time. So maybe one minus epsilon greedy policy, because it's 95% greedy, 5% exploring. That's actually a more accurate description of the algorithm. But for historical reasons, the name epsilon greedy policy is what has stuck. And so this is the name that people use to refer to the policy that explores actually epsilon fraction of the time rather than this greedy epsilon fraction of the time. Lastly, one of the trick that sometimes user enforcement learning is to start off epsilon high. So initially, you are taking random actions a lot of the time, and then gradually decrease it so that over time, you are less likely to take actions randomly and more likely to use your improving estimates of the Q function to pick good actions. So for example, in the lunar lander exercise, you might start off with epsilon very, very high, maybe even epsilon equals 1.0. So you're just picking actions completely at random initially, and then gradually decrease it all the way down to say, 0.01. So that eventually you're taking greedy actions 99% of the time and acting randomly only a very small 1% of the time. If this seems complicated, don't worry about it. We'll provide the code in the practice lab, in the Jupyter lab that shows you how to do this. If you were to implement the algorithm as we've described it with the more efficient neural network architecture and with an epsilon greedy exploration policy, you find that it will work pretty well on the lunar lander. One of the things that I've noticed for reinforcement learning algorithm is that compared to supervised learning, they're more finicky in terms of the choice of hyperparameters. So for example, in supervised learning, if you set the learning rate a little bit too small, then maybe the algorithm will take longer to learn. Maybe it takes three times as long to train, which is annoying, but maybe not that bad. Whereas in reinforcement learning, find that if you set the value of epsilon not quite as well or set other parameters not quite as well, it doesn't take three times as long to learn. It may take 10 times or 100 times as long to learn. Reinforcement learning algorithms, I think because they're less mature than supervised learning algorithms, are much more finicky to little choices of parameters like that. It actually sometimes is frankly more frustrating to tune these parameters for reinforcement learning algorithm compared to a supervised learning algorithm. But again, if you're worried about the practice lab, the programming exercise, we'll give you a sense of good parameters to use in the programming exercise so that you should be able to do that and successfully land the lunar lander, hopefully without too many problems. In the next optional video, I want to describe a couple more algorithm refinements, mini batching and also using soft updates, even without these additional refinements, the algorithm will work okay. But these are additional refinements that make the algorithm run much faster. And it's okay if you skip this video, we've provided everything you need in the practice lab to hopefully successfully complete it. But if you're interested in learning about more of these details of tuning reinforcement learning algorithms, then come with me and let's see in the next video, mini batching and soft updates.
[{"start": 0.0, "end": 8.32, "text": " In the learning algorithm that we developed, even while you're still learning how to approximate"}, {"start": 8.32, "end": 13.88, "text": " QFSA, you need to take some actions in the lunar lander."}, {"start": 13.88, "end": 17.080000000000002, "text": " So how do you pick those actions while you're still learning?"}, {"start": 17.080000000000002, "end": 21.56, "text": " The most common way to do so is to use something called an epsilon greedy policy."}, {"start": 21.56, "end": 24.48, "text": " Let's take a look at how that works."}, {"start": 24.48, "end": 27.88, "text": " Here's the algorithm that you saw earlier."}, {"start": 27.88, "end": 33.6, "text": " One of the steps in the algorithm is to take actions in the lunar lander."}, {"start": 33.6, "end": 38.72, "text": " So when the learning algorithm is still running, we don't really know what's the best action"}, {"start": 38.72, "end": 39.96, "text": " to take in every state."}, {"start": 39.96, "end": 42.8, "text": " If we did, we'd already be done learning."}, {"start": 42.8, "end": 48.4, "text": " But even while we're still learning and don't have a very good estimate of QFSA yet, how"}, {"start": 48.4, "end": 52.84, "text": " do we take actions in this step of the learning algorithm?"}, {"start": 52.84, "end": 54.64, "text": " Let's look at some options."}, {"start": 54.64, "end": 60.76, "text": " When you're in some state S, we might not want to take actions totally at random because"}, {"start": 60.76, "end": 64.12, "text": " that will often be a bad action."}, {"start": 64.12, "end": 72.44, "text": " So one natural option would be to pick, whenever we're in state S, pick an action A that maximizes"}, {"start": 72.44, "end": 74.36, "text": " QFSA."}, {"start": 74.36, "end": 80.66, "text": " So we may say, even if QFSA is not a great estimate of the Q function, let's just do"}, {"start": 80.66, "end": 87.24, "text": " our best and use our current guess of QFSA and pick the action A that maximizes it."}, {"start": 87.24, "end": 92.96, "text": " It turns out this may work okay, but isn't the best option."}, {"start": 92.96, "end": 96.52, "text": " Instead, here's what is commonly done."}, {"start": 96.52, "end": 102.94, "text": " Here's option two, which is most of the time, let's say with probability 0.95, pick the"}, {"start": 102.94, "end": 107.08, "text": " action that maximizes QFSA."}, {"start": 107.08, "end": 113.0, "text": " So most of the time, we'll try to pick a good action using our current guess of QFSA."}, {"start": 113.0, "end": 119.84, "text": " But a small fraction of the time, let's say 5% of the time, we'll pick an action A randomly."}, {"start": 119.84, "end": 123.2, "text": " Why do we want to occasionally pick an action randomly?"}, {"start": 123.2, "end": 125.0, "text": " Well here's why."}, {"start": 125.0, "end": 131.12, "text": " Suppose there's some strange reason that QFSA was initialized randomly so that the learning"}, {"start": 131.12, "end": 137.56, "text": " algorithm thinks that firing the main thrusters is never a good idea, maybe the neural network"}, {"start": 137.56, "end": 145.5, "text": " parameters were initialized so that Q of S main is always very low."}, {"start": 145.5, "end": 150.36, "text": " If that's the case, then the neural network, because it's trying to pick the action A that"}, {"start": 150.36, "end": 155.92000000000002, "text": " maximizes QFSA, it will never ever try firing the main thruster."}, {"start": 155.92000000000002, "end": 161.04000000000002, "text": " And because it never ever tries firing the main thruster, it will never learn that firing"}, {"start": 161.04, "end": 164.4, "text": " the main thruster is actually sometimes a good idea."}, {"start": 164.4, "end": 170.6, "text": " And so because of the random initialization, if the neural network somehow initially gets"}, {"start": 170.6, "end": 177.35999999999999, "text": " stuck in its mind that something's a bad idea just by chance, then option one means that"}, {"start": 177.35999999999999, "end": 183.35999999999999, "text": " it will never try out those actions and discover that maybe it's actually a good idea to take"}, {"start": 183.35999999999999, "end": 186.6, "text": " that action, like firing the main thruster sometimes."}, {"start": 186.6, "end": 192.32, "text": " So under option two, on every step, we have some small probability of trying out different"}, {"start": 192.32, "end": 200.22, "text": " actions so that the neural network can learn to overcome its own possible preconceptions"}, {"start": 200.22, "end": 204.92, "text": " about what might be a bad idea that turns out not to be the case."}, {"start": 204.92, "end": 212.57999999999998, "text": " This idea of picking actions randomly is sometimes called an exploration step because we're going"}, {"start": 212.58, "end": 217.52, "text": " to try out something that may not be the best idea, but we're going to just try out some"}, {"start": 217.52, "end": 222.64000000000001, "text": " action in some circumstance to explore and learn more about an action in a circumstance"}, {"start": 222.64000000000001, "end": 226.36, "text": " where we may not have had as much experience before."}, {"start": 226.36, "end": 234.60000000000002, "text": " Taking an action that maximizes QFSA, sometimes this is called a greedy action because we're"}, {"start": 234.60000000000002, "end": 240.64000000000001, "text": " trying to actually maximize our return by picking this."}, {"start": 240.64, "end": 246.55999999999997, "text": " Or in the reinforcement learning literature, sometimes you also hear this as an exploitation"}, {"start": 246.55999999999997, "end": 247.55999999999997, "text": " step."}, {"start": 247.55999999999997, "end": 250.95999999999998, "text": " I know that exploitation is not a good thing."}, {"start": 250.95999999999998, "end": 256.03999999999996, "text": " Nobody should ever exploit anyone else, but historically this was the term user-enforced"}, {"start": 256.03999999999996, "end": 260.68, "text": " learning to say, let's exploit everything we've learned to do the best we can."}, {"start": 260.68, "end": 266.03999999999996, "text": " So in the reinforcement learning literature, sometimes you hear people talk about the exploration"}, {"start": 266.04, "end": 271.92, "text": " versus exploitation trade-off, which refers to how often do you take actions randomly"}, {"start": 271.92, "end": 279.0, "text": " or take actions that may not be the best in order to learn more versus trying to maximize"}, {"start": 279.0, "end": 283.70000000000005, "text": " your return by say taking the action that maximizes QFSA."}, {"start": 283.70000000000005, "end": 292.12, "text": " This approach that is option two has a name is called an epsilon greedy policy where here"}, {"start": 292.12, "end": 299.04, "text": " epsilon is 0.05, is the probability of picking an action randomly."}, {"start": 299.04, "end": 306.16, "text": " And this is the most common way to make your reinforcement learning algorithm explore a"}, {"start": 306.16, "end": 311.6, "text": " little bit, even while occasionally or maybe most of the time taking greedy actions."}, {"start": 311.6, "end": 317.76, "text": " Oh, and by the way, a lot of people have commented that the name epsilon greedy policy is confusing"}, {"start": 317.76, "end": 323.44, "text": " because you're actually being greedy 95% of the time, not 5% of the time."}, {"start": 323.44, "end": 330.48, "text": " So maybe one minus epsilon greedy policy, because it's 95% greedy, 5% exploring."}, {"start": 330.48, "end": 333.48, "text": " That's actually a more accurate description of the algorithm."}, {"start": 333.48, "end": 338.96, "text": " But for historical reasons, the name epsilon greedy policy is what has stuck."}, {"start": 338.96, "end": 344.48, "text": " And so this is the name that people use to refer to the policy that explores actually"}, {"start": 344.48, "end": 349.92, "text": " epsilon fraction of the time rather than this greedy epsilon fraction of the time."}, {"start": 349.92, "end": 355.12, "text": " Lastly, one of the trick that sometimes user enforcement learning is to start off epsilon"}, {"start": 355.12, "end": 356.46000000000004, "text": " high."}, {"start": 356.46000000000004, "end": 364.12, "text": " So initially, you are taking random actions a lot of the time, and then gradually decrease"}, {"start": 364.12, "end": 371.48, "text": " it so that over time, you are less likely to take actions randomly and more likely to"}, {"start": 371.48, "end": 377.16, "text": " use your improving estimates of the Q function to pick good actions."}, {"start": 377.16, "end": 383.04, "text": " So for example, in the lunar lander exercise, you might start off with epsilon very, very"}, {"start": 383.04, "end": 386.24, "text": " high, maybe even epsilon equals 1.0."}, {"start": 386.24, "end": 390.72, "text": " So you're just picking actions completely at random initially, and then gradually decrease"}, {"start": 390.72, "end": 394.62, "text": " it all the way down to say, 0.01."}, {"start": 394.62, "end": 401.44, "text": " So that eventually you're taking greedy actions 99% of the time and acting randomly only"}, {"start": 401.44, "end": 404.16, "text": " a very small 1% of the time."}, {"start": 404.16, "end": 406.71999999999997, "text": " If this seems complicated, don't worry about it."}, {"start": 406.71999999999997, "end": 411.92, "text": " We'll provide the code in the practice lab, in the Jupyter lab that shows you how to do"}, {"start": 411.92, "end": 413.24, "text": " this."}, {"start": 413.24, "end": 417.96, "text": " If you were to implement the algorithm as we've described it with the more efficient"}, {"start": 417.96, "end": 423.6, "text": " neural network architecture and with an epsilon greedy exploration policy, you find that it"}, {"start": 423.6, "end": 427.08, "text": " will work pretty well on the lunar lander."}, {"start": 427.08, "end": 432.44, "text": " One of the things that I've noticed for reinforcement learning algorithm is that compared to supervised"}, {"start": 432.44, "end": 437.0, "text": " learning, they're more finicky in terms of the choice of hyperparameters."}, {"start": 437.0, "end": 442.03999999999996, "text": " So for example, in supervised learning, if you set the learning rate a little bit too"}, {"start": 442.03999999999996, "end": 445.76, "text": " small, then maybe the algorithm will take longer to learn."}, {"start": 445.76, "end": 451.24, "text": " Maybe it takes three times as long to train, which is annoying, but maybe not that bad."}, {"start": 451.24, "end": 456.76, "text": " Whereas in reinforcement learning, find that if you set the value of epsilon not quite"}, {"start": 456.76, "end": 461.71999999999997, "text": " as well or set other parameters not quite as well, it doesn't take three times as long"}, {"start": 461.71999999999997, "end": 462.71999999999997, "text": " to learn."}, {"start": 462.71999999999997, "end": 467.52, "text": " It may take 10 times or 100 times as long to learn."}, {"start": 467.52, "end": 471.28, "text": " Reinforcement learning algorithms, I think because they're less mature than supervised"}, {"start": 471.28, "end": 476.88, "text": " learning algorithms, are much more finicky to little choices of parameters like that."}, {"start": 476.88, "end": 482.84, "text": " It actually sometimes is frankly more frustrating to tune these parameters for reinforcement"}, {"start": 482.84, "end": 487.28, "text": " learning algorithm compared to a supervised learning algorithm."}, {"start": 487.28, "end": 492.11999999999995, "text": " But again, if you're worried about the practice lab, the programming exercise, we'll give"}, {"start": 492.11999999999995, "end": 496.44, "text": " you a sense of good parameters to use in the programming exercise so that you should be"}, {"start": 496.44, "end": 502.59999999999997, "text": " able to do that and successfully land the lunar lander, hopefully without too many problems."}, {"start": 502.59999999999997, "end": 508.79999999999995, "text": " In the next optional video, I want to describe a couple more algorithm refinements, mini"}, {"start": 508.8, "end": 514.72, "text": " batching and also using soft updates, even without these additional refinements, the"}, {"start": 514.72, "end": 516.52, "text": " algorithm will work okay."}, {"start": 516.52, "end": 521.12, "text": " But these are additional refinements that make the algorithm run much faster."}, {"start": 521.12, "end": 525.64, "text": " And it's okay if you skip this video, we've provided everything you need in the practice"}, {"start": 525.64, "end": 528.52, "text": " lab to hopefully successfully complete it."}, {"start": 528.52, "end": 532.52, "text": " But if you're interested in learning about more of these details of tuning reinforcement"}, {"start": 532.52, "end": 537.8, "text": " learning algorithms, then come with me and let's see in the next video, mini batching"}, {"start": 537.8, "end": 539.04, "text": " and soft updates."}]