10.4 Reinforcement Learning formalism | Making decisions: Policies in reinforcement learning -ML| Ng
 Let's formalize how a reinforcement learning algorithm picks actions. In this video, you'll learn about what is a policy in a reinforcement learning algorithm. Let's take a look. As we've seen, there are many different ways that you can take actions in a reinforcement learning problem. For example, we could decide to always go for the nearer reward. So you go left if this leftmost reward is nearer, or go right if this rightmost reward is nearer. Another way we could choose actions is to always go for the larger reward. Or we could always go for the smaller reward. It doesn't seem like a good idea, but it is another option. Or you could choose to go left unless you're just one step away from the lesser reward, in which case you go for that one. In reinforcement learning, our goal is to come up with a function which is called a policy pi whose job it is to take as input any state s and map it to some action a that it wants us to take. So for example, for this policy here at the bottom, this policy would say that if you're in state two, then it maps us to the left action. If you're in state three, the policy says go left. If you're in state four, also go left. And if you're in state five, go right. And so pi applied to state s tells us what action it wants us to take in that state. And so the goal of reinforcement learning is to find a policy pi or pi of s that tells you what action to take in every state so as to maximize the return. By the way, I don't know if policy is the most descriptive term of what pi is, but it's one of those terms that's become standard in reinforcement learning. Maybe calling pi a controller rather than a policy would be more natural terminology. But policy is what everyone in reinforcement learning now calls this. In the last video, we've gone through quite a few concepts in reinforcement learning from states to actions to rewards to returns to policies. Let's do a quick review of them in the next video and then we'll go on to start developing algorithms for finding good policies. Let's go on to the next video.
[{"start": 0.0, "end": 6.12, "text": " Let's formalize how a reinforcement learning algorithm picks actions."}, {"start": 6.12, "end": 10.88, "text": " In this video, you'll learn about what is a policy in a reinforcement learning algorithm."}, {"start": 10.88, "end": 12.76, "text": " Let's take a look."}, {"start": 12.76, "end": 17.88, "text": " As we've seen, there are many different ways that you can take actions in a reinforcement"}, {"start": 17.88, "end": 19.64, "text": " learning problem."}, {"start": 19.64, "end": 25.96, "text": " For example, we could decide to always go for the nearer reward."}, {"start": 25.96, "end": 32.0, "text": " So you go left if this leftmost reward is nearer, or go right if this rightmost reward"}, {"start": 32.0, "end": 34.28, "text": " is nearer."}, {"start": 34.28, "end": 39.88, "text": " Another way we could choose actions is to always go for the larger reward."}, {"start": 39.88, "end": 43.480000000000004, "text": " Or we could always go for the smaller reward."}, {"start": 43.480000000000004, "end": 47.24, "text": " It doesn't seem like a good idea, but it is another option."}, {"start": 47.24, "end": 52.88, "text": " Or you could choose to go left unless you're just one step away from the lesser reward,"}, {"start": 52.88, "end": 55.68, "text": " in which case you go for that one."}, {"start": 55.68, "end": 63.0, "text": " In reinforcement learning, our goal is to come up with a function which is called a"}, {"start": 63.0, "end": 73.68, "text": " policy pi whose job it is to take as input any state s and map it to some action a that"}, {"start": 73.68, "end": 75.96000000000001, "text": " it wants us to take."}, {"start": 75.96000000000001, "end": 82.28, "text": " So for example, for this policy here at the bottom, this policy would say that if you're"}, {"start": 82.28, "end": 87.84, "text": " in state two, then it maps us to the left action."}, {"start": 87.84, "end": 91.52, "text": " If you're in state three, the policy says go left."}, {"start": 91.52, "end": 94.76, "text": " If you're in state four, also go left."}, {"start": 94.76, "end": 97.84, "text": " And if you're in state five, go right."}, {"start": 97.84, "end": 105.24000000000001, "text": " And so pi applied to state s tells us what action it wants us to take in that state."}, {"start": 105.24, "end": 112.96, "text": " And so the goal of reinforcement learning is to find a policy pi or pi of s that tells"}, {"start": 112.96, "end": 117.67999999999999, "text": " you what action to take in every state so as to maximize the return."}, {"start": 117.67999999999999, "end": 125.03999999999999, "text": " By the way, I don't know if policy is the most descriptive term of what pi is, but it's"}, {"start": 125.03999999999999, "end": 128.56, "text": " one of those terms that's become standard in reinforcement learning."}, {"start": 128.56, "end": 134.51999999999998, "text": " Maybe calling pi a controller rather than a policy would be more natural terminology."}, {"start": 134.52, "end": 139.68, "text": " But policy is what everyone in reinforcement learning now calls this."}, {"start": 139.68, "end": 144.28, "text": " In the last video, we've gone through quite a few concepts in reinforcement learning from"}, {"start": 144.28, "end": 148.60000000000002, "text": " states to actions to rewards to returns to policies."}, {"start": 148.60000000000002, "end": 152.64000000000001, "text": " Let's do a quick review of them in the next video and then we'll go on to start developing"}, {"start": 152.64000000000001, "end": 155.92000000000002, "text": " algorithms for finding good policies."}, {"start": 155.92, "end": 178.56, "text": " Let's go on to the next video."}]