10.5 Reinforcement Learning formalism | Review of key concepts -[Machine Learning| Andrew Ng]
 We've developed a reinforcement learning formalism using the six-state mass rover example. Let's do a quick review of the key concepts and also see how this set of concepts can be used for other applications as well. Some of the concepts we've discussed are states of a reinforcement learning problem, the set of actions, the rewards, a discount factor, then how rewards and the discount factor are together used to compute the return, and then finally a policy whose job it is to help you pick actions so as to maximize the return. For the mass rover example, we had six states that we numbered one to six and the actions were to go left or to go right. The rewards were 100 for the leftmost state, 40 for the rightmost state, and 0 in between, and I was using a discount factor of 0.5. The return was given by this formula and we could have different policies pi that pick actions, depending on what state you're in. This same formalism of states, actions, rewards, and so on can be used for many other applications as well. Take the problem of flying an autonomous helicopter. The set of states would be the set of possible positions and orientations and speeds and so on of the helicopter. The possible actions would be the set of possible ways to move the control stick of a helicopter. And the rewards may be a plus one if it's flying well and a negative 1000 if it doesn't feel really bad or crashes. So a reward function that tells you how well the helicopter is flying. The discount factor, a number slightly less than one, maybe say 0.99. And then based on the rewards and the discount factor, you compute the return using the same formula. And the job of reinforcement learning algorithm would be to find some policy pi of s so that given as input the position of the helicopter s, it tells you what action to take, that is, tells you how to move the control sticks. Here's one more example. Here's a game playing one. Say you want to use reinforcement learning to learn to play chess. The state of this problem would be the position of all the pieces on the board. By the way, if you play chess and know the rules well, I know that there's a little bit more information than just the position of the pieces as important for chess. I'll simplify a little bit for this video. The actions are the possible legal moves in the game. And then a common choice of reward would be if you give your system a reward of plus one, if it wins a game, minus one, if it loses a game, and a reward of zero, if it ties a game. For chess, usually a discount factor very close to one would be used. So maybe 0.99 or even 0.995 or 0.999. And the return uses the same formula as the other applications. And once again, the goal is given a board position to pick a good action using a policy pie. This formalism of a reinforcement learning application actually has a name. It's called a Markov decision process. And I know that sounds like a big technical complicated term. But if you ever hear this term Markov decision process or MDP for short, that's just the formalism that we've been talking about in the last few videos. The term Markov in the MDP or Markov decision process refers to that the future only depends on the current state and not on anything that might have occurred prior to getting to the current state. In other words, in a Markov decision process, the future depends only on where you are now, not on how you got here. One other way to think of the Markov decision process formalism is that we have a robot or some other agent that we wish to control. And what we get to do is choose actions A and based on those actions, something will happen in the world or in the environment, such as opposition in the world changes, or we get to sample a piece of rock and execute the science mission. The way we choose the actions A is with a policy pie. And based on what happens in the world, we then get to see or we observe back what state we're in as well as what reward are that we get. And so you sometimes see different authors use a diagram like this to represent the Markov decision process or the MDP formalism. But this is just another way of illustrating the set of concepts that you learned about in the last few videos. So you now know how a reinforcement learning problem works. In the next video, we'll start to develop an algorithm for picking good actions. The first step to that will be to define and then eventually learn to compute the state action value function. This turns out to be one of the key quantities for when we want to develop a learning algorithm. Let's go on to the next video to see what is this state action value function.
[{"start": 0.0, "end": 8.76, "text": " We've developed a reinforcement learning formalism using the six-state mass rover example."}, {"start": 8.76, "end": 13.44, "text": " Let's do a quick review of the key concepts and also see how this set of concepts can"}, {"start": 13.44, "end": 16.84, "text": " be used for other applications as well."}, {"start": 16.84, "end": 23.32, "text": " Some of the concepts we've discussed are states of a reinforcement learning problem, the set"}, {"start": 23.32, "end": 30.0, "text": " of actions, the rewards, a discount factor, then how rewards and the discount factor are"}, {"start": 30.0, "end": 36.4, "text": " together used to compute the return, and then finally a policy whose job it is to help you"}, {"start": 36.4, "end": 39.36, "text": " pick actions so as to maximize the return."}, {"start": 39.36, "end": 45.5, "text": " For the mass rover example, we had six states that we numbered one to six and the actions"}, {"start": 45.5, "end": 48.72, "text": " were to go left or to go right."}, {"start": 48.72, "end": 56.12, "text": " The rewards were 100 for the leftmost state, 40 for the rightmost state, and 0 in between,"}, {"start": 56.12, "end": 59.72, "text": " and I was using a discount factor of 0.5."}, {"start": 59.72, "end": 65.08, "text": " The return was given by this formula and we could have different policies pi that pick"}, {"start": 65.08, "end": 68.03999999999999, "text": " actions, depending on what state you're in."}, {"start": 68.03999999999999, "end": 73.36, "text": " This same formalism of states, actions, rewards, and so on can be used for many other applications"}, {"start": 73.36, "end": 74.84, "text": " as well."}, {"start": 74.84, "end": 78.28, "text": " Take the problem of flying an autonomous helicopter."}, {"start": 78.28, "end": 84.56, "text": " The set of states would be the set of possible positions and orientations and speeds and"}, {"start": 84.56, "end": 86.56, "text": " so on of the helicopter."}, {"start": 86.56, "end": 93.44, "text": " The possible actions would be the set of possible ways to move the control stick of a helicopter."}, {"start": 93.44, "end": 99.36, "text": " And the rewards may be a plus one if it's flying well and a negative 1000 if it doesn't"}, {"start": 99.36, "end": 101.4, "text": " feel really bad or crashes."}, {"start": 101.4, "end": 105.56, "text": " So a reward function that tells you how well the helicopter is flying."}, {"start": 105.56, "end": 111.0, "text": " The discount factor, a number slightly less than one, maybe say 0.99."}, {"start": 111.0, "end": 116.04, "text": " And then based on the rewards and the discount factor, you compute the return using the same"}, {"start": 116.04, "end": 117.88, "text": " formula."}, {"start": 117.88, "end": 124.56, "text": " And the job of reinforcement learning algorithm would be to find some policy pi of s so that"}, {"start": 124.56, "end": 130.08, "text": " given as input the position of the helicopter s, it tells you what action to take, that"}, {"start": 130.08, "end": 132.56, "text": " is, tells you how to move the control sticks."}, {"start": 132.56, "end": 134.04, "text": " Here's one more example."}, {"start": 134.04, "end": 135.56, "text": " Here's a game playing one."}, {"start": 135.56, "end": 139.23999999999998, "text": " Say you want to use reinforcement learning to learn to play chess."}, {"start": 139.23999999999998, "end": 144.79999999999998, "text": " The state of this problem would be the position of all the pieces on the board."}, {"start": 144.79999999999998, "end": 149.51999999999998, "text": " By the way, if you play chess and know the rules well, I know that there's a little bit"}, {"start": 149.51999999999998, "end": 153.56, "text": " more information than just the position of the pieces as important for chess."}, {"start": 153.56, "end": 156.64, "text": " I'll simplify a little bit for this video."}, {"start": 156.64, "end": 161.0, "text": " The actions are the possible legal moves in the game."}, {"start": 161.0, "end": 166.4, "text": " And then a common choice of reward would be if you give your system a reward of plus one,"}, {"start": 166.4, "end": 172.8, "text": " if it wins a game, minus one, if it loses a game, and a reward of zero, if it ties a"}, {"start": 172.8, "end": 174.04, "text": " game."}, {"start": 174.04, "end": 178.92000000000002, "text": " For chess, usually a discount factor very close to one would be used."}, {"start": 178.92000000000002, "end": 185.32, "text": " So maybe 0.99 or even 0.995 or 0.999."}, {"start": 185.32, "end": 188.8, "text": " And the return uses the same formula as the other applications."}, {"start": 188.8, "end": 197.28, "text": " And once again, the goal is given a board position to pick a good action using a policy"}, {"start": 197.28, "end": 198.4, "text": " pie."}, {"start": 198.4, "end": 203.76000000000002, "text": " This formalism of a reinforcement learning application actually has a name."}, {"start": 203.76000000000002, "end": 206.76000000000002, "text": " It's called a Markov decision process."}, {"start": 206.76000000000002, "end": 210.20000000000002, "text": " And I know that sounds like a big technical complicated term."}, {"start": 210.20000000000002, "end": 216.96, "text": " But if you ever hear this term Markov decision process or MDP for short, that's just the"}, {"start": 216.96, "end": 220.68, "text": " formalism that we've been talking about in the last few videos."}, {"start": 220.68, "end": 228.12, "text": " The term Markov in the MDP or Markov decision process refers to that the future only depends"}, {"start": 228.12, "end": 233.12, "text": " on the current state and not on anything that might have occurred prior to getting to the"}, {"start": 233.12, "end": 234.4, "text": " current state."}, {"start": 234.4, "end": 240.76000000000002, "text": " In other words, in a Markov decision process, the future depends only on where you are now,"}, {"start": 240.76000000000002, "end": 242.84, "text": " not on how you got here."}, {"start": 242.84, "end": 250.24, "text": " One other way to think of the Markov decision process formalism is that we have a robot"}, {"start": 250.24, "end": 255.32, "text": " or some other agent that we wish to control."}, {"start": 255.32, "end": 264.32, "text": " And what we get to do is choose actions A and based on those actions, something will"}, {"start": 264.32, "end": 271.0, "text": " happen in the world or in the environment, such as opposition in the world changes, or"}, {"start": 271.0, "end": 275.0, "text": " we get to sample a piece of rock and execute the science mission."}, {"start": 275.0, "end": 278.68, "text": " The way we choose the actions A is with a policy pie."}, {"start": 278.68, "end": 285.16, "text": " And based on what happens in the world, we then get to see or we observe back what state"}, {"start": 285.16, "end": 291.62, "text": " we're in as well as what reward are that we get."}, {"start": 291.62, "end": 297.32, "text": " And so you sometimes see different authors use a diagram like this to represent the Markov"}, {"start": 297.32, "end": 300.56, "text": " decision process or the MDP formalism."}, {"start": 300.56, "end": 305.04, "text": " But this is just another way of illustrating the set of concepts that you learned about"}, {"start": 305.04, "end": 306.72, "text": " in the last few videos."}, {"start": 306.72, "end": 311.72, "text": " So you now know how a reinforcement learning problem works."}, {"start": 311.72, "end": 316.56, "text": " In the next video, we'll start to develop an algorithm for picking good actions."}, {"start": 316.56, "end": 321.52, "text": " The first step to that will be to define and then eventually learn to compute the state"}, {"start": 321.52, "end": 323.28, "text": " action value function."}, {"start": 323.28, "end": 329.6, "text": " This turns out to be one of the key quantities for when we want to develop a learning algorithm."}, {"start": 329.6, "end": 334.08000000000004, "text": " Let's go on to the next video to see what is this state action value function."}]