10.2 Reinforcement Learning formalism | Mars rover example -[ML |Andrew Ng]
 To flesh out the reinforcement learning formalism, instead of looking at something as complicated as a helicopter or a robot dog, we're going to use a simplified example that's loosely inspired by the Mars Rover. This is adapted from an example due to Stanford professor Emma Brunskell and one of my collaborators Jack Rite-Argerwal who had actually written code that is actually controlling the Mars Rover right now. It also helped me talk through and help develop this example. Let's take a look. We'll develop reinforcement learning using a simplified example inspired by the Mars Rover. In this application, the Rover can be in any of six positions, as shown by the six boxes here. The Rover might start off, say, in this position in the fourth box shown here. The position of the Mars Rover is called the state in reinforcement learning. I'm going to call these six states, state one, state two, state three, state four, state five, and state six. So the Rover is starting off in state four. Now, the Rover was sent to Mars to try to carry out different science missions. It can go to different places to use its sensors, such as a drill or radar or spectrometer, to analyze the rock at different places on the planet, or go to different places to take interesting pictures for scientists on Earth to look at. In this example, state one, here on the left, has a very interesting surface that scientists would love for the robot to sample. State six also has a pretty interesting surface that scientists would quite like the Rover to sample but not as interesting as state one. So we would more like to carry out the science mission at state one than at state six, but state one is further away. The way we will reflect state one being potentially more valuable is through the reward function. So the reward at state one is 100, and the reward at state six is 40, and the rewards at all of the other states in between, I'm going to write as a reward of zero because there's not as much interesting science to be done at these states two, three, four, and five. On each step, the Rover gets to choose one of two actions. It can either go to the left or it can go to the right. So the question is, what should the Rover do? In reinforcement learning, we pay a lot of attention to the rewards because that's how we know if the robot is doing well or poorly. So let's look at some examples of what might happen. If the robot were to go left starting from state four, then initially starting from state four, it will receive a reward of zero, and after going left, it gets to state three where it receives again a reward of zero, then it gets to state two, receives a reward of zero, and finally it gets to state one where it receives a reward of 100. For this application, I'm going to assume that when it gets to either state one or state six, that the day ends. So in reinforcement learning, we sometimes call this a terminal state. What that means is that after it gets to one of these terminal states, it gets a reward at that state, but then nothing more happens after that. Maybe the robots run out of fuel or run out of time for the day, which is why it only gets to either enjoy the 100 or the 40 reward, but then that's it for the day, and it doesn't get to earn additional rewards after that. Now, instead of going left, the robot could also choose to go to the right, in which case from state four, it would first have a reward of zero, and then it'll move right and get to state five, have another reward of zero, and then it will get to this other terminal state on the right, state six, and get a reward of 40. But going left and going right aren't the only options. One thing the robot could do is it could start from state four and decide to move to the right. So it goes from state four to five, gets a reward of zero in state four, reward of zero in state five, and then maybe it changes its mind and it decides to start going to the left. As follows, in which case, it would get a reward of zero at state four, at state three, at state two, and then a reward of 100 when it gets to state one. In this sequence of actions and states, the robot is wasting a bit of time. So this maybe isn't such a great way to take actions, but it is one choice that the algorithm could pick. But hopefully, it won't pick this one. So to summarize, at every time step, the robot is in some state, which I'm going to call s, and it gets to choose an action, and it also enjoys some rewards, r of s, that it gets from that state. As a result of this action, it gets to some new state, s prime. So as a concrete example, when the robot was in state four and it took the action, go left, it enjoyed, well, maybe didn't enjoy the reward of zero associated with that state four, and it wound up in a new state three. When you learn about specific reinforcement learning algorithms, you see that these four things, the state, action, the reward, and the next state, which is what happens basically every time you take an action, that this be a core elements of what reinforcement learning algorithms will look at when deciding how to take actions. Just for clarity, the reward here, r of s, this is the reward associated with this state. So this reward of zero is associated with the state four rather than with the state three. So that's the formalism of how a reinforcement learning application works. In the next video, let's take a look at how we specify exactly what we want the reinforcement learning algorithm to do. In particular, we'll talk about an important idea in reinforcement learning called the return. Let's go on to the next video to see what that means.
[{"start": 0.0, "end": 5.5600000000000005, "text": " To flesh out the reinforcement learning formalism,"}, {"start": 5.5600000000000005, "end": 10.8, "text": " instead of looking at something as complicated as a helicopter or a robot dog,"}, {"start": 10.8, "end": 17.240000000000002, "text": " we're going to use a simplified example that's loosely inspired by the Mars Rover."}, {"start": 17.240000000000002, "end": 22.16, "text": " This is adapted from an example due to Stanford professor Emma Brunskell and one of"}, {"start": 22.16, "end": 26.52, "text": " my collaborators Jack Rite-Argerwal who had actually written code"}, {"start": 26.52, "end": 29.48, "text": " that is actually controlling the Mars Rover right now."}, {"start": 29.48, "end": 32.96, "text": " It also helped me talk through and help develop this example."}, {"start": 32.96, "end": 34.24, "text": " Let's take a look."}, {"start": 34.24, "end": 38.28, "text": " We'll develop reinforcement learning using"}, {"start": 38.28, "end": 43.24, "text": " a simplified example inspired by the Mars Rover."}, {"start": 43.24, "end": 48.84, "text": " In this application, the Rover can be in any of six positions,"}, {"start": 48.84, "end": 51.72, "text": " as shown by the six boxes here."}, {"start": 51.72, "end": 54.120000000000005, "text": " The Rover might start off, say,"}, {"start": 54.120000000000005, "end": 58.480000000000004, "text": " in this position in the fourth box shown here."}, {"start": 58.48, "end": 64.08, "text": " The position of the Mars Rover is called the state in reinforcement learning."}, {"start": 64.08, "end": 66.47999999999999, "text": " I'm going to call these six states,"}, {"start": 66.47999999999999, "end": 69.2, "text": " state one, state two, state three,"}, {"start": 69.2, "end": 72.36, "text": " state four, state five, and state six."}, {"start": 72.36, "end": 75.8, "text": " So the Rover is starting off in state four."}, {"start": 75.8, "end": 81.84, "text": " Now, the Rover was sent to Mars to try to carry out different science missions."}, {"start": 81.84, "end": 86.24, "text": " It can go to different places to use its sensors,"}, {"start": 86.24, "end": 89.19999999999999, "text": " such as a drill or radar or spectrometer,"}, {"start": 89.19999999999999, "end": 92.64, "text": " to analyze the rock at different places on the planet,"}, {"start": 92.64, "end": 94.36, "text": " or go to different places to take"}, {"start": 94.36, "end": 96.96, "text": " interesting pictures for scientists on Earth to look at."}, {"start": 96.96, "end": 99.83999999999999, "text": " In this example, state one, here on the left,"}, {"start": 99.83999999999999, "end": 105.47999999999999, "text": " has a very interesting surface that scientists would love for the robot to sample."}, {"start": 105.47999999999999, "end": 110.0, "text": " State six also has a pretty interesting surface that scientists would"}, {"start": 110.0, "end": 114.11999999999999, "text": " quite like the Rover to sample but not as interesting as state one."}, {"start": 114.12, "end": 121.80000000000001, "text": " So we would more like to carry out the science mission at state one than at state six,"}, {"start": 121.80000000000001, "end": 124.12, "text": " but state one is further away."}, {"start": 124.12, "end": 131.32, "text": " The way we will reflect state one being potentially more valuable is through the reward function."}, {"start": 131.32, "end": 135.88, "text": " So the reward at state one is 100,"}, {"start": 135.88, "end": 139.72, "text": " and the reward at state six is 40,"}, {"start": 139.72, "end": 143.12, "text": " and the rewards at all of the other states in between,"}, {"start": 143.12, "end": 146.56, "text": " I'm going to write as a reward of zero because there's"}, {"start": 146.56, "end": 152.08, "text": " not as much interesting science to be done at these states two, three, four, and five."}, {"start": 152.08, "end": 156.92000000000002, "text": " On each step, the Rover gets to choose one of two actions."}, {"start": 156.92000000000002, "end": 162.64000000000001, "text": " It can either go to the left or it can go to the right."}, {"start": 162.64000000000001, "end": 166.20000000000002, "text": " So the question is, what should the Rover do?"}, {"start": 166.20000000000002, "end": 170.72, "text": " In reinforcement learning, we pay a lot of attention to the rewards because that's how"}, {"start": 170.72, "end": 174.08, "text": " we know if the robot is doing well or poorly."}, {"start": 174.08, "end": 178.0, "text": " So let's look at some examples of what might happen."}, {"start": 178.0, "end": 181.52, "text": " If the robot were to go left starting from state four,"}, {"start": 181.52, "end": 185.44, "text": " then initially starting from state four,"}, {"start": 185.44, "end": 187.68, "text": " it will receive a reward of zero,"}, {"start": 187.68, "end": 189.64, "text": " and after going left,"}, {"start": 189.64, "end": 193.52, "text": " it gets to state three where it receives again a reward of zero,"}, {"start": 193.52, "end": 195.36, "text": " then it gets to state two,"}, {"start": 195.36, "end": 197.04, "text": " receives a reward of zero,"}, {"start": 197.04, "end": 199.8, "text": " and finally it gets to state one where it receives"}, {"start": 199.8, "end": 201.96, "text": " a reward of 100."}, {"start": 201.96, "end": 207.72, "text": " For this application, I'm going to assume that when it gets to either state one or state six,"}, {"start": 207.72, "end": 209.52, "text": " that the day ends."}, {"start": 209.52, "end": 211.12, "text": " So in reinforcement learning,"}, {"start": 211.12, "end": 215.32000000000002, "text": " we sometimes call this a terminal state."}, {"start": 215.32000000000002, "end": 219.60000000000002, "text": " What that means is that after it gets to one of these terminal states,"}, {"start": 219.60000000000002, "end": 221.44, "text": " it gets a reward at that state,"}, {"start": 221.44, "end": 223.72000000000003, "text": " but then nothing more happens after that."}, {"start": 223.72000000000003, "end": 227.36, "text": " Maybe the robots run out of fuel or run out of time for the day,"}, {"start": 227.36, "end": 234.36, "text": " which is why it only gets to either enjoy the 100 or the 40 reward,"}, {"start": 234.36, "end": 236.4, "text": " but then that's it for the day,"}, {"start": 236.4, "end": 239.32000000000002, "text": " and it doesn't get to earn additional rewards after that."}, {"start": 239.32000000000002, "end": 241.68, "text": " Now, instead of going left,"}, {"start": 241.68, "end": 244.52, "text": " the robot could also choose to go to the right,"}, {"start": 244.52, "end": 246.4, "text": " in which case from state four,"}, {"start": 246.4, "end": 250.84, "text": " it would first have a reward of zero,"}, {"start": 250.84, "end": 253.4, "text": " and then it'll move right and get to state five,"}, {"start": 253.4, "end": 255.52, "text": " have another reward of zero,"}, {"start": 255.52, "end": 258.92, "text": " and then it will get to this other terminal state on the right,"}, {"start": 258.92, "end": 261.72, "text": " state six, and get a reward of 40."}, {"start": 261.72, "end": 266.52, "text": " But going left and going right aren't the only options."}, {"start": 266.52, "end": 272.84000000000003, "text": " One thing the robot could do is it could start from state four and decide to move to the right."}, {"start": 272.84000000000003, "end": 275.36, "text": " So it goes from state four to five,"}, {"start": 275.36, "end": 277.40000000000003, "text": " gets a reward of zero in state four,"}, {"start": 277.40000000000003, "end": 279.0, "text": " reward of zero in state five,"}, {"start": 279.0, "end": 283.32, "text": " and then maybe it changes its mind and it decides to start going to the left."}, {"start": 283.32, "end": 287.84, "text": " As follows, in which case, it would get a reward of zero at state four,"}, {"start": 287.84, "end": 289.8, "text": " at state three, at state two,"}, {"start": 289.8, "end": 293.24, "text": " and then a reward of 100 when it gets to state one."}, {"start": 293.24, "end": 296.8, "text": " In this sequence of actions and states,"}, {"start": 296.8, "end": 298.88, "text": " the robot is wasting a bit of time."}, {"start": 298.88, "end": 302.2, "text": " So this maybe isn't such a great way to take actions,"}, {"start": 302.2, "end": 304.96, "text": " but it is one choice that the algorithm could pick."}, {"start": 304.96, "end": 307.12, "text": " But hopefully, it won't pick this one."}, {"start": 307.12, "end": 310.92, "text": " So to summarize, at every time step,"}, {"start": 310.92, "end": 313.40000000000003, "text": " the robot is in some state,"}, {"start": 313.40000000000003, "end": 315.64000000000004, "text": " which I'm going to call s,"}, {"start": 315.64000000000004, "end": 319.40000000000003, "text": " and it gets to choose an action,"}, {"start": 319.40000000000003, "end": 323.44, "text": " and it also enjoys some rewards,"}, {"start": 323.44, "end": 326.64000000000004, "text": " r of s, that it gets from that state."}, {"start": 326.64000000000004, "end": 328.68, "text": " As a result of this action,"}, {"start": 328.68, "end": 332.08000000000004, "text": " it gets to some new state, s prime."}, {"start": 332.08000000000004, "end": 334.0, "text": " So as a concrete example,"}, {"start": 334.0, "end": 339.08000000000004, "text": " when the robot was in state four and it took the action, go left,"}, {"start": 339.08, "end": 341.12, "text": " it enjoyed, well,"}, {"start": 341.12, "end": 346.03999999999996, "text": " maybe didn't enjoy the reward of zero associated with that state four,"}, {"start": 346.03999999999996, "end": 349.4, "text": " and it wound up in a new state three."}, {"start": 349.4, "end": 352.8, "text": " When you learn about specific reinforcement learning algorithms,"}, {"start": 352.8, "end": 355.4, "text": " you see that these four things,"}, {"start": 355.4, "end": 357.2, "text": " the state, action, the reward,"}, {"start": 357.2, "end": 361.24, "text": " and the next state, which is what happens basically every time you take an action,"}, {"start": 361.24, "end": 365.03999999999996, "text": " that this be a core elements of what reinforcement learning algorithms will"}, {"start": 365.03999999999996, "end": 368.79999999999995, "text": " look at when deciding how to take actions."}, {"start": 368.8, "end": 372.28000000000003, "text": " Just for clarity, the reward here,"}, {"start": 372.28000000000003, "end": 376.04, "text": " r of s, this is the reward associated with this state."}, {"start": 376.04, "end": 381.44, "text": " So this reward of zero is associated with the state four rather than with the state three."}, {"start": 381.44, "end": 386.96000000000004, "text": " So that's the formalism of how a reinforcement learning application works."}, {"start": 386.96000000000004, "end": 390.36, "text": " In the next video, let's take a look at how we specify"}, {"start": 390.36, "end": 393.88, "text": " exactly what we want the reinforcement learning algorithm to do."}, {"start": 393.88, "end": 396.40000000000003, "text": " In particular, we'll talk about an important idea in"}, {"start": 396.4, "end": 398.96, "text": " reinforcement learning called the return."}, {"start": 398.96, "end": 426.52, "text": " Let's go on to the next video to see what that means."}]