10.7 State-action value function | State-action value function example -[Machine Learning|Andrew Ng]
 Using the Mars Rover example, you've seen what the values of QSA are like. In order to keep holding our intuition about reinforcement learning problems and how the values of QSA change depending on the problem, we've provided an optional lab that lets you play around, modify the Mars Rover example and see for yourself how Q of SA will change. Let's take a look. Here's a Jupyter notebook that I hope you play with. After watching this video, I'm going to run these helper functions. Now, notice here that this specifies the number of states, that are two actions, so I wouldn't change these. This specifies the terminal left and the terminal right rewards, which has been 140, and then zero was the rewards of the intermediate states. The discount factor gamma was 0.5, and let's ignore the misstep probability for now. We'll talk about that in a later video. With these values, if you run this code, this will compute and visualize the optimal policy, as well as the Q function, Q of SA. You learn later about how to develop a learning algorithm to estimate or compute Q of SA yourself. For now, don't worry about what code we had written to compute Q of SA, but you see that the values here, Q of SA are the values we saw in the lecture. Now, here's where the fun starts. Let's change around some of the values and see how these things change. I'm going to update the terminal right reward to a much smaller value, say it's only 10. If I now rerun the code, look at how Q of SA changes. It now thinks that if you're in state five, then if you go left and behave optimally, you get 6.25, whereas if you go right and behave optimally after that, you get a return of only five. Now, when the reward at the right is so small, it's only 10, even when you're so close to it, you rather go left all the way. In fact, the optimal policy is now to go left from every single state. Let's make some other changes. I'm going to change the terminal right reward back to 40, but let me change the discount factor to 0.9. With a discount factor that's closer to one, this makes the mouse rover less impatient. It's willing to take longer to hold out for a higher reward because rewards in the future are not multiplied by 0.5 to some high power, it's multiplied by 0.9 to some high power, and so it's willing to be more patient because rewards in the future are not discounted or multiplied by as small a number, as when the discount was 0.5. So let's rerun the code. And now you see this is Q of sA for the different states. And now for state five, going left actually gives you a higher reward of 65.61 compared to 36. Notice by the way that 36 is 0.9 times this terminal reward of 40. So these numbers make sense. But when it's more patient, it's willing to go to the left even when you're in state five. Now let's change gamma to a much smaller number like 0.3. So this very heavily discounts rewards in the future. This makes it incredibly impatient. So let me rerun this code. And now the behavior has changed. Notice that now in state four is not going to have the patience to go for the larger 100 reward because the discount factor gamma is now so small is 0.3, it would rather go for the reward of 40. Even though it's a much smaller reward is closer, and that's what it would choose to do. So I hope that you can get a sense by playing around with these numbers yourself and running this code how the values of Q of SA change as well as how the optimal return, which you notice is a larger of these two numbers QSA, how that changes as well as how the optimal policy also changes. So I hope you go and play with the optional lab and change the reward function and change the discount factor gamma, and try different values and see for yourself how the values of Q of SA change, how the optimal return from different states change, and how the optimal policy changes depending on these different values. And by doing so, I hope that will sharpen your intuition about how these different quantities are affected depending on the rewards and so on in reinforcement learning application. After you play through the lab, we then be ready to come back and talk about what's probably the single most important equation in reinforcement learning, which is something called the Bellman equation. So I hope you have fun playing with the optional lab. And after that, let's come back to talk about Bellman equations.
[{"start": 0.0, "end": 4.2, "text": " Using the Mars Rover example,"}, {"start": 4.2, "end": 7.88, "text": " you've seen what the values of QSA are like."}, {"start": 7.88, "end": 11.24, "text": " In order to keep holding our intuition about"}, {"start": 11.24, "end": 14.48, "text": " reinforcement learning problems and how the values of"}, {"start": 14.48, "end": 17.76, "text": " QSA change depending on the problem,"}, {"start": 17.76, "end": 21.28, "text": " we've provided an optional lab that lets you play around,"}, {"start": 21.28, "end": 23.84, "text": " modify the Mars Rover example and see for"}, {"start": 23.84, "end": 27.560000000000002, "text": " yourself how Q of SA will change. Let's take a look."}, {"start": 27.56, "end": 30.919999999999998, "text": " Here's a Jupyter notebook that I hope you play with."}, {"start": 30.919999999999998, "end": 32.58, "text": " After watching this video,"}, {"start": 32.58, "end": 35.1, "text": " I'm going to run these helper functions."}, {"start": 35.1, "end": 39.519999999999996, "text": " Now, notice here that this specifies the number of states,"}, {"start": 39.519999999999996, "end": 40.76, "text": " that are two actions,"}, {"start": 40.76, "end": 43.04, "text": " so I wouldn't change these."}, {"start": 43.04, "end": 46.58, "text": " This specifies the terminal left and the terminal right rewards,"}, {"start": 46.58, "end": 48.519999999999996, "text": " which has been 140,"}, {"start": 48.519999999999996, "end": 53.04, "text": " and then zero was the rewards of the intermediate states."}, {"start": 53.04, "end": 55.519999999999996, "text": " The discount factor gamma was 0.5,"}, {"start": 55.52, "end": 58.480000000000004, "text": " and let's ignore the misstep probability for now."}, {"start": 58.480000000000004, "end": 60.56, "text": " We'll talk about that in a later video."}, {"start": 60.56, "end": 62.580000000000005, "text": " With these values,"}, {"start": 62.580000000000005, "end": 65.22, "text": " if you run this code,"}, {"start": 65.22, "end": 69.32000000000001, "text": " this will compute and visualize the optimal policy,"}, {"start": 69.32000000000001, "end": 73.44, "text": " as well as the Q function, Q of SA."}, {"start": 73.44, "end": 75.96000000000001, "text": " You learn later about how to develop"}, {"start": 75.96000000000001, "end": 80.12, "text": " a learning algorithm to estimate or compute Q of SA yourself."}, {"start": 80.12, "end": 85.12, "text": " For now, don't worry about what code we had written to compute Q of SA,"}, {"start": 85.12, "end": 87.32000000000001, "text": " but you see that the values here,"}, {"start": 87.32000000000001, "end": 91.12, "text": " Q of SA are the values we saw in the lecture."}, {"start": 91.12, "end": 93.32000000000001, "text": " Now, here's where the fun starts."}, {"start": 93.32000000000001, "end": 97.64, "text": " Let's change around some of the values and see how these things change."}, {"start": 97.64, "end": 103.32000000000001, "text": " I'm going to update the terminal right reward to a much smaller value,"}, {"start": 103.32000000000001, "end": 104.92, "text": " say it's only 10."}, {"start": 104.92, "end": 107.16, "text": " If I now rerun the code,"}, {"start": 107.16, "end": 110.48, "text": " look at how Q of SA changes."}, {"start": 110.48, "end": 114.44, "text": " It now thinks that if you're in state five,"}, {"start": 114.44, "end": 117.2, "text": " then if you go left and behave optimally,"}, {"start": 117.2, "end": 118.96, "text": " you get 6.25,"}, {"start": 118.96, "end": 121.96, "text": " whereas if you go right and behave optimally after that,"}, {"start": 121.96, "end": 123.96, "text": " you get a return of only five."}, {"start": 123.96, "end": 126.96, "text": " Now, when the reward at the right is so small,"}, {"start": 126.96, "end": 129.8, "text": " it's only 10, even when you're so close to it,"}, {"start": 129.8, "end": 132.64, "text": " you rather go left all the way."}, {"start": 132.64, "end": 137.6, "text": " In fact, the optimal policy is now to go left from every single state."}, {"start": 137.6, "end": 139.26, "text": " Let's make some other changes."}, {"start": 139.26, "end": 142.78, "text": " I'm going to change the terminal right reward back to 40,"}, {"start": 142.78, "end": 148.48, "text": " but let me change the discount factor to 0.9."}, {"start": 148.48, "end": 152.04, "text": " With a discount factor that's closer to one,"}, {"start": 152.04, "end": 155.28, "text": " this makes the mouse rover less impatient."}, {"start": 155.28, "end": 160.84, "text": " It's willing to take longer to hold out for a higher reward because rewards in"}, {"start": 160.84, "end": 165.48, "text": " the future are not multiplied by 0.5 to some high power,"}, {"start": 165.48, "end": 168.6, "text": " it's multiplied by 0.9 to some high power,"}, {"start": 168.6, "end": 172.24, "text": " and so it's willing to be more patient because"}, {"start": 172.24, "end": 177.60000000000002, "text": " rewards in the future are not discounted or multiplied by as small a number,"}, {"start": 177.60000000000002, "end": 180.12, "text": " as when the discount was 0.5."}, {"start": 180.12, "end": 182.44, "text": " So let's rerun the code."}, {"start": 182.44, "end": 187.56, "text": " And now you see this is Q of sA for the different states."}, {"start": 187.56, "end": 190.42000000000002, "text": " And now for state five,"}, {"start": 190.42000000000002, "end": 198.56, "text": " going left actually gives you a higher reward of 65.61 compared to 36."}, {"start": 198.56, "end": 203.32, "text": " Notice by the way that 36 is 0.9 times this terminal reward of 40."}, {"start": 203.32, "end": 204.8, "text": " So these numbers make sense."}, {"start": 204.8, "end": 206.12, "text": " But when it's more patient,"}, {"start": 206.12, "end": 209.84, "text": " it's willing to go to the left even when you're in state five."}, {"start": 209.84, "end": 215.2, "text": " Now let's change gamma to a much smaller number like 0.3."}, {"start": 215.2, "end": 218.52, "text": " So this very heavily discounts rewards in the future."}, {"start": 218.52, "end": 221.02, "text": " This makes it incredibly impatient."}, {"start": 221.02, "end": 223.04, "text": " So let me rerun this code."}, {"start": 223.04, "end": 224.82, "text": " And now the behavior has changed."}, {"start": 224.82, "end": 230.84, "text": " Notice that now in state four is not going to have"}, {"start": 230.84, "end": 235.4, "text": " the patience to go for the larger 100 reward because"}, {"start": 235.4, "end": 238.76, "text": " the discount factor gamma is now so small is 0.3,"}, {"start": 238.76, "end": 242.0, "text": " it would rather go for the reward of 40."}, {"start": 242.0, "end": 244.76, "text": " Even though it's a much smaller reward is closer,"}, {"start": 244.76, "end": 246.51999999999998, "text": " and that's what it would choose to do."}, {"start": 246.51999999999998, "end": 250.6, "text": " So I hope that you can get a sense by playing around with"}, {"start": 250.6, "end": 253.72, "text": " these numbers yourself and running this code how"}, {"start": 253.72, "end": 259.36, "text": " the values of Q of SA change as well as how the optimal return,"}, {"start": 259.36, "end": 263.92, "text": " which you notice is a larger of these two numbers QSA,"}, {"start": 263.92, "end": 269.28, "text": " how that changes as well as how the optimal policy also changes."}, {"start": 269.28, "end": 273.32, "text": " So I hope you go and play with the optional lab and"}, {"start": 273.32, "end": 277.84, "text": " change the reward function and change the discount factor gamma,"}, {"start": 277.84, "end": 283.16, "text": " and try different values and see for yourself how the values of Q of SA change,"}, {"start": 283.16, "end": 285.96000000000004, "text": " how the optimal return from different states change,"}, {"start": 285.96000000000004, "end": 290.52000000000004, "text": " and how the optimal policy changes depending on these different values."}, {"start": 290.52000000000004, "end": 291.96000000000004, "text": " And by doing so,"}, {"start": 291.96000000000004, "end": 296.92, "text": " I hope that will sharpen your intuition about how these different quantities"}, {"start": 296.92, "end": 303.24, "text": " are affected depending on the rewards and so on in reinforcement learning application."}, {"start": 303.24, "end": 305.20000000000005, "text": " After you play through the lab,"}, {"start": 305.20000000000005, "end": 308.6, "text": " we then be ready to come back and talk about what's probably"}, {"start": 308.6, "end": 311.74, "text": " the single most important equation in reinforcement learning,"}, {"start": 311.74, "end": 314.48, "text": " which is something called the Bellman equation."}, {"start": 314.48, "end": 318.16, "text": " So I hope you have fun playing with the optional lab."}, {"start": 318.16, "end": 342.52000000000004, "text": " And after that, let's come back to talk about Bellman equations."}]