10.10 Continuous State Spaces | Example of continuous state space applications -[ML | Andrew Ng]
 Many robotic control applications, including the lunar lander application that you work on in the practice lab, have continuous state spaces. Let's take a look at what that means and how to generalize the concepts we've talked about to these continuous state spaces. The simplified Mars rover example we use, had used a discrete set of states. What that means is that simplified Mars rover could only be in one of six possible positions. But most robots can be in more than one of six or any discrete number of positions. Instead, they can be in any of a very large number of continuous value positions. For example, if the Mars rover could be anywhere on a line, so its position was indicated by a number ranging from zero to six kilometers, where any number in between is valid, that would be an example of a continuous state space. Because the position would be represented by a number, such as that is 2.7 kilometers along or 4.8 kilometers, or any other number between zero and six. Let's look at another example. I'm going to use for this example, the application of controlling a car or a truck. Here's a toy car, right here, toy truck. This one belongs to my daughter. If you're building a self-driving car or a self-driving truck, and you want to control this to drive smoothly, then the state of this truck might include a few numbers, such as its x-position, its y-position, maybe its orientation, what way is it facing. Assuming the truck stays on the ground, you probably don't need to worry about how tall it is, how high up it is. This state would include x, y, and its angle theta, as well as maybe its speed in x direction, the speed in the y direction, and how quickly it is turning. Is it turning at one degree per second, or is it turning at 30 degrees per second, or is it turning really quickly at 90 degrees per second? For a truck or a car, the state might include not just one number, like how many kilometers it is along on this line, but it might include six numbers. Its x-position, its y-position, its orientation, which I'm going to denote using Greek alphabet theta, as well as its velocity in the x direction, which I'm going to denote using x dot. That means how quickly is this x coordinate changing, y dot, how quickly is the y coordinate changing, and then finally theta dot, which is how quickly is the angle of the car changing. Whereas for the 6th state Mars Rover example, the state was just one of six possible numbers. It could be 1, 2, 3, 4, 5, or 6. For the car, the state would comprise this vector of six numbers and any of these numbers can take on any value within this valid range. For example, theta should range between 0 and 360 degrees. Let's look at another example. What if you're building a reinforcement learning algorithm to control an autonomous helicopter? How would you characterize the position of a helicopter? To illustrate, I have with me here a small toy helicopter. The position of the helicopter would include its x-position, such as how far north or south is the helicopter, its y-position, maybe how far on the east-west axis is the helicopter, and then also z, the height of the helicopter above ground. But other than the position, the helicopter also has an orientation. Conventionally, one way to capture this orientation is with three additional numbers. One of which captures the row of the helicopter. Is it rolling to the left or the right? The pitch, is it pitching forward or pitching up, pitching back? Then finally, the yaw, which is what's the compass orientation is it facing? Is it facing north or east or south or west? To summarize, the state of the helicopter includes its position in the, say, north-south direction, its position in the east-west direction, y, its height above ground, and also the row, the pitch, and then also the yaw of the helicopter. To write this down, the state therefore includes the position x, y, z, and then the row, pitch, and yaw denoted with Greek alphabets, Phi, Theta, and Omega. But to control the helicopter, we also need to know its speed in the x direction, in the y direction, and in the z direction, as well as its rate of turning, also called the angular velocity. So how fast is this row changing, and how fast is this pitch changing, and how fast is this yaw changing? So this is actually the state used to control autonomous helicopters. It's this list of 12 numbers that is input to a policy, and the job of a policy is to look at these 12 numbers and decide what's an appropriate action to take in a helicopter. So in a continuous state reinforcement learning problem, or a continuous state Markov decision process, continuous state MTP, the state of the problem isn't just one of a small number of possible discrete values, like a number from one to six. Instead, it's a vector of numbers, any of which could take any of a large number of values. In the practice lab for this week, you get to implement for yourself a reinforcement learning algorithm applied to a simulated lunar lander application, landing something on the moon in simulation. Let's take a look in the next video at what that application entails, since that would be another continuous state application.
[{"start": 0.0, "end": 4.5600000000000005, "text": " Many robotic control applications,"}, {"start": 4.5600000000000005, "end": 9.16, "text": " including the lunar lander application that you work on in the practice lab,"}, {"start": 9.16, "end": 11.52, "text": " have continuous state spaces."}, {"start": 11.52, "end": 14.52, "text": " Let's take a look at what that means and how to generalize"}, {"start": 14.52, "end": 18.34, "text": " the concepts we've talked about to these continuous state spaces."}, {"start": 18.34, "end": 21.92, "text": " The simplified Mars rover example we use,"}, {"start": 21.92, "end": 24.8, "text": " had used a discrete set of states."}, {"start": 24.8, "end": 29.48, "text": " What that means is that simplified Mars rover could only be in"}, {"start": 29.48, "end": 32.36, "text": " one of six possible positions."}, {"start": 32.36, "end": 39.52, "text": " But most robots can be in more than one of six or any discrete number of positions."}, {"start": 39.52, "end": 41.68, "text": " Instead, they can be in any of"}, {"start": 41.68, "end": 46.32, "text": " a very large number of continuous value positions."}, {"start": 46.32, "end": 52.24, "text": " For example, if the Mars rover could be anywhere on a line,"}, {"start": 52.24, "end": 59.46, "text": " so its position was indicated by a number ranging from zero to six kilometers,"}, {"start": 59.46, "end": 61.74, "text": " where any number in between is valid,"}, {"start": 61.74, "end": 66.2, "text": " that would be an example of a continuous state space."}, {"start": 66.2, "end": 70.28, "text": " Because the position would be represented by a number,"}, {"start": 70.28, "end": 75.56, "text": " such as that is 2.7 kilometers along or 4.8 kilometers,"}, {"start": 75.56, "end": 78.06, "text": " or any other number between zero and six."}, {"start": 78.06, "end": 79.9, "text": " Let's look at another example."}, {"start": 79.9, "end": 82.28, "text": " I'm going to use for this example,"}, {"start": 82.28, "end": 85.52000000000001, "text": " the application of controlling a car or a truck."}, {"start": 85.52000000000001, "end": 87.7, "text": " Here's a toy car, right here, toy truck."}, {"start": 87.7, "end": 89.72, "text": " This one belongs to my daughter."}, {"start": 89.72, "end": 93.04, "text": " If you're building a self-driving car or a self-driving truck,"}, {"start": 93.04, "end": 95.76, "text": " and you want to control this to drive smoothly,"}, {"start": 95.76, "end": 99.72, "text": " then the state of this truck might include a few numbers,"}, {"start": 99.72, "end": 103.88, "text": " such as its x-position, its y-position,"}, {"start": 103.88, "end": 107.16, "text": " maybe its orientation, what way is it facing."}, {"start": 107.16, "end": 109.12, "text": " Assuming the truck stays on the ground,"}, {"start": 109.12, "end": 113.32000000000001, "text": " you probably don't need to worry about how tall it is, how high up it is."}, {"start": 113.32, "end": 119.52, "text": " This state would include x, y, and its angle theta,"}, {"start": 119.52, "end": 122.88, "text": " as well as maybe its speed in x direction,"}, {"start": 122.88, "end": 124.67999999999999, "text": " the speed in the y direction,"}, {"start": 124.67999999999999, "end": 126.19999999999999, "text": " and how quickly it is turning."}, {"start": 126.19999999999999, "end": 128.23999999999998, "text": " Is it turning at one degree per second,"}, {"start": 128.23999999999998, "end": 130.28, "text": " or is it turning at 30 degrees per second,"}, {"start": 130.28, "end": 133.6, "text": " or is it turning really quickly at 90 degrees per second?"}, {"start": 133.6, "end": 136.79999999999998, "text": " For a truck or a car,"}, {"start": 136.79999999999998, "end": 140.95999999999998, "text": " the state might include not just one number,"}, {"start": 140.96, "end": 144.0, "text": " like how many kilometers it is along on this line,"}, {"start": 144.0, "end": 146.28, "text": " but it might include six numbers."}, {"start": 146.28, "end": 149.20000000000002, "text": " Its x-position, its y-position,"}, {"start": 149.20000000000002, "end": 154.56, "text": " its orientation, which I'm going to denote using Greek alphabet theta,"}, {"start": 154.56, "end": 157.68, "text": " as well as its velocity in the x direction,"}, {"start": 157.68, "end": 160.0, "text": " which I'm going to denote using x dot."}, {"start": 160.0, "end": 163.84, "text": " That means how quickly is this x coordinate changing,"}, {"start": 163.84, "end": 167.60000000000002, "text": " y dot, how quickly is the y coordinate changing,"}, {"start": 167.60000000000002, "end": 169.38, "text": " and then finally theta dot,"}, {"start": 169.38, "end": 174.84, "text": " which is how quickly is the angle of the car changing."}, {"start": 174.84, "end": 178.14, "text": " Whereas for the 6th state Mars Rover example,"}, {"start": 178.14, "end": 181.72, "text": " the state was just one of six possible numbers."}, {"start": 181.72, "end": 185.56, "text": " It could be 1, 2, 3, 4, 5, or 6."}, {"start": 185.56, "end": 190.32, "text": " For the car, the state would comprise this vector of"}, {"start": 190.32, "end": 194.04, "text": " six numbers and any of these numbers can take on"}, {"start": 194.04, "end": 198.0, "text": " any value within this valid range."}, {"start": 198.0, "end": 203.4, "text": " For example, theta should range between 0 and 360 degrees."}, {"start": 203.4, "end": 205.56, "text": " Let's look at another example."}, {"start": 205.56, "end": 207.04, "text": " What if you're building"}, {"start": 207.04, "end": 211.2, "text": " a reinforcement learning algorithm to control an autonomous helicopter?"}, {"start": 211.2, "end": 214.36, "text": " How would you characterize the position of a helicopter?"}, {"start": 214.36, "end": 217.84, "text": " To illustrate, I have with me here a small toy helicopter."}, {"start": 217.84, "end": 222.08, "text": " The position of the helicopter would include its x-position,"}, {"start": 222.08, "end": 225.6, "text": " such as how far north or south is the helicopter,"}, {"start": 225.6, "end": 231.12, "text": " its y-position, maybe how far on the east-west axis is the helicopter,"}, {"start": 231.12, "end": 235.16, "text": " and then also z, the height of the helicopter above ground."}, {"start": 235.16, "end": 237.72, "text": " But other than the position,"}, {"start": 237.72, "end": 240.64, "text": " the helicopter also has an orientation."}, {"start": 240.64, "end": 245.57999999999998, "text": " Conventionally, one way to capture this orientation is with three additional numbers."}, {"start": 245.57999999999998, "end": 249.24, "text": " One of which captures the row of the helicopter."}, {"start": 249.24, "end": 251.28, "text": " Is it rolling to the left or the right?"}, {"start": 251.28, "end": 255.04, "text": " The pitch, is it pitching forward or pitching up, pitching back?"}, {"start": 255.04, "end": 256.96, "text": " Then finally, the yaw,"}, {"start": 256.96, "end": 260.0, "text": " which is what's the compass orientation is it facing?"}, {"start": 260.0, "end": 262.71999999999997, "text": " Is it facing north or east or south or west?"}, {"start": 262.71999999999997, "end": 268.4, "text": " To summarize, the state of the helicopter includes its position in the,"}, {"start": 268.4, "end": 270.2, "text": " say, north-south direction,"}, {"start": 270.2, "end": 273.32, "text": " its position in the east-west direction, y,"}, {"start": 273.32, "end": 276.92, "text": " its height above ground, and also the row,"}, {"start": 276.92, "end": 282.14, "text": " the pitch, and then also the yaw of the helicopter."}, {"start": 282.14, "end": 283.88, "text": " To write this down,"}, {"start": 283.88, "end": 287.71999999999997, "text": " the state therefore includes the position x, y,"}, {"start": 287.71999999999997, "end": 291.96, "text": " z, and then the row, pitch,"}, {"start": 291.96, "end": 296.32, "text": " and yaw denoted with Greek alphabets,"}, {"start": 296.32, "end": 299.52, "text": " Phi, Theta, and Omega."}, {"start": 299.52, "end": 300.96, "text": " But to control the helicopter,"}, {"start": 300.96, "end": 305.68, "text": " we also need to know its speed in the x direction,"}, {"start": 305.68, "end": 307.2, "text": " in the y direction,"}, {"start": 307.2, "end": 308.92, "text": " and in the z direction,"}, {"start": 308.92, "end": 311.32, "text": " as well as its rate of turning,"}, {"start": 311.32, "end": 313.24, "text": " also called the angular velocity."}, {"start": 313.24, "end": 315.92, "text": " So how fast is this row changing,"}, {"start": 315.92, "end": 318.28000000000003, "text": " and how fast is this pitch changing,"}, {"start": 318.28000000000003, "end": 321.40000000000003, "text": " and how fast is this yaw changing?"}, {"start": 321.40000000000003, "end": 326.12, "text": " So this is actually the state used to control autonomous helicopters."}, {"start": 326.12, "end": 331.64, "text": " It's this list of 12 numbers that is input to a policy,"}, {"start": 331.64, "end": 335.36, "text": " and the job of a policy is to look at these 12 numbers and"}, {"start": 335.36, "end": 338.6, "text": " decide what's an appropriate action to take in a helicopter."}, {"start": 338.6, "end": 342.2, "text": " So in a continuous state reinforcement learning problem,"}, {"start": 342.2, "end": 345.32, "text": " or a continuous state Markov decision process,"}, {"start": 345.32, "end": 346.96, "text": " continuous state MTP,"}, {"start": 346.96, "end": 350.28, "text": " the state of the problem isn't just one of"}, {"start": 350.28, "end": 352.96, "text": " a small number of possible discrete values,"}, {"start": 352.96, "end": 354.76, "text": " like a number from one to six."}, {"start": 354.76, "end": 358.59999999999997, "text": " Instead, it's a vector of numbers,"}, {"start": 358.59999999999997, "end": 363.12, "text": " any of which could take any of a large number of values."}, {"start": 363.12, "end": 365.68, "text": " In the practice lab for this week,"}, {"start": 365.68, "end": 367.64, "text": " you get to implement for yourself"}, {"start": 367.64, "end": 370.0, "text": " a reinforcement learning algorithm applied to"}, {"start": 370.0, "end": 373.44, "text": " a simulated lunar lander application,"}, {"start": 373.44, "end": 376.76, "text": " landing something on the moon in simulation."}, {"start": 376.76, "end": 381.0, "text": " Let's take a look in the next video at what that application entails,"}, {"start": 381.0, "end": 400.84, "text": " since that would be another continuous state application."}]