10.13 Continuous State Spaces|Algorithm refinement Improved neural network architecture-ML Andrew Ng
 In the last video, we saw a neural network architecture that would input the state in action and attempt to output the Q function, Q of sA. It turns out that there's a change to neural network architecture that makes this algorithm much more efficient. So most implementations of DQN actually use this more efficient architecture that we'll see in this video. Let's take a look. This was the neural network architecture we saw previously where it would input 12 numbers and output Q of sA. Whenever we are in some state s, we would have to carry out inference in the neural network separately four times to compute these four values so as to pick the action A that gives us the largest Q value. This is inefficient because we have to carry out inference four times from every single state. Instead, it turns out to be more efficient to train a single neural network to output all four of these values simultaneously. This is what it looks like. Here's a modified neural network architecture where the input is eight numbers corresponding to the state of the lunar lander. It then goes through the neural network with 64 units in the first hidden layer, 64 units in the second hidden layer. Now the output unit has four output units. The job of the neural network is to have the four output units output Q of s nothing, Q of s left, Q of s main, and Q of s right. The job of the neural network is to compute simultaneously the Q value for all four possible actions for when we are in the state s. This turns out to be more efficient because given a state s, we can run inference just once and get all four of these values and then very quickly pick the action A that maximizes Q of sA. You notice also in Bellman's equations, there's a step in which we have to compute max over A prime, Q of s prime A prime, this is multiplied by gamma and then there was plus R of s up here. This neural network also makes it much more efficient to compute this because we're getting Q of s prime A prime for all actions A prime at the same time. So you then just pick the max to compute this value for the right hand side of Bellman's equations. This change to the neural network architecture makes the algorithm much more efficient and so we will be using this architecture in the practice lab. Next, there's one other idea that will help the algorithm a lot, which is something called an Epsilon greedy policy, which affects how you choose actions even while you're still learning. Let's take a look at the next video at what that means.
[{"start": 0.0, "end": 7.640000000000001, "text": " In the last video, we saw a neural network architecture that would input the state in"}, {"start": 7.640000000000001, "end": 12.82, "text": " action and attempt to output the Q function, Q of sA."}, {"start": 12.82, "end": 17.12, "text": " It turns out that there's a change to neural network architecture that makes this algorithm"}, {"start": 17.12, "end": 18.92, "text": " much more efficient."}, {"start": 18.92, "end": 24.28, "text": " So most implementations of DQN actually use this more efficient architecture that we'll"}, {"start": 24.28, "end": 25.28, "text": " see in this video."}, {"start": 25.28, "end": 27.240000000000002, "text": " Let's take a look."}, {"start": 27.24, "end": 33.36, "text": " This was the neural network architecture we saw previously where it would input 12 numbers"}, {"start": 33.36, "end": 35.64, "text": " and output Q of sA."}, {"start": 35.64, "end": 41.56, "text": " Whenever we are in some state s, we would have to carry out inference in the neural"}, {"start": 41.56, "end": 48.84, "text": " network separately four times to compute these four values so as to pick the action A that"}, {"start": 48.84, "end": 52.16, "text": " gives us the largest Q value."}, {"start": 52.16, "end": 58.12, "text": " This is inefficient because we have to carry out inference four times from every single"}, {"start": 58.12, "end": 59.12, "text": " state."}, {"start": 59.12, "end": 64.36, "text": " Instead, it turns out to be more efficient to train a single neural network to output"}, {"start": 64.36, "end": 69.22, "text": " all four of these values simultaneously."}, {"start": 69.22, "end": 70.75999999999999, "text": " This is what it looks like."}, {"start": 70.75999999999999, "end": 78.19999999999999, "text": " Here's a modified neural network architecture where the input is eight numbers corresponding"}, {"start": 78.19999999999999, "end": 81.36, "text": " to the state of the lunar lander."}, {"start": 81.36, "end": 86.03999999999999, "text": " It then goes through the neural network with 64 units in the first hidden layer, 64 units"}, {"start": 86.03999999999999, "end": 87.92, "text": " in the second hidden layer."}, {"start": 87.92, "end": 92.68, "text": " Now the output unit has four output units."}, {"start": 92.68, "end": 99.56, "text": " The job of the neural network is to have the four output units output Q of s nothing, Q"}, {"start": 99.56, "end": 104.4, "text": " of s left, Q of s main, and Q of s right."}, {"start": 104.4, "end": 110.44, "text": " The job of the neural network is to compute simultaneously the Q value for all four possible"}, {"start": 110.44, "end": 115.52, "text": " actions for when we are in the state s."}, {"start": 115.52, "end": 120.72, "text": " This turns out to be more efficient because given a state s, we can run inference just"}, {"start": 120.72, "end": 128.04, "text": " once and get all four of these values and then very quickly pick the action A that maximizes"}, {"start": 128.04, "end": 129.88, "text": " Q of sA."}, {"start": 129.88, "end": 135.96, "text": " You notice also in Bellman's equations, there's a step in which we have to compute max over"}, {"start": 135.96, "end": 142.96, "text": " A prime, Q of s prime A prime, this is multiplied by gamma and then there was plus R of s up"}, {"start": 142.96, "end": 144.28, "text": " here."}, {"start": 144.28, "end": 149.54000000000002, "text": " This neural network also makes it much more efficient to compute this because we're getting"}, {"start": 149.54000000000002, "end": 153.56, "text": " Q of s prime A prime for all actions A prime at the same time."}, {"start": 153.56, "end": 159.08, "text": " So you then just pick the max to compute this value for the right hand side of Bellman's"}, {"start": 159.08, "end": 160.08, "text": " equations."}, {"start": 160.08, "end": 163.76000000000002, "text": " This change to the neural network architecture makes the algorithm much more efficient and"}, {"start": 163.76, "end": 168.2, "text": " so we will be using this architecture in the practice lab."}, {"start": 168.2, "end": 172.76, "text": " Next, there's one other idea that will help the algorithm a lot, which is something called"}, {"start": 172.76, "end": 177.35999999999999, "text": " an Epsilon greedy policy, which affects how you choose actions even while you're still"}, {"start": 177.35999999999999, "end": 178.35999999999999, "text": " learning."}, {"start": 178.36, "end": 194.72000000000003, "text": " Let's take a look at the next video at what that means."}]