4.6 Neural Networks Model | More complex neural networks --[Machine Learning | Andrew Ng]
 In the last video, you learned about the neural network layer and how that takes as input a vector of numbers and in turn outputs another vector of numbers. In this video, let's use that layer to build a more complex neural network. And through this, I hope that the notation that we're using for neural networks will become clearer and more concrete as well. Let's take a look. This is the running example that I'm going to use throughout this video as an example of a more complex neural network. This network has four layers, not counting the input layer, which is also called layer zero, where layers one, two, and three are hidden layers and layer four is the output layer and layer zero as usual is the input layer. By convention, when we say that a neural network has four layers, that includes all the hidden layers and the output layer, but we don't count the input layer. So this is a neural network with four layers in the conventional way of counting layers in the network. Let's zoom in to layer three, which is the third and final hidden layer to look at the computations of that layer. Layer three inputs a vector, a superscript square bracket two that was computed by the previous layer and it outputs a three, which is another vector. So what is the computation that layer three does in order to go from a two to a three? If it has three neurons or we call it three hidden units, then it has parameters w1, b1, w2, b2, and w3, b3 and it computes a1 equals sigmoid of w1 dot product with this input to the layer plus b1 and it computes a2 equals sigmoid of w2 dot product with again a2, the input to the layer plus b2 and so on to get a3 and then the output of this layer is a vector comprising a1, a2, and a3. And again by convention if we want to more explicitly denote that all of these are quantities associated with layer three, then we add in all of these superscript square brackets three here to denote that these parameters w and b are the parameters associated with neurons in layer three and that these activations are activations with layer three. Notice that this term here is w1 superscript square bracket three meaning the parameters associated with layer three dot product with a superscript square bracket two, which was the output of layer two, which became the input to layer three. So that's why there's a three here because there's a parameter associated with layer three dot product with and there's a two there because it's the output of layer two. Now let's just do a quick double check of our understanding of this. I'm going to hide the superscripts and subscripts associated with the second neuron and without rewinding this video, go ahead and rewind if you want but you know prefer you not, but without rewinding this video, are you able to think through what are the missing superscripts and subscripts in this equation and fill them in yourself? Why don't you take a look at the end video quiz and see if you can figure out what are the appropriate superscripts and subscripts for this equation over here. So to recap, A3 is activation associated with layer three for the second neuron, hence this is a two. There's a parameter associated with the third layer for the second neuron. This is A2, same as above, and then plus B3, two. So hopefully that makes sense. Here's the more general form of this equation for an arbitrary layer L and for an arbitrary unit J, which is that A deactivation output of layer L unit J like A32, that's going to be the sigmoid function applied to this term, which is the weight vector of layer L such as layer three for the Jth unit. So there's two again in the example above. And so that's dot product with A deactivation value of, and notice this is not L, this is L minus one, like the two above here, because you're dot producting with the output from the previous layer, and then plus B, the parameter for this layer for that unit J. And so this gives you the activation of layer L unit J, where the superscript in square brackets L denotes layer L and the subscript J denotes unit J. And when building neural networks, unit J refers to the Jth neuron. So use those terms a little bit interchangeably, where each unit is a single neuron in a layer. G here is the sigmoid function. In the context of a neural network, G has another name, which is also called the activation function because G outputs this activation value. So when I say activation function, I mean this function G here. And so far, the only activation function you've seen is the sigmoid function. But next week, we'll look at when other functions than the sigmoid function can be plugged in in place of G as well. But so the activation function is just that function that outputs these activation values. And just one last piece of notation in order to make all this notation consistent. I'm also doing to give the input vector X another name, which is a zero. So this way, the same equation also works for the first layer, where when L is equal to one deactivations of the first layer that is a one will be sigmoid times the weights dot product with a zero, which is just this input feature vector X. So with this notation, you now know how to compute the activation values of any layer in a neural network as a function of the parameters as well as the activations of the previous layer. So you now know how to compute the activations of any layer given the activations of the previous layer. Let's put this into an inference algorithm for a neural network. In other words, how to get a neural network to make predictions. Let's go see that in the next video.
[{"start": 0.0, "end": 7.2, "text": " In the last video, you learned about the neural network layer and how that takes as input"}, {"start": 7.2, "end": 12.6, "text": " a vector of numbers and in turn outputs another vector of numbers."}, {"start": 12.6, "end": 17.6, "text": " In this video, let's use that layer to build a more complex neural network."}, {"start": 17.6, "end": 22.72, "text": " And through this, I hope that the notation that we're using for neural networks will"}, {"start": 22.72, "end": 25.32, "text": " become clearer and more concrete as well."}, {"start": 25.32, "end": 26.54, "text": " Let's take a look."}, {"start": 26.54, "end": 31.2, "text": " This is the running example that I'm going to use throughout this video as an example"}, {"start": 31.2, "end": 34.0, "text": " of a more complex neural network."}, {"start": 34.0, "end": 39.1, "text": " This network has four layers, not counting the input layer, which is also called layer"}, {"start": 39.1, "end": 46.64, "text": " zero, where layers one, two, and three are hidden layers and layer four is the output"}, {"start": 46.64, "end": 50.96, "text": " layer and layer zero as usual is the input layer."}, {"start": 50.96, "end": 56.519999999999996, "text": " By convention, when we say that a neural network has four layers, that includes all the hidden"}, {"start": 56.52, "end": 60.28, "text": " layers and the output layer, but we don't count the input layer."}, {"start": 60.28, "end": 65.08, "text": " So this is a neural network with four layers in the conventional way of counting layers"}, {"start": 65.08, "end": 67.88, "text": " in the network."}, {"start": 67.88, "end": 74.04, "text": " Let's zoom in to layer three, which is the third and final hidden layer to look at the"}, {"start": 74.04, "end": 77.32000000000001, "text": " computations of that layer."}, {"start": 77.32000000000001, "end": 83.92, "text": " Layer three inputs a vector, a superscript square bracket two that was computed by the"}, {"start": 83.92, "end": 92.04, "text": " previous layer and it outputs a three, which is another vector."}, {"start": 92.04, "end": 100.44, "text": " So what is the computation that layer three does in order to go from a two to a three?"}, {"start": 100.44, "end": 108.68, "text": " If it has three neurons or we call it three hidden units, then it has parameters w1, b1,"}, {"start": 108.68, "end": 120.16000000000001, "text": " w2, b2, and w3, b3 and it computes a1 equals sigmoid of w1 dot product with this input"}, {"start": 120.16000000000001, "end": 128.96, "text": " to the layer plus b1 and it computes a2 equals sigmoid of w2 dot product with again a2, the"}, {"start": 128.96, "end": 136.52, "text": " input to the layer plus b2 and so on to get a3 and then the output of this layer is a"}, {"start": 136.52, "end": 142.8, "text": " vector comprising a1, a2, and a3."}, {"start": 142.8, "end": 149.44, "text": " And again by convention if we want to more explicitly denote that all of these are quantities"}, {"start": 149.44, "end": 155.44, "text": " associated with layer three, then we add in all of these superscript square brackets three"}, {"start": 155.44, "end": 161.20000000000002, "text": " here to denote that these parameters w and b are the parameters associated with neurons"}, {"start": 161.2, "end": 168.6, "text": " in layer three and that these activations are activations with layer three."}, {"start": 168.6, "end": 173.88, "text": " Notice that this term here is w1 superscript square bracket three meaning the parameters"}, {"start": 173.88, "end": 180.56, "text": " associated with layer three dot product with a superscript square bracket two, which was"}, {"start": 180.56, "end": 184.39999999999998, "text": " the output of layer two, which became the input to layer three."}, {"start": 184.39999999999998, "end": 188.2, "text": " So that's why there's a three here because there's a parameter associated with layer"}, {"start": 188.2, "end": 194.04, "text": " three dot product with and there's a two there because it's the output of layer two."}, {"start": 194.04, "end": 198.51999999999998, "text": " Now let's just do a quick double check of our understanding of this."}, {"start": 198.51999999999998, "end": 207.28, "text": " I'm going to hide the superscripts and subscripts associated with the second neuron and without"}, {"start": 207.28, "end": 212.51999999999998, "text": " rewinding this video, go ahead and rewind if you want but you know prefer you not, but"}, {"start": 212.52, "end": 218.4, "text": " without rewinding this video, are you able to think through what are the missing superscripts"}, {"start": 218.4, "end": 222.32000000000002, "text": " and subscripts in this equation and fill them in yourself?"}, {"start": 222.32000000000002, "end": 226.44, "text": " Why don't you take a look at the end video quiz and see if you can figure out what are"}, {"start": 226.44, "end": 248.32, "text": " the appropriate superscripts and subscripts for this equation over here."}, {"start": 248.32, "end": 256.56, "text": " So to recap, A3 is activation associated with layer three for the second neuron, hence this"}, {"start": 256.56, "end": 257.56, "text": " is a two."}, {"start": 257.56, "end": 262.24, "text": " There's a parameter associated with the third layer for the second neuron."}, {"start": 262.24, "end": 267.48, "text": " This is A2, same as above, and then plus B3, two."}, {"start": 267.48, "end": 270.36, "text": " So hopefully that makes sense."}, {"start": 270.36, "end": 275.12, "text": " Here's the more general form of this equation for an arbitrary layer L and for an arbitrary"}, {"start": 275.12, "end": 285.56, "text": " unit J, which is that A deactivation output of layer L unit J like A32, that's going to"}, {"start": 285.56, "end": 293.56, "text": " be the sigmoid function applied to this term, which is the weight vector of layer L such"}, {"start": 293.56, "end": 296.24, "text": " as layer three for the Jth unit."}, {"start": 296.24, "end": 299.08, "text": " So there's two again in the example above."}, {"start": 299.08, "end": 305.44, "text": " And so that's dot product with A deactivation value of, and notice this is not L, this is"}, {"start": 305.44, "end": 311.88, "text": " L minus one, like the two above here, because you're dot producting with the output from"}, {"start": 311.88, "end": 318.47999999999996, "text": " the previous layer, and then plus B, the parameter for this layer for that unit J."}, {"start": 318.47999999999996, "end": 325.52, "text": " And so this gives you the activation of layer L unit J, where the superscript in square"}, {"start": 325.52, "end": 330.79999999999995, "text": " brackets L denotes layer L and the subscript J denotes unit J."}, {"start": 330.79999999999995, "end": 335.59999999999997, "text": " And when building neural networks, unit J refers to the Jth neuron."}, {"start": 335.59999999999997, "end": 341.28, "text": " So use those terms a little bit interchangeably, where each unit is a single neuron in a layer."}, {"start": 341.28, "end": 343.76, "text": " G here is the sigmoid function."}, {"start": 343.76, "end": 349.15999999999997, "text": " In the context of a neural network, G has another name, which is also called the activation"}, {"start": 349.15999999999997, "end": 353.59999999999997, "text": " function because G outputs this activation value."}, {"start": 353.6, "end": 358.84000000000003, "text": " So when I say activation function, I mean this function G here."}, {"start": 358.84000000000003, "end": 363.72, "text": " And so far, the only activation function you've seen is the sigmoid function."}, {"start": 363.72, "end": 368.36, "text": " But next week, we'll look at when other functions than the sigmoid function can be plugged in"}, {"start": 368.36, "end": 369.8, "text": " in place of G as well."}, {"start": 369.8, "end": 376.94, "text": " But so the activation function is just that function that outputs these activation values."}, {"start": 376.94, "end": 384.56, "text": " And just one last piece of notation in order to make all this notation consistent."}, {"start": 384.56, "end": 390.96, "text": " I'm also doing to give the input vector X another name, which is a zero."}, {"start": 390.96, "end": 396.52, "text": " So this way, the same equation also works for the first layer, where when L is equal"}, {"start": 396.52, "end": 402.76, "text": " to one deactivations of the first layer that is a one will be sigmoid times the weights"}, {"start": 402.76, "end": 408.96, "text": " dot product with a zero, which is just this input feature vector X."}, {"start": 408.96, "end": 414.71999999999997, "text": " So with this notation, you now know how to compute the activation values of any layer"}, {"start": 414.71999999999997, "end": 420.59999999999997, "text": " in a neural network as a function of the parameters as well as the activations of the previous"}, {"start": 420.59999999999997, "end": 421.59999999999997, "text": " layer."}, {"start": 421.59999999999997, "end": 426.84, "text": " So you now know how to compute the activations of any layer given the activations of the"}, {"start": 426.84, "end": 428.4, "text": " previous layer."}, {"start": 428.4, "end": 432.44, "text": " Let's put this into an inference algorithm for a neural network."}, {"start": 432.44, "end": 436.16, "text": " In other words, how to get a neural network to make predictions."}, {"start": 436.16, "end": 463.16, "text": " Let's go see that in the next video."}]