4.12 Neural network implementation in Python | General implementation of forward propagation-ML Ng
 In the last video, you saw how to implement forward prop in Python, but by hard coding lines of code for every single neuron. Let's now take a look at the more general implementation of forward prop in Python. Similar to the previous video, my goal in this video is to show you the code so that when you see it again in the practice lab and the optional labs, you know how to interpret it. So as we walk through this example, don't worry about taking notes on every single line of code. If you can read through the code and understand it, that's definitely enough. So what you can do is write a function to implement a dense layer that is a single layer of a neural network. So I'm going to define the dense function, which takes as input the activation from the previous layer, as well as the parameters w and b for the neurons in a given layer. Using the example from the previous video, if layer one has three neurons, and if w1 and w2 and w3 are these, then what we'll do is stack all of these weight vectors into a matrix. This is going to be a two by three matrix, where the first column is the parameter w11, the second column is the parameter w12, and the third column is the parameter w13. And then in a similar way, if you have parameters b, b11 equals negative 1, b12 equals 1, and so on, then we're going to stack these three numbers into one D array b as follows, negative 1, 1, 2. So what the dense function will do is take as input the activation from the previous layer, and a here could be a0, which is equal to x, or the activation from a later layer, as well as the w parameters stacked in columns, like shown on the right, as well as the b parameters also stacked into one D array, like shown to the left over there. And what this function will do is input a, the activation from the previous layer, and will output the activations from the columns layer. So let's step through the code for doing this. Here's the code. First, units equals w dot shape one. So w here is a two by three matrix, and so the number of columns is three. That's equal to the number of units in this layer. So here, units would be equal to three. And looking at the shape of w, it's just a way of pulling out the number of hidden units, or the number of units in this layer. Next, we set a to be an array of zeros with as many elements as there are units. So in this example, we need to output three activation values. So this just initializes a to be zero, zero, zero, an array of three zeros. Next, we go through a for loop to compute the first, second, and third elements of a. So for j and range units, so j goes from zero to units minus one, so it goes from zero, one, two, indexing from zero and Python as usual. This command, w equals capital W colon comma j, this is how you pull out the j column of a matrix in Python. So the first time through this loop, this will pull out the first column of w, and so will pull out w one one. The second time through this loop, when you're computing the activation of the second unit, the pull out the second column corresponding to w one two, and so on for the third time through this loop. And then you compute z using the usual formula as a thought product between that parameter w and the activation that you had received plus b j. And then you compute the activation a j equals g sigmoid function applied to z. So three times through this loop and you computed the values for all three values of this vector of activations a, and then finally you return a. So what the dense function does is it inputs the activations from the previous layer and given the parameters for the current layer, it returns the activations for the next layer. So given the dense function, here's how you can string together a few dense layers sequentially in order to implement forward prop in the neural network. Given the input features x, you can then compute the activations a one to be a one equals dens of x w one b one, where here w one b one are the parameters, sometimes also called the weights of the first hidden layer. Then you can compute a two as dens of now a one, which you just computed above and w two b two, which are the parameters or weights of this second hidden layer and then compute a three and a four. And if this is a neural network with four layers, then the final output f of x is just equal to a four. And so you return f of x. Notice that here I'm using a capital W because one of the notational conventions from linear algebra is to use uppercase or a capital alphabets when it's referring to a matrix and lower case to refer to vectors and scalars. So because it's a matrix, this is capital W. So that's it. You now know how to implement forward prop yourself from scratch and you get to see all this code and run it and practice it yourself in the practice lab coming after this as well. I think that even when you're using powerful libraries like TensorFlow, it's helpful to know how it works under the hood. Because in case something goes wrong, in case something runs really slowly or you have a strange result, or it looks like there's a bug, your ability to understand what's actually going on will make you much more effective when debugging your code. When I run machine learning algorithms a lot of the time, frankly, it doesn't work. So if you're not the first time. And so I find that my ability to debug my code, be it TensorFlow code or something else, is really important to being an effective machine learning engineer. So even when you're using TensorFlow or some other framework, I hope that you find this deeper understanding useful for your own applications and for debugging your own machine learning algorithms as well. So that's it. That's the last required video of this week with code in it. In the next video, I'd like to dive into what I think is a fun and fascinating topic, which is what is the relationship between neural networks and AI or AGI, artificial general intelligence. This is a controversial topic, but because it's been so widely discussed, I want to share with you some thoughts on this. So when you are asked, are neural networks at all on the path to human level intelligence, you have a framework for thinking about that question. Let's go take a look at that fun topic, I think, in the next video.
[{"start": 0.0, "end": 7.96, "text": " In the last video, you saw how to implement forward prop in Python, but by hard coding"}, {"start": 7.96, "end": 10.88, "text": " lines of code for every single neuron."}, {"start": 10.88, "end": 15.88, "text": " Let's now take a look at the more general implementation of forward prop in Python."}, {"start": 15.88, "end": 20.32, "text": " Similar to the previous video, my goal in this video is to show you the code so that"}, {"start": 20.32, "end": 25.76, "text": " when you see it again in the practice lab and the optional labs, you know how to interpret"}, {"start": 25.76, "end": 26.76, "text": " it."}, {"start": 26.76, "end": 30.92, "text": " So as we walk through this example, don't worry about taking notes on every single line"}, {"start": 30.92, "end": 32.120000000000005, "text": " of code."}, {"start": 32.120000000000005, "end": 36.58, "text": " If you can read through the code and understand it, that's definitely enough."}, {"start": 36.58, "end": 42.82, "text": " So what you can do is write a function to implement a dense layer that is a single layer"}, {"start": 42.82, "end": 45.120000000000005, "text": " of a neural network."}, {"start": 45.120000000000005, "end": 50.52, "text": " So I'm going to define the dense function, which takes as input the activation from the"}, {"start": 50.52, "end": 59.24, "text": " previous layer, as well as the parameters w and b for the neurons in a given layer."}, {"start": 59.24, "end": 70.0, "text": " Using the example from the previous video, if layer one has three neurons, and if w1"}, {"start": 70.0, "end": 78.88, "text": " and w2 and w3 are these, then what we'll do is stack all of these weight vectors into"}, {"start": 78.88, "end": 79.88, "text": " a matrix."}, {"start": 79.88, "end": 89.84, "text": " This is going to be a two by three matrix, where the first column is the parameter w11,"}, {"start": 89.84, "end": 96.91999999999999, "text": " the second column is the parameter w12, and the third column is the parameter w13."}, {"start": 96.91999999999999, "end": 105.88, "text": " And then in a similar way, if you have parameters b, b11 equals negative 1, b12 equals 1, and"}, {"start": 105.88, "end": 113.28, "text": " so on, then we're going to stack these three numbers into one D array b as follows, negative"}, {"start": 113.28, "end": 114.96, "text": " 1, 1, 2."}, {"start": 114.96, "end": 119.8, "text": " So what the dense function will do is take as input the activation from the previous"}, {"start": 119.8, "end": 127.82, "text": " layer, and a here could be a0, which is equal to x, or the activation from a later layer,"}, {"start": 127.82, "end": 135.48, "text": " as well as the w parameters stacked in columns, like shown on the right, as well as the b"}, {"start": 135.48, "end": 143.32, "text": " parameters also stacked into one D array, like shown to the left over there."}, {"start": 143.32, "end": 150.16, "text": " And what this function will do is input a, the activation from the previous layer, and"}, {"start": 150.16, "end": 154.39999999999998, "text": " will output the activations from the columns layer."}, {"start": 154.39999999999998, "end": 157.51999999999998, "text": " So let's step through the code for doing this."}, {"start": 157.51999999999998, "end": 159.32, "text": " Here's the code."}, {"start": 159.32, "end": 163.39999999999998, "text": " First, units equals w dot shape one."}, {"start": 163.4, "end": 170.72, "text": " So w here is a two by three matrix, and so the number of columns is three."}, {"start": 170.72, "end": 173.70000000000002, "text": " That's equal to the number of units in this layer."}, {"start": 173.70000000000002, "end": 177.08, "text": " So here, units would be equal to three."}, {"start": 177.08, "end": 182.92000000000002, "text": " And looking at the shape of w, it's just a way of pulling out the number of hidden units,"}, {"start": 182.92000000000002, "end": 185.64000000000001, "text": " or the number of units in this layer."}, {"start": 185.64000000000001, "end": 192.20000000000002, "text": " Next, we set a to be an array of zeros with as many elements as there are units."}, {"start": 192.2, "end": 196.6, "text": " So in this example, we need to output three activation values."}, {"start": 196.6, "end": 202.92, "text": " So this just initializes a to be zero, zero, zero, an array of three zeros."}, {"start": 202.92, "end": 209.0, "text": " Next, we go through a for loop to compute the first, second, and third elements of a."}, {"start": 209.0, "end": 214.79999999999998, "text": " So for j and range units, so j goes from zero to units minus one, so it goes from zero,"}, {"start": 214.79999999999998, "end": 218.28, "text": " one, two, indexing from zero and Python as usual."}, {"start": 218.28, "end": 227.56, "text": " This command, w equals capital W colon comma j, this is how you pull out the j column of"}, {"start": 227.56, "end": 230.2, "text": " a matrix in Python."}, {"start": 230.2, "end": 236.0, "text": " So the first time through this loop, this will pull out the first column of w, and so"}, {"start": 236.0, "end": 239.12, "text": " will pull out w one one."}, {"start": 239.12, "end": 243.56, "text": " The second time through this loop, when you're computing the activation of the second unit,"}, {"start": 243.56, "end": 249.52, "text": " the pull out the second column corresponding to w one two, and so on for the third time"}, {"start": 249.52, "end": 251.44, "text": " through this loop."}, {"start": 251.44, "end": 257.0, "text": " And then you compute z using the usual formula as a thought product between that parameter"}, {"start": 257.0, "end": 263.24, "text": " w and the activation that you had received plus b j."}, {"start": 263.24, "end": 269.32, "text": " And then you compute the activation a j equals g sigmoid function applied to z."}, {"start": 269.32, "end": 274.44, "text": " So three times through this loop and you computed the values for all three values of this vector"}, {"start": 274.44, "end": 279.44, "text": " of activations a, and then finally you return a."}, {"start": 279.44, "end": 284.76, "text": " So what the dense function does is it inputs the activations from the previous layer and"}, {"start": 284.76, "end": 290.78, "text": " given the parameters for the current layer, it returns the activations for the next layer."}, {"start": 290.78, "end": 296.84, "text": " So given the dense function, here's how you can string together a few dense layers sequentially"}, {"start": 296.84, "end": 300.96, "text": " in order to implement forward prop in the neural network."}, {"start": 300.96, "end": 310.5, "text": " Given the input features x, you can then compute the activations a one to be a one equals dens"}, {"start": 310.5, "end": 317.71999999999997, "text": " of x w one b one, where here w one b one are the parameters, sometimes also called the"}, {"start": 317.71999999999997, "end": 321.5, "text": " weights of the first hidden layer."}, {"start": 321.5, "end": 330.0, "text": " Then you can compute a two as dens of now a one, which you just computed above and w"}, {"start": 330.0, "end": 336.08, "text": " two b two, which are the parameters or weights of this second hidden layer and then compute"}, {"start": 336.08, "end": 338.76, "text": " a three and a four."}, {"start": 338.76, "end": 345.4, "text": " And if this is a neural network with four layers, then the final output f of x is just"}, {"start": 345.4, "end": 346.8, "text": " equal to a four."}, {"start": 346.8, "end": 350.44, "text": " And so you return f of x."}, {"start": 350.44, "end": 356.28, "text": " Notice that here I'm using a capital W because one of the notational conventions from linear"}, {"start": 356.28, "end": 362.56, "text": " algebra is to use uppercase or a capital alphabets when it's referring to a matrix and lower"}, {"start": 362.56, "end": 365.9, "text": " case to refer to vectors and scalars."}, {"start": 365.9, "end": 369.46, "text": " So because it's a matrix, this is capital W."}, {"start": 369.46, "end": 370.46, "text": " So that's it."}, {"start": 370.46, "end": 375.04, "text": " You now know how to implement forward prop yourself from scratch and you get to see all"}, {"start": 375.04, "end": 381.20000000000005, "text": " this code and run it and practice it yourself in the practice lab coming after this as well."}, {"start": 381.20000000000005, "end": 385.88, "text": " I think that even when you're using powerful libraries like TensorFlow, it's helpful to"}, {"start": 385.88, "end": 388.32000000000005, "text": " know how it works under the hood."}, {"start": 388.32000000000005, "end": 392.76, "text": " Because in case something goes wrong, in case something runs really slowly or you have a"}, {"start": 392.76, "end": 398.08000000000004, "text": " strange result, or it looks like there's a bug, your ability to understand what's actually"}, {"start": 398.08000000000004, "end": 402.24, "text": " going on will make you much more effective when debugging your code."}, {"start": 402.24, "end": 406.84000000000003, "text": " When I run machine learning algorithms a lot of the time, frankly, it doesn't work."}, {"start": 406.84000000000003, "end": 408.64, "text": " So if you're not the first time."}, {"start": 408.64, "end": 414.16, "text": " And so I find that my ability to debug my code, be it TensorFlow code or something else,"}, {"start": 414.16, "end": 418.36, "text": " is really important to being an effective machine learning engineer."}, {"start": 418.36, "end": 423.76, "text": " So even when you're using TensorFlow or some other framework, I hope that you find this"}, {"start": 423.76, "end": 429.92, "text": " deeper understanding useful for your own applications and for debugging your own machine learning"}, {"start": 429.92, "end": 432.6, "text": " algorithms as well."}, {"start": 432.6, "end": 434.3, "text": " So that's it."}, {"start": 434.3, "end": 438.08000000000004, "text": " That's the last required video of this week with code in it."}, {"start": 438.08000000000004, "end": 442.68, "text": " In the next video, I'd like to dive into what I think is a fun and fascinating topic, which"}, {"start": 442.68, "end": 449.0, "text": " is what is the relationship between neural networks and AI or AGI, artificial general"}, {"start": 449.0, "end": 450.64, "text": " intelligence."}, {"start": 450.64, "end": 455.16, "text": " This is a controversial topic, but because it's been so widely discussed, I want to share"}, {"start": 455.16, "end": 457.6, "text": " with you some thoughts on this."}, {"start": 457.6, "end": 464.54, "text": " So when you are asked, are neural networks at all on the path to human level intelligence,"}, {"start": 464.54, "end": 468.08000000000004, "text": " you have a framework for thinking about that question."}, {"start": 468.08, "end": 488.47999999999996, "text": " Let's go take a look at that fun topic, I think, in the next video."}]