4.17 Vectorization (optional) | Matrix multiplication code-- [Machine Learning - Andrew Ng]
 So, without further ado, let's jump into the vectorized implementation of a neural network. We'll look at the code that you have seen in an earlier video, and hopefully, MatMul, that is that matrix multiplication calculation, will make more sense. Let's jump in. So you saw previously how you can take the matrix A and compute A transpose times W, resulting in this matrix here, Z. In code, if this is the matrix A, this is a numpy array with the elements corresponding to what I wrote on top, then A transpose, which I'm going to write as A T, is going to be this matrix here, with, again, the columns of A now laid out in rows instead. And by the way, instead of setting up A T this way, another way to compute A T in numpy array would be to write A T equals A dot T. That's the transpose function that takes the columns of the matrix and lays them on the side. In code, here's how you initialize the matrix W, this is another 2D numpy array. And then to compute Z equals A transpose times W, you would write Z equals NP dot MatMul A T comma W. And that will compute this matrix Z over here, giving you this result down here. And by the way, if you read others' code, sometimes you see Z equals A T and then the at symbol W. This is an alternative way of calling the MatMul function, although I find using NP dot MatMul to be clearer. And so in the code you see in this class, we just use the MatMul function like this, rather than this at symbol. So let's look at what a vectorized implementation of what prop looks like. I'm going to set A transpose to be equal to the input feature values 217. So these are just the usual input feature values, 200 degrees, roasting coffee for 17 minutes. So this is a 1 by 2 matrix, and I'm going to take the parameters W1, W2, and W3 and stack them in columns like this to form this matrix capital W. And the values b1, b2, b3, I'm going to put into a 1 by 3 matrix that is this matrix B as follows. And it turns out that if you were to compute Z equals A transpose W plus B, that will result in these three numbers. And that's computed by taking the input feature values and multiplying that by the first column and then adding B to get 165. Taking these feature values, dot producting with the second column that is a weights W2 and adding B2 to get negative 531. And these feature values dot product with the weights W3 plus B3 to get 900. Feel free to pause the video if you wish to double check these calculations, but this gives you the values of Z11, Z12, and Z13. And then finally, if the function G applies the sigmoid function to these three numbers element-wise, that this applies the sigmoid function to 165, to negative 531, and to 900, then you end up with A equals G of this matrix Z ends up being 101. And it's 101 because sigmoid of 165 is so close to 1 that, you know, up to numerical round off is basically 1 and these are basically 0 and 1. Now let's look at how you implement this in code. A transpose is equal to this, is this 1 by 2 array of 217. The matrix W is this 2 by 3 matrix and B is this 1 by 3 matrix. And so the way you can implement for prop in the layer is dense input A transpose WB is equal to Z equals matmul A transpose times W plus B. So that just implements this line of code. And then A out, that is the output of this layer, is equal to G, the activation function applied element-wise to this matrix Z. And you return A out and that gives you this value. In case you're comparing the slide with the slides a few videos back, there was just one little difference, which was by convention, the way this is implemented in TensorFlow, rather than calling this variable Xt, we call it just A and rather than calling this variable At, we were calling it An, which is why this too is a correct implementation of the code. And there is a convention in TensorFlow that individual examples are actually laid out in rows in the matrix X rather than in the matrix X transpose, which is why the code implementation actually looks like this in TensorFlow. But this explains why with just a few lines of code, you can implement for prop in the neural network and moreover get a huge speed bonus because matmul matrix multiplication can be done very efficiently using fast hardware and get a huge bonus because modern computers are very good at implementing matrix multiplication such as matmul efficiently. That's the last video of this week. Thanks for sticking with me all the way through the end of these optional videos. For the rest of this week, I hope you also take a look at the quizzes and the practice labs and also the optional labs to exercise this material even more deeply. You now know how to do inference and for prop in a neural network, which I think is really cool. So congratulations. After you have gone through the quizzes in the labs, please also come back and in the next week, we'll look at how to actually train a neural network. So I look forward to seeing you next week.
[{"start": 0.0, "end": 7.28, "text": " So, without further ado, let's jump into the vectorized implementation of a neural network."}, {"start": 7.28, "end": 12.120000000000001, "text": " We'll look at the code that you have seen in an earlier video, and hopefully, MatMul,"}, {"start": 12.120000000000001, "end": 16.28, "text": " that is that matrix multiplication calculation, will make more sense."}, {"start": 16.28, "end": 17.28, "text": " Let's jump in."}, {"start": 17.28, "end": 23.44, "text": " So you saw previously how you can take the matrix A and compute A transpose times W,"}, {"start": 23.44, "end": 27.76, "text": " resulting in this matrix here, Z."}, {"start": 27.76, "end": 35.120000000000005, "text": " In code, if this is the matrix A, this is a numpy array with the elements corresponding"}, {"start": 35.120000000000005, "end": 40.68, "text": " to what I wrote on top, then A transpose, which I'm going to write as A T, is going"}, {"start": 40.68, "end": 49.760000000000005, "text": " to be this matrix here, with, again, the columns of A now laid out in rows instead."}, {"start": 49.760000000000005, "end": 57.120000000000005, "text": " And by the way, instead of setting up A T this way, another way to compute A T in numpy"}, {"start": 57.12, "end": 61.599999999999994, "text": " array would be to write A T equals A dot T."}, {"start": 61.599999999999994, "end": 66.44, "text": " That's the transpose function that takes the columns of the matrix and lays them on the"}, {"start": 66.44, "end": 67.44, "text": " side."}, {"start": 67.44, "end": 75.36, "text": " In code, here's how you initialize the matrix W, this is another 2D numpy array."}, {"start": 75.36, "end": 84.56, "text": " And then to compute Z equals A transpose times W, you would write Z equals NP dot MatMul"}, {"start": 84.56, "end": 95.64, "text": " A T comma W. And that will compute this matrix Z over here, giving you this result down here."}, {"start": 95.64, "end": 102.16, "text": " And by the way, if you read others' code, sometimes you see Z equals A T and then the"}, {"start": 102.16, "end": 109.72, "text": " at symbol W. This is an alternative way of calling the MatMul function, although I find"}, {"start": 109.72, "end": 112.76, "text": " using NP dot MatMul to be clearer."}, {"start": 112.76, "end": 116.60000000000001, "text": " And so in the code you see in this class, we just use the MatMul function like this,"}, {"start": 116.60000000000001, "end": 119.28, "text": " rather than this at symbol."}, {"start": 119.28, "end": 124.2, "text": " So let's look at what a vectorized implementation of what prop looks like."}, {"start": 124.2, "end": 131.68, "text": " I'm going to set A transpose to be equal to the input feature values 217."}, {"start": 131.68, "end": 137.76, "text": " So these are just the usual input feature values, 200 degrees, roasting coffee for 17"}, {"start": 137.76, "end": 138.76, "text": " minutes."}, {"start": 138.76, "end": 147.72, "text": " So this is a 1 by 2 matrix, and I'm going to take the parameters W1, W2, and W3 and"}, {"start": 147.72, "end": 157.0, "text": " stack them in columns like this to form this matrix capital W. And the values b1, b2, b3,"}, {"start": 157.0, "end": 163.68, "text": " I'm going to put into a 1 by 3 matrix that is this matrix B as follows."}, {"start": 163.68, "end": 172.04000000000002, "text": " And it turns out that if you were to compute Z equals A transpose W plus B, that will result"}, {"start": 172.04000000000002, "end": 174.84, "text": " in these three numbers."}, {"start": 174.84, "end": 182.60000000000002, "text": " And that's computed by taking the input feature values and multiplying that by the first column"}, {"start": 182.60000000000002, "end": 187.36, "text": " and then adding B to get 165."}, {"start": 187.36, "end": 192.92000000000002, "text": " Taking these feature values, dot producting with the second column that is a weights W2"}, {"start": 192.92, "end": 196.51999999999998, "text": " and adding B2 to get negative 531."}, {"start": 196.51999999999998, "end": 204.95999999999998, "text": " And these feature values dot product with the weights W3 plus B3 to get 900."}, {"start": 204.95999999999998, "end": 208.35999999999999, "text": " Feel free to pause the video if you wish to double check these calculations, but this"}, {"start": 208.35999999999999, "end": 216.95999999999998, "text": " gives you the values of Z11, Z12, and Z13."}, {"start": 216.96, "end": 223.20000000000002, "text": " And then finally, if the function G applies the sigmoid function to these three numbers"}, {"start": 223.20000000000002, "end": 229.68, "text": " element-wise, that this applies the sigmoid function to 165, to negative 531, and to 900,"}, {"start": 229.68, "end": 237.32, "text": " then you end up with A equals G of this matrix Z ends up being 101."}, {"start": 237.32, "end": 242.88, "text": " And it's 101 because sigmoid of 165 is so close to 1 that, you know, up to numerical"}, {"start": 242.88, "end": 247.48, "text": " round off is basically 1 and these are basically 0 and 1."}, {"start": 247.48, "end": 250.79999999999998, "text": " Now let's look at how you implement this in code."}, {"start": 250.79999999999998, "end": 257.32, "text": " A transpose is equal to this, is this 1 by 2 array of 217."}, {"start": 257.32, "end": 265.56, "text": " The matrix W is this 2 by 3 matrix and B is this 1 by 3 matrix."}, {"start": 265.56, "end": 272.56, "text": " And so the way you can implement for prop in the layer is dense input A transpose WB"}, {"start": 272.56, "end": 278.4, "text": " is equal to Z equals matmul A transpose times W plus B."}, {"start": 278.4, "end": 282.08, "text": " So that just implements this line of code."}, {"start": 282.08, "end": 289.84000000000003, "text": " And then A out, that is the output of this layer, is equal to G, the activation function"}, {"start": 289.84000000000003, "end": 294.56, "text": " applied element-wise to this matrix Z."}, {"start": 294.56, "end": 299.64, "text": " And you return A out and that gives you this value."}, {"start": 299.64, "end": 304.65999999999997, "text": " In case you're comparing the slide with the slides a few videos back, there was just one"}, {"start": 304.65999999999997, "end": 310.56, "text": " little difference, which was by convention, the way this is implemented in TensorFlow,"}, {"start": 310.56, "end": 317.08, "text": " rather than calling this variable Xt, we call it just A and rather than calling this variable"}, {"start": 317.08, "end": 325.44, "text": " At, we were calling it An, which is why this too is a correct implementation of the code."}, {"start": 325.44, "end": 331.64, "text": " And there is a convention in TensorFlow that individual examples are actually laid out"}, {"start": 331.64, "end": 337.84, "text": " in rows in the matrix X rather than in the matrix X transpose, which is why the code"}, {"start": 337.84, "end": 341.08, "text": " implementation actually looks like this in TensorFlow."}, {"start": 341.08, "end": 346.2, "text": " But this explains why with just a few lines of code, you can implement for prop in the"}, {"start": 346.2, "end": 353.38, "text": " neural network and moreover get a huge speed bonus because matmul matrix multiplication"}, {"start": 353.38, "end": 360.84, "text": " can be done very efficiently using fast hardware and get a huge bonus because modern computers"}, {"start": 360.84, "end": 366.71999999999997, "text": " are very good at implementing matrix multiplication such as matmul efficiently."}, {"start": 366.71999999999997, "end": 368.96, "text": " That's the last video of this week."}, {"start": 368.96, "end": 373.68, "text": " Thanks for sticking with me all the way through the end of these optional videos."}, {"start": 373.68, "end": 377.71999999999997, "text": " For the rest of this week, I hope you also take a look at the quizzes and the practice"}, {"start": 377.71999999999997, "end": 383.04, "text": " labs and also the optional labs to exercise this material even more deeply."}, {"start": 383.04, "end": 387.40000000000003, "text": " You now know how to do inference and for prop in a neural network, which I think is really"}, {"start": 387.40000000000003, "end": 388.40000000000003, "text": " cool."}, {"start": 388.40000000000003, "end": 389.40000000000003, "text": " So congratulations."}, {"start": 389.40000000000003, "end": 394.20000000000005, "text": " After you have gone through the quizzes in the labs, please also come back and in the"}, {"start": 394.20000000000005, "end": 398.08000000000004, "text": " next week, we'll look at how to actually train a neural network."}, {"start": 398.08, "end": 413.71999999999997, "text": " So I look forward to seeing you next week."}]