4.16 Vectorization (optional) | Matrix multiplication rules-- [Machine Learning - Andrew Ng]
 So, let's take a look at the general form of how you multiply two matrices together. And then in the last video after this one, we'll take this and apply it to the vectorized implementation of a neural network. Let's dive in. Here's a matrix A, which is a two by three matrix, because it has two rows and three columns. As before, I'd encourage you to think of the columns of this matrix as three vectors, vectors A1, A2, and A3. And what we're going to do is take A transpose and multiply that with a matrix W. The first, what is A transpose? Well, A transpose is obtained by taking the first column of A and laying it on the side like this, and then taking the second column of A and laying it on the side like this, and then the third column of A and laying it on the side like that. And so these rows are now A1 transpose, A2 transpose, and A3 transpose. Next, here's a matrix W. I encourage you to think of W as vectors W1, W2, W3, and W4 stacked together as so. Let's look at how you then compute A transpose times W. Now, notice that I've also used slightly different shades of orange to denote the different columns of A, where the same shade corresponds to numbers that we think of as grouped together into a vector. And that same shade is used to indicate different rows of A transpose, because the different rows of A transpose are A1 transpose, A2 transpose, and A3 transpose. And in a similar way, I've used different shades to denote the different columns of W, because the numbers of the same shade of blue are the ones that are grouped together to form the vectors W1, or W2, or W3, or W4. Now, let's look at how you can compute A transpose times W. I'm going to draw vertical bars with the different shades of blue, and horizontal bars with the different shades of orange to indicate which elements of Z, that is A transpose W, are influenced or affected by the different rows of A transpose, and which are influenced or affected by the different columns of W. So, for example, let's look at the first column of W. So, that's W1, as indicated by the lightest shade of blue here. So, W1 will influence or will correspond to this first column of Z, shown here, by this lightest shade of blue. And the values of this second column of W, that is W2, as indicated by this second lightest shade of blue, will affect the values computed in the second column of Z, and so on. For the third and fourth columns. Correspondingly, let's look at A transpose. A1 transpose is the first row of A transpose, as indicated by the lightest shade of orange, and A1 transpose will affect or influence or correspond to the values in the first row of Z. And A2 transpose will influence the second row of Z, and A3 transpose will influence or correspond to this third row of Z. So, let's figure out how to compute the matrix Z, which is going to be a 3 by 4 matrix. So, we have 12 numbers altogether. Let's start off and figure out how to compute the number in the first row in the first column of Z. So, this upper left most element here. Because this is the first row in the first column corresponding to the lightest shade of orange and the lightest shade of blue, the way you compute that is to grab the first row of A transpose and the first column of W and take their inner product or the dot product. And so, this number is going to be 1, 2 dot product with 3, 4, which is 1 times 3 plus 2 times 4, which is equal to 11. Let's look at a second example. How would you compute this number, this element of Z? So, this is in the third row, row 1, row 2, row 3. So, this is row 3 and the second column, column 1, column 2. So, to compute the number in row 3, column 2 of Z, you would now grab row 3 of A transpose and column 2 of W and dot product those together. Notice that this corresponds to the darkest shade of orange and the second lightest shade of blue. And to compute this, this is 0.1 times 5 plus 0.2 times 6, which is 0.5 plus 1.2, which is equal to 1.7. So, to compute the number in row 3, column 2 of Z, you grab the third row, row 3 of A transpose and column 2 of W. Let's look at one more example and let's see if you can figure this one out. This is row 2, column 3 of the matrix Z. Why don't you take a look and see if you can figure out which row and which column to grab to dot product together and therefore what is the number that will go in this element of this matrix. Maybe you got that you should be grabbing row 2 of A transpose and column 3 of W and when you dot product that together, you have A2 transpose W3 is negative 1 times 7 plus negative 2 times 8, which is negative 7 plus negative 16, which is equal to negative 23. And so that's how you compute this element of the matrix Z. And it turns out if you do this for every element of the matrix Z, then you can compute all of the numbers in this matrix, which turns out to look like that. Feel free to pause the video if you want and pick any element and double check that the formula we've been going through gives you the right value for Z. I just want to point out one last interesting requirement for multiplying matrices together, which is that X transpose here is a 3 by 2 matrix because it has 3 rows and 2 columns and W here is a 2 by 4 matrix because it has 2 rows and 4 columns. One requirement in order to multiply 2 matrices together is that this number must match that number and that's because you can only take dot products between vectors that are the same length. So you can take the dot product between a vector with 2 numbers and that's because you can take the inner product between a vector of length 2 only with another vector of length 2. You can't take the inner product between a vector of length 2 with a vector of length 3, for example. And that's why matrix multiplication is valid only if the number of columns of the first matrix, that is A transpose here, is equal to the number of rows of the second matrix, that is the number of rows of W here. So that when you take dot products during this process, you're taking dot products of vectors of the same size. And then the other observation is that the output Z equals A transpose W, the dimensions of Z is 3 by 4. And so the output of this multiplication will have the same number of rows as X transpose and the same number of columns as W. And so that too is another property of matrix multiplication. So that's matrix multiplication. All these videos are optional, so thank you for sticking with me through these. And if you're interested, later in this week, there are also some purely optional quizzes to let you practice some more of these calculations yourself as well. So with that, let's take what we've learned about matrix multiplication and apply it back to the vectorized implementation of a neural network. I have to say, the first time I understood the vectorized implementation, I thought it was actually really cool. I've been implementing neural networks for a while myself without the vectorized implementation. And when I finally understood the vectorized implementation and implemented it that way for the first time, it ran blazingly much faster than anything I've ever done before. And I thought, wow, I wish I had figured this out earlier. The vectorized implementation, it is a little bit complicated, but it makes neural networks run much faster. So let's take a look at that in the next video.
[{"start": 0.0, "end": 8.6, "text": " So, let's take a look at the general form of how you multiply two matrices together."}, {"start": 8.6, "end": 13.92, "text": " And then in the last video after this one, we'll take this and apply it to the vectorized"}, {"start": 13.92, "end": 16.68, "text": " implementation of a neural network."}, {"start": 16.68, "end": 17.68, "text": " Let's dive in."}, {"start": 17.68, "end": 25.48, "text": " Here's a matrix A, which is a two by three matrix, because it has two rows and three"}, {"start": 25.48, "end": 26.48, "text": " columns."}, {"start": 26.48, "end": 33.6, "text": " As before, I'd encourage you to think of the columns of this matrix as three vectors, vectors"}, {"start": 33.6, "end": 37.6, "text": " A1, A2, and A3."}, {"start": 37.6, "end": 45.04, "text": " And what we're going to do is take A transpose and multiply that with a matrix W."}, {"start": 45.04, "end": 46.480000000000004, "text": " The first, what is A transpose?"}, {"start": 46.480000000000004, "end": 52.56, "text": " Well, A transpose is obtained by taking the first column of A and laying it on the side"}, {"start": 52.56, "end": 57.2, "text": " like this, and then taking the second column of A and laying it on the side like this,"}, {"start": 57.2, "end": 60.88, "text": " and then the third column of A and laying it on the side like that."}, {"start": 60.88, "end": 68.2, "text": " And so these rows are now A1 transpose, A2 transpose, and A3 transpose."}, {"start": 68.2, "end": 80.48, "text": " Next, here's a matrix W. I encourage you to think of W as vectors W1, W2, W3, and W4 stacked"}, {"start": 80.48, "end": 82.34, "text": " together as so."}, {"start": 82.34, "end": 88.36, "text": " Let's look at how you then compute A transpose times W. Now, notice that I've also used slightly"}, {"start": 88.36, "end": 97.08, "text": " different shades of orange to denote the different columns of A, where the same shade corresponds"}, {"start": 97.08, "end": 101.68, "text": " to numbers that we think of as grouped together into a vector."}, {"start": 101.68, "end": 107.0, "text": " And that same shade is used to indicate different rows of A transpose, because the different"}, {"start": 107.0, "end": 112.28, "text": " rows of A transpose are A1 transpose, A2 transpose, and A3 transpose."}, {"start": 112.28, "end": 118.4, "text": " And in a similar way, I've used different shades to denote the different columns of W,"}, {"start": 118.4, "end": 123.32, "text": " because the numbers of the same shade of blue are the ones that are grouped together to"}, {"start": 123.32, "end": 128.08, "text": " form the vectors W1, or W2, or W3, or W4."}, {"start": 128.08, "end": 136.54, "text": " Now, let's look at how you can compute A transpose times W. I'm going to draw vertical bars"}, {"start": 136.54, "end": 142.06, "text": " with the different shades of blue, and horizontal bars with the different shades of orange to"}, {"start": 142.06, "end": 152.56, "text": " indicate which elements of Z, that is A transpose W, are influenced or affected by the different"}, {"start": 152.56, "end": 159.32, "text": " rows of A transpose, and which are influenced or affected by the different columns of W."}, {"start": 159.32, "end": 165.68, "text": " So, for example, let's look at the first column of W. So, that's W1, as indicated by the"}, {"start": 165.68, "end": 168.72, "text": " lightest shade of blue here."}, {"start": 168.72, "end": 179.1, "text": " So, W1 will influence or will correspond to this first column of Z, shown here, by this"}, {"start": 179.1, "end": 181.16, "text": " lightest shade of blue."}, {"start": 181.16, "end": 188.52, "text": " And the values of this second column of W, that is W2, as indicated by this second lightest"}, {"start": 188.52, "end": 195.48000000000002, "text": " shade of blue, will affect the values computed in the second column of Z, and so on."}, {"start": 195.48, "end": 198.44, "text": " For the third and fourth columns."}, {"start": 198.44, "end": 203.51999999999998, "text": " Correspondingly, let's look at A transpose."}, {"start": 203.51999999999998, "end": 209.88, "text": " A1 transpose is the first row of A transpose, as indicated by the lightest shade of orange,"}, {"start": 209.88, "end": 216.23999999999998, "text": " and A1 transpose will affect or influence or correspond to the values in the first row"}, {"start": 216.23999999999998, "end": 224.14, "text": " of Z. And A2 transpose will influence the second row of Z, and A3 transpose will influence"}, {"start": 224.14, "end": 226.88, "text": " or correspond to this third row of Z."}, {"start": 226.88, "end": 234.79999999999998, "text": " So, let's figure out how to compute the matrix Z, which is going to be a 3 by 4 matrix."}, {"start": 234.79999999999998, "end": 237.51999999999998, "text": " So, we have 12 numbers altogether."}, {"start": 237.51999999999998, "end": 244.07999999999998, "text": " Let's start off and figure out how to compute the number in the first row in the first column"}, {"start": 244.07999999999998, "end": 247.48, "text": " of Z. So, this upper left most element here."}, {"start": 247.48, "end": 251.51999999999998, "text": " Because this is the first row in the first column corresponding to the lightest shade"}, {"start": 251.52, "end": 256.68, "text": " of orange and the lightest shade of blue, the way you compute that is to grab the first"}, {"start": 256.68, "end": 262.96000000000004, "text": " row of A transpose and the first column of W and take their inner product or the dot"}, {"start": 262.96000000000004, "end": 263.96000000000004, "text": " product."}, {"start": 263.96000000000004, "end": 272.64, "text": " And so, this number is going to be 1, 2 dot product with 3, 4, which is 1 times 3 plus"}, {"start": 272.64, "end": 277.0, "text": " 2 times 4, which is equal to 11."}, {"start": 277.0, "end": 278.84000000000003, "text": " Let's look at a second example."}, {"start": 278.84, "end": 284.35999999999996, "text": " How would you compute this number, this element of Z?"}, {"start": 284.35999999999996, "end": 288.0, "text": " So, this is in the third row, row 1, row 2, row 3."}, {"start": 288.0, "end": 292.2, "text": " So, this is row 3 and the second column, column 1, column 2."}, {"start": 292.2, "end": 303.28, "text": " So, to compute the number in row 3, column 2 of Z, you would now grab row 3 of A transpose"}, {"start": 303.28, "end": 307.76, "text": " and column 2 of W and dot product those together."}, {"start": 307.76, "end": 312.71999999999997, "text": " Notice that this corresponds to the darkest shade of orange and the second lightest shade"}, {"start": 312.71999999999997, "end": 314.2, "text": " of blue."}, {"start": 314.2, "end": 324.84, "text": " And to compute this, this is 0.1 times 5 plus 0.2 times 6, which is 0.5 plus 1.2, which"}, {"start": 324.84, "end": 326.52, "text": " is equal to 1.7."}, {"start": 326.52, "end": 333.15999999999997, "text": " So, to compute the number in row 3, column 2 of Z, you grab the third row, row 3 of A"}, {"start": 333.15999999999997, "end": 336.92, "text": " transpose and column 2 of W."}, {"start": 336.92, "end": 342.44, "text": " Let's look at one more example and let's see if you can figure this one out."}, {"start": 342.44, "end": 348.0, "text": " This is row 2, column 3 of the matrix Z."}, {"start": 348.0, "end": 354.44, "text": " Why don't you take a look and see if you can figure out which row and which column to grab"}, {"start": 354.44, "end": 360.56, "text": " to dot product together and therefore what is the number that will go in this element"}, {"start": 360.56, "end": 362.44, "text": " of this matrix."}, {"start": 362.44, "end": 370.4, "text": " Maybe you got that you should be grabbing row 2 of A transpose and column 3 of W and"}, {"start": 370.4, "end": 377.64, "text": " when you dot product that together, you have A2 transpose W3 is negative 1 times 7 plus"}, {"start": 377.64, "end": 384.72, "text": " negative 2 times 8, which is negative 7 plus negative 16, which is equal to negative 23."}, {"start": 384.72, "end": 388.6, "text": " And so that's how you compute this element of the matrix Z."}, {"start": 388.6, "end": 393.8, "text": " And it turns out if you do this for every element of the matrix Z, then you can compute"}, {"start": 393.8, "end": 399.04, "text": " all of the numbers in this matrix, which turns out to look like that."}, {"start": 399.04, "end": 403.0, "text": " Feel free to pause the video if you want and pick any element and double check that the"}, {"start": 403.0, "end": 408.68, "text": " formula we've been going through gives you the right value for Z."}, {"start": 408.68, "end": 416.72, "text": " I just want to point out one last interesting requirement for multiplying matrices together,"}, {"start": 416.72, "end": 423.72, "text": " which is that X transpose here is a 3 by 2 matrix because it has 3 rows and 2 columns"}, {"start": 423.72, "end": 430.28000000000003, "text": " and W here is a 2 by 4 matrix because it has 2 rows and 4 columns."}, {"start": 430.28000000000003, "end": 437.44000000000005, "text": " One requirement in order to multiply 2 matrices together is that this number must match that"}, {"start": 437.44000000000005, "end": 443.92, "text": " number and that's because you can only take dot products between vectors that are the"}, {"start": 443.92, "end": 444.92, "text": " same length."}, {"start": 444.92, "end": 450.96000000000004, "text": " So you can take the dot product between a vector with 2 numbers and that's because you"}, {"start": 450.96000000000004, "end": 456.76, "text": " can take the inner product between a vector of length 2 only with another vector of length"}, {"start": 456.76, "end": 457.76, "text": " 2."}, {"start": 457.76, "end": 461.72, "text": " You can't take the inner product between a vector of length 2 with a vector of length"}, {"start": 461.72, "end": 463.72, "text": " 3, for example."}, {"start": 463.72, "end": 470.0, "text": " And that's why matrix multiplication is valid only if the number of columns of the first"}, {"start": 470.0, "end": 476.32, "text": " matrix, that is A transpose here, is equal to the number of rows of the second matrix,"}, {"start": 476.32, "end": 479.56, "text": " that is the number of rows of W here."}, {"start": 479.56, "end": 484.36, "text": " So that when you take dot products during this process, you're taking dot products of"}, {"start": 484.36, "end": 487.26, "text": " vectors of the same size."}, {"start": 487.26, "end": 495.08, "text": " And then the other observation is that the output Z equals A transpose W, the dimensions"}, {"start": 495.08, "end": 497.9, "text": " of Z is 3 by 4."}, {"start": 497.9, "end": 505.23999999999995, "text": " And so the output of this multiplication will have the same number of rows as X transpose"}, {"start": 505.23999999999995, "end": 513.52, "text": " and the same number of columns as W. And so that too is another property of matrix multiplication."}, {"start": 513.52, "end": 516.4, "text": " So that's matrix multiplication."}, {"start": 516.4, "end": 520.92, "text": " All these videos are optional, so thank you for sticking with me through these."}, {"start": 520.92, "end": 526.04, "text": " And if you're interested, later in this week, there are also some purely optional quizzes"}, {"start": 526.04, "end": 530.48, "text": " to let you practice some more of these calculations yourself as well."}, {"start": 530.48, "end": 535.0, "text": " So with that, let's take what we've learned about matrix multiplication and apply it back"}, {"start": 535.0, "end": 537.88, "text": " to the vectorized implementation of a neural network."}, {"start": 537.88, "end": 542.5999999999999, "text": " I have to say, the first time I understood the vectorized implementation, I thought it"}, {"start": 542.5999999999999, "end": 544.7199999999999, "text": " was actually really cool."}, {"start": 544.7199999999999, "end": 550.4399999999999, "text": " I've been implementing neural networks for a while myself without the vectorized implementation."}, {"start": 550.4399999999999, "end": 555.36, "text": " And when I finally understood the vectorized implementation and implemented it that way"}, {"start": 555.36, "end": 560.52, "text": " for the first time, it ran blazingly much faster than anything I've ever done before."}, {"start": 560.52, "end": 563.8000000000001, "text": " And I thought, wow, I wish I had figured this out earlier."}, {"start": 563.8000000000001, "end": 568.24, "text": " The vectorized implementation, it is a little bit complicated, but it makes neural networks"}, {"start": 568.24, "end": 570.16, "text": " run much faster."}, {"start": 570.16, "end": 586.52, "text": " So let's take a look at that in the next video."}]