4.14 Vectorization (optional) | How neural networks are implemented efficiently-- [ML- Andrew Ng]
 One of the reasons that deep learning researchers have been able to scale up neural networks and build really large neural networks over the last decade is because neural networks can be vectorized. They can be implemented very efficiently using matrix multiplications. And it turns out that parallel computing hardware, including GPUs, but also some CPU functions are very good at doing very large matrix multiplications. In this video, we'll take a look at how these vectorized implementations of neural networks work. Without these ideas, I don't think deep learning would be anywhere near success in scale today. Here on the left is the code that you had seen previously of how you would implement for a prop or for a propagation in a single layer. This here is the input w, the weights of the first, second, and third neurons, say, parameters b. And then this is the same code as what you saw before. And this will output three numbers, say, like that. And if you actually implement this computation, you get 101. It turns out you can develop a vectorized implementation of this function as follows. Set x to be equal to this. Notice the double square brackets. So this is now a 2D array, like in TensorFlow. W is the same as before. And b, I'm now using capital B, is also a 1 by 3 2D array. And then it turns out that all of these steps, this for loop inside, can be replaced with just a couple lines of code. Z equals np.matmul. Matmul is how NumPy carries out matrix multiplication, where now x and w are both matrices. And so you just multiply them together. And it turns out that this for loop, all of these lines of code, can be replaced with just a couple lines of code, which gives a vectorized implementation of this function. So you compute z, which is now a matrix again, as NumPy.matmul between a in and w, where here a in and w are both matrices. And matmul is how NumPy carries out a matrix multiplication. It multiplies two matrices together and then adds the matrix b to it. And then a here, or a out, is equal to the activation function g, that is a sigmoid function, multiplied element-wise to this matrix z. And then you finally return a out. So this is what the code looks like. Notice that in the vectorized implementation, all of these quantities, x, which is fed into the value of a in, as well as w, b, as well as z, and a out, all of these are now 2D arrays. All of these are matrices. And this turns out to be a very efficient implementation of one step of forward propagation through a dense layer in the neural network. So this is code for a vectorized implementation of forward prop in a neural network. But what is this code doing and how does it actually work? And what is this matmul actually doing? In the next two videos, both also optional, we'll go over matrix multiplication and how that works. If you're familiar with linear algebra, if you're familiar with vectors, matrices, transposes, and matrix matrix multiplications, you can safely just quickly skim over these two videos and jump to the last video of this week. And then in the last video of this week, also optional, we'll dive into more detail to explain how matmul gives you this vectorized implementation. And so with that, let's go on to the next video where we'll take a look at what matrix multiplication is.
[{"start": 0.0, "end": 7.16, "text": " One of the reasons that deep learning researchers have been able to scale up neural networks"}, {"start": 7.16, "end": 12.24, "text": " and build really large neural networks over the last decade is because neural networks"}, {"start": 12.24, "end": 14.16, "text": " can be vectorized."}, {"start": 14.16, "end": 19.32, "text": " They can be implemented very efficiently using matrix multiplications."}, {"start": 19.32, "end": 26.0, "text": " And it turns out that parallel computing hardware, including GPUs, but also some CPU functions"}, {"start": 26.0, "end": 30.48, "text": " are very good at doing very large matrix multiplications."}, {"start": 30.48, "end": 36.0, "text": " In this video, we'll take a look at how these vectorized implementations of neural networks"}, {"start": 36.0, "end": 37.0, "text": " work."}, {"start": 37.0, "end": 42.620000000000005, "text": " Without these ideas, I don't think deep learning would be anywhere near success in scale today."}, {"start": 42.620000000000005, "end": 49.28, "text": " Here on the left is the code that you had seen previously of how you would implement"}, {"start": 49.28, "end": 55.400000000000006, "text": " for a prop or for a propagation in a single layer."}, {"start": 55.4, "end": 64.6, "text": " This here is the input w, the weights of the first, second, and third neurons, say, parameters"}, {"start": 64.6, "end": 65.6, "text": " b."}, {"start": 65.6, "end": 68.92, "text": " And then this is the same code as what you saw before."}, {"start": 68.92, "end": 72.46, "text": " And this will output three numbers, say, like that."}, {"start": 72.46, "end": 77.72, "text": " And if you actually implement this computation, you get 101."}, {"start": 77.72, "end": 86.56, "text": " It turns out you can develop a vectorized implementation of this function as follows."}, {"start": 86.56, "end": 89.8, "text": " Set x to be equal to this."}, {"start": 89.8, "end": 91.62, "text": " Notice the double square brackets."}, {"start": 91.62, "end": 96.75999999999999, "text": " So this is now a 2D array, like in TensorFlow."}, {"start": 96.75999999999999, "end": 98.75999999999999, "text": " W is the same as before."}, {"start": 98.76, "end": 108.96000000000001, "text": " And b, I'm now using capital B, is also a 1 by 3 2D array."}, {"start": 108.96000000000001, "end": 116.12, "text": " And then it turns out that all of these steps, this for loop inside, can be replaced with"}, {"start": 116.12, "end": 118.04, "text": " just a couple lines of code."}, {"start": 118.04, "end": 120.24000000000001, "text": " Z equals np.matmul."}, {"start": 120.24, "end": 129.84, "text": " Matmul is how NumPy carries out matrix multiplication, where now x and w are both matrices."}, {"start": 129.84, "end": 133.16, "text": " And so you just multiply them together."}, {"start": 133.16, "end": 137.76, "text": " And it turns out that this for loop, all of these lines of code, can be replaced with"}, {"start": 137.76, "end": 145.24, "text": " just a couple lines of code, which gives a vectorized implementation of this function."}, {"start": 145.24, "end": 154.60000000000002, "text": " So you compute z, which is now a matrix again, as NumPy.matmul between a in and w, where"}, {"start": 154.60000000000002, "end": 157.76000000000002, "text": " here a in and w are both matrices."}, {"start": 157.76000000000002, "end": 162.8, "text": " And matmul is how NumPy carries out a matrix multiplication."}, {"start": 162.8, "end": 167.64000000000001, "text": " It multiplies two matrices together and then adds the matrix b to it."}, {"start": 167.64000000000001, "end": 175.04000000000002, "text": " And then a here, or a out, is equal to the activation function g, that is a sigmoid function,"}, {"start": 175.04, "end": 178.72, "text": " multiplied element-wise to this matrix z."}, {"start": 178.72, "end": 182.4, "text": " And then you finally return a out."}, {"start": 182.4, "end": 185.92, "text": " So this is what the code looks like."}, {"start": 185.92, "end": 191.23999999999998, "text": " Notice that in the vectorized implementation, all of these quantities, x, which is fed into"}, {"start": 191.23999999999998, "end": 198.84, "text": " the value of a in, as well as w, b, as well as z, and a out, all of these are now 2D arrays."}, {"start": 198.84, "end": 201.28, "text": " All of these are matrices."}, {"start": 201.28, "end": 208.16, "text": " And this turns out to be a very efficient implementation of one step of forward propagation"}, {"start": 208.16, "end": 210.8, "text": " through a dense layer in the neural network."}, {"start": 210.8, "end": 217.28, "text": " So this is code for a vectorized implementation of forward prop in a neural network."}, {"start": 217.28, "end": 220.88, "text": " But what is this code doing and how does it actually work?"}, {"start": 220.88, "end": 224.48, "text": " And what is this matmul actually doing?"}, {"start": 224.48, "end": 230.68, "text": " In the next two videos, both also optional, we'll go over matrix multiplication and how"}, {"start": 230.68, "end": 232.36, "text": " that works."}, {"start": 232.36, "end": 238.6, "text": " If you're familiar with linear algebra, if you're familiar with vectors, matrices, transposes,"}, {"start": 238.6, "end": 244.84, "text": " and matrix matrix multiplications, you can safely just quickly skim over these two videos"}, {"start": 244.84, "end": 247.64000000000001, "text": " and jump to the last video of this week."}, {"start": 247.64000000000001, "end": 252.84, "text": " And then in the last video of this week, also optional, we'll dive into more detail to explain"}, {"start": 252.84, "end": 257.0, "text": " how matmul gives you this vectorized implementation."}, {"start": 257.0, "end": 261.32, "text": " And so with that, let's go on to the next video where we'll take a look at what matrix"}, {"start": 261.32, "end": 288.36, "text": " multiplication is."}]