4.7 Neural Networks Model |Inference : making predictions (forward propagation)- [ML- Andrew Ng]
 Let's take what we've learned and put it together into an algorithm to let your neural network make inferences or make predictions. This will be an algorithm called forward propagation. Let's take a look. I'm going to use as a multi-fingered example, handwritten digit recognition. And for simplicity, we're just going to distinguish between the handwritten digits 0 and 1. So it's just a binary classification problem where we're going to input an image and classify is this the digit 0 or the digit 1. And you get to play with this yourself later this week in the practice lab as well. For the example on this slide, I'm going to use an 8 by 8 image. And so this image of a 1 is this grid or matrix of 8 by 8 or 64 pixel intensity values where 255 denotes a bright white pixel and 0 would denote a black pixel and different numbers are different shades of gray in between the shades of black and white. Given these 64 input features, we're going to use a neural network with two hidden layers where the first hidden layer has 25 neurons or 25 units. Second hidden layer has 15 neurons or 15 units, and then finally the output layer outputs. What's the chance of this being 1 versus 0? So let's step through the sequence of computations that the neural network would need to make to go from the input x, this 8 by 8 or 64 numbers, to the predicted probability a3. The first computation is to go from x to a1. And that's what the first layer or the first hidden layer does. It carries out a computation of a superscript square bracket 1 equals this formula on the right. Notice that a1 has 25 numbers because this hidden layer has 25 units, which is why the parameters go from w1 through w25 as well as b1 through b25. And I've written x here, but I could also have written a0 here because by convention deactivation of layer 0, that is a0, is equal to the input feature value x. So that lets us compute a1. The next step is to compute a2. Looking at the second hidden layer, it then carries out this computation where a2 is a function of a1 and is computed as the sigmoid activation function applied to w dot product a1 plus the corresponding value of b. Notice that layer 2 has 15 neurons, so 15 units, which is why the parameters here run from w1 through w15 and b1 through b15. Now we've computed a2. The final step is then to compute a3. And we do so using a very similar computation. Only now this third layer, the output layer, has just one unit, which is why there's just one output here. So a3 is just a scalar. And finally, you can optionally take a3 subscript 1 and threshold it at 0.5 to come up with a binary classification label. Is this the digit 1? Yes or no? So the sequence of computations first takes x and then computes a1 and then computes a2 and then computes a3, which is also the output of the neural network. You can also write that as f of x. So remember when we learned about linear regression and logistic regression, we used f of x to denote the outputs of linear regression or logistic regression. So we can also use f of x to denote the function computed by the neural network as a function of x. Because this computation goes from left to right, you start from x, then compute a1, then a2, then a3, this algorithm is also called forward propagation because you're propagating the activations of the neurons. So you're making these computations in the forward direction from left to right. And this is in contrast to a different algorithm called backward propagation or back propagation, which is used for learning. And that's something you learn about next week. And by the way, this type of neural network architecture where you have more hidden units initially, and then the number of hidden units decreases as you get closer to the output layer, that's also a pretty typical choice when choosing neural network architectures. And you see more examples of this in the practice lab as well. So that's neural network inference using the forward propagation algorithm. And with this, you'd be able to download the parameters of a neural network that someone else had trained and posted on the internet. And you'd be able to carry out inference on your new data using their neural network. Now that you've seen the math and the algorithm, let's take a look at how you can actually implement this in TensorFlow. Specifically, let's take a look at this in the next video.
[{"start": 0.0, "end": 6.48, "text": " Let's take what we've learned and put it together into an algorithm to let your neural network"}, {"start": 6.48, "end": 9.48, "text": " make inferences or make predictions."}, {"start": 9.48, "end": 12.8, "text": " This will be an algorithm called forward propagation."}, {"start": 12.8, "end": 13.8, "text": " Let's take a look."}, {"start": 13.8, "end": 19.52, "text": " I'm going to use as a multi-fingered example, handwritten digit recognition."}, {"start": 19.52, "end": 26.48, "text": " And for simplicity, we're just going to distinguish between the handwritten digits 0 and 1."}, {"start": 26.48, "end": 31.04, "text": " So it's just a binary classification problem where we're going to input an image and classify"}, {"start": 31.04, "end": 34.52, "text": " is this the digit 0 or the digit 1."}, {"start": 34.52, "end": 39.16, "text": " And you get to play with this yourself later this week in the practice lab as well."}, {"start": 39.16, "end": 44.08, "text": " For the example on this slide, I'm going to use an 8 by 8 image."}, {"start": 44.08, "end": 52.56, "text": " And so this image of a 1 is this grid or matrix of 8 by 8 or 64 pixel intensity values where"}, {"start": 52.56, "end": 59.800000000000004, "text": " 255 denotes a bright white pixel and 0 would denote a black pixel and different numbers"}, {"start": 59.800000000000004, "end": 65.64, "text": " are different shades of gray in between the shades of black and white."}, {"start": 65.64, "end": 73.0, "text": " Given these 64 input features, we're going to use a neural network with two hidden layers"}, {"start": 73.0, "end": 77.68, "text": " where the first hidden layer has 25 neurons or 25 units."}, {"start": 77.68, "end": 84.24000000000001, "text": " Second hidden layer has 15 neurons or 15 units, and then finally the output layer outputs."}, {"start": 84.24000000000001, "end": 87.56, "text": " What's the chance of this being 1 versus 0?"}, {"start": 87.56, "end": 92.12, "text": " So let's step through the sequence of computations that the neural network would need to make"}, {"start": 92.12, "end": 101.22, "text": " to go from the input x, this 8 by 8 or 64 numbers, to the predicted probability a3."}, {"start": 101.22, "end": 105.72, "text": " The first computation is to go from x to a1."}, {"start": 105.72, "end": 109.24, "text": " And that's what the first layer or the first hidden layer does."}, {"start": 109.24, "end": 115.56, "text": " It carries out a computation of a superscript square bracket 1 equals this formula on the"}, {"start": 115.56, "end": 116.56, "text": " right."}, {"start": 116.56, "end": 125.0, "text": " Notice that a1 has 25 numbers because this hidden layer has 25 units, which is why the"}, {"start": 125.0, "end": 131.64, "text": " parameters go from w1 through w25 as well as b1 through b25."}, {"start": 131.64, "end": 136.92, "text": " And I've written x here, but I could also have written a0 here because by convention"}, {"start": 136.92, "end": 143.72, "text": " deactivation of layer 0, that is a0, is equal to the input feature value x."}, {"start": 143.72, "end": 146.48, "text": " So that lets us compute a1."}, {"start": 146.48, "end": 150.2, "text": " The next step is to compute a2."}, {"start": 150.2, "end": 156.48, "text": " Looking at the second hidden layer, it then carries out this computation where a2 is a"}, {"start": 156.48, "end": 165.2, "text": " function of a1 and is computed as the sigmoid activation function applied to w dot product"}, {"start": 165.2, "end": 169.12, "text": " a1 plus the corresponding value of b."}, {"start": 169.12, "end": 174.64, "text": " Notice that layer 2 has 15 neurons, so 15 units, which is why the parameters here run"}, {"start": 174.64, "end": 181.26, "text": " from w1 through w15 and b1 through b15."}, {"start": 181.26, "end": 183.57999999999998, "text": " Now we've computed a2."}, {"start": 183.58, "end": 188.04000000000002, "text": " The final step is then to compute a3."}, {"start": 188.04000000000002, "end": 192.0, "text": " And we do so using a very similar computation."}, {"start": 192.0, "end": 197.64000000000001, "text": " Only now this third layer, the output layer, has just one unit, which is why there's just"}, {"start": 197.64000000000001, "end": 199.64000000000001, "text": " one output here."}, {"start": 199.64000000000001, "end": 202.8, "text": " So a3 is just a scalar."}, {"start": 202.8, "end": 209.84, "text": " And finally, you can optionally take a3 subscript 1 and threshold it at 0.5 to come up with"}, {"start": 209.84, "end": 212.28, "text": " a binary classification label."}, {"start": 212.28, "end": 213.48000000000002, "text": " Is this the digit 1?"}, {"start": 213.48, "end": 215.11999999999998, "text": " Yes or no?"}, {"start": 215.11999999999998, "end": 220.84, "text": " So the sequence of computations first takes x and then computes a1 and then computes a2"}, {"start": 220.84, "end": 225.35999999999999, "text": " and then computes a3, which is also the output of the neural network."}, {"start": 225.35999999999999, "end": 228.76, "text": " You can also write that as f of x."}, {"start": 228.76, "end": 235.39999999999998, "text": " So remember when we learned about linear regression and logistic regression, we used f of x to"}, {"start": 235.39999999999998, "end": 239.14, "text": " denote the outputs of linear regression or logistic regression."}, {"start": 239.14, "end": 244.76, "text": " So we can also use f of x to denote the function computed by the neural network as a function"}, {"start": 244.76, "end": 247.04, "text": " of x."}, {"start": 247.04, "end": 251.23999999999998, "text": " Because this computation goes from left to right, you start from x, then compute a1,"}, {"start": 251.23999999999998, "end": 258.0, "text": " then a2, then a3, this algorithm is also called forward propagation because you're propagating"}, {"start": 258.0, "end": 260.2, "text": " the activations of the neurons."}, {"start": 260.2, "end": 266.36, "text": " So you're making these computations in the forward direction from left to right."}, {"start": 266.36, "end": 272.0, "text": " And this is in contrast to a different algorithm called backward propagation or back propagation,"}, {"start": 272.0, "end": 273.0, "text": " which is used for learning."}, {"start": 273.0, "end": 276.0, "text": " And that's something you learn about next week."}, {"start": 276.0, "end": 281.04, "text": " And by the way, this type of neural network architecture where you have more hidden units"}, {"start": 281.04, "end": 285.76, "text": " initially, and then the number of hidden units decreases as you get closer to the output"}, {"start": 285.76, "end": 290.78000000000003, "text": " layer, that's also a pretty typical choice when choosing neural network architectures."}, {"start": 290.78000000000003, "end": 294.0, "text": " And you see more examples of this in the practice lab as well."}, {"start": 294.0, "end": 299.84, "text": " So that's neural network inference using the forward propagation algorithm."}, {"start": 299.84, "end": 304.68, "text": " And with this, you'd be able to download the parameters of a neural network that someone"}, {"start": 304.68, "end": 307.76, "text": " else had trained and posted on the internet."}, {"start": 307.76, "end": 313.88, "text": " And you'd be able to carry out inference on your new data using their neural network."}, {"start": 313.88, "end": 318.44, "text": " Now that you've seen the math and the algorithm, let's take a look at how you can actually"}, {"start": 318.44, "end": 320.4, "text": " implement this in TensorFlow."}, {"start": 320.4, "end": 324.12, "text": " Specifically, let's take a look at this in the next video."}]