5.9 Multiclass Classification | Improved implementation of softmax --[Machine Learning | Andrew Ng]
 The implementation that you saw in the last video of a neural network with a softmax layer will work okay, but there's an even better way to implement it. Let's take a look at what can go wrong with that implementation and also how to make it better. Let me show you two different ways of computing the same quantity in a computer. Option one, we can set x equals to 2 over 10,000. Option two, we can set x equals 1 plus 1 over 10,000 minus 1 minus 1 over 10,000. I wish you'd first compute this and then compute this and you take the difference. And if you simplify this expression, this turns out to be equal to 2 over 10,000. Let me illustrate this in this notebook. So first, let's set x equals 2 over 10,000 and print the result to a lot of decimal points of accuracy. Okay, that looks pretty good. Second, let me set x equals, I'm going to insist on computing 1 over 1 plus 10,000 and then subtract from that 1 minus 1 over 10,000 and let's print that out. And oh, okay, this looks a little bit off. As if there's some round off error. Because a computer has only a finite amount of memory to store each number, called a floating point number in this case, depending on how you decide to compute the value to over 10,000, the result can have more or less numerical round off error. And it turns out that while the way we have been computing the cost function for softmax is correct, there's a different way of formulating it that reduces these numerical round off errors, leading to more accurate computations within TensorFlow. Let me first explain this a little bit more detail using logistic regression. And then we will show how these ideas apply to improving our implementation of softmax. So first, let me illustrate these ideas using logistic regression. And then we'll move on to show how to improve your implementation of softmax as well. Recall that for logistic regression, if you want to compute the loss function, for a given example, you would first compute this output activation A, which is g of z, or 1 over 1 plus e is negative z. And then you compute the loss using this expression over here. And in fact, this is what the code would look like for a logistic output layer with this binary cross entropy loss. And for logistic regression, this works okay. And usually the numerical round off errors aren't that bad. But it turns out that if you allow TensorFlow to not have to compute A as an intermediate term, but instead, if you tell TensorFlow that the loss is this expression down here, and all I've done is I've taken A and expanded it into this expression down here, then TensorFlow can rearrange terms in this expression and come up with a more numerically accurate way to compute this loss function. And so whereas the original procedure was like insisting on computing as an intermediate value, 1 plus 1 over 10,000, and another intermediate value, 1 minus 1 over 10,000, and then manipulating these two to get 2 over 10,000, this original implementation was insisting on explicitly computing A as an intermediate quantity. But instead, by specifying this expression at the bottom directly as a loss function, it gives TensorFlow more flexibility in terms of how to compute this and whether or not it wants to compute A explicitly. And so the code you can use to do this is shown here. And what this does is it sets the output layer to just use a linear activation function, and it puts both the activation function, 1 over 1 plus e to the negative z, as well as this cross entropy loss into the specification of the loss function over here. And that's what this from logits equals true argument causes TensorFlow to do. And in case you're wondering what the logits are, it's basically this number z. So TensorFlow will compute z as an intermediate value, but it can rearrange terms to make this become computed more accurately. One downside of this code is it becomes a little bit less legible, but this causes TensorFlow to have a little bit less numerical round off error. Now in the case of logistic regression, either of these implementations actually works okay. But the numerical round off errors can get worse when it comes to softmax. Now let's take this idea and apply it to softmax regression. Recall what you saw in the last video was you compute the activations as follows. The activations is g of z1 through z10, where a1, for example, is e to the z1 divided by the sum of the e to the zj's. And then the loss was this, depending on what is the actual value of y is negative log of aj for one of the aj's. And so this was the code that we had to do this computation in two separate steps. But once again, if you instead specify that the loss is if y is equal to 1 is negative log of this formula, and so on, if y is equal to 10 is this formula, then this gives TensorFlow the ability to rearrange terms and compute this in a more numerically accurate way. Just to give you some intuition for why TensorFlow might want to do this, it turns out if one of the z's is really small, then e to a negative small number becomes very, very small. Or if one of the z's is a very large number, then e to the z can become a very, very large number. And by rearranging terms, TensorFlow can avoid some of these very small or very large numbers and therefore come up with a more accurate computation for the loss function. So the code for doing this is shown here. In the output layer, we're now just using a linear activation function. So the output layer just computes z1 through z10. And this whole computation of the loss is then captured in the loss function over here, where again, we have the firm log of z equals true parameter. So once again, these two pieces of code do pretty much the same thing, except that the version that is recommended is more numerically accurate, although unfortunately, it is a little bit harder to read as well. So if you're reading someone else's code and you see this and you wonder what's going on, it's actually equivalent to the original implementation and these in concept, except that is more numerically accurate. The numerical round off errors for logistic regression aren't that bad, but it is recommended that you use this implementation down at the bottom instead. And conceptually, this code does the same thing as the first version that you had previously, except that it is a little bit more numerically accurate. Although the downside is maybe just a little bit harder to interpret as well. Now, there's just one more detail, which is that we've now changed the neural network to use a linear activation function rather than a softmax activation function. And so the neural network's final layer no longer outputs these probabilities a1 through a10, it is instead outputting z1 through z10. And I didn't talk about it in the case of logistic regression, but if you were combining the output logistic function with the loss function, then for logistic regression, you also have to change the code this way to take the output value and map it through the logistic function in order to actually get the probability. So you now know how to do multiclass classification with a softmax output layer, and also how to do it in a numerically stable way. Before wrapping up multiclass classification, I want to share with you one other type of classification problem called a multi-label classification problem. Let's talk about that in the next video.
[{"start": 0.0, "end": 8.92, "text": " The implementation that you saw in the last video of a neural network with a softmax layer"}, {"start": 8.92, "end": 13.68, "text": " will work okay, but there's an even better way to implement it."}, {"start": 13.68, "end": 18.04, "text": " Let's take a look at what can go wrong with that implementation and also how to make it"}, {"start": 18.04, "end": 19.52, "text": " better."}, {"start": 19.52, "end": 25.32, "text": " Let me show you two different ways of computing the same quantity in a computer."}, {"start": 25.32, "end": 29.560000000000002, "text": " Option one, we can set x equals to 2 over 10,000."}, {"start": 29.56, "end": 38.6, "text": " Option two, we can set x equals 1 plus 1 over 10,000 minus 1 minus 1 over 10,000."}, {"start": 38.6, "end": 44.56, "text": " I wish you'd first compute this and then compute this and you take the difference."}, {"start": 44.56, "end": 51.28, "text": " And if you simplify this expression, this turns out to be equal to 2 over 10,000."}, {"start": 51.28, "end": 53.44, "text": " Let me illustrate this in this notebook."}, {"start": 53.44, "end": 59.72, "text": " So first, let's set x equals 2 over 10,000 and print the result to a lot of decimal"}, {"start": 59.72, "end": 60.72, "text": " points of accuracy."}, {"start": 60.72, "end": 62.72, "text": " Okay, that looks pretty good."}, {"start": 62.72, "end": 69.88, "text": " Second, let me set x equals, I'm going to insist on computing 1 over 1 plus 10,000 and"}, {"start": 69.88, "end": 74.8, "text": " then subtract from that 1 minus 1 over 10,000 and let's print that out."}, {"start": 74.8, "end": 77.68, "text": " And oh, okay, this looks a little bit off."}, {"start": 77.68, "end": 81.0, "text": " As if there's some round off error."}, {"start": 81.0, "end": 86.76, "text": " Because a computer has only a finite amount of memory to store each number, called a floating"}, {"start": 86.76, "end": 94.08, "text": " point number in this case, depending on how you decide to compute the value to over 10,000,"}, {"start": 94.08, "end": 99.6, "text": " the result can have more or less numerical round off error."}, {"start": 99.6, "end": 106.14, "text": " And it turns out that while the way we have been computing the cost function for softmax"}, {"start": 106.14, "end": 111.92, "text": " is correct, there's a different way of formulating it that reduces these numerical round off"}, {"start": 111.92, "end": 117.28, "text": " errors, leading to more accurate computations within TensorFlow."}, {"start": 117.28, "end": 121.5, "text": " Let me first explain this a little bit more detail using logistic regression."}, {"start": 121.5, "end": 128.12, "text": " And then we will show how these ideas apply to improving our implementation of softmax."}, {"start": 128.12, "end": 132.52, "text": " So first, let me illustrate these ideas using logistic regression."}, {"start": 132.52, "end": 138.56, "text": " And then we'll move on to show how to improve your implementation of softmax as well."}, {"start": 138.56, "end": 145.60000000000002, "text": " Recall that for logistic regression, if you want to compute the loss function, for a given"}, {"start": 145.60000000000002, "end": 150.68, "text": " example, you would first compute this output activation A, which is g of z, or 1 over 1"}, {"start": 150.68, "end": 152.84, "text": " plus e is negative z."}, {"start": 152.84, "end": 158.8, "text": " And then you compute the loss using this expression over here."}, {"start": 158.8, "end": 167.88000000000002, "text": " And in fact, this is what the code would look like for a logistic output layer with this"}, {"start": 167.88000000000002, "end": 171.04000000000002, "text": " binary cross entropy loss."}, {"start": 171.04000000000002, "end": 173.16000000000003, "text": " And for logistic regression, this works okay."}, {"start": 173.16000000000003, "end": 177.4, "text": " And usually the numerical round off errors aren't that bad."}, {"start": 177.4, "end": 184.84, "text": " But it turns out that if you allow TensorFlow to not have to compute A as an intermediate"}, {"start": 184.84, "end": 191.4, "text": " term, but instead, if you tell TensorFlow that the loss is this expression down here,"}, {"start": 191.4, "end": 199.84, "text": " and all I've done is I've taken A and expanded it into this expression down here, then TensorFlow"}, {"start": 199.84, "end": 206.24, "text": " can rearrange terms in this expression and come up with a more numerically accurate way"}, {"start": 206.24, "end": 209.12, "text": " to compute this loss function."}, {"start": 209.12, "end": 214.84, "text": " And so whereas the original procedure was like insisting on computing as an intermediate"}, {"start": 214.84, "end": 224.88, "text": " value, 1 plus 1 over 10,000, and another intermediate value, 1 minus 1 over 10,000, and then manipulating"}, {"start": 224.88, "end": 230.68, "text": " these two to get 2 over 10,000, this original implementation was insisting on explicitly"}, {"start": 230.68, "end": 235.56, "text": " computing A as an intermediate quantity."}, {"start": 235.56, "end": 240.72, "text": " But instead, by specifying this expression at the bottom directly as a loss function,"}, {"start": 240.72, "end": 246.16, "text": " it gives TensorFlow more flexibility in terms of how to compute this and whether or not"}, {"start": 246.16, "end": 248.94, "text": " it wants to compute A explicitly."}, {"start": 248.94, "end": 255.06, "text": " And so the code you can use to do this is shown here."}, {"start": 255.06, "end": 261.88, "text": " And what this does is it sets the output layer to just use a linear activation function,"}, {"start": 261.88, "end": 267.71999999999997, "text": " and it puts both the activation function, 1 over 1 plus e to the negative z, as well"}, {"start": 267.71999999999997, "end": 275.4, "text": " as this cross entropy loss into the specification of the loss function over here."}, {"start": 275.4, "end": 282.24, "text": " And that's what this from logits equals true argument causes TensorFlow to do."}, {"start": 282.24, "end": 286.6, "text": " And in case you're wondering what the logits are, it's basically this number z."}, {"start": 286.6, "end": 292.92, "text": " So TensorFlow will compute z as an intermediate value, but it can rearrange terms to make"}, {"start": 292.92, "end": 296.44, "text": " this become computed more accurately."}, {"start": 296.44, "end": 302.76000000000005, "text": " One downside of this code is it becomes a little bit less legible, but this causes TensorFlow"}, {"start": 302.76000000000005, "end": 305.56, "text": " to have a little bit less numerical round off error."}, {"start": 305.56, "end": 311.48, "text": " Now in the case of logistic regression, either of these implementations actually works okay."}, {"start": 311.48, "end": 316.8, "text": " But the numerical round off errors can get worse when it comes to softmax."}, {"start": 316.8, "end": 320.84000000000003, "text": " Now let's take this idea and apply it to softmax regression."}, {"start": 320.84000000000003, "end": 326.72, "text": " Recall what you saw in the last video was you compute the activations as follows."}, {"start": 326.72, "end": 335.24, "text": " The activations is g of z1 through z10, where a1, for example, is e to the z1 divided by"}, {"start": 335.24, "end": 338.68, "text": " the sum of the e to the zj's."}, {"start": 338.68, "end": 343.64, "text": " And then the loss was this, depending on what is the actual value of y is negative log of"}, {"start": 343.64, "end": 346.36, "text": " aj for one of the aj's."}, {"start": 346.36, "end": 353.52, "text": " And so this was the code that we had to do this computation in two separate steps."}, {"start": 353.52, "end": 362.72, "text": " But once again, if you instead specify that the loss is if y is equal to 1 is negative"}, {"start": 362.72, "end": 374.24, "text": " log of this formula, and so on, if y is equal to 10 is this formula, then this gives TensorFlow"}, {"start": 374.24, "end": 381.12, "text": " the ability to rearrange terms and compute this in a more numerically accurate way."}, {"start": 381.12, "end": 386.92, "text": " Just to give you some intuition for why TensorFlow might want to do this, it turns out if one"}, {"start": 386.92, "end": 393.28000000000003, "text": " of the z's is really small, then e to a negative small number becomes very, very small."}, {"start": 393.28000000000003, "end": 397.0, "text": " Or if one of the z's is a very large number, then e to the z can become a very, very large"}, {"start": 397.0, "end": 398.36, "text": " number."}, {"start": 398.36, "end": 403.72, "text": " And by rearranging terms, TensorFlow can avoid some of these very small or very large numbers"}, {"start": 403.72, "end": 408.90000000000003, "text": " and therefore come up with a more accurate computation for the loss function."}, {"start": 408.90000000000003, "end": 411.92, "text": " So the code for doing this is shown here."}, {"start": 411.92, "end": 416.36, "text": " In the output layer, we're now just using a linear activation function."}, {"start": 416.36, "end": 421.28000000000003, "text": " So the output layer just computes z1 through z10."}, {"start": 421.28000000000003, "end": 430.36, "text": " And this whole computation of the loss is then captured in the loss function over here,"}, {"start": 430.36, "end": 434.44, "text": " where again, we have the firm log of z equals true parameter."}, {"start": 434.44, "end": 440.08000000000004, "text": " So once again, these two pieces of code do pretty much the same thing, except that the"}, {"start": 440.08000000000004, "end": 445.64, "text": " version that is recommended is more numerically accurate, although unfortunately, it is a"}, {"start": 445.64, "end": 448.12, "text": " little bit harder to read as well."}, {"start": 448.12, "end": 452.56, "text": " So if you're reading someone else's code and you see this and you wonder what's going on,"}, {"start": 452.56, "end": 457.76, "text": " it's actually equivalent to the original implementation and these in concept, except that is more"}, {"start": 457.76, "end": 458.76, "text": " numerically accurate."}, {"start": 458.76, "end": 466.32, "text": " The numerical round off errors for logistic regression aren't that bad, but it is recommended"}, {"start": 466.32, "end": 471.5, "text": " that you use this implementation down at the bottom instead."}, {"start": 471.5, "end": 477.24, "text": " And conceptually, this code does the same thing as the first version that you had previously,"}, {"start": 477.24, "end": 482.18, "text": " except that it is a little bit more numerically accurate."}, {"start": 482.18, "end": 486.2, "text": " Although the downside is maybe just a little bit harder to interpret as well."}, {"start": 486.2, "end": 491.56, "text": " Now, there's just one more detail, which is that we've now changed the neural network"}, {"start": 491.56, "end": 497.0, "text": " to use a linear activation function rather than a softmax activation function."}, {"start": 497.0, "end": 503.36, "text": " And so the neural network's final layer no longer outputs these probabilities a1 through"}, {"start": 503.36, "end": 509.46, "text": " a10, it is instead outputting z1 through z10."}, {"start": 509.46, "end": 515.12, "text": " And I didn't talk about it in the case of logistic regression, but if you were combining"}, {"start": 515.12, "end": 520.6, "text": " the output logistic function with the loss function, then for logistic regression, you"}, {"start": 520.6, "end": 526.44, "text": " also have to change the code this way to take the output value and map it through the logistic"}, {"start": 526.44, "end": 530.08, "text": " function in order to actually get the probability."}, {"start": 530.08, "end": 536.6, "text": " So you now know how to do multiclass classification with a softmax output layer, and also how"}, {"start": 536.6, "end": 540.2, "text": " to do it in a numerically stable way."}, {"start": 540.2, "end": 544.8800000000001, "text": " Before wrapping up multiclass classification, I want to share with you one other type of"}, {"start": 544.8800000000001, "end": 549.9200000000001, "text": " classification problem called a multi-label classification problem."}, {"start": 549.92, "end": 556.92, "text": " Let's talk about that in the next video."}]