3.4 Cost function | Cost function for logistic regression  --[Machine Learning | Andrew Ng]
 Remember that the cost function gives you a way to measure how well a specific set of parameters fits the training data, and it thereby gives you a way to try to choose better parameters. In this video, we'll look at how the squared error cost function is not an ideal cost function for logistic regression, and we'll take a look at a different cost function that can help us choose better parameters for logistic regression. Notice what the training set for a logistic regression model might look like, where here each row might correspond to a patient that was paying a visit to the doctor and wound up with some sort of diagnosis. As before, we'll use m to denote the number of training examples. Each training example has one or more features, such as the tumor size, the patient's age, and so on, for a total of n features. And so let's call the features x1 through xn. And since this is a binary classification task, the target label y takes on only two values, either 0 or 1. And finally, the logistic regression model is defined by this equation. Okay, so the question you want to answer is, given this training set, how can you choose parameters w and b? Recall for linear regression, this is the squared error cost function. The only thing I've changed is that I put the one half inside the summation instead of outside the summation. And you might remember that in the case of linear regression, where f of x is the linear function w dot x plus b, the cost function looks like this is a convex function or a bow shape or a hammock shape. And so gradient descent will look like this, where you take one step, one step, one step, and so on to converge at the global minimum. Now, you could try to use the same cost function for logistic regression, but it turns out that if I were to write f of x equals 1 over 1 plus e to the negative w x plus b, and plot the cost function using this value of f of x, then the cost will look like this. This becomes what's called a non-convex cost function, it's not convex. And what this means is that if you were to try to use gradient descent, there are lots of local minima that you can get stuck in. So it turns out that for logistic regression, this squared error cost function is not a good choice. Instead, there will be a different cost function that can make the cost function convex again, so that gradient descent can be guaranteed to converge to the global minimum. The only thing I've changed is that I put the one half inside the summation instead of outside the summation. This will make the math you see later on the slide a little bit simpler. In order to build a new cost function, one that we'll use for logistic regression, I'm going to change a little bit the definition of the cost function j of w and b. In particular, if you look inside the summation, let's call this term inside the loss on a single training example. And I'm going to denote the loss via this capital L, and is a function of the prediction of the learning algorithm f of x, as well as of the true label y. And so the loss, given the predicted f of x and the true label y, is equal in this case to one half of the squared difference. We'll see shortly that by choosing a different form for this loss function, we'll be able to keep the overall cost function, which is one over m times the sum of these loss functions, to be a convex function. Now the loss function inputs f of x and the true label y, and tells us how well we're doing on that example. I'm going to just write down here the definition of the loss function we'll use for logistic regression. If the label y is equal to one, then the loss is negative log of f of x. And if the label y is equal to zero, then the loss is negative log of one minus f of x. Let's take a look at why this loss function hopefully makes sense. Let's first consider the case of y equals one, and plot what this function looks like to gain some intuition about what this loss function is doing. And remember, the loss function measures how well you're doing on one training example, and is by summing up the losses on all of the training examples that you then get the cost function, which measures how well you're doing on the entire training set. So if you plot log of f, it looks like this curve here, where f here is on the horizontal axis. And so a plot of negative of the log of f looks like this, where we just flip the curve along the horizontal axis. Notice that it intersects the horizontal axis at f equals one, and continues downward from there. Now f is the output of logistic regression. Thus, f is always between zero and one, because the output of logistic regression is always between zero and one. The only part of the function that's relevant is therefore this part over here, corresponding to f between zero and one. So let's zoom in and take a closer look at this part of the graph. If the algorithm predicts a probability close to one, and the true label is one, then the loss is very small, it's pretty much zero, because you're very close to the right answer. Now continuing with the example of the true label y being one, so say it really is a malignant tumor, if the algorithm predicts 0.5, then the loss is at this point here, which is a bit higher, but not that high. Whereas in contrast, if the algorithm were to have output 0.1, if it thinks that there's only a 10% chance of the tumor being malignant, but y really is one, it really is malignant, then the loss is this much higher value over here. So when y is equal to one, the loss function incentivizes, or nudges, or helps push the algorithm to make more accurate predictions, because the loss is lowest when it predicts values close to one. Now on this slide, we've been looking at what the loss is when y is equal to one. On this slide, let's look at the second part of the loss function corresponding to when y is equal to zero. In this case, the loss is negative log of one minus f of x. When this function is plotted, it actually looks like this. The range of f is limited to zero to one, because logistic regression only outputs values between zero and one, and if we zoom in, this is what it looks like. So in this plot, corresponding to y equals zero, the vertical axis shows the value of the loss for different values of f of x. So when f is zero or very close to zero, the loss is also going to be very small, which means that if the true label is zero and the model's prediction is very close to zero, well, you nearly got it right. So the loss is appropriately very close to zero. And the larger the value of f of x gets, the bigger the loss, because the prediction is further from the true label zero. And in fact, as that prediction approaches one, the loss actually approaches infinity. Going back to the tumor prediction example, this is if a model predicts that the patient's tumor is almost certain to be malignant, say 99.9% chance of malignancy, but it turns out to actually not be malignant. So y equals zero, then we penalize the model with a very high loss. So in this case of y equals zero, similar to the case of y equals one on the previous slide, the further the prediction f of x is away from the true value of y, the higher the loss. And in fact, if f of x approaches zero, the loss here actually goes really, really large and in fact approaches infinity. So when the true label is one, the algorithm is strongly incentivized not to predict something too close to zero. So in this video, you saw why the squared error cost function doesn't work well for logistic regression. We also defined the loss for a single training example and came up with a new definition for the loss function for logistic regression. It turns out that with this choice of loss function, the overall cost function will be convex and thus you can reliably use gradient descent to take you to the global minimum. Proving that this function is convex is beyond the scope of this course. You may remember that the cost function is a function of the entire training set and is therefore the average or one over m times the sum of the loss function on the individual training examples. So the cost on a certain set of parameters w and b is equal to one over m times the sum over all the training examples of the loss on the training examples. And if you can find the value of the parameters w and b that minimizes this, then you'd have a pretty good set of values for the parameters w and b for logistic regression. In the upcoming optional lab, you get to take a look at how the squared error cost function doesn't work very well for classification because you see that the surface plot results in a very wiggly cost surface with many local minima. Then you take a look at the new logistic loss function and as you can see here, this produces a nice and smooth convex surface plot that does not have all those local minima. So please take a look at the code and the plots after this video. Alright so we've seen a lot in this video. In the next video, let's go back and take the loss function for a single training example and use that to define the overall cost function for the entire training set. And we'll also figure out a simpler way to write out the cost function, which will then later allow us to run gradient descent to find good parameters for logistic regression. Let's go on to the next video.
[{"start": 0.0, "end": 6.96, "text": " Remember that the cost function gives you a way to measure how well a specific set of"}, {"start": 6.96, "end": 12.92, "text": " parameters fits the training data, and it thereby gives you a way to try to choose better"}, {"start": 12.92, "end": 13.92, "text": " parameters."}, {"start": 13.92, "end": 20.52, "text": " In this video, we'll look at how the squared error cost function is not an ideal cost function"}, {"start": 20.52, "end": 25.16, "text": " for logistic regression, and we'll take a look at a different cost function that can"}, {"start": 25.16, "end": 29.28, "text": " help us choose better parameters for logistic regression."}, {"start": 29.28, "end": 35.1, "text": " Notice what the training set for a logistic regression model might look like, where here"}, {"start": 35.1, "end": 41.44, "text": " each row might correspond to a patient that was paying a visit to the doctor and wound"}, {"start": 41.44, "end": 44.84, "text": " up with some sort of diagnosis."}, {"start": 44.84, "end": 51.32, "text": " As before, we'll use m to denote the number of training examples."}, {"start": 51.32, "end": 57.64, "text": " Each training example has one or more features, such as the tumor size, the patient's age,"}, {"start": 57.64, "end": 62.0, "text": " and so on, for a total of n features."}, {"start": 62.0, "end": 66.72, "text": " And so let's call the features x1 through xn."}, {"start": 66.72, "end": 73.44, "text": " And since this is a binary classification task, the target label y takes on only two values,"}, {"start": 73.44, "end": 75.96000000000001, "text": " either 0 or 1."}, {"start": 75.96000000000001, "end": 81.28, "text": " And finally, the logistic regression model is defined by this equation."}, {"start": 81.28, "end": 87.04, "text": " Okay, so the question you want to answer is, given this training set, how can you choose"}, {"start": 87.04, "end": 90.72000000000001, "text": " parameters w and b?"}, {"start": 90.72000000000001, "end": 95.56, "text": " Recall for linear regression, this is the squared error cost function."}, {"start": 95.56, "end": 100.52000000000001, "text": " The only thing I've changed is that I put the one half inside the summation instead"}, {"start": 100.52000000000001, "end": 102.68, "text": " of outside the summation."}, {"start": 102.68, "end": 107.4, "text": " And you might remember that in the case of linear regression, where f of x is the linear"}, {"start": 107.4, "end": 114.68, "text": " function w dot x plus b, the cost function looks like this is a convex function or a"}, {"start": 114.68, "end": 116.88000000000001, "text": " bow shape or a hammock shape."}, {"start": 116.88, "end": 122.19999999999999, "text": " And so gradient descent will look like this, where you take one step, one step, one step,"}, {"start": 122.19999999999999, "end": 125.44, "text": " and so on to converge at the global minimum."}, {"start": 125.44, "end": 131.68, "text": " Now, you could try to use the same cost function for logistic regression, but it turns out"}, {"start": 131.68, "end": 139.72, "text": " that if I were to write f of x equals 1 over 1 plus e to the negative w x plus b, and plot"}, {"start": 139.72, "end": 146.64, "text": " the cost function using this value of f of x, then the cost will look like this."}, {"start": 146.64, "end": 152.51999999999998, "text": " This becomes what's called a non-convex cost function, it's not convex."}, {"start": 152.51999999999998, "end": 158.23999999999998, "text": " And what this means is that if you were to try to use gradient descent, there are lots"}, {"start": 158.23999999999998, "end": 161.83999999999997, "text": " of local minima that you can get stuck in."}, {"start": 161.83999999999997, "end": 167.79999999999998, "text": " So it turns out that for logistic regression, this squared error cost function is not a"}, {"start": 167.79999999999998, "end": 168.79999999999998, "text": " good choice."}, {"start": 168.79999999999998, "end": 175.72, "text": " Instead, there will be a different cost function that can make the cost function convex again,"}, {"start": 175.72, "end": 180.44, "text": " so that gradient descent can be guaranteed to converge to the global minimum."}, {"start": 180.44, "end": 184.6, "text": " The only thing I've changed is that I put the one half inside the summation instead"}, {"start": 184.6, "end": 186.72, "text": " of outside the summation."}, {"start": 186.72, "end": 191.88, "text": " This will make the math you see later on the slide a little bit simpler."}, {"start": 191.88, "end": 196.72, "text": " In order to build a new cost function, one that we'll use for logistic regression, I'm"}, {"start": 196.72, "end": 203.0, "text": " going to change a little bit the definition of the cost function j of w and b."}, {"start": 203.0, "end": 209.48, "text": " In particular, if you look inside the summation, let's call this term inside the loss on a"}, {"start": 209.48, "end": 212.84, "text": " single training example."}, {"start": 212.84, "end": 220.78, "text": " And I'm going to denote the loss via this capital L, and is a function of the prediction"}, {"start": 220.78, "end": 228.04, "text": " of the learning algorithm f of x, as well as of the true label y."}, {"start": 228.04, "end": 235.44, "text": " And so the loss, given the predicted f of x and the true label y, is equal in this case"}, {"start": 235.44, "end": 239.35999999999999, "text": " to one half of the squared difference."}, {"start": 239.35999999999999, "end": 244.28, "text": " We'll see shortly that by choosing a different form for this loss function, we'll be able"}, {"start": 244.28, "end": 251.07999999999998, "text": " to keep the overall cost function, which is one over m times the sum of these loss functions,"}, {"start": 251.07999999999998, "end": 253.28, "text": " to be a convex function."}, {"start": 253.28, "end": 261.28, "text": " Now the loss function inputs f of x and the true label y, and tells us how well we're"}, {"start": 261.28, "end": 264.28, "text": " doing on that example."}, {"start": 264.28, "end": 269.52, "text": " I'm going to just write down here the definition of the loss function we'll use for logistic"}, {"start": 269.52, "end": 271.4, "text": " regression."}, {"start": 271.4, "end": 279.2, "text": " If the label y is equal to one, then the loss is negative log of f of x."}, {"start": 279.2, "end": 287.15999999999997, "text": " And if the label y is equal to zero, then the loss is negative log of one minus f of"}, {"start": 287.15999999999997, "end": 289.4, "text": " x."}, {"start": 289.4, "end": 295.96, "text": " Let's take a look at why this loss function hopefully makes sense."}, {"start": 295.96, "end": 301.96, "text": " Let's first consider the case of y equals one, and plot what this function looks like"}, {"start": 301.96, "end": 306.64, "text": " to gain some intuition about what this loss function is doing."}, {"start": 306.64, "end": 312.2, "text": " And remember, the loss function measures how well you're doing on one training example,"}, {"start": 312.2, "end": 316.68, "text": " and is by summing up the losses on all of the training examples that you then get the"}, {"start": 316.68, "end": 322.71999999999997, "text": " cost function, which measures how well you're doing on the entire training set."}, {"start": 322.71999999999997, "end": 331.44, "text": " So if you plot log of f, it looks like this curve here, where f here is on the horizontal"}, {"start": 331.44, "end": 332.76, "text": " axis."}, {"start": 332.76, "end": 339.56, "text": " And so a plot of negative of the log of f looks like this, where we just flip the curve"}, {"start": 339.56, "end": 342.36, "text": " along the horizontal axis."}, {"start": 342.36, "end": 348.71999999999997, "text": " Notice that it intersects the horizontal axis at f equals one, and continues downward from"}, {"start": 348.71999999999997, "end": 351.15999999999997, "text": " there."}, {"start": 351.15999999999997, "end": 354.2, "text": " Now f is the output of logistic regression."}, {"start": 354.2, "end": 360.46, "text": " Thus, f is always between zero and one, because the output of logistic regression is always"}, {"start": 360.46, "end": 363.26, "text": " between zero and one."}, {"start": 363.26, "end": 370.46, "text": " The only part of the function that's relevant is therefore this part over here, corresponding"}, {"start": 370.46, "end": 373.41999999999996, "text": " to f between zero and one."}, {"start": 373.41999999999996, "end": 378.71999999999997, "text": " So let's zoom in and take a closer look at this part of the graph."}, {"start": 378.71999999999997, "end": 386.35999999999996, "text": " If the algorithm predicts a probability close to one, and the true label is one, then the"}, {"start": 386.36, "end": 394.36, "text": " loss is very small, it's pretty much zero, because you're very close to the right answer."}, {"start": 394.36, "end": 400.16, "text": " Now continuing with the example of the true label y being one, so say it really is a malignant"}, {"start": 400.16, "end": 409.48, "text": " tumor, if the algorithm predicts 0.5, then the loss is at this point here, which is a"}, {"start": 409.48, "end": 412.16, "text": " bit higher, but not that high."}, {"start": 412.16, "end": 419.0, "text": " Whereas in contrast, if the algorithm were to have output 0.1, if it thinks that there's"}, {"start": 419.0, "end": 425.84000000000003, "text": " only a 10% chance of the tumor being malignant, but y really is one, it really is malignant,"}, {"start": 425.84000000000003, "end": 431.24, "text": " then the loss is this much higher value over here."}, {"start": 431.24, "end": 437.02000000000004, "text": " So when y is equal to one, the loss function incentivizes, or nudges, or helps push the"}, {"start": 437.02, "end": 442.62, "text": " algorithm to make more accurate predictions, because the loss is lowest when it predicts"}, {"start": 442.62, "end": 445.24, "text": " values close to one."}, {"start": 445.24, "end": 450.64, "text": " Now on this slide, we've been looking at what the loss is when y is equal to one."}, {"start": 450.64, "end": 455.08, "text": " On this slide, let's look at the second part of the loss function corresponding to when"}, {"start": 455.08, "end": 457.4, "text": " y is equal to zero."}, {"start": 457.4, "end": 463.56, "text": " In this case, the loss is negative log of one minus f of x."}, {"start": 463.56, "end": 468.16, "text": " When this function is plotted, it actually looks like this."}, {"start": 468.16, "end": 474.72, "text": " The range of f is limited to zero to one, because logistic regression only outputs values"}, {"start": 474.72, "end": 480.94, "text": " between zero and one, and if we zoom in, this is what it looks like."}, {"start": 480.94, "end": 488.16, "text": " So in this plot, corresponding to y equals zero, the vertical axis shows the value of"}, {"start": 488.16, "end": 493.48, "text": " the loss for different values of f of x."}, {"start": 493.48, "end": 500.44, "text": " So when f is zero or very close to zero, the loss is also going to be very small, which"}, {"start": 500.44, "end": 505.12, "text": " means that if the true label is zero and the model's prediction is very close to zero,"}, {"start": 505.12, "end": 507.56, "text": " well, you nearly got it right."}, {"start": 507.56, "end": 512.16, "text": " So the loss is appropriately very close to zero."}, {"start": 512.16, "end": 517.22, "text": " And the larger the value of f of x gets, the bigger the loss, because the prediction is"}, {"start": 517.22, "end": 520.76, "text": " further from the true label zero."}, {"start": 520.76, "end": 527.36, "text": " And in fact, as that prediction approaches one, the loss actually approaches infinity."}, {"start": 527.36, "end": 532.68, "text": " Going back to the tumor prediction example, this is if a model predicts that the patient's"}, {"start": 532.68, "end": 539.36, "text": " tumor is almost certain to be malignant, say 99.9% chance of malignancy, but it turns out"}, {"start": 539.36, "end": 541.12, "text": " to actually not be malignant."}, {"start": 541.12, "end": 547.24, "text": " So y equals zero, then we penalize the model with a very high loss."}, {"start": 547.24, "end": 552.36, "text": " So in this case of y equals zero, similar to the case of y equals one on the previous"}, {"start": 552.36, "end": 557.6800000000001, "text": " slide, the further the prediction f of x is away from the true value of y, the higher"}, {"start": 557.6800000000001, "end": 558.6800000000001, "text": " the loss."}, {"start": 558.6800000000001, "end": 566.8, "text": " And in fact, if f of x approaches zero, the loss here actually goes really, really large"}, {"start": 566.8, "end": 569.84, "text": " and in fact approaches infinity."}, {"start": 569.84, "end": 576.26, "text": " So when the true label is one, the algorithm is strongly incentivized not to predict something"}, {"start": 576.26, "end": 578.88, "text": " too close to zero."}, {"start": 578.88, "end": 582.88, "text": " So in this video, you saw why the squared error cost function doesn't work well for"}, {"start": 582.88, "end": 585.36, "text": " logistic regression."}, {"start": 585.36, "end": 593.12, "text": " We also defined the loss for a single training example and came up with a new definition"}, {"start": 593.12, "end": 597.96, "text": " for the loss function for logistic regression."}, {"start": 597.96, "end": 603.4, "text": " It turns out that with this choice of loss function, the overall cost function will be"}, {"start": 603.4, "end": 610.0799999999999, "text": " convex and thus you can reliably use gradient descent to take you to the global minimum."}, {"start": 610.0799999999999, "end": 615.3199999999999, "text": " Proving that this function is convex is beyond the scope of this course."}, {"start": 615.3199999999999, "end": 621.3199999999999, "text": " You may remember that the cost function is a function of the entire training set and"}, {"start": 621.3199999999999, "end": 628.24, "text": " is therefore the average or one over m times the sum of the loss function on the individual"}, {"start": 628.24, "end": 630.36, "text": " training examples."}, {"start": 630.36, "end": 639.2, "text": " So the cost on a certain set of parameters w and b is equal to one over m times the sum"}, {"start": 639.2, "end": 644.48, "text": " over all the training examples of the loss on the training examples."}, {"start": 644.48, "end": 651.24, "text": " And if you can find the value of the parameters w and b that minimizes this, then you'd have"}, {"start": 651.24, "end": 657.08, "text": " a pretty good set of values for the parameters w and b for logistic regression."}, {"start": 657.08, "end": 662.5, "text": " In the upcoming optional lab, you get to take a look at how the squared error cost function"}, {"start": 662.5, "end": 667.7800000000001, "text": " doesn't work very well for classification because you see that the surface plot results"}, {"start": 667.7800000000001, "end": 672.84, "text": " in a very wiggly cost surface with many local minima."}, {"start": 672.84, "end": 680.36, "text": " Then you take a look at the new logistic loss function and as you can see here, this produces"}, {"start": 680.36, "end": 687.0, "text": " a nice and smooth convex surface plot that does not have all those local minima."}, {"start": 687.0, "end": 692.2, "text": " So please take a look at the code and the plots after this video."}, {"start": 692.2, "end": 695.64, "text": " Alright so we've seen a lot in this video."}, {"start": 695.64, "end": 700.68, "text": " In the next video, let's go back and take the loss function for a single training example"}, {"start": 700.68, "end": 706.0, "text": " and use that to define the overall cost function for the entire training set."}, {"start": 706.0, "end": 710.88, "text": " And we'll also figure out a simpler way to write out the cost function, which will then"}, {"start": 710.88, "end": 716.64, "text": " later allow us to run gradient descent to find good parameters for logistic regression."}, {"start": 716.64, "end": 718.24, "text": " Let's go on to the next video."}]