6.5 Bias and variance | Regularization and bias/variance -[Machine Learning | Andrew Ng]
 You saw in the last video how different choices of the degree of polynomial d affects the bias and variance of your learning algorithm and therefore its overall performance. In this video, let's take a look at how regularization, specifically the choice of the regularization parameter lambda, affects the bias and variance and therefore the overall performance of the algorithm. This, it turns out, will be helpful for when you want to choose a good value of lambda of the regularization parameter for your algorithm. Let's take a look. In this example, I'm going to use a fourth order polynomial, but we're going to fit this model using regularization, where here the value of lambda is the regularization parameter that controls how much you trade off keeping the parameters w small versus fitting the training data well. Let's start with the example of setting lambda to be a very large value. Say lambda is equal to 10,000. If you were to do so, you would end up fitting a model that looks roughly like this. Because if lambda were very, very large, then the algorithm is highly motivated to keep these parameters w very small, and so you end up with w1, w2, really all of these parameters would be very close to zero. The model ends up being f of x is just approximately b, a constant value, which is why you end up with a model like this. This model clearly has high bias and it underfits the training data because it doesn't even do well on the training set, and J train is large. Let's take a look at the other extreme. Let's say you set lambda to be a very small value. So with a small value of lambda, in fact, let's go to extreme of setting lambda equals zero. With that choice of lambda, there is no regularization, and so we're just fitting a four-folder polynomial with no regularization, and you end up with that curve that you saw previously that overfits the data. What we saw previously was when you have a model like this, J train is small, but JCV is much larger than J train. JCV is large, and so this indicates we have high variance and it overfits this data. It would be if you have some intermediate value of lambda, not really large, 10,000, but not so small as zero, that hopefully you get a model that looks like this, that is just right and fits the data well with small J train and small JCV. If you are trying to decide what is a good value of lambda to use for the regularization parameter, cross-validation gives you a way to do so as well. Let's take a look at how we could do so, and just as a reminder, the problem we're addressing is if you're fitting a four-folder polynomial, so that's the model, and you're using regularization, how can you choose a good value of lambda? This would be a procedure similar to what you had seen for choosing the degree of polynomial D using cross-validation. Specifically, let's say we try to fit a model using lambda equals zero, and so we would minimize the cost function using lambda equals zero and end up with some parameters w1, b1, and you can then compute the cross-validation error, JCV of w1, b1, and now let's try a different value of lambda. Let's say you try lambda equals 0.01, then again, minimizing the cost function gives you a second set of parameters, w2, b2, and you can also see how well that does on the cross-validation set and so on. Let's keep trying other values of lambda, and in this example, I'm going to try doubling it to lambda equals 0.02, and so that would give you JCV of w3, b3, and so on. Let's double the gain and double the gain. After doubling this number of times, you end up with lambda approximately equal to 10, and that would give you parameters w12, b12, and JCV w12, b12, and by trying out the large range of possible values for lambda, fitting parameters using those different regularization parameters and then evaluating the performance on the cross-validation set, you can then try to pick what is the best value for the regularization parameter. Concretely, if in this example, you find that JCV of w5, b5 has the lowest value of all of these different cross-validation errors, you might then decide to pick this value for lambda and so use w5, b5 as the chosen parameters. And finally, if you want to report out an estimate of the generalization error, you would then report out the test set error J test of w5, b5. To further hone intuition about what this algorithm is doing, let's take a look at how training error and cross-validation error vary as a function of the parameter lambda. So in this figure, I've changed the x-axis again. Notice that the x-axis here is annotated with the value of the regularization parameter lambda. And if we look at the extreme of lambda equals zero, here on the left, that corresponds to not using any regularization and so that's where we want to end up with this very wiggly curve if lambda was small or was even zero. And in that case, we have a high variance model and so J train is going to be small and JCV is going to be large because it does great on the training data but does much worse on the cross-validation data. This extreme on the right with very large values of lambda, say lambda equals 10,000, ends up with fitting a model that looks like that. So this has high bias, it underfits the data and it turns out J train will be high and JCV will be high as well. And in fact, if you were to look at how J train varies as a function of lambda, you find that J train will go up like this because in the optimization cost function, the larger lambda is, the more the algorithm is trying to keep w squared small, that is the more weight is given to this regularization term and thus the less attention is paid to actually doing well on the training set. Right, this term on the left is J train, so the more it's trying to keep the parameters small, the less good a job it does on minimizing the training error. So that's why as lambda increases, the training error J train will tend to increase like so. Now how about the cross-validation error? Turns out the cross-validation error will look like this because we've seen that if lambda is too small or too large, then it doesn't do well on the cross-validation set. It either overfits here on the left or underfits here on the right and there'll be some intermediate value of lambda that causes the algorithm to perform best. And what cross-validation is doing is trying out a lot of different values of lambda. This is what we saw on the last slide, try out lambda equals 0, lambda equals 0.01, lambda equals 0.02, try out a lot of different values of lambda and evaluate the cross-validation error at a lot of these different points and then hopefully pick a value that has low cross-validation error and this will hopefully correspond to a good model for your application. If you compare this diagram to the one that we had in the previous video where the horizontal axis was the degree of polynomial, these two diagrams look a little bit, not mathematically and not in any formal way, but they look a little bit like mirror images of each other. And that's because when you're fitting a degree of polynomial, the left part of this curve corresponded to overfitting and high bias, the right part corresponded to underfitting and high variance, whereas in this one, high variance was on the left and high bias was on the right. But that's why these two images are a little bit like mirror images of each other. But in both cases, cross-validation, evaluating different values can help you choose a good value of t or a good value of lambda. So that's how the choice of regularization parameter lambda affects the bias and variance and overall performance of your algorithm. And you've also seen how you can use cross-validation to make a good choice for the regularization parameter lambda. Now so far, we've talked about how having a high training set error, high J train is indicative of high bias and how having a high cross-validation error, JCV, specifically if it's much higher than J train, how that's indicative of a variance problem. But what do these words high or much higher actually mean? Let's take a look at that in the next video where we'll look at how you can look at the numbers J train and JCV and judge if it's high or low. And it turns out that one further refinement of these ideas, that is establishing a baseline level of performance for your learning algorithm, will make it much easier for you to look at these numbers J train, JCV and judge if they're high or low. Let's take a look at what all this means in the next video.
[{"start": 0.0, "end": 8.44, "text": " You saw in the last video how different choices of the degree of polynomial d affects the"}, {"start": 8.44, "end": 13.36, "text": " bias and variance of your learning algorithm and therefore its overall performance."}, {"start": 13.36, "end": 18.84, "text": " In this video, let's take a look at how regularization, specifically the choice of the regularization"}, {"start": 18.84, "end": 23.96, "text": " parameter lambda, affects the bias and variance and therefore the overall performance of the"}, {"start": 23.96, "end": 24.96, "text": " algorithm."}, {"start": 24.96, "end": 29.96, "text": " This, it turns out, will be helpful for when you want to choose a good value of lambda"}, {"start": 29.96, "end": 33.08, "text": " of the regularization parameter for your algorithm."}, {"start": 33.08, "end": 34.44, "text": " Let's take a look."}, {"start": 34.44, "end": 39.760000000000005, "text": " In this example, I'm going to use a fourth order polynomial, but we're going to fit this"}, {"start": 39.760000000000005, "end": 48.08, "text": " model using regularization, where here the value of lambda is the regularization parameter"}, {"start": 48.08, "end": 54.84, "text": " that controls how much you trade off keeping the parameters w small versus fitting the"}, {"start": 54.84, "end": 57.28, "text": " training data well."}, {"start": 57.28, "end": 63.440000000000005, "text": " Let's start with the example of setting lambda to be a very large value."}, {"start": 63.440000000000005, "end": 67.44, "text": " Say lambda is equal to 10,000."}, {"start": 67.44, "end": 75.84, "text": " If you were to do so, you would end up fitting a model that looks roughly like this."}, {"start": 75.84, "end": 81.4, "text": " Because if lambda were very, very large, then the algorithm is highly motivated to keep"}, {"start": 81.4, "end": 88.88000000000001, "text": " these parameters w very small, and so you end up with w1, w2, really all of these parameters"}, {"start": 88.88000000000001, "end": 92.96000000000001, "text": " would be very close to zero."}, {"start": 92.96000000000001, "end": 99.08000000000001, "text": " The model ends up being f of x is just approximately b, a constant value, which is why you end"}, {"start": 99.08000000000001, "end": 102.44, "text": " up with a model like this."}, {"start": 102.44, "end": 109.96000000000001, "text": " This model clearly has high bias and it underfits the training data because it doesn't even"}, {"start": 109.96, "end": 117.11999999999999, "text": " do well on the training set, and J train is large."}, {"start": 117.11999999999999, "end": 121.55999999999999, "text": " Let's take a look at the other extreme."}, {"start": 121.55999999999999, "end": 128.0, "text": " Let's say you set lambda to be a very small value."}, {"start": 128.0, "end": 133.56, "text": " So with a small value of lambda, in fact, let's go to extreme of setting lambda equals"}, {"start": 133.56, "end": 135.28, "text": " zero."}, {"start": 135.28, "end": 141.12, "text": " With that choice of lambda, there is no regularization, and so we're just fitting a four-folder polynomial"}, {"start": 141.12, "end": 148.64, "text": " with no regularization, and you end up with that curve that you saw previously that overfits"}, {"start": 148.64, "end": 151.12, "text": " the data."}, {"start": 151.12, "end": 157.08, "text": " What we saw previously was when you have a model like this, J train is small, but JCV"}, {"start": 157.08, "end": 159.48, "text": " is much larger than J train."}, {"start": 159.48, "end": 167.56, "text": " JCV is large, and so this indicates we have high variance and it overfits this data."}, {"start": 167.56, "end": 173.48, "text": " It would be if you have some intermediate value of lambda, not really large, 10,000,"}, {"start": 173.48, "end": 179.04, "text": " but not so small as zero, that hopefully you get a model that looks like this, that is"}, {"start": 179.04, "end": 186.51999999999998, "text": " just right and fits the data well with small J train and small JCV."}, {"start": 186.52, "end": 192.44, "text": " If you are trying to decide what is a good value of lambda to use for the regularization"}, {"start": 192.44, "end": 197.8, "text": " parameter, cross-validation gives you a way to do so as well."}, {"start": 197.8, "end": 202.68, "text": " Let's take a look at how we could do so, and just as a reminder, the problem we're addressing"}, {"start": 202.68, "end": 209.12, "text": " is if you're fitting a four-folder polynomial, so that's the model, and you're using regularization,"}, {"start": 209.12, "end": 212.36, "text": " how can you choose a good value of lambda?"}, {"start": 212.36, "end": 217.12, "text": " This would be a procedure similar to what you had seen for choosing the degree of polynomial"}, {"start": 217.12, "end": 219.4, "text": " D using cross-validation."}, {"start": 219.4, "end": 227.04000000000002, "text": " Specifically, let's say we try to fit a model using lambda equals zero, and so we would"}, {"start": 227.04000000000002, "end": 238.52, "text": " minimize the cost function using lambda equals zero and end up with some parameters w1, b1,"}, {"start": 238.52, "end": 245.8, "text": " and you can then compute the cross-validation error, JCV of w1, b1, and now let's try a"}, {"start": 245.8, "end": 247.0, "text": " different value of lambda."}, {"start": 247.0, "end": 252.74, "text": " Let's say you try lambda equals 0.01, then again, minimizing the cost function gives"}, {"start": 252.74, "end": 259.44, "text": " you a second set of parameters, w2, b2, and you can also see how well that does on the"}, {"start": 259.44, "end": 262.68, "text": " cross-validation set and so on."}, {"start": 262.68, "end": 267.66, "text": " Let's keep trying other values of lambda, and in this example, I'm going to try doubling"}, {"start": 267.66, "end": 276.16, "text": " it to lambda equals 0.02, and so that would give you JCV of w3, b3, and so on."}, {"start": 276.16, "end": 278.72, "text": " Let's double the gain and double the gain."}, {"start": 278.72, "end": 283.6, "text": " After doubling this number of times, you end up with lambda approximately equal to 10,"}, {"start": 283.6, "end": 295.04, "text": " and that would give you parameters w12, b12, and JCV w12, b12, and by trying out the large"}, {"start": 295.04, "end": 300.24, "text": " range of possible values for lambda, fitting parameters using those different regularization"}, {"start": 300.24, "end": 305.36, "text": " parameters and then evaluating the performance on the cross-validation set, you can then"}, {"start": 305.36, "end": 309.64000000000004, "text": " try to pick what is the best value for the regularization parameter."}, {"start": 309.64000000000004, "end": 319.36, "text": " Concretely, if in this example, you find that JCV of w5, b5 has the lowest value of all"}, {"start": 319.36, "end": 325.12, "text": " of these different cross-validation errors, you might then decide to pick this value for"}, {"start": 325.12, "end": 331.84000000000003, "text": " lambda and so use w5, b5 as the chosen parameters."}, {"start": 331.84000000000003, "end": 336.96000000000004, "text": " And finally, if you want to report out an estimate of the generalization error, you"}, {"start": 336.96000000000004, "end": 343.2, "text": " would then report out the test set error J test of w5, b5."}, {"start": 343.2, "end": 347.76, "text": " To further hone intuition about what this algorithm is doing, let's take a look at how"}, {"start": 347.76, "end": 354.28, "text": " training error and cross-validation error vary as a function of the parameter lambda."}, {"start": 354.28, "end": 357.76, "text": " So in this figure, I've changed the x-axis again."}, {"start": 357.76, "end": 364.0, "text": " Notice that the x-axis here is annotated with the value of the regularization parameter"}, {"start": 364.0, "end": 366.59999999999997, "text": " lambda."}, {"start": 366.59999999999997, "end": 374.44, "text": " And if we look at the extreme of lambda equals zero, here on the left, that corresponds to"}, {"start": 374.44, "end": 379.16, "text": " not using any regularization and so that's where we want to end up with this very wiggly"}, {"start": 379.16, "end": 384.12, "text": " curve if lambda was small or was even zero."}, {"start": 384.12, "end": 391.84, "text": " And in that case, we have a high variance model and so J train is going to be small"}, {"start": 391.84, "end": 397.72, "text": " and JCV is going to be large because it does great on the training data but does much worse"}, {"start": 397.72, "end": 400.56, "text": " on the cross-validation data."}, {"start": 400.56, "end": 406.28000000000003, "text": " This extreme on the right with very large values of lambda, say lambda equals 10,000,"}, {"start": 406.28000000000003, "end": 409.48, "text": " ends up with fitting a model that looks like that."}, {"start": 409.48, "end": 416.04, "text": " So this has high bias, it underfits the data and it turns out J train will be high and"}, {"start": 416.04, "end": 420.2, "text": " JCV will be high as well."}, {"start": 420.2, "end": 426.52, "text": " And in fact, if you were to look at how J train varies as a function of lambda, you"}, {"start": 426.52, "end": 433.52, "text": " find that J train will go up like this because in the optimization cost function, the larger"}, {"start": 433.52, "end": 439.08, "text": " lambda is, the more the algorithm is trying to keep w squared small, that is the more"}, {"start": 439.08, "end": 444.47999999999996, "text": " weight is given to this regularization term and thus the less attention is paid to actually"}, {"start": 444.47999999999996, "end": 446.35999999999996, "text": " doing well on the training set."}, {"start": 446.35999999999996, "end": 451.0, "text": " Right, this term on the left is J train, so the more it's trying to keep the parameters"}, {"start": 451.0, "end": 455.96, "text": " small, the less good a job it does on minimizing the training error."}, {"start": 455.96, "end": 463.12, "text": " So that's why as lambda increases, the training error J train will tend to increase like so."}, {"start": 463.12, "end": 465.71999999999997, "text": " Now how about the cross-validation error?"}, {"start": 465.71999999999997, "end": 471.71999999999997, "text": " Turns out the cross-validation error will look like this because we've seen that if"}, {"start": 471.71999999999997, "end": 477.35999999999996, "text": " lambda is too small or too large, then it doesn't do well on the cross-validation set."}, {"start": 477.35999999999996, "end": 485.15999999999997, "text": " It either overfits here on the left or underfits here on the right and there'll be some intermediate"}, {"start": 485.16, "end": 490.68, "text": " value of lambda that causes the algorithm to perform best."}, {"start": 490.68, "end": 496.8, "text": " And what cross-validation is doing is trying out a lot of different values of lambda."}, {"start": 496.8, "end": 500.8, "text": " This is what we saw on the last slide, try out lambda equals 0, lambda equals 0.01, lambda"}, {"start": 500.8, "end": 506.64000000000004, "text": " equals 0.02, try out a lot of different values of lambda and evaluate the cross-validation"}, {"start": 506.64000000000004, "end": 515.0400000000001, "text": " error at a lot of these different points and then hopefully pick a value that has low cross-validation"}, {"start": 515.04, "end": 521.52, "text": " error and this will hopefully correspond to a good model for your application."}, {"start": 521.52, "end": 527.4399999999999, "text": " If you compare this diagram to the one that we had in the previous video where the horizontal"}, {"start": 527.4399999999999, "end": 533.7199999999999, "text": " axis was the degree of polynomial, these two diagrams look a little bit, not mathematically"}, {"start": 533.7199999999999, "end": 539.76, "text": " and not in any formal way, but they look a little bit like mirror images of each other."}, {"start": 539.76, "end": 545.56, "text": " And that's because when you're fitting a degree of polynomial, the left part of this curve"}, {"start": 545.56, "end": 551.56, "text": " corresponded to overfitting and high bias, the right part corresponded to underfitting"}, {"start": 551.56, "end": 558.58, "text": " and high variance, whereas in this one, high variance was on the left and high bias was"}, {"start": 558.58, "end": 559.58, "text": " on the right."}, {"start": 559.58, "end": 563.4399999999999, "text": " But that's why these two images are a little bit like mirror images of each other."}, {"start": 563.4399999999999, "end": 569.12, "text": " But in both cases, cross-validation, evaluating different values can help you choose a good"}, {"start": 569.12, "end": 572.28, "text": " value of t or a good value of lambda."}, {"start": 572.28, "end": 577.26, "text": " So that's how the choice of regularization parameter lambda affects the bias and variance"}, {"start": 577.26, "end": 580.28, "text": " and overall performance of your algorithm."}, {"start": 580.28, "end": 585.42, "text": " And you've also seen how you can use cross-validation to make a good choice for the regularization"}, {"start": 585.42, "end": 588.0, "text": " parameter lambda."}, {"start": 588.0, "end": 594.14, "text": " Now so far, we've talked about how having a high training set error, high J train is"}, {"start": 594.14, "end": 600.3199999999999, "text": " indicative of high bias and how having a high cross-validation error, JCV, specifically"}, {"start": 600.3199999999999, "end": 606.9399999999999, "text": " if it's much higher than J train, how that's indicative of a variance problem."}, {"start": 606.9399999999999, "end": 611.04, "text": " But what do these words high or much higher actually mean?"}, {"start": 611.04, "end": 614.64, "text": " Let's take a look at that in the next video where we'll look at how you can look at the"}, {"start": 614.64, "end": 619.48, "text": " numbers J train and JCV and judge if it's high or low."}, {"start": 619.48, "end": 624.24, "text": " And it turns out that one further refinement of these ideas, that is establishing a baseline"}, {"start": 624.24, "end": 627.88, "text": " level of performance for your learning algorithm, will make it much easier for you to look at"}, {"start": 627.88, "end": 633.08, "text": " these numbers J train, JCV and judge if they're high or low."}, {"start": 633.08, "end": 650.0400000000001, "text": " Let's take a look at what all this means in the next video."}]