6.7 Bias and variance | Learning curves --[Machine Learning | Andrew Ng]
 Learning curves are a way to help understand how your learning algorithm is doing as a function of the amount of experience it has, whereby experience I mean, for example, the number of training examples it has. Let's take a look. Let me plot learning curves for a model that fits a second order polynomial quadratic function like so. And I'm going to plot both J C V, the cross-validation error, as well as J train, the training error. So on this figure, the horizontal axis is going to be M train. That is the training set size or the number of examples that the algorithm can learn from. And on the vertical axis, I'm going to plot the error. And by error, I mean either J C V or J train. So let's start by plotting the cross-validation error. It will look something like this. So that's what J C V of W B will look like. And it's maybe no surprise that as M train, the training set size gets bigger, then you learn a better model. And so the cross-validation error goes down. Now, let's plot J train of W B of what the training error looks like as the training set size gets bigger. It turns out that the training error will actually look like this. That as the training set size gets bigger, the training set error actually increases. Let's take a look at why this is the case. We'll start with an example of when you have just a single training example. Well, if you were to fit a quadratic model to this, you can fit, you know, easily a straight line or a curve and your training error will be zero. How about if you have two training examples like this? Well, you can again fit straight line and achieve zero training error. In fact, if you have three training examples, the quadratic function can still fit this very well and get pretty much zero training error. But now, if your training set gets a little bit bigger, say you have four training examples, then it gets a little bit harder to fit all four examples perfectly. And you may get a curve that looks like this, fits it pretty well, but you know, a little bit off in a few places here and there. And so when you have increased the training set size to four, the training error has actually gone up a little bit. How about when you have five training examples? Well, again, you can fit it pretty well, but it gets even a little bit harder to fit all of them perfectly. And when you have an even larger training set, it just gets harder and harder to fit every single one of your training examples perfectly. So to recap, when you have a very small number of training examples, like one or two or even three, it's relatively easy to get zero or very small training error. But when you have a larger training set, it's harder for quadratic function to fit all the training examples perfectly, which is why as the training set gets bigger, the training error increases because it's harder to fit all of the training examples perfectly. Notice one other thing about these curves, which is the cross validation error will be typically higher than the training error because you fit the parameters to the training set and so you expect to do at least a little bit better or when M is small, maybe even a lot better on the training set than on the cross validation set. Let's now take a look at what the learning curves will look like for an algorithm with high bias versus one with high variance. Let's start at the high bias or the under fitting case. Recall that an example of high bias would be if you're fitting a linear function to a curve that looks like this. If you were to plot the training error, then the training error will go up like so, as you'd expect. And in fact, this curve of training error may start to flatten out or we call it plateau, meaning flatten out after a while. And that's because as you get more and more training examples, when you're fitting the simple linear function, your model doesn't actually change that much more. Fitting a straight line and even as you get more and more and more examples, there's just not that much more to change, which is why the average training error flattens out after a while. And similarly, your cross validation error will come down and also flatten out after a while, which is why JCV again is higher than JTrain, but JCV will tend to look like that. And it's because beyond a certain point, even as you get more and more and more examples, not much is going to change about the straight line you're fitting. It's just too simple a model to be fitting to this much data, which is why both of these curves, JCV and JTrain, tend to flatten after a while. And if you had a measure of that baseline level of performance, such as human level performance, then it'll tend to be a value that is lower than your JTrain and your JCV. So human level performance may look like this. And there's a big gap between the baseline level performance and JTrain, which was our indicator for this algorithm having high bias. That is, one could hope to be doing much better if only we could fit a more complex function than just a straight line. Now, one interesting thing about this plot is you can ask, what do you think will happen if you could have a much bigger training set? So what would it look like if we could increase M even further than the right of this plot and go further to the right as follows? Well, you can imagine if you were to extend both of these curves to the right, they both sort of flatten out and both of them will probably just continue to be flat like that. And no matter how far you extend to the right of this plot, these two curves, they will never, you know, somehow find a way to dip down to this human level of performance or just keep on being kind of flat like this pretty much forever, no matter how large the training set gets. So that gives this conclusion, maybe a little bit surprising, that if a learning algorithm has high bias, getting more training data will not by itself help that much. And I know that we're used to thinking that having more data is good. But if your algorithm has high bias, then if the only thing you do is throw more training data at it, that by itself will not ever let you bring down the error rate that much. And it's because of this really, no matter how many more examples you add to this figure, the straight learning fitting just isn't going to get that much better. And that's why before investing a lot of effort into collecting more training data, it's worth checking if your learning algorithm has high bias, because if it does, then you probably need to do some other things other than just throw more training data at it. Let's now take a look at what the learning curve looks like for learning algorithm with high variance. You might remember that if you were to fit a four-fold polynomial with small lambda, say, or even lambda equals zero, then you get a curve that looks like this. And even though it fits the training data very well, it doesn't generalize. Let's now look at what a learning curve might look like in this high variance scenario. JTrain will be going up as the training set size increases, so you get a curve that looks like this, and JCV will be much higher. So your cross-validation error is much higher than your training error. And the fact there's a huge gap here is what can tell you that this high variance is doing much better on the training set than it's doing on your cross-validation set. If you were to plot a baseline level of performance, such as human level performance, you may find that it turns out to be here that JTrain can sometimes be even lower than the human level of performance, or maybe human level performance is a little bit lower than this. But when you're overfitting the training set, you may be able to fit the training set so well to have an unrealistically low error, such as zero error in this example over here, which is actually better than how well humans would actually be able to predict housing prices or whatever the application you're working on. But again, the signal for high variance is whether JCV is much higher than JTrain. And when you have high variance, then increasing the training set size could help a lot. And in particular, if we could extrapolate these curves to the right, increase MTrain, then the training error will continue to go up, but then the cross-validation error hopefully will come down and approach JTrain. And so in this scenario, it might be possible just by increasing the training set size to lower the cross-validation error and to get your algorithm to perform better and better. And this is unlike the high bias case where if the only thing you do is get more training data, that won't actually help your learning algorithm performance much. So to summarize, if a learning algorithm suffers from high variance, then getting more training data is indeed likely to help. Because extrapolating to the right of this curve, you see that you can expect JCV to keep on coming down. And in this example, just by getting more training data allows the algorithm to go from this relatively high cross-validation error to get much closer to human level performance. You can see that if you were to add a lot more training examples and continue to fit a four-folder polynomial, then you can just get a better four-folder polynomial fit to this data than just very weakly curve up on top. So if you're building a machine learning application, you could plot the learning curve if you want. That is, you can take different subsets of your training sets. And even if you have, say, a thousand training examples, you could train a model on just a hundred training examples and look at the training error and the cross-validation error, then train a model on 200 examples, holding out 800 examples and just not using them for now, and plot J train and JCV and so on and repeat and plot out what the learning curve looks like. And if you were to visualize it that way, then that could be another way for you to see if your learning curve looks more like a high bias or high variance one. One downside of plotting learning curves like this is something I've done, but one downside is it is computationally quite expensive to train so many different models using different size subsets of your training set. So in practice, it isn't done that often. But nonetheless, I find that having this mental visual picture in my head of what the training set looks like, sometimes that helps me to think through what I think my learning algorithm is doing and whether it has high bias or high variance. So I know we've gone through a lot about bias and variance. Let's go back to our earlier example of if you have trained a model for housing price prediction, how does bias and variance help you decide what to do next? Let's go back to that earlier example, which I hope will now make a lot more sense to you. Let's do that in the next video.
[{"start": 0.0, "end": 9.4, "text": " Learning curves are a way to help understand how your learning algorithm is doing as a function of the amount of experience it has,"}, {"start": 9.4, "end": 14.0, "text": " whereby experience I mean, for example, the number of training examples it has."}, {"start": 14.0, "end": 22.04, "text": " Let's take a look. Let me plot learning curves for a model that fits a second order polynomial quadratic function like so."}, {"start": 22.04, "end": 29.400000000000002, "text": " And I'm going to plot both J C V, the cross-validation error, as well as J train, the training error."}, {"start": 29.4, "end": 36.64, "text": " So on this figure, the horizontal axis is going to be M train."}, {"start": 36.64, "end": 41.36, "text": " That is the training set size or the number of examples that the algorithm can learn from."}, {"start": 41.36, "end": 44.4, "text": " And on the vertical axis, I'm going to plot the error."}, {"start": 44.4, "end": 47.84, "text": " And by error, I mean either J C V or J train."}, {"start": 47.84, "end": 51.04, "text": " So let's start by plotting the cross-validation error."}, {"start": 51.04, "end": 53.56, "text": " It will look something like this."}, {"start": 53.56, "end": 58.72, "text": " So that's what J C V of W B will look like."}, {"start": 58.72, "end": 63.44, "text": " And it's maybe no surprise that as M train, the training set size gets bigger,"}, {"start": 63.44, "end": 65.4, "text": " then you learn a better model."}, {"start": 65.4, "end": 68.64, "text": " And so the cross-validation error goes down."}, {"start": 68.64, "end": 77.36, "text": " Now, let's plot J train of W B of what the training error looks like as the training set size gets bigger."}, {"start": 77.36, "end": 82.2, "text": " It turns out that the training error will actually look like this."}, {"start": 82.2, "end": 88.44, "text": " That as the training set size gets bigger, the training set error actually increases."}, {"start": 88.44, "end": 90.39999999999999, "text": " Let's take a look at why this is the case."}, {"start": 90.39999999999999, "end": 95.39999999999999, "text": " We'll start with an example of when you have just a single training example."}, {"start": 95.39999999999999, "end": 99.36, "text": " Well, if you were to fit a quadratic model to this, you can fit, you know,"}, {"start": 99.36, "end": 103.75999999999999, "text": " easily a straight line or a curve and your training error will be zero."}, {"start": 103.75999999999999, "end": 106.68, "text": " How about if you have two training examples like this?"}, {"start": 106.68, "end": 111.68, "text": " Well, you can again fit straight line and achieve zero training error."}, {"start": 111.68, "end": 116.68, "text": " In fact, if you have three training examples, the quadratic function can still fit this very well"}, {"start": 116.68, "end": 119.80000000000001, "text": " and get pretty much zero training error."}, {"start": 119.80000000000001, "end": 125.04, "text": " But now, if your training set gets a little bit bigger, say you have four training examples,"}, {"start": 125.04, "end": 129.6, "text": " then it gets a little bit harder to fit all four examples perfectly."}, {"start": 129.6, "end": 132.4, "text": " And you may get a curve that looks like this, fits it pretty well,"}, {"start": 132.4, "end": 135.20000000000002, "text": " but you know, a little bit off in a few places here and there."}, {"start": 135.20000000000002, "end": 140.24, "text": " And so when you have increased the training set size to four,"}, {"start": 140.24, "end": 144.04000000000002, "text": " the training error has actually gone up a little bit."}, {"start": 144.04000000000002, "end": 146.4, "text": " How about when you have five training examples?"}, {"start": 146.4, "end": 152.20000000000002, "text": " Well, again, you can fit it pretty well, but it gets even a little bit harder to fit all of them perfectly."}, {"start": 152.20000000000002, "end": 156.28, "text": " And when you have an even larger training set, it just gets harder and harder"}, {"start": 156.28, "end": 159.92000000000002, "text": " to fit every single one of your training examples perfectly."}, {"start": 159.92000000000002, "end": 163.6, "text": " So to recap, when you have a very small number of training examples,"}, {"start": 163.6, "end": 171.48000000000002, "text": " like one or two or even three, it's relatively easy to get zero or very small training error."}, {"start": 171.48, "end": 176.67999999999998, "text": " But when you have a larger training set, it's harder for quadratic function"}, {"start": 176.67999999999998, "end": 182.83999999999997, "text": " to fit all the training examples perfectly, which is why as the training set gets bigger,"}, {"start": 182.83999999999997, "end": 189.76, "text": " the training error increases because it's harder to fit all of the training examples perfectly."}, {"start": 189.76, "end": 194.32, "text": " Notice one other thing about these curves, which is the cross validation error"}, {"start": 194.32, "end": 201.28, "text": " will be typically higher than the training error because you fit the parameters to the training set"}, {"start": 201.28, "end": 205.6, "text": " and so you expect to do at least a little bit better or when M is small,"}, {"start": 205.6, "end": 211.52, "text": " maybe even a lot better on the training set than on the cross validation set."}, {"start": 211.52, "end": 219.72, "text": " Let's now take a look at what the learning curves will look like for an algorithm with high bias versus one with high variance."}, {"start": 219.72, "end": 223.92000000000002, "text": " Let's start at the high bias or the under fitting case."}, {"start": 223.92000000000002, "end": 230.52, "text": " Recall that an example of high bias would be if you're fitting a linear function to a curve that looks like this."}, {"start": 230.52, "end": 238.96, "text": " If you were to plot the training error, then the training error will go up like so, as you'd expect."}, {"start": 238.96, "end": 248.92000000000002, "text": " And in fact, this curve of training error may start to flatten out or we call it plateau, meaning flatten out after a while."}, {"start": 248.92000000000002, "end": 255.56, "text": " And that's because as you get more and more training examples, when you're fitting the simple linear function,"}, {"start": 255.56, "end": 258.44, "text": " your model doesn't actually change that much more."}, {"start": 258.44, "end": 264.2, "text": " Fitting a straight line and even as you get more and more and more examples, there's just not that much more to change,"}, {"start": 264.2, "end": 270.12, "text": " which is why the average training error flattens out after a while."}, {"start": 270.12, "end": 276.88, "text": " And similarly, your cross validation error will come down and also flatten out after a while,"}, {"start": 276.88, "end": 284.0, "text": " which is why JCV again is higher than JTrain, but JCV will tend to look like that."}, {"start": 284.0, "end": 289.2, "text": " And it's because beyond a certain point, even as you get more and more and more examples,"}, {"start": 289.2, "end": 291.8, "text": " not much is going to change about the straight line you're fitting."}, {"start": 291.8, "end": 302.08, "text": " It's just too simple a model to be fitting to this much data, which is why both of these curves, JCV and JTrain, tend to flatten after a while."}, {"start": 302.08, "end": 309.84, "text": " And if you had a measure of that baseline level of performance, such as human level performance,"}, {"start": 309.84, "end": 315.47999999999996, "text": " then it'll tend to be a value that is lower than your JTrain and your JCV."}, {"start": 315.47999999999996, "end": 318.59999999999997, "text": " So human level performance may look like this."}, {"start": 318.59999999999997, "end": 323.32, "text": " And there's a big gap between the baseline level performance and JTrain,"}, {"start": 323.32, "end": 328.91999999999996, "text": " which was our indicator for this algorithm having high bias."}, {"start": 328.91999999999996, "end": 337.91999999999996, "text": " That is, one could hope to be doing much better if only we could fit a more complex function than just a straight line."}, {"start": 337.92, "end": 343.76, "text": " Now, one interesting thing about this plot is you can ask,"}, {"start": 343.76, "end": 349.32, "text": " what do you think will happen if you could have a much bigger training set?"}, {"start": 349.32, "end": 358.48, "text": " So what would it look like if we could increase M even further than the right of this plot and go further to the right as follows?"}, {"start": 358.48, "end": 362.36, "text": " Well, you can imagine if you were to extend both of these curves to the right,"}, {"start": 362.36, "end": 368.6, "text": " they both sort of flatten out and both of them will probably just continue to be flat like that."}, {"start": 368.6, "end": 372.56, "text": " And no matter how far you extend to the right of this plot, these two curves,"}, {"start": 372.56, "end": 382.2, "text": " they will never, you know, somehow find a way to dip down to this human level of performance or just keep on being kind of flat like this pretty much forever,"}, {"start": 382.2, "end": 385.52000000000004, "text": " no matter how large the training set gets."}, {"start": 385.52, "end": 392.71999999999997, "text": " So that gives this conclusion, maybe a little bit surprising, that if a learning algorithm has high bias,"}, {"start": 392.71999999999997, "end": 397.24, "text": " getting more training data will not by itself help that much."}, {"start": 397.24, "end": 402.24, "text": " And I know that we're used to thinking that having more data is good."}, {"start": 402.24, "end": 409.35999999999996, "text": " But if your algorithm has high bias, then if the only thing you do is throw more training data at it,"}, {"start": 409.35999999999996, "end": 412.91999999999996, "text": " that by itself will not ever let you bring down the error rate that much."}, {"start": 412.92, "end": 417.24, "text": " And it's because of this really, no matter how many more examples you add to this figure,"}, {"start": 417.24, "end": 421.76, "text": " the straight learning fitting just isn't going to get that much better."}, {"start": 421.76, "end": 426.92, "text": " And that's why before investing a lot of effort into collecting more training data,"}, {"start": 426.92, "end": 431.40000000000003, "text": " it's worth checking if your learning algorithm has high bias, because if it does,"}, {"start": 431.40000000000003, "end": 437.52000000000004, "text": " then you probably need to do some other things other than just throw more training data at it."}, {"start": 437.52, "end": 443.0, "text": " Let's now take a look at what the learning curve looks like for learning algorithm with high variance."}, {"start": 443.0, "end": 448.91999999999996, "text": " You might remember that if you were to fit a four-fold polynomial with small lambda,"}, {"start": 448.91999999999996, "end": 453.4, "text": " say, or even lambda equals zero, then you get a curve that looks like this."}, {"start": 453.4, "end": 458.88, "text": " And even though it fits the training data very well, it doesn't generalize."}, {"start": 458.88, "end": 464.96, "text": " Let's now look at what a learning curve might look like in this high variance scenario."}, {"start": 464.96, "end": 470.2, "text": " JTrain will be going up as the training set size increases,"}, {"start": 470.2, "end": 476.15999999999997, "text": " so you get a curve that looks like this, and JCV will be much higher."}, {"start": 476.15999999999997, "end": 480.32, "text": " So your cross-validation error is much higher than your training error."}, {"start": 480.32, "end": 487.03999999999996, "text": " And the fact there's a huge gap here is what can tell you that this high variance is doing much better"}, {"start": 487.03999999999996, "end": 491.24, "text": " on the training set than it's doing on your cross-validation set."}, {"start": 491.24, "end": 496.08, "text": " If you were to plot a baseline level of performance, such as human level performance,"}, {"start": 496.08, "end": 503.92, "text": " you may find that it turns out to be here that JTrain can sometimes be even lower than the human level of performance,"}, {"start": 503.92, "end": 507.56, "text": " or maybe human level performance is a little bit lower than this."}, {"start": 507.56, "end": 513.44, "text": " But when you're overfitting the training set, you may be able to fit the training set so well"}, {"start": 513.44, "end": 518.64, "text": " to have an unrealistically low error, such as zero error in this example over here,"}, {"start": 518.64, "end": 524.3199999999999, "text": " which is actually better than how well humans would actually be able to predict housing prices"}, {"start": 524.3199999999999, "end": 526.16, "text": " or whatever the application you're working on."}, {"start": 526.16, "end": 532.76, "text": " But again, the signal for high variance is whether JCV is much higher than JTrain."}, {"start": 532.76, "end": 539.84, "text": " And when you have high variance, then increasing the training set size could help a lot."}, {"start": 539.84, "end": 545.3199999999999, "text": " And in particular, if we could extrapolate these curves to the right, increase MTrain,"}, {"start": 545.32, "end": 551.48, "text": " then the training error will continue to go up, but then the cross-validation error"}, {"start": 551.48, "end": 556.2, "text": " hopefully will come down and approach JTrain."}, {"start": 556.2, "end": 563.2, "text": " And so in this scenario, it might be possible just by increasing the training set size"}, {"start": 563.2, "end": 568.7600000000001, "text": " to lower the cross-validation error and to get your algorithm to perform better and better."}, {"start": 568.7600000000001, "end": 574.72, "text": " And this is unlike the high bias case where if the only thing you do is get more training data,"}, {"start": 574.72, "end": 578.2, "text": " that won't actually help your learning algorithm performance much."}, {"start": 578.2, "end": 582.76, "text": " So to summarize, if a learning algorithm suffers from high variance,"}, {"start": 582.76, "end": 587.44, "text": " then getting more training data is indeed likely to help."}, {"start": 587.44, "end": 593.8000000000001, "text": " Because extrapolating to the right of this curve, you see that you can expect JCV to keep on coming down."}, {"start": 593.8000000000001, "end": 598.84, "text": " And in this example, just by getting more training data allows the algorithm to go from this"}, {"start": 598.84, "end": 604.84, "text": " relatively high cross-validation error to get much closer to human level performance."}, {"start": 604.84, "end": 611.12, "text": " You can see that if you were to add a lot more training examples and continue to fit a four-folder polynomial,"}, {"start": 611.12, "end": 618.84, "text": " then you can just get a better four-folder polynomial fit to this data than just very weakly curve up on top."}, {"start": 618.84, "end": 624.52, "text": " So if you're building a machine learning application, you could plot the learning curve if you want."}, {"start": 624.52, "end": 627.4000000000001, "text": " That is, you can take different subsets of your training sets."}, {"start": 627.4, "end": 633.6, "text": " And even if you have, say, a thousand training examples, you could train a model on just a hundred training examples"}, {"start": 633.6, "end": 639.28, "text": " and look at the training error and the cross-validation error, then train a model on 200 examples,"}, {"start": 639.28, "end": 645.92, "text": " holding out 800 examples and just not using them for now, and plot J train and JCV and so on and repeat"}, {"start": 645.92, "end": 648.9599999999999, "text": " and plot out what the learning curve looks like."}, {"start": 648.9599999999999, "end": 655.0, "text": " And if you were to visualize it that way, then that could be another way for you to see if your learning curve"}, {"start": 655.0, "end": 658.36, "text": " looks more like a high bias or high variance one."}, {"start": 658.36, "end": 661.48, "text": " One downside of plotting learning curves like this is something I've done,"}, {"start": 661.48, "end": 667.32, "text": " but one downside is it is computationally quite expensive to train so many different models"}, {"start": 667.32, "end": 670.56, "text": " using different size subsets of your training set."}, {"start": 670.56, "end": 673.4, "text": " So in practice, it isn't done that often."}, {"start": 673.4, "end": 679.4, "text": " But nonetheless, I find that having this mental visual picture in my head of what the training set looks like,"}, {"start": 679.4, "end": 684.0, "text": " sometimes that helps me to think through what I think my learning algorithm is doing"}, {"start": 684.0, "end": 687.8, "text": " and whether it has high bias or high variance."}, {"start": 687.8, "end": 691.88, "text": " So I know we've gone through a lot about bias and variance."}, {"start": 691.88, "end": 697.48, "text": " Let's go back to our earlier example of if you have trained a model for housing price prediction,"}, {"start": 697.48, "end": 701.24, "text": " how does bias and variance help you decide what to do next?"}, {"start": 701.24, "end": 705.84, "text": " Let's go back to that earlier example, which I hope will now make a lot more sense to you."}, {"start": 705.84, "end": 715.84, "text": " Let's do that in the next video."}]