6.8 Bias and variance | Deciding what to try next revisited --[Machine Learning | Andrew Ng]
 You've seen how by looking at JTrain and JCV, that is the training error and the cross-validation error, or maybe even plotting a learning curve, you can try to get a sense of whether your learning algorithm has high bias or high variance. This is a procedure I routinely do when I'm training a learning algorithm. I'll often look at the training error and the cross-validation error to try to decide if my algorithm has high bias or high variance. It turns out this will help you make better decisions about what to try next in order to improve the performance of your learning algorithm. Let's look at an example. This is actually the example that you had seen earlier. If you've implemented regularized linear regression on predicting housing prices, but your algorithm makes unacceptably large errors in its predictions, what do you try next? And these were the six ideas that we had when we had looked over this slide earlier. Get more training examples, try a smaller set of features, additional features, and so on. It turns out that each of these six items either helps fix a high variance or a high bias problem. And in particular, if your learning algorithm has high bias, three of these techniques will be useful. If your learning algorithm has high variance, then a different three of these techniques will be useful. Let's see if we can figure out which is which. First one is get more training examples. We saw in the last video that if your algorithm has high bias, then if the only thing we do is get more training data, that by itself probably won't help that much. But in contrast, if your algorithm had high variance, say it was overfitting to a very small training set, then getting more training examples will help a lot. So this first option of getting more training examples helps to fix a high variance problem. How about the other five? Do you think you can figure out which of the remaining five fix high bias or high variance problems? I'm going to go through the rest of them in this video in a minute, but if you want, be free to pause the video and see if you can think through these five other things by yourself. So feel free to pause the video and just kidding, that was me pausing, not your video pausing. But seriously, if you want, go ahead and pause the video and think through it if you want or not. And we'll go over these with you in a minute. How about trying a smaller set of features? Sometimes if your learning algorithm has too many features, then it gives the algorithm too much flexibility to fit very complicated models. This is a little bit like if you had x squared, x cubed, x to the fourth, x to the fifth, and so on. And if only you were to eliminate a few of these, then your model won't be so complex and won't have such high variance. So if you suspect that your algorithm has a lot of features that are not actually relevant or helpful to predicting a housing price, or if you suspect that you had even somewhat redundant features, then eliminating or reducing the number of features will help reduce the flexibility of your algorithm to overfit the data. And so this is a tactic that will help you to fix high variance. Conversely, getting additional features that's just adding additional features is kind of the opposite of going to a smaller set of features. This will help you to fix a high bias problem. As a concrete example, if you were trying to predict the price of a house just based on the size, but it turns out that the price of a house also really depends on the number of bedrooms and on the number of floors and on the age of a house, then the algorithm will never do that well unless you add in those additional features. So that's a high bias problem because you just can't do that well on the training set when you know only the size. Because only when you tell the algorithm how many bedrooms are there, how many floors are there, what's the age of the house, that it finally has enough information to even do better on the training set. And so adding additional features is a way to fix a high bias problem. Adding polynomial features is a little bit like adding additional features. So if your linear function, straight line, can't fit the training set that well, then adding additional polynomial features could help you do better on the training set. And helping you do better on the training set is a way to fix a high bias problem. And then decreasing lambda means to use a lower value for the regularization parameter. And that means we're going to pay less attention to this term and pay more attention to this term to try to do better on the training set. And again, that helps you to fix a high bias problem. And finally, increasing lambda, well, that's the opposite of this. But that says you're overfitting the data. And so increasing lambda would make sense if it's overfitting the training set, just putting too much attention to fit in the training set, but at the expense of generalizing to new examples. And so increasing lambda would force the algorithm to fit a smoother function, maybe a less wiggly function and use this to fix a high variance problem. So I realized that this was a lot of stuff on the slide. But the takeaways I hope you have are, if you find that your algorithm has high variance, then the two main ways to fix that are either get more training data or simplify your model. And by simplify model, I mean, either get a smaller set of features or increase the regularization parameter lambda. So your algorithm has less flexibility to fit very complex, very wiggly curves. Conversely, if your algorithm has high bias, then that means it's not doing well even on the training set. So if that's the case, the main fixes are to make your model more powerful or to give it more flexibility to fit more complex or more wiggly functions. And so some ways to do that are to give it additional features or add these polynomial features or to decrease the regularization parameter lambda. By the way, in case you're wondering if you should fix high bias by reducing the training set size, that doesn't actually help. If you reduce the training set size, you will fit the training set better, but that tends to worsen your cross validation error and the performance of your learning algorithm. So don't randomly throw away training examples just to try to fix a high bias problem. One of my PhD students from Stanford, many years after he'd already graduated from Stanford, once said to me that while he was studying at Stanford, he learned about bias and variance and felt like he got it, he understood it. But that subsequently, after many years of work experience at a few different companies, he realized that bias and variance is one of those concepts that takes a short time to learn, but takes a lifetime to master. Those were his exact words. And I think bias and variance is one of those very powerful ideas. When I'm training learning algorithms, I almost always try to figure out if it is high bias or high variance. But the way you go about addressing it systematically is something that I think you will keep on getting better at through repeated practice. But you'll find, I think, that understanding these ideas will help you be much more effective at how you decide what to try next when developing a learning algorithm. Now I know that we did go through a lot in this video. And if you feel like, boy, there's just a lot of stuff here, it's okay, don't worry about it. Later this week in the practice labs and practice quizzes, we'll have also additional opportunities to go over these ideas so that you can get additional practice with thinking about bias and variance of different learning algorithms. So if it seems like a lot right now, it's okay. You get to practice these ideas later this week and hopefully deepen your understanding of them at that time. Before moving on, bias and variance also are very useful when thinking about how to train a neural network. So in the next video, let's take a look at these concepts applied to neural network training. Let's go on to the next video.
[{"start": 0.0, "end": 8.56, "text": " You've seen how by looking at JTrain and JCV, that is the training error and the cross-validation"}, {"start": 8.56, "end": 14.0, "text": " error, or maybe even plotting a learning curve, you can try to get a sense of whether your"}, {"start": 14.0, "end": 17.68, "text": " learning algorithm has high bias or high variance."}, {"start": 17.68, "end": 21.32, "text": " This is a procedure I routinely do when I'm training a learning algorithm."}, {"start": 21.32, "end": 25.48, "text": " I'll often look at the training error and the cross-validation error to try to decide"}, {"start": 25.48, "end": 28.240000000000002, "text": " if my algorithm has high bias or high variance."}, {"start": 28.24, "end": 32.8, "text": " It turns out this will help you make better decisions about what to try next in order"}, {"start": 32.8, "end": 35.96, "text": " to improve the performance of your learning algorithm."}, {"start": 35.96, "end": 37.96, "text": " Let's look at an example."}, {"start": 37.96, "end": 41.72, "text": " This is actually the example that you had seen earlier."}, {"start": 41.72, "end": 48.36, "text": " If you've implemented regularized linear regression on predicting housing prices, but your algorithm"}, {"start": 48.36, "end": 53.2, "text": " makes unacceptably large errors in its predictions, what do you try next?"}, {"start": 53.2, "end": 57.28, "text": " And these were the six ideas that we had when we had looked over this slide earlier."}, {"start": 57.28, "end": 61.32, "text": " Get more training examples, try a smaller set of features, additional features, and"}, {"start": 61.32, "end": 62.32, "text": " so on."}, {"start": 62.32, "end": 69.48, "text": " It turns out that each of these six items either helps fix a high variance or a high"}, {"start": 69.48, "end": 71.64, "text": " bias problem."}, {"start": 71.64, "end": 77.28, "text": " And in particular, if your learning algorithm has high bias, three of these techniques will"}, {"start": 77.28, "end": 78.62, "text": " be useful."}, {"start": 78.62, "end": 82.8, "text": " If your learning algorithm has high variance, then a different three of these techniques"}, {"start": 82.8, "end": 83.8, "text": " will be useful."}, {"start": 83.8, "end": 87.6, "text": " Let's see if we can figure out which is which."}, {"start": 87.6, "end": 91.92, "text": " First one is get more training examples."}, {"start": 91.92, "end": 97.96, "text": " We saw in the last video that if your algorithm has high bias, then if the only thing we do"}, {"start": 97.96, "end": 104.03999999999999, "text": " is get more training data, that by itself probably won't help that much."}, {"start": 104.03999999999999, "end": 109.24, "text": " But in contrast, if your algorithm had high variance, say it was overfitting to a very"}, {"start": 109.24, "end": 115.53999999999999, "text": " small training set, then getting more training examples will help a lot."}, {"start": 115.53999999999999, "end": 122.88, "text": " So this first option of getting more training examples helps to fix a high variance problem."}, {"start": 122.88, "end": 124.47999999999999, "text": " How about the other five?"}, {"start": 124.47999999999999, "end": 129.79999999999998, "text": " Do you think you can figure out which of the remaining five fix high bias or high variance"}, {"start": 129.79999999999998, "end": 130.79999999999998, "text": " problems?"}, {"start": 130.79999999999998, "end": 134.0, "text": " I'm going to go through the rest of them in this video in a minute, but if you want, be"}, {"start": 134.0, "end": 138.94, "text": " free to pause the video and see if you can think through these five other things by yourself."}, {"start": 138.94, "end": 145.84, "text": " So feel free to pause the video and just kidding, that was me pausing, not your video pausing."}, {"start": 145.84, "end": 149.12, "text": " But seriously, if you want, go ahead and pause the video and think through it if you want"}, {"start": 149.12, "end": 150.12, "text": " or not."}, {"start": 150.12, "end": 153.07999999999998, "text": " And we'll go over these with you in a minute."}, {"start": 153.07999999999998, "end": 157.48, "text": " How about trying a smaller set of features?"}, {"start": 157.48, "end": 162.76, "text": " Sometimes if your learning algorithm has too many features, then it gives the algorithm"}, {"start": 162.76, "end": 167.16, "text": " too much flexibility to fit very complicated models."}, {"start": 167.16, "end": 173.28, "text": " This is a little bit like if you had x squared, x cubed, x to the fourth, x to the fifth,"}, {"start": 173.28, "end": 174.88, "text": " and so on."}, {"start": 174.88, "end": 180.76, "text": " And if only you were to eliminate a few of these, then your model won't be so complex"}, {"start": 180.76, "end": 183.88, "text": " and won't have such high variance."}, {"start": 183.88, "end": 190.0, "text": " So if you suspect that your algorithm has a lot of features that are not actually relevant"}, {"start": 190.0, "end": 195.51999999999998, "text": " or helpful to predicting a housing price, or if you suspect that you had even somewhat"}, {"start": 195.52, "end": 202.44, "text": " redundant features, then eliminating or reducing the number of features will help reduce the"}, {"start": 202.44, "end": 206.56, "text": " flexibility of your algorithm to overfit the data."}, {"start": 206.56, "end": 210.48000000000002, "text": " And so this is a tactic that will help you to fix high variance."}, {"start": 210.48000000000002, "end": 214.92000000000002, "text": " Conversely, getting additional features that's just adding additional features is kind of"}, {"start": 214.92000000000002, "end": 219.20000000000002, "text": " the opposite of going to a smaller set of features."}, {"start": 219.20000000000002, "end": 223.04000000000002, "text": " This will help you to fix a high bias problem."}, {"start": 223.04, "end": 226.5, "text": " As a concrete example, if you were trying to predict the price of a house just based"}, {"start": 226.5, "end": 232.44, "text": " on the size, but it turns out that the price of a house also really depends on the number"}, {"start": 232.44, "end": 238.35999999999999, "text": " of bedrooms and on the number of floors and on the age of a house, then the algorithm"}, {"start": 238.35999999999999, "end": 241.76, "text": " will never do that well unless you add in those additional features."}, {"start": 241.76, "end": 247.72, "text": " So that's a high bias problem because you just can't do that well on the training set"}, {"start": 247.72, "end": 250.16, "text": " when you know only the size."}, {"start": 250.16, "end": 254.2, "text": " Because only when you tell the algorithm how many bedrooms are there, how many floors are"}, {"start": 254.2, "end": 257.8, "text": " there, what's the age of the house, that it finally has enough information to even do"}, {"start": 257.8, "end": 260.48, "text": " better on the training set."}, {"start": 260.48, "end": 266.64, "text": " And so adding additional features is a way to fix a high bias problem."}, {"start": 266.64, "end": 271.26, "text": " Adding polynomial features is a little bit like adding additional features."}, {"start": 271.26, "end": 276.36, "text": " So if your linear function, straight line, can't fit the training set that well, then"}, {"start": 276.36, "end": 281.16, "text": " adding additional polynomial features could help you do better on the training set."}, {"start": 281.16, "end": 286.44, "text": " And helping you do better on the training set is a way to fix a high bias problem."}, {"start": 286.44, "end": 294.40000000000003, "text": " And then decreasing lambda means to use a lower value for the regularization parameter."}, {"start": 294.40000000000003, "end": 298.88, "text": " And that means we're going to pay less attention to this term and pay more attention to this"}, {"start": 298.88, "end": 301.64, "text": " term to try to do better on the training set."}, {"start": 301.64, "end": 305.84000000000003, "text": " And again, that helps you to fix a high bias problem."}, {"start": 305.84, "end": 309.76, "text": " And finally, increasing lambda, well, that's the opposite of this."}, {"start": 309.76, "end": 312.94, "text": " But that says you're overfitting the data."}, {"start": 312.94, "end": 317.35999999999996, "text": " And so increasing lambda would make sense if it's overfitting the training set, just"}, {"start": 317.35999999999996, "end": 323.03999999999996, "text": " putting too much attention to fit in the training set, but at the expense of generalizing to"}, {"start": 323.03999999999996, "end": 324.85999999999996, "text": " new examples."}, {"start": 324.85999999999996, "end": 330.79999999999995, "text": " And so increasing lambda would force the algorithm to fit a smoother function, maybe a less wiggly"}, {"start": 330.79999999999995, "end": 335.79999999999995, "text": " function and use this to fix a high variance problem."}, {"start": 335.8, "end": 339.94, "text": " So I realized that this was a lot of stuff on the slide."}, {"start": 339.94, "end": 345.8, "text": " But the takeaways I hope you have are, if you find that your algorithm has high variance,"}, {"start": 345.8, "end": 353.86, "text": " then the two main ways to fix that are either get more training data or simplify your model."}, {"start": 353.86, "end": 360.04, "text": " And by simplify model, I mean, either get a smaller set of features or increase the"}, {"start": 360.04, "end": 362.44, "text": " regularization parameter lambda."}, {"start": 362.44, "end": 368.16, "text": " So your algorithm has less flexibility to fit very complex, very wiggly curves."}, {"start": 368.16, "end": 374.88, "text": " Conversely, if your algorithm has high bias, then that means it's not doing well even on"}, {"start": 374.88, "end": 376.2, "text": " the training set."}, {"start": 376.2, "end": 382.0, "text": " So if that's the case, the main fixes are to make your model more powerful or to give"}, {"start": 382.0, "end": 386.44, "text": " it more flexibility to fit more complex or more wiggly functions."}, {"start": 386.44, "end": 391.44, "text": " And so some ways to do that are to give it additional features or add these polynomial"}, {"start": 391.44, "end": 396.0, "text": " features or to decrease the regularization parameter lambda."}, {"start": 396.0, "end": 402.04, "text": " By the way, in case you're wondering if you should fix high bias by reducing the training"}, {"start": 402.04, "end": 404.8, "text": " set size, that doesn't actually help."}, {"start": 404.8, "end": 409.04, "text": " If you reduce the training set size, you will fit the training set better, but that tends"}, {"start": 409.04, "end": 413.04, "text": " to worsen your cross validation error and the performance of your learning algorithm."}, {"start": 413.04, "end": 418.15999999999997, "text": " So don't randomly throw away training examples just to try to fix a high bias problem."}, {"start": 418.16, "end": 423.20000000000005, "text": " One of my PhD students from Stanford, many years after he'd already graduated from Stanford,"}, {"start": 423.20000000000005, "end": 428.98, "text": " once said to me that while he was studying at Stanford, he learned about bias and variance"}, {"start": 428.98, "end": 431.64000000000004, "text": " and felt like he got it, he understood it."}, {"start": 431.64000000000004, "end": 436.86, "text": " But that subsequently, after many years of work experience at a few different companies,"}, {"start": 436.86, "end": 441.32000000000005, "text": " he realized that bias and variance is one of those concepts that takes a short time"}, {"start": 441.32000000000005, "end": 444.64000000000004, "text": " to learn, but takes a lifetime to master."}, {"start": 444.64000000000004, "end": 447.0, "text": " Those were his exact words."}, {"start": 447.0, "end": 451.96, "text": " And I think bias and variance is one of those very powerful ideas."}, {"start": 451.96, "end": 456.72, "text": " When I'm training learning algorithms, I almost always try to figure out if it is high bias"}, {"start": 456.72, "end": 458.28, "text": " or high variance."}, {"start": 458.28, "end": 464.56, "text": " But the way you go about addressing it systematically is something that I think you will keep on"}, {"start": 464.56, "end": 468.64, "text": " getting better at through repeated practice."}, {"start": 468.64, "end": 473.96, "text": " But you'll find, I think, that understanding these ideas will help you be much more effective"}, {"start": 473.96, "end": 479.64, "text": " at how you decide what to try next when developing a learning algorithm."}, {"start": 479.64, "end": 483.56, "text": " Now I know that we did go through a lot in this video."}, {"start": 483.56, "end": 486.56, "text": " And if you feel like, boy, there's just a lot of stuff here, it's okay, don't worry"}, {"start": 486.56, "end": 487.56, "text": " about it."}, {"start": 487.56, "end": 493.91999999999996, "text": " Later this week in the practice labs and practice quizzes, we'll have also additional opportunities"}, {"start": 493.91999999999996, "end": 498.91999999999996, "text": " to go over these ideas so that you can get additional practice with thinking about bias"}, {"start": 498.91999999999996, "end": 501.4, "text": " and variance of different learning algorithms."}, {"start": 501.4, "end": 504.44, "text": " So if it seems like a lot right now, it's okay."}, {"start": 504.44, "end": 508.56, "text": " You get to practice these ideas later this week and hopefully deepen your understanding"}, {"start": 508.56, "end": 511.76, "text": " of them at that time."}, {"start": 511.76, "end": 517.52, "text": " Before moving on, bias and variance also are very useful when thinking about how to train"}, {"start": 517.52, "end": 519.36, "text": " a neural network."}, {"start": 519.36, "end": 524.64, "text": " So in the next video, let's take a look at these concepts applied to neural network training."}, {"start": 524.64, "end": 531.64, "text": " Let's go on to the next video."}]