6.9 Bias and variance | Bias/variance and neural networks --[Machine Learning | Andrew Ng]
 We've seen that high bias or high variance are both bad in the sense that they hurt the performance of your algorithm. One of the reasons that neural networks have been so successful is because neural networks, together with the idea of big data or hopefully having large datasets, is giving us a new way of new ways to address both high bias and high variance. Let's take a look. You saw that if you're fitting different order polynomials to a dataset, then if you were to fit a linear model like this on the left, you have a pretty simple model that can have high bias. Whereas if you were to fit a complex model, then you might suffer from high variance. And there's this trade-off between bias and variance. In our example, it was choosing a second-order polynomial that helps you make a trade-off and pick a model with lowest possible cross-validation error. And so before the days of neural networks, machine learning engineers talked a lot about this bias-variance trade-off in which you had to balance the complexity, that is, the degree of polynomial or the regularization parameter lambda to make bias and variance both not be too high. And if you hear machine learning engineers talk about the bias-variance trade-off, this is what they're referring to, where if you have too simple a model, you have high bias, too complex a model, high variance, and you have to find a trade-off between these two bad things to find hopefully the best possible outcome. But it turns out that neural networks offer us away all of this dilemma of having to trade off bias and variance with some caveats. And it turns out that large neural networks, when trained on small to moderate-sized datasets, are low-bias machines. And what I mean by that is if you make your neural network large enough, you can almost always fit your training set well. So unless your training set is not enormous. And what this means is this gives us a new recipe to try to reduce bias or reduce variance as needed without needing to really trade off between the two of them. So let me share with you a simple recipe that isn't always applicable, but if it applies, can be very powerful for getting an accurate model using a neural network, which is first train your algorithm on your training set and then ask, does it do well on the training set? So measure J train and see if it is high. And by high, I mean, for example, relative to human level performance or some baseline level performance. And if it is not doing well, then you have a high bias problem, high training set error. And one way to reduce bias is to just use a bigger neural network. And by bigger neural network, I mean, either more hidden layers or more hidden units per layer. And you can then keep on going through this loop and make your neural network bigger and bigger until it does well on the training set, meaning it achieves a level of error on your training set that is roughly comparable to the target level of error you hope to get to, which could be human level performance. After it does well on the training set, so the answer to that question is yes, you would then ask, does it do well on the cross validation set? In other words, does it have high variance? And if the answer is no, then you can conclude that the algorithm has high variance because it does well on the training set, does not do well on the cross validation set. So that big gap in JCV and JTrain indicates you probably have a high variance problem. And if you have a high variance problem, then one way to try to fix it is to get more data. So you get more data and go back and retrain the model and just double check that it does well on the training set. If not, have a bigger network or if it does, see if it does well on the cross validation set and if not, get more data. Then if you can keep on going round and round and round this loop until eventually it does well on the cross validation set, then you're probably done because now you have a model that does well on the cross validation set and hopefully will also generalize to new examples as well. Now of course there are limitations to the application of this recipe. Training a bigger neural network does reduce bias, but at some point it does get computationally expensive. That's why the rise of neural networks has been really assisted by the rise of very fast computers, including especially GPUs or graphics processor units. Hardware traditionally used to speed up computer graphics, but that turns out has been very useful for speeding up neural networks as well. But even with hardware accelerators, beyond a certain point, the neural networks are still large and take so long to train, it becomes infeasible. And then of course the other limitation is more data. Sometimes you can only get so much data and beyond a certain point, it's hard to get much more data. But I think this recipe explains a lot of the rise of deep learning in the last several years, which is for applications where you do have access to a lot of data. Then being able to train large neural networks allows you to eventually get pretty good performance on a lot of applications. One thing that was implicit in this slide that may not have been obvious is that as you're developing a learning algorithm, sometimes you find that you have high bias, in which case you do things like increase the neural network. But then after you increase the neural network, you may find that you have high variance, in which case you might do other things like collect more data. Then during the hours or days or weeks you're developing a machine learning algorithm, at different points you may have high bias or high variance and it can change. But it's depending on whether your algorithm has high bias or high variance at that time that that can help give guidance for what you should be trying next. When you're training a neural network, one thing that people have asked me before is, hey Andrew, whether my neural network is too big, will that create a high variance problem? It turns out that a large neural network with well-chosen regularization will usually do as well or better than a smaller one. And so, for example, if you have a small neural network like this and you were to switch to a much larger neural network like this, you would think that the risk of overfitting goes up significantly. But it turns out that if you were to regularize this larger neural network appropriately, then this larger neural network usually will do at least as well or better than the smaller one, so long as the regularization is chosen appropriately. So another way of saying this is that it almost never hurts to go to a larger neural network so long as you regularize appropriately. With one caveat, which is that when you train a larger neural network, it does become more computationally expensive. So the main way it hurts is it will slow down your training and your inference process. And very briefly, to regularize a neural network, this is what you do. If the cost function for your neural network is the average loss, and so the loss here could be squared error or logistic loss, then the regularization term for a neural network looks like pretty much what you expect is lambda over 2m times the sum of w squared, where this is the sum over all weights w in the neural network. And similar to regularization for linear regression and logistic regression, we usually don't regularize the parameters b in a neural network, although in practice it makes very little difference whether you do so or not. And the way you would implement regularization in TensorFlow is, recall that this was the code for implementing an un-regularized handwritten digit classification model. We create three layers like so with number of hidden units, activation, and then create a sequential model with the three layers. If you want to add regularization, then you would just add this extra term kernel regularizer equals L2 and then 0.01, where that's the value of lambda. TensorFlow actually lets you choose different values of lambda for different layers, although for simplicity, you can choose the same value of lambda for all the weights in all of the different layers as follows. And then this will allow you to implement regularization in your neural network. So to summarize, two takeaways I hope you have from this video are, one, it hardly ever hurts to have a larger neural network so long as you regularize appropriately. One caveat being that having a larger neural network can slow down your out-of-room, so maybe that's the one way it hurts, but it shouldn't hurt your out-of-room's performance for the most part. And in fact, it could even help it significantly. And second, so long as your training set isn't too large, then a neural network, especially a large neural network, is often a low-bias machine. It just fits very complicated functions very well, which is why when I'm training neural networks, I find that I'm often fighting various problems rather than bias problems, at least if the neural network is large enough. So the rise of deep learning has really changed the way that machine learning practitioners think about bias and variance. Having said that, even when you're training a neural network, measuring bias and variance and using that to guide what you do next is often a very hopeful thing to do. So that's it for bias and variance. Let's go on to the next video where we'll take all the ideas we've learned and see how they fit in to the development process of machine learning systems. And I hope that we'll tie a lot of these pieces together to give you practical advice for how to quickly move forward in the development of your machine learning systems.
[{"start": 0.0, "end": 6.4, "text": " We've seen that high bias or high variance are both bad in the sense that they hurt the"}, {"start": 6.4, "end": 8.88, "text": " performance of your algorithm."}, {"start": 8.88, "end": 14.0, "text": " One of the reasons that neural networks have been so successful is because neural networks,"}, {"start": 14.0, "end": 19.42, "text": " together with the idea of big data or hopefully having large datasets, is giving us a new"}, {"start": 19.42, "end": 24.560000000000002, "text": " way of new ways to address both high bias and high variance."}, {"start": 24.560000000000002, "end": 25.6, "text": " Let's take a look."}, {"start": 25.6, "end": 32.28, "text": " You saw that if you're fitting different order polynomials to a dataset, then if you were"}, {"start": 32.28, "end": 38.400000000000006, "text": " to fit a linear model like this on the left, you have a pretty simple model that can have"}, {"start": 38.400000000000006, "end": 39.400000000000006, "text": " high bias."}, {"start": 39.400000000000006, "end": 45.760000000000005, "text": " Whereas if you were to fit a complex model, then you might suffer from high variance."}, {"start": 45.760000000000005, "end": 50.32, "text": " And there's this trade-off between bias and variance."}, {"start": 50.32, "end": 57.08, "text": " In our example, it was choosing a second-order polynomial that helps you make a trade-off"}, {"start": 57.08, "end": 62.44, "text": " and pick a model with lowest possible cross-validation error."}, {"start": 62.44, "end": 67.96000000000001, "text": " And so before the days of neural networks, machine learning engineers talked a lot about"}, {"start": 67.96000000000001, "end": 75.36, "text": " this bias-variance trade-off in which you had to balance the complexity, that is, the"}, {"start": 75.36, "end": 80.8, "text": " degree of polynomial or the regularization parameter lambda to make bias and variance"}, {"start": 80.8, "end": 83.44, "text": " both not be too high."}, {"start": 83.44, "end": 88.2, "text": " And if you hear machine learning engineers talk about the bias-variance trade-off, this"}, {"start": 88.2, "end": 93.03999999999999, "text": " is what they're referring to, where if you have too simple a model, you have high bias,"}, {"start": 93.03999999999999, "end": 96.96000000000001, "text": " too complex a model, high variance, and you have to find a trade-off between these two"}, {"start": 96.96000000000001, "end": 101.9, "text": " bad things to find hopefully the best possible outcome."}, {"start": 101.9, "end": 108.08000000000001, "text": " But it turns out that neural networks offer us away all of this dilemma of having to trade"}, {"start": 108.08000000000001, "end": 111.44000000000001, "text": " off bias and variance with some caveats."}, {"start": 111.44000000000001, "end": 119.48, "text": " And it turns out that large neural networks, when trained on small to moderate-sized datasets,"}, {"start": 119.48, "end": 121.84, "text": " are low-bias machines."}, {"start": 121.84, "end": 128.34, "text": " And what I mean by that is if you make your neural network large enough, you can almost"}, {"start": 128.34, "end": 131.16, "text": " always fit your training set well."}, {"start": 131.16, "end": 133.79999999999998, "text": " So unless your training set is not enormous."}, {"start": 133.79999999999998, "end": 140.16, "text": " And what this means is this gives us a new recipe to try to reduce bias or reduce variance"}, {"start": 140.16, "end": 144.51999999999998, "text": " as needed without needing to really trade off between the two of them."}, {"start": 144.51999999999998, "end": 149.92, "text": " So let me share with you a simple recipe that isn't always applicable, but if it applies,"}, {"start": 149.92, "end": 158.2, "text": " can be very powerful for getting an accurate model using a neural network, which is first"}, {"start": 158.2, "end": 163.28, "text": " train your algorithm on your training set and then ask, does it do well on the training"}, {"start": 163.28, "end": 164.28, "text": " set?"}, {"start": 164.28, "end": 168.44, "text": " So measure J train and see if it is high."}, {"start": 168.44, "end": 173.64, "text": " And by high, I mean, for example, relative to human level performance or some baseline"}, {"start": 173.64, "end": 176.07999999999998, "text": " level performance."}, {"start": 176.07999999999998, "end": 182.67999999999998, "text": " And if it is not doing well, then you have a high bias problem, high training set error."}, {"start": 182.68, "end": 188.4, "text": " And one way to reduce bias is to just use a bigger neural network."}, {"start": 188.4, "end": 193.24, "text": " And by bigger neural network, I mean, either more hidden layers or more hidden units per"}, {"start": 193.24, "end": 194.84, "text": " layer."}, {"start": 194.84, "end": 199.96, "text": " And you can then keep on going through this loop and make your neural network bigger and"}, {"start": 199.96, "end": 204.56, "text": " bigger until it does well on the training set, meaning it achieves a level of error"}, {"start": 204.56, "end": 210.86, "text": " on your training set that is roughly comparable to the target level of error you hope to get"}, {"start": 210.86, "end": 215.0, "text": " to, which could be human level performance."}, {"start": 215.0, "end": 218.92000000000002, "text": " After it does well on the training set, so the answer to that question is yes, you would"}, {"start": 218.92000000000002, "end": 222.28, "text": " then ask, does it do well on the cross validation set?"}, {"start": 222.28, "end": 226.52, "text": " In other words, does it have high variance?"}, {"start": 226.52, "end": 232.10000000000002, "text": " And if the answer is no, then you can conclude that the algorithm has high variance because"}, {"start": 232.10000000000002, "end": 235.60000000000002, "text": " it does well on the training set, does not do well on the cross validation set."}, {"start": 235.6, "end": 243.0, "text": " So that big gap in JCV and JTrain indicates you probably have a high variance problem."}, {"start": 243.0, "end": 247.51999999999998, "text": " And if you have a high variance problem, then one way to try to fix it is to get more data."}, {"start": 247.51999999999998, "end": 253.04, "text": " So you get more data and go back and retrain the model and just double check that it does"}, {"start": 253.04, "end": 254.4, "text": " well on the training set."}, {"start": 254.4, "end": 258.44, "text": " If not, have a bigger network or if it does, see if it does well on the cross validation"}, {"start": 258.44, "end": 261.6, "text": " set and if not, get more data."}, {"start": 261.6, "end": 265.76000000000005, "text": " Then if you can keep on going round and round and round this loop until eventually it does"}, {"start": 265.76000000000005, "end": 271.6, "text": " well on the cross validation set, then you're probably done because now you have a model"}, {"start": 271.6, "end": 277.08000000000004, "text": " that does well on the cross validation set and hopefully will also generalize to new"}, {"start": 277.08000000000004, "end": 278.92, "text": " examples as well."}, {"start": 278.92, "end": 283.64000000000004, "text": " Now of course there are limitations to the application of this recipe."}, {"start": 283.64000000000004, "end": 288.52000000000004, "text": " Training a bigger neural network does reduce bias, but at some point it does get computationally"}, {"start": 288.52000000000004, "end": 289.92, "text": " expensive."}, {"start": 289.92, "end": 296.0, "text": " That's why the rise of neural networks has been really assisted by the rise of very fast"}, {"start": 296.0, "end": 301.92, "text": " computers, including especially GPUs or graphics processor units."}, {"start": 301.92, "end": 306.12, "text": " Hardware traditionally used to speed up computer graphics, but that turns out has been very"}, {"start": 306.12, "end": 308.88, "text": " useful for speeding up neural networks as well."}, {"start": 308.88, "end": 313.32, "text": " But even with hardware accelerators, beyond a certain point, the neural networks are still"}, {"start": 313.32, "end": 316.98, "text": " large and take so long to train, it becomes infeasible."}, {"start": 316.98, "end": 321.28000000000003, "text": " And then of course the other limitation is more data."}, {"start": 321.28000000000003, "end": 326.12, "text": " Sometimes you can only get so much data and beyond a certain point, it's hard to get much"}, {"start": 326.12, "end": 327.12, "text": " more data."}, {"start": 327.12, "end": 333.48, "text": " But I think this recipe explains a lot of the rise of deep learning in the last several"}, {"start": 333.48, "end": 338.28000000000003, "text": " years, which is for applications where you do have access to a lot of data."}, {"start": 338.28000000000003, "end": 345.26, "text": " Then being able to train large neural networks allows you to eventually get pretty good performance"}, {"start": 345.26, "end": 347.12, "text": " on a lot of applications."}, {"start": 347.12, "end": 352.94, "text": " One thing that was implicit in this slide that may not have been obvious is that as"}, {"start": 352.94, "end": 358.56, "text": " you're developing a learning algorithm, sometimes you find that you have high bias, in which"}, {"start": 358.56, "end": 361.08, "text": " case you do things like increase the neural network."}, {"start": 361.08, "end": 365.48, "text": " But then after you increase the neural network, you may find that you have high variance,"}, {"start": 365.48, "end": 369.52, "text": " in which case you might do other things like collect more data."}, {"start": 369.52, "end": 375.79999999999995, "text": " Then during the hours or days or weeks you're developing a machine learning algorithm, at"}, {"start": 375.79999999999995, "end": 379.88, "text": " different points you may have high bias or high variance and it can change."}, {"start": 379.88, "end": 384.71999999999997, "text": " But it's depending on whether your algorithm has high bias or high variance at that time"}, {"start": 384.71999999999997, "end": 389.4, "text": " that that can help give guidance for what you should be trying next."}, {"start": 389.4, "end": 394.28, "text": " When you're training a neural network, one thing that people have asked me before is,"}, {"start": 394.28, "end": 401.52, "text": " hey Andrew, whether my neural network is too big, will that create a high variance problem?"}, {"start": 401.52, "end": 408.59999999999997, "text": " It turns out that a large neural network with well-chosen regularization will usually do"}, {"start": 408.59999999999997, "end": 411.84, "text": " as well or better than a smaller one."}, {"start": 411.84, "end": 418.94, "text": " And so, for example, if you have a small neural network like this and you were to switch to"}, {"start": 418.94, "end": 424.7, "text": " a much larger neural network like this, you would think that the risk of overfitting goes"}, {"start": 424.7, "end": 426.7, "text": " up significantly."}, {"start": 426.7, "end": 432.36, "text": " But it turns out that if you were to regularize this larger neural network appropriately,"}, {"start": 432.36, "end": 438.24, "text": " then this larger neural network usually will do at least as well or better than the smaller"}, {"start": 438.24, "end": 442.56, "text": " one, so long as the regularization is chosen appropriately."}, {"start": 442.56, "end": 448.12, "text": " So another way of saying this is that it almost never hurts to go to a larger neural network"}, {"start": 448.12, "end": 450.98, "text": " so long as you regularize appropriately."}, {"start": 450.98, "end": 456.0, "text": " With one caveat, which is that when you train a larger neural network, it does become more"}, {"start": 456.0, "end": 457.68, "text": " computationally expensive."}, {"start": 457.68, "end": 463.08, "text": " So the main way it hurts is it will slow down your training and your inference process."}, {"start": 463.08, "end": 469.52, "text": " And very briefly, to regularize a neural network, this is what you do."}, {"start": 469.52, "end": 475.24, "text": " If the cost function for your neural network is the average loss, and so the loss here"}, {"start": 475.24, "end": 482.08, "text": " could be squared error or logistic loss, then the regularization term for a neural network"}, {"start": 482.08, "end": 489.56, "text": " looks like pretty much what you expect is lambda over 2m times the sum of w squared,"}, {"start": 489.56, "end": 494.12, "text": " where this is the sum over all weights w in the neural network."}, {"start": 494.12, "end": 499.2, "text": " And similar to regularization for linear regression and logistic regression, we usually don't"}, {"start": 499.2, "end": 503.92, "text": " regularize the parameters b in a neural network, although in practice it makes very little"}, {"start": 503.92, "end": 506.88, "text": " difference whether you do so or not."}, {"start": 506.88, "end": 512.74, "text": " And the way you would implement regularization in TensorFlow is, recall that this was the"}, {"start": 512.74, "end": 519.0600000000001, "text": " code for implementing an un-regularized handwritten digit classification model."}, {"start": 519.0600000000001, "end": 524.9200000000001, "text": " We create three layers like so with number of hidden units, activation, and then create"}, {"start": 524.9200000000001, "end": 528.02, "text": " a sequential model with the three layers."}, {"start": 528.02, "end": 535.12, "text": " If you want to add regularization, then you would just add this extra term kernel regularizer"}, {"start": 535.12, "end": 541.12, "text": " equals L2 and then 0.01, where that's the value of lambda."}, {"start": 541.12, "end": 545.52, "text": " TensorFlow actually lets you choose different values of lambda for different layers, although"}, {"start": 545.52, "end": 550.8, "text": " for simplicity, you can choose the same value of lambda for all the weights in all of the"}, {"start": 550.8, "end": 552.76, "text": " different layers as follows."}, {"start": 552.76, "end": 558.36, "text": " And then this will allow you to implement regularization in your neural network."}, {"start": 558.36, "end": 564.04, "text": " So to summarize, two takeaways I hope you have from this video are, one, it hardly ever"}, {"start": 564.04, "end": 570.16, "text": " hurts to have a larger neural network so long as you regularize appropriately."}, {"start": 570.16, "end": 574.64, "text": " One caveat being that having a larger neural network can slow down your out-of-room, so"}, {"start": 574.64, "end": 578.4399999999999, "text": " maybe that's the one way it hurts, but it shouldn't hurt your out-of-room's performance"}, {"start": 578.4399999999999, "end": 579.4399999999999, "text": " for the most part."}, {"start": 579.44, "end": 583.0400000000001, "text": " And in fact, it could even help it significantly."}, {"start": 583.0400000000001, "end": 589.1600000000001, "text": " And second, so long as your training set isn't too large, then a neural network, especially"}, {"start": 589.1600000000001, "end": 592.62, "text": " a large neural network, is often a low-bias machine."}, {"start": 592.62, "end": 598.1600000000001, "text": " It just fits very complicated functions very well, which is why when I'm training neural"}, {"start": 598.1600000000001, "end": 603.36, "text": " networks, I find that I'm often fighting various problems rather than bias problems, at least"}, {"start": 603.36, "end": 605.72, "text": " if the neural network is large enough."}, {"start": 605.72, "end": 610.28, "text": " So the rise of deep learning has really changed the way that machine learning practitioners"}, {"start": 610.28, "end": 612.48, "text": " think about bias and variance."}, {"start": 612.48, "end": 616.5, "text": " Having said that, even when you're training a neural network, measuring bias and variance"}, {"start": 616.5, "end": 623.32, "text": " and using that to guide what you do next is often a very hopeful thing to do."}, {"start": 623.32, "end": 626.32, "text": " So that's it for bias and variance."}, {"start": 626.32, "end": 630.4, "text": " Let's go on to the next video where we'll take all the ideas we've learned and see how"}, {"start": 630.4, "end": 634.36, "text": " they fit in to the development process of machine learning systems."}, {"start": 634.36, "end": 639.76, "text": " And I hope that we'll tie a lot of these pieces together to give you practical advice for"}, {"start": 639.76, "end": 668.3199999999999, "text": " how to quickly move forward in the development of your machine learning systems."}]