6.3 Evaluating and choosing models | Model selection and training/cross validation/test sets-ML Ng
 In the last video, you saw how to use a test set to evaluate the performance of a model. Let's make one further refinement to that idea in this video, which will allow you to use a technique to automatically choose a good model for your machine learning algorithm. One thing we've seen is that once the model's parameters w and b have been fit to the training set, the training error may not be a good indicator of how well the algorithm will do or how well it will generalize to new examples that were not in the training set. And in particular, for this example, the training error will be pretty much zero, and that's likely much lower than the actual generalization error. And by that I mean the average error on new examples that were not in the training set. And what you saw in the last video is that J-Test, the performance of the algorithm on examples is not trained on, that that will be a better indicator of how well the model will likely do on new data. And by that I mean other data that's not in the training set. Let's take a look at how this affects how we might use a test set to choose a model for a given machine learning application. So if fitting a function to predict housing prices or some other regression problem, one model you might consider is to fit a linear model like this. And this is a first order polynomial, and I'm going to use D equals one on this slide to denote fitting a one or first order polynomial. If you were to fit a model like this to your training set, you'd get some parameters W and B, and you can then compute J-Test to estimate how well this will generalize to new data. And on this slide, I'm going to use W1, B1, superscript there, to denote that these are the parameters you get if you were to fit a first order polynomial or degree one, D equals one polynomial. Now you might also consider fitting a second order polynomial or quadratic model. So this is the model. And if you were to fit this to your training set, you would get some parameters W2, B2, and you can then similarly evaluate those parameters on your test set and get J-Test W2, B2. And this would give you a sense of how well the second order polynomial does. And you can go on to try D equals three, that's a third order or a degree three polynomial that looks like this, and fit parameters, and similarly get J-Test. And you might keep doing this until, say, you try up to a 10th order polynomial and you end up with J-Test of W10, B10. That gives you a sense of how well the 10th order polynomial is doing. So one procedure you could try, this turns out not to be the best procedure, but one thing you could try is look at all of these J-Tests and see which one gives you the lowest value. And say you find that J-Test for the fifth order polynomial for W5, B5 turns out to be the lowest. If that's the case, then you might decide that the fifth order polynomial D equals five does best and choose that model for your application. And if you want to estimate how well this model performs, one thing you could do, but this turns out to be a slightly flawed procedure, is to report the test set error J-Test W5, B5. The reason this procedure is flawed is J-Test of W5, B5 is likely to be an optimistic estimate of the generalization error. In other words, it is likely to be lower than the actual generalization error. And the reason is, in the procedure we talked about on the slide, we basically fit one extra parameter, which is D, the degree of polynomial, and we chose this parameter using the test set. So on the previous slide, we saw that if you were to fit WB to the training data, then the training data would be an overly optimistic estimate of generalization error. And it turns out too, that if we were to choose the parameter D using the test set, then the test set J-Test is now an overly optimistic, that is lower than actual estimates of the generalization error. So the procedure on this particular slide is flawed and I don't recommend using this. Instead, if you want to automatically choose a model, such as decide what degree polynomial to use, here's how you modify the training and testing procedure in order to carry out model selection, whereby model selection, I mean, choosing amongst different models, such as these 10 different models that you might contemplate using for your machine learning application. The way we'll modify the procedure is, instead of splitting your data into just two subsets, the training set and the test set, we're going to split your data into three different subsets, which we're going to call the training set, the cross validation set, and then also the test set. So using our example from before of these 10 training examples, we might split it into putting 60% of the data into the training set. And so the notation we'll use for the training set portion will be the same as before, except that now mtrain, the number of training examples will be 6. And we might put 20% of the data into the cross validation set. And the notation I'm going to use is xCV of 1, yCV of 1 for the first cross validation example. So CV stands for cross validation, all the way down to xCV of mCV and yCV of mCV. So here mCV equals 2 in this example is the number of cross validation examples. And then finally, we have the test set same as before. So x1 through xm tests and y1 through ym tests, where m tests here is equal to 2. This is the number of test examples. We'll see on the next slide how to use the cross validation set. So the way we'll modify the procedure is you've already seen the training set and the test set. And we're going to introduce a new subset of the data called the cross validation set. The name cross validation refers to that this is an extra data set that we're going to use to check or cross check the validity or really the accuracy of different models. I don't think it's a great name, but that is what people in machine learning have gotten to call this extra data set. You may also hear people call this the validation set for short, just few syllables then cross validation. Or in some applications, people also call this the development set means basically the same thing. Or for short, sometimes you hear people call this the dev set, but all of these terms mean the same thing as cross validation set. I personally use the term dev set the most often because it's the shortest, fastest way to say it, but cross validation is probably used a little bit more often by machine learning practitioners. So armed with these three subsets of the data, training set, cross validation set and test set, you can then compute the training error, the cross validation error and the test error using these three formulas. Where as usual, none of these terms include the regularization term that is included in the training objective. And this new term in the middle, the cross validation error is just the average over your MCV cross validation examples of the average say squared error. And this term, in addition to being called cross validation error, is also commonly called the validation error for short or even the development set error or the dev error. Armed with these three measures of learning algorithm performance, this is how you can then go about carrying out model selection. You can with the 10 models, same as earlier on the slide with D equals one, D equals two, all the way up to a 10th degree or the 10th order polynomial, you can then fit the parameters W1, B1. But instead of evaluating this on your test set, you would instead evaluate these parameters on your cross validation sets and compute JCV of W1, B1. And similarly for the second model, you get JCV of W2, B2 and all the way down to JCV of W10, B10. Then in order to choose a model, you would look at which model has the lowest cross validation error. And concretely, let's say that JCV of W4, B4 is lowest, then what that means is you would pick this fourth order polynomial as the model you will use for this application. Finally, if you want to report out an estimate of the generalization error of how well this model will do on new data, you would do so using that third subset of your data, the test set and you report out J test of W4, B4. And you notice that throughout this entire procedure, you had fit these parameters using the training set, you then chose the parameter D or chose the degree of polynomial using the cross validation set. And so up until this point, you've not fit any parameters, either W or B or D to the test set. And that's why J test in this example will be a fair estimate of the generalization error of this model that has parameters W4, B4. So this gives a better procedure for model selection. And it lets you automatically make a decision like what order polynomial to choose for your linear regression model. This model selection procedure also works for choosing among other types of models. For example, choosing a neural network architecture. If you are fitting a model for handwritten digit recognition, you might consider three models like these, maybe even a larger set of models than just three, but here are a few different neural networks of small, somewhat larger and then even larger. To help you decide how many layers should your neural network have and how many hidden units per layer should you have. You can then train all three of these models and end up with parameters W1, B1 for the first model, W2, B2 for the second model and W3, B3 for the third model. And you can then evaluate the neural network's performance using JCV using your cross validation set. And with a classification problem, JCV can be the percentage of examples. And since this is a classification problem, JCV, the most common choice would be to compute this as a fraction of cross validation examples that the algorithm has misclassified. And you would compute this using all three models and then pick the model with the lowest cross validation error. So if in this example, this has the lowest cross validation error, you would then pick the second neural network and use parameters trained on this model. And finally, if you want to report out an estimate of the generalization error, you then use the test set to estimate how well the neural network that you just chose will do. So in machine learning practice, it's considered best practice to make all the decisions you want to make regarding your learning algorithm, such as how to choose parameters, what degree polynomial to use, but make decisions only looking at the training set and cross validation set and to not use the test set at all to make decisions about your model. And only after you've made all those decisions, then finally take the model you have designed and evaluated on your test set. And that procedure ensures that you haven't accidentally fit anything to the test set, so that your test set becomes still a fair and not overly optimistic estimate of the generalization error of your algorithm. So it's considered best practice in machine learning that if you have to make decisions about your model, such as fitting parameters or choosing the model architecture, such as neural network architecture or degree of polynomial if you're fitting linear regression, to make all those decisions only using your training set and your cross validation set and to not look at the test set at all while you're still making decisions regarding your learning algorithm. And it's only after you've come up with one model that's your final model to only then evaluate it on the test set. And because you haven't made any decisions using the test set, that ensures that your test set is a fair and not overly optimistic estimate of how well your model will generalize to new data. So that's model selection. And this is actually a very widely used procedure. I use this all the time to automatically choose what model to use for a given machine learning application. Now earlier this week, I mentioned running diagnostics to decide how to improve the performance of a learning algorithm. Now that you have a way to evaluate learning algorithms and even automatically choose a model, let's dive more deeply into examples of some diagnostics. The most powerful diagnostic that I know of and that I use for a lot of machine learning applications is one called bias and variance. Let's take a look at what that means in the next video.
[{"start": 0.0, "end": 7.68, "text": " In the last video, you saw how to use a test set to evaluate the performance of a model."}, {"start": 7.68, "end": 11.8, "text": " Let's make one further refinement to that idea in this video, which will allow you to"}, {"start": 11.8, "end": 17.34, "text": " use a technique to automatically choose a good model for your machine learning algorithm."}, {"start": 17.34, "end": 23.52, "text": " One thing we've seen is that once the model's parameters w and b have been fit to the training"}, {"start": 23.52, "end": 29.52, "text": " set, the training error may not be a good indicator of how well the algorithm will do"}, {"start": 29.52, "end": 34.8, "text": " or how well it will generalize to new examples that were not in the training set."}, {"start": 34.8, "end": 40.56, "text": " And in particular, for this example, the training error will be pretty much zero, and that's"}, {"start": 40.56, "end": 44.68, "text": " likely much lower than the actual generalization error."}, {"start": 44.68, "end": 51.519999999999996, "text": " And by that I mean the average error on new examples that were not in the training set."}, {"start": 51.519999999999996, "end": 55.64, "text": " And what you saw in the last video is that J-Test, the performance of the algorithm on"}, {"start": 55.64, "end": 60.88, "text": " examples is not trained on, that that will be a better indicator of how well the model"}, {"start": 60.88, "end": 63.36, "text": " will likely do on new data."}, {"start": 63.36, "end": 67.84, "text": " And by that I mean other data that's not in the training set."}, {"start": 67.84, "end": 73.68, "text": " Let's take a look at how this affects how we might use a test set to choose a model"}, {"start": 73.68, "end": 76.72, "text": " for a given machine learning application."}, {"start": 76.72, "end": 82.2, "text": " So if fitting a function to predict housing prices or some other regression problem, one"}, {"start": 82.2, "end": 86.04, "text": " model you might consider is to fit a linear model like this."}, {"start": 86.04, "end": 90.8, "text": " And this is a first order polynomial, and I'm going to use D equals one on this slide"}, {"start": 90.8, "end": 95.92, "text": " to denote fitting a one or first order polynomial."}, {"start": 95.92, "end": 100.2, "text": " If you were to fit a model like this to your training set, you'd get some parameters W"}, {"start": 100.2, "end": 106.92, "text": " and B, and you can then compute J-Test to estimate how well this will generalize to"}, {"start": 106.92, "end": 107.96000000000001, "text": " new data."}, {"start": 107.96, "end": 114.32, "text": " And on this slide, I'm going to use W1, B1, superscript there, to denote that these are"}, {"start": 114.32, "end": 119.36, "text": " the parameters you get if you were to fit a first order polynomial or degree one, D"}, {"start": 119.36, "end": 122.83999999999999, "text": " equals one polynomial."}, {"start": 122.83999999999999, "end": 129.12, "text": " Now you might also consider fitting a second order polynomial or quadratic model."}, {"start": 129.12, "end": 131.64, "text": " So this is the model."}, {"start": 131.64, "end": 139.11999999999998, "text": " And if you were to fit this to your training set, you would get some parameters W2, B2,"}, {"start": 139.11999999999998, "end": 146.16, "text": " and you can then similarly evaluate those parameters on your test set and get J-Test"}, {"start": 146.16, "end": 147.64, "text": " W2, B2."}, {"start": 147.64, "end": 151.0, "text": " And this would give you a sense of how well the second order polynomial does."}, {"start": 151.0, "end": 157.56, "text": " And you can go on to try D equals three, that's a third order or a degree three polynomial"}, {"start": 157.56, "end": 163.56, "text": " that looks like this, and fit parameters, and similarly get J-Test."}, {"start": 163.56, "end": 168.92000000000002, "text": " And you might keep doing this until, say, you try up to a 10th order polynomial and"}, {"start": 168.92000000000002, "end": 173.44, "text": " you end up with J-Test of W10, B10."}, {"start": 173.44, "end": 177.9, "text": " That gives you a sense of how well the 10th order polynomial is doing."}, {"start": 177.9, "end": 183.32, "text": " So one procedure you could try, this turns out not to be the best procedure, but one"}, {"start": 183.32, "end": 190.6, "text": " thing you could try is look at all of these J-Tests and see which one gives you the lowest"}, {"start": 190.6, "end": 191.84, "text": " value."}, {"start": 191.84, "end": 200.07999999999998, "text": " And say you find that J-Test for the fifth order polynomial for W5, B5 turns out to be"}, {"start": 200.07999999999998, "end": 202.16, "text": " the lowest."}, {"start": 202.16, "end": 207.16, "text": " If that's the case, then you might decide that the fifth order polynomial D equals five"}, {"start": 207.16, "end": 211.88, "text": " does best and choose that model for your application."}, {"start": 211.88, "end": 216.2, "text": " And if you want to estimate how well this model performs, one thing you could do, but"}, {"start": 216.2, "end": 222.79999999999998, "text": " this turns out to be a slightly flawed procedure, is to report the test set error J-Test W5,"}, {"start": 222.79999999999998, "end": 224.79999999999998, "text": " B5."}, {"start": 224.79999999999998, "end": 233.24, "text": " The reason this procedure is flawed is J-Test of W5, B5 is likely to be an optimistic estimate"}, {"start": 233.24, "end": 235.28, "text": " of the generalization error."}, {"start": 235.28, "end": 241.72, "text": " In other words, it is likely to be lower than the actual generalization error."}, {"start": 241.72, "end": 247.44, "text": " And the reason is, in the procedure we talked about on the slide, we basically fit one extra"}, {"start": 247.44, "end": 254.76, "text": " parameter, which is D, the degree of polynomial, and we chose this parameter using the test"}, {"start": 254.76, "end": 256.16, "text": " set."}, {"start": 256.16, "end": 262.72, "text": " So on the previous slide, we saw that if you were to fit WB to the training data, then"}, {"start": 262.72, "end": 269.36, "text": " the training data would be an overly optimistic estimate of generalization error."}, {"start": 269.36, "end": 274.08000000000004, "text": " And it turns out too, that if we were to choose the parameter D using the test set, then the"}, {"start": 274.08000000000004, "end": 279.96000000000004, "text": " test set J-Test is now an overly optimistic, that is lower than actual estimates of the"}, {"start": 279.96000000000004, "end": 282.1, "text": " generalization error."}, {"start": 282.1, "end": 287.08000000000004, "text": " So the procedure on this particular slide is flawed and I don't recommend using this."}, {"start": 287.08000000000004, "end": 292.96000000000004, "text": " Instead, if you want to automatically choose a model, such as decide what degree polynomial"}, {"start": 292.96000000000004, "end": 298.96000000000004, "text": " to use, here's how you modify the training and testing procedure in order to carry out"}, {"start": 298.96, "end": 305.08, "text": " model selection, whereby model selection, I mean, choosing amongst different models,"}, {"start": 305.08, "end": 309.59999999999997, "text": " such as these 10 different models that you might contemplate using for your machine learning"}, {"start": 309.59999999999997, "end": 311.84, "text": " application."}, {"start": 311.84, "end": 316.91999999999996, "text": " The way we'll modify the procedure is, instead of splitting your data into just two subsets,"}, {"start": 316.91999999999996, "end": 321.79999999999995, "text": " the training set and the test set, we're going to split your data into three different subsets,"}, {"start": 321.79999999999995, "end": 326.88, "text": " which we're going to call the training set, the cross validation set, and then also the"}, {"start": 326.88, "end": 328.91999999999996, "text": " test set."}, {"start": 328.92, "end": 336.96000000000004, "text": " So using our example from before of these 10 training examples, we might split it into"}, {"start": 336.96000000000004, "end": 342.46000000000004, "text": " putting 60% of the data into the training set."}, {"start": 342.46000000000004, "end": 348.6, "text": " And so the notation we'll use for the training set portion will be the same as before, except"}, {"start": 348.6, "end": 353.68, "text": " that now mtrain, the number of training examples will be 6."}, {"start": 353.68, "end": 360.32, "text": " And we might put 20% of the data into the cross validation set."}, {"start": 360.32, "end": 368.12, "text": " And the notation I'm going to use is xCV of 1, yCV of 1 for the first cross validation"}, {"start": 368.12, "end": 369.12, "text": " example."}, {"start": 369.12, "end": 377.68, "text": " So CV stands for cross validation, all the way down to xCV of mCV and yCV of mCV."}, {"start": 377.68, "end": 383.96, "text": " So here mCV equals 2 in this example is the number of cross validation examples."}, {"start": 383.96, "end": 387.36, "text": " And then finally, we have the test set same as before."}, {"start": 387.36, "end": 398.36, "text": " So x1 through xm tests and y1 through ym tests, where m tests here is equal to 2."}, {"start": 398.36, "end": 400.2, "text": " This is the number of test examples."}, {"start": 400.2, "end": 404.48, "text": " We'll see on the next slide how to use the cross validation set."}, {"start": 404.48, "end": 411.16, "text": " So the way we'll modify the procedure is you've already seen the training set and the test"}, {"start": 411.16, "end": 412.16, "text": " set."}, {"start": 412.16, "end": 418.8, "text": " And we're going to introduce a new subset of the data called the cross validation set."}, {"start": 418.8, "end": 424.32, "text": " The name cross validation refers to that this is an extra data set that we're going to use"}, {"start": 424.32, "end": 430.52000000000004, "text": " to check or cross check the validity or really the accuracy of different models."}, {"start": 430.52, "end": 435.15999999999997, "text": " I don't think it's a great name, but that is what people in machine learning have gotten"}, {"start": 435.15999999999997, "end": 437.91999999999996, "text": " to call this extra data set."}, {"start": 437.91999999999996, "end": 443.56, "text": " You may also hear people call this the validation set for short, just few syllables then cross"}, {"start": 443.56, "end": 444.56, "text": " validation."}, {"start": 444.56, "end": 450.0, "text": " Or in some applications, people also call this the development set means basically the"}, {"start": 450.0, "end": 451.08, "text": " same thing."}, {"start": 451.08, "end": 455.91999999999996, "text": " Or for short, sometimes you hear people call this the dev set, but all of these terms mean"}, {"start": 455.91999999999996, "end": 458.79999999999995, "text": " the same thing as cross validation set."}, {"start": 458.8, "end": 464.2, "text": " I personally use the term dev set the most often because it's the shortest, fastest way"}, {"start": 464.2, "end": 468.52000000000004, "text": " to say it, but cross validation is probably used a little bit more often by machine learning"}, {"start": 468.52000000000004, "end": 470.40000000000003, "text": " practitioners."}, {"start": 470.40000000000003, "end": 475.24, "text": " So armed with these three subsets of the data, training set, cross validation set and test"}, {"start": 475.24, "end": 481.88, "text": " set, you can then compute the training error, the cross validation error and the test error"}, {"start": 481.88, "end": 484.28000000000003, "text": " using these three formulas."}, {"start": 484.28, "end": 488.79999999999995, "text": " Where as usual, none of these terms include the regularization term that is included in"}, {"start": 488.79999999999995, "end": 490.52, "text": " the training objective."}, {"start": 490.52, "end": 494.59999999999997, "text": " And this new term in the middle, the cross validation error is just the average over"}, {"start": 494.59999999999997, "end": 501.35999999999996, "text": " your MCV cross validation examples of the average say squared error."}, {"start": 501.35999999999996, "end": 507.52, "text": " And this term, in addition to being called cross validation error, is also commonly called"}, {"start": 507.52, "end": 513.1999999999999, "text": " the validation error for short or even the development set error or the dev error."}, {"start": 513.2, "end": 518.5600000000001, "text": " Armed with these three measures of learning algorithm performance, this is how you can"}, {"start": 518.5600000000001, "end": 522.2, "text": " then go about carrying out model selection."}, {"start": 522.2, "end": 529.72, "text": " You can with the 10 models, same as earlier on the slide with D equals one, D equals two,"}, {"start": 529.72, "end": 535.8000000000001, "text": " all the way up to a 10th degree or the 10th order polynomial, you can then fit the parameters"}, {"start": 535.8000000000001, "end": 539.2800000000001, "text": " W1, B1."}, {"start": 539.28, "end": 543.9599999999999, "text": " But instead of evaluating this on your test set, you would instead evaluate these parameters"}, {"start": 543.9599999999999, "end": 549.88, "text": " on your cross validation sets and compute JCV of W1, B1."}, {"start": 549.88, "end": 557.8, "text": " And similarly for the second model, you get JCV of W2, B2 and all the way down to JCV"}, {"start": 557.8, "end": 561.72, "text": " of W10, B10."}, {"start": 561.72, "end": 569.1999999999999, "text": " Then in order to choose a model, you would look at which model has the lowest cross validation"}, {"start": 569.2, "end": 570.5600000000001, "text": " error."}, {"start": 570.5600000000001, "end": 578.6400000000001, "text": " And concretely, let's say that JCV of W4, B4 is lowest, then what that means is you"}, {"start": 578.6400000000001, "end": 584.48, "text": " would pick this fourth order polynomial as the model you will use for this application."}, {"start": 584.48, "end": 590.5200000000001, "text": " Finally, if you want to report out an estimate of the generalization error of how well this"}, {"start": 590.5200000000001, "end": 597.2800000000001, "text": " model will do on new data, you would do so using that third subset of your data, the"}, {"start": 597.28, "end": 602.1999999999999, "text": " test set and you report out J test of W4, B4."}, {"start": 602.1999999999999, "end": 608.0799999999999, "text": " And you notice that throughout this entire procedure, you had fit these parameters using"}, {"start": 608.0799999999999, "end": 614.52, "text": " the training set, you then chose the parameter D or chose the degree of polynomial using"}, {"start": 614.52, "end": 616.52, "text": " the cross validation set."}, {"start": 616.52, "end": 621.8399999999999, "text": " And so up until this point, you've not fit any parameters, either W or B or D to the"}, {"start": 621.8399999999999, "end": 623.12, "text": " test set."}, {"start": 623.12, "end": 630.32, "text": " And that's why J test in this example will be a fair estimate of the generalization error"}, {"start": 630.32, "end": 635.42, "text": " of this model that has parameters W4, B4."}, {"start": 635.42, "end": 639.84, "text": " So this gives a better procedure for model selection."}, {"start": 639.84, "end": 645.6, "text": " And it lets you automatically make a decision like what order polynomial to choose for your"}, {"start": 645.6, "end": 647.44, "text": " linear regression model."}, {"start": 647.44, "end": 652.6, "text": " This model selection procedure also works for choosing among other types of models."}, {"start": 652.6, "end": 655.9200000000001, "text": " For example, choosing a neural network architecture."}, {"start": 655.9200000000001, "end": 662.28, "text": " If you are fitting a model for handwritten digit recognition, you might consider three"}, {"start": 662.28, "end": 666.5600000000001, "text": " models like these, maybe even a larger set of models than just three, but here are a"}, {"start": 666.5600000000001, "end": 672.52, "text": " few different neural networks of small, somewhat larger and then even larger."}, {"start": 672.52, "end": 677.36, "text": " To help you decide how many layers should your neural network have and how many hidden"}, {"start": 677.36, "end": 679.6800000000001, "text": " units per layer should you have."}, {"start": 679.68, "end": 688.8, "text": " You can then train all three of these models and end up with parameters W1, B1 for the"}, {"start": 688.8, "end": 696.0799999999999, "text": " first model, W2, B2 for the second model and W3, B3 for the third model."}, {"start": 696.0799999999999, "end": 703.3599999999999, "text": " And you can then evaluate the neural network's performance using JCV using your cross validation"}, {"start": 703.3599999999999, "end": 704.74, "text": " set."}, {"start": 704.74, "end": 711.64, "text": " And with a classification problem, JCV can be the percentage of examples."}, {"start": 711.64, "end": 717.2, "text": " And since this is a classification problem, JCV, the most common choice would be to compute"}, {"start": 717.2, "end": 723.64, "text": " this as a fraction of cross validation examples that the algorithm has misclassified."}, {"start": 723.64, "end": 730.8, "text": " And you would compute this using all three models and then pick the model with the lowest"}, {"start": 730.8, "end": 733.16, "text": " cross validation error."}, {"start": 733.16, "end": 739.3199999999999, "text": " So if in this example, this has the lowest cross validation error, you would then pick"}, {"start": 739.3199999999999, "end": 746.4, "text": " the second neural network and use parameters trained on this model."}, {"start": 746.4, "end": 750.7199999999999, "text": " And finally, if you want to report out an estimate of the generalization error, you"}, {"start": 750.7199999999999, "end": 756.16, "text": " then use the test set to estimate how well the neural network that you just chose will"}, {"start": 756.16, "end": 757.4, "text": " do."}, {"start": 757.4, "end": 764.36, "text": " So in machine learning practice, it's considered best practice to make all the decisions you"}, {"start": 764.36, "end": 769.8, "text": " want to make regarding your learning algorithm, such as how to choose parameters, what degree"}, {"start": 769.8, "end": 775.4399999999999, "text": " polynomial to use, but make decisions only looking at the training set and cross validation"}, {"start": 775.4399999999999, "end": 780.9599999999999, "text": " set and to not use the test set at all to make decisions about your model."}, {"start": 780.9599999999999, "end": 786.74, "text": " And only after you've made all those decisions, then finally take the model you have designed"}, {"start": 786.74, "end": 789.72, "text": " and evaluated on your test set."}, {"start": 789.72, "end": 795.96, "text": " And that procedure ensures that you haven't accidentally fit anything to the test set,"}, {"start": 795.96, "end": 800.8, "text": " so that your test set becomes still a fair and not overly optimistic estimate of the"}, {"start": 800.8, "end": 803.32, "text": " generalization error of your algorithm."}, {"start": 803.32, "end": 808.44, "text": " So it's considered best practice in machine learning that if you have to make decisions"}, {"start": 808.44, "end": 813.32, "text": " about your model, such as fitting parameters or choosing the model architecture, such as"}, {"start": 813.32, "end": 818.6, "text": " neural network architecture or degree of polynomial if you're fitting linear regression, to make"}, {"start": 818.6, "end": 825.24, "text": " all those decisions only using your training set and your cross validation set and to not"}, {"start": 825.24, "end": 830.5600000000001, "text": " look at the test set at all while you're still making decisions regarding your learning algorithm."}, {"start": 830.5600000000001, "end": 836.08, "text": " And it's only after you've come up with one model that's your final model to only then"}, {"start": 836.08, "end": 838.8000000000001, "text": " evaluate it on the test set."}, {"start": 838.8, "end": 843.4, "text": " And because you haven't made any decisions using the test set, that ensures that your"}, {"start": 843.4, "end": 850.24, "text": " test set is a fair and not overly optimistic estimate of how well your model will generalize"}, {"start": 850.24, "end": 851.56, "text": " to new data."}, {"start": 851.56, "end": 853.52, "text": " So that's model selection."}, {"start": 853.52, "end": 856.76, "text": " And this is actually a very widely used procedure."}, {"start": 856.76, "end": 862.0, "text": " I use this all the time to automatically choose what model to use for a given machine learning"}, {"start": 862.0, "end": 864.12, "text": " application."}, {"start": 864.12, "end": 870.28, "text": " Now earlier this week, I mentioned running diagnostics to decide how to improve the performance"}, {"start": 870.28, "end": 872.14, "text": " of a learning algorithm."}, {"start": 872.14, "end": 876.76, "text": " Now that you have a way to evaluate learning algorithms and even automatically choose a"}, {"start": 876.76, "end": 881.28, "text": " model, let's dive more deeply into examples of some diagnostics."}, {"start": 881.28, "end": 885.84, "text": " The most powerful diagnostic that I know of and that I use for a lot of machine learning"}, {"start": 885.84, "end": 889.1800000000001, "text": " applications is one called bias and variance."}, {"start": 889.18, "end": 894.28, "text": " Let's take a look at what that means in the next video."}]