6.2 Evaluating and choosing models | Evaluating a model  -[Machine Learning | Andrew Ng]
 Let's say you've trained a machine learning model. How do you evaluate that model's performance? You find that having a systematic way to evaluate performance will also help paint a clearer path for how to then improve this performance. So let's take a look at how to evaluate a model. Let's take the example of learning to predict housing prices as a function of the size. Let's say you've trained a model to predict housing prices as a function of the size x. And for the model that is a fourth order polynomial, so it features x, x squared, x cubed, and x to the fourth. Because we fit a fourth order polynomial to a training set with five data points, this fits the training data really well. But we don't like this model very much because even though the model fits the training data well, we think it will fail to generalize to new examples that aren't in the training set. So when you are predicting prices just a single feature the size of the house, you could plot the model like this and we could see that the curve is very wiggly. So we know this probably isn't a good model. But if you were fitting this model with even more features, say we had x1 the size of the house, number of bedrooms, the number of floors of the house, also the age of the home and years, then it becomes much harder to plot f because f is now a function of x1 through x4. And how do you plot a four dimensional function? So in order to tell if your model is doing well, especially for applications where you have more than one or two features, which makes it difficult to plot f of x, we need some more systematic way to evaluate how well your model is doing. Here's a technique that you can use. If you have a training set, and this is a small training set with just 10 examples listed here, rather than taking all your data to train the parameters w and p of the model, you can instead split the training set into two subsets. I'm going to draw a line here. And let's put 70% of the data into the first part. And I'm going to call that the training set. And the second part of the data, let's say 30% of the data, I'm going to put into a test set. And what we're going to do is train the models parameters on the training sets on this first 70% or so of the data, and then we'll test this performance on this test set. In notation, I'm going to use x1, y1, same as before, to denote the training examples through xm, ym, except that now to make explicit. So in this little example, we would have seven training examples. And to introduce one new piece of notation, I'm going to use m subscript train. m train is a number of training examples, which in this small data set is seven. So the subscript train just emphasizes if we're looking at the training set portion of the data. And for the test sets, I'm going to use the notation x1 subscript test, y1 subscript test to denote the first test example. And this goes all the way to xm test, subscript test, ym test, subscript test. And m test is the number of test examples, which in this case is three. And it's not uncommon to split your data set according to maybe a 70-30 split or 80-20 split with most of your data going into the training set and then a smaller fraction going into the test set. So in order to train a model and evaluate it, this is what it would look like if you're using linear regression with a squared error cost. Start off by fitting the parameters by minimizing the cost function j of wb. So this is a usual cost function. Minimize over wb of this squared error cost plus regularization term, longer over 2m times sum of the wj squared. And then to tell how well this model is doing, you would compute j test of wb, which is equal to the average error on the test set. And that's just equal to 1 over 2 times m test. That's the number of test examples. And then of sum over all the examples from i equals 1 to the number of test examples of the squared error on each of the test examples like so. So it's a prediction on the i-th test example input minus the actual price of the house on the i-th test example squared. And notice that the test error formula j test, it does not include that regularization term. And this will give you a sense of how well your learning algorithm is doing. One other quantity that's often useful to compute as well is the training error, which is a measure of how well your learning algorithm is doing on the training set. So let me define j train of wb to be equal to the average over the training sets 1 over 2m or 1 over 2m subscript train of sum over your training sets of this squared error term. And once again, this does not include the regularization term, unlike the cost function that you were minimizing to fit the parameters. So in a model like what we saw earlier in this video, j train of wb will be low because the average error on your training examples will be zero or very close to zero. So j train will be very close to zero. But if you had a few additional examples in your test set that the algorithm had not trained on then those test examples might look like these. And there's a large gap between what the algorithm is predicting as the estimated housing price and the actual value of those housing prices. And so j tests will be high. So seeing that j test is high on this model gives you a way to realize that even though it does great on the training set is actually not so good at generalizing to new examples to new data points that were not in the training set. So that was regression with squared error costs. Now let's take a look at how you'd apply this procedure to a classification problem. For example, if you were classifying between handwritten digits that are either zero or one. So same as before, you fit the parameters by minimizing the cost function to find the parameters wb. For example, if you were training logistic regression, then this would be the cost function j of wb where this is the usual logistic loss function and then plus also the regularization term and to compute the test error. J test is then the average over your test examples. That's that 30% of the data that wasn't in the training set of the logistic loss on your test set. And the training error you could also compute using this formula is the average logistic loss on your training data that the algorithm was using to minimize the cost function j of wb. Well what I describe here will work okay for figuring out if your learning algorithm is doing well. I've seen how well it's doing in terms of test error. When applying machine learning to classification problems, there's actually one other definition of J test and J train that is maybe even more commonly used, which is instead of using the logistic loss to compute the test error and the training error to instead measure what's the fraction of the test set and the fraction of the training set that the algorithm has misclassified. So specifically on the test set, you can have the algorithm make a prediction one or zero on every test example. So recall y hat, we would predict as one if f of x is greater than or equal to 0.5 and zero if it's less than 0.5. And you can then count up in the test set the fraction of examples where y hat is not equal to the actual ground truth label y in the test set. So concretely, if you were classifying handwritten digits zero or one by new classification toss, then J test would be the fraction of that test set where zero was classified as one or one classified as zero. And similarly J train is a fraction of the training set that has been misclassified. Taking a data set and splitting it into a training set and a separate test set gives you a way to systematically evaluate how well your learning algorithm is doing. By computing both J test and J train, you can now measure how it's doing on the test set and on the training set. This procedure is one step to what you being able to automatically choose what model to use for a given machine learning application. For example, if you're trying to predict housing prices, should you fit a straight line to your data or fit a second order polynomial or third order or fourth order polynomial? It turns out that with one further refinement to the idea you saw in this video, you'll be able to have an algorithm help you to automatically make that type of decision well. Let's take a look at how to do that in the next video.
[{"start": 0.0, "end": 4.48, "text": " Let's say you've trained a machine learning model."}, {"start": 4.48, "end": 6.88, "text": " How do you evaluate that model's performance?"}, {"start": 6.88, "end": 12.040000000000001, "text": " You find that having a systematic way to evaluate performance will also help paint a clearer"}, {"start": 12.040000000000001, "end": 14.84, "text": " path for how to then improve this performance."}, {"start": 14.84, "end": 17.92, "text": " So let's take a look at how to evaluate a model."}, {"start": 17.92, "end": 24.96, "text": " Let's take the example of learning to predict housing prices as a function of the size."}, {"start": 24.96, "end": 31.04, "text": " Let's say you've trained a model to predict housing prices as a function of the size x."}, {"start": 31.04, "end": 38.32, "text": " And for the model that is a fourth order polynomial, so it features x, x squared, x cubed, and"}, {"start": 38.32, "end": 40.52, "text": " x to the fourth."}, {"start": 40.52, "end": 46.68, "text": " Because we fit a fourth order polynomial to a training set with five data points, this"}, {"start": 46.68, "end": 50.36, "text": " fits the training data really well."}, {"start": 50.36, "end": 57.32, "text": " But we don't like this model very much because even though the model fits the training data"}, {"start": 57.32, "end": 61.879999999999995, "text": " well, we think it will fail to generalize to new examples that aren't in the training"}, {"start": 61.879999999999995, "end": 63.96, "text": " set."}, {"start": 63.96, "end": 69.88, "text": " So when you are predicting prices just a single feature the size of the house, you could plot"}, {"start": 69.88, "end": 74.56, "text": " the model like this and we could see that the curve is very wiggly."}, {"start": 74.56, "end": 77.84, "text": " So we know this probably isn't a good model."}, {"start": 77.84, "end": 82.84, "text": " But if you were fitting this model with even more features, say we had x1 the size of the"}, {"start": 82.84, "end": 87.28, "text": " house, number of bedrooms, the number of floors of the house, also the age of the home and"}, {"start": 87.28, "end": 95.80000000000001, "text": " years, then it becomes much harder to plot f because f is now a function of x1 through"}, {"start": 95.80000000000001, "end": 97.08000000000001, "text": " x4."}, {"start": 97.08000000000001, "end": 102.96000000000001, "text": " And how do you plot a four dimensional function?"}, {"start": 102.96, "end": 108.03999999999999, "text": " So in order to tell if your model is doing well, especially for applications where you"}, {"start": 108.03999999999999, "end": 113.72, "text": " have more than one or two features, which makes it difficult to plot f of x, we need"}, {"start": 113.72, "end": 118.6, "text": " some more systematic way to evaluate how well your model is doing."}, {"start": 118.6, "end": 121.56, "text": " Here's a technique that you can use."}, {"start": 121.56, "end": 125.91999999999999, "text": " If you have a training set, and this is a small training set with just 10 examples listed"}, {"start": 125.91999999999999, "end": 132.54, "text": " here, rather than taking all your data to train the parameters w and p of the model,"}, {"start": 132.54, "end": 136.07999999999998, "text": " you can instead split the training set into two subsets."}, {"start": 136.07999999999998, "end": 138.35999999999999, "text": " I'm going to draw a line here."}, {"start": 138.35999999999999, "end": 144.48, "text": " And let's put 70% of the data into the first part."}, {"start": 144.48, "end": 147.76, "text": " And I'm going to call that the training set."}, {"start": 147.76, "end": 153.88, "text": " And the second part of the data, let's say 30% of the data, I'm going to put into a test"}, {"start": 153.88, "end": 155.68, "text": " set."}, {"start": 155.68, "end": 160.92, "text": " And what we're going to do is train the models parameters on the training sets on this first"}, {"start": 160.92, "end": 168.07999999999998, "text": " 70% or so of the data, and then we'll test this performance on this test set."}, {"start": 168.07999999999998, "end": 177.16, "text": " In notation, I'm going to use x1, y1, same as before, to denote the training examples"}, {"start": 177.16, "end": 186.48, "text": " through xm, ym, except that now to make explicit."}, {"start": 186.48, "end": 190.88, "text": " So in this little example, we would have seven training examples."}, {"start": 190.88, "end": 197.35999999999999, "text": " And to introduce one new piece of notation, I'm going to use m subscript train."}, {"start": 197.35999999999999, "end": 203.22, "text": " m train is a number of training examples, which in this small data set is seven."}, {"start": 203.22, "end": 208.16, "text": " So the subscript train just emphasizes if we're looking at the training set portion"}, {"start": 208.16, "end": 209.95999999999998, "text": " of the data."}, {"start": 209.96, "end": 218.32, "text": " And for the test sets, I'm going to use the notation x1 subscript test, y1 subscript test"}, {"start": 218.32, "end": 220.96, "text": " to denote the first test example."}, {"start": 220.96, "end": 230.96, "text": " And this goes all the way to xm test, subscript test, ym test, subscript test."}, {"start": 230.96, "end": 236.20000000000002, "text": " And m test is the number of test examples, which in this case is three."}, {"start": 236.2, "end": 243.6, "text": " And it's not uncommon to split your data set according to maybe a 70-30 split or 80-20"}, {"start": 243.6, "end": 248.39999999999998, "text": " split with most of your data going into the training set and then a smaller fraction going"}, {"start": 248.39999999999998, "end": 250.67999999999998, "text": " into the test set."}, {"start": 250.67999999999998, "end": 256.82, "text": " So in order to train a model and evaluate it, this is what it would look like if you're"}, {"start": 256.82, "end": 261.4, "text": " using linear regression with a squared error cost."}, {"start": 261.4, "end": 266.44, "text": " Start off by fitting the parameters by minimizing the cost function j of wb."}, {"start": 266.44, "end": 269.23999999999995, "text": " So this is a usual cost function."}, {"start": 269.23999999999995, "end": 277.03999999999996, "text": " Minimize over wb of this squared error cost plus regularization term, longer over 2m times"}, {"start": 277.03999999999996, "end": 281.28, "text": " sum of the wj squared."}, {"start": 281.28, "end": 289.79999999999995, "text": " And then to tell how well this model is doing, you would compute j test of wb, which is equal"}, {"start": 289.8, "end": 293.16, "text": " to the average error on the test set."}, {"start": 293.16, "end": 297.64, "text": " And that's just equal to 1 over 2 times m test."}, {"start": 297.64, "end": 299.84000000000003, "text": " That's the number of test examples."}, {"start": 299.84000000000003, "end": 305.68, "text": " And then of sum over all the examples from i equals 1 to the number of test examples"}, {"start": 305.68, "end": 310.40000000000003, "text": " of the squared error on each of the test examples like so."}, {"start": 310.40000000000003, "end": 318.72, "text": " So it's a prediction on the i-th test example input minus the actual price of the house"}, {"start": 318.72, "end": 323.32000000000005, "text": " on the i-th test example squared."}, {"start": 323.32000000000005, "end": 330.92, "text": " And notice that the test error formula j test, it does not include that regularization term."}, {"start": 330.92, "end": 335.8, "text": " And this will give you a sense of how well your learning algorithm is doing."}, {"start": 335.8, "end": 340.92, "text": " One other quantity that's often useful to compute as well is the training error, which"}, {"start": 340.92, "end": 346.32000000000005, "text": " is a measure of how well your learning algorithm is doing on the training set."}, {"start": 346.32, "end": 352.84, "text": " So let me define j train of wb to be equal to the average over the training sets 1 over"}, {"start": 352.84, "end": 360.71999999999997, "text": " 2m or 1 over 2m subscript train of sum over your training sets of this squared error term."}, {"start": 360.71999999999997, "end": 365.88, "text": " And once again, this does not include the regularization term, unlike the cost function"}, {"start": 365.88, "end": 369.2, "text": " that you were minimizing to fit the parameters."}, {"start": 369.2, "end": 377.64, "text": " So in a model like what we saw earlier in this video, j train of wb will be low because"}, {"start": 377.64, "end": 384.48, "text": " the average error on your training examples will be zero or very close to zero."}, {"start": 384.48, "end": 387.36, "text": " So j train will be very close to zero."}, {"start": 387.36, "end": 391.96, "text": " But if you had a few additional examples in your test set that the algorithm had not trained"}, {"start": 391.96, "end": 396.96, "text": " on then those test examples might look like these."}, {"start": 396.96, "end": 401.91999999999996, "text": " And there's a large gap between what the algorithm is predicting as the estimated housing price"}, {"start": 401.91999999999996, "end": 405.28, "text": " and the actual value of those housing prices."}, {"start": 405.28, "end": 408.34, "text": " And so j tests will be high."}, {"start": 408.34, "end": 415.32, "text": " So seeing that j test is high on this model gives you a way to realize that even though"}, {"start": 415.32, "end": 420.9, "text": " it does great on the training set is actually not so good at generalizing to new examples"}, {"start": 420.9, "end": 424.96, "text": " to new data points that were not in the training set."}, {"start": 424.96, "end": 428.71999999999997, "text": " So that was regression with squared error costs."}, {"start": 428.71999999999997, "end": 433.52, "text": " Now let's take a look at how you'd apply this procedure to a classification problem."}, {"start": 433.52, "end": 438.47999999999996, "text": " For example, if you were classifying between handwritten digits that are either zero or"}, {"start": 438.47999999999996, "end": 439.79999999999995, "text": " one."}, {"start": 439.79999999999995, "end": 444.32, "text": " So same as before, you fit the parameters by minimizing the cost function to find the"}, {"start": 444.32, "end": 446.12, "text": " parameters wb."}, {"start": 446.12, "end": 451.08, "text": " For example, if you were training logistic regression, then this would be the cost function"}, {"start": 451.08, "end": 460.44, "text": " j of wb where this is the usual logistic loss function and then plus also the regularization"}, {"start": 460.44, "end": 464.44, "text": " term and to compute the test error."}, {"start": 464.44, "end": 470.71999999999997, "text": " J test is then the average over your test examples."}, {"start": 470.71999999999997, "end": 477.02, "text": " That's that 30% of the data that wasn't in the training set of the logistic loss on your"}, {"start": 477.02, "end": 479.12, "text": " test set."}, {"start": 479.12, "end": 484.48, "text": " And the training error you could also compute using this formula is the average logistic"}, {"start": 484.48, "end": 489.96, "text": " loss on your training data that the algorithm was using to minimize the cost function j"}, {"start": 489.96, "end": 491.96, "text": " of wb."}, {"start": 491.96, "end": 497.4, "text": " Well what I describe here will work okay for figuring out if your learning algorithm is"}, {"start": 497.4, "end": 498.4, "text": " doing well."}, {"start": 498.4, "end": 501.68, "text": " I've seen how well it's doing in terms of test error."}, {"start": 501.68, "end": 507.0, "text": " When applying machine learning to classification problems, there's actually one other definition"}, {"start": 507.0, "end": 513.68, "text": " of J test and J train that is maybe even more commonly used, which is instead of using the"}, {"start": 513.68, "end": 519.44, "text": " logistic loss to compute the test error and the training error to instead measure what's"}, {"start": 519.44, "end": 524.88, "text": " the fraction of the test set and the fraction of the training set that the algorithm has"}, {"start": 524.88, "end": 526.68, "text": " misclassified."}, {"start": 526.68, "end": 535.92, "text": " So specifically on the test set, you can have the algorithm make a prediction one or zero"}, {"start": 535.92, "end": 537.68, "text": " on every test example."}, {"start": 537.68, "end": 544.4, "text": " So recall y hat, we would predict as one if f of x is greater than or equal to 0.5 and"}, {"start": 544.4, "end": 547.56, "text": " zero if it's less than 0.5."}, {"start": 547.56, "end": 553.68, "text": " And you can then count up in the test set the fraction of examples where y hat is not"}, {"start": 553.68, "end": 558.52, "text": " equal to the actual ground truth label y in the test set."}, {"start": 558.52, "end": 565.64, "text": " So concretely, if you were classifying handwritten digits zero or one by new classification toss,"}, {"start": 565.64, "end": 570.88, "text": " then J test would be the fraction of that test set where zero was classified as one"}, {"start": 570.88, "end": 572.92, "text": " or one classified as zero."}, {"start": 572.92, "end": 578.8, "text": " And similarly J train is a fraction of the training set that has been misclassified."}, {"start": 578.8, "end": 583.6, "text": " Taking a data set and splitting it into a training set and a separate test set gives"}, {"start": 583.6, "end": 587.8, "text": " you a way to systematically evaluate how well your learning algorithm is doing."}, {"start": 587.8, "end": 592.92, "text": " By computing both J test and J train, you can now measure how it's doing on the test"}, {"start": 592.92, "end": 595.3199999999999, "text": " set and on the training set."}, {"start": 595.32, "end": 600.5600000000001, "text": " This procedure is one step to what you being able to automatically choose what model to"}, {"start": 600.5600000000001, "end": 603.8000000000001, "text": " use for a given machine learning application."}, {"start": 603.8000000000001, "end": 608.4000000000001, "text": " For example, if you're trying to predict housing prices, should you fit a straight line to"}, {"start": 608.4000000000001, "end": 613.24, "text": " your data or fit a second order polynomial or third order or fourth order polynomial?"}, {"start": 613.24, "end": 617.48, "text": " It turns out that with one further refinement to the idea you saw in this video, you'll"}, {"start": 617.48, "end": 622.08, "text": " be able to have an algorithm help you to automatically make that type of decision well."}, {"start": 622.08, "end": 626.0, "text": " Let's take a look at how to do that in the next video."}]