6.16 Skewed datasets (optional) | Error metrics for skewed datasets -[Machine Learning | Andrew Ng]
 If you're working on a machine learning application where the ratio of positive to negative examples is very skewed, very far from 50-50, then it turns out that the usual error metrics like accuracy don't work that well. Let's start with an example. Let's say you're training a binary classifier to detect a rare disease in patients based on lab tests or based on other data from the patients. So y is equal to 1 if the disease is present and y is equal to 0 otherwise. And suppose you find that you've achieved 1% error on the test set, so you have a 99% correct diagnosis. This seems like a great outcome, right? But it turns out that if this is a rare disease, so y is equal to 1 very rarely, then this may not be as impressive as it sounds. Specifically, if it is a rare disease and if only 0.5% of the patients in your population have the disease, then if instead you wrote a program that just said print y equals 0, it predicts y equals 0 all the time, this very simple even non-learning algorithm, because it just says y equals 0 all the time, this will actually have 99.5% accuracy or 0.5% error. So this really dumb algorithm outperforms your learning algorithm, which had 1% error, much worse than 0.5% error. But I think a piece of software that just prints y equals 0 is not a very useful diagnostic tool. What this really means is that you can't tell if getting 1% error is actually a good result or a bad result. And in particular, if you have one algorithm that achieves 99.5% accuracy, different one that achieves 99.2% accuracy, different one that achieves 99.6% accuracy, it's difficult to know which of these is actually the best algorithm. Because if you have an algorithm that achieves 0.5% error and a different one that achieves 1.2% error, it's difficult to know which of these is the best algorithm, because the one with the lowest error may be a not particularly useful prediction like this that always predicts y equals 0 and never ever diagnose any patient as having this disease. And quite possibly an algorithm that has 1% error, but that at least diagnoses some patients as having the disease could be more useful than just printing y equals 0 all the time. When working on problems with skewed datasets, we usually use a different error metric rather than just classification error to figure out how well your learning algorithm is doing. In particular, a common pair of error metrics are precision and recall, which we'll define on the slide. In this example, y equals 1 will be the rare class such as a rare disease that we may want to detect. And in particular, to evaluate a learning algorithm's performance with one rare class, it'll be useful to construct what's called a confusion matrix, which is a 2x2 matrix or a 2x2 table that looks like this. On the axis on top, I'm going to write the actual class, which could be 1 or 0. And on the vertical axis, I'm going to write the predicted class, which is what did your learning algorithm predict on a given example, 1 or 0. To evaluate your algorithm's performance on the cross-validation set or the test set, say, we would then count up how many examples was the actual class 1 and the predicted class 1. So here you have 100 cross-validation examples, and on 15 of them, the learning algorithm had predicted 1 and the actual label was also 1. And over here, you would count up the number of examples in, say, your cross-validation set where the actual class was 0 and your algorithm predicted 1. So maybe you have five examples there. And here, predicted class 0, actual class 1. So you have 10 examples and, let's say, 70 examples with predicted class 0 and actual class 0. In this example, the skew isn't as extreme as what I had on the previous slide because in these 100 examples in your cross-validation set, we have a total of 25 examples where the actual class was 1 and 75 where the actual class was 0 by adding up these numbers vertically. And you notice also that I'm using different colors to indicate these four cells in the table. I'm actually going to give names to these four cells. When the actual class is 1 and the predicted class is 1, we're going to call that a true positive. Because you predicted positive and it was true, there's a positive example. In this cell on the lower right, where the actual class is 0 and the predicted class is 0, we'll call that a true negative because you predicted negative and it was true. It really was a negative example. This cell on the upper right is called a false positive because the algorithm predicted positive but it was false. It's not actually positive. So this is called a false positive. And this cell is called the number of false negatives because the algorithm predicted 0 but it was false. It wasn't actually negative. The actual label, the actual class was 1. Having divided the classifications into these four cells, two common metrics you might compute are then the precision and recall. Here's what they mean. The precision of learning algorithm computes of all the patients where we predicted y is equal to 1. What fraction actually has the rare disease? In other words, precision is defined as the number of true positives divided by the number classified as positive. In other words, of all the examples you predicted as positive, what fraction do we actually get right? Another way to write this formula would be true positives divided by true positives plus false positives because it is by summing this cell and this cell that you end up with the total number that was predicted as positive. In this example, the numerator true positives would be 15 and divided by 15 plus 5 and so that's 15 over 20 or three quarters or 0.75. So we say that this algorithm has a precision of 75% because of all the things that predicted as positive, of all the patients that it thought has this rare disease, it was right 75% of the time. The second metric that is useful to compute is recall. And recall asks, of all the patients that actually have the rare disease, what fraction do we correctly detect as having it? So recall is defined as the number of true positives divided by the number of actual positives. And alternatively, we can write that as number of true positives divided by the number of actual positives is, well, it's this cell plus this cell. So it's actually the number of true positives plus the number of false negatives because it's by summing up this upper left cell and this lower left cell that you get the number of actual positive examples. And so in our example, this would be 15 divided by 15 plus 10, which is 15 over 25, which is 0.6 or 60%. So this learning algorithm would have 0.75 precision and 0.60 recall. And you notice that this will help you detect if a learning algorithm is just printing y equals 0 all the time, because if it predicts 0 all the time, then the numerator of both of these quantities will be 0. It has no true positives. And the recall metric in particular helps you detect if the learning algorithm is predicting 0 all the time. Because if your learning algorithm just prints y equals 0, then the number of true positives will be 0 because it never predicts positive. And so the recall will be equal to 0 divided by the number of actual positives, which is equal to 0. And in general, a learning algorithm with either 0 precision or 0 recall is not a useful algorithm. But just as a side note, if an algorithm actually predicts 0 all the time, precision actually becomes undefined because it's actually 0 over 0. But in practice, if an algorithm doesn't predict even a single positive, we'll just say that precision is also equal to 0. But computing precision and recall makes it easier for you to spot if a learning algorithm. But we'll find that computing both precision and recall makes it easier to spot if an algorithm is both reasonably accurate in that when it says a patient has a disease, there's a good chance the patient has a disease, such as 0.75 chance in this example. And also making sure that of all the patients that have the disease is hoping to diagnose a reasonable fraction of them, such as here is finding 60% of them. So when you have a rare class looking at precision and recall and making sure that both numbers are decently high, that hopefully helps reassure you that your learning algorithm is actually useful. And the term recall was motivated by this observation that if you have a group of patients or a population of patients, then recall measures of all the patients that have the disease, how many would you have accurately diagnosed as having it. So when you have skewed classes or a rare class that you want to detect, precision and recall helps you tell if your learning algorithms is making good predictions or useful predictions. Now that we have these metrics for telling how well your learning algorithm is doing, in the next video, let's take a look at how to trade off between precision and recall to try to optimize the performance of your learning algorithm.
[{"start": 0.0, "end": 7.5600000000000005, "text": " If you're working on a machine learning application where the ratio of positive to negative examples"}, {"start": 7.5600000000000005, "end": 13.620000000000001, "text": " is very skewed, very far from 50-50, then it turns out that the usual error metrics like"}, {"start": 13.620000000000001, "end": 16.0, "text": " accuracy don't work that well."}, {"start": 16.0, "end": 18.48, "text": " Let's start with an example."}, {"start": 18.48, "end": 24.64, "text": " Let's say you're training a binary classifier to detect a rare disease in patients based"}, {"start": 24.64, "end": 29.86, "text": " on lab tests or based on other data from the patients."}, {"start": 29.86, "end": 38.28, "text": " So y is equal to 1 if the disease is present and y is equal to 0 otherwise."}, {"start": 38.28, "end": 44.36, "text": " And suppose you find that you've achieved 1% error on the test set, so you have a 99%"}, {"start": 44.36, "end": 45.36, "text": " correct diagnosis."}, {"start": 45.36, "end": 48.519999999999996, "text": " This seems like a great outcome, right?"}, {"start": 48.519999999999996, "end": 55.34, "text": " But it turns out that if this is a rare disease, so y is equal to 1 very rarely, then this"}, {"start": 55.34, "end": 58.44, "text": " may not be as impressive as it sounds."}, {"start": 58.44, "end": 65.6, "text": " Specifically, if it is a rare disease and if only 0.5% of the patients in your population"}, {"start": 65.6, "end": 72.56, "text": " have the disease, then if instead you wrote a program that just said print y equals 0,"}, {"start": 72.56, "end": 78.52, "text": " it predicts y equals 0 all the time, this very simple even non-learning algorithm, because"}, {"start": 78.52, "end": 86.92, "text": " it just says y equals 0 all the time, this will actually have 99.5% accuracy or 0.5%"}, {"start": 86.92, "end": 88.8, "text": " error."}, {"start": 88.8, "end": 94.36, "text": " So this really dumb algorithm outperforms your learning algorithm, which had 1% error,"}, {"start": 94.36, "end": 98.08, "text": " much worse than 0.5% error."}, {"start": 98.08, "end": 104.6, "text": " But I think a piece of software that just prints y equals 0 is not a very useful diagnostic"}, {"start": 104.6, "end": 105.78, "text": " tool."}, {"start": 105.78, "end": 111.72, "text": " What this really means is that you can't tell if getting 1% error is actually a good result"}, {"start": 111.72, "end": 113.92, "text": " or a bad result."}, {"start": 113.92, "end": 121.52, "text": " And in particular, if you have one algorithm that achieves 99.5% accuracy, different one"}, {"start": 121.52, "end": 129.52, "text": " that achieves 99.2% accuracy, different one that achieves 99.6% accuracy, it's difficult"}, {"start": 129.52, "end": 134.12, "text": " to know which of these is actually the best algorithm."}, {"start": 134.12, "end": 140.96, "text": " Because if you have an algorithm that achieves 0.5% error and a different one that achieves"}, {"start": 140.96, "end": 150.04000000000002, "text": " 1.2% error, it's difficult to know which of these is the best algorithm, because the one"}, {"start": 150.04000000000002, "end": 156.16, "text": " with the lowest error may be a not particularly useful prediction like this that always predicts"}, {"start": 156.16, "end": 161.32, "text": " y equals 0 and never ever diagnose any patient as having this disease."}, {"start": 161.32, "end": 167.04000000000002, "text": " And quite possibly an algorithm that has 1% error, but that at least diagnoses some patients"}, {"start": 167.04, "end": 173.64, "text": " as having the disease could be more useful than just printing y equals 0 all the time."}, {"start": 173.64, "end": 180.72, "text": " When working on problems with skewed datasets, we usually use a different error metric rather"}, {"start": 180.72, "end": 186.04, "text": " than just classification error to figure out how well your learning algorithm is doing."}, {"start": 186.04, "end": 192.82, "text": " In particular, a common pair of error metrics are precision and recall, which we'll define"}, {"start": 192.82, "end": 193.82, "text": " on the slide."}, {"start": 193.82, "end": 200.68, "text": " In this example, y equals 1 will be the rare class such as a rare disease that we may want"}, {"start": 200.68, "end": 201.68, "text": " to detect."}, {"start": 201.68, "end": 208.16, "text": " And in particular, to evaluate a learning algorithm's performance with one rare class,"}, {"start": 208.16, "end": 215.07999999999998, "text": " it'll be useful to construct what's called a confusion matrix, which is a 2x2 matrix"}, {"start": 215.07999999999998, "end": 218.6, "text": " or a 2x2 table that looks like this."}, {"start": 218.6, "end": 224.6, "text": " On the axis on top, I'm going to write the actual class, which could be 1 or 0."}, {"start": 224.6, "end": 229.6, "text": " And on the vertical axis, I'm going to write the predicted class, which is what did your"}, {"start": 229.6, "end": 234.48, "text": " learning algorithm predict on a given example, 1 or 0."}, {"start": 234.48, "end": 239.76, "text": " To evaluate your algorithm's performance on the cross-validation set or the test set,"}, {"start": 239.76, "end": 246.24, "text": " say, we would then count up how many examples was the actual class 1 and the predicted class"}, {"start": 246.24, "end": 247.24, "text": " 1."}, {"start": 247.24, "end": 253.4, "text": " So here you have 100 cross-validation examples, and on 15 of them, the learning algorithm"}, {"start": 253.4, "end": 258.64, "text": " had predicted 1 and the actual label was also 1."}, {"start": 258.64, "end": 263.08, "text": " And over here, you would count up the number of examples in, say, your cross-validation"}, {"start": 263.08, "end": 268.48, "text": " set where the actual class was 0 and your algorithm predicted 1."}, {"start": 268.48, "end": 271.44, "text": " So maybe you have five examples there."}, {"start": 271.44, "end": 274.52, "text": " And here, predicted class 0, actual class 1."}, {"start": 274.52, "end": 279.91999999999996, "text": " So you have 10 examples and, let's say, 70 examples with predicted class 0 and actual"}, {"start": 279.91999999999996, "end": 282.03999999999996, "text": " class 0."}, {"start": 282.03999999999996, "end": 289.28, "text": " In this example, the skew isn't as extreme as what I had on the previous slide because"}, {"start": 289.28, "end": 297.35999999999996, "text": " in these 100 examples in your cross-validation set, we have a total of 25 examples where"}, {"start": 297.36, "end": 307.28000000000003, "text": " the actual class was 1 and 75 where the actual class was 0 by adding up these numbers vertically."}, {"start": 307.28000000000003, "end": 312.52000000000004, "text": " And you notice also that I'm using different colors to indicate these four cells in the"}, {"start": 312.52000000000004, "end": 313.52000000000004, "text": " table."}, {"start": 313.52000000000004, "end": 317.12, "text": " I'm actually going to give names to these four cells."}, {"start": 317.12, "end": 321.72, "text": " When the actual class is 1 and the predicted class is 1, we're going to call that a true"}, {"start": 321.72, "end": 323.86, "text": " positive."}, {"start": 323.86, "end": 328.2, "text": " Because you predicted positive and it was true, there's a positive example."}, {"start": 328.2, "end": 332.44, "text": " In this cell on the lower right, where the actual class is 0 and the predicted class"}, {"start": 332.44, "end": 338.28000000000003, "text": " is 0, we'll call that a true negative because you predicted negative and it was true."}, {"start": 338.28000000000003, "end": 341.48, "text": " It really was a negative example."}, {"start": 341.48, "end": 350.72, "text": " This cell on the upper right is called a false positive because the algorithm predicted positive"}, {"start": 350.72, "end": 351.72, "text": " but it was false."}, {"start": 351.72, "end": 353.08000000000004, "text": " It's not actually positive."}, {"start": 353.08, "end": 355.91999999999996, "text": " So this is called a false positive."}, {"start": 355.91999999999996, "end": 361.52, "text": " And this cell is called the number of false negatives because the algorithm predicted"}, {"start": 361.52, "end": 362.52, "text": " 0 but it was false."}, {"start": 362.52, "end": 364.2, "text": " It wasn't actually negative."}, {"start": 364.2, "end": 368.8, "text": " The actual label, the actual class was 1."}, {"start": 368.8, "end": 375.12, "text": " Having divided the classifications into these four cells, two common metrics you might compute"}, {"start": 375.12, "end": 378.79999999999995, "text": " are then the precision and recall."}, {"start": 378.79999999999995, "end": 380.59999999999997, "text": " Here's what they mean."}, {"start": 380.6, "end": 386.08000000000004, "text": " The precision of learning algorithm computes of all the patients where we predicted y is"}, {"start": 386.08000000000004, "end": 387.32000000000005, "text": " equal to 1."}, {"start": 387.32000000000005, "end": 390.72, "text": " What fraction actually has the rare disease?"}, {"start": 390.72, "end": 399.22, "text": " In other words, precision is defined as the number of true positives divided by the number"}, {"start": 399.22, "end": 403.12, "text": " classified as positive."}, {"start": 403.12, "end": 407.56, "text": " In other words, of all the examples you predicted as positive, what fraction do we actually"}, {"start": 407.56, "end": 409.88, "text": " get right?"}, {"start": 409.88, "end": 420.4, "text": " Another way to write this formula would be true positives divided by true positives plus"}, {"start": 420.4, "end": 430.0, "text": " false positives because it is by summing this cell and this cell that you end up with the"}, {"start": 430.0, "end": 434.71999999999997, "text": " total number that was predicted as positive."}, {"start": 434.72, "end": 443.92, "text": " In this example, the numerator true positives would be 15 and divided by 15 plus 5 and so"}, {"start": 443.92, "end": 448.68, "text": " that's 15 over 20 or three quarters or 0.75."}, {"start": 448.68, "end": 454.88000000000005, "text": " So we say that this algorithm has a precision of 75% because of all the things that predicted"}, {"start": 454.88000000000005, "end": 460.72, "text": " as positive, of all the patients that it thought has this rare disease, it was right 75% of"}, {"start": 460.72, "end": 461.72, "text": " the time."}, {"start": 461.72, "end": 467.84000000000003, "text": " The second metric that is useful to compute is recall."}, {"start": 467.84000000000003, "end": 472.84000000000003, "text": " And recall asks, of all the patients that actually have the rare disease, what fraction"}, {"start": 472.84000000000003, "end": 476.22, "text": " do we correctly detect as having it?"}, {"start": 476.22, "end": 482.96000000000004, "text": " So recall is defined as the number of true positives divided by the number of actual"}, {"start": 482.96000000000004, "end": 485.56, "text": " positives."}, {"start": 485.56, "end": 492.66, "text": " And alternatively, we can write that as number of true positives divided by the number of"}, {"start": 492.66, "end": 496.48, "text": " actual positives is, well, it's this cell plus this cell."}, {"start": 496.48, "end": 502.48, "text": " So it's actually the number of true positives plus the number of false negatives because"}, {"start": 502.48, "end": 506.6, "text": " it's by summing up this upper left cell and this lower left cell that you get the number"}, {"start": 506.6, "end": 509.64, "text": " of actual positive examples."}, {"start": 509.64, "end": 519.0, "text": " And so in our example, this would be 15 divided by 15 plus 10, which is 15 over 25, which"}, {"start": 519.0, "end": 523.24, "text": " is 0.6 or 60%."}, {"start": 523.24, "end": 529.48, "text": " So this learning algorithm would have 0.75 precision and 0.60 recall."}, {"start": 529.48, "end": 536.64, "text": " And you notice that this will help you detect if a learning algorithm is just printing y"}, {"start": 536.64, "end": 544.52, "text": " equals 0 all the time, because if it predicts 0 all the time, then the numerator of both"}, {"start": 544.52, "end": 548.04, "text": " of these quantities will be 0."}, {"start": 548.04, "end": 550.96, "text": " It has no true positives."}, {"start": 550.96, "end": 557.24, "text": " And the recall metric in particular helps you detect if the learning algorithm is predicting"}, {"start": 557.24, "end": 559.56, "text": " 0 all the time."}, {"start": 559.56, "end": 568.92, "text": " Because if your learning algorithm just prints y equals 0, then the number of true positives"}, {"start": 568.92, "end": 572.2199999999999, "text": " will be 0 because it never predicts positive."}, {"start": 572.2199999999999, "end": 578.4799999999999, "text": " And so the recall will be equal to 0 divided by the number of actual positives, which is"}, {"start": 578.4799999999999, "end": 580.3, "text": " equal to 0."}, {"start": 580.3, "end": 585.8399999999999, "text": " And in general, a learning algorithm with either 0 precision or 0 recall is not a useful"}, {"start": 585.8399999999999, "end": 586.8399999999999, "text": " algorithm."}, {"start": 586.84, "end": 593.08, "text": " But just as a side note, if an algorithm actually predicts 0 all the time, precision actually"}, {"start": 593.08, "end": 596.76, "text": " becomes undefined because it's actually 0 over 0."}, {"start": 596.76, "end": 601.5600000000001, "text": " But in practice, if an algorithm doesn't predict even a single positive, we'll just say that"}, {"start": 601.5600000000001, "end": 604.24, "text": " precision is also equal to 0."}, {"start": 604.24, "end": 610.96, "text": " But computing precision and recall makes it easier for you to spot if a learning algorithm."}, {"start": 610.96, "end": 616.44, "text": " But we'll find that computing both precision and recall makes it easier to spot if an algorithm"}, {"start": 616.44, "end": 622.8800000000001, "text": " is both reasonably accurate in that when it says a patient has a disease, there's a good"}, {"start": 622.8800000000001, "end": 627.5600000000001, "text": " chance the patient has a disease, such as 0.75 chance in this example."}, {"start": 627.5600000000001, "end": 632.24, "text": " And also making sure that of all the patients that have the disease is hoping to diagnose"}, {"start": 632.24, "end": 637.5600000000001, "text": " a reasonable fraction of them, such as here is finding 60% of them."}, {"start": 637.5600000000001, "end": 643.6800000000001, "text": " So when you have a rare class looking at precision and recall and making sure that both numbers"}, {"start": 643.68, "end": 650.16, "text": " are decently high, that hopefully helps reassure you that your learning algorithm is actually"}, {"start": 650.16, "end": 651.64, "text": " useful."}, {"start": 651.64, "end": 658.4399999999999, "text": " And the term recall was motivated by this observation that if you have a group of patients"}, {"start": 658.4399999999999, "end": 664.64, "text": " or a population of patients, then recall measures of all the patients that have the disease,"}, {"start": 664.64, "end": 669.12, "text": " how many would you have accurately diagnosed as having it."}, {"start": 669.12, "end": 674.96, "text": " So when you have skewed classes or a rare class that you want to detect, precision and"}, {"start": 674.96, "end": 682.0, "text": " recall helps you tell if your learning algorithms is making good predictions or useful predictions."}, {"start": 682.0, "end": 686.36, "text": " Now that we have these metrics for telling how well your learning algorithm is doing,"}, {"start": 686.36, "end": 691.76, "text": " in the next video, let's take a look at how to trade off between precision and recall"}, {"start": 691.76, "end": 699.4399999999999, "text": " to try to optimize the performance of your learning algorithm."}]