8.11 Anomaly Detection | Developing and evaluating an anomaly detection system-- [ML | Andrew Ng]
 I'd like to share with you some practical tips for developing an anomaly detection system. One of the key ideas will be that if you can have a way to evaluate a system even as it's being developed, you'll be able to make decisions and change a system and improve it much more quickly. Let's take a look at what that means. When you're developing a learning algorithm, say choosing different features or trying different values of the parameters like epsilon, making decisions about whether or not to change a feature in a certain way or to increase or decrease epsilon or other parameters, making those decisions is much easier if you have a way of evaluating the learning algorithm. This is sometimes called row number evaluation, meaning that if you can quickly change the algorithm in some way, such as change a feature or change a parameter, and have a way of computing a number that tells you if the algorithm got better or worse, then it makes it much easier to decide whether or not to stick with that change to the algorithm. This is how it's often done in anomaly detection, which is even though we've mainly been talking about unlabeled data, I'm going to change that assumption a bit and assume that we have some labeled data, including just a small number usually of previously observed anomalies. So maybe after making airplane engines for a few years, you've just seen a few airplane engines that were anomalous. And for examples that you know are anomalous, I'm going to associate a label y equals one to indicate this anomalous. And for examples that we think are normal, I'm going to associate the label y equals zero. And so the training set that the anomaly detection algorithm will learn from is still this unlabeled training set of x one through x m. And I'm going to think of all of these examples as ones that we'll just assume are normal and not anomalous. So y is equal to zero. In practice, if a few anomalous examples were to slip into this training set, your algorithm will still usually do okay. To evaluate your algorithm, to come up with a way for you to have a real number evaluation, it turns out to be very useful if you have a small number of anomalous examples so that you can create a cross validation set, which I'm going to denote x cv one, y cv one through x cv m cv and y cv m cv. This is similar notation as you had seen in the second course of the specialization and similarly have a test set of some number of examples where both the cross validation and the test sets hopefully include a few anomalous examples. In other words, the cross validation and test sets will have a few examples with y equals one, but also a lot of examples where y is equal to zero. And again, in practice, anomaly detection algorithm will work okay. If there are some examples, they're actually anomalous, but they were accidentally labeled with y equals zero. Let's illustrate this with the aircraft engine example. Let's say you have been manufacturing aircraft engines for years and so you've collected data from 10,000 goods or normal engines. But over the years, you had also collected data from 20 flawed or anomalous engines. Usually the number of anomalous engines that is y equals one will be much smaller. And so it would not be atypical to apply this type of algorithm with anywhere from say two to 50 known anomalies. We're going to take this data set and break it up into a training set, a cross validation set and a test set. Here's one example. I'm going to put 6,000 good engines into the training set. And again, if there are a couple of anomalous engines that got slipped into this set is actually okay. I wouldn't worry too much about that. And then let's put 2,000 good engines and 10 of the known anomalies into the cross validation set and a separate 2,000 good and 10 anomalous engines into the test set. What you can do then is train the algorithm on the training set, fit the Gaussian distributions to these 6,000 examples. And then on the cross validation set, you can see how many of the anomalous engines it correctly flags. And so for example, you could use the cross validation set to tune the parameter epsilon and set it higher or lower depending on whether the algorithm seems to be reliably detecting these 10 anomalies without taking too many of these 2,000 good engines and flagging them as anomalies. And after you have tuned the parameter epsilon and maybe also added or subtracted or tuned the features xj, you can then take the algorithm and evaluate it on your test set to see how many of these 10 anomalous engines it finds as well as how many mistakes it makes by flagging the good engines as anomalous ones. Notice that this is still primarily an unsupervised learning algorithm because the training set really has no labels or they all have labels that we're assuming to be y equals zero. And so we learn from the training set by fitting the Gaussian distributions as you saw in the previous video. But it turns out if you're building a practical anomaly detection system, having a small number of anomalies to use to evaluate the algorithm in your cross validation and test sets is very helpful for tuning the algorithm. Because the number of flawed engines is so small, there's one other alternative that I often see people use for anomaly detection, which is to not use a test set, but to have just a training set and a cross validation set. So in this example, you would still train on 6,000 good engines, but take the remainder of the data, the 4,000 remaining good engines as well as all the anomalies and put them in the cross validation set, and you would then tune the parameters epsilon and add or subtract features xj to try to get it to do as well as possible as evaluated on the cross validation set. If you have very, very few flawed engines, so if you had only two flawed engines, then this really makes sense to put all of that in the cross validation set and you just don't have enough data to create a totally separate test set that is distinct from your cross validation set. The downside of this alternative here is that after you've tuned your algorithm, you don't have a fair way to tell how well this will actually do on future examples because you don't have a test set. But when your data set is small, especially when the number of anomalies you have in your data set is small, this might be the best alternative you have. And so I see this done quite often as well when you just don't have enough data to create a separate test set. And if this is the case, just be aware that there's a higher risk that you would have overfit some of your decisions around epsilon and choice of features and so on to the cross validation set. And so its performance on real data in the future may not be as good as you were expecting. Now let's take a closer look at how to actually evaluate the algorithm on your cross validation sets or on the test set. Here's what you do. You would first fit the model p of x on the training set. So this was a 6,000 examples of good engines. Then on any cross validation or test set example x, you would compute p of x and you would predict y equals 1, that is anomalous if p of x is less than epsilon, and you predict y is zero if p of x is greater than or equal to epsilon. And so based on this, you can now look at how accurately this algorithm's predictions on the cross validation or test set matches the labels why you have the cross validation or the test sets. In the third week of the second course, we had had a couple optional videos on how to handle highly skewed data distributions where the number of positive examples, y equals 1, can be much smaller than the number of negative examples where y equals zero. And this is the case as well for many anomaly detection applications where the number of anomalies in your cross validation set is much smaller. In our previous example, we had maybe 10 positive examples and 2,000 negative examples because we had 10 anomalies and 2,000 normal examples. If you saw those optional videos, you may recall that we saw it could be useful to compute things like the true positive, false positive, false negative, and true negative rates, or to compute precision recall or F1 score, and that these are alternative metrics to classification accuracy that could work better when your data distribution is very skewed. So if you saw that video, you might consider applying those types of evaluation metrics as well to tell how well your learning algorithm is doing at finding that small handful of anomalies or positive examples amidst this much larger set of negative examples of normal plane engines. If you didn't watch that video, don't worry about it. It's okay. The intuition I hope you get is to use the cross validation set to just look at how many anomalies it's finding and also how many normal engines it is incorrectly flagging as an anomaly, and then to just use that to try to choose a good choice for the parameter epsilon. So you find that the practical process of building an anomaly detection system is much easier if you actually have just a small number of labeled examples of known anomalies. Now this does raise a question. If you have a few labeled examples, should you still be using an unsupervised learning algorithm? Why not take those labeled examples and use a supervised learning algorithm instead? In the next video, let's take a look at a comparison between anomaly detection and supervised learning and when you might prefer one over the other. Let's go on to the next video.
[{"start": 0.0, "end": 7.8, "text": " I'd like to share with you some practical tips for developing an anomaly detection system."}, {"start": 7.8, "end": 13.84, "text": " One of the key ideas will be that if you can have a way to evaluate a system even as it's"}, {"start": 13.84, "end": 19.16, "text": " being developed, you'll be able to make decisions and change a system and improve it much more"}, {"start": 19.16, "end": 20.16, "text": " quickly."}, {"start": 20.16, "end": 22.400000000000002, "text": " Let's take a look at what that means."}, {"start": 22.400000000000002, "end": 26.84, "text": " When you're developing a learning algorithm, say choosing different features or trying"}, {"start": 26.84, "end": 33.08, "text": " different values of the parameters like epsilon, making decisions about whether or not to change"}, {"start": 33.08, "end": 37.92, "text": " a feature in a certain way or to increase or decrease epsilon or other parameters, making"}, {"start": 37.92, "end": 44.2, "text": " those decisions is much easier if you have a way of evaluating the learning algorithm."}, {"start": 44.2, "end": 50.3, "text": " This is sometimes called row number evaluation, meaning that if you can quickly change the"}, {"start": 50.3, "end": 56.019999999999996, "text": " algorithm in some way, such as change a feature or change a parameter, and have a way of computing"}, {"start": 56.02, "end": 61.92, "text": " a number that tells you if the algorithm got better or worse, then it makes it much easier"}, {"start": 61.92, "end": 66.44, "text": " to decide whether or not to stick with that change to the algorithm."}, {"start": 66.44, "end": 72.68, "text": " This is how it's often done in anomaly detection, which is even though we've mainly been talking"}, {"start": 72.68, "end": 79.30000000000001, "text": " about unlabeled data, I'm going to change that assumption a bit and assume that we have"}, {"start": 79.3, "end": 87.08, "text": " some labeled data, including just a small number usually of previously observed anomalies."}, {"start": 87.08, "end": 93.32, "text": " So maybe after making airplane engines for a few years, you've just seen a few airplane"}, {"start": 93.32, "end": 96.1, "text": " engines that were anomalous."}, {"start": 96.1, "end": 103.8, "text": " And for examples that you know are anomalous, I'm going to associate a label y equals one"}, {"start": 103.8, "end": 107.16, "text": " to indicate this anomalous."}, {"start": 107.16, "end": 112.64, "text": " And for examples that we think are normal, I'm going to associate the label y equals"}, {"start": 112.64, "end": 113.64, "text": " zero."}, {"start": 113.64, "end": 121.16, "text": " And so the training set that the anomaly detection algorithm will learn from is still this unlabeled"}, {"start": 121.16, "end": 125.08, "text": " training set of x one through x m."}, {"start": 125.08, "end": 132.96, "text": " And I'm going to think of all of these examples as ones that we'll just assume are normal"}, {"start": 132.96, "end": 134.44, "text": " and not anomalous."}, {"start": 134.44, "end": 137.24, "text": " So y is equal to zero."}, {"start": 137.24, "end": 142.76, "text": " In practice, if a few anomalous examples were to slip into this training set, your algorithm"}, {"start": 142.76, "end": 145.24, "text": " will still usually do okay."}, {"start": 145.24, "end": 153.44, "text": " To evaluate your algorithm, to come up with a way for you to have a real number evaluation,"}, {"start": 153.44, "end": 161.4, "text": " it turns out to be very useful if you have a small number of anomalous examples so that"}, {"start": 161.4, "end": 168.24, "text": " you can create a cross validation set, which I'm going to denote x cv one, y cv one through"}, {"start": 168.24, "end": 170.64000000000001, "text": " x cv m cv and y cv m cv."}, {"start": 170.64000000000001, "end": 176.24, "text": " This is similar notation as you had seen in the second course of the specialization and"}, {"start": 176.24, "end": 185.28, "text": " similarly have a test set of some number of examples where both the cross validation and"}, {"start": 185.28, "end": 192.36, "text": " the test sets hopefully include a few anomalous examples."}, {"start": 192.36, "end": 197.16, "text": " In other words, the cross validation and test sets will have a few examples with y equals"}, {"start": 197.16, "end": 201.88, "text": " one, but also a lot of examples where y is equal to zero."}, {"start": 201.88, "end": 206.88, "text": " And again, in practice, anomaly detection algorithm will work okay."}, {"start": 206.88, "end": 212.44, "text": " If there are some examples, they're actually anomalous, but they were accidentally labeled"}, {"start": 212.44, "end": 214.16, "text": " with y equals zero."}, {"start": 214.16, "end": 218.92, "text": " Let's illustrate this with the aircraft engine example."}, {"start": 218.92, "end": 223.44, "text": " Let's say you have been manufacturing aircraft engines for years and so you've collected"}, {"start": 223.44, "end": 228.12, "text": " data from 10,000 goods or normal engines."}, {"start": 228.12, "end": 236.35999999999999, "text": " But over the years, you had also collected data from 20 flawed or anomalous engines."}, {"start": 236.35999999999999, "end": 242.48, "text": " Usually the number of anomalous engines that is y equals one will be much smaller."}, {"start": 242.48, "end": 250.56, "text": " And so it would not be atypical to apply this type of algorithm with anywhere from say two"}, {"start": 250.56, "end": 254.28, "text": " to 50 known anomalies."}, {"start": 254.28, "end": 258.88, "text": " We're going to take this data set and break it up into a training set, a cross validation"}, {"start": 258.88, "end": 260.59999999999997, "text": " set and a test set."}, {"start": 260.59999999999997, "end": 262.03999999999996, "text": " Here's one example."}, {"start": 262.03999999999996, "end": 267.4, "text": " I'm going to put 6,000 good engines into the training set."}, {"start": 267.4, "end": 272.47999999999996, "text": " And again, if there are a couple of anomalous engines that got slipped into this set is"}, {"start": 272.47999999999996, "end": 273.47999999999996, "text": " actually okay."}, {"start": 273.47999999999996, "end": 276.84, "text": " I wouldn't worry too much about that."}, {"start": 276.84, "end": 284.03999999999996, "text": " And then let's put 2,000 good engines and 10 of the known anomalies into the cross validation"}, {"start": 284.03999999999996, "end": 291.79999999999995, "text": " set and a separate 2,000 good and 10 anomalous engines into the test set."}, {"start": 291.8, "end": 299.2, "text": " What you can do then is train the algorithm on the training set, fit the Gaussian distributions"}, {"start": 299.2, "end": 301.76, "text": " to these 6,000 examples."}, {"start": 301.76, "end": 309.18, "text": " And then on the cross validation set, you can see how many of the anomalous engines"}, {"start": 309.18, "end": 312.08000000000004, "text": " it correctly flags."}, {"start": 312.08000000000004, "end": 318.44, "text": " And so for example, you could use the cross validation set to tune the parameter epsilon"}, {"start": 318.44, "end": 326.92, "text": " and set it higher or lower depending on whether the algorithm seems to be reliably detecting"}, {"start": 326.92, "end": 332.6, "text": " these 10 anomalies without taking too many of these 2,000 good engines and flagging them"}, {"start": 332.6, "end": 335.28, "text": " as anomalies."}, {"start": 335.28, "end": 340.64, "text": " And after you have tuned the parameter epsilon and maybe also added or subtracted or tuned"}, {"start": 340.64, "end": 348.84, "text": " the features xj, you can then take the algorithm and evaluate it on your test set to see how"}, {"start": 348.84, "end": 355.32, "text": " many of these 10 anomalous engines it finds as well as how many mistakes it makes by flagging"}, {"start": 355.32, "end": 358.76, "text": " the good engines as anomalous ones."}, {"start": 358.76, "end": 366.0, "text": " Notice that this is still primarily an unsupervised learning algorithm because the training set"}, {"start": 366.0, "end": 371.96, "text": " really has no labels or they all have labels that we're assuming to be y equals zero."}, {"start": 371.96, "end": 376.92, "text": " And so we learn from the training set by fitting the Gaussian distributions as you saw in the"}, {"start": 376.92, "end": 378.76, "text": " previous video."}, {"start": 378.76, "end": 384.64, "text": " But it turns out if you're building a practical anomaly detection system, having a small number"}, {"start": 384.64, "end": 390.84, "text": " of anomalies to use to evaluate the algorithm in your cross validation and test sets is"}, {"start": 390.84, "end": 394.2, "text": " very helpful for tuning the algorithm."}, {"start": 394.2, "end": 399.96, "text": " Because the number of flawed engines is so small, there's one other alternative that"}, {"start": 399.96, "end": 407.32, "text": " I often see people use for anomaly detection, which is to not use a test set, but to have"}, {"start": 407.32, "end": 410.84, "text": " just a training set and a cross validation set."}, {"start": 410.84, "end": 415.8, "text": " So in this example, you would still train on 6,000 good engines, but take the remainder"}, {"start": 415.8, "end": 421.4, "text": " of the data, the 4,000 remaining good engines as well as all the anomalies and put them"}, {"start": 421.4, "end": 426.52, "text": " in the cross validation set, and you would then tune the parameters epsilon and add or"}, {"start": 426.52, "end": 432.52, "text": " subtract features xj to try to get it to do as well as possible as evaluated on the cross"}, {"start": 432.52, "end": 434.59999999999997, "text": " validation set."}, {"start": 434.59999999999997, "end": 441.64, "text": " If you have very, very few flawed engines, so if you had only two flawed engines, then"}, {"start": 441.64, "end": 446.64, "text": " this really makes sense to put all of that in the cross validation set and you just don't"}, {"start": 446.64, "end": 451.76, "text": " have enough data to create a totally separate test set that is distinct from your cross"}, {"start": 451.76, "end": 453.32, "text": " validation set."}, {"start": 453.32, "end": 458.64, "text": " The downside of this alternative here is that after you've tuned your algorithm, you don't"}, {"start": 458.64, "end": 465.03999999999996, "text": " have a fair way to tell how well this will actually do on future examples because you"}, {"start": 465.03999999999996, "end": 466.8, "text": " don't have a test set."}, {"start": 466.8, "end": 471.03999999999996, "text": " But when your data set is small, especially when the number of anomalies you have in your"}, {"start": 471.03999999999996, "end": 475.0, "text": " data set is small, this might be the best alternative you have."}, {"start": 475.0, "end": 480.36, "text": " And so I see this done quite often as well when you just don't have enough data to create"}, {"start": 480.36, "end": 482.32, "text": " a separate test set."}, {"start": 482.32, "end": 487.28, "text": " And if this is the case, just be aware that there's a higher risk that you would have"}, {"start": 487.28, "end": 492.52, "text": " overfit some of your decisions around epsilon and choice of features and so on to the cross"}, {"start": 492.52, "end": 493.96, "text": " validation set."}, {"start": 493.96, "end": 501.4, "text": " And so its performance on real data in the future may not be as good as you were expecting."}, {"start": 501.4, "end": 507.2, "text": " Now let's take a closer look at how to actually evaluate the algorithm on your cross validation"}, {"start": 507.2, "end": 509.64, "text": " sets or on the test set."}, {"start": 509.64, "end": 511.08, "text": " Here's what you do."}, {"start": 511.08, "end": 515.28, "text": " You would first fit the model p of x on the training set."}, {"start": 515.28, "end": 519.12, "text": " So this was a 6,000 examples of good engines."}, {"start": 519.12, "end": 527.3199999999999, "text": " Then on any cross validation or test set example x, you would compute p of x and you would"}, {"start": 527.32, "end": 533.6400000000001, "text": " predict y equals 1, that is anomalous if p of x is less than epsilon, and you predict"}, {"start": 533.6400000000001, "end": 539.24, "text": " y is zero if p of x is greater than or equal to epsilon."}, {"start": 539.24, "end": 546.2, "text": " And so based on this, you can now look at how accurately this algorithm's predictions"}, {"start": 546.2, "end": 554.12, "text": " on the cross validation or test set matches the labels why you have the cross validation"}, {"start": 554.12, "end": 555.8800000000001, "text": " or the test sets."}, {"start": 555.88, "end": 561.68, "text": " In the third week of the second course, we had had a couple optional videos on how to"}, {"start": 561.68, "end": 569.32, "text": " handle highly skewed data distributions where the number of positive examples, y equals"}, {"start": 569.32, "end": 574.8, "text": " 1, can be much smaller than the number of negative examples where y equals zero."}, {"start": 574.8, "end": 579.8, "text": " And this is the case as well for many anomaly detection applications where the number of"}, {"start": 579.8, "end": 584.2, "text": " anomalies in your cross validation set is much smaller."}, {"start": 584.2, "end": 591.36, "text": " In our previous example, we had maybe 10 positive examples and 2,000 negative examples because"}, {"start": 591.36, "end": 595.84, "text": " we had 10 anomalies and 2,000 normal examples."}, {"start": 595.84, "end": 601.32, "text": " If you saw those optional videos, you may recall that we saw it could be useful to compute"}, {"start": 601.32, "end": 605.84, "text": " things like the true positive, false positive, false negative, and true negative rates, or"}, {"start": 605.84, "end": 611.6400000000001, "text": " to compute precision recall or F1 score, and that these are alternative metrics to classification"}, {"start": 611.64, "end": 617.76, "text": " accuracy that could work better when your data distribution is very skewed."}, {"start": 617.76, "end": 623.04, "text": " So if you saw that video, you might consider applying those types of evaluation metrics"}, {"start": 623.04, "end": 629.92, "text": " as well to tell how well your learning algorithm is doing at finding that small handful of"}, {"start": 629.92, "end": 636.36, "text": " anomalies or positive examples amidst this much larger set of negative examples of normal"}, {"start": 636.36, "end": 637.36, "text": " plane engines."}, {"start": 637.36, "end": 640.52, "text": " If you didn't watch that video, don't worry about it."}, {"start": 640.52, "end": 641.52, "text": " It's okay."}, {"start": 641.52, "end": 647.0, "text": " The intuition I hope you get is to use the cross validation set to just look at how many"}, {"start": 647.0, "end": 654.0, "text": " anomalies it's finding and also how many normal engines it is incorrectly flagging as an anomaly,"}, {"start": 654.0, "end": 660.1999999999999, "text": " and then to just use that to try to choose a good choice for the parameter epsilon."}, {"start": 660.1999999999999, "end": 666.48, "text": " So you find that the practical process of building an anomaly detection system is much"}, {"start": 666.48, "end": 673.88, "text": " easier if you actually have just a small number of labeled examples of known anomalies."}, {"start": 673.88, "end": 676.12, "text": " Now this does raise a question."}, {"start": 676.12, "end": 680.64, "text": " If you have a few labeled examples, should you still be using an unsupervised learning"}, {"start": 680.64, "end": 681.64, "text": " algorithm?"}, {"start": 681.64, "end": 686.72, "text": " Why not take those labeled examples and use a supervised learning algorithm instead?"}, {"start": 686.72, "end": 692.6, "text": " In the next video, let's take a look at a comparison between anomaly detection and supervised"}, {"start": 692.6, "end": 696.24, "text": " learning and when you might prefer one over the other."}, {"start": 696.24, "end": 697.84, "text": " Let's go on to the next video."}]