8.9 Anomaly Detection | Gaussian (normal) distribution -- [Machine Learning | Andrew Ng]
 In order to apply anomaly detection, we're going to need to use the Gaussian distribution, which is also called the normal distribution. So when you hear me say either Gaussian distribution or normal distribution, they mean exactly the same thing. And if you've heard of the bell-shaped distribution, that also refers to the same thing. But if you haven't heard of the bell-shaped distribution, that's fine too. But let's take a look at what is the Gaussian or the normal distribution. Say X is a number, and if X is a random number, sometimes called a random variable, but if X can take on random values, and if the probability of X is given by a Gaussian or a normal distribution with mean parameter mu and with variance sigma squared, what that means is that the probability of X looks like a curve that goes like this. The center, the middle of the curve, is given by the mean, mu, and the standard deviation or the width of this curve is given by that variance parameter sigma. Technically, sigma is called the standard deviation, and the square of sigma, or sigma squared, is called the variance of the distribution. And this curve here shows what is P of X or the probability of X. If you've heard of the bell-shaped curve, this is that bell-shaped curve because a lot of classic bells, say in towers, were kind of shaped like this with the bell clapper hanging down here. And so the shape of this curve is vaguely reminiscent of the shape of the large bells that you will still find in some old buildings today. Better looking than my hand-drawn one, there's a picture of the Liberty Bell, and indeed the Liberty Bell's shape on top is a vaguely bell-shaped curve. If you're wondering what this P of X really means, here's one way to interpret it. It means that if you were to get, say, 100 numbers drawn from this probability distribution, and you were to plot a histogram of these 100 numbers drawn from this distribution, you might get a histogram that looks like this. And so it looks vaguely bell-shaped. And what this curve on the left indicates is not if you have just 100 examples or a thousand or a million or a billion, but if you had a practically infinite number of examples and you were to draw a histogram of this practically infinite number of examples with a very, very fine histogram bin, then you end up with essentially this bell-shaped curve here on the left. The formula for P of X is given by this expression. P of X equals 1 over square root 2 pi. Pi here is that 3.14159, it was about 22 over 7, a ratio of a circle's diameter to circumference, times sigma times e to the negative x minus mu, the mean parameter squared, divided by 2 sigma squared. And for any given value of mu and sigma, if you were to plot this function as a function of x, you get this type of bell-shaped curve that is centered at mu and with the width of this bell-shaped curve being determined by the parameter sigma. Now let's look at a few examples of how changing mu and sigma will affect the Gaussian distribution. First, let me set mu equals to 0 and sigma equals 1. Here's my plot of a Gaussian distribution with mean 0, mu equals 0, and standard deviation sigma equals 1. You notice that this distribution is centered at 0 and that is the standard deviation sigma is equal to 1. Now let's reduce the standard deviation sigma to 0.5. If you plot the Gaussian distribution with mu equals 0 and sigma equals 0.5, it now looks like this. Notice that it's still centered at 0 because mu is 0, but it's become a much thinner curve because sigma is now 1 half. And you might recall that sigma is the standard deviation is 0.5, whereas sigma squared is also called the variance and so that's equal to 0.5 squared or 0.25. You may have heard that probabilities always have to sum up to 1, so that's why the area under the curve is always equal to 1, which is why when the Gaussian distribution becomes skinnier it has to become taller as well. Let's look at another value of mu and sigma. Now I'm going to increase sigma to 2, so the standard deviation is 2 and the variance is 4. This now creates a much wider distribution because sigma here is now much larger and because it's now a wider distribution it's become shorter as well because the area under the curve is still equal to 1. And finally let's try changing the mean parameter mu and I'll leave sigma equals 0.5. In this case the center of the distribution mu moves over here to the right, but the width of the distribution is the same as the one on top because the standard deviation is 0.5 in both of these cases on the right. So this is how different choices of mu and sigma affect the Gaussian distribution. When you're applying this to anomaly detection here's what you have to do. You're given a data set of m examples and here x is just a number and here I plotted the training sets with 11 examples and what we have to do is try to estimate what are good choices for the mean parameter mu as well as for the variance parameter sigma squared. And given a data set like this it would seem that a Gaussian distribution may be looking like that with a center here and a standard deviation kind of like that. This might be a pretty good fit to the data. The way you would compute mu and sigma squared mathematically is our estimate for mu will be just the average of all the returning examples. So it's 1 over m times sum from i equals 1 through m of the values of your training examples and the value we would use to estimate sigma squared will be 1 over m times will be the average of the squared difference between the examples and that mu that you just estimated here on the left. It turns out that if you implement these two formulas in code with this value for mu and this value for sigma squared then you pretty much get the Gaussian distribution that I hand drew on top and this will give you a choice of mu and sigma for a Gaussian distribution so that it kind of looks like the 11 training examples might have been drawn from this Gaussian distribution. If you've taken an advanced statistics class you may have heard that these formulas for mu and sigma squared are technically called the maximum likelihood estimates for mu and sigma and some statistics classes will tell you to use the formula 1 over m minus 1 instead of 1 over m. In practice using 1 over m or 1 over m minus 1 makes very little difference. I always use 1 over m but there are some other properties of dividing by m minus 1 that some statisticians prefer. But if you don't understand what I just said don't worry about it. All you need to know is that if you set mu according to this formula and sigma squared according to this formula you get a pretty good estimate of mu and sigma and in particular you get a Gaussian distribution that will be a plausible probability distribution in terms of what's the probability distribution that the training examples had come from. You can probably guess what comes next. If you were to get an example over here then p of x is pretty high whereas if you were to get an example way out here then p of x is pretty low which is why we would consider this example okay not really anomalous, not a lot like the other ones, whereas an example way out here to be pretty unusual compared to the examples we've seen and therefore more anomalous because p of x which is the height of this curve is much lower over here on the left compared to this point over here closer to the middle. Now we've done this only for when x is a number as if you had only a single feature for your anomaly detection problem. For practical anomaly detection applications you usually have a lot of different features so you've now seen how the Gaussian distribution works if x is a single number. This corresponds to if say you had just one feature for your anomaly detection problem but for practical anomaly detection applications you will have many features two or three or some even larger number n of features. Let's take what you saw for a single Gaussian and use it to build a more sophisticated anomaly detection algorithm that can handle multiple features. Let's go through that in the next video.
[{"start": 0.0, "end": 8.84, "text": " In order to apply anomaly detection, we're going to need to use the Gaussian distribution,"}, {"start": 8.84, "end": 12.46, "text": " which is also called the normal distribution."}, {"start": 12.46, "end": 17.900000000000002, "text": " So when you hear me say either Gaussian distribution or normal distribution, they mean exactly"}, {"start": 17.900000000000002, "end": 19.44, "text": " the same thing."}, {"start": 19.44, "end": 25.16, "text": " And if you've heard of the bell-shaped distribution, that also refers to the same thing."}, {"start": 25.16, "end": 29.04, "text": " But if you haven't heard of the bell-shaped distribution, that's fine too."}, {"start": 29.04, "end": 34.08, "text": " But let's take a look at what is the Gaussian or the normal distribution."}, {"start": 34.08, "end": 40.2, "text": " Say X is a number, and if X is a random number, sometimes called a random variable, but if"}, {"start": 40.2, "end": 48.92, "text": " X can take on random values, and if the probability of X is given by a Gaussian or a normal distribution"}, {"start": 48.92, "end": 58.92, "text": " with mean parameter mu and with variance sigma squared, what that means is that the probability"}, {"start": 58.92, "end": 65.1, "text": " of X looks like a curve that goes like this."}, {"start": 65.1, "end": 74.16, "text": " The center, the middle of the curve, is given by the mean, mu, and the standard deviation"}, {"start": 74.16, "end": 81.36, "text": " or the width of this curve is given by that variance parameter sigma."}, {"start": 81.36, "end": 87.08, "text": " Technically, sigma is called the standard deviation, and the square of sigma, or sigma"}, {"start": 87.08, "end": 90.8, "text": " squared, is called the variance of the distribution."}, {"start": 90.8, "end": 96.36, "text": " And this curve here shows what is P of X or the probability of X."}, {"start": 96.36, "end": 101.88, "text": " If you've heard of the bell-shaped curve, this is that bell-shaped curve because a lot"}, {"start": 101.88, "end": 110.16, "text": " of classic bells, say in towers, were kind of shaped like this with the bell clapper"}, {"start": 110.16, "end": 111.6, "text": " hanging down here."}, {"start": 111.6, "end": 117.08, "text": " And so the shape of this curve is vaguely reminiscent of the shape of the large bells"}, {"start": 117.08, "end": 121.96, "text": " that you will still find in some old buildings today."}, {"start": 121.96, "end": 127.03999999999999, "text": " Better looking than my hand-drawn one, there's a picture of the Liberty Bell, and indeed"}, {"start": 127.03999999999999, "end": 132.88, "text": " the Liberty Bell's shape on top is a vaguely bell-shaped curve."}, {"start": 132.88, "end": 140.4, "text": " If you're wondering what this P of X really means, here's one way to interpret it."}, {"start": 140.4, "end": 147.44, "text": " It means that if you were to get, say, 100 numbers drawn from this probability distribution,"}, {"start": 147.44, "end": 153.84, "text": " and you were to plot a histogram of these 100 numbers drawn from this distribution,"}, {"start": 153.84, "end": 156.04000000000002, "text": " you might get a histogram that looks like this."}, {"start": 156.04000000000002, "end": 159.04000000000002, "text": " And so it looks vaguely bell-shaped."}, {"start": 159.04000000000002, "end": 166.32, "text": " And what this curve on the left indicates is not if you have just 100 examples or a"}, {"start": 166.32, "end": 172.79999999999998, "text": " thousand or a million or a billion, but if you had a practically infinite number of examples"}, {"start": 172.79999999999998, "end": 178.68, "text": " and you were to draw a histogram of this practically infinite number of examples with a very, very"}, {"start": 178.68, "end": 187.16, "text": " fine histogram bin, then you end up with essentially this bell-shaped curve here on the left."}, {"start": 187.16, "end": 192.04, "text": " The formula for P of X is given by this expression."}, {"start": 192.04, "end": 197.04, "text": " P of X equals 1 over square root 2 pi."}, {"start": 197.04, "end": 206.23999999999998, "text": " Pi here is that 3.14159, it was about 22 over 7, a ratio of a circle's diameter to circumference,"}, {"start": 206.23999999999998, "end": 214.64, "text": " times sigma times e to the negative x minus mu, the mean parameter squared, divided by"}, {"start": 214.64, "end": 217.88, "text": " 2 sigma squared."}, {"start": 217.88, "end": 224.96, "text": " And for any given value of mu and sigma, if you were to plot this function as a function"}, {"start": 224.96, "end": 231.04, "text": " of x, you get this type of bell-shaped curve that is centered at mu and with the width"}, {"start": 231.04, "end": 237.56, "text": " of this bell-shaped curve being determined by the parameter sigma."}, {"start": 237.56, "end": 244.72, "text": " Now let's look at a few examples of how changing mu and sigma will affect the Gaussian distribution."}, {"start": 244.72, "end": 250.68, "text": " First, let me set mu equals to 0 and sigma equals 1."}, {"start": 250.68, "end": 257.28, "text": " Here's my plot of a Gaussian distribution with mean 0, mu equals 0, and standard deviation"}, {"start": 257.28, "end": 260.32, "text": " sigma equals 1."}, {"start": 260.32, "end": 267.8, "text": " You notice that this distribution is centered at 0 and that is the standard deviation sigma"}, {"start": 267.8, "end": 269.56, "text": " is equal to 1."}, {"start": 269.56, "end": 276.16, "text": " Now let's reduce the standard deviation sigma to 0.5."}, {"start": 276.16, "end": 283.04, "text": " If you plot the Gaussian distribution with mu equals 0 and sigma equals 0.5, it now looks"}, {"start": 283.04, "end": 284.8, "text": " like this."}, {"start": 284.8, "end": 291.4, "text": " Notice that it's still centered at 0 because mu is 0, but it's become a much thinner curve"}, {"start": 291.4, "end": 296.24, "text": " because sigma is now 1 half."}, {"start": 296.24, "end": 303.12, "text": " And you might recall that sigma is the standard deviation is 0.5, whereas sigma squared is"}, {"start": 303.12, "end": 310.2, "text": " also called the variance and so that's equal to 0.5 squared or 0.25."}, {"start": 310.2, "end": 315.12, "text": " You may have heard that probabilities always have to sum up to 1, so that's why the area"}, {"start": 315.12, "end": 321.44, "text": " under the curve is always equal to 1, which is why when the Gaussian distribution becomes"}, {"start": 321.44, "end": 325.2, "text": " skinnier it has to become taller as well."}, {"start": 325.2, "end": 328.03999999999996, "text": " Let's look at another value of mu and sigma."}, {"start": 328.03999999999996, "end": 334.76, "text": " Now I'm going to increase sigma to 2, so the standard deviation is 2 and the variance is"}, {"start": 334.76, "end": 336.76, "text": " 4."}, {"start": 336.76, "end": 344.68, "text": " This now creates a much wider distribution because sigma here is now much larger and"}, {"start": 344.68, "end": 349.12, "text": " because it's now a wider distribution it's become shorter as well because the area under"}, {"start": 349.12, "end": 352.56, "text": " the curve is still equal to 1."}, {"start": 352.56, "end": 362.04, "text": " And finally let's try changing the mean parameter mu and I'll leave sigma equals 0.5."}, {"start": 362.04, "end": 370.24, "text": " In this case the center of the distribution mu moves over here to the right, but the width"}, {"start": 370.24, "end": 375.6, "text": " of the distribution is the same as the one on top because the standard deviation is 0.5"}, {"start": 375.6, "end": 378.64, "text": " in both of these cases on the right."}, {"start": 378.64, "end": 385.76, "text": " So this is how different choices of mu and sigma affect the Gaussian distribution."}, {"start": 385.76, "end": 390.59999999999997, "text": " When you're applying this to anomaly detection here's what you have to do."}, {"start": 390.59999999999997, "end": 399.44, "text": " You're given a data set of m examples and here x is just a number and here I plotted"}, {"start": 399.44, "end": 406.2, "text": " the training sets with 11 examples and what we have to do is try to estimate what are"}, {"start": 406.2, "end": 415.8, "text": " good choices for the mean parameter mu as well as for the variance parameter sigma squared."}, {"start": 415.8, "end": 422.48, "text": " And given a data set like this it would seem that a Gaussian distribution may be looking"}, {"start": 422.48, "end": 428.59999999999997, "text": " like that with a center here and a standard deviation kind of like that."}, {"start": 428.59999999999997, "end": 432.64, "text": " This might be a pretty good fit to the data."}, {"start": 432.64, "end": 439.56, "text": " The way you would compute mu and sigma squared mathematically is our estimate for mu will"}, {"start": 439.56, "end": 442.91999999999996, "text": " be just the average of all the returning examples."}, {"start": 442.91999999999996, "end": 450.68, "text": " So it's 1 over m times sum from i equals 1 through m of the values of your training examples"}, {"start": 450.68, "end": 459.68, "text": " and the value we would use to estimate sigma squared will be 1 over m times will be the"}, {"start": 459.68, "end": 466.92, "text": " average of the squared difference between the examples and that mu that you just estimated"}, {"start": 466.92, "end": 469.02, "text": " here on the left."}, {"start": 469.02, "end": 474.16, "text": " It turns out that if you implement these two formulas in code with this value for mu and"}, {"start": 474.16, "end": 478.72, "text": " this value for sigma squared then you pretty much get the Gaussian distribution that I"}, {"start": 478.72, "end": 485.36, "text": " hand drew on top and this will give you a choice of mu and sigma for a Gaussian distribution"}, {"start": 485.36, "end": 490.16, "text": " so that it kind of looks like the 11 training examples might have been drawn from this Gaussian"}, {"start": 490.16, "end": 492.96000000000004, "text": " distribution."}, {"start": 492.96000000000004, "end": 498.36, "text": " If you've taken an advanced statistics class you may have heard that these formulas for"}, {"start": 498.36, "end": 503.72, "text": " mu and sigma squared are technically called the maximum likelihood estimates for mu and"}, {"start": 503.72, "end": 511.08000000000004, "text": " sigma and some statistics classes will tell you to use the formula 1 over m minus 1 instead"}, {"start": 511.08000000000004, "end": 513.38, "text": " of 1 over m."}, {"start": 513.38, "end": 518.92, "text": " In practice using 1 over m or 1 over m minus 1 makes very little difference."}, {"start": 518.92, "end": 526.96, "text": " I always use 1 over m but there are some other properties of dividing by m minus 1 that some"}, {"start": 526.96, "end": 529.52, "text": " statisticians prefer."}, {"start": 529.52, "end": 532.96, "text": " But if you don't understand what I just said don't worry about it."}, {"start": 532.96, "end": 539.26, "text": " All you need to know is that if you set mu according to this formula and sigma squared"}, {"start": 539.26, "end": 545.48, "text": " according to this formula you get a pretty good estimate of mu and sigma and in particular"}, {"start": 545.48, "end": 551.48, "text": " you get a Gaussian distribution that will be a plausible probability distribution in"}, {"start": 551.48, "end": 556.96, "text": " terms of what's the probability distribution that the training examples had come from."}, {"start": 556.96, "end": 560.4, "text": " You can probably guess what comes next."}, {"start": 560.4, "end": 569.36, "text": " If you were to get an example over here then p of x is pretty high whereas if you were"}, {"start": 569.36, "end": 576.56, "text": " to get an example way out here then p of x is pretty low which is why we would consider"}, {"start": 576.56, "end": 582.64, "text": " this example okay not really anomalous, not a lot like the other ones, whereas an example"}, {"start": 582.64, "end": 588.36, "text": " way out here to be pretty unusual compared to the examples we've seen and therefore more"}, {"start": 588.36, "end": 594.04, "text": " anomalous because p of x which is the height of this curve is much lower over here on the"}, {"start": 594.04, "end": 599.4, "text": " left compared to this point over here closer to the middle."}, {"start": 599.4, "end": 606.24, "text": " Now we've done this only for when x is a number as if you had only a single feature for your"}, {"start": 606.24, "end": 608.84, "text": " anomaly detection problem."}, {"start": 608.84, "end": 615.28, "text": " For practical anomaly detection applications you usually have a lot of different features"}, {"start": 615.28, "end": 621.8399999999999, "text": " so you've now seen how the Gaussian distribution works if x is a single number."}, {"start": 621.8399999999999, "end": 628.48, "text": " This corresponds to if say you had just one feature for your anomaly detection problem"}, {"start": 628.48, "end": 634.4399999999999, "text": " but for practical anomaly detection applications you will have many features two or three or"}, {"start": 634.4399999999999, "end": 638.1999999999999, "text": " some even larger number n of features."}, {"start": 638.1999999999999, "end": 643.4399999999999, "text": " Let's take what you saw for a single Gaussian and use it to build a more sophisticated anomaly"}, {"start": 643.44, "end": 647.12, "text": " detection algorithm that can handle multiple features."}, {"start": 647.12, "end": 674.12, "text": " Let's go through that in the next video."}]