3.2 Classification | Logistic regression  --[Machine Learning | Andrew Ng]
 Let's talk about logistic regression, which is probably the single most widely used classification algorithm in the world. This is something that I use all the time in my work. Let's continue with the example of classifying whether a tumor is malignant, where as before we're going to use the label 1 or yes the positive class to represent malignant tumors and 0 or no negative examples to represent benign tumors. Here's a graph of the dataset where the horizontal axis is the tumor size and the vertical axis takes on only values of 0 and 1 because it's a classification problem. You saw in the last video that linear regression is not a good algorithm for this problem. In contrast, what logistic regression will end up doing is fit a curve that looks like this, a sort of S-shaped curve to this dataset. And so for this example, if a patient comes in with a tumor of this size, which I'm showing on the x-axis, then the algorithm will output 0.7, suggesting that it's closer or maybe more likely to be malignant and benign. We'll say more later what 0.7 actually means in this context, but the output label y is never 0.7, is only ever 0 or 1. To build up to the logistic regression algorithm, there's an important mathematical function I'd like to describe which is called the sigmoid function, sometimes also referred to as the logistic function. The sigmoid function looks like this. Notice that the x-axis of the graphs on the left and right are different. In the graph to the left, on the x-axis is the tumor size, so it's all positive numbers. Whereas in the graph on the right, you have 0 down here, and the horizontal axis takes on both negative and positive values, and I've labeled the horizontal axis z. And I'm showing here just a range of negative 3 to plus 3. So the sigmoid function outputs values between 0 and 1. And if I use g of z to denote this function, then the formula of g of z is equal to 1 over 1 plus e to the negative z, where here e is a mathematical constant that takes on a value of about 2.7, and so e to the negative z is that mathematical constant to the power of negative z. Notice if z were really big, say 100, e to the negative z is e to the negative 100, which is a tiny, tiny number. So this ends up being 1 over 1 plus a tiny little number, and so the denominator will be basically very, very close to 1, which is why when z is large, g of z, that is the sigmoid function of z, is going to be very close to 1. And conversely, you can also check for yourself that when z is a very large negative number, then g of z becomes 1 over a giant number, which is why g of z is very close to 0. So that's why the sigmoid function has this shape, where it starts very close to 0 and slowly builds up, or grows, to the value of 1. Also, in the sigmoid function, when z is equal to 0, then e to the negative z is e to the negative 0, which is equal to 1, and so g of z is equal to 1 over 1 plus 1, which is 0.5. So that's why it passes the vertical axis at 0.5. Now let's use this to build up to the logistic regression algorithm. We're going to do this in two steps. In the first step, I hope you remember that a straight line function, like a linear regression function, can be defined as W dot product of x plus b. So let's store this value in a variable, which I'm going to call z. And this will turn out to be the same z as the one you saw on the previous slide, but we'll get to that in a minute. The next step then is to take this value of z and pass it to the sigmoid function, also called the logistic function g. So now g of z then outputs a value computed by this formula, 1 over 1 plus e to the negative z, that's going to be between 0 and 1. When you take these two equations and put them together, they then give you the logistic regression model f of x, which is equal to g of w x plus b, or equivalently g of z, which is equal to this formula over here. So this is the logistic regression model. And what it does is it inputs a feature or set of features x, and it outputs a number between 0 and 1. Next, let's take a look at how to interpret the output of logistic regression. We'll return to the tumor classification example. The way I'd encourage you to think of logistic regression's output is to think of it as outputting the probability that the class or the label y will be equal to 1, given a certain input x. So, for example, in this application where x is the tumor size and y is either 0 or 1, if you have a patient come in and she has a tumor of a certain size x, and if based on this input x, the model outputs 0.7, then what that means is that the model is predicting or the model thinks there's a 70% chance that the true label y will be equal to 1 for this patient. In other words, the model is telling us that it thinks the patient has a 70% chance of their tumor turning out to be malignant. Now, let me ask you a question. See if you can get this right. We know that y has to be either 0 or 1. So if y has a 70% chance of being 1, what is the chance that it is 0? So y has got to be either 0 or 1, and thus the probability of it being 0 or 1, these two numbers have to add up to 1 or to 100% chance. So that's why if the chance of y being 1 is 0.7 or 70% chance, then the chance of it being 0 has got to be 0.3 or 30% chance. If someday you read research papers or blog posts about logistic regression, sometimes you see this notation that f of x is equal to p of y equals 1, given the input features x and with parameters w and b. What the semicolon here is used to denote is just that w and b are parameters that affect this computation of what is the probability of y being equal to 1, given the input feature x. For the purpose of this class, don't worry too much about what this vertical line and what the semicolon mean. You don't need to remember or follow any of this mathematical notation for this class. I'm mentioning this only because you may see this in other places. In the optional lab that follows this video, you also get to see how the sigmoid function is implemented in code. You can see a plot that uses the sigmoid function so as to do better on the classification task that you saw in the previous optional lab. Remember that the code will be provided to you, so you just have to run it. I hope you take a look and get familiar with the code. So congrats on getting here. You now know what is the logistic regression model as well as the mathematical formula that defines logistic regression. For a long time, a lot of internet advertising was actually driven by basically a slight variation of logistic regression. This was very lucrative for some large companies, and this was basically the algorithm that decided what ad was shown to you and many others on some large websites. Now there's even more to learn about this algorithm. In the next video, we'll take a look at the details of logistic regression. We'll look at some visualizations and also examine something called the decision boundary. This will give you a few different ways to map the numbers that this model outputs, such as 0.3 or 0.7 or 0.65, to a prediction of whether y is actually 0 or 1. So let's go on to the next video to learn more about logistic regression.
[{"start": 0.0, "end": 6.0600000000000005, "text": " Let's talk about logistic regression, which is probably the single most widely used classification"}, {"start": 6.0600000000000005, "end": 7.54, "text": " algorithm in the world."}, {"start": 7.54, "end": 10.94, "text": " This is something that I use all the time in my work."}, {"start": 10.94, "end": 17.22, "text": " Let's continue with the example of classifying whether a tumor is malignant, where as before"}, {"start": 17.22, "end": 22.900000000000002, "text": " we're going to use the label 1 or yes the positive class to represent malignant tumors"}, {"start": 22.900000000000002, "end": 27.62, "text": " and 0 or no negative examples to represent benign tumors."}, {"start": 27.62, "end": 34.86, "text": " Here's a graph of the dataset where the horizontal axis is the tumor size and the vertical axis"}, {"start": 34.86, "end": 41.14, "text": " takes on only values of 0 and 1 because it's a classification problem."}, {"start": 41.14, "end": 46.68, "text": " You saw in the last video that linear regression is not a good algorithm for this problem."}, {"start": 46.68, "end": 53.540000000000006, "text": " In contrast, what logistic regression will end up doing is fit a curve that looks like"}, {"start": 53.54, "end": 59.339999999999996, "text": " this, a sort of S-shaped curve to this dataset."}, {"start": 59.339999999999996, "end": 65.62, "text": " And so for this example, if a patient comes in with a tumor of this size, which I'm showing"}, {"start": 65.62, "end": 73.66, "text": " on the x-axis, then the algorithm will output 0.7, suggesting that it's closer or maybe"}, {"start": 73.66, "end": 77.5, "text": " more likely to be malignant and benign."}, {"start": 77.5, "end": 84.66, "text": " We'll say more later what 0.7 actually means in this context, but the output label y is"}, {"start": 84.66, "end": 89.5, "text": " never 0.7, is only ever 0 or 1."}, {"start": 89.5, "end": 94.46000000000001, "text": " To build up to the logistic regression algorithm, there's an important mathematical function"}, {"start": 94.46000000000001, "end": 101.06, "text": " I'd like to describe which is called the sigmoid function, sometimes also referred to as the"}, {"start": 101.06, "end": 103.66, "text": " logistic function."}, {"start": 103.66, "end": 107.02, "text": " The sigmoid function looks like this."}, {"start": 107.02, "end": 112.16, "text": " Notice that the x-axis of the graphs on the left and right are different."}, {"start": 112.16, "end": 118.97999999999999, "text": " In the graph to the left, on the x-axis is the tumor size, so it's all positive numbers."}, {"start": 118.97999999999999, "end": 126.1, "text": " Whereas in the graph on the right, you have 0 down here, and the horizontal axis takes"}, {"start": 126.1, "end": 132.38, "text": " on both negative and positive values, and I've labeled the horizontal axis z."}, {"start": 132.38, "end": 138.74, "text": " And I'm showing here just a range of negative 3 to plus 3."}, {"start": 138.74, "end": 142.62, "text": " So the sigmoid function outputs values between 0 and 1."}, {"start": 142.62, "end": 151.26, "text": " And if I use g of z to denote this function, then the formula of g of z is equal to 1 over"}, {"start": 151.26, "end": 158.7, "text": " 1 plus e to the negative z, where here e is a mathematical constant that takes on a value"}, {"start": 158.7, "end": 164.5, "text": " of about 2.7, and so e to the negative z is that mathematical constant to the power of"}, {"start": 164.5, "end": 167.33999999999997, "text": " negative z."}, {"start": 167.33999999999997, "end": 175.5, "text": " Notice if z were really big, say 100, e to the negative z is e to the negative 100, which"}, {"start": 175.5, "end": 178.35999999999999, "text": " is a tiny, tiny number."}, {"start": 178.35999999999999, "end": 185.89999999999998, "text": " So this ends up being 1 over 1 plus a tiny little number, and so the denominator will"}, {"start": 185.9, "end": 193.14000000000001, "text": " be basically very, very close to 1, which is why when z is large, g of z, that is the"}, {"start": 193.14000000000001, "end": 198.46, "text": " sigmoid function of z, is going to be very close to 1."}, {"start": 198.46, "end": 206.14000000000001, "text": " And conversely, you can also check for yourself that when z is a very large negative number,"}, {"start": 206.14000000000001, "end": 215.64000000000001, "text": " then g of z becomes 1 over a giant number, which is why g of z is very close to 0."}, {"start": 215.64, "end": 221.88, "text": " So that's why the sigmoid function has this shape, where it starts very close to 0 and"}, {"start": 221.88, "end": 226.57999999999998, "text": " slowly builds up, or grows, to the value of 1."}, {"start": 226.57999999999998, "end": 234.89999999999998, "text": " Also, in the sigmoid function, when z is equal to 0, then e to the negative z is e to the"}, {"start": 234.89999999999998, "end": 243.94, "text": " negative 0, which is equal to 1, and so g of z is equal to 1 over 1 plus 1, which is"}, {"start": 243.94, "end": 245.94, "text": " 0.5."}, {"start": 245.94, "end": 251.5, "text": " So that's why it passes the vertical axis at 0.5."}, {"start": 251.5, "end": 256.18, "text": " Now let's use this to build up to the logistic regression algorithm."}, {"start": 256.18, "end": 258.98, "text": " We're going to do this in two steps."}, {"start": 258.98, "end": 264.7, "text": " In the first step, I hope you remember that a straight line function, like a linear regression"}, {"start": 264.7, "end": 271.92, "text": " function, can be defined as W dot product of x plus b."}, {"start": 271.92, "end": 277.90000000000003, "text": " So let's store this value in a variable, which I'm going to call z."}, {"start": 277.90000000000003, "end": 282.14000000000004, "text": " And this will turn out to be the same z as the one you saw on the previous slide, but"}, {"start": 282.14000000000004, "end": 284.18, "text": " we'll get to that in a minute."}, {"start": 284.18, "end": 291.78000000000003, "text": " The next step then is to take this value of z and pass it to the sigmoid function, also"}, {"start": 291.78000000000003, "end": 295.66, "text": " called the logistic function g."}, {"start": 295.66, "end": 303.94, "text": " So now g of z then outputs a value computed by this formula, 1 over 1 plus e to the negative"}, {"start": 303.94, "end": 308.90000000000003, "text": " z, that's going to be between 0 and 1."}, {"start": 308.90000000000003, "end": 314.22, "text": " When you take these two equations and put them together, they then give you the logistic"}, {"start": 314.22, "end": 328.18, "text": " regression model f of x, which is equal to g of w x plus b, or equivalently g of z, which"}, {"start": 328.18, "end": 332.86, "text": " is equal to this formula over here."}, {"start": 332.86, "end": 336.78000000000003, "text": " So this is the logistic regression model."}, {"start": 336.78000000000003, "end": 342.72, "text": " And what it does is it inputs a feature or set of features x, and it outputs a number"}, {"start": 342.72, "end": 344.82000000000005, "text": " between 0 and 1."}, {"start": 344.82000000000005, "end": 351.54, "text": " Next, let's take a look at how to interpret the output of logistic regression."}, {"start": 351.54, "end": 355.46000000000004, "text": " We'll return to the tumor classification example."}, {"start": 355.46000000000004, "end": 361.78000000000003, "text": " The way I'd encourage you to think of logistic regression's output is to think of it as outputting"}, {"start": 361.78000000000003, "end": 369.62, "text": " the probability that the class or the label y will be equal to 1, given a certain input"}, {"start": 369.62, "end": 370.62, "text": " x."}, {"start": 370.62, "end": 379.02, "text": " So, for example, in this application where x is the tumor size and y is either 0 or 1,"}, {"start": 379.02, "end": 384.78000000000003, "text": " if you have a patient come in and she has a tumor of a certain size x, and if based"}, {"start": 384.78000000000003, "end": 393.02, "text": " on this input x, the model outputs 0.7, then what that means is that the model is predicting"}, {"start": 393.02, "end": 399.86, "text": " or the model thinks there's a 70% chance that the true label y will be equal to 1 for this"}, {"start": 399.86, "end": 400.86, "text": " patient."}, {"start": 400.86, "end": 407.74, "text": " In other words, the model is telling us that it thinks the patient has a 70% chance of"}, {"start": 407.74, "end": 410.62, "text": " their tumor turning out to be malignant."}, {"start": 410.62, "end": 414.3, "text": " Now, let me ask you a question."}, {"start": 414.3, "end": 416.7, "text": " See if you can get this right."}, {"start": 416.7, "end": 421.34000000000003, "text": " We know that y has to be either 0 or 1."}, {"start": 421.34000000000003, "end": 428.98, "text": " So if y has a 70% chance of being 1, what is the chance that it is 0?"}, {"start": 428.98, "end": 435.66, "text": " So y has got to be either 0 or 1, and thus the probability of it being 0 or 1, these"}, {"start": 435.66, "end": 440.38, "text": " two numbers have to add up to 1 or to 100% chance."}, {"start": 440.38, "end": 447.34000000000003, "text": " So that's why if the chance of y being 1 is 0.7 or 70% chance, then the chance of it being"}, {"start": 447.34000000000003, "end": 452.70000000000005, "text": " 0 has got to be 0.3 or 30% chance."}, {"start": 452.70000000000005, "end": 457.94, "text": " If someday you read research papers or blog posts about logistic regression, sometimes"}, {"start": 457.94, "end": 467.38, "text": " you see this notation that f of x is equal to p of y equals 1, given the input features"}, {"start": 467.38, "end": 471.98, "text": " x and with parameters w and b."}, {"start": 471.98, "end": 478.7, "text": " What the semicolon here is used to denote is just that w and b are parameters that affect"}, {"start": 478.7, "end": 484.56, "text": " this computation of what is the probability of y being equal to 1, given the input feature"}, {"start": 484.56, "end": 486.1, "text": " x."}, {"start": 486.1, "end": 489.70000000000005, "text": " For the purpose of this class, don't worry too much about what this vertical line and"}, {"start": 489.70000000000005, "end": 492.26000000000005, "text": " what the semicolon mean."}, {"start": 492.26000000000005, "end": 497.18, "text": " You don't need to remember or follow any of this mathematical notation for this class."}, {"start": 497.18, "end": 501.40000000000003, "text": " I'm mentioning this only because you may see this in other places."}, {"start": 501.40000000000003, "end": 506.22, "text": " In the optional lab that follows this video, you also get to see how the sigmoid function"}, {"start": 506.22, "end": 508.74, "text": " is implemented in code."}, {"start": 508.74, "end": 514.5400000000001, "text": " You can see a plot that uses the sigmoid function so as to do better on the classification task"}, {"start": 514.54, "end": 517.5799999999999, "text": " that you saw in the previous optional lab."}, {"start": 517.5799999999999, "end": 521.78, "text": " Remember that the code will be provided to you, so you just have to run it."}, {"start": 521.78, "end": 525.9399999999999, "text": " I hope you take a look and get familiar with the code."}, {"start": 525.9399999999999, "end": 528.0999999999999, "text": " So congrats on getting here."}, {"start": 528.0999999999999, "end": 533.6999999999999, "text": " You now know what is the logistic regression model as well as the mathematical formula"}, {"start": 533.6999999999999, "end": 536.66, "text": " that defines logistic regression."}, {"start": 536.66, "end": 542.18, "text": " For a long time, a lot of internet advertising was actually driven by basically a slight"}, {"start": 542.18, "end": 544.8599999999999, "text": " variation of logistic regression."}, {"start": 544.8599999999999, "end": 548.78, "text": " This was very lucrative for some large companies, and this was basically the algorithm that"}, {"start": 548.78, "end": 554.3399999999999, "text": " decided what ad was shown to you and many others on some large websites."}, {"start": 554.3399999999999, "end": 557.66, "text": " Now there's even more to learn about this algorithm."}, {"start": 557.66, "end": 562.66, "text": " In the next video, we'll take a look at the details of logistic regression."}, {"start": 562.66, "end": 568.9, "text": " We'll look at some visualizations and also examine something called the decision boundary."}, {"start": 568.9, "end": 573.9399999999999, "text": " This will give you a few different ways to map the numbers that this model outputs, such"}, {"start": 573.9399999999999, "end": 582.74, "text": " as 0.3 or 0.7 or 0.65, to a prediction of whether y is actually 0 or 1."}, {"start": 582.74, "end": 599.54, "text": " So let's go on to the next video to learn more about logistic regression."}]