3.1 Classification | Motivations  --[Machine Learning | Andrew Ng]
 Welcome to the third week of this course. By the end of this week, you have completed the first course of this specialization. So let's jump in. Last week, you learned about linear regression, which predicts a number. This week, you learned about classification, where your output variable y can take on only one of a small handful of possible values, instead of any number in an infinite range of numbers. It turns out that linear regression is not a good algorithm for classification problems. Let's take a look at why, and this will lead us into a different algorithm called logistic regression, which is one of the most popular and most widely used learning algorithms today. Here are some examples of classification problems. Recall the example of trying to figure out whether an email is spam. So the answer you want to output is going to be either a no or a yes. Another example would be figuring out if an online financial transaction is fraudulent. Fighting online financial fraud is something I once worked on, and it was strangely exhilarating because I knew there were forces out there trying to steal money, and my team's job was to stop them. So the problem is, given a financial transaction, can your learning algorithm figure out is this transaction fraudulent, such as was this credit card stolen? Another example we've touched on before was trying to classify a tumor as malignant versus not. In each of these problems, the variable that you want to predict can only be one of two possible values, no or yes. This type of classification problem, where there are only two possible outputs, is called binary classification, where the word binary refers to there being only two possible classes, or two possible categories. In these problems, I will use the terms class and category relatively interchangeably. They mean basically the same thing. By convention, we can refer to these two classes or categories in a few common ways. We often designate classes as no or yes, or sometimes equivalently, false or true, or very commonly using the numbers 0 or 1, following the common convention in computer science with 0 denoting false and 1 denoting true. I'm usually going to use the numbers 0 and 1 to represent the answer why, because that will fit in most easily with the types of learning algorithms we want to implement. But when we talk about it, we'll often say no or yes, or false or true, as well. One of the terminology is commonly used is to call the false or zero class the negative class and the true or the one class the positive class. For example, for spam classification, an email that is not spam may be referred to as a negative example because the output to the question of is it spam? The output is no or zero. In contrast, an email that is spam might be referred to as a positive training example because the answer to is it spam is yes or true or one. To be clear, negative and positive do not necessarily mean bad versus good or evil versus good. It's just that negative and positive examples are used to convey the concepts of absence or zero or false versus the presence or true or one of something you might be looking for, such as the absence or presence of the spam units or the spam property of an email or the absence of presence of fraudulent activity or absence of presence of malignancy in a tumor. Between non-spam and spam emails, which one you call false or zero and which one you call true or one is a little bit arbitrary. Often either choice could work. So a different engineer might actually swap it around and have the positive class be the presence of a good email or the positive class be the presence of a real financial transaction or a healthy patient. So how do you build a classification algorithm? Here's the example of a training set for classifying if a tumor is malignant. A class one positive class, yes class or benign class zero or negative class. I plotted both the tumor size on the horizontal axis as well as the label Y on the vertical axis. By the way, in week one, when we first talked about classification, this is how we previously visualized it on the number line, except that now we're calling the classes zero and one and plotting them on the vertical axis. Now one thing you could try on this training set is to apply the algorithm you already know linear regression and try to fit a straight line to the data. If you do that, maybe the straight line looks like this, right? And that's your f of X. Linear regression predicts not just the values zero and one, but all numbers between zero and one or even less than zero or greater than one. But here we want to predict categories. One thing you could try is to pick a threshold of say 0.5 so that if the model outputs a value below 0.5, then you predict Y equals zero or not malignant. And if the model outputs a number equal to or greater than 0.5, then predict Y equals one or malignant. Notice that this threshold value 0.5 intersects the best fit straight line at this point. So if you draw this vertical line here, everything to the left ends up with a prediction of Y equals zero and everything on the right ends up with a prediction of Y equals one. Now for this particular data set, it looks like linear regression could do something reasonable. But now let's see what happens if your data set has one more training example. This one way over here on the right. Let's also extend the horizontal axis. Notice that this training example shouldn't really change how you classify the data points. This vertical dividing line that we drew just now still makes sense as the cutoff where two meters smaller than this should be classified as zero and two meters greater than this should be classified as one. But once you've added this extra training example on the right, the best fit line for linear regression will shift over like this. And if you continue using the threshold of 0.5, you now notice that everything to the left of this point is predicted as zero, non-malignant, and everything to the right of this point is predicted to be one or malignant. This isn't what we want, because adding that example way to the right shouldn't change any of our conclusions about how to classify malignant versus benign tumors. But if you try to do this with linear regression, adding this one example, which feels like it shouldn't be changing anything, it ends up with us learning a much worse function for this classification problem. Clearly, when a tumor is large, we want the algorithm to classify it as malignant. So what we just saw was linear regression causes the best fit line when we added one more example to the right to shift over, and thus the dividing line also called the decision boundary to shift over to the right. You learn more about the decision boundary in the next video. You also learn about an algorithm called logistic regression, where the output value of the outcome will always be between 0 and 1, and the algorithm will avoid these problems that we're seeing on the slide. By the way, one thing confusing about the name logistic regression is that even though it has the word regression in it, it's actually used for classification. Don't be confused by the name, which was given for historical reasons. It's actually used to solve binary classification problems where the output label y is either 0 or 1. In the upcoming optional lab, you also get to take a look at what happens when you try to use linear regression for classification. Sometimes you get lucky and it may work, but often it will not work well, which is why I don't use linear regression myself for classification. In the optional lab, you see an interactive plot that attempts to classify between two categories, and you hopefully notice how this often doesn't work very well, which is okay, because that motivates the need for a different model to do classification tasks. So please check out this optional lab, and after that, we'll go on to the next video to look at logistic regression for classification.
[{"start": 0.0, "end": 3.72, "text": " Welcome to the third week of this course."}, {"start": 3.72, "end": 8.4, "text": " By the end of this week, you have completed the first course of this specialization."}, {"start": 8.4, "end": 10.200000000000001, "text": " So let's jump in."}, {"start": 10.200000000000001, "end": 15.14, "text": " Last week, you learned about linear regression, which predicts a number."}, {"start": 15.14, "end": 20.0, "text": " This week, you learned about classification, where your output variable y can take on only"}, {"start": 20.0, "end": 25.3, "text": " one of a small handful of possible values, instead of any number in an infinite range"}, {"start": 25.3, "end": 27.080000000000002, "text": " of numbers."}, {"start": 27.08, "end": 32.96, "text": " It turns out that linear regression is not a good algorithm for classification problems."}, {"start": 32.96, "end": 38.28, "text": " Let's take a look at why, and this will lead us into a different algorithm called logistic"}, {"start": 38.28, "end": 43.68, "text": " regression, which is one of the most popular and most widely used learning algorithms today."}, {"start": 43.68, "end": 47.36, "text": " Here are some examples of classification problems."}, {"start": 47.36, "end": 52.64, "text": " Recall the example of trying to figure out whether an email is spam."}, {"start": 52.64, "end": 58.26, "text": " So the answer you want to output is going to be either a no or a yes."}, {"start": 58.26, "end": 64.8, "text": " Another example would be figuring out if an online financial transaction is fraudulent."}, {"start": 64.8, "end": 70.6, "text": " Fighting online financial fraud is something I once worked on, and it was strangely exhilarating"}, {"start": 70.6, "end": 75.64, "text": " because I knew there were forces out there trying to steal money, and my team's job was"}, {"start": 75.64, "end": 77.58, "text": " to stop them."}, {"start": 77.58, "end": 83.72, "text": " So the problem is, given a financial transaction, can your learning algorithm figure out is"}, {"start": 83.72, "end": 89.75999999999999, "text": " this transaction fraudulent, such as was this credit card stolen?"}, {"start": 89.75999999999999, "end": 95.92, "text": " Another example we've touched on before was trying to classify a tumor as malignant versus"}, {"start": 95.92, "end": 98.0, "text": " not."}, {"start": 98.0, "end": 103.64, "text": " In each of these problems, the variable that you want to predict can only be one of two"}, {"start": 103.64, "end": 106.98, "text": " possible values, no or yes."}, {"start": 106.98, "end": 111.60000000000001, "text": " This type of classification problem, where there are only two possible outputs, is called"}, {"start": 111.60000000000001, "end": 118.28, "text": " binary classification, where the word binary refers to there being only two possible classes,"}, {"start": 118.28, "end": 122.04, "text": " or two possible categories."}, {"start": 122.04, "end": 130.2, "text": " In these problems, I will use the terms class and category relatively interchangeably."}, {"start": 130.2, "end": 132.48000000000002, "text": " They mean basically the same thing."}, {"start": 132.48, "end": 138.89999999999998, "text": " By convention, we can refer to these two classes or categories in a few common ways."}, {"start": 138.89999999999998, "end": 147.07999999999998, "text": " We often designate classes as no or yes, or sometimes equivalently, false or true, or"}, {"start": 147.07999999999998, "end": 154.48, "text": " very commonly using the numbers 0 or 1, following the common convention in computer science"}, {"start": 154.48, "end": 159.04, "text": " with 0 denoting false and 1 denoting true."}, {"start": 159.04, "end": 165.2, "text": " I'm usually going to use the numbers 0 and 1 to represent the answer why, because that"}, {"start": 165.2, "end": 170.48, "text": " will fit in most easily with the types of learning algorithms we want to implement."}, {"start": 170.48, "end": 177.6, "text": " But when we talk about it, we'll often say no or yes, or false or true, as well."}, {"start": 177.6, "end": 183.12, "text": " One of the terminology is commonly used is to call the false or zero class the negative"}, {"start": 183.12, "end": 189.44, "text": " class and the true or the one class the positive class."}, {"start": 189.44, "end": 196.0, "text": " For example, for spam classification, an email that is not spam may be referred to as a negative"}, {"start": 196.0, "end": 200.32, "text": " example because the output to the question of is it spam?"}, {"start": 200.32, "end": 203.72, "text": " The output is no or zero."}, {"start": 203.72, "end": 210.28, "text": " In contrast, an email that is spam might be referred to as a positive training example"}, {"start": 210.28, "end": 216.72, "text": " because the answer to is it spam is yes or true or one."}, {"start": 216.72, "end": 222.08, "text": " To be clear, negative and positive do not necessarily mean bad versus good or evil versus"}, {"start": 222.08, "end": 223.08, "text": " good."}, {"start": 223.08, "end": 227.96, "text": " It's just that negative and positive examples are used to convey the concepts of absence"}, {"start": 227.96, "end": 234.08, "text": " or zero or false versus the presence or true or one of something you might be looking for,"}, {"start": 234.08, "end": 240.8, "text": " such as the absence or presence of the spam units or the spam property of an email or"}, {"start": 240.8, "end": 247.36, "text": " the absence of presence of fraudulent activity or absence of presence of malignancy in a"}, {"start": 247.36, "end": 248.36, "text": " tumor."}, {"start": 248.36, "end": 254.28, "text": " Between non-spam and spam emails, which one you call false or zero and which one you call"}, {"start": 254.28, "end": 258.04, "text": " true or one is a little bit arbitrary."}, {"start": 258.04, "end": 260.52000000000004, "text": " Often either choice could work."}, {"start": 260.52, "end": 264.91999999999996, "text": " So a different engineer might actually swap it around and have the positive class be the"}, {"start": 264.91999999999996, "end": 271.64, "text": " presence of a good email or the positive class be the presence of a real financial transaction"}, {"start": 271.64, "end": 274.64, "text": " or a healthy patient."}, {"start": 274.64, "end": 278.84, "text": " So how do you build a classification algorithm?"}, {"start": 278.84, "end": 283.91999999999996, "text": " Here's the example of a training set for classifying if a tumor is malignant."}, {"start": 283.92, "end": 291.64000000000004, "text": " A class one positive class, yes class or benign class zero or negative class."}, {"start": 291.64000000000004, "end": 297.96000000000004, "text": " I plotted both the tumor size on the horizontal axis as well as the label Y on the vertical"}, {"start": 297.96000000000004, "end": 299.6, "text": " axis."}, {"start": 299.6, "end": 304.96000000000004, "text": " By the way, in week one, when we first talked about classification, this is how we previously"}, {"start": 304.96000000000004, "end": 310.8, "text": " visualized it on the number line, except that now we're calling the classes zero and one"}, {"start": 310.8, "end": 314.8, "text": " and plotting them on the vertical axis."}, {"start": 314.8, "end": 320.0, "text": " Now one thing you could try on this training set is to apply the algorithm you already"}, {"start": 320.0, "end": 325.32, "text": " know linear regression and try to fit a straight line to the data."}, {"start": 325.32, "end": 329.16, "text": " If you do that, maybe the straight line looks like this, right?"}, {"start": 329.16, "end": 332.02, "text": " And that's your f of X."}, {"start": 332.02, "end": 337.68, "text": " Linear regression predicts not just the values zero and one, but all numbers between zero"}, {"start": 337.68, "end": 342.04, "text": " and one or even less than zero or greater than one."}, {"start": 342.04, "end": 346.44, "text": " But here we want to predict categories."}, {"start": 346.44, "end": 354.12, "text": " One thing you could try is to pick a threshold of say 0.5 so that if the model outputs a"}, {"start": 354.12, "end": 361.24, "text": " value below 0.5, then you predict Y equals zero or not malignant."}, {"start": 361.24, "end": 367.88, "text": " And if the model outputs a number equal to or greater than 0.5, then predict Y equals"}, {"start": 367.88, "end": 370.92, "text": " one or malignant."}, {"start": 370.92, "end": 378.64, "text": " Notice that this threshold value 0.5 intersects the best fit straight line at this point."}, {"start": 378.64, "end": 384.46000000000004, "text": " So if you draw this vertical line here, everything to the left ends up with a prediction of Y"}, {"start": 384.46, "end": 391.88, "text": " equals zero and everything on the right ends up with a prediction of Y equals one."}, {"start": 391.88, "end": 396.52, "text": " Now for this particular data set, it looks like linear regression could do something"}, {"start": 396.52, "end": 397.52, "text": " reasonable."}, {"start": 397.52, "end": 403.24, "text": " But now let's see what happens if your data set has one more training example."}, {"start": 403.24, "end": 406.32, "text": " This one way over here on the right."}, {"start": 406.32, "end": 409.96, "text": " Let's also extend the horizontal axis."}, {"start": 409.96, "end": 415.71999999999997, "text": " Notice that this training example shouldn't really change how you classify the data points."}, {"start": 415.71999999999997, "end": 420.28, "text": " This vertical dividing line that we drew just now still makes sense as the cutoff where"}, {"start": 420.28, "end": 425.47999999999996, "text": " two meters smaller than this should be classified as zero and two meters greater than this should"}, {"start": 425.47999999999996, "end": 427.76, "text": " be classified as one."}, {"start": 427.76, "end": 431.64, "text": " But once you've added this extra training example on the right, the best fit line for"}, {"start": 431.64, "end": 436.15999999999997, "text": " linear regression will shift over like this."}, {"start": 436.16, "end": 442.12, "text": " And if you continue using the threshold of 0.5, you now notice that everything to the"}, {"start": 442.12, "end": 449.32000000000005, "text": " left of this point is predicted as zero, non-malignant, and everything to the right of this point"}, {"start": 449.32000000000005, "end": 453.70000000000005, "text": " is predicted to be one or malignant."}, {"start": 453.70000000000005, "end": 459.6, "text": " This isn't what we want, because adding that example way to the right shouldn't change"}, {"start": 459.6, "end": 465.36, "text": " any of our conclusions about how to classify malignant versus benign tumors."}, {"start": 465.36, "end": 470.7, "text": " But if you try to do this with linear regression, adding this one example, which feels like"}, {"start": 470.7, "end": 475.32, "text": " it shouldn't be changing anything, it ends up with us learning a much worse function"}, {"start": 475.32, "end": 477.28000000000003, "text": " for this classification problem."}, {"start": 477.28000000000003, "end": 483.76, "text": " Clearly, when a tumor is large, we want the algorithm to classify it as malignant."}, {"start": 483.76, "end": 490.0, "text": " So what we just saw was linear regression causes the best fit line when we added one"}, {"start": 490.0, "end": 497.16, "text": " more example to the right to shift over, and thus the dividing line also called the decision"}, {"start": 497.16, "end": 501.24, "text": " boundary to shift over to the right."}, {"start": 501.24, "end": 505.74, "text": " You learn more about the decision boundary in the next video."}, {"start": 505.74, "end": 511.36, "text": " You also learn about an algorithm called logistic regression, where the output value of the"}, {"start": 511.36, "end": 517.36, "text": " outcome will always be between 0 and 1, and the algorithm will avoid these problems that"}, {"start": 517.36, "end": 519.12, "text": " we're seeing on the slide."}, {"start": 519.12, "end": 524.68, "text": " By the way, one thing confusing about the name logistic regression is that even though"}, {"start": 524.68, "end": 529.68, "text": " it has the word regression in it, it's actually used for classification."}, {"start": 529.68, "end": 533.92, "text": " Don't be confused by the name, which was given for historical reasons."}, {"start": 533.92, "end": 539.04, "text": " It's actually used to solve binary classification problems where the output label y is either"}, {"start": 539.04, "end": 541.84, "text": " 0 or 1."}, {"start": 541.84, "end": 546.28, "text": " In the upcoming optional lab, you also get to take a look at what happens when you try"}, {"start": 546.28, "end": 551.24, "text": " to use linear regression for classification."}, {"start": 551.24, "end": 557.16, "text": " Sometimes you get lucky and it may work, but often it will not work well, which is why"}, {"start": 557.16, "end": 561.8, "text": " I don't use linear regression myself for classification."}, {"start": 561.8, "end": 566.52, "text": " In the optional lab, you see an interactive plot that attempts to classify between two"}, {"start": 566.52, "end": 573.64, "text": " categories, and you hopefully notice how this often doesn't work very well, which is okay,"}, {"start": 573.64, "end": 578.6, "text": " because that motivates the need for a different model to do classification tasks."}, {"start": 578.6, "end": 583.84, "text": " So please check out this optional lab, and after that, we'll go on to the next video"}, {"start": 583.84, "end": 604.24, "text": " to look at logistic regression for classification."}]