6.6 Bias and variance |Establishing a baseline level of performance -[Machine Learning | Andrew Ng]
 Let's look at some concrete numbers for what JTrain and JCV might be and see how you can judge if a learning algorithm has high bias or high variance. And for the examples in this video, I'm going to use as a running example the application of speech recognition, which is something I've worked on multiple times over the years. Let's take a look. A lot of users doing web search on a mobile phone will use speech recognition rather than type on the tiny keyboards on their phones because speaking to a phone is often faster than typing. And so typical audio that a web search engine would get would be like this. What is today's weather? Or like this. Coffee shops near me. And there's the job of the speech recognition algorithm to output the transcripts, what is today's weather or coffee shops near me. Now if you were to train a speech recognition system and measure the training error, and the training error means what's the percentage of audio clips in your training set that the algorithm does not transcribe correctly in its entirety. Let's say the training error for this data set is 10.8%, meaning that it transcribes it perfectly for 89.2% of your training set, but makes some mistake in 10.8% of your training set. And if you were to also measure your speech recognition algorithm's performance on a separate cross-validation set, let's say it gets 14.8% error. So if you were to look at these numbers, it looks like the training error is really high, it got 10% wrong, and then the cross-validation error is higher, but getting 10% of even your training set wrong, that seems pretty high. And it seems like that 10% error would lead you to conclude it has to have high bias because it's not doing well on your training set. But it turns out that when analyzing speech recognition, it's useful to also measure one other thing, which is what is the human level of performance. In other words, how well can even humans transcribe speech accurately from these audio clips? And concretely, let's say that you measure how well fluent speakers can transcribe audio clips and you find that it is 10.6%, and you find that human level performance achieves 10.6% error. Why is human level error so high? It turns out that for web search, there are a lot of audio clips that sound like this. I'm going to navigate to somewhere on the... And there's a lot of noisy audio where really no one can accurately transcribe what was said because of the noise in the audio. And if even a human makes 10.6% error, then it seems difficult to expect a learning algorithm to do much better. And so in order to judge if the training error is high, it turns out to be more useful to see if the training error is much higher than a human level of performance. And in this example, it does just 0.2% worse than humans. Given that humans are actually really good at recognizing speech, I think if I can build a speech recognition system that achieves 10.6% error matching human performance, I'd be pretty happy. So it's just doing a little bit worse than humans. But in contrast, the gap or the difference between JCV and JTrain is much larger. There's actually a 4% gap there. So whereas previously we had said maybe 10.8% error means this is high bias, when we benchmark it to human level performance, we see that the algorithm is actually doing quite well on the training set. But the bigger problem is the cross validation error is much higher than the training error, which is why I would conclude that this algorithm actually has more of a variance problem than a bias problem. So it turns out when judging if the training error is high, it's often useful to establish a baseline level of performance. And by baseline level of performance, I mean, what is the level of error you can reasonably hope your learning algorithm to eventually get to? And one common way to establish a baseline level of performance is to measure how well humans can do on this task. Because humans are really good at understanding speech data or processing images or understanding text, human level performance is often a good benchmark when you are using unstructured data such as audio, images or text. Another way to estimate a baseline level of performance is if there's some competing algorithm, maybe a previous implementation that someone else implemented or even a competitor's algorithm to establish a baseline level of performance, if you can measure that. Or sometimes you might guess based on prior experience. And if you have access to this baseline level of performance, that is, what is the level of error you can reasonably hope to get to or what is the design level of performance that you want your algorithm to get to? Then when judging, if an algorithm has high bias or variance, you would look at the baseline level of performance and the training error and the cross validation error. And the two key quantities to measure are then what is the difference between training error and the baseline level that you hope to get to? So this is 0.2 and if this is large, then you would say you have a high bias problem and you would then also look at this gap between your training error and your cross validation error. And if this is high, then you would conclude you have a high variance problem. And that's why in this example, we concluded we have a high variance problem. Whereas, let's look at the second example. If the baseline level of performance, that is, human level performance and training error and cross validation error look like this, then this first gap is 4.4%. And so there's actually a big gap. The training error is much higher than what humans can do and what we hope to get to. Whereas the cross validation error is just a little bit bigger than the training error. And so if your training error and cross validation error look like this, I would say this algorithm has high bias. And so by looking at these numbers, training error and cross validation error, you can get a sense, intuitively or informally, of the degree to which your algorithm has a high bias or high variance problem. And so just to summarize, this gap between these first two numbers gives you a sense of whether you have a high bias problem and the gap between these two numbers give you a sense of whether you have a high variance problem. And sometimes the baseline level of performance could be 0%. If your goal is to achieve perfect performance, then the baseline level of performance could be 0%. But for some applications, like the speech recognition application, where some audio is just noisy, then the baseline level of performance could be much higher than 0. And the method described on this slide will give you a better read in terms of whether your algorithm suffers from bias or variance. And by the way, it is possible for your algorithm to have high bias and high variance. So concretely, if you get numbers like these, then the gap between the baseline and the training error is large, that would be 4.6%. And the gap between training error and cross validation error is also large, this is 4.2%. And so if it looks like this, you would conclude that your algorithm has high bias and high variance, although hopefully this won't happen that often for your learning applications. So to summarize, we've seen that looking at whether your training error is large is a way to tell if your algorithm has high bias. But on applications where the data is sometimes just noisy and is infeasible or unrealistic to ever expect to get to zero error, then it's useful to establish this baseline level of performance so that rather than just asking, is my training error large? You can ask, is my training error large relative to what I hope I could get to eventually, such as is my training error large relative to what humans can do on the task? And that gives you a more accurate read on how far away you are in terms of a training error from where you hope to get to. And then similarly, looking at whether your cross validation error is much larger than your training error gives you a sense of whether or not your algorithm may have a high variance problem as well. And in practice, this is how I often will look at these numbers to judge if my learning algorithm has a high bias or high variance problem. Now to further hone our intuition about how a learning algorithm is doing, there's one other thing that I found useful to think about, which is the learning curve. Let's take a look at what that means in the next video.
[{"start": 0.0, "end": 8.0, "text": " Let's look at some concrete numbers for what JTrain and JCV might be and see how you can"}, {"start": 8.0, "end": 12.280000000000001, "text": " judge if a learning algorithm has high bias or high variance."}, {"start": 12.280000000000001, "end": 17.36, "text": " And for the examples in this video, I'm going to use as a running example the application"}, {"start": 17.36, "end": 22.2, "text": " of speech recognition, which is something I've worked on multiple times over the years."}, {"start": 22.2, "end": 23.2, "text": " Let's take a look."}, {"start": 23.2, "end": 29.64, "text": " A lot of users doing web search on a mobile phone will use speech recognition rather than"}, {"start": 29.64, "end": 34.44, "text": " type on the tiny keyboards on their phones because speaking to a phone is often faster"}, {"start": 34.44, "end": 36.44, "text": " than typing."}, {"start": 36.44, "end": 42.52, "text": " And so typical audio that a web search engine would get would be like this."}, {"start": 42.52, "end": 44.44, "text": " What is today's weather?"}, {"start": 44.44, "end": 45.44, "text": " Or like this."}, {"start": 45.44, "end": 47.16, "text": " Coffee shops near me."}, {"start": 47.16, "end": 51.0, "text": " And there's the job of the speech recognition algorithm to output the transcripts, what"}, {"start": 51.0, "end": 54.72, "text": " is today's weather or coffee shops near me."}, {"start": 54.72, "end": 61.04, "text": " Now if you were to train a speech recognition system and measure the training error, and"}, {"start": 61.04, "end": 66.76, "text": " the training error means what's the percentage of audio clips in your training set that the"}, {"start": 66.76, "end": 70.88, "text": " algorithm does not transcribe correctly in its entirety."}, {"start": 70.88, "end": 77.36, "text": " Let's say the training error for this data set is 10.8%, meaning that it transcribes"}, {"start": 77.36, "end": 85.28, "text": " it perfectly for 89.2% of your training set, but makes some mistake in 10.8% of your training"}, {"start": 85.28, "end": 86.76, "text": " set."}, {"start": 86.76, "end": 91.8, "text": " And if you were to also measure your speech recognition algorithm's performance on a separate"}, {"start": 91.8, "end": 97.96000000000001, "text": " cross-validation set, let's say it gets 14.8% error."}, {"start": 97.96000000000001, "end": 103.24, "text": " So if you were to look at these numbers, it looks like the training error is really high,"}, {"start": 103.24, "end": 109.52, "text": " it got 10% wrong, and then the cross-validation error is higher, but getting 10% of even your"}, {"start": 109.52, "end": 112.8, "text": " training set wrong, that seems pretty high."}, {"start": 112.8, "end": 117.6, "text": " And it seems like that 10% error would lead you to conclude it has to have high bias because"}, {"start": 117.6, "end": 119.75999999999999, "text": " it's not doing well on your training set."}, {"start": 119.75999999999999, "end": 125.47999999999999, "text": " But it turns out that when analyzing speech recognition, it's useful to also measure one"}, {"start": 125.47999999999999, "end": 130.72, "text": " other thing, which is what is the human level of performance."}, {"start": 130.72, "end": 136.72, "text": " In other words, how well can even humans transcribe speech accurately from these audio clips?"}, {"start": 136.72, "end": 144.24, "text": " And concretely, let's say that you measure how well fluent speakers can transcribe audio"}, {"start": 144.24, "end": 150.32, "text": " clips and you find that it is 10.6%, and you find that human level performance achieves"}, {"start": 150.32, "end": 153.36, "text": " 10.6% error."}, {"start": 153.36, "end": 157.16, "text": " Why is human level error so high?"}, {"start": 157.16, "end": 163.0, "text": " It turns out that for web search, there are a lot of audio clips that sound like this."}, {"start": 163.0, "end": 166.56, "text": " I'm going to navigate to somewhere on the..."}, {"start": 166.56, "end": 171.6, "text": " And there's a lot of noisy audio where really no one can accurately transcribe what was"}, {"start": 171.6, "end": 175.01999999999998, "text": " said because of the noise in the audio."}, {"start": 175.01999999999998, "end": 182.48, "text": " And if even a human makes 10.6% error, then it seems difficult to expect a learning algorithm"}, {"start": 182.48, "end": 184.0, "text": " to do much better."}, {"start": 184.0, "end": 190.08, "text": " And so in order to judge if the training error is high, it turns out to be more useful to"}, {"start": 190.08, "end": 195.88, "text": " see if the training error is much higher than a human level of performance."}, {"start": 195.88, "end": 200.12, "text": " And in this example, it does just 0.2% worse than humans."}, {"start": 200.12, "end": 203.64, "text": " Given that humans are actually really good at recognizing speech, I think if I can build"}, {"start": 203.64, "end": 208.76, "text": " a speech recognition system that achieves 10.6% error matching human performance, I'd"}, {"start": 208.76, "end": 209.76, "text": " be pretty happy."}, {"start": 209.76, "end": 213.0, "text": " So it's just doing a little bit worse than humans."}, {"start": 213.0, "end": 219.56, "text": " But in contrast, the gap or the difference between JCV and JTrain is much larger."}, {"start": 219.56, "end": 222.48, "text": " There's actually a 4% gap there."}, {"start": 222.48, "end": 230.88, "text": " So whereas previously we had said maybe 10.8% error means this is high bias, when we benchmark"}, {"start": 230.88, "end": 235.12, "text": " it to human level performance, we see that the algorithm is actually doing quite well"}, {"start": 235.12, "end": 236.88, "text": " on the training set."}, {"start": 236.88, "end": 243.4, "text": " But the bigger problem is the cross validation error is much higher than the training error,"}, {"start": 243.4, "end": 249.04, "text": " which is why I would conclude that this algorithm actually has more of a variance problem than"}, {"start": 249.04, "end": 251.0, "text": " a bias problem."}, {"start": 251.0, "end": 259.76, "text": " So it turns out when judging if the training error is high, it's often useful to establish"}, {"start": 259.76, "end": 262.04, "text": " a baseline level of performance."}, {"start": 262.04, "end": 266.52, "text": " And by baseline level of performance, I mean, what is the level of error you can reasonably"}, {"start": 266.52, "end": 271.2, "text": " hope your learning algorithm to eventually get to?"}, {"start": 271.2, "end": 276.88, "text": " And one common way to establish a baseline level of performance is to measure how well"}, {"start": 276.88, "end": 279.46, "text": " humans can do on this task."}, {"start": 279.46, "end": 284.71999999999997, "text": " Because humans are really good at understanding speech data or processing images or understanding"}, {"start": 284.71999999999997, "end": 291.03999999999996, "text": " text, human level performance is often a good benchmark when you are using unstructured data"}, {"start": 291.03999999999996, "end": 294.78, "text": " such as audio, images or text."}, {"start": 294.78, "end": 300.15999999999997, "text": " Another way to estimate a baseline level of performance is if there's some competing algorithm,"}, {"start": 300.15999999999997, "end": 306.23999999999995, "text": " maybe a previous implementation that someone else implemented or even a competitor's algorithm"}, {"start": 306.23999999999995, "end": 311.4, "text": " to establish a baseline level of performance, if you can measure that."}, {"start": 311.4, "end": 316.96, "text": " Or sometimes you might guess based on prior experience."}, {"start": 316.96, "end": 322.76, "text": " And if you have access to this baseline level of performance, that is, what is the level"}, {"start": 322.76, "end": 327.12, "text": " of error you can reasonably hope to get to or what is the design level of performance"}, {"start": 327.12, "end": 329.56, "text": " that you want your algorithm to get to?"}, {"start": 329.56, "end": 335.59999999999997, "text": " Then when judging, if an algorithm has high bias or variance, you would look at the baseline"}, {"start": 335.59999999999997, "end": 340.36, "text": " level of performance and the training error and the cross validation error."}, {"start": 340.36, "end": 346.2, "text": " And the two key quantities to measure are then what is the difference between training"}, {"start": 346.2, "end": 350.4, "text": " error and the baseline level that you hope to get to?"}, {"start": 350.4, "end": 356.58, "text": " So this is 0.2 and if this is large, then you would say you have a high bias problem"}, {"start": 356.58, "end": 362.28, "text": " and you would then also look at this gap between your training error and your cross validation"}, {"start": 362.28, "end": 363.28, "text": " error."}, {"start": 363.28, "end": 367.47999999999996, "text": " And if this is high, then you would conclude you have a high variance problem."}, {"start": 367.47999999999996, "end": 372.64, "text": " And that's why in this example, we concluded we have a high variance problem."}, {"start": 372.64, "end": 376.47999999999996, "text": " Whereas, let's look at the second example."}, {"start": 376.48, "end": 381.08000000000004, "text": " If the baseline level of performance, that is, human level performance and training error"}, {"start": 381.08000000000004, "end": 388.44, "text": " and cross validation error look like this, then this first gap is 4.4%."}, {"start": 388.44, "end": 389.76, "text": " And so there's actually a big gap."}, {"start": 389.76, "end": 394.8, "text": " The training error is much higher than what humans can do and what we hope to get to."}, {"start": 394.8, "end": 399.12, "text": " Whereas the cross validation error is just a little bit bigger than the training error."}, {"start": 399.12, "end": 404.04, "text": " And so if your training error and cross validation error look like this, I would say this algorithm"}, {"start": 404.04, "end": 406.72, "text": " has high bias."}, {"start": 406.72, "end": 411.32, "text": " And so by looking at these numbers, training error and cross validation error, you can"}, {"start": 411.32, "end": 417.04, "text": " get a sense, intuitively or informally, of the degree to which your algorithm has a high"}, {"start": 417.04, "end": 419.92, "text": " bias or high variance problem."}, {"start": 419.92, "end": 425.74, "text": " And so just to summarize, this gap between these first two numbers gives you a sense"}, {"start": 425.74, "end": 431.72, "text": " of whether you have a high bias problem and the gap between these two numbers give you"}, {"start": 431.72, "end": 435.32000000000005, "text": " a sense of whether you have a high variance problem."}, {"start": 435.32000000000005, "end": 438.48, "text": " And sometimes the baseline level of performance could be 0%."}, {"start": 438.48, "end": 442.64000000000004, "text": " If your goal is to achieve perfect performance, then the baseline level of performance could"}, {"start": 442.64000000000004, "end": 444.48, "text": " be 0%."}, {"start": 444.48, "end": 448.32000000000005, "text": " But for some applications, like the speech recognition application, where some audio"}, {"start": 448.32000000000005, "end": 452.84000000000003, "text": " is just noisy, then the baseline level of performance could be much higher than 0."}, {"start": 452.84000000000003, "end": 457.24, "text": " And the method described on this slide will give you a better read in terms of whether"}, {"start": 457.24, "end": 460.36, "text": " your algorithm suffers from bias or variance."}, {"start": 460.36, "end": 465.72, "text": " And by the way, it is possible for your algorithm to have high bias and high variance."}, {"start": 465.72, "end": 472.64, "text": " So concretely, if you get numbers like these, then the gap between the baseline and the"}, {"start": 472.64, "end": 477.52000000000004, "text": " training error is large, that would be 4.6%."}, {"start": 477.52000000000004, "end": 483.44, "text": " And the gap between training error and cross validation error is also large, this is 4.2%."}, {"start": 483.44, "end": 487.68, "text": " And so if it looks like this, you would conclude that your algorithm has high bias and high"}, {"start": 487.68, "end": 493.24, "text": " variance, although hopefully this won't happen that often for your learning applications."}, {"start": 493.24, "end": 498.0, "text": " So to summarize, we've seen that looking at whether your training error is large is a"}, {"start": 498.0, "end": 501.44, "text": " way to tell if your algorithm has high bias."}, {"start": 501.44, "end": 507.36, "text": " But on applications where the data is sometimes just noisy and is infeasible or unrealistic"}, {"start": 507.36, "end": 512.6800000000001, "text": " to ever expect to get to zero error, then it's useful to establish this baseline level"}, {"start": 512.6800000000001, "end": 516.5600000000001, "text": " of performance so that rather than just asking, is my training error large?"}, {"start": 516.56, "end": 522.1199999999999, "text": " You can ask, is my training error large relative to what I hope I could get to eventually,"}, {"start": 522.1199999999999, "end": 526.64, "text": " such as is my training error large relative to what humans can do on the task?"}, {"start": 526.64, "end": 531.3199999999999, "text": " And that gives you a more accurate read on how far away you are in terms of a training"}, {"start": 531.3199999999999, "end": 534.28, "text": " error from where you hope to get to."}, {"start": 534.28, "end": 538.0, "text": " And then similarly, looking at whether your cross validation error is much larger than"}, {"start": 538.0, "end": 543.1199999999999, "text": " your training error gives you a sense of whether or not your algorithm may have a high variance"}, {"start": 543.1199999999999, "end": 545.04, "text": " problem as well."}, {"start": 545.04, "end": 549.04, "text": " And in practice, this is how I often will look at these numbers to judge if my learning"}, {"start": 549.04, "end": 552.68, "text": " algorithm has a high bias or high variance problem."}, {"start": 552.68, "end": 557.52, "text": " Now to further hone our intuition about how a learning algorithm is doing, there's one"}, {"start": 557.52, "end": 563.1999999999999, "text": " other thing that I found useful to think about, which is the learning curve."}, {"start": 563.2, "end": 576.08, "text": " Let's take a look at what that means in the next video."}]