4.2 Neural Networks Intuition |Neurons and the brain --[Machine Learning | Andrew Ng]
 When neural networks were first invented many decades ago, the original motivation was the right software that could mimic how the human brain or how the biological brain learns and thinks. And even though today, neural networks, sometimes also called artificial neural networks, have become very different than how any of us might think about how the brain actually works and learns, some of the biological motivation still remains in the way we think about artificial neural networks or computer neural networks today. So let's start by taking a look at how the brain works and how that relates to neural networks. The human brain, or maybe more generally the biological brain, demonstrates a higher level or more capable level of intelligence than anything else we've been able to build so far. And so neural networks have started with the motivation of trying to build software to mimic the brain. Work in neural networks had started back in the 1950s and then it fell out of favor for a while. Then in the 1980s and early 1990s, they gained in popularity again and showed tremendous traction in some applications like handwritten digit recognition, which were used even back then to read postal codes for routing mail and for reading dollar figures in handwritten checks. But then it fell out of favor again in the late 1990s and it was from about 2005 that it enjoyed a resurgence and also became maybe rebranded a little bit with deep learning. One of the things that surprised me back then was deep learning and neural networks meant very similar things, but I maybe underappreciated at the time that the term deep learning just sounds much better because it's deep and it's learning. And so that turned out to be the brand that took off in the last decade or decade and a half. And since then, neural networks have revolutionized application area after application area. I think the first application area that modern neural networks or deep learning had a huge impact on was probably speech recognition, where we started to see much better speech recognition systems due to modern deep learning and authors such as Lee Dang and Jeff Hinton were instrumental to this. And then it started to make inroads into computer vision. And sometimes people still speak of the ImageNet moment in 2012 and that was maybe a bigger splash where it then caught broader imagination and had a big impact on computer vision. Then the next few years, it made its inroads into text or into natural language processing and so on and so forth. And now neural networks are used in everything from climate change to medical imaging to online advertising to product recommendations and really lots of application areas of machine learning now use neural networks. Even though today's neural networks have almost nothing to do with how the brain learns, there was the early motivation of trying to build software to mimic the brain. So how does the brain work? Here's a diagram illustrating what neurons in a brain looks like. All of human thought is from neurons like these in your brain and mind sending electrical impulses and sometimes forming new connections with other neurons. Given a neuron like this one, it has a number of inputs where it receives electrical impulses from other neurons. And then this neuron that I've circled carries out some computations and will then send this output to other neurons via these electrical impulses. And this upper neurons output in turn becomes the input to this neuron down below, which again, advocates inputs from multiple other neurons to then maybe send his own output to yet other neurons. And this is the stuff of which human thought is made. Here's a simplified diagram of a biological neuron. A neuron comprises a cell body shown here on the left. And if you have taken a class in biology, you may recognize this to be the nucleus of the neuron. And as we saw on the previous slide, the neuron has different inputs. And in a biological neuron, the input wires are called the dendrites. And it then occasionally sends electrical impulses to other neurons via the output wire, which is called the axon. Don't worry about these biological terms. If you saw them in a biology class, you may remember them, but you don't really need to memorize any of these terms for the purpose of building artificial neural networks. But this biological neuron may then send electrical impulses that becomes the input to another neuron. So the artificial neural network uses a very simplified mathematical model of what a biological neuron does. And I'm going to draw a little circle here to denote a single neuron. And what a neuron does is it takes some inputs, one or more inputs, which are just numbers, and it does some computation and it outputs some other number, which then could be an input to a second neuron shown here on the right. When you're building an artificial neural network or a deep learning algorithm, rather than building one neuron at a time, you often want to simulate many such neurons at the same time. And so when in this diagram, I'm drawing three neurons, and what these neurons do collectively is input a few numbers, carry out some computation, and output some other numbers. Now at this point, I'd like to give one big caveat, which is that even though I made a loose analogy between biological neurons and artificial neurons, I think that today we have almost no idea how the human brain works. In fact, every few years, neuroscientists make some fundamental breakthrough about how the brain works, and I think we'll continue to do so for the foreseeable future. And that to me is a sign that there are many breakthroughs that are yet to be discovered about how the brain actually works, and thus attempts to blindly mimic what we know of the human brain today, which is frankly very little, probably won't get us that far to our building role intelligence, certainly not with our current level of knowledge in neuroscience. Having said that, even with these extremely simplified models of a neuron, which we'll talk about, we'll be able to build really powerful deep learning algorithms. And so as you go deeper into neural networks and into deep learning, even though the origins were biologically motivated, don't take the biological motivation too seriously. In fact, those of us that do research in deep learning have shifted away from looking to biological motivation that much, but instead are just using engineering principles to figure out how to build algorithms that are more effective. But I think it might still be fun to speculate and think about how biological neurons work every now and then. The ideas of neural networks have been around for many decades. So a few people have asked me, hey, Andrew, why now? Why is it that only in the last handful of years that neural networks have really taken off? This is a picture I draw for them when I'm asked that question, and that maybe you could draw for others as well if they ask you that question. Let me plot on the horizontal axis, the amount of data you have for a problem, and on the vertical axis, the performance or the accuracy of a learning algorithm applied to that problem. Over the last couple of decades, with the rise of the internet, the rise of mobile phones, the digitalization of our society, the amount of data we have for a lot of applications has steadily marched to the right. A lot of records that used to be on paper, such as if you order something, rather than it being on a piece of paper, that's much more likely to be a digital record. Or health record, if you see a doctor, it's much more likely to be digital now compared to on pieces of paper. And so in many application areas, the amount of digital data has exploded. And what we saw was with traditional machine learning algorithms, such as logistic regression and linear regression, even as you fed those algorithms more data, it was very difficult to get the performance to keep on going up. So it was as if the traditional learning algorithms like linear regression and logistic regression, they just weren't able to scale with the amount of data we could now feed it. And they weren't able to take effective advantage of all this data we had for different applications. And what AI researchers started to observe was that if you were to train a small neural network on this data set, then the performance maybe looks like this. And if you were to train a medium sized neural network, meaning one with more neurons in it, this performance may look like that. And if you were to train a very large neural network, meaning one with a lot of these artificial neurons, then for some applications, the performance would just keep on going up. And so this meant two things. It meant that for a certain class of applications where you do have a lot of data, sometimes you hear the term big data toss around, if you're able to train a very large neural network to take advantage of that huge amount of data you have, then you could attain performance on anything ranging from speech recognition to image recognition to natural language processing applications and many more that just were not possible with earlier generations of learning algorithms. And this caused deep learning algorithms to take off. And this too is why faster computer processes, including the rise of GPUs or graphics processor units, this is hardware originally designed to generate nice looking computer graphics, but turned out to be really powerful for deep learning as well. That was also a major force in allowing deep learning algorithms to become what it is today. That's how neural networks got started as well as why they took off so quickly in the last several years. Let's now dive more deeply into the details of how a neural network actually works. Please go on to the next video.
[{"start": 0.0, "end": 5.64, "text": " When neural networks were first invented many decades ago, the original motivation was the"}, {"start": 5.64, "end": 11.64, "text": " right software that could mimic how the human brain or how the biological brain learns and"}, {"start": 11.64, "end": 12.64, "text": " thinks."}, {"start": 12.64, "end": 17.68, "text": " And even though today, neural networks, sometimes also called artificial neural networks, have"}, {"start": 17.68, "end": 22.44, "text": " become very different than how any of us might think about how the brain actually works and"}, {"start": 22.44, "end": 28.32, "text": " learns, some of the biological motivation still remains in the way we think about artificial"}, {"start": 28.32, "end": 31.28, "text": " neural networks or computer neural networks today."}, {"start": 31.28, "end": 36.28, "text": " So let's start by taking a look at how the brain works and how that relates to neural"}, {"start": 36.28, "end": 37.28, "text": " networks."}, {"start": 37.28, "end": 43.24, "text": " The human brain, or maybe more generally the biological brain, demonstrates a higher level"}, {"start": 43.24, "end": 47.6, "text": " or more capable level of intelligence than anything else we've been able to build so"}, {"start": 47.6, "end": 48.6, "text": " far."}, {"start": 48.6, "end": 53.92, "text": " And so neural networks have started with the motivation of trying to build software to"}, {"start": 53.92, "end": 56.08, "text": " mimic the brain."}, {"start": 56.08, "end": 61.56, "text": " Work in neural networks had started back in the 1950s and then it fell out of favor for"}, {"start": 61.56, "end": 62.879999999999995, "text": " a while."}, {"start": 62.879999999999995, "end": 69.32, "text": " Then in the 1980s and early 1990s, they gained in popularity again and showed tremendous"}, {"start": 69.32, "end": 75.0, "text": " traction in some applications like handwritten digit recognition, which were used even back"}, {"start": 75.0, "end": 81.48, "text": " then to read postal codes for routing mail and for reading dollar figures in handwritten"}, {"start": 81.48, "end": 82.48, "text": " checks."}, {"start": 82.48, "end": 90.04, "text": " But then it fell out of favor again in the late 1990s and it was from about 2005 that"}, {"start": 90.04, "end": 97.64, "text": " it enjoyed a resurgence and also became maybe rebranded a little bit with deep learning."}, {"start": 97.64, "end": 102.44, "text": " One of the things that surprised me back then was deep learning and neural networks meant"}, {"start": 102.44, "end": 108.4, "text": " very similar things, but I maybe underappreciated at the time that the term deep learning just"}, {"start": 108.4, "end": 112.28, "text": " sounds much better because it's deep and it's learning."}, {"start": 112.28, "end": 116.52, "text": " And so that turned out to be the brand that took off in the last decade or decade and"}, {"start": 116.52, "end": 117.52, "text": " a half."}, {"start": 117.52, "end": 123.28, "text": " And since then, neural networks have revolutionized application area after application area."}, {"start": 123.28, "end": 127.96000000000001, "text": " I think the first application area that modern neural networks or deep learning had a huge"}, {"start": 127.96000000000001, "end": 133.0, "text": " impact on was probably speech recognition, where we started to see much better speech"}, {"start": 133.0, "end": 138.44, "text": " recognition systems due to modern deep learning and authors such as Lee Dang and Jeff Hinton"}, {"start": 138.44, "end": 141.08, "text": " were instrumental to this."}, {"start": 141.08, "end": 145.32000000000002, "text": " And then it started to make inroads into computer vision."}, {"start": 145.32000000000002, "end": 151.9, "text": " And sometimes people still speak of the ImageNet moment in 2012 and that was maybe a bigger"}, {"start": 151.9, "end": 158.52, "text": " splash where it then caught broader imagination and had a big impact on computer vision."}, {"start": 158.52, "end": 163.84, "text": " Then the next few years, it made its inroads into text or into natural language processing"}, {"start": 163.84, "end": 165.04000000000002, "text": " and so on and so forth."}, {"start": 165.04000000000002, "end": 170.0, "text": " And now neural networks are used in everything from climate change to medical imaging to"}, {"start": 170.0, "end": 175.36, "text": " online advertising to product recommendations and really lots of application areas of machine"}, {"start": 175.36, "end": 178.08, "text": " learning now use neural networks."}, {"start": 178.08, "end": 184.08, "text": " Even though today's neural networks have almost nothing to do with how the brain learns, there"}, {"start": 184.08, "end": 189.52, "text": " was the early motivation of trying to build software to mimic the brain."}, {"start": 189.52, "end": 192.08, "text": " So how does the brain work?"}, {"start": 192.08, "end": 197.0, "text": " Here's a diagram illustrating what neurons in a brain looks like."}, {"start": 197.0, "end": 202.84, "text": " All of human thought is from neurons like these in your brain and mind sending electrical"}, {"start": 202.84, "end": 207.6, "text": " impulses and sometimes forming new connections with other neurons."}, {"start": 207.6, "end": 214.24, "text": " Given a neuron like this one, it has a number of inputs where it receives electrical impulses"}, {"start": 214.24, "end": 216.2, "text": " from other neurons."}, {"start": 216.2, "end": 222.08, "text": " And then this neuron that I've circled carries out some computations and will then send this"}, {"start": 222.08, "end": 227.56, "text": " output to other neurons via these electrical impulses."}, {"start": 227.56, "end": 233.60000000000002, "text": " And this upper neurons output in turn becomes the input to this neuron down below, which"}, {"start": 233.60000000000002, "end": 238.64000000000001, "text": " again, advocates inputs from multiple other neurons to then maybe send his own output"}, {"start": 238.64000000000001, "end": 240.4, "text": " to yet other neurons."}, {"start": 240.4, "end": 245.24, "text": " And this is the stuff of which human thought is made."}, {"start": 245.24, "end": 249.96, "text": " Here's a simplified diagram of a biological neuron."}, {"start": 249.96, "end": 254.72, "text": " A neuron comprises a cell body shown here on the left."}, {"start": 254.72, "end": 260.72, "text": " And if you have taken a class in biology, you may recognize this to be the nucleus of"}, {"start": 260.72, "end": 262.40000000000003, "text": " the neuron."}, {"start": 262.40000000000003, "end": 267.24, "text": " And as we saw on the previous slide, the neuron has different inputs."}, {"start": 267.24, "end": 273.44, "text": " And in a biological neuron, the input wires are called the dendrites."}, {"start": 273.44, "end": 278.92, "text": " And it then occasionally sends electrical impulses to other neurons via the output wire,"}, {"start": 278.92, "end": 281.04, "text": " which is called the axon."}, {"start": 281.04, "end": 283.0, "text": " Don't worry about these biological terms."}, {"start": 283.0, "end": 287.36, "text": " If you saw them in a biology class, you may remember them, but you don't really need to"}, {"start": 287.36, "end": 292.92, "text": " memorize any of these terms for the purpose of building artificial neural networks."}, {"start": 292.92, "end": 298.78000000000003, "text": " But this biological neuron may then send electrical impulses that becomes the input to another"}, {"start": 298.78000000000003, "end": 300.72, "text": " neuron."}, {"start": 300.72, "end": 308.32, "text": " So the artificial neural network uses a very simplified mathematical model of what a biological"}, {"start": 308.32, "end": 310.2, "text": " neuron does."}, {"start": 310.2, "end": 316.96, "text": " And I'm going to draw a little circle here to denote a single neuron."}, {"start": 316.96, "end": 324.44, "text": " And what a neuron does is it takes some inputs, one or more inputs, which are just numbers,"}, {"start": 324.44, "end": 329.8, "text": " and it does some computation and it outputs some other number, which then could be an"}, {"start": 329.8, "end": 334.84, "text": " input to a second neuron shown here on the right."}, {"start": 334.84, "end": 339.0, "text": " When you're building an artificial neural network or a deep learning algorithm, rather"}, {"start": 339.0, "end": 345.44, "text": " than building one neuron at a time, you often want to simulate many such neurons at the"}, {"start": 345.44, "end": 346.79999999999995, "text": " same time."}, {"start": 346.79999999999995, "end": 355.28, "text": " And so when in this diagram, I'm drawing three neurons, and what these neurons do collectively"}, {"start": 355.28, "end": 362.47999999999996, "text": " is input a few numbers, carry out some computation, and output some other numbers."}, {"start": 362.48, "end": 367.68, "text": " Now at this point, I'd like to give one big caveat, which is that even though I made a"}, {"start": 367.68, "end": 373.40000000000003, "text": " loose analogy between biological neurons and artificial neurons, I think that today we"}, {"start": 373.40000000000003, "end": 376.96000000000004, "text": " have almost no idea how the human brain works."}, {"start": 376.96000000000004, "end": 381.72, "text": " In fact, every few years, neuroscientists make some fundamental breakthrough about how the"}, {"start": 381.72, "end": 386.32, "text": " brain works, and I think we'll continue to do so for the foreseeable future."}, {"start": 386.32, "end": 390.90000000000003, "text": " And that to me is a sign that there are many breakthroughs that are yet to be discovered"}, {"start": 390.9, "end": 395.96, "text": " about how the brain actually works, and thus attempts to blindly mimic what we know of"}, {"start": 395.96, "end": 401.44, "text": " the human brain today, which is frankly very little, probably won't get us that far to"}, {"start": 401.44, "end": 405.64, "text": " our building role intelligence, certainly not with our current level of knowledge in"}, {"start": 405.64, "end": 407.76, "text": " neuroscience."}, {"start": 407.76, "end": 412.64, "text": " Having said that, even with these extremely simplified models of a neuron, which we'll"}, {"start": 412.64, "end": 417.91999999999996, "text": " talk about, we'll be able to build really powerful deep learning algorithms."}, {"start": 417.92, "end": 423.16, "text": " And so as you go deeper into neural networks and into deep learning, even though the origins"}, {"start": 423.16, "end": 428.52000000000004, "text": " were biologically motivated, don't take the biological motivation too seriously."}, {"start": 428.52000000000004, "end": 433.08000000000004, "text": " In fact, those of us that do research in deep learning have shifted away from looking to"}, {"start": 433.08000000000004, "end": 439.32, "text": " biological motivation that much, but instead are just using engineering principles to figure"}, {"start": 439.32, "end": 441.88, "text": " out how to build algorithms that are more effective."}, {"start": 441.88, "end": 446.82, "text": " But I think it might still be fun to speculate and think about how biological neurons work"}, {"start": 446.82, "end": 448.52, "text": " every now and then."}, {"start": 448.52, "end": 451.84, "text": " The ideas of neural networks have been around for many decades."}, {"start": 451.84, "end": 455.04, "text": " So a few people have asked me, hey, Andrew, why now?"}, {"start": 455.04, "end": 460.0, "text": " Why is it that only in the last handful of years that neural networks have really taken"}, {"start": 460.0, "end": 461.36, "text": " off?"}, {"start": 461.36, "end": 466.24, "text": " This is a picture I draw for them when I'm asked that question, and that maybe you could"}, {"start": 466.24, "end": 469.76, "text": " draw for others as well if they ask you that question."}, {"start": 469.76, "end": 475.32, "text": " Let me plot on the horizontal axis, the amount of data you have for a problem, and on the"}, {"start": 475.32, "end": 482.36, "text": " vertical axis, the performance or the accuracy of a learning algorithm applied to that problem."}, {"start": 482.36, "end": 487.36, "text": " Over the last couple of decades, with the rise of the internet, the rise of mobile phones,"}, {"start": 487.36, "end": 492.76, "text": " the digitalization of our society, the amount of data we have for a lot of applications"}, {"start": 492.76, "end": 494.88, "text": " has steadily marched to the right."}, {"start": 494.88, "end": 500.56, "text": " A lot of records that used to be on paper, such as if you order something, rather than"}, {"start": 500.56, "end": 505.0, "text": " it being on a piece of paper, that's much more likely to be a digital record."}, {"start": 505.0, "end": 509.44, "text": " Or health record, if you see a doctor, it's much more likely to be digital now compared"}, {"start": 509.44, "end": 512.64, "text": " to on pieces of paper."}, {"start": 512.64, "end": 518.06, "text": " And so in many application areas, the amount of digital data has exploded."}, {"start": 518.06, "end": 523.48, "text": " And what we saw was with traditional machine learning algorithms, such as logistic regression"}, {"start": 523.48, "end": 529.94, "text": " and linear regression, even as you fed those algorithms more data, it was very difficult"}, {"start": 529.94, "end": 533.32, "text": " to get the performance to keep on going up."}, {"start": 533.32, "end": 538.5600000000001, "text": " So it was as if the traditional learning algorithms like linear regression and logistic regression,"}, {"start": 538.5600000000001, "end": 542.72, "text": " they just weren't able to scale with the amount of data we could now feed it."}, {"start": 542.72, "end": 548.46, "text": " And they weren't able to take effective advantage of all this data we had for different applications."}, {"start": 548.46, "end": 554.88, "text": " And what AI researchers started to observe was that if you were to train a small neural"}, {"start": 554.88, "end": 560.4200000000001, "text": " network on this data set, then the performance maybe looks like this."}, {"start": 560.42, "end": 566.0799999999999, "text": " And if you were to train a medium sized neural network, meaning one with more neurons in"}, {"start": 566.0799999999999, "end": 568.92, "text": " it, this performance may look like that."}, {"start": 568.92, "end": 573.7199999999999, "text": " And if you were to train a very large neural network, meaning one with a lot of these artificial"}, {"start": 573.7199999999999, "end": 579.5999999999999, "text": " neurons, then for some applications, the performance would just keep on going up."}, {"start": 579.5999999999999, "end": 581.36, "text": " And so this meant two things."}, {"start": 581.36, "end": 585.8399999999999, "text": " It meant that for a certain class of applications where you do have a lot of data, sometimes"}, {"start": 585.84, "end": 592.36, "text": " you hear the term big data toss around, if you're able to train a very large neural network"}, {"start": 592.36, "end": 597.9200000000001, "text": " to take advantage of that huge amount of data you have, then you could attain performance"}, {"start": 597.9200000000001, "end": 603.5600000000001, "text": " on anything ranging from speech recognition to image recognition to natural language processing"}, {"start": 603.5600000000001, "end": 608.12, "text": " applications and many more that just were not possible with earlier generations of learning"}, {"start": 608.12, "end": 609.48, "text": " algorithms."}, {"start": 609.48, "end": 613.0400000000001, "text": " And this caused deep learning algorithms to take off."}, {"start": 613.04, "end": 621.16, "text": " And this too is why faster computer processes, including the rise of GPUs or graphics processor"}, {"start": 621.16, "end": 628.04, "text": " units, this is hardware originally designed to generate nice looking computer graphics,"}, {"start": 628.04, "end": 632.3199999999999, "text": " but turned out to be really powerful for deep learning as well."}, {"start": 632.3199999999999, "end": 639.4399999999999, "text": " That was also a major force in allowing deep learning algorithms to become what it is today."}, {"start": 639.44, "end": 643.96, "text": " That's how neural networks got started as well as why they took off so quickly in the"}, {"start": 643.96, "end": 645.5600000000001, "text": " last several years."}, {"start": 645.5600000000001, "end": 650.1600000000001, "text": " Let's now dive more deeply into the details of how a neural network actually works."}, {"start": 650.16, "end": 670.1999999999999, "text": " Please go on to the next video."}]