5.12 Additional Neural Network Concepts | Additional Layer Types --[Machine Learning | Andrew Ng]
 All the neural network layers we've used so far have been the dense layer type in which every neuron in a layer gets as its inputs all the activations from the previous layer. And it turns out that just using the dense layer type, you can actually build some pretty powerful learning algorithms. And to help you build further intuition about what neural networks can do, it turns out that there are some other types of layers as well with other properties. In this video, I'd like to briefly touch on this and give you an example of a different type of neural network layer. Let's take a look. To recap, in the dense layer that we've been using, the activation of a neuron in, say, the second hidden layer is a function of every single activation value from the previous layer of A1. But it turns out that for some applications, someone designing a neural network may choose to use a different type of layer. One other layer type that you may see in some work is called a convolutional layer. Let me illustrate this with an example. So what I'm showing on the left is the input X, which is a handwritten digit 9. And what I'm going to do is construct a hidden layer, which will compute different activations as functions of this input image X. But here's something I can do. For the first hidden unit, which I've drawn in blue, rather than saying this neuron can look at all the pixels in this image, I might say this neuron can only look at the pixels in this little rectangular region. The second neuron, which I'm going to illustrate in magenta, is also not going to look at the entire input image X. Instead, it's only going to look at the pixels in a limited region of the image. And so on for the third neuron and the fourth neuron, and so on and so forth, down to the last neuron, which maybe looks only at that region of the image. So why might you want to do this? Why won't you let every neuron look at all the pixels, but instead look at only some of the pixels? Well, some of the benefits are, first, it speeds up computation. And second advantage is that a neural network that uses this type of layer called a convolutional layer can need less training data. Or alternatively, it can also be less prone to overfitting. You'd heard me talk a bit about overfitting in the previous course, but this is something that we'll dive into greater detail on next week as well, when we talk about practical tips for using learning algorithms. And this type of layer, where each neuron only looks at a region of the input image, is called a convolutional layer. It was a researcher, Yang Le Kun, who had figured out a lot of the details of how to get convolutional layers to work and popularize their use. Let me illustrate in more detail a convolutional layer. And if you have multiple convolutional layers in a neural network, sometimes that's called a convolutional neural network. To illustrate the convolutional layer or convolutional neural network, on this slide, I'm going to use instead of a 2D image input, I'm going to use a one dimensional input. And the motivating example I'm going to use is classification of EKG signals or electrocardiograms. So if you put two electrodes on your chest, you will record voltages that look like this that correspond to your heartbeat. This is actually something that my Stanford research group did research on. We're actually reading EKG signals that actually look like this to try to diagnose if a patient may have a heart issue. So an EKG signal, an electrocardiogram, ECG in some places, EKG in some places, is just a list of numbers corresponding to the height of the surface at different points in time. So you may have, say, 100 numbers corresponding to the height of this curve at 100 different points of time. And the learning task is, given this time series, given this EKG signal, to classify, say whether this patient has a heart disease or some diagnosable heart condition, here's what a convolutional neural network might do. So I'm going to take the EKG signal and rotate it 90 degrees to lay it on the side. And so we have here 100 inputs, X1, X2, all the way through X100, like so. And when I construct the first hidden layer, instead of having the first hidden unit take as input all 100 numbers, let me have the first hidden unit look at only X1 through X20. So that corresponds to looking at just a small window of this EKG signal. The second hidden layer, shown in a different color here, will look at X11 through X30, so it looks at a different window in this EKG signal. And the third hidden layer looks at another window, X21 through X40, and so on. And the final hidden unit in this example will look at X81 through X100, so it looks at a small window toward the end of this EKG time series. So this is a convolutional layer, because each unit in this layer looks at only a limited window of the input. Now this layer of the neural network has nine units. The next layer can also be a convolutional layer. So in the second hidden layer, let me architect my first unit, not to look at all nine activations from the previous layer, but to look at, say, just the first five activations from the previous layer. And then my second unit in this second hidden layer may look at just another five numbers, say A3 to A7. And the third and final hidden unit in this layer will only look at A5 through A9. And then maybe finally, these activations, A2, gets inputs to a sigmoid unit that does look at all three of these values of A2 in order to make a binary classification regarding presence or absence of heart disease. So this is an example of a neural network with the first hidden layer being a convolutional layer, the second hidden layer also being a convolutional layer, and then the output layer being a sigmoid layer. And it turns out that with convolutional layers, you have many architectural choices, such as how big is the window of inputs that a single neuron should look at, and how many neurons should each layer have. And by choosing those architectural parameters effectively, you can build new versions of neural networks that can be even more effective than the dense layer for some applications. To recap, that's it for the convolutional layer and convolutional neural networks. I'm not going to go deeper into convolutional networks in this class, and you don't need to know anything about them to do the homeworks and finish this class successfully. But I hope that you find this additional intuition that neural networks can have other types of layers as well to be useful. And in fact, if you sometimes hear about the latest cutting edge architectures, like a transformer model or an LSTM or an attention model, a lot of this research in neural networks even today pertains to researchers trying to invent new types of layers for neural networks and plugging these different types of layers together as building blocks to form even more complex and hopefully more powerful neural networks. So that's it for the required videos for this week. Thank you and congrats on sticking with me all the way through this. And I look forward to seeing you next week also, where we'll start to talk about practical advice for how you can build machine learning systems. I hope that the tips you learn next week will help you become much more effective at building useful machine learning systems. And I look forward also to seeing you next week.
[{"start": 0.0, "end": 8.6, "text": " All the neural network layers we've used so far have been the dense layer type in which"}, {"start": 8.6, "end": 15.58, "text": " every neuron in a layer gets as its inputs all the activations from the previous layer."}, {"start": 15.58, "end": 20.16, "text": " And it turns out that just using the dense layer type, you can actually build some pretty"}, {"start": 20.16, "end": 22.6, "text": " powerful learning algorithms."}, {"start": 22.6, "end": 28.16, "text": " And to help you build further intuition about what neural networks can do, it turns out"}, {"start": 28.16, "end": 32.480000000000004, "text": " that there are some other types of layers as well with other properties."}, {"start": 32.480000000000004, "end": 37.96, "text": " In this video, I'd like to briefly touch on this and give you an example of a different"}, {"start": 37.96, "end": 39.56, "text": " type of neural network layer."}, {"start": 39.56, "end": 41.16, "text": " Let's take a look."}, {"start": 41.16, "end": 49.44, "text": " To recap, in the dense layer that we've been using, the activation of a neuron in, say,"}, {"start": 49.44, "end": 55.74, "text": " the second hidden layer is a function of every single activation value from the previous"}, {"start": 55.74, "end": 63.32, "text": " layer of A1. But it turns out that for some applications, someone designing a neural network"}, {"start": 63.32, "end": 67.32000000000001, "text": " may choose to use a different type of layer."}, {"start": 67.32000000000001, "end": 73.8, "text": " One other layer type that you may see in some work is called a convolutional layer."}, {"start": 73.8, "end": 76.26, "text": " Let me illustrate this with an example."}, {"start": 76.26, "end": 82.64, "text": " So what I'm showing on the left is the input X, which is a handwritten digit 9."}, {"start": 82.64, "end": 88.28, "text": " And what I'm going to do is construct a hidden layer, which will compute different activations"}, {"start": 88.28, "end": 91.52, "text": " as functions of this input image X."}, {"start": 91.52, "end": 93.64, "text": " But here's something I can do."}, {"start": 93.64, "end": 99.5, "text": " For the first hidden unit, which I've drawn in blue, rather than saying this neuron can"}, {"start": 99.5, "end": 105.86, "text": " look at all the pixels in this image, I might say this neuron can only look at the pixels"}, {"start": 105.86, "end": 108.88, "text": " in this little rectangular region."}, {"start": 108.88, "end": 113.19999999999999, "text": " The second neuron, which I'm going to illustrate in magenta, is also not going to look at the"}, {"start": 113.19999999999999, "end": 119.39999999999999, "text": " entire input image X. Instead, it's only going to look at the pixels in a limited region"}, {"start": 119.39999999999999, "end": 121.03999999999999, "text": " of the image."}, {"start": 121.03999999999999, "end": 128.24, "text": " And so on for the third neuron and the fourth neuron, and so on and so forth, down to the"}, {"start": 128.24, "end": 135.07999999999998, "text": " last neuron, which maybe looks only at that region of the image."}, {"start": 135.07999999999998, "end": 136.84, "text": " So why might you want to do this?"}, {"start": 136.84, "end": 141.6, "text": " Why won't you let every neuron look at all the pixels, but instead look at only some"}, {"start": 141.6, "end": 142.6, "text": " of the pixels?"}, {"start": 142.6, "end": 149.24, "text": " Well, some of the benefits are, first, it speeds up computation."}, {"start": 149.24, "end": 154.28, "text": " And second advantage is that a neural network that uses this type of layer called a convolutional"}, {"start": 154.28, "end": 157.72, "text": " layer can need less training data."}, {"start": 157.72, "end": 162.2, "text": " Or alternatively, it can also be less prone to overfitting."}, {"start": 162.2, "end": 166.64000000000001, "text": " You'd heard me talk a bit about overfitting in the previous course, but this is something"}, {"start": 166.64, "end": 171.48, "text": " that we'll dive into greater detail on next week as well, when we talk about practical"}, {"start": 171.48, "end": 175.55999999999997, "text": " tips for using learning algorithms."}, {"start": 175.55999999999997, "end": 183.23999999999998, "text": " And this type of layer, where each neuron only looks at a region of the input image,"}, {"start": 183.23999999999998, "end": 186.32, "text": " is called a convolutional layer."}, {"start": 186.32, "end": 190.72, "text": " It was a researcher, Yang Le Kun, who had figured out a lot of the details of how to"}, {"start": 190.72, "end": 195.23999999999998, "text": " get convolutional layers to work and popularize their use."}, {"start": 195.24, "end": 201.32000000000002, "text": " Let me illustrate in more detail a convolutional layer."}, {"start": 201.32000000000002, "end": 206.10000000000002, "text": " And if you have multiple convolutional layers in a neural network, sometimes that's called"}, {"start": 206.10000000000002, "end": 209.24, "text": " a convolutional neural network."}, {"start": 209.24, "end": 214.12, "text": " To illustrate the convolutional layer or convolutional neural network, on this slide, I'm going to"}, {"start": 214.12, "end": 221.20000000000002, "text": " use instead of a 2D image input, I'm going to use a one dimensional input."}, {"start": 221.2, "end": 229.28, "text": " And the motivating example I'm going to use is classification of EKG signals or electrocardiograms."}, {"start": 229.28, "end": 234.64, "text": " So if you put two electrodes on your chest, you will record voltages that look like this"}, {"start": 234.64, "end": 237.51999999999998, "text": " that correspond to your heartbeat."}, {"start": 237.51999999999998, "end": 241.83999999999997, "text": " This is actually something that my Stanford research group did research on."}, {"start": 241.83999999999997, "end": 247.79999999999998, "text": " We're actually reading EKG signals that actually look like this to try to diagnose if a patient"}, {"start": 247.79999999999998, "end": 250.5, "text": " may have a heart issue."}, {"start": 250.5, "end": 257.8, "text": " So an EKG signal, an electrocardiogram, ECG in some places, EKG in some places, is just"}, {"start": 257.8, "end": 263.94, "text": " a list of numbers corresponding to the height of the surface at different points in time."}, {"start": 263.94, "end": 269.68, "text": " So you may have, say, 100 numbers corresponding to the height of this curve at 100 different"}, {"start": 269.68, "end": 272.52, "text": " points of time."}, {"start": 272.52, "end": 279.68, "text": " And the learning task is, given this time series, given this EKG signal, to classify,"}, {"start": 279.68, "end": 286.56, "text": " say whether this patient has a heart disease or some diagnosable heart condition, here's"}, {"start": 286.56, "end": 289.84000000000003, "text": " what a convolutional neural network might do."}, {"start": 289.84000000000003, "end": 294.48, "text": " So I'm going to take the EKG signal and rotate it 90 degrees to lay it on the side."}, {"start": 294.48, "end": 301.16, "text": " And so we have here 100 inputs, X1, X2, all the way through X100, like so."}, {"start": 301.16, "end": 308.52, "text": " And when I construct the first hidden layer, instead of having the first hidden unit take"}, {"start": 308.52, "end": 315.0, "text": " as input all 100 numbers, let me have the first hidden unit look at only X1 through"}, {"start": 315.0, "end": 316.32, "text": " X20."}, {"start": 316.32, "end": 322.03999999999996, "text": " So that corresponds to looking at just a small window of this EKG signal."}, {"start": 322.03999999999996, "end": 328.12, "text": " The second hidden layer, shown in a different color here, will look at X11 through X30,"}, {"start": 328.12, "end": 332.35999999999996, "text": " so it looks at a different window in this EKG signal."}, {"start": 332.35999999999996, "end": 337.32, "text": " And the third hidden layer looks at another window, X21 through X40, and so on."}, {"start": 337.32, "end": 344.24, "text": " And the final hidden unit in this example will look at X81 through X100, so it looks"}, {"start": 344.24, "end": 349.76, "text": " at a small window toward the end of this EKG time series."}, {"start": 349.76, "end": 354.84, "text": " So this is a convolutional layer, because each unit in this layer looks at only a limited"}, {"start": 354.84, "end": 357.84, "text": " window of the input."}, {"start": 357.84, "end": 363.58, "text": " Now this layer of the neural network has nine units."}, {"start": 363.58, "end": 368.52, "text": " The next layer can also be a convolutional layer."}, {"start": 368.52, "end": 377.96, "text": " So in the second hidden layer, let me architect my first unit, not to look at all nine activations"}, {"start": 377.96, "end": 384.09999999999997, "text": " from the previous layer, but to look at, say, just the first five activations from the previous"}, {"start": 384.09999999999997, "end": 385.4, "text": " layer."}, {"start": 385.4, "end": 392.84, "text": " And then my second unit in this second hidden layer may look at just another five numbers,"}, {"start": 392.84, "end": 395.08, "text": " say A3 to A7."}, {"start": 395.08, "end": 401.52, "text": " And the third and final hidden unit in this layer will only look at A5 through A9."}, {"start": 401.52, "end": 409.67999999999995, "text": " And then maybe finally, these activations, A2, gets inputs to a sigmoid unit that does"}, {"start": 409.67999999999995, "end": 417.12, "text": " look at all three of these values of A2 in order to make a binary classification regarding"}, {"start": 417.12, "end": 420.03999999999996, "text": " presence or absence of heart disease."}, {"start": 420.04, "end": 425.04, "text": " So this is an example of a neural network with the first hidden layer being a convolutional"}, {"start": 425.04, "end": 429.56, "text": " layer, the second hidden layer also being a convolutional layer, and then the output"}, {"start": 429.56, "end": 432.68, "text": " layer being a sigmoid layer."}, {"start": 432.68, "end": 438.1, "text": " And it turns out that with convolutional layers, you have many architectural choices, such"}, {"start": 438.1, "end": 443.04, "text": " as how big is the window of inputs that a single neuron should look at, and how many"}, {"start": 443.04, "end": 445.94, "text": " neurons should each layer have."}, {"start": 445.94, "end": 451.71999999999997, "text": " And by choosing those architectural parameters effectively, you can build new versions of"}, {"start": 451.71999999999997, "end": 457.24, "text": " neural networks that can be even more effective than the dense layer for some applications."}, {"start": 457.24, "end": 462.48, "text": " To recap, that's it for the convolutional layer and convolutional neural networks."}, {"start": 462.48, "end": 467.2, "text": " I'm not going to go deeper into convolutional networks in this class, and you don't need"}, {"start": 467.2, "end": 472.28, "text": " to know anything about them to do the homeworks and finish this class successfully."}, {"start": 472.28, "end": 477.76, "text": " But I hope that you find this additional intuition that neural networks can have other types"}, {"start": 477.76, "end": 480.52, "text": " of layers as well to be useful."}, {"start": 480.52, "end": 485.64, "text": " And in fact, if you sometimes hear about the latest cutting edge architectures, like a"}, {"start": 485.64, "end": 492.03999999999996, "text": " transformer model or an LSTM or an attention model, a lot of this research in neural networks"}, {"start": 492.03999999999996, "end": 498.47999999999996, "text": " even today pertains to researchers trying to invent new types of layers for neural networks"}, {"start": 498.48, "end": 503.40000000000003, "text": " and plugging these different types of layers together as building blocks to form even more"}, {"start": 503.40000000000003, "end": 507.84000000000003, "text": " complex and hopefully more powerful neural networks."}, {"start": 507.84000000000003, "end": 511.0, "text": " So that's it for the required videos for this week."}, {"start": 511.0, "end": 515.16, "text": " Thank you and congrats on sticking with me all the way through this."}, {"start": 515.16, "end": 520.86, "text": " And I look forward to seeing you next week also, where we'll start to talk about practical"}, {"start": 520.86, "end": 525.0, "text": " advice for how you can build machine learning systems."}, {"start": 525.0, "end": 530.48, "text": " I hope that the tips you learn next week will help you become much more effective at building"}, {"start": 530.48, "end": 532.72, "text": " useful machine learning systems."}, {"start": 532.72, "end": 555.72, "text": " And I look forward also to seeing you next week."}]