5.3 Activation Functions | Alternatives to the sigmoid activation  --[Machine Learning | Andrew Ng]
 So far, we've been using the sigmoid activation function in all the nodes, in the hidden layers, and in the output layer. And we have started that way because we were building up neural networks by taking logistic regression and creating a lot of logistic regression units and stringing them together. But if you use other activation functions, your neural network can become much more powerful. Let's take a look at how to do that. Recall the demand prediction example from last week, where given price, shipping cost, marketing material, you would try to predict if something is highly affordable, if there's good awareness and high perceived quality, and based on that, try to predict if it's a top seller. But this assumes that awareness is maybe binary, it's either people are aware or they are not. But it seems like the degree to which possible buyers are aware of the t-shirt you're selling may not be binary. They can be a little bit aware, somewhat aware, extremely aware, or it could have gone completely viral. So rather than modeling awareness as a binary number, 0 or 1, that you try to estimate the probability of awareness, or rather than modeling awareness as just a number between 0 and 1, maybe awareness should be any non-negative number, because there can be any non-negative value of awareness going from 0 up to very, very large numbers. So whereas previously we had used this equation to calculate the activation of that second hidden unit estimating awareness, where g was the sigmoid function, and thus goes between 0 and 1. If you want to allow a 1, 2 to potentially take on much larger positive values, we can instead swap in a different activation function. It turns out that a very common choice of activation function in neural networks is this function. It looks like this. It goes, if z is this, then g of z is 0 to the left, and then this is straight line, 35 degrees to the right of 0. And so when z is greater than or equal to 0, g of z is just equal to z, that is to the right half of this diagram. And the mathematical equation for this is g of z equals max of 0, z. Be free to verify for yourself that max of 0, z results in this curve that I've drawn over here. And if a 1, 2 is g of z for this value of z, then a, the activation value, can now take on 0 or any non-negative value. This activation function has a name. It goes by the name ReLU with this funny capitalization. And ReLU stands for, again, a somewhat arcane term, but it stands for rectified linear unit. Don't worry too much about what rectified means and what linear unit means. This was just a name that the authors had given to this particular activation function when they came up with it. But most people in deep learning just say ReLU to refer to this g of z. More generally, you have a choice of what to use for g of z. And sometimes we'll use a different choice than the sigmoid activation function. Here are the most commonly used activation functions. You saw the sigmoid activation function, g of z equals the sigmoid function. On the last slide, we just looked at the ReLU, or rectified linear unit, g of z equals max of 0, z. There's one other activation function which is worth mentioning, which is called the linear activation function, which is just g of z equals to z. Sometimes if you use the linear activation function, people will say we're not using any activation function because if a is g of z, where g of z equals z, then a is just equal to this, w dot x plus b, say. And so it's as if there was no g in there at all. So when you are using this linear activation function, g of z, sometimes people will say, well, we're not using any activation function. Although in this clause, I will refer to using the linear activation function rather than no activation function. But if you hear someone else use that terminology, that's what they mean. It just refers to the linear activation function. And these three are probably by far the most commonly used activation functions in neural networks. Later this week, we'll touch on the fourth one called the softmax activation function. But with these activation functions, you'll be able to build a rich variety of powerful neural networks. So when building a neural network, for each neuron, do you want to use the sigmoid activation function or the ReLU activation function or a linear activation function? How do you choose between these different activation functions? Let's take a look at that in the next video.
[{"start": 0.0, "end": 8.76, "text": " So far, we've been using the sigmoid activation function in all the nodes, in the hidden layers,"}, {"start": 8.76, "end": 11.08, "text": " and in the output layer."}, {"start": 11.08, "end": 16.28, "text": " And we have started that way because we were building up neural networks by taking logistic"}, {"start": 16.28, "end": 22.84, "text": " regression and creating a lot of logistic regression units and stringing them together."}, {"start": 22.84, "end": 27.84, "text": " But if you use other activation functions, your neural network can become much more powerful."}, {"start": 27.84, "end": 30.4, "text": " Let's take a look at how to do that."}, {"start": 30.4, "end": 35.36, "text": " Recall the demand prediction example from last week, where given price, shipping cost,"}, {"start": 35.36, "end": 41.0, "text": " marketing material, you would try to predict if something is highly affordable, if there's"}, {"start": 41.0, "end": 44.879999999999995, "text": " good awareness and high perceived quality, and based on that, try to predict if it's"}, {"start": 44.879999999999995, "end": 46.64, "text": " a top seller."}, {"start": 46.64, "end": 54.16, "text": " But this assumes that awareness is maybe binary, it's either people are aware or they are not."}, {"start": 54.16, "end": 60.699999999999996, "text": " But it seems like the degree to which possible buyers are aware of the t-shirt you're selling"}, {"start": 60.699999999999996, "end": 62.0, "text": " may not be binary."}, {"start": 62.0, "end": 66.92, "text": " They can be a little bit aware, somewhat aware, extremely aware, or it could have gone completely"}, {"start": 66.92, "end": 68.24, "text": " viral."}, {"start": 68.24, "end": 74.8, "text": " So rather than modeling awareness as a binary number, 0 or 1, that you try to estimate the"}, {"start": 74.8, "end": 80.72, "text": " probability of awareness, or rather than modeling awareness as just a number between 0 and 1,"}, {"start": 80.72, "end": 86.2, "text": " maybe awareness should be any non-negative number, because there can be any non-negative"}, {"start": 86.2, "end": 91.9, "text": " value of awareness going from 0 up to very, very large numbers."}, {"start": 91.9, "end": 98.96000000000001, "text": " So whereas previously we had used this equation to calculate the activation of that second"}, {"start": 98.96000000000001, "end": 105.24, "text": " hidden unit estimating awareness, where g was the sigmoid function, and thus goes between"}, {"start": 105.24, "end": 107.32, "text": " 0 and 1."}, {"start": 107.32, "end": 114.67999999999999, "text": " If you want to allow a 1, 2 to potentially take on much larger positive values, we can"}, {"start": 114.67999999999999, "end": 118.96, "text": " instead swap in a different activation function."}, {"start": 118.96, "end": 123.96, "text": " It turns out that a very common choice of activation function in neural networks is"}, {"start": 123.96, "end": 125.58, "text": " this function."}, {"start": 125.58, "end": 126.91999999999999, "text": " It looks like this."}, {"start": 126.91999999999999, "end": 136.28, "text": " It goes, if z is this, then g of z is 0 to the left, and then this is straight line,"}, {"start": 136.28, "end": 139.84, "text": " 35 degrees to the right of 0."}, {"start": 139.84, "end": 148.24, "text": " And so when z is greater than or equal to 0, g of z is just equal to z, that is to the"}, {"start": 148.24, "end": 151.52, "text": " right half of this diagram."}, {"start": 151.52, "end": 159.12, "text": " And the mathematical equation for this is g of z equals max of 0, z."}, {"start": 159.12, "end": 168.6, "text": " Be free to verify for yourself that max of 0, z results in this curve that I've drawn"}, {"start": 168.6, "end": 170.1, "text": " over here."}, {"start": 170.1, "end": 177.88, "text": " And if a 1, 2 is g of z for this value of z, then a, the activation value, can now take"}, {"start": 177.88, "end": 182.9, "text": " on 0 or any non-negative value."}, {"start": 182.9, "end": 185.52, "text": " This activation function has a name."}, {"start": 185.52, "end": 190.12, "text": " It goes by the name ReLU with this funny capitalization."}, {"start": 190.12, "end": 196.8, "text": " And ReLU stands for, again, a somewhat arcane term, but it stands for rectified linear unit."}, {"start": 196.8, "end": 200.18, "text": " Don't worry too much about what rectified means and what linear unit means."}, {"start": 200.18, "end": 204.72, "text": " This was just a name that the authors had given to this particular activation function"}, {"start": 204.72, "end": 206.48000000000002, "text": " when they came up with it."}, {"start": 206.48000000000002, "end": 212.72, "text": " But most people in deep learning just say ReLU to refer to this g of z."}, {"start": 212.72, "end": 217.92, "text": " More generally, you have a choice of what to use for g of z."}, {"start": 217.92, "end": 223.07999999999998, "text": " And sometimes we'll use a different choice than the sigmoid activation function."}, {"start": 223.07999999999998, "end": 227.56, "text": " Here are the most commonly used activation functions."}, {"start": 227.56, "end": 232.84, "text": " You saw the sigmoid activation function, g of z equals the sigmoid function."}, {"start": 232.84, "end": 238.44, "text": " On the last slide, we just looked at the ReLU, or rectified linear unit, g of z equals max"}, {"start": 238.44, "end": 240.6, "text": " of 0, z."}, {"start": 240.6, "end": 244.92, "text": " There's one other activation function which is worth mentioning, which is called the linear"}, {"start": 244.92, "end": 250.44, "text": " activation function, which is just g of z equals to z."}, {"start": 250.44, "end": 254.48, "text": " Sometimes if you use the linear activation function, people will say we're not using"}, {"start": 254.48, "end": 262.36, "text": " any activation function because if a is g of z, where g of z equals z, then a is just"}, {"start": 262.36, "end": 266.36, "text": " equal to this, w dot x plus b, say."}, {"start": 266.36, "end": 269.36, "text": " And so it's as if there was no g in there at all."}, {"start": 269.36, "end": 275.64, "text": " So when you are using this linear activation function, g of z, sometimes people will say,"}, {"start": 275.64, "end": 278.6, "text": " well, we're not using any activation function."}, {"start": 278.6, "end": 283.64, "text": " Although in this clause, I will refer to using the linear activation function rather than"}, {"start": 283.64, "end": 285.52000000000004, "text": " no activation function."}, {"start": 285.52000000000004, "end": 288.44, "text": " But if you hear someone else use that terminology, that's what they mean."}, {"start": 288.44, "end": 291.82, "text": " It just refers to the linear activation function."}, {"start": 291.82, "end": 298.32, "text": " And these three are probably by far the most commonly used activation functions in neural"}, {"start": 298.32, "end": 299.64, "text": " networks."}, {"start": 299.64, "end": 304.88, "text": " Later this week, we'll touch on the fourth one called the softmax activation function."}, {"start": 304.88, "end": 310.8, "text": " But with these activation functions, you'll be able to build a rich variety of powerful"}, {"start": 310.8, "end": 312.48, "text": " neural networks."}, {"start": 312.48, "end": 318.24, "text": " So when building a neural network, for each neuron, do you want to use the sigmoid activation"}, {"start": 318.24, "end": 323.65999999999997, "text": " function or the ReLU activation function or a linear activation function?"}, {"start": 323.65999999999997, "end": 327.24, "text": " How do you choose between these different activation functions?"}, {"start": 327.24, "end": 328.96000000000004, "text": " Let's take a look at that in the next video."}]