5.5 Activation Functions | Why do we need activation functions? --[Machine Learning | Andrew Ng]
 Let's take a look at why neural networks need activation functions and why they just don't work if we were to use the linear activation function in every neuron in the neural network. Recall this demand prediction example. What would happen if we were to use a linear activation function for all of the nodes in this neural network? It turns out that this big neural network will become no different than just linear regression. And so this would defeat the entire purpose of using a neural network because it would then just not be able to fit anything more complex than the linear regression model that we learned about in the first course. Let's illustrate this with a simpler example. Let's look at the example of a neural network where the input x is just a number and we have one hidden unit with parameters w1 and b1 that outputs a1, which is here just a number. And then the second layer is the output layer and it has also just one output unit with parameters w2 and b2 and that outputs a2, which is also just a number, just a scalar, which is the output of the neural network f of x. Let's see what this neural network would do if we were to use the linear activation function g of z equals z everywhere. So to compute a1 as a function of x, the neural network would use a1 equals g of w1 times x plus b1, but g of z is equal to z, so this is just w1 times x plus b1. Then a2 is equal to w2 times a1 plus b2 because g of z equals z. And let me take this expression for a1 and substitute it in there. So that becomes w2 times w1 x plus b1 plus b2. And if we simplify, this becomes w2 w1 times x plus w2 b1 plus b2. And it turns out that if I were to set w equals w2 times w1 and set b equals this quantity over here, then what we've just shown is that a2 is equal to wx plus b. So w is just a linear function of the input x. And rather than using a neural network with one hidden layer and one output layer, we might as well have just used a linear regression model. If you're familiar with linear algebra, this result comes from the fact that a linear function of a linear function is itself a linear function. And this is why having multiple layers in a neural network doesn't let the neural network compute any more complex features or learn anything more complex than just a linear function. So in the general case, if you had a neural network with multiple layers like this, and say you were to use a linear activation function for all the hidden layers, and also use a linear activation function for the output layer, then it turns out this model will compute an output that is completely equivalent to linear regression. The output a4 can be expressed as a linear function of the input features x plus b. Or alternatively, if we were to still use a linear activation function for all the hidden layers, for these three hidden layers here, but we were to use a logistic activation function for the output layer, then it turns out you can show that this model becomes equivalent to logistic regression. And a4 in this case can be expressed as 1 over 1 plus e to the negative wx plus b for some values of w and b. And so this big neural network doesn't do anything that you can't also do with logistic regression. That's why a common rule of thumb is don't use the linear activation function in the hidden layers of your neural network. And in fact, I recommend typically using the ReLU activation function should do just fine. So that's why a neural network needs activation functions other than just the linear activation function everywhere. So far, you've learned to build neural networks for binary classification problems, where y is either 0 or 1, as well as for regression problems, where y can take negative or positive values or maybe just positive and non-negative values. In the next video, I'd like to share with you a generalization of what you've seen so far for classification. In particular, when y doesn't just take on two values, but may take on three or four or 10 or even more categorical values. Let's take a look at how you can build a neural network for that type of classification problem.
[{"start": 0.0, "end": 8.700000000000001, "text": " Let's take a look at why neural networks need activation functions and why they just don't"}, {"start": 8.700000000000001, "end": 15.860000000000001, "text": " work if we were to use the linear activation function in every neuron in the neural network."}, {"start": 15.860000000000001, "end": 19.06, "text": " Recall this demand prediction example."}, {"start": 19.06, "end": 24.82, "text": " What would happen if we were to use a linear activation function for all of the nodes in"}, {"start": 24.82, "end": 26.26, "text": " this neural network?"}, {"start": 26.26, "end": 31.46, "text": " It turns out that this big neural network will become no different than just linear"}, {"start": 31.46, "end": 32.82, "text": " regression."}, {"start": 32.82, "end": 37.74, "text": " And so this would defeat the entire purpose of using a neural network because it would"}, {"start": 37.74, "end": 44.06, "text": " then just not be able to fit anything more complex than the linear regression model that"}, {"start": 44.06, "end": 46.980000000000004, "text": " we learned about in the first course."}, {"start": 46.980000000000004, "end": 51.44, "text": " Let's illustrate this with a simpler example."}, {"start": 51.44, "end": 57.08, "text": " Let's look at the example of a neural network where the input x is just a number and we"}, {"start": 57.08, "end": 68.53999999999999, "text": " have one hidden unit with parameters w1 and b1 that outputs a1, which is here just a number."}, {"start": 68.53999999999999, "end": 74.94, "text": " And then the second layer is the output layer and it has also just one output unit with"}, {"start": 74.94, "end": 81.74, "text": " parameters w2 and b2 and that outputs a2, which is also just a number, just a scalar,"}, {"start": 81.74, "end": 85.58, "text": " which is the output of the neural network f of x."}, {"start": 85.58, "end": 90.82, "text": " Let's see what this neural network would do if we were to use the linear activation function"}, {"start": 90.82, "end": 95.38, "text": " g of z equals z everywhere."}, {"start": 95.38, "end": 105.17999999999999, "text": " So to compute a1 as a function of x, the neural network would use a1 equals g of w1 times"}, {"start": 105.17999999999999, "end": 115.53999999999999, "text": " x plus b1, but g of z is equal to z, so this is just w1 times x plus b1."}, {"start": 115.53999999999999, "end": 123.97999999999999, "text": " Then a2 is equal to w2 times a1 plus b2 because g of z equals z."}, {"start": 123.98, "end": 129.68, "text": " And let me take this expression for a1 and substitute it in there."}, {"start": 129.68, "end": 139.9, "text": " So that becomes w2 times w1 x plus b1 plus b2."}, {"start": 139.9, "end": 152.86, "text": " And if we simplify, this becomes w2 w1 times x plus w2 b1 plus b2."}, {"start": 152.86, "end": 161.58, "text": " And it turns out that if I were to set w equals w2 times w1 and set b equals this quantity"}, {"start": 161.58, "end": 168.26000000000002, "text": " over here, then what we've just shown is that a2 is equal to wx plus b."}, {"start": 168.26000000000002, "end": 174.34, "text": " So w is just a linear function of the input x."}, {"start": 174.34, "end": 178.06, "text": " And rather than using a neural network with one hidden layer and one output layer, we"}, {"start": 178.06, "end": 181.9, "text": " might as well have just used a linear regression model."}, {"start": 181.9, "end": 187.86, "text": " If you're familiar with linear algebra, this result comes from the fact that a linear function"}, {"start": 187.86, "end": 191.18, "text": " of a linear function is itself a linear function."}, {"start": 191.18, "end": 195.36, "text": " And this is why having multiple layers in a neural network doesn't let the neural network"}, {"start": 195.36, "end": 202.78, "text": " compute any more complex features or learn anything more complex than just a linear function."}, {"start": 202.78, "end": 210.14000000000001, "text": " So in the general case, if you had a neural network with multiple layers like this, and"}, {"start": 210.14, "end": 214.7, "text": " say you were to use a linear activation function for all the hidden layers, and also use a"}, {"start": 214.7, "end": 221.29999999999998, "text": " linear activation function for the output layer, then it turns out this model will compute"}, {"start": 221.29999999999998, "end": 225.5, "text": " an output that is completely equivalent to linear regression."}, {"start": 225.5, "end": 233.89999999999998, "text": " The output a4 can be expressed as a linear function of the input features x plus b."}, {"start": 233.89999999999998, "end": 239.42, "text": " Or alternatively, if we were to still use a linear activation function for all the hidden"}, {"start": 239.42, "end": 245.38, "text": " layers, for these three hidden layers here, but we were to use a logistic activation function"}, {"start": 245.38, "end": 251.11999999999998, "text": " for the output layer, then it turns out you can show that this model becomes equivalent"}, {"start": 251.11999999999998, "end": 253.73999999999998, "text": " to logistic regression."}, {"start": 253.73999999999998, "end": 261.26, "text": " And a4 in this case can be expressed as 1 over 1 plus e to the negative wx plus b for"}, {"start": 261.26, "end": 263.41999999999996, "text": " some values of w and b."}, {"start": 263.41999999999996, "end": 268.9, "text": " And so this big neural network doesn't do anything that you can't also do with logistic"}, {"start": 268.9, "end": 269.97999999999996, "text": " regression."}, {"start": 269.97999999999996, "end": 273.78, "text": " That's why a common rule of thumb is don't use the linear activation function in the"}, {"start": 273.78, "end": 276.06, "text": " hidden layers of your neural network."}, {"start": 276.06, "end": 282.78, "text": " And in fact, I recommend typically using the ReLU activation function should do just fine."}, {"start": 282.78, "end": 288.78, "text": " So that's why a neural network needs activation functions other than just the linear activation"}, {"start": 288.78, "end": 290.94, "text": " function everywhere."}, {"start": 290.94, "end": 295.82, "text": " So far, you've learned to build neural networks for binary classification problems, where"}, {"start": 295.82, "end": 302.98, "text": " y is either 0 or 1, as well as for regression problems, where y can take negative or positive"}, {"start": 302.98, "end": 307.62, "text": " values or maybe just positive and non-negative values."}, {"start": 307.62, "end": 311.86, "text": " In the next video, I'd like to share with you a generalization of what you've seen"}, {"start": 311.86, "end": 315.18, "text": " so far for classification."}, {"start": 315.18, "end": 321.98, "text": " In particular, when y doesn't just take on two values, but may take on three or four"}, {"start": 321.98, "end": 326.1, "text": " or 10 or even more categorical values."}, {"start": 326.1, "end": 353.1, "text": " Let's take a look at how you can build a neural network for that type of classification problem."}]