Why is activation needed at all in neural network? I watched the Risto Siilasmaa video on Machine Learning. It's very well explained, but the question emerged that at what stage should we use the activation function and why we need it at all. I know that by definition the activation function transforms the sum of w*x+b to a number between some lower and upper limit.

In the video Risto Siilasmaa explains that in the training process there are the following steps:


Start with random weights.
Calculate the outcome (sum of w*x+b) - we know what it should be because we know what image we gave to the system.
Calculate the error.
Nudge all weights to reduce the error.


But what about the activation? Where to place it in the previous list? Before the error calculation?
And what would happen if we omitted it altogether? Just calculate the outcome and error and nudge the weights? Is it because the error calculation doesn't work well when the value of outcome isn't transformed between some lower and upper limit?
Generally the activation is part of the model and gets applied for each neuron, so definitely before the error calculation. What the activation function is depends on what task you are solving and where the neuron of interest is. In principle the activation function $f$ would go to the calculation of the outcome

$$ y = f(Wx + b)$$

For output neurons, if you are doing classification, then $f$ should map between 0 and 1, since you'll interpret the outcome as a probability. For regression $f$ could be just the identity. 

For hidden (i.e. non-output neurons), you definitely want to use a non-linear $f$. The reason is that the neural network would otherwise be equivalent to a regular linear model. So the non-linear activations are needed to harvest the expressive power of neural networks.

For deep learning the most popular $f$ for hidden neurons would probably be the rectified linear unit (relu)

$$ f(x) = \max(0,x)$$