Which functions can be activation functions?
The main characteristic of an activation function is to bring a non-linearity property into the neural network. For the hidden layer, there is no need for the function to be bounded. The last layer should use a function whose range corresponds to what you want.
For regression, you usually re-scale your output data to $[-1,1]$ or $[0,1]$ and you use a tanh (hyperbolic tangent) or sigmoid function in the last layer
For classification, you want to obtain probabilities: use a softmax function in the last layer.
For the hidden layers some functions are better than others :

The gradient should be fast to compute (from the perspective of your computer).

If you use too many hidden layers, you will have the vanishing gradient problem if the derivative of your activation is too close to zero. You need a large zone of the domain with a derivative not close to zero.


In practice, the ReLU function defined as $f(x)=\max(0, x)$ works very well and is very simple.