Learning the activation function in a neural network? Neural networks use specific activation functions, commonly used ones are tanh, ReLu.

I have seen that people have experimented with continuously parametrices activation functions, for instance here. Here, the additional parameter $\alpha$ which interpolates from logarithmus to linear to exponental, is optimized during the back-propagation step of the training.

I wonder whether much more general types of activation functions have been considered, such as Taylor-expansion of an unknown function where the coefficients are parameters in the training? It seems is a lot of degrees of freedom, that are not considered conventionally.

So my questions:


What is known about optimimality of actionation functions, apart from empirally testing? Are there mathematical proofs, for instance, that show some necessary properties for ideal learning (for instance, is monotonicity necessary, or could a sinusoidal activation function lead to optimal results too?)
If the answer to question 1 is no: Is there much research investigating the topic of learning activation functions?
Monotonicity is necessary! A neuron (perceptron) is either activated or not activated given an input.  If you're trying to learn "red" and red should activate the neuron; the more red it's the most likely it should be activated (This monotonic gradient is what this activation function represents).  If at certain redness it activates but at more redness it deactivates it makes no sense e.g. it can't learn red (The case for non-monotonic functions).

That they're differentiable functions makes it practical for the back propagation algorithm.

What they explain in your linked article is that they're trying to better approximate other functions e.g. sinusoidal.  They're not trying sinuosoidal as an activation function. 
They're combining several monotonic functions to create yet another monotonic function, that is also differentiable.