Why don't they use all kinds of non-linear functions in Neural Network Activation Functions?
I am wondering why they choose functions that always go up and to the right? Why not use [...] ones that are directed down instead of up, etc.?

The activation function is applied to a weighted sum of the inputs of the neuron. Replacing a function that is &quot;directed up&quot; with one that is &quot;directed down&quot; is equivalent to changing the weights. There's no particular reason to prefer an increasing function to a decreasing one, it just happens to be the convention.

Why not use all kinds of crazy input functions, those that fluctuate up and down

In principle, you could. People are using the functions that work best. If somebody found a &quot;crazy&quot; activation function that yielded good results it would certainly get used. However, using a &quot;crazy&quot; activation function probably makes it much harder to train the network. Part of the reason why e.g. ReLU is popular is because its simplicity makes training easier.