Using the cosine activation function seems to perform very badly
Cosine is not a commonly used activation function.

Looking at the Wikipedia page describing common activation functions, it is not listed.

And one of the desirable properties of activation functions described on that page is:


  Approximates identity near the origin: When activation functions have this property, the neural network will learn efficiently when its weights are initialized with small random values. When the activation function does not approximate identity near the origin, special care must be used when initializing the weights.


$cos(0) = 1$, a basic cosine function does not have this property. Combined with its periodic nature, this makes it look like it could be particularly tricky to get correct starting conditions and other hyper-parameters in order to have a network learn whilst using it.

In addition, cosine is not monotonic, which means that error surface is likely to be more complex than for e.g. sigmoid.

I suggest trying with a low learning rate, and initialising all the bias values to $-\frac{\pi}{2}$. Maybe reduce the variance in initial weights a little too, just to start off with things close to zero. Essentially this is starting with $sin()$. Caveat: not that I have tried this myself, just an educated guess, so I would be interested to know if that helps at all with stability.