Is computational power of Neural networks related to the activation function
Just a note:  


rational-weighted recurrent $NN$s having boolean activation functions (simple thresholds) are equivalent to finite state automata (Minsky, "Computation: finite and infinite machines",  1967);
rational-weighted recurrent $NN$s having linear sigmoid activation functions  are equivalent to Turing Machines (Siegelmann and Sontag, "On the computational power of neural nets", 1995);
real-weighted recurrent $NN$s having linear sigmoid activation functions are more powerful than Turing Machines (Siegelmann and Sontag, "Analog computation via neural networks", 1993);


but ...


real-weighted recurrent $NN$s with Gaussian noise on the outputs cannot recognize arbitrary regular languages (Maass and Sontag, "Analog Neural Nets with Gaussian or Other Common NoiseDistributions Cannot Recognize Arbitrary Regular Languages" , 1995);