Combining multiple neural networks with different activation functions I have 3 neural networks where each has as a different activation function: Sigmoid, Tanh and Softmax. I am planning to average their final predictions, but as we know the functions doesn't have the same range values. 

P = (P1 + P2 + P3)/3 


Where 0 &lt; P1 &lt; 1, -1 &lt; P2 &lt; 1, 0 &lt; P3 &lt; 1

Can I directly average the predictions or I need to perform a normalization to have all prediction fall into the same interval ?
As you are trying to average out the values, and given the three have different domains, it makes sense to bring all in same domain before averaging out. You can normalize P2 (tanh) to 0-1 and then average the values.

If you want to try another way, you can combine these three networks and take the three outputs p1, p2, and p3 and input it to some more dense layers and finally predict the single output. This way instead of averaging out, you are bringing in some nonlinearity which will be helpful to learn the task.