Why are sigmoid/tanh activation function still used for deep NN when we have ReLU? Looks like ReLU is better then sigmoid or tanh for deep neural networks from all aspects:


simple
more biologically plausible
no gradient to vanish
better performance
sparsity


And I see only one advantage of sigmoid/tanh: they are bounded. It means that your activations won't blow up as you keep training and your network parameters won't take off to the sky.

Why should not we forget about sigmoid/tanh for deep neural networks?
In certain network structures having symmetric activation layers has advantages (certain autoencoders for example)
In certain scenarios having an activation with mean 0 is important (so tanh makes sense).
Sigmoid activation in the output layer is still important for classification


In 95% of the cases ReLU is much better though.