Why are sigmoid/tanh activation function still used for deep NN when we have ReLU?
In certain network structures having symmetric activation layers has advantages (certain autoencoders for example)
In certain scenarios having an activation with mean 0 is important (so tanh makes sense).
Sigmoid activation in the output layer is still important for classification


In 95% of the cases ReLU is much better though.