Why is the "dying ReLU" problem not present in most modern deep learning architectures? The $ReLU(x) = max(0,x)$ function is an often used activation function in neural networks.
However it has been shown that it can suffer from the dying Relu problem (see
also What is the &quot;dying ReLU&quot; problem in neural networks?)
Given this problem with the ReLU function and the often seen suggestion to use a leaky ReLU instead, why is it that to this day ReLU remains the most used activation function in modern deep learning architectures? Is it simply a theoretical problem that does not often occur in practice? And if so, why does it not occur often in practice? Is it because as the width of a network becomes larger the probability of dead ReLUs becomes smaller (see Dying ReLU and Initialization: Theory and Numerical Examples
)?
We moved away from sigmoid and tanh activation functions due to the vanishing gradient problem and avoid RNN's due to exploding gradients but it seems like we haven't moved away from ReLUs and their dead gradient? I would like to get more insights in to why.
In practice, dead ReLUs connections are not a major issue. Most deep learning networks can still learn an adequate representations with only sub-selection of possible connections. This is possible because deep learning networks are highly over-parameterized.
Even with the possible drawbacks backs of the dying ReLUs problem, the computational effectiveness and efficiency of ReLUs still make them one of the best options currently available.