Title: The Shattered Gradients Problem: If resnets are the answer, then what is the question?,
Abstract: A long-standing obstacle to progress in deep learning is the problem of
vanishing and exploding gradients. Although, the problem has largely been
overcome via carefully constructed initializations and batch normalization,
architectures incorporating skip-connections such as highway and resnets
perform much better than standard feedforward architectures despite well-chosen
initialization and batch normalization. In this paper, we identify the
shattered gradients problem. Specifically, we show that the correlation between
gradients in standard feedforward networks decays exponentially with depth
resulting in gradients that resemble white noise whereas, in contrast, the
gradients in architectures with skip-connections are far more resistant to
shattering, decaying sublinearly. Detailed empirical evidence is presented in
support of the analysis, on both fully-connected networks and convnets.
Finally, we present a new "looks linear" (LL) initialization that prevents
shattering, with preliminary experiments showing the new initialization allows
to train very deep networks without the addition of skip-connections.