Are there relatively new research papers that describe how to make back-propagation more efficient? I read Yann LeCun's paper Efficient BackProp, which was published in 2000. I looked for similar but more recent papers on Arxiv, but I have not yet found any.
Are there relatively new research papers that describe how to make back-propagation more efficient?
So, I am looking for papers similar to Efficient Backprop by LeCun but newer. The papers could describe why ReLU now &quot;dominates&quot; tanh or even sigmoid (but tanh was Yann's favorite, as explained in the paper). ReLU is just one thing I am interested in, but the paper could also analyze e.g. the inputs from a statistical standpoint.
Here is a paper that explains why ReLU rules.
What we want is to disentangle data of different classes. In order to do that, we need a discontinuous mapping for the data. ReLU easily allows for that. It is even better than LeakyReLU, sigmoid and tanh in that regard. Also, the reason any of the activations work is because of the floating point error, there is inadvertently a discontinuous mapping for the whole data. I have also explained it here.