In how few updates can a multi layer neural net be trained?
In Don't Decay the Learning Rate, Increase the Batch Size, Smith et al. train ResNet-50 on ImageNet to 76.1% with only 2500 updates. Has anyone done it in less?

In The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent Sankararaman et al. present the concept of gradient confusion which slows convergence, and show that depth increases gradient confusion while overparameterization on width and skip connections reduces gradient confusion.

In Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash, Mikami et al. train ResNet-50 on ImageNet to 75.29% with some number of updates, but I can't do the math to compute out how many updates they must have used.

To summarise:


Larger mini-batches help, but give diminishing returns and cause other issues.
Deeper models tend to require more updates.
Wider layers need fewer updates.
Skip connections, label smoothing, and batch norm might help.
The best I have found so far on ImageNet/ResNet-50 to >75% is 2500 updates.