EDGE OF STOCHASTIC STABILITY: REVISITING THE EDGE OF STABILITY FOR SGD
Edge of Stability;Stochastic Gradient Descent;Neural Networks
We posted the wrong version by mistake and noticed only after the deadline.

Best regards,

The Authors
I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.
@misc{
andreyev2024edge,
title={{EDGE} {OF} {STOCHASTIC} {STABILITY}: {REVISITING} {THE} {EDGE} {OF} {STABILITY} {FOR} {SGD}},
author={Arseniy Andreyev and Pierfrancesco Beneventano},
year={2024},
url={https://openreview.net/forum?id=djnvlNHQR4}
}
Recent findings by Cohen et al. (2021) demonstrate that during the training of neural networks with full batch gradient descent at a step size of $\eta$, the sharpness—defined as the largest eigenvalue of the full batch Hessian—consistently stabilizes at $2/\eta$. These results have significant implications for generalization and convergence. Unfortunately, this was observed not to be the case of mini-batch stochastic gradient descent (SGD), thus limiting the broader applicability of these findings.
We empirically discover that SGD trains in a different regime we call Edge of Stochastic Stability. In this regime, what hovers at $2/\eta$ is, instead, the average over the batches of the largest eigenvalue of the Hessian of the mini batch loss—which is always bigger than the sharpness.
This implies that the sharpness is generally lower when training with smaller batches or bigger learning rate, providing a basis for the observed implicit regularization effect of SGD towards flatter minima and a number of well established empirical phenomena.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Arseniy_Andreyev1 ~Pierfrancesco_Beneventano1
Arseniy Andreyev Pierfrancesco Beneventano
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Edge of Stability Stochastic Gradient Descent Neural Networks
I certify that there is no acknowledgement section in this submission for double blind review.
andreyev|edge_of_stochastic_stability_revisiting_the_edge_of_stability_for_sgd
/pdf/0425fa90c1721bf5d2efecae99db29ccf5a90401.pdf
optimization
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
No
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
EDGE OF STOCHASTIC STABILITY: REVISITING THE EDGE OF STABILITY FOR SGD
ICLR 2025 Conference Withdrawn Submission
ICLR.cc/2025/Conference/Withdrawn_Submission