A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with Stochastic Gradient Descent
Adam;almost sure convergence;l1 convergence;sample complexity;analysis framework
Some tiny typos need to fix.
I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.
We present a novel framework for analyzing Adam, which demonstrates that Adam's convergence guarantees can be aligned with those of SGD.
@misc{
jin2024a,
title={A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with Stochastic Gradient Descent},
author={Ruinan Jin and Xiao Li and Yaoliang Yu and Baoxiang Wang},
year={2024},
url={https://openreview.net/forum?id=mKZsokwMRb}
}
Adaptive Moment Estimation (Adam) is a cornerstone optimization algorithm in deep learning, widely recognized for its flexibility with adaptive learning rates and efficiency in handling large-scale data. However, despite its practical success, the theoretical understanding of Adam's convergence has been constrained by stringent assumptions, such as almost surely bounded stochastic gradients or uniformly bounded gradients, which are more restrictive than those typically required for analyzing stochastic gradient descent (SGD).

In this paper, we introduce a novel and comprehensive framework for analyzing the convergence properties of Adam. This framework offers a versatile approach to establishing Adam's convergence. Specifically, we prove that Adam achieves asymptotic (last iterate sense) convergence in both the almost sure sense and the \(L_1\) sense under the relaxed assumptions typically used for SGD, namely \(L\)-smoothness and the ABC inequality. Meanwhile, under the same assumptions, we show that Adam attains non-asymptotic sample complexity bounds similar to those of SGD.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Ruinan_Jin3 ~Xiao_Li5 ~Yaoliang_Yu1 ~Baoxiang_Wang1
Ruinan Jin Xiao Li Yaoliang Yu Baoxiang Wang
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Adam almost sure convergence l1 convergence sample complexity analysis framework
I certify that there is no acknowledgement section in this submission for double blind review.
jin|a_comprehensive_framework_for_analyzing_the_convergence_of_adam_bridging_the_gap_with_stochastic_gradient_descent
/pdf/c72c22b612e10526b1010133825edbc55ee9461a.pdf
optimization
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with Stochastic Gradient Descent
ICLR 2025 Conference Withdrawn Submission
ICLR.cc/2025/Conference/Withdrawn_Submission