Steepest Descent in the Modular Norm
Adam;Shampoo;Prodigy;optimizers;optimization;steepest descent;norms;modular norm;spectral norm
We decided we want to further hone the manuscript prior to conference submission. We are withdrawing now to avoid wasting reviewer time.
I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.
We cast optimizers like Adam and Shampoo as steepest descent methods under different norms. Generalizing this idea opens up a new design space for training algorithms
@misc{
bernstein2024steepest,
title={Steepest Descent in the Modular Norm},
author={Jeremy Bernstein and Laker Newhouse},
year={2024},
url={https://openreview.net/forum?id=JimfKP7qrU}
}
An old idea in optimization theory says that since the gradient is a dual vector it may not be subtracted from the weights without first being mapped to the primal space where the weights live. We take this idea seriously in this paper and construct such a gradient mapping for neural networks. We call our mapping modular dualization, and show how it may be used to derive steepest descent optimization methods in an architecture-aware norm called the modular norm. We prove that the Adam and Shampoo optimizers (without exponential moving averages) emerge as special cases of modular dualization. We go on to develop new special-purpose dualize functions for linear, embedding and convolution layers. And we contribute a novel, GPU-friendly algorithm for dualizing under the spectral norm. We hope that the technique of modular dualization might enable a next generation of fast and scalable optimizers that are well-matched to general neural architectures.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Jeremy_Bernstein1 ~Laker_Newhouse1
Jeremy Bernstein Laker Newhouse
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Adam Shampoo Prodigy optimizers optimization steepest descent norms modular norm spectral norm
I certify that there is no acknowledgement section in this submission for double blind review.
bernstein|steepest_descent_in_the_modular_norm
optimization
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Steepest Descent in the Modular Norm
ICLR 2025 Conference Withdrawn Submission
ICLR.cc/2025/Conference/Withdrawn_Submission