A second-order-like optimizer with adaptive gradient scaling for deep learning
deep learning;second-order methods;stochastic optimization;dynamical systems
Yes
No ethics review needed.
See above.
The derivation of the algorithm is clear and the performance often seems competitive with AdamW. Large-scale models such as ViT-B/32 and GPT-2 were trained.
This paper introduces INNAprop, a combination of the INNA optimizer with RMSprop gradient scaling. The resulting method has memory requirements similar to methods such as AdamW. Experiments show that this optimizer is competitive with AdamW.
The INNAprop algorithm seems like a relatively straightforward extension from INNA. From what I can tell Algorithm 2 can be derived directly from the INNA equations in Table 2 and adding the factor $\\frac{1}{\\sqrt{v_{k+1}} + \\varepsilon}$ to the gradient. The motivation from a continuous dynamical systems perspective and the derivation done in appendix B seem equivalent to those from the INNA paper. This means that the theoretical insights from this paper seem limited.

This in itself wouldn't be a bad thing if the paper could make strong experimental claims about this small tweak to INNA leading to consistently stronger performance. However, this seems difficult to conclude for a few reasons, such as:

* No comparisons were made with INNA (without the RMSprop scaling) or other algorithms (Adam, ADAGRAD, SGD)
* In figures 3 and 6 INNAprop is sometimes indistinguishable from AdamW

Another surprise is that the optimal values for $\\alpha$ and $\\beta$ seem quite different to those found in the INNA paper (which ends up using $(0.1, 0.1)$ and $(0.5, 0.1)$) and it would be great to understand why. (In general, I am not convinced that the comparison with AdamW was the way to go, given that the proposed algorithm is far more similar to INNA. It would be insightful to know how the RMSprop extension changes INNA's behaviour and the choice of optimal hyperparameters for INNA.)

I understand that with limited compute it can be difficult to run large amounts of experiments, but I don't think that in it's current form this paper really provides new insights to the community: The RMSprop addition to INNA is pretty straightforward, and whether it really makes a difference is still hard to say.
Yes
No ethics review needed.
Can you explain how you "systematically favored AdamW through the choice of recommended hyperparameters (scheduler, learning rates, weight decay)"? In more established tasks, I feel this is a reasonable claim. However, where INNAprop showed wins (finetuning on an uncommon dataset and training tiny GPT-2 models), I expect that standard AdamW hyperparameters are suboptimal. How much compute budget was spent tuning AdamW compared to INNAprop? I am also concerned that two different settings of INNAprop are shown in all experiments against a single choice of hyperparameters for AdamW.

In the appendix, you derived the DINAdam variant. Do you also explore this algorithm empirically?

With my initial review, I am recommending that the paper be rejected. I believe the paper is technically correct and the proposed algorithm is sufficiently novel. However, I do not believe that the algorithm provides a meaningful practical contribution for the wider community at this time. My concern is that a practitioner would always choose AdamW over INNAprop due to the marginal expected gains, increased risk of failure, and the additional overhead of parameter tuning required. My score would change if the authors could provide clear guidance on when a practitioner would choose INNAprop over AdamW and expect a win. 

Minor points:

- "we provide quite extensive experiments" vague statement in abstract. I would prefer that this be made more precise.
- "extra second-order geometrical intelligence" vague technical term. What do you mean by this? I think this should be clarified in the main text.
I thought the introduction was strong and successfully explained the need for improved optimization algorithm algorithms in terms of real-world impact. The authors back this up by deriving a memory-efficient version of their algorithm matching the memory overhead of existing approaches.

A thorough discussion of the algorithm and how it can be derived is provided. The authors also discuss comparisons between INNAprop and several other methods in the literature, in terms of its derivation. This is a valuable contribution that helps to place the proposed method among existing work. On the surface, I felt that the combination of the two methods could be of limited novelty. However, after reviewing the derivation I feel this constitutes a reasonably strong contribution.

The paper is clear and well-written. The authors provide a thorough set of diverse experiments to investigate the proposed method and include multiple runs to account for variability. From my review of the manuscript, all results look sensible and technically sound.
This paper proposes a new first-order optimization algorithm: INNAprop. The algorithm combines the INNA method with RMSProp to derive a novel optimization scheme. The authors evaluate INNAprop empirically against AdamW, the de facto standard deep learning optimizer. They explore a set of diverse tasks spanning image classification and natural language processing using several different classes of deep learning architectures.
Claiming second-order without computing second-order statistics doesn't seem accurate to me. The finite difference approximation may not be sufficiently accurate to achieve these benefits. However, I am not overly familiar with this approximation that the authors adopt from prior work. One way to alleviate this concern would be to provide additional justification for this terminology. Or, to include empirical evidence that the finite-difference approach is providing a good approximation of the actual second order statistics.

One of the claimed contributions is that INNAprop "matches or outperforms AdamW in both training speed and final accuracy on benchmarks". However, Fig. 2 shows the faster training speed of AdamW (the final error bars overlap for test accuracy so I'd agree that they match performance). Fig. 3 shows INNAprop as faster to converge for ResNet, but slower once again for ViT. The test accuracy is once again matched in both experiments. Fig. 4 shows a similar convergence speed with a modest (~1%) increase in accuracy for INNAprop. Finally, Fig. 5 shows a modest win for INNAprop on the smaller settings and matched performance for the larger model. For the LoRA finetuning (Fig. 6), there is a small win for the small setting and INNAprop matches AdamW on the medium and large setting.

The improvement over Adam variants is small across the board. The notable wins are on vision finetuning (Fig. 4) and small-scale GPT-2 training (Fig. 5). There are many methods that propose similarly slim margins and none are adopted by the community. Furthermore, under scrutiny from third-party implementations, the gains are often difficult to replicate [1]. Sadly, my honest opinion is that this method will not gain traction with the community --- it is unclear when it should be used over AdamW.

While often spurious, it is common for optimization algorithms to include theoretical analysis of their convergence. I do not consider this a critical weakness of this paper.

[1] "Descending through a Crowded Valley — Benchmarking Deep Learning Optimizers", Schmidt et al.
Yes
No ethics review needed.
I noticed that you mentioned the lack of computational resources two times (lines 259 and 359). You should know, that I am very sympathetic to that challenge. Still, my suggestion is that you bite the apple and invest that extra time and computational cost --- to really make a convincing case for your method :-)
It is a comprehensive and well-written paper that does a good job of explaining the motivation for, and the math behind, the proposed method. I do appreciate the main thesis of the work, and the effort that went into it. I like the discussion of, and approach to, hyperparameter tuning, as well as the comprehensive Appendix.
The authors introduce a second-order optimization method, INNAprop, which combines the INNA method with RMSprop, while not increasing the memory-footprint. The method is evaluated on a few image classification benchmarks, and GPT-2 with the OpenWebText dataset, and directly compared to AdamW with somewhat favorable results.
My main objection is the empirical evaluation. Firstly, the reported improvements (on test sets) are generally very marginal. Secondly, the results are not pushed to SOTA-level --- even for the chosen architectures (that are pretty old and pretty small). Thirdly, there are too few experiments to convince me that the proposed method really works better than, or even as well as, AdamW. Since the method is not faster, or less computationally demanding, I find myself unconvinced of its overall usefulness.
Yes
No ethics review needed.
-   How does INNAprop compare in terms of wall-clock time to AdamW or Sophia?
-   How does INNAprop compare to Sophia/Lion and other more recent high performing optimizers?
-   Is the increase in test accuracy of INNAprop compared to AdamW statistically significant? What are the error ranges?
The paper is well written, and easy to follow.
-    The derivation of the algorithm is well motivated.
-    They leverages second order information without requiring the hessian.
-    The algorithm seems to perform on par or even outperform AdamW in certain settings like language modeling.
-    The experiments cover a good amount of tasks between image classification and language modeling.
-    The authors open-source the code.
The paper combines RMSprop adaptive gradient scaling to the existing INNA optimizer, which is based on the Dynamical Inertial Newton. The obtained optimizer leverages second-order information while having similar computational requirements to first-order methods. With appropriate hyper-parameters the optimizer matches and in some cases outperforms AdamW.
-   The introduction of the paper makes a big deal about the energy consumption of training large language models, but the authors don't measure or show how their method compares to the competitors in terms of wall-clock time in their experiments. Particularly interesting would be seeing how wall-clock time on ImageNet or language modeling tasks compare.
-   The method lacks comparison with more recent optimizers other than AdamW. Sophia for example is another fast and efficient second-order optimizer paper, which the author cite, but don't compare with. Lion is another recent and high performing optimizer, which was cited but not compared with either.
-   Numeric results are reported without standard deviation/standard error or confidence intervals
Using a second-order in time minimization system with Hessian friction and gradient adaption à la RMSprop, we introduce and benchmark a new  stochastic second-order optimizer.
@inproceedings{
anonymous2024a,
title={A second-order-like optimizer with adaptive gradient scaling for deep learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=r4Q86nBQka},
note={under review}
}
In this empirical article, we introduce INNAprop, an optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. It leverages second-order information and rescaling while keeping the memory requirements of standard DL methods as AdamW or SGD with momentum. After having recalled our geometrical motivations, we provide quite extensive experiments. On image classification (CIFAR-10, ImageNet) and language modeling (GPT-2), INNAprop consistently matches or outperforms AdamW both in training speed and accuracy, with minimal hyperparameter tuning in large-scale settings. Our code is publicly available at \url{https://github.com/innaprop/innaprop}.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
deep learning second-order methods stochastic optimization dynamical systems
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/705e437ff5e871b806aa9dc6b4202be0a6e91a6f.pdf
optimization
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
A second-order-like optimizer with adaptive gradient scaling for deep learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission