Exploring the Impact of Activation Functions in Training Neural ODEs
Neural ODEs;Gradient Descent;Neural Tangent Kernel (NTK)
Yes
No ethics review needed.
I don't think it's quite right (or at least I don't understand exactly why, but I'm not a specialist in Neural ODEs) that the title of the paper focuses on the impact of activations and their regularities, whereas most of your results use these as hypotheses for theorem proving. In my opinion, you should assume a more theoretical title (or present more results comparing activations, but it seems to me that this is a slightly different job than what you present, which is amply sufficient from a theoretical point of view).
- precise, rigorously demonstrated theoretical results
- clear presentation
- extension of fundamental NTK results to neural ODEs, using classical ODE analytical tools (Picard theorem, Groenwald theorem etc).
This paper presents several theoretical results about Neural ODEs, built around sufficient regularity of activations:
- It proves that "optimize then discretize" setting, built upon the use of classical ODE solvers, is well-posed under the assumption of Lipschitz-continuous activations (proposition 1).
- It proves that "discretize-then-optimize" setting, built as an Euler scheme discretization of the ODE by growing the depth of a residual neural network, is also well posed in the sense that the solution of the discretization scheme converges to the actual solution of the ODE in the infinite depth limit, if the activations and there derivatives are lipschitz-continuous (proposition 2). It has to be noted that authors gives explicit convergence rates.
- It shows that under the "discretize-then-optimize" setting with fixed depth L, the associated residual network converges in infinite width limit to a fixed kernel with explicit form, under the assumption of Lipschitz continuity of activations (proposition 3).
- It shows that under the assumption of Lipschitz continuity of activations, the infinite depth and width limits commute, implying that the neural ODE converges to a Gaussian process with covariance given as the infinite depth limit of the covariance of proposition 3 (theorem 1).
- Similar results are shown for the convergence of the NTK (proposition 4 and theorem 2)
- In section 5, authors show that the resulting limiting covariance of the gaussian process as well as NTK are Semi-positive definite (lemma 3, proposition 5, corrolary 1), resulting in derivation of explicit convergences rates (theorem 3).
Finally empirical evidences of the theoretical results are presented in section 6.
- Typo : the index of the last h in equation 1 should probably be T instead of t
- Figure 1 : a,b,c should rather be in log-log scale to better appreciate the slope comparison with 1/L.
Yes
No ethics review needed.
1.
Could you provide a numerical experiment that specifically highlights the convergence problem of the "Discretize-then-optimize" method when using an activation function that does not meet the Lipschitz continuity requirements? This would strengthen the paper by providing empirical support for your theoretical analysis.

2.
How does the choice of numerical ODE solver in the "Optimize-then-discretize" method impact the accuracy of the gradient computation and the overall training dynamics? Exploring the sensitivity of the method to different solvers would be valuable.

3.
What are the practical implications of your findings for the choice of activation functions in real-world applications of Neural ODEs? Providing guidance on activation function selection based on the desired trade-off between theoretical guarantees and empirical performance would be beneficial.

Addressing these points would significantly improve the paper and provide a more comprehensive understanding of the impact of activation functions on Neural ODE training.
Theoretical Analysis: The paper offers a rigorous mathematical analysis of the well-posedness of Neural ODE solutions and their gradients, focusing on the impact of activation function smoothness. The authors clearly demonstrate the importance of Lipschitz continuity for both the activation function and its derivative to ensure stable and accurate gradient computation.

NTK Analysis: The paper extends the concept of the Neural Tangent Kernel (NTK) to the continuous setting of Neural ODEs. This is a valuable contribution, as it provides insights into the training dynamics of these models. The authors show that the NTK converges to a deterministic limit under certain conditions, which helps explain the convergence properties of gradient descent in Neural ODE training.

Global Convergence Proof: The authors establish the global convergence of Neural ODEs under gradient descent in the overparameterized regime, under assumptions about the activation function's nonlinearity and non-polynomial nature. This theoretical result provides a strong foundation for understanding the optimization behavior of Neural ODEs.

Empirical Validation: The paper presents several numerical experiments to validate the theoretical findings. These experiments demonstrate the convergence of Neural ODE outputs and gradients to their finite-depth ResNet approximations, the behavior of the NTK, and the convergence properties of Neural ODEs under gradient descent.
This paper investigates the role of activation functions in the training dynamics of Neural ODEs, particularly focusing on the convergence and stability properties of two key methods: "Optimize-then-discretize" and "Discretize-then-optimize." The authors establish theoretical guarantees for the convergence of both methods under certain conditions on the activation functions, and they provide numerical experiments to support their findings.
Lack of Demonstration for "Discretize-then-optimize" Convergence Issue: While the paper theoretically analyzes the convergence problem of "Discretize-then-optimize" with non-smooth activation functions, it does not provide a dedicated numerical experiment to showcase this issue. This would be valuable for empirically confirming the theoretical claims and for understanding the practical implications of using non-smooth activations with this method.

The lack of a numerical demonstration of the "Discretize-then-optimize" convergence problem weakens the overall impact of the paper. Without empirical evidence, the theoretical claims regarding the convergence issue remain unsubstantiated, and the practical implications of using non-smooth activation functions with this method remain unclear.
Yes
No ethics review needed.
No questions.
The paper is well-written. The ideas presented are novel and represent a significant contribution in the context of neural ODEs. The main strength is the theoretical foundation of the work.
The authors studied activation functions in the training dynamics of neural ODEs. They showed that the choice of activation function has significant impacts on the dynamic and asymptotic behavior. Additionally, they propose a new mathematical framework to study continuous models from the perspective of approximation theory. Finally, the authors conduct experiments to empirically validate their theoretical results.
The proofs are generally well-written and clear, but since they are in the appendix, the flow of the paper's main narrative can sometimes be interrupted.
Yes
No ethics review needed.
See weakness.
(1) The paper extends NTK theory from general regression problems with ReLU nets to Neural ODEs with ResNet architectures. The analysis and results are novel for Neural ODEs.

(2) The empirical results validate the theorems.
This paper studies the convergence of gradient descent for training neural ODEs from the neural tangent kernel perspectives, with experimental results validating the theoretical claims.
(1) Some prior work has extended NTK theory to analyze CNNs (for image classification tasks), PINNs, Transformers, and various gradient-based algorithms beyond vanilla gradient descent. However, related references are missing. 

(2) SGD is more commonly used in practice, making a convergence analysis of SGD particularly valuable and interesting.

(3) The results are specific to ODEs of the form given in Equation (2). Theorem 2 does not hold for PINNs if the underlying PDE is nonlinear. How would Theorem 2 apply if the ODE were not limited to this special case?

(4) This paper claims that the nonlinear and non-polynomial nature of the activation function is crucial for convergence. However, I believe the convergence results (Theorem 3) may also hold with polynomial activation functions, provided that Condition (1) of Assumption 1 is further strengthened to ensure linear independence.

(5) From the title, the topic of the paper is exploring the smoothness of activation functions in neural ODEs. However, after reading your paper, your main contribution is developing the global convergence of GD in training neural ODEs.
@inproceedings{
anonymous2024exploring,
title={Exploring the Impact of Activation Functions in Training Neural {ODE}s},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=AoraWUmpLU},
note={under review}
}
Neural Ordinary Differential Equations (ODEs) have been successful in various applications due to their continuous nature and parameter-sharing efficiency. However, these unique characteristics also introduce challenges in training, particularly with respect to gradient computation accuracy and convergence analysis. In this paper, we address these challenges by investigating the impact of activation functions. We demonstrate that the properties of activation functions—specifically smoothness and nonlinearity—are critical to the training dynamics. Smooth activation functions guarantee globally unique solutions for both forward and backward ODEs, while sufficient nonlinearity is essential for maintaining the spectral properties of the Neural Tangent Kernel (NTK) during training. Together, these properties enable us to establish the global convergence of Neural ODEs under gradient descent in overparameterized regimes. Our theoretical findings are validated by numerical experiments, which not only support our analysis but also provide practical guidelines for scaling Neural ODEs, potentially leading to faster training and improved performance in real-world applications.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Neural ODEs Gradient Descent Neural Tangent Kernel (NTK)
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/ae481a879c2a856fe1029169bbf2d234b0c1c5e3.pdf
learning theory
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/ced94840a96fd0d49127b367d9dd6de1b74ff9d3.pdf
Exploring the Impact of Activation Functions in Training Neural ODEs
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission