ANaGRAM: A Natural Gradient Relative to Adapted Model for efficient PINNs learning
PINNs;SciML;PDEs;Natural Gradient;Neural Tangent Kernel
Yes
No ethics review needed.
1. Implementing an adaptive selection strategy for the SVD cutoff factor based on the eigenvalue spectrum could make the method easier to use.

2. Developing an adaptive method for selecting collocation points based on the geometry of the domain could improve convergence rates.
1.The paper's use of differential geometry to formulate a natural gradient specifically for PINNs counts as a theoretical contribution. By aligning gradient updates with the functional space geometry, ANaGRAM offers a alternative to gradient-based methods, potentially addressing some of the known limitations in PINN optimization.

2 NaGRAM's computational complexity, which scales favorably with parameter and batch sizes, is an improvement over traditional natural gradient methods.

3. Experimental results indicate that ANaGRAM consistently achieves lower error rates compared to baseline methods like E-NGD and standard optimizers (Adam, L-BFGS).
The paper "ANaGRAM: A Natural Gradient Relative to Adapted Model for Efficient PINNs Learning" introduces an innovative natural gradient optimization algorithm tailored for Physics-Informed Neural Networks (PINNs). Named ANaGRAM, this approach leverages differential geometric perspectives to optimize PINNs, aiming to enhance computational efficiency and accuracy in solving Partial Differential Equations (PDEs) in a machine learning framework.
1. The current implementation of ANaGRAM requires manual tuning of hyperparameters, such as the cutoff factor for SVD. 

2. The choice of collocation points (batch points) is limited to heuristics in this study. Developing an adaptive method for selecting these points could further improve ANaGRAM's performance.

3. The results indicate that the efficacy of ANaGRAM may diminish for nonlinear differential operators, where the method's equivalence with traditional approaches is less certain.
Yes
No ethics review needed.
I find the empirical investigations sufficient for this paper. It might be even better to have a high dimensional example. For example, would ANaGRAM still perform better than E-NGD when tested on 100 D Laplace eqn?
The paper is very well written. The proposed method is elegant and well motivated from the theory perspective. Empirical results are well-done and show good evidence for the proposed ANaGRAM.
The paper proposes a new algorithm -- ANaGRAM-- that is well suited for learning PDEs with boundary conditions. The algorithm makes use of a clever manipulation of natural gradient of Amari & Douglas. The natural gradient, PINN and some elements from differential geometry are introduced. The proposed algorithm can be interpreted as follows: compute the integration of the euclidean gradient using by "weighting" with the appropriate Green's function and use pseudo-inverse of the differential as a pre-conditioner (Theorem 1). Experiements show that ANaGRAM learns faster than Adam, GD, E-NGD across well-known low-dimensional PDEs.
I quite like the paper. Even though I am not an expert of PINNs at all, I am guessing that this paper may be exciting for that community. I found the math well written, which may be not very common for an ICLR submission. 

Typos:
- Eq 2: $h_i \to h_p $
- line 147: "his" comp. cost $ \to $ "its" comp. cost
- line 199: maybe add what $\Delta_{\theta_r}$ means here (singular values) 

The performance of ANaGRAM is quite similar to E-NGD for Heat eqn and 5D Laplace eqn. ANaGRAM is faster by a tiny margin. I did not find a discussion of this in the paper. Can the authors comment on this? In particular, what would be the expected advantage of  ANaGRAM over E-NGS in words?
Yes
No ethics review needed.
1. It's interesting that ANaGRAM usually has the largest variance among all methods, is there a reason for that? I was thinking it should have small variance since it's more well conditioned.

2. For some experiments, it seems the difference between different methods is more stark for L2 loss compared to test Loss. Any explanation for this?

3. Could you explain more about the relationship between NTK and equation 19 (line 223)?
This paper has an interesting and supportive theoretical results for their results.
This paper proposed a natural gradient based optimizer called ANaGRAM for PINNs. The key algorithmic contribution of the paper includes a novel approach to discretize the natural gradient by connecting the natural gradient with a linear regression in L2 space, which is then combined with the fact that PINNs can be regarded as a least square regression to adapt the method for PINNs. A major theoretical contribution of this is showing that the natural gradient for PINNs is actually Green’s function under some conditions. 

In summary, this paper proposed a novel optimization approach which can be used to help improve the convergence of the usually ill-condition problems in PINNs.
1. There are many typos 126 il conditioned -> ill condition 186 hapens -> happens 201: writes -> write 274 quadraic....
2. The flow of the paper may be better by revisiting some parts. For example, at the end of the section 2, authors illustrate the relationship between the natural gradient and the linear regression problem. However, the section 3.1 has the title indicating the same thing. Maybe the end of section 2 can be moved to section 3.1, otherwise, I don't see the reason why section 3.1 has the current titles.
3. The experiments are not enough. Usually people also include Burgers, Schrodinge, etc.
4. The paper's main theoretical result is the equivalence of Green's function and natural gradient. It would be helpful to include some toy examples to justify this finding.
Yes
No ethics review needed.
Please see the weakness part.

One typo: Lines 126-127: ill conditioned
(1) The natural gradient descent approach is theoretically sound, offering a promising solution for improving the convergence of regression problems and PINNs. Notably, while gradient descent may struggle with PINNs involving nonlinear PDEs, natural gradient descent effectively addresses this limitation.


(2) The paper's presentation is clear and well-structured.
This paper studies the natural gradient descent, which is theoretically and empirically believed to improve the convergence of training PINNs. Theoretical analysis on the natural gradient and training PINNs bridges connections on Green function theory.
(1) The algorithm in the paper is not novel. The general natural GD method requires O(P^3) computational costs, due to the inversion of the Gram matrix of \hat{\phi} (G = \hat{\phi} \hat{\phi}^{T} is in size of P \times P). This paper applies SVD directly on \hat{\phi}. Moreover, the line 6 of the algorithm is exactly equivalent to the vanilla natural GD algorithm (rewriting the pseudo-inverse with the SVD). Therefore, I do not find the novelty of the algorithm. 


(2) In Equations (18) and (19), it seems that the paper tries to explain the consistency by approximating the integral by finite sum of some observations. However, the results of Corollary 1 are relatively weak. The existence of certain points, as stated, follows naturally from standard regularity conditions (e.g., smoothness) of functions. The result is therefore not surprising and lacks substantial implications. It is better if the paper can alternatively prove that if we randomly pick P training samples as observations and with high probability, the difference between du and \hat{\phi} \hat{f} is upper bounded, e.g., O(1/P^{\alpha}), where \alpha is a constant depending on function’s conditions and dimensionality. Such findings would be more meaningful for machine learning.


(3) The ANaGRAM for PINNs closely resembles existing natural GD for training PINNs (https://arxiv.org/abs/2408.00573). For example, in Equation (31) of the paper, the pseudo-inverse of (J J^{T}) is equivalent to the SVD of \hat{\phi} in this paper. 


(4) The natural GD requires the full-batch training, which limits its adaptability to stochastic algorithms. We know that the SGD-like algorithms are widely used in deep learning due to their benefits, such as improving the generalization and bypassing spurious local minima. A more practical approach would be one that can adapt effectively to a stochastic setting.
Yes
No ethics review needed.
- Can the authors add a reference for the computational complexity of the SVD for the rectangular solve?

- Why is the proposed method performing better than ENGD for the linear equations? I would expect similar performance as it is the same algorithm (up to numerical linear algebra).

- it is very common to include damping into natural gradient methods (adding a scaled identity term to the square Gramian). Can this be realized in this setting?
- The viewpoint outlined above leads to a performant method, that works better than ENGD for nonlinear PDEs.

- The improvement in complexity for the linear solve is helpful.
The manuscript proposes a natural gradient method related to the ENGD optimizer proposed by (Müller and Zeinhofer). The main contributions are:

- In the context of PINNs, the authors view the ansatz/parametric model as a neural network followed by the application of the PDE and boundary operator. This means that the model is defined on some parameter space $\Theta$, taking values in the product space $L^2(\Omega) \times L^2(\partial\Omega)$ via the intermediate space $\mathcal H$ which is related to the PDE and boundary operators of the specific problem. This mathematical trick of changing what is the model and what is the loss function allows to view PINNs as a standard regression problem. This viewpoint motivates the authors to use an $L^2(\Omega)\times L^2(\partial\Omega)$ based natural gradient method, which agrees with ENGD for linear PDEs but performs better than ENGD for nonlinear PDEs

- The second contribution is on the numerical linear algebra level. Instead of solving the normal equations of the regression, the authors propose to use an SVD directly for the Jacobian of the model. Here, when I write Jacobian I mean the discretized differential of the parametrization, which is denoted by $\hat \phi_{\theta(t)}$ in the manuscript. This gives an improvement in computational complexity from $\mathcal O(P^3)$ for solving the normal equations, to $\mathcal O (\min(N^2P, P^2N) )$, as the Jacobian is of shape $(N, P)$, where $P$ is the dimension of the parameter space $\Theta$ and $N$ is the number of sampling points.
- The viewpoint of absorbing the differential operators into the model can also be understood as a Gauss-Newton method in function space compare to (Jnini et al, arXiv:2402.10680). Moreover, it leads to the standard Gauss-Newton algorithm in parameter space. The manuscript thus does not provide a novel algorithm, but a novel derivation of a known algorithm. While I like this idea and it is certainly a contribution, I think it is not significant enough for an exceptional venue such as ICLR.
We propose a new natural gradient that scales as $\min(P^2S, S^2P)$, $P$ the number of parameters and $S$ the batch size, focusing on its application to PINNs traning.
@inproceedings{
anonymous2024anagram,
title={{AN}a{GRAM}: A Natural Gradient Relative to Adapted Model for efficient {PINN}s learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=o1IiiNIoaA},
note={under review}
}
In the recent years, Physics Informed Neural Networks (PINNs) have received strong interest as a method to solve PDE driven systems, in particular for data assimilation purpose. This method is still in its infancy, with many shortcomings and failures that remain not properly understood.
In this paper we propose a natural gradient approach to PINNs which contributes to speed-up and improve the accuracy of the training.
Based on an  in depth analysis of the differential geometric structures of the problem, we come up with two distinct contributions:
(i) a new natural gradient algorithm that scales as $\min(P^2S, S^2P)$, where $P$ is the number of parameters, and $S$ the batch size;
(ii) a mathematically principled reformulation of the PINNs problem that allows the extension of natural gradient to it, with proved connections to Green's function theory.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
PINNs SciML PDEs Natural Gradient Neural Tangent Kernel
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/52975f7f6d49242f71bcbd2fcb7d091394d441b5.pdf
applications to physical sciences (physics, chemistry, biology, etc.)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/fbf05a05b52ef3eca7a6fbbd2ee5ce83b5631975.pdf
ANaGRAM: A Natural Gradient Relative to Adapted Model for efficient PINNs learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission