Equivariant Polynomial Functional Networks
neural functional network;equivariant model;polynomial layer;monomial matrix group
Yes
No ethics review needed.
1. How difficult is the implementation of the proposed nonlinear layers? Can the authors provide pseudo code?

2. Can the trained MAGEP-NFN be used on the wider networks (e.g. predict generalization for larger CNNs than in the Small CNN Zoo dataset or retraining the model would be necessary?
1. The paper tackles a challenging problem of modeling neural network weights and their symmetries.
2. The proposed nonlinear equivariant layers appear to be original and interesting. 
3. The paper seems theoretically sound, however, I haven't checked the correctness.
4. The results are better than for the linear polynomials (Monomial-NFN).
The paper proposes a new method to model neural network weights based on Equivariant Neural Functional Networks. Compared to previous approaches, this method is claimed to be more expressive while still efficient. This is achieved by adding nonlinear (polynomial) equivariant layers acting on weights. The paper presents an extensive theoretical framework and shows improved empirical performance on the INR-based tasks and generalization prediction.
The paper's main weaknesses are experiments and writing/presentation.

### The experiments are weak in several aspects:
1. The paper claims "low memory consumption and efficient running time", however these claims are not supported by empirical evidence. The previous paper (Tran et al., 2024) shows that it can reduce memory from 6GB to 0.5GB, however, the extra (stable) polynomial terms may increase memory/runtime of the method in this paper. 
2. No comparison to graph-based approaches (Lim et al.,2023; Kofinas et al., 2024) in Table 1-4 (ideally in terms of memory/efficiency vs results), so the significance of the results is hard to estimate.
3. Some inconsistencies in evaluation, which is not explained, making comparison to previous literature difficult. For example, in (Kofinas et al., 2024) for MNIST the INR classification performance was about 90-95% and for FashionMNIST 70-75%, while in this paper it's  77 and 62 respectively. On the Small CNN Zoo dataset (Unterthineretal.,2020) the method achieves 0.933, while graph-based approaches (Kofinas et al., 2024) achieve better results (0.935).
4. The shortcoming of the method is that it requires some manual design of the polynomials for each architecture, so it's unclear how to extend it to other architectures such as transformers or even if it's possible to use a trained MAGEP-NFN on the wider networks.
5. The experiments on INR and generalization prediction for simple small CNNs are not sufficient. Previous works (Lim et al.,2023; Kofinas et al., 2024) added more practical experiments with learning to optimize or diverse transformer architectures. Otherwise, the proposed approach appears to be overengineered for some specific toy cases. 

### Writing and presentation
1. "These specialized networks are known as neural functional networks (NFNs)(Zhouetal.,2024b)" is a questionable statement.
2. "graph-based equivariant NFNs suffer from high memory consumption and long running times" is an overstatement given that Tran et al., 2024 showed the memory consumption of those methods is at maximum 6GB and runtime at max 4.5 hours. So in practice, this does not seem to be a problem for any modern GPUs. Please rewrite or scale the experiments.
2. Related works is not informative, because it just list existing papers without discussing how the current work is connected (improved, different, etc.) to them. 
3. The whole Section 3, if I understand correctly, seems to be background (notation introduction, etc. from Tran 2024), and it's quite lengthy, which makes the amount of the original content in the paper quite small.
4. In section 4 it's not described how the equations 20-21 are different from the previous work (Tran 2024) or section 3. It would be more clear to directly contrast nonlinear and linear layers in this context to understand the contribution better.
5. Contribution 2 of the paper (linear independence) is overstated - there is no comprehensive study in the main text. In contrast, the graph-based approached handled diverse and complex architectures (such as transformers) more easily. 
6. The paper could be more accessible, e.g. by having illustrations of the proposed vs previous method.
Yes
No ethics review needed.
- How would one apply this approach to methods that support arbitrary computation graphs such as [1]?



**Reference**

[1] Universal neural functionals. Zhou et al.
This paper brings both theoretical and empirical contributions to NFNs.
The techniques proposed in the paper are well-motivated and rigorously derived.
Experiments and mathematical details are quite thorough, especially given how mathematically involved these approaches tend to be (I was able to follow the derivation reasonably well).

Overall, I believe MAGEP-NFN provides a concrete improvement in the line of work on NFNs and points towards a direction to improve NFNs further.
This paper proposes monomial matrix group equivariant polynomial neural functional networks (MAGEP-NFN), a new class of neural functionals whose input and output are the weights of neural networks. MAGEP-NFN builds on a prior work Monomial NFN which adds scale equivariance and sign equivariance of the original NFN which is only equivariant to neuron permutations. MAGEP-NDF introduces *stable polynomial terms* which are the product of a series of adjacent weights and biases of a neural networks that are also equivariant under different neuron permutations. These stable polynomial terms can be thought of as some "features" of the original weights, similar to how polynomials are used as features in kernels. MAGEP-NFN layers can process these stable polynomials in addition to the original weights. Experimental results demonstrate that MAGEP-NFNs outperform previous NFNs that use weight-sharing schemes in terms of generalization prediction and perform similarly on INR editing tasks.
The paper makes some unsubstantiated claims about its advantages over prior methods and GNN-based methods in terms of runtime and memory consumption. Specifically, while the paper emphasizes that MAGEP-NFN enjoys superior runtime and memory efficiency, there are no comparisons of these quantities for MAGEP-NFN and relevant baselines (For example, it is not clear to me that MAGEP-NFN should be more efficient than vanilla NFN). It would be good to add a runtime and memory efficiency comparison to the paper which in my opinion would make the paper stronger. 

Furthermore, while the theoretical contribution is nice and sound, MAGEP-NFN does not seem to bring substantial empirical improvement over relevant baselines.

**Minor**
- Use `` and '' for quotations.
Yes
No ethics review needed.
- Due to the lack of insights in the methodological section, several questions arise. For instance,
   - What is the degree of the resulting polynomial? Shouldn’t this be chosen as a hyperparameter by the user?
   - I assume that for different activation functions (or more precisely for different symmetries induced by the activation function) the resulting layers should be different. However, this is not clear to me from the Equations 20 and 21. Could the authors explain what are the differences?
   - What are the matrices $\Phi$ and $\Psi$? They are kind of abruptly introduced.


**Minor:**
- **Related work and background**:  I have the impression that these two sections (2 and 3) are quite similar to those of the Monomial-NFN paper. Especially, regarding the background section, I felt it was too long, which might not be necessary, since the reader can be pointed to Monomial-NFN or ScaleGMN to find a detailed explanation of the background.
- The paper misses an important citation: “Equivariant Polynomials for Graph Neural Networks”, Puny et al., ICML’23. This work discusses equivariant polynomials for permutation symmetries present in graphs.
- **Significance and Motivation**. Similarly to many equivariant machine learning tasks, designing models that are equivariant to parameter symmetries, typically requires dealing with a computational complexity vs expressivity dilemma. The authors correctly identify this dilemma, promptly (since NFNs/metanetworks are increasingly becoming more popular), and with their approach, they attempt to find a better trade-off.
- **Novelty**. This is the first work that proposes polynomials equivariant to parameter symmetries and may potentially mark a first step towards improving this paradigm as an alternative to linear layers + non-linearities, as well as GNNs.
- **Empirical performance**. In practice, the improvement in expressivity is shown to be reflected in the downstream performance in most of the tested cases.
This paper follows a recent line of work on *neural functional networks (NFNs)*, i.e. neural networks that are specifically designed to process other neural networks. In specific, the first NFNs are composed of linear layers (interleaved with non-linearities) that are equivariant to the *permutation* symmetries of the input NNs, while a recent follow-up called Monomial-NFN follows the same strategy, but additionally accounting for the *scaling* symmetries. However, the latter turned out to suffer from limited expressivity, due to the extensive parameter sharing that resulted from the existing symmetries. 

To overcome this limitation, the authors in the present manuscript propose a Polynomial-NFN, i.e. a combination of polynomial layers (potentially interleaved with non-linearities) that are equivariant to permutation + scaling symmetries. The paper mainly focuses on the characterisation of polynomial equivariant layers, using a construction named stable polynomials. Experimentally, the method is tested on a variety of common NFN benchmarks showing improved performance in the majority of the cases.
- **Presentation and Exposition**. My major concern with this paper is that the methodological/technical contributions are not well-presented. In particular,
   - The methodological part (Section 4) is quite hard to follow and the information provided is highly technical. This is not necessarily an issue, but currently, the authors do not provide any intuition/insights or simpler explanations of their thought process (why did they choose this proof technique, i.e. via stable polynomials), derivations (e.g. a proof sketch) and resulting equivariant/invariant layers (why do they look like that). 
   - Therefore, I am afraid that this problem might be reflected in **reproducibility/extensibility**, as it will be hard for the interested reader to adopt/re-implement this method, and even harder to deeply understand the structure of the resulting layers. 
   - I  would recommend the authors invest more in explanations and intuition about each part of their methodological section, most importantly in explaining the terms involved in the equivariant/invariant layer equations.


- **Claims and Theory**. Another major concern is that the authors have made a few claims that are not adequately supported with evidence:
   - First, they argue that their method improves expressivity compared to Monomial-NFN. Intuitively, probably this is indeed the case, but the authors have not provided any theoretical argument to support this claim.
   - Additionally, the authors mention that GNNs have increased memory consumption and runtime. As far as I can see, this claim is also not supported with, at least, experimental evidence. I suggest the authors report the above metrics and compare those of their method to GNNs (metanetworks), NFN, Monomial-NFN, and ScaleGMN (see below), to improve our understanding of the trade-offs. Additionally, a computational complexity table with comparisons would help a lot.
   - Finally, the theoretical contributions need some improvements. For example, it is unclear what is the role of Proposition 4.2. and Theorem 4.3. (why are they important?). Additionally, as far as I understand Theorem 4.4. characterises all G-invariant polynomials (by the way, of what degree?). Is there a similar argument for G-equivariant polynomials?


- **Method**. Additionally, examining the method per se, even if it improves expressivity, I believe it inherits several of the limitations of Monomial-NFN:
   - Only certain types of activation functions can be used.
   - It is hard to extend to other activation functions
   - It cannot operate on diverse architectures.
I advise the authors to mention those clearly and discuss potential options to address them.

- **Experiments**.
   - Apart from the memory/runtime comparisons that this paper is missing, I think that the authors should include another baseline: “Scale Equivariant Meta Networks”, Kalogeropoulos et al., NeurIPS’24. This is going to be a fairer competitor as opposed to Monomial-NFN and it would be interesting to observe the performance vs runtime tradeoffs.
   - I am wondering why in the editing experiment we don’t observe any performance improvement.
Yes
The manuscript is, in extensive parts of the first three sections, almost identical to the work of Tran et al. [1].
This is especially true in the first 5 pages (until the end of section 3, line 281).
Often times, whole paragraphs bear striking similarity, and only a few words differ between the two paragraphs.

More specifically:
- In the introduction: lines 36-59.
- In the related work section: lines 104-127
- In section 3: Equations 3-10, 14, 15. Lines 216-221. Proposition 3.3.

[1] Tran et al. Monomial Matrix Group Equivariant Neural Functional Networks. NeurIPS 2024.
Yes, Research integrity issues (e.g., plagiarism, dual submission)
[1] What does the outer sum  $\sum_{L\geq s>t\geq 0}$ in equation 20 denote? Is it a double sum over $s$ and $t$?

[2] Related to question 1, in equations 20-21, is it necessary to sum over the whole range of $s$ and $t$? This seems like a very inefficient operation for very deep networks (e.g. ResNet 101). Isn't it possible to sum over a few adjacent layer indices? An ablation study on the number of hops would be useful here.

[3] What is the computational complexity of the method compared to having linear layers?

[4] Can the method work in a combined Small CNN Zoo dataset with both ReLU and tanh activations (and potentially other similar groups)?

**Minor questions**:

[5] In Eq. 15, shouldn't it be $\pi_{i}(j)$ instead of $\pi_{i}^{-1}(j)$?
The manuscript is fairly clear and easy to follow, despite being math heavy.

NFNs are a very promising research direction, and making them faster and more efficient is key to scaling them up to more real-world applications.
Hence, the fact that the proposed method can reach good performance with a reduced number of parameters
compared to previous works is very promising, and a significant step forward.
This work proposes a novel Neural Functional Network (NFN), i.e. a neural network that takes the weights of neural networks as input data. The method, termed MAGEP-NFN, belongs to the family of parameter-sharing NFNs and incorporates both the permutation symmetries of neural networks and the scaling symmetries of common non-linearities, such as ReLU and tanh. The proposed method alleviates the reduced expressivity of previous works by introducing non-linear equivariant layers through polynomials of the input weights and biases. MAGEP-NFN achieves competitive results in INR tasks and CNN generalization tasks, while having far fewer parameters than the compared baselines.
The numbers reported for NFN in the INR classification experiments are considerably worse than
the official ones. Namely, NFN scores $46.6 \pm 0.072$ in CIFAR10, $92.9 \pm 0.218$ on
MNIST and $75.6 \pm 1.07$ on FashionMNIST. This also means that the performance of
MAGEP-NFN is quite poor. According to the appendix E, the authors use the same datasets as Zhou et al. Is there a different setup used in the experiments?

Almost no explanation is given for equations 20 and 21, that comprise the core functionality of the proposed method.
These equations are very math-heavy, and it is very hard for the reader to understand the usefulness of all individual components.
Further explanation of each module could provide more intuition about the method. Further, both formulas could benefit from some visualizations that show the connections between layers and the flow of information. Finally, it is unclear if all the individual components contribute to the effectiveness of the method. An ablation study on those components could make the method more efficient, without sacrificing accuracy.
We propose a novel monomial matrix group equivariant polynomial neural functional network (MAGEP-NFN) that enhances model expressivity while maintaining low memory consumption and running time.
@inproceedings{
anonymous2024equivariant,
title={Equivariant Polynomial Functional Networks},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=qto91DryES},
note={under review}
}
Neural Functional Networks (NFNs) have gained increasing interest due to their wide range of applications, including extracting information from implicit representations of data, editing network weights, and evaluating policies. A key design principle of NFNs is their adherence to the permutation and scaling symmetries inherent in the connectionist structure of the input neural networks.  Recent NFNs have been proposed with permutation and scaling equivariance based on either graph-based message-passing mechanisms or parameter-sharing mechanisms. However, graph-based equivariant NFNs suffer from high memory consumption and long running times. On the other hand, parameter-sharing-based NFNs built upon equivariant linear layers exhibit lower memory consumption and faster running time, yet their expressivity is limited due to the large size of the symmetric group of the input neural networks. The challenge of designing a permutation and scaling equivariant NFN that maintains low memory consumption and running time while preserving expressivity remains unresolved. In this paper, we propose a novel solution with the development of MAGEP-NFN (**M**onomial m**A**trix **G**roup **E**quivariant **P**olynomial **NFN**). Our approach follows the parameter-sharing mechanism but differs from previous works by constructing a nonlinear equivariant layer represented as a polynomial in the input weights. This polynomial formulation enables us to incorporate additional relationships between weights from different input hidden layers, enhancing the model's expressivity while keeping memory consumption and running time low, thereby addressing the aforementioned challenge. We provide empirical evidence demonstrating that MAGEP-NFN achieves competitive performance and efficiency compared to existing baselines.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
neural functional network equivariant model polynomial layer monomial matrix group
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/be4ca38e97772c7187bb43d833b4b2037623f60d.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/7ad0795cbc199b2900afef71eb4c5f2c9ac753da.zip
Equivariant Polynomial Functional Networks
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission