MotherNet: Fast Training and Inference via Hyper-Network Transformers
hypernetwork;tabular data;meta-learning;foundational models
This is a very fair and insightful review, demonstrating the reviewer’s profound understanding and expertise in this field.
Yes
No ethics review needed.
1. **Scalability Constraints**: While MotherNet's performance on small datasets is impressive, it shares the quadratic memory limitations of standard transformers, restricting its use to smaller datasets. Have you considered modifying MotherNet to incorporate larger context or memory-efficient transformer architectures, such as those designed for long-context handling or reduced attention complexity? How might these adaptations impact its scalability and performance?

2. **Limited Domain Exploration**: While the focus on tabular data showcases MotherNet’s strengths, its hypernetwork and in-context learning capabilities seem promising for other types of data, such as text or images. Do you see potential for adapting MotherNet to these modalities? If so, what adjustments would be necessary to leverage its in-context learning for different types of tasks?
**Strengths**:
- **Creative Approach**: MotherNet is an inventive application of hypernetworks and transformer-based architectures, demonstrating how in-context learning can effectively generate task-specific models without gradient descent.
- **Efficiency and Speed**: The method achieves significant inference speed improvements over TabPFN, offering practical advantages for use cases requiring fast, on-demand predictions.
- **Eliminates Hyper-Parameter Tuning**: MotherNet operates without per-dataset hyper-parameter tuning or gradient-based training, simplifying the pipeline for real-world applications.
- **Strong Empirical Evidence**: The paper provides a thorough comparison with a range of baselines, including TabPFN, HyperFast, and traditional models like XGBoost and Random Forests, illustrating MotherNet’s consistent performance across benchmarks.
- **Open-Source Contribution**: The work includes an open-source implementation, supporting reproducibility and facilitating future research in the field.
- **Thoughtful Methodology**: The architecture decisions, such as low-rank decomposition of weights, are well-documented and provide insights into balancing memory efficiency and model capability.
- **Solid Trade-off Analysis**: The authors give a clear breakdown of the trade-offs in terms of training and inference time, presenting scenarios where MotherNet excels over other approaches.
The authors propose a novel transformer based hypernetwork model that can be 'in-context' prompted with a supervised dataset and it can generate weights for a small neural network that can generalize well on this new task. Their approach borrows ideas from meta-learning, hypernetworks, transformers and distillation, and demonstrates a modern and effective way to achieve large speed ups when a task-specific task model is trained, while also retaining the generalization capability of a full training run. 

Overall I find the creativity, elegance and rigor demonstrated in this paper very refreshing and commendable. 

I have a bunch of questions, and no paper is without weakness, but I find the work a reasonable and worthwhile contribution to the field.
- **Scalability Constraints**: MotherNet, like TabPFN, is bound by the quadratic memory requirements of transformers, limiting its usability for datasets larger than around 5,000 samples. This could be viewed as a major limitation for more extensive applications.
- **Presentation Gaps**: Some sections, particularly in the methodology and results, could benefit from improved clarity and structure to enhance readability and understanding.
- **Comparison Depth**: Although the evaluation includes multiple models, comparisons with newer, more diverse architectures would strengthen the paper’s impact and situate MotherNet more firmly in the broader ML landscape.


And my favourite 'weakness'
- **Limited Domain Exploration**: The paper focuses solely on tabular data, leaving questions about whether MotherNet’s advantages could extend to other data modalities like text or images. This work could be a neat contender for in-context learning paradigms across the board. Hypernetworks are an intriguing idea.
Yes
No ethics review needed.
- Since the model could handle 30,000 data points on GPU and 100,000 data points on CPU, why didn't you evaluate on larger datasets (or in the case of Tabzilla, with a larger training set)?

- In general, how do you see this paper contributing to future research or practical applications? I think it's interesting that this sort of model is possible, but I'm having trouble seeing what the broader contributions are, especially because I'm not convinced the experimental results show a practical niche for the model as proposed.

- What are the main contributions of this paper compared to Hyperfast specifically? The performance of the two models is only compared on very small datasets. But beyond that, I'd like to know how you'd compare this work to Hyperfast because a comparison isn't given in Section 2.

Minor issues:  
- On page 5, how did you get the number 25,738? I calculate the number of low-rank weights as 33,088 ($2hr+Nr$).
- In Section 3.1, $r$ is used to represent a rank whereas in Section 3.2, it's used for the number of input features, which is a bit confusing.
- On page 6, the text says that MotherNet outperforms MLP-Distill in terms of normalized AUC, but the results table only shows it outperforming in terms of raw AUC.
- There are a few references to "Table 4.1" and "Table 4.2" that are incorrect.
- The fact that this approach works well is surprising and compelling. It appears that training on synthetic datasets has a useful regularizing effect in generating MLPs compared to standard MLPs that are just trained on the dataset in question.

- The presentation is generally very clear.
This paper shows that in-context tabular models can be trained to generate MLP weights directly. The resulting MLPs generally outperform standard trained MLPs and are competitive with other tabular predictive models. The paper focuses on small datasets due to limitations of the in-context modelling being used.
- The experimental results aren't very compelling on the whole. The CC-18 evaluation is limited to very small datasets and the actual differences in AUC between the top ten models are very small. Given the scale of the data and relative effectiveness of very simple models like logistic regression, it's tough to see the computational efficiency of the proposed model as actually mattering in practice, especially since it's using GPU hardware. The Tabzilla evaluation provides a wider range of dataset sizes, but in this regime, the proposed method performs about as well as an untuned XGBoost, and significantly worse than tuned GBDT models (in addition to TabPFN).

- I think the use of ensembling also weakens the experimental results. While TabPFN did this in their paper, it's generally the case that ensembling any stochastically trained model can improve performance, and I think that using ensembling on some methods and not others isn't a fair comparison.

- I'm not confident about the contributions of this paper beyond an interesting result - I'd like to hear more from the authors on that (see below).
Yes
No ethics review needed.
-
I like the overall idea of the project. This paper shows an interesting new path with certain benefits over the original TabPFN, including the high inference efficiency.

*(Assuming that the authors will share the code and the model checkpoints linked in the paper)*
To me, a big positive thing is the provided code for training MotherNet and the model weights. Training hypernetworks is non-trivial and costly, and I like that this project gives the community a ready-to-use hypernetwork baseline.

Continuing the previous point, the proposed model seem to outperform HyperFast, one of the previous (and the only?) tabular hypernetwork. So MotherNet advances the niche of tabular hypernetworks, and gives a better starting point for future work in this direction.

I also appreciate that the paper explicitly communicates the scope of dataset sizes (<5000 objects), as well as the analysis of some failure cases in Appendix.
This study proposes a classification hypernetwork for tabular data, called MotherNet. A hypernetwork is a "parent" network that, given training data, produces the weights for another "child" network that is immediately ready to make predictions on the test data. This allows completely avoiding the traditional gradient-based training and hyperparameter tuning.

To learn how to produce good child networks, MotherNet itself is trained on synthetic data for a long time (28 days on one GPU A100). The synthetic data generation is inherited from the TabPFN project. The parent network is a 12-layer Transformer, same as TabPFN. The child network is a lightweight 2-layer MLP. Since the actual predictions are made by the child network, it means that the inference efficiency is high.

On datasets of size <5000 objects, a competitive performance of MotherNet is reported.
**Benchmarks**

I have mixed feelings about the benchmarks. I tend to agree that MotherNet outperforms HyperFast, however, the overall ranking of models is less convincing to me. I can imagine that the relative performance of the methods will change significantly in a different setup, and especially on different datasets.

Regarding datasets, if I understand correctly, there are three parts of results:

- (Figure 2 and Table 1) The performance on datasets with unusual ranking of models, in particular with the high performance of a linear model, as directly noted in the paper on L357. It is unclear how reliable are the conclusions made on these datasets.
- (Figure 6) The performance on the "validation" datasets. My understanding is that these results are somewhat additional, and thus presented in Appendix, not in the main text. I should admit I do not fully understand the role of the validation datasets, so I may be missing something.
- (Table 2, Figure 3, Table 6) The results on the TabZilla benchmark. There are also certain unusual results, e.g. the linear model outperforming MLP, or SVM outperforming MLP-rtdl.

Regarding hyperparameters:
- I noticed that the number of training epochs for MLP and ResNet is tuned within the grid (10, 100, 1000). I can imagine that on some datasets all three values are suboptimal, e.g. 10 can lead to underfitting, and 100 and 1000 can lead to overfitting.
- If the "choice" distritbution for HyperOpt is a categorical distribution that is order-unaware, then this can be misleading for the hyperparameter tuning engine (hard to tell to what extent).

**Related work**

Section 2 (related work) does not cover a whole family of methods that, I think, is highly relevant. However, this seems to be easy to fix, plus, some of the methods are already used as baselines. Namely, I imply the traditional machine learning models that should be trained from scratch for each task. That includes classic ML algorithms (linear models, tree-based methods, etc.) and DL architectures (there are too many of them to list; I suggest taking any popular baseline, e.g. FT-Transformer from the same paper as the already used ResNet, and traverse the citation graph backwards and forwards).

**Analysis**

In my opinion, the analysis of the MotherNet's performance should be extended. I suggest the following experiments:

(1) Finetuning the child network produced by MotherNet. Even if it is not how MotherNet is supposed to be used in practice, it would show the full potential of the MotherNet system. If this experiment is not presented in this paper, then it can become a somewhat must have experiment for future researchers willing to use MotherNet as a baseline. I recommend conducting this experiment directly in this submission.

(2) Extensive hyperparameter tuning (learning rate, weight decay, dropout, etc.) for a plain MLP of the *exactly* same architecture as the child network. Otherwise, it is unclear how optimal are the weights produced by MotherNet. For example, if the traditional tune-and-train approach gives significantly better results for the child architecture, then it can be the preferable approach when the task performance is more important than the (almost) zero cost of the MotherNet's forward pass (plus, training on small datasets is cheap).
Yes
No ethics review needed.
I strongly encourage the authors to restructure Section 4 and rearrange the results. Also, a stronger argument is necessary to show why MotherNet is better than tree-based methods and TabPFN.
1. The pipeline is efficient during training/fitting and inference. 
2. Empirical evaluations include an extensive set of baselines, including distillation methods, and two frequently-used benchmarks for tabular data.
MotherNet performs in-context learning on tabular data via generating a feed-forward network from a pre-trained transformer to perform prediction. The work is a synthesis of ideas from TabPFN and meta-learning/hypernetworks. MotherNet’s efficiency stems from requiring only a forward pass on the pre-trained model and the simplicity of the feed-forward MLP. Experiments are conducted on the OpenML CC-18 and TabZilla benchmarks.
1. MotherNet is not a competitive method based on the average rank numbers. It is outperformed on CC-18 by MLP-Distill and TabPFN. Such is the case for TabZilla, but the error intervals are too wide for any conclusions to be drawn. This experiment is not effective. Pre-training time is not rigorously compared as part of the method’s cost.
2. The paper itself is of subpar quality, especially in the presentation of results. There are too many references to appendix figures and tables, which should be discouraged and used sparingly. Paradoxically, the paper does not even take up the full page limit of 10 pages. There are errors in references (e.g. line 405 - Table 4.2). Understanding the results is very difficult. The critical difference diagrams are not explained. 
3. Novelty is limited as the method is heavily influenced by TabPFN. It could be seen as an extension.
MotherNet is a foundational hypernetwork that can produce trained neural networks for small tabular datasets using in-context learning without finetuning.
@inproceedings{
anonymous2024mothernet,
title={MotherNet: Fast Training and Inference via Hyper-Network Transformers},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=6H4jRWKFc3},
note={under review}
}
Foundation models are transforming machine learning across many modalities, with in-context learning replacing classical model training. Recent work on tabular data hints at a similar opportunity to build foundation models for classification for numerical data. However, existing meta-learning approaches can not compete with tree-based methods in terms of inference time. In this paper, we propose MotherNet, a hypernetwork architecture trained on synthetic classification tasks that, once prompted with a never-seen-before training set generates the weights of a trained ``child'' neural-network by in-context learning using a single forward pass. In contrast to most existing hypernetworks that are usually trained for relatively constrained multi-task settings, MotherNet can create models for multiclass classification on arbitrary tabular datasets without any dataset specific gradient descent.
The child network generated by MotherNet outperforms neural networks trained using gradient descent on small datasets, and is competitive with predictions by TabPFN and standard ML methods like Gradient Boosting. Unlike a direct application of TabPFN, MotherNet generated networks are highly efficient at inference time.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
hypernetwork tabular data meta-learning foundational models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/35eba59460dba770c8248a7f5c2e0eef94cd8af0.pdf
transfer learning, meta learning, and lifelong learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
MotherNet: Fast Training and Inference via Hyper-Network Transformers
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission