MCNC: Manifold-Constrained Reparameterization for Neural Compression
Model Compression;LoRA;PEFT;Transformers;ViT
Yes
No ethics review needed.
Can the author clarify how does the benefit of the proposed approach over baselines translate to practical computational performance gains?
+ The proposed algorithm outperforms baselines in the high compression rate regions
+ The idea is interesting and shows that there exists a set of parameters in a DNN that lies in the lower-dimension manifold.  
+ Evaluations are thorough
The paper proposed a new approach to compress deep neural networks. The basic idea is to enforce the DNN parameters to lie on a k-dimension manifold for every chunk of d parameters where k << d. The proposed approach uses a generator model that maps a low-dimensional uniform distribution (k-dimension) to a high-dimensional uniform distribution (d-dimension). Evaluations on computer vision and NLP tasks demonstrate that the proposed approaches can achieve extremely high compression rates.
The paper presents an interesting idea of compressing neural network parameters into a low dimension manifold. Instead of training a generator to produce the neural network parameters, the proposed approach can use a randomly-initialized generator to project a trainable k-dimension vector into a d-dimension vector as parameters. I find the idea very interesting. The empirical results also demonstrate that the approach can achieve better accuracy under extreme compression rates. 

With that said, here are a few concerns on the work
- The benefits of achieving extreme compression rates under the high accuracy loss are not clear to me. Using table 1's result as an example, with 1% percentage of model size, the proposed approach can achieve an accuracy of 49% compared to the baselines which are 33%-39% on ViT-Ti model. However, compared to the baseline model accuracy 76.2%, it is a huge accuracy drop (more than 20%). Under this context, one might consider using a different model variant that is smaller but achieves a higher accuracy than compressing the ViT-Ti model.  Although the proposed method can achieve better accuracy compared to baselines under high compression rates, I am concerned about how likely such advantages would be useful in practice due to the significant accuracy drop compared to the non-compressed or lightly-compressed model. 

- Table 4 highlights the same concern. The trainable parameters from the LoRA model are already minimal compared to the number of parameters in the LLaMA-2 7B and 13B. With the quantization from NOLA and MCNC, both of them slow down the training throughput of the model. I agree with the advantages of MCNC compared to NOLA, but neither approaches are appealing compared to running the original LoRA method.
Yes
No ethics review needed.
Please address weaknesses above.
The paper presents very simple yet powerful idea of essentially hashing high-dimensional weights into lower-dimensional manifold. Instead of hashing directly from higher to lower dimension, the proposed approach fixes the inverts of the hashing function (i.e., generator), and learns the "hashed" outputs via sgd. This idea has deep roots in random projections/random hashing literature, and as such I am very glad that similar approach can be exploited for the parameter efficient training of the neural networks.
The paper presents a new approach to parametrize the network parameters: via another generator model. Specifically, weights are split into blocks(vectors) of size $d$ (i.e., $\mathbb{R}^d$) and parametrized to be coming from a $k$-dimensional space ($\mathbb{R}^k$) via mapping $\phi: \mathbb{R}^k \to \mathbb{R}^d $. The mapping $\phi$ is defined as a compact feed-forward neural network with sine activation with number of layers and hidden dimensions being hyper-parameters of choice; though for most experiments there are not more 3 layers with hidden dimensions of not more 1024. The mapping $\phi$ is randomly initialized, and fixed. The only trainable parameters are the actual inputs to the $\phi$, i.e., the inverted image of parameters in this lower-dimensional space. The authors demonstrate that such a simple setup allows to train from scratch and finetune various models in parameter efficient way.
Overall, I think the paper does a good job in validating the soundness and effectiveness of the proposed approach, however, I think it can be significantly strengthened:
1. In the exposition of the idea, the actual architecture of the generator network comes very late; with actual numbers/settings coming in appendix! I think, the simplicity of the approach must be showcased from the beginning, I highly suggest to add simple pytorch code showing how easy is to plug and play with the method
2. Similar to the previous weakness, I think authors should consider recommending some bulletproof settings that would work for most of the cases (like number of layer in $\phi$, number of dimensions, some rules of thumb, how long to train, optimizer settings) to easy any potential adoption: at the end of the day, the ideas have very limited impact if not used in practice. 
3. Theoretical guarantees of any sorts would be welcome as well: how universal is such a generator network? can it be used to train/finetune any neural network or it has limitations?
4. While the papers literature review is extensive, I think the connection to hashing must be explored in depth and more papers included into review (some suggestions: Structured Multi-Hashing for Model Compression,  Lossy weight encoding for deep neural network compression)
Yes
No ethics review needed.
1. How was the wasserstein distance computed for Fig 2? If I understand correctly, there is no closed form expression to compute this metric in higher dimensions. 
2. Why does ViT-S have better accuracy for smaller models in Tab 1? Is it an artifact of high variance in the results?
3. How were reconstruction GFLOPs computed for Table 4?
4. How much slower is training MCNC as compared to baseline LoRA?
1. Strong empirical results, with comparisons against other compression methods on various modalities and architectures demonstrating the proposed method’s effectiveness.
2. Extensive ablation studies for each component of the method, as well as for choices of hyper-parameters, giving some intuition on how 
3. Demonstration of real world speedups
The paper proposes a reparameterization to train neural networks which leads to a lower memory footprint for transferring and storing the parameters. The paper achieves this by re-parameterizing the network parameters to lie on a lower dimensional spherical manifold. The method uses a “generator” network for this, mapping k-dimensional vectors onto a d-dimensional sphere. The k-dimensional vectors are then trained. The paper compares this method against multiple model compression and efficient training methods, demonstrating performance gains.
1. My main gripe with the paper is that the method is not very well motivated. For example, the paper claims that a randomly initialized generator is enough to span the entire space of weights in a higher dimension, but never shows much empirical evidence beyond an experiment in low dimensions. It is not clear if this result also holds in higher dimensions. I would have ideally liked to see a theoretical result around the expressivity of the manifold constrained reparametrization, but even an empirical investigation showing that one does not lose out on expressivity would be great.  
2. Continuing on my previous point, from table 10, it appears that a trained generator is better for larger models. However, this result is down-played in the text description, with the narrative claiming that the improvement is "marginal".
A method that compresses large models by limiting parameters to low-dimensional manifolds. It achieves superior compression of CNNs, ViTs, and LLMs across tasks compared to current techniques.
@inproceedings{
anonymous2024mcnc,
title={{MCNC}: Manifold-Constrained Reparameterization for Neural Compression},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=VMV8gefvq8},
note={under review}
}
The outstanding performance of large foundational models across diverse tasks, from computer vision to speech and natural language processing, has significantly increased their demand. However, storing and transmitting these models poses significant challenges due to their massive size (e.g., 750GB for Llama 3.1 405B). Recent literature has focused on compressing the original weights or reducing the number of parameters required for fine-tuning these models. These compression methods generally constrain the parameter space, for example, through low-rank reparametrization (e.g., LoRA), pruning, or quantization (e.g., QLoRA) during or after the model training. In this paper, we present a novel model compression method, which we term Manifold-Constrained Neural Compression (MCNC). This method constrains the parameter space to low-dimensional pre-defined and frozen nonlinear manifolds, which effectively cover this space. Given the prevalence of good solutions in over-parameterized deep neural networks, we show that by constraining the parameter space to our proposed manifold, we can identify high-quality solutions while achieving unprecedented compression rates across a wide variety of tasks and architectures. Through extensive experiments in computer vision and natural language processing tasks, we demonstrate that our method significantly outperforms state-of-the-art baselines in terms of compression, accuracy, and/or model reconstruction time.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Model Compression LoRA PEFT Transformers ViT
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/cf431c13a6c0aa29d1dd65d25f2e19db96426ca7.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/cdbedd7b86d080d8458fe8a779b5b3f41692e79a.zip
MCNC: Manifold-Constrained Reparameterization for Neural Compression
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission