Investigating the Effectiveness of HyperTuning via Gisting
hypernetworks;llm;parameter-efficient fine-tuning;prefix tuning
Yes
No ethics review needed.
In second paragraph of 3.2, the authors says they use QLoRA for hyper network training, but in the last paragraph of 3.3, the authors states they fine-tuned all model parameters and no LoRA is used. I am a bit confused.
1. The paper is easy to read. 
2. The paper has a very detailed discussion on each experiment.
This paper introduce a set of Gisting based hyper network called HyperLlama for generating soft prefix tokens for downstream tasks. The prefix tokens acts similar to few shot example in in-context learning. The experiments show that their HyperLlama is effective in generating soft prefix tokens, but they underperformed compared to multi-task fine-tuned models with attention to in-context examples.
It seems that the paper has no technical contribution to the community. The introduced HyperLlama follows the setting of HyperTuning. 
The paper introduced a set of new models (HyperLlama) for sure, but I do not think it is a valid technical contribution. Further, the introduced models generally underperform few-shot language models by large margins.
Yes
No ethics review needed.
(1) What are the advantages of this paper's approach compared to existing Gist methods? Additionally, I would like to understand the benefits of separating the model components.
(2) As highlighted in the title, it would be helpful if the paper demonstrated the effectiveness of the method with numerical evidence compared to existing methods.
The paper is written in a way that makes the field and methodology easy to understand.
- This study introduces HyperLlama, a Gisting-based hypernetwork designed to generate soft prefixes for a frozen downstream Llama-2 model.
- Through experiments on P3, S-NI, and Symbol Tuning datasets, they demonstrate that HyperLlama can compress few-shot example information into soft prefixes,
- HyperLlama-generated soft prefixes also serve as strong initializations for further prefix tuning, supporting efficient fine-tuning.
The Discussion section describes the strengths and weaknesses of HyperLlama well, but these aspects are not fully addressed in the paper. For example, while it is mentioned that HyperLlama saves resources during inference, the authors should provide numerical evidence to support this.

I believe that the Gist (Mu et al., 2023) and the HyperTuning (phang et al. 2023) have both made substantial contributions to this field, and I have some concerns that this paper primarily builds on these approaches by combining them in its experiments.
Yes
No ethics review needed.
Just like the weaknesses above.
1. The introduction of HyperLlama leverages Gisting to generate soft prefixes, compressing task-specific information efficiently.

2. Comprehensive experiments across multiple datasets demonstrate HyperLlama's strengths, especially in initializing soft prefixes for prefix tuning, showing improved performance over random initializations.
The authors introduce HyperLlama, a Gisting-based hypernetwork designed to generate soft prefixes for a frozen downstream Llama-2 model. Their experiments across diverse datasets demonstrate that HyperLlama effectively compresses information from few-shot examples into soft prefixes, outperforming baselines that lack access to additional examples. Additionally, they show that HyperLlama-generated soft prefixes provide superior initializations for further prefix tuning.
1. The paper lacks clarity in several areas, including the introduction of motivation, the description of the methodology, and the presentation of experimental results and figures.

2. The experiments are limited to the Llama-2-7B model, and there are no evaluations on newer models (such as Llama-3) or larger-scale models (like 13B), which limits the generalizability of the findings.

3. HyperLlama struggles with tasks that rely on precise output formats or highly contextual few-shot examples, affecting its generalizability.

4. The method’s effectiveness is contingent on Gisting’s ability to compress information accurately, which may vary across different types of tasks.
Yes
No ethics review needed.
Please answer the Weaknesses.
1, The figures are good and especially helpful for understanding the main idea of this paper.

2, This paper has detailed training details.
This paper introduced a Gisting-based hypernetworks. This hypernetwork is built on Llama-2 and is able to generate task-specific soft prefixes based on few-shot inputs. To do so, the authors use two LLMs. One acts as the hypernetwork and the other one acts as the downstream network. To enable training, the hypernetwork is trained with QLoRA (plus additional embeddings) to produce Gist tokens to append to the downstream network.
1, Table 1 has not been cited in the paper. What is its role in this paper?

2, At training, the hypernetwork and the downstream network are quantized to 4-bit. Are they full-precision when used in the test stage? If it is, how to solve the gap between the quantized model and the full-precision model?

3, To generate Gist tokens, two forwards with LLM are needed, which may incur high costs.

4, Even though I am not an expert in the area, I still recommend the author reorganize this paper for clear expression.
Yes
No ethics review needed.
Please refer to the Weakness section
1. This paper proposes a novel method to generate gist tokens using hypernetwork, which can compress few shot examples of different tasks into gist tokens.
2. The author's motivation for designing this hypernetwork is very clear, and sufficient experiments are provided to prove the good performance of gist token on P3 and S-IN datasets, and its insufficient performance on Symbolic-tuning task. 
3. The author has demonstrated through sufficient experiments that using the soft token generated by the hypernetwork as the initial token of prefix finetuning can improve the efficiency of prefix finetuning.
This work proposes a novel HyperLlama, which uses a hypernetwork to generate gist tokens to capture information from few-shot examples. The author provides sufficient experimental evidence that HyperLlama performs well on P3 and S-IN datasets, but performs not that good on Symbolic-tuning task. However, this work needs to strengthen its analysis of HyperLlama's performance deficiencies.
1. As noted in Section 3.3, "Compression Hyperpretraining" is essential for the downstream model (specifically the Llama 2 model in this study) to support gist tokens. However, it remains uncertain whether this additional pretraining step ("Compression Hyperpretraining"), following the standard pretraining, might adversely affect the model's performance on other tasks.
2. As discussed in Section 5, HyperLlama does not exhibit superior performance on the Symbolic-turing task. The paper lacks an in-depth analysis of the factors contributing to this shortcoming. Specifically, it is not clear whether the hypernetwork is unable to extract all relevant information from the few-shot examples, or if the gist tokens are insufficient in storing the information from the few-shot examples.
3. The study lacks an ablation analysis regarding the number of few-shot examples and the number of gist tokens. In other words, it is not explored whether increasing the number of few-shot examples, while keeping the number of gist tokens constant, would lead to an improvement in performance.
We use Gisting to train Llama-2 to efficient generate parameters for model adaptation
@inproceedings{
anonymous2024investigating,
title={Investigating the Effectiveness of HyperTuning via Gisting},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=EjCrfVFZTx},
note={under review}
}
Gisting (Mu et al., 2023) is a simple method for training models to compress information into fewer token representations using a modified attention mask, and can serve as an economical approach to training Transformer-based hypernetworks. We introduce HyperLlama, a set of Gisting-based hypernetworks built on Llama-2 models that generates task-specific soft prefixes based on few-shot inputs. In experiments across P3, Super-NaturalInstructions and Symbol Tuning datasets, we show that HyperLlama models can effectively compress information from few-shot examples into soft prefixes. However, they still underperform multi-task fine-tuned language models with full attention over few-shot in-context examples. We also show that HyperLlama-generated soft prefixes can serve as better initializations for further prefix tuning. Overall, Gisting-based hypernetworks are economical and easy to implement, but have mixed empirical performance.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
hypernetworks llm parameter-efficient fine-tuning prefix tuning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/6182f02a588206d1efc89879c4a8a12e64cd0ce7.pdf
transfer learning, meta learning, and lifelong learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Investigating the Effectiveness of HyperTuning via Gisting
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission