Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass
language model; efficient adaptation; fine-tuning; prompting
Yes
No ethics review needed.
- From what I understand, generative adapters are Hypernetworks [[Ha et al](https://arxiv.org/abs/1609.09106)], but that hasn't been discussed anywhere. Do the authors agree with this assessment, and if so, how would they contextualize similar work using hypernetworks for continual learning [[Oswald et al](https://arxiv.org/abs/1906.00695), [Vladymyrov et al](https://arxiv.org/abs/2301.04584)]?
- Why is the continual pretraining baseline so much lower than the generative adapter baseline? This sticks out and is surprising to me, and makes me think it is poorly tuned.
- I think a valuable baseline (especially since continual pretraining seems to underperform) might be using the generative adapter loss function (Eqn 5) but optimize the weights of the model directly instead of the generative adapter. If I understood correctly, I don't see any reason why this shouldn't be possible, and if generative adapter is in the ballpark of this baseline, it's a positive signal. Do the authors agree with this assessment?
- The method is simple and well-motivated.
- The method works nicely in a streaming setting to continually update weights using lower number of FLOPs due to being able to reuse weights computed at the previous time step of the stream.
- The paper is clearly written and easy to follow.
This paper proposes a neural module called generative adapter, which once trained can accept a stream of input contexts to adapt transformers on the fly. The paper shows that generative adapters work well at lower computational cost than in-context learning on several NLP tasks.
- I think the Ultragist baseline should be reported in $\S 4.1$ and  $\S 4.2$, not just $\S 4.3$. Gisting is a natural comparison and I am glad to see it compared against in some form, but I think it could be reported more widely in the paper.
- Using llama-2-7b doesn't provide much value in my opinion. The 4k context length makes Figure 2b hard to reason about, and I don't see why all llama-2-7b experiments couldn't be done with llama-3.1-8b instead for similar cost.
Yes
No ethics review needed.
How does GenerativeAdapter perform in scenarios where the context information is dynamic or frequently changing, such as real-time conversational agents? It would be interesting to see if the method’s efficiency and adaptability hold in continuously evolving contexts.
1. The paper presents an efficient approach to adapting LLMs in real-time by using a lightweight, bilinear adapter generator. This mechanism effectively reduces computational costs associated with full model fine-tuning and inference-time overhead due to prompting, which is particularly advantageous for applications on edge devices.

2. The authors evaluate *GenerativeAdapter* across diverse, practical contexts such as document-based knowledge acquisition, in-context learning, and user personalization. These evaluations, especially in a personalized setting, demonstrate the adaptability of the method and its effectiveness in various contexts that align with real-world demands.
The paper introduces GenerativeAdapter, a method for adapting pretrained large language models (LLMs) using lightweight, parameter-efficient adapters generated by a self-supervised generator network. This approach aims to enhance LLM adaptability without incurring the computational cost of fine-tuning or the inference overhead of prompting. Evaluations are conducted across three main areas: factual knowledge acquisition, learning from demonstrations, and user personalization, showing that GenerativeAdapter is effective in reducing computational load and increasing model adaptability to various contexts.
1. The authors provide detailed descriptions of the model architecture, adapter generation mechanism, and datasets. However, hyperparameter settings and implementation details could be expanded, especially regarding stability mechanisms like SVD normalization, which could impact reproducibility. It is unclear if the code or models will be open-sourced for verification.
2. The analysis of *GenerativeAdapter*’s performance in different scenarios is comprehensive; however, the lack of prompt  engineering effort for baselines could introduce an unfair comparison. A clarification on the extent of prompt engineering would ensure a fairer assessment.
3. While *GenerativeAdapter* achieves superior performance over specific context lengths and adaptation scenarios, claims of general "efficiency" for "personalization" are strong, given the specific constraints of the study. The authors should emphasize that observed benefits apply to controlled experiments rather than broad generalizations.
4. The paper introduces concepts like "test-time contextualization" and "parameter-efficient" without specific definitions, which could hinder comprehension for readers less familiar with these terms. Explicit definitions or brief explanations in the methodology section would help maintain clarity.
Yes
No ethics review needed.
- Is there a suspected reason why the unnormalized adaptor contains extreme values?
- In the formulations in section 2.3, there is a  G(x_{1:m}). In this case, shouldn't the hidden states input to the generator, not the input tokens?
-  In the completion loss formulation, shouldn't the output just x_{m+1}, not x{m+1},...,x_n?
- The paper addresses the critical challenge of adapting LMs to incorporate new knowledge effectively, a fundamental issue since LMs can only utilize the information they were originally trained on. 
- It introduces an interesting approach of training another generator to generate an adaptor and optimize adaptation cost
The paper proposes GenerativeAdapter, which is a method to introduce unseen knowledge to LMs without finetuning cost or extra prompts. It generates a new adapter for the target unseen knowledge and utilize it to adapt the LMs.
- Some of the important experiments are either missing or have questionable setting
    - It is suspicious that the the settings for experiment 4.1 are fair for all of the baselines. It seems like the knowledge that each baseline has seen are different.
    - In section 5, the paper mentions the preliminary study shows the quality of the adapter generator is highly correlated with the final evaluation metrics. It would have been better if the results that support this claim is included in the paper. Many times, the training objective is not exactly aligned with the downstream task.
    - It would be better if the experiment to see how the adaptor affects the original ability of the LMs is included in the paper.
Yes
No ethics review needed.
I don't have any questions.
1. The topic is important. 
2. The idea of this paper is novel and clean, inspired by fast weights. 
3. The experimental results show the effectiveness of this method. it is impressive to see that model is able to inject 32k tokens into the parameters.
This paper proposes a novel method to integrate the information into the parameters. The experimental results are comprehensive and demonstrate the effectiveness of this method.
1. **Missing related works**: GenerativeAdaptor adds the adaptor which is quite similar to memory-based methods, where people would add a memory module on the large language models [1,2]. Perhaps the paper should discuss some related papers and the correlations between them. 

2. **Size of the additional parameters**: As stated in line 264, GenerativeAdaptor introduces 500 million parameters, however, even using 500 million parameters cannot achieve comparative performance as In-context Prompting (as shown in Figure 2(a)). Consider saving all the tokens in the context, which leads to 32768 * 4096 = 134 million parameters. This is fine, as the proposed method has smaller computation cost than storing all the previous tokens, but this might be something to consider, as it shows that the parameters may not be fully utilized, or the information in the parameters might be overly sparse. 

[1] Memllm: Finetuning llms to use an explicit read-write memory   
[2] MemoryLLM: Towards Self-Updatable Large Language Models
@inproceedings{
anonymous2024generative,
title={Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=bc3sUsS6ck},
note={under review}
}
Large language models (LLMs) acquire substantial knowledge during pretraining but often need adaptation to new contexts, tasks, or domains, typically achieved through fine-tuning or prompting. However, fine-tuning incurs significant training costs, while prompting increases inference overhead. Inspired by fast weight memory, we introduce GenerativeAdapter, an effective and efficient adaptation method that encode test-time context into language model parameters with a single forward pass.
GenerativeAdapter augments a frozen pretrained LM with a lightweight adapter generator, trained via self-supervised learning, to produce parameter-efficient adapters.
Notably, our generator is general-purpose, i.e., one generator can adapt the corresponding base model for all langauge processing scenarios.
We apply GenerativeAdapter to two pretrained LMs (Mistral-7B-Instruct and Llama2-7B-Chat) and evaluate the adapted models across  knowledge acquisition from documents, learning from demonstrations, and personalization for users.
In StreamingQA, our approach is effective in injecting knowledge into the LM's parameters, achieving a 63.5\% improvement in F1 score over the model with supervised fine-tuning (from $19.5$ to $31.5$) for contexts as long as 32K tokens.
In the MetaICL in-context learning evaluation, our method achieves an average accuracy of $44.9$ across 26 tasks, outperforming the base model. 
On MSC, our method proves to be highly competitive in memorizing user information from conversations with a 4x reduction in computation and memory costs compared to 
prompting with full conversation history.
Overall, GenerativeAdapter provides a viable solution for adapting large LMs to evolving information and providing tailored user experience, while reducing training and inference costs relative to traditional fine-tuning and prompting techniques.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
language model; efficient adaptation; fine-tuning; prompting
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/cb0bb315513d54b413c16c509ef1e99c4c320ab7.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission