Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning
Catastrophic forgetting; Large language model; Instruction tuning
Yes
No ethics review needed.
See above
* The forgetting issue during fine-tuning foundation models is an important problem.
* The observations and analysis through the function vector in this paper is very interesting, which I believe would be interesting to many.
*  The finding that CF in LLMs primarily stems from biases in function activation rather than the overwriting of task processing functions is not obvious, but I think the authors has done a good job in confiming this. 
* The writing is easy to follow.
The paper investigates Catastrophic Forgetting in Large Language Models during continual learning, revealing that forgetting is influenced by specific tasks and model characteristics, and proposes a novel training methodology that utilizes a function vector to stabilize performance and reduce CF, with empirical validation across multiple benchmarks.
Overall, I like the findings in the paper and the authers have done extensive investigations. I am interested in comparing the proposed method with model averaging, which is shown in [1] that model averaging is very effectivenss in terms of mitigating forgetting. Furthermore, is it possible to analyze the effectivenss of model averaging through function vector? 


[1] Yong Lin, et.al., Mitigating the alignment tax of RLHF.
Yes
No ethics review needed.
1. It is unclear in methodology of training with function vector guided loss. Since attention head will change at every iteration during training, when do you impose the KL divergence-based regularization. Since, it requires separate computation using ICL-based examples, how do you update model weights through backdrop?  Kindly provide more details on the FV loss computation. 
2. The FV-based method may require additional computational resources, especially during interventions where multiple layers are evaluated. Since to extract FV, one has to run the model on a set of counterfactual inputs to isolate activation patterns and then select top attention with the highest causal effect. The FV-guided KL-divergence loss aligns logits produced with and without FV interventions. This requires computing model outputs twice. There needs to be a discussion on the computational time and memory overhead required for the proposed approach. 
3. While training on a task, regularization is imposed just based on the most recently trained task or do you consider all previously trained tasks? The loss function in equation 4 and 5 may be modified to consider the state of training on a particular task in the sequence. 
4. There are some typos in the paper Line 107: bracket missing in citation, Line 970: it should be We instead of I, Line 231  : sequensce --> sequence.
1. The idea of using regularization on the function vectors and the function vector guided loss to enable continual instruction fine tuning seems novel and is well motivated in the paper. 
2. The paper presents a strong ablation study for the proposed methods across various settings, demonstrating their generalizability by conducting experiments using multiple llama checkpoints, classification and generation tasks and evaluating across generalizability and supervised finetuning on multiple datasets.
3. The experiment studies which clearly demonstrate the ability of  function vectors to significantly improve the performance on continual instruction fine tuning.
The paper explores catastrophic forgetting in continual instruction finetuning across various settings, discovering that model forgetting is influenced by the specific training tasks and the models themselves. It explores correlation between function vectors and catastrophic forgetting, and  propose a function vector-based approach to eliminate catastrophic forgetting. The paper has conducted ablation studies across various tasks, methods and models to demonstrate the generalizability of the method using different llama and mixtral models with classification and generation tasks and evaluating it for supervised finetuning and model generalizability.
1. There is a lack of clarity in the proposed methodology. i.e function vector guided training design. It would help to understand  the proposed methodology by providing the overall  the optimisation objective and end-to-end training algorithm. In the current state, it is difficult to reproduce the results that the authors mentioned in the paper without the details on the proposed methodology. 
2. The  presentation  of the paper and the readability of some sections are difficult to follow.   Notations are unnecessarily complicated. 
3. The experimental results though convincing across most of the experimental setups, its hard to make decisive conclusions especially on the core idea of function vectors and their effectiveness in mitigating catastrophic forgetting.
4. Some experiments and  observations made related to catastrophic forgetting are redundant and may not contribute much. For example, forgetting will be model-dependent since performance on different datasets and tasks is model-dependent.

More details on the above mentioned points are listed below.

1. The authors claim that “Forgetting coincides with changes in FV similarity between tasks”. However, it is difficult to observe the same from the evidence provided because the trend is not constant across all tasks and models; sometimes, a slight change in similarity changes the task performance drastically(i.e. Fig 2(a) on count object and hellaswag datasets), in some cases even though similarity is decreasing, the performance is increasing. It is very difficult to establish a clear relation from the empirical results provided. A suggestion is to  extend the number of tasks or conduct a statistical test using correlation metrics. 
2. The observation that generation tasks lead to greater forgetting cannot be measured directly, even though the same evaluation metric (Rouge) is used for both types of tasks. Rouge is more sensitive to longer sequences because it relies on n-grams overlap. The tasks have different difficulties; one way to measure them is by comparing them against some upper bound baseline and calculating forgetting in terms of percentage loss.
3. The computation of the FV guided loss (eqn 4) is not very clear. If the set S consists attention head with top-10 CE,  is that going to be same before training on the current task and after training. Is this loss recomputed at every iteration of the optimization step? 
4. Although function vectors improve the performance of existing CL methods, it can be helpful to compare against  finetuning models individually, to see how much gap there is between methods with function vector guided fine tuning and the optimal performance that can be achieved by fine tuning.
Yes
No ethics review needed.
1. Related to weakness 2: can you more concretely establish the FV similarity is correlated with forgetting, perhaps with a correlation plot? I want to better understand how much signal FV provides on forgetting as a whole.
2. Related to weakness 2: can you show how FV measures forgetting compared to other metrics one might consider. For example, after training on a task, does FV similarity to the previous task better predict forgetting of the previous task than L2 distance to the previous model?
3. Related to weakness 3: Why is FV a measure of latent task identification? What exactly is the argument for this?
4. Related to weakness 3: an alternate hypothesis for the results in section 5 is that after finetuning, only certain layers of the model have changed significantly while a lot of the other layers have remained unchanged. So when you make the activation intervention and search over the layers to find the optimal one, you have just undone the effect of finetuning in the activations by returning it to the behavior of an earlier model upto that later and the rest of the model isn't that different after finetuning so activations pass through and your recover the behavior of the earlier model. How would you refute this alternate hypothesis?
5. Related to weakness 4: did I understand this correctly? If so, can you make this clear/provide some evidence that if you train with the activations for task 1 regularized, it does not mitigate forgetting on task 2 (another unrelated task). If this is true, this is an important limitation that should be made clear (I want to emphasize that it isn't a limitation of the paper but rather would make the paper stronger be explaining how this method should be used).
6. A claim for the training method is that it does not effect plasticity. I would like to understand the limitations of this: can we finetune on a very large dataset of a new task with this regularization and still maintain plasticity? When does the regularization start hurting? What are the limits. One way you could do this is train on a FT dataset with millions of samples (maybe OpenMathInstruct-2) to see if this regularization limits learnability in extreme cases.
**Originality**: This was an interesting approach/hypothesis for studying forgetting that I quite enjoyed.
**Quality**: The authors perform a thorough series of experiments investigating their claims and evidence ranges from empirical measurements, mechanistic interventions, and training interventions. 
**Clarity**: The work is well contextualized in related work.
**Significance**: This work is a really interesting way to connect mechanistic interpretability to studying and mitigating forgetting demonstrating practical ways in which tools from mechanistic interpretability can be used to understand diverse phenomena.
This paper empirically studies catastrophic forgetting (CF) from the perspective of function vectors (FV) and makes the following core contributions:
1. It shows that there is a correlation between the change in function vector for a task and degradation in performance on the task.
2. It argues that CF is due to the inability to associate the input with the latent task vector rather than a degradation in the ability to solve a tasks conditioned on the input and correct latent task vector. It supports this with mechanistic interventions to the task vector. 
3. It presents a method to regularize FVs during training to mitigate forgetting.
1. **Clarity**: The paper dense and the writing is terse and sometimes hard to follow. The figures have a lot of information on them, are very small and it is quite difficult to extract takeaways from them. Overall the paper is often hard to follow and it is hard to trace the evidence for the claims. A lot of points through the paper seem like observations and the connection to one of the core scientific claims isn't always clear. I think this paper would benefit greatly from better organization. Probably most important is to explain the figures/tables better *in the captions* and actually point out what the takeaways are/what we should learn from them/what points it is substantiating. I understand this is challenging because of page constraints and the amount of information presented in the paper: my recommendation would be to move anything non-essential to the core scientific claims to the appendix. Also, the authors could present the results for 1 model and add the rest of the models to the appendix given that the models are all of the same scale.
2. **FV and forgetting**: One of the key claims in this paper is change in FV is correlated with forgetting, i.e. this is a useful measure that "explains" the phenomena of forgetting. The evidence did not convince me of this: in Figure 2, there are points where FV similarity is high, 5 shot performance is high but 0-shot performance is low. Object Count especially is unconvincing (0shot and 5shot switch and don't seem to be correlated with FV).
3. **Mechanistic interventions**: While the mechanistic interventions in Section 5 are quite interesting, the paper does not consider any alternated explanations for this behavior (see questions). It is also not clear to me why FV is a measure of latent task identification.
4. **Limitation of training method**: If I understand correctly, this training method regularizes the FV of particular tasks and thus prevents forgetting of the particular tasks being regularized. It doesn't affect forgetting as a whole? While this is not a weakness, I think this should be clarified and the method should be explained as such.
Yes
No ethics review needed.
Here are some minor/typo suggestions:
1. Line 107, the references to Peng et al. and Chung et al. should be \citep instead of \cite.
2. Line 143, missing a blank between "using" and "SuperNI".
3. Line 162, the reference to Zhao et al. should be \citep instead of \cite.
4. Line 200, missing a blank between "." and "Unless".
5. Line 200, the reference to Hu et al. should be \citep instead of \cite.
6. Line 201, "learning rate of 1e-4" should be "learning rate of $1e^{-4}$".
7. Line 143/157/197, appending a "." after each boldface phrase will be more consistent with your writing style in Line 240/249/257 and so on.
8. Line 357,  "... trigger its corresponding latent variable θT (PM(θT|x))." could be rewritten as "... trigger its corresponding Function Vector θT (i.e., PM(θT|x))." for more clarification.
9. It would be great to provide a one-sentence summary at the end of each paragraph in Section 7 about the connection between the related works and this study.
10. The Conclusion section seems to be more consistent with the full manuscript if it is described in the present tense.
1. The manuscript provides a thorough study of the catastrophic forgetting problem. 
2. The most insightful contribution of this work is posing a new hypothesis on this problem. 
3. Its technical contribution is significant as the authors first introduce the Function Vector as a tool in this community.
4. The paper is well-written and well-organized.
This paper studies the catastrophic forgetting (CF) problem in LLMs with the tool of Function Vector (FV). The research starts with reproducing the CF problem on a series of four tasks across 4 publicly available LLMs. In addition, the researchers study the differences in FVs between continue-trained and untrained LLMs on each task. They show that there is a strong correlation between the differences in FVs and the CF issue. Their theoretical analysis posts a hypothesis on the bias of the FV θT elicited by the input x, (i.e., p(θT|x)). The authors further conducted several experiments to well support their hypothesis. Finally, they designed a new loss for alleviating the CF problem in continuing learning, where they encourage LLMs to minimize the differences between model prediction and the FV-intervented prediction while keeping the hidden representations of X the same as un-trained ones.
I cannot see any significant weakness. One nitpick would be expanding the background of Function Vector in Sec. 2.2, especially providing the dimension or shape of each notation. It will help the audiences understand the procedure for computing FVs more easily.
@inproceedings{
anonymous2024unlocking,
title={Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=gc8QAQfXv6},
note={under review}
}
Catastrophic forgetting (CF) poses a significant challenge in machine learning, where a model forgets previously learned information upon learning new tasks. 
Despite the advanced capabilities of Large Language Models (LLMs), they continue to face challenges with CF during continual learning. The majority of existing research focuses on analyzing forgetting patterns through a singular training sequence, thereby overlooking the intricate effects that diverse tasks have on model behavior.
Our study explores CF across various settings, discovering that model forgetting is influenced by both the specific training tasks and the models themselves. To this end, we interpret forgetting by examining the function vector (FV), a compact representation of functions in LLMs, offering a model-dependent indicator for the occurrence of CF. Through theoretical and empirical analyses, we demonstrated that CF in LLMs primarily stems from biases in function activation rather than the overwriting of task processing functions.
Leveraging these insights, we propose a novel function vector guided training methodology, incorporating a regularization technique to stabilize the FV and mitigate forgetting. Empirical tests on four benchmarks confirm the effectiveness of our proposed training method, substantiating our theoretical framework concerning CF and model function dynamics. We plan to make our code publicly accessible in the near future.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Catastrophic forgetting; Large language model; Instruction tuning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/b55c3952bac52c3824b890c3a62820fdac113ea3.pdf
transfer learning, meta learning, and lifelong learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Unlocking the Power of Function Vectors for Characterizing and Mitigating Catastrophic Forgetting in Continual Instruction Tuning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission