Locate-then-Unlearn: An Effective Method of Multi-Task Continuous Learning for Large Language Models
Machine unlearning;Continue learning;Model editing
Yes
No ethics review needed.
* What is the definition of “tasks” in this work? The distribution of data, even under the same tasks, seems to affect the selection of task-specific neurons. Thus, whether improvement will transfer to different datasets from the same tasks is unexamined.

* How are the toy datasets constructed in section 3.2? Are they a subsample of the underlying task data?

* Regarding the design of false answers in section 4.3., the false knowledge can be presented in multiple forms than what’s already in the dataset, which could thus not capture the potential wrong knowledge and influence the effect of unlearning. 

*Typos*

* Title: Continuous -> Continual

* L409-410: implement -> implementing; this sentence is difficult to parse.
* This work presents solid experimental results and ablation studies, showing the effectiveness of each component.

* The main novelty lies in finding task-specific neurons for more efficient re-learning.
This work proposes to unlearn task-specific neurons to enable more efficient multi-task learning. The proposed approach, locate-then-unlearn, first locates task-relevant neurons, deduplicates the neurons that are relevant for multiple tasks, subtracts the corresponding parameters to unlearn false answers, and relearns the objective tasks.

The proposed approach shows higher performance than five baseline methods on five datasets in the multi-task learning scenario, and improvements on three out of five datasets in the single-task setting (with the remaining two getting close performance with direct full fun-tuning).
* The causal effect between the neurons and different tasks requires more careful examination. Although the approach shows empirical improvement,  the assumption that neurons are task-specific might require further evidence.

* This approach is dependent on the selected data used to locate neurons (section 3.2.) and unlearn tasks (section 3.3). The authors provide several ablation studies on the method components, while it may be worthwhile to conduct ablations on the choice of data.
Yes
No ethics concerns are found.
No ethics review needed.
- How does the method compare with other PEFT methods, such as prompt-tuning, lora-hub?
- It would be better if the authors can discuss for each new task, how many parameters need to be stored (the percentage of the whole transformer model)?
- The method is intuitively reasonable. By detecting task-related and task-specific neurons, the methods make multi-task learning more parameter-efficient.
- The experiment results show that the proposed method can outperform knowledge editing methods such as ROME and MEMIT on five datasets.
- The authors conduct extensive analysis. The unlearning hyperparameter has a best performing peak, while the threshold is rather robust. The task-specific neurons are important, and tuning them on unrelated tasks lead to worse results. Visualization shows that task-specific neurons are well-separated.
- The writing of the paper is quite clear.
This paper proposes locate-then-unlearn, to identify and unlearn task-specific neurons in transformer models for multi-task learning. Specifically, the authors propose to identify task-related neurons from top-k important scores for each task, and identify task-specific neurons by removing overlapping neurons shared by different tasks. Finally, the authors fine-tune the selected task-specific parameters with unlearning data, and remove the parameter offset (from fine-tuning) from the original parameters. Experiment results demonstrate that the proposed locate-then-unlearn outperform other knowledge editing baselines on five datasets. They also conduct extensive analysis to demonstrate that the setting of unlearning hyperparameter is important, task-specific neurons work the best on corresponding related tasks.
- Since the authors claim that their methods contribute to parameter-efficient multi-task learning, it would be better to compare with methods such as prompt-tuning, lora-hub.
- It would be better if the authors could release their code upon acceptance.
Yes
No ethics review needed.
See weaknesses.
1. The proposed approach is novel.  
2. The proposed approach is well-motivated.  
3. The results show good performance.
This paper proposes Locate-then-Unlearn framework to improve LLM continual learning by identifying and selectively unlearning task-specific neurons. By localizing neurons specific to individual tasks and unlearning them, the approach reduces interference between tasks and enhances model efficiency. Experiment results and analyses across multiple datasets demonstrate good performance in both single-task and multi-task settings.
1. One problem of LLM continual learning is catastrophic forgetting, it would be better if the authors can further discuss how the proposed approach handle the issue.  
2. For LLM continual learning, it would be better to compare model performance on the benchmark datasets before and after training, so the readers will learn the effect of the learning process on previously learnt task. Currently, the authors just report the performance after training. It would be a plus if the authors can include the model performance before training.  
3. The paper claims many times that the proposed approach "significantly" outperforms existing approaches, but it seems the significance test is missing. So, it is hard to justify whether the improvement is significant.
@inproceedings{
anonymous2024locatethenunlearn,
title={Locate-then-Unlearn: An Effective Method of Multi-Task Continuous Learning for Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=cJ9qoVZbPd},
note={under review}
}
Nowadays large language models (LLMs) have achieved remarkable success in various NLP tasks. However, they often misinterpret human instructions and generate incorrect or outdated responses, highlighting the need for more effective continual learning techniques. While recent efforts have introduced unlearning methods to remove erroneous knowledge, existing approaches still struggle in multi-task learning scenarios.

To overcome these limitations, we propose \textbf{locate-then-unlearn}, a novel framework that identifies and selectively unlearns task-specific neurons to enable efficient multi-task learning. We hypothesize that LLM neurons can be broadly categorized into task-specific neurons for handling individual tasks, and general neurons to maintain the model’s foundational capabilities. To accurately identify task-specific neurons, we utilize a two-stage locating process: (1) ranking task-related neurons based on their importance to each task, and (2) identifying task-specific neurons by applying intervention to assess how neuron activity impacts task performance, isolating those most critical to each task. We conduct comprehensive evaluations in two experimental setups: single-task specialization and multi-task generalization. The results show that our method significantly improves performance across both settings. This indicates that our method effectively balances model efficiency and accuracy in multi-task continual learning.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Machine unlearning Continue learning Model editing
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c9f4867ee0911c1ee02528dfdb032d5d3f9ebd9f.pdf
interpretability and explainable AI
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Locate-then-Unlearn: An Effective Method of Multi-Task Continuous Learning for Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission