Selective Task Group Updates for Multi-Task Optimization
Multi-Task Learning;Multi-Task Optimization;Proximal Inter-Task Affinity
Yes
NA
No ethics review needed.
Please see the weaknesses.
1. Solid analysis from a theoretical perspective: The paper provides theoretical insights to explain the effectiveness of the proposed method, including (i) the benefits of sequential updating of groups, and (ii) the role of incorporating task-specific parameters in reducing conflicts.

2. The paper is well-written and well-organized.
This paper presents a multi-task learning method that adaptively groups tasks based on proximal inter-task affinity and then sequentially updates each group. It provides a theoretical explanation of the benefits of sequentially updating task groups and the role of incorporating task-specific parameters in reducing conflicts. Experimental results demonstrate the method's superior performance across various benchmarks.
1. Based on proximal inter-task affinity, what principle do we use for task grouping? Discussion on other principles should be included. For example, in [1], they use the Fisher Information Matrix, grouping the most heterogeneous tasks to mitigate conflicts.

2. The motivation for introducing proximal inter-task affinity: After reading Appendix A.1, I still find it difficult to understand the motivation for introducing proximal inter-task affinity.

3. Sequential learning on tasks [1], domains [3,4], and mini-batches [2] for alignment has been studied previously. It would be beneficial to compare the different theoretical perspectives between the proposed method and these prior studies.

[1] Task Groupings Regularization: Data-Free Meta-Learning with Heterogeneous Pre-trained Models, ICML 2024  
[2] On the Origin of Implicit Regularization in Stochastic Gradient Descent, ICLR 2021  
[3] Sequential Learning for Domain Generalization, ECCV 2022  
[4] Gradient Matching for Domain Generalization, ICLR 2022
Yes
No ethics review needed.
As I mentioned in the weaknesses section, I do not interpret the author’s theoretical analysis as indicating that the proposed multiple-objective method can be effectively solved on a global scale.
However, I am not suggesting the necessity of a stringent theoretical foundation.
I would like to see evidence that the author’s method can consistently provide an optimal point.
Demonstrating that the proposed method is robust across different batch sizes, numbers of groups, and optimization hyperparameters would effectively support its consistency and reliability.
1. Experimental Analysis
In the field of deep learning, the analysis of batch sequences has not been extensively explored. The author argues that grouping certain objectives in multi-objective problems can be significantly beneficial from a global perspective and has demonstrated this experimentally. In cases where the multi-task learning (MTL) results outperform those of single-task learning (STL), the author’s method consistently achieves the highest performance, which serves as strong empirical support for the validity of the proposed approach.

2. Logical Approach
The author’s approach to deriving proxy task affinity is reasonable.
By utilizing the loss after task parameter updates, the author effectively reduced additional computations
Also, the proposed method has been theoretically proven to remain useful.
Additionally, the explanation of the benefits of grouping derived from Theorem 2 is clearly written and easy to understand.
The author proposes a method for addressing multi-objective problems by grouping objectives.
In the process of considering relations between tasks (objectives), the concept of inter-task affinity was introduced, but additional computation was reduced by focusing on the update of task-specific parameters.
Additionally, introducing the concept of affinity to group objectives is the author’s original idea and has been thoroughly analyzed both theoretically and experimentally.
In my understanding, some questions remain regarding the actual utility of certain theoretical approaches.
The author addresses the utility of multiple objectives in a local context, but optimization in the field of deep learning is far more complex.
In practice, grouping the same classes together for optimization in classification tasks may be optimal for the currently updated classes locally; however, it is challenging to reach a global optimum.
I would like to see additional experimental evaluation on this matter.
I will leave detailed suggestions in the questions section.
Yes
No ethics review needed.
Please refer to the weaknesss section.
1. This paper investigates an important problem of multi-task learning. 
2. This paper is well-written and easy to follow.
3.  Realizing that traditional solutions focus on optimizing shared parameters but neglect task-specific ones, the authors delve into the concept of proximal inter-task affinity, making this paper well-motivated.
4.  The proposed method is new to me and gives a fresh perspective to further improve the performance of MTL.
5. This approach is said to improve multi-task performance by enhancing the learning of task-specific parameters and is shown to outperform previous methods, being scalable to different architectures and task numbers.
This paper discusses the challenges of multi-task learning, where training multiple tasks together in one architecture can lead to negative transfer or performance degradation. Traditional solutions focus on optimizing shared parameters but neglect task-specific ones. The proposed solution involves grouping tasks selectively and updating them in each batch, along with an algorithm that adapts to determine effective task grouping. The concept of proximal inter-task affinity is introduced to track task relations during optimization. This approach is said to improve multi-task performance by enhancing the learning of task-specific parameters and is shown to outperform previous methods, being scalable to different architectures and task numbers.
1. The task grouping result in Figure 3c seems out of converge. Will the number of groups further increase as the iteration becomes larger?
2. Why Nash-MTL is not reported in Table 2?
3. In the theoretical analysis (Section 4), the authors explain how this sequential update strategy can improve multi-task performance from an optimization standpoint. What about the generalization standpoint? I think the generalization of a model is more important.
4. In real-world applications, a typical MTL problem may have only a few tasks (e.g., 3). Will the proposed method work in such a circumstance? What is the task grouping result if there are only three tasks?
Yes
No ethics review needed.
1. Why is the number of groups in Figure 3c not an integer?

2. Line 523, "Table 4c" means "Figure 4c"?

3. Can you compare the performance of random grouping during optimization?
1. The paper proposes a novel optimization method to optimize the multi-task learning process from a new perspective.

2. The method demonstrates promising results on the NYUv2, PASCAL-Context, and Taskonomy datasets.

3. The paper provides a theoretical explanation of the advantages of sequential optimization of task groups and provides an analysis of convergence.
This paper proposes a novel MTL method in which tasks are dynamically grouped according to the conflict of different tasks during the optimization process and the model is optimized based on these grouped tasks. The method introduces only a limited computation cost and the experiment results show strong performance of their method compared to several existing MTL methods
1. There are some incorrect statements in the article. For example, in Line 092, “This perspective is not addressed in traditional multi-task optimization, which typically focuses solely on the learning of shared parameters.” is wrong, because the learning of task-specific parameters is considered in IMTL [1].

2. The conclusion in Line 373-374, “This suggests that grouping tasks with proximal inter-task affinity and subsequently updating these groups sequentially result in lower multi-task optimization. sequentially result in lower multi-task loss compared to jointly backpropagating all tasks.” does not seem relevant to the theorem above. Can you give a more detailed explanation?

3. For the conclusion in Line 525-526, “We observe that the affinity decay rate ... within a reasonable range.”, there is a lack of experimental results on the performance of models with different $\beta$.

[1] Liu, L., Li, Y., Kuang, Z., Xue, J.-H., Chen, Y., Yang, W., Liao, Q., & Zhang, W. Towards Impartial Multi-task Learning. ICLR, 2021.
@inproceedings{
anonymous2024selective,
title={Selective Task Group Updates for Multi-Task Optimization},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=EdNSQHaaMR},
note={under review}
}
Multi-task learning enables the acquisition of task-generic knowledge by training multiple tasks within a unified architecture. However, training all tasks together in a single architecture can lead to performance degradation, known as negative transfer, which is a main concern in multi-task learning. Previous works have addressed this issue by optimizing the multi-task network through gradient manipulation or weighted loss adjustments. However, their optimization strategy focuses on addressing task imbalance in shared parameters, neglecting the learning of task-specific parameters. As a result, they show limitations in mitigating negative transfer, since the learning of shared space and task-specific information influences each other during optimization. To address this, we propose a different approach to enhance multi-task performance by selectively grouping tasks and updating them for each batch during optimization. We introduce an algorithm that adaptively determines how to effectively group tasks and update them during the learning process. To track inter-task relations and optimize multi-task networks simultaneously, we propose proximal inter-task affinity, which can be measured during the optimization process. We provide a theoretical analysis on how dividing tasks into multiple groups and updating them sequentially significantly affects multi-task performance by enhancing the learning of task-specific parameters. Our methods substantially outperform previous multi-task optimization approaches and are scalable to different architectures and various numbers of tasks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multi-Task Learning Multi-Task Optimization Proximal Inter-Task Affinity
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/502322edef2ca5fa69f70593559aca975ea1b1bc.pdf
learning theory
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Selective Task Group Updates for Multi-Task Optimization
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission