AutoScale: Combining Multi-Task Optimization with Linear Scalarization
Multi-task Learning;Autonomous Driving
Yes
No ethics review needed.
In Table 1, the iteration time is reported. Could the authors also provide the total computational time? Additionally, what is the computational time for each method shown in Table 2?
Multi-task optimization and learning is essential in machine learning. The approach of using multi-task learning metrics to determine the weights in the linear scalarization seems novel. The proposed method is computationally more feasible than an exhaustive grid search of weights.
This paper explores the connection between multi-task optimization (MTO) and linear scalarization, proposing the use of multi-task learning metrics such as gradient magnitude similarity and condition number to guide the determination of weights in linear scalarization. The proposed approach, named AutoScale, includes detailed discussions on the initial multi-task learning phase and various strategies for optimizing the weights, which gives insights into its implementation.
1. While using MTL metrics to estimate the optimal weight appears reasonable based on the results shown in Figure 2, it is unclear whether these findings would hold consistently across other datasets. Additionally, the evaluation metric—a single average of the accuracy of multiple tasks—may not fully capture the nuanced definition of "good" and "bad" results, especially in a multi-task learning context. More careful discussions are needed to justify the use of MLT metrics.
2. The paper could benefit from more comparisons with linear scalarization approaches that directly use predictive performance metrics as criteria for determining weights.
3. Beyond the unitary and grid-searched weights discussed, the authors might consider exploring other weight determination methods, such as those surveyed by Royer et al. (2024), to provide a broader context and benchmark for AutoScale.
Yes
No ethics review needed.
What are G1, G2, G3, …, B1 in Figure 2?
 
I feel the whole structure needs to be rewritten. For example, Delta m is in Figure 3 and authors wrote on Line 257 that Figure 3 shows clear correlations. However, Delta m is not defined until Section 5. Key definitions need to be given in the main text.
 
In Section 4.1, authors propose three cost functions. Are these cost functions related to the literature. Are there any reasonings on choosing these? Why are these functions helpful to the problem? Authors should explain the intuitions.
 
In Section 4.2, what is the fifth method? Exponential fit? What are the conclusions from Figure 4?

Is Time s/iter the running time? For example, the last column of Table 1. This should be defined clearly.

Authors show the running time, different values of alpha (Figure 6) etc. If these are to demonstrate the generalization or robustness, authors should show them in all datasets.
This work is the first to explore relationship between MTO and linear secularization.

Experiments show the benefit of AutoScale in terms of evaluation metrics and running time.
This work studies the relationship between MTO and linear scalarization, which is over debate in the literature. Authors use some metrics to demonstrate the positive relationship. Based on this observation, a new two-phase pipeline for MTL is proposed and demonstrated in multiple datasets.
Notations are not given before their formal definitions, which creates barriers of reading.

Some explanations are not given, for example, why are those cost functions defined in that way?

Results (figure/table) are not clearly explained.
Yes
No ethics review needed.
1. Can authors discuss why linear scalaization correlates with some MTO metrics while is independent of others. What are commonality of these correlated MTO metrics?

2. What are benefits of Autoscale compared with traditional MTO algorithms?
The paper is well-organized and the idea is clear.
The presentation is overall clear, and the experiments seem extensive and convincing.
The paper proposes AutoScale, a new approach that combines MTO with linear scalarization for efficient and effective MTL. The authors explore the synergy between MTOs and linear scalarization, proposing that certain MTO metrics can guide weight selection to improve scalarization efficiency. AutoScale operates in two stages: an exploration phase to gather gradient and loss data using an MTO method, and a scalarization phase that leverages optimized weights derived from key MTO metrics. Extensive experiments demonstrate that AutoScale outperforms prior MTO methods and nearly matches the performance of grid-searched scalarization weights, without the associated search costs.
1. The technical contributions seem limited since the proposed methodology is pretty straightforward.
2. In the experiments, the number of tasks are too small (i.e. up to 4 tasks), which are not common in multi-task learning settings.
3. The code is unavailable, and it is impossible to reproduce the results.
4. Some typos. For example, Line 306 Exploration -> exploration
Line 311: Linear Scalarization -> linear scalarization
@inproceedings{
anonymous2024autoscale,
title={AutoScale: Combining Multi-Task Optimization with Linear Scalarization},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=VvxuD3cdJx},
note={under review}
}
Multi-task learning is favored due to its efficiency and potential transfer learning achieved by sharing networks across tasks. While a series of multi-task optimization algorithms (MTOs) have been proposed to solve MTL optimization challenges and enhance performance, recent research claims that simple linear scalarization, which sums per-task loss with a carefully searched weight set, is sufficient, casting doubt on the added value of more complex MTO algorithms. In this paper, we provide a novel perspective that linear scalarization and MTOs are closely related and can be combined to yield high performance and efficiency. We show, for the first time, that a well-performing linear scalarization exhibits specific characteristics of certain optimization metrics proposed by MTOs, such as high task gradient magnitude similarity and low condition number, via an extensive empirical study. We then propose AutoScale, an efficient pipeline that leverages these influential metrics to guide the search for optimal linear scalarization weights. AutoScale shows superior performance than prior MTOs and performs close to the searched weight performance consistently across different datasets.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Multi-task Learning Autonomous Driving
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/7f3dd41e6e7501c406574587f10ff83ca500d4e8.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
AutoScale: Combining Multi-Task Optimization with Linear Scalarization
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission