Rethinking the Bias of Foundation Model under Long-tailed Distribution
foundation model;long-tailed learning
Yes
No ethics review needed.
1. How do you quantify parameter imbalance without access to the pre-training data? What metrics are used?

2. Can you provide theoretical justification for why semantic factors should be treated as confounders in the causal framework?

3. What is the computational overhead of the proposed method compared to standard fine-tuning?

4. How does the method perform when the degree of imbalance varies between pre-training and fine-tuning?
The paper presents a novel perspective on long-tail learning by considering the dual impact of pre-training and fine-tuning imbalance, which has been largely overlooked in previous work. The causal inference framework provides a principled way to analyze and address these challenges.
This paper addresses the long-tail learning problem in foundation model fine-tuning from a novel perspective by considering both parameter imbalance (from pre-training) and data imbalance (from downstream tasks). The authors propose a causal inference framework to analyze these two types of imbalance and introduce a backdoor adjustment method to learn the true causal effects. Experiments conducted on three long-tailed datasets (ImageNet-LT, Places365-LT, iNaturalist2018) with various foundation models demonstrate the effectiveness of the proposed approach.
1. The definition and quantification of "parameter imbalance" lack rigorous theoretical justification. The causal relationship between pre-training data imbalance and parameter imbalance is not thoroughly explained. The selection of semantic factors as confounders needs stronger theoretical support.

2. The claim that parameter imbalance has greater impact than data imbalance needs more empirical evidence.

3. Incomplete ablation studies for key components and lack of analysis on computational overhead and efficiency.
Yes
No ethics review needed.
1. In Fig. 3, how is the right picture plotted? How is the parameter imbalance computed for each sample?
2. In equation. 7, how is $P(Y=y|bc)$ implemented in your experiments. Do you run each model once and generate a $P(Y=y|bc)$? and how these models are jointly trained (or fine-tuned)?
3. In lines 184-195, it mentioned that "$P_P(Y)$ is inaccessible and we cannot use Eq. (1)". How is $P_P(Y)$  related to Eq. (1) where the symbol is not mentioned.
1. [Contribution] The proposed argument is important for the community, which is a generally existing problem while rarely mentioned. The propose of this problem is imperative.
2. [Soundness] The authors conduct a large number of experiments on SOTA models and popular long-tail datasets to demonstrate their arguments.
3. [Presentation] This paper is well organized. The step-by-step exploration of the problem provides an intriguing reading experience.
In the context of foundation model and PEFT, this paper defines two imbalance terms for long-tailed data, i.e., data-imbalance and parameter imbalance. The paper argues that only considering data imbalance is insufficient for PEFT on foundation models, and proposes an effective strategy to address the problem of parameter imbalance from a causal graph perspective. Numerical experiments have been conducted to support the arguments, and their proposed strategies.

The method's lack of efficiency is analyzed and discussed.
Although the introduced problem is important, the solution is kind of trivial. The simple combination of different models could partially resolve the problem, while each of the foundation models contains a large number of parameters, which is not applicable.
Yes
No ethics review needed.
1. In Table 8, it seems that as M increases, performance improves. Do the authors think this trend can persist for larger values of M? Is there any theory or insight that can support this observed trend?

2. In Table 7, why can the proposed method also improve performance in D-Many? I suppose such methods would sacrifice the performance on D-Many for improvements on long tail classes.
- They made a clear observation from empirical results that parameter imbalance is more dominant than data imbalance.

- They show that previous methods such as GLA and re-balancing cannot resolve the parameter imbalance issue.

- The paper is well-written and easy to understand.
This work considers the bias of foundation models when the data for fine-tuning suffers from class imbalance. They show that parameter imbalance is more dominant than data imbalance. Then, they propose a method based on an assumed causal graph and backdoor adjustment. Experiments show that the proposed method is effective.
- The argument that C is a confounder is confusing. The model will predict Y using the representation B, it is hard to believe there is a direct causal effect of C on Y. I would suggest the authors to show the direct causal effect or confounding bias with experiments.

- The proposed method utilized \textit{backdoor adjustment}, which is also used in some previous work in the literature of long tail classification such as [1]. The authors did not point out the difference.

[1] Tang, Kaihua, Jianqiang Huang, and Hanwang Zhang. "Long-tailed classification by keeping the good and removing the bad momentum causal effect." _Advances in neural information processing systems_ 33 (2020): 1513-1524.
Yes
No ethics review needed.
1. Could you explain a bit more about the concept of partial semantic factor and the causal graph in Section 5.1? What is a data-balanced representation, and could you provide an example of  $C \rightarrow X$?

2. In Section 4.1, the authors claim that although the fine-tuning-based method is able to address parameter imbalance, bias still exists; the results shown in figure 3 for Places365 looks unreasonably bad and could you provide more details on this part of experiments?
1. The paper identifies the importance of parameter imbalance and thoroughly analyzes the phenomenon with examples. It also highlights the limitations of logic adjustment-based methods in mitigating parameter imbalance.

2. The authors put significant effort into conducting thorough experiments.
The paper aims to solve the problem of long-tailed learning and illustrates the limitations of rebalancing methods due to biases in foundation models resulting from the imbalance in pre-training data. Based on a causal structure graph and back-door adjustment, the paper proposes a method to mitigate the spurious correlations introduced to the foundation model.
1. The writing can be greatly improved. For example, the paper mentions Figures 2 and 3 in the introduction without further explaining what they mean. More explanation on figure 2 and 3 in the introduction or detailed caption for figures 2 and 3 would be preferred. Authors also refer to “D-many,” “D-Medium,” and “D-Few” without explicitly explaining what they are. Given the frequency of these terms, I would expect more details on what they means e.g., the definition and data processing etc. 

2. Regarding the experimental results,  the improvement of the proposed method looks slightly marginal given that fine-tuning based method can also alleviate the bias. Maybe a side-by-side comparison of the improvement of the proposed method and fine-tuning based method would be beneficial. Moreoever, the results are mostly in tables, visual examples similar to those in Figure 4, that illustrate how the method reduces spurious correlations could be beneficial. To be more precise, what does the heatmap looks like after the application of the proposed method? Has the problem of partial semantic factor been alleviated from the visualization?
We propose a novel backdoor adjustment to solve imbalance problem.
@inproceedings{
anonymous2024rethinking,
title={Rethinking the Bias of Foundation Model under Long-tailed Distribution},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=l6K688mhDT},
note={under review}
}
Long-tailed learning has garnered increasing attention due to its practical significance. Among the various approaches, the fine-tuning paradigm has gained considerable interest with the advent of foundation models. However, most existing methods primarily focus on leveraging knowledge from these models, overlooking the inherent biases introduced by the imbalanced training data they rely on. In this paper, we examine how such imbalances affect long-tailed downstream tasks. Specifically, we refer to the biases in foundation models and downstream tasks as parameter imbalance and data imbalance, respectively. Through fine-tuning, we observe that parameter imbalance plays a more critical role, while data imbalance can be mitigated using existing re-balancing strategies. Moreover, we find that parameter imbalance cannot be effectively addressed by current re-balancing techniques, such as adjusting the logits, during training, unlike data imbalance. To tackle both imbalances simultaneously, we constitute a causal structure graph and view the partial semantic factor as the confounder, which brings spurious correlations between input samples and labels. To resolve the negative effects of this, we propose a novel backdoor adjustment method that learns the true causal effect between input samples and labels, rather than merely fitting the correlations in the data. Experimental results validate the effectiveness of our method.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
foundation model long-tailed learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/17433bbc989ec2db5f6942265cb9dc55be1fbb61.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Rethinking the Bias of Foundation Model under Long-tailed Distribution
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission