FairLoRA: Unpacking Bias Mitigation in Vision Models with Fairness-Regularized Low-Rank Adaptation
LoRA;ViTs;Fairness;Vision Models;Bias mitigation
Yes
No ethics review needed.
see weaknesses.
1. The investigation of fairness on Lora is interesting.
This paper proposes to achieve Lora fairness by reducing the variance of per group loss at a mini-batch level, which can reduce the performance disparities of the Lora across all classes. Experiments on Aircrafts with 100 classes and Waterbirds with 2 classes show that the proposed method is somewhat effective.
1. This paper is difficult to read. 
- Figures: Fig.1, and 2,5 are not well illustrated, the font is too small to read clearly.  Also, the low variance between losses does not necessarily lead to good performance/fairness.
- Writing: The story is not clearly or well organized. Fairness is not well defined in the classification tasks (which is also very limited) involved in this paper. The fairness can be defined between classes on a specific task, or between different metrics, or between different tasks, or even on different domains, which is not even mentioned in this paper. The distinctions/relationships between fairness, good performance, and good generalization ability are not discussed.
2. Poor experiments.
- Experiments on two datasets are not convincing enough, and the dataset Waterbirds only has two classes, which is not a good choice to demonstrate fairness.
- Fairness may be largely affected by the long-tail distribution of the training data, and there are no experiments. Also, there should be some comparisons between strategies for long-tail problems and the proposed method.
- Experiments only show min and avg metrics, without the best metrics. It is doubted that the best group may be degenerated by the proposed method.
3. Limited novelty.
- Minimizing the group loss variance is straightforward and seems effective, but there should be more explanations, especially on why the results do not degenerate.
Yes
No ethics review needed.
1. If possible, it would be beneficial to include further comparisons and analyses, particularly visualizations, discussions of concurrent papers, and detailed analysis of the EOD in the Geode datasets, as noted in the Weaknesses section.

2. As I am not an expert in ML fairness, I’m curious about how you divide the datasets into different groups. Are the groups derived from inherent divisions in the dataset, random divisions, or based on the original VLM’s performance on each sample during inference? Could different group division strategies affect the performance and conclusions? It would be helpful if you could clarify the exact process for group division in each dataset and discuss how alternative division strategies might influence your results and conclusions about fairness.

3. In figure 5, why a LoRA Rank of 32 experiences a sudden drop in Accuracy and Min F1, compared with 16 and 64? Similarly, in Figure 9 there is a sudden drop of performance when LoRA rank is set to 16, compared with neighbored rank values. You may further analyze that if this pattern is consistent across different models or datasets, and what implications this might have for choosing optimal LoRA ranks in fairness-aware fine-tuning.
1. This paper is the first to introduce a fairness-based fine-tuning technique using LoRA, as stated.

2. It presents multiple fairness metrics (Sec. 5) and conducts extensive experiments, supplementing previous fairness-centered evaluation metrics and contributing to future research.

3. The comprehensive experiments are visually presented with colorful figures, enhancing clarity and accessibility, especially for researchers who may not be familiar with ML fairness concepts.

4. Using these metrics, the paper also examines potential factors influencing FairLoRA’s fairness, such as LoRA ranks, distribution shifts from pretraining data, and task types, further contributing to the research community.
This paper introduces a novel fairness-based finetuning technique based on LoRA with a fairness-specific regularizer, studies multiple fairness metrics, and analyzes factors for bias mitigation. Experiments conducted on various vision models including ViT, DiNO, and CLIP support the discovery and analysis.
1. The analysis on handling distribution shifts seems to be insufficient, as it includes only FID-based quantitative comparisons. Would it be possible to conduct additional experiments, such as t-SNE, to better demonstrate the effectiveness of distribution shift handling? For instance, showing the t-SNE point distributions before and after fine-tuning or comparing t-SNE point distributions of fine-tuned models with vanilla LoRA and FineLoRA could add valuable insights.

2. Several recent papers [1][2] also focus on LoRA fairness, discussing similar concepts, such as fairness metrics like EOD and LoRA ranks as fairness factors. Both papers were submitted in May 2024, more than three months before the ICLR submission deadline, yet they are not mentioned in the discussion. It might be helpful to clarify the distinctions in motivation, methodology, experiments, and findings between Submission 11853 and these related works. For instance, [1] notes that "lower ranks may retain the bias of the original model." Does this finding complement or challenge your conclusions? Since [1][2] evaluate LoRA fairness in LLMs rather than vision-language models like CLIP, could extending your experiments to LLMs provide additional insights?

3. In the Geode dataset, FairLoRA’s "Eod Max" value is significantly lower than that of the other methods (Fig. 4, 7). Are there additional comparisons or analyses to further investigate this? Providing a detailed analysis of why FairLoRA performs exceptionally well on this metric for this dataset, as well as exploring whether this trend holds across different model architectures or LoRA ranks, could strengthen the findings.

[1]	Das S, Romanelli M, Tran C, et al. Low-rank finetuning for LLMs: A fairness perspective[J]. arXiv preprint arXiv:2405.18572, 2024.

[2]	Ding Z, Liu K Z, Peetathawatchai P, et al. On Fairness of Low-Rank Adaptation of Large Models[J]. arXiv preprint arXiv:2405.17512, 2024.
Yes
No ethics review needed.
- Explain the "Group" strategies.

- Using text to explain the formulations.

- How about adopt your approach on segmentation task.
- The topic of fairness in LoRA (Low-Rank Adaptation) is important and worthy of study.
- The proposed method of reducing group-wise variance is simple yet effective.
- The motivation behind the proposed method is clear.
- The method is evaluated on several visual foundation models, and the results are convincing.
This paper studies the fairness problem of the popular LoRA (Low-Rank Adaptation) technique. The authors propose a simple yet effective strategy for improving the fairness of LoRA, specifically by reducing the variance of per-group loss. The results are promising.
- How are the groups divided in page 4, line 212?
- The formulation in page 4, line 208 lacks a number.
- How are the groups divided in Section 5, "MEASURING FAIRNESS"? What is the difference between these groups and the groups mentioned in page 4, line 212?
- In my opinion, fairness means consistent results among different categories, such as cats and dogs. What is the relationship and difference between the category group and the groups you defined?
- The formulation of "Equalized Opportunity Difference (EOD)" is abstract. Please add a textual explanation.
-  This paper only studies the recognition problem. How does the proposed method generalize to more fine-grained tasks, such as segmentation? Category imbalance is a common issue in segmentation tasks.
We introduce FairLoRA, a fairness aware LoRA, that aims to mitigate bias in fine-tuning with LoRA and measure the impact across metrics.
@inproceedings{
anonymous2024fairlora,
title={FairLo{RA}: Unpacking Bias Mitigation in Vision Models with Fairness-Regularized Low-Rank Adaptation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=pB3KeBCnQs},
note={under review}
}
Recent advances in parameter-efficient fine-tuning methods, such as Low Rank Adaptation (LoRA), have gained significant attention for their ability to efficiently adapt large foundational models to various downstream tasks. These methods are appreciated for achieving performance comparable to full fine-tuning on aggregate-level metrics, while significantly reducing computational costs. To systematically address fairness in LLMs previous studies fine-tune on fairness specific data using a larger LoRA rank than typically used. In this paper, we introduce FairLoRA, a novel fairness-specific regularizer for LoRA aimed at reducing performance disparities across data subgroups by minimizing per-class variance in loss. To the best of our knowledge, we are the first to introduce a fairness based finetuning through LoRA. Our results demonstrate that the need for higher ranks to mitigate bias is not universal; it depends on factors such as the pre-trained model, dataset, and task. More importantly, we systematically evaluate FairLoRA across various vision models, including ViT, DiNO, and CLIP, in scenarios involving distribution shifts. We further emphasize the necessity of using multiple fairness metrics to obtain a holistic assessment of fairness, rather than relying solely on the metric optimized during training.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LoRA ViTs Fairness Vision Models Bias mitigation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/fcc202b82878ecaf1215ec1acb3dd6ccea0ff2b8.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/ce8871765e455b0539bab3cc26804b74d31d67ac.pdf
FairLoRA: Unpacking Bias Mitigation in Vision Models with Fairness-Regularized Low-Rank Adaptation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission