Selective LoRA for Domain-Aligned Dataset Generation in Urban-Scene Segmentation
Dataset Generation;Urban-scene Segmentation
Yes
No ethics review needed.
Please see Weaknesses. If the authors address my concerns, I will raise my rate.
Additionally, It would be better if:
1. Provide quantitative metrics on the quality of the generated segmentation maps, comparing them to ground truth or to maps generated by other methods. Discuss any observed differences in segmentation map quality between their method and baseline approaches, particularly in relation to the selective learning process.
2. If possible, include a qualitative analysis (e.g., visual examples) of any artifacts or inconsistencies in the generated segmentation maps that might be attributed to the selective learning process.
1. The paper is well-organized, the motivation description is clear and concise. The pipeline figures are easy to understand.
2. The core idea is interesting, use language distinction to learn concept difference, and eliminate the requirement of paired visual data to learn specific concepts. I think the learning procedure is practicable. 
3. The experimental settings are extensive, including both in-domain, few-shot and damain generalization.
The paper introduces a new approach to generate training samples with sepecific concepts variants. The proposed method learn specific concepts, such as style or viewpoint, by selectively manipulate the gradients. The method claim that it improve domain alignment and sample diversity.  In experiments, the method are compared with baseline and the DatasetDM, and results show improvements in in-domain, few-shot segmentation. The wide-scope experiments prove its practicability.
1. The method mainly focus on introduce their Selective LoRA. However, the whole pipline includes training a label generator (stage 3 in Figure 2). The technical details in this part are not well delivered. How the label generator receive the intermediate features from T2I models and generate semantic maps?  In addition, in line 190-197, the authors say their use Mask2Former as label generator as same as DatasetDM. As I know, the DatasetDM use only "perceptual decoder" which only includes a decoder architecture instead of whole Mask2Former segmentaiton. Clarifying this distinction could provide a clearer understanding of the contributions of the current approach.
2. While the method aims for simultaneous sample and segmentation map generation, it requires a two-stage training process for the T2I model and label generator separately, contrasting with DatasetDM’s one-stage training. This additional stage could indeed limit practicality for real-time or large-scale augmentation, and a comparison in training efficiency or practical adaptability would be beneficial.
3. The dataset used for evaluation is Cityscapes and BDD100K, which includes only city streets. Since singlar scene makes learning specific concepts changes easiler, it would be improved if the authors prove their method on more general dataset, e.g. coco, ade20k. Since the main comparison method DatasetDM use more general dataset, I wonder performances of Selective LoRA on other datasets.
4. If the selective learning process affects the reliability of generated segmentation maps? The authors seem not provide relavant discussion.
5. Minor erros: the box of viewpoints and styles in stage 4) of Figure 2 are reversed.

Reference:
[1] DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models, NIPS 2023
Yes
No ethics review needed.
Q1: From the design motivation, it seems that training all LoRA parameters may lead to overfitting, resulting in reduced diversity in the generated images. In contrast, Selective LoRA selects a subset of parameters that are most associated with the concepts, effectively training fewer parameters and better preserving the original T2I model's capabilities. The original LoRA setting applies training to all linear layers in the UNet with LoRA. I wonder if training LoRA only on certain layers' cross-attention (few parameters) could achieve a similar effect as Selective LoRA.

Q2: I hope the authors can address the concerns raised in the "Weaknesses" section.
S1: The paper "Selective LoRA" introduces a new training strategy for fine-tuning pretrained T2I models to generate diverse datasets for segmentation tasks, addressing the challenge of data scarcity.
S2: Extensive experiments show that the generated datasets enhance the performance of prior segmentation models in urban-scene segmentation.
The paper presents a method for fine-tuning pre-trained T2I models to generate datasets specifically for urban-scene segmentation, addressing the challenge of data scarcity. Traditional methods often utilize pre-trained T2I models directly or apply LoRA for fine-tuning, which can lead to generated samples that fail to align with the target domain or lack diversity. To overcome these issues, the paper introduces Selective LoRA, a novel fine-tuning approach that selectively identifies and updates the weights that are most closely associated with specific concepts for domain alignment. This approach reduces the number of parameters that need training, improving training efficiency while ensuring that the original T2I model's generalizability is preserved. Extensive experiments demonstrate that the generated datasets improve the performance of previous segmentation models in urban-scene segmentation.
W1: The writing in this paper is somewhat challenging to understand, and the technical descriptions lack clarity, which can lead to confusion during reading. Below are a few examples, though not exhaustive. For instance, in Figures 3 and 4, what do the layer indices represent? Are they the projection layers for all attention in the network? However, according to the description in line 251, it seems that all linear layers in the network are being trained with LoRA. Additionally, Section 3 only covers the first two stages, with the third and fourth stages not being described in detail, making this part less clear. The structure of the experimental section is also somewhat disorganized. 

W2: The design of the tables lacks standardization, leading to confusion for the reader. Here are a few examples, though not exhaustive. For instance, many tables do not clearly explain what the numerical values represent, making interpretation difficult. In Table 3, the baseline names should be listed directly. Additionally, the entries under the "Data Ratio" column correspond to various methods, which creates some confusion. Furthermore, for the methods used to generate datasets that enhance baseline performance in Table 3, it would be clearer to label them as "Baseline + Real FT" rather than just "Real FT."

W3: Additionally, I noticed that the baseline appears to be from a 2022 paper. Are there any more recent baselines available for comparison?
W4: Some modules may not appear particularly novel from a technical perspective. LoRA are also commonly used in various papers.
Yes
No ethics review needed.
Additional minor questions are listed as follows.
 1.  The method uses selective LoRA to solve the problem of data scarcity in cross-domain segmentation. However, there are some similar methods to select LoRA weights in other fields, like LoRA-SP [3], GS-LoRA [2], Tied-LoRA [1] etc.. The authors should discuss these papers.  

[1] Tied-LoRA: Enhancing parameter efficiency of LoRA with Weight Tying  

[2] Continual Forgetting for Pre-trained Vision Models 

[3] LoRA-SP: Streamlined Partial Parameter Adaptation for Resource-Efficient Fine-Tuning of Large Language Models

2. The authors should compare more text-driven or image-driven generated dataset baselines, such as Instruct-Pix2Pix [4], PTDiffSeg [5], DATUM [6], etc.

[4] InstructPix2Pix: Learning to Follow Image Editing Instructions 

[5] Prompting Diffusion Representations for Cross-Domain Semantic Segmentation 

[6] One-shot Unsupervised Domain Adaptation with Personalized Diffusion Models
1. The paper is well-written and easy to follow.
2. The idea of using concept loss to find important weights is reasonable.
3. The ablation studies and analytical experiments are interesting and inspiring.
This paper addresses the challenge of data scarcity in semantic segmentation by generating datasets through fine-tuned text-to-image generation models. Existing methods often overfit and memorize training data, limiting their ability to generate diverse and well-aligned samples. This paper proposes Selective LoRA that selectively identifies and updates only the weights associated with necessary concepts for domain alignment while leveraging the pretrained knowledge of the image generation model to produce more informative samples.
The authors demonstrate its effectiveness in generating datasets for urban-scene segmentation.
1. The proposed method outlines how to fine-tune LoRA for generating informative target-style images. However, the definition of "informative samples" is not clear. This lack of clarity may hinder the reader's understanding of the intended contributions. For example, Figure 1 would benefit from including examples of informative data to provide a clearer context for what constitutes an informative sample. 

2. In Figure 1(b), the results of the LoRA-finetuned images for both foggy and night-time conditions appear remarkably similar, suggesting that the fine-tuning process may not have effectively generated between these two target styles. It raises concerns about the method's capability compared to the pretrained approach. 

3. The proposed Selective LoRA generates images in a specific style and containing particular content, but it lacks a comparative analysis with existing text-driven diffusion models, such as Instruct-Pix2Pix. A comparison in terms of both the quality of generated images and adaptation performance would significantly enhance the paper's contributions and provide the reader with a clearer understanding of how the proposed method stands in relation to established techniques.

4. I find the results presented in Table 3 somewhat confusing. If I understand correctly, the baseline results are derived from fine-tuning Mask2Former using generated images, while RealFT represents the results from fine-tuning on real data. However, it is unclear how the authors obtained the labels for the generated data. Were these results obtained through an unsupervised training approach, or was an additional decoder trained similarly to the DatasetDM?
Yes
No ethics review needed.
I believe that the proposed method demonstrates sufficient technical novelty and shows effectiveness through its quantitative experimental results. However, I think it would be beneficial for the authors to revise certain aspects to make the paper easier to understand. Improving the clarity of the text would enhance the overall presentation of the work.
The proposed method differentiate the weights to align with target domain while preserving valuable information of the pretarined model. From this, the proposed method selectively fine-tunes the model. This approach effectively addresses challenges when to adapt large pretrained models to different domains, enabling better domain specific performance without losing the benefits of the pretrained features.

The data scarcity problem addressed in this paper is a critical issue not only for semantic segmentation but also for a variety of vision tasks. Therefore, the proposed technique for generating image and ground truth pair datasets can be considered a core technology in the advancement of deep learning.
This paper proposes a novel approach to address the data scarcity in semantic segmentation by generating datasets (image-mask pairs) using text-to-image models. To solve the issue, it is necessary that the generative images align with the target doman and provide useful information beyond the training dataset. Therefore, this paper intorudces Selective LoRA, a finetuning approach for the pretrained text-to-image model that preserves the distributional diversity of the original pretrained model while aligning with the target domain. The proposed method selectively update weights for key concepts, such as style and viewpoint, which need to be aligned or to maintain the diversity. The authors shows that the proposed method generates datasets with desired distribution via ablation studies and improves in both in-domain and domain generalization settings.
I believe that the proposed method tackles an important problem and offers a reasonable approach to addressing the challenges, which I highly commend. However, there are some weaknesses to consider.

First, the method relies on Stable Diffusion, trained on a large dataset. Although leveraging the distributional diversity learned by this pretrained model is the motivation behind the approach, it inherently sets an upper bound on the applicability of the proposed method based on the knowledge of the pretrained model. This is a fundamental limitation.

Defining the desired concepts, identifying the critical parts of the architecture where these concepts are expressed, and retraining the model are all highly manual processes that depend heavily on individual characteristics of target data . To define the desired concepts, the user of this method must analyze the distributions of the pretrained model and the target domain, identify the differing concepts, and guide the process with appropriate text prompts. Additionally, finding the associated weights and determining their importance requires experimental work, which lacks standardized criteria.

Generating images aligned with the desired distribution is crucial, but creating high-quality masks to accompany these images is equally important for semantic segmentation models. This aspect has not been sufficiently addressed. While the current method leverages intermediate features, there could be consideration of various other ways to generate masks from the images. Given that the proposed method utilizes a model trained on a large dataset, it might also be worth exploring the use of models like SAM (Segment Anything Model) for mask generation (of course, there are lots of candidates). I am not requring additional experiments using SAM. It woud be beneficial to analyze the quality of the masks for the generated images. 

From a presentation, the paper is challenging to read. Figures 2 and 3 are difficult to understand, and it is not easy to infer the intended meaning from the related sections in the text. Additionally, the paper mentions L_Concept, but the figures use terms like L_style and L_viewpoint, which are not defined in the main text, causing confusion. The authors should clarify this and revise the figures accordingly. More detailed explanations about the process of creating text prompts are also necessary.

The experimental setup, including the ablation study, is not sufficiently explained. For instance, in experiments like those in Table 3, it is unclear how extensive the generated dataset is and how it is used.

Overall, the paper needs to be written in a way that is easier to understand.
We propose Selective LoRA, a novel fine-tuning method that generates well-aligned and informative segmentation datasets by updating only weights related to desired concepts. We improve existing urban-scene segmentation models in various settings.
@inproceedings{
anonymous2024selective,
title={Selective Lo{RA} for Domain-Aligned Dataset Generation in Urban-Scene Segmentation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=2TiU1JTdSQ},
note={under review}
}
This paper addresses the challenge of data scarcity in semantic segmentation by generating datasets through fine-tuned text-to-image generation models, reducing the costs of image acquisition and labeling. Segmentation dataset generation faces two key challenges: 1) aligning generated samples with the target domain and 2) producing informative samples beyond the training data. Existing methods often overfit and memorize training data, limiting their ability to generate diverse and well-aligned samples. To overcome these issues, we propose Selective LoRA, a novel fine-tuning approach that selectively identifies and updates only the weights associated with necessary concepts (e.g., style or viewpoint) for domain alignment while leveraging the pretrained knowledge of the image generation model to produce more informative samples. Our approach ensures effective domain alignment and enhances sample diversity.
We demonstrate its effectiveness in generating datasets for urban-scene segmentation, outperforming baseline and state-of-the-art methods in in-domain (few-shot and fully-supervised) settings, as well as domain generalization tasks, especially under challenging conditions such as adverse weather and varying illumination, further highlighting its superiority.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Dataset Generation Urban-scene Segmentation
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/2d54cc9c7c34de08be4a4734b3344660c4f68e12.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Selective LoRA for Domain-Aligned Dataset Generation in Urban-Scene Segmentation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission