Open-Set Domain Adaptation Under Background Distribution Shift: Challenges and A Provably Efficient Solution
Distribution Shift;Open Set Domain Adaptation;Novel Category Detection;Anomaly Detection;Out of Distribution
Yes
No ethics review needed.
NA
- identification and formalization of understudied OSDA challenges
- Specific attention to challenging real-world scenarios like low-proportion novel classes
This paper addresses a gap in Open-Set Domain Adaptation (OSDA): handling both novel class detection and distribution shifts in known classes ("background shift"). The authors propose a scalable solution combining principled novelty detection with shared representation learning, demonstrating strong results across multiple domains.
- All datasets are semi-synthetic. It would be interesting if authors can show results on practical datasets. 
- While the problem addressed is quite interesting theoretically, practical relevance is unclear. As mentioned above, the paper may benefit from including results on datasets which inherently satisfy the problem discussed in the paper
- The paper may also benefit from discussing how using stronger CLIP models alters the behavior of the baselines and the proposed method.
Yes
No ethics review needed.
See Weaknesses
- Strong theoretical foundation with formal analysis of background shift in OSDA, including necessary/sufficient conditions and limitations of existing approaches.
- Comprehensive empirical evaluation across multiple modalities (images and text) and varying novel class ratios.
This paper addresses Open-Set Domain Adaptation under a specific scenario referred to as "background shift," where known classes have partial overlap between source and target domains, while novel classes remain distinct. The authors introduce CoLOR (Constrained Learning for Open-set Recognition), a multitask learning framework that applies constraints on target data to reduce bias in the feature extractor. The key insight is that traditional partial domain alignment methods struggle when source-private classes significantly outnumber common classes. CoLOR addresses this by jointly optimizing for closed-set classification and novel class detection through a constrained learning approach. The method is evaluated comprehensively across three datasets (CIFAR100, SUN397, Amazon Reviews), demonstrating improved performance, particularly in scenarios with low novel class ratios. The theoretical contributions include a formal analysis of necessary/sufficient conditions for OSDA under background shift.
- While background shift is justified with potential applications in medical imaging (e.g., identifying known and novel tumor cells), this application is not addressed in the experiments. Including other real-world cases would strengthen the relevance of background shift beyond this initial mention.

- The abstract could be improved by defining “background shift” and avoiding vague phrases like "principled methods."

- The claim “we observe that existing OSDA methods are not robust to the distribution shifts we consider” is strong, but the baselines used are limited. Adding recent OSDA and UniDA baselines would support this claim.
Suggested OSDA baselines: Adjustment and Alignment for OSDA [1], Open-Set Domain Adaptation for Semantic Segmentation [2], Source-Free Progressive Graph Learning for OSDA [3].
Suggested UniDA baselines: Compressive Attention Matching for UniDA [4], Classifiers of Prototypes and Reciprocal Points for UniDA [5].

- The paper assumes “notable separability” between known and novel classes, which might be feasible for images or text using pretrained models like ViT-CLIP. However, this separability may not hold in fields like medical imaging, where pretrained models aren’t available.

- The authors should clarify how they determine the number of classes (k) for any additional classifier head and discuss the impact of varying k.

- In Table 2, BODA outperforms CoLOR on CIFAR100, but this result isn’t highlighted in bold.

- In Table 3, only one baseline is used, which is outdated. The chosen method isn’t clear either, given the three references listed (‘These baselines are domain discriminator (DD), Elkan & Noto (2008); du Plessis et al. (2014); Garg et al. (2021)’).

- Is the “OSCR” metric referring to the commonly used H-score in open-set papers?

- The method is sensitive to hyperparameters like the FPR; details on selecting FPR and setting the threshold  β for optimization should be included.

-  The code is missing .
 
[1] LI, Wuyang, LIU, Jie, HAN, Bo, et al. Adjustment and alignment for unbiased open set domain adaptation. In : Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023. p. 24110-24119.

[2] CHOE, Seun-An, SHIN, Ah-Hyung, PARK, Keon-Hee, et al. Open-Set Domain Adaptation for Semantic Segmentation. In : Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024. p. 23943-23953.

[3] LUO, Yadan, WANG, Zijian, CHEN, Zhuoxiao, et al. Source-free progressive graph learning for open-set domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, vol. 45, no 9, p. 11240-11255.

[4] ZHU, Didi, LI, Yinchuan, YUAN, Junkun, et al. Universal domain adaptation via compressive attention matching. In : Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023. p. 6974-6985.

[5] HUR, Sungsu, SHIN, Inkyu, PARK, Kwanyong, et al. Learning classifiers of prototypes and reciprocal points for universal domain adaptation. In : Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023. p. 531-540.
Yes
No ethics review needed.
1. Why did the authors define background shift not as a distinction between support of $P_{T,[k]}$ and $P_S$ i.e. $Supp(P_{T,[k]}) \neq Supp(P_S)$ but rather as $Supp(P_{T,[k]}) \subseteq Supp(P_S)$???? 

2. Could you elaborate on the differences in assumptions and results between Theorem 1 in this manuscript and Theorem 4.3 in [1]? If there are differences, please explain how these differences are reflected in the proof process of Theorem 1.

3. I am curious whether the recent UDA methods demonstrate low performance even when applied in OSDA with background shift.
1. Introducing background shift in OSDA community is interesting and novel.

2. Proposed method is based on theoretical evidence.

3. Experiments are well structured to verify the effectiveness of background shift.
This work suggests a new Open-Set Domain Adaptation (OSDA) scenario, “background shift” which is a distribution shift with overlapping supports between source and target distributions exists within the known classes. Also, the existing OSDA methods are experimentally shown to be insufficient in addressing background shifts. To handling this new scenario, this work proposes a new method dubbed as CoLOR. Experimental results show the effectiveness of the proposed method.
1. This work has limited novelty and originality since it relies heavily on [1], which provide theoretical analysis of OOD novel category detection. Lemma 1 and Theorem 1 in this manuscript are quite similar claim with Proposition 3.1 and Theorem 4.3 in [1] though they focus different task. Furthermore, it is unclear that Theorem 1 in this manuscript can be easily extended on OSDA since there is no proof in the manuscript and supplementary material.

2. Proposed method is rather incremental. I think it is concatenate the cross-entropy loss on [1].

3. Baselines for performance comparison is insufficient and outdated. I think several recent UDA methods should be included.

[1] Birds of an Odd Feather: Guaranteed Out-of-Distribution (OOD) Novel Category Detection
Yes
-
No ethics review needed.
See questions posed in the above section.
- The setting of OSDA is both an important and challenging area  

- The paper uses an OOD method in combination with learning a classifier to do OSDA which is a reasonable approach which seems to work well

- Method performs well on chosen datasets compared to baselines
This work sets out to address the open set domain adaptation (OSDA) problem under so-called background shifts. This problem amounts to solving the domain adaptation problem while also identifying datapoints which are novel classes which were not seen in the source domain. Domain adaptation is the problem of finding a model which performs well in a target domain when it has been trained on labeled data from some source domain and unlabeled data from the target domain. Background shifts are defined as any distribution shift between source and target domains which retains overlapping support between source and target domain distributions with the non-novel label set. 

The authors show that strong positivity and overlapping support is not a sufficient condition to ensure better than random performance. 
The work proposes a model partly based on previous work in OOD detection. The work proposes to use this method to detect the novel classes while at the same time learning to do the classification. The architecture of the model has a shared base structure which is used for both the detection and classification heads.

The model is then compared on image and text dataset benchmark against several baselines. It performs favourably on these tasks compared to baselines, especially when the ratio of novel class datapoints is low.
- The introduction and related work together makes up ~3 pages which is a substantial part of the paper.

- It seems rather inefficient to solve the problem for a large number of possible values of $\alpha$. How does this compare to the other baseline methods? 


- The experimental details are somewhat unclear. Some things are not clearly stated. 
    * How many repetitions of experiments were done?
    * How is the separability of the novel classes ensured? 

- Why does the amount of 'wins' not tally up to the total number of repetitions? I assume that this is what the Wins column in table 2 means, since I cannot find it explained in the text. Does this mean that the models tie in performance in some runs? 

- Not clear to me how the choice of $\beta$ threshold impacts the results. 

Typos and other comments:

- Unclear what the bolding in table 3b means, it almost seems random.

- Figure 2 is small and admits a large amount of white space for seemingly little reason.

- The bullet points headers for the paragraphs in the introduction seem distracting to me


line 283: largrangian
line 355: Expertiments
line 385: table table
line 480: Refer 5,6,7,10 in for further results. - Supposed to be reference to some section here?
line 481: AUORC
Yes
No ethics review needed.
Please see Weaknesses above.
1. This paper addresses the overlooked issue of background distribution shifts in OSDA, relevant for dynamic applications like medical imaging and autonomous systems.

2. Extensive experiments across diverse datasets validate CoLOR's performance and robustness against strong baselines.

3. The paper is well-organized, with clear explanations, detailed methodology, and effective visual aids for complex concepts.
This paper investigates Open-Set Domain Adaptation (OSDA) under a background distribution shift. In OSDA, the objective is to classify known categories while detecting novel, previously unobserved classes in the target domain. This study addresses background shifts, where the distribution of known classes in source and target domains differs. The authors present a scalable method named CoLOR (Constrained Learning for Open-Set Recognition) to robustly classify known classes and detect novel ones under this shift. They demonstrate that CoLOR outperforms existing baselines by jointly learning representations for both classification and novelty detection.
1. The separability assumption (Assumption 1) is central to CoLOR’s effectiveness. Could you discuss scenarios where this assumption might not hold in practical applications? How would CoLOR perform if the separability or background shift assumptions were relaxed or violated?

2. The writing quality could be enhanced by ensuring consistency in formatting, particularly in the Related Works section, where the subheading styles appear inconsistent. Standardizing these subheading formats would improve the paper’s overall readability and professionalism.
We study the problem of Open-Set Domain Adaptation under distribution shift of non-novel classes and study the impact of various factors affecting the performance of existing methods such as novel class ratio.
@inproceedings{
anonymous2024openset,
title={Open-Set Domain Adaptation Under Background Distribution Shift: Challenges and A Provably Efficient Solution},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=Yd2GeHRSlJ},
note={under review}
}
In Open-Set Domain Adaptation (OSDA) we wish to perform classification in a target domain which contains a novel class along with $k$ non-novel classes. This work formally studies OSDA under the assumption that classes are separable, and the supports of source and target domains coincide, while other aspects of the distribution may change.
We develop a simple and scalable method that attains robustness to distribution shift and is guaranteed to solve the problem, while showing that it cannot be solved under weaker conditions that have been studied for OSDA in the past, particularly in the presence of covariate shift. We formally define the realistic assumptions within the scope of OSDA problem that the previous literature has either overlooked or not explicitly addressed. In a thorough empirical evaluation on both image and text data, we observe that existing OSDA methods are not robust to the distribution shifts we consider.
The results demonstrate the efficacy of joint representation learning for classification of known classes and detection of novel ones using principled methods. We find that optimizing these two objectives in unison leads to mutual improvements in task performance contrary to what might be expected when objectives are considered independently. Our rigorous empirical study also examines how OSDA performance under distribution shift is affected by parameters of the problem such as the size of novel class. 
 Taken together, our observations emphasize the importance of formalizing assumptions under which OSDA methods operate and to develop appropriate methodology that are capable of scaling with large datasets and models for different scenarios of OSDA.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Distribution Shift Open Set Domain Adaptation Novel Category Detection Anomaly Detection Out of Distribution
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/9217344a88ae8af3d57bb7fac61e358069afca35.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Open-Set Domain Adaptation Under Background Distribution Shift: Challenges and A Provably Efficient Solution
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission