Distributionally Robust Policy Learning under Concept Drifts
distributionally robust optimization;offline policy learning;concept drift;bandit learning;reinforcement learning.
Yes
No ethics review needed.
How are the rates here compared with existing rates?

Are there lower bounds of value estimation and regret?
Separating concept shift from covariate shift offers a fresh and insightful perspective in this field. The methods developed in this paper come with strong theoretical guarantees: the policy value estimator exhibits asymptotic normality, and the policy learning algorithm has $n^{-1/2}$ sub-optimality gap, improving previous results. Additionally, experiments conducted in a synthetic environment demonstrate that the proposed algorithm consistently outperforms baseline methods, further supporting the effectiveness of the approach.
This paper studies robust policy learning under the concept drift, where the distributional shift occurs only in the conditional reward distribution. The authors first develop a doubly-robust estimator for the worst-case policy value under concept drift, which is proved to be asymptotic normal, and then design a robust policy learning algorithm which achieves $n^{-1/2}$ regret.
While the paper is motivated by the idea that existing methods may be suboptimal under concept shift alone, it does not clearly demonstrate how its proposed rates improve upon existing ones. The improvement noted in Remark 4.3 seems more a result of advanced theorem-proving techniques (e.g., chaining) rather than a genuine improvement in performance metrics. Additionally, while the introduction raises the question of "optimal worst-case average performance" (lines 75-77), the paper lacks an optimality analysis of the derived bounds. Consequently, the theoretical contributions do not align strongly with the paper's motivation and feel somewhat insufficient in addressing the posed questions.

Furthermore, the experiments are conducted solely in a synthetic environment, with no practical examples illustrating the benefits of the proposed algorithm.
Yes
No ethics review needed.
see the summary
see the summary
This paper studies the problem of distributionally robust contextual bandit, where the uncertainty only lies in the conditional reward distribution P(Y|X).

Some major comments are listed below:
1. First, Lemma 2.3 cannot be claimed as the contribution of this paper, as it is a standard result in the literature of KL constrained DRO. The authors shall provide credit to existing literature properly. 
Kullback-Leibler Divergence Constrained Distributionally Robust Optimization, Hu and Hong
Moreover, min over \eta in (3) has a closed form solution. Please check theorem 1 in the above paper. Therefore, there is no need for an iterative method to obtain \eta^*.

2. The technical presentation is hard to follow. Notations are not defined or not clearly introduced before use. For example g_\pi is not defined. What does propensity score function \pi_0 represent is not clear. Does the propensity score function mean the same thing as the behavior policy? 

3. Assumption 3.3. holds if $\theta^*_{\pi}(x)$ is continuous in x. Can the authors provide examples for this to hold?

4. The problem in this paper is in an offline setting, however, assumption 2.1 guarantees that any action can be visited in the training dataset with high probability, and therefore, the most challenging part of offline RL, e..g, partial coverage, is not addressed in this paper. How the performance depends on the concentrability coefficient is not clear from this paper. This limits the significance of this work. 

5. This paper investigates only distributional shift in the conditional reward, however, this does not seems to require development of new techniques comparing to distribution shift in the entire P(x,y).
see the summary
Yes
No ethics review needed.
The authors could also expand on how to choose the uncertainty set parameter, as the results depend heavily on this choice. Additionally, they could discuss why they opted for KL-divergence over alternatives like Wasserstein distance, which might offer different insights or benefits in the context of distributional robustness.
This paper discusses distributionally robust policy learning under concept shift. It relies on standard assumptions and considers a KL-divergence-based evaluation framework. It presents a duality result and provides an efficient estimator based on this result, with guarantees for asymptotic estimation rates and asymptotic normality.

The result is interesting and proposed concept shift makes sense because in practice, considering the worst-case joint distribution of the covariate and the outcome can be too conservative.
This paper addresses distributionally robust policy learning under concept drift, focusing on cases where only the conditional relationship between covariates and outcomes changes. Using a KL-divergence-based framework, the authors define an uncertainty set that constrains shifts in the conditional distribution and develop a computational approach supported by duality to evaluate and optimize policies efficiently. They provide theoretical guarantees, including asymptotic normality and convergence rates, adding robustness to the methodology. Initial numerical results suggest the approach is less conservative than traditional joint distribution shift methods.
- The duality and theoretical results appear fairly standard, with proofs similar to existing literature. 
- The numerical results are very preliminary and lack in-depth discussion.  For example, $𝑥$ is assumed to follow a Gaussian distribution. 
- Additionally, the authors should expand on the results in Table 1. While they claim their method is better, the table mainly shows it is less conservative (which is expected, given the smaller uncertainty set they consider compared to the joint distribution worst-case uncertainty set). 
- They should provide more justification for why this choice makes sense in practice and discuss how to empirically determine the appropriate uncertainty type for policy learning (joint distribution shift vs. concept shift), which is not covered in the experiments.
@inproceedings{
anonymous2024distributionally,
title={Distributionally Robust Policy Learning under Concept Drifts},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=TbRFc2HPoN},
note={under review}
}
Distributionally robust policy learning aims to find a policy that performs well 
under the worst-case distributional shift, and yet most existing methods for 
robust policy learning consider the worst-case joint distribution of 
the covariate and the outcome. The joint-modeling strategy can be unnecessarily conservative
when we have more information on the source of distributional shifts. This paper studies
a more nuanced problem --- robust policy learning under the concept drift, 
when only the conditional relationship between the outcome and the covariate changes. 
To this end, we first provide a doubly-robust estimator for evaluating
the worst-case average reward of a given policy under a set of perturbed conditional distributions. 
We show that the policy value estimator enjoys asymptotic normality even if the nuisance parameters 
are estimated with a slower-than-root-$n$ rate. We then propose a learning algorithm that outputs the policy maximizing the 
estimated policy value within a given policy class $\Pi$, and show
that the sub-optimality gap of the proposed algorithm is of the order 
$\kappa(\Pi)n^{-1/2}$, with $\kappa(\Pi)$ is the entropy integral of $\Pi$ under the Hamming distance
and $n$ is the sample size.
The proposed methods are implemented and evaluated in numerical studies, 
demonstrating substantial improvement compared with existing benchmarks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
distributionally robust optimization offline policy learning concept drift bandit learning reinforcement learning.
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/ff6f7d7d7e18d3bdaf340d90748dfcda50a3e8dd.pdf
causal reasoning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Distributionally Robust Policy Learning under Concept Drifts
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission