Model Risk-sensitive Offline Reinforcement Learning
risk-sensitive offline reinforcement learning;reinforcement learning;offline reinforcement learning;risk;model risk
Yes
No ethics review needed.
1) It is mentioned that when $\epsilon \leq (\mu_z-\mu)^2 + (\sigma_z - \sigma)^2$ holds, the quantile function doesn't have a solution. Under what distribution conditions does this occur, and how often is it observed? It would be useful to know if the strictly-less-than case occurs with relative infrequency, or very often.

2) Baselines ORAAC, CODAC, and 1R2R are mentioned in the introduction and related works, but only the first two are used as comparison baselines. Is there a reason the most recent method was excluded?
The motivation is clear; the problem statement, issues with existing works, and connections to the provided solution are clean and easy to follow. I also appreciate the label brackets underneath parts of the equations. 

The contribution addresses the issue both distributional shifts in data and model inaccuracies, which are critical challenges in offline RL.
This paper addresses the problem of model risk in offline reinforcement learning, which arises from distributional shifts in the learning dataset. The authors provide a critic ensemble framework, integrating well-researched predecessor methods into a critic ensemble for risk estimation that can be applied regardless of the transition distribution or differences in distributional support. Experimental results in two domains demonstrate the superiority of the method and highlight the tradeoffs between robustness and high performance.
Most of the formalizations are cited directly from previous works, with the exception of 4.1. Given that 4.1 is only half of the proposed methods section, this makes the contribution seem more incremental. Are there changes or improvements that can be made to any of these components to better suit them to the domain? As an example, the Bellamn equation in Eq. 4 could be extended to measures such as minimum reward, expected value (over actions), or regret.

There are a few minor spelling errors, like 'qunatile' in the Fig. 4 caption.
Yes
No ethics review needed.
1. I appreciate the illustrative explanation of the calculation for $ F_{ens.}^{-1} $; however, is there a mathematical formula for this calculation? Additionally, were gradients preserved when calculating $ F_{ens.}^{-1} $ during experiments?
2. In the experiments, how was the offline dataset generated?
1. **Novelty:** To the best of my knowledge, this is a novel formulation for risk-sensitive offline RL, with no similar approach appearing in current literature. Additionally, the use of a critic ensemble and a Fourier feature network for variance and bias reduction is a thoughtful design choice.
2. **Clarity:** The paper is well-organized and self-contained, providing a clear presentation of the problem, algorithmic approach, and experimental setup.
3. **Comprehensive Experiments:** The evaluation on finance and self-driving applications aligns well with the risk-sensitive focus of the paper. Extensive experiments, including a range of baselines, lend further credibility to the results.
This work introduces a framework and learning algorithm, MR-IQN, for offline reinforcement learning in a risk-sensitive context. The MR-IQN algorithm aims to identify a policy that minimizes model risk while remaining close to the behavior policy derived from the dataset. The authors evaluate the algorithm on both finance and self-driving datasets, demonstrating that it outperforms baseline methods in terms of model risk measures and average return.
**Major Issues:**

1. The formulation of risk-sensitive offline RL optimizes model risk regularized by the dataset's behavior policy. It is unclear how this formulation guarantees that the optimality considers both model risk and expected return, especially when the behavior policy in the dataset may not be optimal, as is often the case in offline RL.
2. There is a lack of theoretical insight into the algorithm. Specifically, it is not evident why the proposed loss function in Equation (11) adequately solves the problem defined in Equation (10). Additional theoretical justification would enhance understanding of the algorithm's effectiveness.

**Suggested Improvements:**

1. In the experimental section, an ablation study on ensemble size (i.e., number of critics) would help substantiate the value of using an ensemble for performance gains.
2. The authors choose $\mu$ as the minimum value among critics' estimations and $\sigma$ as the maximum, aiming for conservatism. It would strengthen the argument if they defined conservatism and justified the choices of $\mu$ and $\sigma$.

**Minor Issues:**

The following suggestions may improve clarity:

1. In Equation (3), the form $ F^{-1}_{\phi}(U[0, 1]) $ is not clearly defined by Definition 1.
2. Is $ Z_{\pi} $ intended to mean the same as $ Z^{\pi} $? If so, consistent notation is recommended.
3. In lines 275-276, the notation $ E_{p_j} $ appears undefined or unclear.
4. In Figure 4, the x-axis and y-axis require explicit labeling for clarity.
5. In Tables 1 and 2, the meaning of "Average" should be clarified.
Yes
No ethics review needed.
See above
1. The approach of minimizing model risk in offline RL using a critic ensemble, rather than focusing on conventional risk, is an interesting and novel idea.
2. MR-IQN demonstrates strong results in the presented environments, indicating promising potential.
This paper introduces MR-IQN, a model-risk-sensitive offline RL algorithm, which aims to minimize risk in worst-case scenarios instead of conventional risk. Building on TD3+BC, it incorporates a critic ensemble to estimate model risk and employs a Fourier feature quantile regression network to mitigate spectral bias, allowing for precise distributional statistics estimation. MR-IQN outperforms baseline algorithms CODAC and ORAAC in two finance environments, Forex and Stock, as well as in the Airsim environment.
1. The experimental section feels underdeveloped, as MR-IQN and the baselines are only tested in three environments. Expanding the evaluation to include a broader set of benchmarks, such as those in D4RL (as done in CODAC and ORAAC), would significantly strengthen the results.
2. The choice of metrics in Table 1 raises some concerns. Evaluating performance at the 50% quantile is unusual, as this metric doesn’t emphasize rare events, which seems at odds with the motivation of risk-sensitive RL. To the best of my knowledge, previous literature typically uses the 10% quantile. Could you clarify why CV@R(50%) was selected for this study?
3. The explanation of model risk is somewhat unclear, and Figure 1 is particularly challenging to interpret. A more detailed and structured presentation of model risk concepts would improve readability and understanding.
Yes
No ethics review needed.
- Why can we translate the objective (8) to the objective (10)?

- In step 1, why does MR-IQN choose the **smallest** expectation and the **largest** deviation?

- Why do Fourier features benefit a lot, are there any explanations?

- Are there ablation studies about the number of ensembled critics?
- Considering spectral risk measure is interesting in risk-sensitive RL as it seems a general form of several risk metrics like CVaR, worst case, and Wang risk.

- Extensive ablation studies show the importance of each component in MR-IQN.

- Fig. 4 explains the advantage of Fourier feature clearly.
This work considers the risk-sensitive setting in offline RL, which is important for real-world applications. This work proposes to minimize the worst-case risks across a set of plausible alternative scenarios rather than solely focusing on minimizing estimated risk. The proposed method, MR-IQN, introduces a critic-ensemble criterion method, the learned Fourier feature framework, and the IQN framework to address spectral bias in neural networks. Extensive experiments show that MR-IQN can significantly reduce the different kinds of risk, like CVaR and Wang risk.
- It seems that the algorithm is a direct application of previous theoretical results (Bernard et al., 2023) as the major theoretical results (Thm 1, Cor 1) are from (Bernard et al., 2023). What is the major challenge in applying these theoretical analyses to your methods?

- There seems to be a gap between the objective 10 (can be simplified by Cor. 1) and the proposed algorithm. Why can MR-IQN handle objective 10? As objective 10 is a constrained optimization problem and hard to solve directly, do you use bi-level optimization, or use the Lagrange multiplier method, or some other methods? It will make the work more solid by providing a more detailed explanation of how MR-IQN addresses the constrained optimization problem in objective 10, including any specific optimization techniques they use.

- Lack of some related work on several risk-sensitive RL methods including different risk metrics which are closely related to this work, like CVaR [1-4], EaR [5], worst case [6], and so on.

Reference:

[1] TRC: Trust region conditional value at risk for safe reinforcement learning

[2] Towards safe reinforcement learning via constraining conditional value-at-risk

[3] Distributional reinforcement learning for risk-sensitive policies

[4] Efficient risk-averse reinforcement learning

[5] Risk-sensitive reinforcement learning via Entropic-VaR optimization

[6] Provably efficient risk-sensitive reinforcement learning: Iterated cvar and worst path
We proposed a model risk-sensitive offline RL framework, devising the critic-ensemble criterion to capture model risk effectively. To ensure the precision of model risk calculation, we employed Fourier feature networks.
@inproceedings{
anonymous2024model,
title={Model Risk-sensitive Offline Reinforcement Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=h6k4809xVV},
note={under review}
}
Offline reinforcement learning (RL) is becoming critical in risk-sensitive areas such as finance and autonomous driving, where incorrect decisions can lead to substantial financial loss or compromised safety. However, traditional risk-sensitive offline RL methods often struggle with accurately assessing risk, with minor errors in the estimated return potentially causing significant inaccuracies of risk estimation. These challenges are intensified by distribution shifts inherent in offline RL. To mitigate these issues, we propose a model risk-sensitive offline RL framework designed to minimize the worst-case of risks across a set of plausible alternative scenarios rather than solely focusing on minimizing estimated risk. We present a critic-ensemble criterion method that identifies the plausible alternative scenarios without introducing additional hyperparameters. We also incorporate the learned Fourier feature framework and the IQN framework to address spectral bias in neural networks, which can otherwise lead to severe errors in calculating model risk. Our experiments in finance and self-driving scenarios demonstrate that the proposed framework significantly reduces risk, by $11.2\%$ to $18.5\%$, compared to the most outperforming risk-sensitive offline RL baseline, particularly in highly uncertain environments.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
risk-sensitive offline reinforcement learning reinforcement learning offline reinforcement learning risk model risk
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/ed3a5f3dadc917a77d012613e1108a18371fec91.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/5bc8ea93bf60c16e0827e0d95eda6a0527733b8b.zip
Model Risk-sensitive Offline Reinforcement Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission