Learn With Imagination: Safe Set Guided State-wise Constrained Policy Optimization
Deep Reinforcement Learning;Safe Control;Safety Index;Zero Training Violations;Imaginary Cost
This paper is desk rejected because the authors are not anonymized.
Submission Desk Rejected by Program Chairs
Safe Set Guided State-wise Constrained Policy Optimization (S-3PO) is a novel algorithm generates state-wise safe optimal policies with zero training violations.
@misc{
li2024learn,
title={Learn With Imagination: Safe Set Guided State-wise Constrained Policy Optimization},
author={Feihan Li and Yifan Sun and Weiye Zhao and Rui Chen and Tianhao Wei and Changliu Liu},
year={2024},
url={https://openreview.net/forum?id=PAzVN4EEkj}
}
Deep reinforcement learning (RL) excels in various control tasks, yet the absence of safety guarantees hampers its real-world applicability. In particular, explorations during learning usually results in safety violations, while the RL agent learns from those mistakes. On the other hand, safe control techniques ensure persistent safety satisfaction but demand strong priors on system dynamics, which is usually hard to obtain in practice. To address these problems, we present Safe Set Guided State-wise Constrained Policy Optimization (S-3PO), a pioneering algorithm generating state-wise safe optimal policies with zero training violations, i.e., learning without mistakes. S-3PO first employs a safety-oriented monitor with black-box dynamics to ensure safe exploration. It then enforces a unique cost for the RL agent to converge to optimal behaviors within safety constraints.  S-3PO outperforms existing methods in high-dimensional robotics tasks, managing state-wise constraints with zero training violation. This innovation marks a significant stride towards real-world safe RL deployment.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Feihan_Li1 ~Yifan_Sun9 ~Weiye_Zhao1 ~Rui_Chen11 ~Tianhao_Wei1 ~Changliu_Liu1
Feihan Li Yifan Sun Weiye Zhao Rui Chen Tianhao Wei Changliu Liu
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Deep Reinforcement Learning Safe Control Safety Index Zero Training Violations Imaginary Cost
I certify that there is no acknowledgement section in this submission for double blind review.
li|learn_with_imagination_safe_set_guided_statewise_constrained_policy_optimization
/pdf/90aabcb4a4db376251ef5d1ad42d6a6d4bfcb050.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/593eaa67a71741e20d7d836a80fa76d84a323624.zip
Learn With Imagination: Safe Set Guided State-wise Constrained Policy Optimization
ICLR 2025 Conference Desk Rejected Submission
ICLR.cc/2025/Conference/Desk_Rejected_Submission