Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPs
Reinforcement learning;action constraints
Yes
No ethics review needed.
1. How can the augmented MDP approach guarantee that actions remain within constraints? Even if complete avoidance of constraint violations is not feasible, it would be useful to establish an understanding, such as defining a convergence rate and probability that constraint violations will diminish to a certain margin of error over time.

2. Can the scalability of the proposed ARAM framework be demonstrated in high-dimensional action spaces, through a comparative analysis with existing ACRL methods?

3. Although the authors state "... we directly leverage the multi-objective extension of SAC that can learn policies under all the penalty weights." (page 2), it seems that multi-objective reinforcement learning approaches might be sensitive to hyperparameters, such as the penalty weight. Is ARAM similarly sensitive to this hyperparameter, particularly in its convergence behavior? Additionally, how is the penalty weight selected in practice?
1. The computational overhead caused by solving QPs is challenging in existing ACRL methods, particularly scaling to high-dimensional action spaces.

2. The paper integrates a multitude of concepts such as action-constrained reinforcement learning, Acceptance-rejection
method, and Augmented MDPs. These topics have been at the forefront of recent research trends.

3. The paper is well-organized, with clear explanations of the background, the proposed methods, theoretical foundations (including Proposition 1), and detailed experimental setups.
The paper presents acceptance-rejection augmented MDPs, a novel framework proposed to improve action-constrained reinforcement learning. The method aims to address the computational inefficiencies and architectural complexities in existing Action-Constrained Reinforcement Learning (ACRL), by incorporating two key innovations: an acceptance-rejection mechanism to filter out infeasible actions, and an augmented MDP to optimize policy training towards feasible regions by penalizing constraint violations. The framework is empirically tested against state-of-the-art ACRL benchmarks in robotic control and resource allocation domains. Overall, the idea of combining the acceptance-rejection method with the augmented MDP is interesting, but the work could be improved in both the methodology and the experimental demonstration (see Weakness and Questions below for further details).
1. The two main components, i.e., the acceptance-rejection method and augmented unconstrained MDPs, seem somewhat contradictory. They represent opposing optimization strategies: the acceptance-rejection method directly filters out actions outside the feasible set, potentially leading to overly conservative decisions; then, the augmented MDP penalizes constraint violations in a gradual way, providing a “soft” learning approach for the agent. However, this softer approach does not fully guarantee that actions remain within the feasible set.

2. While some theoretical insights are provided, including optimality equivalence in Proposition 1, key theoretical gaps remain unaddressed. Specifically, for augmented MDPs, there is no guarantee that actions won’t violate constraints, i.e., a critical issue in action-constrained reinforcement learning.

3. The introduction highlights the computational challenges of scaling quadratic programs to high-dimensional action spaces. However, the experimental analysis in Section 5 primarily explores action constraints in relatively lower-dimensional spaces. The performance of ARAM in high-dimensional action spaces (such as complex robotic manipulators) would benefit from more in-depth analysis.
Yes
No ethics review needed.
- Does the choice of different target distributions in ARM have a significant impact on the final results?
- Is the ARM’s acceptance rate analyzed in detail across the different stages of training? It would be useful to understand if and how the acceptance rate varies in the proposed setup.
- What role do quadratic programs (QP) play in ARAM? QPs are not included in Algorithm 1 or Figure 2.
- How is MORL implemented in ARAM? Does it significantly increase computational requirements?
- The layout of Algorithm 1 in the paper needs adjustment.
- Well-written paper with clear objectives.
- Although the two key improvements (ARM and AUTO-MDP) are straightforward, the experimental results demonstrate the elegance and efficiency of this method in addressing the ACRL problem.
- The paper is well-supported with experiments that compare ARAM against various benchmarks, showcasing improvements in training speed, constraint satisfaction, and inference time.
- The authors have the intention to share the code.
This paper proposes a novel framework for action-constrained reinforcement learning, named ARAM, which integrates an acceptance-rejection method and an augmented Markov decision process to alleviate the low action acceptance rate problem under ARM. ARAM aims to avoid the computational burden associated with quadratic programs in existing ACRL methods while achieving less constraint violations. The authors demonstrate through experiments on robot control and resource allocation tasks that ARAM achieves faster training, better constraint satisfaction, and reduced action inference time compared to various recent ACRL algorithms.
- The experiments are only conducted in simple simulation environments. Applying the method to real-world scenarios would make the results more impressive and convincing.
- Lacks experimental comparisons of the ARM acceptance rate.
- The paper lacks a discussion on the limitations of the method.
Yes
No ethics review needed.
1. Could you provide the value of $M$ and $\kappa$ for all the tasks?  Do you need extra tuning on these two hyperparameters? 
2. Could you provide the computation infrastructure?
3. Is it possible that ARAM with a single setting of fixed $\lambda$  outperforms other baselines across MuJoCo locomotion tasks? It is suggested that the authors conduct an experiment with a single fixed λ across all MuJoCo tasks and report those results, or explain why such an experiment might not be feasible or meaningful.
4. I find in Table 2 that NFWPO enjoys the highest valid action rate. That indicates that it requires a smaller number of QP operations, as opposed to DPre+ and SPre+. Yet in Figure 4, NFWPO shows the highest number of QP operations, which seems conflict to with the analysis in Lines 456-462. Please clarify this apparent contradiction and, if necessary, correct either the data or the analysis.
5. In the last row of Table 2, 0.77 is much lower than 0.84, thus I suggest not making it bold.
6. As claimed in the abstract, "We propose a generic and computationally efficient framework that can adapt a standard unconstrained RL method to ACRL through two modifications". Could the authors provide ARAM techniques with another backbone standard RL algorithm to support this claim?
1. This paper is well-written and easy to follow.
2. I appreciate the experiments in terms of the considerable evaluation metrics.
3. I am not very familiar with the ACRL field. From my perspective and limited knowledge of this field, I appreciate the proposed direction to solve the ACRL problem, I think it is a new try.
This paper aims to adapt the standard unconstrained RL method to ACRL via two proposed techniques: an acceptance-rejection method and an augmented two-objective MDP. It compares ARAM with various recent benchmark ACRL algorithms on MuJoCo tasks and Resource allocation for networked systems, showing the superiority of ARAM in terms of training efficiency, valid action rate, and per-action inference time.
1. The proposed preference distribution tuning procedure seems a bit redundant. It is better to compare ARAM with fixed $\lambda$  to the baselines for a fair comparison. Otherwise, it is hard to distinguish whether the main components (acceptance-rejection method and AUTO-MDP) are more important or the preference distribution tuning procedure. As we can observe in Figure 11, ARAM is not quite robust to the  $\lambda$, and a single set of  $\lambda$  could not achieve satisfactory performance across all tasks.  To make the comparison more fair and isolate the effects of the main components, it would be helpful to suggest that the authors include results for ARAM with a fixed λ alongside the current results in the main paper and hope to see the superiority of ARAM without $\lambda$-tuning. This would allow readers to better understand the relative contributions of the acceptance-rejection method, AUTO-MDP, and preference tuning. Of course, you may not need to fully resolve this weakness, yet I suggest explicitly mentioning it as a limitation. 
2. Several experimental details have not been provided. For example, what is the value of M and $\kappa$ for all the tasks, and other parameters used? They are quite important details.
Yes
No ethics review needed.
1. What metrics were used to evaluate the acceptance rate during the experiments, and how were these metrics calculated?
2. Is there a systematic approach to determine the optimal penalty weight, or is it left to empirical tuning?
3. Conducting more experiments on hyperparameter sensitivity is better.
4. Whether introducing AUTO-MDP in the theoretical aspect will affect the guarantee of convergence, while the current article only discusses the optimality.
The framework significantly reduces the computational burden associated with traditional ACRL methods, which often rely on solving complex quadratic programs (QPs). By utilizing the acceptance-rejection method, the need for QP operations is largely obviated, leading to faster training times.
This work presents a groundbreaking approach to learning control policies that adhere to strict action constraints, making it perfect for safety-critical and resource-constrained applications. The authors introduce a novel framework that enhances traditional reinforcement learning methods by utilizing the acceptance-rejection method and an augmented two-objective Markov decision process.
The effectiveness of the acceptance-rejection method heavily relies on the quality of the initial policy. If the initial policy is far from optimal, it may result in a high number of rejected actions, which can hinder learning progress and increase computational overhead. While the augmented MDP approach aims to improve acceptance rates, it introduces additional complexity to the learning process. The design and tuning of the penalty function and self-loop transitions may require careful consideration, and improper tuning could lead to suboptimal performance.
@inproceedings{
anonymous2024efficient,
title={Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented {MDP}s},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=AgMpK7z4bz},
note={under review}
}
Action-constrained reinforcement learning (ACRL) is a generic framework for learning control policies with zero action constraint violation, which is required by various safety-critical and resource-constrained applications. The existing ACRL methods can typically achieve favorable constraint satisfaction but at the cost of either high computational burden incurred by the quadratic programs (QP) or increased architectural complexity due to the use of sophisticated generative models. In this paper, we propose a generic and computationally efficient framework that can adapt a standard unconstrained RL method to ACRL through two modifications: (i) To enforce the action constraints, we leverage the classic acceptance-rejection method, where we treat the unconstrained policy as the proposal distribution and derive a modified policy with feasible actions. (ii) To improve the acceptance rate of the proposal distribution, we construct an augmented two-objective Markov decision process (MDP), which include additional self-loop state transitions and a penalty signal for the rejected actions. This augmented MDP incentives the learned policy to stay close to the feasible action sets. Through extensive experiments in both robot control and resource allocation domains, we demonstrate that the proposed framework enjoys faster training progress, better constraint satisfaction, and a lower action inference time simultaneously than the state-of-the-art ACRL methods.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Reinforcement learning action constraints
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/1b3a220325a75aaa01c4a7b1c81fb0f66650910c.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/964ca4ab3f00377d3f5245f3ec551f404c2aee3a.zip
Efficient Action-Constrained Reinforcement Learning via Acceptance-Rejection Method and Augmented MDPs
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission