Achieving Zero Constraint Violation for Concave Utility Constrained   Reinforcement Learning via Primal-Dual Approach
Reinforcement learning (RL) is widely used in applications where one needs to perform sequential decision-making while interacting with the environment. The standard RL problem with safety constraints can be formulated via constrained Markov Decision Processes (CMDP), which is linear in objective and constraints in occupancy measure space, where the problem becomes challenging in the case where model is unknown apriori. The problem further becomes challenging when the decision requirement includes optimizing a concave utility while satisfying some nonlinear safety constraints (we call this problem concave CMDP (CCMDP)). In the literature, various algorithms are available to solve CMDP problems in a model-free manner to achieve $\epsilon$-optimal cumulative reward with $\epsilon$ feasible policies. An $\epsilon$-feasible policy implies that it suffers from constraint violation. An important question here is whether we can achieve $\epsilon$-optimal objective with zero constraint violations or not. To answer that in affirmative, we advocate the use of randomized primal-dual approach to solve the CCMDP problems and propose a conservative stochastic primal-dual algorithm (CSPDA). We prove that CSPDA exhibit $\tilde{\mathcal{O}}\left(1/\epsilon^2\right)$ sample complexity to achieve $\epsilon$-optimal concave utility with zero constraint violations for the CCMDP, and CMDP as well which is a special case. In the prior works, the best available sample complexity for the $\epsilon$-optimal policy for CMDP with zero constraint violation is $\tilde{\mathcal{O}}\left(1/\epsilon^5\right)$. Hence, the proposed algorithm provides a significant improvement as compared to the state of the art.