Reinforcement Learning with Adaptive Control Regularization for Safe Control of Critical Systems
Haozhe Tian Homayoun Hamedmoghadam Robert Shorten Pietro Ferraro



The Power of Resets in Online Reinforcement Learning
    Zakaria Mhammed
 Dylan J. Foster
 Alexander Rakhlin


    
=========================================================





Reinforcement Learning (RL) is a powerful method for controlling dynamic systems, but its learning mechanism can lead to unpredictable actions that undermine the safety of critical systems. Here, we propose RL with Adaptive Control Regularization (RL-ACR) that ensures RL safety by combining the RL policy with a control regularizer that hard-codes safety constraints over forecasted system behaviors. The adaptability is achieved by using a learnable focus weight trained to maximize the cumulative reward of the policy combination. As the RL policy improves through off-policy learning, the focus weight improves the initial sub-optimum strategy by gradually relying more on the RL policy. We demonstrate the effectiveness of RL-ACR in a critical medical control application and further investigate its performance in four classic control environments.





§ INTRODUCTION



A wide array of control applications, ranging from medical to engineering, fundamentally deals with critical systems, i.e., systems of vital importance where control actions have to guarantee no harm to the system functionality. Examples include managing nuclear fusion <cit.>, performing robotic surgeries <cit.>, and devising patient treatment strategy <cit.>. Due to the critical nature of these systems, ensuring control safety and reliability is paramount.

 Reinforcement Learning (RL), which identifies the optimum policy through interactions with the environment,  has seen substantial success in controlling complex systems <cit.>. However, RL's search for optimal policy involves trial-and-error which can violate safety constraints in critical system applications <cit.>. Despite RL's potential in critical domains <cit.>, it remains a challenge to develop reliable RL-based algorithms for real-world single-life applications, where the control must succeed from the first trial <cit.>. The existing safe RL algorithms either fail to ensure safety during the training phase <cit.> or require significant computational overhead for action verification <cit.>. As a result, traditional control methods are often favored for critical applications, even though their performance is heavily dependent on the availability of accurate environment models.

Here, we address RL's safety issue when an estimated model of the environment can be built to derive a control prior. This scenario is representative of many critical real-life applications <cit.>. Consider an application where drug doses are prescribed to regulate a patient's health status, which is a single-life setting where no harm to the patient is tolerated. An estimated model can be built from other patients' records to predict patient responses and guarantee adherence to safety bounds (set based on clinical knowledge). However, individual variations in new patients' responses, which deviates from the estimated model, pose a significant challenge in control adaptability and patient-treatment performance.

We propose a method, RL with Adaptive Control Regularization (RL-ACR), that simultaneously shows safety and adaptability properties required for critical applications in single-life settings.
The method interacts with the actual environment using two parallel agents. The first (safe) agent is initialized with the estimated model and solves a constrained optimization that hard-codes safety. The second (adaptive) agent is a model-free RL agent which enables adaptability by learning from the actual environment interactions. Our proposed method introduces a focus mechanism that dynamically learns to combine the two agents' policies, allowing the RL agent to explore the environment's dynamics while correcting its effect with the safe agent's policy until the optimum policy is found.

The idea of combining the RL policy with another policy is implemented in <cit.> as well, but with a fixed weight which prevents convergence to the optimum policy. To the best of our knowledge, RL-ACR is the first work that automatically learns to combine a safe policy with the RL policy, allowing for unbiased convergence to the optimum policy. Furthermore, RL-ACR is readily applicable to the environment and ensures safe control even during training, which is another significant benefit that promotes applications of RL in critical systems.

The proposed algorithm is tested under extensive numerical experiments including comparisons with different baseline methods. We demonstrate the performance of the RL-ACR in real-world critical control with an important medical application. Estimated models are widely available for this problem, but the model parameters can differ considerably between patients making individualized adaptation crucial for control performance.


















§ PRELIMINARIES



RL is a data-driven approach that learns from interactions with the environment. Instead of relying on an explicit environment model, RL acts in each time step and observes the corresponding state transition and reward. The objective of an RL agent is to learn a policy that maximizes the cumulative reward, estimated from past observations. Formally, the environment is modeled as a Markov Decision Process (MDP) defined by (𝒮, 𝒜, P, r, γ), where 𝒮 is the observation space, 𝒜 is the action space, P(s, a): 𝒮×𝒜→𝒮 is the state transition probability, r(s, a): 𝒮×𝒜→ℝ is the reward function, and γ∈ (0, 1) is the discount factor that exponentially reduces future rewards to prioritize immediate rewards. The expected future cumulative reward for state-action pair (s, a)∈𝒮×𝒜 is characterized by the Q-function Q^π: 𝒮×𝒜→ℝ, which is defined as follows:

    Q^π(s, a) = 𝔼[ ∑_t=0^∞γ^tr(s_t, a_t) | π, s_0=s, a_0=a],

where π:𝒮→𝒜 is a policy. At any state s∈𝒮, the optimum policy is determined by π^⋆ = _π Q^π(s, π(s)) to maximize the expected cumulative reward. Therefore, the optimality of π depends on an accurate Q-function estimation. A fundamental aspect of this estimation is the application of the Bellman equation <cit.>:

    Q^π(s, a) = r(s, a) + γ𝔼_s'∼ P(s, a)[ Q^π(s', π(s') ],

In modern RL, Q^π and π are respectively approximated with neural networks Q_ϕ(·) and π_θ, where ϕ and θ are the learnable parameters. An approach to stabilize learning is using a Replay Buffer 𝒟, where the experience in the environment is recorded as transitions e_t = (s_t, a_t, s_t+1, r_t) at each time step t and a batch ℬ of transitions from 𝒟 are sampled to update ϕ and θ <cit.>.


In this work, we are interested in taking safe control actions that can differ from the learned RL policy. RL approaches that allow different acting policy and learned policy are referred to as Off-policy RL.  A state-of-the-art off-policy RL algorithm is Soft Actor-Critic (SAC), which uses a stochastic policy to capture environmental uncertainties and prevent sticking to sub-optimal policies. Similar to <cit.>, SAC uses the clipped double Q-learning and maintains two Q-networks Q_ϕ_i(·), i=1, 2. To update the Q-networks, gradient descent is performed using:

    ∇_ϕ_i1/|ℬ|∑_(s, a, s', r, d)∈ℬ (Q_ϕ_i(s, a) - y)^2,     i=1,2

where d∈{0,1} equals 1 if 𝐬' is a terminal state of the environment, and equals 0 otherwise. The objective y in <ref> is given by:

    y = r + γ(1-d)(min_i=1,2Q_ϕ_targ,i(s', a') - αlogπ_θ(a' | s')),     a'∼π_θ(s'),

where logπ_θ(a' | s') is the entropy regularization term introduced by <cit.> to encourage random actions and facilitate exploration. Target Q-networks ϕ_targ, i(·) proposed by <cit.> are used to reduce drastic changes in value estimates and stabilize training. The target Q-network parameters are initialized with ϕ_targ, i = ϕ_i, i=1,2. Each time ϕ_1 and ϕ_2 are updated, ϕ_targ, 1, ϕ_targ, 2 slowly track the update using τ∈(0, 1):

    ϕ_targ,i = τϕ_targ,i + (1-τ) ϕ_i,   i=1,2.

To update the policy network π_θ(·), gradient ascent is performed using:

    ∇_θ1/|ℬ|∑_s∈ℬ(min_i=1,2Q_ϕ,i(s, π_θ(s)) -αlogπ_θ(a|s)).

Note that the policy network relies on Q values estimated by Q_ϕ_i(·) instead of ϕ_targ, i(·) to allow quick policy updates in response to new information.




§ METHODOLOGY




Here, we propose RL-ACR, an algorithm designed for the safe operation of critical dynamical systems. A schematic view of the procedure in RL-ACR is shown in <ref>. RL-ACR is composed of three main modules: The MPC module takes the form of a classic model predictive control (MPC) implementation. The RL module follows an off-policy paradigm, enabling learning from the combined MPC and RL policy, which is different from the RL strategy. The focus module learns a focus weight β that determines how to mix the MPC action a_MPC and RL action a_RL. The three modules function together when interacting with the environment, among them the RL module and the focus module are dynamically updated from historical observations.

The workflow of RL-ACR is outlined as follows: i) the RL module allows stochastic exploration and adaptation to the actual environment, ii) the Function Regularizer (the MPC module) generates a policy with guaranteed safety within a forecasted period by considering hard-coded safety constraints in an optimization problem, and iii) the adaptive action combination adjusts the focus weight β based on observations in the actual environment, allowing gradual switch to the RL policy as the RL module is trained. <ref> shows the pseudocode for updating RL-ACR. The detailed workflow will be explained in the following sections. The details of the RL module follow the SAC <cit.> paradigm introduced in <ref> and is omitted in this section.






 §.§ Function regularizer


RL-ACR regularizes the RL action using an MPC module, which uses an in-built estimated system model to forecast the system behavior under different policies. At any state s_t, the MPC module finds an action a_t that optimizes the N-step system forecast (the horizon), while ensuring safety by formalizing safety measures as constraints. During each step, the following constrained optimization problem is solved in the MPC module:

    min_a_t:t+N-1∑_k=t^t+N-1 J_k(s_k, a_k) + J_N(s_t+N)
       s.t.: 
            s_k+1 = f(s_k, a_k) 
            s_min≤ s_k+1≤ s_max, ∀ k∈{t, …, t+N}
            a_min≤ a_k≤ a_max, ∀ k∈{t, …, t+N-1},

where J_k(s_k, a_k) is the stage cost function at time step k, and J_N(·) is the terminal cost function. The estimated model f(·) characterizes the dynamics of the environment. Solving the optimization problem in <ref> yields a sequence of N actions a_t:t+N-1, with only the first term a_t adopted at time step t, i.e. the MPC policy π_MPC(s_t) = a_t. As the system transits from s_t to s_t+1 as a result of a_t, the optimization (<ref>) is performed again over the new horizon {t+1:t+1+N} to obtain a_t+1. Thereby, MPC iteratively solves for the N-step optimum action in each time step and steers the environment to the desired state.
The optimization in <ref> is efficiently solved using the interior-point filter line-search algorithm <cit.>.







 §.§ Adaptive action combination

The combined policy in the focus module is as follows:

    a(s) = β a_MPC(s) + (1-β) a_RL(s) 
       a_MPC =π_MPC, a_RL(s)=π_θ, β∈ [0, 1].

The weight β is initialized to be 1, prioritizing the MPC's policy when learning begins. As RL gradually improves through interactions with the environment, the agent increases its confidence in the RL policy by updating β to lower values. As proved in <cit.>, the combined policy in <ref> is equivalent to regularizing the RL policy a_RL with the function regularizer a_MPC.


    Assuming the convergence of the RL policy π_θ, the mean of the combined action a̅(s) is the solution to the following regularized optimization with regularization parameter λ=β/1-β:
    
    a̅(s) = _a a(s)-_π_θQ(s, π_θ(s))_Σ + β/1-βa(s)-a_MPC(s)_Σ.




The proof of <ref> is provided in the appendix. During the early stage of training, β is close to one, resulting in strong regularization from the MPC policy. The MPC module ensures safety using two features: 1) hard-coded safe constraints and 2) predictive capability. When a breach of constraint is forecasted, MPC sacrifices high rewards to ensure safety.

To enable updates of networks and the focus weight β, RL-ACR keeps track of the transition e = (s, a, s', r, d, a_MPC) using a replay buffer 𝒟 after each interaction with the environment. Compared to the original replay buffer framework proposed by <cit.> and <cit.>, we store an augmented e with an additional term a_MPC to enable the update of β.



The update of the focus weight β utilizes a batch ℬ of randomly sampled past transitions from 𝒟. The objective is to find the β that generates the combined action a that maximizes the expected return:

    _β∈ [0, 1]𝔼_s∼ p_π[Q^π_θ,β(s, β a_MPC+(1-β) π_θ(s)],

where p_π denotes the state distribution induced by following the policy π. Assume RL converges to the optimum policy π^⋆ that satisfies Q_ϕ(s,π^⋆(s)) > Q_ϕ(s, a), ∀ a∈𝒜∖π^⋆(s), then the optmality in <ref> is achieved by β=0. Because if β≠ 0 then βπ_MPC(s)+(1-β) π_θ(s) = βπ_MPC(s)+(1-β) π^⋆(s) ≠π^⋆(s). The inequality follows because π_MPC≠π^⋆ as the MPC is based on a sub-optimum model.


    Let D_TV(·, ·) denote the total variance distance. The combined policy π has the following bias:
    
    D_TV(π, π^⋆) ≥  D_TV(π_MPC, π^⋆) - (1-β) D_TV(π_MPC, π_θ).

    However, assume RL converges to the optimum policy π^⋆ that satisfies Q^π^⋆(s,π^⋆(s)) > Q^π(s, a), ∀ a∈𝒜∖π^⋆(s), the combined policy π is unbiased with D_TV(π, π^⋆) = 0.



The proof of <ref> is given in the appendix, which shows that by adaptively updating β, the convergence of the combined policy can be achieved with common assumptions.


Practically, learning an optimal β can be seen as a separate update step after updating the Q-networks and the policy network. To estimate Q^π_θ, β(·) in <ref>, we utilize the updated Q-networks of the RL module. To stabilize training, the clipped double-Q learning <cit.> was also used in the β update. Suppose Q_ϕ_i(·), i=1,2, and π_θ(·) are updated by a batch ℬ of past experiences. Then, β is updated by gradient ascent using:

    ∇_β1/|ℬ|∑_(s, a_MPC)∈ℬmin_i=1,2 Q_ϕ_i(s, β a_MPC+(1-β) π_θ(s)),

Note that the updated Q-networks and policy network are used in <ref> to allow quick response to new information.

Although β converges to 0 theoretically, to avoid violation of the action bounds, β is clipped between 0 and 1 in practice. To allow adequate learning of the Q-networks and the policy network, the gradient ∇_β is clipped to the range [-c, c], c>0. With a fixed learning rate lr, the maximum reduction of ∇_β in each step is bounded by c· lr.







§ NUMERICAL EXPERIMENTS



In this section, we validate RL-ACR in the practical setting mentioned in <ref>, where only the estimated models are available to RL-ACR and baselines, but the actual environment models are unknown. To simulate this, we set different model parameters for the estimated models and the actual environment models (see <ref> for details).


The baseline methods used in the experiments are: 
1) Soft Actor-Critic (SAC), a state-of-the-art RL algorithm,
2) Constrained Policy Optimization (CPO), a commonly used risk-aware safe RL benchmark based on the trust region method <cit.>, and
3) Model Predictive Control (MPC), a widely adopted safe control method for critical systems.
Since RL-ACR contains an estimated model through the MPC module, to make the comparison fair, we also include 4) pretrained SAC (SAC-PT) and 5) pretrained CPO (CPO-PT) in the comparison.



SAC's implementation follows Stable-Baselines3 <cit.> and <cit.>.
The MPC implementation is based on do-mpc <cit.>, in which the optimization in <ref> relies on <cit.> and <cit.>.
CPO's implementation follows <cit.>.
The set of hyperparameters used by RL-ACR is detailed in <ref>.












 §.§ Bi-hormonal Glucose Regulation




Regulating the blood glucose level is a critical medical control problem.
The objective is to inject hormones for maintaining the blood glucose level, denoted by G, especially in the face of meal-induced carbohydrate disturbances. Crossing certain safe boundaries of G can lead to catastrophic health consequences (hyperglycemia or hypoglycemia).
Here, a recent blood glucose model proposed by <cit.> and <cit.> is used, which extends the conventional single-action (insulin injection) models by a second action (glucagon injection).
The extended action space leads to potentially better regulation performance but also increases the regulation complexity.
The system has 12 state variables modeled by a set of Ordinary Differential Equations (ODEs):

    𝐱̇ =    [Q̇_1, Q̇_2, ẋ_1, ẋ_2, ẋ_3, Ṡ_1, Ṡ_2, İ, Ż_1, Ż_2, Ṅ, Ẏ]^T 
    
               =    f(𝐱, 𝐚, U_G),

where 𝐚=[a_I, a_N] are the actions, with a_I and a_N being the amount of injected insulin and glucagon, respectively. U_G denotes the disturbance of carbohydrates in a meal. Among the 12 states, only Q_1 is observable through the measurable blood glucose level G since Q_1 = G V_G and V_G is a fixed parameter for glucose distribution volume.
State feedback is applied to Q_1 to maintain close-loop control. See <ref> for the detailed system model in <ref>.

In this environment, a time-varying disturbance U_G ∝ t · e^-c· t derived by <cit.> is introduced over time to simulate the ingestion of 80 g of carbohydrates. The objective is to regulate G to the target range between 3.9 and 7.8 mmol/L (shaded green in <ref>). Failure is defined as G reaching G>25 mmol/L (hyperglycemia) or G<3 mmol/L (hypoglycemia); these unsafe ranges are shaded red in <ref>. The parameters of the estimated model (available to the MPC module) and the actual environment are listed in Appendix <ref>.

The RL reward for this environment is a piece-wise linear function given in <ref> in the Appendix. The MPC setting follows <cit.>, where the stage and terminal costs are given by J_k(𝐬, 𝐚) = J_N(𝐬, 𝐚)=3e^3×(G-6)^2+  8e^4 a_I + 2e^4 a_I a_N + 1e^6 a_N + 5e^7max(0, 6-G). For CPO, the constraint function is c=10max(0, 6-G)^2.









Safety test. We first compare the safety of the methods in regulating blood glucose. The first row of <ref> shows the number of failed episodes (G exceeding [3, 25] bounds) out of 100 consecutive runs while the methods (except for MPC) learn the environment through interactions. It is seen that CPO does not offer any safety guarantee during the initial stage of training. SAC consistently fails during its policy exploration, which is unacceptable in critical applications. This issue persists even if the SAC is trained by the estimated model. Only MPC and RL-ACR guarantee safety in all episodes (SAC, SAC-PT, CPO, and CPO-PT failed in all 100 episodes).


Performance test.
As indicated in <ref>, only MPC and RL-ACR can be safely applied to the actual environment in a single-life setting. CPO, SAC, and SAC-PT violate the safety bounds every run. SAC-PT, being trained on the estimated model, is overall safer compared to SAC and CPO (see other rows of <ref>), so we keep it in the rest of our performance analysis.


<ref> shows the performance comparison in the glucose regulation environment.
SAC-PT causes severe hypoglycemia when deployed to the actual model, but MPC and RL-ACR keep glucose levels within the safe range.
RL-ACR achieves the best performance, clearly improving the MPC policy based on an estimated model.
<ref> shows the MPC action a_MPC and the RL action a_RL.
The combined action β a_MPC+(1-β) a_RL (the green line in <ref>) is the actual policy of RL-ACR.
It can be seen in <ref> that MPC only injects small doses of insulin to avoid the danger of hypoglycemia, which results in poor management of high G level under disturbance. Meanwhile, model-free RL finds a better policy, resulting in better overall performance.
The performance metrics are summarised later in <ref>





 §.§ Classic Control Environments



In this section, we perform experiments on the continuous versions of the four classic control environments: Acrobot, Mountain Car, Pendulum, and Cart Pole <cit.>. Depending on the specific environment, we define failure as either reaching states that violate the safety bounds (for Cart Pole) or failing to reach the control objective within the allocated time frame (for  Acrobot, Mountain Car, and Pendulum).
For more details, such as the MPC cost function and CPO constraint functions, see <ref>.
Notice that, for environments where failure is defined as not achieving the objective in a given time (Acrobot, Mountain Car, and Pendulum), CPO triggers the recovery policy proposal and minimizes the constraint cost first <cit.>.


 The Acrobot environment simulates two links connected by a joint, with one end of the connected links fixed. The links start facing downward. The objective is to swing the free end (the tip) above a given target height as quickly as possible by applying torque to the joint. Failure is defined as the tip not reaching the target height in 200 time steps. The estimated model and the actual environment are detailed in <ref>.

The Mountain Car environment simulates a car placed in the valley of a sinusoidal. The objective is to drive the car to the target location on the sinusoidal by applying force either left or right. Failure is defined as the car not reaching the target location in 1000 time steps. The estimated model and the actual environment are detailed in <ref>.







The Pendulum environment simulates a hanging pole fixed at one end. The pendulum starts facing downward. The objective is to swing the pole upright and keep it in that position.
Failure occurs if the pole does not attain and maintain an upward position (angular velocity not exceeding ±0.1) within 300 time steps.
The estimated model and the actual environment are detailed in <ref>.



The Cart Pole environment simulates a pole attached to a cart that moves along a track. The pole starts with a 6-degree tilt. The objective is to maintain an upright position of the pole by applying force to the cart. Failure is defined as the tilt exceeding the ± 12 degrees angle. The estimated model and the actual environment are detailed in <ref>.


Safety test. We first compare the safety of the methods using the concept of failure defined for each environment. We run the methods for 100 consecutive episodes on each environment. <ref> shows the number of episodes that terminate with failure.
CPO and SAC cause failures during training in all environments.
SAC-PT, even with a strategy trained by the estimated model, still fails due to its stochastic policy exploration in the actual environment.
Only MPC and RL-ACR can guarantee safety in all episodes, as suggested by 0 failed episodes in all environments reported in <ref>.



Performance test. As indicated in <ref>, SAC-PT, being trained on the estimated model, is overall safer compared to SAC and CPO, so we keep it in the performance analysis.

<ref> shows the behavior of SAC-PT, MPC, and our RL-ACR in the classic control environments. 
<ref> (b) shows an instance where the SAC-PT strategy, although fully trained on an estimated model, still has a biased understanding of the car position and fails in the actual environment.
Despite only having an inaccurate model, MPC achieves the control objectives without failures.
Our results suggest that RL-ACR learns similar strategies to MPC (note the similar trajectories in <ref>).
This shows that the off-policy RL module of RL-ACR effectively learns from the sub-optimum MPC policy.
However, the adaptation to the actual environment enabled by model-free RL allows RL-ACR to achieve the control objective faster (see <ref> (a), (b), and (c), and note the significantly lower number of oscillations in the RL-ACR trajectory in <ref> (d)).

<ref> summarizes the performance metrics of SAC-PT, MPC, and RL-ACR on all environments.




As <ref> shows, RL-ACR outperforms MPC in all environments. This demonstrates RL-ACR's ability to adapt to the actual environment and improve the original sub-optimum MPC policy. RL-ACR also achieves lower NCEs in four out of five environments. This further testifies a better policy is found by RL-ACR, as the stronger performance on other metrics does not come at the cost of taking more drastic actions.



















§ CONCLUSIONS AND DISCUSSION



Controlling critical systems, where unsafe control actions have catastrophic consequences, has huge applications from engineering to medicine. We show that appropriate embedding of a control regularization can ensure safety as RL exploration searches for the optimum policy. The proposed method, RL-ACR, dynamically tunes the combination of an RL agent and a control regularizer via a learnable focus weight, which prevents early incorporation of poorly learned RL policy. As revealed in extensive numerical experiments in a single-life setting, the safety of RL-ACR outperforms the baselines. Meanwhile, RL-ACR was able to learn from explorations in the actual environment and find policies with stronger performance.


One limitation of our setting is the assumption that the estimated model has a reasonable accuracy to ensure safety in the actual environment. Although this assumption is common in the safe RL literature, one possible direction for future works is to design algorithms with more flexibility against deviations of the estimated model from the actual environment. A potential approach is to update the estimated model parameters using observations in the actual environment. However, one practical concern is that all parameters are assumed to be unknown and need updating, but only a small number of transitions are observed in the actual environment. In this case, managing controllability, convergence, and safety requires careful design and tuning.




§ RELATED WORKS

Similar to the characterization by <cit.> and <cit.>, we divide existing safe RL works into roughly two categories. The first category does not require knowledge of the system dynamics. These methods often rely on probabilistic approaches <cit.> or constrained MDP <cit.>. In more recent works, learned models are used to filter out unsafe actions <cit.>. However, these methods need to observe failures to estimate the safety cost, thus do not ensure safety during training. Since this work requires safety while training in the actual environment, these methods do not apply.

The second category assumes known system dynamics. Some works use the control barrier function to enforce safety <cit.>. However, these works require a given valid safe set, which is unobtainable when the estimated model and the actual environment are different. Some methods compute a model-based projection to verify the safety of actions <cit.>. However, the scalability of these verification-based methods is a huge problem. Although <cit.> proposed to use neurosymbolic representations to alleviate the computational cost of verification, the computational cost is still high.

<cit.> and <cit.> proposed to use MPC as the policy generator, and use RL to dynamically tune the MPC parameters in the cost functions and the system model. Assuming the discrepancy between the estimated model and the actual environment exists in the form of model parameters, the tuning increases MPC performance. However, this is a strong assumption since there are other discrepancies, such as neglected dynamics, discretization errors, etc. Instead, RL-ACR in this work can theoretically converge to the optimum policy.

It is important to note that, although the MPC policy accelerates the learning of the RL agent, RL-ACR is not a special case of transferring a learned policy. 
The main role of MPC in RL-ACR is to keep the actions safe in the actual environment, instead of transferring knowledge.
Transfer learning in RL studies the effective reuse of knowledge, especially across different tasks <cit.>.  By reusing prior knowledge, transferred RL agents skip the initial random trial-and-error and drastically increase sampling efficiency <cit.>. However, transferred RL agents are not inherently risk-aware, and thus can still steer the actual environment into unsafely. For this reason, transferred RL policies are rarely considered a valid method for ensuring safety. On the other hand, the control regularizer in this work forecasts system behaviors and hard-codes safety into constrained optimizations.


plainnat

















§ APPENDIX









 §.§ Proof of Lemma 1



    Assuming the convergence of the RL policy π_θ, the mean of the combined action a̅(s) is the solution to the following regularized optimization with regularization parameter λ=β/1-β:
    
    a̅(s) = _a a(s)-_π_θQ(s, π_θ(s))_Σ + β/1-βa(s)-a_MPC(s)_Σ.




    For a fixed β, the proof is similar to the proof by <cit.>. Because the RL policy is a Gaussian distributed policy, i.e. π_θ(a|s)∼𝒩(a̅_RL(s), Σ), the combined policy is also a Gaussian distribution: 
    
    a(s)∼𝒩(β a_MPC + (1-β) a̅_RL(s), (1-β)^2Σ).

    The distribution is equivalent to the product of two multivariate Gaussian distributions:
    
    𝒩(β a_MPC + (1-β) a̅_RL(s), (1-β)^2Σ) = 1/1-β𝒩(a_MPC, 1/βΣ)·𝒩(a̅_RL(s), 1/1-βΣ)

    Define a_1-a_2_Σ = (a_1-a_2)^TΣ^-1(a_1-a_2). The probability density function can be written as:
    
    f(a(s)) =    (1-β)^-1(2π)^-k/2β^-k/2|Σ|^1/2exp(-β/2a(s) - a_MPC(s)_Σ)
       × (2π)^-k/2(1-β)^-k/2|Σ|^1/2exp(-1-β/2a(s) - a̅_RL(s)_Σ) 
    
            =    c exp(1-β/2( 
            -a(s) - a̅_RL(s)_Σ
            -β/1-βa(s) - a_MPC(s)_Σ)),
    
            c =   1/(2π)^kβ^k/2(1-β)^(k+2)/2 |Σ|.

    Since β is in range (0, 1), f(a(s)) monotonically decreases. Therefore, the probability of a(s) is maximized when the exponential term is minimized, leading to the following optimization problem:
    
    a̅(s) = _a a(s)-a̅_RL(s)_Σ + β/1-βa(s)-a_MPC(s)_Σ.

    Assuming the RL policy π_θ converges to _π_θQ(s, π_θ(s)), <ref> can be written as:
    
    a̅(s) = _a a(s)-_π_θQ(s, π_θ(s))_Σ + β/1-βa(s)-a_MPC(s)_Σ.











 §.§ Proof of Theorem 1



    Let D_TV(·, ·) denote the total variance distance. The combined policy π has the following bias:
    
    D_TV(π, π^⋆) ≥  D_TV(π_MPC, π^⋆) - (1-β) D_TV(π_MPC, π_θ).

    However, assume RL converges to the optimum policy π^⋆ that satisfies Q^π^⋆(s,π^⋆(s)) > Q^π(s, a), ∀ a∈𝒜∖π^⋆(s), the combined policy π is unbiased with D_TV(π, π^⋆) = 0.



    The proof for the lower bound is similar to the proof <cit.>. First, we expand the total variance distance between the MPC policy and the combined policy as:
    
    D_TV(π_MPC, π)    = sup_(s, a)∈𝒮×𝒜|π_MPC - βπ_MPC - (1-β)π_θ| 
       = (1-β) D_TV(π_MPC, π_θ)

    Following triangle inequality, the total variance distance between the combined policy and the optimal policy is as follows:
    
    D_TV(π, π^⋆)≥ D_TV(π_MPC, π^⋆) - D_TV(π_MPC, π_θ)


    To prove the upper bound on D_TV(π, π^⋆) after convergence, we first examine the convergence of β. According to the assumption, for state s, there exists one optimum policy Q^π^⋆(s,π^⋆(s)) > Q^π(s, a), ∀ a∈𝒜∖π^⋆(s). If RL learns the optimum action (i.e. π_θ(s)=π^⋆(s)), then the optimum in <ref> is achieved by β=0. Because if β≠ 0 then βπ_MPC(s)+(1-β) π_θ(s) = βπ_MPC(s)+(1-β) π^⋆(s) ≠π^⋆(s). The inequality follows as π_MPC≠π^⋆ because the MPC is based on a sub-optimum model. Because β=0, the combined policy has variance equals to π_θ and mean equals to the following according to <ref>:
    
    a̅(s)    = _a a(s)-_π_θQ(s, π_θ(s))_Σ + β/1-βa(s)-a_MPC(s)_Σ
       = _a a(s)-_π_θQ(s, π_θ(s))_Σ = a̅_RL.

    Thus the the combined policy π is unbiased with D_TV(π, π^⋆) = D_TV(π_θ, π^⋆) = 0







 §.§ Hyperparameters

The same set of parameters was utilized in all environments to demonstrate the robustness of RL-ACR against the selection of hyperparameters.
The hyperparameters in RL-ACR are given in <ref>.








 §.§ Estimated models and the actual environment models





  §.§.§ Bi-hormonla Glucose Model


The blood glucose model used in this work contains 12 state variables regulated by the ODEs given below:

    Q̇_1    = -F_01^c(G) - x_1Q_1 + k_12Q_2 - F_R  
       + (1-x_3)EGP_0 +c_convU_G + YQ_1
    Q̇_2    = x_1Q_1 - (k_12+x_2)Q_2
    ẋ_1    = -k_a1x_1 + k_b1I
    ẋ_2    = -k_a2x_2 + k_b2I
    ẋ_3    = -k_a3x_3 + k_b3I
    Ṡ_1    = u_I - S_1/t_max, I
    Ṡ_2    = S_1/t_max, I - S_2/t_max, I
    İ   = S_2/V_I t_max, I - k_e I
    Ż_1    = u_N - Z_1/t_max, N
    Ż_2    = Z_1/t_max, N - Z_2/t_max, N
    Ṅ   = -k_N(N - N_b) + Z_2/V_N t_max,N
    Ẏ   = -pY + pS_N(N - N_b),


The last four equations form the glucagon subsystem proposed by <cit.> and <cit.>.
These newly proposed dynamics allow an expanded action space.
While conventional blood glucose models have one action, insulin injection <cit.>, the glucose model in this work has two actions u_I and u_N, representing the injection of insulin and glucagon. 
The measurable blood glucose mass G=Q_1/V_G is lowered by insulin injection and raised by glucagon injection. The intermediate variables depend on the value of G, as shown below:

    F^c_01 = F_01,     G≥4.5mmol L^-1
    F_01 G/4.5,    otherwise, 
       F_R = 0.003(G-9)V_G,     G≥9mmol L^-1
    0,    otherwise.


The term U_G represents the time-varying disturbance caused by the injection of D_G of carbohydrates:

    U_G = D_G A_G/t_max, G^2· t · e^-t/t_max, G,

the model parameters adopted by the estimated model and the actual environment are given by <ref>. The reward function for blood glucose regulation is a piece-wise linear function given below:

    r = 
        
        -400× (6-G),     G<6
    
        0,     6≤ G< 7.8
    
        -40/7.2, × (G-7.8)     7.8≤ G< 20
    
        -300,     7.8≤ G <25
    
        -2000,     G≥ 25
        .







  §.§.§ Acrobot Implementation



In this work, the system model utilized by MPC and RL-ACR for the Acrobot environment is defined as:

    θ̈_1    = -(d_2θ̈_̈1̈+ϕ_1)/d_1
    θ̈_2    = a + d_2ϕ_1/d_1 - m_2 l_1 l_2 c θ̇_1^2 sinθ_2 - ϕ_2/m_2 l_2^2 c^2 + I_2 - d_2^2/d_1,

where a is the action, the default model parameters are given in <ref>, and the intermediate variables are:

    d_1    = m_1l_1^2c^2 + m_2 (l_1^2+l_2^2c^2+2l_1l_2ccosθ_2) + I_1 + I_2 
    
    d_2    = m_2 (l_2^2 c^2 + l_1 l_2 ccosθ_2) + I_2 
    ϕ_2    = m_2 l_2 c g cos(θ_1 + θ_2 - π /2) 
    ϕ_1    =  - m_2 l_1 l_2 cθ̇_2^2  sinθ_2 - 2m_2 l_1 l_2 cθ̇_1 θ̇_2 sinθ_2
              + (m_1 l_1 c + m_2 l_1)gcos(θ_1 - π / 2) + ϕ_2.


For this particular environment, we utilized a custom RL reward function. This allows SAC and CPO to achieve the control objective within 100 episodes, thus benefiting the comparison in <ref>. The reward function is:

    r = 
        
        -2 - l_1 cosθ_1 -l_2 cosθ(θ_1 + θ_2),     - l_1 cosθ_1 -l_2 cosθ(θ_1 + θ_2)<l_2
    
        +100,    otherwise
    .









  §.§.§ Mountain Car Implementation



In this work, the system model utilized by MPC and RL-ACR for the Mountain Car environment is defined as:

    x'    = x + v 
    
    v'    = v + ac_f - 0.0025 cos (3(x+x_0)),

where a is the action, the default model parameters are given in <ref>.








  §.§.§ Pendulum Implementation



In this work, the system model utilized by MPC and RL-ACR for the Pendulum environment is defined as:

    θ'          = θ + θ̇Δ t 
    θ̇'    = θ̇' + (3g/2lsinθ + 3/ml^2a)Δ t,

where a is the action, the default model parameters are given in <ref>.








  §.§.§ Cart Pole Implementation



In this work, the system model utilized by MPC and RL-ACR for the Acrobot environment is defined as:

    θ̈   = gsinθ - dcosθ/l(4/3 - m_pcos^2θ/(m_p + m_c))
    ẍ   = d - m_p l θ̈cosθ/m_p + m_c,

where a is the action, the default model parameters are given in <ref>, and the intermediate variable d is:

    d = a c_f + m_p l θ̇^2 sinθ/m_p + m_c










 §.§ MPC Costs and CPO constraints for the Classic Control Environments


Here we illustrate the CPO constraints and the MPC cost functions in the classic control environments in <ref>.

For the Acrobot environment, the CPO constraint function we utilized was c = 1 + cosθ_1 + cos(θ_1 + θ_2). The MPC costs we utilized are:

    J_k(𝐬, 𝐚) = J_N(𝐬, 𝐚) = 100 × (2 + cosθ_1 + cos(θ_1 + θ_2))^2.



For the Mountain Car environment, the CPO constraint function we utilized was c = max(0, 0.45-x)^2. The MPC costs we utilized are:

    J_k(𝐬, 𝐚) = J_N(𝐬, 𝐚) = max(0, 0.45-x)^2.


For the Pendulum environment, the CPO constraint function we utilized was c = 2max(0, |θ|-4)^2. The MPC costs we utilized are:

    J_k(𝐬, 𝐚)    = 10θ^2 + 0.1θ̇^2
    
        J_N(𝐬, 𝐚)    = 80θ^2 + 0.1θ̇^2.


For the Cart Pole environment, the CPO constraint function we utilized was c = max(0, |θ|-6)^2. The MPC costs we utilized are:

    J_k(𝐬, 𝐚) = J_N(𝐬, 𝐚) = 100x^2+100θ^2.



