Runtime Learning Machine
Runtime Learning;Deep Reinforcement Learning;Safety;Unknown Unknown;Autonomous Systems
Wait
Response to Reviewer mpnQ: Part II
**Requirement 3: Attracting Toward Safety Envelope.** The center of the HA-Teacher's patch must be located within the safety envelope. If it is not, the system's state may become trapped within the patch. This situation can cause the HA-Teacher to dominate the machine during runtime learning, preventing the HP-Student (the DRL agent) from independently learning a high-performance action policy.

In the paper, Equations (16), (18), (23), and (28) to (31) detail the design of HA-Teacher, which is formally stated in Theorem 6.7. The resulting properties are outlined in Items 1-3 of Theorem 6.7. Meanwhile, Items 1-3 mean that Requirements 1, 3, and 2 are, respectively, satisfied. The theoretical proof and Equations (18), (23), and (28) to (31) are pretty lengthy, so we have moved them to Appendix D due to page limitations. This may have led to the reviewer’s impression that the design of HA-Teacher lacks formal rigor. To address this concern, we have highlighted sentences on page 7 and in Remark 6.9 in teal color, clearly stating that the design of HA-Teacher simultaneously and strictly satisfies the proposed Requirements 1-3.

-----

**W3: While the selected experimental domains are relevant, further discussion on generalizability to applications like autonomous driving or complex industrial robots would broaden the framework’s impact.**

**Answer:**
Response to Reviewer mpnQ: Part I (Continued)
**W1: The design details for the teacher, a core element of this framework, could be expanded, particularly in scenarios where unknowns extend beyond simple disturbances to include hardware or sensor faults.**

**Answer:** We appreciate the valuable suggestion from the reviewer. Our system's sensors generate two types of data: samplings of the system state (including velocities, positions, roll, yaw, pitch, etc.) and samplings of control commands. In our experiments with the cart-pole system and the real A1 quadruped robot, we introduced faults in the controller sensors by injecting unknown-unknown noise. This noise was generated according to a randomized Beta distribution, explained in detail in Appendix F of the paper as one method for representing unknown unknowns. Initially, we did not intentionally induce faults in the system-state sensors because the sensor readings are inherently noisy. Besides, the real robots' center of mass (CoM) positions are estimated using the Kalman filter that relies on velocity samplings, making estimation errors unavoidable.

Also, in response to Reviewer Z8i5's comments, we have added two additional benchmarks: 1) the locomotion of the Go2 quadruped robot in dynamic, uneven, and unstructured environments, and 2) waypoint tracking for the 2D Quadrotor. In these new experiments, we introduced system-state sensor faults by injecting unknown-unknown noise. Due to page limitations, these new benchmarks are presented in Appendix I in the revised paper (**will be available in 3 days**). Additionally, the demonstration video showcasing the runtime learning system in the presence of sensor faults for both the system state and the controller will be available at [Go2 Demo Video (anonymous link)](...) (**will be available in 3 days**).

Our unknowns in the experiment with the real A1 quadruped robot go beyond simple disturbances, which include:
1. **Beta**: Disturbances injected into HP-Student's action/control commands.
2. **PD**: Random and sudden payload (around 4 lbs) drops on the robot's back. 
3. **Kick**: Random and sudden kick by a human.
4. **DoS**: A real denial-of-service fault of the platform, which can be caused by task scheduling latency, communication delay, communication block, etc., is unknown to us. 
5. **SP**:  A sudden side push. 

To be more convincing, we consider three combinations of these unknown unknowns applied to the runtime learning stage: i) **Beta + PD**, ii) **Beta + DoS + Kick**, and iii) **Beta + SP**. The demonstration video is available at [A1 Robot: unknown unknowns (anonymous link)](https://www.dropbox.com/scl/fi/jjxlym8xtdxqgr7d5hclp/unk.mp4?rlkey=kh76ojwhspqgslzolmowuuu0l&st=igbjnji3&dl=0). To clarify this critical information, we highlighted the five different unknowns in bold black font in the revised paper.

-----

**W2: Additionally, the theoretical justification for the teacher’s resilience against unknown unknowns could be strengthened, as the current analysis lacks formal rigor or quantitative measures.**

**Answer:** We appreciate the reviewer’s insightful question. A rigorous and verifiable design is essential for HA-Teacher in our runtime learning machine. One of the primary motivations for this is the Sim2Real gap, as prevalent DRL typically involves training within a simulator. Additionally, there are challenges posed by unknown unknowns, the unpredictability of DNNs, and the difficulty in verifying their functions due to the vast parameter space and nonlinear activations. These make it nearly impossible to enable DRL with verified safety in safety-critical systems that operate in time-sensitive physical environments. These challenges inspire us to develop HA-Teacher, a verified yet simplified design focused exclusively on safety-critical functions. As a complement to DRL, HA-Teacher has two main objectives: i) to correct any unsafe learning conducted by HP-Student (the DRL agent), and ii) to back up the safety of the real plants. We conclude that the design of the HA-Teacher must be formal and rigorous regarding safety assurance. The actions of HA-Teacher must have verifiable safety, with rigorous theoretical foundations and proof. To achieve this, we propose three essential requirements (outlined in Requirements 6.2-6.4 of the paper) that the designed HA-Teacher must simultaneously fulfill. If these requirements are not met, we cannot trust HA-Teacher to correct unsafe learning or ensure safety effectively.  We will now summarize these three requirements.

**Requirement 1: Conformity with Safety Regulations.** It requires that the HA-Teacher's real-time patches be subsets of the safety set. Otherwise, the HA-Teacher will not be able to ensure safety.

**Requirement 2: Conformity with Operation Regulations.** It is necessary to confine the real-time actions of HA-Teacher within a physically-feasible bounded action space.
Response to Reviewer mpnQ: Part I
Yes
No ethics review needed.
Could you provide more insights into the teacher's handling of complex unknowns, such as hardware failures or significant sensor errors?

What theoretical guarantees exist for the teacher's performance in completely unanticipated scenarios?

How might this system perform in applications outside of the tested environments, such as in higher dimensional or more complex environments?
The paper presents an innovative hierarchical safety mechanism where the HA-Teacher manages safety while the HP-Student prioritizes performance optimization within safety constraints. This interactive approach is practical and well-suited to the paper’s focus on real-world, safety-critical applications. The experimental setup, involving both a cart-pole system and a quadruped robot, is comprehensive and shows the method’s robustness across different unknowns. The results suggest that the system can maintain stability through real-time patching by the HA-Teacher, showing practical value in environments where safety and performance trade-offs are essential. Additionally, the framework addresses the Sim2Real gap, which remains a significant challenge in deploying DRL systems in physical settings.
This paper introduces a "Runtime Learning Machine" designed for autonomous systems operating in safety-critical environments. The machine combines three main components: an HP-Student (high-performance learner), an HA-Teacher (high-assurance safety monitor), and a Coordinator. The HP-Student learns in real-time from physical environments, with the HA-Teacher correcting unsafe actions and enforcing safety. This design aims to address critical challenges in deep reinforcement learning, including tolerating unknown unknowns and bridging the Sim2Real gap.
The design details for the teacher, a core element of this framework, could be expanded, particularly in scenarios where unknowns extend beyond simple disturbances to include hardware or sensor faults. Additionally, the theoretical justification for the teacher’s resilience against unknown unknowns could be strengthened, as the current analysis lacks formal rigor or quantitative measures. While the selected experimental domains are relevant, further discussion on generalizability to applications like autonomous driving or complex industrial robots would broaden the framework’s impact. Although comparisons with other DRL methods are made, the analysis could be enhanced by discussing specific scenarios where other methods may outperform this framework or identifying potential limitations.
Yes
/
No ethics review needed.
1. How often does the teacher’s intervention occur during runtime, and is there an impact on the agent's performance with frequent switches?
2. If the teacher is adjusted or modified during runtime, how is its performance or reliability guaranteed in changing conditions?
1. The paper addresses a important safety issue in learning-based systems, particularly focusing on challenges like unknown unknowns and Sim2Real gap.
2. The paper presents solid theorem.
3. The evalution is performed on a real robot, which exhibits the practicality of this work.
The paper presents a novel approach to runtime learning for agents, guided by a verified teacher during runtime to ensure safe operation. This teacher-student framework leverages physics-regulated deep reinforcement learning (PhyRL) and aims to address safety concerns arising from unknown variables and Sim2Real gaps. The validation of the approach is conducted on two benchmarks, including a real quadruped robot, which adds to the practical value of this research.
Though the results look impressive, I found some parts of the paper needs further clarification and a few more discussion about related works are expected.
1. The paper lacks details on the verifiability of the PhyRL structure, especially regarding how the teacher's safety is guaranteed in unknown environments. In Equation 2, the system safety is only related to its state. Does this mean that this paper assumes such simple constraint and not consider how the environment change impacts on the states?
2. The paper should discuss whether switching back to the teacher affects operational performance, as such switches may potentially degrade it. Additionally, the evaluation would benefit from presenting metrics on how frequently the teacher’s guidance is triggered.
3. The work currently assumes a single-step safety model, where each action ensures safety only for the immediate next state. However, a broader perspective on end-to-end safety—considering the effects over several steps or even entire trajectories—is the cases that usually happen in reality.
4. There are lots of verified learning works that are not cited/discussed by this paper.  For example, Neurosymbolic reinforcement learning with formally verified exploration. Neurips 2020.
5. There are only two benchmarks presented. Though one of them is the real robot, it would be necessary to evaluate on a more diverse set of benchmarks.
Yes
No ethics review needed.
1. How is this approach placed in the context of other shielding approaches [1,2]?
2. The triggering condition in Eq 7 is entirely causal (i.e., depends only on the current time and prior to the current time). Why is this better than using the environment model to predict for a future failure from the current state and input?
3. How does the approach generalize to different safety sets (e.g., velocity < 1.5 vs velocity < 3)? Will the entire training process be needed for each change? How long would the training time be for each minor change?
4. Are there any experiments or examples with System Identification as mentioned in Remark 6.5 or Appendix C?

### References:
[1] Safe Reinforcement Learning with Nonlinear Dynamics via Model Predictive Shielding, Bastani, ACC 2020

[2] Dynamic Model Predictive Shielding for Provably Safe Reinforcement Learning, Banerjee et al, arXiv 2024
- The proposed method combines a High Assurance Teacher with a Performance optimizing student to yield strong real word robustness results on a challenging quadruped robot.
- The experiments show scalable computation that can work fast enough for real-time learning using matrix toolboxes and related libraries.
This paper introduces a runtime learning machine for safety-critical autonomous systems, featuring an interactive HP-Student, HA-Teacher, and Coordinator. It ensures lifetime safety by addressing unknowns and the Sim2Real gap, enabling real-time learning for safe, high-performance policies. Experimental results validate its effectiveness on cart-pole and quadruped robot systems.
- The setting considered assumes knowledge of the environment dynamics by the HA-Teacher. System identification is proposed but does not appear to have been experimented with.
- The presentation is cluttered at times and causes some difficulty in parsing. For example, some of the numerous assumptions, remarks, characteristics, and definitions should be consolidated if possible. Section 1.3 is almost entirely italicized.
Runtime Learning Machine for Safety-critical Autonomous Systems
@inproceedings{
anonymous2024runtime,
title={Runtime Learning Machine},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=KCTHM2Ffh3},
note={under review}
}
This paper proposes the Runtime Learning Machine for safety-critical autonomous systems. The learning machine has three interactive components: a high-performance (HP)-Student, a high-assurance (HA)-Teacher, and a Coordinator. The HP-Student is a high-performance but not fully verified Phy-DRL (physics-regulated deep reinforcement learning) agent that performs safe runtime learning in real plants, using real-time sensor data from real-time physical environments. On the other hand, HA-Teacher is a verified but simplified design, focusing on safety-critical functions. As a complementary, HA-Teacher's novelty lies in real-time patch for two missions: i) correcting unsafe learning of HP-Student, and ii) backing up safety. The Coordinator manages the interaction between HP-Student and HA-Teacher. Powered by the three interactive components, the runtime learning machine notably features i) assuring lifetime safety (i.e., safety guarantee in any runtime learning stage, regardless of HP-Student's success), ii) tolerating unknown unknowns, iii) addressing Sim2Real gap, and iv) automatic hierarchy learning (i.e., safety-first learning, and then high-performance learning). Experimental results involving a cart-pole system and a real quadruped robot, as well as comparisons with state-of-the-art safe DRL, fault-tolerant DRL, and approaches for addressing Sim2Real gap, demonstrate the machine's effectiveness and unique features.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Runtime Learning Deep Reinforcement Learning Safety Unknown Unknown Autonomous Systems
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/7d414ad9856f4def76ffa62616267c319b385435.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/45a2d2e6e45d2414e8a2e9c49d266095a00154cf.zip
Runtime Learning Machine
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission