Reinforcement Learning via Lazy-Agent for Environments with Random Delays
Delayed system;Markov decision process;reinforcement learning
Yes
No ethics review needed.
* What is the performance of the approach in the other MuJoCo environments?
Soundness
======
 The approach is sound, and, albeit a small increment over existing approaches, well demonstrated empirically. The paper lacks an in-depth discussion of the benefits and limitations of the approach. 
	
Significance & Related work
=========
There is no related work section in the paper and thus the significance of the work is not clarified.  

Experimentation
=========
The experimental analysis shows the equivalence of RDMDPs to CDMDPs in BPQL in the MuJoCo environments. 


Presentation
=========
The paper is well written.
This paper presents a derivation and a small-scale empirical analysis that shows random-delay MDPs can be transformed to nearly equivalent constant-delay MDPs by employing a lazy-agent that assumes that all states are (constantly) delayed by the maximum delayed times.
Soundness
======
The paper lacks an in-depth discussion of the benefits and limitations of the approach. 
	
Significance & Related work
=========
There is no related work section in the paper and thus the significance of the work is not clarified.  

Experimentation
=========

There is a need, however, for more in-depth ablation experiments that evaluate the impact of assuming states are processed in order, and of highly varied random delays.
Yes
No ethics review needed.
1. Unsure why you need to wait for $t>2_{o_{max}}$ in line 19 of Algorithm 1, instead of just $t>o_{max+2}$, if the purpose is just to wait until you have the next state/augmented state?
2. In section 3.2 it is stated that the state dimension could become infinite in infinite-horizon MDPs, wouldn’t this only occur if $O_{max}$ was infinite? 
3. Could you provide more details on how BPQL mitigates the state space explosion problem, particularly given that the policy is still trained on the augmented state?
1. Nice visual representations e.g. Figure 1 help with understanding
2. The paper proposes a straightforward way to apply an algorithm for constant-delay environments to environments with random delays
3. Convincing number of environments and comparison algorithms used in experiments
This paper proposes a way to transform random delay environments into constant delay environments, by imposing the specified maximum delay onto all observed states, regardless of their actual delay. Experiments show that agents trained under this transformation achieve similar results to agents trained in constant delay environments.
1. Still requires a maximum number of delay time steps to be specified, which may be unrealistic in some environments (e.g. if observations must be sent over a high-latency network, if the observations require lengthy processing). This would admittedly be a challenging setting to address.
2. Only addresses the issue of large augmented state dimensions if the specified max delay is small.
3. I found the explanation of BPQL a bit confusing, as I don’t see how it avoids state space explosion if you are still training a policy on the augmented state.
Yes
No ethics review needed.
See weaknesses
- The authors demonstrate that using Lazy-Agent can convert random-delay Markov decision processes (RDMDPs) into constant-delay MDPs (CDMDPs). This transformation allows established constant-delay solutions to be applied to random-delay environments.
- The proposed method is simple and easy to implement.
The paper introduces Lazy-Agent for handling reinforcement learning in environments with random delays.
Traditionally, state-of-the-art RL techniques assume constant delays, simplifying the learning process but failing to reflect real-world scenarios' complexity.
A Lazy-Agent operates under the assumption that all feedback, such as states or rewards, is delayed by the maximum possible time, even though the actual delay is random.
This method allows the transformation of random-delay environments into constant-delay ones, making it possible to apply conventional RL techniques designed for constant delays.
Rather than reacting immediately, the Lazy-Agent waits for the maximum delay before making decisions.
This simplifies the management of random and unpredictable delays by treating them as constant delays, thereby facilitating the use of existing reinforcement learning methods for delayed feedback.
The paper extends the belief projection-based Q-learning (BPQL) framework, introducing Lazy-BPQL.
- For certain tasks (such as Ant, Hopper, Humanoid, and Pendulum), the performance gains from using Lazy-BPQL are minimal.
The reported improvements in performance, when considering standard deviation, are negligible.
- There is no significant advantage of Lazy Agents over traditional methods in all benchmarks, which raises concerns about the broader applicability of the Lazy Agent model.
We introduce a simple yet effective reinforcement learning agent, referred to as the lazy-agent, which can be applied in environments with random delays.
@inproceedings{
anonymous2024reinforcement,
title={Reinforcement Learning via Lazy-Agent for Environments with Random Delays},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=KRlJN9d1sW},
note={under review}
}
Real-world reinforcement learning applications are often hampered by delayed feedback from environments, which violates the fundamental assumption of the Markovian property and introduces significant challenges. While numerous methods have been proposed for handling environments with constant delays, those with random delays remain largely unexplored owing to their inherent complexity and variability. In this study, we explored environments with random delays and demonstrated that these environments can be transformed into equivalent constant-delay environments by introducing a simple agent called the \textit{lazy-agent}. This approach overcame the challenges posed by the variability of random delays, enabling the application of conventional constant-delay approaches to random-delay environments. The empirical results reveal that our lazy-agents trained in random-delay environments performed almost comparably to the agents trained in constant-delay environments, significantly outperforming other baseline algorithms in terms of asymptotic performance and sample efficiency.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Delayed system Markov decision process reinforcement learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/1c31295411de60fadd3498b8546d2d22f6cce217.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/481ef2537baf8d7b4492ecf6d7ab1987adea3f4f.zip
Reinforcement Learning via Lazy-Agent for Environments with Random Delays
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission