Time to Truncate Trajectory: Stochastic Retrace for Multi-step Off-policy Reinforcement Learning
multi-step off-policy reinforcement learning
I have read and agree with the venue's withdrawal policy on behalf of myself and my co-authors.
@misc{
han2024time,
title={Time to Truncate Trajectory: Stochastic Retrace for Multi-step Off-policy Reinforcement Learning},
author={Seungyub Han and Taehyun Cho and Dohyeong Kim and Kyungjae Lee and Jungwoo Lee},
year={2024},
url={https://openreview.net/forum?id=aZwKz5Tg3m}
}
While off-policy reinforcement learning methods that is based on one-step temporal difference learning have shown to be promising for solving complex decision making problems, multi-step lookahead from behavior policies is still challenging by the discrepancy between behavior policy and target policy.
Several recent works have addressed this challenge by introducing the coefficients to correct the discrepancy such as Retrace and evolving the behavior policy in a similar way to conservative policy iteration such as Peng's $Q(\lambda)$.
However, both methods do not universally work well by the policy evaluation error caused by the value from the later part of a long trajectory data.
In this work, we propose a stochastic truncation method which modify the correction coefficeints of Retrace into the sequence of Bernoulli random variables to remove the later part of trajectory, which degrades off-policy evaluation by adding the unnecessary noise.
Unlike prior method for reducing the off-policy discrepancy,
our stochastic truncation enjoys two strengths form the conservative and non-conservative multi-step RL methods.
We demonstrate that our algorithm, time-to-truncate-trajectory (T4) outperforms various model-free RL methods.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
~Seungyub_Han1 ~Taehyun_Cho1 ~Dohyeong_Kim1 ~Kyungjae_Lee1 ~Jungwoo_Lee1
Seungyub Han Taehyun Cho Dohyeong Kim Kyungjae Lee Jungwoo Lee
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
multi-step off-policy reinforcement learning
Yes, at the sentence level (e.g., fixing grammar, re-wording sentences)
I certify that there is no acknowledgement section in this submission for double blind review.
han|time_to_truncate_trajectory_stochastic_retrace_for_multistep_offpolicy_reinforcement_learning
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
No
Yes
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Time to Truncate Trajectory: Stochastic Retrace for Multi-step Off-policy Reinforcement Learning
ICLR 2025 Conference Withdrawn Submission
ICLR.cc/2025/Conference/Withdrawn_Submission