HP3O: Hybrid-Policy Proximal Policy Optimization with Best Trajectory
Reinforcement learning;proximal policy optimization;hybrid policy;trajectory replay buffer;variance reduction
Yes
No ethics review needed.
- See my comments in the weaknesses.
- How to define or measure the best trajectory in the replay buffer?
- How to implement the proposed replay buffer (with FIFO) but withoug padding? How much memory it would consume?
- Why the variance of HalfCheetach becomes larger when training goes longer?
- How do you calculate the explained variance? what are the inputs and groundtruth?
- High variance in on-policy policy gradient methods, such as PPO, remains a challenge. This work aims to mitigate this issue by proposing a trajectory replay buffer with FIFO stategy so that both on-policy and off-policy experience can be utilized to balance the trade-off between sample efficiency and training stability.  
- Both theoretical analysis and empirical evaluation on continuous control environments are provided to show its effectiveness.
This paper focuses on the issues of high variance and high sample complexity in proximal policy optimization (PPO) algorithm. To mitigate the issues, an approach of Hybrid-Policy Proximal Policy Optimization (HP3O) is proposed by utilizing a trajectory replay buffer with a "first in, first out" (FIFO) strategy. Further, another variant termed HP3O+ is proposed by leveraging a new baseline to enhance the model performance. Theoretical analysis is also provided to show the policy improvement lower bounds for the proposed algorithms. Finally, experiments were conducted on continuous control environments from Gymnasium (Walker, HalfCheetah, Lunarlander, Swimmer, Hopper, and InvertedPendulum), comparing HP3O/HP3O+ with PPO, A2C, and P3O.
- The contribution of this paper looks a little bit incremental to me. The idea of using FIFO strategy for replay buffer is quite straightforward, this is OK, but I don't see why and how this will make a big difference. It's more interesting to compare and analyze FIFO with other strategies, such as prioritized replay buffer, and show why FIFO could work well or much better than the other ones. From the theoretical perspective, what's the improvement of sample complexity over other policy gradient methods? From the empirical evaluation, the current results are insufficient to support the claim. Although the paper acknowledged that fully off-policy method, such as SAC, could work better, it's still necessary to compare the proposed methods with both the state-of-the-art methods and off-policy methods (TD3, SAC, etc.). Currently, HP3O showed the advantage over PPO, A2C, P3O on some domains, but why it can make success and why it fails, it's not clear yet. For the ablation study, it's more expected to see the proposed method with or without FIFO replay buffer and best trajectory.
- There are some missing cross-references, such as "Appendix ?? and ??".
Yes
No ethics review needed.
- If the best trajectory is not added to the replay buffer, which part of the theoretical results no longer holds?
- In Theorem 2, there’s a factor of 2 in the last two terms on the right-hand side. Should these factors be canceled out instead?
- the proposed approach combines the strengths of on-policy and off-policy methods
- offers theoretical insights
- validated the efficacy in continuous control experiments
This paper proposes Hybrid-Policy Proximal Policy Optimization (HP3O) and a variant HP3O+, which incorporate a replay buffer with standard PPO, to improve the sample efficiency while mitigating the high variance common in off-policy algorithms. They derived the policy improvement lower bound for both HP3O and HP3O+. Experiments conducted on continuous control environments demonstrate that the proposed algorithms perform comparably or better than baselines, including PPO, A2C [Peng et al. 2018] and P3O [Chen et al. 2023].
1. The baselines in the experiments are limited: six baseline methods are listed in Table 1 as related works but only one of them is used as a baseline in the experiments. The paper explains that other methods were excluded because “corresponding code bases are either inaccessible or problematic”. However,  at least three of these methods have publicly available codebases, including the most relevant work GePPO.  It would be beneficial to clarify the specific issues that prevents their inclusion:

  - GePPO (official): https://github.com/jqueeney/geppo
  - Policy-on-off PPO (official): https://github.com/rasoolfa/P3O
  - Off-policy PPO (unofficial): https://github.com/Ladun/OffPolicy-PPO

    Given that GePPO and Off-policy PPO are both hybrid methods with theoretical guarantees,  comparing with them would better illustrate the benefits of using a trajectory buffer.

2. Since HP3O+ is a proposed algorithm, its learning curve should ideally be included in Fig. 2. It would strengthen the paper if the authors add the results or elaborate why they’re unnecessary.

3. It seems that the learning curves only report performance of the training episodes. If so, I think that it’s better to report the evaluation performance instead, as is done in SAC.

4. In the limitation section, it is mentioned that dequeuing replay buffer may cause issues in sparse reward setting. On a related note, conducting ablation studies on the buffer size in general settings would provide insights into its impact on performance and stability.

Minor:
- For future work, it would be useful to explore the performance of HP3O in discrete action spaces, such as Atari, as done in PPO.
- Typo on line 282, “the the”
Yes
No ethics review needed.
1. In Algorithm 1 Line 16, should $G_t−V_\phi(s_t)$ be replaced by $\hat{A}_t^{\pi_k}$ to be consistent with HP3O+?
2. Could you provide the final, converged reward for each algorithm in the experiments to better demonstrate their performance?
3. Could you include a discussion on the impact of replay buffer size, either through theoretical analysis or additional simulation results?
1. The hybrid-policy approach of HP3O is an innovative extension of PPO, blending on-policy stability with off-policy sample efficiency by storing recent trajectories in a replay buffer and selectively sampling high-reward experiences.
2. The paper is rigorous both in theory and empirical validation. It presents a clear theoretical framework, including policy improvement bounds that account for the hybrid on-policy/off-policy nature of HP3O.
3. The paper is well-structured and provides detailed explanations of key concepts.
This paper introduces Hybrid-Policy PPO (HP3O), a variant of Proximal Policy Optimization (PPO) designed to improve sample efficiency and reduce high variance. HP3O integrates on-policy and off-policy learning through a replay buffer that stores recent trajectories, using a “first in, first out” (FIFO) strategy to retain only the most recent experiences. By incorporating the trajectory with the highest return along with randomly sampled trajectories from the buffer for policy updates, HP3O empirically reduces variance while preserving a policy improvement guarantee. The authors provide rigorous proofs for the policy improvement lower bounds of both HP3O and its variant, HP3O+. Empirical evaluations across continuous control environments demonstrate the effectiveness of HP3O compared to other baselines.
1. While the paper incorporates the replay buffer concept from off-policy reinforcement learning into PPO, the scope of this structure modification shows limited originality.
2. The training curves in Figures 1 and 2 do not appear to have fully converged, making it difficult to assess the ultimate performance that each model could achieve.
3. The impact of the replay buffer size has not been discussed theoretically or evaluated empirically, leaving an important aspect of the method unexplored.
Yes
No ethics review needed.
1. Does the experiment utilize parallel gymnasium environment settings, or is each method trained with single environment? In practical applications, parallel environments are widely used such as ISAAC simulator with thousands  of robots trained simultaneously, which reduce training times with higher computation costs. Does this method still achieve similar higher performance than other methods? 
2. In this paper, the author claims that ``A notable disadvantage of SAC is that it only works with continuous action spaces. " However, SAC can be utilized in discrete action spaces [1]. In this paper, HP3O(+) is implemented in continuous action spaces. Is this method effective in discrete action space?
3. The experiments are conducted in several Mujoco continuous control tasks. Is this method effective in locomotion tasks with higher action spaces, such as Humanoid-v2 task? 

[1] Christodoulou P. Soft actor-critic for discrete action settings[J]. arXiv preprint arXiv:1910.07207, 2019.
1. The new method HP3O(+) is simple and easy to implement. The effectiveness of HP3O policy improvement is analyzed theoretically in detail. 
2. The PPO is widely used in applications. Thus, this method can be utilized to improve the performance of downstream tasks such as robotics control tasks.
3. The paper is well-written with clear descriptions of the method and the experiments.
This paper proposes a new hybrid-policy RL method called HP3O for continuous control tasks, which employs a trajectory replay buffer to reduce variance  and improve sample efficiency. HP3O is simple but effective with theoretical guarantees on policy improvement. The experiments on Mujoco control tasks demonstrate its effectiveness empirically. This paper is well written and the method is clearly described.
The main concerns lie in the experiments:

1. More experiment results with (more) baseline methods are expected. In this paper, authors compare HP3O(+) with PPO, A2C, P3O, and SAC. However, some results are not shown in the current version: 

   1. The learning curves of SAC & HP3O+ in Fig. 2 are absent. It is recommended that Fig. 2 and Fig. 3 can be merged, with more results of HP3O+ in more experimental tasks.

   2. Similarly, HP3O+ in absent in Fig. 4(a), which is important to demonstrate the performance of HP3O+ on variance.

   3. More baseline methods are expected, such as on-policy methods constructed based on PPO. Besides, comparisons with off-policy methods such as SAC are also meaningful. As illustrated in the paper, HP3O achieves superior sample efficiency compared to SAC, which is normal and acceptable.
2. This paper claims that SAC achieves higher sample efficiency at the cost of runtime complexity. Fig. 4(b) gives the training times of methods in *HalfCheetah* with 1M steps. However, it is more reasonable to give the training times of each method with *same episode returns*, which is more meaningful in practical applications. Besides, more results in other tasks are expected to demonstrate this claim. It is recommended that the curve of episode return - training time is clearer than Fig. 4(b).

3. Some latex errors \ref in the manuscripts.
Proximal Policy Optimization (HP3O) leverages a trajectory replay buffer to enhance the efficiency of utilizing trajectories generated by recent policies.
@inproceedings{
anonymous2024hpo,
title={{HP}3O: Hybrid-Policy Proximal Policy Optimization with Best Trajectory},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=PgR6fziYmJ},
note={under review}
}
Proximal policy optimization (PPO) is one of the most popular state-of-the-art on-policy algorithms that has become a standard baseline in modern reinforcement learning with applications in numerous fields. Though it delivers stable performance with theoretical policy improvement guarantees, high variance and high sample complexity still remain critical challenges in on-policy algorithms. To alleviate these issues, we propose Hybrid-Policy Proximal Policy Optimization (HP3O), which utilizes a trajectory replay buffer to make efficient use of trajectories generated by recent policies. Particularly, the buffer applies the "first in, first out" (FIFO) strategy so as to keep only the recent trajectories to attenuate the data distribution drift. A batch consisting of the trajectory with the best return and other randomly sampled ones from the buffer is used for updating the policy networks. The strategy helps the agent to improve its capability on top of the most recent best performance and in turn reduce variance empirically. We theoretically construct the policy improvement guarantees for the proposed algorithm. HP3O is validated and compared against several baseline algorithms using multiple continuous control environments. Our code is available here.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Reinforcement learning proximal policy optimization hybrid policy trajectory replay buffer variance reduction
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/c6cfcfa77aebd44a0d5ddeb424f5b2505078b7e9.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/138618bd3e6ec87ec6f112a4ccbd33e1f323dc64.zip
HP3O: Hybrid-Policy Proximal Policy Optimization with Best Trajectory
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission