Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generation
offline-to-online reinforcement learning;data augmentation;diffusion models
Yes
No ethics review needed.
1. The current explanation assumes that separate generation is beneficial due to distribution differences, but this could be expanded by quantifying how these differences affect policy optimization. Could the authors provide additional theoretical or empirical justification for why separate generation of offline and online data leads to significant performance gains in O2O RL?
2. How sensitive is the performance of CFDG to changes in the offline-to-online and synthetic data ratios?
1. By analyzing the distinct distributions of offline and online data, the authors identify the benefits of separately augmenting each data type. 
2. The use of conditional diffusion with classifier-free guidance allows the generation of high-quality offline and online samples independently. 
3. Through comprehensive experiments on challenging benchmarks like Locomotion and AntMaze, the paper shows that its approach significantly boosts the performance of multiple O2O RL algorithms.
This paper introduces CFDG, a method that augments both offline and online data separately to enhance offline-to-online reinforcement learning. Experimental results on benchmarks like D4RL demonstrate that integrating CFDG into standard O2O RL algorithms, such as IQL, PEX, and APL, yields an average 15% improvement over prior data augmentation approaches like SynthER and EDIS. CFDG thus provides an effective and adaptable way to boost O2O RL performance through refined data augmentation.
1.The paper mentions that the ratio of offline to online data and the ratio of real data and synthetic data can significantly impact performance but does not explore this aspect in detail. Specifically, a sensitivity analysis of these ratios would help determine optimal values or provide insights into the adaptability of CFDG in diverse O2O RL scenarios.
2. The paper lacks sufficient innovation; the difference in distribution between offline and online data is obvious, and the analysis in Section 3.1 does not provide any new insights. The classifier-free guided diffusion model used for data generation is also based on existing work. The only innovative aspect of the paper is the separate generation of offline and online data.
Yes
No ethics review needed.
No
The motivation is clear
This paper proposes to utilize classifier-free diffusion for data augmentation. More specifically, it distinguishes the difference between online data and offline data and uses conditional diffusion to generate both types.
1. The novelty is limited.
2. The performance improvement is negligible and sensitive to hyperparameters.
3. Ablation studies fail to provide enough insights into the algorithm.
Yes
No ethics review needed.
How can we embed the class identifier when constructing a conditional diffusion model?
The method is designed with simplicity in mind, using a motivating example to illustrate the data distribution and introduce the proposed approach.
To enhance offline-to-online reinforcement learning algorithms, this paper proposes a data augmentation approach called Classifier-Free Diffusion Generation (CFDG). Recognizing the differences between offline and online data, the authors use conditional diffusion to generate both types of data for augmentation during the online phase. This approach aims to improve the quality of sample generation.
1. Incomplete method description of hyperparameters, requiring significant adjustment for implementation.
2. Incomplete sentence, like on line 181.
3. Limited performance improvements, with potentially misleading labels (e.g., antmaze-medium-play IQL).
4. Lack of novelty, as careful adjustments to existing baselines could yield similar results.
@inproceedings{
anonymous2024offlinetoonline,
title={Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=cXxfVkRCHJ},
note={under review}
}
Offline-to-online Reinforcement Learning (O2O RL) aims to perform online fine-tuning on an offline pre-trained policy to minimize costly online interactions. Existing methods have used offline data or online data to generate new data for data augmentation, which has led to performance improvement during online fine-tuning. However, they have not fully analyzed and utilized both types of data simultaneously. Offline data helps prevent agents from settling too early on suboptimal policies by providing diverse data, while online data improves training stability and speeds up convergence. In this paper, we propose a data augmentation approach, Classifier-Free Diffusion Generation (CFDG). Considering the differences between offline data and online data, we use conditional diffusion to generate both types of data for augmentation in the online phase, aiming to improve the quality of sample generation. Experimental results show that CFDG outperforms replaying the two data types or using a standard diffusion model to generate new data. Our method is versatile and can be integrated with existing offline-to-online RL algorithms. By implementing CFDG to popular methods IQL, PEX and APL, we achieve a notable 15% average improvement in empirical performance on the D4RL benchmark like MuJoCo and AntMaze.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
offline-to-online reinforcement learning data augmentation diffusion models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/9456edca01a6df0353c9cccd8b0d2230bd92cbc3.pdf
reinforcement learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission