Strongly agree! I believe online, asynchronous, off-policy DPO has large potential for current “online RL” paradigm, though people usually think DPO=offline. InI did experiments to find DPO is very robust in async, off-policy learning (but only 1-step off, not a large buffer). Although not in the paper, when I enlarged the buffer size (a single parameter change using my codebase), the sample efficiency further boosts!