DPO  was designed as an offline algorithm, but works better online for data constrained reasons. However I believe the most efficient formulation is as s massive distributed async off-policy RL. This allows you to reuse real with AI feedback while massively scaling data.