RLHF can present challenges due to inherent slowness and instability, especially in the case of highly varied outputs . Recently, there has been a shift towards using closed-form losses in RLHF to align LLMs with human preferences. These losses are predominantly HALOs that model human biases, as discussed in , aiming to maximise the margin between preferred and dispreferred generated outputs. This approach offers a mathematical equivalence with RLHF, while effectively addressing inherent limitations.