Feedback-aligned LLMs traditionally undergo fine-tuning with RLHF, where human preferences serve as a reward signal in optimisation . To train a LLM with RLHF, a reinforcement learning optimisation algorithm such as PPO is typically deployed on offline preference data, commonly involving three steps: • Model Training: Typically, a model π is trained for auto-regressive language generation on a large generic corpus. This training operates under the premise that the probability distribution of a sequence of words can be broken down into the product of conditional distributions for the next word . • Reward Model Training: A reference model π ref is employed to optimise π for a downstream task. Typically, the π ref model undergoes finetuning with an auto-regressive objective, using data pertinent to the downstream task. This often involves instruction tuning π ref to regulate the generated outputs. • Reinforcement Learning: The optimisation of π with respect to π ref operates on a triple dataset D = {x, y w , y l }, where x represents the input, and y w and y l denote preferred and dispreferred outputs, respectively, such that y w ≻ y l for x. In the Bradley-Terry model , the probability of y w being preferred over y l in pairwise comparisons can be formulated as follows: