) Here, σ represents the logistic function, and r * denotes the "true" reward function that underlies the preferences. As obtaining the true reward directly from a human would be prohibitively expensive, a reward model r ϕ is trained to act as a surrogate. This is achieved by minimising the negative log-likelihood in human preference data;