For fine-tuning alignment, we focus on Reinforcement Learning from Human Feedback (RLHF) to align the universal models. Although alignment has typically been used to calibrate LLM behaviour , we hypothesise that it can also accelerate learning in crossmodal spaces by rewarding preferred over dispreferred outputs, thus improv- ing inference with minimal training data. We focus on optimisation algorithms using closed-form losses on offline preferences, such as Direct Preference Optimisation (DPO) , Contrastive Preference Optimisation (CPO) , and Kahneman-Tversky Optimisation (KTO) . We incorporate golden data as human preferences and dispreferred synthetic outputs generated by proprietary models into the reward signal (Step2 in Fig. ).