LLM alignment methods can be divided into two categories: test-time and fine-tuning. Test-time alignment techniques, such as prompt engineering and guided decoding , adjust LLMs without altering their weights, although performance relies on the original model. Fine-tuning methods, such as RLHF , are effective but complex, requiring model training and continuous sampling. DPO simplifies RLHF by focusing on a direct optimisation objective from PPO , thereby streamlining the process to the training of supervised policy models. CPO enhances efficiency by utilising a uniform reference model, which reduces memory demands. Other methods employ SFT to optimise the management of the RLHF model and parameter tuning . Here, we explore alignment fine-tuning in a crossmodal setup.