Training
Once the model is created, it can be fine-tuned similar to BART, T5 or any other encoder-decoder model on a dataset of (image, text) pairs.
As you can see, only 2 inputs are required for the model in order to compute a loss: pixel_values (which are the
images) and labels (which are the input_ids of the encoded target sequence).
thon