What Makes Multimodal In-Context Learning Work?
Folco Bertini Baldassini Mustafa Shukor Matthieu Cord Laure Soulier Benjamin Piwowarski

No Train but Gain:
Language Arithmetic for training-free Language Adapters enhancement
    [
    April 29, 2024
======================================================================================


Large Language Models have demonstrated remarkable performance across various tasks, exhibiting the capacity to swiftly acquire new skills, such as through In-Context Learning (ICL) with minimal demonstration examples.  In this work, we present a comprehensive framework for investigating Multimodal ICL (M-ICL) in the context of Large Multimodal Models. We consider the best open-source multimodal models (, IDEFICS, OpenFlamingo) and a wide range of multimodal tasks. Our study unveils several noteworthy findings: (1) M-ICL primarily relies on text-driven mechanisms, showing little to no influence from the image modality. (2) When used with advanced-ICL strategy (like RICES), M-ICL is not better than a simple strategy based on majority voting over context examples. Moreover, we identify several biases and limitations of M-ICL that warrant consideration prior to deployment. Code available at https://gitlab.com/folbaeni/multimodal-iclgitlab.com/folbaeni/multimodal-icl
    




§ INTRODUCTION

Recently, Large Multimodal Models (LMMs) have made considerable progress in comprehending and generating visual and textual content <cit.>. These models can be seamlessly adapted to solve novel tasks, through In-context learning (ICL) <cit.>. It is a training-free approach that consists of augmenting the input prompt with a few pairs (input,output) prepended to the query prompt. This extra context acts as demonstrations that should help the model understand the task at hand. The choice and ordering of examples used in the ICL is decisive to its performance, as observed for retrieval methods <cit.>, and for multimodal tasks by exploiting CLIP <cit.>, exemplified by RICES <cit.>.  
While extensive research has been carried out into conditions and biases of ICL for LLMs <cit.>, extending this knowledge to the multimodal domain is not trivial. Besides, multimodal ICL (M-ICL) presents new challenges and biases  <cit.> that may not be fully addressed by existing unimodal studies. 

In this paper, we propose a comprehensive framework to study M-ICL: using  the best open-source LMM models with M-ICL ability, such as IDEFICS <cit.> and OpenFlamingo <cit.>, we consider a wide range of  multimodal benchmarks that cover Visual Question Answering (VQA), captioning and classification tasks.  To investigate how modalities (image and text) affect the M-ICL behavior, we systematically remove or mix each modality. We then extend our study to approaches that improve ICL with retrieval-based context selection (RICES <cit.>).

To summarize, we propose a comprehensive framework to evaluate the M-ICL behavior in LMMs.  Our empirical study led to the following findings illustrated in <Ref>:
    
        
  * In general, M-ICL is primarily focused on text, overshadowing the role played by images. This is less the case for image captioning and classification tasks.
        
  * For advanced similarity-based context selection M-ICL methods, the LMM models behave so far not better than  a majority voting mechanism over the context demonstrations. 
        
  * We also identify a major flaw in these advanced similarity-based methods. They suffer from recency bias, where the model tends to "copy" the answer of the last example in context. This sheds light on several limitations that should be considered before deployment. 
    



§ RELATED WORK




  
Multimodal models
 have undergone significant advancements recently <cit.>, by moving towards more unified models that can support a myriad of tasks and modalities <cit.>. These models are generally built on top of pre-trained LLMs and visual encoders that are simply connected by a linear transformations <cit.>, or transformer-based mechanisms <cit.>. The level of performance of these models has started to approach those of LLMs, especially after multimodal instruction tuning <cit.>. In addition, several models can now support ICL <cit.>, arguably due to training on interleaved image-text datasets. In this work, we focus on the best open-source models with ICL abilities (IDEFICS <cit.> and OpenFlamingo <cit.>), and in particular, IDEFICS that achieves comparable performance to Flamingo. 



  
In-Context Learning (ICL) is a paradigm that allows language models to learn tasks given only a few demonstrations <cit.> and is particularly effective for tackling more complex and reasoning-based tasks <cit.>.
To explain ICL, studies compare it with gradient descent <cit.> and examine the inner workings of the models <cit.>.
ICL is highly sensitive to the prompt and choice of demonstrations,  <cit.> indicates that the format of the prompt and distribution of the words matter, though the importance of labels is debated <cit.>. Interestingly,  <cit.> discusses task recognition and task learning, where the former requires a few examples to understand the task format, and the latter to reproduce the input-output mapping. This depends on multiple factors such as if the model has been instruction tuned <cit.>, the model size <cit.>, and the semantics of the prompt <cit.>, affecting the necessary number of shots. Studies also identify recency and majority biases <cit.> and order sensitivity <cit.>.



  
Multimodal ICL.
ICL can be extended to multimodal models after training on interleaved image-text datasets  <cit.>.
To further enhance M-ICL, several works try to use better context demonstrations using similarity sampling-based approaches  <cit.>. Despite being effective, especially in handling out-of-distribution tasks  <cit.>, several works have tried to highlight several flaws. In particular, increasing object hallucinations and the limited ability to solve complex tasks such as instruction following or compositional image-text matching <cit.>. In addition, <cit.> study OpenFlamingo

and find that the image plays a marginal role in VQA tasks, raising questions about the effectiveness of ICL in a multimodal context. 



§ ANALYSIS FRAMEWORK OF M-ICL


For M-ICL, LMMs process inputs composed of a query Q and a context C. The query Q includes an image I and an optional associated text T, which can be a question, instruction, or additional information. The context C comprises N  demonstrations (examples) from the training dataset D, each containing images and texts along with their corresponding responses R. M-ICL can be written as follows:



    C=((I_i,T_i,R_i))_i∈ D_C,  O = LMM(C, (I_Q, T_Q))

Our similarity sampling method is RICES <cit.>. 
Given a query Q, it retrieves the N most similar demonstrations from the training set according to S_iq = s(I_i, I_Q) + s(T_i, T_Q), where s represents the similarity score calculated by the visual encoder CLIP <cit.>. These demonstrations are arranged in the context in order of increasing similarity.




 §.§ Research questions & analysis methodology

Our objective is to understand how different modalities affect M-ICL – here text and image. While there are several methods for demonstration retrieval in the literature <cit.>, there's limited work <cit.> for M-ICL and consequently little analysis of these methods. We believe that it is essential to investigate how ICL's sensitivity factors apply to these methods and identify their limitations. We address the following research questions:


RQ1: How does each modality influence M-ICL? To analyze the effect of each modality, we modify the context C by adjusting either I (images) or T (text). We describe the procedure for I, but the same method applies to T. We either completely remove the image component, resulting in a new context defined as ((∅, T_i, R_i))_i∈ D_C, or randomize this modality by using random images from the demonstration dataset. In the later, the altered context is represented as ((I_j, T_i, R_i) | j ≠ i)_i∈ D_C.  
We also conduct experiments with RICES to identify any behavioral differences. 

RQ2: Which kind of shortcuts influence M-ICL?
We are interested in whether M-ICL involves genuine learning from demonstrations, or if it relies on what we name “shortcuts”. Using Generalized Linear Models (GLM) and Spearman's rank correlation, we evaluate the relationship between the similarity of the demonstrations to the query and their performance outcomes. We compare random sampling with RICES to understand M-ICL's behavior, focusing on the improvements attributed to RICES. This analysis aims to understand the reason of these improvements and whether they reveal any emerging behaviors that suggest reliance on shortcuts.
We then turn to the question of what performance gains can be attributed to RICES or to the m-ICL of LLMs. 
More precisely, for classification tasks, we rely on a simple RICES based KNN where the predicted answer O' is given by argmax_R ( ∑_{i ∈ D_C | R_i = R} e^S_iq). For generation tasks (VQA and captioning), we also rely on another set of analysis, since the KNN approach is not the most adapted. 
Finally, we investigate another factor impacting M-ICL, namely the recency bias that complement our analysis on the relationship between the similarity of the context answer to the target one.




 §.§ Experimental setup




  
Datasets In our study, we investigate various tasks, including image captioning, classification, and visual question answering. For captioning, we employ the COCO dataset <cit.> and the Flickr30k dataset <cit.>, where each image is annotated with five captions; we select one caption randomly for our experiments and evaluate using the CIDEr <cit.> metric. In classification, we use the CIFAR-100 <cit.> and ImageNet <cit.> datasets, with 100 and 1000 classes, respectively. The predicted class is the one whose label has the smallest Levenshtein distance to the model's output. We use accuracy as the metric. An alternative would be to instruct the model to choose among all the classes, but this has a high computational cost.
We also examine the Hateful Memes <cit.> and Rendered SST2 <cit.> datasets for detecting hate speech and performing sentiment analysis through OCR, measuring performance by exact match accuracy. For visual question answering, we use the VizWiz <cit.>, VQAv2 <cit.>, OK-VQA <cit.>, TextVQA <cit.>, ScienceQA <cit.> (only items containing images), and MMMU <cit.> datasets, 
covering a range of applications from assisting visually impaired users to requiring scientific reasoning, 
with VQA accuracy as metrics for most, except accuracy for multiple-choice formats for ScienceQA and MMMU. The test set is composed of a maximum of 5000 items, chosen randomly if the original dataset exceeds this number. This set remains the same across all tests, serving as a consistent basis for comparison. Additionally, the entire training dataset is used as the support set for M-ICL demonstrations.





  
Models and ICL details.
We conduct our tests with IDEFICS <cit.>  9B version (for OpenFlamingo, results are reported in the Appendix <ref>). For RICES we use the CLIP version "openai/clip-vit-large-patch14". 
Unless specified, demonstrations are chosen randomly.
For captioning and classification tasks (image-to-text tasks), the demonstrations consists of interleaved image and captions/classes. For VQA datasets, the text consists of the question-answer pairs.
We do not use explicit task instruction, letting the model understand the task from its context. 
We repeat each experiment 3 times and report the averaged results. 




§ RQ1: HOW DOES EACH MODALITY INFLUENCE M-ICL?

In this section, we try to answer RQ1, i.e we investigate the influence of each modality on M-ICL and their interactions by manipulating the context (text or images).
We conduct our study with randomly sampled demonstrations and extend to the retrieval M-ICL such as RICES in <Ref>.
We summarise the results in <Ref>, presenting the scores for the 16-shot scenario with both contexts of altered images (<Ref>) and texts (<Ref>). Additionally, we illustrate the effect of the number of demonstrations in <Ref>. To make values comparable across tasks, we normalize the measures.



 §.§ Images impact M-ICL

In <Ref>, we observe that image-to-text tasks like captioning and classification are highly affected when altering the images. Compared to the context baseline that consists of images and their correct classes/captions, using random images or removing them from the context leads to a significant drop in performance. The  performance for datasets such as CIFAR and ImageNet is close to the level of a zero-shot m-ICL, and for MS-COCO it is even worse. This phenomenon is corroborated in <Ref>, where we show that adding more demonstrations with random images has a strong negative impact on image-to-text tasks, in stark contrast to the initial demonstrations.

To understand the effect of using random images, in <Ref> we examine the model's output in this setup. Our analysis focuses on the most common n-grams found in the captions over the whole dataset (pink) within the context, looking at their frequency in the model's output. We compare the base prompt (blue) when 32 demonstrations are used, against random images setup in 4 shot (orange) and 32 shot (green). In the case of the base prompt, the distribution appears similar to that of the context, indicating a similar input and output distribution of words. However, in scenarios involving random images, there is a noticeable shift towards an over-representation of the most frequent n-grams, and the more demonstrations the more this happens. This suggests that the mismatch between images and their corresponding textual outputs in the demonstrations causes the model to switch to a generic mode, in which it tends to output the most frequent words in the dataset used to construct the ICL context. 



These results suggest that demonstration images influence the performance of M-ICL in image-to-text tasks, and that the model leverages the relationship between visual inputs and textual outputs.
We discuss the potential reasons for this behavior in section <ref>.

We now turn to VQA which exhibits a different behavior. Altering or omitting images results in a minor decrease in performance, typically between 1.2 to 1.5 points from the base prompt (<Ref>). This suggests that the inclusion of textual information (i.e. questions) diminishes the model's dependence on visual data, a topic we explore in the next section.




 §.§ Text drives M-ICL

In VQA, which has both image and text (i.e. questions) as input,  <Ref> illustrates that removing the question (purple) results in an average drop of 3.5 points. Moreover, replacing it with a random question (green) leads to an average decrease of 9.5 points. We further observe (<Ref>) that the decrease worsens with an increasing number of shots[In practice, M-ICL often outputs 'no', the most frequent answer in the dataset]. 

For text-to-image tasks, <Ref> provides also insights into the role of text, as scenarios without images (orange) correspond to a scenario with only text. In classification tasks, where the text has limited information, i.e. just the one-two word labels,  the text-only scenario performs as poorly as the zero-shot setup (black), with only a 0.47% increase in accuracy. However, in captioning, where the text is richer, M-ICL enables capturing the style of the captions and/or the distribution of words, resulting in an average improvement of 31 points over the zero-shot approach.

These results indicate that text influences M-ICL when it carries sufficient semantic content.

In summary, in classification tasks, text has a minor impact compared to images. When text becomes richer, particularly in the context of captioning, the use of text alone can improve zero-shot methods by 31 points. Incorporating images further enhances performance by an additional 20 points, underscoring the importance of both modalities. 
In tasks like VQA, textual information becomes dominant and significantly influences performance, with random text  leading to a significant drop in performance. We conclude that while images do have an impact on M-ICL, textual information takes precedence and drives the model's decision-making process.



 §.§ How do retrieving similar demonstrations affect interactions?
 
In the previous section, demonstrations were randomly sampled. Here, we turn to similarity-based (RICES) M-ICL, and analyze which observations still hold and which don't.
First, <Ref> shows that in most cases RICES leads to better performance. For captioning tasks, the more demonstrations, the better the performance. For VQA, the use of RICES leads to improvements of between 2 and 5% for most datasets. The most significant improvements are in classification, where gains range from 10 to 50%. 



Investigating the factors influencing these improvements and how each modality contributes can help us understand better multimodal interactions. We follow the same procedure as with random sampling: We investigate the effect of disrupting the alignment between visual and textual parts, while maintaining one modality closely related to the query. Additionally, we explore which modality is pivotal for the improvements by computing similarity based on different modality choices.



  
Disrupting image-text alignment  
In <Ref> we observe that there is no significant degradation when removing the images or replacing them with random ones. The context with random images (in blue), where only the demonstration responses resemble the query, yields results comparable to random sampling and is slightly better than removing images.
Furthermore there is no noticeable drop in performance as the number of examples increases, which is different than when using randomly sampled demonstrations (as shown in Appendix <ref>).
On the other hand, random responses (in purple) show a significant decrease in performance (i.e. only the demonstration images are similar to the query's). 

In particular, when substituting the responses in the context by  random ones, the drop is more important in RICES than with random sampling (, as shown in Appendix <Ref>; for random sampling and image-to-text tasks, random image and random label is equivalent). 
Having the wrong responses for images similar to the query, might push the model to naturally output the wrong response as well. 

Overall, the results above suggests that images serve as a prior for the demonstrations, which is confirmed in the analysis conducted in <ref>.





  
Retrieving demonstrations similar to text or image query?
In the case of VQA, the question is composed of text and images. As described in <Ref>, S_iq is the sum of CLIP textual and visual similarities. In <Ref>, to further explore the effect of each modality, we compare this baseline (orange) to using only CLIP image similarity (blue) or CLIP text similarity (pink).
Results vary across different datasets, however for TextVQA, VQAv2, and VizWiz, using image similarity has a better outcome, while  textual similarity is better for MMMU, OK-VQA, and ScienceQA. 
This might be explained by the nature of each dataset: TextVQA, VQAv2, and VizWiz necessitate images for accurate responses, whereas MMMU, OK-VQA, and ScienceQA are more dependent on textual information. To conclude, using the right similarity highly depends on the actual dataset, and there is no clear indication of which to choose for M-ICL models.





§ RQ2: WHICH KIND OF SHORTCUTS INFLUENCE M-ICL?


In this section, we answer to RQ2, i.e. we try to explain the M-ICL behavior with random or similarity-based demonstrations. More precisely, we investigate whether M-ICL performance can be partially explained by the fact that the demonstration responses can be close to the desired response, and the M-ICL model do a “soft copy” of the demonstration responses. Formally, we hypothesize (1) that, given a demonstration i and a query q the similarity function S_iq has a correlation with the CLIP score between the responses R_i and R_q, denoted S_iq^R, i.e. if demonstrations inputs are similar to the query inputs, the same applies for the responses. Furthermore, we also hypothesize (2) that, given a context C composed by demonstrations i, the average of the similarities S^R_iq of the demonstrations responses R_i with the target response R_q correlates with performances, i.e., the closest the context responses to the target one, the better the generated one.

To verify these hypotheses, we compute both General Linear Model (GLM) coefficients and Spearman correlation to characterize the relationship between different factors described above.
In the first column of <Ref>, we compare S_iq and S^R_iq=s(R_i, R_q) with s the CLIP similarity across all demonstrations (hypothesis 1). With RICES, we can observe a positive Spearman correlation, especially for classification (SST2) and text-to-image (COCO) datasets, slightly less for VQA (VQAv2). The regression coefficient, close to 1, shows that the similarities almost match in average. We also observe that correlation drops when using random samples, showing that this relation holds only when looking at more similar demonstrations.
In the second column of <Ref>, we look at the relationship between (a) the average similarity avg(S^R_iq) between a demonstration and target response; and (b) the performance of M-ICL. We again only observe correlation in all cases when using RICES demonstrations.




These observation support our initial explanation of M-ICL performance in the case of RICES (this is less clear otherwise), i.e. RICES is effective because it retrieves responses that closely match the target one.
This raises the question of whether the performance gains from M-ICL are simply due to better context responses acting as shortcuts, or whether there is genuine learning involved, with demonstrations that are more similar to the query proving to be more useful. In what follows, we study two potential shortcuts: one being that M-ICL might simply exploit the presence of more accurate or relevant responses in the context, and the other being that the most similar demonstrations, whose response is probably the same or close to the query's, are the most recent, and the model could be leveraging this recency. The remaining of this section aims to explore and clarify these two possibilities.



 §.§ M-ICL does a majority vote over the demonstrations

We dive into the first possibility, which examines the impact of having more accurate or relevant responses in the context. We aim to assess the effectiveness of M-ICL with demonstrations similar to the query by comparing the performance of M-ICL and a simple RICES KNN baseline described in  <ref>.





<Ref> illustrates that for classification, RICES KNN (blue) obtains similar performances than M-ICL when using the same demonstration (orange), and outperforms the random sampling setup (green). 
In particular RICES M-ICL struggles to surpass RICES KNN, and this is particularly visible for SST-2, where increasing the number of demonstrations decreases the performances for both the KNN and ICL (see Appendix <ref>).  

To further show this majority voting effect, we observed that ensuring that the labels are uniformly distributed with the demonstrations degrades the performance of both M-ICL and the KNN (see Appendix <ref>). 
This suggests that M-ICL leverages similar demonstrations by leveraging the distribution of context responses, rather than actually learning. Said otherwise, in classification tasks, M-ICL's effectiveness is comparable to that of a KNN, and M-ICL does not seem to be useful.



In open-ended generation tasks, i.e. captioning and visual question answering, majority voting is insufficient. Here the baseline method falls short against random sampling and the RICES approach shows small improvements. <Ref> and <Ref> show that there is a correlation especially between the responses and performance while this is not the case for the images and texts. This is more true with RICES, but also present for random sampling in VQA. 

To further analyze this phenomenon, we introduce Oracle RICES which leverages the similarity metric S^R_iq=s(R_i,R_q) that uses the ground truth response R_q.  This approach enables us to select examples with responses that closely match the desired answer. In VQA, if "yes" is the correct answer, the chosen examples will all share this answer despite differences in image or text content.
<Ref> illustrates this method in pink and that it significantly outperforms the others methods, providing an upper limit for the RICES approach. 
This in turn show that (a) for open-ended generation m-ICL can do intelligent soft copy when provided close responses; (b) that the used RICES similarity does not select enough demonstrations with a high target response similarity which can improve substantially the performance.





 §.§ M-ICL tends to copy recent similar responses

Another factor impacting the performance can be the ordering of the demonstrations. In <Ref>,
we compute the GLM coefficients for  S_i^R = s(R_q, R_i) when the performance is the response variable. 
For random sampling, we observe that this coefficient does not depend much on the position, while for RICES the coefficient increases from 0.01 (first rank) to 0.30 (4th rank) in captioning (and similarly in VQA, but to a lesser extent). As we saw earlier, this might be explained by the fact that this coefficient increases with more similar demonstrations. Another possibility is that M-ICL relies more on later ranks. The lines "RICE reverse" show that the latter explanation is truer, since by reversing the RICES order the coefficient still increases (to some extent) with the rank of the demonstration.




To further analyze the impact of this recency phenomenon, we compare the outputs of the model against each demonstration's output. Where there is a complete match between an entire demonstration's response and the full output produced by the model
For multiple matches, only the last one is recorded. Yes/no answers are excluded since in their frequency in VQA would skew the results.
This allows us to measure the extent with which a demonstration response is replicated in the model output. Although we observed that exact copies are extremely rare for random sampling (not shown here), the RICES method shows a frequent replication of the last demonstration (as depicted by the bars in <Ref>). For VQA, the final context response is used 12% of the cases, regardless of the number of shots. For captioning, this ranges from 24% with four shots to 4% with 32 shots. We compare with a variation of RICES where demonstrations are arranged from most to least similar (depicted by the lines). In this setup, the model less frequently replicates the last output, yet the same trend remains, indicating that the model tends to replicate the outputs of the more recent demonstrations over the more similar ones. This demonstrates that when M-ICL is faced with similar demonstrations, a recency bias leans towards replicating the output of the latest ones rather than the most similar. 






§ CONCLUSION

We propose a framework to study ICL in a multimodal context. Our study reveals that M-ICL is primarily text-driven, and that images in the context have little impact on the overall performance. This is exacerbated when using RICES to improve M-ICL. 
We also show that the reason of the success of similarity-based M-ICL is partially due to the fact that such techniques retrieve responses which are more similar to the target one rather than merely retrieving more related demonstrations. The practical consequences are that for classification-based tasks, M-ICL is useless when using RICES, and that for open-ended generation, there is still a gap that could be leveraged between RICES-retrieved responses and ideal ones.
In addition, we show that M-ICL suffers from different biases, such as the ability to replicate the last example in the demonstrations.
Our work sheds light on several limitations and suggests that there is room for improvement regarding M-ICL. Current M-ICL improvements can be brought by M-ICL variants or prompting strategies <cit.>, or better training datasets <cit.>. Our work suggests that working on better retrieval and reducing the biases 
(recency) 
would also benefit this line of models.
Finally, while our findings hold for the best open-source M-ICL models, we recognize it would be  important to study more powerful models such as GPT4-V <cit.> and Gemini <cit.> to check if our conclusion still hold.



§ ACKNOWLEDGMENTS

This work was partly funded by the ANR-23-PEIA-0008, PEPR IA, project "Principes théoriques et algorithmiques de l’apprentissage frugal (SHARP)," and received computing AI and storage resources from GENCI at IDRIS on the Jean Zay supercomputer's V100/A100 partition through grant 2023-AD011014764.



    ieeenat_fullname
    





