<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2406.15334] Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning</title><meta property="og:description" content="The recent success of interleaved Large Multimodal Models (LMMs) in few-shot learning suggests that in-context learning (ICL) with many examples can be promising for learning new tasks. However, this many-shot multimod…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2406.15334">

<!--Generated on Fri Jul  5 19:57:25 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Brandon Huang<sup id="id1.1.id1" class="ltx_sup">1*</sup> &amp;Chancharik Mitra<sup id="id2.2.id2" class="ltx_sup">1*</sup> &amp;Assaf Arbelle<sup id="id3.3.id3" class="ltx_sup">2</sup> &amp;Leonid Karlinsky<sup id="id4.4.id4" class="ltx_sup">3</sup> &amp;Trevor Darrell<sup id="id5.5.id5" class="ltx_sup">1</sup>  Roei Herzig<sup id="id6.6.id6" class="ltx_sup">1, 2</sup> 
<br class="ltx_break">
<br class="ltx_break"><sup id="id7.7.id7" class="ltx_sup">1</sup> University of California, Berkeley  <sup id="id8.8.id8" class="ltx_sup">2</sup> IBM Research  <sup id="id9.9.id9" class="ltx_sup">3</sup> MIT-IBM Watson AI Lab
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id10.id1" class="ltx_p">The recent success of interleaved Large Multimodal Models (LMMs) in few-shot learning suggests that in-context learning (ICL) with many examples can be promising for learning new tasks. However, this <span id="id10.id1.1" class="ltx_text ltx_font_italic">many-shot</span> multimodal ICL setting has one crucial problem: it is fundamentally limited by the model’s context length set at pretraining. The problem is especially prominent in the multimodal domain, which processes both text and images, requiring additional tokens. This motivates the need for a multimodal method to compress many shots into fewer tokens without finetuning. In this work, we enable LMMs to perform multimodal, many-shot in-context learning by leveraging Multimodal Task Vectors (MTV)—compact implicit representations of in-context examples compressed in the model’s attention heads. Specifically, we first demonstrate the existence of such MTV in LMMs and then leverage these extracted MTV to enable many-shot in-context learning for various vision-and-language tasks. Our experiments suggest that MTV can scale in performance with the number of compressed shots and generalize to similar out-of-domain tasks without additional context length for inference.</p>
</div>
<span id="footnotex1" class="ltx_note ltx_role_footnotetext"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">footnotetext: </span><sup id="footnotex1.1" class="ltx_sup">*</sup>Denotes Equal Contribution</span></span></span>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<figure id="S1.F1" class="ltx_figure"><img src="/html/2406.15334/assets/figs/fig1_v5.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="568" height="450" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S1.F1.3.1.1" class="ltx_text" style="font-size:90%;">Figure 1</span>: </span><span id="S1.F1.4.2" class="ltx_text" style="font-size:90%;">
<span id="S1.F1.4.2.1" class="ltx_text ltx_font_bold">Overview of Multimodal Task Vectors (MTV).</span> In the standard multimodal in-context learning (ICL) paradigm (top), the number of shots is limited by an LMM’s context length. We solve this issue by first calculating the mean activations of the last token’s attention heads (grey squares in the LMM) given a set of multimodal ICL examples and then finding a set of attention head locations (yellow squares in the LMM) that best align with the downstream task. These mean activations are then replaced directly in these attention heads locations, enabling many-shot multimodal ICL.
</span></figcaption>
</figure>
<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Large Multimodal Models (LMMs) such as GPT-4V <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib59" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">59</span></a>]</cite>, LLaVA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib50" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">50</span></a>, <a href="#bib.bib49" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">49</span></a>]</cite>, and the BLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib13" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">13</span></a>, <a href="#bib.bib43" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">43</span></a>]</cite> family of models demonstrate state-of-the-art performance on a variety of vision and language (VL) tasks due to their strong reasoning capabilities over both text and images. Recent works show that LMMs pre-trained on interleaved text-image data can do multimodal in-context learning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">6</span></a>, <a href="#bib.bib39" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">39</span></a>]</cite>. In particular, few-shot, in-context learning (ICL) in text-only LLMs has been scaled with an increasing number of examples in long-context language models—a setting called many-shot learning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">1</span></a>]</cite>. A natural question arises on how to perform many-shot learning in the multimodal domain.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">The first issue with directly applying a many-shot learning regimen to LMMs is the intrinsic limitation of context length. This is especially true in the multimodal domain, as LMMs must encode both text and images, whose embeddings are token-expensive. Moreover, long-context language models, which LMMs leverage for reasoning, struggle to use their entire context length effectively for ICL <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib51" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">51</span></a>, <a href="#bib.bib45" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">45</span></a>]</cite>. Secondly, perhaps due to the misalignment of pretraining tasks with ICL, many instruction-tuned LMMs underperform on tasks in the ICL setting <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">16</span></a>]</cite>, suggesting the importance of interleaved LMMs. Finally, there is also the challenge of the increasing memory and run-time required for processing long contexts for every inference call. These challenges motivate a method for compressing multimodal in-context examples into compact, implicit representations. Therefore, in this paper, we propose Multimodal Task Vectors (MTV)—compact representations of multimodal in-context tasks—within the attention heads of LMMs to enable many-shot ICL. In particular, we show the existence of MTV in interleaved LMMs, and we use them to compress large numbers of multimodal ICL examples.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">Recent research in explainability has demonstrated the existence of task vectors in both the language <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib79" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">79</span></a>, <a href="#bib.bib25" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">25</span></a>]</cite> and vision <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib27" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">27</span></a>]</cite> domains. These task vectors are implicit representations of in-context tasks represented by sets of activations in the model. These activations compactly summarize the information in ICL examples. In our work, we go beyond proving the existence of these task vectors in the multimodal domain by demonstrating their ability to compress examples for many-shot ICL in LMMs without the need for finetuning.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">Our method can be described in three steps. First, given a set of many-shot multimodal ICL examples, we calculate the mean activations of the attention heads across multiple inference iterations. Second, to avoid the context length constraint, we select a set of attention heads in the model to store the mean activations of the ICL examples. However, since the downstream task may be zero-shot or use a different number of ICL examples, we select a set of examples aligned with its form. We then use these examples to find an optimal set of LMM head locations where the many-shot examples will be encoded. We refer to these mean activations and locations as MTV, which implicitly encodes the many-shot multimodal examples for use in the downstream task. Finally, for downstream inference, we replace the mean activations from Step 1 with the attention head locations found in Step 2. Since we input examples to the LMM across different iterations in Step 1, Multimodal Task Vectors can implicitly encode more examples than are allowable by the context limit. We find that utilizing many examples for extracting MTV surpasses performance on zero-shot and most standard few-shot ICL settings, suggesting the effectiveness of our method. Another key benefit of our method is that it frees up tokens for the model during downstream inference compared to standard few-shot ICL methods.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">We summarize our main contributions as follows: (i) We show the existence of Multimodal Task Vectors, compact implicit representations of in-context functions in LMMs. (ii) MTV can encode more examples than allowed by an LMM’s context length, enabling both runtime and memory-efficient multimodal many-shot in-context learning.
(iii) MTV surpasses zero-shot and few-shot ICL settings on various VL benchmarks without finetuning.
(iv) MTV can scale to larger numbers of examples and can generalize to similar out-of-domain tasks.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Works</h2>

<div id="S2.p1" class="ltx_para ltx_noindent">
<p id="S2.p1.1" class="ltx_p"><span id="S2.p1.1.1" class="ltx_text ltx_font_bold">Many-Shot In-Context Learning</span>. Few-shot in-context learning (ICL) is a significant area of study in text-only LLMs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib9" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">9</span></a>, <a href="#bib.bib87" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">87</span></a>]</cite>. A natural question arises about the possibility of using a larger number of shots (e.g., hundreds) to further improve performance or learn more complex tasks. Indeed, some early work in text-only <span id="S2.p1.1.2" class="ltx_text ltx_font_italic">many-shot, in-context learning</span> suggests performance on different tasks can scale with a larger number of examples <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib7" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">7</span></a>, <a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">1</span></a>, <a href="#bib.bib44" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">44</span></a>, <a href="#bib.bib45" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">45</span></a>]</cite>.</p>
</div>
<div id="S2.p2" class="ltx_para">
<p id="S2.p2.1" class="ltx_p">However, scaling ICL in text-only LLMs is a challenge due to the intrinsic context length. One method to increase context length in these models is to apply positional interpolation methods <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">10</span></a>, <a href="#bib.bib62" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">62</span></a>]</cite>. However, research on these longer-context models finds that they struggle to use the entire context for ICL <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib51" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">51</span></a>, <a href="#bib.bib45" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">45</span></a>]</cite>. Moreover, as inference on long contexts of inputs is also time and memory-expensive, it is unclear whether simply scaling the context of models is practical for enabling multimodal many-shot ICL. This has led to work that looks to compress explicit input tokens <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib20" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">20</span></a>, <a href="#bib.bib75" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">75</span></a>, <a href="#bib.bib11" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">11</span></a>, <a href="#bib.bib57" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">57</span></a>, <a href="#bib.bib71" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">71</span></a>, <a href="#bib.bib35" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">35</span></a>]</cite>. But crucially, many of these methods require finetuning and only try to preserve performance. Our work is different in that it is the first to enable <span id="S2.p2.1.1" class="ltx_text ltx_font_italic">multimodal</span> models with many-shot ICL capabilities, while also having the benefit of improving on complex VL tasks without finetuning.</p>
</div>
<div id="S2.p3" class="ltx_para ltx_noindent">
<p id="S2.p3.1" class="ltx_p"><span id="S2.p3.1.1" class="ltx_text ltx_font_bold">Task Vectors</span>. Our work builds off of research in text-only and vision-only domains showing that internal representations of these models called task vectors <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib79" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">79</span></a>, <a href="#bib.bib25" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">25</span></a>, <a href="#bib.bib27" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">27</span></a>]</cite> (or function vectors) can encapsulate tasks outlined by ICL examples. Our is the first demonstration of Multimodal Task Vectors (MTV) in LMMs. Going beyond previous work, however, we show that MTV enable LMMs not only to use many-shot, multimodal ICL examples but also scale with more samples, be used alongside explicit ICL shots, and even generalize to unseen classes or similar tasks.</p>
</div>
<div id="S2.p4" class="ltx_para ltx_noindent">
<p id="S2.p4.1" class="ltx_p"><span id="S2.p4.1.1" class="ltx_text ltx_font_bold">Model Domain Adaptation Methods</span>.
As LLM and LMM model architectures have advanced, so have methods to allow these models to generalize beyond their pretraining distributions. Methods like instruction tuning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib86" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">86</span></a>, <a href="#bib.bib50" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">50</span></a>, <a href="#bib.bib5" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">5</span></a>, <a href="#bib.bib67" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">67</span></a>]</cite> have shown strong zero-shot generalization to some out-of-domain tasks, but forgetting remains an issue. One popular solution to this issue involves Parameter Efficient Fine-tuning (PEFT) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib28" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">28</span></a>]</cite>: finetuning either a set of soft prompt input tokens <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib41" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">41</span></a>, <a href="#bib.bib46" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">46</span></a>]</cite>, low-rank model weights <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib30" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">30</span></a>, <a href="#bib.bib14" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">14</span></a>, <a href="#bib.bib96" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">96</span></a>]</cite>, or a separate adapter from the main model <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib97" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">97</span></a>, <a href="#bib.bib18" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">18</span></a>, <a href="#bib.bib31" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">31</span></a>]</cite>.</p>
</div>
<div id="S2.p5" class="ltx_para">
<p id="S2.p5.1" class="ltx_p">Prompting methods are a well-explored area for adapting models without finetuning. LLM prompting includes zero-shot methods <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib36" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">36</span></a>, <a href="#bib.bib83" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">83</span></a>, <a href="#bib.bib81" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">81</span></a>]</cite>, few-shot and ICL methods <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib9" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">9</span></a>, <a href="#bib.bib55" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">55</span></a>, <a href="#bib.bib15" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">15</span></a>, <a href="#bib.bib53" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">53</span></a>]</cite>, expert prompting <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib91" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">91</span></a>]</cite>, and Chain-of-Thought (CoT) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib88" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">88</span></a>, <a href="#bib.bib98" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">98</span></a>]</cite>, with extensions like self-consistency <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib84" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">84</span></a>]</cite>, Tree-of-Thought (ToT) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib92" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">92</span></a>]</cite>, and Graph-of-Thought (GoT) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib8" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">8</span></a>, <a href="#bib.bib93" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">93</span></a>, <a href="#bib.bib40" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">40</span></a>]</cite> for more complex structures. Similar multimodal prompting methods exist for LMMs as well <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib85" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">85</span></a>, <a href="#bib.bib101" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">101</span></a>, <a href="#bib.bib99" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">99</span></a>, <a href="#bib.bib82" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">82</span></a>, <a href="#bib.bib56" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">56</span></a>]</cite>.</p>
</div>
<div id="S2.p6" class="ltx_para ltx_noindent">
<p id="S2.p6.1" class="ltx_p"><span id="S2.p6.1.1" class="ltx_text ltx_font_bold">Large Multimodal Models (LMMs)</span>. The state-of-the-art performance of LMMs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib50" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">50</span></a>, <a href="#bib.bib49" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">49</span></a>, <a href="#bib.bib2" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">2</span></a>, <a href="#bib.bib43" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">43</span></a>, <a href="#bib.bib13" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">13</span></a>, <a href="#bib.bib102" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">102</span></a>, <a href="#bib.bib94" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">94</span></a>, <a href="#bib.bib95" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">95</span></a>, <a href="#bib.bib6" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">6</span></a>, <a href="#bib.bib21" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">21</span></a>, <a href="#bib.bib17" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">17</span></a>]</cite> on multimodal tasks stems from combining LLMs’ reasoning capabilities  <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">3</span></a>, <a href="#bib.bib65" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">65</span></a>, <a href="#bib.bib12" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">12</span></a>, <a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">26</span></a>, <a href="#bib.bib76" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">76</span></a>, <a href="#bib.bib69" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">69</span></a>]</cite> with the perception abilities of vision models. LMMs’ generative reasoning also makes them more applicable to complex tasks than previous contrastive methods <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib64" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">64</span></a>, <a href="#bib.bib42" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">42</span></a>, <a href="#bib.bib43" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">43</span></a>]</cite>. Such tasks include visual question-answering <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib4" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">4</span></a>, <a href="#bib.bib32" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">32</span></a>, <a href="#bib.bib66" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">66</span></a>, <a href="#bib.bib54" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">54</span></a>, <a href="#bib.bib33" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">33</span></a>, <a href="#bib.bib24" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">24</span></a>, <a href="#bib.bib23" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">23</span></a>]</cite> as well as object identification and localization <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib58" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">58</span></a>, <a href="#bib.bib80" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">80</span></a>, <a href="#bib.bib48" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">48</span></a>, <a href="#bib.bib37" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">37</span></a>]</cite>. Visual Programmatic Models (VPMs) are another class of multimodal methods that makes use of in-context APIs code generation <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib74" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">74</span></a>, <a href="#bib.bib19" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">19</span></a>, <a href="#bib.bib22" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">22</span></a>, <a href="#bib.bib72" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">72</span></a>, <a href="#bib.bib68" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">68</span></a>, <a href="#bib.bib52" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">52</span></a>, <a href="#bib.bib70" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">70</span></a>, <a href="#bib.bib63" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">63</span></a>, <a href="#bib.bib90" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">90</span></a>]</cite>. However, context length limits both LMMs’ and VPMs’ ability to use multimodal prompting methods such as ICL <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib9" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">9</span></a>]</cite>. Another key challenge is that many LMMs are pre-trained on single text-image pair data. Recently, many LMM models now pretrain on interleaved text-image data <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">6</span></a>, <a href="#bib.bib73" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">73</span></a>, <a href="#bib.bib2" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">2</span></a>, <a href="#bib.bib39" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">39</span></a>, <a href="#bib.bib34" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">34</span></a>, <a href="#bib.bib16" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">16</span></a>, <a href="#bib.bib100" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">100</span></a>]</cite>, making effective multimodal ICL possible. In our work, MTV goes beyond simple few-shot multimodal ICL and scales to many-shot multimodal ICL.</p>
</div>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Multimodal Task Vectors</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">To address the challenge of performing many-shot multimodal in-context learning, we demonstrate the existence of MTV in LMMs and then leverage them for many-shot multimodal ICL. We begin by describing some background on multimodal ICL and task vectors (Section <a href="#S3.SS1" title="3.1 Preliminaries ‣ 3 Multimodal Task Vectors ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a>). We then introduce our three-step approach: (i) We calculate the mean activations of the attention heads from the many-shot multimodal ICL examples (Section <a href="#S3.SS2" title="3.2 Step 1: Calculate MTV Mean Activations ‣ 3 Multimodal Task Vectors ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>); (ii) We then extract the set of LMM attention heads locations that best align to the downstream task using an adapted version of the REINFORCE <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib89" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">89</span></a>]</cite> algorithm (Section <a href="#S3.SS3" title="3.3 Step 2: Extract MTV Attention Head Locations ‣ 3 Multimodal Task Vectors ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>); and (iii) We replace the calculated mean activation values into the LMM for a downstream task (Section <a href="#S3.SS4" title="3.4 Step 3: Multimodal Task Vector Application ‣ 3 Multimodal Task Vectors ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.4</span></a>). The method overview is also illustrated Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Preliminaries</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.4" class="ltx_p">In the multimodal in-context learning setting, an LMM learns a new task outlined by a set of multimodal examples. The input to the LMM would be outlined as follows:</p>
<table id="S3.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E1.m1.1" class="ltx_math_unparsed" alttext="I_{\mathrm{few}}=[(x_{1}:y_{1}),(x_{2}:y_{2}),\ldots,(x_{n}:y_{n}),Q]" display="block"><semantics id="S3.E1.m1.1a"><mrow id="S3.E1.m1.1b"><msub id="S3.E1.m1.1.1"><mi id="S3.E1.m1.1.1.2">I</mi><mi id="S3.E1.m1.1.1.3">few</mi></msub><mo id="S3.E1.m1.1.2">=</mo><mrow id="S3.E1.m1.1.3"><mo stretchy="false" id="S3.E1.m1.1.3.1">[</mo><mrow id="S3.E1.m1.1.3.2"><mo stretchy="false" id="S3.E1.m1.1.3.2.1">(</mo><msub id="S3.E1.m1.1.3.2.2"><mi id="S3.E1.m1.1.3.2.2.2">x</mi><mn id="S3.E1.m1.1.3.2.2.3">1</mn></msub><mo lspace="0.278em" rspace="0.278em" id="S3.E1.m1.1.3.2.3">:</mo><msub id="S3.E1.m1.1.3.2.4"><mi id="S3.E1.m1.1.3.2.4.2">y</mi><mn id="S3.E1.m1.1.3.2.4.3">1</mn></msub><mo stretchy="false" id="S3.E1.m1.1.3.2.5">)</mo></mrow><mo id="S3.E1.m1.1.3.3">,</mo><mrow id="S3.E1.m1.1.3.4"><mo stretchy="false" id="S3.E1.m1.1.3.4.1">(</mo><msub id="S3.E1.m1.1.3.4.2"><mi id="S3.E1.m1.1.3.4.2.2">x</mi><mn id="S3.E1.m1.1.3.4.2.3">2</mn></msub><mo lspace="0.278em" rspace="0.278em" id="S3.E1.m1.1.3.4.3">:</mo><msub id="S3.E1.m1.1.3.4.4"><mi id="S3.E1.m1.1.3.4.4.2">y</mi><mn id="S3.E1.m1.1.3.4.4.3">2</mn></msub><mo stretchy="false" id="S3.E1.m1.1.3.4.5">)</mo></mrow><mo id="S3.E1.m1.1.3.5">,</mo><mi mathvariant="normal" id="S3.E1.m1.1.3.6">…</mi><mo id="S3.E1.m1.1.3.7">,</mo><mrow id="S3.E1.m1.1.3.8"><mo stretchy="false" id="S3.E1.m1.1.3.8.1">(</mo><msub id="S3.E1.m1.1.3.8.2"><mi id="S3.E1.m1.1.3.8.2.2">x</mi><mi id="S3.E1.m1.1.3.8.2.3">n</mi></msub><mo lspace="0.278em" rspace="0.278em" id="S3.E1.m1.1.3.8.3">:</mo><msub id="S3.E1.m1.1.3.8.4"><mi id="S3.E1.m1.1.3.8.4.2">y</mi><mi id="S3.E1.m1.1.3.8.4.3">n</mi></msub><mo stretchy="false" id="S3.E1.m1.1.3.8.5">)</mo></mrow><mo id="S3.E1.m1.1.3.9">,</mo><mi id="S3.E1.m1.1.3.10">Q</mi><mo stretchy="false" id="S3.E1.m1.1.3.11">]</mo></mrow></mrow><annotation encoding="application/x-tex" id="S3.E1.m1.1c">I_{\mathrm{few}}=[(x_{1}:y_{1}),(x_{2}:y_{2}),\ldots,(x_{n}:y_{n}),Q]</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="S3.SS1.p1.3" class="ltx_p">where the model is prompted to answer a query <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="Q" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><mi id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml">Q</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><ci id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">𝑄</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">Q</annotation></semantics></math> given a set of input-output examples (each <math id="S3.SS1.p1.2.m2.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S3.SS1.p1.2.m2.1a"><msub id="S3.SS1.p1.2.m2.1.1" xref="S3.SS1.p1.2.m2.1.1.cmml"><mi id="S3.SS1.p1.2.m2.1.1.2" xref="S3.SS1.p1.2.m2.1.1.2.cmml">x</mi><mi id="S3.SS1.p1.2.m2.1.1.3" xref="S3.SS1.p1.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m2.1b"><apply id="S3.SS1.p1.2.m2.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.2.m2.1.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1">subscript</csymbol><ci id="S3.SS1.p1.2.m2.1.1.2.cmml" xref="S3.SS1.p1.2.m2.1.1.2">𝑥</ci><ci id="S3.SS1.p1.2.m2.1.1.3.cmml" xref="S3.SS1.p1.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m2.1c">x_{i}</annotation></semantics></math> being a multimodal input and each <math id="S3.SS1.p1.3.m3.1" class="ltx_Math" alttext="y_{i}" display="inline"><semantics id="S3.SS1.p1.3.m3.1a"><msub id="S3.SS1.p1.3.m3.1.1" xref="S3.SS1.p1.3.m3.1.1.cmml"><mi id="S3.SS1.p1.3.m3.1.1.2" xref="S3.SS1.p1.3.m3.1.1.2.cmml">y</mi><mi id="S3.SS1.p1.3.m3.1.1.3" xref="S3.SS1.p1.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.3.m3.1b"><apply id="S3.SS1.p1.3.m3.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.3.m3.1.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1">subscript</csymbol><ci id="S3.SS1.p1.3.m3.1.1.2.cmml" xref="S3.SS1.p1.3.m3.1.1.2">𝑦</ci><ci id="S3.SS1.p1.3.m3.1.1.3.cmml" xref="S3.SS1.p1.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.3.m3.1c">y_{i}</annotation></semantics></math> a text output).</p>
</div>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.1" class="ltx_p">We note that in-context examples are commonly passed sequentially to the LMM, necessarily restricting multimodal ICL to being small numbers of shots due to limited context length. Furthermore, the images require more tokens to embed, which means enabling many-shot ICL is even more challenging in the multimodal domain. To solve this, we utilize our method MTV—which are implicit representations in the model’s attention heads that encode a many-shot multimodal ICL task.</p>
</div>
<div id="S3.SS1.p3" class="ltx_para">
<p id="S3.SS1.p3.13" class="ltx_p">We start with a background on task vectors for some task <math id="S3.SS1.p3.1.m1.1" class="ltx_Math" alttext="j" display="inline"><semantics id="S3.SS1.p3.1.m1.1a"><mi id="S3.SS1.p3.1.m1.1.1" xref="S3.SS1.p3.1.m1.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.1.m1.1b"><ci id="S3.SS1.p3.1.m1.1.1.cmml" xref="S3.SS1.p3.1.m1.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.1.m1.1c">j</annotation></semantics></math>. Given a model <math id="S3.SS1.p3.2.m2.1" class="ltx_Math" alttext="F" display="inline"><semantics id="S3.SS1.p3.2.m2.1a"><mi id="S3.SS1.p3.2.m2.1.1" xref="S3.SS1.p3.2.m2.1.1.cmml">F</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.2.m2.1b"><ci id="S3.SS1.p3.2.m2.1.1.cmml" xref="S3.SS1.p3.2.m2.1.1">𝐹</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.2.m2.1c">F</annotation></semantics></math>, we denote the set of attention-head locations as <math id="S3.SS1.p3.3.m3.2" class="ltx_Math" alttext="\lambda=\{l\mid\forall l\in F\}" display="inline"><semantics id="S3.SS1.p3.3.m3.2a"><mrow id="S3.SS1.p3.3.m3.2.2" xref="S3.SS1.p3.3.m3.2.2.cmml"><mi id="S3.SS1.p3.3.m3.2.2.3" xref="S3.SS1.p3.3.m3.2.2.3.cmml">λ</mi><mo id="S3.SS1.p3.3.m3.2.2.2" xref="S3.SS1.p3.3.m3.2.2.2.cmml">=</mo><mrow id="S3.SS1.p3.3.m3.2.2.1.1" xref="S3.SS1.p3.3.m3.2.2.1.2.cmml"><mo stretchy="false" id="S3.SS1.p3.3.m3.2.2.1.1.2" xref="S3.SS1.p3.3.m3.2.2.1.2.1.cmml">{</mo><mi id="S3.SS1.p3.3.m3.1.1" xref="S3.SS1.p3.3.m3.1.1.cmml">l</mi><mo fence="true" lspace="0em" rspace="0.167em" id="S3.SS1.p3.3.m3.2.2.1.1.3" xref="S3.SS1.p3.3.m3.2.2.1.2.1.cmml">∣</mo><mrow id="S3.SS1.p3.3.m3.2.2.1.1.1" xref="S3.SS1.p3.3.m3.2.2.1.1.1.cmml"><mrow id="S3.SS1.p3.3.m3.2.2.1.1.1.2" xref="S3.SS1.p3.3.m3.2.2.1.1.1.2.cmml"><mo rspace="0.167em" id="S3.SS1.p3.3.m3.2.2.1.1.1.2.1" xref="S3.SS1.p3.3.m3.2.2.1.1.1.2.1.cmml">∀</mo><mi id="S3.SS1.p3.3.m3.2.2.1.1.1.2.2" xref="S3.SS1.p3.3.m3.2.2.1.1.1.2.2.cmml">l</mi></mrow><mo id="S3.SS1.p3.3.m3.2.2.1.1.1.1" xref="S3.SS1.p3.3.m3.2.2.1.1.1.1.cmml">∈</mo><mi id="S3.SS1.p3.3.m3.2.2.1.1.1.3" xref="S3.SS1.p3.3.m3.2.2.1.1.1.3.cmml">F</mi></mrow><mo stretchy="false" id="S3.SS1.p3.3.m3.2.2.1.1.4" xref="S3.SS1.p3.3.m3.2.2.1.2.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.3.m3.2b"><apply id="S3.SS1.p3.3.m3.2.2.cmml" xref="S3.SS1.p3.3.m3.2.2"><eq id="S3.SS1.p3.3.m3.2.2.2.cmml" xref="S3.SS1.p3.3.m3.2.2.2"></eq><ci id="S3.SS1.p3.3.m3.2.2.3.cmml" xref="S3.SS1.p3.3.m3.2.2.3">𝜆</ci><apply id="S3.SS1.p3.3.m3.2.2.1.2.cmml" xref="S3.SS1.p3.3.m3.2.2.1.1"><csymbol cd="latexml" id="S3.SS1.p3.3.m3.2.2.1.2.1.cmml" xref="S3.SS1.p3.3.m3.2.2.1.1.2">conditional-set</csymbol><ci id="S3.SS1.p3.3.m3.1.1.cmml" xref="S3.SS1.p3.3.m3.1.1">𝑙</ci><apply id="S3.SS1.p3.3.m3.2.2.1.1.1.cmml" xref="S3.SS1.p3.3.m3.2.2.1.1.1"><in id="S3.SS1.p3.3.m3.2.2.1.1.1.1.cmml" xref="S3.SS1.p3.3.m3.2.2.1.1.1.1"></in><apply id="S3.SS1.p3.3.m3.2.2.1.1.1.2.cmml" xref="S3.SS1.p3.3.m3.2.2.1.1.1.2"><csymbol cd="latexml" id="S3.SS1.p3.3.m3.2.2.1.1.1.2.1.cmml" xref="S3.SS1.p3.3.m3.2.2.1.1.1.2.1">for-all</csymbol><ci id="S3.SS1.p3.3.m3.2.2.1.1.1.2.2.cmml" xref="S3.SS1.p3.3.m3.2.2.1.1.1.2.2">𝑙</ci></apply><ci id="S3.SS1.p3.3.m3.2.2.1.1.1.3.cmml" xref="S3.SS1.p3.3.m3.2.2.1.1.1.3">𝐹</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.3.m3.2c">\lambda=\{l\mid\forall l\in F\}</annotation></semantics></math> where each location <math id="S3.SS1.p3.4.m4.1" class="ltx_Math" alttext="l" display="inline"><semantics id="S3.SS1.p3.4.m4.1a"><mi id="S3.SS1.p3.4.m4.1.1" xref="S3.SS1.p3.4.m4.1.1.cmml">l</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.4.m4.1b"><ci id="S3.SS1.p3.4.m4.1.1.cmml" xref="S3.SS1.p3.4.m4.1.1">𝑙</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.4.m4.1c">l</annotation></semantics></math> is indexed as <math id="S3.SS1.p3.5.m5.2" class="ltx_Math" alttext="l=(h,m)" display="inline"><semantics id="S3.SS1.p3.5.m5.2a"><mrow id="S3.SS1.p3.5.m5.2.3" xref="S3.SS1.p3.5.m5.2.3.cmml"><mi id="S3.SS1.p3.5.m5.2.3.2" xref="S3.SS1.p3.5.m5.2.3.2.cmml">l</mi><mo id="S3.SS1.p3.5.m5.2.3.1" xref="S3.SS1.p3.5.m5.2.3.1.cmml">=</mo><mrow id="S3.SS1.p3.5.m5.2.3.3.2" xref="S3.SS1.p3.5.m5.2.3.3.1.cmml"><mo stretchy="false" id="S3.SS1.p3.5.m5.2.3.3.2.1" xref="S3.SS1.p3.5.m5.2.3.3.1.cmml">(</mo><mi id="S3.SS1.p3.5.m5.1.1" xref="S3.SS1.p3.5.m5.1.1.cmml">h</mi><mo id="S3.SS1.p3.5.m5.2.3.3.2.2" xref="S3.SS1.p3.5.m5.2.3.3.1.cmml">,</mo><mi id="S3.SS1.p3.5.m5.2.2" xref="S3.SS1.p3.5.m5.2.2.cmml">m</mi><mo stretchy="false" id="S3.SS1.p3.5.m5.2.3.3.2.3" xref="S3.SS1.p3.5.m5.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.5.m5.2b"><apply id="S3.SS1.p3.5.m5.2.3.cmml" xref="S3.SS1.p3.5.m5.2.3"><eq id="S3.SS1.p3.5.m5.2.3.1.cmml" xref="S3.SS1.p3.5.m5.2.3.1"></eq><ci id="S3.SS1.p3.5.m5.2.3.2.cmml" xref="S3.SS1.p3.5.m5.2.3.2">𝑙</ci><interval closure="open" id="S3.SS1.p3.5.m5.2.3.3.1.cmml" xref="S3.SS1.p3.5.m5.2.3.3.2"><ci id="S3.SS1.p3.5.m5.1.1.cmml" xref="S3.SS1.p3.5.m5.1.1">ℎ</ci><ci id="S3.SS1.p3.5.m5.2.2.cmml" xref="S3.SS1.p3.5.m5.2.2">𝑚</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.5.m5.2c">l=(h,m)</annotation></semantics></math> for the <math id="S3.SS1.p3.6.m6.1" class="ltx_Math" alttext="h^{\mathrm{th}}" display="inline"><semantics id="S3.SS1.p3.6.m6.1a"><msup id="S3.SS1.p3.6.m6.1.1" xref="S3.SS1.p3.6.m6.1.1.cmml"><mi id="S3.SS1.p3.6.m6.1.1.2" xref="S3.SS1.p3.6.m6.1.1.2.cmml">h</mi><mi id="S3.SS1.p3.6.m6.1.1.3" xref="S3.SS1.p3.6.m6.1.1.3.cmml">th</mi></msup><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.6.m6.1b"><apply id="S3.SS1.p3.6.m6.1.1.cmml" xref="S3.SS1.p3.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.6.m6.1.1.1.cmml" xref="S3.SS1.p3.6.m6.1.1">superscript</csymbol><ci id="S3.SS1.p3.6.m6.1.1.2.cmml" xref="S3.SS1.p3.6.m6.1.1.2">ℎ</ci><ci id="S3.SS1.p3.6.m6.1.1.3.cmml" xref="S3.SS1.p3.6.m6.1.1.3">th</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.6.m6.1c">h^{\mathrm{th}}</annotation></semantics></math> layer and <math id="S3.SS1.p3.7.m7.1" class="ltx_Math" alttext="m^{\mathrm{th}}" display="inline"><semantics id="S3.SS1.p3.7.m7.1a"><msup id="S3.SS1.p3.7.m7.1.1" xref="S3.SS1.p3.7.m7.1.1.cmml"><mi id="S3.SS1.p3.7.m7.1.1.2" xref="S3.SS1.p3.7.m7.1.1.2.cmml">m</mi><mi id="S3.SS1.p3.7.m7.1.1.3" xref="S3.SS1.p3.7.m7.1.1.3.cmml">th</mi></msup><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.7.m7.1b"><apply id="S3.SS1.p3.7.m7.1.1.cmml" xref="S3.SS1.p3.7.m7.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.7.m7.1.1.1.cmml" xref="S3.SS1.p3.7.m7.1.1">superscript</csymbol><ci id="S3.SS1.p3.7.m7.1.1.2.cmml" xref="S3.SS1.p3.7.m7.1.1.2">𝑚</ci><ci id="S3.SS1.p3.7.m7.1.1.3.cmml" xref="S3.SS1.p3.7.m7.1.1.3">th</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.7.m7.1c">m^{\mathrm{th}}</annotation></semantics></math> attention head. We define the task vectors as follows: (1) the task vector <span id="S3.SS1.p3.13.1" class="ltx_text ltx_font_bold">values</span> <math id="S3.SS1.p3.8.m8.1" class="ltx_Math" alttext="\mu_{j}" display="inline"><semantics id="S3.SS1.p3.8.m8.1a"><msub id="S3.SS1.p3.8.m8.1.1" xref="S3.SS1.p3.8.m8.1.1.cmml"><mi id="S3.SS1.p3.8.m8.1.1.2" xref="S3.SS1.p3.8.m8.1.1.2.cmml">μ</mi><mi id="S3.SS1.p3.8.m8.1.1.3" xref="S3.SS1.p3.8.m8.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.8.m8.1b"><apply id="S3.SS1.p3.8.m8.1.1.cmml" xref="S3.SS1.p3.8.m8.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.8.m8.1.1.1.cmml" xref="S3.SS1.p3.8.m8.1.1">subscript</csymbol><ci id="S3.SS1.p3.8.m8.1.1.2.cmml" xref="S3.SS1.p3.8.m8.1.1.2">𝜇</ci><ci id="S3.SS1.p3.8.m8.1.1.3.cmml" xref="S3.SS1.p3.8.m8.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.8.m8.1c">\mu_{j}</annotation></semantics></math> are a subset of mean activations produced by the attention heads of <math id="S3.SS1.p3.9.m9.1" class="ltx_Math" alttext="F" display="inline"><semantics id="S3.SS1.p3.9.m9.1a"><mi id="S3.SS1.p3.9.m9.1.1" xref="S3.SS1.p3.9.m9.1.1.cmml">F</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.9.m9.1b"><ci id="S3.SS1.p3.9.m9.1.1.cmml" xref="S3.SS1.p3.9.m9.1.1">𝐹</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.9.m9.1c">F</annotation></semantics></math> given examples of a task, and (2) the task vector <span id="S3.SS1.p3.13.2" class="ltx_text ltx_font_bold">locations</span> <math id="S3.SS1.p3.10.m10.1" class="ltx_Math" alttext="\lambda_{j}" display="inline"><semantics id="S3.SS1.p3.10.m10.1a"><msub id="S3.SS1.p3.10.m10.1.1" xref="S3.SS1.p3.10.m10.1.1.cmml"><mi id="S3.SS1.p3.10.m10.1.1.2" xref="S3.SS1.p3.10.m10.1.1.2.cmml">λ</mi><mi id="S3.SS1.p3.10.m10.1.1.3" xref="S3.SS1.p3.10.m10.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.10.m10.1b"><apply id="S3.SS1.p3.10.m10.1.1.cmml" xref="S3.SS1.p3.10.m10.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.10.m10.1.1.1.cmml" xref="S3.SS1.p3.10.m10.1.1">subscript</csymbol><ci id="S3.SS1.p3.10.m10.1.1.2.cmml" xref="S3.SS1.p3.10.m10.1.1.2">𝜆</ci><ci id="S3.SS1.p3.10.m10.1.1.3.cmml" xref="S3.SS1.p3.10.m10.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.10.m10.1c">\lambda_{j}</annotation></semantics></math>, which denotes a subset of the attention head indices per task. Thus, the task vector is <math id="S3.SS1.p3.11.m11.2" class="ltx_Math" alttext="(\mu_{j},\lambda_{j})" display="inline"><semantics id="S3.SS1.p3.11.m11.2a"><mrow id="S3.SS1.p3.11.m11.2.2.2" xref="S3.SS1.p3.11.m11.2.2.3.cmml"><mo stretchy="false" id="S3.SS1.p3.11.m11.2.2.2.3" xref="S3.SS1.p3.11.m11.2.2.3.cmml">(</mo><msub id="S3.SS1.p3.11.m11.1.1.1.1" xref="S3.SS1.p3.11.m11.1.1.1.1.cmml"><mi id="S3.SS1.p3.11.m11.1.1.1.1.2" xref="S3.SS1.p3.11.m11.1.1.1.1.2.cmml">μ</mi><mi id="S3.SS1.p3.11.m11.1.1.1.1.3" xref="S3.SS1.p3.11.m11.1.1.1.1.3.cmml">j</mi></msub><mo id="S3.SS1.p3.11.m11.2.2.2.4" xref="S3.SS1.p3.11.m11.2.2.3.cmml">,</mo><msub id="S3.SS1.p3.11.m11.2.2.2.2" xref="S3.SS1.p3.11.m11.2.2.2.2.cmml"><mi id="S3.SS1.p3.11.m11.2.2.2.2.2" xref="S3.SS1.p3.11.m11.2.2.2.2.2.cmml">λ</mi><mi id="S3.SS1.p3.11.m11.2.2.2.2.3" xref="S3.SS1.p3.11.m11.2.2.2.2.3.cmml">j</mi></msub><mo stretchy="false" id="S3.SS1.p3.11.m11.2.2.2.5" xref="S3.SS1.p3.11.m11.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.11.m11.2b"><interval closure="open" id="S3.SS1.p3.11.m11.2.2.3.cmml" xref="S3.SS1.p3.11.m11.2.2.2"><apply id="S3.SS1.p3.11.m11.1.1.1.1.cmml" xref="S3.SS1.p3.11.m11.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.11.m11.1.1.1.1.1.cmml" xref="S3.SS1.p3.11.m11.1.1.1.1">subscript</csymbol><ci id="S3.SS1.p3.11.m11.1.1.1.1.2.cmml" xref="S3.SS1.p3.11.m11.1.1.1.1.2">𝜇</ci><ci id="S3.SS1.p3.11.m11.1.1.1.1.3.cmml" xref="S3.SS1.p3.11.m11.1.1.1.1.3">𝑗</ci></apply><apply id="S3.SS1.p3.11.m11.2.2.2.2.cmml" xref="S3.SS1.p3.11.m11.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS1.p3.11.m11.2.2.2.2.1.cmml" xref="S3.SS1.p3.11.m11.2.2.2.2">subscript</csymbol><ci id="S3.SS1.p3.11.m11.2.2.2.2.2.cmml" xref="S3.SS1.p3.11.m11.2.2.2.2.2">𝜆</ci><ci id="S3.SS1.p3.11.m11.2.2.2.2.3.cmml" xref="S3.SS1.p3.11.m11.2.2.2.2.3">𝑗</ci></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.11.m11.2c">(\mu_{j},\lambda_{j})</annotation></semantics></math>. For inference, <math id="S3.SS1.p3.12.m12.1" class="ltx_Math" alttext="\mu_{j}" display="inline"><semantics id="S3.SS1.p3.12.m12.1a"><msub id="S3.SS1.p3.12.m12.1.1" xref="S3.SS1.p3.12.m12.1.1.cmml"><mi id="S3.SS1.p3.12.m12.1.1.2" xref="S3.SS1.p3.12.m12.1.1.2.cmml">μ</mi><mi id="S3.SS1.p3.12.m12.1.1.3" xref="S3.SS1.p3.12.m12.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.12.m12.1b"><apply id="S3.SS1.p3.12.m12.1.1.cmml" xref="S3.SS1.p3.12.m12.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.12.m12.1.1.1.cmml" xref="S3.SS1.p3.12.m12.1.1">subscript</csymbol><ci id="S3.SS1.p3.12.m12.1.1.2.cmml" xref="S3.SS1.p3.12.m12.1.1.2">𝜇</ci><ci id="S3.SS1.p3.12.m12.1.1.3.cmml" xref="S3.SS1.p3.12.m12.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.12.m12.1c">\mu_{j}</annotation></semantics></math> replaces the activation values of the heads in <math id="S3.SS1.p3.13.m13.1" class="ltx_Math" alttext="\lambda_{j}" display="inline"><semantics id="S3.SS1.p3.13.m13.1a"><msub id="S3.SS1.p3.13.m13.1.1" xref="S3.SS1.p3.13.m13.1.1.cmml"><mi id="S3.SS1.p3.13.m13.1.1.2" xref="S3.SS1.p3.13.m13.1.1.2.cmml">λ</mi><mi id="S3.SS1.p3.13.m13.1.1.3" xref="S3.SS1.p3.13.m13.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.13.m13.1b"><apply id="S3.SS1.p3.13.m13.1.1.cmml" xref="S3.SS1.p3.13.m13.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.13.m13.1.1.1.cmml" xref="S3.SS1.p3.13.m13.1.1">subscript</csymbol><ci id="S3.SS1.p3.13.m13.1.1.2.cmml" xref="S3.SS1.p3.13.m13.1.1.2">𝜆</ci><ci id="S3.SS1.p3.13.m13.1.1.3.cmml" xref="S3.SS1.p3.13.m13.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.13.m13.1c">\lambda_{j}</annotation></semantics></math>.</p>
</div>
<div id="S3.SS1.p4" class="ltx_para">
<p id="S3.SS1.p4.6" class="ltx_p">In prior work <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib25" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">25</span></a>, <a href="#bib.bib27" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">27</span></a>, <a href="#bib.bib79" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">79</span></a>]</cite>, the calculation of the mean activations <math id="S3.SS1.p4.1.m1.1" class="ltx_Math" alttext="\mu_{j}" display="inline"><semantics id="S3.SS1.p4.1.m1.1a"><msub id="S3.SS1.p4.1.m1.1.1" xref="S3.SS1.p4.1.m1.1.1.cmml"><mi id="S3.SS1.p4.1.m1.1.1.2" xref="S3.SS1.p4.1.m1.1.1.2.cmml">μ</mi><mi id="S3.SS1.p4.1.m1.1.1.3" xref="S3.SS1.p4.1.m1.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.1.m1.1b"><apply id="S3.SS1.p4.1.m1.1.1.cmml" xref="S3.SS1.p4.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p4.1.m1.1.1.1.cmml" xref="S3.SS1.p4.1.m1.1.1">subscript</csymbol><ci id="S3.SS1.p4.1.m1.1.1.2.cmml" xref="S3.SS1.p4.1.m1.1.1.2">𝜇</ci><ci id="S3.SS1.p4.1.m1.1.1.3.cmml" xref="S3.SS1.p4.1.m1.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.1.m1.1c">\mu_{j}</annotation></semantics></math> and the extraction of the attention-head locations <math id="S3.SS1.p4.2.m2.1" class="ltx_Math" alttext="\lambda_{j}" display="inline"><semantics id="S3.SS1.p4.2.m2.1a"><msub id="S3.SS1.p4.2.m2.1.1" xref="S3.SS1.p4.2.m2.1.1.cmml"><mi id="S3.SS1.p4.2.m2.1.1.2" xref="S3.SS1.p4.2.m2.1.1.2.cmml">λ</mi><mi id="S3.SS1.p4.2.m2.1.1.3" xref="S3.SS1.p4.2.m2.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.2.m2.1b"><apply id="S3.SS1.p4.2.m2.1.1.cmml" xref="S3.SS1.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p4.2.m2.1.1.1.cmml" xref="S3.SS1.p4.2.m2.1.1">subscript</csymbol><ci id="S3.SS1.p4.2.m2.1.1.2.cmml" xref="S3.SS1.p4.2.m2.1.1.2">𝜆</ci><ci id="S3.SS1.p4.2.m2.1.1.3.cmml" xref="S3.SS1.p4.2.m2.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.2.m2.1c">\lambda_{j}</annotation></semantics></math> are used together to extract the task vector. Interestingly, we find that these two steps should be decoupled in order to better align with the downstream task. In our work, we calculate the mean activations <math id="S3.SS1.p4.3.m3.1" class="ltx_Math" alttext="\mu_{j}" display="inline"><semantics id="S3.SS1.p4.3.m3.1a"><msub id="S3.SS1.p4.3.m3.1.1" xref="S3.SS1.p4.3.m3.1.1.cmml"><mi id="S3.SS1.p4.3.m3.1.1.2" xref="S3.SS1.p4.3.m3.1.1.2.cmml">μ</mi><mi id="S3.SS1.p4.3.m3.1.1.3" xref="S3.SS1.p4.3.m3.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.3.m3.1b"><apply id="S3.SS1.p4.3.m3.1.1.cmml" xref="S3.SS1.p4.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS1.p4.3.m3.1.1.1.cmml" xref="S3.SS1.p4.3.m3.1.1">subscript</csymbol><ci id="S3.SS1.p4.3.m3.1.1.2.cmml" xref="S3.SS1.p4.3.m3.1.1.2">𝜇</ci><ci id="S3.SS1.p4.3.m3.1.1.3.cmml" xref="S3.SS1.p4.3.m3.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.3.m3.1c">\mu_{j}</annotation></semantics></math> corresponding to the last token specifically to encode a dataset of many-shot multimodal ICL examples by averaging them across multiple inference calls. However, the downstream task may not always be in the same ICL format as the many-shot examples (e.g., the downstream task uses a different number of shots or is zero-shot). To solve this, we use a separate set of examples that are of the exact format of the downstream task to align the extracted attention-head locations <math id="S3.SS1.p4.4.m4.1" class="ltx_Math" alttext="\lambda_{j}" display="inline"><semantics id="S3.SS1.p4.4.m4.1a"><msub id="S3.SS1.p4.4.m4.1.1" xref="S3.SS1.p4.4.m4.1.1.cmml"><mi id="S3.SS1.p4.4.m4.1.1.2" xref="S3.SS1.p4.4.m4.1.1.2.cmml">λ</mi><mi id="S3.SS1.p4.4.m4.1.1.3" xref="S3.SS1.p4.4.m4.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.4.m4.1b"><apply id="S3.SS1.p4.4.m4.1.1.cmml" xref="S3.SS1.p4.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS1.p4.4.m4.1.1.1.cmml" xref="S3.SS1.p4.4.m4.1.1">subscript</csymbol><ci id="S3.SS1.p4.4.m4.1.1.2.cmml" xref="S3.SS1.p4.4.m4.1.1.2">𝜆</ci><ci id="S3.SS1.p4.4.m4.1.1.3.cmml" xref="S3.SS1.p4.4.m4.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.4.m4.1c">\lambda_{j}</annotation></semantics></math> with the inference task. This separation of responsibilities, wherein <math id="S3.SS1.p4.5.m5.1" class="ltx_Math" alttext="\mu_{j}" display="inline"><semantics id="S3.SS1.p4.5.m5.1a"><msub id="S3.SS1.p4.5.m5.1.1" xref="S3.SS1.p4.5.m5.1.1.cmml"><mi id="S3.SS1.p4.5.m5.1.1.2" xref="S3.SS1.p4.5.m5.1.1.2.cmml">μ</mi><mi id="S3.SS1.p4.5.m5.1.1.3" xref="S3.SS1.p4.5.m5.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.5.m5.1b"><apply id="S3.SS1.p4.5.m5.1.1.cmml" xref="S3.SS1.p4.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS1.p4.5.m5.1.1.1.cmml" xref="S3.SS1.p4.5.m5.1.1">subscript</csymbol><ci id="S3.SS1.p4.5.m5.1.1.2.cmml" xref="S3.SS1.p4.5.m5.1.1.2">𝜇</ci><ci id="S3.SS1.p4.5.m5.1.1.3.cmml" xref="S3.SS1.p4.5.m5.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.5.m5.1c">\mu_{j}</annotation></semantics></math> captures the essential information from the many-shot examples and <math id="S3.SS1.p4.6.m6.1" class="ltx_Math" alttext="\lambda_{j}" display="inline"><semantics id="S3.SS1.p4.6.m6.1a"><msub id="S3.SS1.p4.6.m6.1.1" xref="S3.SS1.p4.6.m6.1.1.cmml"><mi id="S3.SS1.p4.6.m6.1.1.2" xref="S3.SS1.p4.6.m6.1.1.2.cmml">λ</mi><mi id="S3.SS1.p4.6.m6.1.1.3" xref="S3.SS1.p4.6.m6.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.6.m6.1b"><apply id="S3.SS1.p4.6.m6.1.1.cmml" xref="S3.SS1.p4.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS1.p4.6.m6.1.1.1.cmml" xref="S3.SS1.p4.6.m6.1.1">subscript</csymbol><ci id="S3.SS1.p4.6.m6.1.1.2.cmml" xref="S3.SS1.p4.6.m6.1.1.2">𝜆</ci><ci id="S3.SS1.p4.6.m6.1.1.3.cmml" xref="S3.SS1.p4.6.m6.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.6.m6.1c">\lambda_{j}</annotation></semantics></math> identifies the specific attention head locations for the downstream task, optimizes the utilization of the encoded information at relevant locations within the model.</p>
</div>
<div id="S3.SS1.p5" class="ltx_para">
<p id="S3.SS1.p5.1" class="ltx_p">Our approach to finding Multimodal Task Vectors (MTV) <math id="S3.SS1.p5.1.m1.2" class="ltx_Math" alttext="(\mu^{\mathrm{MTV}}_{j},\lambda^{\mathrm{MTV}}_{j})" display="inline"><semantics id="S3.SS1.p5.1.m1.2a"><mrow id="S3.SS1.p5.1.m1.2.2.2" xref="S3.SS1.p5.1.m1.2.2.3.cmml"><mo stretchy="false" id="S3.SS1.p5.1.m1.2.2.2.3" xref="S3.SS1.p5.1.m1.2.2.3.cmml">(</mo><msubsup id="S3.SS1.p5.1.m1.1.1.1.1" xref="S3.SS1.p5.1.m1.1.1.1.1.cmml"><mi id="S3.SS1.p5.1.m1.1.1.1.1.2.2" xref="S3.SS1.p5.1.m1.1.1.1.1.2.2.cmml">μ</mi><mi id="S3.SS1.p5.1.m1.1.1.1.1.3" xref="S3.SS1.p5.1.m1.1.1.1.1.3.cmml">j</mi><mi id="S3.SS1.p5.1.m1.1.1.1.1.2.3" xref="S3.SS1.p5.1.m1.1.1.1.1.2.3.cmml">MTV</mi></msubsup><mo id="S3.SS1.p5.1.m1.2.2.2.4" xref="S3.SS1.p5.1.m1.2.2.3.cmml">,</mo><msubsup id="S3.SS1.p5.1.m1.2.2.2.2" xref="S3.SS1.p5.1.m1.2.2.2.2.cmml"><mi id="S3.SS1.p5.1.m1.2.2.2.2.2.2" xref="S3.SS1.p5.1.m1.2.2.2.2.2.2.cmml">λ</mi><mi id="S3.SS1.p5.1.m1.2.2.2.2.3" xref="S3.SS1.p5.1.m1.2.2.2.2.3.cmml">j</mi><mi id="S3.SS1.p5.1.m1.2.2.2.2.2.3" xref="S3.SS1.p5.1.m1.2.2.2.2.2.3.cmml">MTV</mi></msubsup><mo stretchy="false" id="S3.SS1.p5.1.m1.2.2.2.5" xref="S3.SS1.p5.1.m1.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p5.1.m1.2b"><interval closure="open" id="S3.SS1.p5.1.m1.2.2.3.cmml" xref="S3.SS1.p5.1.m1.2.2.2"><apply id="S3.SS1.p5.1.m1.1.1.1.1.cmml" xref="S3.SS1.p5.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p5.1.m1.1.1.1.1.1.cmml" xref="S3.SS1.p5.1.m1.1.1.1.1">subscript</csymbol><apply id="S3.SS1.p5.1.m1.1.1.1.1.2.cmml" xref="S3.SS1.p5.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p5.1.m1.1.1.1.1.2.1.cmml" xref="S3.SS1.p5.1.m1.1.1.1.1">superscript</csymbol><ci id="S3.SS1.p5.1.m1.1.1.1.1.2.2.cmml" xref="S3.SS1.p5.1.m1.1.1.1.1.2.2">𝜇</ci><ci id="S3.SS1.p5.1.m1.1.1.1.1.2.3.cmml" xref="S3.SS1.p5.1.m1.1.1.1.1.2.3">MTV</ci></apply><ci id="S3.SS1.p5.1.m1.1.1.1.1.3.cmml" xref="S3.SS1.p5.1.m1.1.1.1.1.3">𝑗</ci></apply><apply id="S3.SS1.p5.1.m1.2.2.2.2.cmml" xref="S3.SS1.p5.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS1.p5.1.m1.2.2.2.2.1.cmml" xref="S3.SS1.p5.1.m1.2.2.2.2">subscript</csymbol><apply id="S3.SS1.p5.1.m1.2.2.2.2.2.cmml" xref="S3.SS1.p5.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS1.p5.1.m1.2.2.2.2.2.1.cmml" xref="S3.SS1.p5.1.m1.2.2.2.2">superscript</csymbol><ci id="S3.SS1.p5.1.m1.2.2.2.2.2.2.cmml" xref="S3.SS1.p5.1.m1.2.2.2.2.2.2">𝜆</ci><ci id="S3.SS1.p5.1.m1.2.2.2.2.2.3.cmml" xref="S3.SS1.p5.1.m1.2.2.2.2.2.3">MTV</ci></apply><ci id="S3.SS1.p5.1.m1.2.2.2.2.3.cmml" xref="S3.SS1.p5.1.m1.2.2.2.2.3">𝑗</ci></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p5.1.m1.2c">(\mu^{\mathrm{MTV}}_{j},\lambda^{\mathrm{MTV}}_{j})</annotation></semantics></math> allows LMMs to actually leverage many-shot multimodal ICL examples for complex vision-language tasks without being limited by context length. We proceed by first describing how to calculate the mean activations.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Step 1: Calculate MTV Mean Activations</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">The ultimate objective of many-shot multimodal ICL is to use a large number of input-output examples when solving a task <math id="S3.SS2.p1.1.m1.1" class="ltx_Math" alttext="j" display="inline"><semantics id="S3.SS2.p1.1.m1.1a"><mi id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.1b"><ci id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.1c">j</annotation></semantics></math>. However, it is not trivial to get the LMM to see more examples during inference time than its context length allows.</p>
</div>
<div id="S3.SS2.p2" class="ltx_para">
<p id="S3.SS2.p2.8" class="ltx_p">To address this issue, we pass a few-shot input <math id="S3.SS2.p2.1.m1.1" class="ltx_Math" alttext="I_{t}" display="inline"><semantics id="S3.SS2.p2.1.m1.1a"><msub id="S3.SS2.p2.1.m1.1.1" xref="S3.SS2.p2.1.m1.1.1.cmml"><mi id="S3.SS2.p2.1.m1.1.1.2" xref="S3.SS2.p2.1.m1.1.1.2.cmml">I</mi><mi id="S3.SS2.p2.1.m1.1.1.3" xref="S3.SS2.p2.1.m1.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.1.m1.1b"><apply id="S3.SS2.p2.1.m1.1.1.cmml" xref="S3.SS2.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.1.m1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.1.1">subscript</csymbol><ci id="S3.SS2.p2.1.m1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.1.1.2">𝐼</ci><ci id="S3.SS2.p2.1.m1.1.1.3.cmml" xref="S3.SS2.p2.1.m1.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.1.m1.1c">I_{t}</annotation></semantics></math> for each inference call <math id="S3.SS2.p2.2.m2.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S3.SS2.p2.2.m2.1a"><mi id="S3.SS2.p2.2.m2.1.1" xref="S3.SS2.p2.2.m2.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.2.m2.1b"><ci id="S3.SS2.p2.2.m2.1.1.cmml" xref="S3.SS2.p2.2.m2.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.2.m2.1c">t</annotation></semantics></math> for a total of <math id="S3.SS2.p2.3.m3.1" class="ltx_Math" alttext="T&gt;1" display="inline"><semantics id="S3.SS2.p2.3.m3.1a"><mrow id="S3.SS2.p2.3.m3.1.1" xref="S3.SS2.p2.3.m3.1.1.cmml"><mi id="S3.SS2.p2.3.m3.1.1.2" xref="S3.SS2.p2.3.m3.1.1.2.cmml">T</mi><mo id="S3.SS2.p2.3.m3.1.1.1" xref="S3.SS2.p2.3.m3.1.1.1.cmml">&gt;</mo><mn id="S3.SS2.p2.3.m3.1.1.3" xref="S3.SS2.p2.3.m3.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.3.m3.1b"><apply id="S3.SS2.p2.3.m3.1.1.cmml" xref="S3.SS2.p2.3.m3.1.1"><gt id="S3.SS2.p2.3.m3.1.1.1.cmml" xref="S3.SS2.p2.3.m3.1.1.1"></gt><ci id="S3.SS2.p2.3.m3.1.1.2.cmml" xref="S3.SS2.p2.3.m3.1.1.2">𝑇</ci><cn type="integer" id="S3.SS2.p2.3.m3.1.1.3.cmml" xref="S3.SS2.p2.3.m3.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.3.m3.1c">T&gt;1</annotation></semantics></math> inference calls. Each <math id="S3.SS2.p2.4.m4.1" class="ltx_Math" alttext="I_{t}" display="inline"><semantics id="S3.SS2.p2.4.m4.1a"><msub id="S3.SS2.p2.4.m4.1.1" xref="S3.SS2.p2.4.m4.1.1.cmml"><mi id="S3.SS2.p2.4.m4.1.1.2" xref="S3.SS2.p2.4.m4.1.1.2.cmml">I</mi><mi id="S3.SS2.p2.4.m4.1.1.3" xref="S3.SS2.p2.4.m4.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.4.m4.1b"><apply id="S3.SS2.p2.4.m4.1.1.cmml" xref="S3.SS2.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.4.m4.1.1.1.cmml" xref="S3.SS2.p2.4.m4.1.1">subscript</csymbol><ci id="S3.SS2.p2.4.m4.1.1.2.cmml" xref="S3.SS2.p2.4.m4.1.1.2">𝐼</ci><ci id="S3.SS2.p2.4.m4.1.1.3.cmml" xref="S3.SS2.p2.4.m4.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.4.m4.1c">I_{t}</annotation></semantics></math> consists of <math id="S3.SS2.p2.5.m5.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S3.SS2.p2.5.m5.1a"><mi id="S3.SS2.p2.5.m5.1.1" xref="S3.SS2.p2.5.m5.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.5.m5.1b"><ci id="S3.SS2.p2.5.m5.1.1.cmml" xref="S3.SS2.p2.5.m5.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.5.m5.1c">N</annotation></semantics></math> shots (where <math id="S3.SS2.p2.6.m6.1" class="ltx_Math" alttext="N&gt;1" display="inline"><semantics id="S3.SS2.p2.6.m6.1a"><mrow id="S3.SS2.p2.6.m6.1.1" xref="S3.SS2.p2.6.m6.1.1.cmml"><mi id="S3.SS2.p2.6.m6.1.1.2" xref="S3.SS2.p2.6.m6.1.1.2.cmml">N</mi><mo id="S3.SS2.p2.6.m6.1.1.1" xref="S3.SS2.p2.6.m6.1.1.1.cmml">&gt;</mo><mn id="S3.SS2.p2.6.m6.1.1.3" xref="S3.SS2.p2.6.m6.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.6.m6.1b"><apply id="S3.SS2.p2.6.m6.1.1.cmml" xref="S3.SS2.p2.6.m6.1.1"><gt id="S3.SS2.p2.6.m6.1.1.1.cmml" xref="S3.SS2.p2.6.m6.1.1.1"></gt><ci id="S3.SS2.p2.6.m6.1.1.2.cmml" xref="S3.SS2.p2.6.m6.1.1.2">𝑁</ci><cn type="integer" id="S3.SS2.p2.6.m6.1.1.3.cmml" xref="S3.SS2.p2.6.m6.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.6.m6.1c">N&gt;1</annotation></semantics></math>) of multimodal in-context examples in the form of randomly-selected input-output response pairs <math id="S3.SS2.p2.7.m7.1" class="ltx_Math" alttext="(x_{t}:y_{t})" display="inline"><semantics id="S3.SS2.p2.7.m7.1a"><mrow id="S3.SS2.p2.7.m7.1.1.1" xref="S3.SS2.p2.7.m7.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p2.7.m7.1.1.1.2" xref="S3.SS2.p2.7.m7.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p2.7.m7.1.1.1.1" xref="S3.SS2.p2.7.m7.1.1.1.1.cmml"><msub id="S3.SS2.p2.7.m7.1.1.1.1.2" xref="S3.SS2.p2.7.m7.1.1.1.1.2.cmml"><mi id="S3.SS2.p2.7.m7.1.1.1.1.2.2" xref="S3.SS2.p2.7.m7.1.1.1.1.2.2.cmml">x</mi><mi id="S3.SS2.p2.7.m7.1.1.1.1.2.3" xref="S3.SS2.p2.7.m7.1.1.1.1.2.3.cmml">t</mi></msub><mo lspace="0.278em" rspace="0.278em" id="S3.SS2.p2.7.m7.1.1.1.1.1" xref="S3.SS2.p2.7.m7.1.1.1.1.1.cmml">:</mo><msub id="S3.SS2.p2.7.m7.1.1.1.1.3" xref="S3.SS2.p2.7.m7.1.1.1.1.3.cmml"><mi id="S3.SS2.p2.7.m7.1.1.1.1.3.2" xref="S3.SS2.p2.7.m7.1.1.1.1.3.2.cmml">y</mi><mi id="S3.SS2.p2.7.m7.1.1.1.1.3.3" xref="S3.SS2.p2.7.m7.1.1.1.1.3.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="S3.SS2.p2.7.m7.1.1.1.3" xref="S3.SS2.p2.7.m7.1.1.1.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.7.m7.1b"><apply id="S3.SS2.p2.7.m7.1.1.1.1.cmml" xref="S3.SS2.p2.7.m7.1.1.1"><ci id="S3.SS2.p2.7.m7.1.1.1.1.1.cmml" xref="S3.SS2.p2.7.m7.1.1.1.1.1">:</ci><apply id="S3.SS2.p2.7.m7.1.1.1.1.2.cmml" xref="S3.SS2.p2.7.m7.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p2.7.m7.1.1.1.1.2.1.cmml" xref="S3.SS2.p2.7.m7.1.1.1.1.2">subscript</csymbol><ci id="S3.SS2.p2.7.m7.1.1.1.1.2.2.cmml" xref="S3.SS2.p2.7.m7.1.1.1.1.2.2">𝑥</ci><ci id="S3.SS2.p2.7.m7.1.1.1.1.2.3.cmml" xref="S3.SS2.p2.7.m7.1.1.1.1.2.3">𝑡</ci></apply><apply id="S3.SS2.p2.7.m7.1.1.1.1.3.cmml" xref="S3.SS2.p2.7.m7.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p2.7.m7.1.1.1.1.3.1.cmml" xref="S3.SS2.p2.7.m7.1.1.1.1.3">subscript</csymbol><ci id="S3.SS2.p2.7.m7.1.1.1.1.3.2.cmml" xref="S3.SS2.p2.7.m7.1.1.1.1.3.2">𝑦</ci><ci id="S3.SS2.p2.7.m7.1.1.1.1.3.3.cmml" xref="S3.SS2.p2.7.m7.1.1.1.1.3.3">𝑡</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.7.m7.1c">(x_{t}:y_{t})</annotation></semantics></math>, and <math id="S3.SS2.p2.8.m8.1" class="ltx_Math" alttext="Q_{t}" display="inline"><semantics id="S3.SS2.p2.8.m8.1a"><msub id="S3.SS2.p2.8.m8.1.1" xref="S3.SS2.p2.8.m8.1.1.cmml"><mi id="S3.SS2.p2.8.m8.1.1.2" xref="S3.SS2.p2.8.m8.1.1.2.cmml">Q</mi><mi id="S3.SS2.p2.8.m8.1.1.3" xref="S3.SS2.p2.8.m8.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.8.m8.1b"><apply id="S3.SS2.p2.8.m8.1.1.cmml" xref="S3.SS2.p2.8.m8.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.8.m8.1.1.1.cmml" xref="S3.SS2.p2.8.m8.1.1">subscript</csymbol><ci id="S3.SS2.p2.8.m8.1.1.2.cmml" xref="S3.SS2.p2.8.m8.1.1.2">𝑄</ci><ci id="S3.SS2.p2.8.m8.1.1.3.cmml" xref="S3.SS2.p2.8.m8.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.8.m8.1c">Q_{t}</annotation></semantics></math>, which is the query to be answered by the LMM in that iteration.</p>
<table id="S3.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E2.m1.1" class="ltx_math_unparsed" alttext="I_{t}=[(x_{1}:y_{1}),(x_{2}:y_{2}),\ldots,(x_{N}:y_{N}),Q_{t}]" display="block"><semantics id="S3.E2.m1.1a"><mrow id="S3.E2.m1.1b"><msub id="S3.E2.m1.1.1"><mi id="S3.E2.m1.1.1.2">I</mi><mi id="S3.E2.m1.1.1.3">t</mi></msub><mo id="S3.E2.m1.1.2">=</mo><mrow id="S3.E2.m1.1.3"><mo stretchy="false" id="S3.E2.m1.1.3.1">[</mo><mrow id="S3.E2.m1.1.3.2"><mo stretchy="false" id="S3.E2.m1.1.3.2.1">(</mo><msub id="S3.E2.m1.1.3.2.2"><mi id="S3.E2.m1.1.3.2.2.2">x</mi><mn id="S3.E2.m1.1.3.2.2.3">1</mn></msub><mo lspace="0.278em" rspace="0.278em" id="S3.E2.m1.1.3.2.3">:</mo><msub id="S3.E2.m1.1.3.2.4"><mi id="S3.E2.m1.1.3.2.4.2">y</mi><mn id="S3.E2.m1.1.3.2.4.3">1</mn></msub><mo stretchy="false" id="S3.E2.m1.1.3.2.5">)</mo></mrow><mo id="S3.E2.m1.1.3.3">,</mo><mrow id="S3.E2.m1.1.3.4"><mo stretchy="false" id="S3.E2.m1.1.3.4.1">(</mo><msub id="S3.E2.m1.1.3.4.2"><mi id="S3.E2.m1.1.3.4.2.2">x</mi><mn id="S3.E2.m1.1.3.4.2.3">2</mn></msub><mo lspace="0.278em" rspace="0.278em" id="S3.E2.m1.1.3.4.3">:</mo><msub id="S3.E2.m1.1.3.4.4"><mi id="S3.E2.m1.1.3.4.4.2">y</mi><mn id="S3.E2.m1.1.3.4.4.3">2</mn></msub><mo stretchy="false" id="S3.E2.m1.1.3.4.5">)</mo></mrow><mo id="S3.E2.m1.1.3.5">,</mo><mi mathvariant="normal" id="S3.E2.m1.1.3.6">…</mi><mo id="S3.E2.m1.1.3.7">,</mo><mrow id="S3.E2.m1.1.3.8"><mo stretchy="false" id="S3.E2.m1.1.3.8.1">(</mo><msub id="S3.E2.m1.1.3.8.2"><mi id="S3.E2.m1.1.3.8.2.2">x</mi><mi id="S3.E2.m1.1.3.8.2.3">N</mi></msub><mo lspace="0.278em" rspace="0.278em" id="S3.E2.m1.1.3.8.3">:</mo><msub id="S3.E2.m1.1.3.8.4"><mi id="S3.E2.m1.1.3.8.4.2">y</mi><mi id="S3.E2.m1.1.3.8.4.3">N</mi></msub><mo stretchy="false" id="S3.E2.m1.1.3.8.5">)</mo></mrow><mo id="S3.E2.m1.1.3.9">,</mo><msub id="S3.E2.m1.1.3.10"><mi id="S3.E2.m1.1.3.10.2">Q</mi><mi id="S3.E2.m1.1.3.10.3">t</mi></msub><mo stretchy="false" id="S3.E2.m1.1.3.11">]</mo></mrow></mrow><annotation encoding="application/x-tex" id="S3.E2.m1.1c">I_{t}=[(x_{1}:y_{1}),(x_{2}:y_{2}),\ldots,(x_{N}:y_{N}),Q_{t}]</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S3.SS2.p2.10" class="ltx_p">Thus, over <math id="S3.SS2.p2.9.m1.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S3.SS2.p2.9.m1.1a"><mi id="S3.SS2.p2.9.m1.1.1" xref="S3.SS2.p2.9.m1.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.9.m1.1b"><ci id="S3.SS2.p2.9.m1.1.1.cmml" xref="S3.SS2.p2.9.m1.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.9.m1.1c">T</annotation></semantics></math> LMM inference calls, we have a many-shot multimodal dataset (of <math id="S3.SS2.p2.10.m2.1" class="ltx_Math" alttext="N\times T" display="inline"><semantics id="S3.SS2.p2.10.m2.1a"><mrow id="S3.SS2.p2.10.m2.1.1" xref="S3.SS2.p2.10.m2.1.1.cmml"><mi id="S3.SS2.p2.10.m2.1.1.2" xref="S3.SS2.p2.10.m2.1.1.2.cmml">N</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p2.10.m2.1.1.1" xref="S3.SS2.p2.10.m2.1.1.1.cmml">×</mo><mi id="S3.SS2.p2.10.m2.1.1.3" xref="S3.SS2.p2.10.m2.1.1.3.cmml">T</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.10.m2.1b"><apply id="S3.SS2.p2.10.m2.1.1.cmml" xref="S3.SS2.p2.10.m2.1.1"><times id="S3.SS2.p2.10.m2.1.1.1.cmml" xref="S3.SS2.p2.10.m2.1.1.1"></times><ci id="S3.SS2.p2.10.m2.1.1.2.cmml" xref="S3.SS2.p2.10.m2.1.1.2">𝑁</ci><ci id="S3.SS2.p2.10.m2.1.1.3.cmml" xref="S3.SS2.p2.10.m2.1.1.3">𝑇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.10.m2.1c">N\times T</annotation></semantics></math> examples):</p>
<table id="A5.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E3.m1.1" class="ltx_Math" alttext="\displaystyle I_{\mathrm{many}}" display="inline"><semantics id="S3.E3.m1.1a"><msub id="S3.E3.m1.1.1" xref="S3.E3.m1.1.1.cmml"><mi id="S3.E3.m1.1.1.2" xref="S3.E3.m1.1.1.2.cmml">I</mi><mi id="S3.E3.m1.1.1.3" xref="S3.E3.m1.1.1.3.cmml">many</mi></msub><annotation-xml encoding="MathML-Content" id="S3.E3.m1.1b"><apply id="S3.E3.m1.1.1.cmml" xref="S3.E3.m1.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.cmml" xref="S3.E3.m1.1.1">subscript</csymbol><ci id="S3.E3.m1.1.1.2.cmml" xref="S3.E3.m1.1.1.2">𝐼</ci><ci id="S3.E3.m1.1.1.3.cmml" xref="S3.E3.m1.1.1.3">many</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m1.1c">\displaystyle I_{\mathrm{many}}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E3.m2.4" class="ltx_Math" alttext="\displaystyle=[I_{1},I_{2},\ldots,I_{T}]" display="inline"><semantics id="S3.E3.m2.4a"><mrow id="S3.E3.m2.4.4" xref="S3.E3.m2.4.4.cmml"><mi id="S3.E3.m2.4.4.5" xref="S3.E3.m2.4.4.5.cmml"></mi><mo id="S3.E3.m2.4.4.4" xref="S3.E3.m2.4.4.4.cmml">=</mo><mrow id="S3.E3.m2.4.4.3.3" xref="S3.E3.m2.4.4.3.4.cmml"><mo stretchy="false" id="S3.E3.m2.4.4.3.3.4" xref="S3.E3.m2.4.4.3.4.cmml">[</mo><msub id="S3.E3.m2.2.2.1.1.1" xref="S3.E3.m2.2.2.1.1.1.cmml"><mi id="S3.E3.m2.2.2.1.1.1.2" xref="S3.E3.m2.2.2.1.1.1.2.cmml">I</mi><mn id="S3.E3.m2.2.2.1.1.1.3" xref="S3.E3.m2.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S3.E3.m2.4.4.3.3.5" xref="S3.E3.m2.4.4.3.4.cmml">,</mo><msub id="S3.E3.m2.3.3.2.2.2" xref="S3.E3.m2.3.3.2.2.2.cmml"><mi id="S3.E3.m2.3.3.2.2.2.2" xref="S3.E3.m2.3.3.2.2.2.2.cmml">I</mi><mn id="S3.E3.m2.3.3.2.2.2.3" xref="S3.E3.m2.3.3.2.2.2.3.cmml">2</mn></msub><mo id="S3.E3.m2.4.4.3.3.6" xref="S3.E3.m2.4.4.3.4.cmml">,</mo><mi mathvariant="normal" id="S3.E3.m2.1.1" xref="S3.E3.m2.1.1.cmml">…</mi><mo id="S3.E3.m2.4.4.3.3.7" xref="S3.E3.m2.4.4.3.4.cmml">,</mo><msub id="S3.E3.m2.4.4.3.3.3" xref="S3.E3.m2.4.4.3.3.3.cmml"><mi id="S3.E3.m2.4.4.3.3.3.2" xref="S3.E3.m2.4.4.3.3.3.2.cmml">I</mi><mi id="S3.E3.m2.4.4.3.3.3.3" xref="S3.E3.m2.4.4.3.3.3.3.cmml">T</mi></msub><mo stretchy="false" id="S3.E3.m2.4.4.3.3.8" xref="S3.E3.m2.4.4.3.4.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E3.m2.4b"><apply id="S3.E3.m2.4.4.cmml" xref="S3.E3.m2.4.4"><eq id="S3.E3.m2.4.4.4.cmml" xref="S3.E3.m2.4.4.4"></eq><csymbol cd="latexml" id="S3.E3.m2.4.4.5.cmml" xref="S3.E3.m2.4.4.5">absent</csymbol><list id="S3.E3.m2.4.4.3.4.cmml" xref="S3.E3.m2.4.4.3.3"><apply id="S3.E3.m2.2.2.1.1.1.cmml" xref="S3.E3.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.E3.m2.2.2.1.1.1.1.cmml" xref="S3.E3.m2.2.2.1.1.1">subscript</csymbol><ci id="S3.E3.m2.2.2.1.1.1.2.cmml" xref="S3.E3.m2.2.2.1.1.1.2">𝐼</ci><cn type="integer" id="S3.E3.m2.2.2.1.1.1.3.cmml" xref="S3.E3.m2.2.2.1.1.1.3">1</cn></apply><apply id="S3.E3.m2.3.3.2.2.2.cmml" xref="S3.E3.m2.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.E3.m2.3.3.2.2.2.1.cmml" xref="S3.E3.m2.3.3.2.2.2">subscript</csymbol><ci id="S3.E3.m2.3.3.2.2.2.2.cmml" xref="S3.E3.m2.3.3.2.2.2.2">𝐼</ci><cn type="integer" id="S3.E3.m2.3.3.2.2.2.3.cmml" xref="S3.E3.m2.3.3.2.2.2.3">2</cn></apply><ci id="S3.E3.m2.1.1.cmml" xref="S3.E3.m2.1.1">…</ci><apply id="S3.E3.m2.4.4.3.3.3.cmml" xref="S3.E3.m2.4.4.3.3.3"><csymbol cd="ambiguous" id="S3.E3.m2.4.4.3.3.3.1.cmml" xref="S3.E3.m2.4.4.3.3.3">subscript</csymbol><ci id="S3.E3.m2.4.4.3.3.3.2.cmml" xref="S3.E3.m2.4.4.3.3.3.2">𝐼</ci><ci id="S3.E3.m2.4.4.3.3.3.3.cmml" xref="S3.E3.m2.4.4.3.3.3.3">𝑇</ci></apply></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m2.4c">\displaystyle=[I_{1},I_{2},\ldots,I_{T}]</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS2.p3" class="ltx_para">
<p id="S3.SS2.p3.1" class="ltx_p">However, this dataset is still just a disconnected set of few-shot examples. Next, we would like to connect the separate examples into one unified many-shot multimodal ICL representation.</p>
</div>
<div id="S3.SS2.p4" class="ltx_para">
<p id="S3.SS2.p4.4" class="ltx_p">For each inference call, the LMM is given <math id="S3.SS2.p4.1.m1.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S3.SS2.p4.1.m1.1a"><mi id="S3.SS2.p4.1.m1.1.1" xref="S3.SS2.p4.1.m1.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.1.m1.1b"><ci id="S3.SS2.p4.1.m1.1.1.cmml" xref="S3.SS2.p4.1.m1.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.1.m1.1c">N</annotation></semantics></math>-shot ICL examples. We calculate the mean of the activations corresponding to the last token of the input <math id="S3.SS2.p4.2.m2.2" class="ltx_Math" alttext="z_{l,j}" display="inline"><semantics id="S3.SS2.p4.2.m2.2a"><msub id="S3.SS2.p4.2.m2.2.3" xref="S3.SS2.p4.2.m2.2.3.cmml"><mi id="S3.SS2.p4.2.m2.2.3.2" xref="S3.SS2.p4.2.m2.2.3.2.cmml">z</mi><mrow id="S3.SS2.p4.2.m2.2.2.2.4" xref="S3.SS2.p4.2.m2.2.2.2.3.cmml"><mi id="S3.SS2.p4.2.m2.1.1.1.1" xref="S3.SS2.p4.2.m2.1.1.1.1.cmml">l</mi><mo id="S3.SS2.p4.2.m2.2.2.2.4.1" xref="S3.SS2.p4.2.m2.2.2.2.3.cmml">,</mo><mi id="S3.SS2.p4.2.m2.2.2.2.2" xref="S3.SS2.p4.2.m2.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.2.m2.2b"><apply id="S3.SS2.p4.2.m2.2.3.cmml" xref="S3.SS2.p4.2.m2.2.3"><csymbol cd="ambiguous" id="S3.SS2.p4.2.m2.2.3.1.cmml" xref="S3.SS2.p4.2.m2.2.3">subscript</csymbol><ci id="S3.SS2.p4.2.m2.2.3.2.cmml" xref="S3.SS2.p4.2.m2.2.3.2">𝑧</ci><list id="S3.SS2.p4.2.m2.2.2.2.3.cmml" xref="S3.SS2.p4.2.m2.2.2.2.4"><ci id="S3.SS2.p4.2.m2.1.1.1.1.cmml" xref="S3.SS2.p4.2.m2.1.1.1.1">𝑙</ci><ci id="S3.SS2.p4.2.m2.2.2.2.2.cmml" xref="S3.SS2.p4.2.m2.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.2.m2.2c">z_{l,j}</annotation></semantics></math> for each attention head index <math id="S3.SS2.p4.3.m3.1" class="ltx_Math" alttext="\forall l\in\lambda" display="inline"><semantics id="S3.SS2.p4.3.m3.1a"><mrow id="S3.SS2.p4.3.m3.1.1" xref="S3.SS2.p4.3.m3.1.1.cmml"><mrow id="S3.SS2.p4.3.m3.1.1.2" xref="S3.SS2.p4.3.m3.1.1.2.cmml"><mo rspace="0.167em" id="S3.SS2.p4.3.m3.1.1.2.1" xref="S3.SS2.p4.3.m3.1.1.2.1.cmml">∀</mo><mi id="S3.SS2.p4.3.m3.1.1.2.2" xref="S3.SS2.p4.3.m3.1.1.2.2.cmml">l</mi></mrow><mo id="S3.SS2.p4.3.m3.1.1.1" xref="S3.SS2.p4.3.m3.1.1.1.cmml">∈</mo><mi id="S3.SS2.p4.3.m3.1.1.3" xref="S3.SS2.p4.3.m3.1.1.3.cmml">λ</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.3.m3.1b"><apply id="S3.SS2.p4.3.m3.1.1.cmml" xref="S3.SS2.p4.3.m3.1.1"><in id="S3.SS2.p4.3.m3.1.1.1.cmml" xref="S3.SS2.p4.3.m3.1.1.1"></in><apply id="S3.SS2.p4.3.m3.1.1.2.cmml" xref="S3.SS2.p4.3.m3.1.1.2"><csymbol cd="latexml" id="S3.SS2.p4.3.m3.1.1.2.1.cmml" xref="S3.SS2.p4.3.m3.1.1.2.1">for-all</csymbol><ci id="S3.SS2.p4.3.m3.1.1.2.2.cmml" xref="S3.SS2.p4.3.m3.1.1.2.2">𝑙</ci></apply><ci id="S3.SS2.p4.3.m3.1.1.3.cmml" xref="S3.SS2.p4.3.m3.1.1.3">𝜆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.3.m3.1c">\forall l\in\lambda</annotation></semantics></math> (Section <a href="#S3.SS1" title="3.1 Preliminaries ‣ 3 Multimodal Task Vectors ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a>) across <math id="S3.SS2.p4.4.m4.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S3.SS2.p4.4.m4.1a"><mi id="S3.SS2.p4.4.m4.1.1" xref="S3.SS2.p4.4.m4.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.4.m4.1b"><ci id="S3.SS2.p4.4.m4.1.1.cmml" xref="S3.SS2.p4.4.m4.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.4.m4.1c">T</annotation></semantics></math> inference calls, yielding:
</p>
<table id="S3.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E4.m1.6" class="ltx_math_unparsed" alttext="\forall l\in\lambda:\quad\mu_{l,j}=\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}[z_{l,j}\mid I_{t}]=\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}\left[z_{l,j}\mid(x_{1}:y_{1}),(x_{2}:y_{2}),\ldots,(x_{N}:y_{N}),Q_{t}\right]" display="block"><semantics id="S3.E4.m1.6a"><mrow id="S3.E4.m1.6b"><mo rspace="0.167em" id="S3.E4.m1.6.7">∀</mo><mi id="S3.E4.m1.6.8">l</mi><mo id="S3.E4.m1.6.9">∈</mo><mi id="S3.E4.m1.6.10">λ</mi><mo lspace="0.278em" id="S3.E4.m1.6.11">:</mo><mspace width="1.278em" id="S3.E4.m1.6.12"></mspace><msub id="S3.E4.m1.6.13"><mi id="S3.E4.m1.6.13.2">μ</mi><mrow id="S3.E4.m1.2.2.2.4"><mi id="S3.E4.m1.1.1.1.1">l</mi><mo id="S3.E4.m1.2.2.2.4.1">,</mo><mi id="S3.E4.m1.2.2.2.2">j</mi></mrow></msub><mo id="S3.E4.m1.6.14">=</mo><mfrac id="S3.E4.m1.6.15"><mn id="S3.E4.m1.6.15.2">1</mn><mi id="S3.E4.m1.6.15.3">T</mi></mfrac><munderover id="S3.E4.m1.6.16"><mo movablelimits="false" id="S3.E4.m1.6.16.2.2">∑</mo><mrow id="S3.E4.m1.6.16.2.3"><mi id="S3.E4.m1.6.16.2.3.2">t</mi><mo id="S3.E4.m1.6.16.2.3.1">=</mo><mn id="S3.E4.m1.6.16.2.3.3">1</mn></mrow><mi id="S3.E4.m1.6.16.3">T</mi></munderover><mi id="S3.E4.m1.6.17">𝔼</mi><mrow id="S3.E4.m1.6.18"><mo stretchy="false" id="S3.E4.m1.6.18.1">[</mo><msub id="S3.E4.m1.6.18.2"><mi id="S3.E4.m1.6.18.2.2">z</mi><mrow id="S3.E4.m1.4.4.2.4"><mi id="S3.E4.m1.3.3.1.1">l</mi><mo id="S3.E4.m1.4.4.2.4.1">,</mo><mi id="S3.E4.m1.4.4.2.2">j</mi></mrow></msub><mo lspace="0em" rspace="0.167em" id="S3.E4.m1.6.18.3">∣</mo><msub id="S3.E4.m1.6.18.4"><mi id="S3.E4.m1.6.18.4.2">I</mi><mi id="S3.E4.m1.6.18.4.3">t</mi></msub><mo stretchy="false" id="S3.E4.m1.6.18.5">]</mo></mrow><mo id="S3.E4.m1.6.19">=</mo><mfrac id="S3.E4.m1.6.20"><mn id="S3.E4.m1.6.20.2">1</mn><mi id="S3.E4.m1.6.20.3">T</mi></mfrac><munderover id="S3.E4.m1.6.21"><mo movablelimits="false" id="S3.E4.m1.6.21.2.2">∑</mo><mrow id="S3.E4.m1.6.21.2.3"><mi id="S3.E4.m1.6.21.2.3.2">t</mi><mo id="S3.E4.m1.6.21.2.3.1">=</mo><mn id="S3.E4.m1.6.21.2.3.3">1</mn></mrow><mi id="S3.E4.m1.6.21.3">T</mi></munderover><mi id="S3.E4.m1.6.22">𝔼</mi><mrow id="S3.E4.m1.6.23"><mo id="S3.E4.m1.6.23.1">[</mo><msub id="S3.E4.m1.6.23.2"><mi id="S3.E4.m1.6.23.2.2">z</mi><mrow id="S3.E4.m1.6.6.2.4"><mi id="S3.E4.m1.5.5.1.1">l</mi><mo id="S3.E4.m1.6.6.2.4.1">,</mo><mi id="S3.E4.m1.6.6.2.2">j</mi></mrow></msub><mo lspace="0em" rspace="0.167em" id="S3.E4.m1.6.23.3">∣</mo><mrow id="S3.E4.m1.6.23.4"><mo stretchy="false" id="S3.E4.m1.6.23.4.1">(</mo><msub id="S3.E4.m1.6.23.4.2"><mi id="S3.E4.m1.6.23.4.2.2">x</mi><mn id="S3.E4.m1.6.23.4.2.3">1</mn></msub><mo lspace="0.278em" rspace="0.278em" id="S3.E4.m1.6.23.4.3">:</mo><msub id="S3.E4.m1.6.23.4.4"><mi id="S3.E4.m1.6.23.4.4.2">y</mi><mn id="S3.E4.m1.6.23.4.4.3">1</mn></msub><mo stretchy="false" id="S3.E4.m1.6.23.4.5">)</mo></mrow><mo id="S3.E4.m1.6.23.5">,</mo><mrow id="S3.E4.m1.6.23.6"><mo stretchy="false" id="S3.E4.m1.6.23.6.1">(</mo><msub id="S3.E4.m1.6.23.6.2"><mi id="S3.E4.m1.6.23.6.2.2">x</mi><mn id="S3.E4.m1.6.23.6.2.3">2</mn></msub><mo lspace="0.278em" rspace="0.278em" id="S3.E4.m1.6.23.6.3">:</mo><msub id="S3.E4.m1.6.23.6.4"><mi id="S3.E4.m1.6.23.6.4.2">y</mi><mn id="S3.E4.m1.6.23.6.4.3">2</mn></msub><mo stretchy="false" id="S3.E4.m1.6.23.6.5">)</mo></mrow><mo id="S3.E4.m1.6.23.7">,</mo><mi mathvariant="normal" id="S3.E4.m1.6.23.8">…</mi><mo id="S3.E4.m1.6.23.9">,</mo><mrow id="S3.E4.m1.6.23.10"><mo stretchy="false" id="S3.E4.m1.6.23.10.1">(</mo><msub id="S3.E4.m1.6.23.10.2"><mi id="S3.E4.m1.6.23.10.2.2">x</mi><mi id="S3.E4.m1.6.23.10.2.3">N</mi></msub><mo lspace="0.278em" rspace="0.278em" id="S3.E4.m1.6.23.10.3">:</mo><msub id="S3.E4.m1.6.23.10.4"><mi id="S3.E4.m1.6.23.10.4.2">y</mi><mi id="S3.E4.m1.6.23.10.4.3">N</mi></msub><mo stretchy="false" id="S3.E4.m1.6.23.10.5">)</mo></mrow><mo id="S3.E4.m1.6.23.11">,</mo><msub id="S3.E4.m1.6.23.12"><mi id="S3.E4.m1.6.23.12.2">Q</mi><mi id="S3.E4.m1.6.23.12.3">t</mi></msub><mo id="S3.E4.m1.6.23.13">]</mo></mrow></mrow><annotation encoding="application/x-tex" id="S3.E4.m1.6c">\forall l\in\lambda:\quad\mu_{l,j}=\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}[z_{l,j}\mid I_{t}]=\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}\left[z_{l,j}\mid(x_{1}:y_{1}),(x_{2}:y_{2}),\ldots,(x_{N}:y_{N}),Q_{t}\right]</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS2.p5" class="ltx_para">
<p id="S3.SS2.p5.1" class="ltx_p">In this step, we have found the mean activations <math id="S3.SS2.p5.1.m1.2" class="ltx_Math" alttext="\mu_{l,j}" display="inline"><semantics id="S3.SS2.p5.1.m1.2a"><msub id="S3.SS2.p5.1.m1.2.3" xref="S3.SS2.p5.1.m1.2.3.cmml"><mi id="S3.SS2.p5.1.m1.2.3.2" xref="S3.SS2.p5.1.m1.2.3.2.cmml">μ</mi><mrow id="S3.SS2.p5.1.m1.2.2.2.4" xref="S3.SS2.p5.1.m1.2.2.2.3.cmml"><mi id="S3.SS2.p5.1.m1.1.1.1.1" xref="S3.SS2.p5.1.m1.1.1.1.1.cmml">l</mi><mo id="S3.SS2.p5.1.m1.2.2.2.4.1" xref="S3.SS2.p5.1.m1.2.2.2.3.cmml">,</mo><mi id="S3.SS2.p5.1.m1.2.2.2.2" xref="S3.SS2.p5.1.m1.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p5.1.m1.2b"><apply id="S3.SS2.p5.1.m1.2.3.cmml" xref="S3.SS2.p5.1.m1.2.3"><csymbol cd="ambiguous" id="S3.SS2.p5.1.m1.2.3.1.cmml" xref="S3.SS2.p5.1.m1.2.3">subscript</csymbol><ci id="S3.SS2.p5.1.m1.2.3.2.cmml" xref="S3.SS2.p5.1.m1.2.3.2">𝜇</ci><list id="S3.SS2.p5.1.m1.2.2.2.3.cmml" xref="S3.SS2.p5.1.m1.2.2.2.4"><ci id="S3.SS2.p5.1.m1.1.1.1.1.cmml" xref="S3.SS2.p5.1.m1.1.1.1.1">𝑙</ci><ci id="S3.SS2.p5.1.m1.2.2.2.2.cmml" xref="S3.SS2.p5.1.m1.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p5.1.m1.2c">\mu_{l,j}</annotation></semantics></math>, which encode an internal LMM representation of many shots of multimodal ICL examples. In the next subsection, we describe our methodology for selecting the set of attention heads where these mean activations will be used.</p>
</div>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Step 2: Extract MTV Attention Head Locations</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">After Step 1, we now have mean activations for the attention heads of the last token in a given many-shot multimodal task. Yet, we still need to find which set of attention heads <math id="S3.SS3.p1.1.m1.1" class="ltx_Math" alttext="\lambda^{\mathrm{MTV}}_{j}" display="inline"><semantics id="S3.SS3.p1.1.m1.1a"><msubsup id="S3.SS3.p1.1.m1.1.1" xref="S3.SS3.p1.1.m1.1.1.cmml"><mi id="S3.SS3.p1.1.m1.1.1.2.2" xref="S3.SS3.p1.1.m1.1.1.2.2.cmml">λ</mi><mi id="S3.SS3.p1.1.m1.1.1.3" xref="S3.SS3.p1.1.m1.1.1.3.cmml">j</mi><mi id="S3.SS3.p1.1.m1.1.1.2.3" xref="S3.SS3.p1.1.m1.1.1.2.3.cmml">MTV</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.1.m1.1b"><apply id="S3.SS3.p1.1.m1.1.1.cmml" xref="S3.SS3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p1.1.m1.1.1.1.cmml" xref="S3.SS3.p1.1.m1.1.1">subscript</csymbol><apply id="S3.SS3.p1.1.m1.1.1.2.cmml" xref="S3.SS3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p1.1.m1.1.1.2.1.cmml" xref="S3.SS3.p1.1.m1.1.1">superscript</csymbol><ci id="S3.SS3.p1.1.m1.1.1.2.2.cmml" xref="S3.SS3.p1.1.m1.1.1.2.2">𝜆</ci><ci id="S3.SS3.p1.1.m1.1.1.2.3.cmml" xref="S3.SS3.p1.1.m1.1.1.2.3">MTV</ci></apply><ci id="S3.SS3.p1.1.m1.1.1.3.cmml" xref="S3.SS3.p1.1.m1.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.1.m1.1c">\lambda^{\mathrm{MTV}}_{j}</annotation></semantics></math> should be chosen to encode our task.</p>
</div>
<div id="S3.SS3.p2" class="ltx_para">
<p id="S3.SS3.p2.5" class="ltx_p">To choose the set of attention heads, we first prepare a separate set of <math id="S3.SS3.p2.1.m1.1" class="ltx_Math" alttext="S" display="inline"><semantics id="S3.SS3.p2.1.m1.1a"><mi id="S3.SS3.p2.1.m1.1.1" xref="S3.SS3.p2.1.m1.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.1.m1.1b"><ci id="S3.SS3.p2.1.m1.1.1.cmml" xref="S3.SS3.p2.1.m1.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.1.m1.1c">S</annotation></semantics></math> examples specifically aligned to the format of the downstream task. For instance, if the downstream setting is a 2-way, one-shot classification task, then the <math id="S3.SS3.p2.2.m2.1" class="ltx_Math" alttext="S" display="inline"><semantics id="S3.SS3.p2.2.m2.1a"><mi id="S3.SS3.p2.2.m2.1.1" xref="S3.SS3.p2.2.m2.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.2.m2.1b"><ci id="S3.SS3.p2.2.m2.1.1.cmml" xref="S3.SS3.p2.2.m2.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.2.m2.1c">S</annotation></semantics></math> examples should conform to this paradigm. For our explanation, let’s consider a downstream task that is zero-shot such that there is a single query <math id="S3.SS3.p2.3.m3.1" class="ltx_Math" alttext="Q_{s}" display="inline"><semantics id="S3.SS3.p2.3.m3.1a"><msub id="S3.SS3.p2.3.m3.1.1" xref="S3.SS3.p2.3.m3.1.1.cmml"><mi id="S3.SS3.p2.3.m3.1.1.2" xref="S3.SS3.p2.3.m3.1.1.2.cmml">Q</mi><mi id="S3.SS3.p2.3.m3.1.1.3" xref="S3.SS3.p2.3.m3.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.3.m3.1b"><apply id="S3.SS3.p2.3.m3.1.1.cmml" xref="S3.SS3.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.3.m3.1.1.1.cmml" xref="S3.SS3.p2.3.m3.1.1">subscript</csymbol><ci id="S3.SS3.p2.3.m3.1.1.2.cmml" xref="S3.SS3.p2.3.m3.1.1.2">𝑄</ci><ci id="S3.SS3.p2.3.m3.1.1.3.cmml" xref="S3.SS3.p2.3.m3.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.3.m3.1c">Q_{s}</annotation></semantics></math> and corresponding response <math id="S3.SS3.p2.4.m4.1" class="ltx_Math" alttext="R_{s}" display="inline"><semantics id="S3.SS3.p2.4.m4.1a"><msub id="S3.SS3.p2.4.m4.1.1" xref="S3.SS3.p2.4.m4.1.1.cmml"><mi id="S3.SS3.p2.4.m4.1.1.2" xref="S3.SS3.p2.4.m4.1.1.2.cmml">R</mi><mi id="S3.SS3.p2.4.m4.1.1.3" xref="S3.SS3.p2.4.m4.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.4.m4.1b"><apply id="S3.SS3.p2.4.m4.1.1.cmml" xref="S3.SS3.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.1.1.1.cmml" xref="S3.SS3.p2.4.m4.1.1">subscript</csymbol><ci id="S3.SS3.p2.4.m4.1.1.2.cmml" xref="S3.SS3.p2.4.m4.1.1.2">𝑅</ci><ci id="S3.SS3.p2.4.m4.1.1.3.cmml" xref="S3.SS3.p2.4.m4.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.4.m4.1c">R_{s}</annotation></semantics></math> for all <math id="S3.SS3.p2.5.m5.4" class="ltx_Math" alttext="s\in[1,2,\ldots,S]" display="inline"><semantics id="S3.SS3.p2.5.m5.4a"><mrow id="S3.SS3.p2.5.m5.4.5" xref="S3.SS3.p2.5.m5.4.5.cmml"><mi id="S3.SS3.p2.5.m5.4.5.2" xref="S3.SS3.p2.5.m5.4.5.2.cmml">s</mi><mo id="S3.SS3.p2.5.m5.4.5.1" xref="S3.SS3.p2.5.m5.4.5.1.cmml">∈</mo><mrow id="S3.SS3.p2.5.m5.4.5.3.2" xref="S3.SS3.p2.5.m5.4.5.3.1.cmml"><mo stretchy="false" id="S3.SS3.p2.5.m5.4.5.3.2.1" xref="S3.SS3.p2.5.m5.4.5.3.1.cmml">[</mo><mn id="S3.SS3.p2.5.m5.1.1" xref="S3.SS3.p2.5.m5.1.1.cmml">1</mn><mo id="S3.SS3.p2.5.m5.4.5.3.2.2" xref="S3.SS3.p2.5.m5.4.5.3.1.cmml">,</mo><mn id="S3.SS3.p2.5.m5.2.2" xref="S3.SS3.p2.5.m5.2.2.cmml">2</mn><mo id="S3.SS3.p2.5.m5.4.5.3.2.3" xref="S3.SS3.p2.5.m5.4.5.3.1.cmml">,</mo><mi mathvariant="normal" id="S3.SS3.p2.5.m5.3.3" xref="S3.SS3.p2.5.m5.3.3.cmml">…</mi><mo id="S3.SS3.p2.5.m5.4.5.3.2.4" xref="S3.SS3.p2.5.m5.4.5.3.1.cmml">,</mo><mi id="S3.SS3.p2.5.m5.4.4" xref="S3.SS3.p2.5.m5.4.4.cmml">S</mi><mo stretchy="false" id="S3.SS3.p2.5.m5.4.5.3.2.5" xref="S3.SS3.p2.5.m5.4.5.3.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.5.m5.4b"><apply id="S3.SS3.p2.5.m5.4.5.cmml" xref="S3.SS3.p2.5.m5.4.5"><in id="S3.SS3.p2.5.m5.4.5.1.cmml" xref="S3.SS3.p2.5.m5.4.5.1"></in><ci id="S3.SS3.p2.5.m5.4.5.2.cmml" xref="S3.SS3.p2.5.m5.4.5.2">𝑠</ci><list id="S3.SS3.p2.5.m5.4.5.3.1.cmml" xref="S3.SS3.p2.5.m5.4.5.3.2"><cn type="integer" id="S3.SS3.p2.5.m5.1.1.cmml" xref="S3.SS3.p2.5.m5.1.1">1</cn><cn type="integer" id="S3.SS3.p2.5.m5.2.2.cmml" xref="S3.SS3.p2.5.m5.2.2">2</cn><ci id="S3.SS3.p2.5.m5.3.3.cmml" xref="S3.SS3.p2.5.m5.3.3">…</ci><ci id="S3.SS3.p2.5.m5.4.4.cmml" xref="S3.SS3.p2.5.m5.4.4">𝑆</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.5.m5.4c">s\in[1,2,\ldots,S]</annotation></semantics></math>.</p>
</div>
<div id="S3.SS3.p3" class="ltx_para">
<p id="S3.SS3.p3.7" class="ltx_p">From these examples, we utilize an adapted version of the REINFORCE <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib89" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">89</span></a>]</cite> algorithm—an iterative policy optimization method that can be used to find task vector locations. Given an LMM <math id="S3.SS3.p3.1.m1.1" class="ltx_Math" alttext="F" display="inline"><semantics id="S3.SS3.p3.1.m1.1a"><mi id="S3.SS3.p3.1.m1.1.1" xref="S3.SS3.p3.1.m1.1.1.cmml">F</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.1.m1.1b"><ci id="S3.SS3.p3.1.m1.1.1.cmml" xref="S3.SS3.p3.1.m1.1.1">𝐹</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.1.m1.1c">F</annotation></semantics></math>, we first select a proposed set of attention head locations by sampling a Bernoulli distribution over the locations multiple times. Next, we directly replace the values of the selected attention heads with the corresponding mean activations <math id="S3.SS3.p3.2.m2.2" class="ltx_Math" alttext="\mu_{l,j}" display="inline"><semantics id="S3.SS3.p3.2.m2.2a"><msub id="S3.SS3.p3.2.m2.2.3" xref="S3.SS3.p3.2.m2.2.3.cmml"><mi id="S3.SS3.p3.2.m2.2.3.2" xref="S3.SS3.p3.2.m2.2.3.2.cmml">μ</mi><mrow id="S3.SS3.p3.2.m2.2.2.2.4" xref="S3.SS3.p3.2.m2.2.2.2.3.cmml"><mi id="S3.SS3.p3.2.m2.1.1.1.1" xref="S3.SS3.p3.2.m2.1.1.1.1.cmml">l</mi><mo id="S3.SS3.p3.2.m2.2.2.2.4.1" xref="S3.SS3.p3.2.m2.2.2.2.3.cmml">,</mo><mi id="S3.SS3.p3.2.m2.2.2.2.2" xref="S3.SS3.p3.2.m2.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.2.m2.2b"><apply id="S3.SS3.p3.2.m2.2.3.cmml" xref="S3.SS3.p3.2.m2.2.3"><csymbol cd="ambiguous" id="S3.SS3.p3.2.m2.2.3.1.cmml" xref="S3.SS3.p3.2.m2.2.3">subscript</csymbol><ci id="S3.SS3.p3.2.m2.2.3.2.cmml" xref="S3.SS3.p3.2.m2.2.3.2">𝜇</ci><list id="S3.SS3.p3.2.m2.2.2.2.3.cmml" xref="S3.SS3.p3.2.m2.2.2.2.4"><ci id="S3.SS3.p3.2.m2.1.1.1.1.cmml" xref="S3.SS3.p3.2.m2.1.1.1.1">𝑙</ci><ci id="S3.SS3.p3.2.m2.2.2.2.2.cmml" xref="S3.SS3.p3.2.m2.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.2.m2.2c">\mu_{l,j}</annotation></semantics></math>. Then, after prompting the model with the query <math id="S3.SS3.p3.3.m3.1" class="ltx_Math" alttext="Q_{s}" display="inline"><semantics id="S3.SS3.p3.3.m3.1a"><msub id="S3.SS3.p3.3.m3.1.1" xref="S3.SS3.p3.3.m3.1.1.cmml"><mi id="S3.SS3.p3.3.m3.1.1.2" xref="S3.SS3.p3.3.m3.1.1.2.cmml">Q</mi><mi id="S3.SS3.p3.3.m3.1.1.3" xref="S3.SS3.p3.3.m3.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.3.m3.1b"><apply id="S3.SS3.p3.3.m3.1.1.cmml" xref="S3.SS3.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.3.m3.1.1.1.cmml" xref="S3.SS3.p3.3.m3.1.1">subscript</csymbol><ci id="S3.SS3.p3.3.m3.1.1.2.cmml" xref="S3.SS3.p3.3.m3.1.1.2">𝑄</ci><ci id="S3.SS3.p3.3.m3.1.1.3.cmml" xref="S3.SS3.p3.3.m3.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.3.m3.1c">Q_{s}</annotation></semantics></math>, we use the negative cross-entropy loss between the LMM’s output logits and the logits of the ground-truth response <math id="S3.SS3.p3.4.m4.1" class="ltx_Math" alttext="R_{s}" display="inline"><semantics id="S3.SS3.p3.4.m4.1a"><msub id="S3.SS3.p3.4.m4.1.1" xref="S3.SS3.p3.4.m4.1.1.cmml"><mi id="S3.SS3.p3.4.m4.1.1.2" xref="S3.SS3.p3.4.m4.1.1.2.cmml">R</mi><mi id="S3.SS3.p3.4.m4.1.1.3" xref="S3.SS3.p3.4.m4.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.4.m4.1b"><apply id="S3.SS3.p3.4.m4.1.1.cmml" xref="S3.SS3.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.4.m4.1.1.1.cmml" xref="S3.SS3.p3.4.m4.1.1">subscript</csymbol><ci id="S3.SS3.p3.4.m4.1.1.2.cmml" xref="S3.SS3.p3.4.m4.1.1.2">𝑅</ci><ci id="S3.SS3.p3.4.m4.1.1.3.cmml" xref="S3.SS3.p3.4.m4.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.4.m4.1c">R_{s}</annotation></semantics></math> to optimize the Bernoulli distribution. By optimizing the Bernoulli distribution across <math id="S3.SS3.p3.5.m5.1" class="ltx_Math" alttext="S" display="inline"><semantics id="S3.SS3.p3.5.m5.1a"><mi id="S3.SS3.p3.5.m5.1.1" xref="S3.SS3.p3.5.m5.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.5.m5.1b"><ci id="S3.SS3.p3.5.m5.1.1.cmml" xref="S3.SS3.p3.5.m5.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.5.m5.1c">S</annotation></semantics></math> iterations, we are finding the best attention head locations <math id="S3.SS3.p3.6.m6.1" class="ltx_Math" alttext="\lambda^{\mathrm{MTV}}_{j}" display="inline"><semantics id="S3.SS3.p3.6.m6.1a"><msubsup id="S3.SS3.p3.6.m6.1.1" xref="S3.SS3.p3.6.m6.1.1.cmml"><mi id="S3.SS3.p3.6.m6.1.1.2.2" xref="S3.SS3.p3.6.m6.1.1.2.2.cmml">λ</mi><mi id="S3.SS3.p3.6.m6.1.1.3" xref="S3.SS3.p3.6.m6.1.1.3.cmml">j</mi><mi id="S3.SS3.p3.6.m6.1.1.2.3" xref="S3.SS3.p3.6.m6.1.1.2.3.cmml">MTV</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.6.m6.1b"><apply id="S3.SS3.p3.6.m6.1.1.cmml" xref="S3.SS3.p3.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.6.m6.1.1.1.cmml" xref="S3.SS3.p3.6.m6.1.1">subscript</csymbol><apply id="S3.SS3.p3.6.m6.1.1.2.cmml" xref="S3.SS3.p3.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.6.m6.1.1.2.1.cmml" xref="S3.SS3.p3.6.m6.1.1">superscript</csymbol><ci id="S3.SS3.p3.6.m6.1.1.2.2.cmml" xref="S3.SS3.p3.6.m6.1.1.2.2">𝜆</ci><ci id="S3.SS3.p3.6.m6.1.1.2.3.cmml" xref="S3.SS3.p3.6.m6.1.1.2.3">MTV</ci></apply><ci id="S3.SS3.p3.6.m6.1.1.3.cmml" xref="S3.SS3.p3.6.m6.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.6.m6.1c">\lambda^{\mathrm{MTV}}_{j}</annotation></semantics></math> for patching in our mean activations. Finally, we can extract <math id="S3.SS3.p3.7.m7.1" class="ltx_Math" alttext="\lambda^{\mathrm{MTV}}_{j}" display="inline"><semantics id="S3.SS3.p3.7.m7.1a"><msubsup id="S3.SS3.p3.7.m7.1.1" xref="S3.SS3.p3.7.m7.1.1.cmml"><mi id="S3.SS3.p3.7.m7.1.1.2.2" xref="S3.SS3.p3.7.m7.1.1.2.2.cmml">λ</mi><mi id="S3.SS3.p3.7.m7.1.1.3" xref="S3.SS3.p3.7.m7.1.1.3.cmml">j</mi><mi id="S3.SS3.p3.7.m7.1.1.2.3" xref="S3.SS3.p3.7.m7.1.1.2.3.cmml">MTV</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.7.m7.1b"><apply id="S3.SS3.p3.7.m7.1.1.cmml" xref="S3.SS3.p3.7.m7.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.7.m7.1.1.1.cmml" xref="S3.SS3.p3.7.m7.1.1">subscript</csymbol><apply id="S3.SS3.p3.7.m7.1.1.2.cmml" xref="S3.SS3.p3.7.m7.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.7.m7.1.1.2.1.cmml" xref="S3.SS3.p3.7.m7.1.1">superscript</csymbol><ci id="S3.SS3.p3.7.m7.1.1.2.2.cmml" xref="S3.SS3.p3.7.m7.1.1.2.2">𝜆</ci><ci id="S3.SS3.p3.7.m7.1.1.2.3.cmml" xref="S3.SS3.p3.7.m7.1.1.2.3">MTV</ci></apply><ci id="S3.SS3.p3.7.m7.1.1.3.cmml" xref="S3.SS3.p3.7.m7.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.7.m7.1c">\lambda^{\mathrm{MTV}}_{j}</annotation></semantics></math>, the optimized indices of attention heads, by sampling our optimized Bernoulli distribution.</p>
<table id="S3.E5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E5.m1.3" class="ltx_math_unparsed" alttext="\lambda^{\mathrm{MTV}}_{j}=\mathrm{MTV\_EXTRACT}(F,[Q_{1},Q_{2},\ldots,Q_{S})],[R_{1},R_{2},\ldots,R_{S}])" display="block"><semantics id="S3.E5.m1.3a"><mrow id="S3.E5.m1.3b"><msubsup id="S3.E5.m1.3.4"><mi id="S3.E5.m1.3.4.2.2">λ</mi><mi id="S3.E5.m1.3.4.3">j</mi><mi id="S3.E5.m1.3.4.2.3">MTV</mi></msubsup><mo id="S3.E5.m1.3.5">=</mo><mi id="S3.E5.m1.3.6">MTV</mi><mi mathvariant="normal" id="S3.E5.m1.3.7">_</mi><mi id="S3.E5.m1.3.8">EXTRACT</mi><mrow id="S3.E5.m1.3.9"><mo stretchy="false" id="S3.E5.m1.3.9.1">(</mo><mi id="S3.E5.m1.2.2">F</mi><mo id="S3.E5.m1.3.9.2">,</mo><mrow id="S3.E5.m1.3.9.3"><mo stretchy="false" id="S3.E5.m1.3.9.3.1">[</mo><msub id="S3.E5.m1.3.9.3.2"><mi id="S3.E5.m1.3.9.3.2.2">Q</mi><mn id="S3.E5.m1.3.9.3.2.3">1</mn></msub><mo id="S3.E5.m1.3.9.3.3">,</mo><msub id="S3.E5.m1.3.9.3.4"><mi id="S3.E5.m1.3.9.3.4.2">Q</mi><mn id="S3.E5.m1.3.9.3.4.3">2</mn></msub><mo id="S3.E5.m1.3.9.3.5">,</mo><mi mathvariant="normal" id="S3.E5.m1.1.1">…</mi><mo id="S3.E5.m1.3.9.3.6">,</mo><msub id="S3.E5.m1.3.9.3.7"><mi id="S3.E5.m1.3.9.3.7.2">Q</mi><mi id="S3.E5.m1.3.9.3.7.3">S</mi></msub><mo stretchy="false" id="S3.E5.m1.3.9.3.8">)</mo></mrow><mo stretchy="false" id="S3.E5.m1.3.9.4">]</mo></mrow><mo id="S3.E5.m1.3.10">,</mo><mrow id="S3.E5.m1.3.11"><mo stretchy="false" id="S3.E5.m1.3.11.1">[</mo><msub id="S3.E5.m1.3.11.2"><mi id="S3.E5.m1.3.11.2.2">R</mi><mn id="S3.E5.m1.3.11.2.3">1</mn></msub><mo id="S3.E5.m1.3.11.3">,</mo><msub id="S3.E5.m1.3.11.4"><mi id="S3.E5.m1.3.11.4.2">R</mi><mn id="S3.E5.m1.3.11.4.3">2</mn></msub><mo id="S3.E5.m1.3.11.5">,</mo><mi mathvariant="normal" id="S3.E5.m1.3.3">…</mi><mo id="S3.E5.m1.3.11.6">,</mo><msub id="S3.E5.m1.3.11.7"><mi id="S3.E5.m1.3.11.7.2">R</mi><mi id="S3.E5.m1.3.11.7.3">S</mi></msub><mo stretchy="false" id="S3.E5.m1.3.11.8">]</mo></mrow><mo stretchy="false" id="S3.E5.m1.3.12">)</mo></mrow><annotation encoding="application/x-tex" id="S3.E5.m1.3c">\lambda^{\mathrm{MTV}}_{j}=\mathrm{MTV\_EXTRACT}(F,[Q_{1},Q_{2},\ldots,Q_{S})],[R_{1},R_{2},\ldots,R_{S}])</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
</div>
<div id="S3.SS3.p4" class="ltx_para">
<p id="S3.SS3.p4.2" class="ltx_p">It is important to note that MTV_EXTRACT does not require finetuning of the LMM parameters, but rather only inference calls. We describe further the underlying details of our adapted MTV_EXTRACT algorithm in Section <a href="#A2" title="Appendix B Additional Method Details ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a> of the Supplementary. Having found <math id="S3.SS3.p4.1.m1.1" class="ltx_Math" alttext="\lambda^{\mathrm{MTV}}_{j}" display="inline"><semantics id="S3.SS3.p4.1.m1.1a"><msubsup id="S3.SS3.p4.1.m1.1.1" xref="S3.SS3.p4.1.m1.1.1.cmml"><mi id="S3.SS3.p4.1.m1.1.1.2.2" xref="S3.SS3.p4.1.m1.1.1.2.2.cmml">λ</mi><mi id="S3.SS3.p4.1.m1.1.1.3" xref="S3.SS3.p4.1.m1.1.1.3.cmml">j</mi><mi id="S3.SS3.p4.1.m1.1.1.2.3" xref="S3.SS3.p4.1.m1.1.1.2.3.cmml">MTV</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS3.p4.1.m1.1b"><apply id="S3.SS3.p4.1.m1.1.1.cmml" xref="S3.SS3.p4.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p4.1.m1.1.1.1.cmml" xref="S3.SS3.p4.1.m1.1.1">subscript</csymbol><apply id="S3.SS3.p4.1.m1.1.1.2.cmml" xref="S3.SS3.p4.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p4.1.m1.1.1.2.1.cmml" xref="S3.SS3.p4.1.m1.1.1">superscript</csymbol><ci id="S3.SS3.p4.1.m1.1.1.2.2.cmml" xref="S3.SS3.p4.1.m1.1.1.2.2">𝜆</ci><ci id="S3.SS3.p4.1.m1.1.1.2.3.cmml" xref="S3.SS3.p4.1.m1.1.1.2.3">MTV</ci></apply><ci id="S3.SS3.p4.1.m1.1.1.3.cmml" xref="S3.SS3.p4.1.m1.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p4.1.m1.1c">\lambda^{\mathrm{MTV}}_{j}</annotation></semantics></math> and <math id="S3.SS3.p4.2.m2.2" class="ltx_Math" alttext="\mu_{l,j}" display="inline"><semantics id="S3.SS3.p4.2.m2.2a"><msub id="S3.SS3.p4.2.m2.2.3" xref="S3.SS3.p4.2.m2.2.3.cmml"><mi id="S3.SS3.p4.2.m2.2.3.2" xref="S3.SS3.p4.2.m2.2.3.2.cmml">μ</mi><mrow id="S3.SS3.p4.2.m2.2.2.2.4" xref="S3.SS3.p4.2.m2.2.2.2.3.cmml"><mi id="S3.SS3.p4.2.m2.1.1.1.1" xref="S3.SS3.p4.2.m2.1.1.1.1.cmml">l</mi><mo id="S3.SS3.p4.2.m2.2.2.2.4.1" xref="S3.SS3.p4.2.m2.2.2.2.3.cmml">,</mo><mi id="S3.SS3.p4.2.m2.2.2.2.2" xref="S3.SS3.p4.2.m2.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p4.2.m2.2b"><apply id="S3.SS3.p4.2.m2.2.3.cmml" xref="S3.SS3.p4.2.m2.2.3"><csymbol cd="ambiguous" id="S3.SS3.p4.2.m2.2.3.1.cmml" xref="S3.SS3.p4.2.m2.2.3">subscript</csymbol><ci id="S3.SS3.p4.2.m2.2.3.2.cmml" xref="S3.SS3.p4.2.m2.2.3.2">𝜇</ci><list id="S3.SS3.p4.2.m2.2.2.2.3.cmml" xref="S3.SS3.p4.2.m2.2.2.2.4"><ci id="S3.SS3.p4.2.m2.1.1.1.1.cmml" xref="S3.SS3.p4.2.m2.1.1.1.1">𝑙</ci><ci id="S3.SS3.p4.2.m2.2.2.2.2.cmml" xref="S3.SS3.p4.2.m2.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p4.2.m2.2c">\mu_{l,j}</annotation></semantics></math>, we describe in what follows, the final procedure to use MTV for inference.</p>
</div>
</section>
<section id="S3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.4 </span>Step 3: Multimodal Task Vector Application</h3>

<div id="S3.SS4.p1" class="ltx_para">
<p id="S3.SS4.p1.3" class="ltx_p">After we have identified a set of attention heads <math id="S3.SS4.p1.1.m1.1" class="ltx_Math" alttext="\lambda^{\mathrm{MTV}}_{j}" display="inline"><semantics id="S3.SS4.p1.1.m1.1a"><msubsup id="S3.SS4.p1.1.m1.1.1" xref="S3.SS4.p1.1.m1.1.1.cmml"><mi id="S3.SS4.p1.1.m1.1.1.2.2" xref="S3.SS4.p1.1.m1.1.1.2.2.cmml">λ</mi><mi id="S3.SS4.p1.1.m1.1.1.3" xref="S3.SS4.p1.1.m1.1.1.3.cmml">j</mi><mi id="S3.SS4.p1.1.m1.1.1.2.3" xref="S3.SS4.p1.1.m1.1.1.2.3.cmml">MTV</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.1.m1.1b"><apply id="S3.SS4.p1.1.m1.1.1.cmml" xref="S3.SS4.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p1.1.m1.1.1.1.cmml" xref="S3.SS4.p1.1.m1.1.1">subscript</csymbol><apply id="S3.SS4.p1.1.m1.1.1.2.cmml" xref="S3.SS4.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p1.1.m1.1.1.2.1.cmml" xref="S3.SS4.p1.1.m1.1.1">superscript</csymbol><ci id="S3.SS4.p1.1.m1.1.1.2.2.cmml" xref="S3.SS4.p1.1.m1.1.1.2.2">𝜆</ci><ci id="S3.SS4.p1.1.m1.1.1.2.3.cmml" xref="S3.SS4.p1.1.m1.1.1.2.3">MTV</ci></apply><ci id="S3.SS4.p1.1.m1.1.1.3.cmml" xref="S3.SS4.p1.1.m1.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.1.m1.1c">\lambda^{\mathrm{MTV}}_{j}</annotation></semantics></math>, it is straightforward to apply MTV for inference. We denote the set of mean activations <math id="S3.SS4.p1.2.m2.1" class="ltx_Math" alttext="\mu^{\mathrm{MTV}}_{j}" display="inline"><semantics id="S3.SS4.p1.2.m2.1a"><msubsup id="S3.SS4.p1.2.m2.1.1" xref="S3.SS4.p1.2.m2.1.1.cmml"><mi id="S3.SS4.p1.2.m2.1.1.2.2" xref="S3.SS4.p1.2.m2.1.1.2.2.cmml">μ</mi><mi id="S3.SS4.p1.2.m2.1.1.3" xref="S3.SS4.p1.2.m2.1.1.3.cmml">j</mi><mi id="S3.SS4.p1.2.m2.1.1.2.3" xref="S3.SS4.p1.2.m2.1.1.2.3.cmml">MTV</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.2.m2.1b"><apply id="S3.SS4.p1.2.m2.1.1.cmml" xref="S3.SS4.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS4.p1.2.m2.1.1.1.cmml" xref="S3.SS4.p1.2.m2.1.1">subscript</csymbol><apply id="S3.SS4.p1.2.m2.1.1.2.cmml" xref="S3.SS4.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS4.p1.2.m2.1.1.2.1.cmml" xref="S3.SS4.p1.2.m2.1.1">superscript</csymbol><ci id="S3.SS4.p1.2.m2.1.1.2.2.cmml" xref="S3.SS4.p1.2.m2.1.1.2.2">𝜇</ci><ci id="S3.SS4.p1.2.m2.1.1.2.3.cmml" xref="S3.SS4.p1.2.m2.1.1.2.3">MTV</ci></apply><ci id="S3.SS4.p1.2.m2.1.1.3.cmml" xref="S3.SS4.p1.2.m2.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.2.m2.1c">\mu^{\mathrm{MTV}}_{j}</annotation></semantics></math> as follows <math id="S3.SS4.p1.3.m3.4" class="ltx_Math" alttext="\mu^{\mathrm{MTV}}_{j}=\{\mu_{l,j}|\forall l\in\lambda^{\mathrm{MTV}}_{j}\}" display="inline"><semantics id="S3.SS4.p1.3.m3.4a"><mrow id="S3.SS4.p1.3.m3.4.4" xref="S3.SS4.p1.3.m3.4.4.cmml"><msubsup id="S3.SS4.p1.3.m3.4.4.4" xref="S3.SS4.p1.3.m3.4.4.4.cmml"><mi id="S3.SS4.p1.3.m3.4.4.4.2.2" xref="S3.SS4.p1.3.m3.4.4.4.2.2.cmml">μ</mi><mi id="S3.SS4.p1.3.m3.4.4.4.3" xref="S3.SS4.p1.3.m3.4.4.4.3.cmml">j</mi><mi id="S3.SS4.p1.3.m3.4.4.4.2.3" xref="S3.SS4.p1.3.m3.4.4.4.2.3.cmml">MTV</mi></msubsup><mo id="S3.SS4.p1.3.m3.4.4.3" xref="S3.SS4.p1.3.m3.4.4.3.cmml">=</mo><mrow id="S3.SS4.p1.3.m3.4.4.2.2" xref="S3.SS4.p1.3.m3.4.4.2.3.cmml"><mo stretchy="false" id="S3.SS4.p1.3.m3.4.4.2.2.3" xref="S3.SS4.p1.3.m3.4.4.2.3.1.cmml">{</mo><msub id="S3.SS4.p1.3.m3.3.3.1.1.1" xref="S3.SS4.p1.3.m3.3.3.1.1.1.cmml"><mi id="S3.SS4.p1.3.m3.3.3.1.1.1.2" xref="S3.SS4.p1.3.m3.3.3.1.1.1.2.cmml">μ</mi><mrow id="S3.SS4.p1.3.m3.2.2.2.4" xref="S3.SS4.p1.3.m3.2.2.2.3.cmml"><mi id="S3.SS4.p1.3.m3.1.1.1.1" xref="S3.SS4.p1.3.m3.1.1.1.1.cmml">l</mi><mo id="S3.SS4.p1.3.m3.2.2.2.4.1" xref="S3.SS4.p1.3.m3.2.2.2.3.cmml">,</mo><mi id="S3.SS4.p1.3.m3.2.2.2.2" xref="S3.SS4.p1.3.m3.2.2.2.2.cmml">j</mi></mrow></msub><mo lspace="0em" rspace="0.167em" id="S3.SS4.p1.3.m3.4.4.2.2.4" xref="S3.SS4.p1.3.m3.4.4.2.3.1.cmml">|</mo><mrow id="S3.SS4.p1.3.m3.4.4.2.2.2" xref="S3.SS4.p1.3.m3.4.4.2.2.2.cmml"><mrow id="S3.SS4.p1.3.m3.4.4.2.2.2.2" xref="S3.SS4.p1.3.m3.4.4.2.2.2.2.cmml"><mo rspace="0.167em" id="S3.SS4.p1.3.m3.4.4.2.2.2.2.1" xref="S3.SS4.p1.3.m3.4.4.2.2.2.2.1.cmml">∀</mo><mi id="S3.SS4.p1.3.m3.4.4.2.2.2.2.2" xref="S3.SS4.p1.3.m3.4.4.2.2.2.2.2.cmml">l</mi></mrow><mo id="S3.SS4.p1.3.m3.4.4.2.2.2.1" xref="S3.SS4.p1.3.m3.4.4.2.2.2.1.cmml">∈</mo><msubsup id="S3.SS4.p1.3.m3.4.4.2.2.2.3" xref="S3.SS4.p1.3.m3.4.4.2.2.2.3.cmml"><mi id="S3.SS4.p1.3.m3.4.4.2.2.2.3.2.2" xref="S3.SS4.p1.3.m3.4.4.2.2.2.3.2.2.cmml">λ</mi><mi id="S3.SS4.p1.3.m3.4.4.2.2.2.3.3" xref="S3.SS4.p1.3.m3.4.4.2.2.2.3.3.cmml">j</mi><mi id="S3.SS4.p1.3.m3.4.4.2.2.2.3.2.3" xref="S3.SS4.p1.3.m3.4.4.2.2.2.3.2.3.cmml">MTV</mi></msubsup></mrow><mo stretchy="false" id="S3.SS4.p1.3.m3.4.4.2.2.5" xref="S3.SS4.p1.3.m3.4.4.2.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.3.m3.4b"><apply id="S3.SS4.p1.3.m3.4.4.cmml" xref="S3.SS4.p1.3.m3.4.4"><eq id="S3.SS4.p1.3.m3.4.4.3.cmml" xref="S3.SS4.p1.3.m3.4.4.3"></eq><apply id="S3.SS4.p1.3.m3.4.4.4.cmml" xref="S3.SS4.p1.3.m3.4.4.4"><csymbol cd="ambiguous" id="S3.SS4.p1.3.m3.4.4.4.1.cmml" xref="S3.SS4.p1.3.m3.4.4.4">subscript</csymbol><apply id="S3.SS4.p1.3.m3.4.4.4.2.cmml" xref="S3.SS4.p1.3.m3.4.4.4"><csymbol cd="ambiguous" id="S3.SS4.p1.3.m3.4.4.4.2.1.cmml" xref="S3.SS4.p1.3.m3.4.4.4">superscript</csymbol><ci id="S3.SS4.p1.3.m3.4.4.4.2.2.cmml" xref="S3.SS4.p1.3.m3.4.4.4.2.2">𝜇</ci><ci id="S3.SS4.p1.3.m3.4.4.4.2.3.cmml" xref="S3.SS4.p1.3.m3.4.4.4.2.3">MTV</ci></apply><ci id="S3.SS4.p1.3.m3.4.4.4.3.cmml" xref="S3.SS4.p1.3.m3.4.4.4.3">𝑗</ci></apply><apply id="S3.SS4.p1.3.m3.4.4.2.3.cmml" xref="S3.SS4.p1.3.m3.4.4.2.2"><csymbol cd="latexml" id="S3.SS4.p1.3.m3.4.4.2.3.1.cmml" xref="S3.SS4.p1.3.m3.4.4.2.2.3">conditional-set</csymbol><apply id="S3.SS4.p1.3.m3.3.3.1.1.1.cmml" xref="S3.SS4.p1.3.m3.3.3.1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p1.3.m3.3.3.1.1.1.1.cmml" xref="S3.SS4.p1.3.m3.3.3.1.1.1">subscript</csymbol><ci id="S3.SS4.p1.3.m3.3.3.1.1.1.2.cmml" xref="S3.SS4.p1.3.m3.3.3.1.1.1.2">𝜇</ci><list id="S3.SS4.p1.3.m3.2.2.2.3.cmml" xref="S3.SS4.p1.3.m3.2.2.2.4"><ci id="S3.SS4.p1.3.m3.1.1.1.1.cmml" xref="S3.SS4.p1.3.m3.1.1.1.1">𝑙</ci><ci id="S3.SS4.p1.3.m3.2.2.2.2.cmml" xref="S3.SS4.p1.3.m3.2.2.2.2">𝑗</ci></list></apply><apply id="S3.SS4.p1.3.m3.4.4.2.2.2.cmml" xref="S3.SS4.p1.3.m3.4.4.2.2.2"><in id="S3.SS4.p1.3.m3.4.4.2.2.2.1.cmml" xref="S3.SS4.p1.3.m3.4.4.2.2.2.1"></in><apply id="S3.SS4.p1.3.m3.4.4.2.2.2.2.cmml" xref="S3.SS4.p1.3.m3.4.4.2.2.2.2"><csymbol cd="latexml" id="S3.SS4.p1.3.m3.4.4.2.2.2.2.1.cmml" xref="S3.SS4.p1.3.m3.4.4.2.2.2.2.1">for-all</csymbol><ci id="S3.SS4.p1.3.m3.4.4.2.2.2.2.2.cmml" xref="S3.SS4.p1.3.m3.4.4.2.2.2.2.2">𝑙</ci></apply><apply id="S3.SS4.p1.3.m3.4.4.2.2.2.3.cmml" xref="S3.SS4.p1.3.m3.4.4.2.2.2.3"><csymbol cd="ambiguous" id="S3.SS4.p1.3.m3.4.4.2.2.2.3.1.cmml" xref="S3.SS4.p1.3.m3.4.4.2.2.2.3">subscript</csymbol><apply id="S3.SS4.p1.3.m3.4.4.2.2.2.3.2.cmml" xref="S3.SS4.p1.3.m3.4.4.2.2.2.3"><csymbol cd="ambiguous" id="S3.SS4.p1.3.m3.4.4.2.2.2.3.2.1.cmml" xref="S3.SS4.p1.3.m3.4.4.2.2.2.3">superscript</csymbol><ci id="S3.SS4.p1.3.m3.4.4.2.2.2.3.2.2.cmml" xref="S3.SS4.p1.3.m3.4.4.2.2.2.3.2.2">𝜆</ci><ci id="S3.SS4.p1.3.m3.4.4.2.2.2.3.2.3.cmml" xref="S3.SS4.p1.3.m3.4.4.2.2.2.3.2.3">MTV</ci></apply><ci id="S3.SS4.p1.3.m3.4.4.2.2.2.3.3.cmml" xref="S3.SS4.p1.3.m3.4.4.2.2.2.3.3">𝑗</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.3.m3.4c">\mu^{\mathrm{MTV}}_{j}=\{\mu_{l,j}|\forall l\in\lambda^{\mathrm{MTV}}_{j}\}</annotation></semantics></math>.</p>
</div>
<div id="S3.SS4.p2" class="ltx_para">
<p id="S3.SS4.p2.5" class="ltx_p">To run downstream inference on a new query <math id="S3.SS4.p2.1.m1.1" class="ltx_Math" alttext="Q_{\mathrm{new}}" display="inline"><semantics id="S3.SS4.p2.1.m1.1a"><msub id="S3.SS4.p2.1.m1.1.1" xref="S3.SS4.p2.1.m1.1.1.cmml"><mi id="S3.SS4.p2.1.m1.1.1.2" xref="S3.SS4.p2.1.m1.1.1.2.cmml">Q</mi><mi id="S3.SS4.p2.1.m1.1.1.3" xref="S3.SS4.p2.1.m1.1.1.3.cmml">new</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.1.m1.1b"><apply id="S3.SS4.p2.1.m1.1.1.cmml" xref="S3.SS4.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.1.m1.1.1.1.cmml" xref="S3.SS4.p2.1.m1.1.1">subscript</csymbol><ci id="S3.SS4.p2.1.m1.1.1.2.cmml" xref="S3.SS4.p2.1.m1.1.1.2">𝑄</ci><ci id="S3.SS4.p2.1.m1.1.1.3.cmml" xref="S3.SS4.p2.1.m1.1.1.3">new</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.1.m1.1c">Q_{\mathrm{new}}</annotation></semantics></math> with our model <math id="S3.SS4.p2.2.m2.1" class="ltx_Math" alttext="F" display="inline"><semantics id="S3.SS4.p2.2.m2.1a"><mi id="S3.SS4.p2.2.m2.1.1" xref="S3.SS4.p2.2.m2.1.1.cmml">F</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.2.m2.1b"><ci id="S3.SS4.p2.2.m2.1.1.cmml" xref="S3.SS4.p2.2.m2.1.1">𝐹</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.2.m2.1c">F</annotation></semantics></math>, we directly replace the values of attention heads <math id="S3.SS4.p2.3.m3.1" class="ltx_Math" alttext="\lambda^{\mathrm{MTV}}_{j}" display="inline"><semantics id="S3.SS4.p2.3.m3.1a"><msubsup id="S3.SS4.p2.3.m3.1.1" xref="S3.SS4.p2.3.m3.1.1.cmml"><mi id="S3.SS4.p2.3.m3.1.1.2.2" xref="S3.SS4.p2.3.m3.1.1.2.2.cmml">λ</mi><mi id="S3.SS4.p2.3.m3.1.1.3" xref="S3.SS4.p2.3.m3.1.1.3.cmml">j</mi><mi id="S3.SS4.p2.3.m3.1.1.2.3" xref="S3.SS4.p2.3.m3.1.1.2.3.cmml">MTV</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.3.m3.1b"><apply id="S3.SS4.p2.3.m3.1.1.cmml" xref="S3.SS4.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.3.m3.1.1.1.cmml" xref="S3.SS4.p2.3.m3.1.1">subscript</csymbol><apply id="S3.SS4.p2.3.m3.1.1.2.cmml" xref="S3.SS4.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.3.m3.1.1.2.1.cmml" xref="S3.SS4.p2.3.m3.1.1">superscript</csymbol><ci id="S3.SS4.p2.3.m3.1.1.2.2.cmml" xref="S3.SS4.p2.3.m3.1.1.2.2">𝜆</ci><ci id="S3.SS4.p2.3.m3.1.1.2.3.cmml" xref="S3.SS4.p2.3.m3.1.1.2.3">MTV</ci></apply><ci id="S3.SS4.p2.3.m3.1.1.3.cmml" xref="S3.SS4.p2.3.m3.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.3.m3.1c">\lambda^{\mathrm{MTV}}_{j}</annotation></semantics></math> with <math id="S3.SS4.p2.4.m4.1" class="ltx_Math" alttext="\mu^{\mathrm{MTV}}_{j}" display="inline"><semantics id="S3.SS4.p2.4.m4.1a"><msubsup id="S3.SS4.p2.4.m4.1.1" xref="S3.SS4.p2.4.m4.1.1.cmml"><mi id="S3.SS4.p2.4.m4.1.1.2.2" xref="S3.SS4.p2.4.m4.1.1.2.2.cmml">μ</mi><mi id="S3.SS4.p2.4.m4.1.1.3" xref="S3.SS4.p2.4.m4.1.1.3.cmml">j</mi><mi id="S3.SS4.p2.4.m4.1.1.2.3" xref="S3.SS4.p2.4.m4.1.1.2.3.cmml">MTV</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.4.m4.1b"><apply id="S3.SS4.p2.4.m4.1.1.cmml" xref="S3.SS4.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.4.m4.1.1.1.cmml" xref="S3.SS4.p2.4.m4.1.1">subscript</csymbol><apply id="S3.SS4.p2.4.m4.1.1.2.cmml" xref="S3.SS4.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.4.m4.1.1.2.1.cmml" xref="S3.SS4.p2.4.m4.1.1">superscript</csymbol><ci id="S3.SS4.p2.4.m4.1.1.2.2.cmml" xref="S3.SS4.p2.4.m4.1.1.2.2">𝜇</ci><ci id="S3.SS4.p2.4.m4.1.1.2.3.cmml" xref="S3.SS4.p2.4.m4.1.1.2.3">MTV</ci></apply><ci id="S3.SS4.p2.4.m4.1.1.3.cmml" xref="S3.SS4.p2.4.m4.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.4.m4.1c">\mu^{\mathrm{MTV}}_{j}</annotation></semantics></math> and produce the following response <math id="S3.SS4.p2.5.m5.1" class="ltx_Math" alttext="R_{\mathrm{new}}" display="inline"><semantics id="S3.SS4.p2.5.m5.1a"><msub id="S3.SS4.p2.5.m5.1.1" xref="S3.SS4.p2.5.m5.1.1.cmml"><mi id="S3.SS4.p2.5.m5.1.1.2" xref="S3.SS4.p2.5.m5.1.1.2.cmml">R</mi><mi id="S3.SS4.p2.5.m5.1.1.3" xref="S3.SS4.p2.5.m5.1.1.3.cmml">new</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.5.m5.1b"><apply id="S3.SS4.p2.5.m5.1.1.cmml" xref="S3.SS4.p2.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.5.m5.1.1.1.cmml" xref="S3.SS4.p2.5.m5.1.1">subscript</csymbol><ci id="S3.SS4.p2.5.m5.1.1.2.cmml" xref="S3.SS4.p2.5.m5.1.1.2">𝑅</ci><ci id="S3.SS4.p2.5.m5.1.1.3.cmml" xref="S3.SS4.p2.5.m5.1.1.3">new</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.5.m5.1c">R_{\mathrm{new}}</annotation></semantics></math>:</p>
<table id="S3.E6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E6.m1.1" class="ltx_Math" alttext="R_{\mathrm{new}}=F(Q_{new}|\lambda^{\mathrm{MTV}}_{j},\mu^{\mathrm{MTV}}_{j})" display="block"><semantics id="S3.E6.m1.1a"><mrow id="S3.E6.m1.1.1" xref="S3.E6.m1.1.1.cmml"><msub id="S3.E6.m1.1.1.3" xref="S3.E6.m1.1.1.3.cmml"><mi id="S3.E6.m1.1.1.3.2" xref="S3.E6.m1.1.1.3.2.cmml">R</mi><mi id="S3.E6.m1.1.1.3.3" xref="S3.E6.m1.1.1.3.3.cmml">new</mi></msub><mo id="S3.E6.m1.1.1.2" xref="S3.E6.m1.1.1.2.cmml">=</mo><mrow id="S3.E6.m1.1.1.1" xref="S3.E6.m1.1.1.1.cmml"><mi id="S3.E6.m1.1.1.1.3" xref="S3.E6.m1.1.1.1.3.cmml">F</mi><mo lspace="0em" rspace="0em" id="S3.E6.m1.1.1.1.2" xref="S3.E6.m1.1.1.1.2.cmml">​</mo><mrow id="S3.E6.m1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E6.m1.1.1.1.1.1.2" xref="S3.E6.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E6.m1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.cmml"><msub id="S3.E6.m1.1.1.1.1.1.1.4" xref="S3.E6.m1.1.1.1.1.1.1.4.cmml"><mi id="S3.E6.m1.1.1.1.1.1.1.4.2" xref="S3.E6.m1.1.1.1.1.1.1.4.2.cmml">Q</mi><mrow id="S3.E6.m1.1.1.1.1.1.1.4.3" xref="S3.E6.m1.1.1.1.1.1.1.4.3.cmml"><mi id="S3.E6.m1.1.1.1.1.1.1.4.3.2" xref="S3.E6.m1.1.1.1.1.1.1.4.3.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="S3.E6.m1.1.1.1.1.1.1.4.3.1" xref="S3.E6.m1.1.1.1.1.1.1.4.3.1.cmml">​</mo><mi id="S3.E6.m1.1.1.1.1.1.1.4.3.3" xref="S3.E6.m1.1.1.1.1.1.1.4.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.E6.m1.1.1.1.1.1.1.4.3.1a" xref="S3.E6.m1.1.1.1.1.1.1.4.3.1.cmml">​</mo><mi id="S3.E6.m1.1.1.1.1.1.1.4.3.4" xref="S3.E6.m1.1.1.1.1.1.1.4.3.4.cmml">w</mi></mrow></msub><mo fence="false" id="S3.E6.m1.1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.1.3.cmml">|</mo><mrow id="S3.E6.m1.1.1.1.1.1.1.2.2" xref="S3.E6.m1.1.1.1.1.1.1.2.3.cmml"><msubsup id="S3.E6.m1.1.1.1.1.1.1.1.1.1" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E6.m1.1.1.1.1.1.1.1.1.1.2.2" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.2.2.cmml">λ</mi><mi id="S3.E6.m1.1.1.1.1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.3.cmml">j</mi><mi id="S3.E6.m1.1.1.1.1.1.1.1.1.1.2.3" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.2.3.cmml">MTV</mi></msubsup><mo id="S3.E6.m1.1.1.1.1.1.1.2.2.3" xref="S3.E6.m1.1.1.1.1.1.1.2.3.cmml">,</mo><msubsup id="S3.E6.m1.1.1.1.1.1.1.2.2.2" xref="S3.E6.m1.1.1.1.1.1.1.2.2.2.cmml"><mi id="S3.E6.m1.1.1.1.1.1.1.2.2.2.2.2" xref="S3.E6.m1.1.1.1.1.1.1.2.2.2.2.2.cmml">μ</mi><mi id="S3.E6.m1.1.1.1.1.1.1.2.2.2.3" xref="S3.E6.m1.1.1.1.1.1.1.2.2.2.3.cmml">j</mi><mi id="S3.E6.m1.1.1.1.1.1.1.2.2.2.2.3" xref="S3.E6.m1.1.1.1.1.1.1.2.2.2.2.3.cmml">MTV</mi></msubsup></mrow></mrow><mo stretchy="false" id="S3.E6.m1.1.1.1.1.1.3" xref="S3.E6.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E6.m1.1b"><apply id="S3.E6.m1.1.1.cmml" xref="S3.E6.m1.1.1"><eq id="S3.E6.m1.1.1.2.cmml" xref="S3.E6.m1.1.1.2"></eq><apply id="S3.E6.m1.1.1.3.cmml" xref="S3.E6.m1.1.1.3"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.3.1.cmml" xref="S3.E6.m1.1.1.3">subscript</csymbol><ci id="S3.E6.m1.1.1.3.2.cmml" xref="S3.E6.m1.1.1.3.2">𝑅</ci><ci id="S3.E6.m1.1.1.3.3.cmml" xref="S3.E6.m1.1.1.3.3">new</ci></apply><apply id="S3.E6.m1.1.1.1.cmml" xref="S3.E6.m1.1.1.1"><times id="S3.E6.m1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1.2"></times><ci id="S3.E6.m1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.3">𝐹</ci><apply id="S3.E6.m1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1"><csymbol cd="latexml" id="S3.E6.m1.1.1.1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.3">conditional</csymbol><apply id="S3.E6.m1.1.1.1.1.1.1.4.cmml" xref="S3.E6.m1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.4.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.4">subscript</csymbol><ci id="S3.E6.m1.1.1.1.1.1.1.4.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.4.2">𝑄</ci><apply id="S3.E6.m1.1.1.1.1.1.1.4.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.4.3"><times id="S3.E6.m1.1.1.1.1.1.1.4.3.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.4.3.1"></times><ci id="S3.E6.m1.1.1.1.1.1.1.4.3.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.4.3.2">𝑛</ci><ci id="S3.E6.m1.1.1.1.1.1.1.4.3.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.4.3.3">𝑒</ci><ci id="S3.E6.m1.1.1.1.1.1.1.4.3.4.cmml" xref="S3.E6.m1.1.1.1.1.1.1.4.3.4">𝑤</ci></apply></apply><list id="S3.E6.m1.1.1.1.1.1.1.2.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.2.2"><apply id="S3.E6.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1">subscript</csymbol><apply id="S3.E6.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1">superscript</csymbol><ci id="S3.E6.m1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.2.2">𝜆</ci><ci id="S3.E6.m1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.2.3">MTV</ci></apply><ci id="S3.E6.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.1.1.1.3">𝑗</ci></apply><apply id="S3.E6.m1.1.1.1.1.1.1.2.2.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.2.2.2.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.2.2.2">subscript</csymbol><apply id="S3.E6.m1.1.1.1.1.1.1.2.2.2.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="S3.E6.m1.1.1.1.1.1.1.2.2.2.2.1.cmml" xref="S3.E6.m1.1.1.1.1.1.1.2.2.2">superscript</csymbol><ci id="S3.E6.m1.1.1.1.1.1.1.2.2.2.2.2.cmml" xref="S3.E6.m1.1.1.1.1.1.1.2.2.2.2.2">𝜇</ci><ci id="S3.E6.m1.1.1.1.1.1.1.2.2.2.2.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.2.2.2.2.3">MTV</ci></apply><ci id="S3.E6.m1.1.1.1.1.1.1.2.2.2.3.cmml" xref="S3.E6.m1.1.1.1.1.1.1.2.2.2.3">𝑗</ci></apply></list></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E6.m1.1c">R_{\mathrm{new}}=F(Q_{new}|\lambda^{\mathrm{MTV}}_{j},\mu^{\mathrm{MTV}}_{j})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
<p id="S3.SS4.p2.8" class="ltx_p"><math id="S3.SS4.p2.6.m1.1" class="ltx_Math" alttext="R_{\mathrm{new}}" display="inline"><semantics id="S3.SS4.p2.6.m1.1a"><msub id="S3.SS4.p2.6.m1.1.1" xref="S3.SS4.p2.6.m1.1.1.cmml"><mi id="S3.SS4.p2.6.m1.1.1.2" xref="S3.SS4.p2.6.m1.1.1.2.cmml">R</mi><mi id="S3.SS4.p2.6.m1.1.1.3" xref="S3.SS4.p2.6.m1.1.1.3.cmml">new</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.6.m1.1b"><apply id="S3.SS4.p2.6.m1.1.1.cmml" xref="S3.SS4.p2.6.m1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p2.6.m1.1.1.1.cmml" xref="S3.SS4.p2.6.m1.1.1">subscript</csymbol><ci id="S3.SS4.p2.6.m1.1.1.2.cmml" xref="S3.SS4.p2.6.m1.1.1.2">𝑅</ci><ci id="S3.SS4.p2.6.m1.1.1.3.cmml" xref="S3.SS4.p2.6.m1.1.1.3">new</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.6.m1.1c">R_{\mathrm{new}}</annotation></semantics></math> is thus a response generated using many shots of multimodal examples as implicit context via MTV. The key insight of our method is the importance of <math id="S3.SS4.p2.7.m2.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S3.SS4.p2.7.m2.1a"><mi id="S3.SS4.p2.7.m2.1.1" xref="S3.SS4.p2.7.m2.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.7.m2.1b"><ci id="S3.SS4.p2.7.m2.1.1.cmml" xref="S3.SS4.p2.7.m2.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.7.m2.1c">N</annotation></semantics></math> (the number of multimodal examples) and many <math id="S3.SS4.p2.8.m3.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S3.SS4.p2.8.m3.1a"><mi id="S3.SS4.p2.8.m3.1.1" xref="S3.SS4.p2.8.m3.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.8.m3.1b"><ci id="S3.SS4.p2.8.m3.1.1.cmml" xref="S3.SS4.p2.8.m3.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.8.m3.1c">T</annotation></semantics></math> (the number of iterations) during the calculation of MTV. This enables an LMM to go beyond its context length to learn more nuanced properties of the task from seeing many examples. Additionally, insertion of MTV directly into the LMM also obviates the need for any context length during downstream inference, actually <span id="S3.SS4.p2.8.1" class="ltx_text ltx_font_italic">freeing</span> additional context for other use (e.g., an additional prompt, more ICL examples, etc.). Finally, because we align the attention-head locations with the downstream task, MTV can be effectively applied to zero-shot and different ICL settings.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Evaluation</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">In order for LMMs to perform multimodal ICL, it is important for interleaved data to be included in pretraining. We apply our MTV approach to Qwen-VL <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">6</span></a>]</cite>, Idefics2-8B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib38" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">38</span></a>]</cite>, and ViLA-1.5-8B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib47" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">47</span></a>]</cite> three popular interleaved LMMs. For each model, we compare our method to using few-shot ICL across different vision-and-language tasks like VQA and object identification.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Implementation Details</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">We implemented MTV using PyTorch <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">60</span></a>]</cite>. We used each model’s respective official implementation. While the compute and memory requirements differ slightly between models, all our experiments can be run on a single NVIDIA A6000 GPU. For additional information, refer to Supplementary Section <a href="#A3" title="Appendix C Additional Implementation Details ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>. Our model and weights will be released upon acceptance, and our code is in Supplementary.</p>
</div>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Models</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p">In this work, we apply MTV to the following interleaved LMMs as they are better-suited for multimodal ICL as shown by <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">16</span></a>]</cite>: (1) <span id="S4.SS2.p1.1.1" class="ltx_text ltx_font_bold">QwenVL</span> <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">6</span></a>]</cite> is a LLaMA-based model that has the ability to process high-resolution images, and its two-stage pre-training methodology, which includes multi-task finetuning and interleaved text-image data.
(2) <span id="S4.SS2.p1.1.2" class="ltx_text ltx_font_bold">Idefics2-8B</span> <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib39" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">39</span></a>]</cite> is a Mistral-based model that benefits from its pre-training on the expansive OBELICS dataset, which comprises a web-scale collection of interleaved image-text documents. We utilize the base version of the model. This demonstrates multimodal in-context learning abilities.
(3) <span id="S4.SS2.p1.1.3" class="ltx_text ltx_font_bold">ViLA-1.5-8B</span>. ViLA-1.5-8B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib47" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">47</span></a>]</cite> is an architecture that leverages LLaMA-3 as the LLM backbone. As in others, a significant portion of the model’s pretraining data is interleaved text-image data.</p>
</div>
<figure id="S4.T1" class="ltx_table">

<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S4.T1.4.1.1" class="ltx_text" style="font-size:90%;">Table 1</span>: </span><span id="S4.T1.5.2" class="ltx_text ltx_font_bold" style="font-size:90%;">Results<span id="S4.T1.5.2.1" class="ltx_text ltx_font_medium">. (Left) MTV evaluated on VQA datasets. (Right) MTV evaluated on object classification datasets. The baselines are shown in <span id="S4.T1.5.2.1.1" class="ltx_text" style="color:#808080;">gray</span>.</span></span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.T1.st1" class="ltx_table ltx_figure_panel ltx_align_center">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S4.T1.st1.3.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="S4.T1.st1.4.2" class="ltx_text ltx_font_bold" style="font-size:90%;">MTV on VQA Benchmarks</span></figcaption>
<table id="S4.T1.st1.5" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T1.st1.5.1.1" class="ltx_tr">
<td id="S4.T1.st1.5.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt ltx_border_t" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.1.1.1.1.1" class="ltx_p">Model</span>
</span>
</td>
<td id="S4.T1.st1.5.1.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.1.1.2.1.1" class="ltx_p" style="width:108.4pt;">VizWiz</span>
</span>
</td>
<td id="S4.T1.st1.5.1.1.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.1.1.3.1.1" class="ltx_p" style="width:108.4pt;">OK-VQA</span>
</span>
</td>
</tr>
<tr id="S4.T1.st1.5.2.2" class="ltx_tr" style="background-color:#D9D9D9;">
<td id="S4.T1.st1.5.2.2.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.2.2.1.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st1.5.2.2.1.1.1" class="ltx_p">Flamingo 9B</span>
</span>
</td>
<td id="S4.T1.st1.5.2.2.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.2.2.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st1.5.2.2.2.1.1" class="ltx_p" style="width:108.4pt;">28.8</span>
</span>
</td>
<td id="S4.T1.st1.5.2.2.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.2.2.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st1.5.2.2.3.1.1" class="ltx_p" style="width:108.4pt;">44.7</span>
</span>
</td>
</tr>
<tr id="S4.T1.st1.5.3.3" class="ltx_tr" style="background-color:#D9D9D9;">
<td id="S4.T1.st1.5.3.3.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.3.3.1.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st1.5.3.3.1.1.1" class="ltx_p">+4-shot ICL</span>
</span>
</td>
<td id="S4.T1.st1.5.3.3.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.3.3.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st1.5.3.3.2.1.1" class="ltx_p" style="width:108.4pt;">34.9</span>
</span>
</td>
<td id="S4.T1.st1.5.3.3.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.3.3.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st1.5.3.3.3.1.1" class="ltx_p" style="width:108.4pt;">49.3</span>
</span>
</td>
</tr>
<tr id="S4.T1.st1.5.4.4" class="ltx_tr" style="background-color:#D9D9D9;">
<td id="S4.T1.st1.5.4.4.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.4.4.1.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st1.5.4.4.1.1.1" class="ltx_p">+8-shot ICL</span>
</span>
</td>
<td id="S4.T1.st1.5.4.4.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.4.4.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st1.5.4.4.2.1.1" class="ltx_p" style="width:108.4pt;">39.4</span>
</span>
</td>
<td id="S4.T1.st1.5.4.4.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.4.4.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st1.5.4.4.3.1.1" class="ltx_p" style="width:108.4pt;">50.0</span>
</span>
</td>
</tr>
<tr id="S4.T1.st1.5.5.5" class="ltx_tr" style="background-color:#D9D9D9;">
<td id="S4.T1.st1.5.5.5.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.5.5.1.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st1.5.5.5.1.1.1" class="ltx_p">Blip3</span>
</span>
</td>
<td id="S4.T1.st1.5.5.5.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.5.5.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st1.5.5.5.2.1.1" class="ltx_p" style="width:108.4pt;">21.2</span>
</span>
</td>
<td id="S4.T1.st1.5.5.5.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.5.5.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st1.5.5.5.3.1.1" class="ltx_p" style="width:108.4pt;">26.5</span>
</span>
</td>
</tr>
<tr id="S4.T1.st1.5.6.6" class="ltx_tr" style="background-color:#D9D9D9;">
<td id="S4.T1.st1.5.6.6.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.6.6.1.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st1.5.6.6.1.1.1" class="ltx_p">+4-shot ICL</span>
</span>
</td>
<td id="S4.T1.st1.5.6.6.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.6.6.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st1.5.6.6.2.1.1" class="ltx_p" style="width:108.4pt;">38.4</span>
</span>
</td>
<td id="S4.T1.st1.5.6.6.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.6.6.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st1.5.6.6.3.1.1" class="ltx_p" style="width:108.4pt;">49.2</span>
</span>
</td>
</tr>
<tr id="S4.T1.st1.5.7.7" class="ltx_tr" style="background-color:#D9D9D9;">
<td id="S4.T1.st1.5.7.7.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.7.7.1.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st1.5.7.7.1.1.1" class="ltx_p">+8-shot ICL</span>
</span>
</td>
<td id="S4.T1.st1.5.7.7.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.7.7.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st1.5.7.7.2.1.1" class="ltx_p" style="width:108.4pt;">44.3</span>
</span>
</td>
<td id="S4.T1.st1.5.7.7.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.7.7.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st1.5.7.7.3.1.1" class="ltx_p" style="width:108.4pt;">49.1</span>
</span>
</td>
</tr>
<tr id="S4.T1.st1.5.8.8" class="ltx_tr">
<td id="S4.T1.st1.5.8.8.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.8.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.8.8.1.1.1" class="ltx_p">Qwen-VL-7B</span>
</span>
</td>
<td id="S4.T1.st1.5.8.8.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.8.8.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.8.8.2.1.1" class="ltx_p" style="width:108.4pt;">35.2</span>
</span>
</td>
<td id="S4.T1.st1.5.8.8.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.8.8.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.8.8.3.1.1" class="ltx_p" style="width:108.4pt;">58.6</span>
</span>
</td>
</tr>
<tr id="S4.T1.st1.5.9.9" class="ltx_tr">
<td id="S4.T1.st1.5.9.9.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.9.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.9.9.1.1.1" class="ltx_p">+4-shot ICL</span>
</span>
</td>
<td id="S4.T1.st1.5.9.9.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.9.9.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.9.9.2.1.1" class="ltx_p" style="width:108.4pt;">42.0</span>
</span>
</td>
<td id="S4.T1.st1.5.9.9.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.9.9.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.9.9.3.1.1" class="ltx_p" style="width:108.4pt;"><span id="S4.T1.st1.5.9.9.3.1.1.1" class="ltx_text ltx_font_bold">62.0</span></span>
</span>
</td>
</tr>
<tr id="S4.T1.st1.5.10.10" class="ltx_tr">
<td id="S4.T1.st1.5.10.10.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.10.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.10.10.1.1.1" class="ltx_p">+8-shot ICL</span>
</span>
</td>
<td id="S4.T1.st1.5.10.10.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.10.10.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.10.10.2.1.1" class="ltx_p" style="width:108.4pt;">44.3</span>
</span>
</td>
<td id="S4.T1.st1.5.10.10.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.10.10.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.10.10.3.1.1" class="ltx_p" style="width:108.4pt;">61.5</span>
</span>
</td>
</tr>
<tr id="S4.T1.st1.5.11.11" class="ltx_tr">
<td id="S4.T1.st1.5.11.11.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.11.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.11.11.1.1.1" class="ltx_p">+<span id="S4.T1.st1.5.11.11.1.1.1.1" class="ltx_text ltx_font_bold">MTV</span></span>
</span>
</td>
<td id="S4.T1.st1.5.11.11.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.11.11.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.11.11.2.1.1" class="ltx_p" style="width:108.4pt;"><span id="S4.T1.st1.5.11.11.2.1.1.1" class="ltx_text ltx_font_bold">45.6</span></span>
</span>
</td>
<td id="S4.T1.st1.5.11.11.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.11.11.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.11.11.3.1.1" class="ltx_p" style="width:108.4pt;"><span id="S4.T1.st1.5.11.11.3.1.1.1" class="ltx_text ltx_font_bold">62.0</span></span>
</span>
</td>
</tr>
<tr id="S4.T1.st1.5.12.12" class="ltx_tr">
<td id="S4.T1.st1.5.12.12.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.12.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.12.12.1.1.1" class="ltx_p">Idefics2</span>
</span>
</td>
<td id="S4.T1.st1.5.12.12.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.12.12.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.12.12.2.1.1" class="ltx_p" style="width:108.4pt;">31.3</span>
</span>
</td>
<td id="S4.T1.st1.5.12.12.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.12.12.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.12.12.3.1.1" class="ltx_p" style="width:108.4pt;">52.4</span>
</span>
</td>
</tr>
<tr id="S4.T1.st1.5.13.13" class="ltx_tr">
<td id="S4.T1.st1.5.13.13.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.13.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.13.13.1.1.1" class="ltx_p">+4-shot ICL</span>
</span>
</td>
<td id="S4.T1.st1.5.13.13.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.13.13.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.13.13.2.1.1" class="ltx_p" style="width:108.4pt;">40.8</span>
</span>
</td>
<td id="S4.T1.st1.5.13.13.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.13.13.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.13.13.3.1.1" class="ltx_p" style="width:108.4pt;">51.5</span>
</span>
</td>
</tr>
<tr id="S4.T1.st1.5.14.14" class="ltx_tr">
<td id="S4.T1.st1.5.14.14.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.14.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.14.14.1.1.1" class="ltx_p">+8-shot ICL</span>
</span>
</td>
<td id="S4.T1.st1.5.14.14.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.14.14.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.14.14.2.1.1" class="ltx_p" style="width:108.4pt;">43.8</span>
</span>
</td>
<td id="S4.T1.st1.5.14.14.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.14.14.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.14.14.3.1.1" class="ltx_p" style="width:108.4pt;">52.3</span>
</span>
</td>
</tr>
<tr id="S4.T1.st1.5.15.15" class="ltx_tr">
<td id="S4.T1.st1.5.15.15.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.15.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.15.15.1.1.1" class="ltx_p">+<span id="S4.T1.st1.5.15.15.1.1.1.1" class="ltx_text ltx_font_bold">MTV</span></span>
</span>
</td>
<td id="S4.T1.st1.5.15.15.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.15.15.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.15.15.2.1.1" class="ltx_p" style="width:108.4pt;"><span id="S4.T1.st1.5.15.15.2.1.1.1" class="ltx_text ltx_font_bold">52.5</span></span>
</span>
</td>
<td id="S4.T1.st1.5.15.15.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.15.15.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.15.15.3.1.1" class="ltx_p" style="width:108.4pt;"><span id="S4.T1.st1.5.15.15.3.1.1.1" class="ltx_text ltx_font_bold">53.0</span></span>
</span>
</td>
</tr>
<tr id="S4.T1.st1.5.16.16" class="ltx_tr">
<td id="S4.T1.st1.5.16.16.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.16.16.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.16.16.1.1.1" class="ltx_p">Llama3-VILA-1.5-8B</span>
</span>
</td>
<td id="S4.T1.st1.5.16.16.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.16.16.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.16.16.2.1.1" class="ltx_p" style="width:108.4pt;">28.0</span>
</span>
</td>
<td id="S4.T1.st1.5.16.16.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.16.16.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.16.16.3.1.1" class="ltx_p" style="width:108.4pt;">32.8</span>
</span>
</td>
</tr>
<tr id="S4.T1.st1.5.17.17" class="ltx_tr">
<td id="S4.T1.st1.5.17.17.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.17.17.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.17.17.1.1.1" class="ltx_p">+4-shot ICL</span>
</span>
</td>
<td id="S4.T1.st1.5.17.17.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.17.17.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.17.17.2.1.1" class="ltx_p" style="width:108.4pt;">39.3</span>
</span>
</td>
<td id="S4.T1.st1.5.17.17.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.17.17.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.17.17.3.1.1" class="ltx_p" style="width:108.4pt;">35.6</span>
</span>
</td>
</tr>
<tr id="S4.T1.st1.5.18.18" class="ltx_tr">
<td id="S4.T1.st1.5.18.18.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.18.18.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.18.18.1.1.1" class="ltx_p">+8-shot ICL</span>
</span>
</td>
<td id="S4.T1.st1.5.18.18.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.18.18.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.18.18.2.1.1" class="ltx_p" style="width:108.4pt;">44.2</span>
</span>
</td>
<td id="S4.T1.st1.5.18.18.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.18.18.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.18.18.3.1.1" class="ltx_p" style="width:108.4pt;">36.5</span>
</span>
</td>
</tr>
<tr id="S4.T1.st1.5.19.19" class="ltx_tr">
<td id="S4.T1.st1.5.19.19.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb ltx_border_b" style="width:130.1pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.19.19.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.19.19.1.1.1" class="ltx_p">+<span id="S4.T1.st1.5.19.19.1.1.1.1" class="ltx_text ltx_font_bold">MTV</span></span>
</span>
</td>
<td id="S4.T1.st1.5.19.19.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb ltx_border_b" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.19.19.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.19.19.2.1.1" class="ltx_p" style="width:108.4pt;"><span id="S4.T1.st1.5.19.19.2.1.1.1" class="ltx_text ltx_font_bold">55.2</span></span>
</span>
</td>
<td id="S4.T1.st1.5.19.19.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb ltx_border_b" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st1.5.19.19.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st1.5.19.19.3.1.1" class="ltx_p" style="width:108.4pt;"><span id="S4.T1.st1.5.19.19.3.1.1.1" class="ltx_text ltx_font_bold">40.6</span></span>
</span>
</td>
</tr>
</tbody>
</table>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.T1.st2" class="ltx_table ltx_figure_panel ltx_align_center">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S4.T1.st2.3.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="S4.T1.st2.4.2" class="ltx_text ltx_font_bold" style="font-size:90%;">MTV on Object Classification</span></figcaption>
<table id="S4.T1.st2.5" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T1.st2.5.1.1" class="ltx_tr">
<td id="S4.T1.st2.5.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt ltx_border_t" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.1.1.1.1.1" class="ltx_p">Model</span>
</span>
</td>
<td id="S4.T1.st2.5.1.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.1.1.2.1.1" class="ltx_p" style="width:86.7pt;">Flowers</span>
</span>
</td>
<td id="S4.T1.st2.5.1.1.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.1.1.3.1.1" class="ltx_p" style="width:52.0pt;">CUB</span>
</span>
</td>
</tr>
<tr id="S4.T1.st2.5.2.2" class="ltx_tr" style="background-color:#D9D9D9;">
<td id="S4.T1.st2.5.2.2.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.2.2.1.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.2.2.1.1.1" class="ltx_p">LLaVA-1.5-13B</span>
</span>
</td>
<td id="S4.T1.st2.5.2.2.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.2.2.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.2.2.2.1.1" class="ltx_p" style="width:86.7pt;"></span>
</span>
</td>
<td id="S4.T1.st2.5.2.2.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.2.2.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.2.2.3.1.1" class="ltx_p" style="width:52.0pt;"></span>
</span>
</td>
</tr>
<tr id="S4.T1.st2.5.3.3" class="ltx_tr" style="background-color:#D9D9D9;">
<td id="S4.T1.st2.5.3.3.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.3.3.1.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.3.3.1.1.1" class="ltx_p">+ 1-shot ICL</span>
</span>
</td>
<td id="S4.T1.st2.5.3.3.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.3.3.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.3.3.2.1.1" class="ltx_p" style="width:86.7pt;">58.60</span>
</span>
</td>
<td id="S4.T1.st2.5.3.3.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.3.3.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.3.3.3.1.1" class="ltx_p" style="width:52.0pt;">58.24</span>
</span>
</td>
</tr>
<tr id="S4.T1.st2.5.4.4" class="ltx_tr" style="background-color:#D9D9D9;">
<td id="S4.T1.st2.5.4.4.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.4.4.1.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.4.4.1.1.1" class="ltx_p">LLaVA-1.6-13B</span>
</span>
</td>
<td id="S4.T1.st2.5.4.4.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.4.4.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.4.4.2.1.1" class="ltx_p" style="width:86.7pt;"></span>
</span>
</td>
<td id="S4.T1.st2.5.4.4.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.4.4.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.4.4.3.1.1" class="ltx_p" style="width:52.0pt;"></span>
</span>
</td>
</tr>
<tr id="S4.T1.st2.5.5.5" class="ltx_tr" style="background-color:#D9D9D9;">
<td id="S4.T1.st2.5.5.5.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.5.5.1.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.5.5.1.1.1" class="ltx_p">+ 1-shot ICL</span>
</span>
</td>
<td id="S4.T1.st2.5.5.5.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.5.5.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.5.5.2.1.1" class="ltx_p" style="width:86.7pt;">65.58</span>
</span>
</td>
<td id="S4.T1.st2.5.5.5.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.5.5.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.5.5.3.1.1" class="ltx_p" style="width:52.0pt;">67.90</span>
</span>
</td>
</tr>
<tr id="S4.T1.st2.5.6.6" class="ltx_tr" style="background-color:#D9D9D9;">
<td id="S4.T1.st2.5.6.6.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.6.6.1.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.6.6.1.1.1" class="ltx_p">Flamingo 9B</span>
</span>
</td>
<td id="S4.T1.st2.5.6.6.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.6.6.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.6.6.2.1.1" class="ltx_p" style="width:86.7pt;"></span>
</span>
</td>
<td id="S4.T1.st2.5.6.6.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.6.6.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.6.6.3.1.1" class="ltx_p" style="width:52.0pt;"></span>
</span>
</td>
</tr>
<tr id="S4.T1.st2.5.7.7" class="ltx_tr" style="background-color:#D9D9D9;">
<td id="S4.T1.st2.5.7.7.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.7.7.1.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.7.7.1.1.1" class="ltx_p">+ 1-shot ICL 9B</span>
</span>
</td>
<td id="S4.T1.st2.5.7.7.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.7.7.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.7.7.2.1.1" class="ltx_p" style="width:86.7pt;">48.78</span>
</span>
</td>
<td id="S4.T1.st2.5.7.7.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.7.7.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.7.7.3.1.1" class="ltx_p" style="width:52.0pt;">51.2</span>
</span>
</td>
</tr>
<tr id="S4.T1.st2.5.8.8" class="ltx_tr" style="background-color:#D9D9D9;">
<td id="S4.T1.st2.5.8.8.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.8.8.1.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.8.8.1.1.1" class="ltx_p">IDEFICS-9B</span>
</span>
</td>
<td id="S4.T1.st2.5.8.8.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.8.8.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.8.8.2.1.1" class="ltx_p" style="width:86.7pt;"></span>
</span>
</td>
<td id="S4.T1.st2.5.8.8.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.8.8.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.8.8.3.1.1" class="ltx_p" style="width:52.0pt;"></span>
</span>
</td>
</tr>
<tr id="S4.T1.st2.5.9.9" class="ltx_tr" style="background-color:#D9D9D9;">
<td id="S4.T1.st2.5.9.9.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.9.9.1.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.9.9.1.1.1" class="ltx_p">+ 1-shot ICL</span>
</span>
</td>
<td id="S4.T1.st2.5.9.9.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.9.9.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.9.9.2.1.1" class="ltx_p" style="width:86.7pt;">55.29</span>
</span>
</td>
<td id="S4.T1.st2.5.9.9.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.9.9.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.9.9.3.1.1" class="ltx_p" style="width:52.0pt;">62.0</span>
</span>
</td>
</tr>
<tr id="S4.T1.st2.5.10.10" class="ltx_tr" style="background-color:#D9D9D9;">
<td id="S4.T1.st2.5.10.10.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.10.10.1.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.10.10.1.1.1" class="ltx_p">Emu 37B</span>
</span>
</td>
<td id="S4.T1.st2.5.10.10.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.10.10.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.10.10.2.1.1" class="ltx_p" style="width:86.7pt;"></span>
</span>
</td>
<td id="S4.T1.st2.5.10.10.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.10.10.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.10.10.3.1.1" class="ltx_p" style="width:52.0pt;"></span>
</span>
</td>
</tr>
<tr id="S4.T1.st2.5.11.11" class="ltx_tr" style="background-color:#D9D9D9;">
<td id="S4.T1.st2.5.11.11.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.11.11.1.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.11.11.1.1.1" class="ltx_p">+ 1-shot ICL</span>
</span>
</td>
<td id="S4.T1.st2.5.11.11.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.11.11.2.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.11.11.2.1.1" class="ltx_p" style="width:86.7pt;">52.76</span>
</span>
</td>
<td id="S4.T1.st2.5.11.11.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.11.11.3.1" class="ltx_inline-block ltx_align_top" style="background-color:#D9D9D9;">
<span id="S4.T1.st2.5.11.11.3.1.1" class="ltx_p" style="width:52.0pt;">53.56</span>
</span>
</td>
</tr>
<tr id="S4.T1.st2.5.12.12" class="ltx_tr">
<td id="S4.T1.st2.5.12.12.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.12.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.12.12.1.1.1" class="ltx_p">Qwen-VL-7B</span>
</span>
</td>
<td id="S4.T1.st2.5.12.12.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;"></td>
<td id="S4.T1.st2.5.12.12.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;"></td>
</tr>
<tr id="S4.T1.st2.5.13.13" class="ltx_tr">
<td id="S4.T1.st2.5.13.13.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.13.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.13.13.1.1.1" class="ltx_p">+ 1-shot ICL</span>
</span>
</td>
<td id="S4.T1.st2.5.13.13.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.13.13.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.13.13.2.1.1" class="ltx_p" style="width:86.7pt;">55.0</span>
</span>
</td>
<td id="S4.T1.st2.5.13.13.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.13.13.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.13.13.3.1.1" class="ltx_p" style="width:52.0pt;">56.5</span>
</span>
</td>
</tr>
<tr id="S4.T1.st2.5.14.14" class="ltx_tr">
<td id="S4.T1.st2.5.14.14.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.14.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.14.14.1.1.1" class="ltx_p">+ <span id="S4.T1.st2.5.14.14.1.1.1.1" class="ltx_text ltx_font_bold">MTV</span>+1-shot ICL</span>
</span>
</td>
<td id="S4.T1.st2.5.14.14.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.14.14.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.14.14.2.1.1" class="ltx_p" style="width:86.7pt;"><span id="S4.T1.st2.5.14.14.2.1.1.1" class="ltx_text ltx_font_bold">78.1</span></span>
</span>
</td>
<td id="S4.T1.st2.5.14.14.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.14.14.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.14.14.3.1.1" class="ltx_p" style="width:52.0pt;"><span id="S4.T1.st2.5.14.14.3.1.1.1" class="ltx_text ltx_font_bold">80.0</span></span>
</span>
</td>
</tr>
<tr id="S4.T1.st2.5.15.15" class="ltx_tr">
<td id="S4.T1.st2.5.15.15.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.15.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.15.15.1.1.1" class="ltx_p">Idefics2</span>
</span>
</td>
<td id="S4.T1.st2.5.15.15.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;"></td>
<td id="S4.T1.st2.5.15.15.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;"></td>
</tr>
<tr id="S4.T1.st2.5.16.16" class="ltx_tr">
<td id="S4.T1.st2.5.16.16.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.16.16.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.16.16.1.1.1" class="ltx_p">+ 1-shot ICL</span>
</span>
</td>
<td id="S4.T1.st2.5.16.16.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.16.16.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.16.16.2.1.1" class="ltx_p" style="width:86.7pt;">82.8</span>
</span>
</td>
<td id="S4.T1.st2.5.16.16.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.16.16.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.16.16.3.1.1" class="ltx_p" style="width:52.0pt;">88.7</span>
</span>
</td>
</tr>
<tr id="S4.T1.st2.5.17.17" class="ltx_tr">
<td id="S4.T1.st2.5.17.17.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.17.17.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.17.17.1.1.1" class="ltx_p">+ <span id="S4.T1.st2.5.17.17.1.1.1.1" class="ltx_text ltx_font_bold">MTV</span>+1-shot ICL</span>
</span>
</td>
<td id="S4.T1.st2.5.17.17.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.17.17.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.17.17.2.1.1" class="ltx_p" style="width:86.7pt;"><span id="S4.T1.st2.5.17.17.2.1.1.1" class="ltx_text ltx_font_bold">83.8</span></span>
</span>
</td>
<td id="S4.T1.st2.5.17.17.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.17.17.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.17.17.3.1.1" class="ltx_p" style="width:52.0pt;"><span id="S4.T1.st2.5.17.17.3.1.1.1" class="ltx_text ltx_font_bold">89.8</span></span>
</span>
</td>
</tr>
<tr id="S4.T1.st2.5.18.18" class="ltx_tr">
<td id="S4.T1.st2.5.18.18.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.18.18.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.18.18.1.1.1" class="ltx_p">Llama3-VILA-1.5-8B</span>
</span>
</td>
<td id="S4.T1.st2.5.18.18.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;"></td>
<td id="S4.T1.st2.5.18.18.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;"></td>
</tr>
<tr id="S4.T1.st2.5.19.19" class="ltx_tr">
<td id="S4.T1.st2.5.19.19.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.19.19.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.19.19.1.1.1" class="ltx_p">+ 1-shot ICL</span>
</span>
</td>
<td id="S4.T1.st2.5.19.19.2" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.19.19.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.19.19.2.1.1" class="ltx_p" style="width:86.7pt;">87.4</span>
</span>
</td>
<td id="S4.T1.st2.5.19.19.3" class="ltx_td ltx_align_justify ltx_align_top" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.19.19.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.19.19.3.1.1" class="ltx_p" style="width:52.0pt;">88.4</span>
</span>
</td>
</tr>
<tr id="S4.T1.st2.5.20.20" class="ltx_tr">
<td id="S4.T1.st2.5.20.20.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb ltx_border_b" style="width:238.5pt;padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.20.20.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.20.20.1.1.1" class="ltx_p">+ <span id="S4.T1.st2.5.20.20.1.1.1.1" class="ltx_text ltx_font_bold">MTV</span>+1-shot ICL</span>
</span>
</td>
<td id="S4.T1.st2.5.20.20.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb ltx_border_b" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.20.20.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.20.20.2.1.1" class="ltx_p" style="width:86.7pt;"><span id="S4.T1.st2.5.20.20.2.1.1.1" class="ltx_text ltx_font_bold">89.3</span></span>
</span>
</td>
<td id="S4.T1.st2.5.20.20.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb ltx_border_b" style="padding-top:0.5pt;padding-bottom:0.5pt;">
<span id="S4.T1.st2.5.20.20.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T1.st2.5.20.20.3.1.1" class="ltx_p" style="width:52.0pt;"><span id="S4.T1.st2.5.20.20.3.1.1.1" class="ltx_text ltx_font_bold">89.7</span></span>
</span>
</td>
</tr>
</tbody>
</table>
</figure>
</div>
</div>
</figure>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Datasets</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">We briefly describe the tasks and datasets we evaluate our method on. More details about the datasets and their setup can be found in Section <a href="#A3" title="Appendix C Additional Implementation Details ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>.</p>
</div>
<div id="S4.SS3.p2" class="ltx_para ltx_noindent">
<p id="S4.SS3.p2.1" class="ltx_p"><span id="S4.SS3.p2.1.1" class="ltx_text ltx_font_bold">VQA Datasets</span>. We use the following commonly-evaluated datasets which emphasize different aspects of multimodal reasoning, including visual features (VizWiz) and outside knowledge (OK-VQA): (1) <span id="S4.SS3.p2.1.2" class="ltx_text ltx_font_bold">VizWiz</span> <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib23" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">23</span></a>]</cite> consists of images taken by visually impaired individuals paired with questions they pose about these images, making it crucial for developing AI systems that assist in real-world, accessibility-focused visual understanding tasks. (2) <span id="S4.SS3.p2.1.3" class="ltx_text ltx_font_bold">OK-VQA</span> dataset <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">54</span></a>]</cite> is designed to push the boundaries of Visual Question Answering (VQA) by focusing on knowledge-based questions, where answers require external knowledge beyond the image content. (3)</p>
</div>
<div id="S4.SS3.p3" class="ltx_para ltx_noindent">
<p id="S4.SS3.p3.1" class="ltx_p"><span id="S4.SS3.p3.1.1" class="ltx_text ltx_font_bold">Object Classification</span>. We use the following datasets, which are commonly used for object classification in multimodal ICL: (1) The <span id="S4.SS3.p3.1.2" class="ltx_text ltx_font_bold">Flowers</span> dataset <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib58" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">58</span></a>]</cite>, commonly known as the Oxford 102 Flowers dataset, is a collection specifically designed for image-based flower species recognition for fine-grained classification of 102 different categories of flowers. (2) <span id="S4.SS3.p3.1.3" class="ltx_text ltx_font_bold">Caltech’s CUB Dataset on Birds</span> <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib80" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">80</span></a>]</cite> is a well-known resource for evaluating algorithms on the task of object identification, specifically focused on bird species. It features 200 bird species with roughly 30 images each, annotated with key attributes and bounding boxes. Both Flowers and Birds are formatted as 2-way,1-shot classification episodes, with model inputs being a positive and negative image for the class to be identified in the query image. The response format is a short text response.</p>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Results</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">Our main results are shown in Table <a href="#S4.T1" title="Table 1 ‣ 4.2 Models ‣ 4 Evaluation ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. For VQA, we show the results of MTV with 4 shots per 100 iterations to calculate the mean activations and 100 examples for task vector locations (500 examples total). The task vector is extracted using examples from the train set of the dataset and evaluated on the validation set. For object classification, we extract MTV based on a 2-way, one-shot regimen per 100 iterations for both mean activations and task vector locations (200 examples total). The task vector is extracted using a train set of 30% of the object classes and evaluated on the remaining 70% of <span id="S5.p1.1.1" class="ltx_text ltx_font_italic">unseen</span> classes.
We demonstrate how Multimodal Task Vectors outperforms zero-shot and few-shot ICL settings on three different models on VL tasks, highlighting the effectiveness of our method.
Next, we describe the unique capabilities of our method, such as scaling to more samples and showing some generalizations to other tasks. More results can be found in Section <a href="#A1.SS1" title="A.1 Additional Experiments ‣ Appendix A Additional Experiment Results ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.1</span></a> of Supplementary.</p>
</div>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>MTV scales with more examples</h3>

<figure id="S5.F2" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<div id="S5.F2.1" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:216.8pt;">
<img src="/html/2406.15334/assets/figs/plot1.png" id="S5.F2.1.g1" class="ltx_graphics ltx_img_landscape" width="598" height="454" alt="Refer to caption">
</div>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<div id="S5.F2.2" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_center ltx_align_middle" style="width:216.8pt;">
<img src="/html/2406.15334/assets/figs/plot2.png" id="S5.F2.2.g1" class="ltx_graphics ltx_img_landscape" width="598" height="465" alt="Refer to caption">
</div>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S5.F2.5.1.1" class="ltx_text" style="font-size:90%;">Figure 2</span>: </span><span id="S5.F2.6.2" class="ltx_text ltx_font_bold" style="font-size:90%;">Scaling of Qwen-MTV on VizWiz:<span id="S5.F2.6.2.1" class="ltx_text ltx_font_medium"> (Left) We show the effect of varying the number of shots per iteration for a fixed 100 iterations. (Right) We also show the effect of varying numbers of iterations fixing 4 shots per iteration.</span></span></figcaption>
</figure>
<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.1" class="ltx_p">We are interested in evaluating (i) the effect of different numbers of shots used <span id="S5.SS1.p1.1.1" class="ltx_text ltx_font_italic">per iteration</span> to extract MTV and (ii) the effect of different numbers of <span id="S5.SS1.p1.1.2" class="ltx_text ltx_font_italic">iterations</span> used. We test the impact on accuracy when increasing both of these parameters for QwenVL on the VizWiz validation set. In Figure <a href="#S5.F2" title="Figure 2 ‣ 5.1 MTV scales with more examples ‣ 5 Results ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, we show on the left that the optimal number of multimodal ICL shots is 16 shots per iteration. Further, we show on the right side of the figure that 1000 examples yield the best performance. These results illustrate that MTV can effectively scale by utilizing larger numbers of ICL examples per iteration and also in aggregate.</p>
</div>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>MTV works with explicit few-shot examples</h3>

<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.1" class="ltx_p">One of the benefits of MTV over a few-shot ICL is the context length that is saved during inference. This is because the many-shot examples are encoded directly in the activation space rather than in the input token space. Thus, we ask whether the LMM can use the freed context for additional few-shot examples. For object classification, we formulate both Flowers and CUB as a 1-shot comparison between a positive and negative sample to identify the correct class (i.e., 2-way, 1-shot ICL by construction). We report results on 1-shot ICL and MTV with 1-shot classification during inference. MTV+1-shot ICL surpasses 1-shot ICL accuracy on these tasks, showing that MTV can be utilized alongside few-shot examples. Furthermore, it is vital to note that the evaluation classes are completely unseen by MTV. Thus, with just a 1-shot ICL example, MTV is able to generalize to unseen classes.</p>
</div>
</section>
<section id="S5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.3 </span>MTV heads generalize to other tasks</h3>

<figure id="S5.T2" class="ltx_table">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S5.T2.3.1.1" class="ltx_text" style="font-size:90%;">Table 2</span>: </span><span id="S5.T2.4.2" class="ltx_text ltx_font_bold" style="font-size:90%;">Generalization &amp; Method Comparison<span id="S5.T2.4.2.1" class="ltx_text ltx_font_medium"> (Left) MTV-VizWiz evaluated on OK-VQA. (Right) MTV compared to VizWiz finetuning, function vectors <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib79" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">79</span></a>]</cite>, and task vectors <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib27" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">27</span></a>]</cite>.</span></span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S5.T2.st1" class="ltx_table ltx_figure_panel">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S5.T2.st1.2.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="S5.T2.st1.3.2" class="ltx_text" style="font-size:90%;">Attention Head Generalization</span></figcaption>
<div id="S5.T2.st1.4" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:201.4pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(119.9pt,-55.7pt) scale(2.23771512372675,2.23771512372675) ;">
<table id="S5.T2.st1.4.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T2.st1.4.1.1.1" class="ltx_tr">
<th id="S5.T2.st1.4.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding-top:0.5pt;padding-bottom:0.5pt;">Model</th>
<th id="S5.T2.st1.4.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st1.4.1.1.1.2.1" class="ltx_text" style="background-color:#E6E6E6;">VizWiz</span></th>
<th id="S5.T2.st1.4.1.1.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st1.4.1.1.1.3.1" class="ltx_text" style="background-color:#E6E6E6;">OK-VQA</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T2.st1.4.1.2.1" class="ltx_tr">
<th id="S5.T2.st1.4.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">ViLA-1.5-8B</th>
<td id="S5.T2.st1.4.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st1.4.1.2.1.2.1" class="ltx_text" style="background-color:#E6E6E6;">28.0</span></td>
<td id="S5.T2.st1.4.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st1.4.1.2.1.3.1" class="ltx_text" style="background-color:#E6E6E6;">32.8</span></td>
</tr>
<tr id="S5.T2.st1.4.1.3.2" class="ltx_tr">
<th id="S5.T2.st1.4.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:0.5pt;padding-bottom:0.5pt;">   + 4-shot-ICL</th>
<td id="S5.T2.st1.4.1.3.2.2" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st1.4.1.3.2.2.1" class="ltx_text" style="background-color:#E6E6E6;">39.3</span></td>
<td id="S5.T2.st1.4.1.3.2.3" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st1.4.1.3.2.3.1" class="ltx_text" style="background-color:#E6E6E6;">35.6</span></td>
</tr>
<tr id="S5.T2.st1.4.1.4.3" class="ltx_tr">
<th id="S5.T2.st1.4.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:0.5pt;padding-bottom:0.5pt;">   + 8-shot-ICL</th>
<td id="S5.T2.st1.4.1.4.3.2" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st1.4.1.4.3.2.1" class="ltx_text" style="background-color:#E6E6E6;">44.2</span></td>
<td id="S5.T2.st1.4.1.4.3.3" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st1.4.1.4.3.3.1" class="ltx_text" style="background-color:#E6E6E6;">36.5</span></td>
</tr>
<tr id="S5.T2.st1.4.1.5.4" class="ltx_tr">
<th id="S5.T2.st1.4.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding-top:0.5pt;padding-bottom:0.5pt;">   + <span id="S5.T2.st1.4.1.5.4.1.1" class="ltx_text ltx_font_bold">MTV-Vizwiz</span>
</th>
<td id="S5.T2.st1.4.1.5.4.2" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st1.4.1.5.4.2.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6E6;">55.2</span></td>
<td id="S5.T2.st1.4.1.5.4.3" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st1.4.1.5.4.3.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6E6;">38.3</span></td>
</tr>
</tbody>
</table>
</span></div>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S5.T2.st2" class="ltx_table ltx_figure_panel">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S5.T2.st2.2.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="S5.T2.st2.3.2" class="ltx_text" style="font-size:90%;">Comparison to Other Methods</span></figcaption>
<div id="S5.T2.st2.4" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:235.3pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(117.3pt,-63.6pt) scale(2.17838394483478,2.17838394483478) ;">
<table id="S5.T2.st2.4.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T2.st2.4.1.1.1" class="ltx_tr">
<th id="S5.T2.st2.4.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding-top:0.5pt;padding-bottom:0.5pt;">Model</th>
<th id="S5.T2.st2.4.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st2.4.1.1.1.2.1" class="ltx_text" style="background-color:#E6E6E6;">VizWiz</span></th>
<th id="S5.T2.st2.4.1.1.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st2.4.1.1.1.3.1" class="ltx_text" style="background-color:#E6E6E6;">OK-VQA</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T2.st2.4.1.2.1" class="ltx_tr">
<th id="S5.T2.st2.4.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">Qwen-VL-7B</th>
<td id="S5.T2.st2.4.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st2.4.1.2.1.2.1" class="ltx_text" style="background-color:#E6E6E6;">35.2</span></td>
<td id="S5.T2.st2.4.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st2.4.1.2.1.3.1" class="ltx_text" style="background-color:#E6E6E6;">58.6</span></td>
</tr>
<tr id="S5.T2.st2.4.1.3.2" class="ltx_tr">
<th id="S5.T2.st2.4.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:0.5pt;padding-bottom:0.5pt;">   + VizWiz LoRA</th>
<td id="S5.T2.st2.4.1.3.2.2" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st2.4.1.3.2.2.1" class="ltx_text" style="background-color:#E6E6E6;">62.0</span></td>
<td id="S5.T2.st2.4.1.3.2.3" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st2.4.1.3.2.3.1" class="ltx_text" style="background-color:#E6E6E6;">25.1</span></td>
</tr>
<tr id="S5.T2.st2.4.1.4.3" class="ltx_tr">
<th id="S5.T2.st2.4.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:0.5pt;padding-bottom:0.5pt;">   + FV</th>
<td id="S5.T2.st2.4.1.4.3.2" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st2.4.1.4.3.2.1" class="ltx_text" style="background-color:#E6E6E6;">36.4</span></td>
<td id="S5.T2.st2.4.1.4.3.3" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st2.4.1.4.3.3.1" class="ltx_text" style="background-color:#E6E6E6;">59.0</span></td>
</tr>
<tr id="S5.T2.st2.4.1.5.4" class="ltx_tr">
<th id="S5.T2.st2.4.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:0.5pt;padding-bottom:0.5pt;">   + VTV</th>
<td id="S5.T2.st2.4.1.5.4.2" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st2.4.1.5.4.2.1" class="ltx_text" style="background-color:#E6E6E6;">37.0</span></td>
<td id="S5.T2.st2.4.1.5.4.3" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st2.4.1.5.4.3.1" class="ltx_text" style="background-color:#E6E6E6;">58.9</span></td>
</tr>
<tr id="S5.T2.st2.4.1.6.5" class="ltx_tr">
<th id="S5.T2.st2.4.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding-top:0.5pt;padding-bottom:0.5pt;">   + <span id="S5.T2.st2.4.1.6.5.1.1" class="ltx_text ltx_font_bold">MTV</span>
</th>
<td id="S5.T2.st2.4.1.6.5.2" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st2.4.1.6.5.2.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6E6;">45.6</span></td>
<td id="S5.T2.st2.4.1.6.5.3" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T2.st2.4.1.6.5.3.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6E6;">62.0</span></td>
</tr>
</tbody>
</table>
</span></div>
</figure>
</div>
</div>
</figure>
<div id="S5.SS3.p1" class="ltx_para">
<p id="S5.SS3.p1.3" class="ltx_p">In this experiment, we further ask whether the MTV heads <math id="S5.SS3.p1.1.m1.1" class="ltx_Math" alttext="\lambda^{\mathrm{MTV}}_{j}" display="inline"><semantics id="S5.SS3.p1.1.m1.1a"><msubsup id="S5.SS3.p1.1.m1.1.1" xref="S5.SS3.p1.1.m1.1.1.cmml"><mi id="S5.SS3.p1.1.m1.1.1.2.2" xref="S5.SS3.p1.1.m1.1.1.2.2.cmml">λ</mi><mi id="S5.SS3.p1.1.m1.1.1.3" xref="S5.SS3.p1.1.m1.1.1.3.cmml">j</mi><mi id="S5.SS3.p1.1.m1.1.1.2.3" xref="S5.SS3.p1.1.m1.1.1.2.3.cmml">MTV</mi></msubsup><annotation-xml encoding="MathML-Content" id="S5.SS3.p1.1.m1.1b"><apply id="S5.SS3.p1.1.m1.1.1.cmml" xref="S5.SS3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S5.SS3.p1.1.m1.1.1.1.cmml" xref="S5.SS3.p1.1.m1.1.1">subscript</csymbol><apply id="S5.SS3.p1.1.m1.1.1.2.cmml" xref="S5.SS3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S5.SS3.p1.1.m1.1.1.2.1.cmml" xref="S5.SS3.p1.1.m1.1.1">superscript</csymbol><ci id="S5.SS3.p1.1.m1.1.1.2.2.cmml" xref="S5.SS3.p1.1.m1.1.1.2.2">𝜆</ci><ci id="S5.SS3.p1.1.m1.1.1.2.3.cmml" xref="S5.SS3.p1.1.m1.1.1.2.3">MTV</ci></apply><ci id="S5.SS3.p1.1.m1.1.1.3.cmml" xref="S5.SS3.p1.1.m1.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p1.1.m1.1c">\lambda^{\mathrm{MTV}}_{j}</annotation></semantics></math> extracted on one task <math id="S5.SS3.p1.2.m2.1" class="ltx_Math" alttext="j" display="inline"><semantics id="S5.SS3.p1.2.m2.1a"><mi id="S5.SS3.p1.2.m2.1.1" xref="S5.SS3.p1.2.m2.1.1.cmml">j</mi><annotation-xml encoding="MathML-Content" id="S5.SS3.p1.2.m2.1b"><ci id="S5.SS3.p1.2.m2.1.1.cmml" xref="S5.SS3.p1.2.m2.1.1">𝑗</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p1.2.m2.1c">j</annotation></semantics></math> can generalize to a separate, but similar task <math id="S5.SS3.p1.3.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.SS3.p1.3.m3.1a"><mi id="S5.SS3.p1.3.m3.1.1" xref="S5.SS3.p1.3.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.SS3.p1.3.m3.1b"><ci id="S5.SS3.p1.3.m3.1.1.cmml" xref="S5.SS3.p1.3.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p1.3.m3.1c">k</annotation></semantics></math>. To test this, we use the attention heads extracted from ViLA-1.5-8B on VizWiz for use on OK-VQA. Our results on the left of Table <a href="#S5.T2" title="Table 2 ‣ 5.3 MTV heads generalize to other tasks ‣ 5 Results ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> demonstrate that the extracted heads from one task can improve accuracy on another similar task. This generalizability of the heads is significant because it suggests that the heads from MTV may only have to be extracted once to be applied to many other similar tasks. The only calculation necessary then would be the mean activations of the many-shot examples used for the target dataset, making the application of many-shot multimodal ICL even more efficient for similar tasks.</p>
</div>
</section>
<section id="S5.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.4 </span>LoRA Finetuning as an upper bound</h3>

<div id="S5.SS4.p1" class="ltx_para">
<p id="S5.SS4.p1.1" class="ltx_p">In Table <a href="#S5.T2.st2" title="In Table 2 ‣ 5.3 MTV heads generalize to other tasks ‣ 5 Results ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2(b)</span></a>, we compare our method to LoRA finetuning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">29</span></a>]</cite>. To do this, we use LoRA on the same number of examples as MTV uses from the train set and evaluate not only on the validation set but also on the validation set of another similar dataset. In particular, for a ViLA-1.5-8B model finetuned on VizWiz, we report accuracy on both VizWiz and OK-VQA validation sets. It can be seen that LoRA finetuning is indeed an upper bound on the dataset the model was finetuned on. However, we show that LoRA leads to overfitting on the finetuned dataset and forgetting the zero-shot capabilities. In contrast, we also show that MTV not only improves zero-shot capabilities but can generalize to similar tasks with only a few inference examples Table <a href="#S4.T1.st2" title="In Table 1 ‣ 4.2 Models ‣ 4 Evaluation ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1(b)</span></a> and Table <a href="#S5.T2.st1" title="In Table 2 ‣ 5.3 MTV heads generalize to other tasks ‣ 5 Results ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2(a)</span></a>.</p>
</div>
</section>
<section id="S5.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.5 </span>Comparison to other methods</h3>

<div id="S5.SS5.p1" class="ltx_para">
<p id="S5.SS5.p1.1" class="ltx_p">We compare our method to two different methods that can find task vectors: Visual Task Vectors (VTV) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib27" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">27</span></a>]</cite> and Function Vectors (FV) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib79" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">79</span></a>]</cite>. Originally, these works could not be applied as-is to support multimodal ICL, but here, we have implemented a version that follows the original exactly with only minor modifications to allow
performing our evaluated multimodal tasks. More details about the methods can be found in Section <a href="#A2" title="Appendix B Additional Method Details ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a> in the Supplementary. In our experiments Table <a href="#S5.T2.st2" title="In Table 2 ‣ 5.3 MTV heads generalize to other tasks ‣ 5 Results ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2(b)</span></a>, we find that MTV surpasses both methods on VizWiz and OK-VQA. VTV are image-only task vectors that use only one-shot image examples for fixed small <math id="S5.SS5.p1.1.m1.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S5.SS5.p1.1.m1.1a"><mi id="S5.SS5.p1.1.m1.1.1" xref="S5.SS5.p1.1.m1.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S5.SS5.p1.1.m1.1b"><ci id="S5.SS5.p1.1.m1.1.1.cmml" xref="S5.SS5.p1.1.m1.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS5.p1.1.m1.1c">T</annotation></semantics></math> iterations, and they calculate the mean activations and the locations together without aligning to the downstream task. FV are text-only task vectors that use Causal Mediation Analysis <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib61" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">61</span></a>]</cite> to extract task vector locations. But then a vector of mean activations at these locations is simply added after the output of an arbitrary layer. The results suggest the importance of finding the task vectors by decoupling the calculation of the mean activations and locations in two separate steps to perform many-shot multimodal ICL more effectively for complex multimodal tasks.</p>
</div>
</section>
<section id="S5.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.6 </span>Compute and runtime efficiency</h3>

<figure id="S5.T3" class="ltx_table">
<table id="S5.T3.2" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S5.T3.2.1.1" class="ltx_tr">
<th id="S5.T3.2.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T3.2.1.1.1.1" class="ltx_text ltx_font_bold">Metric</span></th>
<th id="S5.T3.2.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T3.2.1.1.2.1" class="ltx_text ltx_font_bold">0-shot</span></th>
<th id="S5.T3.2.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T3.2.1.1.3.1" class="ltx_text ltx_font_bold">4-shot</span></th>
<th id="S5.T3.2.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T3.2.1.1.4.1" class="ltx_text ltx_font_bold">8-shot</span></th>
<th id="S5.T3.2.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T3.2.1.1.5.1" class="ltx_text ltx_font_bold">16-shot</span></th>
<th id="S5.T3.2.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" style="padding-top:0.5pt;padding-bottom:0.5pt;"><span id="S5.T3.2.1.1.6.1" class="ltx_text ltx_font_bold">MTV (400-shot)</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S5.T3.2.2.1" class="ltx_tr">
<th id="S5.T3.2.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">Max GPU Memory (GB)</th>
<td id="S5.T3.2.2.1.2" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">17.4</td>
<td id="S5.T3.2.2.1.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">18.3</td>
<td id="S5.T3.2.2.1.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">19.0</td>
<td id="S5.T3.2.2.1.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">20.6</td>
<td id="S5.T3.2.2.1.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">19.8</td>
</tr>
<tr id="S5.T3.2.3.2" class="ltx_tr">
<th id="S5.T3.2.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding-top:0.5pt;padding-bottom:0.5pt;">Runtime per 100 iterations (min)</th>
<td id="S5.T3.2.3.2.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:0.5pt;padding-bottom:0.5pt;">1.1</td>
<td id="S5.T3.2.3.2.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:0.5pt;padding-bottom:0.5pt;">2.7</td>
<td id="S5.T3.2.3.2.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:0.5pt;padding-bottom:0.5pt;">3.1</td>
<td id="S5.T3.2.3.2.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:0.5pt;padding-bottom:0.5pt;">3.3</td>
<td id="S5.T3.2.3.2.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding-top:0.5pt;padding-bottom:0.5pt;">1.9</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S5.T3.4.1.1" class="ltx_text" style="font-size:90%;">Table 3</span>: </span><span id="S5.T3.5.2" class="ltx_text ltx_font_bold" style="font-size:90%;">Efficiency:<span id="S5.T3.5.2.1" class="ltx_text ltx_font_medium"> We show that even though MTV encodes 400 multimodal ICL examples in the mean activations, it still requires less runtime and memory than 8-shot and 16-shot multimodal ICL.
</span></span></figcaption>
</figure>
<div id="S5.SS6.p1" class="ltx_para">
<p id="S5.SS6.p1.1" class="ltx_p">An important feature of our work is that multimodal ICL examples do not require explicit tokens during inference. Because of this, we are interested in the efficiency gains of our method. Intuitively, the longer MTV extraction time is amortized during downstream inference, where the runtime would be equivalent to the zero-shot case.
Similarly, the memory requirements are maximal during the MTV extraction process but require the same memory as the zero-shot case afterward. In contrast, the ICL tasks have a slower runtime and larger memory requirement throughout due to running inference on <math id="S5.SS6.p1.1.m1.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S5.SS6.p1.1.m1.1a"><mi id="S5.SS6.p1.1.m1.1.1" xref="S5.SS6.p1.1.m1.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S5.SS6.p1.1.m1.1b"><ci id="S5.SS6.p1.1.m1.1.1.cmml" xref="S5.SS6.p1.1.m1.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS6.p1.1.m1.1c">N</annotation></semantics></math> examples <span id="S5.SS6.p1.1.1" class="ltx_text ltx_font_italic">for every iteration</span>. To demonstrate this, we calculate the maximum memory requirement in gigabytes (GB) for ViLA-1.5-8B on VizWiz using different ICL-shot counts and MTV with 400 examples. As shown in Table <a href="#S5.T3" title="Table 3 ‣ 5.6 Compute and runtime efficiency ‣ 5 Results ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, MTV requires less runtime than 16-shot, 8-shot, and 4-shot ICL methods and also requires less memory than 16-shot ICL. These results demonstrate that MTV can encode many multimodal ICL examples with greater efficiency than few-shot methods.</p>
</div>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Conclusion</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">In this work, we present Multimodal Task Vectors a compact, implicit representation that can efficiently encode many-shot multimodal ICL examples for use in complex vision-language tasks. We demonstrate this implicit model representation not only encodes a multimodal ICL task but can also enable many-shot multimodal ICL to surpass zero-shot and few-shot performance on a variety of VL tasks. Our method stands out from previous work in its ability to scale, use additional explicit multimodal ICL examples, and generalize to other similar VL tasks. Our work is a viable way to surpass the limit of context length of an LMM for multimodal ICL and demonstrates clearly that these additional examples aid in multimodal reasoning. Finally, we do not anticipate a specific negative impact, but, as with any Machine Learning method, we recommend exercising caution.</p>
</div>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Limitations</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.1" class="ltx_p">While Multimodal Task Vectors offers substantial benefits for handling complex vision-language tasks compared to finetuning or few-shot ICL, it is important to recognize certain limitations that accompany our approach. MTV requires access to the internal architecture of an LMM, so while it is an effective solution for all open-source models, its application is restricted from proprietary models, such as GPT-4 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib59" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">59</span></a>]</cite> and Gemini <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib78" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">78</span></a>, <a href="#bib.bib77" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">77</span></a>]</cite>. Furthermore, while many-shot ICL is incredibly attractive for many applications, it may not be practical for low-data scenarios where synthetic data <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">1</span></a>]</cite> or the transfer of MTV extracted from another dataset may be required.
We feel these challenges represent great opportunities for future work in the many-shot multimodal in-context learning domain.</p>
</div>
<section id="S7.SS0.SSSx1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Acknowledgements.</h4>

<div id="S7.SS0.SSSx1.p1" class="ltx_para">
<p id="S7.SS0.SSSx1.p1.1" class="ltx_p">We would like to thank Suzie Petryk and Grace Luo for helpful feedback and discussions. This project was supported in part by DoD, including PTG and/or LwLL programs, as well as BAIR’s industrial alliance programs.</p>
</div>
</section>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography" style="font-size:90%;">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib1.5.5.1" class="ltx_text" style="font-size:90%;">Agarwal et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib1.7.1" class="ltx_text" style="font-size:90%;">
Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal M. P. Behbahani, Aleksandra Faust, and Hugo Larochelle.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib1.8.1" class="ltx_text" style="font-size:90%;">Many-shot in-context learning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib1.9.1" class="ltx_text" style="font-size:90%;">2024.
</span>
</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib2.5.5.1" class="ltx_text" style="font-size:90%;">Alayrac et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib2.7.1" class="ltx_text" style="font-size:90%;">
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib2.8.1" class="ltx_text" style="font-size:90%;">Flamingo: a visual language model for few-shot learning.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib2.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib2.10.2" class="ltx_text" style="font-size:90%;">, abs/2204.14198, 2022.
</span>
</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib3.5.5.1" class="ltx_text" style="font-size:90%;">Alfassy et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib3.7.1" class="ltx_text" style="font-size:90%;">
Amit Alfassy, Assaf Arbelle, Oshri Halimi, Sivan Harary, Roei Herzig, Eli Schwartz, Rameswar Panda, Michele Dolfi, Christoph Auer, Peter W. J. Staar, Kate Saenko, Rogerio Feris, and Leonid Karlinsky.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib3.8.1" class="ltx_text" style="font-size:90%;">FETA: Towards specializing foundational models for expert task applications.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib3.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib3.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track</em><span id="bib.bib3.11.3" class="ltx_text" style="font-size:90%;">, 2022.
</span>
</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib4.5.5.1" class="ltx_text" style="font-size:90%;">Antol et al. [2015]</span></span>
<span class="ltx_bibblock"><span id="bib.bib4.7.1" class="ltx_text" style="font-size:90%;">
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib4.8.1" class="ltx_text" style="font-size:90%;">Vqa: Visual question answering.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib4.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib4.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the IEEE international conference on computer vision</em><span id="bib.bib4.11.3" class="ltx_text" style="font-size:90%;">, pages 2425–2433, 2015.
</span>
</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib5.5.5.1" class="ltx_text" style="font-size:90%;">Avraham et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib5.7.1" class="ltx_text" style="font-size:90%;">
Elad Ben Avraham, Roei Herzig, Karttikeya Mangalam, Amir Bar, Anna Rohrbach, Leonid Karlinsky, Trevor Darrell, and Amir Globerson.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib5.8.1" class="ltx_text" style="font-size:90%;">Bringing image scene structure to video via frame-clip consistency of object tokens.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib5.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib5.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Thirty-Sixth Conference on Neural Information Processing Systems</em><span id="bib.bib5.11.3" class="ltx_text" style="font-size:90%;">, 2022.
</span>
</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib6.5.5.1" class="ltx_text" style="font-size:90%;">Bai et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib6.7.1" class="ltx_text" style="font-size:90%;">
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib6.8.1" class="ltx_text" style="font-size:90%;">Qwen-vl: A frontier large vision-language model with versatile abilities.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib6.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib6.10.2" class="ltx_text" style="font-size:90%;">, abs/2308.12966, 2023.
</span>
</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib7.5.5.1" class="ltx_text" style="font-size:90%;">Bertsch et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib7.7.1" class="ltx_text" style="font-size:90%;">
Amanda Bertsch, Maor Ivgi, Uri Alon, Jonathan Berant, Matthew R. Gormley, and Graham Neubig.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib7.8.1" class="ltx_text" style="font-size:90%;">In-context learning with long-context models: An in-depth exploration.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib7.9.1" class="ltx_text" style="font-size:90%;">2024.
</span>
</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib8.5.5.1" class="ltx_text" style="font-size:90%;">Besta et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib8.7.1" class="ltx_text" style="font-size:90%;">
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib8.8.1" class="ltx_text" style="font-size:90%;">Graph of thoughts: Solving elaborate problems with large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib8.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib8.10.2" class="ltx_text" style="font-size:90%;">, abs/2308.09687, 2023.
</span>
</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib9.5.5.1" class="ltx_text" style="font-size:90%;">Brown et al. [2020]</span></span>
<span class="ltx_bibblock"><span id="bib.bib9.7.1" class="ltx_text" style="font-size:90%;">
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib9.8.1" class="ltx_text" style="font-size:90%;">Language models are few-shot learners.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib9.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib9.10.2" class="ltx_text" style="font-size:90%;">, abs/2005.14165, 2020.
</span>
</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib10.5.5.1" class="ltx_text" style="font-size:90%;">Chen et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib10.7.1" class="ltx_text" style="font-size:90%;">
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib10.8.1" class="ltx_text" style="font-size:90%;">Extending context window of large language models via positional interpolation.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib10.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib10.10.2" class="ltx_text" style="font-size:90%;">, abs/2306.15595, 2023.
</span>
</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib11.5.5.1" class="ltx_text" style="font-size:90%;">Chevalier et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib11.7.1" class="ltx_text" style="font-size:90%;">
Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib11.8.1" class="ltx_text" style="font-size:90%;">Adapting language models to compress contexts.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib11.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib11.10.2" class="ltx_text" style="font-size:90%;">, abs/2305.14788, 2023.
</span>
</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib12.5.5.1" class="ltx_text" style="font-size:90%;">Chowdhery et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib12.7.1" class="ltx_text" style="font-size:90%;">
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov,
and Noah Fiedel.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib12.8.1" class="ltx_text" style="font-size:90%;">Palm: Scaling language modeling with pathways.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib12.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">J. Mach. Learn. Res.</em><span id="bib.bib12.10.2" class="ltx_text" style="font-size:90%;">, 24:240:1–240:113, 2022.
</span>
</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib13.5.5.1" class="ltx_text" style="font-size:90%;">Dai et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib13.7.1" class="ltx_text" style="font-size:90%;">
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib13.8.1" class="ltx_text" style="font-size:90%;">Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
</span>
</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib14.5.5.1" class="ltx_text" style="font-size:90%;">Dettmers et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib14.7.1" class="ltx_text" style="font-size:90%;">
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib14.8.1" class="ltx_text" style="font-size:90%;">Qlora: Efficient finetuning of quantized llms.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib14.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib14.10.2" class="ltx_text" style="font-size:90%;">, abs/2305.14314, 2023.
</span>
</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib15.5.5.1" class="ltx_text" style="font-size:90%;">Dong et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib15.7.1" class="ltx_text" style="font-size:90%;">
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib15.8.1" class="ltx_text" style="font-size:90%;">A survey on in-context learning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib15.9.1" class="ltx_text" style="font-size:90%;">2022.
</span>
</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib16.5.5.1" class="ltx_text" style="font-size:90%;">Doveh et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib16.7.1" class="ltx_text" style="font-size:90%;">
Sivan Doveh, Shaked Perek, Muhammad Jehanzeb Mirza, Amit Alfassy, Assaf Arbelle, Shimon Ullman, and Leonid Karlinsky.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib16.8.1" class="ltx_text" style="font-size:90%;">Towards multimodal in-context learning for vision &amp; language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib16.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib16.10.2" class="ltx_text" style="font-size:90%;">, abs/2403.12736, 2024.
</span>
</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib17.5.5.1" class="ltx_text" style="font-size:90%;">Driess et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib17.7.1" class="ltx_text" style="font-size:90%;">
Danny Driess, F. Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Ho Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Peter R. Florence.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib17.8.1" class="ltx_text" style="font-size:90%;">Palm-e: An embodied multimodal language model.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib17.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib17.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">International Conference on Machine Learning</em><span id="bib.bib17.11.3" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib18.5.5.1" class="ltx_text" style="font-size:90%;">Gao et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib18.7.1" class="ltx_text" style="font-size:90%;">
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, W. Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Jiao Qiao.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib18.8.1" class="ltx_text" style="font-size:90%;">Llama-adapter v2: Parameter-efficient visual instruction model.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib18.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib18.10.2" class="ltx_text" style="font-size:90%;">, abs/2304.15010, 2023.
</span>
</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib19.5.5.1" class="ltx_text" style="font-size:90%;">Ge et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib19.7.1" class="ltx_text" style="font-size:90%;">
Jiaxin Ge, Sanjay Subramanian, Baifeng Shi, Roei Herzig, and Trevor Darrell.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib19.8.1" class="ltx_text" style="font-size:90%;">Recursive visual programming.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib19.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib19.10.2" class="ltx_text" style="font-size:90%;">, abs/2312.02249, 2023a.
</span>
</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib20.5.5.1" class="ltx_text" style="font-size:90%;">Ge et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib20.7.1" class="ltx_text" style="font-size:90%;">
Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib20.8.1" class="ltx_text" style="font-size:90%;">In-context autoencoder for context compression in a large language model.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib20.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib20.10.2" class="ltx_text" style="font-size:90%;">, abs/2307.06945, 2023b.
</span>
</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib21.5.5.1" class="ltx_text" style="font-size:90%;">Gong et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib21.7.1" class="ltx_text" style="font-size:90%;">
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qianmengke Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib21.8.1" class="ltx_text" style="font-size:90%;">Multimodal-gpt: A vision and language model for dialogue with humans.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib21.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib21.10.2" class="ltx_text" style="font-size:90%;">, abs/2305.04790, 2023.
</span>
</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib22.4.4.1" class="ltx_text" style="font-size:90%;">Gupta and Kembhavi [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib22.6.1" class="ltx_text" style="font-size:90%;">
Tanmay Gupta and Aniruddha Kembhavi.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib22.7.1" class="ltx_text" style="font-size:90%;">Visual programming: Compositional visual reasoning without training.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib22.8.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)</em><span id="bib.bib22.9.2" class="ltx_text" style="font-size:90%;">, pages 14953–14962, 2022.
</span>
</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib23.5.5.1" class="ltx_text" style="font-size:90%;">Gurari et al. [2018]</span></span>
<span class="ltx_bibblock"><span id="bib.bib23.7.1" class="ltx_text" style="font-size:90%;">
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib23.8.1" class="ltx_text" style="font-size:90%;">Vizwiz grand challenge: Answering visual questions from blind people.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib23.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition</em><span id="bib.bib23.10.2" class="ltx_text" style="font-size:90%;">, pages 3608–3617, 2018.
</span>
</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib24.5.5.1" class="ltx_text" style="font-size:90%;">He et al. [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib24.7.1" class="ltx_text" style="font-size:90%;">
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib24.8.1" class="ltx_text" style="font-size:90%;">Momentum contrast for unsupervised visual representation learning.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib24.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)</em><span id="bib.bib24.10.2" class="ltx_text" style="font-size:90%;">, pages 9726–9735, 2019.
</span>
</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib25.5.5.1" class="ltx_text" style="font-size:90%;">Hendel et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib25.7.1" class="ltx_text" style="font-size:90%;">
Roee Hendel, Mor Geva, and Amir Globerson.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib25.8.1" class="ltx_text" style="font-size:90%;">In-context learning creates task vectors.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib25.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib25.10.2" class="ltx_text" style="font-size:90%;">, abs/2310.15916, 2023.
</span>
</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib26.5.5.1" class="ltx_text" style="font-size:90%;">Herzig et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib26.7.1" class="ltx_text" style="font-size:90%;">
Roei Herzig, Alon Mendelson, Leonid Karlinsky, Assaf Arbelle, Rogerio Feris, Trevor Darrell, and Amir Globerson.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib26.8.1" class="ltx_text" style="font-size:90%;">Incorporating structured representations into pretrained vision \&amp; language models using scene graphs.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib26.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib26.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">The 2023 Conference on Empirical Methods in Natural Language Processing</em><span id="bib.bib26.11.3" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib27.5.5.1" class="ltx_text" style="font-size:90%;">Hojel et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib27.7.1" class="ltx_text" style="font-size:90%;">
Alberto Hojel, Yutong Bai, Trevor Darrell, Amir Globerson, and Amir Bar.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib27.8.1" class="ltx_text" style="font-size:90%;">Finding visual task vectors.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib27.9.1" class="ltx_text" style="font-size:90%;">2024.
</span>
</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib28.5.5.1" class="ltx_text" style="font-size:90%;">Houlsby et al. [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib28.7.1" class="ltx_text" style="font-size:90%;">
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib28.8.1" class="ltx_text" style="font-size:90%;">Parameter-efficient transfer learning for NLP.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib28.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib28.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the 36th International Conference on Machine Learning</em><span id="bib.bib28.11.3" class="ltx_text" style="font-size:90%;">, pages 2790–2799, 2019.
</span>
</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib29.5.5.1" class="ltx_text" style="font-size:90%;">Hu et al. [2021a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib29.7.1" class="ltx_text" style="font-size:90%;">
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib29.8.1" class="ltx_text" style="font-size:90%;">Lora: Low-rank adaptation of large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib29.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2106.09685</em><span id="bib.bib29.10.2" class="ltx_text" style="font-size:90%;">, 2021a.
</span>
</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib30.5.5.1" class="ltx_text" style="font-size:90%;">Hu et al. [2021b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib30.7.1" class="ltx_text" style="font-size:90%;">
J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib30.8.1" class="ltx_text" style="font-size:90%;">Lora: Low-rank adaptation of large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib30.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib30.10.2" class="ltx_text" style="font-size:90%;">, abs/2106.09685, 2021b.
</span>
</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib31.5.5.1" class="ltx_text" style="font-size:90%;">Hu et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib31.7.1" class="ltx_text" style="font-size:90%;">
Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, and Soujanya Poria.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib31.8.1" class="ltx_text" style="font-size:90%;">Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib31.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib31.10.2" class="ltx_text" style="font-size:90%;">, abs/2304.01933, 2023.
</span>
</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib32.4.4.1" class="ltx_text" style="font-size:90%;">Hudson and Manning [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib32.6.1" class="ltx_text" style="font-size:90%;">
Drew A. Hudson and Christopher D. Manning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib32.7.1" class="ltx_text" style="font-size:90%;">Gqa: A new dataset for real-world visual reasoning and compositional question answering.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib32.8.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)</em><span id="bib.bib32.9.2" class="ltx_text" style="font-size:90%;">, pages 6693–6702, 2019.
</span>
</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib33.5.5.1" class="ltx_text" style="font-size:90%;">Jia et al. [2021]</span></span>
<span class="ltx_bibblock"><span id="bib.bib33.7.1" class="ltx_text" style="font-size:90%;">
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib33.8.1" class="ltx_text" style="font-size:90%;">Scaling up visual and vision-language representation learning with noisy text supervision.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib33.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib33.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">International Conference on Machine Learning</em><span id="bib.bib33.11.3" class="ltx_text" style="font-size:90%;">, 2021.
</span>
</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib34.5.5.1" class="ltx_text" style="font-size:90%;">Jiang et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib34.7.1" class="ltx_text" style="font-size:90%;">
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max W.F. Ku, Qian Liu, and Wenhu Chen.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib34.8.1" class="ltx_text" style="font-size:90%;">Mantis: Interleaved multi-image instruction tuning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib34.9.1" class="ltx_text" style="font-size:90%;">arXiv2405.01483, 2024.
</span>
</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib35.5.5.1" class="ltx_text" style="font-size:90%;">Jiang et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib35.7.1" class="ltx_text" style="font-size:90%;">
Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib35.8.1" class="ltx_text" style="font-size:90%;">Llmlingua: Compressing prompts for accelerated inference of large language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib35.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib35.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Conference on Empirical Methods in Natural Language Processing</em><span id="bib.bib35.11.3" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib36.5.5.1" class="ltx_text" style="font-size:90%;">Kojima et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib36.7.1" class="ltx_text" style="font-size:90%;">
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib36.8.1" class="ltx_text" style="font-size:90%;">Large language models are zero-shot reasoners.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib36.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib36.10.2" class="ltx_text" style="font-size:90%;">, abs/2205.11916, 2022.
</span>
</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib37.5.5.1" class="ltx_text" style="font-size:90%;">Krishna et al. [2017]</span></span>
<span class="ltx_bibblock"><span id="bib.bib37.7.1" class="ltx_text" style="font-size:90%;">
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib37.8.1" class="ltx_text" style="font-size:90%;">Visual genome: Connecting language and vision using crowdsourced dense image annotations.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib37.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">International Journal of Computer Vision</em><span id="bib.bib37.10.2" class="ltx_text" style="font-size:90%;">, 123(1):32–73, 2017.
</span>
</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib38.5.5.1" class="ltx_text" style="font-size:90%;">Laurenccon et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib38.7.1" class="ltx_text" style="font-size:90%;">
Hugo Laurenccon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib38.8.1" class="ltx_text" style="font-size:90%;">Obelisc: An open web-scale filtered dataset of interleaved image-text documents.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib38.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib38.10.2" class="ltx_text" style="font-size:90%;">, abs/2306.16527, 2023.
</span>
</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib39.5.5.1" class="ltx_text" style="font-size:90%;">Laurenccon et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib39.7.1" class="ltx_text" style="font-size:90%;">
Hugo Laurenccon, Léo Tronchon, Matthieu Cord, and Victor Sanh.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib39.8.1" class="ltx_text" style="font-size:90%;">What matters when building vision-language models?
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib39.9.1" class="ltx_text" style="font-size:90%;">2024.
</span>
</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib40.5.5.1" class="ltx_text" style="font-size:90%;">Lei et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib40.7.1" class="ltx_text" style="font-size:90%;">
Bin Lei, Pei-Hung Lin, Chunhua Liao, and Caiwen Ding.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib40.8.1" class="ltx_text" style="font-size:90%;">Boosting logical reasoning in large language models through a new framework: The graph of thought.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib40.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib40.10.2" class="ltx_text" style="font-size:90%;">, abs/2308.08614, 2023.
</span>
</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib41.5.5.1" class="ltx_text" style="font-size:90%;">Lester et al. [2021]</span></span>
<span class="ltx_bibblock"><span id="bib.bib41.7.1" class="ltx_text" style="font-size:90%;">
Brian Lester, Rami Al-Rfou, and Noah Constant.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib41.8.1" class="ltx_text" style="font-size:90%;">The power of scale for parameter-efficient prompt tuning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib41.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib41.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Conference on Empirical Methods in Natural Language Processing</em><span id="bib.bib41.11.3" class="ltx_text" style="font-size:90%;">, 2021.
</span>
</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib42.5.5.1" class="ltx_text" style="font-size:90%;">Li et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib42.7.1" class="ltx_text" style="font-size:90%;">
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib42.8.1" class="ltx_text" style="font-size:90%;">Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib42.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2201.12086</em><span id="bib.bib42.10.2" class="ltx_text" style="font-size:90%;">, 2022.
</span>
</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib43.5.5.1" class="ltx_text" style="font-size:90%;">Li et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib43.7.1" class="ltx_text" style="font-size:90%;">
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib43.8.1" class="ltx_text" style="font-size:90%;">BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib43.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib43.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">ICML</em><span id="bib.bib43.11.3" class="ltx_text" style="font-size:90%;">, 2023a.
</span>
</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib44.5.5.1" class="ltx_text" style="font-size:90%;">Li et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib44.7.1" class="ltx_text" style="font-size:90%;">
Mukai Li, Shansan Gong, Jiangtao Feng, Yiheng Xu, Jinchao Zhang, Zhiyong Wu, and Lingpeng Kong.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib44.8.1" class="ltx_text" style="font-size:90%;">In-context learning with many demonstration examples.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib44.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib44.10.2" class="ltx_text" style="font-size:90%;">, abs/2302.04931, 2023b.
</span>
</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib45.5.5.1" class="ltx_text" style="font-size:90%;">Li et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib45.7.1" class="ltx_text" style="font-size:90%;">
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib45.8.1" class="ltx_text" style="font-size:90%;">Long-context llms struggle with long in-context learning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib45.9.1" class="ltx_text" style="font-size:90%;">2024.
</span>
</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib46.4.4.1" class="ltx_text" style="font-size:90%;">Li and Liang [2021]</span></span>
<span class="ltx_bibblock"><span id="bib.bib46.6.1" class="ltx_text" style="font-size:90%;">
Xiang Lisa Li and Percy Liang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib46.7.1" class="ltx_text" style="font-size:90%;">Prefix-tuning: Optimizing continuous prompts for generation.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib46.8.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)</em><span id="bib.bib46.9.2" class="ltx_text" style="font-size:90%;">, abs/2101.00190, 2021.
</span>
</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib47.5.5.1" class="ltx_text" style="font-size:90%;">Lin et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib47.7.1" class="ltx_text" style="font-size:90%;">
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib47.8.1" class="ltx_text" style="font-size:90%;">Vila: On pre-training for visual language models, 2023.
</span>
</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib48.5.5.1" class="ltx_text" style="font-size:90%;">Lin et al. [2014]</span></span>
<span class="ltx_bibblock"><span id="bib.bib48.7.1" class="ltx_text" style="font-size:90%;">
Tsung-Yi Lin, M. Maire, Serge J. Belongie, James Hays, P. Perona, D. Ramanan, Piotr Dollár, and C. L. Zitnick.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib48.8.1" class="ltx_text" style="font-size:90%;">Microsoft coco: Common objects in context.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib48.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib48.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">ECCV</em><span id="bib.bib48.11.3" class="ltx_text" style="font-size:90%;">, 2014.
</span>
</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib49.5.5.1" class="ltx_text" style="font-size:90%;">Liu et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib49.7.1" class="ltx_text" style="font-size:90%;">
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib49.8.1" class="ltx_text" style="font-size:90%;">Improved baselines with visual instruction tuning, 2023a.
</span>
</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib50.5.5.1" class="ltx_text" style="font-size:90%;">Liu et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib50.7.1" class="ltx_text" style="font-size:90%;">
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib50.8.1" class="ltx_text" style="font-size:90%;">Visual instruction tuning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib50.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib50.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">NeurIPS</em><span id="bib.bib50.11.3" class="ltx_text" style="font-size:90%;">, 2023b.
</span>
</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib51.5.5.1" class="ltx_text" style="font-size:90%;">Liu et al. [2023c]</span></span>
<span class="ltx_bibblock"><span id="bib.bib51.7.1" class="ltx_text" style="font-size:90%;">
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib51.8.1" class="ltx_text" style="font-size:90%;">Lost in the middle: How language models use long contexts.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib51.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Transactions of the Association for Computational Linguistics</em><span id="bib.bib51.10.2" class="ltx_text" style="font-size:90%;">, 12:157–173, 2023c.
</span>
</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib52.5.5.1" class="ltx_text" style="font-size:90%;">Lu et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib52.7.1" class="ltx_text" style="font-size:90%;">
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib52.8.1" class="ltx_text" style="font-size:90%;">Chameleon: Plug-and-play compositional reasoning with large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib52.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib52.10.2" class="ltx_text" style="font-size:90%;">, abs/2304.09842, 2023.
</span>
</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib53.5.5.1" class="ltx_text" style="font-size:90%;">Ma et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib53.7.1" class="ltx_text" style="font-size:90%;">
Huan Ma, Changqing Zhang, Yatao Bian, Lemao Liu, Zhirui Zhang, Peilin Zhao, Shu Zhang, H. Fu, Qinghua Hu, and Bing Wu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib53.8.1" class="ltx_text" style="font-size:90%;">Fairness-guided few-shot prompting for large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib53.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib53.10.2" class="ltx_text" style="font-size:90%;">, abs/2303.13217, 2023.
</span>
</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib54.5.5.1" class="ltx_text" style="font-size:90%;">Marino et al. [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib54.7.1" class="ltx_text" style="font-size:90%;">
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib54.8.1" class="ltx_text" style="font-size:90%;">Ok-vqa: A visual question answering benchmark requiring external knowledge.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib54.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)</em><span id="bib.bib54.10.2" class="ltx_text" style="font-size:90%;">, pages 3190–3199, 2019.
</span>
</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib55.5.5.1" class="ltx_text" style="font-size:90%;">Min et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib55.7.1" class="ltx_text" style="font-size:90%;">
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib55.8.1" class="ltx_text" style="font-size:90%;">Rethinking the role of demonstrations: What makes in-context learning work?
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib55.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib55.10.2" class="ltx_text" style="font-size:90%;">, abs/2202.12837, 2022.
</span>
</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib56.5.5.1" class="ltx_text" style="font-size:90%;">Mitra et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib56.7.1" class="ltx_text" style="font-size:90%;">
Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib56.8.1" class="ltx_text" style="font-size:90%;">Compositional chain of thought prompting for large multimodal models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib56.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib56.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)</em><span id="bib.bib56.11.3" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib57.5.5.1" class="ltx_text" style="font-size:90%;">Mu et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib57.7.1" class="ltx_text" style="font-size:90%;">
Jesse Mu, Xiang Lisa Li, and Noah D. Goodman.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib57.8.1" class="ltx_text" style="font-size:90%;">Learning to compress prompts with gist tokens.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib57.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib57.10.2" class="ltx_text" style="font-size:90%;">, abs/2304.08467, 2023.
</span>
</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib58.4.4.1" class="ltx_text" style="font-size:90%;">Nilsback and Zisserman [2008]</span></span>
<span class="ltx_bibblock"><span id="bib.bib58.6.1" class="ltx_text" style="font-size:90%;">
Maria-Elena Nilsback and Andrew Zisserman.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib58.7.1" class="ltx_text" style="font-size:90%;">Automated flower classification over a large number of classes.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib58.8.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">2008 Sixth Indian Conference on Computer Vision, Graphics &amp; Image Processing</em><span id="bib.bib58.9.2" class="ltx_text" style="font-size:90%;">, pages 722–729, 2008.
</span>
</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib59.4.4.1" class="ltx_text" style="font-size:90%;">OpenAI [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib59.6.1" class="ltx_text" style="font-size:90%;">
OpenAI.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib59.7.1" class="ltx_text" style="font-size:90%;">Gpt-4 technical report.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib59.8.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib59.9.2" class="ltx_text" style="font-size:90%;">, abs/2303.08774, 2023.
</span>
</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib60.5.5.1" class="ltx_text" style="font-size:90%;">Paszke et al. [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib60.7.1" class="ltx_text" style="font-size:90%;">
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib60.8.1" class="ltx_text" style="font-size:90%;">Pytorch: An imperative style, high-performance deep learning library.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib60.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Advances in neural information processing systems</em><span id="bib.bib60.10.2" class="ltx_text" style="font-size:90%;">, 32, 2019.
</span>
</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib61.4.4.1" class="ltx_text" style="font-size:90%;">Pearl [2001]</span></span>
<span class="ltx_bibblock"><span id="bib.bib61.6.1" class="ltx_text" style="font-size:90%;">
Judea Pearl.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib61.7.1" class="ltx_text" style="font-size:90%;">Direct and indirect effects.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib61.8.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Probabilistic and Causal Inference</em><span id="bib.bib61.9.2" class="ltx_text" style="font-size:90%;">, 2001.
</span>
</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib62.5.5.1" class="ltx_text" style="font-size:90%;">Peng et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib62.7.1" class="ltx_text" style="font-size:90%;">
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib62.8.1" class="ltx_text" style="font-size:90%;">Yarn: Efficient context window extension of large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib62.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib62.10.2" class="ltx_text" style="font-size:90%;">, abs/2309.00071, 2023.
</span>
</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib63.5.5.1" class="ltx_text" style="font-size:90%;">Qin et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib63.7.1" class="ltx_text" style="font-size:90%;">
Yujia Qin, Shi Liang, Yining Ye, Kunlun Zhu, Lan Yan, Ya-Ting Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Marc H. Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib63.8.1" class="ltx_text" style="font-size:90%;">Toolllm: Facilitating large language models to master 16000+ real-world apis.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib63.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib63.10.2" class="ltx_text" style="font-size:90%;">, abs/2307.16789, 2023.
</span>
</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib64.5.5.1" class="ltx_text" style="font-size:90%;">Radford et al. [2021]</span></span>
<span class="ltx_bibblock"><span id="bib.bib64.7.1" class="ltx_text" style="font-size:90%;">
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib64.8.1" class="ltx_text" style="font-size:90%;">Learning transferable visual models from natural language supervision.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib64.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib64.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">International Conference on Machine Learning</em><span id="bib.bib64.11.3" class="ltx_text" style="font-size:90%;">, pages 8748–8763. PMLR, 2021.
</span>
</span>
</li>
<li id="bib.bib65" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib65.5.5.1" class="ltx_text" style="font-size:90%;">Raffel et al. [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib65.7.1" class="ltx_text" style="font-size:90%;">
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib65.8.1" class="ltx_text" style="font-size:90%;">Exploring the limits of transfer learning with a unified text-to-text transformer.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib65.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">J. Mach. Learn. Res.</em><span id="bib.bib65.10.2" class="ltx_text" style="font-size:90%;">, 21:140:1–140:67, 2019.
</span>
</span>
</li>
<li id="bib.bib66" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib66.5.5.1" class="ltx_text" style="font-size:90%;">Saikh et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib66.7.1" class="ltx_text" style="font-size:90%;">
Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib66.8.1" class="ltx_text" style="font-size:90%;">Scienceqa: a novel resource for question answering on scholarly articles.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib66.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">International Journal on Digital Libraries</em><span id="bib.bib66.10.2" class="ltx_text" style="font-size:90%;">, 23:289 – 301, 2022.
</span>
</span>
</li>
<li id="bib.bib67" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib67.5.5.1" class="ltx_text" style="font-size:90%;">Sanh et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib67.7.1" class="ltx_text" style="font-size:90%;">
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib67.8.1" class="ltx_text" style="font-size:90%;">Multitask prompted training enables zero-shot task generalization.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib67.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib67.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">International Conference on Learning Representations</em><span id="bib.bib67.11.3" class="ltx_text" style="font-size:90%;">, 2022.
</span>
</span>
</li>
<li id="bib.bib68" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib68.5.5.1" class="ltx_text" style="font-size:90%;">Schick et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib68.7.1" class="ltx_text" style="font-size:90%;">
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib68.8.1" class="ltx_text" style="font-size:90%;">Toolformer: Language models can teach themselves to use tools.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib68.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib68.10.2" class="ltx_text" style="font-size:90%;">, abs/2302.04761, 2023.
</span>
</span>
</li>
<li id="bib.bib69" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib69.5.5.1" class="ltx_text" style="font-size:90%;">Shang et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib69.7.1" class="ltx_text" style="font-size:90%;">
Chuyi Shang, Amos You, Sanjay Subramanian, Trevor Darrell, and Roei Herzig.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib69.8.1" class="ltx_text" style="font-size:90%;">Traveler: A multi-lmm agent framework for video question-answering.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib69.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib69.10.2" class="ltx_text" style="font-size:90%;">, abs/2404.01476, 2024.
</span>
</span>
</li>
<li id="bib.bib70" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib70.5.5.1" class="ltx_text" style="font-size:90%;">Shen et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib70.7.1" class="ltx_text" style="font-size:90%;">
Yongliang Shen, Kaitao Song, Xu Tan, Dong Sheng Li, Weiming Lu, and Yue Ting Zhuang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib70.8.1" class="ltx_text" style="font-size:90%;">Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib70.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib70.10.2" class="ltx_text" style="font-size:90%;">, abs/2303.17580, 2023.
</span>
</span>
</li>
<li id="bib.bib71" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib71.5.5.1" class="ltx_text" style="font-size:90%;">Snell et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib71.7.1" class="ltx_text" style="font-size:90%;">
Charles Burton Snell, Dan Klein, and Ruiqi Zhong.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib71.8.1" class="ltx_text" style="font-size:90%;">Learning by distilling context.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib71.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib71.10.2" class="ltx_text" style="font-size:90%;">, abs/2209.15189, 2022.
</span>
</span>
</li>
<li id="bib.bib72" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib72.5.5.1" class="ltx_text" style="font-size:90%;">Subramanian et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib72.7.1" class="ltx_text" style="font-size:90%;">
Sanjay Subramanian, Medhini G. Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, and Dan Klein.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib72.8.1" class="ltx_text" style="font-size:90%;">Modular visual question answering via code generation.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib72.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib72.10.2" class="ltx_text" style="font-size:90%;">, abs/2306.05392, 2023.
</span>
</span>
</li>
<li id="bib.bib73" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib73.5.5.1" class="ltx_text" style="font-size:90%;">Sun et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib73.7.1" class="ltx_text" style="font-size:90%;">
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib73.8.1" class="ltx_text" style="font-size:90%;">Generative multimodal models are in-context learners.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib73.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib73.10.2" class="ltx_text" style="font-size:90%;">, abs/2312.13286, 2023.
</span>
</span>
</li>
<li id="bib.bib74" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib74.5.5.1" class="ltx_text" style="font-size:90%;">Sur’is et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib74.7.1" class="ltx_text" style="font-size:90%;">
D’idac Sur’is, Sachit Menon, and Carl Vondrick.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib74.8.1" class="ltx_text" style="font-size:90%;">Vipergpt: Visual inference via python execution for reasoning.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib74.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib74.10.2" class="ltx_text" style="font-size:90%;">, abs/2303.08128, 2023.
</span>
</span>
</li>
<li id="bib.bib75" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib75.5.5.1" class="ltx_text" style="font-size:90%;">Tan et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib75.7.1" class="ltx_text" style="font-size:90%;">
Sijun Tan, Xiuyu Li, Shishir G. Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph E. Gonzalez, and Raluca A. Popa.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib75.8.1" class="ltx_text" style="font-size:90%;">Lloco: Learning long contexts offline.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib75.9.1" class="ltx_text" style="font-size:90%;">2024.
</span>
</span>
</li>
<li id="bib.bib76" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib76.5.5.1" class="ltx_text" style="font-size:90%;">Tay et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib76.7.1" class="ltx_text" style="font-size:90%;">
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier García, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib76.8.1" class="ltx_text" style="font-size:90%;">Ul2: Unifying language learning paradigms.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib76.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib76.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">International Conference on Learning Representations</em><span id="bib.bib76.11.3" class="ltx_text" style="font-size:90%;">, 2022.
</span>
</span>
</li>
<li id="bib.bib77" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib77.4.4.1" class="ltx_text" style="font-size:90%;">Team [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib77.6.1" class="ltx_text" style="font-size:90%;">
Gemini Team.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib77.7.1" class="ltx_text" style="font-size:90%;">Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib77.8.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib77.9.2" class="ltx_text" style="font-size:90%;">, abs/2403.05530, 2024.
</span>
</span>
</li>
<li id="bib.bib78" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib78.5.5.1" class="ltx_text" style="font-size:90%;">Team et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib78.7.1" class="ltx_text" style="font-size:90%;">
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib78.8.1" class="ltx_text" style="font-size:90%;">Gemini: a family of highly capable multimodal models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib78.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2312.11805</em><span id="bib.bib78.10.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib79" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib79.5.5.1" class="ltx_text" style="font-size:90%;">Todd et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib79.7.1" class="ltx_text" style="font-size:90%;">
Eric Todd, Millicent Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib79.8.1" class="ltx_text" style="font-size:90%;">Function vectors in large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib79.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib79.10.2" class="ltx_text" style="font-size:90%;">, abs/2310.15213, 2023.
</span>
</span>
</li>
<li id="bib.bib80" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib80.5.5.1" class="ltx_text" style="font-size:90%;">Wah et al. [2011]</span></span>
<span class="ltx_bibblock"><span id="bib.bib80.7.1" class="ltx_text" style="font-size:90%;">
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge J. Belongie.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib80.8.1" class="ltx_text" style="font-size:90%;">The caltech-ucsd birds-200-2011 dataset.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib80.9.1" class="ltx_text" style="font-size:90%;">2011.
</span>
</span>
</li>
<li id="bib.bib81" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib81.5.5.1" class="ltx_text" style="font-size:90%;">Wan et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib81.7.1" class="ltx_text" style="font-size:90%;">
Xingchen Wan, Ruoxi Sun, Hanjun Dai, Sercan Ö. Arik, and Tomas Pfister.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib81.8.1" class="ltx_text" style="font-size:90%;">Better zero-shot reasoning with self-adaptive prompting.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib81.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib81.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Annual Meeting of the Association for Computational Linguistics</em><span id="bib.bib81.11.3" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib82" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib82.5.5.1" class="ltx_text" style="font-size:90%;">Wang et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib82.7.1" class="ltx_text" style="font-size:90%;">
Lei Wang, Yilang Hu, Jiabang He, Xingdong Xu, Ning Liu, Hui juan Liu, and Hengtao Shen.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib82.8.1" class="ltx_text" style="font-size:90%;">T-sciq: Teaching multimodal chain-of-thought reasoning via large language model signals for science question answering.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib82.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib82.10.2" class="ltx_text" style="font-size:90%;">, abs/2305.03453, 2023a.
</span>
</span>
</li>
<li id="bib.bib83" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib83.5.5.1" class="ltx_text" style="font-size:90%;">Wang et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib83.7.1" class="ltx_text" style="font-size:90%;">
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib83.8.1" class="ltx_text" style="font-size:90%;">Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib83.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib83.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Annual Meeting of the Association for Computational Linguistics</em><span id="bib.bib83.11.3" class="ltx_text" style="font-size:90%;">, 2023b.
</span>
</span>
</li>
<li id="bib.bib84" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib84.5.5.1" class="ltx_text" style="font-size:90%;">Wang et al. [2022a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib84.7.1" class="ltx_text" style="font-size:90%;">
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Huai hsin Chi, and Denny Zhou.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib84.8.1" class="ltx_text" style="font-size:90%;">Self-consistency improves chain of thought reasoning in language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib84.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib84.10.2" class="ltx_text" style="font-size:90%;">, abs/2203.11171, 2022a.
</span>
</span>
</li>
<li id="bib.bib85" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib85.5.5.1" class="ltx_text" style="font-size:90%;">Wang et al. [2022b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib85.7.1" class="ltx_text" style="font-size:90%;">
Zhenhailong Wang, Manling Li, Ruochen Xu, Luowei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem, Shih-Fu Chang, Mohit Bansal, and Heng Ji.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib85.8.1" class="ltx_text" style="font-size:90%;">Language models with image descriptors are strong few-shot video-language learners.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib85.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib85.10.2" class="ltx_text" style="font-size:90%;">, abs/2205.10747, 2022b.
</span>
</span>
</li>
<li id="bib.bib86" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib86.5.5.1" class="ltx_text" style="font-size:90%;">Wei et al. [2021]</span></span>
<span class="ltx_bibblock"><span id="bib.bib86.7.1" class="ltx_text" style="font-size:90%;">
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib86.8.1" class="ltx_text" style="font-size:90%;">Finetuned language models are zero-shot learners.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib86.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib86.10.2" class="ltx_text" style="font-size:90%;">, abs/2109.01652, 2021.
</span>
</span>
</li>
<li id="bib.bib87" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib87.5.5.1" class="ltx_text" style="font-size:90%;">Wei et al. [2022a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib87.7.1" class="ltx_text" style="font-size:90%;">
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Huai hsin Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib87.8.1" class="ltx_text" style="font-size:90%;">Emergent abilities of large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib87.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Trans. Mach. Learn. Res.</em><span id="bib.bib87.10.2" class="ltx_text" style="font-size:90%;">, 2022, 2022a.
</span>
</span>
</li>
<li id="bib.bib88" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib88.5.5.1" class="ltx_text" style="font-size:90%;">Wei et al. [2022b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib88.7.1" class="ltx_text" style="font-size:90%;">
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib88.8.1" class="ltx_text" style="font-size:90%;">Chain of thought prompting elicits reasoning in large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib88.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib88.10.2" class="ltx_text" style="font-size:90%;">, abs/2201.11903, 2022b.
</span>
</span>
</li>
<li id="bib.bib89" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib89.4.4.1" class="ltx_text" style="font-size:90%;">Williams [2004]</span></span>
<span class="ltx_bibblock"><span id="bib.bib89.6.1" class="ltx_text" style="font-size:90%;">
Ronald J. Williams.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib89.7.1" class="ltx_text" style="font-size:90%;">Simple statistical gradient-following algorithms for connectionist reinforcement learning.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib89.8.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Machine Learning</em><span id="bib.bib89.9.2" class="ltx_text" style="font-size:90%;">, 8:229–256, 2004.
</span>
</span>
</li>
<li id="bib.bib90" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib90.5.5.1" class="ltx_text" style="font-size:90%;">Wu et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib90.7.1" class="ltx_text" style="font-size:90%;">
Chenfei Wu, Sheng-Kai Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib90.8.1" class="ltx_text" style="font-size:90%;">Visual chatgpt: Talking, drawing and editing with visual foundation models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib90.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib90.10.2" class="ltx_text" style="font-size:90%;">, abs/2303.04671, 2023.
</span>
</span>
</li>
<li id="bib.bib91" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib91.5.5.1" class="ltx_text" style="font-size:90%;">Xu et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib91.7.1" class="ltx_text" style="font-size:90%;">
Benfeng Xu, An Yang, Junyang Lin, Quang Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib91.8.1" class="ltx_text" style="font-size:90%;">Expertprompting: Instructing large language models to be distinguished experts.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib91.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib91.10.2" class="ltx_text" style="font-size:90%;">, abs/2305.14688, 2023.
</span>
</span>
</li>
<li id="bib.bib92" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib92.5.5.1" class="ltx_text" style="font-size:90%;">Yao et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib92.7.1" class="ltx_text" style="font-size:90%;">
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib92.8.1" class="ltx_text" style="font-size:90%;">Tree of thoughts: Deliberate problem solving with large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib92.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib92.10.2" class="ltx_text" style="font-size:90%;">, abs/2305.10601, 2023a.
</span>
</span>
</li>
<li id="bib.bib93" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib93.5.5.1" class="ltx_text" style="font-size:90%;">Yao et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib93.7.1" class="ltx_text" style="font-size:90%;">
Yao Yao, Z. Li, and Hai Zhao.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib93.8.1" class="ltx_text" style="font-size:90%;">Beyond chain-of-thought, effective graph-of-thought reasoning in large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib93.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib93.10.2" class="ltx_text" style="font-size:90%;">, abs/2305.16582, 2023b.
</span>
</span>
</li>
<li id="bib.bib94" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib94.5.5.1" class="ltx_text" style="font-size:90%;">Ye et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib94.7.1" class="ltx_text" style="font-size:90%;">
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yi Zhou, Junyan Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qiang Qi, Ji Zhang, and Feiyan Huang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib94.8.1" class="ltx_text" style="font-size:90%;">mplug-owl: Modularization empowers large language models with multimodality.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib94.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib94.10.2" class="ltx_text" style="font-size:90%;">, abs/2304.14178, 2023a.
</span>
</span>
</li>
<li id="bib.bib95" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib95.5.5.1" class="ltx_text" style="font-size:90%;">Ye et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib95.7.1" class="ltx_text" style="font-size:90%;">
Qinghao Ye, Haiyang Xu, Jiabo Ye, Mingshi Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib95.8.1" class="ltx_text" style="font-size:90%;">mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib95.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib95.10.2" class="ltx_text" style="font-size:90%;">, abs/2311.04257, 2023b.
</span>
</span>
</li>
<li id="bib.bib96" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib96.5.5.1" class="ltx_text" style="font-size:90%;">Zhang et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib96.7.1" class="ltx_text" style="font-size:90%;">
Qingru Zhang, Minshuo Chen, Alexander W. Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib96.8.1" class="ltx_text" style="font-size:90%;">Adaptive budget allocation for parameter-efficient fine-tuning.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib96.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib96.10.2" class="ltx_text" style="font-size:90%;">, abs/2303.10512, 2023a.
</span>
</span>
</li>
<li id="bib.bib97" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib97.5.5.1" class="ltx_text" style="font-size:90%;">Zhang et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib97.7.1" class="ltx_text" style="font-size:90%;">
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Jiao Qiao.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib97.8.1" class="ltx_text" style="font-size:90%;">Llama-adapter: Efficient fine-tuning of language models with zero-init attention.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib97.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib97.10.2" class="ltx_text" style="font-size:90%;">, abs/2303.16199, 2023b.
</span>
</span>
</li>
<li id="bib.bib98" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib98.5.5.1" class="ltx_text" style="font-size:90%;">Zhang et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib98.7.1" class="ltx_text" style="font-size:90%;">
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alexander J. Smola.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib98.8.1" class="ltx_text" style="font-size:90%;">Automatic chain of thought prompting in large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib98.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib98.10.2" class="ltx_text" style="font-size:90%;">, abs/2210.03493, 2022.
</span>
</span>
</li>
<li id="bib.bib99" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib99.5.5.1" class="ltx_text" style="font-size:90%;">Zhang et al. [2023c]</span></span>
<span class="ltx_bibblock"><span id="bib.bib99.7.1" class="ltx_text" style="font-size:90%;">
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alexander J. Smola.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib99.8.1" class="ltx_text" style="font-size:90%;">Multimodal chain-of-thought reasoning in language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib99.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib99.10.2" class="ltx_text" style="font-size:90%;">, abs/2302.00923, 2023c.
</span>
</span>
</li>
<li id="bib.bib100" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib100.5.5.1" class="ltx_text" style="font-size:90%;">Zhao et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib100.7.1" class="ltx_text" style="font-size:90%;">
Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib100.8.1" class="ltx_text" style="font-size:90%;">Mmicl: Empowering vision-language model with multi-modal in-context learning.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib100.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib100.10.2" class="ltx_text" style="font-size:90%;">, abs/2309.07915, 2023.
</span>
</span>
</li>
<li id="bib.bib101" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib101.5.5.1" class="ltx_text" style="font-size:90%;">Zheng et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib101.7.1" class="ltx_text" style="font-size:90%;">
Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib101.8.1" class="ltx_text" style="font-size:90%;">Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib101.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib101.10.2" class="ltx_text" style="font-size:90%;">, abs/2310.16436, 2023.
</span>
</span>
</li>
<li id="bib.bib102" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib102.5.5.1" class="ltx_text" style="font-size:90%;">Zhu et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib102.7.1" class="ltx_text" style="font-size:90%;">
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib102.8.1" class="ltx_text" style="font-size:90%;">Minigpt-4: Enhancing vision-language understanding with advanced large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib102.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2304.10592</em><span id="bib.bib102.10.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<div id="p1" class="ltx_para">
<span id="p1.1" class="ltx_ERROR undefined">\maketitlesupplementary</span>
</div>
<div id="p2" class="ltx_para">
<p id="p2.1" class="ltx_p">Here, we provide additional information about our experimental results, qualitative examples, implementation details, and datasets. Specifically, Section <a href="#A1" title="Appendix A Additional Experiment Results ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a> provides more experiment results, Section <a href="#A2" title="Appendix B Additional Method Details ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a> provides additional method details, Section <a href="#A3" title="Appendix C Additional Implementation Details ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a> provides additional implementation details, and Section <a href="#A4" title="Appendix D Qualitative Visualizations ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D</span></a> provides qualitative visualizations to illustrate our approach.</p>
</div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Additional Experiment Results</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.1" class="ltx_p">We present several additional experiments that further demonstrate the benefits of our MTV approach.</p>
</div>
<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>Additional Experiments</h3>

<div id="A1.SS1.p1" class="ltx_para">
<p id="A1.SS1.p1.1" class="ltx_p">Here we provide additional ablations that further illustrate different characteristics of MTV.</p>
</div>
<div id="A1.SS1.p2" class="ltx_para ltx_noindent">
<p id="A1.SS1.p2.2" class="ltx_p"><span id="A1.SS1.p2.2.1" class="ltx_text ltx_font_bold">Attention head generalization on object classification tasks Table <a href="#A1.T4.st1" title="In Table 4 ‣ A.1 Additional Experiments ‣ Appendix A Additional Experiment Results ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4(a)</span></a></span>. We also test generalization for object classification tasks identical to the formulation described in Section <a href="#S5.SS3" title="5.3 MTV heads generalize to other tasks ‣ 5 Results ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.3</span></a>. For clarity, MTV shows another kind of generalization when it is leveraged alongside additional explicit ICL samples. This capability is described in Section <a href="#S5.SS2" title="5.2 MTV works with explicit few-shot examples ‣ 5 Results ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.2</span></a>. To summarize our experiment, we calculate MTV using the Flowers dataset using 1-shot ICL example for 100 iterations for both the mean activations <math id="A1.SS1.p2.1.m1.1" class="ltx_Math" alttext="\mu^{\mathrm{MTV}}_{j}" display="inline"><semantics id="A1.SS1.p2.1.m1.1a"><msubsup id="A1.SS1.p2.1.m1.1.1" xref="A1.SS1.p2.1.m1.1.1.cmml"><mi id="A1.SS1.p2.1.m1.1.1.2.2" xref="A1.SS1.p2.1.m1.1.1.2.2.cmml">μ</mi><mi id="A1.SS1.p2.1.m1.1.1.3" xref="A1.SS1.p2.1.m1.1.1.3.cmml">j</mi><mi id="A1.SS1.p2.1.m1.1.1.2.3" xref="A1.SS1.p2.1.m1.1.1.2.3.cmml">MTV</mi></msubsup><annotation-xml encoding="MathML-Content" id="A1.SS1.p2.1.m1.1b"><apply id="A1.SS1.p2.1.m1.1.1.cmml" xref="A1.SS1.p2.1.m1.1.1"><csymbol cd="ambiguous" id="A1.SS1.p2.1.m1.1.1.1.cmml" xref="A1.SS1.p2.1.m1.1.1">subscript</csymbol><apply id="A1.SS1.p2.1.m1.1.1.2.cmml" xref="A1.SS1.p2.1.m1.1.1"><csymbol cd="ambiguous" id="A1.SS1.p2.1.m1.1.1.2.1.cmml" xref="A1.SS1.p2.1.m1.1.1">superscript</csymbol><ci id="A1.SS1.p2.1.m1.1.1.2.2.cmml" xref="A1.SS1.p2.1.m1.1.1.2.2">𝜇</ci><ci id="A1.SS1.p2.1.m1.1.1.2.3.cmml" xref="A1.SS1.p2.1.m1.1.1.2.3">MTV</ci></apply><ci id="A1.SS1.p2.1.m1.1.1.3.cmml" xref="A1.SS1.p2.1.m1.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p2.1.m1.1c">\mu^{\mathrm{MTV}}_{j}</annotation></semantics></math> and the attention head locations <math id="A1.SS1.p2.2.m2.1" class="ltx_Math" alttext="\lambda^{\mathrm{MTV}}_{j}" display="inline"><semantics id="A1.SS1.p2.2.m2.1a"><msubsup id="A1.SS1.p2.2.m2.1.1" xref="A1.SS1.p2.2.m2.1.1.cmml"><mi id="A1.SS1.p2.2.m2.1.1.2.2" xref="A1.SS1.p2.2.m2.1.1.2.2.cmml">λ</mi><mi id="A1.SS1.p2.2.m2.1.1.3" xref="A1.SS1.p2.2.m2.1.1.3.cmml">j</mi><mi id="A1.SS1.p2.2.m2.1.1.2.3" xref="A1.SS1.p2.2.m2.1.1.2.3.cmml">MTV</mi></msubsup><annotation-xml encoding="MathML-Content" id="A1.SS1.p2.2.m2.1b"><apply id="A1.SS1.p2.2.m2.1.1.cmml" xref="A1.SS1.p2.2.m2.1.1"><csymbol cd="ambiguous" id="A1.SS1.p2.2.m2.1.1.1.cmml" xref="A1.SS1.p2.2.m2.1.1">subscript</csymbol><apply id="A1.SS1.p2.2.m2.1.1.2.cmml" xref="A1.SS1.p2.2.m2.1.1"><csymbol cd="ambiguous" id="A1.SS1.p2.2.m2.1.1.2.1.cmml" xref="A1.SS1.p2.2.m2.1.1">superscript</csymbol><ci id="A1.SS1.p2.2.m2.1.1.2.2.cmml" xref="A1.SS1.p2.2.m2.1.1.2.2">𝜆</ci><ci id="A1.SS1.p2.2.m2.1.1.2.3.cmml" xref="A1.SS1.p2.2.m2.1.1.2.3">MTV</ci></apply><ci id="A1.SS1.p2.2.m2.1.1.3.cmml" xref="A1.SS1.p2.2.m2.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p2.2.m2.1c">\lambda^{\mathrm{MTV}}_{j}</annotation></semantics></math>. Then, we apply MTV to the CUB task <span id="A1.SS1.p2.2.2" class="ltx_text ltx_font_italic">using the same set of attention head locations from Flowers</span>. We just calculate the mean activations for the CUB dataset using a 1-shot for 100 iterations (halving our data requirement for this specific scenario). Once again, we find that the heads of MTV can indeed generalize between similar classes.</p>
</div>
<figure id="A1.T4" class="ltx_table">

<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="A1.T4.3.1.1" class="ltx_text" style="font-size:90%;">Table 4</span>: </span><span id="A1.T4.4.2" class="ltx_text ltx_font_bold" style="font-size:90%;">Generalization &amp; Direct ICL Comparison<span id="A1.T4.4.2.1" class="ltx_text ltx_font_medium"> (Left) MTV-Flowers evaluated on OK-VQA. (Right) Direct comparison of MTV extracted from 4-shots, 1-iteration (MTV_4shot_1it) compared to 4-shot ICL</span></span></figcaption><div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A1.T4.st1" class="ltx_table ltx_figure_panel">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A1.T4.st1.2.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="A1.T4.st1.3.2" class="ltx_text" style="font-size:90%;">Attention Head Generalization</span></figcaption>
<div id="A1.T4.st1.4" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:133.3pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(99.7pt,-30.6pt) scale(1.8508775146387,1.8508775146387) ;">
<table id="A1.T4.st1.4.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T4.st1.4.1.1.1" class="ltx_tr">
<th id="A1.T4.st1.4.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding-top:0.5pt;padding-bottom:0.5pt;">Model</th>
<th id="A1.T4.st1.4.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="A1.T4.st1.4.1.1.1.2.1" class="ltx_text" style="background-color:#E6E6E6;">Flowers</span></th>
<th id="A1.T4.st1.4.1.1.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="A1.T4.st1.4.1.1.1.3.1" class="ltx_text" style="background-color:#E6E6E6;">CUB</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T4.st1.4.1.2.1" class="ltx_tr">
<th id="A1.T4.st1.4.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">ViLA-1.5-8B</th>
<td id="A1.T4.st1.4.1.2.1.2" class="ltx_td ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;"></td>
<td id="A1.T4.st1.4.1.2.1.3" class="ltx_td ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;"></td>
</tr>
<tr id="A1.T4.st1.4.1.3.2" class="ltx_tr">
<th id="A1.T4.st1.4.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:0.5pt;padding-bottom:0.5pt;">   + 1-shot-ICL</th>
<td id="A1.T4.st1.4.1.3.2.2" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="A1.T4.st1.4.1.3.2.2.1" class="ltx_text" style="background-color:#E6E6E6;">87.4</span></td>
<td id="A1.T4.st1.4.1.3.2.3" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="A1.T4.st1.4.1.3.2.3.1" class="ltx_text" style="background-color:#E6E6E6;">88.4</span></td>
</tr>
<tr id="A1.T4.st1.4.1.4.3" class="ltx_tr">
<th id="A1.T4.st1.4.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding-top:0.5pt;padding-bottom:0.5pt;">   + <span id="A1.T4.st1.4.1.4.3.1.1" class="ltx_text ltx_font_bold">MTV-Flowers</span>+1-shot-ICL</th>
<td id="A1.T4.st1.4.1.4.3.2" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="A1.T4.st1.4.1.4.3.2.1" class="ltx_text" style="background-color:#E6E6E6;">89.3</span></td>
<td id="A1.T4.st1.4.1.4.3.3" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="A1.T4.st1.4.1.4.3.3.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6E6;">89.9</span></td>
</tr>
</tbody>
</table>
</span></div>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="A1.T4.st2" class="ltx_table ltx_figure_panel">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A1.T4.st2.2.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="A1.T4.st2.3.2" class="ltx_text" style="font-size:90%;">Comparison to Other Methods</span></figcaption>
<div id="A1.T4.st2.4" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:147.7pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(111.1pt,-37.9pt) scale(2.05182671633516,2.05182671633516) ;">
<table id="A1.T4.st2.4.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T4.st2.4.1.1.1" class="ltx_tr">
<th id="A1.T4.st2.4.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding-top:0.5pt;padding-bottom:0.5pt;">Model</th>
<th id="A1.T4.st2.4.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="A1.T4.st2.4.1.1.1.2.1" class="ltx_text" style="background-color:#E6E6E6;">VizWiz</span></th>
<th id="A1.T4.st2.4.1.1.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="A1.T4.st2.4.1.1.1.3.1" class="ltx_text" style="background-color:#E6E6E6;">OK-VQA</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T4.st2.4.1.2.1" class="ltx_tr">
<th id="A1.T4.st2.4.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-top:0.5pt;padding-bottom:0.5pt;">ViLA-1.5-8B</th>
<td id="A1.T4.st2.4.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="A1.T4.st2.4.1.2.1.2.1" class="ltx_text" style="background-color:#E6E6E6;">28.0</span></td>
<td id="A1.T4.st2.4.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="A1.T4.st2.4.1.2.1.3.1" class="ltx_text" style="background-color:#E6E6E6;">32.8</span></td>
</tr>
<tr id="A1.T4.st2.4.1.3.2" class="ltx_tr">
<th id="A1.T4.st2.4.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-top:0.5pt;padding-bottom:0.5pt;">   + 4-shot-ICL</th>
<td id="A1.T4.st2.4.1.3.2.2" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="A1.T4.st2.4.1.3.2.2.1" class="ltx_text" style="background-color:#E6E6E6;">39.3</span></td>
<td id="A1.T4.st2.4.1.3.2.3" class="ltx_td ltx_align_center" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="A1.T4.st2.4.1.3.2.3.1" class="ltx_text" style="background-color:#E6E6E6;">35.6</span></td>
</tr>
<tr id="A1.T4.st2.4.1.4.3" class="ltx_tr">
<th id="A1.T4.st2.4.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding-top:0.5pt;padding-bottom:0.5pt;">   + <span id="A1.T4.st2.4.1.4.3.1.1" class="ltx_text ltx_font_bold">MTV</span>_4shot_1it</th>
<td id="A1.T4.st2.4.1.4.3.2" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="A1.T4.st2.4.1.4.3.2.1" class="ltx_text" style="background-color:#E6E6E6;">57.4</span></td>
<td id="A1.T4.st2.4.1.4.3.3" class="ltx_td ltx_align_center ltx_border_bb" style="background-color:#E6E6E6;padding-top:0.5pt;padding-bottom:0.5pt;"><span id="A1.T4.st2.4.1.4.3.3.1" class="ltx_text" style="background-color:#E6E6E6;">40.0</span></td>
</tr>
</tbody>
</table>
</span></div>
</figure>
</div>
</div>
</figure>
<div id="A1.SS1.p3" class="ltx_para ltx_noindent">
<p id="A1.SS1.p3.1" class="ltx_p"><span id="A1.SS1.p3.1.1" class="ltx_text ltx_font_bold">MTV one-to-one comparison with ICL Table <a href="#A1.T4.st2" title="In Table 4 ‣ A.1 Additional Experiments ‣ Appendix A Additional Experiment Results ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4(b)</span></a></span>. Although not directly comparable, we consider an extreme case of MTV where we encode only 4-shots of ICL examples for 1 iteration. This matches the exact setting used in standard 4-shot ICL. Interestingly, MTV applied to both VizWiz and OK-VQA exceeds performance on the 4-shot-ICL case and even MTV formulated on 4-shots per 100 iterations for calculating the mean activations. This result suggests that there may be scope for MTV to be effective in both high and low-data regimens. More research needs to be done to explore this idea.</p>
</div>
<div id="A1.SS1.p4" class="ltx_para ltx_noindent">
<p id="A1.SS1.p4.2" class="ltx_p"><span id="A1.SS1.p4.2.1" class="ltx_text ltx_font_bold">Effect of permutation order of examples</span>. We consider applying five random seeds to both 4-shot-ICL and MTV extracted on 4-shots per 100 iterations on VizWiz. We find the 4-shot-ICL average and standard deviation to be 41.3 % (<math id="A1.SS1.p4.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A1.SS1.p4.1.m1.1a"><mo id="A1.SS1.p4.1.m1.1.1" xref="A1.SS1.p4.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A1.SS1.p4.1.m1.1b"><csymbol cd="latexml" id="A1.SS1.p4.1.m1.1.1.cmml" xref="A1.SS1.p4.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p4.1.m1.1c">\pm</annotation></semantics></math> .8%) and the MTV average and standard deviation to be 45.2 % (<math id="A1.SS1.p4.2.m2.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A1.SS1.p4.2.m2.1a"><mo id="A1.SS1.p4.2.m2.1.1" xref="A1.SS1.p4.2.m2.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A1.SS1.p4.2.m2.1b"><csymbol cd="latexml" id="A1.SS1.p4.2.m2.1.1.cmml" xref="A1.SS1.p4.2.m2.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p4.2.m2.1c">\pm</annotation></semantics></math> .7 %). This suggests that MTV is stable across different permutations of the given ICL examples.</p>
</div>
<div id="A1.SS1.p5" class="ltx_para ltx_noindent">
<p id="A1.SS1.p5.1" class="ltx_p"><span id="A1.SS1.p5.1.1" class="ltx_text ltx_font_bold">Scaling on Flowers Dataset</span>.</p>
</div>
<figure id="A1.F3" class="ltx_figure"><img src="/html/2406.15334/assets/figs/plot4.png" id="A1.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="359" height="279" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A1.F3.3.1.1" class="ltx_text" style="font-size:90%;">Figure 3</span>: </span><span id="A1.F3.4.2" class="ltx_text" style="font-size:90%;">
<span id="A1.F3.4.2.1" class="ltx_text ltx_font_bold">Efficiency.</span> We show that for Flowers, MTV does scale to but only up to 100 examples in our experiments.
</span></figcaption>
</figure>
<div id="A1.SS1.p6" class="ltx_para">
<p id="A1.SS1.p6.1" class="ltx_p">We provide additional results on the scaling property of MTV on the Flowers dataset. We again note that the examples are <span id="A1.SS1.p6.1.1" class="ltx_text ltx_font_italic">2-way</span>, one-shot examples with 2 examples (one positive and one negative) for each sample. As in the main paper, we fix 1 shot per iteration to calculate the mean activations, scaling up to 500 total examples used. Our results show that there is a saturation of MTV at 100 examples (i.e., 1 example per 100 iterations). While this still indicates some scaling as the result is an improvement over 20 examples, the results show that the task vector can reach its best accuracy with fewer shots depending on the complexity of the task. Future work to probe more deeply into the scaling nature of MTV across different tasks would be valuable.</p>
</div>
</section>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Additional Method Details</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.1" class="ltx_p">Here we provide some additional method details about MTV, Visual Task Vectors (VTV) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib27" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">27</span></a>]</cite>, and Function Vectors <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib79" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">79</span></a>]</cite> (FV).</p>
</div>
<section id="A2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.1 </span>MTV-EXTRACT</h3>

<div id="A2.SS1.p1" class="ltx_para">
<p id="A2.SS1.p1.2" class="ltx_p">We describe the particulars of our MTV-EXTRACT algorithm for finding the set of attention head locations that best align with the downstream task as follows (<math id="A2.SS1.p1.1.m1.1" class="ltx_Math" alttext="Q_{s}" display="inline"><semantics id="A2.SS1.p1.1.m1.1a"><msub id="A2.SS1.p1.1.m1.1.1" xref="A2.SS1.p1.1.m1.1.1.cmml"><mi id="A2.SS1.p1.1.m1.1.1.2" xref="A2.SS1.p1.1.m1.1.1.2.cmml">Q</mi><mi id="A2.SS1.p1.1.m1.1.1.3" xref="A2.SS1.p1.1.m1.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="A2.SS1.p1.1.m1.1b"><apply id="A2.SS1.p1.1.m1.1.1.cmml" xref="A2.SS1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A2.SS1.p1.1.m1.1.1.1.cmml" xref="A2.SS1.p1.1.m1.1.1">subscript</csymbol><ci id="A2.SS1.p1.1.m1.1.1.2.cmml" xref="A2.SS1.p1.1.m1.1.1.2">𝑄</ci><ci id="A2.SS1.p1.1.m1.1.1.3.cmml" xref="A2.SS1.p1.1.m1.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p1.1.m1.1c">Q_{s}</annotation></semantics></math> and <math id="A2.SS1.p1.2.m2.1" class="ltx_Math" alttext="R_{s}" display="inline"><semantics id="A2.SS1.p1.2.m2.1a"><msub id="A2.SS1.p1.2.m2.1.1" xref="A2.SS1.p1.2.m2.1.1.cmml"><mi id="A2.SS1.p1.2.m2.1.1.2" xref="A2.SS1.p1.2.m2.1.1.2.cmml">R</mi><mi id="A2.SS1.p1.2.m2.1.1.3" xref="A2.SS1.p1.2.m2.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="A2.SS1.p1.2.m2.1b"><apply id="A2.SS1.p1.2.m2.1.1.cmml" xref="A2.SS1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="A2.SS1.p1.2.m2.1.1.1.cmml" xref="A2.SS1.p1.2.m2.1.1">subscript</csymbol><ci id="A2.SS1.p1.2.m2.1.1.2.cmml" xref="A2.SS1.p1.2.m2.1.1.2">𝑅</ci><ci id="A2.SS1.p1.2.m2.1.1.3.cmml" xref="A2.SS1.p1.2.m2.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p1.2.m2.1c">R_{s}</annotation></semantics></math> are formatted identically to the downstream task):</p>
</div>
<figure id="alg1" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg1.2.1.1" class="ltx_text ltx_font_bold">Algorithm 1</span> </span> MTV-EXTRACT for finding task vector locations</figcaption>
<div id="alg1.3" class="ltx_listing ltx_listing">
<div id="alg1.l1" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l1.1.1.1" class="ltx_text" style="font-size:80%;">1:</span></span><math id="alg1.l1.m1.1" class="ltx_Math" alttext="F" display="inline"><semantics id="alg1.l1.m1.1a"><mi id="alg1.l1.m1.1.1" xref="alg1.l1.m1.1.1.cmml">F</mi><annotation-xml encoding="MathML-Content" id="alg1.l1.m1.1b"><ci id="alg1.l1.m1.1.1.cmml" xref="alg1.l1.m1.1.1">𝐹</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m1.1c">F</annotation></semantics></math> (LMM), <math id="alg1.l1.m2.1" class="ltx_Math" alttext="S" display="inline"><semantics id="alg1.l1.m2.1a"><mi id="alg1.l1.m2.1.1" xref="alg1.l1.m2.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="alg1.l1.m2.1b"><ci id="alg1.l1.m2.1.1.cmml" xref="alg1.l1.m2.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m2.1c">S</annotation></semantics></math> (examples), <math id="alg1.l1.m3.1" class="ltx_Math" alttext="\mu_{j}" display="inline"><semantics id="alg1.l1.m3.1a"><msub id="alg1.l1.m3.1.1" xref="alg1.l1.m3.1.1.cmml"><mi id="alg1.l1.m3.1.1.2" xref="alg1.l1.m3.1.1.2.cmml">μ</mi><mi id="alg1.l1.m3.1.1.3" xref="alg1.l1.m3.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="alg1.l1.m3.1b"><apply id="alg1.l1.m3.1.1.cmml" xref="alg1.l1.m3.1.1"><csymbol cd="ambiguous" id="alg1.l1.m3.1.1.1.cmml" xref="alg1.l1.m3.1.1">subscript</csymbol><ci id="alg1.l1.m3.1.1.2.cmml" xref="alg1.l1.m3.1.1.2">𝜇</ci><ci id="alg1.l1.m3.1.1.3.cmml" xref="alg1.l1.m3.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m3.1c">\mu_{j}</annotation></semantics></math> (mean activations), <math id="alg1.l1.m4.2" class="ltx_Math" alttext="Q_{s},R_{s}" display="inline"><semantics id="alg1.l1.m4.2a"><mrow id="alg1.l1.m4.2.2.2" xref="alg1.l1.m4.2.2.3.cmml"><msub id="alg1.l1.m4.1.1.1.1" xref="alg1.l1.m4.1.1.1.1.cmml"><mi id="alg1.l1.m4.1.1.1.1.2" xref="alg1.l1.m4.1.1.1.1.2.cmml">Q</mi><mi id="alg1.l1.m4.1.1.1.1.3" xref="alg1.l1.m4.1.1.1.1.3.cmml">s</mi></msub><mo id="alg1.l1.m4.2.2.2.3" xref="alg1.l1.m4.2.2.3.cmml">,</mo><msub id="alg1.l1.m4.2.2.2.2" xref="alg1.l1.m4.2.2.2.2.cmml"><mi id="alg1.l1.m4.2.2.2.2.2" xref="alg1.l1.m4.2.2.2.2.2.cmml">R</mi><mi id="alg1.l1.m4.2.2.2.2.3" xref="alg1.l1.m4.2.2.2.2.3.cmml">s</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="alg1.l1.m4.2b"><list id="alg1.l1.m4.2.2.3.cmml" xref="alg1.l1.m4.2.2.2"><apply id="alg1.l1.m4.1.1.1.1.cmml" xref="alg1.l1.m4.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l1.m4.1.1.1.1.1.cmml" xref="alg1.l1.m4.1.1.1.1">subscript</csymbol><ci id="alg1.l1.m4.1.1.1.1.2.cmml" xref="alg1.l1.m4.1.1.1.1.2">𝑄</ci><ci id="alg1.l1.m4.1.1.1.1.3.cmml" xref="alg1.l1.m4.1.1.1.1.3">𝑠</ci></apply><apply id="alg1.l1.m4.2.2.2.2.cmml" xref="alg1.l1.m4.2.2.2.2"><csymbol cd="ambiguous" id="alg1.l1.m4.2.2.2.2.1.cmml" xref="alg1.l1.m4.2.2.2.2">subscript</csymbol><ci id="alg1.l1.m4.2.2.2.2.2.cmml" xref="alg1.l1.m4.2.2.2.2.2">𝑅</ci><ci id="alg1.l1.m4.2.2.2.2.3.cmml" xref="alg1.l1.m4.2.2.2.2.3">𝑠</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m4.2c">Q_{s},R_{s}</annotation></semantics></math> (queries and responses)

</div>
<div id="alg1.l2" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l2.1.1.1" class="ltx_text" style="font-size:80%;">2:</span></span><math id="alg1.l2.m1.1" class="ltx_Math" alttext="\lambda^{\mathrm{MTV}}_{j}" display="inline"><semantics id="alg1.l2.m1.1a"><msubsup id="alg1.l2.m1.1.1" xref="alg1.l2.m1.1.1.cmml"><mi id="alg1.l2.m1.1.1.2.2" xref="alg1.l2.m1.1.1.2.2.cmml">λ</mi><mi id="alg1.l2.m1.1.1.3" xref="alg1.l2.m1.1.1.3.cmml">j</mi><mi id="alg1.l2.m1.1.1.2.3" xref="alg1.l2.m1.1.1.2.3.cmml">MTV</mi></msubsup><annotation-xml encoding="MathML-Content" id="alg1.l2.m1.1b"><apply id="alg1.l2.m1.1.1.cmml" xref="alg1.l2.m1.1.1"><csymbol cd="ambiguous" id="alg1.l2.m1.1.1.1.cmml" xref="alg1.l2.m1.1.1">subscript</csymbol><apply id="alg1.l2.m1.1.1.2.cmml" xref="alg1.l2.m1.1.1"><csymbol cd="ambiguous" id="alg1.l2.m1.1.1.2.1.cmml" xref="alg1.l2.m1.1.1">superscript</csymbol><ci id="alg1.l2.m1.1.1.2.2.cmml" xref="alg1.l2.m1.1.1.2.2">𝜆</ci><ci id="alg1.l2.m1.1.1.2.3.cmml" xref="alg1.l2.m1.1.1.2.3">MTV</ci></apply><ci id="alg1.l2.m1.1.1.3.cmml" xref="alg1.l2.m1.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m1.1c">\lambda^{\mathrm{MTV}}_{j}</annotation></semantics></math> (optimized attention head locations)

</div>
<div id="alg1.l3" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l3.1.1.1" class="ltx_text" style="font-size:80%;">3:</span></span>Initialize <math id="alg1.l3.m1.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="alg1.l3.m1.1a"><mi id="alg1.l3.m1.1.1" xref="alg1.l3.m1.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="alg1.l3.m1.1b"><ci id="alg1.l3.m1.1.1.cmml" xref="alg1.l3.m1.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l3.m1.1c">\theta</annotation></semantics></math> randomly

</div>
<div id="alg1.l4" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l4.1.1.1" class="ltx_text" style="font-size:80%;">4:</span></span><span id="alg1.l4.2" class="ltx_text ltx_font_bold">for</span> <math id="alg1.l4.m1.1" class="ltx_Math" alttext="s\leftarrow 1" display="inline"><semantics id="alg1.l4.m1.1a"><mrow id="alg1.l4.m1.1.1" xref="alg1.l4.m1.1.1.cmml"><mi id="alg1.l4.m1.1.1.2" xref="alg1.l4.m1.1.1.2.cmml">s</mi><mo stretchy="false" id="alg1.l4.m1.1.1.1" xref="alg1.l4.m1.1.1.1.cmml">←</mo><mn id="alg1.l4.m1.1.1.3" xref="alg1.l4.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.l4.m1.1b"><apply id="alg1.l4.m1.1.1.cmml" xref="alg1.l4.m1.1.1"><ci id="alg1.l4.m1.1.1.1.cmml" xref="alg1.l4.m1.1.1.1">←</ci><ci id="alg1.l4.m1.1.1.2.cmml" xref="alg1.l4.m1.1.1.2">𝑠</ci><cn type="integer" id="alg1.l4.m1.1.1.3.cmml" xref="alg1.l4.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.m1.1c">s\leftarrow 1</annotation></semantics></math> to <math id="alg1.l4.m2.1" class="ltx_Math" alttext="S" display="inline"><semantics id="alg1.l4.m2.1a"><mi id="alg1.l4.m2.1.1" xref="alg1.l4.m2.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="alg1.l4.m2.1b"><ci id="alg1.l4.m2.1.1.cmml" xref="alg1.l4.m2.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.m2.1c">S</annotation></semantics></math> <span id="alg1.l4.3" class="ltx_text ltx_font_bold">do</span>

</div>
<div id="alg1.l5" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l5.2.1.1" class="ltx_text" style="font-size:80%;">5:</span></span>     <span id="alg1.l5.3" class="ltx_text ltx_font_bold">for</span> <math id="alg1.l5.m1.1" class="ltx_Math" alttext="i\leftarrow 1" display="inline"><semantics id="alg1.l5.m1.1a"><mrow id="alg1.l5.m1.1.1" xref="alg1.l5.m1.1.1.cmml"><mi id="alg1.l5.m1.1.1.2" xref="alg1.l5.m1.1.1.2.cmml">i</mi><mo stretchy="false" id="alg1.l5.m1.1.1.1" xref="alg1.l5.m1.1.1.1.cmml">←</mo><mn id="alg1.l5.m1.1.1.3" xref="alg1.l5.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.l5.m1.1b"><apply id="alg1.l5.m1.1.1.cmml" xref="alg1.l5.m1.1.1"><ci id="alg1.l5.m1.1.1.1.cmml" xref="alg1.l5.m1.1.1.1">←</ci><ci id="alg1.l5.m1.1.1.2.cmml" xref="alg1.l5.m1.1.1.2">𝑖</ci><cn type="integer" id="alg1.l5.m1.1.1.3.cmml" xref="alg1.l5.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m1.1c">i\leftarrow 1</annotation></semantics></math> to 32 <span id="alg1.l5.4" class="ltx_text ltx_font_bold">do</span> <span id="alg1.l5.1" class="ltx_text" style="float:right;"><math id="alg1.l5.1.m1.1" class="ltx_Math" alttext="\triangleright" display="inline"><semantics id="alg1.l5.1.m1.1a"><mo id="alg1.l5.1.m1.1.1" xref="alg1.l5.1.m1.1.1.cmml">▷</mo><annotation-xml encoding="MathML-Content" id="alg1.l5.1.m1.1b"><ci id="alg1.l5.1.m1.1.1.cmml" xref="alg1.l5.1.m1.1.1">▷</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.1.m1.1c">\triangleright</annotation></semantics></math> Sampling heads 32 times
</span>
</div>
<div id="alg1.l6" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l6.1.1.1" class="ltx_text" style="font-size:80%;">6:</span></span>         Sample <math id="alg1.l6.m1.2" class="ltx_Math" alttext="\lambda_{i}\sim\text{Bernoulli}(\sigma(\theta))" display="inline"><semantics id="alg1.l6.m1.2a"><mrow id="alg1.l6.m1.2.2" xref="alg1.l6.m1.2.2.cmml"><msub id="alg1.l6.m1.2.2.3" xref="alg1.l6.m1.2.2.3.cmml"><mi id="alg1.l6.m1.2.2.3.2" xref="alg1.l6.m1.2.2.3.2.cmml">λ</mi><mi id="alg1.l6.m1.2.2.3.3" xref="alg1.l6.m1.2.2.3.3.cmml">i</mi></msub><mo id="alg1.l6.m1.2.2.2" xref="alg1.l6.m1.2.2.2.cmml">∼</mo><mrow id="alg1.l6.m1.2.2.1" xref="alg1.l6.m1.2.2.1.cmml"><mtext id="alg1.l6.m1.2.2.1.3" xref="alg1.l6.m1.2.2.1.3a.cmml">Bernoulli</mtext><mo lspace="0em" rspace="0em" id="alg1.l6.m1.2.2.1.2" xref="alg1.l6.m1.2.2.1.2.cmml">​</mo><mrow id="alg1.l6.m1.2.2.1.1.1" xref="alg1.l6.m1.2.2.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l6.m1.2.2.1.1.1.2" xref="alg1.l6.m1.2.2.1.1.1.1.cmml">(</mo><mrow id="alg1.l6.m1.2.2.1.1.1.1" xref="alg1.l6.m1.2.2.1.1.1.1.cmml"><mi id="alg1.l6.m1.2.2.1.1.1.1.2" xref="alg1.l6.m1.2.2.1.1.1.1.2.cmml">σ</mi><mo lspace="0em" rspace="0em" id="alg1.l6.m1.2.2.1.1.1.1.1" xref="alg1.l6.m1.2.2.1.1.1.1.1.cmml">​</mo><mrow id="alg1.l6.m1.2.2.1.1.1.1.3.2" xref="alg1.l6.m1.2.2.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l6.m1.2.2.1.1.1.1.3.2.1" xref="alg1.l6.m1.2.2.1.1.1.1.cmml">(</mo><mi id="alg1.l6.m1.1.1" xref="alg1.l6.m1.1.1.cmml">θ</mi><mo stretchy="false" id="alg1.l6.m1.2.2.1.1.1.1.3.2.2" xref="alg1.l6.m1.2.2.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="alg1.l6.m1.2.2.1.1.1.3" xref="alg1.l6.m1.2.2.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l6.m1.2b"><apply id="alg1.l6.m1.2.2.cmml" xref="alg1.l6.m1.2.2"><csymbol cd="latexml" id="alg1.l6.m1.2.2.2.cmml" xref="alg1.l6.m1.2.2.2">similar-to</csymbol><apply id="alg1.l6.m1.2.2.3.cmml" xref="alg1.l6.m1.2.2.3"><csymbol cd="ambiguous" id="alg1.l6.m1.2.2.3.1.cmml" xref="alg1.l6.m1.2.2.3">subscript</csymbol><ci id="alg1.l6.m1.2.2.3.2.cmml" xref="alg1.l6.m1.2.2.3.2">𝜆</ci><ci id="alg1.l6.m1.2.2.3.3.cmml" xref="alg1.l6.m1.2.2.3.3">𝑖</ci></apply><apply id="alg1.l6.m1.2.2.1.cmml" xref="alg1.l6.m1.2.2.1"><times id="alg1.l6.m1.2.2.1.2.cmml" xref="alg1.l6.m1.2.2.1.2"></times><ci id="alg1.l6.m1.2.2.1.3a.cmml" xref="alg1.l6.m1.2.2.1.3"><mtext id="alg1.l6.m1.2.2.1.3.cmml" xref="alg1.l6.m1.2.2.1.3">Bernoulli</mtext></ci><apply id="alg1.l6.m1.2.2.1.1.1.1.cmml" xref="alg1.l6.m1.2.2.1.1.1"><times id="alg1.l6.m1.2.2.1.1.1.1.1.cmml" xref="alg1.l6.m1.2.2.1.1.1.1.1"></times><ci id="alg1.l6.m1.2.2.1.1.1.1.2.cmml" xref="alg1.l6.m1.2.2.1.1.1.1.2">𝜎</ci><ci id="alg1.l6.m1.1.1.cmml" xref="alg1.l6.m1.1.1">𝜃</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l6.m1.2c">\lambda_{i}\sim\text{Bernoulli}(\sigma(\theta))</annotation></semantics></math>

</div>
<div id="alg1.l7" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l7.1.1.1" class="ltx_text" style="font-size:80%;">7:</span></span>         Replace activations for <math id="alg1.l7.m1.1" class="ltx_Math" alttext="\lambda_{i}" display="inline"><semantics id="alg1.l7.m1.1a"><msub id="alg1.l7.m1.1.1" xref="alg1.l7.m1.1.1.cmml"><mi id="alg1.l7.m1.1.1.2" xref="alg1.l7.m1.1.1.2.cmml">λ</mi><mi id="alg1.l7.m1.1.1.3" xref="alg1.l7.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="alg1.l7.m1.1b"><apply id="alg1.l7.m1.1.1.cmml" xref="alg1.l7.m1.1.1"><csymbol cd="ambiguous" id="alg1.l7.m1.1.1.1.cmml" xref="alg1.l7.m1.1.1">subscript</csymbol><ci id="alg1.l7.m1.1.1.2.cmml" xref="alg1.l7.m1.1.1.2">𝜆</ci><ci id="alg1.l7.m1.1.1.3.cmml" xref="alg1.l7.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l7.m1.1c">\lambda_{i}</annotation></semantics></math> in <math id="alg1.l7.m2.1" class="ltx_Math" alttext="F" display="inline"><semantics id="alg1.l7.m2.1a"><mi id="alg1.l7.m2.1.1" xref="alg1.l7.m2.1.1.cmml">F</mi><annotation-xml encoding="MathML-Content" id="alg1.l7.m2.1b"><ci id="alg1.l7.m2.1.1.cmml" xref="alg1.l7.m2.1.1">𝐹</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l7.m2.1c">F</annotation></semantics></math> with <math id="alg1.l7.m3.2" class="ltx_Math" alttext="\mu_{l,j}" display="inline"><semantics id="alg1.l7.m3.2a"><msub id="alg1.l7.m3.2.3" xref="alg1.l7.m3.2.3.cmml"><mi id="alg1.l7.m3.2.3.2" xref="alg1.l7.m3.2.3.2.cmml">μ</mi><mrow id="alg1.l7.m3.2.2.2.4" xref="alg1.l7.m3.2.2.2.3.cmml"><mi id="alg1.l7.m3.1.1.1.1" xref="alg1.l7.m3.1.1.1.1.cmml">l</mi><mo id="alg1.l7.m3.2.2.2.4.1" xref="alg1.l7.m3.2.2.2.3.cmml">,</mo><mi id="alg1.l7.m3.2.2.2.2" xref="alg1.l7.m3.2.2.2.2.cmml">j</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="alg1.l7.m3.2b"><apply id="alg1.l7.m3.2.3.cmml" xref="alg1.l7.m3.2.3"><csymbol cd="ambiguous" id="alg1.l7.m3.2.3.1.cmml" xref="alg1.l7.m3.2.3">subscript</csymbol><ci id="alg1.l7.m3.2.3.2.cmml" xref="alg1.l7.m3.2.3.2">𝜇</ci><list id="alg1.l7.m3.2.2.2.3.cmml" xref="alg1.l7.m3.2.2.2.4"><ci id="alg1.l7.m3.1.1.1.1.cmml" xref="alg1.l7.m3.1.1.1.1">𝑙</ci><ci id="alg1.l7.m3.2.2.2.2.cmml" xref="alg1.l7.m3.2.2.2.2">𝑗</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l7.m3.2c">\mu_{l,j}</annotation></semantics></math>

</div>
<div id="alg1.l8" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l8.4.1.1" class="ltx_text" style="font-size:80%;">8:</span></span>         Compute output logits <math id="alg1.l8.m1.1" class="ltx_Math" alttext="O_{s}\leftarrow F(Q_{s})" display="inline"><semantics id="alg1.l8.m1.1a"><mrow id="alg1.l8.m1.1.1" xref="alg1.l8.m1.1.1.cmml"><msub id="alg1.l8.m1.1.1.3" xref="alg1.l8.m1.1.1.3.cmml"><mi id="alg1.l8.m1.1.1.3.2" xref="alg1.l8.m1.1.1.3.2.cmml">O</mi><mi id="alg1.l8.m1.1.1.3.3" xref="alg1.l8.m1.1.1.3.3.cmml">s</mi></msub><mo stretchy="false" id="alg1.l8.m1.1.1.2" xref="alg1.l8.m1.1.1.2.cmml">←</mo><mrow id="alg1.l8.m1.1.1.1" xref="alg1.l8.m1.1.1.1.cmml"><mi id="alg1.l8.m1.1.1.1.3" xref="alg1.l8.m1.1.1.1.3.cmml">F</mi><mo lspace="0em" rspace="0em" id="alg1.l8.m1.1.1.1.2" xref="alg1.l8.m1.1.1.1.2.cmml">​</mo><mrow id="alg1.l8.m1.1.1.1.1.1" xref="alg1.l8.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l8.m1.1.1.1.1.1.2" xref="alg1.l8.m1.1.1.1.1.1.1.cmml">(</mo><msub id="alg1.l8.m1.1.1.1.1.1.1" xref="alg1.l8.m1.1.1.1.1.1.1.cmml"><mi id="alg1.l8.m1.1.1.1.1.1.1.2" xref="alg1.l8.m1.1.1.1.1.1.1.2.cmml">Q</mi><mi id="alg1.l8.m1.1.1.1.1.1.1.3" xref="alg1.l8.m1.1.1.1.1.1.1.3.cmml">s</mi></msub><mo stretchy="false" id="alg1.l8.m1.1.1.1.1.1.3" xref="alg1.l8.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l8.m1.1b"><apply id="alg1.l8.m1.1.1.cmml" xref="alg1.l8.m1.1.1"><ci id="alg1.l8.m1.1.1.2.cmml" xref="alg1.l8.m1.1.1.2">←</ci><apply id="alg1.l8.m1.1.1.3.cmml" xref="alg1.l8.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.l8.m1.1.1.3.1.cmml" xref="alg1.l8.m1.1.1.3">subscript</csymbol><ci id="alg1.l8.m1.1.1.3.2.cmml" xref="alg1.l8.m1.1.1.3.2">𝑂</ci><ci id="alg1.l8.m1.1.1.3.3.cmml" xref="alg1.l8.m1.1.1.3.3">𝑠</ci></apply><apply id="alg1.l8.m1.1.1.1.cmml" xref="alg1.l8.m1.1.1.1"><times id="alg1.l8.m1.1.1.1.2.cmml" xref="alg1.l8.m1.1.1.1.2"></times><ci id="alg1.l8.m1.1.1.1.3.cmml" xref="alg1.l8.m1.1.1.1.3">𝐹</ci><apply id="alg1.l8.m1.1.1.1.1.1.1.cmml" xref="alg1.l8.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l8.m1.1.1.1.1.1.1.1.cmml" xref="alg1.l8.m1.1.1.1.1.1">subscript</csymbol><ci id="alg1.l8.m1.1.1.1.1.1.1.2.cmml" xref="alg1.l8.m1.1.1.1.1.1.1.2">𝑄</ci><ci id="alg1.l8.m1.1.1.1.1.1.1.3.cmml" xref="alg1.l8.m1.1.1.1.1.1.1.3">𝑠</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l8.m1.1c">O_{s}\leftarrow F(Q_{s})</annotation></semantics></math> <span id="alg1.l8.3" class="ltx_text" style="float:right;"><math id="alg1.l8.1.m1.1" class="ltx_Math" alttext="\triangleright" display="inline"><semantics id="alg1.l8.1.m1.1a"><mo id="alg1.l8.1.m1.1.1" xref="alg1.l8.1.m1.1.1.cmml">▷</mo><annotation-xml encoding="MathML-Content" id="alg1.l8.1.m1.1b"><ci id="alg1.l8.1.m1.1.1.cmml" xref="alg1.l8.1.m1.1.1">▷</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l8.1.m1.1c">\triangleright</annotation></semantics></math> Pass <math id="alg1.l8.2.m2.1" class="ltx_Math" alttext="Q_{s}" display="inline"><semantics id="alg1.l8.2.m2.1a"><msub id="alg1.l8.2.m2.1.1" xref="alg1.l8.2.m2.1.1.cmml"><mi id="alg1.l8.2.m2.1.1.2" xref="alg1.l8.2.m2.1.1.2.cmml">Q</mi><mi id="alg1.l8.2.m2.1.1.3" xref="alg1.l8.2.m2.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="alg1.l8.2.m2.1b"><apply id="alg1.l8.2.m2.1.1.cmml" xref="alg1.l8.2.m2.1.1"><csymbol cd="ambiguous" id="alg1.l8.2.m2.1.1.1.cmml" xref="alg1.l8.2.m2.1.1">subscript</csymbol><ci id="alg1.l8.2.m2.1.1.2.cmml" xref="alg1.l8.2.m2.1.1.2">𝑄</ci><ci id="alg1.l8.2.m2.1.1.3.cmml" xref="alg1.l8.2.m2.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l8.2.m2.1c">Q_{s}</annotation></semantics></math> to LMM <math id="alg1.l8.3.m3.1" class="ltx_Math" alttext="F" display="inline"><semantics id="alg1.l8.3.m3.1a"><mi id="alg1.l8.3.m3.1.1" xref="alg1.l8.3.m3.1.1.cmml">F</mi><annotation-xml encoding="MathML-Content" id="alg1.l8.3.m3.1b"><ci id="alg1.l8.3.m3.1.1.cmml" xref="alg1.l8.3.m3.1.1">𝐹</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l8.3.m3.1c">F</annotation></semantics></math>
</span>
</div>
<div id="alg1.l9" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l9.1.1.1" class="ltx_text" style="font-size:80%;">9:</span></span>         <math id="alg1.l9.m1.1" class="ltx_Math" alttext="L_{i}\leftarrow" display="inline"><semantics id="alg1.l9.m1.1a"><mrow id="alg1.l9.m1.1.1" xref="alg1.l9.m1.1.1.cmml"><msub id="alg1.l9.m1.1.1.2" xref="alg1.l9.m1.1.1.2.cmml"><mi id="alg1.l9.m1.1.1.2.2" xref="alg1.l9.m1.1.1.2.2.cmml">L</mi><mi id="alg1.l9.m1.1.1.2.3" xref="alg1.l9.m1.1.1.2.3.cmml">i</mi></msub><mo stretchy="false" id="alg1.l9.m1.1.1.1" xref="alg1.l9.m1.1.1.1.cmml">←</mo><mi id="alg1.l9.m1.1.1.3" xref="alg1.l9.m1.1.1.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.l9.m1.1b"><apply id="alg1.l9.m1.1.1.cmml" xref="alg1.l9.m1.1.1"><ci id="alg1.l9.m1.1.1.1.cmml" xref="alg1.l9.m1.1.1.1">←</ci><apply id="alg1.l9.m1.1.1.2.cmml" xref="alg1.l9.m1.1.1.2"><csymbol cd="ambiguous" id="alg1.l9.m1.1.1.2.1.cmml" xref="alg1.l9.m1.1.1.2">subscript</csymbol><ci id="alg1.l9.m1.1.1.2.2.cmml" xref="alg1.l9.m1.1.1.2.2">𝐿</ci><ci id="alg1.l9.m1.1.1.2.3.cmml" xref="alg1.l9.m1.1.1.2.3">𝑖</ci></apply><csymbol cd="latexml" id="alg1.l9.m1.1.1.3.cmml" xref="alg1.l9.m1.1.1.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l9.m1.1c">L_{i}\leftarrow</annotation></semantics></math> Negative Cross-Entropy(<math id="alg1.l9.m2.1" class="ltx_Math" alttext="O_{s}" display="inline"><semantics id="alg1.l9.m2.1a"><msub id="alg1.l9.m2.1.1" xref="alg1.l9.m2.1.1.cmml"><mi id="alg1.l9.m2.1.1.2" xref="alg1.l9.m2.1.1.2.cmml">O</mi><mi id="alg1.l9.m2.1.1.3" xref="alg1.l9.m2.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="alg1.l9.m2.1b"><apply id="alg1.l9.m2.1.1.cmml" xref="alg1.l9.m2.1.1"><csymbol cd="ambiguous" id="alg1.l9.m2.1.1.1.cmml" xref="alg1.l9.m2.1.1">subscript</csymbol><ci id="alg1.l9.m2.1.1.2.cmml" xref="alg1.l9.m2.1.1.2">𝑂</ci><ci id="alg1.l9.m2.1.1.3.cmml" xref="alg1.l9.m2.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l9.m2.1c">O_{s}</annotation></semantics></math>, <math id="alg1.l9.m3.1" class="ltx_Math" alttext="R_{s}" display="inline"><semantics id="alg1.l9.m3.1a"><msub id="alg1.l9.m3.1.1" xref="alg1.l9.m3.1.1.cmml"><mi id="alg1.l9.m3.1.1.2" xref="alg1.l9.m3.1.1.2.cmml">R</mi><mi id="alg1.l9.m3.1.1.3" xref="alg1.l9.m3.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="alg1.l9.m3.1b"><apply id="alg1.l9.m3.1.1.cmml" xref="alg1.l9.m3.1.1"><csymbol cd="ambiguous" id="alg1.l9.m3.1.1.1.cmml" xref="alg1.l9.m3.1.1">subscript</csymbol><ci id="alg1.l9.m3.1.1.2.cmml" xref="alg1.l9.m3.1.1.2">𝑅</ci><ci id="alg1.l9.m3.1.1.3.cmml" xref="alg1.l9.m3.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l9.m3.1c">R_{s}</annotation></semantics></math>)

</div>
<div id="alg1.l10" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l10.1.1.1" class="ltx_text" style="font-size:80%;">10:</span></span>     <span id="alg1.l10.2" class="ltx_text ltx_font_bold">end</span> <span id="alg1.l10.3" class="ltx_text ltx_font_bold">for</span>
</div>
<div id="alg1.l11" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l11.2.1.1" class="ltx_text" style="font-size:80%;">11:</span></span>     <math id="alg1.l11.m1.2" class="ltx_Math" alttext="\theta\leftarrow\text{Adam}(\theta,\nabla_{\theta}\frac{1}{32}\sum_{i=1}^{32}L_{i})" display="inline"><semantics id="alg1.l11.m1.2a"><mrow id="alg1.l11.m1.2.2" xref="alg1.l11.m1.2.2.cmml"><mi id="alg1.l11.m1.2.2.3" xref="alg1.l11.m1.2.2.3.cmml">θ</mi><mo stretchy="false" id="alg1.l11.m1.2.2.2" xref="alg1.l11.m1.2.2.2.cmml">←</mo><mrow id="alg1.l11.m1.2.2.1" xref="alg1.l11.m1.2.2.1.cmml"><mtext id="alg1.l11.m1.2.2.1.3" xref="alg1.l11.m1.2.2.1.3a.cmml">Adam</mtext><mo lspace="0em" rspace="0em" id="alg1.l11.m1.2.2.1.2" xref="alg1.l11.m1.2.2.1.2.cmml">​</mo><mrow id="alg1.l11.m1.2.2.1.1.1" xref="alg1.l11.m1.2.2.1.1.2.cmml"><mo stretchy="false" id="alg1.l11.m1.2.2.1.1.1.2" xref="alg1.l11.m1.2.2.1.1.2.cmml">(</mo><mi id="alg1.l11.m1.1.1" xref="alg1.l11.m1.1.1.cmml">θ</mi><mo id="alg1.l11.m1.2.2.1.1.1.3" xref="alg1.l11.m1.2.2.1.1.2.cmml">,</mo><mrow id="alg1.l11.m1.2.2.1.1.1.1" xref="alg1.l11.m1.2.2.1.1.1.1.cmml"><mrow id="alg1.l11.m1.2.2.1.1.1.1.2" xref="alg1.l11.m1.2.2.1.1.1.1.2.cmml"><msub id="alg1.l11.m1.2.2.1.1.1.1.2.1" xref="alg1.l11.m1.2.2.1.1.1.1.2.1.cmml"><mo rspace="0.167em" id="alg1.l11.m1.2.2.1.1.1.1.2.1.2" xref="alg1.l11.m1.2.2.1.1.1.1.2.1.2.cmml">∇</mo><mi id="alg1.l11.m1.2.2.1.1.1.1.2.1.3" xref="alg1.l11.m1.2.2.1.1.1.1.2.1.3.cmml">θ</mi></msub><mfrac id="alg1.l11.m1.2.2.1.1.1.1.2.2" xref="alg1.l11.m1.2.2.1.1.1.1.2.2.cmml"><mn id="alg1.l11.m1.2.2.1.1.1.1.2.2.2" xref="alg1.l11.m1.2.2.1.1.1.1.2.2.2.cmml">1</mn><mn id="alg1.l11.m1.2.2.1.1.1.1.2.2.3" xref="alg1.l11.m1.2.2.1.1.1.1.2.2.3.cmml">32</mn></mfrac></mrow><mo lspace="0em" rspace="0em" id="alg1.l11.m1.2.2.1.1.1.1.1" xref="alg1.l11.m1.2.2.1.1.1.1.1.cmml">​</mo><mrow id="alg1.l11.m1.2.2.1.1.1.1.3" xref="alg1.l11.m1.2.2.1.1.1.1.3.cmml"><msubsup id="alg1.l11.m1.2.2.1.1.1.1.3.1" xref="alg1.l11.m1.2.2.1.1.1.1.3.1.cmml"><mo id="alg1.l11.m1.2.2.1.1.1.1.3.1.2.2" xref="alg1.l11.m1.2.2.1.1.1.1.3.1.2.2.cmml">∑</mo><mrow id="alg1.l11.m1.2.2.1.1.1.1.3.1.2.3" xref="alg1.l11.m1.2.2.1.1.1.1.3.1.2.3.cmml"><mi id="alg1.l11.m1.2.2.1.1.1.1.3.1.2.3.2" xref="alg1.l11.m1.2.2.1.1.1.1.3.1.2.3.2.cmml">i</mi><mo id="alg1.l11.m1.2.2.1.1.1.1.3.1.2.3.1" xref="alg1.l11.m1.2.2.1.1.1.1.3.1.2.3.1.cmml">=</mo><mn id="alg1.l11.m1.2.2.1.1.1.1.3.1.2.3.3" xref="alg1.l11.m1.2.2.1.1.1.1.3.1.2.3.3.cmml">1</mn></mrow><mn id="alg1.l11.m1.2.2.1.1.1.1.3.1.3" xref="alg1.l11.m1.2.2.1.1.1.1.3.1.3.cmml">32</mn></msubsup><msub id="alg1.l11.m1.2.2.1.1.1.1.3.2" xref="alg1.l11.m1.2.2.1.1.1.1.3.2.cmml"><mi id="alg1.l11.m1.2.2.1.1.1.1.3.2.2" xref="alg1.l11.m1.2.2.1.1.1.1.3.2.2.cmml">L</mi><mi id="alg1.l11.m1.2.2.1.1.1.1.3.2.3" xref="alg1.l11.m1.2.2.1.1.1.1.3.2.3.cmml">i</mi></msub></mrow></mrow><mo stretchy="false" id="alg1.l11.m1.2.2.1.1.1.4" xref="alg1.l11.m1.2.2.1.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l11.m1.2b"><apply id="alg1.l11.m1.2.2.cmml" xref="alg1.l11.m1.2.2"><ci id="alg1.l11.m1.2.2.2.cmml" xref="alg1.l11.m1.2.2.2">←</ci><ci id="alg1.l11.m1.2.2.3.cmml" xref="alg1.l11.m1.2.2.3">𝜃</ci><apply id="alg1.l11.m1.2.2.1.cmml" xref="alg1.l11.m1.2.2.1"><times id="alg1.l11.m1.2.2.1.2.cmml" xref="alg1.l11.m1.2.2.1.2"></times><ci id="alg1.l11.m1.2.2.1.3a.cmml" xref="alg1.l11.m1.2.2.1.3"><mtext id="alg1.l11.m1.2.2.1.3.cmml" xref="alg1.l11.m1.2.2.1.3">Adam</mtext></ci><interval closure="open" id="alg1.l11.m1.2.2.1.1.2.cmml" xref="alg1.l11.m1.2.2.1.1.1"><ci id="alg1.l11.m1.1.1.cmml" xref="alg1.l11.m1.1.1">𝜃</ci><apply id="alg1.l11.m1.2.2.1.1.1.1.cmml" xref="alg1.l11.m1.2.2.1.1.1.1"><times id="alg1.l11.m1.2.2.1.1.1.1.1.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.1"></times><apply id="alg1.l11.m1.2.2.1.1.1.1.2.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.2"><apply id="alg1.l11.m1.2.2.1.1.1.1.2.1.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.2.1"><csymbol cd="ambiguous" id="alg1.l11.m1.2.2.1.1.1.1.2.1.1.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.2.1">subscript</csymbol><ci id="alg1.l11.m1.2.2.1.1.1.1.2.1.2.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.2.1.2">∇</ci><ci id="alg1.l11.m1.2.2.1.1.1.1.2.1.3.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.2.1.3">𝜃</ci></apply><apply id="alg1.l11.m1.2.2.1.1.1.1.2.2.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.2.2"><divide id="alg1.l11.m1.2.2.1.1.1.1.2.2.1.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.2.2"></divide><cn type="integer" id="alg1.l11.m1.2.2.1.1.1.1.2.2.2.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.2.2.2">1</cn><cn type="integer" id="alg1.l11.m1.2.2.1.1.1.1.2.2.3.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.2.2.3">32</cn></apply></apply><apply id="alg1.l11.m1.2.2.1.1.1.1.3.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.3"><apply id="alg1.l11.m1.2.2.1.1.1.1.3.1.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.3.1"><csymbol cd="ambiguous" id="alg1.l11.m1.2.2.1.1.1.1.3.1.1.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.3.1">superscript</csymbol><apply id="alg1.l11.m1.2.2.1.1.1.1.3.1.2.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.3.1"><csymbol cd="ambiguous" id="alg1.l11.m1.2.2.1.1.1.1.3.1.2.1.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.3.1">subscript</csymbol><sum id="alg1.l11.m1.2.2.1.1.1.1.3.1.2.2.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.3.1.2.2"></sum><apply id="alg1.l11.m1.2.2.1.1.1.1.3.1.2.3.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.3.1.2.3"><eq id="alg1.l11.m1.2.2.1.1.1.1.3.1.2.3.1.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.3.1.2.3.1"></eq><ci id="alg1.l11.m1.2.2.1.1.1.1.3.1.2.3.2.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.3.1.2.3.2">𝑖</ci><cn type="integer" id="alg1.l11.m1.2.2.1.1.1.1.3.1.2.3.3.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.3.1.2.3.3">1</cn></apply></apply><cn type="integer" id="alg1.l11.m1.2.2.1.1.1.1.3.1.3.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.3.1.3">32</cn></apply><apply id="alg1.l11.m1.2.2.1.1.1.1.3.2.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.3.2"><csymbol cd="ambiguous" id="alg1.l11.m1.2.2.1.1.1.1.3.2.1.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.3.2">subscript</csymbol><ci id="alg1.l11.m1.2.2.1.1.1.1.3.2.2.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.3.2.2">𝐿</ci><ci id="alg1.l11.m1.2.2.1.1.1.1.3.2.3.cmml" xref="alg1.l11.m1.2.2.1.1.1.1.3.2.3">𝑖</ci></apply></apply></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l11.m1.2c">\theta\leftarrow\text{Adam}(\theta,\nabla_{\theta}\frac{1}{32}\sum_{i=1}^{32}L_{i})</annotation></semantics></math> <span id="alg1.l11.1" class="ltx_text" style="float:right;"><math id="alg1.l11.1.m1.1" class="ltx_Math" alttext="\triangleright" display="inline"><semantics id="alg1.l11.1.m1.1a"><mo id="alg1.l11.1.m1.1.1" xref="alg1.l11.1.m1.1.1.cmml">▷</mo><annotation-xml encoding="MathML-Content" id="alg1.l11.1.m1.1b"><ci id="alg1.l11.1.m1.1.1.cmml" xref="alg1.l11.1.m1.1.1">▷</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l11.1.m1.1c">\triangleright</annotation></semantics></math> Update rule
</span>
</div>
<div id="alg1.l12" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l12.1.1.1" class="ltx_text" style="font-size:80%;">12:</span></span><span id="alg1.l12.2" class="ltx_text ltx_font_bold">end</span> <span id="alg1.l12.3" class="ltx_text ltx_font_bold">for</span>
</div>
<div id="alg1.l13" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l13.2.1.1" class="ltx_text" style="font-size:80%;">13:</span></span>Sample final <math id="alg1.l13.m1.2" class="ltx_Math" alttext="\lambda^{\mathrm{MTV}}_{j}\sim\text{Bernoulli}(\sigma(\theta))" display="inline"><semantics id="alg1.l13.m1.2a"><mrow id="alg1.l13.m1.2.2" xref="alg1.l13.m1.2.2.cmml"><msubsup id="alg1.l13.m1.2.2.3" xref="alg1.l13.m1.2.2.3.cmml"><mi id="alg1.l13.m1.2.2.3.2.2" xref="alg1.l13.m1.2.2.3.2.2.cmml">λ</mi><mi id="alg1.l13.m1.2.2.3.3" xref="alg1.l13.m1.2.2.3.3.cmml">j</mi><mi id="alg1.l13.m1.2.2.3.2.3" xref="alg1.l13.m1.2.2.3.2.3.cmml">MTV</mi></msubsup><mo id="alg1.l13.m1.2.2.2" xref="alg1.l13.m1.2.2.2.cmml">∼</mo><mrow id="alg1.l13.m1.2.2.1" xref="alg1.l13.m1.2.2.1.cmml"><mtext id="alg1.l13.m1.2.2.1.3" xref="alg1.l13.m1.2.2.1.3a.cmml">Bernoulli</mtext><mo lspace="0em" rspace="0em" id="alg1.l13.m1.2.2.1.2" xref="alg1.l13.m1.2.2.1.2.cmml">​</mo><mrow id="alg1.l13.m1.2.2.1.1.1" xref="alg1.l13.m1.2.2.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l13.m1.2.2.1.1.1.2" xref="alg1.l13.m1.2.2.1.1.1.1.cmml">(</mo><mrow id="alg1.l13.m1.2.2.1.1.1.1" xref="alg1.l13.m1.2.2.1.1.1.1.cmml"><mi id="alg1.l13.m1.2.2.1.1.1.1.2" xref="alg1.l13.m1.2.2.1.1.1.1.2.cmml">σ</mi><mo lspace="0em" rspace="0em" id="alg1.l13.m1.2.2.1.1.1.1.1" xref="alg1.l13.m1.2.2.1.1.1.1.1.cmml">​</mo><mrow id="alg1.l13.m1.2.2.1.1.1.1.3.2" xref="alg1.l13.m1.2.2.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l13.m1.2.2.1.1.1.1.3.2.1" xref="alg1.l13.m1.2.2.1.1.1.1.cmml">(</mo><mi id="alg1.l13.m1.1.1" xref="alg1.l13.m1.1.1.cmml">θ</mi><mo stretchy="false" id="alg1.l13.m1.2.2.1.1.1.1.3.2.2" xref="alg1.l13.m1.2.2.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="alg1.l13.m1.2.2.1.1.1.3" xref="alg1.l13.m1.2.2.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l13.m1.2b"><apply id="alg1.l13.m1.2.2.cmml" xref="alg1.l13.m1.2.2"><csymbol cd="latexml" id="alg1.l13.m1.2.2.2.cmml" xref="alg1.l13.m1.2.2.2">similar-to</csymbol><apply id="alg1.l13.m1.2.2.3.cmml" xref="alg1.l13.m1.2.2.3"><csymbol cd="ambiguous" id="alg1.l13.m1.2.2.3.1.cmml" xref="alg1.l13.m1.2.2.3">subscript</csymbol><apply id="alg1.l13.m1.2.2.3.2.cmml" xref="alg1.l13.m1.2.2.3"><csymbol cd="ambiguous" id="alg1.l13.m1.2.2.3.2.1.cmml" xref="alg1.l13.m1.2.2.3">superscript</csymbol><ci id="alg1.l13.m1.2.2.3.2.2.cmml" xref="alg1.l13.m1.2.2.3.2.2">𝜆</ci><ci id="alg1.l13.m1.2.2.3.2.3.cmml" xref="alg1.l13.m1.2.2.3.2.3">MTV</ci></apply><ci id="alg1.l13.m1.2.2.3.3.cmml" xref="alg1.l13.m1.2.2.3.3">𝑗</ci></apply><apply id="alg1.l13.m1.2.2.1.cmml" xref="alg1.l13.m1.2.2.1"><times id="alg1.l13.m1.2.2.1.2.cmml" xref="alg1.l13.m1.2.2.1.2"></times><ci id="alg1.l13.m1.2.2.1.3a.cmml" xref="alg1.l13.m1.2.2.1.3"><mtext id="alg1.l13.m1.2.2.1.3.cmml" xref="alg1.l13.m1.2.2.1.3">Bernoulli</mtext></ci><apply id="alg1.l13.m1.2.2.1.1.1.1.cmml" xref="alg1.l13.m1.2.2.1.1.1"><times id="alg1.l13.m1.2.2.1.1.1.1.1.cmml" xref="alg1.l13.m1.2.2.1.1.1.1.1"></times><ci id="alg1.l13.m1.2.2.1.1.1.1.2.cmml" xref="alg1.l13.m1.2.2.1.1.1.1.2">𝜎</ci><ci id="alg1.l13.m1.1.1.cmml" xref="alg1.l13.m1.1.1">𝜃</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l13.m1.2c">\lambda^{\mathrm{MTV}}_{j}\sim\text{Bernoulli}(\sigma(\theta))</annotation></semantics></math> <span id="alg1.l13.1" class="ltx_text" style="float:right;"><math id="alg1.l13.1.m1.1" class="ltx_Math" alttext="\triangleright" display="inline"><semantics id="alg1.l13.1.m1.1a"><mo id="alg1.l13.1.m1.1.1" xref="alg1.l13.1.m1.1.1.cmml">▷</mo><annotation-xml encoding="MathML-Content" id="alg1.l13.1.m1.1b"><ci id="alg1.l13.1.m1.1.1.cmml" xref="alg1.l13.1.m1.1.1">▷</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l13.1.m1.1c">\triangleright</annotation></semantics></math> Final set of head locations
</span>
</div>
<div id="alg1.l14" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline"><span id="alg1.l14.1.1.1" class="ltx_text" style="font-size:80%;">14:</span></span><span id="alg1.l14.2" class="ltx_text ltx_font_bold">return</span> <math id="alg1.l14.m1.1" class="ltx_Math" alttext="\lambda^{\mathrm{MTV}}_{j}" display="inline"><semantics id="alg1.l14.m1.1a"><msubsup id="alg1.l14.m1.1.1" xref="alg1.l14.m1.1.1.cmml"><mi id="alg1.l14.m1.1.1.2.2" xref="alg1.l14.m1.1.1.2.2.cmml">λ</mi><mi id="alg1.l14.m1.1.1.3" xref="alg1.l14.m1.1.1.3.cmml">j</mi><mi id="alg1.l14.m1.1.1.2.3" xref="alg1.l14.m1.1.1.2.3.cmml">MTV</mi></msubsup><annotation-xml encoding="MathML-Content" id="alg1.l14.m1.1b"><apply id="alg1.l14.m1.1.1.cmml" xref="alg1.l14.m1.1.1"><csymbol cd="ambiguous" id="alg1.l14.m1.1.1.1.cmml" xref="alg1.l14.m1.1.1">subscript</csymbol><apply id="alg1.l14.m1.1.1.2.cmml" xref="alg1.l14.m1.1.1"><csymbol cd="ambiguous" id="alg1.l14.m1.1.1.2.1.cmml" xref="alg1.l14.m1.1.1">superscript</csymbol><ci id="alg1.l14.m1.1.1.2.2.cmml" xref="alg1.l14.m1.1.1.2.2">𝜆</ci><ci id="alg1.l14.m1.1.1.2.3.cmml" xref="alg1.l14.m1.1.1.2.3">MTV</ci></apply><ci id="alg1.l14.m1.1.1.3.cmml" xref="alg1.l14.m1.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l14.m1.1c">\lambda^{\mathrm{MTV}}_{j}</annotation></semantics></math>

</div>
</div>
</figure>
<div id="A2.SS1.p2" class="ltx_para">
<p id="A2.SS1.p2.3" class="ltx_p">We point out a few important factors. It is important to note that none of the parameters of <math id="A2.SS1.p2.1.m1.1" class="ltx_Math" alttext="F" display="inline"><semantics id="A2.SS1.p2.1.m1.1a"><mi id="A2.SS1.p2.1.m1.1.1" xref="A2.SS1.p2.1.m1.1.1.cmml">F</mi><annotation-xml encoding="MathML-Content" id="A2.SS1.p2.1.m1.1b"><ci id="A2.SS1.p2.1.m1.1.1.cmml" xref="A2.SS1.p2.1.m1.1.1">𝐹</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p2.1.m1.1c">F</annotation></semantics></math> are being finetuned through any gradient update. We take the negative cross-entropy (negative as MTV_EXTRACT draws inspiration from REINFORCE <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib89" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">89</span></a>]</cite>, which is a policy optimization algorithm) between the output logits <math id="A2.SS1.p2.2.m2.1" class="ltx_Math" alttext="O_{s}" display="inline"><semantics id="A2.SS1.p2.2.m2.1a"><msub id="A2.SS1.p2.2.m2.1.1" xref="A2.SS1.p2.2.m2.1.1.cmml"><mi id="A2.SS1.p2.2.m2.1.1.2" xref="A2.SS1.p2.2.m2.1.1.2.cmml">O</mi><mi id="A2.SS1.p2.2.m2.1.1.3" xref="A2.SS1.p2.2.m2.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="A2.SS1.p2.2.m2.1b"><apply id="A2.SS1.p2.2.m2.1.1.cmml" xref="A2.SS1.p2.2.m2.1.1"><csymbol cd="ambiguous" id="A2.SS1.p2.2.m2.1.1.1.cmml" xref="A2.SS1.p2.2.m2.1.1">subscript</csymbol><ci id="A2.SS1.p2.2.m2.1.1.2.cmml" xref="A2.SS1.p2.2.m2.1.1.2">𝑂</ci><ci id="A2.SS1.p2.2.m2.1.1.3.cmml" xref="A2.SS1.p2.2.m2.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p2.2.m2.1c">O_{s}</annotation></semantics></math> and the first token of the target response <math id="A2.SS1.p2.3.m3.1" class="ltx_Math" alttext="R_{s}" display="inline"><semantics id="A2.SS1.p2.3.m3.1a"><msub id="A2.SS1.p2.3.m3.1.1" xref="A2.SS1.p2.3.m3.1.1.cmml"><mi id="A2.SS1.p2.3.m3.1.1.2" xref="A2.SS1.p2.3.m3.1.1.2.cmml">R</mi><mi id="A2.SS1.p2.3.m3.1.1.3" xref="A2.SS1.p2.3.m3.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="A2.SS1.p2.3.m3.1b"><apply id="A2.SS1.p2.3.m3.1.1.cmml" xref="A2.SS1.p2.3.m3.1.1"><csymbol cd="ambiguous" id="A2.SS1.p2.3.m3.1.1.1.cmml" xref="A2.SS1.p2.3.m3.1.1">subscript</csymbol><ci id="A2.SS1.p2.3.m3.1.1.2.cmml" xref="A2.SS1.p2.3.m3.1.1.2">𝑅</ci><ci id="A2.SS1.p2.3.m3.1.1.3.cmml" xref="A2.SS1.p2.3.m3.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p2.3.m3.1c">R_{s}</annotation></semantics></math> for a simple update scheme. This along with the choice of 32 samples of the Bernoulli distribution are ones we encourage more experimentation with in future work.</p>
</div>
</section>
<section id="A2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.2 </span>Visual TaskVectors (VTV) Adaptation for Multimodal ICL</h3>

<div id="A2.SS2.p1" class="ltx_para">
<p id="A2.SS2.p1.1" class="ltx_p">Visual Task Vectors (VTV) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib27" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">27</span></a>]</cite> were originally designed to be applied to large vision-transformer-based models. We make as few changes as possible to apply this method for multimodal tasks. We preserve VTVs distinct factors like a the usage of 1-shot examples for both calculation of the mean activations and attention head locations regardless of the format of the downstream task. Furthermore, we fix the number of iterations for both mean activation and attention head calculation at 10. Finally, we replace the proposed MSE loss with a cross-entropy loss that is more suited for an LMM task.</p>
</div>
</section>
<section id="A2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.3 </span>Function Vectors (FV)</h3>

<div id="A2.SS3.p1" class="ltx_para">
<p id="A2.SS3.p1.1" class="ltx_p">Because Function Vectors describe text-only task vectors, we follow the implementation of Function Vectors <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib79" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">79</span></a>]</cite> almost exactly as LLMs and LMMs are similar. The only major change made is the use of many-shot multimodal ICL examples for mean activation calculation. We preserve the lack of an optimization method for the layer used to replace the mean activations. Rather than performing a standard grid search over the set of layers, we set the layer number to 20 as recommended for LLaMA and LLaMA-based models by the paper. The only other difference is the encoding of multimodal ICL examples. Again, due the the similarity between LMMs and text-only LLMs, these tests can be used as needed as long as the multimodal inputs are properly processed by the LMM.</p>
</div>
</section>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Additional Implementation Details</h2>

<div id="A3.p1" class="ltx_para">
<p id="A3.p1.1" class="ltx_p">To run all of our experiments, we use 1 NVIDIA RTX 6000 GPU. Importantly, this includes the runtime and efficiency ablations, which were evaluated on the same GPU for consistency. Please refer to the respective model’s paper for their specific implementation details of the architecture. Besides the output token generation length, which varies depending on the standard setting for each task, we use the default generation parameters (e.g. temperature and no. of beams in beam search) recommended for each model. In the following sections, we describe some of the finer nuances of our MTV-EXTRACT process as well as our implementations of the Visual Task Vectors (VTV) and Function Vectors (FV) implementations.</p>
</div>
<section id="A3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.1 </span>VizWiz</h3>

<div id="A3.SS1.p1" class="ltx_para ltx_noindent">
<p id="A3.SS1.p1.1" class="ltx_p"><span id="A3.SS1.p1.1.1" class="ltx_text ltx_font_bold">Dataset</span>. The VizWiz dataset is designed to challenge and evaluate the capabilities of Large Multimodal Models (LMMs) in understanding and responding to real-world visual questions. This dataset is comprised of images accompanied by spoken questions, which have been transcribed and paired with answers. Each image in this dataset is sourced from visually impaired individuals seeking assistance, thereby incorporating a wide array of everyday challenges they face. This setup is inherently diverse and often requires high-level visual understanding combined with contextual reasoning, making them a robust benchmark for assessing the practical utility of LMMs in assistive technologies. The format of the dataset samples is an image paired with a text question. The LMM is required to provide a short response limited to 10 tokens or respond with “unanswerable" if the question is not answerable give the image.</p>
</div>
<div id="A3.SS1.p2" class="ltx_para">
<p id="A3.SS1.p2.1" class="ltx_p">For this research paper, we specifically utilize the VizWiz dataset to benchmark the performance of our proposed task vectors in multimodal in-context learning (MM-ICL) on a dataset that challenges visual scene understanding of LMMs. We extract MTV on the training set and evaluate on the evaluation set containing 4,319 validation image/question pairs.</p>
</div>
<div id="A3.SS1.p3" class="ltx_para ltx_noindent">
<p id="A3.SS1.p3.1" class="ltx_p"><span id="A3.SS1.p3.1.1" class="ltx_text ltx_font_bold">Inference details</span>. We use the standard VQA question-answer response format that is outlined in the QwenVL repository <a target="_blank" href="https://github.com/QwenLM/Qwen-VL" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/QwenLM/Qwen-VL</a>. Put simply, the LMM is presented with an image and a corresponding text question. The response is then expected in a short text format of no more than 10 tokens (set as the “max_tokens” parameter in the LMM). One nuance is the special answer “unanswerable". We handle this by providing MTV and all baselines with the following prompt for every question: “First carefully understand the given examples. Then use the given image and answer the question in the same way as the examples. If the question can not be answered, respond unanswerable. " The official dataset can be downloaded at <a target="_blank" href="https://vizwiz.org/tasks-and-datasets/vqa/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://vizwiz.org/tasks-and-datasets/vqa/</a>.</p>
</div>
</section>
<section id="A3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.2 </span>OK-VQA</h3>

<div id="A3.SS2.p1" class="ltx_para ltx_noindent">
<p id="A3.SS2.p1.1" class="ltx_p"><span id="A3.SS2.p1.1.1" class="ltx_text ltx_font_bold">Dataset</span>. The OK-VQA dataset, differs from traditional VQA datasets in its focus on necessitating knowledge beyond what is presented in the given images. This dataset encompasses over 14,000 questions that are not merely reliant on visual cues but require associative reasoning with external data sources, making it a unique tool for evaluating AI’s capability in handling complex, knowledge-driven queries. Thus, we evaluate on this dataset to test whether MTV can be beneficial for this type of reasoning.</p>
</div>
<div id="A3.SS2.p2" class="ltx_para">
<p id="A3.SS2.p2.1" class="ltx_p">We once again extract MTV on the train set and evaluate on the validation set. OK-VQA is formatted as an image with a corresponding text question. However, it is important to note that the text question heavily relies on external knowledge to answer. Examples of questions can be found in Section <a href="#A4" title="Appendix D Qualitative Visualizations ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D</span></a>.</p>
</div>
<div id="A3.SS2.p3" class="ltx_para ltx_noindent">
<p id="A3.SS2.p3.1" class="ltx_p"><span id="A3.SS2.p3.1.1" class="ltx_text ltx_font_bold">Inference details</span>. We use the standard VQA question-answer response format that is outlined in the QwenVL repository <a target="_blank" href="https://github.com/QwenLM/Qwen-VL" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/QwenLM/Qwen-VL</a>. Put simply, the LMM is presented with an image and a corresponding text question. The response is then expected in a short text format of no more than 10 tokens (set as the “max_tokens” parameter in the LMM). We do not add any additional prompts or special tokens apart from prompt format or image tokens required by the model being evaluated. The official dataset can be downloaded at <a target="_blank" href="https://okvqa.allenai.org/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://okvqa.allenai.org/</a>.</p>
</div>
</section>
<section id="A3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.3 </span>Flowers</h3>

<div id="A3.SS3.p1" class="ltx_para ltx_noindent">
<p id="A3.SS3.p1.1" class="ltx_p"><span id="A3.SS3.p1.1.1" class="ltx_text ltx_font_bold">Dataset</span>.
Flowers <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib58" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">58</span></a>]</cite> is an object classification dataset that requires fine-grained classification of 102 different flower species. The Flowers dataset is formulated as a 2-way, 1-shot task where one example is the positive sample and the other is the negative sample. In this way, the data poses a unique challenge for MTV having to store examples with two associated images. Thus, given the 2-way examples and the query image, the LMM is tasked with selecting the correct class from the given two options. Examples can be found in Section <a href="#A4" title="Appendix D Qualitative Visualizations ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D</span></a></p>
</div>
<div id="A3.SS3.p2" class="ltx_para ltx_noindent">
<p id="A3.SS3.p2.1" class="ltx_p"><span id="A3.SS3.p2.1.1" class="ltx_text ltx_font_bold">Implementation Details</span>. We use the official data released by the authors which is available at <a target="_blank" href="https://www.robots.ox.ac.uk/~vgg/data/flowers/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.robots.ox.ac.uk/~vgg/data/flowers/</a>. We provide a Python code snippet below showing the Flowers data format:</p>
<div id="A3.SS3.p2.2" class="ltx_listing ltx_lstlisting ltx_listing">
<div class="ltx_listing_data"><a href="data:text/plain;base64,ZGVmIGZvcm1hdF9mbG93ZXIoY3VyX2RhdGEpOgogICAgcG9zID0gY3VyX2RhdGFbInBvcyJdCiAgICBuZWcgPSBjdXJfZGF0YVsibmVnIl0KICAgIHBvc19sYWJlbCA9IGN1cl9kYXRhWyJwb3NfbGFiZWwiXQogICAgbmVnX2xhYmVsID0gY3VyX2RhdGFbIm5lZ19sYWJlbCJdCiAgICBxdWVyeSA9IGN1cl9kYXRhWyJxdWVyeSJdCiAgICByYW5kX251bSA9IHJhbmRvbS5yYW5kaW50KDAsMSkKICAgIGlmIHJhbmRfbnVtID09IDA6CiAgICAgICAgcG9zX2V4YW1wbGUgPSBmIjxpbWc+e3Bvc308L2ltZz5XaGF0IGlzIHRoZSB0eXBlIG9mIGZsb3dlciBpbiB0aGUgaW1hZ2U/IEEue3Bvc19sYWJlbH0gQi57bmVnX2xhYmVsfVxuQW5zd2VyIHdpdGggdGhlIG9wdGlvbidzIGxldHRlciBmcm9tIHRoZSBnaXZlbiBjaG9pY2UgZGlyZWN0bHkuIEFuc3dlcjogQVxuIgoKICAgICAgICBuZWdfZXhhbXBsZSA9IGYiPGltZz57bmVnfTwvaW1nPldoYXQgaXMgdGhlIHR5cGUgb2YgZmxvd2VyIGluIHRoZSBpbWFnZT8gQS57cG9zX2xhYmVsfSBCLntuZWdfbGFiZWx9XG5BbnN3ZXIgd2l0aCB0aGUgb3B0aW9uJ3MgbGV0dGVyIGZyb20gdGhlIGdpdmVuIGNob2ljZSBkaXJlY3RseS4gQW5zd2VyOiBCXG4iCgogICAgICAgIGN1cl9xdWVyeSA9IGYiPGltZz57cXVlcnl9PC9pbWc+V2hhdCBpcyB0aGUgdHlwZSBvZiBmbG93ZXIgaW4gdGhlIGltYWdlPyBBLntwb3NfbGFiZWx9IEIue25lZ19sYWJlbH1cbkFuc3dlciB3aXRoIHRoZSBvcHRpb24ncyBsZXR0ZXIgZnJvbSB0aGUgZ2l2ZW4gY2hvaWNlIGRpcmVjdGx5LiBBbnN3ZXI6IgogICAgICAgIHF1ZXJ5X2xhYmVsID0gIkEiCiAgICAgICAgcmV0dXJuIHBvc19leGFtcGxlICsgbmVnX2V4YW1wbGUgKyBjdXJfcXVlcnksIHF1ZXJ5X2xhYmVsLCAtMQoKICAgIGVsc2U6CiAgICAgICAgcG9zX2V4YW1wbGUgPSBmIjxpbWc+e3Bvc308L2ltZz5XaGF0IGlzIHRoZSB0eXBlIG9mIGZsb3dlciBpbiB0aGUgaW1hZ2U/IEEue25lZ19sYWJlbH0gQi57cG9zX2xhYmVsfVxuQW5zd2VyIHdpdGggdGhlIG9wdGlvbidzIGxldHRlciBmcm9tIHRoZSBnaXZlbiBjaG9pY2UgZGlyZWN0bHkuIEFuc3dlcjogQlxuIgoKICAgICAgICBuZWdfZXhhbXBsZSA9IGYiPGltZz57bmVnfTwvaW1nPldoYXQgaXMgdGhlIHR5cGUgb2YgZmxvd2VyIGluIHRoZSBpbWFnZT8gQS57bmVnX2xhYmVsfSBCLntwb3NfbGFiZWx9XG5BbnN3ZXIgd2l0aCB0aGUgb3B0aW9uJ3MgbGV0dGVyIGZyb20gdGhlIGdpdmVuIGNob2ljZSBkaXJlY3RseS4gQW5zd2VyOiBBXG4iCgogICAgICAgIGN1cl9xdWVyeSA9IGYiPGltZz57cXVlcnl9PC9pbWc+V2hhdCBpcyB0aGUgdHlwZSBvZiBmbG93ZXIgaW4gdGhlIGltYWdlPyBBLntuZWdfbGFiZWx9IEIue3Bvc19sYWJlbH1cbkFuc3dlciB3aXRoIHRoZSBvcHRpb24ncyBsZXR0ZXIgZnJvbSB0aGUgZ2l2ZW4gY2hvaWNlIGRpcmVjdGx5LiBBbnN3ZXI6IgogICAgICAgIHF1ZXJ5X2xhYmVsID0gIkIiCiAgICAgICAgcmV0dXJuIG5lZ19leGFtcGxlICsgcG9zX2V4YW1wbGUgKyBjdXJfcXVlcnksIHF1ZXJ5X2xhYmVsLCAtMQo=" download="">⬇</a></div>
<div id="lstnumberx1" class="ltx_listingline">
<span id="lstnumberx1.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">def</span><span id="lstnumberx1.2" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx1.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">format_flower</span><span id="lstnumberx1.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx1.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_data</span><span id="lstnumberx1.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">):</span>
</div>
<div id="lstnumberx2" class="ltx_listingline">
<span id="lstnumberx2.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx2.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos</span><span id="lstnumberx2.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx2.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx2.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx2.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_data</span><span id="lstnumberx2.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">["</span><span id="lstnumberx2.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos</span><span id="lstnumberx2.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"]</span>
</div>
<div id="lstnumberx3" class="ltx_listingline">
<span id="lstnumberx3.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx3.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg</span><span id="lstnumberx3.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx3.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx3.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx3.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_data</span><span id="lstnumberx3.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">["</span><span id="lstnumberx3.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg</span><span id="lstnumberx3.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"]</span>
</div>
<div id="lstnumberx4" class="ltx_listingline">
<span id="lstnumberx4.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx4.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_label</span><span id="lstnumberx4.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx4.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx4.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx4.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_data</span><span id="lstnumberx4.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">["</span><span id="lstnumberx4.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_label</span><span id="lstnumberx4.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"]</span>
</div>
<div id="lstnumberx5" class="ltx_listingline">
<span id="lstnumberx5.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx5.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_label</span><span id="lstnumberx5.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx5.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx5.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx5.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_data</span><span id="lstnumberx5.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">["</span><span id="lstnumberx5.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_label</span><span id="lstnumberx5.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"]</span>
</div>
<div id="lstnumberx6" class="ltx_listingline">
<span id="lstnumberx6.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx6.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">query</span><span id="lstnumberx6.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx6.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx6.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx6.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_data</span><span id="lstnumberx6.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">["</span><span id="lstnumberx6.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">query</span><span id="lstnumberx6.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"]</span>
</div>
<div id="lstnumberx7" class="ltx_listingline">
<span id="lstnumberx7.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx7.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">rand_num</span><span id="lstnumberx7.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx7.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx7.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx7.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">random</span><span id="lstnumberx7.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx7.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">randint</span><span id="lstnumberx7.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(0,1)</span>
</div>
<div id="lstnumberx8" class="ltx_listingline">
<span id="lstnumberx8.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx8.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">if</span><span id="lstnumberx8.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx8.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">rand_num</span><span id="lstnumberx8.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx8.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">==</span><span id="lstnumberx8.7" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx8.8" class="ltx_text ltx_font_typewriter" style="font-size:90%;">0:</span>
</div>
<div id="lstnumberx9" class="ltx_listingline">
<span id="lstnumberx9.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx9.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_example</span><span id="lstnumberx9.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx9.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">f</span><span id="lstnumberx9.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"&lt;</span><span id="lstnumberx9.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx9.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;{</span><span id="lstnumberx9.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos</span><span id="lstnumberx9.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}&lt;/</span><span id="lstnumberx9.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx9.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;</span><span id="lstnumberx9.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">What</span><span id="lstnumberx9.15" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">is</span><span id="lstnumberx9.17" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx9.19" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">type</span><span id="lstnumberx9.21" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">of</span><span id="lstnumberx9.23" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">flower</span><span id="lstnumberx9.25" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">in</span><span id="lstnumberx9.27" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx9.29" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">image</span><span id="lstnumberx9.31" class="ltx_text ltx_font_typewriter" style="font-size:90%;">?</span><span id="lstnumberx9.32" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">A</span><span id="lstnumberx9.34" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx9.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_label</span><span id="lstnumberx9.36" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}</span><span id="lstnumberx9.37" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.38" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">B</span><span id="lstnumberx9.39" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx9.40" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_label</span><span id="lstnumberx9.41" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}\</span><span id="lstnumberx9.42" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">nAnswer</span><span id="lstnumberx9.43" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.44" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">with</span><span id="lstnumberx9.45" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.46" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx9.47" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.48" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">option</span><span id="lstnumberx9.49" class="ltx_text ltx_font_typewriter" style="font-size:90%;">’</span><span id="lstnumberx9.50" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">s</span><span id="lstnumberx9.51" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.52" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">letter</span><span id="lstnumberx9.53" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.54" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">from</span><span id="lstnumberx9.55" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.56" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx9.57" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.58" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">given</span><span id="lstnumberx9.59" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.60" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">choice</span><span id="lstnumberx9.61" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.62" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">directly</span><span id="lstnumberx9.63" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx9.64" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.65" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Answer</span><span id="lstnumberx9.66" class="ltx_text ltx_font_typewriter" style="font-size:90%;">:</span><span id="lstnumberx9.67" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx9.68" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">A</span><span id="lstnumberx9.69" class="ltx_text ltx_font_typewriter" style="font-size:90%;">\</span><span id="lstnumberx9.70" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">n</span><span id="lstnumberx9.71" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"</span>
</div>
<div id="lstnumberx10" class="ltx_listingline">
</div>
<div id="lstnumberx11" class="ltx_listingline">
<span id="lstnumberx11.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx11.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_example</span><span id="lstnumberx11.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx11.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">f</span><span id="lstnumberx11.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"&lt;</span><span id="lstnumberx11.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx11.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;{</span><span id="lstnumberx11.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg</span><span id="lstnumberx11.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}&lt;/</span><span id="lstnumberx11.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx11.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;</span><span id="lstnumberx11.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">What</span><span id="lstnumberx11.15" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">is</span><span id="lstnumberx11.17" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx11.19" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">type</span><span id="lstnumberx11.21" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">of</span><span id="lstnumberx11.23" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">flower</span><span id="lstnumberx11.25" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">in</span><span id="lstnumberx11.27" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx11.29" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">image</span><span id="lstnumberx11.31" class="ltx_text ltx_font_typewriter" style="font-size:90%;">?</span><span id="lstnumberx11.32" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">A</span><span id="lstnumberx11.34" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx11.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_label</span><span id="lstnumberx11.36" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}</span><span id="lstnumberx11.37" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.38" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">B</span><span id="lstnumberx11.39" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx11.40" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_label</span><span id="lstnumberx11.41" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}\</span><span id="lstnumberx11.42" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">nAnswer</span><span id="lstnumberx11.43" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.44" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">with</span><span id="lstnumberx11.45" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.46" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx11.47" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.48" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">option</span><span id="lstnumberx11.49" class="ltx_text ltx_font_typewriter" style="font-size:90%;">’</span><span id="lstnumberx11.50" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">s</span><span id="lstnumberx11.51" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.52" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">letter</span><span id="lstnumberx11.53" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.54" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">from</span><span id="lstnumberx11.55" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.56" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx11.57" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.58" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">given</span><span id="lstnumberx11.59" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.60" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">choice</span><span id="lstnumberx11.61" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.62" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">directly</span><span id="lstnumberx11.63" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx11.64" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.65" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Answer</span><span id="lstnumberx11.66" class="ltx_text ltx_font_typewriter" style="font-size:90%;">:</span><span id="lstnumberx11.67" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx11.68" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">B</span><span id="lstnumberx11.69" class="ltx_text ltx_font_typewriter" style="font-size:90%;">\</span><span id="lstnumberx11.70" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">n</span><span id="lstnumberx11.71" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"</span>
</div>
<div id="lstnumberx12" class="ltx_listingline">
</div>
<div id="lstnumberx13" class="ltx_listingline">
<span id="lstnumberx13.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx13.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_query</span><span id="lstnumberx13.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx13.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">f</span><span id="lstnumberx13.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"&lt;</span><span id="lstnumberx13.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx13.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;{</span><span id="lstnumberx13.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">query</span><span id="lstnumberx13.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}&lt;/</span><span id="lstnumberx13.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx13.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;</span><span id="lstnumberx13.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">What</span><span id="lstnumberx13.15" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">is</span><span id="lstnumberx13.17" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx13.19" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">type</span><span id="lstnumberx13.21" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">of</span><span id="lstnumberx13.23" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">flower</span><span id="lstnumberx13.25" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">in</span><span id="lstnumberx13.27" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx13.29" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">image</span><span id="lstnumberx13.31" class="ltx_text ltx_font_typewriter" style="font-size:90%;">?</span><span id="lstnumberx13.32" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">A</span><span id="lstnumberx13.34" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx13.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_label</span><span id="lstnumberx13.36" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}</span><span id="lstnumberx13.37" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.38" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">B</span><span id="lstnumberx13.39" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx13.40" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_label</span><span id="lstnumberx13.41" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}\</span><span id="lstnumberx13.42" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">nAnswer</span><span id="lstnumberx13.43" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.44" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">with</span><span id="lstnumberx13.45" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.46" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx13.47" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.48" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">option</span><span id="lstnumberx13.49" class="ltx_text ltx_font_typewriter" style="font-size:90%;">’</span><span id="lstnumberx13.50" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">s</span><span id="lstnumberx13.51" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.52" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">letter</span><span id="lstnumberx13.53" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.54" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">from</span><span id="lstnumberx13.55" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.56" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx13.57" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.58" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">given</span><span id="lstnumberx13.59" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.60" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">choice</span><span id="lstnumberx13.61" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.62" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">directly</span><span id="lstnumberx13.63" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx13.64" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx13.65" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Answer</span><span id="lstnumberx13.66" class="ltx_text ltx_font_typewriter" style="font-size:90%;">:"</span>
</div>
<div id="lstnumberx14" class="ltx_listingline">
<span id="lstnumberx14.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx14.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">query_label</span><span id="lstnumberx14.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx14.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx14.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx14.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"</span><span id="lstnumberx14.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">A</span><span id="lstnumberx14.8" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"</span>
</div>
<div id="lstnumberx15" class="ltx_listingline">
<span id="lstnumberx15.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx15.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">return</span><span id="lstnumberx15.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx15.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_example</span><span id="lstnumberx15.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx15.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">+</span><span id="lstnumberx15.7" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx15.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_example</span><span id="lstnumberx15.9" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx15.10" class="ltx_text ltx_font_typewriter" style="font-size:90%;">+</span><span id="lstnumberx15.11" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx15.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_query</span><span id="lstnumberx15.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx15.14" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx15.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">query_label</span><span id="lstnumberx15.16" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx15.17" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx15.18" class="ltx_text ltx_font_typewriter" style="font-size:90%;">-1</span>
</div>
<div id="lstnumberx16" class="ltx_listingline">
</div>
<div id="lstnumberx17" class="ltx_listingline">
<span id="lstnumberx17.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx17.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">else</span><span id="lstnumberx17.3" class="ltx_text ltx_font_typewriter" style="font-size:90%;">:</span>
</div>
<div id="lstnumberx18" class="ltx_listingline">
<span id="lstnumberx18.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx18.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_example</span><span id="lstnumberx18.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx18.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">f</span><span id="lstnumberx18.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"&lt;</span><span id="lstnumberx18.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx18.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;{</span><span id="lstnumberx18.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos</span><span id="lstnumberx18.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}&lt;/</span><span id="lstnumberx18.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx18.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;</span><span id="lstnumberx18.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">What</span><span id="lstnumberx18.15" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">is</span><span id="lstnumberx18.17" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx18.19" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">type</span><span id="lstnumberx18.21" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">of</span><span id="lstnumberx18.23" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">flower</span><span id="lstnumberx18.25" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">in</span><span id="lstnumberx18.27" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx18.29" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">image</span><span id="lstnumberx18.31" class="ltx_text ltx_font_typewriter" style="font-size:90%;">?</span><span id="lstnumberx18.32" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">A</span><span id="lstnumberx18.34" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx18.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_label</span><span id="lstnumberx18.36" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}</span><span id="lstnumberx18.37" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.38" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">B</span><span id="lstnumberx18.39" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx18.40" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_label</span><span id="lstnumberx18.41" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}\</span><span id="lstnumberx18.42" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">nAnswer</span><span id="lstnumberx18.43" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.44" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">with</span><span id="lstnumberx18.45" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.46" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx18.47" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.48" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">option</span><span id="lstnumberx18.49" class="ltx_text ltx_font_typewriter" style="font-size:90%;">’</span><span id="lstnumberx18.50" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">s</span><span id="lstnumberx18.51" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.52" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">letter</span><span id="lstnumberx18.53" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.54" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">from</span><span id="lstnumberx18.55" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.56" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx18.57" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.58" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">given</span><span id="lstnumberx18.59" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.60" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">choice</span><span id="lstnumberx18.61" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.62" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">directly</span><span id="lstnumberx18.63" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx18.64" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.65" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Answer</span><span id="lstnumberx18.66" class="ltx_text ltx_font_typewriter" style="font-size:90%;">:</span><span id="lstnumberx18.67" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx18.68" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">B</span><span id="lstnumberx18.69" class="ltx_text ltx_font_typewriter" style="font-size:90%;">\</span><span id="lstnumberx18.70" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">n</span><span id="lstnumberx18.71" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"</span>
</div>
<div id="lstnumberx19" class="ltx_listingline">
</div>
<div id="lstnumberx20" class="ltx_listingline">
<span id="lstnumberx20.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx20.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_example</span><span id="lstnumberx20.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx20.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">f</span><span id="lstnumberx20.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"&lt;</span><span id="lstnumberx20.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx20.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;{</span><span id="lstnumberx20.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg</span><span id="lstnumberx20.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}&lt;/</span><span id="lstnumberx20.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx20.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;</span><span id="lstnumberx20.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">What</span><span id="lstnumberx20.15" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">is</span><span id="lstnumberx20.17" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx20.19" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">type</span><span id="lstnumberx20.21" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">of</span><span id="lstnumberx20.23" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">flower</span><span id="lstnumberx20.25" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">in</span><span id="lstnumberx20.27" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx20.29" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">image</span><span id="lstnumberx20.31" class="ltx_text ltx_font_typewriter" style="font-size:90%;">?</span><span id="lstnumberx20.32" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">A</span><span id="lstnumberx20.34" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx20.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_label</span><span id="lstnumberx20.36" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}</span><span id="lstnumberx20.37" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.38" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">B</span><span id="lstnumberx20.39" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx20.40" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_label</span><span id="lstnumberx20.41" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}\</span><span id="lstnumberx20.42" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">nAnswer</span><span id="lstnumberx20.43" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.44" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">with</span><span id="lstnumberx20.45" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.46" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx20.47" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.48" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">option</span><span id="lstnumberx20.49" class="ltx_text ltx_font_typewriter" style="font-size:90%;">’</span><span id="lstnumberx20.50" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">s</span><span id="lstnumberx20.51" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.52" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">letter</span><span id="lstnumberx20.53" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.54" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">from</span><span id="lstnumberx20.55" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.56" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx20.57" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.58" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">given</span><span id="lstnumberx20.59" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.60" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">choice</span><span id="lstnumberx20.61" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.62" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">directly</span><span id="lstnumberx20.63" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx20.64" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.65" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Answer</span><span id="lstnumberx20.66" class="ltx_text ltx_font_typewriter" style="font-size:90%;">:</span><span id="lstnumberx20.67" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx20.68" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">A</span><span id="lstnumberx20.69" class="ltx_text ltx_font_typewriter" style="font-size:90%;">\</span><span id="lstnumberx20.70" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">n</span><span id="lstnumberx20.71" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"</span>
</div>
<div id="lstnumberx21" class="ltx_listingline">
</div>
<div id="lstnumberx22" class="ltx_listingline">
<span id="lstnumberx22.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx22.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_query</span><span id="lstnumberx22.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx22.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">f</span><span id="lstnumberx22.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"&lt;</span><span id="lstnumberx22.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx22.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;{</span><span id="lstnumberx22.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">query</span><span id="lstnumberx22.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}&lt;/</span><span id="lstnumberx22.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx22.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;</span><span id="lstnumberx22.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">What</span><span id="lstnumberx22.15" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">is</span><span id="lstnumberx22.17" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx22.19" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">type</span><span id="lstnumberx22.21" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">of</span><span id="lstnumberx22.23" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">flower</span><span id="lstnumberx22.25" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">in</span><span id="lstnumberx22.27" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx22.29" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">image</span><span id="lstnumberx22.31" class="ltx_text ltx_font_typewriter" style="font-size:90%;">?</span><span id="lstnumberx22.32" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">A</span><span id="lstnumberx22.34" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx22.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_label</span><span id="lstnumberx22.36" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}</span><span id="lstnumberx22.37" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.38" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">B</span><span id="lstnumberx22.39" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx22.40" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_label</span><span id="lstnumberx22.41" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}\</span><span id="lstnumberx22.42" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">nAnswer</span><span id="lstnumberx22.43" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.44" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">with</span><span id="lstnumberx22.45" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.46" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx22.47" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.48" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">option</span><span id="lstnumberx22.49" class="ltx_text ltx_font_typewriter" style="font-size:90%;">’</span><span id="lstnumberx22.50" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">s</span><span id="lstnumberx22.51" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.52" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">letter</span><span id="lstnumberx22.53" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.54" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">from</span><span id="lstnumberx22.55" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.56" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx22.57" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.58" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">given</span><span id="lstnumberx22.59" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.60" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">choice</span><span id="lstnumberx22.61" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.62" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">directly</span><span id="lstnumberx22.63" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx22.64" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx22.65" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Answer</span><span id="lstnumberx22.66" class="ltx_text ltx_font_typewriter" style="font-size:90%;">:"</span>
</div>
<div id="lstnumberx23" class="ltx_listingline">
<span id="lstnumberx23.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx23.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">query_label</span><span id="lstnumberx23.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx23.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx23.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx23.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"</span><span id="lstnumberx23.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">B</span><span id="lstnumberx23.8" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"</span>
</div>
<div id="lstnumberx24" class="ltx_listingline">
<span id="lstnumberx24.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx24.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">return</span><span id="lstnumberx24.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx24.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_example</span><span id="lstnumberx24.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx24.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">+</span><span id="lstnumberx24.7" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx24.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_example</span><span id="lstnumberx24.9" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx24.10" class="ltx_text ltx_font_typewriter" style="font-size:90%;">+</span><span id="lstnumberx24.11" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx24.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_query</span><span id="lstnumberx24.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx24.14" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx24.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">query_label</span><span id="lstnumberx24.16" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx24.17" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx24.18" class="ltx_text ltx_font_typewriter" style="font-size:90%;">-1</span>
</div>
</div>
</div>
</section>
<section id="A3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.4 </span>CUB</h3>

<div id="A3.SS4.p1" class="ltx_para ltx_noindent">
<p id="A3.SS4.p1.1" class="ltx_p"><span id="A3.SS4.p1.1.1" class="ltx_text ltx_font_bold">Dataset</span>.
CUB <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib80" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">80</span></a>]</cite> or CUB-200-2011 is an object classification dataset that tests the fine-grained classification of 200 classes of birds. Similar to the Flowers dataset, CUB is formulated as a 2-way, 1-shot task where one example is the positive sample and the other is the negative sample. In this way, the data poses a unique challenge for MTV having to store examples with two associated images. Thus, given the 2-way examples and the query image, the LMM is tasked with selecting the correct class from the given two options.</p>
</div>
<div id="A3.SS4.p2" class="ltx_para ltx_noindent">
<p id="A3.SS4.p2.1" class="ltx_p"><span id="A3.SS4.p2.1.1" class="ltx_text ltx_font_bold">Implementation Details</span>. We use the official data released by the authors which is available at <a target="_blank" href="https://www.vision.caltech.edu/datasets/cub_200_2011/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.vision.caltech.edu/datasets/cub_200_2011/</a>. We provide a Python code snippet below showing the Flowers data format:</p>
</div>
<div id="A3.SS4.p3" class="ltx_para">
<div id="A3.SS4.p3.1" class="ltx_listing ltx_lstlisting ltx_listing">
<div class="ltx_listing_data"><a href="data:text/plain;base64,ZGVmIGZvcm1hdF9jdWIoY3VyX2RhdGEpOgogICAgcG9zID0gY3VyX2RhdGFbInBvcyJdCiAgICBuZWcgPSBjdXJfZGF0YVsibmVnIl0KICAgIHBvc19sYWJlbCA9IGN1cl9kYXRhWyJwb3NfbGFiZWwiXQogICAgbmVnX2xhYmVsID0gY3VyX2RhdGFbIm5lZ19sYWJlbCJdCiAgICBxdWVyeSA9IGN1cl9kYXRhWyJxdWVyeSJdCiAgICByYW5kX251bSA9IHJhbmRvbS5yYW5kaW50KDAsMSkKICAgIGlmIHJhbmRfbnVtID09IDA6CiAgICAgICAgcG9zX2V4YW1wbGUgPSBmIjxpbWc+e3Bvc308L2ltZz5XaGF0IGlzIHRoZSB0eXBlIG9mIGJpcmQgaW4gdGhlIGltYWdlPyBBLntwb3NfbGFiZWx9IEIue25lZ19sYWJlbH1cbkFuc3dlciB3aXRoIHRoZSBvcHRpb24ncyBsZXR0ZXIgZnJvbSB0aGUgZ2l2ZW4gY2hvaWNlIGRpcmVjdGx5LiBBbnN3ZXI6IEFcbiIKCiAgICAgICAgbmVnX2V4YW1wbGUgPSBmIjxpbWc+e25lZ308L2ltZz5XaGF0IGlzIHRoZSB0eXBlIG9mIGJpcmQgaW4gdGhlIGltYWdlPyBBLntwb3NfbGFiZWx9IEIue25lZ19sYWJlbH1cbkFuc3dlciB3aXRoIHRoZSBvcHRpb24ncyBsZXR0ZXIgZnJvbSB0aGUgZ2l2ZW4gY2hvaWNlIGRpcmVjdGx5LiBBbnN3ZXI6IEJcbiIKCiAgICAgICAgY3VyX3F1ZXJ5ID0gZiI8aW1nPntxdWVyeX08L2ltZz5XaGF0IGlzIHRoZSB0eXBlIG9mIGJpcmQgaW4gdGhlIGltYWdlPyBBLntwb3NfbGFiZWx9IEIue25lZ19sYWJlbH1cbkFuc3dlciB3aXRoIHRoZSBvcHRpb24ncyBsZXR0ZXIgZnJvbSB0aGUgZ2l2ZW4gY2hvaWNlIGRpcmVjdGx5LiBBbnN3ZXI6IgogICAgICAgIHF1ZXJ5X2xhYmVsID0gIkEiCiAgICAgICAgcmV0dXJuIHBvc19leGFtcGxlICsgbmVnX2V4YW1wbGUgKyBjdXJfcXVlcnksIHF1ZXJ5X2xhYmVsLCAtMQoKICAgIGVsc2U6CiAgICAgICAgcG9zX2V4YW1wbGUgPSBmIjxpbWc+e3Bvc308L2ltZz5XaGF0IGlzIHRoZSB0eXBlIG9mIGJpcmQgaW4gdGhlIGltYWdlPyBBLntuZWdfbGFiZWx9IEIue3Bvc19sYWJlbH1cbkFuc3dlciB3aXRoIHRoZSBvcHRpb24ncyBsZXR0ZXIgZnJvbSB0aGUgZ2l2ZW4gY2hvaWNlIGRpcmVjdGx5LiBBbnN3ZXI6IEJcbiIKCiAgICAgICAgbmVnX2V4YW1wbGUgPSBmIjxpbWc+e25lZ308L2ltZz5XaGF0IGlzIHRoZSB0eXBlIG9mIGJpcmQgaW4gdGhlIGltYWdlPyBBLntuZWdfbGFiZWx9IEIue3Bvc19sYWJlbH1cbkFuc3dlciB3aXRoIHRoZSBvcHRpb24ncyBsZXR0ZXIgZnJvbSB0aGUgZ2l2ZW4gY2hvaWNlIGRpcmVjdGx5LiBBbnN3ZXI6IEFcbiIKCiAgICAgICAgY3VyX3F1ZXJ5ID0gZiI8aW1nPntxdWVyeX08L2ltZz5XaGF0IGlzIHRoZSB0eXBlIG9mIGJpcmQgaW4gdGhlIGltYWdlPyBBLntuZWdfbGFiZWx9IEIue3Bvc19sYWJlbH1cbkFuc3dlciB3aXRoIHRoZSBvcHRpb24ncyBsZXR0ZXIgZnJvbSB0aGUgZ2l2ZW4gY2hvaWNlIGRpcmVjdGx5LiBBbnN3ZXI6IgogICAgICAgIHF1ZXJ5X2xhYmVsID0gIkIiCiAgICAgICAgcmV0dXJuIG5lZ19leGFtcGxlICsgcG9zX2V4YW1wbGUgKyBjdXJfcXVlcnksIHF1ZXJ5X2xhYmVsLCAtMQo=" download="">⬇</a></div>
<div id="lstnumberx25" class="ltx_listingline">
<span id="lstnumberx25.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">def</span><span id="lstnumberx25.2" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx25.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">format_cub</span><span id="lstnumberx25.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(</span><span id="lstnumberx25.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_data</span><span id="lstnumberx25.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">):</span>
</div>
<div id="lstnumberx26" class="ltx_listingline">
<span id="lstnumberx26.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx26.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos</span><span id="lstnumberx26.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx26.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx26.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx26.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_data</span><span id="lstnumberx26.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">["</span><span id="lstnumberx26.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos</span><span id="lstnumberx26.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"]</span>
</div>
<div id="lstnumberx27" class="ltx_listingline">
<span id="lstnumberx27.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx27.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg</span><span id="lstnumberx27.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx27.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx27.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx27.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_data</span><span id="lstnumberx27.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">["</span><span id="lstnumberx27.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg</span><span id="lstnumberx27.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"]</span>
</div>
<div id="lstnumberx28" class="ltx_listingline">
<span id="lstnumberx28.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx28.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_label</span><span id="lstnumberx28.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx28.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx28.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx28.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_data</span><span id="lstnumberx28.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">["</span><span id="lstnumberx28.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_label</span><span id="lstnumberx28.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"]</span>
</div>
<div id="lstnumberx29" class="ltx_listingline">
<span id="lstnumberx29.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx29.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_label</span><span id="lstnumberx29.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx29.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx29.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx29.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_data</span><span id="lstnumberx29.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">["</span><span id="lstnumberx29.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_label</span><span id="lstnumberx29.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"]</span>
</div>
<div id="lstnumberx30" class="ltx_listingline">
<span id="lstnumberx30.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx30.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">query</span><span id="lstnumberx30.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx30.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx30.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx30.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_data</span><span id="lstnumberx30.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">["</span><span id="lstnumberx30.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">query</span><span id="lstnumberx30.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"]</span>
</div>
<div id="lstnumberx31" class="ltx_listingline">
<span id="lstnumberx31.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx31.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">rand_num</span><span id="lstnumberx31.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx31.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx31.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx31.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">random</span><span id="lstnumberx31.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx31.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">randint</span><span id="lstnumberx31.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">(0,1)</span>
</div>
<div id="lstnumberx32" class="ltx_listingline">
<span id="lstnumberx32.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx32.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">if</span><span id="lstnumberx32.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx32.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">rand_num</span><span id="lstnumberx32.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx32.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">==</span><span id="lstnumberx32.7" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx32.8" class="ltx_text ltx_font_typewriter" style="font-size:90%;">0:</span>
</div>
<div id="lstnumberx33" class="ltx_listingline">
<span id="lstnumberx33.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx33.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_example</span><span id="lstnumberx33.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx33.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">f</span><span id="lstnumberx33.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"&lt;</span><span id="lstnumberx33.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx33.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;{</span><span id="lstnumberx33.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos</span><span id="lstnumberx33.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}&lt;/</span><span id="lstnumberx33.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx33.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;</span><span id="lstnumberx33.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">What</span><span id="lstnumberx33.15" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">is</span><span id="lstnumberx33.17" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx33.19" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">type</span><span id="lstnumberx33.21" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">of</span><span id="lstnumberx33.23" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">bird</span><span id="lstnumberx33.25" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">in</span><span id="lstnumberx33.27" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx33.29" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">image</span><span id="lstnumberx33.31" class="ltx_text ltx_font_typewriter" style="font-size:90%;">?</span><span id="lstnumberx33.32" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">A</span><span id="lstnumberx33.34" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx33.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_label</span><span id="lstnumberx33.36" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}</span><span id="lstnumberx33.37" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.38" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">B</span><span id="lstnumberx33.39" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx33.40" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_label</span><span id="lstnumberx33.41" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}\</span><span id="lstnumberx33.42" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">nAnswer</span><span id="lstnumberx33.43" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.44" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">with</span><span id="lstnumberx33.45" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.46" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx33.47" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.48" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">option</span><span id="lstnumberx33.49" class="ltx_text ltx_font_typewriter" style="font-size:90%;">’</span><span id="lstnumberx33.50" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">s</span><span id="lstnumberx33.51" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.52" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">letter</span><span id="lstnumberx33.53" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.54" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">from</span><span id="lstnumberx33.55" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.56" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx33.57" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.58" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">given</span><span id="lstnumberx33.59" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.60" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">choice</span><span id="lstnumberx33.61" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.62" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">directly</span><span id="lstnumberx33.63" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx33.64" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.65" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Answer</span><span id="lstnumberx33.66" class="ltx_text ltx_font_typewriter" style="font-size:90%;">:</span><span id="lstnumberx33.67" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx33.68" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">A</span><span id="lstnumberx33.69" class="ltx_text ltx_font_typewriter" style="font-size:90%;">\</span><span id="lstnumberx33.70" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">n</span><span id="lstnumberx33.71" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"</span>
</div>
<div id="lstnumberx34" class="ltx_listingline">
</div>
<div id="lstnumberx35" class="ltx_listingline">
<span id="lstnumberx35.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx35.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_example</span><span id="lstnumberx35.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx35.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">f</span><span id="lstnumberx35.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"&lt;</span><span id="lstnumberx35.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx35.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;{</span><span id="lstnumberx35.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg</span><span id="lstnumberx35.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}&lt;/</span><span id="lstnumberx35.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx35.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;</span><span id="lstnumberx35.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">What</span><span id="lstnumberx35.15" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">is</span><span id="lstnumberx35.17" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx35.19" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">type</span><span id="lstnumberx35.21" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">of</span><span id="lstnumberx35.23" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">bird</span><span id="lstnumberx35.25" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">in</span><span id="lstnumberx35.27" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx35.29" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">image</span><span id="lstnumberx35.31" class="ltx_text ltx_font_typewriter" style="font-size:90%;">?</span><span id="lstnumberx35.32" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">A</span><span id="lstnumberx35.34" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx35.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_label</span><span id="lstnumberx35.36" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}</span><span id="lstnumberx35.37" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.38" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">B</span><span id="lstnumberx35.39" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx35.40" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_label</span><span id="lstnumberx35.41" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}\</span><span id="lstnumberx35.42" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">nAnswer</span><span id="lstnumberx35.43" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.44" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">with</span><span id="lstnumberx35.45" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.46" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx35.47" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.48" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">option</span><span id="lstnumberx35.49" class="ltx_text ltx_font_typewriter" style="font-size:90%;">’</span><span id="lstnumberx35.50" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">s</span><span id="lstnumberx35.51" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.52" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">letter</span><span id="lstnumberx35.53" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.54" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">from</span><span id="lstnumberx35.55" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.56" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx35.57" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.58" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">given</span><span id="lstnumberx35.59" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.60" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">choice</span><span id="lstnumberx35.61" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.62" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">directly</span><span id="lstnumberx35.63" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx35.64" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.65" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Answer</span><span id="lstnumberx35.66" class="ltx_text ltx_font_typewriter" style="font-size:90%;">:</span><span id="lstnumberx35.67" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx35.68" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">B</span><span id="lstnumberx35.69" class="ltx_text ltx_font_typewriter" style="font-size:90%;">\</span><span id="lstnumberx35.70" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">n</span><span id="lstnumberx35.71" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"</span>
</div>
<div id="lstnumberx36" class="ltx_listingline">
</div>
<div id="lstnumberx37" class="ltx_listingline">
<span id="lstnumberx37.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx37.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_query</span><span id="lstnumberx37.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx37.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">f</span><span id="lstnumberx37.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"&lt;</span><span id="lstnumberx37.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx37.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;{</span><span id="lstnumberx37.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">query</span><span id="lstnumberx37.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}&lt;/</span><span id="lstnumberx37.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx37.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;</span><span id="lstnumberx37.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">What</span><span id="lstnumberx37.15" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">is</span><span id="lstnumberx37.17" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx37.19" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">type</span><span id="lstnumberx37.21" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">of</span><span id="lstnumberx37.23" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">bird</span><span id="lstnumberx37.25" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">in</span><span id="lstnumberx37.27" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx37.29" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">image</span><span id="lstnumberx37.31" class="ltx_text ltx_font_typewriter" style="font-size:90%;">?</span><span id="lstnumberx37.32" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">A</span><span id="lstnumberx37.34" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx37.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_label</span><span id="lstnumberx37.36" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}</span><span id="lstnumberx37.37" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.38" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">B</span><span id="lstnumberx37.39" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx37.40" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_label</span><span id="lstnumberx37.41" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}\</span><span id="lstnumberx37.42" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">nAnswer</span><span id="lstnumberx37.43" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.44" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">with</span><span id="lstnumberx37.45" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.46" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx37.47" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.48" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">option</span><span id="lstnumberx37.49" class="ltx_text ltx_font_typewriter" style="font-size:90%;">’</span><span id="lstnumberx37.50" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">s</span><span id="lstnumberx37.51" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.52" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">letter</span><span id="lstnumberx37.53" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.54" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">from</span><span id="lstnumberx37.55" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.56" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx37.57" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.58" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">given</span><span id="lstnumberx37.59" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.60" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">choice</span><span id="lstnumberx37.61" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.62" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">directly</span><span id="lstnumberx37.63" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx37.64" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx37.65" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Answer</span><span id="lstnumberx37.66" class="ltx_text ltx_font_typewriter" style="font-size:90%;">:"</span>
</div>
<div id="lstnumberx38" class="ltx_listingline">
<span id="lstnumberx38.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx38.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">query_label</span><span id="lstnumberx38.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx38.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx38.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx38.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"</span><span id="lstnumberx38.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">A</span><span id="lstnumberx38.8" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"</span>
</div>
<div id="lstnumberx39" class="ltx_listingline">
<span id="lstnumberx39.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx39.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">return</span><span id="lstnumberx39.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx39.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_example</span><span id="lstnumberx39.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx39.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">+</span><span id="lstnumberx39.7" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx39.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_example</span><span id="lstnumberx39.9" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx39.10" class="ltx_text ltx_font_typewriter" style="font-size:90%;">+</span><span id="lstnumberx39.11" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx39.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_query</span><span id="lstnumberx39.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx39.14" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx39.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">query_label</span><span id="lstnumberx39.16" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx39.17" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx39.18" class="ltx_text ltx_font_typewriter" style="font-size:90%;">-1</span>
</div>
<div id="lstnumberx40" class="ltx_listingline">
</div>
<div id="lstnumberx41" class="ltx_listingline">
<span id="lstnumberx41.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">    </span><span id="lstnumberx41.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">else</span><span id="lstnumberx41.3" class="ltx_text ltx_font_typewriter" style="font-size:90%;">:</span>
</div>
<div id="lstnumberx42" class="ltx_listingline">
<span id="lstnumberx42.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx42.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_example</span><span id="lstnumberx42.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx42.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">f</span><span id="lstnumberx42.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"&lt;</span><span id="lstnumberx42.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx42.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;{</span><span id="lstnumberx42.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos</span><span id="lstnumberx42.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}&lt;/</span><span id="lstnumberx42.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx42.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;</span><span id="lstnumberx42.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">What</span><span id="lstnumberx42.15" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">is</span><span id="lstnumberx42.17" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx42.19" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">type</span><span id="lstnumberx42.21" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">of</span><span id="lstnumberx42.23" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">bird</span><span id="lstnumberx42.25" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">in</span><span id="lstnumberx42.27" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx42.29" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">image</span><span id="lstnumberx42.31" class="ltx_text ltx_font_typewriter" style="font-size:90%;">?</span><span id="lstnumberx42.32" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">A</span><span id="lstnumberx42.34" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx42.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_label</span><span id="lstnumberx42.36" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}</span><span id="lstnumberx42.37" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.38" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">B</span><span id="lstnumberx42.39" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx42.40" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_label</span><span id="lstnumberx42.41" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}\</span><span id="lstnumberx42.42" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">nAnswer</span><span id="lstnumberx42.43" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.44" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">with</span><span id="lstnumberx42.45" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.46" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx42.47" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.48" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">option</span><span id="lstnumberx42.49" class="ltx_text ltx_font_typewriter" style="font-size:90%;">’</span><span id="lstnumberx42.50" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">s</span><span id="lstnumberx42.51" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.52" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">letter</span><span id="lstnumberx42.53" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.54" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">from</span><span id="lstnumberx42.55" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.56" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx42.57" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.58" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">given</span><span id="lstnumberx42.59" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.60" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">choice</span><span id="lstnumberx42.61" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.62" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">directly</span><span id="lstnumberx42.63" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx42.64" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.65" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Answer</span><span id="lstnumberx42.66" class="ltx_text ltx_font_typewriter" style="font-size:90%;">:</span><span id="lstnumberx42.67" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx42.68" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">B</span><span id="lstnumberx42.69" class="ltx_text ltx_font_typewriter" style="font-size:90%;">\</span><span id="lstnumberx42.70" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">n</span><span id="lstnumberx42.71" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"</span>
</div>
<div id="lstnumberx43" class="ltx_listingline">
</div>
<div id="lstnumberx44" class="ltx_listingline">
<span id="lstnumberx44.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx44.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_example</span><span id="lstnumberx44.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx44.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">f</span><span id="lstnumberx44.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"&lt;</span><span id="lstnumberx44.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx44.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;{</span><span id="lstnumberx44.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg</span><span id="lstnumberx44.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}&lt;/</span><span id="lstnumberx44.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx44.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;</span><span id="lstnumberx44.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">What</span><span id="lstnumberx44.15" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">is</span><span id="lstnumberx44.17" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx44.19" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">type</span><span id="lstnumberx44.21" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">of</span><span id="lstnumberx44.23" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">bird</span><span id="lstnumberx44.25" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">in</span><span id="lstnumberx44.27" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx44.29" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">image</span><span id="lstnumberx44.31" class="ltx_text ltx_font_typewriter" style="font-size:90%;">?</span><span id="lstnumberx44.32" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">A</span><span id="lstnumberx44.34" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx44.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_label</span><span id="lstnumberx44.36" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}</span><span id="lstnumberx44.37" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.38" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">B</span><span id="lstnumberx44.39" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx44.40" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_label</span><span id="lstnumberx44.41" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}\</span><span id="lstnumberx44.42" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">nAnswer</span><span id="lstnumberx44.43" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.44" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">with</span><span id="lstnumberx44.45" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.46" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx44.47" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.48" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">option</span><span id="lstnumberx44.49" class="ltx_text ltx_font_typewriter" style="font-size:90%;">’</span><span id="lstnumberx44.50" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">s</span><span id="lstnumberx44.51" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.52" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">letter</span><span id="lstnumberx44.53" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.54" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">from</span><span id="lstnumberx44.55" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.56" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx44.57" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.58" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">given</span><span id="lstnumberx44.59" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.60" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">choice</span><span id="lstnumberx44.61" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.62" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">directly</span><span id="lstnumberx44.63" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx44.64" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.65" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Answer</span><span id="lstnumberx44.66" class="ltx_text ltx_font_typewriter" style="font-size:90%;">:</span><span id="lstnumberx44.67" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx44.68" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">A</span><span id="lstnumberx44.69" class="ltx_text ltx_font_typewriter" style="font-size:90%;">\</span><span id="lstnumberx44.70" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">n</span><span id="lstnumberx44.71" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"</span>
</div>
<div id="lstnumberx45" class="ltx_listingline">
</div>
<div id="lstnumberx46" class="ltx_listingline">
<span id="lstnumberx46.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx46.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_query</span><span id="lstnumberx46.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx46.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">f</span><span id="lstnumberx46.7" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"&lt;</span><span id="lstnumberx46.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx46.9" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;{</span><span id="lstnumberx46.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">query</span><span id="lstnumberx46.11" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}&lt;/</span><span id="lstnumberx46.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">img</span><span id="lstnumberx46.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">&gt;</span><span id="lstnumberx46.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">What</span><span id="lstnumberx46.15" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">is</span><span id="lstnumberx46.17" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx46.19" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">type</span><span id="lstnumberx46.21" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">of</span><span id="lstnumberx46.23" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">bird</span><span id="lstnumberx46.25" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">in</span><span id="lstnumberx46.27" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.28" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx46.29" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.30" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">image</span><span id="lstnumberx46.31" class="ltx_text ltx_font_typewriter" style="font-size:90%;">?</span><span id="lstnumberx46.32" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.33" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">A</span><span id="lstnumberx46.34" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx46.35" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_label</span><span id="lstnumberx46.36" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}</span><span id="lstnumberx46.37" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.38" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">B</span><span id="lstnumberx46.39" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.{</span><span id="lstnumberx46.40" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_label</span><span id="lstnumberx46.41" class="ltx_text ltx_font_typewriter" style="font-size:90%;">}\</span><span id="lstnumberx46.42" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">nAnswer</span><span id="lstnumberx46.43" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.44" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">with</span><span id="lstnumberx46.45" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.46" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx46.47" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.48" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">option</span><span id="lstnumberx46.49" class="ltx_text ltx_font_typewriter" style="font-size:90%;">’</span><span id="lstnumberx46.50" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">s</span><span id="lstnumberx46.51" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.52" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">letter</span><span id="lstnumberx46.53" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.54" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">from</span><span id="lstnumberx46.55" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.56" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">the</span><span id="lstnumberx46.57" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.58" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">given</span><span id="lstnumberx46.59" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.60" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">choice</span><span id="lstnumberx46.61" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.62" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">directly</span><span id="lstnumberx46.63" class="ltx_text ltx_font_typewriter" style="font-size:90%;">.</span><span id="lstnumberx46.64" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx46.65" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">Answer</span><span id="lstnumberx46.66" class="ltx_text ltx_font_typewriter" style="font-size:90%;">:"</span>
</div>
<div id="lstnumberx47" class="ltx_listingline">
<span id="lstnumberx47.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx47.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">query_label</span><span id="lstnumberx47.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx47.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">=</span><span id="lstnumberx47.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx47.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"</span><span id="lstnumberx47.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">B</span><span id="lstnumberx47.8" class="ltx_text ltx_font_typewriter" style="font-size:90%;">"</span>
</div>
<div id="lstnumberx48" class="ltx_listingline">
<span id="lstnumberx48.1" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;">        </span><span id="lstnumberx48.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">return</span><span id="lstnumberx48.3" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx48.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">neg_example</span><span id="lstnumberx48.5" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx48.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">+</span><span id="lstnumberx48.7" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx48.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">pos_example</span><span id="lstnumberx48.9" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx48.10" class="ltx_text ltx_font_typewriter" style="font-size:90%;">+</span><span id="lstnumberx48.11" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx48.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">cur_query</span><span id="lstnumberx48.13" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx48.14" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx48.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter" style="font-size:90%;">query_label</span><span id="lstnumberx48.16" class="ltx_text ltx_font_typewriter" style="font-size:90%;">,</span><span id="lstnumberx48.17" class="ltx_text ltx_lst_space ltx_font_typewriter" style="font-size:90%;"> </span><span id="lstnumberx48.18" class="ltx_text ltx_font_typewriter" style="font-size:90%;">-1</span>
</div>
</div>
</div>
</section>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Qualitative Visualizations</h2>

<div id="A4.p1" class="ltx_para">
<p id="A4.p1.1" class="ltx_p">We present further qualitative success and failure cases of <span id="A4.p1.1.1" class="ltx_text ltx_font_bold">QwenVL-MTV</span> in Figure <a href="#A4.F4" title="Figure 4 ‣ Appendix D Qualitative Visualizations ‣ Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> on OK-VQA and Flowers.</p>
</div>
<figure id="A4.F4" class="ltx_figure"><img src="/html/2406.15334/assets/figs/examples.png" id="A4.F4.g1" class="ltx_graphics ltx_centering ltx_img_square" width="598" height="718" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="A4.F4.3.1.1" class="ltx_text" style="font-size:90%;">Figure 4</span>: </span><span id="A4.F4.4.2" class="ltx_text" style="font-size:90%;">
<span id="A4.F4.4.2.1" class="ltx_text ltx_font_bold">Examples.</span> Here we show example outputs of our method compared to zero-shot outputs.</span></figcaption>
</figure>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Licenses and Privacy</h2>

<div id="A5.p1" class="ltx_para">
<p id="A5.p1.1" class="ltx_p">The license, PII, and consent details of each dataset are in the respective papers. In addition, we wish to emphasize that the datasets we use do not contain any harmful or offensive content, as many other papers in the field also use them. Thus, we do not anticipate a specific negative impact, but, as with any machine learning method, we recommend exercising caution.</p>
</div>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2406.15333" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2406.15334" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2406.15334">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2406.15334" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2406.15335" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Fri Jul  5 19:57:25 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2406.15334
The recent success of interleaved Large Multimodal Models (LMMs)  in few-shot learning suggests that in-context learning (ICL)  with many examples can be promising for learning new tasks. However, thismany-shotmultimodal ICL setting has one crucial problem: it is fundamentally limited by the model’s context length set at pretraining. The problem is especially prominent in the multimodal domain, which processes both text and images, requiring additional tokens. This motivates the need for a multimodal method to compress many shots into fewer tokens without finetuning. In this work, we enable LMMs to perform multimodal, many-shot in-context learning by leveraging Multimodal Task Vectors (MTV) —compact implicit representations of in-context examples compressed in the model’s attention heads. Specifically, we first demonstrate the existence of such MTV in LMMs and then leverage these extracted MTV to enable many-shot in-context learning for various vision-and-language tasks. Our experiments suggest that MTV can scale in performance with the number of compressed shots and generalize to similar out-of-domain tasks without additional context length for inference.
abstract
Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
Abstract
2406.15334
1IntroductionFigure 1:Overview of Multimodal Task Vectors (MTV) .In the standard multimodal in-context learning (ICL)  paradigm (top) , the number of shots is limited by an LMM’s context length. We solve this issue by first calculating the mean activations of the last token’s attention heads (grey squares in the LMM)  given a set of multimodal ICL examples and then finding a set of attention head locations (yellow squares in the LMM)  that best align with the downstream task. These mean activations are then replaced directly in these attention heads locations, enabling many-shot multimodal ICL.Large Multimodal Models (LMMs)  such as GPT-4V, LLaVA, and the BLIPfamily of models demonstrate state-of-the-art performance on a variety of vision and language (VL)  tasks due to their strong reasoning capabilities over both text and images. Recent works show that LMMs pre-trained on interleaved text-image data can do multimodal in-context learning. In particular, few-shot, in-context learning (ICL)  in text-only LLMs has been scaled with an increasing number of examples in long-context language models—a setting called many-shot learning. A natural question arises on how to perform many-shot learning in the multimodal domain.The first issue with directly applying a many-shot learning regimen to LMMs is the intrinsic limitation of context length. This is especially true in the multimodal domain, as LMMs must encode both text and images, whose embeddings are token-expensive. Moreover, long-context language models, which LMMs leverage for reasoning, struggle to use their entire context length effectively for ICL. Secondly, perhaps due to the misalignment of pretraining tasks with ICL, many instruction-tuned LMMs underperform on tasks in the ICL setting, suggesting the importance of interleaved LMMs. Finally, there is also the challenge of the increasing memory and run-time required for processing long contexts for every inference call. These challenges motivate a method for compressing multimodal in-context examples into compact, implicit representations. Therefore, in this paper, we propose Multimodal Task Vectors (MTV) —compact representations of multimodal in-context tasks—within the attention heads of LMMs to enable many-shot ICL. In particular, we show the existence of MTV in interleaved LMMs, and we use them to compress large numbers of multimodal ICL examples.Recent research in explainability has demonstrated the existence of task vectors in both the languageand visiondomains. These task vectors are implicit representations of in-context tasks represented by sets of activations in the model. These activations compactly summarize the information in ICL examples. In our work, we go beyond proving the existence of these task vectors in the multimodal domain by demonstrating their ability to compress examples for many-shot ICL in LMMs without the need for finetuning.Our method can be described in three steps. First, given a set of many-shot multimodal ICL examples, we calculate the mean activations of the attention heads across multiple inference iterations. Second, to avoid the context length constraint, we select a set of attention heads in the model to store the mean activations of the ICL examples. However, since the downstream task may be zero-shot or use a different number of ICL examples, we select a set of examples aligned with its form. We then use these examples to find an optimal set of LMM head locations where the many-shot examples will be encoded. We refer to these mean activations and locations as MTV, which implicitly encodes the many-shot multimodal examples for use in the downstream task. Finally, for downstream inference, we replace the mean activations from Step 1 with the attention head locations found in Step 2. Since we input examples to the LMM across different iterations in Step 1, Multimodal Task Vectors can implicitly encode more examples than are allowable by the context limit. We find that utilizing many examples for extracting MTV surpasses performance on zero-shot and most standard few-shot ICL settings, suggesting the effectiveness of our method. Another key benefit of our method is that it frees up tokens for the model during downstream inference compared to standard few-shot ICL methods.We summarize our main contributions as follows: (i)  We show the existence of Multimodal Task Vectors, compact implicit representations of in-context functions in LMMs. (ii)  MTV can encode more examples than allowed by an LMM’s context length, enabling both runtime and memory-efficient multimodal many-shot in-context learning.
(iii)  MTV surpasses zero-shot and few-shot ICL settings on various VL benchmarks without finetuning.
(iv)  MTV can scale to larger numbers of examples and can generalize to similar out-of-domain tasks.
introduction
Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
1Introduction
2406.15334
2Related WorksMany-Shot In-Context Learning. Few-shot in-context learning (ICL)  is a significant area of study in text-only LLMs. A natural question arises about the possibility of using a larger number of shots (e.g., hundreds)  to further improve performance or learn more complex tasks. Indeed, some early work in text-onlymany-shot, in-context learningsuggests performance on different tasks can scale with a larger number of examples.However, scaling ICL in text-only LLMs is a challenge due to the intrinsic context length. One method to increase context length in these models is to apply positional interpolation methods. However, research on these longer-context models finds that they struggle to use the entire context for ICL. Moreover, as inference on long contexts of inputs is also time and memory-expensive, it is unclear whether simply scaling the context of models is practical for enabling multimodal many-shot ICL. This has led to work that looks to compress explicit input tokens. But crucially, many of these methods require finetuning and only try to preserve performance. Our work is different in that it is the first to enablemultimodalmodels with many-shot ICL capabilities, while also having the benefit of improving on complex VL tasks without finetuning.Task Vectors. Our work builds off of research in text-only and vision-only domains showing that internal representations of these models called task vectors(or function vectors)  can encapsulate tasks outlined by ICL examples. Our is the first demonstration of Multimodal Task Vectors (MTV)  in LMMs. Going beyond previous work, however, we show that MTV enable LMMs not only to use many-shot, multimodal ICL examples but also scale with more samples, be used alongside explicit ICL shots, and even generalize to unseen classes or similar tasks.Model Domain Adaptation Methods.
As LLM and LMM model architectures have advanced, so have methods to allow these models to generalize beyond their pretraining distributions. Methods like instruction tuninghave shown strong zero-shot generalization to some out-of-domain tasks, but forgetting remains an issue. One popular solution to this issue involves Parameter Efficient Fine-tuning (PEFT) : finetuning either a set of soft prompt input tokens, low-rank model weights, or a separate adapter from the main model.Prompting methods are a well-explored area for adapting models without finetuning. LLM prompting includes zero-shot methods, few-shot and ICL methods, expert prompting, and Chain-of-Thought (CoT) , with extensions like self-consistency, Tree-of-Thought (ToT) , and Graph-of-Thought (GoT) for more complex structures. Similar multimodal prompting methods exist for LMMs as well.Large Multimodal Models (LMMs) . The state-of-the-art performance of LMMson multimodal tasks stems from combining LLMs’ reasoning capabilitieswith the perception abilities of vision models. LMMs’ generative reasoning also makes them more applicable to complex tasks than previous contrastive methods. Such tasks include visual question-answeringas well as object identification and localization. Visual Programmatic Models (VPMs)  are another class of multimodal methods that makes use of in-context APIs code generation. However, context length limits both LMMs’ and VPMs’ ability to use multimodal prompting methods such as ICL. Another key challenge is that many LMMs are pre-trained on single text-image pair data. Recently, many LMM models now pretrain on interleaved text-image data, making effective multimodal ICL possible. In our work, MTV goes beyond simple few-shot multimodal ICL and scales to many-shot multimodal ICL.
body
Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
2Related Works
2406.15334
3Multimodal Task VectorsTo address the challenge of performing many-shot multimodal in-context learning, we demonstrate the existence of MTV in LMMs and then leverage them for many-shot multimodal ICL. We begin by describing some background on multimodal ICL and task vectors (Section3.1) . We then introduce our three-step approach: (i)  We calculate the mean activations of the attention heads from the many-shot multimodal ICL examples (Section3.2) ; (ii)  We then extract the set of LMM attention heads locations that best align to the downstream task using an adapted version of the REINFORCEalgorithm (Section3.3) ; and (iii)  We replace the calculated mean activation values into the LMM for a downstream task (Section3.4) . The method overview is also illustrated Figure1.3.1PreliminariesIn the multimodal in-context learning setting, an LMM learns a new task outlined by a set of multimodal examples. The input to the LMM would be outlined as follows:(1) where the model is prompted to answer a querygiven a set of input-output examples (eachbeing a multimodal input and eacha text output) .We note that in-context examples are commonly passed sequentially to the LMM, necessarily restricting multimodal ICL to being small numbers of shots due to limited context length. Furthermore, the images require more tokens to embed, which means enabling many-shot ICL is even more challenging in the multimodal domain. To solve this, we utilize our method MTV—which are implicit representations in the model’s attention heads that encode a many-shot multimodal ICL task.We start with a background on task vectors for some task. Given a model, we denote the set of attention-head locations aswhere each locationis indexed asfor thelayer andattention head. We define the task vectors as follows: (1)  the task vectorvaluesare a subset of mean activations produced by the attention heads ofgiven examples of a task, and (2)  the task vectorlocations, which denotes a subset of the attention head indices per task. Thus, the task vector is. For inference,replaces the activation values of the heads in.In prior work, the calculation of the mean activationsand the extraction of the attention-head locationsare used together to extract the task vector. Interestingly, we find that these two steps should be decoupled in order to better align with the downstream task. In our work, we calculate the mean activationscorresponding to the last token specifically to encode a dataset of many-shot multimodal ICL examples by averaging them across multiple inference calls. However, the downstream task may not always be in the same ICL format as the many-shot examples (e.g., the downstream task uses a different number of shots or is zero-shot) . To solve this, we use a separate set of examples that are of the exact format of the downstream task to align the extracted attention-head locationswith the inference task. This separation of responsibilities, whereincaptures the essential information from the many-shot examples andidentifies the specific attention head locations for the downstream task, optimizes the utilization of the encoded information at relevant locations within the model.Our approach to finding Multimodal Task Vectors (MTV) allows LMMs to actually leverage many-shot multimodal ICL examples for complex vision-language tasks without being limited by context length. We proceed by first describing how to calculate the mean activations.3.2Step 1: Calculate MTV Mean ActivationsThe ultimate objective of many-shot multimodal ICL is to use a large number of input-output examples when solving a task. However, it is not trivial to get the LMM to see more examples during inference time than its context length allows.To address this issue, we pass a few-shot inputfor each inference callfor a total ofinference calls. Eachconsists ofshots (where)  of multimodal in-context examples in the form of randomly-selected input-output response pairs, and, which is the query to be answered by the LMM in that iteration.(2) Thus, overLMM inference calls, we have a many-shot multimodal dataset (ofexamples) :(3) However, this dataset is still just a disconnected set of few-shot examples. Next, we would like to connect the separate examples into one unified many-shot multimodal ICL representation.For each inference call, the LMM is given-shot ICL examples. We calculate the mean of the activations corresponding to the last token of the inputfor each attention head index(Section3.1)  acrossinference calls, yielding:(4) In this step, we have found the mean activations, which encode an internal LMM representation of many shots of multimodal ICL examples. In the next subsection, we describe our methodology for selecting the set of attention heads where these mean activations will be used.3.3Step 2: Extract MTV Attention Head LocationsAfter Step 1, we now have mean activations for the attention heads of the last token in a given many-shot multimodal task. Yet, we still need to find which set of attention headsshould be chosen to encode our task.To choose the set of attention heads, we first prepare a separate set ofexamples specifically aligned to the format of the downstream task. For instance, if the downstream setting is a 2-way, one-shot classification task, then theexamples should conform to this paradigm. For our explanation, let’s consider a downstream task that is zero-shot such that there is a single queryand corresponding responsefor all.From these examples, we utilize an adapted version of the REINFORCEalgorithm—an iterative policy optimization method that can be used to find task vector locations. Given an LMM, we first select a proposed set of attention head locations by sampling a Bernoulli distribution over the locations multiple times. Next, we directly replace the values of the selected attention heads with the corresponding mean activations. Then, after prompting the model with the query, we use the negative cross-entropy loss between the LMM’s output logits and the logits of the ground-truth responseto optimize the Bernoulli distribution. By optimizing the Bernoulli distribution acrossiterations, we are finding the best attention head locationsfor patching in our mean activations. Finally, we can extract, the optimized indices of attention heads, by sampling our optimized Bernoulli distribution.(5) It is important to note that MTV_EXTRACT does not require finetuning of the LMM parameters, but rather only inference calls. We describe further the underlying details of our adapted MTV_EXTRACT algorithm in SectionBof the Supplementary. Having foundand, we describe in what follows, the final procedure to use MTV for inference.3.4Step 3: Multimodal Task Vector ApplicationAfter we have identified a set of attention heads, it is straightforward to apply MTV for inference. We denote the set of mean activationsas follows.To run downstream inference on a new querywith our model, we directly replace the values of attention headswithand produce the following response:(6) is thus a response generated using many shots of multimodal examples as implicit context via MTV. The key insight of our method is the importance of(the number of multimodal examples)  and many(the number of iterations)  during the calculation of MTV. This enables an LMM to go beyond its context length to learn more nuanced properties of the task from seeing many examples. Additionally, insertion of MTV directly into the LMM also obviates the need for any context length during downstream inference, actuallyfreeingadditional context for other use (e.g., an additional prompt, more ICL examples, etc.) . Finally, because we align the attention-head locations with the downstream task, MTV can be effectively applied to zero-shot and different ICL settings.
body
Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
3Multimodal Task Vectors
2406.15334
4EvaluationIn order for LMMs to perform multimodal ICL, it is important for interleaved data to be included in pretraining. We apply our MTV approach to Qwen-VL, Idefics2-8B, and ViLA-1.5-8Bthree popular interleaved LMMs. For each model, we compare our method to using few-shot ICL across different vision-and-language tasks like VQA and object identification.4.1Implementation DetailsWe implemented MTV using PyTorch. We used each model’s respective official implementation. While the compute and memory requirements differ slightly between models, all our experiments can be run on a single NVIDIA A6000 GPU. For additional information, refer to Supplementary SectionC. Our model and weights will be released upon acceptance, and our code is in Supplementary.4.2ModelsIn this work, we apply MTV to the following interleaved LMMs as they are better-suited for multimodal ICL as shown by: (1) QwenVLis a LLaMA-based model that has the ability to process high-resolution images, and its two-stage pre-training methodology, which includes multi-task finetuning and interleaved text-image data.
(2) Idefics2-8Bis a Mistral-based model that benefits from its pre-training on the expansive OBELICS dataset, which comprises a web-scale collection of interleaved image-text documents. We utilize the base version of the model. This demonstrates multimodal in-context learning abilities.
(3) ViLA-1.5-8B. ViLA-1.5-8Bis an architecture that leverages LLaMA-3 as the LLM backbone. As in others, a significant portion of the model’s pretraining data is interleaved text-image data.Table 1:Results. (Left)  MTV evaluated on VQA datasets. (Right)  MTV evaluated on object classification datasets. The baselines are shown ingray.(a) MTV on VQA BenchmarksModelVizWizOK-VQAFlamingo 9B28.844.7+4-shot ICL34.949.3+8-shot ICL39.450.0Blip321.226.5+4-shot ICL38.449.2+8-shot ICL44.349.1Qwen-VL-7B35.258.6+4-shot ICL42.062.0+8-shot ICL44.361.5+MTV45.662.0Idefics231.352.4+4-shot ICL40.851.5+8-shot ICL43.852.3+MTV52.553.0Llama3-VILA-1.5-8B28.032.8+4-shot ICL39.335.6+8-shot ICL44.236.5+MTV55.240.6(b) MTV on Object ClassificationModelFlowersCUBLLaVA-1.5-13B+ 1-shot ICL58.6058.24LLaVA-1.6-13B+ 1-shot ICL65.5867.90Flamingo 9B+ 1-shot ICL 9B48.7851.2IDEFICS-9B+ 1-shot ICL55.2962.0Emu 37B+ 1-shot ICL52.7653.56Qwen-VL-7B+ 1-shot ICL55.056.5+MTV+1-shot ICL78.180.0Idefics2+ 1-shot ICL82.888.7+MTV+1-shot ICL83.889.8Llama3-VILA-1.5-8B+ 1-shot ICL87.488.4+MTV+1-shot ICL89.389.74.3DatasetsWe briefly describe the tasks and datasets we evaluate our method on. More details about the datasets and their setup can be found in SectionC.VQA Datasets. We use the following commonly-evaluated datasets which emphasize different aspects of multimodal reasoning, including visual features (VizWiz)  and outside knowledge (OK-VQA) : (1) VizWizconsists of images taken by visually impaired individuals paired with questions they pose about these images, making it crucial for developing AI systems that assist in real-world, accessibility-focused visual understanding tasks. (2) OK-VQAdatasetis designed to push the boundaries of Visual Question Answering (VQA)  by focusing on knowledge-based questions, where answers require external knowledge beyond the image content. (3) Object Classification. We use the following datasets, which are commonly used for object classification in multimodal ICL: (1)  TheFlowersdataset, commonly known as the Oxford 102 Flowers dataset, is a collection specifically designed for image-based flower species recognition for fine-grained classification of 102 different categories of flowers. (2) Caltech’s CUB Dataset on Birdsis a well-known resource for evaluating algorithms on the task of object identification, specifically focused on bird species. It features 200 bird species with roughly 30 images each, annotated with key attributes and bounding boxes. Both Flowers and Birds are formatted as 2-way,1-shot classification episodes, with model inputs being a positive and negative image for the class to be identified in the query image. The response format is a short text response.
body
Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
4Evaluation
2406.15334
5ResultsOur main results are shown in Table1. For VQA, we show the results of MTV with 4 shots per 100 iterations to calculate the mean activations and 100 examples for task vector locations (500 examples total) . The task vector is extracted using examples from the train set of the dataset and evaluated on the validation set. For object classification, we extract MTV based on a 2-way, one-shot regimen per 100 iterations for both mean activations and task vector locations (200 examples total) . The task vector is extracted using a train set of 30% of the object classes and evaluated on the remaining 70% ofunseenclasses.
We demonstrate how Multimodal Task Vectors outperforms zero-shot and few-shot ICL settings on three different models on VL tasks, highlighting the effectiveness of our method.
Next, we describe the unique capabilities of our method, such as scaling to more samples and showing some generalizations to other tasks. More results can be found in SectionA.1of Supplementary.5.1MTV scales with more examplesFigure 2:Scaling of Qwen-MTV on VizWiz:(Left)  We show the effect of varying the number of shots per iteration for a fixed 100 iterations. (Right)  We also show the effect of varying numbers of iterations fixing 4 shots per iteration.We are interested in evaluating (i)  the effect of different numbers of shots usedper iterationto extract MTV and (ii)  the effect of different numbers ofiterationsused. We test the impact on accuracy when increasing both of these parameters for QwenVL on the VizWiz validation set. In Figure2, we show on the left that the optimal number of multimodal ICL shots is 16 shots per iteration. Further, we show on the right side of the figure that 1000 examples yield the best performance. These results illustrate that MTV can effectively scale by utilizing larger numbers of ICL examples per iteration and also in aggregate.5.2MTV works with explicit few-shot examplesOne of the benefits of MTV over a few-shot ICL is the context length that is saved during inference. This is because the many-shot examples are encoded directly in the activation space rather than in the input token space. Thus, we ask whether the LMM can use the freed context for additional few-shot examples. For object classification, we formulate both Flowers and CUB as a 1-shot comparison between a positive and negative sample to identify the correct class (i.e., 2-way, 1-shot ICL by construction) . We report results on 1-shot ICL and MTV with 1-shot classification during inference. MTV+1-shot ICL surpasses 1-shot ICL accuracy on these tasks, showing that MTV can be utilized alongside few-shot examples. Furthermore, it is vital to note that the evaluation classes are completely unseen by MTV. Thus, with just a 1-shot ICL example, MTV is able to generalize to unseen classes.5.3MTV heads generalize to other tasksTable 2:Generalization & Method Comparison(Left)  MTV-VizWiz evaluated on OK-VQA. (Right)  MTV compared to VizWiz finetuning, function vectors, and task vectors.(a) Attention Head GeneralizationModelVizWizOK-VQAViLA-1.5-8B28.032.8+ 4-shot-ICL39.335.6+ 8-shot-ICL44.236.5+MTV-Vizwiz55.238.3(b) Comparison to Other MethodsModelVizWizOK-VQAQwen-VL-7B35.258.6+ VizWiz LoRA62.025.1+ FV36.459.0+ VTV37.058.9+MTV45.662.0In this experiment, we further ask whether the MTV headsextracted on one taskcan generalize to a separate, but similar task. To test this, we use the attention heads extracted from ViLA-1.5-8B on VizWiz for use on OK-VQA. Our results on the left of Table2demonstrate that the extracted heads from one task can improve accuracy on another similar task. This generalizability of the heads is significant because it suggests that the heads from MTV may only have to be extracted once to be applied to many other similar tasks. The only calculation necessary then would be the mean activations of the many-shot examples used for the target dataset, making the application of many-shot multimodal ICL even more efficient for similar tasks.5.4LoRA Finetuning as an upper boundIn Table2(b) , we compare our method to LoRA finetuning. To do this, we use LoRA on the same number of examples as MTV uses from the train set and evaluate not only on the validation set but also on the validation set of another similar dataset. In particular, for a ViLA-1.5-8B model finetuned on VizWiz, we report accuracy on both VizWiz and OK-VQA validation sets. It can be seen that LoRA finetuning is indeed an upper bound on the dataset the model was finetuned on. However, we show that LoRA leads to overfitting on the finetuned dataset and forgetting the zero-shot capabilities. In contrast, we also show that MTV not only improves zero-shot capabilities but can generalize to similar tasks with only a few inference examples Table1(b) and Table2(a) .5.5Comparison to other methodsWe compare our method to two different methods that can find task vectors: Visual Task Vectors (VTV) and Function Vectors (FV) . Originally, these works could not be applied as-is to support multimodal ICL, but here, we have implemented a version that follows the original exactly with only minor modifications to allow
performing our evaluated multimodal tasks. More details about the methods can be found in SectionBin the Supplementary. In our experiments Table2(b) , we find that MTV surpasses both methods on VizWiz and OK-VQA. VTV are image-only task vectors that use only one-shot image examples for fixed smalliterations, and they calculate the mean activations and the locations together without aligning to the downstream task. FV are text-only task vectors that use Causal Mediation Analysisto extract task vector locations. But then a vector of mean activations at these locations is simply added after the output of an arbitrary layer. The results suggest the importance of finding the task vectors by decoupling the calculation of the mean activations and locations in two separate steps to perform many-shot multimodal ICL more effectively for complex multimodal tasks.5.6Compute and runtime efficiencyMetric0-shot4-shot8-shot16-shotMTV (400-shot) Max GPU Memory (GB) 17.418.319.020.619.8Runtime per 100 iterations (min) 1.12.73.13.31.9Table 3:Efficiency:We show that even though MTV encodes 400 multimodal ICL examples in the mean activations, it still requires less runtime and memory than 8-shot and 16-shot multimodal ICL.An important feature of our work is that multimodal ICL examples do not require explicit tokens during inference. Because of this, we are interested in the efficiency gains of our method. Intuitively, the longer MTV extraction time is amortized during downstream inference, where the runtime would be equivalent to the zero-shot case.
Similarly, the memory requirements are maximal during the MTV extraction process but require the same memory as the zero-shot case afterward. In contrast, the ICL tasks have a slower runtime and larger memory requirement throughout due to running inference onexamplesfor every iteration. To demonstrate this, we calculate the maximum memory requirement in gigabytes (GB)  for ViLA-1.5-8B on VizWiz using different ICL-shot counts and MTV with 400 examples. As shown in Table3, MTV requires less runtime than 16-shot, 8-shot, and 4-shot ICL methods and also requires less memory than 16-shot ICL. These results demonstrate that MTV can encode many multimodal ICL examples with greater efficiency than few-shot methods.
body
Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
5Results
2406.15334
6ConclusionIn this work, we present Multimodal Task Vectors a compact, implicit representation that can efficiently encode many-shot multimodal ICL examples for use in complex vision-language tasks. We demonstrate this implicit model representation not only encodes a multimodal ICL task but can also enable many-shot multimodal ICL to surpass zero-shot and few-shot performance on a variety of VL tasks. Our method stands out from previous work in its ability to scale, use additional explicit multimodal ICL examples, and generalize to other similar VL tasks. Our work is a viable way to surpass the limit of context length of an LMM for multimodal ICL and demonstrates clearly that these additional examples aid in multimodal reasoning. Finally, we do not anticipate a specific negative impact, but, as with any Machine Learning method, we recommend exercising caution.
body
Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
6Conclusion
2406.15334
7LimitationsWhile Multimodal Task Vectors offers substantial benefits for handling complex vision-language tasks compared to finetuning or few-shot ICL, it is important to recognize certain limitations that accompany our approach. MTV requires access to the internal architecture of an LMM, so while it is an effective solution for all open-source models, its application is restricted from proprietary models, such as GPT-4and Gemini. Furthermore, while many-shot ICL is incredibly attractive for many applications, it may not be practical for low-data scenarios where synthetic dataor the transfer of MTV extracted from another dataset may be required.
We feel these challenges represent great opportunities for future work in the many-shot multimodal in-context learning domain.Acknowledgements.We would like to thank Suzie Petryk and Grace Luo for helpful feedback and discussions. This project was supported in part by DoD, including PTG and/or LwLL programs, as well as BAIR’s industrial alliance programs.
conclusion
Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning
7Limitations