<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.07726] PaliGemma: A versatile 3B VLM for transfer</title><meta property="og:description" content="PaliGemma is an open Vision-Language Model (VLM) that is based on the SigLIP-So400m vision encoder and the Gemma-2B language model.
It is trained to be a versatile and broadly knowledgeable base model that is effective…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="PaliGemma: A versatile 3B VLM for transfer">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="PaliGemma: A versatile 3B VLM for transfer">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.07726">

<!--Generated on Mon Aug  5 16:21:39 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<div id="p1" class="ltx_para">
<span id="p1.1" class="ltx_ERROR undefined">\pdftrailerid</span>
<p id="p1.2" class="ltx_p">redacted



 
 
 
 
 








 

<span id="p1.2.1" class="ltx_ERROR undefined">\correspondingauthor</span>lbeyer,xzhai@google.com





































</p>
</div>
<h1 class="ltx_title ltx_title_document">PaliGemma: A versatile 3B VLM for transfer</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Lucas Beyer
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Core team
</span>
<span class="ltx_contact ltx_role_affiliation">Project lead
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Andreas Steiner
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Core team
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">André Susano Pinto
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Core team
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Alexander Kolesnikov
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Core team
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Xiao Wang
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Core team
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Daniel Salz
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Maxim Neumann
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Ibrahim Alabdulmohsin
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Michael Tschannen
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Emanuele Bugliarello
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Thomas Unterthiner
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Daniel Keysers
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Skanda Koppula
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Fangyu Liu
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Adam Grycner
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Alexey Gritsenko
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Neil Houlsby
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Manoj Kumar
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Keran Rong
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Julian Eisenschlos
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Rishabh Kabra
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Matthias Bauer
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Matko Bošnjak
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Xi Chen
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Matthias Minderer
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Paul Voigtlaender
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Ioana Bica
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Ivana Balazevic
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Joan Puigcerver
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Pinelopi Papalampidi
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Olivier Henaff
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Xi Xiong
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Radu Soricut
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Jeremiah Harmsen
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">
</span></span></span>
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Xiaohua Zhai
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation">Core team
</span>
<span class="ltx_contact ltx_role_affiliation">Project lead
</span></span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id1.id1" class="ltx_p">PaliGemma is an open Vision-Language Model (VLM) that is based on the SigLIP-So400m vision encoder and the Gemma-2B language model.
It is trained to be a versatile and broadly knowledgeable base model that is effective to transfer. It achieves strong performance on a wide variety of open-world tasks.
We evaluate PaliGemma on almost 40 diverse tasks including standard VLM benchmarks, but also more specialized tasks such as remote-sensing and segmentation.</p>
</div>
<section id="S1" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<figure id="S1.F1" class="ltx_figure"><img src="/html/2407.07726/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="415" height="207" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>PaliGemma’s architecture: a SigLIP image encoder feeds into a Gemma decoder LM.</figcaption>
</figure>
<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">PaliGemma is an open model, continuing the line of PaLI vision-language models in a combination with the Gemma family of language models.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">PaLI is a series of state-of-the-art vision-language models, starting with the first PaLI <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>]</cite> showing promising scaling results up to 17 B, using classification pretrained ViT <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib127" title="" class="ltx_ref">127</a>]</cite> and mT5 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib122" title="" class="ltx_ref">122</a>]</cite> language model.
PaLI-X <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib23" title="" class="ltx_ref">23</a>]</cite> and PaLM-E <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib35" title="" class="ltx_ref">35</a>]</cite> then pushed this further, combining ViT-22 B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib28" title="" class="ltx_ref">28</a>]</cite> and a 32 B UL2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib100" title="" class="ltx_ref">100</a>]</cite> language model or the 540 B PaLM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib27" title="" class="ltx_ref">27</a>]</cite> language model, respectively, and getting further increased performance on vision-language tasks, albeit saturating performance on standard image classification and retrieval tasks.
Finally, PaLI-3 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib24" title="" class="ltx_ref">24</a>]</cite> demonstrates that through better pretraining with SigLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib129" title="" class="ltx_ref">129</a>]</cite> and more careful multimodal data curation, a 2 B vision and 3 B language model (<em id="S1.p2.1.1" class="ltx_emph ltx_font_italic">i.e</em>.<span id="S1.p2.1.2" class="ltx_text"></span> a 5 B vision-language model) matches the 10x larger PaLI-X and 100x larger PaLM-E across most benchmarks.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">PaliGemma continues this trend, combining the 400 M SigLIP and the 2 B Gemma models <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib79" title="" class="ltx_ref">79</a>]</cite> into a sub-3 B VLM that still maintains performance comparable to PaLI-X, PaLM-E, and PaLI-3.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">Gemma <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib79" title="" class="ltx_ref">79</a>]</cite> is a family of auto-regressive decoder-only open large language models built from the same research and technology used to create the Gemini <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib7" title="" class="ltx_ref">7</a>]</cite> models.
The models come in different sizes (2 B, 7 B), both pretrained and instruction fine-tuned. PaliGemma uses the 2 B pretrained version.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">The main goal of our work is to provide a versatile base VLM.
Hence, we show that it reaches state-of-the-art results not only on standard COCO captions, VQAv2, InfographicVQA and others, but also on more exotic Remote-Sensing VQA, TallyVQA, several video captioning and QA tasks, as well as referring expression <em id="S1.p5.1.1" class="ltx_emph ltx_font_italic">segmentation</em> (see full task list in Appendix <a href="#A2" title="Appendix B Tasks ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>).</p>
</div>
</section>
<section id="S2" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related work</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">Over the course of the past few years, vision-language models have gained considerable importance in computer vision.
The first generation, spearheaded by CLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib90" title="" class="ltx_ref">90</a>]</cite> and ALIGN <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib47" title="" class="ltx_ref">47</a>]</cite> by scaling up ConVIRT <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib131" title="" class="ltx_ref">131</a>]</cite> and VirTex <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib31" title="" class="ltx_ref">31</a>]</cite>, is an extension of large-scale classification pretraining <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib53" title="" class="ltx_ref">53</a>, <a href="#bib.bib127" title="" class="ltx_ref">127</a>]</cite>, to leverage all data from the web without the need for onerous human labeling, replacing a fixed and large set of classes by a caption embedding instead.
The caption embeddings are mostly obtained using language encoders (similar to BERT <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib32" title="" class="ltx_ref">32</a>]</cite>) and allow to open up the vocabulary of classification and retrieval tasks.
The second generation, akin to T5 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib91" title="" class="ltx_ref">91</a>]</cite> in language, is a unification of captioning and question-answering tasks via generative encoder-decoder modeling <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib26" title="" class="ltx_ref">26</a>, <a href="#bib.bib116" title="" class="ltx_ref">116</a>, <a href="#bib.bib133" title="" class="ltx_ref">133</a>, <a href="#bib.bib107" title="" class="ltx_ref">107</a>]</cite>, often backed by the progress in generative language models.
These were then further scaled up by, among others, Flamingo <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite>, BLIP-2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite> and, PaLI <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>]</cite>.
Finally, most recent works <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib84" title="" class="ltx_ref">84</a>, <a href="#bib.bib67" title="" class="ltx_ref">67</a>, <a href="#bib.bib7" title="" class="ltx_ref">7</a>, <a href="#bib.bib109" title="" class="ltx_ref">109</a>]</cite> perform an additional “instruction tuning” step that is intended to make the raw model more user-friendly.
In addition to building systems, several recent more systematic studies <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib78" title="" class="ltx_ref">78</a>, <a href="#bib.bib57" title="" class="ltx_ref">57</a>, <a href="#bib.bib103" title="" class="ltx_ref">103</a>]</cite> aim to find out what really matters in VLMs.
PaliGemma is an open base VLM without instruction tuning, and this report answers a few more questions regarding what matters.
More discussion in Appendix <a href="#A1" title="Appendix A More related work ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>.</p>
</div>
</section>
<section id="S3" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Model</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">In this section we present details about PaliGemma’s architecture and training.
Several of our decisions are further ablated in Section <a href="#S5" title="5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>.</p>
</div>
<div id="S3.p2" class="ltx_para">
<p id="S3.p2.1" class="ltx_p">At a high level, PaliGemma is a VLM, taking as input one or more images, and a textual description of the task (the prompt or question, which we often refer to as the <span id="S3.p2.1.1" class="ltx_text ltx_font_typewriter">prefix</span>).
PaliGemma then autoregressively generates a prediction in the form of a text string (the answer, which we often refer to as the <span id="S3.p2.1.2" class="ltx_text ltx_font_typewriter">suffix</span>).</p>
</div>
<div id="S3.p3" class="ltx_para">
<p id="S3.p3.1" class="ltx_p">This simple image+text in, text out API is flexible enough to cover many standard tasks, such as image classification, captioning, visual question-answering and dialogue.
Additionally, as shown in the literature, by converting more complex structured outputs into “text”, this API can also cover more tasks such as:
detection <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib21" title="" class="ltx_ref">21</a>]</cite>, instance segmentation <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib24" title="" class="ltx_ref">24</a>, <a href="#bib.bib111" title="" class="ltx_ref">111</a>]</cite>, panoptic segmentation, depth prediction, colorization, and many more <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>, <a href="#bib.bib70" title="" class="ltx_ref">70</a>, <a href="#bib.bib134" title="" class="ltx_ref">134</a>]</cite>.
This conversion can be hand-engineered and task-specific, such as done in pix2seq <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib21" title="" class="ltx_ref">21</a>]</cite> for detection, or learned as is the case for segmentation <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite> and dense output tasks in general.</p>
</div>
<div id="S3.p4" class="ltx_para">
<p id="S3.p4.1" class="ltx_p">During PaliGemma’s pretraining, we limit ourselves to “text” covering natural language, object detection, and instance segmentation, but this API remains versatile and the pretrained models can be finetuned for other output types.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Architecture</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">PaliGemma consists of three components:</p>
</div>
<div id="S3.SS1.p2" class="ltx_para">
<ul id="S3.I1" class="ltx_itemize">
<li id="S3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i1.p1" class="ltx_para">
<p id="S3.I1.i1.p1.1" class="ltx_p">An image encoder, for which we use a publicly available SigLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib129" title="" class="ltx_ref">129</a>]</cite> checkpoint, specifically the “shape optimized” <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>]</cite> ViT-So400m image encoder. This model was contrastively pretrained at large scale via the sigmoid loss, and has shown state-of-the-art performance, especially for its small size.</p>
</div>
</li>
<li id="S3.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i2.p1" class="ltx_para">
<p id="S3.I1.i2.p1.1" class="ltx_p">A decoder-only language model, for which we use the publicly available Gemma-2B v1.0 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib79" title="" class="ltx_ref">79</a>]</cite> raw pretrained checkpoint, which strikes a great balance between performance and size.
As we will show, this language model is good enough to match or surpass the performance of VLMs using much larger language models, including previous PaLIs.</p>
</div>
</li>
<li id="S3.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I1.i3.p1" class="ltx_para">
<p id="S3.I1.i3.p1.1" class="ltx_p">A linear layer projecting SigLIP’s output tokens into the same dimensions as Gemma-2B’s vocab tokens, so they can be concatenated.
In early experiments, we found that more complicated alternatives (<em id="S3.I1.i3.p1.1.1" class="ltx_emph ltx_font_italic">e.g</em>.<span id="S3.I1.i3.p1.1.2" class="ltx_text"></span> MLPs) do not provide a clear advantage, and hence decided to use the simplest option (Sec <a href="#S5.SS5" title="5.5 Connector design ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.5</span></a>).</p>
</div>
</li>
</ul>
</div>
<div id="S3.SS1.p3" class="ltx_para">
<p id="S3.SS1.p3.2" class="ltx_p">The image is passed through the image encoder, which turns it into a sequence of <math id="S3.SS1.p3.1.m1.1" class="ltx_Math" alttext="N_{\text{img}}" display="inline"><semantics id="S3.SS1.p3.1.m1.1a"><msub id="S3.SS1.p3.1.m1.1.1" xref="S3.SS1.p3.1.m1.1.1.cmml"><mi id="S3.SS1.p3.1.m1.1.1.2" xref="S3.SS1.p3.1.m1.1.1.2.cmml">N</mi><mtext id="S3.SS1.p3.1.m1.1.1.3" xref="S3.SS1.p3.1.m1.1.1.3a.cmml">img</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.1.m1.1b"><apply id="S3.SS1.p3.1.m1.1.1.cmml" xref="S3.SS1.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.1.m1.1.1.1.cmml" xref="S3.SS1.p3.1.m1.1.1">subscript</csymbol><ci id="S3.SS1.p3.1.m1.1.1.2.cmml" xref="S3.SS1.p3.1.m1.1.1.2">𝑁</ci><ci id="S3.SS1.p3.1.m1.1.1.3a.cmml" xref="S3.SS1.p3.1.m1.1.1.3"><mtext mathsize="70%" id="S3.SS1.p3.1.m1.1.1.3.cmml" xref="S3.SS1.p3.1.m1.1.1.3">img</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.1.m1.1c">N_{\text{img}}</annotation></semantics></math> tokens.
The text is converted into <math id="S3.SS1.p3.2.m2.1" class="ltx_Math" alttext="N_{\text{txt}}" display="inline"><semantics id="S3.SS1.p3.2.m2.1a"><msub id="S3.SS1.p3.2.m2.1.1" xref="S3.SS1.p3.2.m2.1.1.cmml"><mi id="S3.SS1.p3.2.m2.1.1.2" xref="S3.SS1.p3.2.m2.1.1.2.cmml">N</mi><mtext id="S3.SS1.p3.2.m2.1.1.3" xref="S3.SS1.p3.2.m2.1.1.3a.cmml">txt</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.2.m2.1b"><apply id="S3.SS1.p3.2.m2.1.1.cmml" xref="S3.SS1.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.2.m2.1.1.1.cmml" xref="S3.SS1.p3.2.m2.1.1">subscript</csymbol><ci id="S3.SS1.p3.2.m2.1.1.2.cmml" xref="S3.SS1.p3.2.m2.1.1.2">𝑁</ci><ci id="S3.SS1.p3.2.m2.1.1.3a.cmml" xref="S3.SS1.p3.2.m2.1.1.3"><mtext mathsize="70%" id="S3.SS1.p3.2.m2.1.1.3.cmml" xref="S3.SS1.p3.2.m2.1.1.3">txt</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.2.m2.1c">N_{\text{txt}}</annotation></semantics></math> tokens using Gemma’s SentencePiece <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib56" title="" class="ltx_ref">56</a>]</cite> tokenizer, and embedded with Gemma’s vocabulary embedding layer.
The image tokens are projected with the (zero initialized) linear projection.
Then the sequence of input tokens to the decoder is created as follows (and also as visible in Figure <a href="#S3.F2" title="Figure 2 ‣ 3.1 Architecture ‣ 3 Model ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>):</p>
</div>
<div id="S3.SS1.p4" class="ltx_para">
<pre id="S3.SS1.p4.1" class="ltx_verbatim ltx_font_typewriter" style="font-size:80%;">  tokens = [image tokens...,
            BOS, prefix tokens..., SEP,
            suffix tokens..., EOS, PAD...]
</pre>
</div>
<figure id="S3.F2" class="ltx_figure"><img src="/html/2407.07726/assets/x2.png" id="S3.F2.g1" class="ltx_graphics ltx_centering ltx_img_square" width="207" height="175" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>PaliGemma’s Prefix-LM masking: block attention throughout image and prefix, autoregressive attention on the suffix. Each square indicates whether the row can attend to the column.</figcaption>
</figure>
<div id="S3.SS1.p5" class="ltx_para">
<p id="S3.SS1.p5.1" class="ltx_p">We always resize the image to a fixed square size (224, 448, or 896 pixels).
This leads to a fixed number of image tokens per model variant (respectively 256, 1024, or 4096 tokens), which we place in the front, making image tokens straightforward to interpret without the need for special location markers.
The BOS token then marks the start of text tokens.
We use <span id="S3.SS1.p5.1.1" class="ltx_text ltx_font_typewriter">\n</span> as SEP token, it does not appear in any of our prefixes.
We also tokenize SEP separately to avoid it being merged (by the tokenizer) with either the end of the prefix or the beginning of the suffix.
In order to maximize model capacity for such a small model, we have full (unmasked) attention on the whole input, <em id="S3.SS1.p5.1.2" class="ltx_emph ltx_font_italic">i.e</em>.<span id="S3.SS1.p5.1.3" class="ltx_text"></span> the image and prefix tokens.
In this way, image tokens can also "lookahead" at the task at hand (prefix) in order to update their representation.
The suffix is our output and necessarily covered by an auto-regressive mask, including the PAD tokens.
When we mention sequence length (<math id="S3.SS1.p5.1.m1.1" class="ltx_Math" alttext="N_{\text{txt}}" display="inline"><semantics id="S3.SS1.p5.1.m1.1a"><msub id="S3.SS1.p5.1.m1.1.1" xref="S3.SS1.p5.1.m1.1.1.cmml"><mi id="S3.SS1.p5.1.m1.1.1.2" xref="S3.SS1.p5.1.m1.1.1.2.cmml">N</mi><mtext id="S3.SS1.p5.1.m1.1.1.3" xref="S3.SS1.p5.1.m1.1.1.3a.cmml">txt</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p5.1.m1.1b"><apply id="S3.SS1.p5.1.m1.1.1.cmml" xref="S3.SS1.p5.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p5.1.m1.1.1.1.cmml" xref="S3.SS1.p5.1.m1.1.1">subscript</csymbol><ci id="S3.SS1.p5.1.m1.1.1.2.cmml" xref="S3.SS1.p5.1.m1.1.1.2">𝑁</ci><ci id="S3.SS1.p5.1.m1.1.1.3a.cmml" xref="S3.SS1.p5.1.m1.1.1.3"><mtext mathsize="70%" id="S3.SS1.p5.1.m1.1.1.3.cmml" xref="S3.SS1.p5.1.m1.1.1.3">txt</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p5.1.m1.1c">N_{\text{txt}}</annotation></semantics></math>), we typically mean prefix and suffix combined, ignoring image tokens.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Pretraining</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">The training of PaliGemma follows the same steps as previous PaLI models, with only small modifications. Training consists of several stages, which we detail in this section:</p>
</div>
<div id="S3.SS2.p2" class="ltx_para">
<ul id="S3.I2" class="ltx_itemize">
<li id="S3.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I2.i1.p1" class="ltx_para">
<p id="S3.I2.i1.p1.1" class="ltx_p"><span id="S3.I2.i1.p1.1.1" class="ltx_text ltx_font_bold">Stage0:</span> Unimodal pretraining - we use existing off-the-shelf components.</p>
</div>
</li>
<li id="S3.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I2.i2.p1" class="ltx_para">
<p id="S3.I2.i2.p1.1" class="ltx_p"><span id="S3.I2.i2.p1.1.1" class="ltx_text ltx_font_bold">Stage1:</span> Multimodal pretraining - long pretraining on a carefully chosen mixture of multimodal tasks. Notably, nothing is frozen.</p>
</div>
</li>
<li id="S3.I2.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I2.i3.p1" class="ltx_para">
<p id="S3.I2.i3.p1.1" class="ltx_p"><span id="S3.I2.i3.p1.1.1" class="ltx_text ltx_font_bold">Stage2:</span> Resolution increase - short continued pretraining at higher resolution.</p>
</div>
</li>
<li id="S3.I2.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I2.i4.p1" class="ltx_para">
<p id="S3.I2.i4.p1.1" class="ltx_p"><span id="S3.I2.i4.p1.1.1" class="ltx_text ltx_font_bold">Stage3:</span> Transfer - turn the base model into a task-specific specialist.</p>
</div>
</li>
</ul>
</div>
<section id="S3.SS2.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.2.1 </span>Stage0: Unimodal pretraining</h4>

<div id="S3.SS2.SSS1.p1" class="ltx_para">
<p id="S3.SS2.SSS1.p1.1" class="ltx_p">First, the unimodal components of the model are pretrained individually, in order to benefit from their well-studied and scaled training recipes.
For PaliGemma specifically, we do not perform any custom unimodal pretraining, instead relying on existing publicly available checkpoints.</p>
</div>
<div id="S3.SS2.SSS1.p2" class="ltx_para">
<p id="S3.SS2.SSS1.p2.1" class="ltx_p">Following PaLI-3’s strong experimental results, we use a SigLIP image encoder.
While PaLI-3 (and others <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>, <a href="#bib.bib25" title="" class="ltx_ref">25</a>]</cite>) use a large image model such as ViT-G, we use the much smaller but similarly strong “shape optimized” ViT-So400m model.</p>
</div>
<div id="S3.SS2.SSS1.p3" class="ltx_para">
<p id="S3.SS2.SSS1.p3.1" class="ltx_p">PaLI traditionally uses an encoder-decoder language model; however all recently publicly released language models are decoder-only Transformers.
We opt for the Gemma-2B model, which strikes a good balance between size and performance.
Larger language models, such as the popular 7 B or 70 B sizes, are often significantly better at tasks like mathematical reasoning.
However, PaLI-3 has shown that across a wide range of vision-language tasks, a well-trained small 5 B model (2 B vision + 3 B language) can attain the same performance as the much larger 55 B PaLI-X (22 B vision + 32 B language) and 562 B PaLM-E (22 B vision + 540 B language), including tasks such as ScienceQA.
With PaliGemma we continue this push for smaller models and show that we can keep the same performance with less than 3 B total parameters.</p>
</div>
</section>
<section id="S3.SS2.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.2.2 </span>Stage1: Multimodal pretraining</h4>

<div id="S3.SS2.SSS2.p1" class="ltx_para">
<p id="S3.SS2.SSS2.p1.1" class="ltx_p">In this stage, we combine the unimodal models as explained in Section <a href="#S3.SS1" title="3.1 Architecture ‣ 3 Model ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.1</span></a> and train the whole model on a broad mixture of large-scale vision-language tasks.
Contrary to most recent VLMs, our core goal is to train a base model that fine-tunes well to a wide range of tasks, not merely to align the modalities.
Intuitively, we want a mix of tasks which force the model to acquire a broad range of “skills”, regardless of the task’s user (or benchmark) friendliness out of the box.
More on this in Section <a href="#S3.SS2.SSS5" title="3.2.5 Pretraining task mixture ‣ 3.2 Pretraining ‣ 3 Model ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2.5</span></a>.</p>
</div>
<div id="S3.SS2.SSS2.p2" class="ltx_para">
<p id="S3.SS2.SSS2.p2.1" class="ltx_p">It is common practice, also followed by previous PaLI versions, to keep the image encoder frozen during the first multimodal pretraining stage. This is partially due to findings as in LiT <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib128" title="" class="ltx_ref">128</a>]</cite> reporting multimodal tuning of pretrained image encoders degrading their representations.
However, more recent work such as CapPa <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib106" title="" class="ltx_ref">106</a>]</cite> and LocCa <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib111" title="" class="ltx_ref">111</a>]</cite> have shown that captioning and other harder-to-learn tasks can provide valuable signal to image encoders, allowing them to learn spatial and relational understanding capabilities which contrastive models like CLIP or SigLIP typically lack.
Hence, again in the spirit of learning more skills during pretraining, we depart from common practice and do not freeze the image encoder.
However, the challenges outlined in LiT remain.
In order to avoid destructive supervision signal from the initially unaligned language model,
we use a slow linear warm-up for the image encoder’s learning-rate (Figure <a href="#S3.F3" title="Figure 3 ‣ 3.2.6 Other pretraining details ‣ 3.2 Pretraining ‣ 3 Model ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>), which ensures that the image encoder’s quality is not deteriorated from the initially misaligned gradients coming through the LLM.</p>
</div>
<div id="S3.SS2.SSS2.p3" class="ltx_para">
<p id="S3.SS2.SSS2.p3.2" class="ltx_p">We train Stage1 at resolution 224px (hence, <math id="S3.SS2.SSS2.p3.1.m1.1" class="ltx_Math" alttext="N_{\text{img}}=256" display="inline"><semantics id="S3.SS2.SSS2.p3.1.m1.1a"><mrow id="S3.SS2.SSS2.p3.1.m1.1.1" xref="S3.SS2.SSS2.p3.1.m1.1.1.cmml"><msub id="S3.SS2.SSS2.p3.1.m1.1.1.2" xref="S3.SS2.SSS2.p3.1.m1.1.1.2.cmml"><mi id="S3.SS2.SSS2.p3.1.m1.1.1.2.2" xref="S3.SS2.SSS2.p3.1.m1.1.1.2.2.cmml">N</mi><mtext id="S3.SS2.SSS2.p3.1.m1.1.1.2.3" xref="S3.SS2.SSS2.p3.1.m1.1.1.2.3a.cmml">img</mtext></msub><mo id="S3.SS2.SSS2.p3.1.m1.1.1.1" xref="S3.SS2.SSS2.p3.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS2.p3.1.m1.1.1.3" xref="S3.SS2.SSS2.p3.1.m1.1.1.3.cmml">256</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p3.1.m1.1b"><apply id="S3.SS2.SSS2.p3.1.m1.1.1.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1"><eq id="S3.SS2.SSS2.p3.1.m1.1.1.1.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1.1"></eq><apply id="S3.SS2.SSS2.p3.1.m1.1.1.2.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p3.1.m1.1.1.2.1.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS2.p3.1.m1.1.1.2.2.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1.2.2">𝑁</ci><ci id="S3.SS2.SSS2.p3.1.m1.1.1.2.3a.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS2.p3.1.m1.1.1.2.3.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1.2.3">img</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS2.p3.1.m1.1.1.3.cmml" xref="S3.SS2.SSS2.p3.1.m1.1.1.3">256</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p3.1.m1.1c">N_{\text{img}}=256</annotation></semantics></math> image tokens) and sequence length <math id="S3.SS2.SSS2.p3.2.m2.1" class="ltx_Math" alttext="N_{\text{txt}}=128" display="inline"><semantics id="S3.SS2.SSS2.p3.2.m2.1a"><mrow id="S3.SS2.SSS2.p3.2.m2.1.1" xref="S3.SS2.SSS2.p3.2.m2.1.1.cmml"><msub id="S3.SS2.SSS2.p3.2.m2.1.1.2" xref="S3.SS2.SSS2.p3.2.m2.1.1.2.cmml"><mi id="S3.SS2.SSS2.p3.2.m2.1.1.2.2" xref="S3.SS2.SSS2.p3.2.m2.1.1.2.2.cmml">N</mi><mtext id="S3.SS2.SSS2.p3.2.m2.1.1.2.3" xref="S3.SS2.SSS2.p3.2.m2.1.1.2.3a.cmml">txt</mtext></msub><mo id="S3.SS2.SSS2.p3.2.m2.1.1.1" xref="S3.SS2.SSS2.p3.2.m2.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS2.p3.2.m2.1.1.3" xref="S3.SS2.SSS2.p3.2.m2.1.1.3.cmml">128</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p3.2.m2.1b"><apply id="S3.SS2.SSS2.p3.2.m2.1.1.cmml" xref="S3.SS2.SSS2.p3.2.m2.1.1"><eq id="S3.SS2.SSS2.p3.2.m2.1.1.1.cmml" xref="S3.SS2.SSS2.p3.2.m2.1.1.1"></eq><apply id="S3.SS2.SSS2.p3.2.m2.1.1.2.cmml" xref="S3.SS2.SSS2.p3.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS2.p3.2.m2.1.1.2.1.cmml" xref="S3.SS2.SSS2.p3.2.m2.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS2.p3.2.m2.1.1.2.2.cmml" xref="S3.SS2.SSS2.p3.2.m2.1.1.2.2">𝑁</ci><ci id="S3.SS2.SSS2.p3.2.m2.1.1.2.3a.cmml" xref="S3.SS2.SSS2.p3.2.m2.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS2.p3.2.m2.1.1.2.3.cmml" xref="S3.SS2.SSS2.p3.2.m2.1.1.2.3">txt</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS2.p3.2.m2.1.1.3.cmml" xref="S3.SS2.SSS2.p3.2.m2.1.1.3">128</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p3.2.m2.1c">N_{\text{txt}}=128</annotation></semantics></math> for a total of 1 billion examples.
While we provide an ablation in Section <a href="#S5.SS1" title="5.1 Multimodal pretraining duration ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.1</span></a> showing that a 10x to 30x shorter Stage1 still provides good results on popular benchmarks,
we wish to embue as much visual knowledge to the base model as possible, and cover a broad set of concepts, cultures, and languages <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib89" title="" class="ltx_ref">89</a>, <a href="#bib.bib82" title="" class="ltx_ref">82</a>]</cite>.</p>
</div>
</section>
<section id="S3.SS2.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.2.3 </span>Stage2: Resolution increase</h4>

<div id="S3.SS2.SSS3.p1" class="ltx_para">
<p id="S3.SS2.SSS3.p1.3" class="ltx_p">The model resulting from Stage1 is already a useful base model for many tasks (see example images in Appendix <a href="#A2" title="Appendix B Tasks ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>).
However, it only understands images at <math id="S3.SS2.SSS3.p1.1.m1.1" class="ltx_Math" alttext="224\times 224" display="inline"><semantics id="S3.SS2.SSS3.p1.1.m1.1a"><mrow id="S3.SS2.SSS3.p1.1.m1.1.1" xref="S3.SS2.SSS3.p1.1.m1.1.1.cmml"><mn id="S3.SS2.SSS3.p1.1.m1.1.1.2" xref="S3.SS2.SSS3.p1.1.m1.1.1.2.cmml">224</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS3.p1.1.m1.1.1.1" xref="S3.SS2.SSS3.p1.1.m1.1.1.1.cmml">×</mo><mn id="S3.SS2.SSS3.p1.1.m1.1.1.3" xref="S3.SS2.SSS3.p1.1.m1.1.1.3.cmml">224</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS3.p1.1.m1.1b"><apply id="S3.SS2.SSS3.p1.1.m1.1.1.cmml" xref="S3.SS2.SSS3.p1.1.m1.1.1"><times id="S3.SS2.SSS3.p1.1.m1.1.1.1.cmml" xref="S3.SS2.SSS3.p1.1.m1.1.1.1"></times><cn type="integer" id="S3.SS2.SSS3.p1.1.m1.1.1.2.cmml" xref="S3.SS2.SSS3.p1.1.m1.1.1.2">224</cn><cn type="integer" id="S3.SS2.SSS3.p1.1.m1.1.1.3.cmml" xref="S3.SS2.SSS3.p1.1.m1.1.1.3">224</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS3.p1.1.m1.1c">224\times 224</annotation></semantics></math> pixel resolution, which is too small for several tasks.
For instance, detection and segmentation of smaller objects, and tasks related to reading smaller texts such as charts, infographics, or documents, all strongly benefit from higher resolution (see Table <a href="#S4.T1" title="Table 1 ‣ 4 Results ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>).
Hence, we train two further model checkpoints for increased resolution, first to <math id="S3.SS2.SSS3.p1.2.m2.1" class="ltx_Math" alttext="448\times 448" display="inline"><semantics id="S3.SS2.SSS3.p1.2.m2.1a"><mrow id="S3.SS2.SSS3.p1.2.m2.1.1" xref="S3.SS2.SSS3.p1.2.m2.1.1.cmml"><mn id="S3.SS2.SSS3.p1.2.m2.1.1.2" xref="S3.SS2.SSS3.p1.2.m2.1.1.2.cmml">448</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS3.p1.2.m2.1.1.1" xref="S3.SS2.SSS3.p1.2.m2.1.1.1.cmml">×</mo><mn id="S3.SS2.SSS3.p1.2.m2.1.1.3" xref="S3.SS2.SSS3.p1.2.m2.1.1.3.cmml">448</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS3.p1.2.m2.1b"><apply id="S3.SS2.SSS3.p1.2.m2.1.1.cmml" xref="S3.SS2.SSS3.p1.2.m2.1.1"><times id="S3.SS2.SSS3.p1.2.m2.1.1.1.cmml" xref="S3.SS2.SSS3.p1.2.m2.1.1.1"></times><cn type="integer" id="S3.SS2.SSS3.p1.2.m2.1.1.2.cmml" xref="S3.SS2.SSS3.p1.2.m2.1.1.2">448</cn><cn type="integer" id="S3.SS2.SSS3.p1.2.m2.1.1.3.cmml" xref="S3.SS2.SSS3.p1.2.m2.1.1.3">448</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS3.p1.2.m2.1c">448\times 448</annotation></semantics></math> and then to <math id="S3.SS2.SSS3.p1.3.m3.1" class="ltx_Math" alttext="896\times 896" display="inline"><semantics id="S3.SS2.SSS3.p1.3.m3.1a"><mrow id="S3.SS2.SSS3.p1.3.m3.1.1" xref="S3.SS2.SSS3.p1.3.m3.1.1.cmml"><mn id="S3.SS2.SSS3.p1.3.m3.1.1.2" xref="S3.SS2.SSS3.p1.3.m3.1.1.2.cmml">896</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.SSS3.p1.3.m3.1.1.1" xref="S3.SS2.SSS3.p1.3.m3.1.1.1.cmml">×</mo><mn id="S3.SS2.SSS3.p1.3.m3.1.1.3" xref="S3.SS2.SSS3.p1.3.m3.1.1.3.cmml">896</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS3.p1.3.m3.1b"><apply id="S3.SS2.SSS3.p1.3.m3.1.1.cmml" xref="S3.SS2.SSS3.p1.3.m3.1.1"><times id="S3.SS2.SSS3.p1.3.m3.1.1.1.cmml" xref="S3.SS2.SSS3.p1.3.m3.1.1.1"></times><cn type="integer" id="S3.SS2.SSS3.p1.3.m3.1.1.2.cmml" xref="S3.SS2.SSS3.p1.3.m3.1.1.2">896</cn><cn type="integer" id="S3.SS2.SSS3.p1.3.m3.1.1.3.cmml" xref="S3.SS2.SSS3.p1.3.m3.1.1.3">896</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS3.p1.3.m3.1c">896\times 896</annotation></semantics></math> pixel resolution.</p>
</div>
<div id="S3.SS2.SSS3.p2" class="ltx_para">
<p id="S3.SS2.SSS3.p2.1" class="ltx_p">Since stage1 took care of providing the model with a broad set of knowledge and skill, stage2 can focus on extending the model’s ability to parse higher-resolution images.
We thus run Stage2 with fewer total examples, while increasing the cost and information density of each example.
For resolution 448, we train for an additional 50 M examples, and for resolution 896, we add another 10 M examples.</p>
</div>
<div id="S3.SS2.SSS3.p3" class="ltx_para">
<p id="S3.SS2.SSS3.p3.1" class="ltx_p">For simplicity, Stage2 consists of the exact same mixture of tasks and datasets as Stage1, but with significantly increased sampling of tasks that require high resolution.
Additionally, these upweighted tasks all can be modified to provide much longer suffix sequence lengths.
For instance, for OCR tasks, we can simply request the model to read <em id="S3.SS2.SSS3.p3.1.1" class="ltx_emph ltx_font_italic">all</em> text on the image in left-to-right, top-to-bottom order.
For detection and segmentation tasks, we can request the model to detect or segment <em id="S3.SS2.SSS3.p3.1.2" class="ltx_emph ltx_font_italic">all</em> objects for which annotation is provided.
Hence, we also increase the text sequence length to <math id="S3.SS2.SSS3.p3.1.m1.1" class="ltx_Math" alttext="N_{\text{txt}}=512" display="inline"><semantics id="S3.SS2.SSS3.p3.1.m1.1a"><mrow id="S3.SS2.SSS3.p3.1.m1.1.1" xref="S3.SS2.SSS3.p3.1.m1.1.1.cmml"><msub id="S3.SS2.SSS3.p3.1.m1.1.1.2" xref="S3.SS2.SSS3.p3.1.m1.1.1.2.cmml"><mi id="S3.SS2.SSS3.p3.1.m1.1.1.2.2" xref="S3.SS2.SSS3.p3.1.m1.1.1.2.2.cmml">N</mi><mtext id="S3.SS2.SSS3.p3.1.m1.1.1.2.3" xref="S3.SS2.SSS3.p3.1.m1.1.1.2.3a.cmml">txt</mtext></msub><mo id="S3.SS2.SSS3.p3.1.m1.1.1.1" xref="S3.SS2.SSS3.p3.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS3.p3.1.m1.1.1.3" xref="S3.SS2.SSS3.p3.1.m1.1.1.3.cmml">512</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS3.p3.1.m1.1b"><apply id="S3.SS2.SSS3.p3.1.m1.1.1.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1"><eq id="S3.SS2.SSS3.p3.1.m1.1.1.1.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.1"></eq><apply id="S3.SS2.SSS3.p3.1.m1.1.1.2.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS3.p3.1.m1.1.1.2.1.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS3.p3.1.m1.1.1.2.2.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.2.2">𝑁</ci><ci id="S3.SS2.SSS3.p3.1.m1.1.1.2.3a.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS3.p3.1.m1.1.1.2.3.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.2.3">txt</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS3.p3.1.m1.1.1.3.cmml" xref="S3.SS2.SSS3.p3.1.m1.1.1.3">512</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS3.p3.1.m1.1c">N_{\text{txt}}=512</annotation></semantics></math> tokens.</p>
</div>
<div id="S3.SS2.SSS3.p4" class="ltx_para">
<p id="S3.SS2.SSS3.p4.1" class="ltx_p">While PaLI has always had this resolution increasing stage, and for image classification the importance of resolution is long known <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib105" title="" class="ltx_ref">105</a>, <a href="#bib.bib53" title="" class="ltx_ref">53</a>]</cite>, several recent works <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib110" title="" class="ltx_ref">110</a>, <a href="#bib.bib117" title="" class="ltx_ref">117</a>, <a href="#bib.bib78" title="" class="ltx_ref">78</a>]</cite> have raised the importance of resolution in VLMs too.
We add to this body of knowledge by providing several ablation studies regarding Stage2 in Section <a href="#S5.SS7" title="5.7 Image resolution ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.7</span></a>.</p>
</div>
</section>
<section id="S3.SS2.SSS4" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.2.4 </span>Stage3: Transfer</h4>

<div id="S3.SS2.SSS4.p1" class="ltx_para">
<p id="S3.SS2.SSS4.p1.1" class="ltx_p">The result of Stages 1 and 2 is a family of three PaliGemma checkpoints, at 224px, 448px, and 896px resolution,
which are pre-equipped with broad visual knowledge.
However, these checkpoints are not “user (or benchmark) friendly” as their pretraining has focused solely on density of learning signal, as opposed to usable interface.</p>
</div>
<div id="S3.SS2.SSS4.p2" class="ltx_para">
<p id="S3.SS2.SSS4.p2.1" class="ltx_p">These base models need to be transferred to serve their intended final purpose.
That could take the form of fine-tuning on a specific, specialized task, such as COCO Captions, Remote Sensing VQA, Video Captioning, or InfographicQA. Adapt to new inputs such as multiple images (NLVR2) or bounding boxes draw in the image (WidgetCap).
Or it could take the form of instruction <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib67" title="" class="ltx_ref">67</a>]</cite> or even chat <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite> tuning.</p>
</div>
<div id="S3.SS2.SSS4.p3" class="ltx_para">
<p id="S3.SS2.SSS4.p3.1" class="ltx_p">To show the effectiveness of the base models, we transfer them to a wide range of individual academic benchmarks, using a simple unified transfer recipe with few hyper-parameters.
And to showcase the versatility beyond academic tasks, we also provide a “mix” transfer checkpoint, which transfers to a subset of these tasks at the same time, along with detailed captioning and long question-answering data.
While this is not instruction tuning, it is a step in that direction.</p>
</div>
<div id="S3.SS2.SSS4.p4" class="ltx_para">
<p id="S3.SS2.SSS4.p4.1" class="ltx_p">We also transfer PaliGemma to tasks which take multiple images as input.
NLVR2 is one such task, which asks one question about two images, and requires looking at both to give the correct answer.
Other such tasks are standard short-video understanding tasks subsampled to 16 frames.
In all these cases, we follow PaLI-3 and encode each image separately, then concatenate the image tokens without any special separator or embedding tokens.
Thus, 16 frames at 224px resolution result in <math id="S3.SS2.SSS4.p4.1.m1.1" class="ltx_Math" alttext="N_{\text{img}}=4096" display="inline"><semantics id="S3.SS2.SSS4.p4.1.m1.1a"><mrow id="S3.SS2.SSS4.p4.1.m1.1.1" xref="S3.SS2.SSS4.p4.1.m1.1.1.cmml"><msub id="S3.SS2.SSS4.p4.1.m1.1.1.2" xref="S3.SS2.SSS4.p4.1.m1.1.1.2.cmml"><mi id="S3.SS2.SSS4.p4.1.m1.1.1.2.2" xref="S3.SS2.SSS4.p4.1.m1.1.1.2.2.cmml">N</mi><mtext id="S3.SS2.SSS4.p4.1.m1.1.1.2.3" xref="S3.SS2.SSS4.p4.1.m1.1.1.2.3a.cmml">img</mtext></msub><mo id="S3.SS2.SSS4.p4.1.m1.1.1.1" xref="S3.SS2.SSS4.p4.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS2.SSS4.p4.1.m1.1.1.3" xref="S3.SS2.SSS4.p4.1.m1.1.1.3.cmml">4096</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS4.p4.1.m1.1b"><apply id="S3.SS2.SSS4.p4.1.m1.1.1.cmml" xref="S3.SS2.SSS4.p4.1.m1.1.1"><eq id="S3.SS2.SSS4.p4.1.m1.1.1.1.cmml" xref="S3.SS2.SSS4.p4.1.m1.1.1.1"></eq><apply id="S3.SS2.SSS4.p4.1.m1.1.1.2.cmml" xref="S3.SS2.SSS4.p4.1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS4.p4.1.m1.1.1.2.1.cmml" xref="S3.SS2.SSS4.p4.1.m1.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS4.p4.1.m1.1.1.2.2.cmml" xref="S3.SS2.SSS4.p4.1.m1.1.1.2.2">𝑁</ci><ci id="S3.SS2.SSS4.p4.1.m1.1.1.2.3a.cmml" xref="S3.SS2.SSS4.p4.1.m1.1.1.2.3"><mtext mathsize="70%" id="S3.SS2.SSS4.p4.1.m1.1.1.2.3.cmml" xref="S3.SS2.SSS4.p4.1.m1.1.1.2.3">img</mtext></ci></apply><cn type="integer" id="S3.SS2.SSS4.p4.1.m1.1.1.3.cmml" xref="S3.SS2.SSS4.p4.1.m1.1.1.3">4096</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS4.p4.1.m1.1c">N_{\text{img}}=4096</annotation></semantics></math> image tokens, the same amount as a single image at 896px resolution.</p>
</div>
<div id="S3.SS2.SSS4.p5" class="ltx_para">
<p id="S3.SS2.SSS4.p5.1" class="ltx_p">For all transfers, we perform fine-tuning of all the model parameters.
The hyper-parameters we modify per-task are the following, in decreasing order of importance:</p>
<ul id="S3.I3" class="ltx_itemize">
<li id="S3.I3.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I3.i1.p1" class="ltx_para">
<p id="S3.I3.i1.p1.1" class="ltx_p">Resolution (<em id="S3.I3.i1.p1.1.1" class="ltx_emph ltx_font_italic">i.e</em>.<span id="S3.I3.i1.p1.1.2" class="ltx_text"></span> checkpoint): <span id="S3.I3.i1.p1.1.3" class="ltx_text ltx_font_bold">224</span>, 448, 896.</p>
</div>
</li>
<li id="S3.I3.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I3.i2.p1" class="ltx_para">
<p id="S3.I3.i2.p1.1" class="ltx_p">Epochs: <span id="S3.I3.i2.p1.1.1" class="ltx_text ltx_font_bold">1, 3, 10</span>, 30, 100.</p>
</div>
</li>
<li id="S3.I3.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I3.i3.p1" class="ltx_para">
<p id="S3.I3.i3.p1.1" class="ltx_p">Learning-rate: 3e-5, <span id="S3.I3.i3.p1.1.1" class="ltx_text ltx_font_bold">1e-5</span>, 3e-6.</p>
</div>
</li>
<li id="S3.I3.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I3.i4.p1" class="ltx_para">
<p id="S3.I3.i4.p1.1" class="ltx_p">Label-smoothing: <span id="S3.I3.i4.p1.1.1" class="ltx_text ltx_font_bold">0.0</span>, 0.1, 0.3.</p>
</div>
</li>
<li id="S3.I3.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I3.i5.p1" class="ltx_para">
<p id="S3.I3.i5.p1.1" class="ltx_p">Dropout in the LLM: <span id="S3.I3.i5.p1.1.1" class="ltx_text ltx_font_bold">0.0</span>, 0.1, 0.3.</p>
</div>
</li>
<li id="S3.I3.i6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I3.i6.p1" class="ltx_para">
<p id="S3.I3.i6.p1.1" class="ltx_p">Weight decay: <span id="S3.I3.i6.p1.1.1" class="ltx_text ltx_font_bold">0.0</span> or 0.1 <math id="S3.I3.i6.p1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S3.I3.i6.p1.1.m1.1a"><mo id="S3.I3.i6.p1.1.m1.1.1" xref="S3.I3.i6.p1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S3.I3.i6.p1.1.m1.1b"><times id="S3.I3.i6.p1.1.m1.1.1.cmml" xref="S3.I3.i6.p1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S3.I3.i6.p1.1.m1.1c">\times</annotation></semantics></math> learning-rate.</p>
</div>
</li>
<li id="S3.I3.i7" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I3.i7.p1" class="ltx_para">
<p id="S3.I3.i7.p1.1" class="ltx_p">Freeze ViT: <span id="S3.I3.i7.p1.1.1" class="ltx_text ltx_font_bold">false</span>, true.</p>
</div>
</li>
<li id="S3.I3.i8" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S3.I3.i8.p1" class="ltx_para">
<p id="S3.I3.i8.p1.1" class="ltx_p">Beam-search may benefit captioning.</p>
</div>
</li>
</ul>
<p id="S3.SS2.SSS4.p5.2" class="ltx_p">The above are typical values we suggest exploring, with the recommended initial attempt value in bold.
We provide the best setting for each individual task in Appendix <a href="#A10" title="Appendix J Hyper-parameters ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">J</span></a>.
We study the sensitivity to transfer hyper-parameters in Section <a href="#S6.SS2" title="6.2 Transfer hyper-parameter sensitivity ‣ 6 Transferability ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6.2</span></a>, and the “transferability” in general in Section <a href="#S6" title="6 Transferability ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>, showing that good results can be achieved with the aforementioned initial attempt values.</p>
</div>
</section>
<section id="S3.SS2.SSS5" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.2.5 </span>Pretraining task mixture</h4>

<div id="S3.SS2.SSS5.p1" class="ltx_para">
<p id="S3.SS2.SSS5.p1.1" class="ltx_p">Just like for previous PaLI models, the pretraining (Stage1 and Stage2) is designed to result in a model that transfers well, not necessarily a model that is usable out of the box (“0 shot”).
The intuition here is that we want a mix of tasks which force the model to acquire a broad range of “skills”.
We prefix each task with its unique prefix to avoid conflicting learning signals across skills <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib14" title="" class="ltx_ref">14</a>]</cite>.
At transfer time (Stage3), the model then merely needs to recognize which skill is useful for the task, and rewire itself to use that while following the output syntax and vocabulary of the task.
In our experience, these can all be done relatively quickly and based on few examples (Section <a href="#S6.SS3" title="6.3 Transfer with limited examples ‣ 6 Transferability ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6.3</span></a>).
We do not use any of our transfer datasets during pretraining, and furthermore remove all near-duplicates of their images from the pretraining datasets <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib53" title="" class="ltx_ref">53</a>]</cite>.</p>
</div>
<div id="S3.SS2.SSS5.p2" class="ltx_para">
<p id="S3.SS2.SSS5.p2.1" class="ltx_p">Largely following previous PaLI works, these are the pretraining tasks:</p>
</div>
<div id="S3.SS2.SSS5.p3" class="ltx_para">
<p id="S3.SS2.SSS5.p3.1" class="ltx_p"><span id="S3.SS2.SSS5.p3.1.1" class="ltx_text ltx_font_typewriter">caption {lang}
<br class="ltx_break"></span>We include the simple captioning objective on various datasets, including WebLI in over 100 languages, and CC3M-35L. Previous PaLIs use an encoder-decoder language model with the SplitCap objective, however for PaliGemma with the decoder-only language model, plain captioning is a more informative and simpler objective.</p>
</div>
<div id="S3.SS2.SSS5.p4" class="ltx_para">
<p id="S3.SS2.SSS5.p4.1" class="ltx_p"><span id="S3.SS2.SSS5.p4.1.1" class="ltx_text ltx_font_typewriter">ocr
<br class="ltx_break"></span>Concatenation (in raster order) of all text on the image transcribed by a public OCR system. Potentially skipping random snippets of OCR in order to fit sequence length without biasing recognition towards the beginning of raster order.</p>
</div>
<div id="S3.SS2.SSS5.p5" class="ltx_para">
<p id="S3.SS2.SSS5.p5.1" class="ltx_p"><span id="S3.SS2.SSS5.p5.1.1" class="ltx_text ltx_font_typewriter">answer en {question}
<br class="ltx_break"></span>Generated VQA on CC3M-35L following <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite> with questions in 35 languages but English answers.
Additionally, English-only object-centric questions on OpenImages following <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib88" title="" class="ltx_ref">88</a>]</cite>:
<br class="ltx_break">listing: <span id="S3.SS2.SSS5.p5.1.2" class="ltx_text ltx_font_typewriter">What objects are in the image?</span>,
<br class="ltx_break">presence: <span id="S3.SS2.SSS5.p5.1.3" class="ltx_text ltx_font_typewriter">Is {thing} in the image?</span>,
<br class="ltx_break">multi-object presence: <span id="S3.SS2.SSS5.p5.1.4" class="ltx_text ltx_font_typewriter">Which of {thing}, {thing}... are in the image?</span>,
<br class="ltx_break">and newly, counting: <span id="S3.SS2.SSS5.p5.1.5" class="ltx_text ltx_font_typewriter">How many {thing}?</span>.</p>
</div>
<div id="S3.SS2.SSS5.p6" class="ltx_para">
<p id="S3.SS2.SSS5.p6.1" class="ltx_p"><span id="S3.SS2.SSS5.p6.1.1" class="ltx_text ltx_font_typewriter">question {lang} {English answer}
<br class="ltx_break"></span>Generated VQG on CC3M-35L following <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite> generating questions in 35 languages, for a given English answer.</p>
</div>
<div id="S3.SS2.SSS5.p7" class="ltx_para">
<p id="S3.SS2.SSS5.p7.1" class="ltx_p"><span id="S3.SS2.SSS5.p7.1.1" class="ltx_text ltx_font_typewriter">detect {thing} ; {thing} ; ...
<br class="ltx_break"></span>Multi-object detection similar to Pix2Seq <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib21" title="" class="ltx_ref">21</a>]</cite> on generated open-world data via pseudo-labeling as described in OWL-ViTv2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib80" title="" class="ltx_ref">80</a>]</cite>.</p>
</div>
<div id="S3.SS2.SSS5.p8" class="ltx_para">
<p id="S3.SS2.SSS5.p8.1" class="ltx_p"><span id="S3.SS2.SSS5.p8.1.1" class="ltx_text ltx_font_typewriter">segment {thing} ; {thing} ; ...
<br class="ltx_break"></span>Multi-object instance segmentation as in PaLI-3 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib24" title="" class="ltx_ref">24</a>]</cite> on generated open-world data similar to OWL-ViTv2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib80" title="" class="ltx_ref">80</a>]</cite> and SAM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib52" title="" class="ltx_ref">52</a>]</cite>.</p>
</div>
<div id="S3.SS2.SSS5.p9" class="ltx_para">
<p id="S3.SS2.SSS5.p9.1" class="ltx_p"><span id="S3.SS2.SSS5.p9.1.1" class="ltx_text ltx_font_typewriter">caption &lt;ymin&gt;&lt;xmin&gt;&lt;ymax&gt;&lt;xmax&gt;
<br class="ltx_break"></span>Grounded captioning of what is in the box, following LocCa <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib111" title="" class="ltx_ref">111</a>]</cite>. The box is indicated by the same location tokens as used in detection and segmentation: normalized image coordinates binned to 1024 tokens.</p>
</div>
<div id="S3.SS2.SSS5.p10" class="ltx_para">
<p id="S3.SS2.SSS5.p10.1" class="ltx_p">Notably distinct from the widely used LLaVa’s GPT-4 generated instruction following data, none of PaliGemma’s pretraining tasks is the output of a larger commercial VLM.</p>
</div>
<div id="S3.SS2.SSS5.p11" class="ltx_para">
<p id="S3.SS2.SSS5.p11.1" class="ltx_p">Finally, we believe that it is important to detect and remove all images in our pretraining datasets which are near-duplicates of images in the transfer tasks we evaluate in this report <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib53" title="" class="ltx_ref">53</a>]</cite>, as well as a few more popular computer vision benchmarks.
Doing so, we more accurately capture PaliGemma’s capability to transfer to new tasks.</p>
</div>
</section>
<section id="S3.SS2.SSS6" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.2.6 </span>Other pretraining details</h4>

<figure id="S3.F3" class="ltx_figure"><img src="/html/2407.07726/assets/x3.png" id="S3.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="231" height="80" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Learning-rate schedule across stages.</figcaption>
</figure>
<div id="S3.SS2.SSS6.p1" class="ltx_para">
<p id="S3.SS2.SSS6.p1.1" class="ltx_p">Throughout pretraining, we use an "infinite" learning-rate schedule following <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib127" title="" class="ltx_ref">127</a>]</cite>, which provides a straightforward way of chaining several stages without decaying the learning-rate between them.
Figure <a href="#S3.F3" title="Figure 3 ‣ 3.2.6 Other pretraining details ‣ 3.2 Pretraining ‣ 3 Model ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> shows the full schedule: pretraining is one continuous rsqrt curve for all stages.
The transfer can then act as a cooldown, fully annealing the learning rate.
We recommend transferring with a simple setup that tunes the full model using a cosine learning-rate schedule with a short linear warm-up and decaying to zero.
This is not well represented by Figure <a href="#S3.F3" title="Figure 3 ‣ 3.2.6 Other pretraining details ‣ 3.2 Pretraining ‣ 3 Model ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> due to its comparatively short duration.</p>
</div>
<div id="S3.SS2.SSS6.p2" class="ltx_para">
<p id="S3.SS2.SSS6.p2.1" class="ltx_p">The model was entirely trained in the open-source <span id="S3.SS2.SSS6.p2.1.1" class="ltx_text ltx_font_typewriter">big_vision</span> codebase <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib12" title="" class="ltx_ref">12</a>]</cite> on Cloud TPUv5e <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib36" title="" class="ltx_ref">36</a>]</cite>. However, some of the pretraining datasets remain private.
During training, we partition data, as well as model parameters and optimizer state (Zero-DP style <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib92" title="" class="ltx_ref">92</a>]</cite>) across all available devices using JAX <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite> with GSPMD <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib121" title="" class="ltx_ref">121</a>]</cite>. This fully-sharded data-parallel (FSDP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib132" title="" class="ltx_ref">132</a>]</cite>) sharding strategy is achieved by constructing global arrays and annotating the sharding accordingly, with the XLA compiler <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib93" title="" class="ltx_ref">93</a>]</cite> taking care of the concrete implementation of the computation and communication between devices.
We measured a model FLOPS utilization (MFU) of 55%, resulting in 5189 tokens/second/device. Model parameters and optimizer state are kept in float32 to guarantee stable training, but we verified that inference works just as well with bfloat16 model parameters.</p>
</div>
<div id="S3.SS2.SSS6.p3" class="ltx_para">
<p id="S3.SS2.SSS6.p3.1" class="ltx_p">One training run of the final PaliGemma model using TPUv5e-256 takes slightly less than 3 days for Stage1 and 15h for each Stage2. Stage1 sees slightly less than 350 B tokens, and both Stage2 combined about 90 B tokens. Transfers take between 20min and 10h on TPUv3-32, depending on the task.</p>
</div>
<div id="S3.SS2.SSS6.p4" class="ltx_para">
<p id="S3.SS2.SSS6.p4.1" class="ltx_p">In order to avoid model brittleness to different image processing details in different frameworks, we randomize the image preprocessing details such as resize method, JPEG encoding, and apply very slight <span id="S3.SS2.SSS6.p4.1.1" class="ltx_text ltx_font_typewriter">inception_crop</span>.</p>
</div>
</section>
</section>
</section>
<section id="S4" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Results</h2>

<figure id="S4.T1" class="ltx_table">
<figcaption class="ltx_caption ltx_centering" style="font-size:70%;"><span class="ltx_tag ltx_tag_table">Table 1: </span>Results (1 random run of 5) obtained with PaliGemma.
Tasks marked with <math id="S4.T1.2.m1.1" class="ltx_Math" alttext="\llcorner" display="inline"><semantics id="S4.T1.2.m1.1b"><mi mathvariant="normal" id="S4.T1.2.m1.1.1" xref="S4.T1.2.m1.1.1.cmml">⌞</mi><annotation-xml encoding="MathML-Content" id="S4.T1.2.m1.1c"><ci id="S4.T1.2.m1.1.1.cmml" xref="S4.T1.2.m1.1.1">⌞</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.2.m1.1d">\llcorner</annotation></semantics></math> indicate zero-shot evaluation of the transferred model above. Where numbers depend on server submissions we report standard deviation from validation splits. Highlighted rows indicate resolution sensitive tasks. Per-task details and hyper-parameters are in Appendix <a href="#A2" title="Appendix B Tasks ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a> and <a href="#A10" title="Appendix J Hyper-parameters ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">J</span></a>.
</figcaption>
<table id="S4.T1.10" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T1.10.9.1" class="ltx_tr">
<th id="S4.T1.10.9.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt"><span id="S4.T1.10.9.1.1.1" class="ltx_text" style="font-size:70%;">Task</span></th>
<td id="S4.T1.10.9.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T1.10.9.1.2.1" class="ltx_text" style="font-size:70%;">224px</span></td>
<td id="S4.T1.10.9.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T1.10.9.1.3.1" class="ltx_text" style="font-size:70%;">448px</span></td>
<td id="S4.T1.10.9.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T1.10.9.1.4.1" class="ltx_text" style="font-size:70%;">896px</span></td>
</tr>
<tr id="S4.T1.10.10.2" class="ltx_tr">
<th id="S4.T1.10.10.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="4"><span id="S4.T1.10.10.2.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Image captioning</span></th>
</tr>
<tr id="S4.T1.10.11.3" class="ltx_tr">
<th id="S4.T1.10.11.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.11.3.1.1" class="ltx_text" style="font-size:70%;">COCOcap</span></th>
<td id="S4.T1.10.11.3.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.11.3.2.1" class="ltx_text" style="font-size:70%;">141.9 </span><span id="S4.T1.10.11.3.2.2" class="ltx_text" style="font-size:50%;color:#808080;">±0.3</span>
</td>
<td id="S4.T1.10.11.3.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.11.3.3.1" class="ltx_text" style="font-size:70%;">144.6 </span><span id="S4.T1.10.11.3.3.2" class="ltx_text" style="font-size:50%;color:#808080;">±0.5</span>
</td>
<td id="S4.T1.10.11.3.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.11.3.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.3.1" class="ltx_tr">
<th id="S4.T1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<math id="S4.T1.3.1.1.m1.1" class="ltx_Math" alttext="\llcorner" display="inline"><semantics id="S4.T1.3.1.1.m1.1a"><mi mathsize="70%" mathvariant="normal" id="S4.T1.3.1.1.m1.1.1" xref="S4.T1.3.1.1.m1.1.1.cmml">⌞</mi><annotation-xml encoding="MathML-Content" id="S4.T1.3.1.1.m1.1b"><ci id="S4.T1.3.1.1.m1.1.1.cmml" xref="S4.T1.3.1.1.m1.1.1">⌞</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.3.1.1.m1.1c">\llcorner</annotation></semantics></math><span id="S4.T1.3.1.1.1" class="ltx_text" style="font-size:70%;">NoCaps</span>
</th>
<td id="S4.T1.3.1.2" class="ltx_td ltx_align_center">
<span id="S4.T1.3.1.2.1" class="ltx_text" style="font-size:70%;">121.7 </span><span id="S4.T1.3.1.2.2" class="ltx_text" style="font-size:50%;color:#808080;">±0.3</span>
</td>
<td id="S4.T1.3.1.3" class="ltx_td ltx_align_center">
<span id="S4.T1.3.1.3.1" class="ltx_text" style="font-size:70%;">123.6 </span><span id="S4.T1.3.1.3.2" class="ltx_text" style="font-size:50%;color:#808080;">±0.7</span>
</td>
<td id="S4.T1.3.1.4" class="ltx_td ltx_align_center"><span id="S4.T1.3.1.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.12.4" class="ltx_tr">
<th id="S4.T1.10.12.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.12.4.1.1" class="ltx_text" style="font-size:70%;">COCO-35L (en)</span></th>
<td id="S4.T1.10.12.4.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.12.4.2.1" class="ltx_text" style="font-size:70%;">139.2 </span><span id="S4.T1.10.12.4.2.2" class="ltx_text" style="font-size:50%;color:#808080;">±0.4</span>
</td>
<td id="S4.T1.10.12.4.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.12.4.3.1" class="ltx_text" style="font-size:70%;">141.2 </span><span id="S4.T1.10.12.4.3.2" class="ltx_text" style="font-size:50%;color:#808080;">±0.6</span>
</td>
<td id="S4.T1.10.12.4.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.12.4.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.13.5" class="ltx_tr">
<th id="S4.T1.10.13.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.13.5.1.1" class="ltx_text" style="font-size:70%;">COCO-35L (avg34)</span></th>
<td id="S4.T1.10.13.5.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.13.5.2.1" class="ltx_text" style="font-size:70%;">113.7 </span><span id="S4.T1.10.13.5.2.2" class="ltx_text" style="font-size:50%;color:#808080;">±0.3</span>
</td>
<td id="S4.T1.10.13.5.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.13.5.3.1" class="ltx_text" style="font-size:70%;">115.8 </span><span id="S4.T1.10.13.5.3.2" class="ltx_text" style="font-size:50%;color:#808080;">±0.1</span>
</td>
<td id="S4.T1.10.13.5.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.13.5.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.4.2" class="ltx_tr">
<th id="S4.T1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<math id="S4.T1.4.2.1.m1.1" class="ltx_Math" alttext="\llcorner" display="inline"><semantics id="S4.T1.4.2.1.m1.1a"><mi mathsize="70%" mathvariant="normal" id="S4.T1.4.2.1.m1.1.1" xref="S4.T1.4.2.1.m1.1.1.cmml">⌞</mi><annotation-xml encoding="MathML-Content" id="S4.T1.4.2.1.m1.1b"><ci id="S4.T1.4.2.1.m1.1.1.cmml" xref="S4.T1.4.2.1.m1.1.1">⌞</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.4.2.1.m1.1c">\llcorner</annotation></semantics></math><span id="S4.T1.4.2.1.1" class="ltx_text" style="font-size:70%;">XM3600 (en)</span>
</th>
<td id="S4.T1.4.2.2" class="ltx_td ltx_align_center">
<span id="S4.T1.4.2.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.4.2.2.2" class="ltx_text" style="font-size:70%;">78.0 </span><span id="S4.T1.4.2.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.8</span>
</td>
<td id="S4.T1.4.2.3" class="ltx_td ltx_align_center">
<span id="S4.T1.4.2.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.4.2.3.2" class="ltx_text" style="font-size:70%;">80.0 </span><span id="S4.T1.4.2.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.3</span>
</td>
<td id="S4.T1.4.2.4" class="ltx_td ltx_align_center"><span id="S4.T1.4.2.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.5.3" class="ltx_tr">
<th id="S4.T1.5.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<math id="S4.T1.5.3.1.m1.1" class="ltx_Math" alttext="\llcorner" display="inline"><semantics id="S4.T1.5.3.1.m1.1a"><mi mathsize="70%" mathvariant="normal" id="S4.T1.5.3.1.m1.1.1" xref="S4.T1.5.3.1.m1.1.1.cmml">⌞</mi><annotation-xml encoding="MathML-Content" id="S4.T1.5.3.1.m1.1b"><ci id="S4.T1.5.3.1.m1.1.1.cmml" xref="S4.T1.5.3.1.m1.1.1">⌞</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.5.3.1.m1.1c">\llcorner</annotation></semantics></math><span id="S4.T1.5.3.1.1" class="ltx_text" style="font-size:70%;">XM3600 (avg35)</span>
</th>
<td id="S4.T1.5.3.2" class="ltx_td ltx_align_center">
<span id="S4.T1.5.3.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.5.3.2.2" class="ltx_text" style="font-size:70%;">41.9 </span><span id="S4.T1.5.3.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.0</span>
</td>
<td id="S4.T1.5.3.3" class="ltx_td ltx_align_center">
<span id="S4.T1.5.3.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.5.3.3.2" class="ltx_text" style="font-size:70%;">42.4 </span><span id="S4.T1.5.3.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.1</span>
</td>
<td id="S4.T1.5.3.4" class="ltx_td ltx_align_center"><span id="S4.T1.5.3.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.14.6" class="ltx_tr">
<th id="S4.T1.10.14.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.14.6.1.1" class="ltx_text" style="font-size:70%;">Screen2Words</span></th>
<td id="S4.T1.10.14.6.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.14.6.2.1" class="ltx_text" style="font-size:70%;">117.6 </span><span id="S4.T1.10.14.6.2.2" class="ltx_text" style="font-size:50%;color:#808080;">±0.7</span>
</td>
<td id="S4.T1.10.14.6.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.14.6.3.1" class="ltx_text" style="font-size:70%;">119.6 </span><span id="S4.T1.10.14.6.3.2" class="ltx_text" style="font-size:50%;color:#808080;">±0.7</span>
</td>
<td id="S4.T1.10.14.6.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.14.6.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.15.7" class="ltx_tr" style="background-color:#F2F2F2;">
<th id="S4.T1.10.15.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.15.7.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">TextCaps</span></th>
<td id="S4.T1.10.15.7.2" class="ltx_td ltx_align_center"><span id="S4.T1.10.15.7.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">127.5 <span id="S4.T1.10.15.7.2.1.1" class="ltx_text" style="font-size:71%;color:#808080;background-color:#F2F2F2;">±1.0</span></span></td>
<td id="S4.T1.10.15.7.3" class="ltx_td ltx_align_center"><span id="S4.T1.10.15.7.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">153.9 <span id="S4.T1.10.15.7.3.1.1" class="ltx_text" style="font-size:71%;color:#808080;background-color:#F2F2F2;">±0.5</span></span></td>
<td id="S4.T1.10.15.7.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.15.7.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">-</span></td>
</tr>
<tr id="S4.T1.10.16.8" class="ltx_tr" style="background-color:#F2F2F2;">
<th id="S4.T1.10.16.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.16.8.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">SciCap</span></th>
<td id="S4.T1.10.16.8.2" class="ltx_td ltx_align_center"><span id="S4.T1.10.16.8.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">162.3 <span id="S4.T1.10.16.8.2.1.1" class="ltx_text" style="font-size:71%;color:#808080;background-color:#F2F2F2;">±0.7</span></span></td>
<td id="S4.T1.10.16.8.3" class="ltx_td ltx_align_center"><span id="S4.T1.10.16.8.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">181.5 <span id="S4.T1.10.16.8.3.1.1" class="ltx_text" style="font-size:71%;color:#808080;background-color:#F2F2F2;">±0.6</span></span></td>
<td id="S4.T1.10.16.8.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.16.8.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">-</span></td>
</tr>
<tr id="S4.T1.10.17.9" class="ltx_tr" style="background-color:#F2F2F2;">
<th id="S4.T1.10.17.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.17.9.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">WidgetCap</span></th>
<td id="S4.T1.10.17.9.2" class="ltx_td ltx_align_center"><span id="S4.T1.10.17.9.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">136.1 <span id="S4.T1.10.17.9.2.1.1" class="ltx_text" style="font-size:71%;color:#808080;background-color:#F2F2F2;">±1.4</span></span></td>
<td id="S4.T1.10.17.9.3" class="ltx_td ltx_align_center"><span id="S4.T1.10.17.9.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">148.4 <span id="S4.T1.10.17.9.3.1.1" class="ltx_text" style="font-size:71%;color:#808080;background-color:#F2F2F2;">±0.3</span></span></td>
<td id="S4.T1.10.17.9.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.17.9.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">-</span></td>
</tr>
<tr id="S4.T1.10.18.10" class="ltx_tr">
<th id="S4.T1.10.18.10.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="4"><span id="S4.T1.10.18.10.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Visual question answering</span></th>
</tr>
<tr id="S4.T1.10.19.11" class="ltx_tr">
<th id="S4.T1.10.19.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.19.11.1.1" class="ltx_text" style="font-size:70%;">VQAv2</span></th>
<td id="S4.T1.10.19.11.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.19.11.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.19.11.2.2" class="ltx_text" style="font-size:70%;">83.2 </span><span id="S4.T1.10.19.11.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.4</span>
</td>
<td id="S4.T1.10.19.11.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.19.11.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.19.11.3.2" class="ltx_text" style="font-size:70%;">85.6 </span><span id="S4.T1.10.19.11.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.2</span>
</td>
<td id="S4.T1.10.19.11.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.19.11.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.6.4" class="ltx_tr">
<th id="S4.T1.6.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<math id="S4.T1.6.4.1.m1.1" class="ltx_Math" alttext="\llcorner" display="inline"><semantics id="S4.T1.6.4.1.m1.1a"><mi mathsize="70%" mathvariant="normal" id="S4.T1.6.4.1.m1.1.1" xref="S4.T1.6.4.1.m1.1.1.cmml">⌞</mi><annotation-xml encoding="MathML-Content" id="S4.T1.6.4.1.m1.1b"><ci id="S4.T1.6.4.1.m1.1.1.cmml" xref="S4.T1.6.4.1.m1.1.1">⌞</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.6.4.1.m1.1c">\llcorner</annotation></semantics></math><span id="S4.T1.6.4.1.1" class="ltx_text" style="font-size:70%;">MMVP</span>
</th>
<td id="S4.T1.6.4.2" class="ltx_td ltx_align_center">
<span id="S4.T1.6.4.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.6.4.2.2" class="ltx_text" style="font-size:70%;">47.3 </span><span id="S4.T1.6.4.2.3" class="ltx_text ltx_phantom" style="font-size:50%;color:#808080;"><span style="visibility:hidden">±0.0</span></span>
</td>
<td id="S4.T1.6.4.3" class="ltx_td ltx_align_center">
<span id="S4.T1.6.4.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.6.4.3.2" class="ltx_text" style="font-size:70%;">45.3 </span><span id="S4.T1.6.4.3.3" class="ltx_text ltx_phantom" style="font-size:50%;color:#808080;"><span style="visibility:hidden">±0.0</span></span>
</td>
<td id="S4.T1.6.4.4" class="ltx_td ltx_align_center"><span id="S4.T1.6.4.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.7.5" class="ltx_tr">
<th id="S4.T1.7.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<math id="S4.T1.7.5.1.m1.1" class="ltx_Math" alttext="\llcorner" display="inline"><semantics id="S4.T1.7.5.1.m1.1a"><mi mathsize="70%" mathvariant="normal" id="S4.T1.7.5.1.m1.1.1" xref="S4.T1.7.5.1.m1.1.1.cmml">⌞</mi><annotation-xml encoding="MathML-Content" id="S4.T1.7.5.1.m1.1b"><ci id="S4.T1.7.5.1.m1.1.1.cmml" xref="S4.T1.7.5.1.m1.1.1">⌞</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.7.5.1.m1.1c">\llcorner</annotation></semantics></math><span id="S4.T1.7.5.1.1" class="ltx_text" style="font-size:70%;">POPE</span>
</th>
<td id="S4.T1.7.5.2" class="ltx_td ltx_align_center">
<span id="S4.T1.7.5.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.7.5.2.2" class="ltx_text" style="font-size:70%;">86.0 </span><span id="S4.T1.7.5.2.3" class="ltx_text ltx_phantom" style="font-size:50%;color:#808080;"><span style="visibility:hidden">±0.0</span></span>
</td>
<td id="S4.T1.7.5.3" class="ltx_td ltx_align_center">
<span id="S4.T1.7.5.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.7.5.3.2" class="ltx_text" style="font-size:70%;">87.0 </span><span id="S4.T1.7.5.3.3" class="ltx_text ltx_phantom" style="font-size:50%;color:#808080;"><span style="visibility:hidden">±0.0</span></span>
</td>
<td id="S4.T1.7.5.4" class="ltx_td ltx_align_center"><span id="S4.T1.7.5.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.8.6" class="ltx_tr">
<th id="S4.T1.8.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<math id="S4.T1.8.6.1.m1.1" class="ltx_Math" alttext="\llcorner" display="inline"><semantics id="S4.T1.8.6.1.m1.1a"><mi mathsize="70%" mathvariant="normal" id="S4.T1.8.6.1.m1.1.1" xref="S4.T1.8.6.1.m1.1.1.cmml">⌞</mi><annotation-xml encoding="MathML-Content" id="S4.T1.8.6.1.m1.1b"><ci id="S4.T1.8.6.1.m1.1.1.cmml" xref="S4.T1.8.6.1.m1.1.1">⌞</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.8.6.1.m1.1c">\llcorner</annotation></semantics></math><span id="S4.T1.8.6.1.1" class="ltx_text" style="font-size:70%;">Objaverse Multiview</span>
</th>
<td id="S4.T1.8.6.2" class="ltx_td ltx_align_center">
<span id="S4.T1.8.6.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.8.6.2.2" class="ltx_text" style="font-size:70%;">62.7 </span><span id="S4.T1.8.6.2.3" class="ltx_text ltx_phantom" style="font-size:50%;color:#808080;"><span style="visibility:hidden">±0.0</span></span>
</td>
<td id="S4.T1.8.6.3" class="ltx_td ltx_align_center">
<span id="S4.T1.8.6.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.8.6.3.2" class="ltx_text" style="font-size:70%;">62.8 </span><span id="S4.T1.8.6.3.3" class="ltx_text ltx_phantom" style="font-size:50%;color:#808080;"><span style="visibility:hidden">±0.0</span></span>
</td>
<td id="S4.T1.8.6.4" class="ltx_td ltx_align_center"><span id="S4.T1.8.6.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.20.12" class="ltx_tr">
<th id="S4.T1.10.20.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.20.12.1.1" class="ltx_text" style="font-size:70%;">OKVQA</span></th>
<td id="S4.T1.10.20.12.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.20.12.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.20.12.2.2" class="ltx_text" style="font-size:70%;">63.5 </span><span id="S4.T1.10.20.12.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.3</span>
</td>
<td id="S4.T1.10.20.12.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.20.12.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.20.12.3.2" class="ltx_text" style="font-size:70%;">63.2 </span><span id="S4.T1.10.20.12.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.2</span>
</td>
<td id="S4.T1.10.20.12.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.20.12.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.21.13" class="ltx_tr">
<th id="S4.T1.10.21.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.21.13.1.1" class="ltx_text" style="font-size:70%;">AOKVQA-MC</span></th>
<td id="S4.T1.10.21.13.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.21.13.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.21.13.2.2" class="ltx_text" style="font-size:70%;">76.4 </span><span id="S4.T1.10.21.13.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.4</span>
</td>
<td id="S4.T1.10.21.13.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.21.13.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.21.13.3.2" class="ltx_text" style="font-size:70%;">76.9 </span><span id="S4.T1.10.21.13.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.3</span>
</td>
<td id="S4.T1.10.21.13.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.21.13.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.22.14" class="ltx_tr">
<th id="S4.T1.10.22.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.22.14.1.1" class="ltx_text" style="font-size:70%;">AOKVQA-DA</span></th>
<td id="S4.T1.10.22.14.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.22.14.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.22.14.2.2" class="ltx_text" style="font-size:70%;">61.9 </span><span id="S4.T1.10.22.14.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.6</span>
</td>
<td id="S4.T1.10.22.14.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.22.14.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.22.14.3.2" class="ltx_text" style="font-size:70%;">63.2 </span><span id="S4.T1.10.22.14.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.5</span>
</td>
<td id="S4.T1.10.22.14.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.22.14.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.23.15" class="ltx_tr">
<th id="S4.T1.10.23.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.23.15.1.1" class="ltx_text" style="font-size:70%;">GQA</span></th>
<td id="S4.T1.10.23.15.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.23.15.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.23.15.2.2" class="ltx_text" style="font-size:70%;">65.6 </span><span id="S4.T1.10.23.15.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.4</span>
</td>
<td id="S4.T1.10.23.15.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.23.15.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.23.15.3.2" class="ltx_text" style="font-size:70%;">67.0 </span><span id="S4.T1.10.23.15.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.3</span>
</td>
<td id="S4.T1.10.23.15.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.23.15.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.9.7" class="ltx_tr">
<th id="S4.T1.9.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<math id="S4.T1.9.7.1.m1.1" class="ltx_Math" alttext="\llcorner" display="inline"><semantics id="S4.T1.9.7.1.m1.1a"><mi mathsize="70%" mathvariant="normal" id="S4.T1.9.7.1.m1.1.1" xref="S4.T1.9.7.1.m1.1.1.cmml">⌞</mi><annotation-xml encoding="MathML-Content" id="S4.T1.9.7.1.m1.1b"><ci id="S4.T1.9.7.1.m1.1.1.cmml" xref="S4.T1.9.7.1.m1.1.1">⌞</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.9.7.1.m1.1c">\llcorner</annotation></semantics></math><span id="S4.T1.9.7.1.1" class="ltx_text" style="font-size:70%;">xGQA (avg7)</span>
</th>
<td id="S4.T1.9.7.2" class="ltx_td ltx_align_center">
<span id="S4.T1.9.7.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.9.7.2.2" class="ltx_text" style="font-size:70%;">57.3 </span><span id="S4.T1.9.7.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.2</span>
</td>
<td id="S4.T1.9.7.3" class="ltx_td ltx_align_center">
<span id="S4.T1.9.7.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.9.7.3.2" class="ltx_text" style="font-size:70%;">57.9 </span><span id="S4.T1.9.7.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.5</span>
</td>
<td id="S4.T1.9.7.4" class="ltx_td ltx_align_center"><span id="S4.T1.9.7.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.24.16" class="ltx_tr">
<th id="S4.T1.10.24.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.24.16.1.1" class="ltx_text" style="font-size:70%;">NLVR2</span></th>
<td id="S4.T1.10.24.16.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.24.16.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.24.16.2.2" class="ltx_text" style="font-size:70%;">90.0 </span><span id="S4.T1.10.24.16.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.2</span>
</td>
<td id="S4.T1.10.24.16.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.24.16.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.24.16.3.2" class="ltx_text" style="font-size:70%;">88.9 </span><span id="S4.T1.10.24.16.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.3</span>
</td>
<td id="S4.T1.10.24.16.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.24.16.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.8" class="ltx_tr">
<th id="S4.T1.10.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<math id="S4.T1.10.8.1.m1.1" class="ltx_Math" alttext="\llcorner" display="inline"><semantics id="S4.T1.10.8.1.m1.1a"><mi mathsize="70%" mathvariant="normal" id="S4.T1.10.8.1.m1.1.1" xref="S4.T1.10.8.1.m1.1.1.cmml">⌞</mi><annotation-xml encoding="MathML-Content" id="S4.T1.10.8.1.m1.1b"><ci id="S4.T1.10.8.1.m1.1.1.cmml" xref="S4.T1.10.8.1.m1.1.1">⌞</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.10.8.1.m1.1c">\llcorner</annotation></semantics></math><span id="S4.T1.10.8.1.1" class="ltx_text" style="font-size:70%;">MARVL (avg5)</span>
</th>
<td id="S4.T1.10.8.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.8.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.8.2.2" class="ltx_text" style="font-size:70%;">80.6 </span><span id="S4.T1.10.8.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.3</span>
</td>
<td id="S4.T1.10.8.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.8.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.8.3.2" class="ltx_text" style="font-size:70%;">76.8 </span><span id="S4.T1.10.8.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.3</span>
</td>
<td id="S4.T1.10.8.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.8.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.25.17" class="ltx_tr">
<th id="S4.T1.10.25.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.25.17.1.1" class="ltx_text" style="font-size:70%;">AI2D</span></th>
<td id="S4.T1.10.25.17.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.25.17.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.25.17.2.2" class="ltx_text" style="font-size:70%;">72.1 </span><span id="S4.T1.10.25.17.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.7</span>
</td>
<td id="S4.T1.10.25.17.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.25.17.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.25.17.3.2" class="ltx_text" style="font-size:70%;">73.3 </span><span id="S4.T1.10.25.17.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.7</span>
</td>
<td id="S4.T1.10.25.17.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.25.17.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.26.18" class="ltx_tr">
<th id="S4.T1.10.26.18.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.26.18.1.1" class="ltx_text" style="font-size:70%;">ScienceQA</span></th>
<td id="S4.T1.10.26.18.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.26.18.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.26.18.2.2" class="ltx_text" style="font-size:70%;">95.4 </span><span id="S4.T1.10.26.18.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.3</span>
</td>
<td id="S4.T1.10.26.18.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.26.18.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.26.18.3.2" class="ltx_text" style="font-size:70%;">95.9 </span><span id="S4.T1.10.26.18.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.2</span>
</td>
<td id="S4.T1.10.26.18.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.26.18.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.27.19" class="ltx_tr">
<th id="S4.T1.10.27.19.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.27.19.1.1" class="ltx_text" style="font-size:70%;">RSVQA-lr</span></th>
<td id="S4.T1.10.27.19.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.27.19.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.27.19.2.2" class="ltx_text" style="font-size:70%;">92.6 </span><span id="S4.T1.10.27.19.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.3</span>
</td>
<td id="S4.T1.10.27.19.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.27.19.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.27.19.3.2" class="ltx_text" style="font-size:70%;">93.1 </span><span id="S4.T1.10.27.19.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.7</span>
</td>
<td id="S4.T1.10.27.19.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.27.19.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.28.20" class="ltx_tr">
<th id="S4.T1.10.28.20.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.28.20.1.1" class="ltx_text" style="font-size:70%;">RSVQA-hr (test)</span></th>
<td id="S4.T1.10.28.20.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.28.20.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.28.20.2.2" class="ltx_text" style="font-size:70%;">92.6 </span><span id="S4.T1.10.28.20.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.1</span>
</td>
<td id="S4.T1.10.28.20.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.28.20.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.28.20.3.2" class="ltx_text" style="font-size:70%;">92.8 </span><span id="S4.T1.10.28.20.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.1</span>
</td>
<td id="S4.T1.10.28.20.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.28.20.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.29.21" class="ltx_tr">
<th id="S4.T1.10.29.21.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.29.21.1.1" class="ltx_text" style="font-size:70%;">RSVQA-hr (test2)</span></th>
<td id="S4.T1.10.29.21.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.29.21.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.29.21.2.2" class="ltx_text" style="font-size:70%;">90.6 </span><span id="S4.T1.10.29.21.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.1</span>
</td>
<td id="S4.T1.10.29.21.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.29.21.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.29.21.3.2" class="ltx_text" style="font-size:70%;">90.5 </span><span id="S4.T1.10.29.21.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.2</span>
</td>
<td id="S4.T1.10.29.21.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.29.21.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.30.22" class="ltx_tr" style="background-color:#F2F2F2;">
<th id="S4.T1.10.30.22.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.30.22.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">ChartQA (human)</span></th>
<td id="S4.T1.10.30.22.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.30.22.2.1" class="ltx_text ltx_phantom" style="font-size:70%;background-color:#F2F2F2;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.30.22.2.2" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">40.0 <span id="S4.T1.10.30.22.2.2.1" class="ltx_text" style="font-size:71%;color:#808080;background-color:#F2F2F2;">±0.7</span></span>
</td>
<td id="S4.T1.10.30.22.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.30.22.3.1" class="ltx_text ltx_phantom" style="font-size:70%;background-color:#F2F2F2;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.30.22.3.2" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">54.2 <span id="S4.T1.10.30.22.3.2.1" class="ltx_text" style="font-size:71%;color:#808080;background-color:#F2F2F2;">±1.1</span></span>
</td>
<td id="S4.T1.10.30.22.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.30.22.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">-</span></td>
</tr>
<tr id="S4.T1.10.31.23" class="ltx_tr" style="background-color:#F2F2F2;">
<th id="S4.T1.10.31.23.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.31.23.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">ChartQA (aug)</span></th>
<td id="S4.T1.10.31.23.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.31.23.2.1" class="ltx_text ltx_phantom" style="font-size:70%;background-color:#F2F2F2;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.31.23.2.2" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">74.2 <span id="S4.T1.10.31.23.2.2.1" class="ltx_text" style="font-size:71%;color:#808080;background-color:#F2F2F2;">±0.2</span></span>
</td>
<td id="S4.T1.10.31.23.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.31.23.3.1" class="ltx_text ltx_phantom" style="font-size:70%;background-color:#F2F2F2;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.31.23.3.2" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">88.5 <span id="S4.T1.10.31.23.3.2.1" class="ltx_text" style="font-size:71%;color:#808080;background-color:#F2F2F2;">±0.3</span></span>
</td>
<td id="S4.T1.10.31.23.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.31.23.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">-</span></td>
</tr>
<tr id="S4.T1.10.32.24" class="ltx_tr">
<th id="S4.T1.10.32.24.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.32.24.1.1" class="ltx_text" style="font-size:70%;">VizWizVQA</span></th>
<td id="S4.T1.10.32.24.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.32.24.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.32.24.2.2" class="ltx_text" style="font-size:70%;">73.7 </span><span id="S4.T1.10.32.24.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.2</span>
</td>
<td id="S4.T1.10.32.24.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.32.24.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.32.24.3.2" class="ltx_text" style="font-size:70%;">75.5 </span><span id="S4.T1.10.32.24.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.5</span>
</td>
<td id="S4.T1.10.32.24.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.32.24.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.33.25" class="ltx_tr">
<th id="S4.T1.10.33.25.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.33.25.1.1" class="ltx_text" style="font-size:70%;">TallyQA (simple)</span></th>
<td id="S4.T1.10.33.25.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.33.25.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.33.25.2.2" class="ltx_text" style="font-size:70%;">81.7 </span><span id="S4.T1.10.33.25.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.2</span>
</td>
<td id="S4.T1.10.33.25.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.33.25.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.33.25.3.2" class="ltx_text" style="font-size:70%;">84.9 </span><span id="S4.T1.10.33.25.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.1</span>
</td>
<td id="S4.T1.10.33.25.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.33.25.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.34.26" class="ltx_tr">
<th id="S4.T1.10.34.26.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.34.26.1.1" class="ltx_text" style="font-size:70%;">TallyQA (complex)</span></th>
<td id="S4.T1.10.34.26.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.34.26.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.34.26.2.2" class="ltx_text" style="font-size:70%;">69.6 </span><span id="S4.T1.10.34.26.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.1</span>
</td>
<td id="S4.T1.10.34.26.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.34.26.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.34.26.3.2" class="ltx_text" style="font-size:70%;">72.3 </span><span id="S4.T1.10.34.26.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.2</span>
</td>
<td id="S4.T1.10.34.26.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.34.26.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.35.27" class="ltx_tr">
<th id="S4.T1.10.35.27.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.35.27.1.1" class="ltx_text" style="font-size:70%;">CountBenchQA</span></th>
<td id="S4.T1.10.35.27.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.35.27.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.35.27.2.2" class="ltx_text" style="font-size:70%;">81.9 </span><span id="S4.T1.10.35.27.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±1.6</span>
</td>
<td id="S4.T1.10.35.27.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.35.27.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.35.27.3.2" class="ltx_text" style="font-size:70%;">83.1 </span><span id="S4.T1.10.35.27.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±2.1</span>
</td>
<td id="S4.T1.10.35.27.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.35.27.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.36.28" class="ltx_tr">
<th id="S4.T1.10.36.28.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.36.28.1.1" class="ltx_text" style="font-size:70%;">OCR-VQA</span></th>
<td id="S4.T1.10.36.28.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.36.28.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.36.28.2.2" class="ltx_text" style="font-size:70%;">72.3 </span><span id="S4.T1.10.36.28.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.4</span>
</td>
<td id="S4.T1.10.36.28.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.36.28.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.36.28.3.2" class="ltx_text" style="font-size:70%;">74.6 </span><span id="S4.T1.10.36.28.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.4</span>
</td>
<td id="S4.T1.10.36.28.4" class="ltx_td ltx_align_center">
<span id="S4.T1.10.36.28.4.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.36.28.4.2" class="ltx_text" style="font-size:70%;">74.9</span>
</td>
</tr>
<tr id="S4.T1.10.37.29" class="ltx_tr" style="background-color:#F2F2F2;">
<th id="S4.T1.10.37.29.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.37.29.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">TextVQA</span></th>
<td id="S4.T1.10.37.29.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.37.29.2.1" class="ltx_text ltx_phantom" style="font-size:70%;background-color:#F2F2F2;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.37.29.2.2" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">55.5 <span id="S4.T1.10.37.29.2.2.1" class="ltx_text" style="font-size:71%;color:#808080;background-color:#F2F2F2;">±0.2</span></span>
</td>
<td id="S4.T1.10.37.29.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.37.29.3.1" class="ltx_text ltx_phantom" style="font-size:70%;background-color:#F2F2F2;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.37.29.3.2" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">73.2 <span id="S4.T1.10.37.29.3.2.1" class="ltx_text" style="font-size:71%;color:#808080;background-color:#F2F2F2;">±0.2</span></span>
</td>
<td id="S4.T1.10.37.29.4" class="ltx_td ltx_align_center">
<span id="S4.T1.10.37.29.4.1" class="ltx_text ltx_phantom" style="font-size:70%;background-color:#F2F2F2;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.37.29.4.2" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">76.5</span>
</td>
</tr>
<tr id="S4.T1.10.38.30" class="ltx_tr" style="background-color:#F2F2F2;">
<th id="S4.T1.10.38.30.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.38.30.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">DocVQA</span></th>
<td id="S4.T1.10.38.30.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.38.30.2.1" class="ltx_text ltx_phantom" style="font-size:70%;background-color:#F2F2F2;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.38.30.2.2" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">43.7 <span id="S4.T1.10.38.30.2.2.1" class="ltx_text" style="font-size:71%;color:#808080;background-color:#F2F2F2;">±0.5</span></span>
</td>
<td id="S4.T1.10.38.30.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.38.30.3.1" class="ltx_text ltx_phantom" style="font-size:70%;background-color:#F2F2F2;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.38.30.3.2" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">78.0 <span id="S4.T1.10.38.30.3.2.1" class="ltx_text" style="font-size:71%;color:#808080;background-color:#F2F2F2;">±0.3</span></span>
</td>
<td id="S4.T1.10.38.30.4" class="ltx_td ltx_align_center">
<span id="S4.T1.10.38.30.4.1" class="ltx_text ltx_phantom" style="font-size:70%;background-color:#F2F2F2;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.38.30.4.2" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">84.8</span>
</td>
</tr>
<tr id="S4.T1.10.39.31" class="ltx_tr" style="background-color:#F2F2F2;">
<th id="S4.T1.10.39.31.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.39.31.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">InfoVQA</span></th>
<td id="S4.T1.10.39.31.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.39.31.2.1" class="ltx_text ltx_phantom" style="font-size:70%;background-color:#F2F2F2;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.39.31.2.2" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">28.5 <span id="S4.T1.10.39.31.2.2.1" class="ltx_text" style="font-size:71%;color:#808080;background-color:#F2F2F2;">±0.4</span></span>
</td>
<td id="S4.T1.10.39.31.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.39.31.3.1" class="ltx_text ltx_phantom" style="font-size:70%;background-color:#F2F2F2;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.39.31.3.2" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">40.5 <span id="S4.T1.10.39.31.3.2.1" class="ltx_text" style="font-size:71%;color:#808080;background-color:#F2F2F2;">±0.3</span></span>
</td>
<td id="S4.T1.10.39.31.4" class="ltx_td ltx_align_center">
<span id="S4.T1.10.39.31.4.1" class="ltx_text ltx_phantom" style="font-size:70%;background-color:#F2F2F2;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.39.31.4.2" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">47.8</span>
</td>
</tr>
<tr id="S4.T1.10.40.32" class="ltx_tr" style="background-color:#F2F2F2;">
<th id="S4.T1.10.40.32.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.40.32.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">ST-VQA</span></th>
<td id="S4.T1.10.40.32.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.40.32.2.1" class="ltx_text ltx_phantom" style="font-size:70%;background-color:#F2F2F2;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.40.32.2.2" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">63.3 <span id="S4.T1.10.40.32.2.2.1" class="ltx_text" style="font-size:71%;color:#808080;background-color:#F2F2F2;">±0.5</span></span>
</td>
<td id="S4.T1.10.40.32.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.40.32.3.1" class="ltx_text ltx_phantom" style="font-size:70%;background-color:#F2F2F2;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.40.32.3.2" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">81.8 <span id="S4.T1.10.40.32.3.2.1" class="ltx_text" style="font-size:71%;color:#808080;background-color:#F2F2F2;">±0.3</span></span>
</td>
<td id="S4.T1.10.40.32.4" class="ltx_td ltx_align_center">
<span id="S4.T1.10.40.32.4.1" class="ltx_text ltx_phantom" style="font-size:70%;background-color:#F2F2F2;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.40.32.4.2" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">84.4</span>
</td>
</tr>
<tr id="S4.T1.10.41.33" class="ltx_tr">
<th id="S4.T1.10.41.33.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="4"><span id="S4.T1.10.41.33.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Image segmentation</span></th>
</tr>
<tr id="S4.T1.10.42.34" class="ltx_tr">
<th id="S4.T1.10.42.34.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.42.34.1.1" class="ltx_text" style="font-size:70%;">RefCOCO (testA)</span></th>
<td id="S4.T1.10.42.34.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.42.34.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.42.34.2.2" class="ltx_text" style="font-size:70%;">75.7 </span><span id="S4.T1.10.42.34.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.1</span>
</td>
<td id="S4.T1.10.42.34.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.42.34.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.42.34.3.2" class="ltx_text" style="font-size:70%;">77.9 </span><span id="S4.T1.10.42.34.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.1</span>
</td>
<td id="S4.T1.10.42.34.4" class="ltx_td ltx_align_center">
<span id="S4.T1.10.42.34.4.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.42.34.4.2" class="ltx_text" style="font-size:70%;">78.7</span>
</td>
</tr>
<tr id="S4.T1.10.43.35" class="ltx_tr">
<th id="S4.T1.10.43.35.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.43.35.1.1" class="ltx_text" style="font-size:70%;">RefCOCO (testB)</span></th>
<td id="S4.T1.10.43.35.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.43.35.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.43.35.2.2" class="ltx_text" style="font-size:70%;">70.7 </span><span id="S4.T1.10.43.35.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.2</span>
</td>
<td id="S4.T1.10.43.35.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.43.35.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.43.35.3.2" class="ltx_text" style="font-size:70%;">72.4 </span><span id="S4.T1.10.43.35.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.2</span>
</td>
<td id="S4.T1.10.43.35.4" class="ltx_td ltx_align_center">
<span id="S4.T1.10.43.35.4.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.43.35.4.2" class="ltx_text" style="font-size:70%;">73.9</span>
</td>
</tr>
<tr id="S4.T1.10.44.36" class="ltx_tr">
<th id="S4.T1.10.44.36.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.44.36.1.1" class="ltx_text" style="font-size:70%;">RefCOCO+ (testA)</span></th>
<td id="S4.T1.10.44.36.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.44.36.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.44.36.2.2" class="ltx_text" style="font-size:70%;">71.9 </span><span id="S4.T1.10.44.36.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.1</span>
</td>
<td id="S4.T1.10.44.36.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.44.36.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.44.36.3.2" class="ltx_text" style="font-size:70%;">74.2 </span><span id="S4.T1.10.44.36.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.2</span>
</td>
<td id="S4.T1.10.44.36.4" class="ltx_td ltx_align_center">
<span id="S4.T1.10.44.36.4.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.44.36.4.2" class="ltx_text" style="font-size:70%;">76.1</span>
</td>
</tr>
<tr id="S4.T1.10.45.37" class="ltx_tr">
<th id="S4.T1.10.45.37.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.45.37.1.1" class="ltx_text" style="font-size:70%;">RefCOCO+ (testB)</span></th>
<td id="S4.T1.10.45.37.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.45.37.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.45.37.2.2" class="ltx_text" style="font-size:70%;">64.5 </span><span id="S4.T1.10.45.37.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.6</span>
</td>
<td id="S4.T1.10.45.37.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.45.37.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.45.37.3.2" class="ltx_text" style="font-size:70%;">64.5 </span><span id="S4.T1.10.45.37.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.2</span>
</td>
<td id="S4.T1.10.45.37.4" class="ltx_td ltx_align_center">
<span id="S4.T1.10.45.37.4.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.45.37.4.2" class="ltx_text" style="font-size:70%;">66.9</span>
</td>
</tr>
<tr id="S4.T1.10.46.38" class="ltx_tr">
<th id="S4.T1.10.46.38.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.46.38.1.1" class="ltx_text" style="font-size:70%;">RefCOCOg (test)</span></th>
<td id="S4.T1.10.46.38.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.46.38.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.46.38.2.2" class="ltx_text" style="font-size:70%;">68.2 </span><span id="S4.T1.10.46.38.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.1</span>
</td>
<td id="S4.T1.10.46.38.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.46.38.3.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.46.38.3.2" class="ltx_text" style="font-size:70%;">71.0 </span><span id="S4.T1.10.46.38.3.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.1</span>
</td>
<td id="S4.T1.10.46.38.4" class="ltx_td ltx_align_center">
<span id="S4.T1.10.46.38.4.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.46.38.4.2" class="ltx_text" style="font-size:70%;">72.7</span>
</td>
</tr>
<tr id="S4.T1.10.47.39" class="ltx_tr">
<th id="S4.T1.10.47.39.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="4"><span id="S4.T1.10.47.39.1.1" class="ltx_text ltx_font_bold" style="font-size:70%;">Video input</span></th>
</tr>
<tr id="S4.T1.10.48.40" class="ltx_tr">
<th id="S4.T1.10.48.40.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.48.40.1.1" class="ltx_text" style="font-size:70%;">ActivityNet-QA</span></th>
<td id="S4.T1.10.48.40.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.48.40.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.48.40.2.2" class="ltx_text" style="font-size:70%;">50.8 </span><span id="S4.T1.10.48.40.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.4</span>
</td>
<td id="S4.T1.10.48.40.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.48.40.3.1" class="ltx_text" style="font-size:70%;">- </span><span id="S4.T1.10.48.40.3.2" class="ltx_text ltx_phantom" style="font-size:50%;color:#808080;"><span style="visibility:hidden">±0.0</span></span>
</td>
<td id="S4.T1.10.48.40.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.48.40.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.49.41" class="ltx_tr">
<th id="S4.T1.10.49.41.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.49.41.1.1" class="ltx_text" style="font-size:70%;">ActivityNet-CAP</span></th>
<td id="S4.T1.10.49.41.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.49.41.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.49.41.2.2" class="ltx_text" style="font-size:70%;">34.6 </span><span id="S4.T1.10.49.41.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.6</span>
</td>
<td id="S4.T1.10.49.41.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.49.41.3.1" class="ltx_text" style="font-size:70%;">- </span><span id="S4.T1.10.49.41.3.2" class="ltx_text ltx_phantom" style="font-size:50%;color:#808080;"><span style="visibility:hidden">±0.0</span></span>
</td>
<td id="S4.T1.10.49.41.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.49.41.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.50.42" class="ltx_tr">
<th id="S4.T1.10.50.42.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.50.42.1.1" class="ltx_text" style="font-size:70%;">MSRVTT-QA</span></th>
<td id="S4.T1.10.50.42.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.50.42.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.50.42.2.2" class="ltx_text" style="font-size:70%;">50.1 </span><span id="S4.T1.10.50.42.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.1</span>
</td>
<td id="S4.T1.10.50.42.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.50.42.3.1" class="ltx_text" style="font-size:70%;">- </span><span id="S4.T1.10.50.42.3.2" class="ltx_text ltx_phantom" style="font-size:50%;color:#808080;"><span style="visibility:hidden">±0.0</span></span>
</td>
<td id="S4.T1.10.50.42.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.50.42.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.51.43" class="ltx_tr">
<th id="S4.T1.10.51.43.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.51.43.1.1" class="ltx_text" style="font-size:70%;">MSRVTT-CAP</span></th>
<td id="S4.T1.10.51.43.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.51.43.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.51.43.2.2" class="ltx_text" style="font-size:70%;">70.5 </span><span id="S4.T1.10.51.43.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.9</span>
</td>
<td id="S4.T1.10.51.43.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.51.43.3.1" class="ltx_text" style="font-size:70%;">- </span><span id="S4.T1.10.51.43.3.2" class="ltx_text ltx_phantom" style="font-size:50%;color:#808080;"><span style="visibility:hidden">±0.0</span></span>
</td>
<td id="S4.T1.10.51.43.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.51.43.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.52.44" class="ltx_tr">
<th id="S4.T1.10.52.44.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T1.10.52.44.1.1" class="ltx_text" style="font-size:70%;">MSVD-QA</span></th>
<td id="S4.T1.10.52.44.2" class="ltx_td ltx_align_center">
<span id="S4.T1.10.52.44.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.52.44.2.2" class="ltx_text" style="font-size:70%;">60.2 </span><span id="S4.T1.10.52.44.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.3</span>
</td>
<td id="S4.T1.10.52.44.3" class="ltx_td ltx_align_center">
<span id="S4.T1.10.52.44.3.1" class="ltx_text" style="font-size:70%;">- </span><span id="S4.T1.10.52.44.3.2" class="ltx_text ltx_phantom" style="font-size:50%;color:#808080;"><span style="visibility:hidden">±0.0</span></span>
</td>
<td id="S4.T1.10.52.44.4" class="ltx_td ltx_align_center"><span id="S4.T1.10.52.44.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
<tr id="S4.T1.10.53.45" class="ltx_tr">
<th id="S4.T1.10.53.45.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="S4.T1.10.53.45.1.1" class="ltx_text" style="font-size:70%;">VATEX</span></th>
<td id="S4.T1.10.53.45.2" class="ltx_td ltx_align_center ltx_border_bb">
<span id="S4.T1.10.53.45.2.1" class="ltx_text ltx_phantom" style="font-size:70%;"><span style="visibility:hidden">0</span></span><span id="S4.T1.10.53.45.2.2" class="ltx_text" style="font-size:70%;">79.7 </span><span id="S4.T1.10.53.45.2.3" class="ltx_text" style="font-size:50%;color:#808080;">±0.4</span>
</td>
<td id="S4.T1.10.53.45.3" class="ltx_td ltx_align_center ltx_border_bb">
<span id="S4.T1.10.53.45.3.1" class="ltx_text" style="font-size:70%;">- </span><span id="S4.T1.10.53.45.3.2" class="ltx_text ltx_phantom" style="font-size:50%;color:#808080;"><span style="visibility:hidden">±0.0</span></span>
</td>
<td id="S4.T1.10.53.45.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T1.10.53.45.4.1" class="ltx_text" style="font-size:70%;">-</span></td>
</tr>
</tbody>
</table>
</figure>
<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">In order to verify the transferability of PaliGemma to a wide variety of tasks, we transfer the pretrained models on more than 30 academic benchmarks via fine-tuning.
Importantly, none of these tasks or datasets are part of the pretraining data mixture, and their images are explicitly removed from the web-scale pretraining data.
Results are presented in Table <a href="#S4.T1" title="Table 1 ‣ 4 Results ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.</p>
</div>
<div id="S4.p2" class="ltx_para">
<p id="S4.p2.1" class="ltx_p">To select the hyper-parameters for transfer, we first sweep the parameters mentioned in Section <a href="#S3.SS2.SSS4" title="3.2.4 Stage3: Transfer ‣ 3.2 Pretraining ‣ 3 Model ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2.4</span></a>, starting from the recommended value.
We do not necessarily perform the full cross-product, and we sometimes extend or supersample the range, if it seems promising.
Importantly, we make any such decisions and hyper-parameter choices based on the transfer task’s validation split, and if none is provided, we hold out a small “minival” set from the training data.
Once we found good hyper-parameter values for a task, we re-train using the full training and validation data, and report final test numbers.
Details on tasks, metrics, data splits are in Appendix <a href="#A2" title="Appendix B Tasks ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a> and final hyper-parameters in Appendix <a href="#A10" title="Appendix J Hyper-parameters ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">J</span></a>.
In Section <a href="#S6.SS2" title="6.2 Transfer hyper-parameter sensitivity ‣ 6 Transferability ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6.2</span></a> we show that a single recommended value for each hyper-parameter without any exploration works almost as well on most tasks.</p>
</div>
<div id="S4.p3" class="ltx_para">
<p id="S4.p3.1" class="ltx_p">For all but video tasks, we report results on at least two resolutions to provide an impression of which tasks benefit from increased resolution.
We provide many resolution-related ablations in Section <a href="#S5.SS7" title="5.7 Image resolution ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.7</span></a>.</p>
</div>
<div id="S4.p4" class="ltx_para">
<p id="S4.p4.1" class="ltx_p">Notably, we have not found any significant benefit from data augmentation.
We simply resize the input images to a square fixed resolution, even for tasks such as RefCOCO segmentation (more on that in Section <a href="#S5.SS7" title="5.7 Image resolution ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.7</span></a> and Appendix <a href="#A3" title="Appendix C Image augmentations for RefCOCO ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>).</p>
</div>
</section>
<section id="S5" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Ablations</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">We conduct diverse ablations to gain deeper understanding of what matters for training and transferring VLMs.
Unless noted otherwise, all ablations are run with the same setup as the main models, except for making the Stage1 pretraining 10x shorter (<em id="S5.p1.1.1" class="ltx_emph ltx_font_italic">i.e</em>.<span id="S5.p1.1.2" class="ltx_text"></span> 100 M examples seen), and transfer results are reported on validation sets instead of withheld test-sets.
For each experiment, we present only the salient result summary in the main text, but we provide a full per-task breakdown of results in the Appendix.</p>
</div>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Multimodal pretraining duration</h3>

<figure id="S5.F4" class="ltx_figure"><img src="/html/2407.07726/assets/x4.png" id="S5.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="231" height="121" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>Relative regret of transfers when varying the amount of pretraining during Stage 1. Per task plot in appendix <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:sec:app:pt_duration</span>.</figcaption>
</figure>
<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.1" class="ltx_p">With its 1 B examples seen, our multimodal pretraining (Stage1) is on the longer side, similar to BLIP-2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite>, InternVL <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib25" title="" class="ltx_ref">25</a>]</cite>, QwenVL <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>]</cite>, Idefics2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib57" title="" class="ltx_ref">57</a>]</cite> (all around 1 B), but unlike ShareGPT4-v <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib20" title="" class="ltx_ref">20</a>]</cite>, Mini-Gemini <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib63" title="" class="ltx_ref">63</a>]</cite>, LLaVa <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib67" title="" class="ltx_ref">67</a>]</cite> and its derivatives (around 1 M).</p>
</div>
<div id="S5.SS1.p2" class="ltx_para">
<p id="S5.SS1.p2.1" class="ltx_p">To the best of our knowledge, the benefits from longer pretraining have not been studied in isolation.
We run pretrainings of various shorter durations, all the way down to completely skipping Stage1, and show the impact in Figure <a href="#S5.F4" title="Figure 4 ‣ 5.1 Multimodal pretraining duration ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, with a complete break-down across tasks in the Appendix <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:sec:app:pt_duration</span>.
For the case of skipping Stage1, we use the best transfer result when sweeping over three learning-rates for each task.</p>
</div>
<div id="S5.SS1.p3" class="ltx_para">
<p id="S5.SS1.p3.1" class="ltx_p">The result shows that shorter training generally hurts, and skipping Stage1 entirely is the worst setting.
Some task are affected significantly, while others only deteriorate a little, highlighting the need for a broad and diverse set of evaluation tasks.
The <math id="S5.SS1.p3.1.m1.1" class="ltx_Math" alttext="100\," display="inline"><semantics id="S5.SS1.p3.1.m1.1a"><mn id="S5.SS1.p3.1.m1.1.1" xref="S5.SS1.p3.1.m1.1.1.cmml">100</mn><annotation-xml encoding="MathML-Content" id="S5.SS1.p3.1.m1.1b"><cn type="integer" id="S5.SS1.p3.1.m1.1.1.cmml" xref="S5.SS1.p3.1.m1.1.1">100</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p3.1.m1.1c">100\,</annotation></semantics></math>M pretraining duration appears to be a good trade-off for ablations: it is 10x shorter while not significantly hurting any task.</p>
</div>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Causal masking and learning objective</h3>

<figure id="S5.F5" class="ltx_figure"><img src="/html/2407.07726/assets/x5.png" id="S5.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="231" height="119" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>Learning setup for Stage1.
Left: Where to apply the auto-regressive mask and loss. Middle and right: whether to include a task indicator.</figcaption>
</figure>
<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.1" class="ltx_p">We ablate several of our key choices in the pretraining learning objective in Figure <a href="#S5.F5" title="Figure 5 ‣ 5.2 Causal masking and learning objective ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, full per-task breakdown in Appendix <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:sec:app:learning</span>.</p>
</div>
<div id="S5.SS2.p2" class="ltx_para">
<p id="S5.SS2.p2.1" class="ltx_p">First, we investigate design choices for auto-regressive masking.
PaliGemma uses a prefix-LM strategy which allows full (bi-directional) attention on the “input” part of the data, <em id="S5.SS2.p2.1.1" class="ltx_emph ltx_font_italic">i.e</em>.<span id="S5.SS2.p2.1.2" class="ltx_text"></span> the <span id="S5.SS2.p2.1.3" class="ltx_text ltx_font_typewriter">image</span> and <span id="S5.SS2.p2.1.4" class="ltx_text ltx_font_typewriter">prefix</span> tokens, see also Figure <a href="#S3.F2" title="Figure 2 ‣ 3.1 Architecture ‣ 3 Model ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.
The motivation is that it allows more tokens to actively participate in the “thinking” process from the start, as the image tokens now can attend to the prefix tokens which represent the query.
This is empirically confirmed in Figure <a href="#S5.F5" title="Figure 5 ‣ 5.2 Causal masking and learning objective ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> (left), where the green bars also include the auto-regressive masking on the <span id="S5.SS2.p2.1.5" class="ltx_text ltx_font_typewriter">prefix</span> tokens, and the orange bars further extend the auto-regressive mask to the <span id="S5.SS2.p2.1.6" class="ltx_text ltx_font_typewriter">image</span> tokens.
Both sets of bars work, but perform clearly worse than PaliGemma’s prefix-LM setting represented by the blue bar.</p>
</div>
<div id="S5.SS2.p3" class="ltx_para">
<p id="S5.SS2.p3.1" class="ltx_p">Second, we apply the next-token-prediction loss on the <span id="S5.SS2.p3.1.1" class="ltx_text ltx_font_typewriter">suffix</span> (the output) only.
In principle, it can also be applied on the <span id="S5.SS2.p3.1.2" class="ltx_text ltx_font_typewriter">prefix</span> tokens, once they are auto-regressively masked.
This could provide more learning signal to the model by asking it to “guess the question” for example.
Again, Figure <a href="#S5.F5" title="Figure 5 ‣ 5.2 Causal masking and learning objective ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> shows that while it works, doing so clearly reduces average performance.</p>
</div>
<div id="S5.SS2.p4" class="ltx_para">
<p id="S5.SS2.p4.1" class="ltx_p">Finally, we eased the multi-task learning by using task-prefixes <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib14" title="" class="ltx_ref">14</a>]</cite>.
Figure <a href="#S5.F5" title="Figure 5 ‣ 5.2 Causal masking and learning objective ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> (middle) shows that after transfers, this choice of pretraining has no noticeable effect.
However, it would be incorrect to conclude that it has no effect on the model’s training.
Indeed, pretraining validation perplexities on three representative tasks of pretraining are shown in Figure <a href="#S5.F5" title="Figure 5 ‣ 5.2 Causal masking and learning objective ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> (right): when the <span id="S5.SS2.p4.1.1" class="ltx_text ltx_font_typewriter">prefix</span> makes the task obvious, such as in VQA, a task-prefix has no effect.
For tasks where the <span id="S5.SS2.p4.1.2" class="ltx_text ltx_font_typewriter">prefix</span> does not perfectly disambiguate the exact task, however, the model (expectedly) does become noticeably more uncertain in its predictions without task-prefix.</p>
</div>
<div id="S5.SS2.p5" class="ltx_para">
<p id="S5.SS2.p5.1" class="ltx_p">Overall, the prefix-LM with task-prefix and supervision only on the <span id="S5.SS2.p5.1.1" class="ltx_text ltx_font_typewriter">suffix</span> tokens is an effective VLM pretraining objective.</p>
</div>
</section>
<section id="S5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.3 </span>New token initialization</h3>

<figure id="S5.F6" class="ltx_figure"><img src="/html/2407.07726/assets/x6.png" id="S5.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="231" height="119" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>Initialization of new tokens matters. Left: distribution of embedding norms at init. Right: Top: initial loss is better with AvgEmb, but this changes after a few thousand steps. Bottom: the model pretrained with <math id="S5.F6.2.m1.1" class="ltx_Math" alttext="\sigma=0.02" display="inline"><semantics id="S5.F6.2.m1.1b"><mrow id="S5.F6.2.m1.1.1" xref="S5.F6.2.m1.1.1.cmml"><mi id="S5.F6.2.m1.1.1.2" xref="S5.F6.2.m1.1.1.2.cmml">σ</mi><mo id="S5.F6.2.m1.1.1.1" xref="S5.F6.2.m1.1.1.1.cmml">=</mo><mn id="S5.F6.2.m1.1.1.3" xref="S5.F6.2.m1.1.1.3.cmml">0.02</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.F6.2.m1.1c"><apply id="S5.F6.2.m1.1.1.cmml" xref="S5.F6.2.m1.1.1"><eq id="S5.F6.2.m1.1.1.1.cmml" xref="S5.F6.2.m1.1.1.1"></eq><ci id="S5.F6.2.m1.1.1.2.cmml" xref="S5.F6.2.m1.1.1.2">𝜎</ci><cn type="float" id="S5.F6.2.m1.1.1.3.cmml" xref="S5.F6.2.m1.1.1.3">0.02</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.F6.2.m1.1d">\sigma=0.02</annotation></semantics></math> inits transfers better to a task heavily using the new tokens.</figcaption>
</figure>
<div id="S5.SS3.p1" class="ltx_para">
<p id="S5.SS3.p1.1" class="ltx_p">We add new tokens to Gemma’s vocabulary to support PaliGemma’s ability to perform more structured computer vision tasks.
We add 1024 location tokens (<span id="S5.SS3.p1.1.1" class="ltx_text ltx_font_typewriter">&lt;loc0000&gt;</span> to <span id="S5.SS3.p1.1.2" class="ltx_text ltx_font_typewriter">&lt;loc1023&gt;</span>), which correspond to binned normalized image coordinates and are used in detection, referring expression, and grounded captioning tasks.
We also add 128 VQVAE <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib108" title="" class="ltx_ref">108</a>]</cite> tokenized single-object mask tokens <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib83" title="" class="ltx_ref">83</a>, <a href="#bib.bib24" title="" class="ltx_ref">24</a>]</cite> (<span id="S5.SS3.p1.1.3" class="ltx_text ltx_font_typewriter">&lt;seg000&gt;</span> to <span id="S5.SS3.p1.1.4" class="ltx_text ltx_font_typewriter">&lt;seg127&gt;</span>) to support referring expression segmentation.</p>
</div>
<div id="S5.SS3.p2" class="ltx_para">
<p id="S5.SS3.p2.1" class="ltx_p">This poses the question of how to initialize the embeddings of these new tokens, given all other vocabulary tokens have already been trained as part of Gemma’s pretraining.
One option is to use a standard small Gaussian noise initialization (<math id="S5.SS3.p2.1.m1.1" class="ltx_Math" alttext="\sigma=0.02" display="inline"><semantics id="S5.SS3.p2.1.m1.1a"><mrow id="S5.SS3.p2.1.m1.1.1" xref="S5.SS3.p2.1.m1.1.1.cmml"><mi id="S5.SS3.p2.1.m1.1.1.2" xref="S5.SS3.p2.1.m1.1.1.2.cmml">σ</mi><mo id="S5.SS3.p2.1.m1.1.1.1" xref="S5.SS3.p2.1.m1.1.1.1.cmml">=</mo><mn id="S5.SS3.p2.1.m1.1.1.3" xref="S5.SS3.p2.1.m1.1.1.3.cmml">0.02</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS3.p2.1.m1.1b"><apply id="S5.SS3.p2.1.m1.1.1.cmml" xref="S5.SS3.p2.1.m1.1.1"><eq id="S5.SS3.p2.1.m1.1.1.1.cmml" xref="S5.SS3.p2.1.m1.1.1.1"></eq><ci id="S5.SS3.p2.1.m1.1.1.2.cmml" xref="S5.SS3.p2.1.m1.1.1.2">𝜎</ci><cn type="float" id="S5.SS3.p2.1.m1.1.1.3.cmml" xref="S5.SS3.p2.1.m1.1.1.3">0.02</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p2.1.m1.1c">\sigma=0.02</annotation></semantics></math>, blue in Fig <a href="#S5.F6" title="Figure 6 ‣ 5.3 New token initialization ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>).
However, <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib40" title="" class="ltx_ref">40</a>]</cite> argues for matching the average of the pretrained embeddings plus small noise (AvgEmb, mauve in Fig <a href="#S5.F6" title="Figure 6 ‣ 5.3 New token initialization ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>).
We compare these strategies in Figure <a href="#S5.F6" title="Figure 6 ‣ 5.3 New token initialization ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> and see that, while the AvgEmb strategy significantly improves initial perplexity of tasks using the new tokens (top-right zoom-in, step 0), this gain vanishes after a thousand steps of training.
The standard initialization strategy not only results in significantly better Stage1 perplexities at the end of pretraining (top-right, full plot), but also results in significantly better transfer of the model to tasks using these tokens, here RefCOCO segmentation MIoU (bottom right).</p>
</div>
</section>
<section id="S5.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.4 </span>To freeze or not to freeze?</h3>

<div id="S5.SS4.p1" class="ltx_para">
<p id="S5.SS4.p1.1" class="ltx_p">The current common wisdom in VLMs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>, <a href="#bib.bib23" title="" class="ltx_ref">23</a>, <a href="#bib.bib24" title="" class="ltx_ref">24</a>, <a href="#bib.bib58" title="" class="ltx_ref">58</a>, <a href="#bib.bib67" title="" class="ltx_ref">67</a>, <a href="#bib.bib50" title="" class="ltx_ref">50</a>, <a href="#bib.bib60" title="" class="ltx_ref">60</a>, <a href="#bib.bib64" title="" class="ltx_ref">64</a>, <a href="#bib.bib43" title="" class="ltx_ref">43</a>]</cite> is to keep the image encoder and sometimes the LLM frozen during multimodal pretraining (our Stage1).
However, inspired by the positive results from CapPa <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib106" title="" class="ltx_ref">106</a>]</cite> and LocCa <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib111" title="" class="ltx_ref">111</a>]</cite> which show that pretraining an image encoder using captioning objectives essentially solves contrastive’s blind spot <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib41" title="" class="ltx_ref">41</a>]</cite> to relation and localization, we pretrained PaliGemma with no frozen parts.
We now ablate the effect of freezing or tuning various parts of the model during Stage1 in Figure <a href="#S5.F7" title="Figure 7 ‣ 5.4 To freeze or not to freeze? ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>, full per-task breakdown in Appendix <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:sec:app:freeze</span>.
Similar to concurrent works <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib103" title="" class="ltx_ref">103</a>, <a href="#bib.bib78" title="" class="ltx_ref">78</a>]</cite>, we find not freezing any part of the model is indeed advantageous.
First, after transfers, there is no difference to keeping the image encoder frozen (left, TT and TF).
Second, however, the validation perplexity (hence, predictability) of tasks requiring spatial understanding (right, green) is significantly improved.</p>
</div>
<div id="S5.SS4.p2" class="ltx_para">
<p id="S5.SS4.p2.1" class="ltx_p">Further, we show that all other options that include freezing the language model <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib107" title="" class="ltx_ref">107</a>]</cite> are significantly worse.
Finally, resetting (and training, R) any part of the model hurts performance dramatically, confirming that Stage0 (<em id="S5.SS4.p2.1.1" class="ltx_emph ltx_font_italic">i.e</em>.<span id="S5.SS4.p2.1.2" class="ltx_text"></span> leveraging pre-trained components) is indeed crucial for attaining good results.</p>
</div>
<figure id="S5.F7" class="ltx_figure"><img src="/html/2407.07726/assets/x7.png" id="S5.F7.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="231" height="117" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>Training setup for Stage1.
Left: The more is frozen or reset, the more performance deteriorates.
Right: The effect of freezing ViT is most visible in some pretraining perplexities.</figcaption>
</figure>
</section>
<section id="S5.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.5 </span>Connector design</h3>

<div id="S5.SS5.p1" class="ltx_para">
<p id="S5.SS5.p1.1" class="ltx_p">Throughout our experiments we use a linear connector to map SigLiP output embeddings to the inputs of Gemma. Given that an MLP connector <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib66" title="" class="ltx_ref">66</a>]</cite> is a popular choice in the VLM literature, we also ablate this choice.</p>
</div>
<div id="S5.SS5.p2" class="ltx_para">
<p id="S5.SS5.p2.1" class="ltx_p">We consider two connector choices: a linear connector and an MLP (1 hidden layer, with GeLU non-linearity). We also consider two Stage1 pretraining settings: tune all weights (TT), or freeze everything but the connector (FF).</p>
</div>
<div id="S5.SS5.p3" class="ltx_para">
<p id="S5.SS5.p3.4" class="ltx_p">When tuning all weights, average transfer score is nearly identical for linear vs MLP, achieving <math id="S5.SS5.p3.1.m1.1" class="ltx_Math" alttext="77.2" display="inline"><semantics id="S5.SS5.p3.1.m1.1a"><mn id="S5.SS5.p3.1.m1.1.1" xref="S5.SS5.p3.1.m1.1.1.cmml">77.2</mn><annotation-xml encoding="MathML-Content" id="S5.SS5.p3.1.m1.1b"><cn type="float" id="S5.SS5.p3.1.m1.1.1.cmml" xref="S5.SS5.p3.1.m1.1.1">77.2</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS5.p3.1.m1.1c">77.2</annotation></semantics></math> and <math id="S5.SS5.p3.2.m2.1" class="ltx_Math" alttext="77.1" display="inline"><semantics id="S5.SS5.p3.2.m2.1a"><mn id="S5.SS5.p3.2.m2.1.1" xref="S5.SS5.p3.2.m2.1.1.cmml">77.1</mn><annotation-xml encoding="MathML-Content" id="S5.SS5.p3.2.m2.1b"><cn type="float" id="S5.SS5.p3.2.m2.1.1.cmml" xref="S5.SS5.p3.2.m2.1.1">77.1</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS5.p3.2.m2.1c">77.1</annotation></semantics></math> points respectively.
In the “all-frozen” scenario, linear vs MLP achieve <math id="S5.SS5.p3.3.m3.1" class="ltx_Math" alttext="70.7" display="inline"><semantics id="S5.SS5.p3.3.m3.1a"><mn id="S5.SS5.p3.3.m3.1.1" xref="S5.SS5.p3.3.m3.1.1.cmml">70.7</mn><annotation-xml encoding="MathML-Content" id="S5.SS5.p3.3.m3.1b"><cn type="float" id="S5.SS5.p3.3.m3.1.1.cmml" xref="S5.SS5.p3.3.m3.1.1">70.7</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS5.p3.3.m3.1c">70.7</annotation></semantics></math> vs <math id="S5.SS5.p3.4.m4.1" class="ltx_Math" alttext="69.7" display="inline"><semantics id="S5.SS5.p3.4.m4.1a"><mn id="S5.SS5.p3.4.m4.1.1" xref="S5.SS5.p3.4.m4.1.1.cmml">69.7</mn><annotation-xml encoding="MathML-Content" id="S5.SS5.p3.4.m4.1b"><cn type="float" id="S5.SS5.p3.4.m4.1.1.cmml" xref="S5.SS5.p3.4.m4.1.1">69.7</cn></annotation-xml><annotation encoding="application/x-tex" id="S5.SS5.p3.4.m4.1c">69.7</annotation></semantics></math>.
Surprisingly, we observe a small performance deterioration with the MLP connector.</p>
</div>
<div id="S5.SS5.p4" class="ltx_para">
<p id="S5.SS5.p4.1" class="ltx_p">Overall, we conclude that in our case, the linear connector seems preferable to the MLP connector.</p>
</div>
</section>
<section id="S5.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.6 </span>Image encoder: with or without?</h3>

<figure id="S5.F8" class="ltx_figure"><img src="/html/2407.07726/assets/x8.png" id="S5.F8.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="231" height="117" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 8: </span>Not using a SigLiP image encoder at all and instead passing a linear projection of raw RGB patches to Gemma works, but is significantly less sample-efficient.</figcaption>
</figure>
<div id="S5.SS6.p1" class="ltx_para">
<p id="S5.SS6.p1.1" class="ltx_p">Most VLMs follow the setup of having an image encoder, such as CLIP/SigLIP (most works) or VQGAN (the Chameleon line of work <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib2" title="" class="ltx_ref">2</a>, <a href="#bib.bib3" title="" class="ltx_ref">3</a>, <a href="#bib.bib125" title="" class="ltx_ref">125</a>, <a href="#bib.bib101" title="" class="ltx_ref">101</a>]</cite>), to turn the image into soft tokens before passing them to the LLM.
We are aware of only two works that attempt to simplify this overall setup by removing the image encoder entirely and passing raw image patches into a decoder-only LLM, namely Fuyu <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite> and EVE <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib33" title="" class="ltx_ref">33</a>]</cite>.
Unfortunately, the former provides no training details or ablations.
The latter, which is a concurrent work, provides some details about training and various ablations, but with mixed results.</p>
</div>
<div id="S5.SS6.p2" class="ltx_para">
<p id="S5.SS6.p2.1" class="ltx_p">Removing the SigLIP encoder in PaliGemma results in a model of the same unified decoder-only architecture.
We run our Stage1 and transfers in this setting.
Since the architecture significantly changed, we also re-tune the learning-rate of Stage1.
Figure <a href="#S5.F8" title="Figure 8 ‣ 5.6 Image encoder: with or without? ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> (per-task breakdown in Appendix <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:sec:app:enc</span>) shows that while this architecture still significantly lags behind, the scaling with pretraining duration seems potentially promising.
This is especially noteworthy considering that PaliGemma’s SigLIP encoder has seen 40 B image-text pairs during its Stage0 pre-training, while the Fuyu-style model sees images for the first time in the Stage1 pre-training shown here, and only sees up to 1 B of them in our experiment.
This ablation confirms that such decoder-only VLMs might be a promising future direction towards simpler multimodal models, although they currently still suffer in training efficiency due to not being able to reuse vision components.</p>
</div>
</section>
<section id="S5.SS7" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.7 </span>Image resolution</h3>

<div id="S5.SS7.p1" class="ltx_para">
<p id="S5.SS7.p1.1" class="ltx_p">Image resolution in VLMs is an important topic that has recently received increased attention <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib78" title="" class="ltx_ref">78</a>, <a href="#bib.bib57" title="" class="ltx_ref">57</a>, <a href="#bib.bib103" title="" class="ltx_ref">103</a>, <a href="#bib.bib34" title="" class="ltx_ref">34</a>, <a href="#bib.bib117" title="" class="ltx_ref">117</a>]</cite>.
PaliGemma uses a very simple approach to deal with resolution: Stage1 is pretrained at relatively low and economical 224px resolution, and short Stage2 then “upcycles” this checkpoint to higher resolutions (448px and 896px).
Hence, the final PaliGemma model comes with three different checkpoints for three different resolutions.</p>
</div>
<div id="S5.SS7.p2" class="ltx_para">
<p id="S5.SS7.p2.1" class="ltx_p">In this section we justify our approach and the necessity for providing three separate checkpoints, and compare it to a recently popular “windowing” approach.
For the ablation studies, we consider resolutions 224px and 448px, and restrict evaluation to single-image tasks where resolution has significant impact on performance according to Table <a href="#S4.T1" title="Table 1 ‣ 4 Results ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, which we call “Resolution-sensitive” tasks.</p>
</div>
<section id="S5.SS7.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">5.7.1 </span>Resolution or sequence length?</h4>

<figure id="S5.F9" class="ltx_figure"><img src="/html/2407.07726/assets/x9.png" id="S5.F9.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="231" height="117" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 9: </span>Increasing resolution has two effects: increased information content of the input image, and increased model capacity via sequence length. For tasks that benefit from increased resolution, both of these effects contribute roughly equally to the overall gain.</figcaption>
</figure>
<div id="S5.SS7.SSS1.p1" class="ltx_para">
<p id="S5.SS7.SSS1.p1.1" class="ltx_p">We generally see either neutral or improved performance across tasks when increasing the resolution of the input image (see Table <a href="#S4.T1" title="Table 1 ‣ 4 Results ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>).
However, it is unclear whether this improvement comes from the fact that the image has higher resolution and therefore more information, or whether it is thanks to the resulting longer sequence length and thus increased model FLOPs and capacity.
We disentangle these by running Stage2 and transfers at 448 px resolution, <em id="S5.SS7.SSS1.p1.1.1" class="ltx_emph ltx_font_italic">but downscaling each image to 224 px before resizing it to 448 px</em>.
Thus, the model gets to see the information content of a low-res (224 px) image but with the model capacity of the high-res (448 px) setting.</p>
</div>
<div id="S5.SS7.SSS1.p2" class="ltx_para">
<p id="S5.SS7.SSS1.p2.1" class="ltx_p">The result in Fig <a href="#S5.F9" title="Figure 9 ‣ 5.7.1 Resolution or sequence length? ‣ 5.7 Image resolution ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a> shows that, for those tasks for which resolution has an effect at all, the reason for the improved performance is split roughly equally between these two causes.
This is true for every individual task and not just an effect of averaging, see Appendix <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:sec:app:res_or_seqlen</span>.</p>
</div>
</section>
<section id="S5.SS7.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">5.7.2 </span>Need resolution-specific checkpoints?</h4>

<figure id="S5.F10" class="ltx_figure"><img src="/html/2407.07726/assets/x10.png" id="S5.F10.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="231" height="117" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 10: </span>Different ways of increasing resolution.
While resolution during transfers works (orange), it is not the best setup.
Using “windows” works better (mauve, <math id="S5.F10.4.m1.1" class="ltx_Math" alttext="s_{w}" display="inline"><semantics id="S5.F10.4.m1.1b"><msub id="S5.F10.4.m1.1.1" xref="S5.F10.4.m1.1.1.cmml"><mi id="S5.F10.4.m1.1.1.2" xref="S5.F10.4.m1.1.1.2.cmml">s</mi><mi id="S5.F10.4.m1.1.1.3" xref="S5.F10.4.m1.1.1.3.cmml">w</mi></msub><annotation-xml encoding="MathML-Content" id="S5.F10.4.m1.1c"><apply id="S5.F10.4.m1.1.1.cmml" xref="S5.F10.4.m1.1.1"><csymbol cd="ambiguous" id="S5.F10.4.m1.1.1.1.cmml" xref="S5.F10.4.m1.1.1">subscript</csymbol><ci id="S5.F10.4.m1.1.1.2.cmml" xref="S5.F10.4.m1.1.1.2">𝑠</ci><ci id="S5.F10.4.m1.1.1.3.cmml" xref="S5.F10.4.m1.1.1.3">𝑤</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.F10.4.m1.1d">s_{w}</annotation></semantics></math> means image resolution <math id="S5.F10.5.m2.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S5.F10.5.m2.1b"><mi id="S5.F10.5.m2.1.1" xref="S5.F10.5.m2.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S5.F10.5.m2.1c"><ci id="S5.F10.5.m2.1.1.cmml" xref="S5.F10.5.m2.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F10.5.m2.1d">s</annotation></semantics></math> with window-size <math id="S5.F10.6.m3.1" class="ltx_Math" alttext="w" display="inline"><semantics id="S5.F10.6.m3.1b"><mi id="S5.F10.6.m3.1.1" xref="S5.F10.6.m3.1.1.cmml">w</mi><annotation-xml encoding="MathML-Content" id="S5.F10.6.m3.1c"><ci id="S5.F10.6.m3.1.1.cmml" xref="S5.F10.6.m3.1.1">𝑤</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F10.6.m3.1d">w</annotation></semantics></math>),
but simply running Stage2 at increased resolution with no tricks works best, justifying the need to provide multiple checkpoints. Per-task breakdown in Appendix <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:sec:app:res_window</span>.</figcaption>
</figure>
<div id="S5.SS7.SSS2.p1" class="ltx_para">
<p id="S5.SS7.SSS2.p1.1" class="ltx_p">PaliGemma provides one checkpoint per resolution.
But is this really needed? Could we not provide just a single, maybe high-resolution checkpoint, and adapt it as needed during transfers?</p>
</div>
<div id="S5.SS7.SSS2.p2" class="ltx_para">
<p id="S5.SS7.SSS2.p2.1" class="ltx_p">Figure <a href="#S5.F10" title="Figure 10 ‣ 5.7.2 Need resolution-specific checkpoints? ‣ 5.7 Image resolution ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a> shows clearly that providing either only a 224 px or only a 448 px checkpoint would not be sufficient:
Transferring the 448 px checkpoint at 224 px resolution (first bar, orange) leads to significantly worse results than using the 224 px checkpoint (second bar, zero), even though the latter had slightly less pretraining.
Similarly, transferring the 224 px checkpoint at resolution 448 px (third bar, orange), while improving the results significantly, still lags far behind transferring the checkpoint whose native resolution is 448 px (last bar, green).</p>
</div>
<div id="S5.SS7.SSS2.p3" class="ltx_para">
<p id="S5.SS7.SSS2.p3.1" class="ltx_p">Thus, in the absence of flexible-resolution modeling tricks such as FlexiViT <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib13" title="" class="ltx_ref">13</a>]</cite> or NaViT <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite>, we recommend running extended pretraining for increasing resolution (Stage2) and providing separate checkpoints for all supported resolutions.</p>
</div>
</section>
<section id="S5.SS7.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">5.7.3 </span>To resize or to window?</h4>

<div id="S5.SS7.SSS3.p1" class="ltx_para">
<p id="S5.SS7.SSS3.p1.1" class="ltx_p">Another recently common way of increasing input resolution is by “windowing” the models <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib110" title="" class="ltx_ref">110</a>, <a href="#bib.bib117" title="" class="ltx_ref">117</a>, <a href="#bib.bib130" title="" class="ltx_ref">130</a>]</cite>, <em id="S5.SS7.SSS3.p1.1.1" class="ltx_emph ltx_font_italic">i.e</em>.<span id="S5.SS7.SSS3.p1.1.2" class="ltx_text"></span> applying the same model on windows of the model’s native resolution from the higher-resolution image.
We evaluate the simplest variant of this alternative: the 448 px resolution image is cut into four pieces, each one passed through the SigLIP image encoder separately.
All four sets of 256 image embedding tokens are then concatenated and passed to Gemma.
This is also related to using windowed attention in a ViT taking the full image <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib62" title="" class="ltx_ref">62</a>]</cite>.
We experimented with adding extra “window ID” position embeddings to indicate which window tokens come from, but this did not significantly change any of the results.</p>
</div>
<div id="S5.SS7.SSS3.p2" class="ltx_para">
<p id="S5.SS7.SSS3.p2.1" class="ltx_p">The mauve bars in Figure <a href="#S5.F10" title="Figure 10 ‣ 5.7.2 Need resolution-specific checkpoints? ‣ 5.7 Image resolution ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a> correspond to windowing settings.
While overall the performance is worse than that of a native 448px model, the windowing approach can be a promising way of transferring a model to a higher resolution <em id="S5.SS7.SSS3.p2.1.1" class="ltx_emph ltx_font_italic">when no higher-resolution checkpoint is made available</em>, and when running a Stage2-like continued pretraining is not feasible.</p>
</div>
<div id="S5.SS7.SSS3.p3" class="ltx_para">
<p id="S5.SS7.SSS3.p3.1" class="ltx_p">Windowing might still seem preferable for speed reasons.
However, we only observed at most a 5% speedup in training across various setups from windowing.
This is explained by Gemma being significantly larger than ViT-So400m, and the Gemma part of the model is unaffected by windowing.</p>
</div>
</section>
<section id="S5.SS7.SSS4" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">5.7.4 </span>Stage2 mixture re-weighting</h4>

<div id="S5.SS7.SSS4.p1" class="ltx_para">
<p id="S5.SS7.SSS4.p1.1" class="ltx_p">Finally, the pretraining mixture changes between Stage1 and Stage2.
While previous PaLI versions change the mixture makeup, for PaliGemma we always use the full set of tasks, only changing their weighting, sampling “resolution-related” tasks (OCR, detection, segmentation) more frequently at higher resolutions.
As an ablation, we run a Stage2 training with the same mixture ratios as Stage1.
After transfers, this checkpoint is significantly worse on only three tasks (DocVQA, ChartQA, XM3600), but otherwise within per-task variance (Full results in Appendix <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:sec:app:s2_reweight</span>).</p>
</div>
<div id="S5.SS7.SSS4.p2" class="ltx_para">
<p id="S5.SS7.SSS4.p2.1" class="ltx_p">Thus, while changing the mixing ratio helped a little, it seems that when the intent is to train a base model for fine-tuning, the precise mixture ratios might not be as important as when training a model intended for sampling zero-shot, where it would significantly affect sampling frequencies.</p>
</div>
</section>
</section>
</section>
<section id="S6" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Transferability</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">To show that PaliGemma is suitable for transfer under different scenarios, we run experiments that quantify its repeatability, sensitivity to hyper-parameters, and number of transfer examples.</p>
</div>
<section id="S6.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.1 </span>Repeatability (variance)</h3>

<figure id="S6.F11" class="ltx_figure"><img src="/html/2407.07726/assets/x11.png" id="S6.F11.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="231" height="155" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 11: </span>For most tasks, the standard deviation of final metric values between re-runs with different seeds is below <math id="S6.F11.2.m1.1" class="ltx_Math" alttext="0.5" display="inline"><semantics id="S6.F11.2.m1.1b"><mn id="S6.F11.2.m1.1.1" xref="S6.F11.2.m1.1.1.cmml">0.5</mn><annotation-xml encoding="MathML-Content" id="S6.F11.2.m1.1c"><cn type="float" id="S6.F11.2.m1.1.1.cmml" xref="S6.F11.2.m1.1.1">0.5</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.F11.2.m1.1d">0.5</annotation></semantics></math>. This is true both for Stage1 (top, single transfers of three 100M runs), as well as for transfers (bottom, values from Table <a href="#S4.T1" title="Table 1 ‣ 4 Results ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>).</figcaption>
</figure>
<div id="S6.SS1.p1" class="ltx_para">
<p id="S6.SS1.p1.1" class="ltx_p">In the main results (Table <a href="#S4.T1" title="Table 1 ‣ 4 Results ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>), we show the standard deviation across 5 transfer reruns using the best hyper-parameter (listed in <a href="#A10" title="Appendix J Hyper-parameters ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">J</span></a>).
It is generally very small across most tasks, meaning transfer from the pretrained checkpoint is highly repeatable.
In Figure <a href="#S6.F11" title="Figure 11 ‣ 6.1 Repeatability (variance) ‣ 6 Transferability ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a> we further show that the standard deviation of transferring from three reruns of Stage1 falls within the same range, meaning pretraining itself is also highly repeatable.</p>
</div>
</section>
<section id="S6.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.2 </span>Transfer hyper-parameter sensitivity</h3>

<figure id="S6.T2" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 2: </span>Relative regret of using the suggested hyper-parameter setting for all tasks instead of performing hyper-parameter search. Per task plot in Appendix  <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:sec:app:transfer_simple</span>.</figcaption>
<table id="S6.T2.4" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S6.T2.4.5.1" class="ltx_tr">
<td id="S6.T2.4.5.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="S6.T2.4.5.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="S6.T2.4.5.1.1.1.1" class="ltx_p" style="width:68.3pt;"><span id="S6.T2.4.5.1.1.1.1.1" class="ltx_text ltx_font_bold">Rel. Regret</span></span>
</span>
</td>
<td id="S6.T2.4.5.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="S6.T2.4.5.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S6.T2.4.5.1.2.1.1" class="ltx_p" style="width:8.5pt;"><span id="S6.T2.4.5.1.2.1.1.1" class="ltx_text ltx_font_bold">#</span></span>
</span>
</td>
<td id="S6.T2.4.5.1.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:113.8pt;">
<span id="S6.T2.4.5.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="S6.T2.4.5.1.3.1.1" class="ltx_p"><span id="S6.T2.4.5.1.3.1.1.1" class="ltx_text ltx_font_bold">Tasks</span></span>
</span>
</td>
</tr>
<tr id="S6.T2.1.1" class="ltx_tr">
<td id="S6.T2.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S6.T2.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="S6.T2.1.1.1.1.1" class="ltx_p" style="width:68.3pt;"><math id="S6.T2.1.1.1.1.1.m1.2" class="ltx_Math" alttext="[\text{None},2.5\%)" display="inline"><semantics id="S6.T2.1.1.1.1.1.m1.2a"><mrow id="S6.T2.1.1.1.1.1.m1.2.2.1" xref="S6.T2.1.1.1.1.1.m1.2.2.2.cmml"><mo stretchy="false" id="S6.T2.1.1.1.1.1.m1.2.2.1.2" xref="S6.T2.1.1.1.1.1.m1.2.2.2.cmml">[</mo><mtext id="S6.T2.1.1.1.1.1.m1.1.1" xref="S6.T2.1.1.1.1.1.m1.1.1a.cmml">None</mtext><mo id="S6.T2.1.1.1.1.1.m1.2.2.1.3" xref="S6.T2.1.1.1.1.1.m1.2.2.2.cmml">,</mo><mrow id="S6.T2.1.1.1.1.1.m1.2.2.1.1" xref="S6.T2.1.1.1.1.1.m1.2.2.1.1.cmml"><mn id="S6.T2.1.1.1.1.1.m1.2.2.1.1.2" xref="S6.T2.1.1.1.1.1.m1.2.2.1.1.2.cmml">2.5</mn><mo id="S6.T2.1.1.1.1.1.m1.2.2.1.1.1" xref="S6.T2.1.1.1.1.1.m1.2.2.1.1.1.cmml">%</mo></mrow><mo stretchy="false" id="S6.T2.1.1.1.1.1.m1.2.2.1.4" xref="S6.T2.1.1.1.1.1.m1.2.2.2.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.T2.1.1.1.1.1.m1.2b"><interval closure="closed-open" id="S6.T2.1.1.1.1.1.m1.2.2.2.cmml" xref="S6.T2.1.1.1.1.1.m1.2.2.1"><ci id="S6.T2.1.1.1.1.1.m1.1.1a.cmml" xref="S6.T2.1.1.1.1.1.m1.1.1"><mtext id="S6.T2.1.1.1.1.1.m1.1.1.cmml" xref="S6.T2.1.1.1.1.1.m1.1.1">None</mtext></ci><apply id="S6.T2.1.1.1.1.1.m1.2.2.1.1.cmml" xref="S6.T2.1.1.1.1.1.m1.2.2.1.1"><csymbol cd="latexml" id="S6.T2.1.1.1.1.1.m1.2.2.1.1.1.cmml" xref="S6.T2.1.1.1.1.1.m1.2.2.1.1.1">percent</csymbol><cn type="float" id="S6.T2.1.1.1.1.1.m1.2.2.1.1.2.cmml" xref="S6.T2.1.1.1.1.1.m1.2.2.1.1.2">2.5</cn></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S6.T2.1.1.1.1.1.m1.2c">[\text{None},2.5\%)</annotation></semantics></math></span>
</span>
</td>
<td id="S6.T2.1.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S6.T2.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S6.T2.1.1.2.1.1" class="ltx_p" style="width:8.5pt;">37</span>
</span>
</td>
<td id="S6.T2.1.1.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:113.8pt;">
<span id="S6.T2.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="S6.T2.1.1.3.1.1" class="ltx_p">All other tasks</span>
</span>
</td>
</tr>
<tr id="S6.T2.2.2" class="ltx_tr">
<td id="S6.T2.2.2.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S6.T2.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="S6.T2.2.2.1.1.1" class="ltx_p" style="width:68.3pt;"><math id="S6.T2.2.2.1.1.1.m1.2" class="ltx_Math" alttext="[2.5\%,5.0\%)" display="inline"><semantics id="S6.T2.2.2.1.1.1.m1.2a"><mrow id="S6.T2.2.2.1.1.1.m1.2.2.2" xref="S6.T2.2.2.1.1.1.m1.2.2.3.cmml"><mo stretchy="false" id="S6.T2.2.2.1.1.1.m1.2.2.2.3" xref="S6.T2.2.2.1.1.1.m1.2.2.3.cmml">[</mo><mrow id="S6.T2.2.2.1.1.1.m1.1.1.1.1" xref="S6.T2.2.2.1.1.1.m1.1.1.1.1.cmml"><mn id="S6.T2.2.2.1.1.1.m1.1.1.1.1.2" xref="S6.T2.2.2.1.1.1.m1.1.1.1.1.2.cmml">2.5</mn><mo id="S6.T2.2.2.1.1.1.m1.1.1.1.1.1" xref="S6.T2.2.2.1.1.1.m1.1.1.1.1.1.cmml">%</mo></mrow><mo id="S6.T2.2.2.1.1.1.m1.2.2.2.4" xref="S6.T2.2.2.1.1.1.m1.2.2.3.cmml">,</mo><mrow id="S6.T2.2.2.1.1.1.m1.2.2.2.2" xref="S6.T2.2.2.1.1.1.m1.2.2.2.2.cmml"><mn id="S6.T2.2.2.1.1.1.m1.2.2.2.2.2" xref="S6.T2.2.2.1.1.1.m1.2.2.2.2.2.cmml">5.0</mn><mo id="S6.T2.2.2.1.1.1.m1.2.2.2.2.1" xref="S6.T2.2.2.1.1.1.m1.2.2.2.2.1.cmml">%</mo></mrow><mo stretchy="false" id="S6.T2.2.2.1.1.1.m1.2.2.2.5" xref="S6.T2.2.2.1.1.1.m1.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.T2.2.2.1.1.1.m1.2b"><interval closure="closed-open" id="S6.T2.2.2.1.1.1.m1.2.2.3.cmml" xref="S6.T2.2.2.1.1.1.m1.2.2.2"><apply id="S6.T2.2.2.1.1.1.m1.1.1.1.1.cmml" xref="S6.T2.2.2.1.1.1.m1.1.1.1.1"><csymbol cd="latexml" id="S6.T2.2.2.1.1.1.m1.1.1.1.1.1.cmml" xref="S6.T2.2.2.1.1.1.m1.1.1.1.1.1">percent</csymbol><cn type="float" id="S6.T2.2.2.1.1.1.m1.1.1.1.1.2.cmml" xref="S6.T2.2.2.1.1.1.m1.1.1.1.1.2">2.5</cn></apply><apply id="S6.T2.2.2.1.1.1.m1.2.2.2.2.cmml" xref="S6.T2.2.2.1.1.1.m1.2.2.2.2"><csymbol cd="latexml" id="S6.T2.2.2.1.1.1.m1.2.2.2.2.1.cmml" xref="S6.T2.2.2.1.1.1.m1.2.2.2.2.1">percent</csymbol><cn type="float" id="S6.T2.2.2.1.1.1.m1.2.2.2.2.2.cmml" xref="S6.T2.2.2.1.1.1.m1.2.2.2.2.2">5.0</cn></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S6.T2.2.2.1.1.1.m1.2c">[2.5\%,5.0\%)</annotation></semantics></math></span>
</span>
</td>
<td id="S6.T2.2.2.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S6.T2.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="S6.T2.2.2.2.1.1" class="ltx_p" style="width:8.5pt;">2</span>
</span>
</td>
<td id="S6.T2.2.2.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:113.8pt;">
<span id="S6.T2.2.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="S6.T2.2.2.3.1.1" class="ltx_p">ChartQA (human): <span id="S6.T2.2.2.3.1.1.1" class="ltx_text" style="font-size:80%;">3.2%</span></span>
</span>
</td>
</tr>
<tr id="S6.T2.4.6.2" class="ltx_tr">
<td id="S6.T2.4.6.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S6.T2.4.6.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="S6.T2.4.6.2.1.1.1" class="ltx_p" style="width:68.3pt;">RefCOCO (val): <span id="S6.T2.4.6.2.1.1.1.1" class="ltx_text" style="font-size:80%;">4.8%</span></span>
</span>
</td>
<td id="S6.T2.4.6.2.2" class="ltx_td ltx_align_justify ltx_align_top"></td>
<td id="S6.T2.4.6.2.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;"></td>
</tr>
<tr id="S6.T2.3.3" class="ltx_tr">
<td id="S6.T2.3.3.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S6.T2.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="S6.T2.3.3.1.1.1" class="ltx_p" style="width:68.3pt;"><math id="S6.T2.3.3.1.1.1.m1.2" class="ltx_Math" alttext="[5.0\%,10.0\%)" display="inline"><semantics id="S6.T2.3.3.1.1.1.m1.2a"><mrow id="S6.T2.3.3.1.1.1.m1.2.2.2" xref="S6.T2.3.3.1.1.1.m1.2.2.3.cmml"><mo stretchy="false" id="S6.T2.3.3.1.1.1.m1.2.2.2.3" xref="S6.T2.3.3.1.1.1.m1.2.2.3.cmml">[</mo><mrow id="S6.T2.3.3.1.1.1.m1.1.1.1.1" xref="S6.T2.3.3.1.1.1.m1.1.1.1.1.cmml"><mn id="S6.T2.3.3.1.1.1.m1.1.1.1.1.2" xref="S6.T2.3.3.1.1.1.m1.1.1.1.1.2.cmml">5.0</mn><mo id="S6.T2.3.3.1.1.1.m1.1.1.1.1.1" xref="S6.T2.3.3.1.1.1.m1.1.1.1.1.1.cmml">%</mo></mrow><mo id="S6.T2.3.3.1.1.1.m1.2.2.2.4" xref="S6.T2.3.3.1.1.1.m1.2.2.3.cmml">,</mo><mrow id="S6.T2.3.3.1.1.1.m1.2.2.2.2" xref="S6.T2.3.3.1.1.1.m1.2.2.2.2.cmml"><mn id="S6.T2.3.3.1.1.1.m1.2.2.2.2.2" xref="S6.T2.3.3.1.1.1.m1.2.2.2.2.2.cmml">10.0</mn><mo id="S6.T2.3.3.1.1.1.m1.2.2.2.2.1" xref="S6.T2.3.3.1.1.1.m1.2.2.2.2.1.cmml">%</mo></mrow><mo stretchy="false" id="S6.T2.3.3.1.1.1.m1.2.2.2.5" xref="S6.T2.3.3.1.1.1.m1.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.T2.3.3.1.1.1.m1.2b"><interval closure="closed-open" id="S6.T2.3.3.1.1.1.m1.2.2.3.cmml" xref="S6.T2.3.3.1.1.1.m1.2.2.2"><apply id="S6.T2.3.3.1.1.1.m1.1.1.1.1.cmml" xref="S6.T2.3.3.1.1.1.m1.1.1.1.1"><csymbol cd="latexml" id="S6.T2.3.3.1.1.1.m1.1.1.1.1.1.cmml" xref="S6.T2.3.3.1.1.1.m1.1.1.1.1.1">percent</csymbol><cn type="float" id="S6.T2.3.3.1.1.1.m1.1.1.1.1.2.cmml" xref="S6.T2.3.3.1.1.1.m1.1.1.1.1.2">5.0</cn></apply><apply id="S6.T2.3.3.1.1.1.m1.2.2.2.2.cmml" xref="S6.T2.3.3.1.1.1.m1.2.2.2.2"><csymbol cd="latexml" id="S6.T2.3.3.1.1.1.m1.2.2.2.2.1.cmml" xref="S6.T2.3.3.1.1.1.m1.2.2.2.2.1">percent</csymbol><cn type="float" id="S6.T2.3.3.1.1.1.m1.2.2.2.2.2.cmml" xref="S6.T2.3.3.1.1.1.m1.2.2.2.2.2">10.0</cn></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S6.T2.3.3.1.1.1.m1.2c">[5.0\%,10.0\%)</annotation></semantics></math></span>
</span>
</td>
<td id="S6.T2.3.3.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S6.T2.3.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="S6.T2.3.3.2.1.1" class="ltx_p" style="width:8.5pt;">2</span>
</span>
</td>
<td id="S6.T2.3.3.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:113.8pt;">
<span id="S6.T2.3.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="S6.T2.3.3.3.1.1" class="ltx_p">RefCOCOg (val): <span id="S6.T2.3.3.3.1.1.1" class="ltx_text" style="font-size:80%;">5.8%</span></span>
</span>
</td>
</tr>
<tr id="S6.T2.4.7.3" class="ltx_tr">
<td id="S6.T2.4.7.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S6.T2.4.7.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="S6.T2.4.7.3.1.1.1" class="ltx_p" style="width:68.3pt;">ScienceQA: <span id="S6.T2.4.7.3.1.1.1.1" class="ltx_text" style="font-size:80%;">6.7%</span></span>
</span>
</td>
<td id="S6.T2.4.7.3.2" class="ltx_td ltx_align_justify ltx_align_top"></td>
<td id="S6.T2.4.7.3.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;"></td>
</tr>
<tr id="S6.T2.4.4" class="ltx_tr">
<td id="S6.T2.4.4.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S6.T2.4.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="S6.T2.4.4.1.1.1" class="ltx_p" style="width:68.3pt;"><math id="S6.T2.4.4.1.1.1.m1.2" class="ltx_Math" alttext="[10.0\%,100\%]" display="inline"><semantics id="S6.T2.4.4.1.1.1.m1.2a"><mrow id="S6.T2.4.4.1.1.1.m1.2.2.2" xref="S6.T2.4.4.1.1.1.m1.2.2.3.cmml"><mo stretchy="false" id="S6.T2.4.4.1.1.1.m1.2.2.2.3" xref="S6.T2.4.4.1.1.1.m1.2.2.3.cmml">[</mo><mrow id="S6.T2.4.4.1.1.1.m1.1.1.1.1" xref="S6.T2.4.4.1.1.1.m1.1.1.1.1.cmml"><mn id="S6.T2.4.4.1.1.1.m1.1.1.1.1.2" xref="S6.T2.4.4.1.1.1.m1.1.1.1.1.2.cmml">10.0</mn><mo id="S6.T2.4.4.1.1.1.m1.1.1.1.1.1" xref="S6.T2.4.4.1.1.1.m1.1.1.1.1.1.cmml">%</mo></mrow><mo id="S6.T2.4.4.1.1.1.m1.2.2.2.4" xref="S6.T2.4.4.1.1.1.m1.2.2.3.cmml">,</mo><mrow id="S6.T2.4.4.1.1.1.m1.2.2.2.2" xref="S6.T2.4.4.1.1.1.m1.2.2.2.2.cmml"><mn id="S6.T2.4.4.1.1.1.m1.2.2.2.2.2" xref="S6.T2.4.4.1.1.1.m1.2.2.2.2.2.cmml">100</mn><mo id="S6.T2.4.4.1.1.1.m1.2.2.2.2.1" xref="S6.T2.4.4.1.1.1.m1.2.2.2.2.1.cmml">%</mo></mrow><mo stretchy="false" id="S6.T2.4.4.1.1.1.m1.2.2.2.5" xref="S6.T2.4.4.1.1.1.m1.2.2.3.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.T2.4.4.1.1.1.m1.2b"><interval closure="closed" id="S6.T2.4.4.1.1.1.m1.2.2.3.cmml" xref="S6.T2.4.4.1.1.1.m1.2.2.2"><apply id="S6.T2.4.4.1.1.1.m1.1.1.1.1.cmml" xref="S6.T2.4.4.1.1.1.m1.1.1.1.1"><csymbol cd="latexml" id="S6.T2.4.4.1.1.1.m1.1.1.1.1.1.cmml" xref="S6.T2.4.4.1.1.1.m1.1.1.1.1.1">percent</csymbol><cn type="float" id="S6.T2.4.4.1.1.1.m1.1.1.1.1.2.cmml" xref="S6.T2.4.4.1.1.1.m1.1.1.1.1.2">10.0</cn></apply><apply id="S6.T2.4.4.1.1.1.m1.2.2.2.2.cmml" xref="S6.T2.4.4.1.1.1.m1.2.2.2.2"><csymbol cd="latexml" id="S6.T2.4.4.1.1.1.m1.2.2.2.2.1.cmml" xref="S6.T2.4.4.1.1.1.m1.2.2.2.2.1">percent</csymbol><cn type="integer" id="S6.T2.4.4.1.1.1.m1.2.2.2.2.2.cmml" xref="S6.T2.4.4.1.1.1.m1.2.2.2.2.2">100</cn></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S6.T2.4.4.1.1.1.m1.2c">[10.0\%,100\%]</annotation></semantics></math></span>
</span>
</td>
<td id="S6.T2.4.4.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S6.T2.4.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="S6.T2.4.4.2.1.1" class="ltx_p" style="width:8.5pt;">2</span>
</span>
</td>
<td id="S6.T2.4.4.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:113.8pt;">
<span id="S6.T2.4.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="S6.T2.4.4.3.1.1" class="ltx_p">RefCOCO+ (val): <span id="S6.T2.4.4.3.1.1.1" class="ltx_text" style="font-size:80%;">10.7%</span></span>
</span>
</td>
</tr>
<tr id="S6.T2.4.8.4" class="ltx_tr">
<td id="S6.T2.4.8.4.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="S6.T2.4.8.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="S6.T2.4.8.4.1.1.1" class="ltx_p" style="width:68.3pt;">SciCap: <span id="S6.T2.4.8.4.1.1.1.1" class="ltx_text" style="font-size:80%;">60.5%</span></span>
</span>
</td>
<td id="S6.T2.4.8.4.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb"></td>
<td id="S6.T2.4.8.4.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb" style="width:113.8pt;"></td>
</tr>
</tbody>
</table>
</figure>
<div id="S6.SS2.p1" class="ltx_para">
<p id="S6.SS2.p1.1" class="ltx_p">However one question a reader may have is whether the choice of transfer hyper-parameter is important.
To ablate that we run all tasks at 224px and under a single and simple hyper-parameter setup, which was also highlighted in bold in Section <a href="#S3.SS2.SSS4" title="3.2.4 Stage3: Transfer ‣ 3.2 Pretraining ‣ 3 Model ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2.4</span></a>: lr=1e-5, bs=256, no dropout, no label smoothing, no weight decay, and not freezing anything.
The only task dependent parameter is the number of epochs for which we use each task’s best one but cap it at 10.</p>
</div>
<div id="S6.SS2.p2" class="ltx_para">
<p id="S6.SS2.p2.1" class="ltx_p">The results (Table <a href="#S6.T2" title="Table 2 ‣ 6.2 Transfer hyper-parameter sensitivity ‣ 6 Transferability ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>) show that this simplified and single setup works very well for the majority of the tasks. The main exceptions we found were for tasks like RefCOCO and SciCap tasks which seem to benefit significantly from increasing the number of epochs while enabling label smoothing and dropout.</p>
</div>
</section>
<section id="S6.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.3 </span>Transfer with limited examples</h3>

<figure id="S6.F12" class="ltx_figure"><img src="/html/2407.07726/assets/x12.png" id="S6.F12.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="231" height="119" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 12: </span>Relative regret of using a limited number of transfer examples. Thick line represents the median. Per task plot in appendix <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:sec:app:transfer_lowdata</span>.</figcaption>
</figure>
<div id="S6.SS3.p1" class="ltx_para">
<p id="S6.SS3.p1.1" class="ltx_p">To analyze how many examples are needed to make PaliGemma solve a new task, we finetune PaliGemma with limited number of examples (64, 256, 1024, 4096).
We sweep transfer with varying learning rates, epochs and batch size and report the best number without separate minival, to indicate the potential.</p>
</div>
<div id="S6.SS3.p2" class="ltx_para">
<p id="S6.SS3.p2.1" class="ltx_p">We run every setting with 5 different seeds, which also affect which examples are used.
We found this important, as finetuning with limited examples exhibits high variance for some tasks (<em id="S6.SS3.p2.1.1" class="ltx_emph ltx_font_italic">e.g</em>.<span id="S6.SS3.p2.1.2" class="ltx_text"></span> RefCOCO mIOU varied within 10%-30%).
As a note, this variance also occurs when repeating with the same examples, but different batch order.
Importantly, seed selection is not overfitting to the metric as the selected model performs equally well in the validation and test splits.But it does allows us to draw conclusions without needing to solve the open problem of making few-example fine-tuning stable.
</p>
</div>
<div id="S6.SS3.p3" class="ltx_para">
<p id="S6.SS3.p3.1" class="ltx_p">Overall, when comparing the best runs of each hyper-parameter and seed with the results obtained with the full dataset, Figure <a href="#S6.F12" title="Figure 12 ‣ 6.3 Transfer with limited examples ‣ 6 Transferability ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a> shows that it is not necessary to have a transfer dataset in the order of 10 k examples.
The majority of the tasks can reach within 10% of the full-data score when using 4 k examples and 20% when using only 256 examples. In many cases the score with 64 transfer examples are good enough to prototype using PaliGemma for a new application.</p>
</div>
</section>
</section>
<section id="S7" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Noteworthy tidbits</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.1" class="ltx_p">We also briefly discuss several small but interesting discoveries or we made, providing more details on each in the Appendix.</p>
</div>
<div id="S7.p2" class="ltx_para">
<p id="S7.p2.1" class="ltx_p"><span id="S7.p2.1.1" class="ltx_text ltx_font_bold">Plain resize to square for segmentation.</span> We found simple resize to square (<math id="S7.p2.1.m1.1" class="ltx_Math" alttext="224\times 224" display="inline"><semantics id="S7.p2.1.m1.1a"><mrow id="S7.p2.1.m1.1.1" xref="S7.p2.1.m1.1.1.cmml"><mn id="S7.p2.1.m1.1.1.2" xref="S7.p2.1.m1.1.1.2.cmml">224</mn><mo lspace="0.222em" rspace="0.222em" id="S7.p2.1.m1.1.1.1" xref="S7.p2.1.m1.1.1.1.cmml">×</mo><mn id="S7.p2.1.m1.1.1.3" xref="S7.p2.1.m1.1.1.3.cmml">224</mn></mrow><annotation-xml encoding="MathML-Content" id="S7.p2.1.m1.1b"><apply id="S7.p2.1.m1.1.1.cmml" xref="S7.p2.1.m1.1.1"><times id="S7.p2.1.m1.1.1.1.cmml" xref="S7.p2.1.m1.1.1.1"></times><cn type="integer" id="S7.p2.1.m1.1.1.2.cmml" xref="S7.p2.1.m1.1.1.2">224</cn><cn type="integer" id="S7.p2.1.m1.1.1.3.cmml" xref="S7.p2.1.m1.1.1.3">224</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.p2.1.m1.1c">224\times 224</annotation></semantics></math>) to work just as well as popular aspect-ratio preserving zoom and crop augmentations. (Appendix <a href="#A3" title="Appendix C Image augmentations for RefCOCO ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>)</p>
</div>
<div id="S7.p3" class="ltx_para">
<p id="S7.p3.1" class="ltx_p"><span id="S7.p3.1.1" class="ltx_text ltx_font_bold">Counting: introducing CountBenchQA.</span>
Due to skewed number distribution and varying image quality <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib49" title="" class="ltx_ref">49</a>]</cite>, we found TallyQA lacking in its ability to assess current VLM’s ability to count.
This is why we introduce CountBenchQA, a VLM-ready version of the CountBench <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib85" title="" class="ltx_ref">85</a>]</cite> dataset. (Appendix <a href="#A4" title="Appendix D Introducing CountBenchQA ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D</span></a>).</p>
</div>
<div id="S7.p4" class="ltx_para">
<p id="S7.p4.1" class="ltx_p"><span id="S7.p4.1.1" class="ltx_text ltx_font_bold">Issues in published WidgetCaps numbers.</span> We found issues in at least three previous work’s evaluation of WidgetCaps, rendering numerical comparisons invalid. (Appendix <a href="#A5" title="Appendix E Issues with published WidgetCaps numbers ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E</span></a>).</p>
</div>
<div id="S7.p5" class="ltx_para">
<p id="S7.p5.1" class="ltx_p"><span id="S7.p5.1.1" class="ltx_text ltx_font_bold">Image annotations work as well as prompts.</span> Marking the widget to be captioned with a red box in the image gives the same results as indicating it with <span id="S7.p5.1.2" class="ltx_text ltx_font_typewriter">&lt;loc&gt;</span> tokens in the prompt. (Appendix <a href="#A6" title="Appendix F Image annotations work as well as prompts ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">F</span></a>)</p>
</div>
<div id="S7.p6" class="ltx_para">
<p id="S7.p6.1" class="ltx_p"><span id="S7.p6.1.1" class="ltx_text ltx_font_bold">RoPE interpolation unnecessary for upscaling.</span> For Stage2, we tried interpolating the RoPE position indices for the image tokens to preserve their semantics from Stage1. However, we saw no benefit from doing so.</p>
</div>
<div id="S7.p7" class="ltx_para">
<p id="S7.p7.1" class="ltx_p"><span id="S7.p7.1.1" class="ltx_text ltx_font_bold">Zero-shot generalization to 3D renders.</span> Although never explicitly trained for it, PaliGemma generalizes surprisingly well to 3D renders form Objaverse without fine-tuning. (Appendix <a href="#A7" title="Appendix G More details and results with Objaverse ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">G</span></a>)</p>
</div>
<div id="S7.p8" class="ltx_para">
<p id="S7.p8.1" class="ltx_p"><span id="S7.p8.1.1" class="ltx_text ltx_font_bold">Our MMVP result is SOTA by a large margin.</span> PaliGemma at 224px achieves 47.3% paired accuracy, while GPT4-V and Gemini achieve 38.7% and 40.7%, respectively, and all other models including LLaVa perform below chance.</p>
</div>
</section>
<section id="S8" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">8 </span>Conclusion</h2>

<div id="S8.p1" class="ltx_para">
<p id="S8.p1.1" class="ltx_p">PaliGemma is a new, small, open base VLM that shines when transferred to a broad range of tasks.
Our results show that VLMs on the “smaller” side can provide state-of-the-art performance across a wide variety of benchmarks.
We also hope that providing the base model without instruction tuning serves as a useful starting point for further research in instruction tuning, specific applications, and encourages clearer separation of base models and fine-tunes in VLM research.</p>
</div>
<div id="S8.p2" class="ltx_para">
<span id="S8.p2.1" class="ltx_ERROR undefined">\nobibliography</span>
<p id="S8.p2.2" class="ltx_p">*</p>
</div>
</section>
<section id="bib" class="ltx_bibliography" lang="en">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Acharya et al. [2019]</span>
<span class="ltx_bibblock">
M. Acharya, K. Kafle, and C. Kanan.

</span>
<span class="ltx_bibblock">Tallyqa: Answering complex counting questions.

</span>
<span class="ltx_bibblock">In <em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">AAAI</em>, 2019.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Aghajanyan et al. [2022]</span>
<span class="ltx_bibblock">
A. Aghajanyan, B. Huang, C. Ross, V. Karpukhin, H. Xu, N. Goyal, D. Okhonko,
M. Joshi, G. Ghosh, M. Lewis, and L. Zettlemoyer.

</span>
<span class="ltx_bibblock">CM3: A causal masked multimodal model of the internet.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2201.07520, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2201.07520" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2201.07520</a>.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Aghajanyan et al. [2023]</span>
<span class="ltx_bibblock">
A. Aghajanyan, L. Yu, A. Conneau, W. Hsu, K. Hambardzumyan, S. Zhang,
S. Roller, N. Goyal, O. Levy, and L. Zettlemoyer.

</span>
<span class="ltx_bibblock">Scaling laws for generative mixed-modal language models.

</span>
<span class="ltx_bibblock">In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and
J. Scarlett, editors, <em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning,
ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA</em>, volume 202 of
<em id="bib.bib3.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>, pages 265–279. PMLR,
2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.mlr.press/v202/aghajanyan23a.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.mlr.press/v202/aghajanyan23a.html</a>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Agrawal et al. [2019]</span>
<span class="ltx_bibblock">
H. Agrawal, K. Desai, Y. Wang, X. Chen, R. Jain, M. Johnson, D. Batra,
D. Parikh, S. Lee, and P. Anderson.

</span>
<span class="ltx_bibblock">nocaps: novel object captioning at scale.

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE International Conference on Computer
Vision</em>, pages 8948–8957, 2019.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Alabdulmohsin et al. [2023]</span>
<span class="ltx_bibblock">
I. Alabdulmohsin, X. Zhai, A. Kolesnikov, and L. Beyer.

</span>
<span class="ltx_bibblock">Getting vit in shape: Scaling laws for compute-optimal model design.

</span>
<span class="ltx_bibblock">In <em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>, 2023.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Alayrac et al. [2022]</span>
<span class="ltx_bibblock">
J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc,
A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han,
Z. Gong, S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock,
A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals,
A. Zisserman, and K. Simonyan.

</span>
<span class="ltx_bibblock">Flamingo: a visual language model for few-shot learning, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2204.14198" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2204.14198</a>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Anil et al. [2023]</span>
<span class="ltx_bibblock">
R. Anil, S. Borgeaud, Y. Wu, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M.
Dai, A. Hauth, K. Millican, D. Silver, S. Petrov, M. Johnson, I. Antonoglou,
J. Schrittwieser, A. Glaese, J. Chen, E. Pitler, T. P. Lillicrap,
A. Lazaridou, O. Firat, J. Molloy, M. Isard, P. R. Barham, T. Hennigan,
B. Lee, F. Viola, M. Reynolds, Y. Xu, R. Doherty, E. Collins, C. Meyer,
E. Rutherford, E. Moreira, K. Ayoub, M. Goel, G. Tucker, E. Piqueras,
M. Krikun, I. Barr, N. Savinov, I. Danihelka, B. Roelofs, A. White,
A. Andreassen, T. von Glehn, L. Yagati, M. Kazemi, L. Gonzalez, M. Khalman,
J. Sygnowski, and et al.

</span>
<span class="ltx_bibblock">Gemini: A family of highly capable multimodal models.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2312.11805, 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2312.11805" title="" class="ltx_ref">10.48550/ARXIV.2312.11805</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2312.11805" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2312.11805</a>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Aniruddha Kembhavi [2016]</span>
<span class="ltx_bibblock">
E. K. M. S. H. H. A. F. Aniruddha Kembhavi, Mike Salvato.

</span>
<span class="ltx_bibblock">A diagram is worth a dozen images.

</span>
<span class="ltx_bibblock">In <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">European Conference on Computer Vision (ECCV)</em>, 2016.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://api.semanticscholar.org/CorpusID:2682274" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://api.semanticscholar.org/CorpusID:2682274</a>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Baechler et al. [2024]</span>
<span class="ltx_bibblock">
G. Baechler, S. Sunkara, M. Wang, F. Zubach, H. Mansoor, V. Etter, V. Cărbune,
J. Lin, J. Chen, and A. Sharma.

</span>
<span class="ltx_bibblock">Screenai: A vision-language model for ui and infographics
understanding, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2402.04615" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2402.04615</a>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bai et al. [2023]</span>
<span class="ltx_bibblock">
J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and
J. Zhou.

</span>
<span class="ltx_bibblock">Qwen-vl: A versatile vision-language model for understanding,
localization, text reading, and beyond.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2308.12966, 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2308.12966" title="" class="ltx_ref">10.48550/ARXIV.2308.12966</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2308.12966" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2308.12966</a>.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bavishi et al. [2023]</span>
<span class="ltx_bibblock">
R. Bavishi, E. Elsen, C. Hawthorne, M. Nye, A. Odena, A. Somani, and
S. Taşırlar.

</span>
<span class="ltx_bibblock">Introducing our multimodal models, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://www.adept.ai/blog/fuyu-8b" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.adept.ai/blog/fuyu-8b</a>.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Beyer et al. [2022]</span>
<span class="ltx_bibblock">
L. Beyer, X. Zhai, and A. Kolesnikov.

</span>
<span class="ltx_bibblock">Big vision.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/google-research/big_vision" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/google-research/big_vision</a>, 2022.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Beyer et al. [2023a]</span>
<span class="ltx_bibblock">
L. Beyer, P. Izmailov, A. Kolesnikov, M. Caron, S. Kornblith, X. Zhai,
M. Minderer, M. Tschannen, I. Alabdulmohsin, and F. Pavetic.

</span>
<span class="ltx_bibblock">Flexivit: One model for all patch sizes.

</span>
<span class="ltx_bibblock">In <em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition</em>, pages 14496–14506, 2023a.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Beyer et al. [2023b]</span>
<span class="ltx_bibblock">
L. Beyer, B. Wan, G. Madan, F. Pavetic, A. Steiner, A. Kolesnikov, A. S. Pinto,
E. Bugliarello, X. Wang, Q. Yu, et al.

</span>
<span class="ltx_bibblock">A study of autoregressive decoders for multi-tasking in computer
vision.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2303.17376</em>, 2023b.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Biten et al. [2019]</span>
<span class="ltx_bibblock">
A. F. Biten, R. Tito, A. Mafla, L. Gomez, M. Rusinol, C. Jawahar, E. Valveny,
and D. Karatzas.

</span>
<span class="ltx_bibblock">Scene text visual question answering.

</span>
<span class="ltx_bibblock">In <em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">2019 IEEE/CVF International Conference on Computer Vision
(ICCV)</em>. IEEE, Oct. 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.1109/iccv.2019.00439" title="" class="ltx_ref">10.1109/iccv.2019.00439</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://dx.doi.org/10.1109/ICCV.2019.00439" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://dx.doi.org/10.1109/ICCV.2019.00439</a>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bradbury et al. [2018]</span>
<span class="ltx_bibblock">
J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin,
G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang.

</span>
<span class="ltx_bibblock">JAX: composable transformations of Python+NumPy programs,
2018.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://github.com/google/jax" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://github.com/google/jax</a>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cha et al. [2023]</span>
<span class="ltx_bibblock">
J. Cha, W. Kang, J. Mun, and B. Roh.

</span>
<span class="ltx_bibblock">Honeybee: Locality-enhanced projector for multimodal LLM.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2312.06742, 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2312.06742" title="" class="ltx_ref">10.48550/ARXIV.2312.06742</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2312.06742" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2312.06742</a>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Changpinyo et al. [2022]</span>
<span class="ltx_bibblock">
S. Changpinyo, D. Kukliansy, I. Szpektor, X. Chen, N. Ding, and R. Soricut.

</span>
<span class="ltx_bibblock">All you may need for VQA are image captions.

</span>
<span class="ltx_bibblock">In M. Carpuat, M.-C. de Marneffe, and I. V. Meza Ruiz, editors,
<em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2022 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies</em>,
pages 1947–1963, Seattle, United States, July 2022. Association for
Computational Linguistics.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.18653/v1/2022.naacl-main.142" title="" class="ltx_ref">10.18653/v1/2022.naacl-main.142</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2022.naacl-main.142" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2022.naacl-main.142</a>.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen and Dolan [2011]</span>
<span class="ltx_bibblock">
D. L. Chen and W. B. Dolan.

</span>
<span class="ltx_bibblock">Collecting highly parallel data for paraphrase evaluation.

</span>
<span class="ltx_bibblock">In D. Lin, Y. Matsumoto, and R. Mihalcea, editors, <em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">The 49th
Annual Meeting of the Association for Computational Linguistics: Human
Language Technologies, Proceedings of the Conference, 19-24 June, 2011,
Portland, Oregon, USA</em>, pages 190–200. The Association for Computer
Linguistics, 2011.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/P11-1020/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/P11-1020/</a>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. [2023a]</span>
<span class="ltx_bibblock">
L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin.

</span>
<span class="ltx_bibblock">Sharegpt4v: Improving large multi-modal models with better captions.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2311.12793</em>, 2023a.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. [2022a]</span>
<span class="ltx_bibblock">
T. Chen, S. Saxena, L. Li, D. J. Fleet, and G. E. Hinton.

</span>
<span class="ltx_bibblock">Pix2seq: A language modeling framework for object detection.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">ICLR</em>, 2022a.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. [2022b]</span>
<span class="ltx_bibblock">
X. Chen, X. Wang, S. Changpinyo, A. J. Piergiovanni, P. Padlewski, D. Salz,
S. Goodman, A. Grycner, B. Mustafa, L. Beyer, A. Kolesnikov, J. Puigcerver,
N. Ding, K. Rong, H. Akbari, G. Mishra, L. Xue, A. Thapliyal, J. Bradbury,
W. Kuo, M. Seyedhosseini, C. Jia, B. K. Ayan, C. Riquelme, A. Steiner,
A. Angelova, X. Zhai, N. Houlsby, and R. Soricut.

</span>
<span class="ltx_bibblock">PaLI: A jointly-scaled multilingual language-image model.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, arXiv:2209.06794, 2022b.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. [2023b]</span>
<span class="ltx_bibblock">
X. Chen, J. Djolonga, P. Padlewski, B. Mustafa, S. Changpinyo, J. Wu, C. R.
Ruiz, S. Goodman, X. Wang, Y. Tay, S. Shakeri, M. Dehghani, D. Salz,
M. Lucic, M. Tschannen, A. Nagrani, H. Hu, M. Joshi, B. Pang, C. Montgomery,
P. Pietrzyk, M. Ritter, A. J. Piergiovanni, M. Minderer, F. Pavetic,
A. Waters, G. Li, I. Alabdulmohsin, L. Beyer, J. Amelot, K. Lee, A. P.
Steiner, Y. Li, D. Keysers, A. Arnab, Y. Xu, K. Rong, A. Kolesnikov,
M. Seyedhosseini, A. Angelova, X. Zhai, N. Houlsby, and R. Soricut.

</span>
<span class="ltx_bibblock">PaLI-X: On scaling up a multilingual vision and language model.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, arXiv:2305.18565, 2023b.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. [2023c]</span>
<span class="ltx_bibblock">
X. Chen, X. Wang, L. Beyer, A. Kolesnikov, J. Wu, P. Voigtlaender, B. Mustafa,
S. Goodman, I. Alabdulmohsin, P. Padlewski, D. Salz, X. Xiong, D. Vlasic,
F. Pavetic, K. Rong, T. Yu, D. Keysers, X. Zhai, and R. Soricut.

</span>
<span class="ltx_bibblock">PaLI-3 vision language models: Smaller, faster, stronger.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, arXiv:2310.09199, 2023c.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. [2023d]</span>
<span class="ltx_bibblock">
Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu,
L. Lu, B. Li, P. Luo, T. Lu, Y. Qiao, and J. Dai.

</span>
<span class="ltx_bibblock">Internvl: Scaling up vision foundation models and aligning for
generic visual-linguistic tasks.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2312.14238</em>, 2023d.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cho et al. [2021]</span>
<span class="ltx_bibblock">
J. Cho, J. Lei, H. Tan, and M. Bansal.

</span>
<span class="ltx_bibblock">Unifying vision-and-language tasks via text generation.

</span>
<span class="ltx_bibblock">In M. Meila and T. Zhang, editors, <em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 38th
International Conference on Machine Learning</em>, volume 139 of
<em id="bib.bib26.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>, pages 1931–1942. PMLR,
18–24 Jul 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.mlr.press/v139/cho21a.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.mlr.press/v139/cho21a.html</a>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chowdhery et al. [2023]</span>
<span class="ltx_bibblock">
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham,
H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko,
J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif,
N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard,
G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev,
H. Michalewski, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou,
D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan,
S. Agrawal, M. Omernick, A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz,
E. Moreira, R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, B. Saeta,
M. Diaz, O. Firat, M. Catasta, J. Wei, K. Meier-Hellstern, D. Eck, J. Dean,
S. Petrov, and N. Fiedel.

</span>
<span class="ltx_bibblock">Palm: Scaling language modeling with pathways.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">J. Mach. Learn. Res.</em>, 24:240:1–240:113, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://jmlr.org/papers/v24/22-1144.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://jmlr.org/papers/v24/22-1144.html</a>.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dehghani et al. [2023]</span>
<span class="ltx_bibblock">
M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P.
Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, R. Jenatton, L. Beyer,
M. Tschannen, A. Arnab, X. Wang, C. R. Ruiz, M. Minderer, J. Puigcerver,
U. Evci, M. Kumar, S. van Steenkiste, G. F. Elsayed, A. Mahendran, F. Yu,
A. Oliver, F. Huot, J. Bastings, M. Collier, A. A. Gritsenko, V. Birodkar,
C. N. Vasconcelos, Y. Tay, T. Mensink, A. Kolesnikov, F. Pavetic, D. Tran,
T. Kipf, M. Lucic, X. Zhai, D. Keysers, J. J. Harmsen, and N. Houlsby.

</span>
<span class="ltx_bibblock">Scaling vision transformers to 22 billion parameters.

</span>
<span class="ltx_bibblock">In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and
J. Scarlett, editors, <em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning,
ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA</em>, volume 202 of
<em id="bib.bib28.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>, pages 7480–7512. PMLR,
2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.mlr.press/v202/dehghani23a.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.mlr.press/v202/dehghani23a.html</a>.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dehghani et al. [2024]</span>
<span class="ltx_bibblock">
M. Dehghani, B. Mustafa, J. Djolonga, J. Heek, M. Minderer, M. Caron,
A. Steiner, J. Puigcerver, R. Geirhos, I. M. Alabdulmohsin, et al.

</span>
<span class="ltx_bibblock">Patch n’pack: Navit, a vision transformer for any aspect ratio and
resolution.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Deitke et al. [2023]</span>
<span class="ltx_bibblock">
M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt,
L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi.

</span>
<span class="ltx_bibblock">Objaverse: A universe of annotated 3d objects.

</span>
<span class="ltx_bibblock">In <em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">IEEE/CVF Conference on Computer Vision and Pattern
Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023</em>, pages
13142–13153. IEEE, 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.1109/CVPR52729.2023.01263" title="" class="ltx_ref">10.1109/CVPR52729.2023.01263</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.1109/CVPR52729.2023.01263" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1109/CVPR52729.2023.01263</a>.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Desai and Johnson [2021]</span>
<span class="ltx_bibblock">
K. Desai and J. Johnson.

</span>
<span class="ltx_bibblock">Virtex: Learning visual representations from textual annotations.

</span>
<span class="ltx_bibblock">In <em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition</em>, pages 11162–11173, 2021.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Devlin et al. [2019]</span>
<span class="ltx_bibblock">
J. Devlin, M. Chang, K. Lee, and K. Toutanova.

</span>
<span class="ltx_bibblock">BERT: pre-training of deep bidirectional transformers for language
understanding.

</span>
<span class="ltx_bibblock">In J. Burstein, C. Doran, and T. Solorio, editors, <em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">Proceedings
of the 2019 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, NAACL-HLT 2019,
Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)</em>,
pages 4171–4186. Association for Computational Linguistics, 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.18653/V1/N19-1423" title="" class="ltx_ref">10.18653/V1/N19-1423</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.18653/v1/n19-1423" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.18653/v1/n19-1423</a>.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Diao et al. [2024]</span>
<span class="ltx_bibblock">
H. Diao, Y. Cui, X. Li, Y. Wang, H. Lu, and X. Wang.

</span>
<span class="ltx_bibblock">Unveiling encoder-free vision-language models, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2406.11832" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2406.11832</a>.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dong et al. [2024]</span>
<span class="ltx_bibblock">
X. Dong, P. Zhang, Y. Zang, Y. Cao, B. Wang, L. Ouyang, S. Zhang, H. Duan,
W. Zhang, Y. Li, H. Yan, Y. Gao, Z. Chen, X. Zhang, W. Li, J. Li, W. Wang,
K. Chen, C. He, X. Zhang, J. Dai, Y. Qiao, D. Lin, and J. Wang.

</span>
<span class="ltx_bibblock">Internlm-xcomposer2-4khd: A pioneering large vision-language model
handling resolutions from 336 pixels to 4k hd, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2404.06512" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2404.06512</a>.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Driess et al. [2023]</span>
<span class="ltx_bibblock">
D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter,
A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet,
D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff,
A. Zeng, I. Mordatch, and P. Florence.

</span>
<span class="ltx_bibblock">PaLM-E: An embodied multimodal language model.

</span>
<span class="ltx_bibblock">In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and
J. Scarlett, editors, <em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning,
ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA</em>, volume 202 of
<em id="bib.bib35.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>, pages 8469–8488. PMLR,
2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.mlr.press/v202/driess23a.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.mlr.press/v202/driess23a.html</a>.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Google Cloud [20xx]</span>
<span class="ltx_bibblock">
Google Cloud.

</span>
<span class="ltx_bibblock">Introduction to Cloud TPU.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://cloud.google.com/tpu/docs/intro-to-tpu" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://cloud.google.com/tpu/docs/intro-to-tpu</a>, 20xx.

</span>
<span class="ltx_bibblock">Accessed: 2024-07-04.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Goyal et al. [2017]</span>
<span class="ltx_bibblock">
Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh.

</span>
<span class="ltx_bibblock">Making the V in VQA matter: Elevating the role of image
understanding in Visual Question Answering.

</span>
<span class="ltx_bibblock">In <em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">Computer Vision and Pattern Recognition (CVPR)</em>, 2017.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gurari et al. [2018]</span>
<span class="ltx_bibblock">
D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P.
Bigham.

</span>
<span class="ltx_bibblock">Vizwiz grand challenge: Answering visual questions from blind people.

</span>
<span class="ltx_bibblock">In <em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE conference on computer vision and
pattern recognition</em>, pages 3608–3617, 2018.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">He et al. [2024]</span>
<span class="ltx_bibblock">
M. He, Y. Liu, B. Wu, J. Yuan, Y. Wang, T. Huang, and B. Zhao.

</span>
<span class="ltx_bibblock">Efficient multimodal learning from data-centric perspective.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2402.11530, 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2402.11530" title="" class="ltx_ref">10.48550/ARXIV.2402.11530</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2402.11530" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2402.11530</a>.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hewitt [2021]</span>
<span class="ltx_bibblock">
J. Hewitt.

</span>
<span class="ltx_bibblock">Initializing new word embeddings for pretrained language models.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/nlp.stanford.edu/~johnhew//vocab-expansion.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https:/nlp.stanford.edu/~johnhew//vocab-expansion.html</a>, 2021.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hsieh et al. [2024]</span>
<span class="ltx_bibblock">
C.-Y. Hsieh, J. Zhang, Z. Ma, A. Kembhavi, and R. Krishna.

</span>
<span class="ltx_bibblock">Sugarcrepe: fixing hackable benchmarks for vision-language
compositionality.

</span>
<span class="ltx_bibblock">In <em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 37th International Conference on Neural
Information Processing Systems</em>, NIPS ’23, Red Hook, NY, USA, 2024. Curran
Associates Inc.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hsu et al. [2021]</span>
<span class="ltx_bibblock">
T.-Y. Hsu, C. L. Giles, and T.-H. Huang.

</span>
<span class="ltx_bibblock">Scicap: Generating captions for scientific figures.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2110.11624</em>, 2021.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huang et al. [2023]</span>
<span class="ltx_bibblock">
S. Huang, L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K.
Mohammed, B. Patra, Q. Liu, K. Aggarwal, Z. Chi, N. J. B. Bjorck,
V. Chaudhary, S. Som, X. Song, and F. Wei.

</span>
<span class="ltx_bibblock">Language is not all you need: Aligning perception with language
models.

</span>
<span class="ltx_bibblock">In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and
S. Levine, editors, <em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems
36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS
2023, New Orleans, LA, USA, December 10 - 16, 2023</em>, 2023.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huang et al. [2024]</span>
<span class="ltx_bibblock">
Y. Huang, Z. Meng, F. Liu, Y. Su, N. Collier, and Y. Lu.

</span>
<span class="ltx_bibblock">Sparkles: Unlocking chats across multiple images for multimodal
instruction-following models, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=oq5EF8parZ" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=oq5EF8parZ</a>.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hudson and Manning [2019]</span>
<span class="ltx_bibblock">
D. Hudson and C. Manning.

</span>
<span class="ltx_bibblock">Gqa: A new dataset for real-world visual reasoning and compositional
question answering.

</span>
<span class="ltx_bibblock"><em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">Computer Vision and Pattern Recognition (CVPR)</em>,
abs/1902.09506, 2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/arXiv.1902.09506" title="" class="ltx_ref">10.48550/arXiv.1902.09506</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.1902.09506" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.1902.09506</a>.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jaegle et al. [2021]</span>
<span class="ltx_bibblock">
A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira.

</span>
<span class="ltx_bibblock">Perceiver: General perception with iterative attention.

</span>
<span class="ltx_bibblock">In M. Meila and T. Zhang, editors, <em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 38th
International Conference on Machine Learning, ICML 2021, 18-24 July 2021,
Virtual Event</em>, volume 139 of <em id="bib.bib46.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning
Research</em>, pages 4651–4664. PMLR, 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://proceedings.mlr.press/v139/jaegle21a.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://proceedings.mlr.press/v139/jaegle21a.html</a>.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jia et al. [2021]</span>
<span class="ltx_bibblock">
C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li,
and T. Duerig.

</span>
<span class="ltx_bibblock">Scaling up visual and vision-language representation learning with
noisy text supervision.

</span>
<span class="ltx_bibblock">In M. Meila and T. Zhang, editors, <em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 38th
International Conference on Machine Learning, ICML 2021, 18-24 July 2021,
Virtual Event</em>, volume 139 of <em id="bib.bib47.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning
Research</em>, pages 4904–4916. PMLR, 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://proceedings.mlr.press/v139/jia21b.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://proceedings.mlr.press/v139/jia21b.html</a>.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kabra et al. [2024]</span>
<span class="ltx_bibblock">
R. Kabra, L. Matthey, A. Lerchner, and N. J. Mitra.

</span>
<span class="ltx_bibblock">Leveraging vlm-based pipelines to annotate 3d objects.

</span>
<span class="ltx_bibblock">In <em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 41st International Conference on Machine
Learning</em>, volume 235 of <em id="bib.bib48.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>.
PMLR, 2024.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kajić et al. [2024]</span>
<span class="ltx_bibblock">
I. Kajić, O. Wiles, I. Albuquerque, M. Bauer, S. Wang, J. Pont-Tuset, and
A. Nematzadeh.

</span>
<span class="ltx_bibblock">Evaluating numerical reasoning in text-to-image models, 2024.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Karamcheti et al. [2024]</span>
<span class="ltx_bibblock">
S. Karamcheti, S. Nair, A. Balakrishna, P. Liang, T. Kollar, and D. Sadigh.

</span>
<span class="ltx_bibblock">Prismatic vlms: Investigating the design space of
visually-conditioned language models, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2402.07865" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2402.07865</a>.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kazemzadeh et al. [2014]</span>
<span class="ltx_bibblock">
S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg.

</span>
<span class="ltx_bibblock">ReferItGame: Referring to objects in photographs of natural
scenes.

</span>
<span class="ltx_bibblock">In A. Moschitti, B. Pang, and W. Daelemans, editors,
<em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2014 Conference on Empirical Methods in Natural
Language Processing (EMNLP)</em>, pages 787–798, Doha, Qatar, Oct. 2014.
Association for Computational Linguistics.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.3115/v1/D14-1086" title="" class="ltx_ref">10.3115/v1/D14-1086</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/D14-1086" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/D14-1086</a>.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kirillov et al. [2023]</span>
<span class="ltx_bibblock">
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao,
S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollar, and R. Girshick.

</span>
<span class="ltx_bibblock">Segment anything.

</span>
<span class="ltx_bibblock">In <em id="bib.bib52.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF International Conference on
Computer Vision (ICCV)</em>, pages 4015–4026, October 2023.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kolesnikov et al. [2020]</span>
<span class="ltx_bibblock">
A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, and
N. Houlsby.

</span>
<span class="ltx_bibblock">Big transfer (BiT): General visual representation learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib53.1.1" class="ltx_emph ltx_font_italic">European Conference on Computer Vision (ECCV)</em>, 2020.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kolesnikov et al. [2022]</span>
<span class="ltx_bibblock">
A. Kolesnikov, A. S. Pinto, L. Beyer, X. Zhai, J. Harmsen, and N. Houlsby.

</span>
<span class="ltx_bibblock">Uvim: A unified modeling approach for vision with learned guiding
codes.

</span>
<span class="ltx_bibblock">In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh,
editors, <em id="bib.bib54.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems 35: Annual
Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New
Orleans, LA, USA, November 28 - December 9, 2022</em>, 2022.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Krishna et al. [2017]</span>
<span class="ltx_bibblock">
R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. Carlos Niebles.

</span>
<span class="ltx_bibblock">Dense-captioning events in videos.

</span>
<span class="ltx_bibblock">In <em id="bib.bib55.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE international conference on computer
vision</em>, pages 706–715, 2017.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kudo and Richardson [2018]</span>
<span class="ltx_bibblock">
T. Kudo and J. Richardson.

</span>
<span class="ltx_bibblock">SentencePiece: A simple and language independent subword
tokenizer and detokenizer for neural text processing.

</span>
<span class="ltx_bibblock">In E. Blanco and W. Lu, editors, <em id="bib.bib56.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018
Conference on Empirical Methods in Natural Language Processing: System
Demonstrations</em>, pages 66–71, Brussels, Belgium, Nov. 2018. Association for
Computational Linguistics.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.18653/v1/D18-2012" title="" class="ltx_ref">10.18653/v1/D18-2012</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/D18-2012" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/D18-2012</a>.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Laurençon et al. [2024]</span>
<span class="ltx_bibblock">
H. Laurençon, L. Tronchon, M. Cord, and V. Sanh.

</span>
<span class="ltx_bibblock">What matters when building vision-language models?

</span>
<span class="ltx_bibblock"><em id="bib.bib57.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2405.02246, 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2405.02246" title="" class="ltx_ref">10.48550/ARXIV.2405.02246</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2405.02246" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2405.02246</a>.

</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Laurençon et al. [2023]</span>
<span class="ltx_bibblock">
H. Laurençon, L. Saulnier, L. Tronchon, S. Bekman, A. Singh, A. Lozhkov,
T. Wang, S. Karamcheti, A. M. Rush, D. Kiela, M. Cord, and V. Sanh.

</span>
<span class="ltx_bibblock">Obelics: An open web-scale filtered dataset of interleaved image-text
documents, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2306.16527" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2306.16527</a>.

</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. [2022a]</span>
<span class="ltx_bibblock">
J. Li, D. Li, C. Xiong, and S. C. H. Hoi.

</span>
<span class="ltx_bibblock">BLIP: bootstrapping language-image pre-training for unified
vision-language understanding and generation.

</span>
<span class="ltx_bibblock">In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu,
and S. Sabato, editors, <em id="bib.bib59.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning,
ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA</em>, volume 162 of
<em id="bib.bib59.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>, pages 12888–12900. PMLR,
2022a.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.mlr.press/v162/li22n.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.mlr.press/v162/li22n.html</a>.

</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. [2023]</span>
<span class="ltx_bibblock">
J. Li, D. Li, S. Savarese, and S. C. H. Hoi.

</span>
<span class="ltx_bibblock">BLIP-2: bootstrapping language-image pre-training with frozen image
encoders and large language models.

</span>
<span class="ltx_bibblock">In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and
J. Scarlett, editors, <em id="bib.bib60.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning,
ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA</em>, volume 202 of
<em id="bib.bib60.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>, pages 19730–19742. PMLR,
2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://proceedings.mlr.press/v202/li23q.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://proceedings.mlr.press/v202/li23q.html</a>.

</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. [2020]</span>
<span class="ltx_bibblock">
Y. Li, G. Li, L. He, J. Zheng, H. Li, and Z. Guan.

</span>
<span class="ltx_bibblock">Widget captioning: Generating natural language description for
mobileuser interface elements.

</span>
<span class="ltx_bibblock">In <em id="bib.bib61.1.1" class="ltx_emph ltx_font_italic">Conference on Empirical Methods in Natural Language
Processing</em>, 2020.

</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. [2022b]</span>
<span class="ltx_bibblock">
Y. Li, H. Mao, R. Girshick, and K. He.

</span>
<span class="ltx_bibblock">Exploring plain vision transformer backbones for object detection,
2022b.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2203.16527" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2203.16527</a>.

</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. [2024]</span>
<span class="ltx_bibblock">
Y. Li, Y. Zhang, C. Wang, Z. Zhong, Y. Chen, R. Chu, S. Liu, and J. Jia.

</span>
<span class="ltx_bibblock">Mini-gemini: Mining the potential of multi-modality vision language
models.

</span>
<span class="ltx_bibblock"><em id="bib.bib63.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2403.18814</em>, 2024.

</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin et al. [2024]</span>
<span class="ltx_bibblock">
B. Lin, Z. Tang, Y. Ye, J. Cui, B. Zhu, P. Jin, J. Huang, J. Zhang, Y. Pang,
M. Ning, and L. Yuan.

</span>
<span class="ltx_bibblock">Moe-llava: Mixture of experts for large vision-language models, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2401.15947" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2401.15947</a>.

</span>
</li>
<li id="bib.bib65" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin et al. [2014]</span>
<span class="ltx_bibblock">
T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays,
P. Perona, D. Ramanan, P. Doll’a r, and C. L. Zitnick.

</span>
<span class="ltx_bibblock">Microsoft COCO: common objects in context.

</span>
<span class="ltx_bibblock"><em id="bib.bib65.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1405.0312, 2014.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://arxiv.org/abs/1405.0312" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://arxiv.org/abs/1405.0312</a>.

</span>
</li>
<li id="bib.bib66" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. [2023a]</span>
<span class="ltx_bibblock">
H. Liu, C. Li, Y. Li, and Y. J. Lee.

</span>
<span class="ltx_bibblock">Improved baselines with visual instruction tuning,
2023a.

</span>
</li>
<li id="bib.bib67" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. [2023b]</span>
<span class="ltx_bibblock">
H. Liu, C. Li, Q. Wu, and Y. J. Lee.

</span>
<span class="ltx_bibblock">Visual instruction tuning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib67.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>, 2023b.

</span>
</li>
<li id="bib.bib68" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lobry et al. [2020]</span>
<span class="ltx_bibblock">
S. Lobry, D. Marcos, J. Murray, and D. Tuia.

</span>
<span class="ltx_bibblock">Rsvqa: Visual question answering for remote sensing data.

</span>
<span class="ltx_bibblock"><em id="bib.bib68.1.1" class="ltx_emph ltx_font_italic">IEEE Transactions on Geoscience and Remote Sensing</em>,
58(12):8555–8566, Dec. 2020.

</span>
<span class="ltx_bibblock">ISSN 1558-0644.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.1109/tgrs.2020.2988782" title="" class="ltx_ref">10.1109/tgrs.2020.2988782</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://dx.doi.org/10.1109/TGRS.2020.2988782" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://dx.doi.org/10.1109/TGRS.2020.2988782</a>.

</span>
</li>
<li id="bib.bib69" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lu et al. [2023a]</span>
<span class="ltx_bibblock">
J. Lu, C. Clark, S. Lee, Z. Zhang, S. Khosla, R. Marten, D. Hoiem, and
A. Kembhavi.

</span>
<span class="ltx_bibblock">Unified-io 2: Scaling autoregressive multimodal models with vision,
language, audio, and action.

</span>
<span class="ltx_bibblock"><em id="bib.bib69.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2312.17172, 2023a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2312.17172" title="" class="ltx_ref">10.48550/ARXIV.2312.17172</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2312.17172" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2312.17172</a>.

</span>
</li>
<li id="bib.bib70" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lu et al. [2023b]</span>
<span class="ltx_bibblock">
J. Lu, C. Clark, R. Zellers, R. Mottaghi, and A. Kembhavi.

</span>
<span class="ltx_bibblock">UNIFIED-IO: A unified model for vision, language, and multi-modal
tasks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib70.1.1" class="ltx_emph ltx_font_italic">ICLR</em>, 2023b.

</span>
</li>
<li id="bib.bib71" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lu et al. [2022]</span>
<span class="ltx_bibblock">
P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark,
and A. Kalyan.

</span>
<span class="ltx_bibblock">Learn to explain: Multimodal reasoning via thought chains for science
question answering.

</span>
<span class="ltx_bibblock">In <em id="bib.bib71.1.1" class="ltx_emph ltx_font_italic">The 36th Conference on Neural Information Processing Systems
(NeurIPS)</em>, 2022.

</span>
</li>
<li id="bib.bib72" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Luo et al. [2023]</span>
<span class="ltx_bibblock">
T. Luo, C. Rockwell, H. Lee, and J. Johnson.

</span>
<span class="ltx_bibblock">Scalable 3d captioning with pretrained models, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2306.07279" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2306.07279</a>.

</span>
</li>
<li id="bib.bib73" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mao et al. [2016]</span>
<span class="ltx_bibblock">
J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy.

</span>
<span class="ltx_bibblock">Generation and comprehension of unambiguous object descriptions.

</span>
<span class="ltx_bibblock">In <em id="bib.bib73.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR)</em>, June 2016.

</span>
</li>
<li id="bib.bib74" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Marino et al. [2019]</span>
<span class="ltx_bibblock">
K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi.

</span>
<span class="ltx_bibblock">Ok-vqa: A visual question answering benchmark requiring external
knowledge.

</span>
<span class="ltx_bibblock">In <em id="bib.bib74.1.1" class="ltx_emph ltx_font_italic">Conference on Computer Vision and Pattern Recognition
(CVPR)</em>, 2019.

</span>
</li>
<li id="bib.bib75" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Masry et al. [2022]</span>
<span class="ltx_bibblock">
A. Masry, X. L. Do, J. Q. Tan, S. Joty, and E. Hoque.

</span>
<span class="ltx_bibblock">ChartQA: A benchmark for question answering about charts with
visual and logical reasoning.

</span>
<span class="ltx_bibblock">In S. Muresan, P. Nakov, and A. Villavicencio, editors,
<em id="bib.bib75.1.1" class="ltx_emph ltx_font_italic">Findings of the Association for Computational Linguistics: ACL 2022</em>,
pages 2263–2279, Dublin, Ireland, May 2022. Association for Computational
Linguistics.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.18653/v1/2022.findings-acl.177" title="" class="ltx_ref">10.18653/v1/2022.findings-acl.177</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2022.findings-acl.177" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2022.findings-acl.177</a>.

</span>
</li>
<li id="bib.bib76" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mathew et al. [2020]</span>
<span class="ltx_bibblock">
M. Mathew, D. Karatzas, R. Manmatha, and C. V. Jawahar.

</span>
<span class="ltx_bibblock">Docvqa: A dataset for VQA on document images.

</span>
<span class="ltx_bibblock"><em id="bib.bib76.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2007.00398, 2020.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2007.00398" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2007.00398</a>.

</span>
</li>
<li id="bib.bib77" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mathew et al. [2022]</span>
<span class="ltx_bibblock">
M. Mathew, V. Bagal, R. Tito, D. Karatzas, E. Valveny, and C. V. Jawahar.

</span>
<span class="ltx_bibblock">Infographicvqa.

</span>
<span class="ltx_bibblock">In <em id="bib.bib77.1.1" class="ltx_emph ltx_font_italic">2022 IEEE/CVF Winter Conference on Applications of Computer
Vision (WACV)</em>. IEEE, Jan. 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.1109/wacv51458.2022.00264" title="" class="ltx_ref">10.1109/wacv51458.2022.00264</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://dx.doi.org/10.1109/WACV51458.2022.00264" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://dx.doi.org/10.1109/WACV51458.2022.00264</a>.

</span>
</li>
<li id="bib.bib78" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">McKinzie et al. [2024]</span>
<span class="ltx_bibblock">
B. McKinzie, Z. Gan, J. Fauconnier, S. Dodge, B. Zhang, P. Dufter, D. Shah,
X. Du, F. Peng, F. Weers, A. Belyi, H. Zhang, K. Singh, D. Kang, A. Jain,
H. Hè, M. Schwarzer, T. Gunter, X. Kong, A. Zhang, J. Wang, C. Wang,
N. Du, T. Lei, S. Wiseman, G. Yin, M. Lee, Z. Wang, R. Pang, P. Grasch,
A. Toshev, and Y. Yang.

</span>
<span class="ltx_bibblock">MM1: methods, analysis &amp; insights from multimodal LLM
pre-training.

</span>
<span class="ltx_bibblock"><em id="bib.bib78.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2403.09611, 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2403.09611" title="" class="ltx_ref">10.48550/ARXIV.2403.09611</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2403.09611" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2403.09611</a>.

</span>
</li>
<li id="bib.bib79" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mesnard et al. [2024]</span>
<span class="ltx_bibblock">
T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre,
M. Rivière, M. S. Kale, J. Love, P. Tafti, L. Hussenot, A. Chowdhery,
A. Roberts, A. Barua, A. Botev, A. Castro-Ros, A. Slone, A. Héliou,
A. Tacchetti, A. Bulanova, A. Paterson, B. Tsai, B. Shahriari, C. L. Lan,
C. A. Choquette-Choo, C. Crepy, D. Cer, D. Ippolito, D. Reid,
E. Buchatskaya, E. Ni, E. Noland, G. Yan, G. Tucker, G. Muraru,
G. Rozhdestvenskiy, H. Michalewski, I. Tenney, I. Grishchenko, J. Austin,
J. Keeling, J. Labanowski, J. Lespiau, J. Stanway, J. Brennan, J. Chen,
J. Ferret, J. Chiu, and et al.

</span>
<span class="ltx_bibblock">Gemma: Open models based on gemini research and technology.

</span>
<span class="ltx_bibblock"><em id="bib.bib79.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2403.08295, 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2403.08295" title="" class="ltx_ref">10.48550/ARXIV.2403.08295</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2403.08295" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2403.08295</a>.

</span>
</li>
<li id="bib.bib80" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Minderer et al. [2023]</span>
<span class="ltx_bibblock">
M. Minderer, A. A. Gritsenko, and N. Houlsby.

</span>
<span class="ltx_bibblock">Scaling open-vocabulary object detection.

</span>
<span class="ltx_bibblock">In <em id="bib.bib80.1.1" class="ltx_emph ltx_font_italic">Thirty-seventh Conference on Neural Information Processing
Systems</em>, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=mQPNcBWjGc" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=mQPNcBWjGc</a>.

</span>
</li>
<li id="bib.bib81" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mishra et al. [2019]</span>
<span class="ltx_bibblock">
A. Mishra, S. Shekhar, A. K. Singh, and A. Chakraborty.

</span>
<span class="ltx_bibblock">Ocr-vqa: Visual question answering by reading text in images.

</span>
<span class="ltx_bibblock">In <em id="bib.bib81.1.1" class="ltx_emph ltx_font_italic">ICDAR</em>, 2019.

</span>
</li>
<li id="bib.bib82" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nguyen et al. [2024]</span>
<span class="ltx_bibblock">
T. Nguyen, M. Wallingford, S. Santy, W. Ma, S. Oh, L. Schmidt, P. W. Koh, and
R. Krishna.

</span>
<span class="ltx_bibblock">Multilingual diversity improves vision-language representations.

</span>
<span class="ltx_bibblock"><em id="bib.bib82.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2405.16915, 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2405.16915" title="" class="ltx_ref">10.48550/ARXIV.2405.16915</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2405.16915" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2405.16915</a>.

</span>
</li>
<li id="bib.bib83" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ning et al. [2023]</span>
<span class="ltx_bibblock">
J. Ning, C. Li, Z. Zhang, Z. Geng, Q. Dai, K. He, and H. Hu.

</span>
<span class="ltx_bibblock">All in tokens: Unifying output space of visual tasks via soft token.

</span>
<span class="ltx_bibblock"><em id="bib.bib83.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2301.02229</em>, 2023.

</span>
</li>
<li id="bib.bib84" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenAI [2023]</span>
<span class="ltx_bibblock">
OpenAI.

</span>
<span class="ltx_bibblock">Gpt-4 technical report, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2303.08774" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2303.08774</a>.

</span>
</li>
<li id="bib.bib85" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Paiss et al. [2023]</span>
<span class="ltx_bibblock">
R. Paiss, A. Ephrat, O. Tov, S. Zada, I. Mosseri, M. Irani, and T. Dekel.

</span>
<span class="ltx_bibblock">Teaching clip to count to ten.

</span>
<span class="ltx_bibblock"><em id="bib.bib85.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2302.12066</em>, 2023.

</span>
</li>
<li id="bib.bib86" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Peng et al. [2022]</span>
<span class="ltx_bibblock">
Z. Peng, L. Dong, H. Bao, Q. Ye, and F. Wei.

</span>
<span class="ltx_bibblock">Beit v2: Masked image modeling with vector-quantized visual
tokenizers.

</span>
<span class="ltx_bibblock"><em id="bib.bib86.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2208.06366, 2022.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2208.06366" title="" class="ltx_ref">10.48550/ARXIV.2208.06366</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2208.06366" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2208.06366</a>.

</span>
</li>
<li id="bib.bib87" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pfeiffer et al. [2022]</span>
<span class="ltx_bibblock">
J. Pfeiffer, G. Geigle, A. Kamath, J.-M. Steitz, S. Roth, I. Vulić, and
I. Gurevych.

</span>
<span class="ltx_bibblock">xGQA: Cross-lingual visual question answering.

</span>
<span class="ltx_bibblock">In <em id="bib.bib87.1.1" class="ltx_emph ltx_font_italic">Findings of the Association for Computational Linguistics:
ACL 2022</em>, pages 2497–2511, Dublin, Ireland, May 2022. Association for
Computational Linguistics.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.18653/v1/2022.findings-acl.196" title="" class="ltx_ref">10.18653/v1/2022.findings-acl.196</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2022.findings-acl.196" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2022.findings-acl.196</a>.

</span>
</li>
<li id="bib.bib88" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Piergiovanni et al. [2022]</span>
<span class="ltx_bibblock">
A. Piergiovanni, W. Kuo, and A. Angelova.

</span>
<span class="ltx_bibblock">Pre-training image-language transformers for open-vocabulary tasks,
2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2209.04372" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2209.04372</a>.

</span>
</li>
<li id="bib.bib89" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pouget et al. [2024]</span>
<span class="ltx_bibblock">
A. Pouget, L. Beyer, E. Bugliarello, X. Wang, A. P. Steiner, X. Zhai, and
I. Alabdulmohsin.

</span>
<span class="ltx_bibblock">No filter: Cultural and socioeconomic diversity in contrastive
vision-language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib89.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2405.13777, 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2405.13777" title="" class="ltx_ref">10.48550/ARXIV.2405.13777</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2405.13777" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2405.13777</a>.

</span>
</li>
<li id="bib.bib90" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. [2021]</span>
<span class="ltx_bibblock">
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry,
A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever.

</span>
<span class="ltx_bibblock">Learning transferable visual models from natural language
supervision.

</span>
<span class="ltx_bibblock">In <em id="bib.bib90.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning, ICML</em>, 2021.

</span>
</li>
<li id="bib.bib91" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raffel et al. [2020]</span>
<span class="ltx_bibblock">
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou,
W. Li, and P. J. Liu.

</span>
<span class="ltx_bibblock">Exploring the limits of transfer learning with a unified text-to-text
transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib91.1.1" class="ltx_emph ltx_font_italic">J. Mach. Learn. Res.</em>, 21:140:1–140:67, 2020.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://jmlr.org/papers/v21/20-074.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://jmlr.org/papers/v21/20-074.html</a>.

</span>
</li>
<li id="bib.bib92" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rajbhandari et al. [2020]</span>
<span class="ltx_bibblock">
S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He.

</span>
<span class="ltx_bibblock">ZeRO: memory optimizations toward training trillion parameter
models.

</span>
<span class="ltx_bibblock">In C. Cuicchi, I. Qualters, and W. T. Kramer, editors,
<em id="bib.bib92.1.1" class="ltx_emph ltx_font_italic">Proceedings of the International Conference for High Performance
Computing, Networking, Storage and Analysis, SC 2020, Virtual Event /
Atlanta, Georgia, USA, November 9-19, 2020</em>, page 20. IEEE/ACM, 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.1109/SC41405.2020.00024" title="" class="ltx_ref">10.1109/SC41405.2020.00024</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.1109/SC41405.2020.00024" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1109/SC41405.2020.00024</a>.

</span>
</li>
<li id="bib.bib93" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sabne [2020]</span>
<span class="ltx_bibblock">
A. Sabne.

</span>
<span class="ltx_bibblock">XLA : Compiling machine learning for peak performance, 2020.

</span>
</li>
<li id="bib.bib94" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schwenk et al. [2022]</span>
<span class="ltx_bibblock">
D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi.

</span>
<span class="ltx_bibblock">A-okvqa: A benchmark for visual question answering using world
knowledge.

</span>
<span class="ltx_bibblock"><em id="bib.bib94.1.1" class="ltx_emph ltx_font_italic">arXiv</em>, 2022.

</span>
</li>
<li id="bib.bib95" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shoeybi et al. [2019]</span>
<span class="ltx_bibblock">
M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro.

</span>
<span class="ltx_bibblock">Megatron-LM: Training multi-billion parameter language models using
model parallelism.

</span>
<span class="ltx_bibblock"><em id="bib.bib95.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/1909.08053, 2019.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="http://arxiv.org/abs/1909.08053" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://arxiv.org/abs/1909.08053</a>.

</span>
</li>
<li id="bib.bib96" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sidorov et al. [2020]</span>
<span class="ltx_bibblock">
O. Sidorov, R. Hu, M. Rohrbach, and A. Singh.

</span>
<span class="ltx_bibblock">Textcaps: A dataset for image captioning with reading
comprehension.

</span>
<span class="ltx_bibblock">In A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, editors,
<em id="bib.bib96.1.1" class="ltx_emph ltx_font_italic">Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK,
August 23-28, 2020, Proceedings, Part II</em>, volume 12347 of <em id="bib.bib96.2.2" class="ltx_emph ltx_font_italic">Lecture
Notes in Computer Science</em>, pages 742–758. Springer, 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.1007/978-3-030-58536-5_44" title="" class="ltx_ref">10.1007/978-3-030-58536-5_44</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.1007/978-3-030-58536-5_44" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1007/978-3-030-58536-5_44</a>.

</span>
</li>
<li id="bib.bib97" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Singh et al. [2019]</span>
<span class="ltx_bibblock">
A. Singh, V. Natarjan, M. Shah, Y. Jiang, X. Chen, D. Parikh, and M. Rohrbach.

</span>
<span class="ltx_bibblock">Towards vqa models that can read.

</span>
<span class="ltx_bibblock">In <em id="bib.bib97.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition</em>, pages 8317–8326, 2019.

</span>
</li>
<li id="bib.bib98" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Suhr et al. [2019]</span>
<span class="ltx_bibblock">
A. Suhr, S. Zhou, A. Zhang, I. Zhang, H. Bai, and Y. Artzi.

</span>
<span class="ltx_bibblock">A corpus for reasoning about natural language grounded in
photographs.

</span>
<span class="ltx_bibblock">In A. Korhonen, D. Traum, and L. Marquez, editors, <em id="bib.bib98.1.1" class="ltx_emph ltx_font_italic">Proceedings
of the 57th Annual Meeting of the Association for Computational Linguistics</em>,
pages 6418–6428, Florence, Italy, July 2019. Association for Computational
Linguistics.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.18653/v1/P19-1644" title="" class="ltx_ref">10.18653/v1/P19-1644</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/P19-1644" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/P19-1644</a>.

</span>
</li>
<li id="bib.bib99" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sun et al. [2023]</span>
<span class="ltx_bibblock">
Q. Sun, Y. Cui, X. Zhang, F. Zhang, Q. Yu, Z. Luo, Y. Wang, Y. Rao, J. Liu,
T. Huang, and X. Wang.

</span>
<span class="ltx_bibblock">Generative multimodal models are in-context learners.

</span>
<span class="ltx_bibblock"><em id="bib.bib99.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2312.13286, 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2312.13286" title="" class="ltx_ref">10.48550/ARXIV.2312.13286</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2312.13286" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2312.13286</a>.

</span>
</li>
<li id="bib.bib100" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tay et al. [2023]</span>
<span class="ltx_bibblock">
Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, J. Wei, X. Wang, H. W. Chung,
D. Bahri, T. Schuster, H. S. Zheng, D. Zhou, N. Houlsby, and D. Metzler.

</span>
<span class="ltx_bibblock">UL2: unifying language learning paradigms.

</span>
<span class="ltx_bibblock">In <em id="bib.bib100.1.1" class="ltx_emph ltx_font_italic">The Eleventh International Conference on Learning
Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023</em>. OpenReview.net,
2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/pdf?id=6ruVLB727MC" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/pdf?id=6ruVLB727MC</a>.

</span>
</li>
<li id="bib.bib101" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Team [2024]</span>
<span class="ltx_bibblock">
C. Team.

</span>
<span class="ltx_bibblock">Chameleon: Mixed-modal early-fusion foundation models.

</span>
<span class="ltx_bibblock"><em id="bib.bib101.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2405.09818, 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2405.09818" title="" class="ltx_ref">10.48550/ARXIV.2405.09818</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2405.09818" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2405.09818</a>.

</span>
</li>
<li id="bib.bib102" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Thapliyal et al. [2022]</span>
<span class="ltx_bibblock">
A. V. Thapliyal, J. Pont Tuset, X. Chen, and R. Soricut.

</span>
<span class="ltx_bibblock">Crossmodal-3600: A massively multilingual multimodal evaluation
dataset.

</span>
<span class="ltx_bibblock">In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, <em id="bib.bib102.1.1" class="ltx_emph ltx_font_italic">Proceedings
of the 2022 Conference on Empirical Methods in Natural Language Processing</em>,
pages 715–729, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for
Computational Linguistics.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.18653/v1/2022.emnlp-main.45" title="" class="ltx_ref">10.18653/v1/2022.emnlp-main.45</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://aclanthology.org/2022.emnlp-main.45" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://aclanthology.org/2022.emnlp-main.45</a>.

</span>
</li>
<li id="bib.bib103" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tong et al. [2024a]</span>
<span class="ltx_bibblock">
S. Tong, E. Brown, P. Wu, S. Woo, M. Middepogu, S. C. Akula, J. Yang, S. Yang,
A. Iyer, X. Pan, A. Wang, R. Fergus, Y. LeCun, and S. Xie.

</span>
<span class="ltx_bibblock">Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal
LLMs.

</span>
<span class="ltx_bibblock"><em id="bib.bib103.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2406.16860</em>, 2024a.

</span>
</li>
<li id="bib.bib104" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tong et al. [2024b]</span>
<span class="ltx_bibblock">
S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie.

</span>
<span class="ltx_bibblock">Eyes wide shut? exploring the visual shortcomings of multimodal llms,
2024b.

</span>
</li>
<li id="bib.bib105" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. [2022]</span>
<span class="ltx_bibblock">
H. Touvron, A. Vedaldi, M. Douze, and H. Jégou.

</span>
<span class="ltx_bibblock">Fixing the train-test resolution discrepancy, 2022.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/1906.06423" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/1906.06423</a>.

</span>
</li>
<li id="bib.bib106" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tschannen et al. [2023]</span>
<span class="ltx_bibblock">
M. Tschannen, M. Kumar, A. Steiner, X. Zhai, N. Houlsby, and L. Beyer.

</span>
<span class="ltx_bibblock">Image captioners are scalable vision learners too.

</span>
<span class="ltx_bibblock"><em id="bib.bib106.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>, 2023.

</span>
</li>
<li id="bib.bib107" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tsimpoukelli et al. [2021]</span>
<span class="ltx_bibblock">
M. Tsimpoukelli, J. Menick, S. Cabi, S. M. A. Eslami, O. Vinyals, and F. Hill.

</span>
<span class="ltx_bibblock">Multimodal few-shot learning with frozen language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib107.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2106.13884, 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2106.13884" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2106.13884</a>.

</span>
</li>
<li id="bib.bib108" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">van den Oord et al. [2017]</span>
<span class="ltx_bibblock">
A. van den Oord, O. Vinyals, and K. Kavukcuoglu.

</span>
<span class="ltx_bibblock">Neural discrete representation learning.

</span>
<span class="ltx_bibblock">In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus,
S. V. N. Vishwanathan, and R. Garnett, editors, <em id="bib.bib108.1.1" class="ltx_emph ltx_font_italic">Advances in Neural
Information Processing Systems 30: Annual Conference on Neural Information
Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA</em>, pages
6306–6315, 2017.

</span>
</li>
<li id="bib.bib109" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vikhyat [2024]</span>
<span class="ltx_bibblock">
Vikhyat.

</span>
<span class="ltx_bibblock">Moondream.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/vikhyat/moondream" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/vikhyat/moondream</a>, 2024.

</span>
<span class="ltx_bibblock">Accessed: 2024-07-04.

</span>
</li>
<li id="bib.bib110" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Visheratin [2024]</span>
<span class="ltx_bibblock">
A. Visheratin.

</span>
<span class="ltx_bibblock">Breaking resolution curse of vision-language models.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://huggingface.co/blog/visheratin/vlm-resolution-curse" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/blog/visheratin/vlm-resolution-curse</a>,
2024.

</span>
</li>
<li id="bib.bib111" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wan et al. [2024]</span>
<span class="ltx_bibblock">
B. Wan, M. Tschannen, Y. Xian, F. Pavetic, I. Alabdulmohsin, X. Wang, A. S.
Pinto, A. Steiner, L. Beyer, and X. Zhai.

</span>
<span class="ltx_bibblock">Locca: Visual pretraining with location-aware captioners.

</span>
<span class="ltx_bibblock"><em id="bib.bib111.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2403.19596, 2024.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2403.19596" title="" class="ltx_ref">10.48550/ARXIV.2403.19596</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2403.19596" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2403.19596</a>.

</span>
</li>
<li id="bib.bib112" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. [2021]</span>
<span class="ltx_bibblock">
B. Wang, G. Li, X. Zhou, Z. Chen, T. Grossman, and Y. Li.

</span>
<span class="ltx_bibblock">Screen2words: Automatic mobile ui summarization with multimodal
learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib112.1.1" class="ltx_emph ltx_font_italic">The 34th Annual ACM Symposium on User Interface Software and
Technology</em>, pages 498–510, 2021.

</span>
</li>
<li id="bib.bib113" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. [2022a]</span>
<span class="ltx_bibblock">
W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K.
Mohammed, S. Singhal, S. Som, and F. Wei.

</span>
<span class="ltx_bibblock">Image as a foreign language: Beit pretraining for all vision and
vision-language tasks.

</span>
<span class="ltx_bibblock"><em id="bib.bib113.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2208.10442, 2022a.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2208.10442" title="" class="ltx_ref">10.48550/ARXIV.2208.10442</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2208.10442" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2208.10442</a>.

</span>
</li>
<li id="bib.bib114" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. [2023]</span>
<span class="ltx_bibblock">
W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y. Wang, J. Ji, Z. Yang, L. Zhao,
X. Song, J. Xu, B. Xu, J. Li, Y. Dong, M. Ding, and J. Tang.

</span>
<span class="ltx_bibblock">Cogvlm: Visual expert for pretrained language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib114.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2311.03079, 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2311.03079" title="" class="ltx_ref">10.48550/ARXIV.2311.03079</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2311.03079" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2311.03079</a>.

</span>
</li>
<li id="bib.bib115" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. [2019]</span>
<span class="ltx_bibblock">
X. Wang, J. Wu, J. Chen, L. Li, Y.-F. Wang, and W. Y. Wang.

</span>
<span class="ltx_bibblock">Vatex: A large-scale, high-quality multilingual dataset for
video-and-language research.

</span>
<span class="ltx_bibblock">In <em id="bib.bib115.1.1" class="ltx_emph ltx_font_italic">The IEEE International Conference on Computer Vision
(ICCV)</em>, October 2019.

</span>
</li>
<li id="bib.bib116" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. [2022b]</span>
<span class="ltx_bibblock">
Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, and Y. Cao.

</span>
<span class="ltx_bibblock">Simvlm: Simple visual language model pretraining with weak
supervision, 2022b.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2108.10904" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2108.10904</a>.

</span>
</li>
<li id="bib.bib117" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wu and Xie [2023]</span>
<span class="ltx_bibblock">
P. Wu and S. Xie.

</span>
<span class="ltx_bibblock">V*: Guided visual search as a core mechanism in multimodal llms,
2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2312.14135" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2312.14135</a>.

</span>
</li>
<li id="bib.bib118" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xiao et al. [2023]</span>
<span class="ltx_bibblock">
B. Xiao, H. Wu, W. Xu, X. Dai, H. Hu, Y. Lu, M. Zeng, C. Liu, and L. Yuan.

</span>
<span class="ltx_bibblock">Florence-2: Advancing a unified representation for a variety of
vision tasks.

</span>
<span class="ltx_bibblock"><em id="bib.bib118.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2311.06242, 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2311.06242" title="" class="ltx_ref">10.48550/ARXIV.2311.06242</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2311.06242" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2311.06242</a>.

</span>
</li>
<li id="bib.bib119" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. [2017]</span>
<span class="ltx_bibblock">
D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y. Zhuang.

</span>
<span class="ltx_bibblock">Video question answering via gradually refined attention over
appearance and motion.

</span>
<span class="ltx_bibblock">In <em id="bib.bib119.1.1" class="ltx_emph ltx_font_italic">ACM Multimedia</em>, 2017.

</span>
</li>
<li id="bib.bib120" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. [2016]</span>
<span class="ltx_bibblock">
J. Xu, T. Mei, T. Yao, and Y. Rui.

</span>
<span class="ltx_bibblock">Msr-vtt: A large video description dataset for bridging video and
language.

</span>
<span class="ltx_bibblock">In <em id="bib.bib120.1.1" class="ltx_emph ltx_font_italic">2016 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR)</em>, pages 5288–5296, 2016.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.1109/CVPR.2016.571" title="" class="ltx_ref">10.1109/CVPR.2016.571</a>.

</span>
</li>
<li id="bib.bib121" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. [2021]</span>
<span class="ltx_bibblock">
Y. Xu, H. Lee, D. Chen, B. A. Hechtman, Y. Huang, R. Joshi, M. Krikun,
D. Lepikhin, A. Ly, M. Maggioni, R. Pang, N. Shazeer, S. Wang, T. Wang,
Y. Wu, and Z. Chen.

</span>
<span class="ltx_bibblock">GSPMD: general and scalable parallelization for ML computation
graphs.

</span>
<span class="ltx_bibblock"><em id="bib.bib121.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2105.04663, 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2105.04663" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2105.04663</a>.

</span>
</li>
<li id="bib.bib122" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xue et al. [2020]</span>
<span class="ltx_bibblock">
L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua,
and C. Raffel.

</span>
<span class="ltx_bibblock">mt5: A massively multilingual pre-trained text-to-text transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib122.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2010.11934</em>, 2020.

</span>
</li>
<li id="bib.bib123" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yifan Li and Wen [2023]</span>
<span class="ltx_bibblock">
K. Z. J. W. W. X. Z. Yifan Li, Yifan Du and J.-R. Wen.

</span>
<span class="ltx_bibblock">Evaluating object hallucination in large vision-language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib123.1.1" class="ltx_emph ltx_font_italic">The 2023 Conference on Empirical Methods in Natural Language
Processing</em>, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://openreview.net/forum?id=xozJw0kZXF" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openreview.net/forum?id=xozJw0kZXF</a>.

</span>
</li>
<li id="bib.bib124" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yu et al. [2016]</span>
<span class="ltx_bibblock">
L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg.

</span>
<span class="ltx_bibblock">Modeling context in referring expressions.

</span>
<span class="ltx_bibblock">In B. Leibe, J. Matas, N. Sebe, and M. Welling, editors,
<em id="bib.bib124.1.1" class="ltx_emph ltx_font_italic">Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam,
The Netherlands, October 11-14, 2016, Proceedings, Part II</em>, volume 9906 of
<em id="bib.bib124.2.2" class="ltx_emph ltx_font_italic">Lecture Notes in Computer Science</em>, pages 69–85. Springer, 2016.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.1007/978-3-319-46475-6_5" title="" class="ltx_ref">10.1007/978-3-319-46475-6_5</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.1007/978-3-319-46475-6_5" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1007/978-3-319-46475-6_5</a>.

</span>
</li>
<li id="bib.bib125" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yu et al. [2023]</span>
<span class="ltx_bibblock">
L. Yu, B. Shi, R. Pasunuru, B. Muller, O. Golovneva, T. Wang, A. Babu, B. Tang,
B. Karrer, S. Sheynin, C. Ross, A. Polyak, R. Howes, V. Sharma, P. Xu,
H. Tamoyan, O. Ashual, U. Singer, S. Li, S. Zhang, R. James, G. Ghosh,
Y. Taigman, M. Fazel-Zarandi, A. Celikyilmaz, L. Zettlemoyer, and
A. Aghajanyan.

</span>
<span class="ltx_bibblock">Scaling autoregressive multi-modal models: Pretraining and
instruction tuning.

</span>
<span class="ltx_bibblock"><em id="bib.bib125.1.1" class="ltx_emph ltx_font_italic">CoRR</em>, abs/2309.02591, 2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.48550/ARXIV.2309.02591" title="" class="ltx_ref">10.48550/ARXIV.2309.02591</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.48550/arXiv.2309.02591" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.48550/arXiv.2309.02591</a>.

</span>
</li>
<li id="bib.bib126" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yu et al. [2019]</span>
<span class="ltx_bibblock">
Z. Yu, D. Xu, J. Yu, T. Yu, Z. Zhao, Y. Zhuang, and D. Tao.

</span>
<span class="ltx_bibblock">Activitynet-qa: A dataset for understanding complex web videos via
question answering.

</span>
<span class="ltx_bibblock">In <em id="bib.bib126.1.1" class="ltx_emph ltx_font_italic">The Thirty-Third AAAI Conference on Artificial
Intelligence, AAAI 2019, The Thirty-First Innovative Applications of
Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium
on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu,
Hawaii, USA, January 27 - February 1, 2019</em>, pages 9127–9134. AAAI Press,
2019.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.1609/AAAI.V33I01.33019127" title="" class="ltx_ref">10.1609/AAAI.V33I01.33019127</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.1609/aaai.v33i01.33019127" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1609/aaai.v33i01.33019127</a>.

</span>
</li>
<li id="bib.bib127" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhai et al. [2022a]</span>
<span class="ltx_bibblock">
X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer.

</span>
<span class="ltx_bibblock">Scaling vision transformers.

</span>
<span class="ltx_bibblock"><em id="bib.bib127.1.1" class="ltx_emph ltx_font_italic">Computer Vision and Pattern Recognition (CVPR)</em>,
2022a.

</span>
</li>
<li id="bib.bib128" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhai et al. [2022b]</span>
<span class="ltx_bibblock">
X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and
L. Beyer.

</span>
<span class="ltx_bibblock">Lit: Zero-shot transfer with locked-image text tuning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib128.1.1" class="ltx_emph ltx_font_italic">Computer Vision and Pattern Recognition (CVPR)</em>,
2022b.

</span>
</li>
<li id="bib.bib129" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhai et al. [2023]</span>
<span class="ltx_bibblock">
X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer.

</span>
<span class="ltx_bibblock">Sigmoid loss for language image pre-training.

</span>
<span class="ltx_bibblock">In <em id="bib.bib129.1.1" class="ltx_emph ltx_font_italic">ICCV</em>, pages 11941–11952, 2023.

</span>
</li>
<li id="bib.bib130" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. [2024]</span>
<span class="ltx_bibblock">
H. Zhang, H. You, P. Dufter, B. Zhang, C. Chen, H.-Y. Chen, T.-J. Fu, W. Y.
Wang, S.-F. Chang, Z. Gan, and Y. Yang.

</span>
<span class="ltx_bibblock">Ferret-v2: An improved baseline for referring and grounding with
large language models, 2024.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://arxiv.org/abs/2404.07973" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://arxiv.org/abs/2404.07973</a>.

</span>
</li>
<li id="bib.bib131" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. [2020]</span>
<span class="ltx_bibblock">
Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz.

</span>
<span class="ltx_bibblock">Contrastive learning of medical visual representations from paired
images and text.

</span>
<span class="ltx_bibblock"><em id="bib.bib131.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2010.00747</em>, 2020.

</span>
</li>
<li id="bib.bib132" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhao et al. [2023]</span>
<span class="ltx_bibblock">
Y. Zhao, A. Gu, R. Varma, L. Luo, C. Huang, M. Xu, L. Wright, H. Shojanazeri,
M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, P. Damania, B. Nguyen,
G. Chauhan, Y. Hao, A. Mathews, and S. Li.

</span>
<span class="ltx_bibblock">Pytorch FSDP: experiences on scaling fully sharded data parallel.

</span>
<span class="ltx_bibblock"><em id="bib.bib132.1.1" class="ltx_emph ltx_font_italic">Proc. VLDB Endow.</em>, 16(12):3848–3860,
2023.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.14778/3611540.3611569" title="" class="ltx_ref">10.14778/3611540.3611569</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://www.vldb.org/pvldb/vol16/p3848-huang.pdf" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.vldb.org/pvldb/vol16/p3848-huang.pdf</a>.

</span>
</li>
<li id="bib.bib133" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. [2020]</span>
<span class="ltx_bibblock">
L. Zhou, H. Palangi, L. Zhang, H. Hu, J. J. Corso, and J. Gao.

</span>
<span class="ltx_bibblock">Unified vision-language pre-training for image captioning and VQA.

</span>
<span class="ltx_bibblock">In <em id="bib.bib133.1.1" class="ltx_emph ltx_font_italic">The Thirty-Fourth AAAI Conference on Artificial
Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of
Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium
on Educational Advances in Artificial Intelligence, EAAI 2020, New York,
NY, USA, February 7-12, 2020</em>, pages 13041–13049. AAAI Press, 2020.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https:/doi.org/10.1609/AAAI.V34I07.7005" title="" class="ltx_ref">10.1609/AAAI.V34I07.7005</a>.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.1609/aaai.v34i07.7005" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1609/aaai.v34i07.7005</a>.

</span>
</li>
<li id="bib.bib134" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zou et al. [2023]</span>
<span class="ltx_bibblock">
X. Zou, Z. Dou, J. Yang, Z. Gan, L. Li, C. Li, X. Dai, H. Behl, J. Wang,
L. Yuan, N. Peng, L. Wang, Y. J. Lee, and J. Gao.

</span>
<span class="ltx_bibblock">Generalized decoding for pixel, image, and language.

</span>
<span class="ltx_bibblock">In <em id="bib.bib134.1.1" class="ltx_emph ltx_font_italic">Computer Vision and Pattern Recognition (CVPR)</em>, pages
15116–15127, 2023.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="Sx1" class="ltx_section" lang="en">
<h2 class="ltx_title ltx_title_section">Author Contributions</h2>

<section id="Sx1.SSx1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">Model development contributors</h3>

<section id="Sx1.SSx1.SSSx1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Core Contributors</h4>

<div id="Sx1.SSx1.SSSx1.p1" class="ltx_para">
<p id="Sx1.SSx1.SSSx1.p1.1" class="ltx_p">Lucas Beyer 
<br class="ltx_break">Andreas Steiner 
<br class="ltx_break">André Susano Pinto 
<br class="ltx_break">Alexander Kolesnikov 
<br class="ltx_break">Xiao Wang 
<br class="ltx_break">Xiaohua Zhai</p>
</div>
</section>
<section id="Sx1.SSx1.SSSx2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Contributors</h4>

<div id="Sx1.SSx1.SSSx2.p1" class="ltx_para">
<p id="Sx1.SSx1.SSSx2.p1.1" class="ltx_p">Daniel Salz 
<br class="ltx_break">Maxim Neumann 
<br class="ltx_break">Ibrahim Alabdulmohsin 
<br class="ltx_break">Michael Tschannen 
<br class="ltx_break">Emanuele Bugliarello 
<br class="ltx_break">Thomas Unterthiner 
<br class="ltx_break">Daniel Keysers 
<br class="ltx_break">Skanda Koppula 
<br class="ltx_break">Fangyu Liu 
<br class="ltx_break">Adam Grycner 
<br class="ltx_break">Alexey Gritsenko 
<br class="ltx_break">Neil Houlsby 
<br class="ltx_break">Manoj Kumar 
<br class="ltx_break">Keran Rong 
<br class="ltx_break">Julian Eisenschlos 
<br class="ltx_break">Rishabh Kabra 
<br class="ltx_break">Matthias Bauer 
<br class="ltx_break">Matko Bošnjak 
<br class="ltx_break">Xi Chen 
<br class="ltx_break">Matthias Minderer 
<br class="ltx_break">Paul Voigtlaender 
<br class="ltx_break">Ioana Bica 
<br class="ltx_break">Ivana Balazevic 
<br class="ltx_break">Joan Puigcerver 
<br class="ltx_break">Pinelopi Papalampidi 
<br class="ltx_break">Olivier Henaff 
<br class="ltx_break">Xi Xiong 
<br class="ltx_break">Radu Soricut 
<br class="ltx_break">Jeremiah Harmsen</p>
</div>
</section>
<section id="Sx1.SSx1.SSSx3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Leads</h4>

<div id="Sx1.SSx1.SSSx3.p1" class="ltx_para">
<p id="Sx1.SSx1.SSSx3.p1.1" class="ltx_p">Xiaohua Zhai 
<br class="ltx_break">Lucas Beyer</p>
</div>
</section>
</section>
<section id="Sx1.SSx2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">Model release contributors and 
<br class="ltx_break">general support</h3>

<section id="Sx1.SSx2.SSSx1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">PM</h4>

<div id="Sx1.SSx2.SSSx1.p1" class="ltx_para">
<p id="Sx1.SSx2.SSSx1.p1.1" class="ltx_p">Tris Warkentin</p>
</div>
</section>
<section id="Sx1.SSx2.SSSx2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Go-to-Market</h4>

<div id="Sx1.SSx2.SSSx2.p1" class="ltx_para">
<p id="Sx1.SSx2.SSSx2.p1.1" class="ltx_p">Kat Black 
<br class="ltx_break">Luiz Gustavo Martins 
<br class="ltx_break">Glenn Cameron 
<br class="ltx_break">Raj Gundluru 
<br class="ltx_break">Manvinder Singh</p>
</div>
</section>
<section id="Sx1.SSx2.SSSx3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Kaggle</h4>

<div id="Sx1.SSx2.SSSx3.p1" class="ltx_para">
<p id="Sx1.SSx2.SSSx3.p1.1" class="ltx_p">Meg Risdal 
<br class="ltx_break">Nilay Chauhan 
<br class="ltx_break">Nate Keating 
<br class="ltx_break">Nesh Devanathan</p>
</div>
</section>
<section id="Sx1.SSx2.SSSx4" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Documentation</h4>

<div id="Sx1.SSx2.SSSx4.p1" class="ltx_para">
<p id="Sx1.SSx2.SSSx4.p1.1" class="ltx_p">Elisa Bandy 
<br class="ltx_break">Joe Fernandez</p>
</div>
</section>
<section id="Sx1.SSx2.SSSx5" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Ethics and Safety</h4>

<div id="Sx1.SSx2.SSSx5.p1" class="ltx_para">
<p id="Sx1.SSx2.SSSx5.p1.1" class="ltx_p">Antonia Paterson 
<br class="ltx_break">Jenny Brennan 
<br class="ltx_break">Tom Eccles 
<br class="ltx_break">Pankil Botadra 
<br class="ltx_break">Ben Bariach</p>
</div>
</section>
<section id="Sx1.SSx2.SSSx6" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Vertex AI</h4>

<div id="Sx1.SSx2.SSSx6.p1" class="ltx_para">
<p id="Sx1.SSx2.SSSx6.p1.1" class="ltx_p">Lav Rai 
<br class="ltx_break">Minwoo Park 
<br class="ltx_break">Dustin Luong 
<br class="ltx_break">Daniel Vlasic 
<br class="ltx_break">Bo Wu 
<br class="ltx_break">Wenming Ye</p>
</div>
</section>
<section id="Sx1.SSx2.SSSx7" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Keras</h4>

<div id="Sx1.SSx2.SSSx7.p1" class="ltx_para">
<p id="Sx1.SSx2.SSSx7.p1.1" class="ltx_p">Divyashree Sreepathihalli 
<br class="ltx_break">Kiranbir Sodhia 
<br class="ltx_break">Gabriel Rasskin 
<br class="ltx_break">Matthew Watson 
<br class="ltx_break">Varun Singh</p>
</div>
</section>
<section id="Sx1.SSx2.SSSx8" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Gemma Model</h4>

<div id="Sx1.SSx2.SSSx8.p1" class="ltx_para">
<p id="Sx1.SSx2.SSSx8.p1.1" class="ltx_p">Alek Andreev 
<br class="ltx_break">Armand Joulin 
<br class="ltx_break">Surya Bhupatiraju 
<br class="ltx_break">Minh Giang</p>
</div>
</section>
<section id="Sx1.SSx2.SSSx9" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Executive Sponsors</h4>

<div id="Sx1.SSx2.SSSx9.p1" class="ltx_para">
<p id="Sx1.SSx2.SSSx9.p1.1" class="ltx_p">Joelle Barral 
<br class="ltx_break">Zoubin Ghahramani</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
</section>
<section id="A1" class="ltx_appendix" lang="en">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>More related work</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.1" class="ltx_p">There are generally two ways to build vision-language models (VLMs): the first option is to connect a vision encoder to a large language model while the second option is to use a transformer decoder-only architecture to handle both vision and language modalities.</p>
</div>
<div id="A1.p2" class="ltx_para">
<p id="A1.p2.1" class="ltx_p">It is popular to connect frozen image component and language component with lightweight adapters, e.g. linear or MLP projector, resampler <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib46" title="" class="ltx_ref">46</a>, <a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite>, Q-Former <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite>.
Flamingo <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite> uses a perceiver-based <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib46" title="" class="ltx_ref">46</a>]</cite> resampler to connect frozen vision and language models.
Idefics2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib57" title="" class="ltx_ref">57</a>]</cite> shows that the perceiver resampler works significantly better than a linear projector.
BLIP2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite> explores using trainable Q-Former <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite> to align frozen vision and language models.
They first train the Q-Former with frozen image model. Then attach the frozen image model and the Q-Former into frozen language model to continue the Q-Former training.
LLaVA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib67" title="" class="ltx_ref">67</a>]</cite> opted to train a projection layer between frozen vision backbone and frozen language backbone, with GPT-4 generated small but high quality instruction-following data.
Afterwards, they unfreeze the language backbone and finetune the projection layer and language model together.
LLaVA-1.5 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib66" title="" class="ltx_ref">66</a>]</cite> extends this to MLP connector.
Bunny <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib39" title="" class="ltx_ref">39</a>]</cite> opted to use MLP connector for their models.
Honeybee <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib17" title="" class="ltx_ref">17</a>]</cite> introduced locality-enhanced projectors by using convolution and deformable attention, for better spatial understanding.
MM1 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib78" title="" class="ltx_ref">78</a>]</cite> claimed that convolution adaptor <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib17" title="" class="ltx_ref">17</a>]</cite> performs close to average pooling and attention pooling baselines.
Cambrian-1 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib103" title="" class="ltx_ref">103</a>]</cite> performed a thorough study of vision encoders for VLMs, and proposed spatial vision aggregator to better integrate visual tokens.
CogVLM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib114" title="" class="ltx_ref">114</a>]</cite> introduced additional trainable visual expert module in the attention and FFN layers of the frozen language model.
This way, the language model is able to process visual tokens and language tokens with different experts, while keeping the same level of performance for text-only tasks.
We performed an ablation study of linear and MLP vision-language connectors for PaliGemma in Section <a href="#S5.SS5" title="5.5 Connector design ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.5</span></a>.</p>
</div>
<div id="A1.p3" class="ltx_para">
<p id="A1.p3.1" class="ltx_p">There are also methods exploring training both the vision and language components, PaliGemma falls into this category.
Many VLM systems  <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>, <a href="#bib.bib23" title="" class="ltx_ref">23</a>, <a href="#bib.bib24" title="" class="ltx_ref">24</a>, <a href="#bib.bib25" title="" class="ltx_ref">25</a>, <a href="#bib.bib10" title="" class="ltx_ref">10</a>]</cite> follow a multi-stage training procedure, including a stage to train both vision and language components.
PaLI line of work <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>, <a href="#bib.bib23" title="" class="ltx_ref">23</a>, <a href="#bib.bib24" title="" class="ltx_ref">24</a>]</cite> gradually scales up the training resolution with different data mixtures in three stages.
Florence2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib118" title="" class="ltx_ref">118</a>]</cite> and LocCa <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib111" title="" class="ltx_ref">111</a>]</cite> train vision-centric models by modeling very diverse tasks with a universal language interface.
Unified-IO 2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib69" title="" class="ltx_ref">69</a>]</cite> trains a single encoder-decoder multimodal model on an ensemble of 120 datasets.
Kosmos <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib43" title="" class="ltx_ref">43</a>]</cite> trains the language model and the last layer of a CLIP vision model.
BLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib59" title="" class="ltx_ref">59</a>]</cite> proposed to dump COCO like pseudo captions on million scale web data, and then use filters to choose from original noisy captions and pseudo captions to improve the quality of vision-language training data.
BEIT3 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib113" title="" class="ltx_ref">113</a>]</cite> treats image data and language data the same way as discrete tokens. The image data is tokenized by the tokenizer of BEIT2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib86" title="" class="ltx_ref">86</a>]</cite>.
Image, text, and image-text pairs are randomly masked and the model is trained to recover the randomly masked tokens.
EMU2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib99" title="" class="ltx_ref">99</a>]</cite> operates in the continuous visual embedding space and jointly modeling the visual embeddings and text embeddings with a language decoder. The visual embeddings could later be decoded back to image pixels or video clips.</p>
</div>
<div id="A1.p4" class="ltx_para">
<p id="A1.p4.1" class="ltx_p">In the category of decoder-only VLMs,
Fuyu <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite> proposed to use an vision encoder-free architecture, that employs a transformer decoder-only model to process both image and text inputs.
The input image is first patchfied and then linearly projected to a shared continuous embedding space as text tokens, so that the decoder-only model could process image and text tokens seamlessly.
CM3 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib2" title="" class="ltx_ref">2</a>, <a href="#bib.bib3" title="" class="ltx_ref">3</a>, <a href="#bib.bib125" title="" class="ltx_ref">125</a>]</cite> and Chameleon <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib101" title="" class="ltx_ref">101</a>]</cite> proposed to convert images into discrete tokens and then model them together with language in the token space with a shared transformer decoder model.
Our work also shared promising early results of a Fuyu-style decoder-only setup following this line of work, by ablating the SigLIP vision encoder component from PaliGemma in Section <a href="#S5.SS6" title="5.6 Image encoder: with or without? ‣ 5 Ablations ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.6</span></a>.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A2" class="ltx_appendix" lang="en">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Tasks</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.1" class="ltx_p">We provide one training example for each transfer task, exactly as it reaches the model.
The image shown has been resized to <math id="A2.p1.1.m1.1" class="ltx_Math" alttext="224\times 224" display="inline"><semantics id="A2.p1.1.m1.1a"><mrow id="A2.p1.1.m1.1.1" xref="A2.p1.1.m1.1.1.cmml"><mn id="A2.p1.1.m1.1.1.2" xref="A2.p1.1.m1.1.1.2.cmml">224</mn><mo lspace="0.222em" rspace="0.222em" id="A2.p1.1.m1.1.1.1" xref="A2.p1.1.m1.1.1.1.cmml">×</mo><mn id="A2.p1.1.m1.1.1.3" xref="A2.p1.1.m1.1.1.3.cmml">224</mn></mrow><annotation-xml encoding="MathML-Content" id="A2.p1.1.m1.1b"><apply id="A2.p1.1.m1.1.1.cmml" xref="A2.p1.1.m1.1.1"><times id="A2.p1.1.m1.1.1.1.cmml" xref="A2.p1.1.m1.1.1.1"></times><cn type="integer" id="A2.p1.1.m1.1.1.2.cmml" xref="A2.p1.1.m1.1.1.2">224</cn><cn type="integer" id="A2.p1.1.m1.1.1.3.cmml" xref="A2.p1.1.m1.1.1.3">224</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.p1.1.m1.1c">224\times 224</annotation></semantics></math> pixels in order to convey how much can be recognized at that resolution.</p>
</div>
<section id="A2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.1 </span>ActivityNet-CAP: Captioning of short video snippets of activities</h3>

<div id="A2.SS1.p1" class="ltx_para">
<table id="A2.SS1.p1.3" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS1.p1.2.2" class="ltx_tr">
<td id="A2.SS1.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS1.p1.2.2.2.2" class="ltx_inline-block ltx_align_top">
<span id="A2.SS1.p1.2.2.2.2.2" class="ltx_p">
<span id="A2.SS1.p1.2.2.2.2.2.2" class="ltx_tabular ltx_align_middle">
<span id="A2.SS1.p1.1.1.1.1.1.1.1" class="ltx_tr">
<span id="A2.SS1.p1.1.1.1.1.1.1.1.1" class="ltx_td ltx_align_center"><span id="A2.SS1.p1.1.1.1.1.1.1.1.1.1" class="ltx_text"><svg version="1.1" width="67" height="67" overflow="visible"><g transform="translate(0,67) scale(1,-1)"><rect fill="none" height="48.4pt" stroke="#000000" stroke-width="0.4" width="48.4pt" x="0" y="0"></rect><g class="makebox" transform="translate(0,0)"><g transform="translate(0,67) scale(1, -1)"><foreignObject width="67" height="67" overflow="visible"><img src="/html/2407.07726/assets/figures/tasks/activitynet_cap_0.jpg" id="A2.SS1.p1.1.1.1.1.1.1.1.1.1.pic1.1.g1" class="ltx_graphics ltx_img_square" width="67" height="67" alt="[Uncaptioned image]"></foreignObject></g></g></g></svg></span></span></span>
<span id="A2.SS1.p1.2.2.2.2.2.2.2" class="ltx_tr">
<span id="A2.SS1.p1.2.2.2.2.2.2.2.1" class="ltx_td ltx_align_center"><span id="A2.SS1.p1.2.2.2.2.2.2.2.1.1" class="ltx_text"><svg version="1.1" width="67" height="67" overflow="visible"><g transform="translate(0,67) scale(1,-1)"><rect fill="none" height="48.4pt" stroke="#000000" stroke-width="0.4" width="48.4pt" x="0" y="0"></rect><g class="makebox" transform="translate(0,0)"><g transform="translate(0,67) scale(1, -1)"><foreignObject width="67" height="67" overflow="visible"><img src="/html/2407.07726/assets/figures/tasks/activitynet_cap_14.jpg" id="A2.SS1.p1.2.2.2.2.2.2.2.1.1.pic1.1.g1" class="ltx_graphics ltx_img_square" width="67" height="67" alt="[Uncaptioned image]"></foreignObject></g></g></g></svg></span></span></span>
</span></span>
</span>
</td>
<td id="A2.SS1.p1.2.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS1.p1.2.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS1.p1.2.2.3.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS1.p1.2.2.4" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS1.p1.2.2.4.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS1.p1.2.2.4.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS1.p1.2.2.4.1.1.1" class="ltx_text ltx_font_typewriter">"caption en\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS1.p1.3.4.1" class="ltx_tr">
<td id="A2.SS1.p1.3.4.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS1.p1.3.4.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS1.p1.3.4.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS1.p1.3.4.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS1.p1.3.4.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS1.p1.3.4.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS1.p1.3.4.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"The man washes pots and a cutting board."</span></span>
</span>
</td>
</tr>
<tr id="A2.SS1.p1.3.3" class="ltx_tr">
<td id="A2.SS1.p1.3.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS1.p1.3.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS1.p1.3.3.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS1.p1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS1.p1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS1.p1.3.3.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS1.p1.3.3.1.1.1.m1.1" class="ltx_Math" alttext="44775" display="inline"><semantics id="A2.SS1.p1.3.3.1.1.1.m1.1a"><mn id="A2.SS1.p1.3.3.1.1.1.m1.1.1" xref="A2.SS1.p1.3.3.1.1.1.m1.1.1.cmml">44775</mn><annotation-xml encoding="MathML-Content" id="A2.SS1.p1.3.3.1.1.1.m1.1b"><cn type="integer" id="A2.SS1.p1.3.3.1.1.1.m1.1.1.cmml" xref="A2.SS1.p1.3.3.1.1.1.m1.1.1">44775</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p1.3.3.1.1.1.m1.1c">44775</annotation></semantics></math> examples (train+val).</span>
</span>
</td>
</tr>
<tr id="A2.SS1.p1.3.5.2" class="ltx_tr">
<td id="A2.SS1.p1.3.5.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS1.p1.3.5.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS1.p1.3.5.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS1.p1.3.5.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS1.p1.3.5.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS1.p1.3.5.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS1.p1.3.5.2.3.1.1" class="ltx_p" style="width:256.1pt;">CIDEr (test split).</span>
</span>
</td>
</tr>
<tr id="A2.SS1.p1.3.6.3" class="ltx_tr">
<td id="A2.SS1.p1.3.6.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS1.p1.3.6.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS1.p1.3.6.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS1.p1.3.6.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS1.p1.3.6.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS1.p1.3.6.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS1.p1.3.6.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Krishna et al. [<a href="#bib.bib55" title="" class="ltx_ref">55</a>]</cite>.</span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.2 </span>ActivityNet-QA: Answering questions about short video snippets of activities</h3>

<div id="A2.SS2.p1" class="ltx_para">
<table id="A2.SS2.p1.3" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS2.p1.2.2" class="ltx_tr">
<td id="A2.SS2.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS2.p1.2.2.2.2" class="ltx_inline-block ltx_align_top">
<span id="A2.SS2.p1.2.2.2.2.2" class="ltx_p">
<span id="A2.SS2.p1.2.2.2.2.2.2" class="ltx_tabular ltx_align_middle">
<span id="A2.SS2.p1.1.1.1.1.1.1.1" class="ltx_tr">
<span id="A2.SS2.p1.1.1.1.1.1.1.1.1" class="ltx_td ltx_align_center"><span id="A2.SS2.p1.1.1.1.1.1.1.1.1.1" class="ltx_text"><svg version="1.1" width="67" height="67" overflow="visible"><g transform="translate(0,67) scale(1,-1)"><rect fill="none" height="48.4pt" stroke="#000000" stroke-width="0.4" width="48.4pt" x="0" y="0"></rect><g class="makebox" transform="translate(0,0)"><g transform="translate(0,67) scale(1, -1)"><foreignObject width="67" height="67" overflow="visible"><img src="/html/2407.07726/assets/figures/tasks/activitynet_qa_0.jpg" id="A2.SS2.p1.1.1.1.1.1.1.1.1.1.pic1.1.g1" class="ltx_graphics ltx_img_square" width="67" height="67" alt="[Uncaptioned image]"></foreignObject></g></g></g></svg></span></span></span>
<span id="A2.SS2.p1.2.2.2.2.2.2.2" class="ltx_tr">
<span id="A2.SS2.p1.2.2.2.2.2.2.2.1" class="ltx_td ltx_align_center"><span id="A2.SS2.p1.2.2.2.2.2.2.2.1.1" class="ltx_text"><svg version="1.1" width="67" height="67" overflow="visible"><g transform="translate(0,67) scale(1,-1)"><rect fill="none" height="48.4pt" stroke="#000000" stroke-width="0.4" width="48.4pt" x="0" y="0"></rect><g class="makebox" transform="translate(0,0)"><g transform="translate(0,67) scale(1, -1)"><foreignObject width="67" height="67" overflow="visible"><img src="/html/2407.07726/assets/figures/tasks/activitynet_qa_11.jpg" id="A2.SS2.p1.2.2.2.2.2.2.2.1.1.pic1.1.g1" class="ltx_graphics ltx_img_square" width="67" height="67" alt="[Uncaptioned image]"></foreignObject></g></g></g></svg></span></span></span>
</span></span>
</span>
</td>
<td id="A2.SS2.p1.2.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS2.p1.2.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS2.p1.2.2.3.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS2.p1.2.2.4" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS2.p1.2.2.4.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS2.p1.2.2.4.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS2.p1.2.2.4.1.1.1" class="ltx_text ltx_font_typewriter">"answer en what is the person in the video doing\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS2.p1.3.4.1" class="ltx_tr">
<td id="A2.SS2.p1.3.4.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS2.p1.3.4.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS2.p1.3.4.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS2.p1.3.4.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS2.p1.3.4.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS2.p1.3.4.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS2.p1.3.4.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"bathe dog"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS2.p1.3.3" class="ltx_tr">
<td id="A2.SS2.p1.3.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS2.p1.3.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS2.p1.3.3.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS2.p1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS2.p1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS2.p1.3.3.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS2.p1.3.3.1.1.1.m1.1" class="ltx_Math" alttext="43130" display="inline"><semantics id="A2.SS2.p1.3.3.1.1.1.m1.1a"><mn id="A2.SS2.p1.3.3.1.1.1.m1.1.1" xref="A2.SS2.p1.3.3.1.1.1.m1.1.1.cmml">43130</mn><annotation-xml encoding="MathML-Content" id="A2.SS2.p1.3.3.1.1.1.m1.1b"><cn type="integer" id="A2.SS2.p1.3.3.1.1.1.m1.1.1.cmml" xref="A2.SS2.p1.3.3.1.1.1.m1.1.1">43130</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS2.p1.3.3.1.1.1.m1.1c">43130</annotation></semantics></math> examples (train+val).</span>
</span>
</td>
</tr>
<tr id="A2.SS2.p1.3.5.2" class="ltx_tr">
<td id="A2.SS2.p1.3.5.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS2.p1.3.5.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS2.p1.3.5.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS2.p1.3.5.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS2.p1.3.5.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS2.p1.3.5.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS2.p1.3.5.2.3.1.1" class="ltx_p" style="width:256.1pt;">Exact match accuracy (test split).</span>
</span>
</td>
</tr>
<tr id="A2.SS2.p1.3.6.3" class="ltx_tr">
<td id="A2.SS2.p1.3.6.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS2.p1.3.6.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS2.p1.3.6.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS2.p1.3.6.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS2.p1.3.6.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS2.p1.3.6.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS2.p1.3.6.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Yu et al. [<a href="#bib.bib126" title="" class="ltx_ref">126</a>]</cite>.</span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.3 </span>AI2D: VQA on science diagrams</h3>

<div id="A2.SS3.p1" class="ltx_para">
<table id="A2.SS3.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS3.p1.1.1" class="ltx_tr">
<td id="A2.SS3.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS3.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS3.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/ai2d.png" id="A2.SS3.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS3.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS3.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS3.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS3.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS3.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS3.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS3.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en Mayfly and dragonfly are eaten by ? choose from: Bald eagle \t Frog \t Phytoplankton \t None of the above\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS3.p1.2.3.1" class="ltx_tr">
<td id="A2.SS3.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS3.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS3.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS3.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS3.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS3.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS3.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"Frog"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS3.p1.2.2" class="ltx_tr">
<td id="A2.SS3.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS3.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS3.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS3.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS3.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS3.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS3.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="12413" display="inline"><semantics id="A2.SS3.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS3.p1.2.2.1.1.1.m1.1.1" xref="A2.SS3.p1.2.2.1.1.1.m1.1.1.cmml">12413</mn><annotation-xml encoding="MathML-Content" id="A2.SS3.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS3.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS3.p1.2.2.1.1.1.m1.1.1">12413</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS3.p1.2.2.1.1.1.m1.1c">12413</annotation></semantics></math> examples (train split).</span>
</span>
</td>
</tr>
<tr id="A2.SS3.p1.2.4.2" class="ltx_tr">
<td id="A2.SS3.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS3.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS3.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS3.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS3.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS3.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS3.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">Exact match accuracy (test split).</span>
</span>
</td>
</tr>
<tr id="A2.SS3.p1.2.5.3" class="ltx_tr">
<td id="A2.SS3.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS3.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS3.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS3.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS3.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS3.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS3.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Aniruddha Kembhavi [<a href="#bib.bib8" title="" class="ltx_ref">8</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.4 </span>AOKVQA-DA: VQA using world knowledge (direct answer)</h3>

<div id="A2.SS4.p1" class="ltx_para">
<table id="A2.SS4.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS4.p1.1.1" class="ltx_tr">
<td id="A2.SS4.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS4.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS4.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/aokvqa_da.png" id="A2.SS4.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS4.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS4.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS4.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS4.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS4.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS4.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS4.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en What is a sound this animal makes?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS4.p1.2.3.1" class="ltx_tr">
<td id="A2.SS4.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS4.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS4.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS4.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS4.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS4.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS4.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"meow"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS4.p1.2.2" class="ltx_tr">
<td id="A2.SS4.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS4.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS4.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS4.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS4.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS4.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS4.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="18201" display="inline"><semantics id="A2.SS4.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS4.p1.2.2.1.1.1.m1.1.1" xref="A2.SS4.p1.2.2.1.1.1.m1.1.1.cmml">18201</mn><annotation-xml encoding="MathML-Content" id="A2.SS4.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS4.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS4.p1.2.2.1.1.1.m1.1.1">18201</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS4.p1.2.2.1.1.1.m1.1c">18201</annotation></semantics></math> examples (train+val splits).</span>
</span>
</td>
</tr>
<tr id="A2.SS4.p1.2.4.2" class="ltx_tr">
<td id="A2.SS4.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS4.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS4.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS4.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS4.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS4.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS4.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">Exact match accuracy (test server).</span>
</span>
</td>
</tr>
<tr id="A2.SS4.p1.2.5.3" class="ltx_tr">
<td id="A2.SS4.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS4.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS4.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS4.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS4.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS4.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS4.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Schwenk et al. [<a href="#bib.bib94" title="" class="ltx_ref">94</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.5 </span>AOKVQA-MC: VQA using world knowledge (multiple choice)</h3>

<div id="A2.SS5.p1" class="ltx_para">
<table id="A2.SS5.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS5.p1.1.1" class="ltx_tr">
<td id="A2.SS5.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS5.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS5.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/aokvqa_mc.jpg" id="A2.SS5.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS5.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS5.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS5.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS5.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS5.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS5.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS5.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en What has the yellow object been drawn on to resemble? choose from: eagle \t face \t dog \t star\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS5.p1.2.3.1" class="ltx_tr">
<td id="A2.SS5.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS5.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS5.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS5.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS5.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS5.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS5.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"face"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS5.p1.2.2" class="ltx_tr">
<td id="A2.SS5.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS5.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS5.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS5.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS5.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS5.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS5.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="18201" display="inline"><semantics id="A2.SS5.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS5.p1.2.2.1.1.1.m1.1.1" xref="A2.SS5.p1.2.2.1.1.1.m1.1.1.cmml">18201</mn><annotation-xml encoding="MathML-Content" id="A2.SS5.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS5.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS5.p1.2.2.1.1.1.m1.1.1">18201</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS5.p1.2.2.1.1.1.m1.1c">18201</annotation></semantics></math> examples (train+val splits).</span>
</span>
</td>
</tr>
<tr id="A2.SS5.p1.2.4.2" class="ltx_tr">
<td id="A2.SS5.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS5.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS5.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS5.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS5.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS5.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS5.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">Accuracy (test server).</span>
</span>
</td>
</tr>
<tr id="A2.SS5.p1.2.5.3" class="ltx_tr">
<td id="A2.SS5.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS5.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS5.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS5.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS5.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS5.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS5.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Schwenk et al. [<a href="#bib.bib94" title="" class="ltx_ref">94</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.6 </span>ChartQA: VQA about charts with logical reasoning</h3>

<div id="A2.SS6.p1" class="ltx_para">
<table id="A2.SS6.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS6.p1.1.1" class="ltx_tr">
<td id="A2.SS6.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS6.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS6.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/chartqa.jpg" id="A2.SS6.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS6.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS6.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS6.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS6.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS6.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS6.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS6.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"What country accounted for 15 percent of the world's machine tool production in 2020?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS6.p1.2.3.1" class="ltx_tr">
<td id="A2.SS6.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS6.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS6.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS6.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS6.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS6.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS6.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"Germany"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS6.p1.2.2" class="ltx_tr">
<td id="A2.SS6.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS6.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS6.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train</span>
</span>
</td>
<td id="A2.SS6.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS6.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS6.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS6.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="30219" display="inline"><semantics id="A2.SS6.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS6.p1.2.2.1.1.1.m1.1.1" xref="A2.SS6.p1.2.2.1.1.1.m1.1.1.cmml">30219</mn><annotation-xml encoding="MathML-Content" id="A2.SS6.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS6.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS6.p1.2.2.1.1.1.m1.1.1">30219</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS6.p1.2.2.1.1.1.m1.1c">30219</annotation></semantics></math> examples (human+augmented train and validation splits).</span>
</span>
</td>
</tr>
<tr id="A2.SS6.p1.2.4.2" class="ltx_tr">
<td id="A2.SS6.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS6.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS6.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS6.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS6.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS6.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS6.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">Relaxed match accuracy. ChartQA-human (human test split), ChartQA-aug (augmented test split).</span>
</span>
</td>
</tr>
<tr id="A2.SS6.p1.2.5.3" class="ltx_tr">
<td id="A2.SS6.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS6.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS6.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS6.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS6.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS6.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS6.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Masry et al. [<a href="#bib.bib75" title="" class="ltx_ref">75</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS7" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.7 </span>COCOcap: COCO image captioning task</h3>

<div id="A2.SS7.p1" class="ltx_para">
<table id="A2.SS7.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS7.p1.1.1" class="ltx_tr">
<td id="A2.SS7.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="4">
<span id="A2.SS7.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS7.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/cococap.jpg" id="A2.SS7.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS7.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS7.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS7.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS7.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS7.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS7.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS7.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"caption en\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS7.p1.2.3.1" class="ltx_tr">
<td id="A2.SS7.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS7.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS7.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS7.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS7.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS7.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS7.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"A full view of a living room with pillows and books. "</span></span>
</span>
</td>
</tr>
<tr id="A2.SS7.p1.2.2" class="ltx_tr">
<td id="A2.SS7.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS7.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS7.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS7.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS7.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS7.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS7.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="113287" display="inline"><semantics id="A2.SS7.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS7.p1.2.2.1.1.1.m1.1.1" xref="A2.SS7.p1.2.2.1.1.1.m1.1.1.cmml">113287</mn><annotation-xml encoding="MathML-Content" id="A2.SS7.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS7.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS7.p1.2.2.1.1.1.m1.1.1">113287</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS7.p1.2.2.1.1.1.m1.1c">113287</annotation></semantics></math> images each with 5 captions (train+restval splits).</span>
</span>
</td>
</tr>
<tr id="A2.SS7.p1.2.4.2" class="ltx_tr">
<td id="A2.SS7.p1.2.4.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS7.p1.2.4.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS7.p1.2.4.2.1.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS7.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS7.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS7.p1.2.4.2.2.1.1" class="ltx_p" style="width:256.1pt;">CIDEr score (val split).</span>
</span>
</td>
</tr>
<tr id="A2.SS7.p1.2.5.3" class="ltx_tr">
<td id="A2.SS7.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS7.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS7.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS7.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS7.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS7.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS7.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Lin et al. [<a href="#bib.bib65" title="" class="ltx_ref">65</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS8" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.8 </span>NoCaps: Novel object captioning</h3>

<div id="A2.SS8.p1" class="ltx_para">
<table id="A2.SS8.p1.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS8.p1.1.1" class="ltx_tr">
<td id="A2.SS8.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="4">
<span id="A2.SS8.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS8.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/nocaps.jpg" id="A2.SS8.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS8.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS8.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS8.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS8.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS8.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS8.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS8.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"caption en\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS8.p1.1.2.1" class="ltx_tr">
<td id="A2.SS8.p1.1.2.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS8.p1.1.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS8.p1.1.2.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS8.p1.1.2.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS8.p1.1.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS8.p1.1.2.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS8.p1.1.2.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"up close shot of street light with banners and tree's in the distance"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS8.p1.1.3.2" class="ltx_tr">
<td id="A2.SS8.p1.1.3.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS8.p1.1.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS8.p1.1.3.2.1.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS8.p1.1.3.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS8.p1.1.3.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS8.p1.1.3.2.2.1.1" class="ltx_p" style="width:256.1pt;">Zero-shot evaluation of the model trained for COCOcap.</span>
</span>
</td>
</tr>
<tr id="A2.SS8.p1.1.4.3" class="ltx_tr">
<td id="A2.SS8.p1.1.4.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS8.p1.1.4.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS8.p1.1.4.3.1.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS8.p1.1.4.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS8.p1.1.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS8.p1.1.4.3.2.1.1" class="ltx_p" style="width:256.1pt;">CIDEr score (val split).</span>
</span>
</td>
</tr>
<tr id="A2.SS8.p1.1.5.4" class="ltx_tr">
<td id="A2.SS8.p1.1.5.4.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS8.p1.1.5.4.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS8.p1.1.5.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS8.p1.1.5.4.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS8.p1.1.5.4.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS8.p1.1.5.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS8.p1.1.5.4.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Agrawal et al. [<a href="#bib.bib4" title="" class="ltx_ref">4</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS9" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.9 </span>COCO-35L: COCO captions translated in 35 languages</h3>

<div id="A2.SS9.p1" class="ltx_para">
<table id="A2.SS9.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS9.p1.1.1" class="ltx_tr">
<td id="A2.SS9.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS9.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS9.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/coco35l.jpg" id="A2.SS9.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS9.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS9.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS9.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS9.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS9.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS9.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS9.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"caption no\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS9.p1.2.3.1" class="ltx_tr">
<td id="A2.SS9.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS9.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS9.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS9.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS9.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS9.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS9.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"pizza serveres p\u00e5 bordet med gafler og kniver . ."</span></span>
</span>
</td>
</tr>
<tr id="A2.SS9.p1.2.2" class="ltx_tr">
<td id="A2.SS9.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS9.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS9.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS9.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS9.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS9.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS9.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="113287" display="inline"><semantics id="A2.SS9.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS9.p1.2.2.1.1.1.m1.1.1" xref="A2.SS9.p1.2.2.1.1.1.m1.1.1.cmml">113287</mn><annotation-xml encoding="MathML-Content" id="A2.SS9.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS9.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS9.p1.2.2.1.1.1.m1.1.1">113287</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS9.p1.2.2.1.1.1.m1.1c">113287</annotation></semantics></math> images each with 5 captions in each of the 35 languages (train split).</span>
</span>
</td>
</tr>
<tr id="A2.SS9.p1.2.4.2" class="ltx_tr">
<td id="A2.SS9.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS9.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS9.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS9.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS9.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS9.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS9.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">Mean of CIDEr on each of the 35 languages (dev split).</span>
</span>
</td>
</tr>
<tr id="A2.SS9.p1.2.5.3" class="ltx_tr">
<td id="A2.SS9.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS9.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS9.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS9.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS9.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS9.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS9.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Thapliyal et al. [<a href="#bib.bib102" title="" class="ltx_ref">102</a>]</cite></span>
</span>
</td>
</tr>
<tr id="A2.SS9.p1.2.6.4" class="ltx_tr">
<td id="A2.SS9.p1.2.6.4.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS9.p1.2.6.4.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS9.p1.2.6.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS9.p1.2.6.4.2.1.1" class="ltx_p" style="width:42.7pt;">Note:</span>
</span>
</td>
<td id="A2.SS9.p1.2.6.4.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS9.p1.2.6.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS9.p1.2.6.4.3.1.1" class="ltx_p" style="width:256.1pt;">The original raw data comes pre-tokenized with sometimes odd punctuation.</span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS10" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.10 </span>XM3600: caption geographically-diverse images in 36 languages</h3>

<div id="A2.SS10.p1" class="ltx_para">
<table id="A2.SS10.p1.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS10.p1.1.1" class="ltx_tr">
<td id="A2.SS10.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS10.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS10.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/xm3600.jpg" id="A2.SS10.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS10.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS10.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS10.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS10.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS10.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS10.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS10.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"caption pt\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS10.p1.1.2.1" class="ltx_tr">
<td id="A2.SS10.p1.1.2.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS10.p1.1.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS10.p1.1.2.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS10.p1.1.2.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS10.p1.1.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS10.p1.1.2.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS10.p1.1.2.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"Silhoueta de um corvo"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS10.p1.1.3.2" class="ltx_tr">
<td id="A2.SS10.p1.1.3.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS10.p1.1.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS10.p1.1.3.2.1.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS10.p1.1.3.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS10.p1.1.3.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS10.p1.1.3.2.2.1.1" class="ltx_p" style="width:256.1pt;">Zero-shot evaluation of the model trained for COCO-35L.</span>
</span>
</td>
</tr>
<tr id="A2.SS10.p1.1.4.3" class="ltx_tr">
<td id="A2.SS10.p1.1.4.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS10.p1.1.4.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS10.p1.1.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS10.p1.1.4.3.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS10.p1.1.4.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS10.p1.1.4.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS10.p1.1.4.3.3.1.1" class="ltx_p" style="width:256.1pt;">Mean of CIDEr on each of the 36 languages (dev split).</span>
</span>
</td>
</tr>
<tr id="A2.SS10.p1.1.5.4" class="ltx_tr">
<td id="A2.SS10.p1.1.5.4.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS10.p1.1.5.4.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS10.p1.1.5.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS10.p1.1.5.4.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS10.p1.1.5.4.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS10.p1.1.5.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS10.p1.1.5.4.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Thapliyal et al. [<a href="#bib.bib102" title="" class="ltx_ref">102</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS11" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.11 </span>DocVQA: VQA on document images</h3>

<div id="A2.SS11.p1" class="ltx_para">
<table id="A2.SS11.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS11.p1.1.1" class="ltx_tr">
<td id="A2.SS11.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS11.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS11.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/docvqa.jpg" id="A2.SS11.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS11.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS11.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS11.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS11.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS11.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS11.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS11.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"What is the arrival time in GSO?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS11.p1.2.3.1" class="ltx_tr">
<td id="A2.SS11.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS11.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS11.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS11.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS11.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS11.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS11.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"9:33 p.m."</span></span>
</span>
</td>
</tr>
<tr id="A2.SS11.p1.2.2" class="ltx_tr">
<td id="A2.SS11.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS11.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS11.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS11.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS11.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS11.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS11.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="44812" display="inline"><semantics id="A2.SS11.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS11.p1.2.2.1.1.1.m1.1.1" xref="A2.SS11.p1.2.2.1.1.1.m1.1.1.cmml">44812</mn><annotation-xml encoding="MathML-Content" id="A2.SS11.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS11.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS11.p1.2.2.1.1.1.m1.1.1">44812</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS11.p1.2.2.1.1.1.m1.1c">44812</annotation></semantics></math> examples (train+val splits)</span>
</span>
</td>
</tr>
<tr id="A2.SS11.p1.2.4.2" class="ltx_tr">
<td id="A2.SS11.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS11.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS11.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS11.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS11.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS11.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS11.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">ANLS (test server).</span>
</span>
</td>
</tr>
<tr id="A2.SS11.p1.2.5.3" class="ltx_tr">
<td id="A2.SS11.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS11.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS11.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS11.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS11.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS11.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS11.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Mathew et al. [<a href="#bib.bib76" title="" class="ltx_ref">76</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS12" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.12 </span>GQA: VQA on image scene graphs</h3>

<div id="A2.SS12.p1" class="ltx_para">
<table id="A2.SS12.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS12.p1.1.1" class="ltx_tr">
<td id="A2.SS12.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS12.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS12.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/gqa.jpg" id="A2.SS12.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS12.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS12.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS12.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS12.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS12.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS12.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS12.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en Where in the picture is the plate, in the bottom or in the top?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS12.p1.2.3.1" class="ltx_tr">
<td id="A2.SS12.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS12.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS12.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS12.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS12.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS12.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS12.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"top"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS12.p1.2.2" class="ltx_tr">
<td id="A2.SS12.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS12.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS12.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS12.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS12.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS12.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS12.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="1075062" display="inline"><semantics id="A2.SS12.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS12.p1.2.2.1.1.1.m1.1.1" xref="A2.SS12.p1.2.2.1.1.1.m1.1.1.cmml">1075062</mn><annotation-xml encoding="MathML-Content" id="A2.SS12.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS12.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS12.p1.2.2.1.1.1.m1.1.1">1075062</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS12.p1.2.2.1.1.1.m1.1c">1075062</annotation></semantics></math> examples (train+val balanced splits).</span>
</span>
</td>
</tr>
<tr id="A2.SS12.p1.2.4.2" class="ltx_tr">
<td id="A2.SS12.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS12.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS12.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS12.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS12.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS12.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS12.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">Exact-match accuracy (testdev balanced split).</span>
</span>
</td>
</tr>
<tr id="A2.SS12.p1.2.5.3" class="ltx_tr">
<td id="A2.SS12.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS12.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS12.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS12.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS12.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS12.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS12.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Hudson and Manning [<a href="#bib.bib45" title="" class="ltx_ref">45</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS13" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.13 </span>xGQA: Cross-lingual VQA</h3>

<div id="A2.SS13.p1" class="ltx_para">
<table id="A2.SS13.p1.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS13.p1.1.1" class="ltx_tr">
<td id="A2.SS13.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="4">
<span id="A2.SS13.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS13.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/xgqa.jpg" id="A2.SS13.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS13.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS13.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS13.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS13.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS13.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS13.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS13.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en Gibt es irgendwelche Schubladen oder Tische?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS13.p1.1.2.1" class="ltx_tr">
<td id="A2.SS13.p1.1.2.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS13.p1.1.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS13.p1.1.2.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS13.p1.1.2.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS13.p1.1.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS13.p1.1.2.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS13.p1.1.2.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"no"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS13.p1.1.3.2" class="ltx_tr">
<td id="A2.SS13.p1.1.3.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS13.p1.1.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS13.p1.1.3.2.1.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS13.p1.1.3.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS13.p1.1.3.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS13.p1.1.3.2.2.1.1" class="ltx_p" style="width:256.1pt;">Zero-shot evaluation of the model trained for GQA.</span>
</span>
</td>
</tr>
<tr id="A2.SS13.p1.1.4.3" class="ltx_tr">
<td id="A2.SS13.p1.1.4.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS13.p1.1.4.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS13.p1.1.4.3.1.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS13.p1.1.4.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS13.p1.1.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS13.p1.1.4.3.2.1.1" class="ltx_p" style="width:256.1pt;">Exact-match accuracy (test zero-shot splits).</span>
</span>
</td>
</tr>
<tr id="A2.SS13.p1.1.5.4" class="ltx_tr">
<td id="A2.SS13.p1.1.5.4.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS13.p1.1.5.4.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS13.p1.1.5.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS13.p1.1.5.4.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS13.p1.1.5.4.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS13.p1.1.5.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS13.p1.1.5.4.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Pfeiffer et al. [<a href="#bib.bib87" title="" class="ltx_ref">87</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS14" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.14 </span>InfoVQA: VQA on infographics</h3>

<div id="A2.SS14.p1" class="ltx_para">
<table id="A2.SS14.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS14.p1.1.1" class="ltx_tr">
<td id="A2.SS14.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS14.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS14.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/infovqa.jpg" id="A2.SS14.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS14.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS14.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS14.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS14.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS14.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS14.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS14.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en WHich team has most number of FB likes in Tennessee\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS14.p1.2.3.1" class="ltx_tr">
<td id="A2.SS14.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS14.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS14.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS14.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS14.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS14.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS14.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"titans"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS14.p1.2.2" class="ltx_tr">
<td id="A2.SS14.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS14.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS14.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS14.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS14.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS14.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS14.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="26747" display="inline"><semantics id="A2.SS14.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS14.p1.2.2.1.1.1.m1.1.1" xref="A2.SS14.p1.2.2.1.1.1.m1.1.1.cmml">26747</mn><annotation-xml encoding="MathML-Content" id="A2.SS14.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS14.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS14.p1.2.2.1.1.1.m1.1.1">26747</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS14.p1.2.2.1.1.1.m1.1c">26747</annotation></semantics></math> examples (train + val).</span>
</span>
</td>
</tr>
<tr id="A2.SS14.p1.2.4.2" class="ltx_tr">
<td id="A2.SS14.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS14.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS14.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS14.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS14.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS14.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS14.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">ANLS (test server).</span>
</span>
</td>
</tr>
<tr id="A2.SS14.p1.2.5.3" class="ltx_tr">
<td id="A2.SS14.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS14.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS14.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS14.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS14.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS14.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS14.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Mathew et al. [<a href="#bib.bib77" title="" class="ltx_ref">77</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS15" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.15 </span>MSRVTT-CAP: Open-domain short video captioning</h3>

<div id="A2.SS15.p1" class="ltx_para">
<table id="A2.SS15.p1.3" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS15.p1.2.2" class="ltx_tr">
<td id="A2.SS15.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS15.p1.2.2.2.2" class="ltx_inline-block ltx_align_top">
<span id="A2.SS15.p1.2.2.2.2.2" class="ltx_p">
<span id="A2.SS15.p1.2.2.2.2.2.2" class="ltx_tabular ltx_align_middle">
<span id="A2.SS15.p1.1.1.1.1.1.1.1" class="ltx_tr">
<span id="A2.SS15.p1.1.1.1.1.1.1.1.1" class="ltx_td ltx_align_center"><span id="A2.SS15.p1.1.1.1.1.1.1.1.1.1" class="ltx_text"><svg version="1.1" width="67" height="67" overflow="visible"><g transform="translate(0,67) scale(1,-1)"><rect fill="none" height="48.4pt" stroke="#000000" stroke-width="0.4" width="48.4pt" x="0" y="0"></rect><g class="makebox" transform="translate(0,0)"><g transform="translate(0,67) scale(1, -1)"><foreignObject width="67" height="67" overflow="visible"><img src="/html/2407.07726/assets/figures/tasks/msrvtt_cap_0.jpg" id="A2.SS15.p1.1.1.1.1.1.1.1.1.1.pic1.1.g1" class="ltx_graphics ltx_img_square" width="67" height="67" alt="[Uncaptioned image]"></foreignObject></g></g></g></svg></span></span></span>
<span id="A2.SS15.p1.2.2.2.2.2.2.2" class="ltx_tr">
<span id="A2.SS15.p1.2.2.2.2.2.2.2.1" class="ltx_td ltx_align_center"><span id="A2.SS15.p1.2.2.2.2.2.2.2.1.1" class="ltx_text"><svg version="1.1" width="67" height="67" overflow="visible"><g transform="translate(0,67) scale(1,-1)"><rect fill="none" height="48.4pt" stroke="#000000" stroke-width="0.4" width="48.4pt" x="0" y="0"></rect><g class="makebox" transform="translate(0,0)"><g transform="translate(0,67) scale(1, -1)"><foreignObject width="67" height="67" overflow="visible"><img src="/html/2407.07726/assets/figures/tasks/msrvtt_cap_11.jpg" id="A2.SS15.p1.2.2.2.2.2.2.2.1.1.pic1.1.g1" class="ltx_graphics ltx_img_square" width="67" height="67" alt="[Uncaptioned image]"></foreignObject></g></g></g></svg></span></span></span>
</span></span>
</span>
</td>
<td id="A2.SS15.p1.2.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS15.p1.2.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS15.p1.2.2.3.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS15.p1.2.2.4" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS15.p1.2.2.4.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS15.p1.2.2.4.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS15.p1.2.2.4.1.1.1" class="ltx_text ltx_font_typewriter">"caption en\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS15.p1.3.4.1" class="ltx_tr">
<td id="A2.SS15.p1.3.4.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS15.p1.3.4.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS15.p1.3.4.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS15.p1.3.4.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS15.p1.3.4.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS15.p1.3.4.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS15.p1.3.4.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"a woman is presenting a baby stroller"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS15.p1.3.3" class="ltx_tr">
<td id="A2.SS15.p1.3.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS15.p1.3.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS15.p1.3.3.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS15.p1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS15.p1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS15.p1.3.3.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS15.p1.3.3.1.1.1.m1.1" class="ltx_Math" alttext="4965" display="inline"><semantics id="A2.SS15.p1.3.3.1.1.1.m1.1a"><mn id="A2.SS15.p1.3.3.1.1.1.m1.1.1" xref="A2.SS15.p1.3.3.1.1.1.m1.1.1.cmml">4965</mn><annotation-xml encoding="MathML-Content" id="A2.SS15.p1.3.3.1.1.1.m1.1b"><cn type="integer" id="A2.SS15.p1.3.3.1.1.1.m1.1.1.cmml" xref="A2.SS15.p1.3.3.1.1.1.m1.1.1">4965</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS15.p1.3.3.1.1.1.m1.1c">4965</annotation></semantics></math> examples (train+valid).</span>
</span>
</td>
</tr>
<tr id="A2.SS15.p1.3.5.2" class="ltx_tr">
<td id="A2.SS15.p1.3.5.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS15.p1.3.5.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS15.p1.3.5.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS15.p1.3.5.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS15.p1.3.5.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS15.p1.3.5.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS15.p1.3.5.2.3.1.1" class="ltx_p" style="width:256.1pt;">CIDEr (test split).</span>
</span>
</td>
</tr>
<tr id="A2.SS15.p1.3.6.3" class="ltx_tr">
<td id="A2.SS15.p1.3.6.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS15.p1.3.6.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS15.p1.3.6.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS15.p1.3.6.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS15.p1.3.6.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS15.p1.3.6.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS15.p1.3.6.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Xu et al. [<a href="#bib.bib120" title="" class="ltx_ref">120</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS16" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.16 </span>MSRVTT-QA: Open-domain sort video question answering</h3>

<div id="A2.SS16.p1" class="ltx_para">
<table id="A2.SS16.p1.3" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS16.p1.2.2" class="ltx_tr">
<td id="A2.SS16.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS16.p1.2.2.2.2" class="ltx_inline-block ltx_align_top">
<span id="A2.SS16.p1.2.2.2.2.2" class="ltx_p">
<span id="A2.SS16.p1.2.2.2.2.2.2" class="ltx_tabular ltx_align_middle">
<span id="A2.SS16.p1.1.1.1.1.1.1.1" class="ltx_tr">
<span id="A2.SS16.p1.1.1.1.1.1.1.1.1" class="ltx_td ltx_align_center"><span id="A2.SS16.p1.1.1.1.1.1.1.1.1.1" class="ltx_text"><svg version="1.1" width="67" height="67" overflow="visible"><g transform="translate(0,67) scale(1,-1)"><rect fill="none" height="48.4pt" stroke="#000000" stroke-width="0.4" width="48.4pt" x="0" y="0"></rect><g class="makebox" transform="translate(0,0)"><g transform="translate(0,67) scale(1, -1)"><foreignObject width="67" height="67" overflow="visible"><img src="/html/2407.07726/assets/figures/tasks/msrvtt_qa_0.jpg" id="A2.SS16.p1.1.1.1.1.1.1.1.1.1.pic1.1.g1" class="ltx_graphics ltx_img_square" width="67" height="67" alt="[Uncaptioned image]"></foreignObject></g></g></g></svg></span></span></span>
<span id="A2.SS16.p1.2.2.2.2.2.2.2" class="ltx_tr">
<span id="A2.SS16.p1.2.2.2.2.2.2.2.1" class="ltx_td ltx_align_center"><span id="A2.SS16.p1.2.2.2.2.2.2.2.1.1" class="ltx_text"><svg version="1.1" width="67" height="67" overflow="visible"><g transform="translate(0,67) scale(1,-1)"><rect fill="none" height="48.4pt" stroke="#000000" stroke-width="0.4" width="48.4pt" x="0" y="0"></rect><g class="makebox" transform="translate(0,0)"><g transform="translate(0,67) scale(1, -1)"><foreignObject width="67" height="67" overflow="visible"><img src="/html/2407.07726/assets/figures/tasks/msrvtt_qa_15.jpg" id="A2.SS16.p1.2.2.2.2.2.2.2.1.1.pic1.1.g1" class="ltx_graphics ltx_img_square" width="67" height="67" alt="[Uncaptioned image]"></foreignObject></g></g></g></svg></span></span></span>
</span></span>
</span>
</td>
<td id="A2.SS16.p1.2.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS16.p1.2.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS16.p1.2.2.3.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS16.p1.2.2.4" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS16.p1.2.2.4.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS16.p1.2.2.4.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS16.p1.2.2.4.1.1.1" class="ltx_text ltx_font_typewriter">"answer en what does a person play?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS16.p1.3.4.1" class="ltx_tr">
<td id="A2.SS16.p1.3.4.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS16.p1.3.4.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS16.p1.3.4.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS16.p1.3.4.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS16.p1.3.4.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS16.p1.3.4.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS16.p1.3.4.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"keyboard"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS16.p1.3.3" class="ltx_tr">
<td id="A2.SS16.p1.3.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS16.p1.3.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS16.p1.3.3.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS16.p1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS16.p1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS16.p1.3.3.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS16.p1.3.3.1.1.1.m1.1" class="ltx_Math" alttext="121646" display="inline"><semantics id="A2.SS16.p1.3.3.1.1.1.m1.1a"><mn id="A2.SS16.p1.3.3.1.1.1.m1.1.1" xref="A2.SS16.p1.3.3.1.1.1.m1.1.1.cmml">121646</mn><annotation-xml encoding="MathML-Content" id="A2.SS16.p1.3.3.1.1.1.m1.1b"><cn type="integer" id="A2.SS16.p1.3.3.1.1.1.m1.1.1.cmml" xref="A2.SS16.p1.3.3.1.1.1.m1.1.1">121646</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS16.p1.3.3.1.1.1.m1.1c">121646</annotation></semantics></math> examples (train+valid splits).</span>
</span>
</td>
</tr>
<tr id="A2.SS16.p1.3.5.2" class="ltx_tr">
<td id="A2.SS16.p1.3.5.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS16.p1.3.5.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS16.p1.3.5.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS16.p1.3.5.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS16.p1.3.5.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS16.p1.3.5.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS16.p1.3.5.2.3.1.1" class="ltx_p" style="width:256.1pt;">Exact match accuracy (test split).</span>
</span>
</td>
</tr>
<tr id="A2.SS16.p1.3.6.3" class="ltx_tr">
<td id="A2.SS16.p1.3.6.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS16.p1.3.6.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS16.p1.3.6.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS16.p1.3.6.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS16.p1.3.6.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS16.p1.3.6.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS16.p1.3.6.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Xu et al. [<a href="#bib.bib119" title="" class="ltx_ref">119</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS17" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.17 </span>MSVD-QA: Answering questions about short video segments of events</h3>

<div id="A2.SS17.p1" class="ltx_para">
<table id="A2.SS17.p1.3" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS17.p1.2.2" class="ltx_tr">
<td id="A2.SS17.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS17.p1.2.2.2.2" class="ltx_inline-block ltx_align_top">
<span id="A2.SS17.p1.2.2.2.2.2" class="ltx_p">
<span id="A2.SS17.p1.2.2.2.2.2.2" class="ltx_tabular ltx_align_middle">
<span id="A2.SS17.p1.1.1.1.1.1.1.1" class="ltx_tr">
<span id="A2.SS17.p1.1.1.1.1.1.1.1.1" class="ltx_td ltx_align_center"><span id="A2.SS17.p1.1.1.1.1.1.1.1.1.1" class="ltx_text"><svg version="1.1" width="67" height="67" overflow="visible"><g transform="translate(0,67) scale(1,-1)"><rect fill="none" height="48.4pt" stroke="#000000" stroke-width="0.4" width="48.4pt" x="0" y="0"></rect><g class="makebox" transform="translate(0,0)"><g transform="translate(0,67) scale(1, -1)"><foreignObject width="67" height="67" overflow="visible"><img src="/html/2407.07726/assets/figures/tasks/msvd_qa_0.jpg" id="A2.SS17.p1.1.1.1.1.1.1.1.1.1.pic1.1.g1" class="ltx_graphics ltx_img_square" width="67" height="67" alt="[Uncaptioned image]"></foreignObject></g></g></g></svg></span></span></span>
<span id="A2.SS17.p1.2.2.2.2.2.2.2" class="ltx_tr">
<span id="A2.SS17.p1.2.2.2.2.2.2.2.1" class="ltx_td ltx_align_center"><span id="A2.SS17.p1.2.2.2.2.2.2.2.1.1" class="ltx_text"><svg version="1.1" width="67" height="67" overflow="visible"><g transform="translate(0,67) scale(1,-1)"><rect fill="none" height="48.4pt" stroke="#000000" stroke-width="0.4" width="48.4pt" x="0" y="0"></rect><g class="makebox" transform="translate(0,0)"><g transform="translate(0,67) scale(1, -1)"><foreignObject width="67" height="67" overflow="visible"><img src="/html/2407.07726/assets/figures/tasks/msvd_qa_7.jpg" id="A2.SS17.p1.2.2.2.2.2.2.2.1.1.pic1.1.g1" class="ltx_graphics ltx_img_square" width="67" height="67" alt="[Uncaptioned image]"></foreignObject></g></g></g></svg></span></span></span>
</span></span>
</span>
</td>
<td id="A2.SS17.p1.2.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS17.p1.2.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS17.p1.2.2.3.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS17.p1.2.2.4" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS17.p1.2.2.4.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS17.p1.2.2.4.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS17.p1.2.2.4.1.1.1" class="ltx_text ltx_font_typewriter">"answer en what is a woman wrapping?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS17.p1.3.4.1" class="ltx_tr">
<td id="A2.SS17.p1.3.4.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS17.p1.3.4.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS17.p1.3.4.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS17.p1.3.4.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS17.p1.3.4.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS17.p1.3.4.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS17.p1.3.4.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"food"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS17.p1.3.3" class="ltx_tr">
<td id="A2.SS17.p1.3.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS17.p1.3.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS17.p1.3.3.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS17.p1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS17.p1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS17.p1.3.3.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS17.p1.3.3.1.1.1.m1.1" class="ltx_Math" alttext="29749" display="inline"><semantics id="A2.SS17.p1.3.3.1.1.1.m1.1a"><mn id="A2.SS17.p1.3.3.1.1.1.m1.1.1" xref="A2.SS17.p1.3.3.1.1.1.m1.1.1.cmml">29749</mn><annotation-xml encoding="MathML-Content" id="A2.SS17.p1.3.3.1.1.1.m1.1b"><cn type="integer" id="A2.SS17.p1.3.3.1.1.1.m1.1.1.cmml" xref="A2.SS17.p1.3.3.1.1.1.m1.1.1">29749</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS17.p1.3.3.1.1.1.m1.1c">29749</annotation></semantics></math> examples (train+valid splits).</span>
</span>
</td>
</tr>
<tr id="A2.SS17.p1.3.5.2" class="ltx_tr">
<td id="A2.SS17.p1.3.5.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS17.p1.3.5.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS17.p1.3.5.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS17.p1.3.5.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS17.p1.3.5.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS17.p1.3.5.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS17.p1.3.5.2.3.1.1" class="ltx_p" style="width:256.1pt;">Exact match accuracy (test split).</span>
</span>
</td>
</tr>
<tr id="A2.SS17.p1.3.6.3" class="ltx_tr">
<td id="A2.SS17.p1.3.6.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS17.p1.3.6.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS17.p1.3.6.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS17.p1.3.6.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS17.p1.3.6.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS17.p1.3.6.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS17.p1.3.6.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Xu et al. [<a href="#bib.bib119" title="" class="ltx_ref">119</a>]</cite> based on <cite class="ltx_cite ltx_citemacro_citet">Chen and Dolan [<a href="#bib.bib19" title="" class="ltx_ref">19</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS18" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.18 </span>NLVR2: Reasoning about natural language grounded in photographs</h3>

<div id="A2.SS18.p1" class="ltx_para">
<table id="A2.SS18.p1.3" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS18.p1.2.2" class="ltx_tr">
<td id="A2.SS18.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS18.p1.2.2.2.2" class="ltx_inline-block ltx_align_top">
<span id="A2.SS18.p1.2.2.2.2.2" class="ltx_p">
<span id="A2.SS18.p1.2.2.2.2.2.2" class="ltx_tabular ltx_align_middle">
<span id="A2.SS18.p1.1.1.1.1.1.1.1" class="ltx_tr">
<span id="A2.SS18.p1.1.1.1.1.1.1.1.1" class="ltx_td ltx_align_center"><span id="A2.SS18.p1.1.1.1.1.1.1.1.1.1" class="ltx_text"><svg version="1.1" width="67" height="67" overflow="visible"><g transform="translate(0,67) scale(1,-1)"><rect fill="none" height="48.4pt" stroke="#000000" stroke-width="0.4" width="48.4pt" x="0" y="0"></rect><g class="makebox" transform="translate(0,0)"><g transform="translate(0,67) scale(1, -1)"><foreignObject width="67" height="67" overflow="visible"><img src="/html/2407.07726/assets/figures/tasks/nlvr2_0.jpg" id="A2.SS18.p1.1.1.1.1.1.1.1.1.1.pic1.1.g1" class="ltx_graphics ltx_img_square" width="67" height="67" alt="[Uncaptioned image]"></foreignObject></g></g></g></svg></span></span></span>
<span id="A2.SS18.p1.2.2.2.2.2.2.2" class="ltx_tr">
<span id="A2.SS18.p1.2.2.2.2.2.2.2.1" class="ltx_td ltx_align_center"><span id="A2.SS18.p1.2.2.2.2.2.2.2.1.1" class="ltx_text"><svg version="1.1" width="67" height="67" overflow="visible"><g transform="translate(0,67) scale(1,-1)"><rect fill="none" height="48.4pt" stroke="#000000" stroke-width="0.4" width="48.4pt" x="0" y="0"></rect><g class="makebox" transform="translate(0,0)"><g transform="translate(0,67) scale(1, -1)"><foreignObject width="67" height="67" overflow="visible"><img src="/html/2407.07726/assets/figures/tasks/nlvr2_1.jpg" id="A2.SS18.p1.2.2.2.2.2.2.2.1.1.pic1.1.g1" class="ltx_graphics ltx_img_square" width="67" height="67" alt="[Uncaptioned image]"></foreignObject></g></g></g></svg></span></span></span>
</span></span>
</span>
</td>
<td id="A2.SS18.p1.2.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS18.p1.2.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS18.p1.2.2.3.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS18.p1.2.2.4" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS18.p1.2.2.4.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS18.p1.2.2.4.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS18.p1.2.2.4.1.1.1" class="ltx_text ltx_font_typewriter">"answer en Each image shows one opened laptop angled so the screen faces rightward.\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS18.p1.3.4.1" class="ltx_tr">
<td id="A2.SS18.p1.3.4.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS18.p1.3.4.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS18.p1.3.4.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS18.p1.3.4.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS18.p1.3.4.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS18.p1.3.4.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS18.p1.3.4.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"False"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS18.p1.3.3" class="ltx_tr">
<td id="A2.SS18.p1.3.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS18.p1.3.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS18.p1.3.3.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS18.p1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS18.p1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS18.p1.3.3.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS18.p1.3.3.1.1.1.m1.1" class="ltx_Math" alttext="93355" display="inline"><semantics id="A2.SS18.p1.3.3.1.1.1.m1.1a"><mn id="A2.SS18.p1.3.3.1.1.1.m1.1.1" xref="A2.SS18.p1.3.3.1.1.1.m1.1.1.cmml">93355</mn><annotation-xml encoding="MathML-Content" id="A2.SS18.p1.3.3.1.1.1.m1.1b"><cn type="integer" id="A2.SS18.p1.3.3.1.1.1.m1.1.1.cmml" xref="A2.SS18.p1.3.3.1.1.1.m1.1.1">93355</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS18.p1.3.3.1.1.1.m1.1c">93355</annotation></semantics></math> examples (train+dev splits).</span>
</span>
</td>
</tr>
<tr id="A2.SS18.p1.3.5.2" class="ltx_tr">
<td id="A2.SS18.p1.3.5.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS18.p1.3.5.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS18.p1.3.5.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS18.p1.3.5.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS18.p1.3.5.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS18.p1.3.5.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS18.p1.3.5.2.3.1.1" class="ltx_p" style="width:256.1pt;">Accuracy (test split).</span>
</span>
</td>
</tr>
<tr id="A2.SS18.p1.3.6.3" class="ltx_tr">
<td id="A2.SS18.p1.3.6.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS18.p1.3.6.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS18.p1.3.6.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS18.p1.3.6.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS18.p1.3.6.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS18.p1.3.6.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS18.p1.3.6.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Suhr et al. [<a href="#bib.bib98" title="" class="ltx_ref">98</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS19" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.19 </span>OCR-VQA: VQA by reading text in images</h3>

<div id="A2.SS19.p1" class="ltx_para">
<table id="A2.SS19.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS19.p1.1.1" class="ltx_tr">
<td id="A2.SS19.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS19.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS19.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/ocrvqa.jpg" id="A2.SS19.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS19.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS19.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS19.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS19.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS19.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS19.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS19.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en Is this a comedy book?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS19.p1.2.3.1" class="ltx_tr">
<td id="A2.SS19.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS19.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS19.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS19.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS19.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS19.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS19.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"Yes"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS19.p1.2.2" class="ltx_tr">
<td id="A2.SS19.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS19.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS19.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS19.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS19.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS19.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS19.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="901717" display="inline"><semantics id="A2.SS19.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS19.p1.2.2.1.1.1.m1.1.1" xref="A2.SS19.p1.2.2.1.1.1.m1.1.1.cmml">901717</mn><annotation-xml encoding="MathML-Content" id="A2.SS19.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS19.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS19.p1.2.2.1.1.1.m1.1.1">901717</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS19.p1.2.2.1.1.1.m1.1c">901717</annotation></semantics></math> examples (train+val splits).</span>
</span>
</td>
</tr>
<tr id="A2.SS19.p1.2.4.2" class="ltx_tr">
<td id="A2.SS19.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS19.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS19.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS19.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS19.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS19.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS19.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">Exact match accuracy (test split).</span>
</span>
</td>
</tr>
<tr id="A2.SS19.p1.2.5.3" class="ltx_tr">
<td id="A2.SS19.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS19.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS19.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS19.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS19.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS19.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS19.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Mishra et al. [<a href="#bib.bib81" title="" class="ltx_ref">81</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS20" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.20 </span>OKVQA: Outside knowledge VQA</h3>

<div id="A2.SS20.p1" class="ltx_para">
<table id="A2.SS20.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS20.p1.1.1" class="ltx_tr">
<td id="A2.SS20.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS20.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS20.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/okvqa.jpg" id="A2.SS20.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS20.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS20.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS20.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS20.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS20.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS20.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS20.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en What type of race is this?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS20.p1.2.3.1" class="ltx_tr">
<td id="A2.SS20.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS20.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS20.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS20.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS20.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS20.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS20.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"truck"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS20.p1.2.2" class="ltx_tr">
<td id="A2.SS20.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS20.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS20.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS20.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS20.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS20.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS20.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="9009" display="inline"><semantics id="A2.SS20.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS20.p1.2.2.1.1.1.m1.1.1" xref="A2.SS20.p1.2.2.1.1.1.m1.1.1.cmml">9009</mn><annotation-xml encoding="MathML-Content" id="A2.SS20.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS20.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS20.p1.2.2.1.1.1.m1.1.1">9009</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS20.p1.2.2.1.1.1.m1.1c">9009</annotation></semantics></math> examples (train split).</span>
</span>
</td>
</tr>
<tr id="A2.SS20.p1.2.4.2" class="ltx_tr">
<td id="A2.SS20.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS20.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS20.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS20.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS20.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS20.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS20.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">Exact match accuracy (val split).</span>
</span>
</td>
</tr>
<tr id="A2.SS20.p1.2.5.3" class="ltx_tr">
<td id="A2.SS20.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS20.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS20.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS20.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS20.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS20.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS20.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Marino et al. [<a href="#bib.bib74" title="" class="ltx_ref">74</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS21" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.21 </span>RefCOCO_seg: Referring expression segmentation</h3>

<div id="A2.SS21.p1" class="ltx_para">
<table id="A2.SS21.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS21.p1.1.1" class="ltx_tr">
<td id="A2.SS21.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS21.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS21.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/refcoco_seg.jpg" id="A2.SS21.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS21.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS21.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS21.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS21.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS21.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS21.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS21.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"the giraffe on the right standing tall\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS21.p1.2.3.1" class="ltx_tr">
<td id="A2.SS21.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS21.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS21.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS21.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS21.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS21.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS21.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"&lt;loc0347&gt;&lt;loc0553&gt;&lt;loc0788&gt;&lt;loc0749&gt;&lt;seg093&gt;&lt;seg106&gt;&lt;seg114&gt;&lt;seg078&gt;&lt;seg064&gt;&lt;seg012&gt;&lt;seg031&gt;&lt;seg055&gt;&lt;seg050&gt;&lt;seg012&gt;&lt;seg083&gt;&lt;seg118&gt;&lt;seg084&gt;&lt;seg078&gt;&lt;seg127&gt;&lt;seg121&gt;"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS21.p1.2.2" class="ltx_tr">
<td id="A2.SS21.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS21.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS21.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS21.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS21.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS21.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS21.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="24407" display="inline"><semantics id="A2.SS21.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS21.p1.2.2.1.1.1.m1.1.1" xref="A2.SS21.p1.2.2.1.1.1.m1.1.1.cmml">24407</mn><annotation-xml encoding="MathML-Content" id="A2.SS21.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS21.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS21.p1.2.2.1.1.1.m1.1.1">24407</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS21.p1.2.2.1.1.1.m1.1c">24407</annotation></semantics></math> examples: The combined training sets of RefCOCO, RefCOCO+, and RefCOCOg, <em id="A2.SS21.p1.2.2.1.1.1.1" class="ltx_emph ltx_font_italic">but with all val and test images removed</em>, see LocCa <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib111" title="" class="ltx_ref">111</a>]</cite> for details.</span>
</span>
</td>
</tr>
<tr id="A2.SS21.p1.2.4.2" class="ltx_tr">
<td id="A2.SS21.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS21.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS21.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS21.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS21.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS21.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS21.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">mean intersection over union (mIoU) on all test splits.</span>
</span>
</td>
</tr>
<tr id="A2.SS21.p1.2.5.3" class="ltx_tr">
<td id="A2.SS21.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS21.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS21.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS21.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS21.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS21.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS21.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;">RefCOCO and RefCOCO+: <cite class="ltx_cite ltx_citemacro_citet">Yu et al. [<a href="#bib.bib124" title="" class="ltx_ref">124</a>], Kazemzadeh et al. [<a href="#bib.bib51" title="" class="ltx_ref">51</a>]</cite> and RefCOCOg:  <cite class="ltx_cite ltx_citemacro_citet">Mao et al. [<a href="#bib.bib73" title="" class="ltx_ref">73</a>]</cite>.</span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS22" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.22 </span>RSVQA-hr: VQA for remote sensing (high res)</h3>

<div id="A2.SS22.p1" class="ltx_para">
<table id="A2.SS22.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS22.p1.1.1" class="ltx_tr">
<td id="A2.SS22.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS22.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS22.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/rsvqa_hr.jpg" id="A2.SS22.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS22.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS22.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS22.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS22.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS22.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS22.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS22.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en Are there more commercial buildings than roads?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS22.p1.2.3.1" class="ltx_tr">
<td id="A2.SS22.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS22.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS22.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS22.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS22.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS22.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS22.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"no"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS22.p1.2.2" class="ltx_tr">
<td id="A2.SS22.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS22.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS22.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS22.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS22.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS22.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS22.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="432239" display="inline"><semantics id="A2.SS22.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS22.p1.2.2.1.1.1.m1.1.1" xref="A2.SS22.p1.2.2.1.1.1.m1.1.1.cmml">432239</mn><annotation-xml encoding="MathML-Content" id="A2.SS22.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS22.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS22.p1.2.2.1.1.1.m1.1.1">432239</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS22.p1.2.2.1.1.1.m1.1c">432239</annotation></semantics></math> examples (non-numeric train+val splits).</span>
</span>
</td>
</tr>
<tr id="A2.SS22.p1.2.4.2" class="ltx_tr">
<td id="A2.SS22.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS22.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS22.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS22.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS22.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS22.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS22.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">Mean accuracy across questions types (reported on test and test2 splits).</span>
</span>
</td>
</tr>
<tr id="A2.SS22.p1.2.5.3" class="ltx_tr">
<td id="A2.SS22.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS22.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS22.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS22.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS22.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS22.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS22.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Lobry et al. [<a href="#bib.bib68" title="" class="ltx_ref">68</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS23" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.23 </span>RSVQA-lr: VQA for remote sensing (low res)</h3>

<div id="A2.SS23.p1" class="ltx_para">
<table id="A2.SS23.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS23.p1.1.1" class="ltx_tr">
<td id="A2.SS23.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS23.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS23.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/rsvqa_lr.jpg" id="A2.SS23.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS23.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS23.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS23.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS23.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS23.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS23.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS23.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en Is the number of buildings equal to the number of water areas in the image?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS23.p1.2.3.1" class="ltx_tr">
<td id="A2.SS23.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS23.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS23.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS23.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS23.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS23.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS23.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"no"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS23.p1.2.2" class="ltx_tr">
<td id="A2.SS23.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS23.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS23.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS23.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS23.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS23.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS23.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="47173" display="inline"><semantics id="A2.SS23.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS23.p1.2.2.1.1.1.m1.1.1" xref="A2.SS23.p1.2.2.1.1.1.m1.1.1.cmml">47173</mn><annotation-xml encoding="MathML-Content" id="A2.SS23.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS23.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS23.p1.2.2.1.1.1.m1.1.1">47173</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS23.p1.2.2.1.1.1.m1.1c">47173</annotation></semantics></math> (non-numeric train+val splits).</span>
</span>
</td>
</tr>
<tr id="A2.SS23.p1.2.4.2" class="ltx_tr">
<td id="A2.SS23.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS23.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS23.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS23.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS23.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS23.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS23.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">Average accuracy across questions types (non-numeric test split).</span>
</span>
</td>
</tr>
<tr id="A2.SS23.p1.2.5.3" class="ltx_tr">
<td id="A2.SS23.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS23.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS23.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS23.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS23.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS23.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS23.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Lobry et al. [<a href="#bib.bib68" title="" class="ltx_ref">68</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS24" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.24 </span>SciCap: Captions for scientific figures</h3>

<div id="A2.SS24.p1" class="ltx_para">
<table id="A2.SS24.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS24.p1.1.1" class="ltx_tr">
<td id="A2.SS24.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS24.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS24.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/scicap.jpg" id="A2.SS24.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS24.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS24.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS24.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS24.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS24.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS24.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS24.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"caption en\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS24.p1.2.3.1" class="ltx_tr">
<td id="A2.SS24.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS24.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS24.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS24.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS24.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS24.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS24.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"end-to-end delay under different traffic load ."</span></span>
</span>
</td>
</tr>
<tr id="A2.SS24.p1.2.2" class="ltx_tr">
<td id="A2.SS24.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS24.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS24.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS24.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS24.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS24.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS24.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="120188" display="inline"><semantics id="A2.SS24.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS24.p1.2.2.1.1.1.m1.1.1" xref="A2.SS24.p1.2.2.1.1.1.m1.1.1.cmml">120188</mn><annotation-xml encoding="MathML-Content" id="A2.SS24.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS24.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS24.p1.2.2.1.1.1.m1.1.1">120188</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS24.p1.2.2.1.1.1.m1.1c">120188</annotation></semantics></math> examples (first sentence, no subfigure train+val splits).</span>
</span>
</td>
</tr>
<tr id="A2.SS24.p1.2.4.2" class="ltx_tr">
<td id="A2.SS24.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS24.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS24.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS24.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS24.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS24.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS24.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">CIDEr (first sentence, no subfigure test split).</span>
</span>
</td>
</tr>
<tr id="A2.SS24.p1.2.5.3" class="ltx_tr">
<td id="A2.SS24.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS24.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS24.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS24.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS24.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS24.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS24.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Hsu et al. [<a href="#bib.bib42" title="" class="ltx_ref">42</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS25" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.25 </span>ScienceQA: Science question answering</h3>

<div id="A2.SS25.p1" class="ltx_para">
<table id="A2.SS25.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS25.p1.1.1" class="ltx_tr">
<td id="A2.SS25.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS25.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS25.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/science_qa.png" id="A2.SS25.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS25.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS25.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS25.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS25.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS25.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS25.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS25.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"Question: Think about the magnetic force between the magnets in each pair. Which of the following statements is true?\nContext: The images below show two pairs of magnets. The magnets in different pairs do not affect each other. All the magnets shown are made of the same material.\nOptions: (A) The magnetic force is weaker in Pair 1., (B) The magnetic force is weaker in Pair 2., (C) The strength of the magnetic force is the same in both pairs.\nAnswer:\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS25.p1.2.3.1" class="ltx_tr">
<td id="A2.SS25.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS25.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS25.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS25.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS25.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS25.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS25.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"The answer is A."</span></span>
</span>
</td>
</tr>
<tr id="A2.SS25.p1.2.2" class="ltx_tr">
<td id="A2.SS25.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS25.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS25.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS25.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS25.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS25.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS25.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="8315" display="inline"><semantics id="A2.SS25.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS25.p1.2.2.1.1.1.m1.1.1" xref="A2.SS25.p1.2.2.1.1.1.m1.1.1.cmml">8315</mn><annotation-xml encoding="MathML-Content" id="A2.SS25.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS25.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS25.p1.2.2.1.1.1.m1.1.1">8315</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS25.p1.2.2.1.1.1.m1.1c">8315</annotation></semantics></math> examples (train+val splits).</span>
</span>
</td>
</tr>
<tr id="A2.SS25.p1.2.4.2" class="ltx_tr">
<td id="A2.SS25.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS25.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS25.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS25.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS25.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS25.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS25.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">Exact-match accuracy (test split).</span>
</span>
</td>
</tr>
<tr id="A2.SS25.p1.2.5.3" class="ltx_tr">
<td id="A2.SS25.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS25.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS25.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS25.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS25.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS25.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS25.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Lu et al. [<a href="#bib.bib71" title="" class="ltx_ref">71</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS26" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.26 </span>Screen2Words: Mobile UI summarization</h3>

<div id="A2.SS26.p1" class="ltx_para">
<table id="A2.SS26.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS26.p1.1.1" class="ltx_tr">
<td id="A2.SS26.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS26.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS26.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/screen2words.jpg" id="A2.SS26.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS26.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS26.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS26.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS26.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS26.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS26.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS26.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"caption en\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS26.p1.2.3.1" class="ltx_tr">
<td id="A2.SS26.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS26.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS26.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS26.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS26.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS26.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS26.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"screen showing general settings page"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS26.p1.2.2" class="ltx_tr">
<td id="A2.SS26.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS26.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS26.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS26.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS26.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS26.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS26.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="18107" display="inline"><semantics id="A2.SS26.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS26.p1.2.2.1.1.1.m1.1.1" xref="A2.SS26.p1.2.2.1.1.1.m1.1.1.cmml">18107</mn><annotation-xml encoding="MathML-Content" id="A2.SS26.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS26.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS26.p1.2.2.1.1.1.m1.1.1">18107</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS26.p1.2.2.1.1.1.m1.1c">18107</annotation></semantics></math> (train+dev splits).</span>
</span>
</td>
</tr>
<tr id="A2.SS26.p1.2.4.2" class="ltx_tr">
<td id="A2.SS26.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS26.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS26.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS26.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS26.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS26.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS26.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">CIDEr (test split).</span>
</span>
</td>
</tr>
<tr id="A2.SS26.p1.2.5.3" class="ltx_tr">
<td id="A2.SS26.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS26.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS26.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS26.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS26.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS26.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS26.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Wang et al. [<a href="#bib.bib112" title="" class="ltx_ref">112</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS27" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.27 </span>ST-VQA: Scene text VQA</h3>

<div id="A2.SS27.p1" class="ltx_para">
<table id="A2.SS27.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS27.p1.1.1" class="ltx_tr">
<td id="A2.SS27.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS27.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS27.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/stvqa.jpg" id="A2.SS27.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS27.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS27.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS27.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS27.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS27.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS27.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS27.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en What company is this?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS27.p1.2.3.1" class="ltx_tr">
<td id="A2.SS27.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS27.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS27.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS27.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS27.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS27.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS27.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"microsoft"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS27.p1.2.2" class="ltx_tr">
<td id="A2.SS27.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS27.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS27.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS27.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS27.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS27.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS27.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="26074" display="inline"><semantics id="A2.SS27.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS27.p1.2.2.1.1.1.m1.1.1" xref="A2.SS27.p1.2.2.1.1.1.m1.1.1.cmml">26074</mn><annotation-xml encoding="MathML-Content" id="A2.SS27.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS27.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS27.p1.2.2.1.1.1.m1.1.1">26074</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS27.p1.2.2.1.1.1.m1.1c">26074</annotation></semantics></math> examples (train+val).</span>
</span>
</td>
</tr>
<tr id="A2.SS27.p1.2.4.2" class="ltx_tr">
<td id="A2.SS27.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS27.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS27.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS27.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS27.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS27.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS27.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">ANLS (test server).</span>
</span>
</td>
</tr>
<tr id="A2.SS27.p1.2.5.3" class="ltx_tr">
<td id="A2.SS27.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS27.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS27.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS27.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS27.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS27.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS27.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Biten et al. [<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS28" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.28 </span>TallyQA: Complex counting questions</h3>

<div id="A2.SS28.p1" class="ltx_para">
<table id="A2.SS28.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS28.p1.1.1" class="ltx_tr">
<td id="A2.SS28.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS28.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS28.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/tallyqa.jpg" id="A2.SS28.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS28.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS28.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS28.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS28.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS28.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS28.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS28.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en How many bowls are in the picture?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS28.p1.2.3.1" class="ltx_tr">
<td id="A2.SS28.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS28.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS28.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS28.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS28.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS28.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS28.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"1"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS28.p1.2.2" class="ltx_tr">
<td id="A2.SS28.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS28.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS28.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS28.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS28.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS28.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS28.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="249318" display="inline"><semantics id="A2.SS28.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS28.p1.2.2.1.1.1.m1.1.1" xref="A2.SS28.p1.2.2.1.1.1.m1.1.1.cmml">249318</mn><annotation-xml encoding="MathML-Content" id="A2.SS28.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS28.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS28.p1.2.2.1.1.1.m1.1.1">249318</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS28.p1.2.2.1.1.1.m1.1c">249318</annotation></semantics></math> examples (train split).</span>
</span>
</td>
</tr>
<tr id="A2.SS28.p1.2.4.2" class="ltx_tr">
<td id="A2.SS28.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS28.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS28.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS28.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS28.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS28.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS28.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">Accuracy (test split) reported on simple and complex splits.</span>
</span>
</td>
</tr>
<tr id="A2.SS28.p1.2.5.3" class="ltx_tr">
<td id="A2.SS28.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS28.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS28.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS28.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS28.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS28.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS28.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Acharya et al. [<a href="#bib.bib1" title="" class="ltx_ref">1</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS29" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.29 </span>CountBenchQA: Evaluate counting in a structured, controlled way</h3>

<div id="A2.SS29.p1" class="ltx_para">
<table id="A2.SS29.p1.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS29.p1.1.1" class="ltx_tr">
<td id="A2.SS29.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS29.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS29.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/countbenchqa.jpg" id="A2.SS29.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS29.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS29.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS29.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS29.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS29.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS29.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS29.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en How many sculptures are there in the image?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS29.p1.1.2.1" class="ltx_tr">
<td id="A2.SS29.p1.1.2.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS29.p1.1.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS29.p1.1.2.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS29.p1.1.2.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS29.p1.1.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS29.p1.1.2.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS29.p1.1.2.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"7"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS29.p1.1.3.2" class="ltx_tr">
<td id="A2.SS29.p1.1.3.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS29.p1.1.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS29.p1.1.3.2.1.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS29.p1.1.3.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS29.p1.1.3.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS29.p1.1.3.2.2.1.1" class="ltx_p" style="width:256.1pt;">Zero-shot evaluation of the model trained for TallyQA.</span>
</span>
</td>
</tr>
<tr id="A2.SS29.p1.1.4.3" class="ltx_tr">
<td id="A2.SS29.p1.1.4.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS29.p1.1.4.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS29.p1.1.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS29.p1.1.4.3.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS29.p1.1.4.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS29.p1.1.4.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS29.p1.1.4.3.3.1.1" class="ltx_p" style="width:256.1pt;">Exact match (full test split).</span>
</span>
</td>
</tr>
<tr id="A2.SS29.p1.1.5.4" class="ltx_tr">
<td id="A2.SS29.p1.1.5.4.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS29.p1.1.5.4.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS29.p1.1.5.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS29.p1.1.5.4.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS29.p1.1.5.4.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS29.p1.1.5.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS29.p1.1.5.4.3.1.1" class="ltx_p" style="width:256.1pt;">Introduced in this report, based on <cite class="ltx_cite ltx_citemacro_citet">Paiss et al. [<a href="#bib.bib85" title="" class="ltx_ref">85</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS30" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.30 </span>TextCaps: Image captioning with reading comprehension</h3>

<div id="A2.SS30.p1" class="ltx_para">
<table id="A2.SS30.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS30.p1.1.1" class="ltx_tr">
<td id="A2.SS30.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS30.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS30.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/textcaps.jpg" id="A2.SS30.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS30.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS30.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS30.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS30.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS30.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS30.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS30.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"caption en\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS30.p1.2.3.1" class="ltx_tr">
<td id="A2.SS30.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS30.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS30.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS30.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS30.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS30.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS30.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"green and red beer can for tsingtao inside a restaurant."</span></span>
</span>
</td>
</tr>
<tr id="A2.SS30.p1.2.2" class="ltx_tr">
<td id="A2.SS30.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS30.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS30.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS30.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS30.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS30.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS30.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="21953" display="inline"><semantics id="A2.SS30.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS30.p1.2.2.1.1.1.m1.1.1" xref="A2.SS30.p1.2.2.1.1.1.m1.1.1.cmml">21953</mn><annotation-xml encoding="MathML-Content" id="A2.SS30.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS30.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS30.p1.2.2.1.1.1.m1.1.1">21953</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS30.p1.2.2.1.1.1.m1.1c">21953</annotation></semantics></math> examples (train split).</span>
</span>
</td>
</tr>
<tr id="A2.SS30.p1.2.4.2" class="ltx_tr">
<td id="A2.SS30.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS30.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS30.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS30.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS30.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS30.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS30.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">CIDEr (test split).</span>
</span>
</td>
</tr>
<tr id="A2.SS30.p1.2.5.3" class="ltx_tr">
<td id="A2.SS30.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS30.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS30.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS30.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS30.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS30.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS30.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Sidorov et al. [<a href="#bib.bib96" title="" class="ltx_ref">96</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS31" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.31 </span>TextVQA: Visual reasoning based on text in images</h3>

<div id="A2.SS31.p1" class="ltx_para">
<table id="A2.SS31.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS31.p1.1.1" class="ltx_tr">
<td id="A2.SS31.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS31.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS31.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/textvqa.jpg" id="A2.SS31.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS31.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS31.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS31.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS31.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS31.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS31.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS31.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en what brand of watch is this?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS31.p1.2.3.1" class="ltx_tr">
<td id="A2.SS31.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS31.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS31.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS31.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS31.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS31.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS31.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"breitling"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS31.p1.2.2" class="ltx_tr">
<td id="A2.SS31.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS31.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS31.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS31.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS31.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS31.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS31.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="39602" display="inline"><semantics id="A2.SS31.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS31.p1.2.2.1.1.1.m1.1.1" xref="A2.SS31.p1.2.2.1.1.1.m1.1.1.cmml">39602</mn><annotation-xml encoding="MathML-Content" id="A2.SS31.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS31.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS31.p1.2.2.1.1.1.m1.1.1">39602</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS31.p1.2.2.1.1.1.m1.1c">39602</annotation></semantics></math> (train+val splits).</span>
</span>
</td>
</tr>
<tr id="A2.SS31.p1.2.4.2" class="ltx_tr">
<td id="A2.SS31.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS31.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS31.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS31.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS31.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS31.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS31.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">test server (test-std).</span>
</span>
</td>
</tr>
<tr id="A2.SS31.p1.2.5.3" class="ltx_tr">
<td id="A2.SS31.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS31.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS31.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS31.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS31.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS31.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS31.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Singh et al. [<a href="#bib.bib97" title="" class="ltx_ref">97</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS32" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.32 </span>VATEX-CAP: Broad video captioning</h3>

<div id="A2.SS32.p1" class="ltx_para">
<table id="A2.SS32.p1.3" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS32.p1.2.2" class="ltx_tr">
<td id="A2.SS32.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS32.p1.2.2.2.2" class="ltx_inline-block ltx_align_top">
<span id="A2.SS32.p1.2.2.2.2.2" class="ltx_p">
<span id="A2.SS32.p1.2.2.2.2.2.2" class="ltx_tabular ltx_align_middle">
<span id="A2.SS32.p1.1.1.1.1.1.1.1" class="ltx_tr">
<span id="A2.SS32.p1.1.1.1.1.1.1.1.1" class="ltx_td ltx_align_center"><span id="A2.SS32.p1.1.1.1.1.1.1.1.1.1" class="ltx_text"><svg version="1.1" width="67" height="67" overflow="visible"><g transform="translate(0,67) scale(1,-1)"><rect fill="none" height="48.4pt" stroke="#000000" stroke-width="0.4" width="48.4pt" x="0" y="0"></rect><g class="makebox" transform="translate(0,0)"><g transform="translate(0,67) scale(1, -1)"><foreignObject width="67" height="67" overflow="visible"><img src="/html/2407.07726/assets/figures/tasks/vatex_cap_0.jpg" id="A2.SS32.p1.1.1.1.1.1.1.1.1.1.pic1.1.g1" class="ltx_graphics ltx_img_square" width="67" height="67" alt="[Uncaptioned image]"></foreignObject></g></g></g></svg></span></span></span>
<span id="A2.SS32.p1.2.2.2.2.2.2.2" class="ltx_tr">
<span id="A2.SS32.p1.2.2.2.2.2.2.2.1" class="ltx_td ltx_align_center"><span id="A2.SS32.p1.2.2.2.2.2.2.2.1.1" class="ltx_text"><svg version="1.1" width="67" height="67" overflow="visible"><g transform="translate(0,67) scale(1,-1)"><rect fill="none" height="48.4pt" stroke="#000000" stroke-width="0.4" width="48.4pt" x="0" y="0"></rect><g class="makebox" transform="translate(0,0)"><g transform="translate(0,67) scale(1, -1)"><foreignObject width="67" height="67" overflow="visible"><img src="/html/2407.07726/assets/figures/tasks/vatex_cap_15.jpg" id="A2.SS32.p1.2.2.2.2.2.2.2.1.1.pic1.1.g1" class="ltx_graphics ltx_img_square" width="67" height="67" alt="[Uncaptioned image]"></foreignObject></g></g></g></svg></span></span></span>
</span></span>
</span>
</td>
<td id="A2.SS32.p1.2.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS32.p1.2.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS32.p1.2.2.3.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS32.p1.2.2.4" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS32.p1.2.2.4.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS32.p1.2.2.4.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS32.p1.2.2.4.1.1.1" class="ltx_text ltx_font_typewriter">"caption en\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS32.p1.3.4.1" class="ltx_tr">
<td id="A2.SS32.p1.3.4.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS32.p1.3.4.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS32.p1.3.4.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS32.p1.3.4.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS32.p1.3.4.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS32.p1.3.4.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS32.p1.3.4.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"A person swims in a pool and touches the wall while others cheer."</span></span>
</span>
</td>
</tr>
<tr id="A2.SS32.p1.3.3" class="ltx_tr">
<td id="A2.SS32.p1.3.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS32.p1.3.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS32.p1.3.3.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS32.p1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS32.p1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS32.p1.3.3.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS32.p1.3.3.1.1.1.m1.1" class="ltx_Math" alttext="24850" display="inline"><semantics id="A2.SS32.p1.3.3.1.1.1.m1.1a"><mn id="A2.SS32.p1.3.3.1.1.1.m1.1.1" xref="A2.SS32.p1.3.3.1.1.1.m1.1.1.cmml">24850</mn><annotation-xml encoding="MathML-Content" id="A2.SS32.p1.3.3.1.1.1.m1.1b"><cn type="integer" id="A2.SS32.p1.3.3.1.1.1.m1.1.1.cmml" xref="A2.SS32.p1.3.3.1.1.1.m1.1.1">24850</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS32.p1.3.3.1.1.1.m1.1c">24850</annotation></semantics></math> examples (train + valid splits).</span>
</span>
</td>
</tr>
<tr id="A2.SS32.p1.3.5.2" class="ltx_tr">
<td id="A2.SS32.p1.3.5.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS32.p1.3.5.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS32.p1.3.5.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS32.p1.3.5.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS32.p1.3.5.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS32.p1.3.5.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS32.p1.3.5.2.3.1.1" class="ltx_p" style="width:256.1pt;">CIDEr (test split).</span>
</span>
</td>
</tr>
<tr id="A2.SS32.p1.3.6.3" class="ltx_tr">
<td id="A2.SS32.p1.3.6.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS32.p1.3.6.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS32.p1.3.6.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS32.p1.3.6.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS32.p1.3.6.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS32.p1.3.6.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS32.p1.3.6.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Wang et al. [<a href="#bib.bib115" title="" class="ltx_ref">115</a>]</cite>.</span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS33" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.33 </span>VizWizVQA: VQA from people who are blind</h3>

<div id="A2.SS33.p1" class="ltx_para">
<table id="A2.SS33.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS33.p1.1.1" class="ltx_tr">
<td id="A2.SS33.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS33.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS33.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/vizwizvqa.jpg" id="A2.SS33.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS33.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS33.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS33.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS33.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS33.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS33.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS33.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en What colors are in the charm of this necklace?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS33.p1.2.3.1" class="ltx_tr">
<td id="A2.SS33.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS33.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS33.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS33.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS33.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS33.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS33.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"silver green blue"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS33.p1.2.2" class="ltx_tr">
<td id="A2.SS33.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS33.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS33.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS33.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS33.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS33.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS33.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="24842" display="inline"><semantics id="A2.SS33.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS33.p1.2.2.1.1.1.m1.1.1" xref="A2.SS33.p1.2.2.1.1.1.m1.1.1.cmml">24842</mn><annotation-xml encoding="MathML-Content" id="A2.SS33.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS33.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS33.p1.2.2.1.1.1.m1.1.1">24842</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS33.p1.2.2.1.1.1.m1.1c">24842</annotation></semantics></math> examples (train+val).</span>
</span>
</td>
</tr>
<tr id="A2.SS33.p1.2.4.2" class="ltx_tr">
<td id="A2.SS33.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS33.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS33.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS33.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS33.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS33.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS33.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">test server (test-std).</span>
</span>
</td>
</tr>
<tr id="A2.SS33.p1.2.5.3" class="ltx_tr">
<td id="A2.SS33.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS33.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS33.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS33.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS33.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS33.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS33.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Gurari et al. [<a href="#bib.bib38" title="" class="ltx_ref">38</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS34" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.34 </span>VQAV2: Visual Question Answering</h3>

<div id="A2.SS34.p1" class="ltx_para">
<table id="A2.SS34.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS34.p1.1.1" class="ltx_tr">
<td id="A2.SS34.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS34.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS34.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/vqav2.jpg" id="A2.SS34.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS34.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS34.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS34.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS34.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS34.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS34.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS34.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en Does this look like a place you'd love to swim?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS34.p1.2.3.1" class="ltx_tr">
<td id="A2.SS34.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS34.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS34.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS34.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS34.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS34.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS34.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"yes"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS34.p1.2.2" class="ltx_tr">
<td id="A2.SS34.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS34.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS34.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS34.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS34.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS34.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS34.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="658111" display="inline"><semantics id="A2.SS34.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS34.p1.2.2.1.1.1.m1.1.1" xref="A2.SS34.p1.2.2.1.1.1.m1.1.1.cmml">658111</mn><annotation-xml encoding="MathML-Content" id="A2.SS34.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS34.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS34.p1.2.2.1.1.1.m1.1.1">658111</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS34.p1.2.2.1.1.1.m1.1c">658111</annotation></semantics></math> examples (train+validation)</span>
</span>
</td>
</tr>
<tr id="A2.SS34.p1.2.4.2" class="ltx_tr">
<td id="A2.SS34.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS34.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS34.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS34.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS34.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS34.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS34.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS34.p1.2.4.2.3.1.1.1" class="ltx_text ltx_font_typewriter">VQAV2</span>: test server (test-std).</span>
</span>
</td>
</tr>
<tr id="A2.SS34.p1.2.5.3" class="ltx_tr">
<td id="A2.SS34.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS34.p1.2.5.3.2" class="ltx_td ltx_align_top"></td>
<td id="A2.SS34.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS34.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS34.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS34.p1.2.5.3.3.1.1.1" class="ltx_text ltx_font_typewriter">VQAV2 (minival)</span>: computed on local split of 10k examples of the validation set.</span>
</span>
</td>
</tr>
<tr id="A2.SS34.p1.2.6.4" class="ltx_tr">
<td id="A2.SS34.p1.2.6.4.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS34.p1.2.6.4.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS34.p1.2.6.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS34.p1.2.6.4.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS34.p1.2.6.4.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS34.p1.2.6.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS34.p1.2.6.4.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Goyal et al. [<a href="#bib.bib37" title="" class="ltx_ref">37</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS35" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.35 </span>MMVP: Hard questions about CLIP-blind image pairs</h3>

<div id="A2.SS35.p1" class="ltx_para">
<table id="A2.SS35.p1.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS35.p1.1.1" class="ltx_tr">
<td id="A2.SS35.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS35.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS35.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/mmvp.jpg" id="A2.SS35.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS35.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS35.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS35.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS35.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS35.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS35.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS35.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en From which angle is this image taken?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS35.p1.1.2.1" class="ltx_tr">
<td id="A2.SS35.p1.1.2.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS35.p1.1.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS35.p1.1.2.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS35.p1.1.2.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS35.p1.1.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS35.p1.1.2.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS35.p1.1.2.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"Front"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS35.p1.1.3.2" class="ltx_tr">
<td id="A2.SS35.p1.1.3.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS35.p1.1.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS35.p1.1.3.2.1.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS35.p1.1.3.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS35.p1.1.3.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS35.p1.1.3.2.2.1.1" class="ltx_p" style="width:256.1pt;">Zero-shot evaluation of the model trained for VQAv2.</span>
</span>
</td>
</tr>
<tr id="A2.SS35.p1.1.4.3" class="ltx_tr">
<td id="A2.SS35.p1.1.4.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS35.p1.1.4.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS35.p1.1.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS35.p1.1.4.3.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS35.p1.1.4.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS35.p1.1.4.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS35.p1.1.4.3.3.1.1" class="ltx_p" style="width:256.1pt;">Paired accuracy.</span>
</span>
</td>
</tr>
<tr id="A2.SS35.p1.1.5.4" class="ltx_tr">
<td id="A2.SS35.p1.1.5.4.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS35.p1.1.5.4.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS35.p1.1.5.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS35.p1.1.5.4.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS35.p1.1.5.4.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS35.p1.1.5.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS35.p1.1.5.4.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Tong et al. [<a href="#bib.bib104" title="" class="ltx_ref">104</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS36" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.36 </span>POPE: Object presence as yes/no VQA (object hallucination)</h3>

<div id="A2.SS36.p1" class="ltx_para">
<table id="A2.SS36.p1.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS36.p1.1.1" class="ltx_tr">
<td id="A2.SS36.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS36.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS36.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/pope.jpg" id="A2.SS36.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS36.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS36.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS36.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS36.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS36.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS36.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS36.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en Is there a car in the image?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS36.p1.1.2.1" class="ltx_tr">
<td id="A2.SS36.p1.1.2.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS36.p1.1.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS36.p1.1.2.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS36.p1.1.2.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS36.p1.1.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS36.p1.1.2.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS36.p1.1.2.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"no"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS36.p1.1.3.2" class="ltx_tr">
<td id="A2.SS36.p1.1.3.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS36.p1.1.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS36.p1.1.3.2.1.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS36.p1.1.3.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS36.p1.1.3.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS36.p1.1.3.2.2.1.1" class="ltx_p" style="width:256.1pt;">Zero-shot evaluation of the model trained for VQAv2.</span>
</span>
</td>
</tr>
<tr id="A2.SS36.p1.1.4.3" class="ltx_tr">
<td id="A2.SS36.p1.1.4.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS36.p1.1.4.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS36.p1.1.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS36.p1.1.4.3.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS36.p1.1.4.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS36.p1.1.4.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS36.p1.1.4.3.3.1.1" class="ltx_p" style="width:256.1pt;">Exact match with “yes” or “no”. The overall score is the average of accuracies on the <span id="A2.SS36.p1.1.4.3.3.1.1.1" class="ltx_text ltx_font_typewriter">random</span>, <span id="A2.SS36.p1.1.4.3.3.1.1.2" class="ltx_text ltx_font_typewriter">popular</span>, and <span id="A2.SS36.p1.1.4.3.3.1.1.3" class="ltx_text ltx_font_typewriter">adversarial</span> splits, which all have the same size.</span>
</span>
</td>
</tr>
<tr id="A2.SS36.p1.1.5.4" class="ltx_tr">
<td id="A2.SS36.p1.1.5.4.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS36.p1.1.5.4.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS36.p1.1.5.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS36.p1.1.5.4.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS36.p1.1.5.4.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS36.p1.1.5.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS36.p1.1.5.4.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Yifan Li and Wen [<a href="#bib.bib123" title="" class="ltx_ref">123</a>]</cite>.</span>
</span>
</td>
</tr>
<tr id="A2.SS36.p1.1.6.5" class="ltx_tr">
<td id="A2.SS36.p1.1.6.5.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS36.p1.1.6.5.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS36.p1.1.6.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS36.p1.1.6.5.2.1.1" class="ltx_p" style="width:42.7pt;">Note:</span>
</span>
</td>
<td id="A2.SS36.p1.1.6.5.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS36.p1.1.6.5.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS36.p1.1.6.5.3.1.1" class="ltx_p" style="width:256.1pt;">Car, not cat.</span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS37" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.37 </span>Objaverse Multiview: view-consistent 3D object annotation</h3>

<div id="A2.SS37.p1" class="ltx_para">
<table id="A2.SS37.p1.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS37.p1.1.1" class="ltx_tr">
<td id="A2.SS37.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS37.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS37.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/objaverse.jpg" id="A2.SS37.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_landscape" width="138" height="96" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS37.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS37.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS37.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS37.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS37.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS37.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS37.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"answer en Describe the object in the image?\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS37.p1.1.2.1" class="ltx_tr">
<td id="A2.SS37.p1.1.2.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS37.p1.1.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS37.p1.1.2.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS37.p1.1.2.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS37.p1.1.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS37.p1.1.2.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS37.p1.1.2.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"banana"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS37.p1.1.3.2" class="ltx_tr">
<td id="A2.SS37.p1.1.3.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS37.p1.1.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS37.p1.1.3.2.1.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS37.p1.1.3.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS37.p1.1.3.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS37.p1.1.3.2.2.1.1" class="ltx_p" style="width:256.1pt;">Zero-shot evaluation of the model trained for VQAv2.</span>
</span>
</td>
</tr>
<tr id="A2.SS37.p1.1.4.3" class="ltx_tr">
<td id="A2.SS37.p1.1.4.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS37.p1.1.4.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS37.p1.1.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS37.p1.1.4.3.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS37.p1.1.4.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS37.p1.1.4.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS37.p1.1.4.3.3.1.1" class="ltx_p" style="width:256.1pt;">Cosine similarity according to Universal Sentence Encoder v4.</span>
</span>
</td>
</tr>
<tr id="A2.SS37.p1.1.5.4" class="ltx_tr">
<td id="A2.SS37.p1.1.5.4.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS37.p1.1.5.4.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS37.p1.1.5.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS37.p1.1.5.4.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS37.p1.1.5.4.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS37.p1.1.5.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS37.p1.1.5.4.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Kabra et al. [<a href="#bib.bib48" title="" class="ltx_ref">48</a>]</cite></span>
</span>
</td>
</tr>
<tr id="A2.SS37.p1.1.6.5" class="ltx_tr">
<td id="A2.SS37.p1.1.6.5.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS37.p1.1.6.5.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS37.p1.1.6.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS37.p1.1.6.5.2.1.1" class="ltx_p" style="width:42.7pt;">Note:</span>
</span>
</td>
<td id="A2.SS37.p1.1.6.5.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS37.p1.1.6.5.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS37.p1.1.6.5.3.1.1" class="ltx_p" style="width:256.1pt;">Each image is seen individually, the final prediction is the score-weighted average. 44k objects, 1100 categories, 8 views rendered per object, 4 prompts per object view.</span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
</section>
<section id="A2.SS38" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.38 </span>WidgetCap: Descriptions for Mobile User Interface Elements</h3>

<div id="A2.SS38.p1" class="ltx_para">
<table id="A2.SS38.p1.2" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A2.SS38.p1.1.1" class="ltx_tr">
<td id="A2.SS38.p1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;" rowspan="3">
<span id="A2.SS38.p1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS38.p1.1.1.1.1.1" class="ltx_p"><img src="/html/2407.07726/assets/figures/tasks/widgetcap.jpg" id="A2.SS38.p1.1.1.1.1.1.g1" class="ltx_graphics ltx_img_square" width="138" height="138" alt="[Uncaptioned image]"></span>
</span>
</td>
<td id="A2.SS38.p1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS38.p1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS38.p1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Prefix:</span>
</span>
</td>
<td id="A2.SS38.p1.1.1.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS38.p1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS38.p1.1.1.3.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS38.p1.1.1.3.1.1.1" class="ltx_text ltx_font_typewriter">"caption en\n"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS38.p1.2.3.1" class="ltx_tr">
<td id="A2.SS38.p1.2.3.1.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS38.p1.2.3.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS38.p1.2.3.1.1.1.1" class="ltx_p" style="width:42.7pt;">Suffix:</span>
</span>
</td>
<td id="A2.SS38.p1.2.3.1.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS38.p1.2.3.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS38.p1.2.3.1.2.1.1" class="ltx_p" style="width:256.1pt;"><span id="A2.SS38.p1.2.3.1.2.1.1.1" class="ltx_text ltx_font_typewriter">"fast forward"</span></span>
</span>
</td>
</tr>
<tr id="A2.SS38.p1.2.2" class="ltx_tr">
<td id="A2.SS38.p1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS38.p1.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS38.p1.2.2.2.1.1" class="ltx_p" style="width:42.7pt;">Train:</span>
</span>
</td>
<td id="A2.SS38.p1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS38.p1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS38.p1.2.2.1.1.1" class="ltx_p" style="width:256.1pt;"><math id="A2.SS38.p1.2.2.1.1.1.m1.1" class="ltx_Math" alttext="44704" display="inline"><semantics id="A2.SS38.p1.2.2.1.1.1.m1.1a"><mn id="A2.SS38.p1.2.2.1.1.1.m1.1.1" xref="A2.SS38.p1.2.2.1.1.1.m1.1.1.cmml">44704</mn><annotation-xml encoding="MathML-Content" id="A2.SS38.p1.2.2.1.1.1.m1.1b"><cn type="integer" id="A2.SS38.p1.2.2.1.1.1.m1.1.1.cmml" xref="A2.SS38.p1.2.2.1.1.1.m1.1.1">44704</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS38.p1.2.2.1.1.1.m1.1c">44704</annotation></semantics></math> examples (train+dev splits). We annotate the widget to be captioned by overlaying a red bounding box in the image input.</span>
</span>
</td>
</tr>
<tr id="A2.SS38.p1.2.4.2" class="ltx_tr">
<td id="A2.SS38.p1.2.4.2.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS38.p1.2.4.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS38.p1.2.4.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS38.p1.2.4.2.2.1.1" class="ltx_p" style="width:42.7pt;">Metric:</span>
</span>
</td>
<td id="A2.SS38.p1.2.4.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS38.p1.2.4.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS38.p1.2.4.2.3.1.1" class="ltx_p" style="width:256.1pt;">CIDEr (test split).</span>
</span>
</td>
</tr>
<tr id="A2.SS38.p1.2.5.3" class="ltx_tr">
<td id="A2.SS38.p1.2.5.3.1" class="ltx_td ltx_align_middle" style="width:113.8pt;"></td>
<td id="A2.SS38.p1.2.5.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS38.p1.2.5.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS38.p1.2.5.3.2.1.1" class="ltx_p" style="width:42.7pt;">Reference:</span>
</span>
</td>
<td id="A2.SS38.p1.2.5.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A2.SS38.p1.2.5.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A2.SS38.p1.2.5.3.3.1.1" class="ltx_p" style="width:256.1pt;"><cite class="ltx_cite ltx_citemacro_citet">Li et al. [<a href="#bib.bib61" title="" class="ltx_ref">61</a>]</cite></span>
</span>
</td>
</tr>
</tbody>
</table>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
<section id="A3" class="ltx_appendix" lang="en">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Image augmentations for RefCOCO</h2>

<figure id="A3.2" class="ltx_figure">
<div id="A3.2.3" class="ltx_block">
<figure id="A3.F13" class="ltx_figure ltx_figure_panel ltx_align_center ltx_align_middle" style="width:208.1pt;"><img src="/html/2407.07726/assets/x13.png" id="A3.1.1.g1" class="ltx_graphics ltx_figure_panel ltx_img_landscape" width="461" height="235" alt="Refer to caption">
<br class="ltx_break ltx_break">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 13: </span>Three different ways to get <math id="A3.F13.2.m1.1" class="ltx_Math" alttext="448\times 448" display="inline"><semantics id="A3.F13.2.m1.1b"><mrow id="A3.F13.2.m1.1.1" xref="A3.F13.2.m1.1.1.cmml"><mn id="A3.F13.2.m1.1.1.2" xref="A3.F13.2.m1.1.1.2.cmml">448</mn><mo lspace="0.222em" rspace="0.222em" id="A3.F13.2.m1.1.1.1" xref="A3.F13.2.m1.1.1.1.cmml">×</mo><mn id="A3.F13.2.m1.1.1.3" xref="A3.F13.2.m1.1.1.3.cmml">448</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.F13.2.m1.1c"><apply id="A3.F13.2.m1.1.1.cmml" xref="A3.F13.2.m1.1.1"><times id="A3.F13.2.m1.1.1.1.cmml" xref="A3.F13.2.m1.1.1.1"></times><cn type="integer" id="A3.F13.2.m1.1.1.2.cmml" xref="A3.F13.2.m1.1.1.2">448</cn><cn type="integer" id="A3.F13.2.m1.1.1.3.cmml" xref="A3.F13.2.m1.1.1.3">448</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.F13.2.m1.1d">448\times 448</annotation></semantics></math> images during training: plain resizing, random aspect-ratio preserving crop, or the latter with an additional random “zoom-out” augmentation.
Two options at inference-time: plain resizing or performing an aspect-ratio preserving resize with padding.
Lines are not training curves, but multiple training runs of 5, 10, 20, and 30 epochs.</figcaption>
</figure>
<figure id="A3.F14" class="ltx_figure ltx_figure_panel ltx_align_center ltx_align_middle" style="width:208.1pt;"><img src="/html/2407.07726/assets/x14.png" id="A3.2.2.g1" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_landscape" width="461" height="240" alt="Refer to caption">
<br class="ltx_break ltx_break">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 14: </span>Wide, outer bars show performance without label-smoothing and dropout (LSDO), thin inner bars with.
10 and 30 denote epochs.
First, LSDO always hurts the 224 px 10 ep setting but always helps the 448 px 30 ep setting.
Second, resize to square overfits in longer training, but adding LSDO remediates this and allows such simple method to outperform more complicated image-augmentation based ones.
</figcaption>
</figure>
</div>
</figure>
<div id="A3.p1" class="ltx_para">
<p id="A3.p1.1" class="ltx_p">The current wisdom regarding structured output tasks such as detection and segmentation is that image augmentations are crucial, and especially keeping the original aspect ratio while performing zoom-like augmentations is key.</p>
</div>
<div id="A3.p2" class="ltx_para">
<p id="A3.p2.1" class="ltx_p">We found this <em id="A3.p2.1.1" class="ltx_emph ltx_font_italic">not</em> to be the case, at least in the setting of pretrained VLMs.
After extensive experiments, we found that simply resizing the image to a square <math id="A3.p2.1.m1.1" class="ltx_Math" alttext="224\times 224" display="inline"><semantics id="A3.p2.1.m1.1a"><mrow id="A3.p2.1.m1.1.1" xref="A3.p2.1.m1.1.1.cmml"><mn id="A3.p2.1.m1.1.1.2" xref="A3.p2.1.m1.1.1.2.cmml">224</mn><mo lspace="0.222em" rspace="0.222em" id="A3.p2.1.m1.1.1.1" xref="A3.p2.1.m1.1.1.1.cmml">×</mo><mn id="A3.p2.1.m1.1.1.3" xref="A3.p2.1.m1.1.1.3.cmml">224</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.p2.1.m1.1b"><apply id="A3.p2.1.m1.1.1.cmml" xref="A3.p2.1.m1.1.1"><times id="A3.p2.1.m1.1.1.1.cmml" xref="A3.p2.1.m1.1.1.1"></times><cn type="integer" id="A3.p2.1.m1.1.1.2.cmml" xref="A3.p2.1.m1.1.1.2">224</cn><cn type="integer" id="A3.p2.1.m1.1.1.3.cmml" xref="A3.p2.1.m1.1.1.3">224</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p2.1.m1.1c">224\times 224</annotation></semantics></math> pixels achieves best (and state-of-the-art) performance.</p>
</div>
<div id="A3.p3" class="ltx_para">
<p id="A3.p3.1" class="ltx_p">Figure <a href="#A3.F13" title="Figure 13 ‣ Appendix C Image augmentations for RefCOCO ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a> shows that one needs to explore both training-time resizing technique and inference-time strategy simultaneously.
It seems natural that, when performing aspect-ratio preserving resize with padding at inference time, the training augemntations need to reflect that to some degree.
However, one can also simply resize test images to a square at inference time, and then resize the output logits back to the original resolution for evaluation.
Doing so works very well in tandem with simple resizing to square at training time and, as one might expect, does not work well with aspect-preserving resizes during training.</p>
</div>
<div id="A3.p4" class="ltx_para">
<p id="A3.p4.1" class="ltx_p">Besides this training-inference mismatch, Figure <a href="#A3.F13" title="Figure 13 ‣ Appendix C Image augmentations for RefCOCO ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a> also shows that the plain resize suffers from overfitting when training for longer than 10 epochs, while cropping and zooming do not, as they also act as image augmentations.
However, overfitting can also be combatted via other, simpler techniques than image processing.</p>
</div>
<div id="A3.p5" class="ltx_para">
<p id="A3.p5.1" class="ltx_p">Figure <a href="#A3.F14" title="Figure 14 ‣ Appendix C Image augmentations for RefCOCO ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">14</span></a> shows that adding label-smoothing and dropout to the training not only completely eliminates the overfitting, but also allows the simple resize strategy to achieve the same or better performance than the more complicated image augmentation ones.
Hence, for our final setting that reaches state-of-the-art results, we stick to simple image resizing with increased training schedule, label-smoothing, and dropout.</p>
</div>
</section>
<section id="A4" class="ltx_appendix" lang="en">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Introducing CountBenchQA</h2>

<div id="A4.p1" class="ltx_para">
<p id="A4.p1.1" class="ltx_p">TallyQA is the only dataset commonly used to evaluate counting, an important skill for VLMs.
However, we found that it has two issues <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib49" title="" class="ltx_ref">49</a>]</cite>:
First, the ground-truth label distribution is highly skewed towards small counts (0, 1, 2).
Second, while the dataset is large, the ground-truth labels themselves are rather noisy, and the validation and test splits have not undergone any further verification or clean-up.</p>
</div>
<div id="A4.p2" class="ltx_para">
<p id="A4.p2.1" class="ltx_p">The recently introduced CountBench <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib85" title="" class="ltx_ref">85</a>]</cite> dataset of 540 images fixes both these shortcomings.
Images and captions are taken from the LAION-400M image-text dataset and each caption mentions the main object class present in the image and specifies its count.
The correspondence of images, counts and captions has been manually verified.</p>
</div>
<div id="A4.p3" class="ltx_para">
<p id="A4.p3.1" class="ltx_p">There are 540 images in total with 60 images per count in [2, 3, … 10].
There is no 0 or 1 count, the smallest count is 2.
The data is not originally available in a VQA format but just as (image, caption, count) triplets.
To construct CountbenchQA, we manually annotated the dataset with questions of the form <span id="A4.p3.1.1" class="ltx_text ltx_font_typewriter">How many {object class} are there in the image?</span>, where we extracted the <span id="A4.p3.1.2" class="ltx_text ltx_font_typewriter">{object class}</span> from the caption manually.
where the object class was overly specific or difficult to understand we replaced them with simpler versions (<em id="A4.p3.1.3" class="ltx_emph ltx_font_italic">e.g</em>.<span id="A4.p3.1.4" class="ltx_text"></span> we changed “glass star christmas tree decorations” to "stars", and “founders of Kappa Sigma” to “people”.)
There are only 3 exceptions deviating from the above template (these were needed to ensure correctness of the answer in ambiguous cases) and these are:</p>
<ul id="A4.I1" class="ltx_itemize">
<li id="A4.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A4.I1.i1.p1" class="ltx_para">
<p id="A4.I1.i1.p1.1" class="ltx_p">How many people are in the foreground of this image?</p>
</div>
</li>
<li id="A4.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A4.I1.i2.p1" class="ltx_para">
<p id="A4.I1.i2.p1.1" class="ltx_p">How many stories does this cottage have?</p>
</div>
</li>
<li id="A4.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A4.I1.i3.p1" class="ltx_para">
<p id="A4.I1.i3.p1.1" class="ltx_p">How many petals does each flower have in this image?</p>
</div>
</li>
</ul>
<p id="A4.p3.2" class="ltx_p">For some counts, several images look extremely similar.
For 9 items, several images are button/icon symbols with slightly different colours/symbols on them, arranged in a regular grid.</p>
</div>
<div id="A4.p4" class="ltx_para">
<p id="A4.p4.1" class="ltx_p">Out of the 540 images, several are not available anymore; the parquet file provided at <a target="_blank" href="https://huggingface.co/datasets/nielsr/countbench" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/datasets/nielsr/countbench</a> contains entries for all 540 examples but only image data for 490 of those examples.
This is why we have two splits, although we computed our metrics using the full split and have not recomputed them since.
Our dataset is available at <a target="_blank" href="https://github.com/google-research/big_vision/blob/main/big_vision/datasets/countbenchqa/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/google-research/big_vision/blob/main/big_vision/datasets/countbenchqa/</a>.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A5" class="ltx_appendix" lang="en">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Issues with published WidgetCaps numbers</h2>

<div id="A5.p1" class="ltx_para">
<p id="A5.p1.2" class="ltx_p">We have found that at least two prior works reporting results on WidgetCaps do so wrongly.
First, previous PaLI numbers (PaLI-X: 153.0, PaLI-3: 159.8) are reported on the validation set while implying they are test-set results.
This makes comparison with test-set numbers such as our 148.4 invalid.
However, a re-run of our transfers in a comparable setting (training on <span id="A5.p1.2.1" class="ltx_text ltx_font_typewriter">train</span>, evaluating on <span id="A5.p1.2.2" class="ltx_text ltx_font_typewriter">dev</span>) achieves a validation set CIDEr of 140.2 at <math id="A5.p1.1.m1.1" class="ltx_Math" alttext="224\," display="inline"><semantics id="A5.p1.1.m1.1a"><mn id="A5.p1.1.m1.1.1" xref="A5.p1.1.m1.1.1.cmml">224</mn><annotation-xml encoding="MathML-Content" id="A5.p1.1.m1.1b"><cn type="integer" id="A5.p1.1.m1.1.1.cmml" xref="A5.p1.1.m1.1.1">224</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.p1.1.m1.1c">224\,</annotation></semantics></math>px and 155.2 at <math id="A5.p1.2.m2.1" class="ltx_Math" alttext="448\," display="inline"><semantics id="A5.p1.2.m2.1a"><mn id="A5.p1.2.m2.1.1" xref="A5.p1.2.m2.1.1.cmml">448</mn><annotation-xml encoding="MathML-Content" id="A5.p1.2.m2.1b"><cn type="integer" id="A5.p1.2.m2.1.1.cmml" xref="A5.p1.2.m2.1.1">448</cn></annotation-xml><annotation encoding="application/x-tex" id="A5.p1.2.m2.1c">448\,</annotation></semantics></math>px resolution, placing it firmly between PaLI-X and PaLI-3 at much smaller model size and resolution.</p>
</div>
<div id="A5.p2" class="ltx_para">
<p id="A5.p2.1" class="ltx_p">Furthermore, ScreenAI <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib9" title="" class="ltx_ref">9</a>]</cite> had a mistake in the CIDEr computation for WidgetCap, using only a single out of the five provided ground-truth captions.
Re-computing their CIDEr score using all five ground-truth captions changes the score from the 167 originally reported in the paper to 156.4, which is close to PaliGemma’s 148.4.</p>
</div>
</section>
<section id="A6" class="ltx_appendix" lang="en">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix F </span>Image annotations work as well as prompts</h2>

<div id="A6.p1" class="ltx_para">
<p id="A6.p1.1" class="ltx_p">The WidgetCaps requires captioning a specific widget in the image.
The widget is given by bounding-box coordinates, which are typically provided in the prompt, including in previous PaLI versions.</p>
</div>
<div id="A6.p2" class="ltx_para">
<p id="A6.p2.1" class="ltx_p">Here, we found that drawing a red box in the image during transfers, and not indicating it in the prompt, performs equally well.
For example, the best validation result when sweeping hyper-parameters in both settings equally, was 135.99 CIDEr for the drawn red box, and 135.28 for the box coordinates as <span id="A6.p2.1.1" class="ltx_text ltx_font_typewriter">&lt;loc&gt;</span> tokens in the prompt.
This difference is well within re-run variation.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A7" class="ltx_appendix" lang="en">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix G </span>More details and results with Objaverse</h2>

<figure id="A7.F15" class="ltx_figure"><img src="/html/2407.07726/assets/x15.png" id="A7.F15.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="332" height="211" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 15: </span>One representative example of PaliGemma’s predictions across the different views.</figcaption>
</figure>
<figure id="A7.T3" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>Results of various Objaverse captioning methods. The mean, standard deviation, and standard error of the mean are computed across the 44 k examples, after the per-example aggregation.</figcaption>
<table id="A7.T3.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A7.T3.1.1.1" class="ltx_tr">
<th id="A7.T3.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_border_tt">
<span id="A7.T3.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T3.1.1.1.1.1.1" class="ltx_p" style="width:227.6pt;"><span id="A7.T3.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Model</span></span>
</span>
</th>
<th id="A7.T3.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">
<span id="A7.T3.1.1.1.2.1" class="ltx_text ltx_font_bold">Score</span> (mean)</th>
<th id="A7.T3.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">(stddev)</th>
<th id="A7.T3.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">(stderr)</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A7.T3.1.2.1" class="ltx_tr">
<td id="A7.T3.1.2.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A7.T3.1.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T3.1.2.1.1.1.1" class="ltx_p" style="width:227.6pt;">CAP3D (Luo et al., 2023)</span>
</span>
</td>
<td id="A7.T3.1.2.1.2" class="ltx_td ltx_align_center ltx_border_tt">36.6</td>
<td id="A7.T3.1.2.1.3" class="ltx_td ltx_align_center ltx_border_tt">-</td>
<td id="A7.T3.1.2.1.4" class="ltx_td ltx_align_center ltx_border_tt">-</td>
</tr>
<tr id="A7.T3.1.3.2" class="ltx_tr">
<td id="A7.T3.1.3.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T3.1.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T3.1.3.2.1.1.1" class="ltx_p" style="width:227.6pt;">PaLI-X (55B, 756x756, mixed VQA transfer)</span>
</span>
</td>
<td id="A7.T3.1.3.2.2" class="ltx_td ltx_align_center">59.0</td>
<td id="A7.T3.1.3.2.3" class="ltx_td ltx_align_center">29.1</td>
<td id="A7.T3.1.3.2.4" class="ltx_td ltx_align_center">0.1</td>
</tr>
<tr id="A7.T3.1.4.3" class="ltx_tr">
<td id="A7.T3.1.4.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T3.1.4.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T3.1.4.3.1.1.1" class="ltx_p" style="width:227.6pt;">PaLI-3 (5B, 448x448, mixed VQA transfer)</span>
</span>
</td>
<td id="A7.T3.1.4.3.2" class="ltx_td ltx_align_center"><span id="A7.T3.1.4.3.2.1" class="ltx_text ltx_font_bold">64.3</span></td>
<td id="A7.T3.1.4.3.3" class="ltx_td ltx_align_center">29.2</td>
<td id="A7.T3.1.4.3.4" class="ltx_td ltx_align_center">0.1</td>
</tr>
<tr id="A7.T3.1.5.4" class="ltx_tr">
<td id="A7.T3.1.5.4.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A7.T3.1.5.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T3.1.5.4.1.1.1" class="ltx_p" style="width:227.6pt;">PaliGemma (3B, 224x224)</span>
</span>
</td>
<td id="A7.T3.1.5.4.2" class="ltx_td ltx_align_center ltx_border_t">58.4</td>
<td id="A7.T3.1.5.4.3" class="ltx_td ltx_align_center ltx_border_t">29.5</td>
<td id="A7.T3.1.5.4.4" class="ltx_td ltx_align_center ltx_border_t">0.1</td>
</tr>
<tr id="A7.T3.1.6.5" class="ltx_tr">
<td id="A7.T3.1.6.5.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T3.1.6.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T3.1.6.5.1.1.1" class="ltx_p" style="width:227.6pt;">PaliGemma (3B, 224x224, VQAv2 transfer)</span>
</span>
</td>
<td id="A7.T3.1.6.5.2" class="ltx_td ltx_align_center">62.7</td>
<td id="A7.T3.1.6.5.3" class="ltx_td ltx_align_center">28.2</td>
<td id="A7.T3.1.6.5.4" class="ltx_td ltx_align_center">0.1</td>
</tr>
<tr id="A7.T3.1.7.6" class="ltx_tr">
<td id="A7.T3.1.7.6.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A7.T3.1.7.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T3.1.7.6.1.1.1" class="ltx_p" style="width:227.6pt;">PaliGemma (3B, 448x448)</span>
</span>
</td>
<td id="A7.T3.1.7.6.2" class="ltx_td ltx_align_center">60.4</td>
<td id="A7.T3.1.7.6.3" class="ltx_td ltx_align_center">28.9</td>
<td id="A7.T3.1.7.6.4" class="ltx_td ltx_align_center">0.1</td>
</tr>
<tr id="A7.T3.1.8.7" class="ltx_tr">
<td id="A7.T3.1.8.7.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A7.T3.1.8.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A7.T3.1.8.7.1.1.1" class="ltx_p" style="width:227.6pt;">PaliGemma (3B, 448x448, VQAv2 transfer)</span>
</span>
</td>
<td id="A7.T3.1.8.7.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="A7.T3.1.8.7.2.1" class="ltx_text ltx_font_bold">62.8</span></td>
<td id="A7.T3.1.8.7.3" class="ltx_td ltx_align_center ltx_border_bb">28.3</td>
<td id="A7.T3.1.8.7.4" class="ltx_td ltx_align_center ltx_border_bb">0.1</td>
</tr>
</tbody>
</table>
</figure>
<div id="A7.p1" class="ltx_para">
<p id="A7.p1.1" class="ltx_p">Objaverse <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib30" title="" class="ltx_ref">30</a>]</cite> is an internet-scale collection of 800 k diverse but noisily or unannotated 3D models.
They were uploaded by 100 k artists to the Sketchfab platform.
A subset of 44 k objects called Objaverse-LVIS is accompanied by human-verified categories.
These can be used to validate object class predictions.
This amounts to only 5% of examples with human-verified labels in a restricted subset of categories.</p>
</div>
<div id="A7.p2" class="ltx_para">
<p id="A7.p2.1" class="ltx_p">A compelling use of VLMs is to infer the type of each Objaverse object in an open-vocabulary setting, providing multiple views of each object to the model.
Furthermore, this is an interesting evaluation benchmark for VLMs: the images come from a more specialized distribution of 3D renders (often untextured), and the query of object type is something we should expect a VLM to handle well.</p>
</div>
<div id="A7.p3" class="ltx_para">
<p id="A7.p3.1" class="ltx_p">The previously most popular method, CAP3D <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib72" title="" class="ltx_ref">72</a>]</cite>, is an ad-hoc combination of BLIP2, CLIP, and GPT4, where GPT4 summarizes annotations created by BLIP2 and CLIP.
Given access to log-probabilities, <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib48" title="" class="ltx_ref">48</a>]</cite> show that the aggregating the predictions of each view informed by their log-probabilities (<span id="A7.p3.1.1" class="ltx_text ltx_font_typewriter">ScoreAgg</span>), largely outperforms CAP3D while being significantly simpler.
See <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib48" title="" class="ltx_ref">48</a>]</cite> for more details on the exact setup.</p>
</div>
<div id="A7.p4" class="ltx_para">
<p id="A7.p4.1" class="ltx_p">We additionally evaluate PaLI-3, and the open-weight PaliGemma base and VQAv2 fine-tune.
We use four VQA prompts to probe for the type of each object: (i) “What is this?” (ii) “What type of object is this?” (iii) “What is in the image?” (iv) “Describe the object in the image”. This produces four sets of five sampled responses per view, which are aggregated using ScoreAgg to produce a final prediction.
The results are summarized in Table <a href="#A7.T3" title="Table 3 ‣ Appendix G More details and results with Objaverse ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> and show that PaliGemma has clear 3D object understanding out of the box, and the VQAv2 fine-tune performs even better.
Finally, we also show a representative example of raw predictions in Figure <a href="#A7.F15" title="Figure 15 ‣ Appendix G More details and results with Objaverse ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">15</span></a>; a banana for s(c)ale, if you will.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A8" class="ltx_appendix" lang="en">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix H </span>Multitask transfer</h2>

<figure id="A8.T4" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>Results when multitasking. Table 1 is the best achievable single-task result with per-task tuned hyper-paramters. “Single Simple” is then the per-task performance when using the same simplified hyper-parameter setting for all tasks. Finally, “Multi” is the multitasking setup where a single model performs all tasks, either with or without prefix indicating the task. The number in parenthesis is the relative regret versus the preceding column.</figcaption>
<table id="A8.T4.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A8.T4.1.1.1" class="ltx_tr">
<td id="A8.T4.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_tt" style="width:113.8pt;">
<span id="A8.T4.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.1.1.1.1.1" class="ltx_p"><span id="A8.T4.1.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Metric</span></span>
</span>
</td>
<td id="A8.T4.1.1.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="A8.T4.1.1.1.2.1" class="ltx_text ltx_font_bold">Table 1</span></td>
<td id="A8.T4.1.1.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="A8.T4.1.1.1.3.1" class="ltx_text ltx_font_bold">Single Simple</span></td>
<td id="A8.T4.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="A8.T4.1.1.1.4.1" class="ltx_text ltx_font_bold">Multi Prefix</span></td>
<td id="A8.T4.1.1.1.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="A8.T4.1.1.1.5.1" class="ltx_text ltx_font_bold">Multi No Prefix</span></td>
</tr>
<tr id="A8.T4.1.2.2" class="ltx_tr">
<td id="A8.T4.1.2.2.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:113.8pt;">
<span id="A8.T4.1.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.2.2.1.1.1" class="ltx_p">AI2D</span>
</span>
</td>
<td id="A8.T4.1.2.2.2" class="ltx_td ltx_align_center ltx_border_t">
<span id="A8.T4.1.2.2.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>72.1</td>
<td id="A8.T4.1.2.2.3" class="ltx_td ltx_align_center ltx_border_t">
<span id="A8.T4.1.2.2.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>73.1</td>
<td id="A8.T4.1.2.2.4" class="ltx_td ltx_align_center ltx_border_t">
<span id="A8.T4.1.2.2.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>72.3 <span id="A8.T4.1.2.2.4.2" class="ltx_text" style="font-size:80%;">(-1.0%)</span>
</td>
<td id="A8.T4.1.2.2.5" class="ltx_td ltx_align_center ltx_border_t">
<span id="A8.T4.1.2.2.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>72.3 <span id="A8.T4.1.2.2.5.2" class="ltx_text" style="font-size:80%;">(-1.1%)</span>
</td>
</tr>
<tr id="A8.T4.1.3.3" class="ltx_tr">
<td id="A8.T4.1.3.3.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.3.3.1.1.1" class="ltx_p">COCOcap</span>
</span>
</td>
<td id="A8.T4.1.3.3.2" class="ltx_td ltx_align_center">141.9</td>
<td id="A8.T4.1.3.3.3" class="ltx_td ltx_align_center">141.7</td>
<td id="A8.T4.1.3.3.4" class="ltx_td ltx_align_center">139.5 <span id="A8.T4.1.3.3.4.1" class="ltx_text" style="font-size:80%;">(-1.6%)</span>
</td>
<td id="A8.T4.1.3.3.5" class="ltx_td ltx_align_center">138.9 <span id="A8.T4.1.3.3.5.1" class="ltx_text" style="font-size:80%;">(-2.0%)</span>
</td>
</tr>
<tr id="A8.T4.1.4.4" class="ltx_tr">
<td id="A8.T4.1.4.4.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.4.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.4.4.1.1.1" class="ltx_p">NoCaps</span>
</span>
</td>
<td id="A8.T4.1.4.4.2" class="ltx_td ltx_align_center">121.7</td>
<td id="A8.T4.1.4.4.3" class="ltx_td ltx_align_center">121.6</td>
<td id="A8.T4.1.4.4.4" class="ltx_td ltx_align_center">118.4 <span id="A8.T4.1.4.4.4.1" class="ltx_text" style="font-size:80%;">(-2.6%)</span>
</td>
<td id="A8.T4.1.4.4.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.4.4.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>98.9 <span id="A8.T4.1.4.4.5.2" class="ltx_text" style="font-size:80%;">(-18.7%)</span>
</td>
</tr>
<tr id="A8.T4.1.5.5" class="ltx_tr">
<td id="A8.T4.1.5.5.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.5.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.5.5.1.1.1" class="ltx_p">DocVQA (val)</span>
</span>
</td>
<td id="A8.T4.1.5.5.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.5.5.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>37.8</td>
<td id="A8.T4.1.5.5.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.5.5.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>38.6</td>
<td id="A8.T4.1.5.5.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.5.5.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>32.2 <span id="A8.T4.1.5.5.4.2" class="ltx_text" style="font-size:80%;">(-16.7%)</span>
</td>
<td id="A8.T4.1.5.5.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.5.5.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>31.3 <span id="A8.T4.1.5.5.5.2" class="ltx_text" style="font-size:80%;">(-19.0%)</span>
</td>
</tr>
<tr id="A8.T4.1.6.6" class="ltx_tr">
<td id="A8.T4.1.6.6.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.6.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.6.6.1.1.1" class="ltx_p">GQA</span>
</span>
</td>
<td id="A8.T4.1.6.6.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.6.6.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>65.6</td>
<td id="A8.T4.1.6.6.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.6.6.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>65.4</td>
<td id="A8.T4.1.6.6.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.6.6.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>64.9 <span id="A8.T4.1.6.6.4.2" class="ltx_text" style="font-size:80%;">(-0.7%)</span>
</td>
<td id="A8.T4.1.6.6.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.6.6.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>64.2 <span id="A8.T4.1.6.6.5.2" class="ltx_text" style="font-size:80%;">(-1.8%)</span>
</td>
</tr>
<tr id="A8.T4.1.7.7" class="ltx_tr">
<td id="A8.T4.1.7.7.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.7.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.7.7.1.1.1" class="ltx_p">xGQA (avg7)</span>
</span>
</td>
<td id="A8.T4.1.7.7.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.7.7.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>57.3</td>
<td id="A8.T4.1.7.7.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.7.7.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>56.9</td>
<td id="A8.T4.1.7.7.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.7.7.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>55.8 <span id="A8.T4.1.7.7.4.2" class="ltx_text" style="font-size:80%;">(-2.0%)</span>
</td>
<td id="A8.T4.1.7.7.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.7.7.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>52.6 <span id="A8.T4.1.7.7.5.2" class="ltx_text" style="font-size:80%;">(-7.7%)</span>
</td>
</tr>
<tr id="A8.T4.1.8.8" class="ltx_tr">
<td id="A8.T4.1.8.8.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.8.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.8.8.1.1.1" class="ltx_p">InfoVQA (val)</span>
</span>
</td>
<td id="A8.T4.1.8.8.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.8.8.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>25.5</td>
<td id="A8.T4.1.8.8.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.8.8.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>25.0</td>
<td id="A8.T4.1.8.8.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.8.8.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>20.8 <span id="A8.T4.1.8.8.4.2" class="ltx_text" style="font-size:80%;">(-17.0%)</span>
</td>
<td id="A8.T4.1.8.8.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.8.8.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>21.4 <span id="A8.T4.1.8.8.5.2" class="ltx_text" style="font-size:80%;">(-14.4%)</span>
</td>
</tr>
<tr id="A8.T4.1.9.9" class="ltx_tr">
<td id="A8.T4.1.9.9.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.9.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.9.9.1.1.1" class="ltx_p">OCR-VQA</span>
</span>
</td>
<td id="A8.T4.1.9.9.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.9.9.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>72.3</td>
<td id="A8.T4.1.9.9.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.9.9.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>73.3</td>
<td id="A8.T4.1.9.9.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.9.9.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>71.4 <span id="A8.T4.1.9.9.4.2" class="ltx_text" style="font-size:80%;">(-2.6%)</span>
</td>
<td id="A8.T4.1.9.9.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.9.9.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>71.4 <span id="A8.T4.1.9.9.5.2" class="ltx_text" style="font-size:80%;">(-2.6%)</span>
</td>
</tr>
<tr id="A8.T4.1.10.10" class="ltx_tr">
<td id="A8.T4.1.10.10.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.10.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.10.10.1.1.1" class="ltx_p">OKVQA</span>
</span>
</td>
<td id="A8.T4.1.10.10.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.10.10.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>63.5</td>
<td id="A8.T4.1.10.10.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.10.10.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>63.1</td>
<td id="A8.T4.1.10.10.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.10.10.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>67.0 <span id="A8.T4.1.10.10.4.2" class="ltx_text" style="font-size:80%;">(+6.2%)</span>
</td>
<td id="A8.T4.1.10.10.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.10.10.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>58.9 <span id="A8.T4.1.10.10.5.2" class="ltx_text" style="font-size:80%;">(-6.6%)</span>
</td>
</tr>
<tr id="A8.T4.1.11.11" class="ltx_tr">
<td id="A8.T4.1.11.11.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.11.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.11.11.1.1.1" class="ltx_p">RefCOCO (val)</span>
</span>
</td>
<td id="A8.T4.1.11.11.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.11.11.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>73.4</td>
<td id="A8.T4.1.11.11.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.11.11.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>69.9</td>
<td id="A8.T4.1.11.11.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.11.11.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>68.4 <span id="A8.T4.1.11.11.4.2" class="ltx_text" style="font-size:80%;">(-2.1%)</span>
</td>
<td id="A8.T4.1.11.11.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.11.11.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>68.3 <span id="A8.T4.1.11.11.5.2" class="ltx_text" style="font-size:80%;">(-2.3%)</span>
</td>
</tr>
<tr id="A8.T4.1.12.12" class="ltx_tr">
<td id="A8.T4.1.12.12.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.12.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.12.12.1.1.1" class="ltx_p">RefCOCO+ (val)</span>
</span>
</td>
<td id="A8.T4.1.12.12.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.12.12.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>68.3</td>
<td id="A8.T4.1.12.12.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.12.12.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>61.0</td>
<td id="A8.T4.1.12.12.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.12.12.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>61.2 <span id="A8.T4.1.12.12.4.2" class="ltx_text" style="font-size:80%;">(+0.3%)</span>
</td>
<td id="A8.T4.1.12.12.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.12.12.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>60.9 <span id="A8.T4.1.12.12.5.2" class="ltx_text" style="font-size:80%;">(-0.2%)</span>
</td>
</tr>
<tr id="A8.T4.1.13.13" class="ltx_tr">
<td id="A8.T4.1.13.13.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.13.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.13.13.1.1.1" class="ltx_p">RefCOCOg (val)</span>
</span>
</td>
<td id="A8.T4.1.13.13.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.13.13.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>67.7</td>
<td id="A8.T4.1.13.13.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.13.13.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>63.7</td>
<td id="A8.T4.1.13.13.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.13.13.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>61.8 <span id="A8.T4.1.13.13.4.2" class="ltx_text" style="font-size:80%;">(-3.1%)</span>
</td>
<td id="A8.T4.1.13.13.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.13.13.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>61.8 <span id="A8.T4.1.13.13.5.2" class="ltx_text" style="font-size:80%;">(-3.0%)</span>
</td>
</tr>
<tr id="A8.T4.1.14.14" class="ltx_tr">
<td id="A8.T4.1.14.14.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.14.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.14.14.1.1.1" class="ltx_p">RSVQA-hr (test)</span>
</span>
</td>
<td id="A8.T4.1.14.14.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.14.14.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>92.6</td>
<td id="A8.T4.1.14.14.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.14.14.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>92.7</td>
<td id="A8.T4.1.14.14.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.14.14.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>92.1 <span id="A8.T4.1.14.14.4.2" class="ltx_text" style="font-size:80%;">(-0.6%)</span>
</td>
<td id="A8.T4.1.14.14.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.14.14.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>92.2 <span id="A8.T4.1.14.14.5.2" class="ltx_text" style="font-size:80%;">(-0.5%)</span>
</td>
</tr>
<tr id="A8.T4.1.15.15" class="ltx_tr">
<td id="A8.T4.1.15.15.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.15.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.15.15.1.1.1" class="ltx_p">RSVQA-hr (test2)</span>
</span>
</td>
<td id="A8.T4.1.15.15.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.15.15.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>90.6</td>
<td id="A8.T4.1.15.15.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.15.15.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>90.8</td>
<td id="A8.T4.1.15.15.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.15.15.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>90.0 <span id="A8.T4.1.15.15.4.2" class="ltx_text" style="font-size:80%;">(-0.9%)</span>
</td>
<td id="A8.T4.1.15.15.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.15.15.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>90.2 <span id="A8.T4.1.15.15.5.2" class="ltx_text" style="font-size:80%;">(-0.7%)</span>
</td>
</tr>
<tr id="A8.T4.1.16.16" class="ltx_tr">
<td id="A8.T4.1.16.16.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.16.16.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.16.16.1.1.1" class="ltx_p">RSVQA-lr</span>
</span>
</td>
<td id="A8.T4.1.16.16.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.16.16.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>92.6</td>
<td id="A8.T4.1.16.16.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.16.16.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>93.7</td>
<td id="A8.T4.1.16.16.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.16.16.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>93.4 <span id="A8.T4.1.16.16.4.2" class="ltx_text" style="font-size:80%;">(-0.3%)</span>
</td>
<td id="A8.T4.1.16.16.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.16.16.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>93.0 <span id="A8.T4.1.16.16.5.2" class="ltx_text" style="font-size:80%;">(-0.8%)</span>
</td>
</tr>
<tr id="A8.T4.1.17.17" class="ltx_tr">
<td id="A8.T4.1.17.17.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.17.17.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.17.17.1.1.1" class="ltx_p">SciCap</span>
</span>
</td>
<td id="A8.T4.1.17.17.2" class="ltx_td ltx_align_center">162.3</td>
<td id="A8.T4.1.17.17.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.17.17.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>64.0</td>
<td id="A8.T4.1.17.17.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.17.17.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>74.7 <span id="A8.T4.1.17.17.4.2" class="ltx_text" style="font-size:80%;">(+16.7%)</span>
</td>
<td id="A8.T4.1.17.17.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.17.17.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>74.7 <span id="A8.T4.1.17.17.5.2" class="ltx_text" style="font-size:80%;">(+16.6%)</span>
</td>
</tr>
<tr id="A8.T4.1.18.18" class="ltx_tr">
<td id="A8.T4.1.18.18.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.18.18.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.18.18.1.1.1" class="ltx_p">Screen2Words</span>
</span>
</td>
<td id="A8.T4.1.18.18.2" class="ltx_td ltx_align_center">117.6</td>
<td id="A8.T4.1.18.18.3" class="ltx_td ltx_align_center">114.7</td>
<td id="A8.T4.1.18.18.4" class="ltx_td ltx_align_center">111.6 <span id="A8.T4.1.18.18.4.1" class="ltx_text" style="font-size:80%;">(-2.7%)</span>
</td>
<td id="A8.T4.1.18.18.5" class="ltx_td ltx_align_center">110.6 <span id="A8.T4.1.18.18.5.1" class="ltx_text" style="font-size:80%;">(-3.5%)</span>
</td>
</tr>
<tr id="A8.T4.1.19.19" class="ltx_tr">
<td id="A8.T4.1.19.19.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.19.19.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.19.19.1.1.1" class="ltx_p">ST-VQA (val)</span>
</span>
</td>
<td id="A8.T4.1.19.19.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.19.19.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>61.6</td>
<td id="A8.T4.1.19.19.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.19.19.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>62.0</td>
<td id="A8.T4.1.19.19.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.19.19.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>58.3 <span id="A8.T4.1.19.19.4.2" class="ltx_text" style="font-size:80%;">(-6.0%)</span>
</td>
<td id="A8.T4.1.19.19.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.19.19.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>57.8 <span id="A8.T4.1.19.19.5.2" class="ltx_text" style="font-size:80%;">(-6.8%)</span>
</td>
</tr>
<tr id="A8.T4.1.20.20" class="ltx_tr">
<td id="A8.T4.1.20.20.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.20.20.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.20.20.1.1.1" class="ltx_p">TallyQA (simple)</span>
</span>
</td>
<td id="A8.T4.1.20.20.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.20.20.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>81.7</td>
<td id="A8.T4.1.20.20.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.20.20.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>81.3</td>
<td id="A8.T4.1.20.20.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.20.20.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>80.4 <span id="A8.T4.1.20.20.4.2" class="ltx_text" style="font-size:80%;">(-1.1%)</span>
</td>
<td id="A8.T4.1.20.20.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.20.20.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>80.7 <span id="A8.T4.1.20.20.5.2" class="ltx_text" style="font-size:80%;">(-0.7%)</span>
</td>
</tr>
<tr id="A8.T4.1.21.21" class="ltx_tr">
<td id="A8.T4.1.21.21.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.21.21.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.21.21.1.1.1" class="ltx_p">TallyQA (complex)</span>
</span>
</td>
<td id="A8.T4.1.21.21.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.21.21.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>69.6</td>
<td id="A8.T4.1.21.21.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.21.21.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>69.6</td>
<td id="A8.T4.1.21.21.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.21.21.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>69.6 <span id="A8.T4.1.21.21.4.2" class="ltx_text" style="font-size:80%;">(-0.1%)</span>
</td>
<td id="A8.T4.1.21.21.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.21.21.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>69.9 <span id="A8.T4.1.21.21.5.2" class="ltx_text" style="font-size:80%;">(+0.3%)</span>
</td>
</tr>
<tr id="A8.T4.1.22.22" class="ltx_tr">
<td id="A8.T4.1.22.22.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.22.22.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.22.22.1.1.1" class="ltx_p">CountBenchQA</span>
</span>
</td>
<td id="A8.T4.1.22.22.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.22.22.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>81.9</td>
<td id="A8.T4.1.22.22.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.22.22.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>83.5</td>
<td id="A8.T4.1.22.22.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.22.22.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>80.6 <span id="A8.T4.1.22.22.4.2" class="ltx_text" style="font-size:80%;">(-3.5%)</span>
</td>
<td id="A8.T4.1.22.22.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.22.22.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>78.3 <span id="A8.T4.1.22.22.5.2" class="ltx_text" style="font-size:80%;">(-6.2%)</span>
</td>
</tr>
<tr id="A8.T4.1.23.23" class="ltx_tr">
<td id="A8.T4.1.23.23.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.23.23.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.23.23.1.1.1" class="ltx_p">TextCaps</span>
</span>
</td>
<td id="A8.T4.1.23.23.2" class="ltx_td ltx_align_center">127.5</td>
<td id="A8.T4.1.23.23.3" class="ltx_td ltx_align_center">128.3</td>
<td id="A8.T4.1.23.23.4" class="ltx_td ltx_align_center">122.5 <span id="A8.T4.1.23.23.4.1" class="ltx_text" style="font-size:80%;">(-4.5%)</span>
</td>
<td id="A8.T4.1.23.23.5" class="ltx_td ltx_align_center">122.3 <span id="A8.T4.1.23.23.5.1" class="ltx_text" style="font-size:80%;">(-4.6%)</span>
</td>
</tr>
<tr id="A8.T4.1.24.24" class="ltx_tr">
<td id="A8.T4.1.24.24.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.24.24.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.24.24.1.1.1" class="ltx_p">TextVQA (val)</span>
</span>
</td>
<td id="A8.T4.1.24.24.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.24.24.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>59.0</td>
<td id="A8.T4.1.24.24.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.24.24.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>57.9</td>
<td id="A8.T4.1.24.24.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.24.24.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>56.0 <span id="A8.T4.1.24.24.4.2" class="ltx_text" style="font-size:80%;">(-3.2%)</span>
</td>
<td id="A8.T4.1.24.24.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.24.24.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>56.5 <span id="A8.T4.1.24.24.5.2" class="ltx_text" style="font-size:80%;">(-2.5%)</span>
</td>
</tr>
<tr id="A8.T4.1.25.25" class="ltx_tr">
<td id="A8.T4.1.25.25.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.25.25.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.25.25.1.1.1" class="ltx_p">VQAv2 (minival)</span>
</span>
</td>
<td id="A8.T4.1.25.25.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.25.25.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>82.1</td>
<td id="A8.T4.1.25.25.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.25.25.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>81.9</td>
<td id="A8.T4.1.25.25.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.25.25.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>83.4 <span id="A8.T4.1.25.25.4.2" class="ltx_text" style="font-size:80%;">(+1.8%)</span>
</td>
<td id="A8.T4.1.25.25.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.25.25.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>82.9 <span id="A8.T4.1.25.25.5.2" class="ltx_text" style="font-size:80%;">(+1.2%)</span>
</td>
</tr>
<tr id="A8.T4.1.26.26" class="ltx_tr">
<td id="A8.T4.1.26.26.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.26.26.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.26.26.1.1.1" class="ltx_p">VizWizVQA (val)</span>
</span>
</td>
<td id="A8.T4.1.26.26.2" class="ltx_td ltx_align_center">
<span id="A8.T4.1.26.26.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>76.4</td>
<td id="A8.T4.1.26.26.3" class="ltx_td ltx_align_center">
<span id="A8.T4.1.26.26.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>75.9</td>
<td id="A8.T4.1.26.26.4" class="ltx_td ltx_align_center">
<span id="A8.T4.1.26.26.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>75.7 <span id="A8.T4.1.26.26.4.2" class="ltx_text" style="font-size:80%;">(-0.2%)</span>
</td>
<td id="A8.T4.1.26.26.5" class="ltx_td ltx_align_center">
<span id="A8.T4.1.26.26.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>76.0 <span id="A8.T4.1.26.26.5.2" class="ltx_text" style="font-size:80%;">(+0.2%)</span>
</td>
</tr>
<tr id="A8.T4.1.27.27" class="ltx_tr">
<td id="A8.T4.1.27.27.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:113.8pt;">
<span id="A8.T4.1.27.27.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.27.27.1.1.1" class="ltx_p">WidgetCap</span>
</span>
</td>
<td id="A8.T4.1.27.27.2" class="ltx_td ltx_align_center">136.1</td>
<td id="A8.T4.1.27.27.3" class="ltx_td ltx_align_center">136.4</td>
<td id="A8.T4.1.27.27.4" class="ltx_td ltx_align_center">129.1 <span id="A8.T4.1.27.27.4.1" class="ltx_text" style="font-size:80%;">(-5.3%)</span>
</td>
<td id="A8.T4.1.27.27.5" class="ltx_td ltx_align_center">131.5 <span id="A8.T4.1.27.27.5.1" class="ltx_text" style="font-size:80%;">(-3.6%)</span>
</td>
</tr>
<tr id="A8.T4.1.28.28" class="ltx_tr">
<td id="A8.T4.1.28.28.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb ltx_border_t" style="width:113.8pt;">
<span id="A8.T4.1.28.28.1.1" class="ltx_inline-block ltx_align_top">
<span id="A8.T4.1.28.28.1.1.1" class="ltx_p">Average</span>
</span>
</td>
<td id="A8.T4.1.28.28.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">
<span id="A8.T4.1.28.28.2.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>84.6</td>
<td id="A8.T4.1.28.28.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">
<span id="A8.T4.1.28.28.3.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>80.2</td>
<td id="A8.T4.1.28.28.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">
<span id="A8.T4.1.28.28.4.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>78.9 <span id="A8.T4.1.28.28.4.2" class="ltx_text" style="font-size:80%;">(-2.0%)</span>
</td>
<td id="A8.T4.1.28.28.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">
<span id="A8.T4.1.28.28.5.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">0</span></span>77.6 <span id="A8.T4.1.28.28.5.2" class="ltx_text" style="font-size:80%;">(-3.5%)</span>
</td>
</tr>
</tbody>
</table>
</figure>
<div id="A8.p1" class="ltx_para">
<p id="A8.p1.1" class="ltx_p">Transferring to individual tasks by fine-tuning is good when all one cares about is having a model that solves a specific task.
However, it is often desirable to have a single generalist model with a conversational interface.
This is typically achieved by instruction tuning, <em id="A8.p1.1.1" class="ltx_emph ltx_font_italic">i.e</em>.<span id="A8.p1.1.2" class="ltx_text"></span> fine-tuning on a mixture of a diverse dataset.
We verify that PaliGemma is well-suited for this type of transfer: we transfer it to a mix of 27 of our datasets (excluding some tasks, such as multi-image, for simplicity).
We follow the “simplified” hyper-parameter from Section <a href="#S6.SS2" title="6.2 Transfer hyper-parameter sensitivity ‣ 6 Transferability ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6.2</span></a> for the mixture, and mix uniformly such that each individual task goes through 10 epochs, no matter its size.</p>
</div>
<div id="A8.p2" class="ltx_para">
<p id="A8.p2.1" class="ltx_p">The results in Table <a href="#A8.T4" title="Table 4 ‣ Appendix H Multitask transfer ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> show that there is a dramatic change in a few tasks, while most win or lose just a little.
Surprisingly, the overall average shows the largest loss comes from changing to a unified hyper-parameter, followed by multitasking, and removing per-task prefix indicator comes with the smallest regret.
However, it should be noted that this multitasking setup was not tuned much.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A9" class="ltx_appendix" lang="en">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix I </span>Inference</h2>

<div id="A9.p1" class="ltx_para">
<p id="A9.p1.1" class="ltx_p">See table <a href="#A9.T5" title="Table 5 ‣ Appendix I Inference ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> for inference measurements using the code from <span id="A9.p1.1.1" class="ltx_text ltx_font_typewriter">big_vision</span>.</p>
</div>
<figure id="A9.T5" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>
Inference measurements on a TPUv3 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib36" title="" class="ltx_ref">36</a>]</cite> with 8 devices (4 chips).
Even though FSDP sharding <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib132" title="" class="ltx_ref">132</a>]</cite> is very efficient at training time (Section <a href="#S3.SS2.SSS6" title="3.2.6 Other pretraining details ‣ 3.2 Pretraining ‣ 3 Model ‣ PaliGemma: A versatile 3B VLM for transfer" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2.6</span></a>),
at inference time it has the downside that each device needs process at least one single example and needs to read all model parameters. It reduces the memory requirements but it still processes one single example as slow as using a single device with a larger memory.
Megatron-style sharding <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib95" title="" class="ltx_ref">95</a>]</cite> on the other hand shards both the parameters <span id="A9.T5.2.1" class="ltx_text ltx_font_italic">and</span> the activations, and can make use of multiple accelerators in parallel to process a single example and significantly reduce the total number of memory reads per device.
</figcaption>
<table id="A9.T5.3" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A9.T5.3.1.1" class="ltx_tr">
<td id="A9.T5.3.1.1.1" class="ltx_td ltx_align_justify ltx_align_bottom ltx_border_tt" style="width:34.1pt;" rowspan="2">
<span id="A9.T5.3.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.1.1.1.1.1" class="ltx_p"><span id="A9.T5.3.1.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Sharding</span></span>
</span>
</td>
<td id="A9.T5.3.1.1.2" class="ltx_td ltx_align_justify ltx_align_bottom ltx_border_tt" style="width:39.8pt;" rowspan="2">
<span id="A9.T5.3.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.1.1.2.1.1" class="ltx_p"><span id="A9.T5.3.1.1.2.1.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Params</span></span>
</span>
</td>
<td id="A9.T5.3.1.1.3" class="ltx_td ltx_align_justify ltx_align_bottom ltx_border_tt" style="width:28.5pt;" rowspan="2">
<span id="A9.T5.3.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.1.1.3.1.1" class="ltx_p"><span id="A9.T5.3.1.1.3.1.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Batch</span></span>
</span>
</td>
<td id="A9.T5.3.1.1.4" class="ltx_td ltx_align_center ltx_align_middle ltx_border_tt" colspan="2"><span id="A9.T5.3.1.1.4.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Walltime [ms]</span></td>
<td id="A9.T5.3.1.1.5" class="ltx_td ltx_align_center ltx_align_middle ltx_border_tt" colspan="2"><span id="A9.T5.3.1.1.5.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Tokens/sec</span></td>
<td id="A9.T5.3.1.1.6" class="ltx_td ltx_align_center ltx_align_middle ltx_border_tt" colspan="2"><span id="A9.T5.3.1.1.6.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Utilization [%]</span></td>
<td id="A9.T5.3.1.1.7" class="ltx_td ltx_align_center ltx_align_middle ltx_border_tt" colspan="2"><span id="A9.T5.3.1.1.7.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Memory [GiB]</span></td>
</tr>
<tr id="A9.T5.3.2.2" class="ltx_tr">
<td id="A9.T5.3.2.2.1" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.2.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.2.2.1.1.1" class="ltx_p"><span id="A9.T5.3.2.2.1.1.1.1" class="ltx_text" style="font-size:90%;">Prefill</span></span>
</span>
</td>
<td id="A9.T5.3.2.2.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.2.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.2.2.2.1.1" class="ltx_p"><span id="A9.T5.3.2.2.2.1.1.1" class="ltx_text" style="font-size:90%;">Extend</span></span>
</span>
</td>
<td id="A9.T5.3.2.2.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:34.1pt;">
<span id="A9.T5.3.2.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.2.2.3.1.1" class="ltx_p"><span id="A9.T5.3.2.2.3.1.1.1" class="ltx_text" style="font-size:90%;">Prefill</span></span>
</span>
</td>
<td id="A9.T5.3.2.2.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.2.2.4.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.2.2.4.1.1" class="ltx_p"><span id="A9.T5.3.2.2.4.1.1.1" class="ltx_text" style="font-size:90%;">Extend</span></span>
</span>
</td>
<td id="A9.T5.3.2.2.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.2.2.5.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.2.2.5.1.1" class="ltx_p"><span id="A9.T5.3.2.2.5.1.1.1" class="ltx_text" style="font-size:90%;">Prefill</span></span>
</span>
</td>
<td id="A9.T5.3.2.2.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.2.2.6.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.2.2.6.1.1" class="ltx_p"><span id="A9.T5.3.2.2.6.1.1.1" class="ltx_text" style="font-size:90%;">Extend</span></span>
</span>
</td>
<td id="A9.T5.3.2.2.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.2.2.7.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.2.2.7.1.1" class="ltx_p"><span id="A9.T5.3.2.2.7.1.1.1" class="ltx_text" style="font-size:90%;">Prefill</span></span>
</span>
</td>
<td id="A9.T5.3.2.2.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.2.2.8.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.2.2.8.1.1" class="ltx_p"><span id="A9.T5.3.2.2.8.1.1.1" class="ltx_text" style="font-size:90%;">Extend</span></span>
</span>
</td>
</tr>
<tr id="A9.T5.3.3.3" class="ltx_tr">
<td id="A9.T5.3.3.3.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:34.1pt;">
<span id="A9.T5.3.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.3.3.1.1.1" class="ltx_p"><span id="A9.T5.3.3.3.1.1.1.1" class="ltx_text" style="font-size:90%;">FSDP</span></span>
</span>
</td>
<td id="A9.T5.3.3.3.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:39.8pt;">
<span id="A9.T5.3.3.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.3.3.2.1.1" class="ltx_p"><span id="A9.T5.3.3.3.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:90%;">float32</span></span>
</span>
</td>
<td id="A9.T5.3.3.3.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:28.5pt;">
<span id="A9.T5.3.3.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.3.3.3.1.1" class="ltx_p"><span id="A9.T5.3.3.3.3.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.3.3.3.1.1.2" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.3.3.3.1.1.3" class="ltx_text" style="font-size:90%;">1</span></span>
</span>
</td>
<td id="A9.T5.3.3.3.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:28.5pt;">
<span id="A9.T5.3.3.3.4.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.3.3.4.1.1" class="ltx_p"><span id="A9.T5.3.3.3.4.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.3.3.4.1.1.2" class="ltx_text" style="font-size:90%;">130</span></span>
</span>
</td>
<td id="A9.T5.3.3.3.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:28.5pt;">
<span id="A9.T5.3.3.3.5.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.3.3.5.1.1" class="ltx_p"><span id="A9.T5.3.3.3.5.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.3.3.5.1.1.2" class="ltx_text" style="font-size:90%;">14.7</span></span>
</span>
</td>
<td id="A9.T5.3.3.3.6" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:34.1pt;">
<span id="A9.T5.3.3.3.6.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.3.3.6.1.1" class="ltx_p"><span id="A9.T5.3.3.3.6.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.3.3.6.1.1.2" class="ltx_text" style="font-size:90%;">3953</span></span>
</span>
</td>
<td id="A9.T5.3.3.3.7" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:28.5pt;">
<span id="A9.T5.3.3.3.7.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.3.3.7.1.1" class="ltx_p"><span id="A9.T5.3.3.3.7.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.3.3.7.1.1.2" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.3.3.7.1.1.3" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.3.3.7.1.1.4" class="ltx_text" style="font-size:90%;">68</span></span>
</span>
</td>
<td id="A9.T5.3.3.3.8" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:28.5pt;">
<span id="A9.T5.3.3.3.8.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.3.3.8.1.1" class="ltx_p"><span id="A9.T5.3.3.3.8.1.1.1" class="ltx_text" style="font-size:90%;">28.8</span></span>
</span>
</td>
<td id="A9.T5.3.3.3.9" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:28.5pt;">
<span id="A9.T5.3.3.3.9.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.3.3.9.1.1" class="ltx_p"><span id="A9.T5.3.3.3.9.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.3.3.9.1.1.2" class="ltx_text" style="font-size:90%;">0.58</span></span>
</span>
</td>
<td id="A9.T5.3.3.3.10" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:28.5pt;">
<span id="A9.T5.3.3.3.10.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.3.3.10.1.1" class="ltx_p"><span id="A9.T5.3.3.3.10.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.3.3.10.1.1.2" class="ltx_text" style="font-size:90%;">3.48</span></span>
</span>
</td>
<td id="A9.T5.3.3.3.11" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:28.5pt;">
<span id="A9.T5.3.3.3.11.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.3.3.11.1.1" class="ltx_p"><span id="A9.T5.3.3.3.11.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.3.3.11.1.1.2" class="ltx_text" style="font-size:90%;">1.69</span></span>
</span>
</td>
</tr>
<tr id="A9.T5.3.4.4" class="ltx_tr">
<td id="A9.T5.3.4.4.1" class="ltx_td ltx_align_middle" style="width:34.1pt;"></td>
<td id="A9.T5.3.4.4.2" class="ltx_td ltx_align_middle" style="width:39.8pt;"></td>
<td id="A9.T5.3.4.4.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.4.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.4.4.3.1.1" class="ltx_p"><span id="A9.T5.3.4.4.3.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.4.4.3.1.1.2" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.4.4.3.1.1.3" class="ltx_text" style="font-size:90%;">8</span></span>
</span>
</td>
<td id="A9.T5.3.4.4.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.4.4.4.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.4.4.4.1.1" class="ltx_p"><span id="A9.T5.3.4.4.4.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.4.4.4.1.1.2" class="ltx_text" style="font-size:90%;">129</span></span>
</span>
</td>
<td id="A9.T5.3.4.4.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.4.4.5.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.4.4.5.1.1" class="ltx_p"><span id="A9.T5.3.4.4.5.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.4.4.5.1.1.2" class="ltx_text" style="font-size:90%;">14.7</span></span>
</span>
</td>
<td id="A9.T5.3.4.4.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:34.1pt;">
<span id="A9.T5.3.4.4.6.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.4.4.6.1.1" class="ltx_p"><span id="A9.T5.3.4.4.6.1.1.1" class="ltx_text" style="font-size:90%;">31656</span></span>
</span>
</td>
<td id="A9.T5.3.4.4.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.4.4.7.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.4.4.7.1.1" class="ltx_p"><span id="A9.T5.3.4.4.7.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.4.4.7.1.1.2" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.4.4.7.1.1.3" class="ltx_text" style="font-size:90%;">546</span></span>
</span>
</td>
<td id="A9.T5.3.4.4.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.4.4.8.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.4.4.8.1.1" class="ltx_p"><span id="A9.T5.3.4.4.8.1.1.1" class="ltx_text" style="font-size:90%;">28.8</span></span>
</span>
</td>
<td id="A9.T5.3.4.4.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.4.4.9.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.4.4.9.1.1" class="ltx_p"><span id="A9.T5.3.4.4.9.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.4.4.9.1.1.2" class="ltx_text" style="font-size:90%;">0.58</span></span>
</span>
</td>
<td id="A9.T5.3.4.4.10" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.4.4.10.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.4.4.10.1.1" class="ltx_p"><span id="A9.T5.3.4.4.10.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.4.4.10.1.1.2" class="ltx_text" style="font-size:90%;">3.48</span></span>
</span>
</td>
<td id="A9.T5.3.4.4.11" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.4.4.11.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.4.4.11.1.1" class="ltx_p"><span id="A9.T5.3.4.4.11.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.4.4.11.1.1.2" class="ltx_text" style="font-size:90%;">1.69</span></span>
</span>
</td>
</tr>
<tr id="A9.T5.3.5.5" class="ltx_tr">
<td id="A9.T5.3.5.5.1" class="ltx_td ltx_align_middle" style="width:34.1pt;"></td>
<td id="A9.T5.3.5.5.2" class="ltx_td ltx_align_middle" style="width:39.8pt;"></td>
<td id="A9.T5.3.5.5.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.5.5.3.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.5.5.3.1.1" class="ltx_p"><span id="A9.T5.3.5.5.3.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.5.5.3.1.1.2" class="ltx_text" style="font-size:90%;">64</span></span>
</span>
</td>
<td id="A9.T5.3.5.5.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.5.5.4.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.5.5.4.1.1" class="ltx_p"><span id="A9.T5.3.5.5.4.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.5.5.4.1.1.2" class="ltx_text" style="font-size:90%;">782</span></span>
</span>
</td>
<td id="A9.T5.3.5.5.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.5.5.5.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.5.5.5.1.1" class="ltx_p"><span id="A9.T5.3.5.5.5.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.5.5.5.1.1.2" class="ltx_text" style="font-size:90%;">14.7</span></span>
</span>
</td>
<td id="A9.T5.3.5.5.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:34.1pt;">
<span id="A9.T5.3.5.5.6.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.5.5.6.1.1" class="ltx_p"><span id="A9.T5.3.5.5.6.1.1.1" class="ltx_text" style="font-size:90%;">41909</span></span>
</span>
</td>
<td id="A9.T5.3.5.5.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.5.5.7.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.5.5.7.1.1" class="ltx_p"><span id="A9.T5.3.5.5.7.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.5.5.7.1.1.2" class="ltx_text" style="font-size:90%;">4365</span></span>
</span>
</td>
<td id="A9.T5.3.5.5.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.5.5.8.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.5.5.8.1.1" class="ltx_p"><span id="A9.T5.3.5.5.8.1.1.1" class="ltx_text" style="font-size:90%;">38.2</span></span>
</span>
</td>
<td id="A9.T5.3.5.5.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.5.5.9.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.5.5.9.1.1" class="ltx_p"><span id="A9.T5.3.5.5.9.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.5.5.9.1.1.2" class="ltx_text" style="font-size:90%;">4.46</span></span>
</span>
</td>
<td id="A9.T5.3.5.5.10" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.5.5.10.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.5.5.10.1.1" class="ltx_p"><span id="A9.T5.3.5.5.10.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.5.5.10.1.1.2" class="ltx_text" style="font-size:90%;">2.72</span></span>
</span>
</td>
<td id="A9.T5.3.5.5.11" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.5.5.11.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.5.5.11.1.1" class="ltx_p"><span id="A9.T5.3.5.5.11.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.5.5.11.1.1.2" class="ltx_text" style="font-size:90%;">1.87</span></span>
</span>
</td>
</tr>
<tr id="A9.T5.3.6.6" class="ltx_tr">
<td id="A9.T5.3.6.6.1" class="ltx_td ltx_align_middle" style="width:34.1pt;"></td>
<td id="A9.T5.3.6.6.2" class="ltx_td ltx_align_middle" style="width:39.8pt;"></td>
<td id="A9.T5.3.6.6.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.6.6.3.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.6.6.3.1.1" class="ltx_p"><span id="A9.T5.3.6.6.3.1.1.1" class="ltx_text" style="font-size:90%;">512</span></span>
</span>
</td>
<td id="A9.T5.3.6.6.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.6.6.4.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.6.6.4.1.1" class="ltx_p"><span id="A9.T5.3.6.6.4.1.1.1" class="ltx_text" style="font-size:90%;">6244</span></span>
</span>
</td>
<td id="A9.T5.3.6.6.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.6.6.5.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.6.6.5.1.1" class="ltx_p"><span id="A9.T5.3.6.6.5.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.6.6.5.1.1.2" class="ltx_text" style="font-size:90%;">35.1</span></span>
</span>
</td>
<td id="A9.T5.3.6.6.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:34.1pt;">
<span id="A9.T5.3.6.6.6.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.6.6.6.1.1" class="ltx_p"><span id="A9.T5.3.6.6.6.1.1.1" class="ltx_text" style="font-size:90%;">41984</span></span>
</span>
</td>
<td id="A9.T5.3.6.6.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.6.6.7.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.6.6.7.1.1" class="ltx_p"><span id="A9.T5.3.6.6.7.1.1.1" class="ltx_text" style="font-size:90%;">14596</span></span>
</span>
</td>
<td id="A9.T5.3.6.6.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.6.6.8.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.6.6.8.1.1" class="ltx_p"><span id="A9.T5.3.6.6.8.1.1.1" class="ltx_text" style="font-size:90%;">38.2</span></span>
</span>
</td>
<td id="A9.T5.3.6.6.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.6.6.9.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.6.6.9.1.1" class="ltx_p"><span id="A9.T5.3.6.6.9.1.1.1" class="ltx_text" style="font-size:90%;">14.79</span></span>
</span>
</td>
<td id="A9.T5.3.6.6.10" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.6.6.10.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.6.6.10.1.1" class="ltx_p"><span id="A9.T5.3.6.6.10.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.6.6.10.1.1.2" class="ltx_text" style="font-size:90%;">7.48</span></span>
</span>
</td>
<td id="A9.T5.3.6.6.11" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.6.6.11.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.6.6.11.1.1" class="ltx_p"><span id="A9.T5.3.6.6.11.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.6.6.11.1.1.2" class="ltx_text" style="font-size:90%;">3.21</span></span>
</span>
</td>
</tr>
<tr id="A9.T5.3.7.7" class="ltx_tr">
<td id="A9.T5.3.7.7.1" class="ltx_td ltx_align_middle" style="width:34.1pt;"></td>
<td id="A9.T5.3.7.7.2" class="ltx_td ltx_align_justify ltx_align_middle" style="width:39.8pt;">
<span id="A9.T5.3.7.7.2.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.7.7.2.1.1" class="ltx_p"><span id="A9.T5.3.7.7.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:90%;">bfloat16</span></span>
</span>
</td>
<td id="A9.T5.3.7.7.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.7.7.3.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.7.7.3.1.1" class="ltx_p"><span id="A9.T5.3.7.7.3.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.7.7.3.1.1.2" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.7.7.3.1.1.3" class="ltx_text" style="font-size:90%;">1</span></span>
</span>
</td>
<td id="A9.T5.3.7.7.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.7.7.4.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.7.7.4.1.1" class="ltx_p"><span id="A9.T5.3.7.7.4.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.7.7.4.1.1.2" class="ltx_text" style="font-size:90%;">112</span></span>
</span>
</td>
<td id="A9.T5.3.7.7.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.7.7.5.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.7.7.5.1.1" class="ltx_p"><span id="A9.T5.3.7.7.5.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.7.7.5.1.1.2" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.7.7.5.1.1.3" class="ltx_text" style="font-size:90%;">8.3</span></span>
</span>
</td>
<td id="A9.T5.3.7.7.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:34.1pt;">
<span id="A9.T5.3.7.7.6.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.7.7.6.1.1" class="ltx_p"><span id="A9.T5.3.7.7.6.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.7.7.6.1.1.2" class="ltx_text" style="font-size:90%;">4568</span></span>
</span>
</td>
<td id="A9.T5.3.7.7.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.7.7.7.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.7.7.7.1.1" class="ltx_p"><span id="A9.T5.3.7.7.7.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.7.7.7.1.1.2" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.7.7.7.1.1.3" class="ltx_text" style="font-size:90%;">121</span></span>
</span>
</td>
<td id="A9.T5.3.7.7.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.7.7.8.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.7.7.8.1.1" class="ltx_p"><span id="A9.T5.3.7.7.8.1.1.1" class="ltx_text" style="font-size:90%;">33.3</span></span>
</span>
</td>
<td id="A9.T5.3.7.7.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.7.7.9.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.7.7.9.1.1" class="ltx_p"><span id="A9.T5.3.7.7.9.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.7.7.9.1.1.2" class="ltx_text" style="font-size:90%;">1.01</span></span>
</span>
</td>
<td id="A9.T5.3.7.7.10" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.7.7.10.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.7.7.10.1.1" class="ltx_p"><span id="A9.T5.3.7.7.10.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.7.7.10.1.1.2" class="ltx_text" style="font-size:90%;">1.75</span></span>
</span>
</td>
<td id="A9.T5.3.7.7.11" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.7.7.11.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.7.7.11.1.1" class="ltx_p"><span id="A9.T5.3.7.7.11.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.7.7.11.1.1.2" class="ltx_text" style="font-size:90%;">0.64</span></span>
</span>
</td>
</tr>
<tr id="A9.T5.3.8.8" class="ltx_tr">
<td id="A9.T5.3.8.8.1" class="ltx_td ltx_align_middle" style="width:34.1pt;"></td>
<td id="A9.T5.3.8.8.2" class="ltx_td ltx_align_middle" style="width:39.8pt;"></td>
<td id="A9.T5.3.8.8.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.8.8.3.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.8.8.3.1.1" class="ltx_p"><span id="A9.T5.3.8.8.3.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.8.8.3.1.1.2" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.8.8.3.1.1.3" class="ltx_text" style="font-size:90%;">8</span></span>
</span>
</td>
<td id="A9.T5.3.8.8.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.8.8.4.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.8.8.4.1.1" class="ltx_p"><span id="A9.T5.3.8.8.4.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.8.8.4.1.1.2" class="ltx_text" style="font-size:90%;">112</span></span>
</span>
</td>
<td id="A9.T5.3.8.8.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.8.8.5.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.8.8.5.1.1" class="ltx_p"><span id="A9.T5.3.8.8.5.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.8.8.5.1.1.2" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.8.8.5.1.1.3" class="ltx_text" style="font-size:90%;">8.2</span></span>
</span>
</td>
<td id="A9.T5.3.8.8.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:34.1pt;">
<span id="A9.T5.3.8.8.6.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.8.8.6.1.1" class="ltx_p"><span id="A9.T5.3.8.8.6.1.1.1" class="ltx_text" style="font-size:90%;">36516</span></span>
</span>
</td>
<td id="A9.T5.3.8.8.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.8.8.7.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.8.8.7.1.1" class="ltx_p"><span id="A9.T5.3.8.8.7.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.8.8.7.1.1.2" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.8.8.7.1.1.3" class="ltx_text" style="font-size:90%;">972</span></span>
</span>
</td>
<td id="A9.T5.3.8.8.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.8.8.8.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.8.8.8.1.1" class="ltx_p"><span id="A9.T5.3.8.8.8.1.1.1" class="ltx_text" style="font-size:90%;">33.3</span></span>
</span>
</td>
<td id="A9.T5.3.8.8.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.8.8.9.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.8.8.9.1.1" class="ltx_p"><span id="A9.T5.3.8.8.9.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.8.8.9.1.1.2" class="ltx_text" style="font-size:90%;">1.01</span></span>
</span>
</td>
<td id="A9.T5.3.8.8.10" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.8.8.10.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.8.8.10.1.1" class="ltx_p"><span id="A9.T5.3.8.8.10.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.8.8.10.1.1.2" class="ltx_text" style="font-size:90%;">1.75</span></span>
</span>
</td>
<td id="A9.T5.3.8.8.11" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.8.8.11.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.8.8.11.1.1" class="ltx_p"><span id="A9.T5.3.8.8.11.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.8.8.11.1.1.2" class="ltx_text" style="font-size:90%;">0.64</span></span>
</span>
</td>
</tr>
<tr id="A9.T5.3.9.9" class="ltx_tr">
<td id="A9.T5.3.9.9.1" class="ltx_td ltx_align_middle" style="width:34.1pt;"></td>
<td id="A9.T5.3.9.9.2" class="ltx_td ltx_align_middle" style="width:39.8pt;"></td>
<td id="A9.T5.3.9.9.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.9.9.3.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.9.9.3.1.1" class="ltx_p"><span id="A9.T5.3.9.9.3.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.9.9.3.1.1.2" class="ltx_text" style="font-size:90%;">64</span></span>
</span>
</td>
<td id="A9.T5.3.9.9.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.9.9.4.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.9.9.4.1.1" class="ltx_p"><span id="A9.T5.3.9.9.4.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.9.9.4.1.1.2" class="ltx_text" style="font-size:90%;">779</span></span>
</span>
</td>
<td id="A9.T5.3.9.9.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.9.9.5.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.9.9.5.1.1" class="ltx_p"><span id="A9.T5.3.9.9.5.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.9.9.5.1.1.2" class="ltx_text" style="font-size:90%;">10.2</span></span>
</span>
</td>
<td id="A9.T5.3.9.9.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:34.1pt;">
<span id="A9.T5.3.9.9.6.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.9.9.6.1.1" class="ltx_p"><span id="A9.T5.3.9.9.6.1.1.1" class="ltx_text" style="font-size:90%;">42056</span></span>
</span>
</td>
<td id="A9.T5.3.9.9.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.9.9.7.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.9.9.7.1.1" class="ltx_p"><span id="A9.T5.3.9.9.7.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.9.9.7.1.1.2" class="ltx_text" style="font-size:90%;">6249</span></span>
</span>
</td>
<td id="A9.T5.3.9.9.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.9.9.8.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.9.9.8.1.1" class="ltx_p"><span id="A9.T5.3.9.9.8.1.1.1" class="ltx_text" style="font-size:90%;">38.3</span></span>
</span>
</td>
<td id="A9.T5.3.9.9.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.9.9.9.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.9.9.9.1.1" class="ltx_p"><span id="A9.T5.3.9.9.9.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.9.9.9.1.1.2" class="ltx_text" style="font-size:90%;">6.36</span></span>
</span>
</td>
<td id="A9.T5.3.9.9.10" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.9.9.10.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.9.9.10.1.1" class="ltx_p"><span id="A9.T5.3.9.9.10.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.9.9.10.1.1.2" class="ltx_text" style="font-size:90%;">1.87</span></span>
</span>
</td>
<td id="A9.T5.3.9.9.11" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.9.9.11.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.9.9.11.1.1" class="ltx_p"><span id="A9.T5.3.9.9.11.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.9.9.11.1.1.2" class="ltx_text" style="font-size:90%;">0.81</span></span>
</span>
</td>
</tr>
<tr id="A9.T5.3.10.10" class="ltx_tr">
<td id="A9.T5.3.10.10.1" class="ltx_td ltx_align_middle" style="width:34.1pt;"></td>
<td id="A9.T5.3.10.10.2" class="ltx_td ltx_align_middle" style="width:39.8pt;"></td>
<td id="A9.T5.3.10.10.3" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.10.10.3.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.10.10.3.1.1" class="ltx_p"><span id="A9.T5.3.10.10.3.1.1.1" class="ltx_text" style="font-size:90%;">512</span></span>
</span>
</td>
<td id="A9.T5.3.10.10.4" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.10.10.4.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.10.10.4.1.1" class="ltx_p"><span id="A9.T5.3.10.10.4.1.1.1" class="ltx_text" style="font-size:90%;">6242</span></span>
</span>
</td>
<td id="A9.T5.3.10.10.5" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.10.10.5.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.10.10.5.1.1" class="ltx_p"><span id="A9.T5.3.10.10.5.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.10.10.5.1.1.2" class="ltx_text" style="font-size:90%;">30.9</span></span>
</span>
</td>
<td id="A9.T5.3.10.10.6" class="ltx_td ltx_align_justify ltx_align_middle" style="width:34.1pt;">
<span id="A9.T5.3.10.10.6.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.10.10.6.1.1" class="ltx_p"><span id="A9.T5.3.10.10.6.1.1.1" class="ltx_text" style="font-size:90%;">41994</span></span>
</span>
</td>
<td id="A9.T5.3.10.10.7" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.10.10.7.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.10.10.7.1.1" class="ltx_p"><span id="A9.T5.3.10.10.7.1.1.1" class="ltx_text" style="font-size:90%;">16568</span></span>
</span>
</td>
<td id="A9.T5.3.10.10.8" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.10.10.8.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.10.10.8.1.1" class="ltx_p"><span id="A9.T5.3.10.10.8.1.1.1" class="ltx_text" style="font-size:90%;">38.2</span></span>
</span>
</td>
<td id="A9.T5.3.10.10.9" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.10.10.9.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.10.10.9.1.1" class="ltx_p"><span id="A9.T5.3.10.10.9.1.1.1" class="ltx_text" style="font-size:90%;">16.78</span></span>
</span>
</td>
<td id="A9.T5.3.10.10.10" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.10.10.10.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.10.10.10.1.1" class="ltx_p"><span id="A9.T5.3.10.10.10.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.10.10.10.1.1.2" class="ltx_text" style="font-size:90%;">6.28</span></span>
</span>
</td>
<td id="A9.T5.3.10.10.11" class="ltx_td ltx_align_justify ltx_align_middle" style="width:28.5pt;">
<span id="A9.T5.3.10.10.11.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.10.10.11.1.1" class="ltx_p"><span id="A9.T5.3.10.10.11.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.10.10.11.1.1.2" class="ltx_text" style="font-size:90%;">2.14</span></span>
</span>
</td>
</tr>
<tr id="A9.T5.3.11.11" class="ltx_tr">
<td id="A9.T5.3.11.11.1" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:34.1pt;">
<span id="A9.T5.3.11.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.11.11.1.1.1" class="ltx_p"><span id="A9.T5.3.11.11.1.1.1.1" class="ltx_text" style="font-size:90%;">Megatron</span></span>
</span>
</td>
<td id="A9.T5.3.11.11.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:39.8pt;">
<span id="A9.T5.3.11.11.2.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.11.11.2.1.1" class="ltx_p"><span id="A9.T5.3.11.11.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:90%;">float32</span></span>
</span>
</td>
<td id="A9.T5.3.11.11.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:28.5pt;">
<span id="A9.T5.3.11.11.3.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.11.11.3.1.1" class="ltx_p"><span id="A9.T5.3.11.11.3.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.11.11.3.1.1.2" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.11.11.3.1.1.3" class="ltx_text" style="font-size:90%;">1</span></span>
</span>
</td>
<td id="A9.T5.3.11.11.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:28.5pt;">
<span id="A9.T5.3.11.11.4.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.11.11.4.1.1" class="ltx_p"><span id="A9.T5.3.11.11.4.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.11.11.4.1.1.2" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.11.11.4.1.1.3" class="ltx_text" style="font-size:90%;">22</span></span>
</span>
</td>
<td id="A9.T5.3.11.11.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:28.5pt;">
<span id="A9.T5.3.11.11.5.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.11.11.5.1.1" class="ltx_p"><span id="A9.T5.3.11.11.5.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.11.11.5.1.1.2" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.11.11.5.1.1.3" class="ltx_text" style="font-size:90%;">8.2</span></span>
</span>
</td>
<td id="A9.T5.3.11.11.6" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:34.1pt;">
<span id="A9.T5.3.11.11.6.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.11.11.6.1.1" class="ltx_p"><span id="A9.T5.3.11.11.6.1.1.1" class="ltx_text" style="font-size:90%;">23014</span></span>
</span>
</td>
<td id="A9.T5.3.11.11.7" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:28.5pt;">
<span id="A9.T5.3.11.11.7.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.11.11.7.1.1" class="ltx_p"><span id="A9.T5.3.11.11.7.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.11.11.7.1.1.2" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.11.11.7.1.1.3" class="ltx_text" style="font-size:90%;">122</span></span>
</span>
</td>
<td id="A9.T5.3.11.11.8" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:28.5pt;">
<span id="A9.T5.3.11.11.8.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.11.11.8.1.1" class="ltx_p"><span id="A9.T5.3.11.11.8.1.1.1" class="ltx_text" style="font-size:90%;">20.6</span></span>
</span>
</td>
<td id="A9.T5.3.11.11.9" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:28.5pt;">
<span id="A9.T5.3.11.11.9.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.11.11.9.1.1" class="ltx_p"><span id="A9.T5.3.11.11.9.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.11.11.9.1.1.2" class="ltx_text" style="font-size:90%;">0.11</span></span>
</span>
</td>
<td id="A9.T5.3.11.11.10" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:28.5pt;">
<span id="A9.T5.3.11.11.10.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.11.11.10.1.1" class="ltx_p"><span id="A9.T5.3.11.11.10.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.11.11.10.1.1.2" class="ltx_text" style="font-size:90%;">2.23</span></span>
</span>
</td>
<td id="A9.T5.3.11.11.11" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_t" style="width:28.5pt;">
<span id="A9.T5.3.11.11.11.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.11.11.11.1.1" class="ltx_p"><span id="A9.T5.3.11.11.11.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.11.11.11.1.1.2" class="ltx_text" style="font-size:90%;">1.61</span></span>
</span>
</td>
</tr>
<tr id="A9.T5.3.12.12" class="ltx_tr">
<td id="A9.T5.3.12.12.1" class="ltx_td ltx_align_middle ltx_border_bb" style="width:34.1pt;"></td>
<td id="A9.T5.3.12.12.2" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb" style="width:39.8pt;">
<span id="A9.T5.3.12.12.2.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.12.12.2.1.1" class="ltx_p"><span id="A9.T5.3.12.12.2.1.1.1" class="ltx_text ltx_font_typewriter" style="font-size:90%;">bfloat16</span></span>
</span>
</td>
<td id="A9.T5.3.12.12.3" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb" style="width:28.5pt;">
<span id="A9.T5.3.12.12.3.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.12.12.3.1.1" class="ltx_p"><span id="A9.T5.3.12.12.3.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.12.12.3.1.1.2" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.12.12.3.1.1.3" class="ltx_text" style="font-size:90%;">1</span></span>
</span>
</td>
<td id="A9.T5.3.12.12.4" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb" style="width:28.5pt;">
<span id="A9.T5.3.12.12.4.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.12.12.4.1.1" class="ltx_p"><span id="A9.T5.3.12.12.4.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.12.12.4.1.1.2" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.12.12.4.1.1.3" class="ltx_text" style="font-size:90%;">17</span></span>
</span>
</td>
<td id="A9.T5.3.12.12.5" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb" style="width:28.5pt;">
<span id="A9.T5.3.12.12.5.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.12.12.5.1.1" class="ltx_p"><span id="A9.T5.3.12.12.5.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.12.12.5.1.1.2" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.12.12.5.1.1.3" class="ltx_text" style="font-size:90%;">5.5</span></span>
</span>
</td>
<td id="A9.T5.3.12.12.6" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb" style="width:34.1pt;">
<span id="A9.T5.3.12.12.6.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.12.12.6.1.1" class="ltx_p"><span id="A9.T5.3.12.12.6.1.1.1" class="ltx_text" style="font-size:90%;">30079</span></span>
</span>
</td>
<td id="A9.T5.3.12.12.7" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb" style="width:28.5pt;">
<span id="A9.T5.3.12.12.7.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.12.12.7.1.1" class="ltx_p"><span id="A9.T5.3.12.12.7.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.12.12.7.1.1.2" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.12.12.7.1.1.3" class="ltx_text" style="font-size:90%;">181</span></span>
</span>
</td>
<td id="A9.T5.3.12.12.8" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb" style="width:28.5pt;">
<span id="A9.T5.3.12.12.8.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.12.12.8.1.1" class="ltx_p"><span id="A9.T5.3.12.12.8.1.1.1" class="ltx_text" style="font-size:90%;">26.9</span></span>
</span>
</td>
<td id="A9.T5.3.12.12.9" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb" style="width:28.5pt;">
<span id="A9.T5.3.12.12.9.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.12.12.9.1.1" class="ltx_p"><span id="A9.T5.3.12.12.9.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.12.12.9.1.1.2" class="ltx_text" style="font-size:90%;">0.27</span></span>
</span>
</td>
<td id="A9.T5.3.12.12.10" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb" style="width:28.5pt;">
<span id="A9.T5.3.12.12.10.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.12.12.10.1.1" class="ltx_p"><span id="A9.T5.3.12.12.10.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.12.12.10.1.1.2" class="ltx_text" style="font-size:90%;">0.92</span></span>
</span>
</td>
<td id="A9.T5.3.12.12.11" class="ltx_td ltx_align_justify ltx_align_middle ltx_border_bb" style="width:28.5pt;">
<span id="A9.T5.3.12.12.11.1" class="ltx_inline-block ltx_align_top">
<span id="A9.T5.3.12.12.11.1.1" class="ltx_p"><span id="A9.T5.3.12.12.11.1.1.1" class="ltx_text ltx_phantom" style="font-size:90%;"><span style="visibility:hidden">0</span></span><span id="A9.T5.3.12.12.11.1.1.2" class="ltx_text" style="font-size:90%;">0.82</span></span>
</span>
</td>
</tr>
</tbody>
</table>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A10" class="ltx_appendix" lang="en">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix J </span>Hyper-parameters</h2>

<div id="A10.p1" class="ltx_para">
<p id="A10.p1.1" class="ltx_p">For each transfer task, we provide the exact hyper-parameters with which the final score and the publicly released checkpoint was obtained.
Tuning was done by looking at a validation-set if available, otherwise on a small “minival” held out from the training set.
Most tuning was done at 224px resolution, and either carried over as-is to higher resolutions, or further tuned only slightly at higher resolution.
Zero-shot evaluations were done on the checkpoint obtained for the fine-tune task and using hyper-parameters selected based on the fine-tune task validation metric. This can lead to worse performance as the hyper-parameters were not selected for generalization to different test distributions.
The full configuration of every transfer task can be found in the provided source-code.</p>
</div>
<figure id="A10.T6" class="ltx_table">
<table id="A10.T6.1" class="ltx_tabular">
<thead class="ltx_thead">
<tr id="A10.T6.1.1.1" class="ltx_tr">
<th id="A10.T6.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" rowspan="2"><span id="A10.T6.1.1.1.1.1" class="ltx_text" style="font-size:70%;">Task</span></th>
<th id="A10.T6.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" rowspan="2"><span id="A10.T6.1.1.1.2.1" class="ltx_text" style="font-size:70%;">Res</span></th>
<th id="A10.T6.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" rowspan="2"><span id="A10.T6.1.1.1.3.1" class="ltx_text" style="font-size:70%;">Epochs</span></th>
<th id="A10.T6.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A10.T6.1.1.1.4.1" class="ltx_text" style="font-size:70%;">Batch</span></th>
<th id="A10.T6.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A10.T6.1.1.1.5.1" class="ltx_text" style="font-size:70%;">Learning</span></th>
<th id="A10.T6.1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A10.T6.1.1.1.6.1" class="ltx_text" style="font-size:70%;">Weight</span></th>
<th id="A10.T6.1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A10.T6.1.1.1.7.1" class="ltx_text" style="font-size:70%;">LLM</span></th>
<th id="A10.T6.1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A10.T6.1.1.1.8.1" class="ltx_text" style="font-size:70%;">Label</span></th>
<th id="A10.T6.1.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A10.T6.1.1.1.9.1" class="ltx_text" style="font-size:70%;">Freeze</span></th>
<th id="A10.T6.1.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A10.T6.1.1.1.10.1" class="ltx_text" style="font-size:70%;">Decode</span></th>
</tr>
<tr id="A10.T6.1.2.2" class="ltx_tr">
<th id="A10.T6.1.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A10.T6.1.2.2.1.1" class="ltx_text" style="font-size:70%;">size</span></th>
<th id="A10.T6.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A10.T6.1.2.2.2.1" class="ltx_text" style="font-size:70%;">rate</span></th>
<th id="A10.T6.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A10.T6.1.2.2.3.1" class="ltx_text" style="font-size:70%;">decay</span></th>
<th id="A10.T6.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A10.T6.1.2.2.4.1" class="ltx_text" style="font-size:70%;">dropout</span></th>
<th id="A10.T6.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A10.T6.1.2.2.5.1" class="ltx_text" style="font-size:70%;">smoothing</span></th>
<th id="A10.T6.1.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A10.T6.1.2.2.6.1" class="ltx_text" style="font-size:70%;">ViT</span></th>
<th id="A10.T6.1.2.2.7" class="ltx_td ltx_th ltx_th_column"></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A10.T6.1.3.1" class="ltx_tr">
<td id="A10.T6.1.3.1.1" class="ltx_td ltx_align_center ltx_border_t"><span id="A10.T6.1.3.1.1.1" class="ltx_text" style="font-size:70%;">COCOcap</span></td>
<td id="A10.T6.1.3.1.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A10.T6.1.3.1.2.1" class="ltx_text" style="font-size:70%;">224</span></td>
<td id="A10.T6.1.3.1.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A10.T6.1.3.1.3.1" class="ltx_text" style="font-size:70%;">5</span></td>
<td id="A10.T6.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A10.T6.1.3.1.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t"><span id="A10.T6.1.3.1.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.3.1.6" class="ltx_td ltx_align_center ltx_border_t"><span id="A10.T6.1.3.1.6.1" class="ltx_text" style="font-size:70%;">1e-06</span></td>
<td id="A10.T6.1.3.1.7" class="ltx_td ltx_align_center ltx_border_t"><span id="A10.T6.1.3.1.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.3.1.8" class="ltx_td ltx_align_center ltx_border_t"><span id="A10.T6.1.3.1.8.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.3.1.9" class="ltx_td ltx_align_center ltx_border_t"><span id="A10.T6.1.3.1.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.3.1.10" class="ltx_td ltx_align_center ltx_border_t"><span id="A10.T6.1.3.1.10.1" class="ltx_text" style="font-size:70%;">beam n=2</span></td>
</tr>
<tr id="A10.T6.1.4.2" class="ltx_tr">
<td id="A10.T6.1.4.2.1" class="ltx_td"></td>
<td id="A10.T6.1.4.2.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.4.2.2.1" class="ltx_text" style="font-size:70%;">448</span></td>
<td id="A10.T6.1.4.2.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.4.2.3.1" class="ltx_text" style="font-size:70%;">5</span></td>
<td id="A10.T6.1.4.2.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.4.2.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.4.2.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.4.2.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.4.2.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.4.2.6.1" class="ltx_text" style="font-size:70%;">1e-06</span></td>
<td id="A10.T6.1.4.2.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.4.2.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.4.2.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.4.2.8.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.4.2.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.4.2.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.4.2.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.4.2.10.1" class="ltx_text" style="font-size:70%;">beam n=2</span></td>
</tr>
<tr id="A10.T6.1.5.3" class="ltx_tr" style="background-color:#F2F2F2;">
<td id="A10.T6.1.5.3.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.5.3.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">COCO-35L</span></td>
<td id="A10.T6.1.5.3.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.5.3.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">224</span></td>
<td id="A10.T6.1.5.3.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.5.3.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">25</span></td>
<td id="A10.T6.1.5.3.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.5.3.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">256</span></td>
<td id="A10.T6.1.5.3.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.5.3.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.5.3.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.5.3.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-06</span></td>
<td id="A10.T6.1.5.3.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.5.3.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.5.3.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.5.3.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.5.3.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.5.3.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.5.3.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.5.3.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.6.4" class="ltx_tr">
<td id="A10.T6.1.6.4.1" class="ltx_td"></td>
<td id="A10.T6.1.6.4.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.6.4.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">448</span></td>
<td id="A10.T6.1.6.4.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.6.4.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">25</span></td>
<td id="A10.T6.1.6.4.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.6.4.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">256</span></td>
<td id="A10.T6.1.6.4.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.6.4.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.6.4.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.6.4.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-06</span></td>
<td id="A10.T6.1.6.4.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.6.4.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.6.4.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.6.4.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.6.4.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.6.4.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.6.4.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.6.4.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.7.5" class="ltx_tr">
<td id="A10.T6.1.7.5.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.7.5.1.1" class="ltx_text" style="font-size:70%;">Screen2Words</span></td>
<td id="A10.T6.1.7.5.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.7.5.2.1" class="ltx_text" style="font-size:70%;">224</span></td>
<td id="A10.T6.1.7.5.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.7.5.3.1" class="ltx_text" style="font-size:70%;">10</span></td>
<td id="A10.T6.1.7.5.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.7.5.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.7.5.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.7.5.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.7.5.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.7.5.6.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.7.5.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.7.5.7.1" class="ltx_text" style="font-size:70%;">0.3</span></td>
<td id="A10.T6.1.7.5.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.7.5.8.1" class="ltx_text" style="font-size:70%;">0.2</span></td>
<td id="A10.T6.1.7.5.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.7.5.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.7.5.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.7.5.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.8.6" class="ltx_tr">
<td id="A10.T6.1.8.6.1" class="ltx_td"></td>
<td id="A10.T6.1.8.6.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.8.6.2.1" class="ltx_text" style="font-size:70%;">448</span></td>
<td id="A10.T6.1.8.6.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.8.6.3.1" class="ltx_text" style="font-size:70%;">10</span></td>
<td id="A10.T6.1.8.6.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.8.6.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.8.6.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.8.6.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.8.6.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.8.6.6.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.8.6.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.8.6.7.1" class="ltx_text" style="font-size:70%;">0.3</span></td>
<td id="A10.T6.1.8.6.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.8.6.8.1" class="ltx_text" style="font-size:70%;">0.2</span></td>
<td id="A10.T6.1.8.6.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.8.6.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.8.6.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.8.6.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.9.7" class="ltx_tr" style="background-color:#F2F2F2;">
<td id="A10.T6.1.9.7.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.9.7.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">TextCaps</span></td>
<td id="A10.T6.1.9.7.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.9.7.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">224</span></td>
<td id="A10.T6.1.9.7.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.9.7.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">5</span></td>
<td id="A10.T6.1.9.7.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.9.7.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">256</span></td>
<td id="A10.T6.1.9.7.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.9.7.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.9.7.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.9.7.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-06</span></td>
<td id="A10.T6.1.9.7.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.9.7.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.9.7.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.9.7.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.9.7.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.9.7.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.9.7.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.9.7.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">beam n=3</span></td>
</tr>
<tr id="A10.T6.1.10.8" class="ltx_tr">
<td id="A10.T6.1.10.8.1" class="ltx_td"></td>
<td id="A10.T6.1.10.8.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.10.8.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">448</span></td>
<td id="A10.T6.1.10.8.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.10.8.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">5</span></td>
<td id="A10.T6.1.10.8.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.10.8.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">256</span></td>
<td id="A10.T6.1.10.8.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.10.8.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.10.8.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.10.8.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-06</span></td>
<td id="A10.T6.1.10.8.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.10.8.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.10.8.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.10.8.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.10.8.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.10.8.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.10.8.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.10.8.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">beam n=3</span></td>
</tr>
<tr id="A10.T6.1.11.9" class="ltx_tr">
<td id="A10.T6.1.11.9.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.11.9.1.1" class="ltx_text" style="font-size:70%;">SciCap</span></td>
<td id="A10.T6.1.11.9.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.11.9.2.1" class="ltx_text" style="font-size:70%;">224</span></td>
<td id="A10.T6.1.11.9.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.11.9.3.1" class="ltx_text" style="font-size:70%;">80</span></td>
<td id="A10.T6.1.11.9.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.11.9.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.11.9.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.11.9.5.1" class="ltx_text" style="font-size:70%;">3e-05</span></td>
<td id="A10.T6.1.11.9.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.11.9.6.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.11.9.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.11.9.7.1" class="ltx_text" style="font-size:70%;">0.1</span></td>
<td id="A10.T6.1.11.9.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.11.9.8.1" class="ltx_text" style="font-size:70%;">0.1</span></td>
<td id="A10.T6.1.11.9.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.11.9.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.11.9.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.11.9.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.12.10" class="ltx_tr">
<td id="A10.T6.1.12.10.1" class="ltx_td"></td>
<td id="A10.T6.1.12.10.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.12.10.2.1" class="ltx_text" style="font-size:70%;">448</span></td>
<td id="A10.T6.1.12.10.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.12.10.3.1" class="ltx_text" style="font-size:70%;">80</span></td>
<td id="A10.T6.1.12.10.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.12.10.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.12.10.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.12.10.5.1" class="ltx_text" style="font-size:70%;">3e-05</span></td>
<td id="A10.T6.1.12.10.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.12.10.6.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.12.10.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.12.10.7.1" class="ltx_text" style="font-size:70%;">0.1</span></td>
<td id="A10.T6.1.12.10.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.12.10.8.1" class="ltx_text" style="font-size:70%;">0.1</span></td>
<td id="A10.T6.1.12.10.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.12.10.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.12.10.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.12.10.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.13.11" class="ltx_tr" style="background-color:#F2F2F2;">
<td id="A10.T6.1.13.11.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.13.11.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">WidgetCap</span></td>
<td id="A10.T6.1.13.11.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.13.11.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">224</span></td>
<td id="A10.T6.1.13.11.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.13.11.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">4</span></td>
<td id="A10.T6.1.13.11.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.13.11.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">64</span></td>
<td id="A10.T6.1.13.11.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.13.11.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3e-06</span></td>
<td id="A10.T6.1.13.11.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.13.11.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3e-07</span></td>
<td id="A10.T6.1.13.11.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.13.11.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.1</span></td>
<td id="A10.T6.1.13.11.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.13.11.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.1</span></td>
<td id="A10.T6.1.13.11.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.13.11.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.13.11.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.13.11.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.14.12" class="ltx_tr">
<td id="A10.T6.1.14.12.1" class="ltx_td"></td>
<td id="A10.T6.1.14.12.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.14.12.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">448</span></td>
<td id="A10.T6.1.14.12.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.14.12.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">4</span></td>
<td id="A10.T6.1.14.12.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.14.12.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">64</span></td>
<td id="A10.T6.1.14.12.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.14.12.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3e-06</span></td>
<td id="A10.T6.1.14.12.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.14.12.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3e-07</span></td>
<td id="A10.T6.1.14.12.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.14.12.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.1</span></td>
<td id="A10.T6.1.14.12.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.14.12.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.1</span></td>
<td id="A10.T6.1.14.12.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.14.12.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.14.12.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.14.12.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.15.13" class="ltx_tr">
<td id="A10.T6.1.15.13.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.15.13.1.1" class="ltx_text" style="font-size:70%;">VQAv2</span></td>
<td id="A10.T6.1.15.13.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.15.13.2.1" class="ltx_text" style="font-size:70%;">224</span></td>
<td id="A10.T6.1.15.13.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.15.13.3.1" class="ltx_text" style="font-size:70%;">10</span></td>
<td id="A10.T6.1.15.13.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.15.13.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.15.13.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.15.13.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.15.13.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.15.13.6.1" class="ltx_text" style="font-size:70%;">1e-06</span></td>
<td id="A10.T6.1.15.13.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.15.13.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.15.13.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.15.13.8.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.15.13.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.15.13.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.15.13.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.15.13.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.16.14" class="ltx_tr">
<td id="A10.T6.1.16.14.1" class="ltx_td"></td>
<td id="A10.T6.1.16.14.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.16.14.2.1" class="ltx_text" style="font-size:70%;">448</span></td>
<td id="A10.T6.1.16.14.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.16.14.3.1" class="ltx_text" style="font-size:70%;">10</span></td>
<td id="A10.T6.1.16.14.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.16.14.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.16.14.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.16.14.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.16.14.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.16.14.6.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.16.14.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.16.14.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.16.14.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.16.14.8.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.16.14.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.16.14.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.16.14.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.16.14.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.17.15" class="ltx_tr" style="background-color:#F2F2F2;">
<td id="A10.T6.1.17.15.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.17.15.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">OKVQA</span></td>
<td id="A10.T6.1.17.15.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.17.15.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">224</span></td>
<td id="A10.T6.1.17.15.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.17.15.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">10</span></td>
<td id="A10.T6.1.17.15.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.17.15.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">128</span></td>
<td id="A10.T6.1.17.15.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.17.15.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">5e-06</span></td>
<td id="A10.T6.1.17.15.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.17.15.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.17.15.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.17.15.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.17.15.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.17.15.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.17.15.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.17.15.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.17.15.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.17.15.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.18.16" class="ltx_tr">
<td id="A10.T6.1.18.16.1" class="ltx_td"></td>
<td id="A10.T6.1.18.16.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.18.16.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">448</span></td>
<td id="A10.T6.1.18.16.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.18.16.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">10</span></td>
<td id="A10.T6.1.18.16.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.18.16.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">128</span></td>
<td id="A10.T6.1.18.16.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.18.16.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">5e-06</span></td>
<td id="A10.T6.1.18.16.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.18.16.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.18.16.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.18.16.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.18.16.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.18.16.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.18.16.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.18.16.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.18.16.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.18.16.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.19.17" class="ltx_tr">
<td id="A10.T6.1.19.17.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.19.17.1.1" class="ltx_text" style="font-size:70%;">AOKVQA-MC</span></td>
<td id="A10.T6.1.19.17.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.19.17.2.1" class="ltx_text" style="font-size:70%;">224</span></td>
<td id="A10.T6.1.19.17.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.19.17.3.1" class="ltx_text" style="font-size:70%;">15</span></td>
<td id="A10.T6.1.19.17.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.19.17.4.1" class="ltx_text" style="font-size:70%;">128</span></td>
<td id="A10.T6.1.19.17.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.19.17.5.1" class="ltx_text" style="font-size:70%;">5e-06</span></td>
<td id="A10.T6.1.19.17.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.19.17.6.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.19.17.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.19.17.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.19.17.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.19.17.8.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.19.17.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.19.17.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.19.17.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.19.17.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.20.18" class="ltx_tr">
<td id="A10.T6.1.20.18.1" class="ltx_td"></td>
<td id="A10.T6.1.20.18.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.20.18.2.1" class="ltx_text" style="font-size:70%;">448</span></td>
<td id="A10.T6.1.20.18.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.20.18.3.1" class="ltx_text" style="font-size:70%;">15</span></td>
<td id="A10.T6.1.20.18.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.20.18.4.1" class="ltx_text" style="font-size:70%;">128</span></td>
<td id="A10.T6.1.20.18.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.20.18.5.1" class="ltx_text" style="font-size:70%;">5e-06</span></td>
<td id="A10.T6.1.20.18.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.20.18.6.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.20.18.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.20.18.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.20.18.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.20.18.8.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.20.18.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.20.18.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.20.18.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.20.18.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.21.19" class="ltx_tr" style="background-color:#F2F2F2;">
<td id="A10.T6.1.21.19.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.21.19.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">AOKVQA-DA</span></td>
<td id="A10.T6.1.21.19.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.21.19.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">224</span></td>
<td id="A10.T6.1.21.19.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.21.19.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">10</span></td>
<td id="A10.T6.1.21.19.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.21.19.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">128</span></td>
<td id="A10.T6.1.21.19.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.21.19.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">5e-06</span></td>
<td id="A10.T6.1.21.19.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.21.19.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.21.19.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.21.19.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.21.19.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.21.19.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.21.19.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.21.19.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.21.19.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.21.19.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.22.20" class="ltx_tr">
<td id="A10.T6.1.22.20.1" class="ltx_td"></td>
<td id="A10.T6.1.22.20.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.22.20.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">448</span></td>
<td id="A10.T6.1.22.20.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.22.20.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">10</span></td>
<td id="A10.T6.1.22.20.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.22.20.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">128</span></td>
<td id="A10.T6.1.22.20.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.22.20.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">5e-06</span></td>
<td id="A10.T6.1.22.20.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.22.20.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.22.20.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.22.20.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.22.20.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.22.20.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.22.20.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.22.20.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.22.20.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.22.20.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.23.21" class="ltx_tr">
<td id="A10.T6.1.23.21.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.23.21.1.1" class="ltx_text" style="font-size:70%;">GQA</span></td>
<td id="A10.T6.1.23.21.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.23.21.2.1" class="ltx_text" style="font-size:70%;">224</span></td>
<td id="A10.T6.1.23.21.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.23.21.3.1" class="ltx_text" style="font-size:70%;">1</span></td>
<td id="A10.T6.1.23.21.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.23.21.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.23.21.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.23.21.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.23.21.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.23.21.6.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.23.21.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.23.21.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.23.21.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.23.21.8.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.23.21.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.23.21.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.23.21.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.23.21.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.24.22" class="ltx_tr">
<td id="A10.T6.1.24.22.1" class="ltx_td"></td>
<td id="A10.T6.1.24.22.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.24.22.2.1" class="ltx_text" style="font-size:70%;">448</span></td>
<td id="A10.T6.1.24.22.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.24.22.3.1" class="ltx_text" style="font-size:70%;">1</span></td>
<td id="A10.T6.1.24.22.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.24.22.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.24.22.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.24.22.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.24.22.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.24.22.6.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.24.22.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.24.22.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.24.22.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.24.22.8.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.24.22.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.24.22.9.1" class="ltx_text" style="font-size:70%;">True</span></td>
<td id="A10.T6.1.24.22.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.24.22.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.25.23" class="ltx_tr" style="background-color:#F2F2F2;">
<td id="A10.T6.1.25.23.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.25.23.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">NLVR2</span></td>
<td id="A10.T6.1.25.23.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.25.23.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">224</span></td>
<td id="A10.T6.1.25.23.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.25.23.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3</span></td>
<td id="A10.T6.1.25.23.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.25.23.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">256</span></td>
<td id="A10.T6.1.25.23.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.25.23.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.25.23.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.25.23.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-06</span></td>
<td id="A10.T6.1.25.23.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.25.23.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.25.23.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.25.23.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.25.23.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.25.23.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.25.23.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.25.23.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.26.24" class="ltx_tr">
<td id="A10.T6.1.26.24.1" class="ltx_td"></td>
<td id="A10.T6.1.26.24.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.26.24.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">448</span></td>
<td id="A10.T6.1.26.24.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.26.24.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">10</span></td>
<td id="A10.T6.1.26.24.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.26.24.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">256</span></td>
<td id="A10.T6.1.26.24.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.26.24.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3e-06</span></td>
<td id="A10.T6.1.26.24.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.26.24.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3e-07</span></td>
<td id="A10.T6.1.26.24.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.26.24.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.26.24.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.26.24.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.26.24.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.26.24.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.26.24.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.26.24.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.27.25" class="ltx_tr">
<td id="A10.T6.1.27.25.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.27.25.1.1" class="ltx_text" style="font-size:70%;">AI2D</span></td>
<td id="A10.T6.1.27.25.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.27.25.2.1" class="ltx_text" style="font-size:70%;">224</span></td>
<td id="A10.T6.1.27.25.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.27.25.3.1" class="ltx_text" style="font-size:70%;">10</span></td>
<td id="A10.T6.1.27.25.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.27.25.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.27.25.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.27.25.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.27.25.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.27.25.6.1" class="ltx_text" style="font-size:70%;">1e-06</span></td>
<td id="A10.T6.1.27.25.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.27.25.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.27.25.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.27.25.8.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.27.25.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.27.25.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.27.25.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.27.25.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.28.26" class="ltx_tr">
<td id="A10.T6.1.28.26.1" class="ltx_td"></td>
<td id="A10.T6.1.28.26.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.28.26.2.1" class="ltx_text" style="font-size:70%;">448</span></td>
<td id="A10.T6.1.28.26.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.28.26.3.1" class="ltx_text" style="font-size:70%;">10</span></td>
<td id="A10.T6.1.28.26.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.28.26.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.28.26.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.28.26.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.28.26.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.28.26.6.1" class="ltx_text" style="font-size:70%;">1e-06</span></td>
<td id="A10.T6.1.28.26.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.28.26.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.28.26.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.28.26.8.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.28.26.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.28.26.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.28.26.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.28.26.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.29.27" class="ltx_tr" style="background-color:#F2F2F2;">
<td id="A10.T6.1.29.27.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.29.27.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">ScienceQA</span></td>
<td id="A10.T6.1.29.27.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.29.27.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">224</span></td>
<td id="A10.T6.1.29.27.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.29.27.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">20</span></td>
<td id="A10.T6.1.29.27.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.29.27.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">128</span></td>
<td id="A10.T6.1.29.27.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.29.27.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.29.27.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.29.27.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.29.27.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.29.27.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.29.27.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.29.27.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.29.27.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.29.27.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">True</span></td>
<td id="A10.T6.1.29.27.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.29.27.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.30.28" class="ltx_tr">
<td id="A10.T6.1.30.28.1" class="ltx_td"></td>
<td id="A10.T6.1.30.28.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.30.28.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">448</span></td>
<td id="A10.T6.1.30.28.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.30.28.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">20</span></td>
<td id="A10.T6.1.30.28.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.30.28.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">128</span></td>
<td id="A10.T6.1.30.28.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.30.28.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.30.28.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.30.28.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.30.28.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.30.28.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.30.28.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.30.28.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.30.28.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.30.28.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">True</span></td>
<td id="A10.T6.1.30.28.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.30.28.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.31.29" class="ltx_tr">
<td id="A10.T6.1.31.29.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.31.29.1.1" class="ltx_text" style="font-size:70%;">RSVQA-lr</span></td>
<td id="A10.T6.1.31.29.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.31.29.2.1" class="ltx_text" style="font-size:70%;">224</span></td>
<td id="A10.T6.1.31.29.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.31.29.3.1" class="ltx_text" style="font-size:70%;">3</span></td>
<td id="A10.T6.1.31.29.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.31.29.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.31.29.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.31.29.5.1" class="ltx_text" style="font-size:70%;">3e-06</span></td>
<td id="A10.T6.1.31.29.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.31.29.6.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.31.29.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.31.29.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.31.29.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.31.29.8.1" class="ltx_text" style="font-size:70%;">0.2</span></td>
<td id="A10.T6.1.31.29.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.31.29.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.31.29.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.31.29.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.32.30" class="ltx_tr">
<td id="A10.T6.1.32.30.1" class="ltx_td"></td>
<td id="A10.T6.1.32.30.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.32.30.2.1" class="ltx_text" style="font-size:70%;">448</span></td>
<td id="A10.T6.1.32.30.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.32.30.3.1" class="ltx_text" style="font-size:70%;">3</span></td>
<td id="A10.T6.1.32.30.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.32.30.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.32.30.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.32.30.5.1" class="ltx_text" style="font-size:70%;">3e-06</span></td>
<td id="A10.T6.1.32.30.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.32.30.6.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.32.30.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.32.30.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.32.30.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.32.30.8.1" class="ltx_text" style="font-size:70%;">0.2</span></td>
<td id="A10.T6.1.32.30.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.32.30.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.32.30.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.32.30.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.33.31" class="ltx_tr" style="background-color:#F2F2F2;">
<td id="A10.T6.1.33.31.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.33.31.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">RSVQA-hr</span></td>
<td id="A10.T6.1.33.31.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.33.31.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">224</span></td>
<td id="A10.T6.1.33.31.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.33.31.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1</span></td>
<td id="A10.T6.1.33.31.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.33.31.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">256</span></td>
<td id="A10.T6.1.33.31.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.33.31.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.33.31.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.33.31.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.33.31.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.33.31.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.33.31.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.33.31.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.33.31.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.33.31.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.33.31.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.33.31.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.34.32" class="ltx_tr">
<td id="A10.T6.1.34.32.1" class="ltx_td"></td>
<td id="A10.T6.1.34.32.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.34.32.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">448</span></td>
<td id="A10.T6.1.34.32.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.34.32.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1</span></td>
<td id="A10.T6.1.34.32.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.34.32.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">256</span></td>
<td id="A10.T6.1.34.32.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.34.32.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.34.32.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.34.32.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.34.32.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.34.32.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.34.32.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.34.32.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.34.32.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.34.32.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.34.32.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.34.32.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.35.33" class="ltx_tr">
<td id="A10.T6.1.35.33.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.35.33.1.1" class="ltx_text" style="font-size:70%;">ChartQA</span></td>
<td id="A10.T6.1.35.33.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.35.33.2.1" class="ltx_text" style="font-size:70%;">224</span></td>
<td id="A10.T6.1.35.33.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.35.33.3.1" class="ltx_text" style="font-size:70%;">30</span></td>
<td id="A10.T6.1.35.33.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.35.33.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.35.33.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.35.33.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.35.33.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.35.33.6.1" class="ltx_text" style="font-size:70%;">1e-06</span></td>
<td id="A10.T6.1.35.33.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.35.33.7.1" class="ltx_text" style="font-size:70%;">0.1</span></td>
<td id="A10.T6.1.35.33.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.35.33.8.1" class="ltx_text" style="font-size:70%;">0.2</span></td>
<td id="A10.T6.1.35.33.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.35.33.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.35.33.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.35.33.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.36.34" class="ltx_tr">
<td id="A10.T6.1.36.34.1" class="ltx_td"></td>
<td id="A10.T6.1.36.34.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.36.34.2.1" class="ltx_text" style="font-size:70%;">448</span></td>
<td id="A10.T6.1.36.34.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.36.34.3.1" class="ltx_text" style="font-size:70%;">30</span></td>
<td id="A10.T6.1.36.34.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.36.34.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.36.34.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.36.34.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.36.34.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.36.34.6.1" class="ltx_text" style="font-size:70%;">1e-06</span></td>
<td id="A10.T6.1.36.34.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.36.34.7.1" class="ltx_text" style="font-size:70%;">0.1</span></td>
<td id="A10.T6.1.36.34.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.36.34.8.1" class="ltx_text" style="font-size:70%;">0.2</span></td>
<td id="A10.T6.1.36.34.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.36.34.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.36.34.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.36.34.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.37.35" class="ltx_tr" style="background-color:#F2F2F2;">
<td id="A10.T6.1.37.35.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.37.35.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">VizWizVQA</span></td>
<td id="A10.T6.1.37.35.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.37.35.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">224</span></td>
<td id="A10.T6.1.37.35.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.37.35.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">10</span></td>
<td id="A10.T6.1.37.35.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.37.35.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">256</span></td>
<td id="A10.T6.1.37.35.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.37.35.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.37.35.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.37.35.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.37.35.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.37.35.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.37.35.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.37.35.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.37.35.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.37.35.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.37.35.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.37.35.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.38.36" class="ltx_tr">
<td id="A10.T6.1.38.36.1" class="ltx_td"></td>
<td id="A10.T6.1.38.36.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.38.36.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">448</span></td>
<td id="A10.T6.1.38.36.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.38.36.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">10</span></td>
<td id="A10.T6.1.38.36.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.38.36.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">256</span></td>
<td id="A10.T6.1.38.36.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.38.36.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.38.36.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.38.36.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.38.36.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.38.36.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.38.36.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.38.36.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.38.36.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.38.36.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.38.36.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.38.36.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.39.37" class="ltx_tr">
<td id="A10.T6.1.39.37.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.39.37.1.1" class="ltx_text" style="font-size:70%;">TallyQA</span></td>
<td id="A10.T6.1.39.37.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.39.37.2.1" class="ltx_text" style="font-size:70%;">224</span></td>
<td id="A10.T6.1.39.37.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.39.37.3.1" class="ltx_text" style="font-size:70%;">2</span></td>
<td id="A10.T6.1.39.37.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.39.37.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.39.37.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.39.37.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.39.37.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.39.37.6.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.39.37.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.39.37.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.39.37.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.39.37.8.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.39.37.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.39.37.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.39.37.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.39.37.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.40.38" class="ltx_tr">
<td id="A10.T6.1.40.38.1" class="ltx_td"></td>
<td id="A10.T6.1.40.38.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.40.38.2.1" class="ltx_text" style="font-size:70%;">448</span></td>
<td id="A10.T6.1.40.38.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.40.38.3.1" class="ltx_text" style="font-size:70%;">2</span></td>
<td id="A10.T6.1.40.38.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.40.38.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.40.38.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.40.38.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.40.38.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.40.38.6.1" class="ltx_text" style="font-size:70%;">1e-06</span></td>
<td id="A10.T6.1.40.38.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.40.38.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.40.38.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.40.38.8.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.40.38.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.40.38.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.40.38.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.40.38.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.41.39" class="ltx_tr" style="background-color:#F2F2F2;">
<td id="A10.T6.1.41.39.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.41.39.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">OCR-VQA</span></td>
<td id="A10.T6.1.41.39.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.41.39.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">224</span></td>
<td id="A10.T6.1.41.39.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.41.39.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3</span></td>
<td id="A10.T6.1.41.39.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.41.39.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">128</span></td>
<td id="A10.T6.1.41.39.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.41.39.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3e-06</span></td>
<td id="A10.T6.1.41.39.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.41.39.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.41.39.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.41.39.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.41.39.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.41.39.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.41.39.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.41.39.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.41.39.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.41.39.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.42.40" class="ltx_tr">
<td id="A10.T6.1.42.40.1" class="ltx_td"></td>
<td id="A10.T6.1.42.40.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.42.40.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">448</span></td>
<td id="A10.T6.1.42.40.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.42.40.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3</span></td>
<td id="A10.T6.1.42.40.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.42.40.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">128</span></td>
<td id="A10.T6.1.42.40.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.42.40.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3e-06</span></td>
<td id="A10.T6.1.42.40.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.42.40.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.42.40.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.42.40.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.42.40.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.42.40.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.42.40.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.42.40.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.42.40.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.42.40.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.43.41" class="ltx_tr">
<td id="A10.T6.1.43.41.1" class="ltx_td"></td>
<td id="A10.T6.1.43.41.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.43.41.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">896</span></td>
<td id="A10.T6.1.43.41.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.43.41.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3</span></td>
<td id="A10.T6.1.43.41.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.43.41.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">128</span></td>
<td id="A10.T6.1.43.41.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.43.41.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.43.41.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.43.41.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.43.41.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.43.41.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.43.41.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.43.41.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.43.41.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.43.41.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.43.41.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.43.41.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.44.42" class="ltx_tr">
<td id="A10.T6.1.44.42.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.44.42.1.1" class="ltx_text" style="font-size:70%;">TextVQA</span></td>
<td id="A10.T6.1.44.42.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.44.42.2.1" class="ltx_text" style="font-size:70%;">224</span></td>
<td id="A10.T6.1.44.42.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.44.42.3.1" class="ltx_text" style="font-size:70%;">5</span></td>
<td id="A10.T6.1.44.42.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.44.42.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.44.42.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.44.42.5.1" class="ltx_text" style="font-size:70%;">3e-06</span></td>
<td id="A10.T6.1.44.42.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.44.42.6.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.44.42.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.44.42.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.44.42.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.44.42.8.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.44.42.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.44.42.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.44.42.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.44.42.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.45.43" class="ltx_tr">
<td id="A10.T6.1.45.43.1" class="ltx_td"></td>
<td id="A10.T6.1.45.43.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.45.43.2.1" class="ltx_text" style="font-size:70%;">448</span></td>
<td id="A10.T6.1.45.43.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.45.43.3.1" class="ltx_text" style="font-size:70%;">10</span></td>
<td id="A10.T6.1.45.43.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.45.43.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.45.43.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.45.43.5.1" class="ltx_text" style="font-size:70%;">3e-06</span></td>
<td id="A10.T6.1.45.43.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.45.43.6.1" class="ltx_text" style="font-size:70%;">3e-07</span></td>
<td id="A10.T6.1.45.43.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.45.43.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.45.43.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.45.43.8.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.45.43.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.45.43.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.45.43.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.45.43.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.46.44" class="ltx_tr">
<td id="A10.T6.1.46.44.1" class="ltx_td"></td>
<td id="A10.T6.1.46.44.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.46.44.2.1" class="ltx_text" style="font-size:70%;">896</span></td>
<td id="A10.T6.1.46.44.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.46.44.3.1" class="ltx_text" style="font-size:70%;">10</span></td>
<td id="A10.T6.1.46.44.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.46.44.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.46.44.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.46.44.5.1" class="ltx_text" style="font-size:70%;">3e-06</span></td>
<td id="A10.T6.1.46.44.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.46.44.6.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.46.44.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.46.44.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.46.44.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.46.44.8.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.46.44.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.46.44.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.46.44.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.46.44.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.47.45" class="ltx_tr" style="background-color:#F2F2F2;">
<td id="A10.T6.1.47.45.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.47.45.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">DocVQA</span></td>
<td id="A10.T6.1.47.45.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.47.45.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">224</span></td>
<td id="A10.T6.1.47.45.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.47.45.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">10</span></td>
<td id="A10.T6.1.47.45.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.47.45.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">256</span></td>
<td id="A10.T6.1.47.45.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.47.45.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.47.45.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.47.45.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-06</span></td>
<td id="A10.T6.1.47.45.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.47.45.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.47.45.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.47.45.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.47.45.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.47.45.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.47.45.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.47.45.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.48.46" class="ltx_tr">
<td id="A10.T6.1.48.46.1" class="ltx_td"></td>
<td id="A10.T6.1.48.46.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.48.46.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">448</span></td>
<td id="A10.T6.1.48.46.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.48.46.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">10</span></td>
<td id="A10.T6.1.48.46.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.48.46.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">256</span></td>
<td id="A10.T6.1.48.46.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.48.46.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.48.46.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.48.46.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-06</span></td>
<td id="A10.T6.1.48.46.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.48.46.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.48.46.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.48.46.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.48.46.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.48.46.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.48.46.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.48.46.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.49.47" class="ltx_tr">
<td id="A10.T6.1.49.47.1" class="ltx_td"></td>
<td id="A10.T6.1.49.47.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.49.47.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">896</span></td>
<td id="A10.T6.1.49.47.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.49.47.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">10</span></td>
<td id="A10.T6.1.49.47.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.49.47.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">256</span></td>
<td id="A10.T6.1.49.47.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.49.47.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.49.47.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.49.47.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-06</span></td>
<td id="A10.T6.1.49.47.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.49.47.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.49.47.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.49.47.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.49.47.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.49.47.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.49.47.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.49.47.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.50.48" class="ltx_tr">
<td id="A10.T6.1.50.48.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.50.48.1.1" class="ltx_text" style="font-size:70%;">InfoVQA</span></td>
<td id="A10.T6.1.50.48.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.50.48.2.1" class="ltx_text" style="font-size:70%;">224</span></td>
<td id="A10.T6.1.50.48.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.50.48.3.1" class="ltx_text" style="font-size:70%;">3</span></td>
<td id="A10.T6.1.50.48.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.50.48.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.50.48.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.50.48.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.50.48.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.50.48.6.1" class="ltx_text" style="font-size:70%;">1e-06</span></td>
<td id="A10.T6.1.50.48.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.50.48.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.50.48.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.50.48.8.1" class="ltx_text" style="font-size:70%;">0.4</span></td>
<td id="A10.T6.1.50.48.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.50.48.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.50.48.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.50.48.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.51.49" class="ltx_tr">
<td id="A10.T6.1.51.49.1" class="ltx_td"></td>
<td id="A10.T6.1.51.49.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.51.49.2.1" class="ltx_text" style="font-size:70%;">448</span></td>
<td id="A10.T6.1.51.49.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.51.49.3.1" class="ltx_text" style="font-size:70%;">3</span></td>
<td id="A10.T6.1.51.49.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.51.49.4.1" class="ltx_text" style="font-size:70%;">128</span></td>
<td id="A10.T6.1.51.49.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.51.49.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.51.49.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.51.49.6.1" class="ltx_text" style="font-size:70%;">1e-06</span></td>
<td id="A10.T6.1.51.49.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.51.49.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.51.49.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.51.49.8.1" class="ltx_text" style="font-size:70%;">0.4</span></td>
<td id="A10.T6.1.51.49.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.51.49.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.51.49.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.51.49.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.52.50" class="ltx_tr">
<td id="A10.T6.1.52.50.1" class="ltx_td"></td>
<td id="A10.T6.1.52.50.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.52.50.2.1" class="ltx_text" style="font-size:70%;">896</span></td>
<td id="A10.T6.1.52.50.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.52.50.3.1" class="ltx_text" style="font-size:70%;">3</span></td>
<td id="A10.T6.1.52.50.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.52.50.4.1" class="ltx_text" style="font-size:70%;">32</span></td>
<td id="A10.T6.1.52.50.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.52.50.5.1" class="ltx_text" style="font-size:70%;">3e-06</span></td>
<td id="A10.T6.1.52.50.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.52.50.6.1" class="ltx_text" style="font-size:70%;">3e-07</span></td>
<td id="A10.T6.1.52.50.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.52.50.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.52.50.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.52.50.8.1" class="ltx_text" style="font-size:70%;">0.4</span></td>
<td id="A10.T6.1.52.50.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.52.50.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.52.50.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.52.50.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.53.51" class="ltx_tr" style="background-color:#F2F2F2;">
<td id="A10.T6.1.53.51.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.53.51.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">ST-VQA</span></td>
<td id="A10.T6.1.53.51.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.53.51.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">224</span></td>
<td id="A10.T6.1.53.51.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.53.51.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3</span></td>
<td id="A10.T6.1.53.51.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.53.51.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">256</span></td>
<td id="A10.T6.1.53.51.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.53.51.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.53.51.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.53.51.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-06</span></td>
<td id="A10.T6.1.53.51.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.53.51.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.53.51.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.53.51.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.1</span></td>
<td id="A10.T6.1.53.51.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.53.51.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.53.51.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.53.51.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.54.52" class="ltx_tr">
<td id="A10.T6.1.54.52.1" class="ltx_td"></td>
<td id="A10.T6.1.54.52.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.54.52.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">448</span></td>
<td id="A10.T6.1.54.52.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.54.52.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3</span></td>
<td id="A10.T6.1.54.52.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.54.52.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">128</span></td>
<td id="A10.T6.1.54.52.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.54.52.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.54.52.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.54.52.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-06</span></td>
<td id="A10.T6.1.54.52.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.54.52.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.54.52.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.54.52.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.1</span></td>
<td id="A10.T6.1.54.52.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.54.52.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.54.52.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.54.52.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.55.53" class="ltx_tr">
<td id="A10.T6.1.55.53.1" class="ltx_td"></td>
<td id="A10.T6.1.55.53.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.55.53.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">896</span></td>
<td id="A10.T6.1.55.53.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.55.53.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3</span></td>
<td id="A10.T6.1.55.53.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.55.53.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">32</span></td>
<td id="A10.T6.1.55.53.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.55.53.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3e-06</span></td>
<td id="A10.T6.1.55.53.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.55.53.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3e-07</span></td>
<td id="A10.T6.1.55.53.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.55.53.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.55.53.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.55.53.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.1</span></td>
<td id="A10.T6.1.55.53.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.55.53.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.55.53.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.55.53.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.56.54" class="ltx_tr">
<td id="A10.T6.1.56.54.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.56.54.1.1" class="ltx_text" style="font-size:70%;">RefCOCO</span></td>
<td id="A10.T6.1.56.54.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.56.54.2.1" class="ltx_text" style="font-size:70%;">224</span></td>
<td id="A10.T6.1.56.54.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.56.54.3.1" class="ltx_text" style="font-size:70%;">100</span></td>
<td id="A10.T6.1.56.54.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.56.54.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.56.54.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.56.54.5.1" class="ltx_text" style="font-size:70%;">3e-05</span></td>
<td id="A10.T6.1.56.54.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.56.54.6.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.56.54.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.56.54.7.1" class="ltx_text" style="font-size:70%;">0.1</span></td>
<td id="A10.T6.1.56.54.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.56.54.8.1" class="ltx_text" style="font-size:70%;">0.3</span></td>
<td id="A10.T6.1.56.54.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.56.54.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.56.54.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.56.54.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.57.55" class="ltx_tr">
<td id="A10.T6.1.57.55.1" class="ltx_td"></td>
<td id="A10.T6.1.57.55.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.57.55.2.1" class="ltx_text" style="font-size:70%;">448</span></td>
<td id="A10.T6.1.57.55.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.57.55.3.1" class="ltx_text" style="font-size:70%;">100</span></td>
<td id="A10.T6.1.57.55.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.57.55.4.1" class="ltx_text" style="font-size:70%;">256</span></td>
<td id="A10.T6.1.57.55.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.57.55.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.57.55.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.57.55.6.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.57.55.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.57.55.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.57.55.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.57.55.8.1" class="ltx_text" style="font-size:70%;">0.3</span></td>
<td id="A10.T6.1.57.55.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.57.55.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.57.55.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.57.55.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.58.56" class="ltx_tr">
<td id="A10.T6.1.58.56.1" class="ltx_td"></td>
<td id="A10.T6.1.58.56.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.58.56.2.1" class="ltx_text" style="font-size:70%;">896</span></td>
<td id="A10.T6.1.58.56.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.58.56.3.1" class="ltx_text" style="font-size:70%;">100</span></td>
<td id="A10.T6.1.58.56.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.58.56.4.1" class="ltx_text" style="font-size:70%;">64</span></td>
<td id="A10.T6.1.58.56.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.58.56.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.58.56.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.58.56.6.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.58.56.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.58.56.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.58.56.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.58.56.8.1" class="ltx_text" style="font-size:70%;">0.3</span></td>
<td id="A10.T6.1.58.56.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.58.56.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.58.56.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.58.56.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.59.57" class="ltx_tr" style="background-color:#F2F2F2;">
<td id="A10.T6.1.59.57.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.59.57.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">ActivityNet-QA</span></td>
<td id="A10.T6.1.59.57.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.59.57.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">224</span></td>
<td id="A10.T6.1.59.57.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.59.57.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1</span></td>
<td id="A10.T6.1.59.57.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.59.57.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">128</span></td>
<td id="A10.T6.1.59.57.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.59.57.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.59.57.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.59.57.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-06</span></td>
<td id="A10.T6.1.59.57.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.59.57.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.59.57.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.59.57.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.59.57.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.59.57.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.59.57.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.59.57.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.60.58" class="ltx_tr">
<td id="A10.T6.1.60.58.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.60.58.1.1" class="ltx_text" style="font-size:70%;">ActivityNet-CAP</span></td>
<td id="A10.T6.1.60.58.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.60.58.2.1" class="ltx_text" style="font-size:70%;">224</span></td>
<td id="A10.T6.1.60.58.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.60.58.3.1" class="ltx_text" style="font-size:70%;">1</span></td>
<td id="A10.T6.1.60.58.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.60.58.4.1" class="ltx_text" style="font-size:70%;">128</span></td>
<td id="A10.T6.1.60.58.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.60.58.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.60.58.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.60.58.6.1" class="ltx_text" style="font-size:70%;">1e-06</span></td>
<td id="A10.T6.1.60.58.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.60.58.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.60.58.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.60.58.8.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.60.58.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.60.58.9.1" class="ltx_text" style="font-size:70%;">True</span></td>
<td id="A10.T6.1.60.58.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.60.58.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.61.59" class="ltx_tr" style="background-color:#F2F2F2;">
<td id="A10.T6.1.61.59.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.61.59.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">MSRVTT-QA</span></td>
<td id="A10.T6.1.61.59.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.61.59.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">224</span></td>
<td id="A10.T6.1.61.59.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.61.59.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1</span></td>
<td id="A10.T6.1.61.59.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.61.59.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">128</span></td>
<td id="A10.T6.1.61.59.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.61.59.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1e-05</span></td>
<td id="A10.T6.1.61.59.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.61.59.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.61.59.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.61.59.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.61.59.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.61.59.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.61.59.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.61.59.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">True</span></td>
<td id="A10.T6.1.61.59.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.61.59.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.62.60" class="ltx_tr">
<td id="A10.T6.1.62.60.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.62.60.1.1" class="ltx_text" style="font-size:70%;">MSRVTT-CAP</span></td>
<td id="A10.T6.1.62.60.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.62.60.2.1" class="ltx_text" style="font-size:70%;">224</span></td>
<td id="A10.T6.1.62.60.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.62.60.3.1" class="ltx_text" style="font-size:70%;">20</span></td>
<td id="A10.T6.1.62.60.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.62.60.4.1" class="ltx_text" style="font-size:70%;">128</span></td>
<td id="A10.T6.1.62.60.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.62.60.5.1" class="ltx_text" style="font-size:70%;">1e-05</span></td>
<td id="A10.T6.1.62.60.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.62.60.6.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.62.60.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.62.60.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.62.60.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.62.60.8.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.62.60.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.62.60.9.1" class="ltx_text" style="font-size:70%;">True</span></td>
<td id="A10.T6.1.62.60.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.62.60.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
<tr id="A10.T6.1.63.61" class="ltx_tr" style="background-color:#F2F2F2;">
<td id="A10.T6.1.63.61.1" class="ltx_td ltx_align_center"><span id="A10.T6.1.63.61.1.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">MSVD-QA</span></td>
<td id="A10.T6.1.63.61.2" class="ltx_td ltx_align_center"><span id="A10.T6.1.63.61.2.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">224</span></td>
<td id="A10.T6.1.63.61.3" class="ltx_td ltx_align_center"><span id="A10.T6.1.63.61.3.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">1</span></td>
<td id="A10.T6.1.63.61.4" class="ltx_td ltx_align_center"><span id="A10.T6.1.63.61.4.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">128</span></td>
<td id="A10.T6.1.63.61.5" class="ltx_td ltx_align_center"><span id="A10.T6.1.63.61.5.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3e-06</span></td>
<td id="A10.T6.1.63.61.6" class="ltx_td ltx_align_center"><span id="A10.T6.1.63.61.6.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">3e-07</span></td>
<td id="A10.T6.1.63.61.7" class="ltx_td ltx_align_center"><span id="A10.T6.1.63.61.7.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.63.61.8" class="ltx_td ltx_align_center"><span id="A10.T6.1.63.61.8.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">0.0</span></td>
<td id="A10.T6.1.63.61.9" class="ltx_td ltx_align_center"><span id="A10.T6.1.63.61.9.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">False</span></td>
<td id="A10.T6.1.63.61.10" class="ltx_td ltx_align_center"><span id="A10.T6.1.63.61.10.1" class="ltx_text" style="font-size:70%;background-color:#F2F2F2;">greedy</span></td>
</tr>
<tr id="A10.T6.1.64.62" class="ltx_tr">
<td id="A10.T6.1.64.62.1" class="ltx_td ltx_align_center ltx_border_bb"><span id="A10.T6.1.64.62.1.1" class="ltx_text" style="font-size:70%;">VATEX</span></td>
<td id="A10.T6.1.64.62.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="A10.T6.1.64.62.2.1" class="ltx_text" style="font-size:70%;">224</span></td>
<td id="A10.T6.1.64.62.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="A10.T6.1.64.62.3.1" class="ltx_text" style="font-size:70%;">10</span></td>
<td id="A10.T6.1.64.62.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="A10.T6.1.64.62.4.1" class="ltx_text" style="font-size:70%;">128</span></td>
<td id="A10.T6.1.64.62.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="A10.T6.1.64.62.5.1" class="ltx_text" style="font-size:70%;">3e-06</span></td>
<td id="A10.T6.1.64.62.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="A10.T6.1.64.62.6.1" class="ltx_text" style="font-size:70%;">3e-07</span></td>
<td id="A10.T6.1.64.62.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="A10.T6.1.64.62.7.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.64.62.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="A10.T6.1.64.62.8.1" class="ltx_text" style="font-size:70%;">0.0</span></td>
<td id="A10.T6.1.64.62.9" class="ltx_td ltx_align_center ltx_border_bb"><span id="A10.T6.1.64.62.9.1" class="ltx_text" style="font-size:70%;">False</span></td>
<td id="A10.T6.1.64.62.10" class="ltx_td ltx_align_center ltx_border_bb"><span id="A10.T6.1.64.62.10.1" class="ltx_text" style="font-size:70%;">greedy</span></td>
</tr>
</tbody>
</table>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A11" class="ltx_appendix" lang="en">
<h2 class="ltx_title ltx_title_appendix" style="font-size:70%;">
<span class="ltx_tag ltx_tag_appendix">Appendix K </span>Full per-task results of ablations</h2>

<section id="A11.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsection">K.1 </span>Pretraining duration</h3>

<div id="A11.SS1.p1" class="ltx_para">
<p id="A11.SS1.p1.1" class="ltx_p"><span id="A11.SS1.p1.1.1" class="ltx_text" style="font-size:70%;">See Figure </span><span class="ltx_ref ltx_missing_label ltx_ref_self" style="font-size:70%;">LABEL:fig:app:pt_duration</span><span id="A11.SS1.p1.1.2" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
<figure id="A11.SS1.1" class="ltx_figure"><img src="/html/2407.07726/assets/x16.png" id="A11.SS1.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="308" alt="[Uncaptioned image]">
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A11.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsection">K.2 </span>Masking and learning objective</h3>

<div id="A11.SS2.p1" class="ltx_para">
<p id="A11.SS2.p1.1" class="ltx_p"><span id="A11.SS2.p1.1.1" class="ltx_text" style="font-size:70%;">See Figure </span><span class="ltx_ref ltx_missing_label ltx_ref_self" style="font-size:70%;">LABEL:fig:app:learning</span><span id="A11.SS2.p1.1.2" class="ltx_text" style="font-size:70%;"> and Figure </span><span class="ltx_ref ltx_missing_label ltx_ref_self" style="font-size:70%;">LABEL:fig:app:learning_prefix</span><span id="A11.SS2.p1.1.3" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
<figure id="A11.SS2.1" class="ltx_figure"><img src="/html/2407.07726/assets/x17.png" id="A11.SS2.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="308" alt="[Uncaptioned image]">
</figure>
<figure id="A11.SS2.2" class="ltx_figure"><img src="/html/2407.07726/assets/x18.png" id="A11.SS2.2.g1" class="ltx_graphics ltx_img_landscape" width="461" height="308" alt="[Uncaptioned image]">
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A11.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsection">K.3 </span>To freeze or not to freeze?</h3>

<div id="A11.SS3.p1" class="ltx_para">
<p id="A11.SS3.p1.1" class="ltx_p"><span id="A11.SS3.p1.1.1" class="ltx_text" style="font-size:70%;">See Figure </span><span class="ltx_ref ltx_missing_label ltx_ref_self" style="font-size:70%;">LABEL:fig:app:freeze</span><span id="A11.SS3.p1.1.2" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
<figure id="A11.SS3.1" class="ltx_figure"><img src="/html/2407.07726/assets/x19.png" id="A11.SS3.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="308" alt="[Uncaptioned image]">
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A11.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsection">K.4 </span>Image encoder: with or without?</h3>

<div id="A11.SS4.p1" class="ltx_para">
<p id="A11.SS4.p1.1" class="ltx_p"><span id="A11.SS4.p1.1.1" class="ltx_text" style="font-size:70%;">See Figure </span><span class="ltx_ref ltx_missing_label ltx_ref_self" style="font-size:70%;">LABEL:fig:app:enc</span><span id="A11.SS4.p1.1.2" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
<figure id="A11.SS4.1" class="ltx_figure"><img src="/html/2407.07726/assets/x20.png" id="A11.SS4.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="308" alt="[Uncaptioned image]">
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A11.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsection">K.5 </span>Resolution or sequence length?</h3>

<div id="A11.SS5.p1" class="ltx_para">
<p id="A11.SS5.p1.1" class="ltx_p"><span id="A11.SS5.p1.1.1" class="ltx_text" style="font-size:70%;">See Figure </span><span class="ltx_ref ltx_missing_label ltx_ref_self" style="font-size:70%;">LABEL:fig:app:res_or_seqlen</span><span id="A11.SS5.p1.1.2" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
<figure id="A11.SS5.1" class="ltx_figure"><img src="/html/2407.07726/assets/x21.png" id="A11.SS5.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="308" alt="[Uncaptioned image]">
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A11.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsection">K.6 </span>Resolution-specific checkpoints and windowing</h3>

<div id="A11.SS6.p1" class="ltx_para">
<p id="A11.SS6.p1.1" class="ltx_p"><span id="A11.SS6.p1.1.1" class="ltx_text" style="font-size:70%;">See Figure </span><span class="ltx_ref ltx_missing_label ltx_ref_self" style="font-size:70%;">LABEL:fig:app:res_window</span><span id="A11.SS6.p1.1.2" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
<figure id="A11.SS6.1" class="ltx_figure"><img src="/html/2407.07726/assets/x22.png" id="A11.SS6.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="308" alt="[Uncaptioned image]">
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A11.SS7" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsection">K.7 </span>Stage2 mixture re-weighting</h3>

<div id="A11.SS7.p1" class="ltx_para">
<p id="A11.SS7.p1.1" class="ltx_p"><span id="A11.SS7.p1.1.1" class="ltx_text" style="font-size:70%;">See Figure </span><span class="ltx_ref ltx_missing_label ltx_ref_self" style="font-size:70%;">LABEL:fig:app:s2_reweight</span><span id="A11.SS7.p1.1.2" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
<figure id="A11.SS7.1" class="ltx_figure"><img src="/html/2407.07726/assets/x23.png" id="A11.SS7.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="195" alt="[Uncaptioned image]">
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A11.SS8" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsection">K.8 </span>Transfer with simple hyper parameters</h3>

<div id="A11.SS8.p1" class="ltx_para">
<p id="A11.SS8.p1.1" class="ltx_p"><span id="A11.SS8.p1.1.1" class="ltx_text" style="font-size:70%;">See Figure </span><span class="ltx_ref ltx_missing_label ltx_ref_self" style="font-size:70%;">LABEL:fig:app:transfer_simple</span><span id="A11.SS8.p1.1.2" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
<figure id="A11.SS8.1" class="ltx_figure"><img src="/html/2407.07726/assets/x24.png" id="A11.SS8.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="308" alt="[Uncaptioned image]">
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A11.SS9" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection" style="font-size:70%;">
<span class="ltx_tag ltx_tag_subsection">K.9 </span>Transfer with limited examples</h3>

<div id="A11.SS9.p1" class="ltx_para">
<p id="A11.SS9.p1.1" class="ltx_p"><span id="A11.SS9.p1.1.1" class="ltx_text" style="font-size:70%;">See Figure </span><span class="ltx_ref ltx_missing_label ltx_ref_self" style="font-size:70%;">LABEL:fig:app:transfer_lowdata</span><span id="A11.SS9.p1.1.2" class="ltx_text" style="font-size:70%;">.</span></p>
</div>
<figure id="A11.SS9.1" class="ltx_figure"><img src="/html/2407.07726/assets/x25.png" id="A11.SS9.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="308" alt="[Uncaptioned image]">
</figure>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.07725" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.07726" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.07726">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.07726" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.07727" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 16:21:39 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.07726
PaliGemma is an open Vision-Language Model (VLM)  that is based on the SigLIP-So400m vision encoder and the Gemma-2B language model.
It is trained to be a versatile and broadly knowledgeable base model that is effective to transfer. It achieves strong performance on a wide variety of open-world tasks.
We evaluate PaliGemma on almost 40 diverse tasks including standard VLM benchmarks, but also more specialized tasks such as remote-sensing and segmentation.
abstract
PaliGemma: A versatile 3B VLM for transfer
Abstract
2407.07726
1IntroductionFigure 1:PaliGemma’s architecture: a SigLIP image encoder feeds into a Gemma decoder LM.PaliGemma is an open model, continuing the line of PaLI vision-language models in a combination with the Gemma family of language models.PaLI is a series of state-of-the-art vision-language models, starting with the first PaLIshowing promising scaling results up to 17 B, using classification pretrained ViTand mT5language model.
PaLI-Xand PaLM-Ethen pushed this further, combining ViT-22 Band a 32 B UL2language model or the 540 B PaLMlanguage model, respectively, and getting further increased performance on vision-language tasks, albeit saturating performance on standard image classification and retrieval tasks.
Finally, PaLI-3demonstrates that through better pretraining with SigLIPand more careful multimodal data curation, a 2 B vision and 3 B language model (i.e.a 5 B vision-language model)  matches the 10x larger PaLI-X and 100x larger PaLM-E across most benchmarks.PaliGemma continues this trend, combining the 400 M SigLIP and the 2 B Gemma modelsinto a sub-3 B VLM that still maintains performance comparable to PaLI-X, PaLM-E, and PaLI-3.Gemmais a family of auto-regressive decoder-only open large language models built from the same research and technology used to create the Geminimodels.
The models come in different sizes (2 B, 7 B) , both pretrained and instruction fine-tuned. PaliGemma uses the 2 B pretrained version.The main goal of our work is to provide a versatile base VLM.
Hence, we show that it reaches state-of-the-art results not only on standard COCO captions, VQAv2, InfographicVQA and others, but also on more exotic Remote-Sensing VQA, TallyVQA, several video captioning and QA tasks, as well as referring expressionsegmentation(see full task list in AppendixB) .
introduction
PaliGemma: A versatile 3B VLM for transfer
1Introduction
2407.07726
2Related workOver the course of the past few years, vision-language models have gained considerable importance in computer vision.
The first generation, spearheaded by CLIPand ALIGNby scaling up ConVIRTand VirTex, is an extension of large-scale classification pretraining, to leverage all data from the web without the need for onerous human labeling, replacing a fixed and large set of classes by a caption embedding instead.
The caption embeddings are mostly obtained using language encoders (similar to BERT)  and allow to open up the vocabulary of classification and retrieval tasks.
The second generation, akin to T5in language, is a unification of captioning and question-answering tasks via generative encoder-decoder modeling, often backed by the progress in generative language models.
These were then further scaled up by, among others, Flamingo, BLIP-2and, PaLI.
Finally, most recent worksperform an additional “instruction tuning” step that is intended to make the raw model more user-friendly.
In addition to building systems, several recent more systematic studiesaim to find out what really matters in VLMs.
PaliGemma is an open base VLM without instruction tuning, and this report answers a few more questions regarding what matters.
More discussion in AppendixA.
body
PaliGemma: A versatile 3B VLM for transfer
2Related work
2407.07726
3ModelIn this section we present details about PaliGemma’s architecture and training.
Several of our decisions are further ablated in Section5.At a high level, PaliGemma is a VLM, taking as input one or more images, and a textual description of the task (the prompt or question, which we often refer to as theprefix) .
PaliGemma then autoregressively generates a prediction in the form of a text string (the answer, which we often refer to as thesuffix) .This simple image+text in, text out API is flexible enough to cover many standard tasks, such as image classification, captioning, visual question-answering and dialogue.
Additionally, as shown in the literature, by converting more complex structured outputs into “text”, this API can also cover more tasks such as:
detection, instance segmentation, panoptic segmentation, depth prediction, colorization, and many more.
This conversion can be hand-engineered and task-specific, such as done in pix2seqfor detection, or learned as is the case for segmentationand dense output tasks in general.During PaliGemma’s pretraining, we limit ourselves to “text” covering natural language, object detection, and instance segmentation, but this API remains versatile and the pretrained models can be finetuned for other output types.3.1ArchitecturePaliGemma consists of three components:•An image encoder, for which we use a publicly available SigLIPcheckpoint, specifically the “shape optimized”ViT-So400m image encoder. This model was contrastively pretrained at large scale via the sigmoid loss, and has shown state-of-the-art performance, especially for its small size.•A decoder-only language model, for which we use the publicly available Gemma-2B v1.0raw pretrained checkpoint, which strikes a great balance between performance and size.
As we will show, this language model is good enough to match or surpass the performance of VLMs using much larger language models, including previous PaLIs.•A linear layer projecting SigLIP’s output tokens into the same dimensions as Gemma-2B’s vocab tokens, so they can be concatenated.
In early experiments, we found that more complicated alternatives (e.g.MLPs)  do not provide a clear advantage, and hence decided to use the simplest option (Sec5.5) .The image is passed through the image encoder, which turns it into a sequence oftokens.
The text is converted intotokens using Gemma’s SentencePiecetokenizer, and embedded with Gemma’s vocabulary embedding layer.
The image tokens are projected with the (zero initialized)  linear projection.
Then the sequence of input tokens to the decoder is created as follows (and also as visible in Figure2) :tokens = [image tokens...,
            BOS, prefix tokens..., SEP,
            suffix tokens..., EOS, PAD...]Figure 2:PaliGemma’s Prefix-LM masking: block attention throughout image and prefix, autoregressive attention on the suffix. Each square indicates whether the row can attend to the column.We always resize the image to a fixed square size (224, 448, or 896 pixels) .
This leads to a fixed number of image tokens per model variant (respectively 256, 1024, or 4096 tokens) , which we place in the front, making image tokens straightforward to interpret without the need for special location markers.
The BOS token then marks the start of text tokens.
We use\nas SEP token, it does not appear in any of our prefixes.
We also tokenize SEP separately to avoid it being merged (by the tokenizer)  with either the end of the prefix or the beginning of the suffix.
In order to maximize model capacity for such a small model, we have full (unmasked)  attention on the whole input,i.e.the image and prefix tokens.
In this way, image tokens can also "lookahead" at the task at hand (prefix)  in order to update their representation.
The suffix is our output and necessarily covered by an auto-regressive mask, including the PAD tokens.
When we mention sequence length () , we typically mean prefix and suffix combined, ignoring image tokens.3.2PretrainingThe training of PaliGemma follows the same steps as previous PaLI models, with only small modifications. Training consists of several stages, which we detail in this section:•Stage0:Unimodal pretraining - we use existing off-the-shelf components.•Stage1:Multimodal pretraining - long pretraining on a carefully chosen mixture of multimodal tasks. Notably, nothing is frozen.•Stage2:Resolution increase - short continued pretraining at higher resolution.•Stage3:Transfer - turn the base model into a task-specific specialist.3.2.1Stage0: Unimodal pretrainingFirst, the unimodal components of the model are pretrained individually, in order to benefit from their well-studied and scaled training recipes.
For PaliGemma specifically, we do not perform any custom unimodal pretraining, instead relying on existing publicly available checkpoints.Following PaLI-3’s strong experimental results, we use a SigLIP image encoder.
While PaLI-3 (and others)  use a large image model such as ViT-G, we use the much smaller but similarly strong “shape optimized” ViT-So400m model.PaLI traditionally uses an encoder-decoder language model; however all recently publicly released language models are decoder-only Transformers.
We opt for the Gemma-2B model, which strikes a good balance between size and performance.
Larger language models, such as the popular 7 B or 70 B sizes, are often significantly better at tasks like mathematical reasoning.
However, PaLI-3 has shown that across a wide range of vision-language tasks, a well-trained small 5 B model (2 B vision + 3 B language)  can attain the same performance as the much larger 55 B PaLI-X (22 B vision + 32 B language)  and 562 B PaLM-E (22 B vision + 540 B language) , including tasks such as ScienceQA.
With PaliGemma we continue this push for smaller models and show that we can keep the same performance with less than 3 B total parameters.3.2.2Stage1: Multimodal pretrainingIn this stage, we combine the unimodal models as explained in Section3.1and train the whole model on a broad mixture of large-scale vision-language tasks.
Contrary to most recent VLMs, our core goal is to train a base model that fine-tunes well to a wide range of tasks, not merely to align the modalities.
Intuitively, we want a mix of tasks which force the model to acquire a broad range of “skills”, regardless of the task’s user (or benchmark)  friendliness out of the box.
More on this in Section3.2.5.It is common practice, also followed by previous PaLI versions, to keep the image encoder frozen during the first multimodal pretraining stage. This is partially due to findings as in LiTreporting multimodal tuning of pretrained image encoders degrading their representations.
However, more recent work such as CapPaand LocCahave shown that captioning and other harder-to-learn tasks can provide valuable signal to image encoders, allowing them to learn spatial and relational understanding capabilities which contrastive models like CLIP or SigLIP typically lack.
Hence, again in the spirit of learning more skills during pretraining, we depart from common practice and do not freeze the image encoder.
However, the challenges outlined in LiT remain.
In order to avoid destructive supervision signal from the initially unaligned language model,
we use a slow linear warm-up for the image encoder’s learning-rate (Figure3) , which ensures that the image encoder’s quality is not deteriorated from the initially misaligned gradients coming through the LLM.We train Stage1 at resolution 224px (hence,image tokens)  and sequence lengthfor a total of 1 billion examples.
While we provide an ablation in Section5.1showing that a 10x to 30x shorter Stage1 still provides good results on popular benchmarks,
we wish to embue as much visual knowledge to the base model as possible, and cover a broad set of concepts, cultures, and languages.3.2.3Stage2: Resolution increaseThe model resulting from Stage1 is already a useful base model for many tasks (see example images in AppendixB) .
However, it only understands images atpixel resolution, which is too small for several tasks.
For instance, detection and segmentation of smaller objects, and tasks related to reading smaller texts such as charts, infographics, or documents, all strongly benefit from higher resolution (see Table1) .
Hence, we train two further model checkpoints for increased resolution, first toand then topixel resolution.Since stage1 took care of providing the model with a broad set of knowledge and skill, stage2 can focus on extending the model’s ability to parse higher-resolution images.
We thus run Stage2 with fewer total examples, while increasing the cost and information density of each example.
For resolution 448, we train for an additional 50 M examples, and for resolution 896, we add another 10 M examples.For simplicity, Stage2 consists of the exact same mixture of tasks and datasets as Stage1, but with significantly increased sampling of tasks that require high resolution.
Additionally, these upweighted tasks all can be modified to provide much longer suffix sequence lengths.
For instance, for OCR tasks, we can simply request the model to readalltext on the image in left-to-right, top-to-bottom order.
For detection and segmentation tasks, we can request the model to detect or segmentallobjects for which annotation is provided.
Hence, we also increase the text sequence length totokens.While PaLI has always had this resolution increasing stage, and for image classification the importance of resolution is long known, several recent workshave raised the importance of resolution in VLMs too.
We add to this body of knowledge by providing several ablation studies regarding Stage2 in Section5.7.3.2.4Stage3: TransferThe result of Stages 1 and 2 is a family of three PaliGemma checkpoints, at 224px, 448px, and 896px resolution,
which are pre-equipped with broad visual knowledge.
However, these checkpoints are not “user (or benchmark)  friendly” as their pretraining has focused solely on density of learning signal, as opposed to usable interface.These base models need to be transferred to serve their intended final purpose.
That could take the form of fine-tuning on a specific, specialized task, such as COCO Captions, Remote Sensing VQA, Video Captioning, or InfographicQA. Adapt to new inputs such as multiple images (NLVR2)  or bounding boxes draw in the image (WidgetCap) .
Or it could take the form of instructionor even chattuning.To show the effectiveness of the base models, we transfer them to a wide range of individual academic benchmarks, using a simple unified transfer recipe with few hyper-parameters.
And to showcase the versatility beyond academic tasks, we also provide a “mix” transfer checkpoint, which transfers to a subset of these tasks at the same time, along with detailed captioning and long question-answering data.
While this is not instruction tuning, it is a step in that direction.We also transfer PaliGemma to tasks which take multiple images as input.
NLVR2 is one such task, which asks one question about two images, and requires looking at both to give the correct answer.
Other such tasks are standard short-video understanding tasks subsampled to 16 frames.
In all these cases, we follow PaLI-3 and encode each image separately, then concatenate the image tokens without any special separator or embedding tokens.
Thus, 16 frames at 224px resolution result inimage tokens, the same amount as a single image at 896px resolution.For all transfers, we perform fine-tuning of all the model parameters.
The hyper-parameters we modify per-task are the following, in decreasing order of importance:•Resolution (i.e.checkpoint) :224, 448, 896.•Epochs:1, 3, 10, 30, 100.•Learning-rate: 3e-5,1e-5, 3e-6.•Label-smoothing:0.0, 0.1, 0.3.•Dropout in the LLM:0.0, 0.1, 0.3.•Weight decay:0.0or 0.1learning-rate.•Freeze ViT:false, true.•Beam-search may benefit captioning.The above are typical values we suggest exploring, with the recommended initial attempt value in bold.
We provide the best setting for each individual task in AppendixJ.
We study the sensitivity to transfer hyper-parameters in Section6.2, and the “transferability” in general in Section6, showing that good results can be achieved with the aforementioned initial attempt values.3.2.5Pretraining task mixtureJust like for previous PaLI models, the pretraining (Stage1 and Stage2)  is designed to result in a model that transfers well, not necessarily a model that is usable out of the box (“0 shot”) .
The intuition here is that we want a mix of tasks which force the model to acquire a broad range of “skills”.
We prefix each task with its unique prefix to avoid conflicting learning signals across skills.
At transfer time (Stage3) , the model then merely needs to recognize which skill is useful for the task, and rewire itself to use that while following the output syntax and vocabulary of the task.
In our experience, these can all be done relatively quickly and based on few examples (Section6.3) .
We do not use any of our transfer datasets during pretraining, and furthermore remove all near-duplicates of their images from the pretraining datasets.Largely following previous PaLI works, these are the pretraining tasks:caption {lang}We include the simple captioning objective on various datasets, including WebLI in over 100 languages, and CC3M-35L. Previous PaLIs use an encoder-decoder language model with the SplitCap objective, however for PaliGemma with the decoder-only language model, plain captioning is a more informative and simpler objective.ocrConcatenation (in raster order)  of all text on the image transcribed by a public OCR system. Potentially skipping random snippets of OCR in order to fit sequence length without biasing recognition towards the beginning of raster order.answer en {question}Generated VQA on CC3M-35L followingwith questions in 35 languages but English answers.
Additionally, English-only object-centric questions on OpenImages following:listing:What objects are in the image?,presence:Is {thing} in the image?,multi-object presence:Which of {thing}, {thing}... are in the image?,and newly, counting:How many {thing}?.question {lang} {English answer}Generated VQG on CC3M-35L followinggenerating questions in 35 languages, for a given English answer.detect {thing} ; {thing} ; ...Multi-object detection similar to Pix2Seqon generated open-world data via pseudo-labeling as described in OWL-ViTv2.segment {thing} ; {thing} ; ...Multi-object instance segmentation as in PaLI-3on generated open-world data similar to OWL-ViTv2and SAM.caption <ymin><xmin><ymax><xmax>Grounded captioning of what is in the box, following LocCa. The box is indicated by the same location tokens as used in detection and segmentation: normalized image coordinates binned to 1024 tokens.Notably distinct from the widely used LLaVa’s GPT-4 generated instruction following data, none of PaliGemma’s pretraining tasks is the output of a larger commercial VLM.Finally, we believe that it is important to detect and remove all images in our pretraining datasets which are near-duplicates of images in the transfer tasks we evaluate in this report, as well as a few more popular computer vision benchmarks.
Doing so, we more accurately capture PaliGemma’s capability to transfer to new tasks.3.2.6Other pretraining detailsFigure 3:Learning-rate schedule across stages.Throughout pretraining, we use an "infinite" learning-rate schedule following, which provides a straightforward way of chaining several stages without decaying the learning-rate between them.
Figure3shows the full schedule: pretraining is one continuous rsqrt curve for all stages.
The transfer can then act as a cooldown, fully annealing the learning rate.
We recommend transferring with a simple setup that tunes the full model using a cosine learning-rate schedule with a short linear warm-up and decaying to zero.
This is not well represented by Figure3due to its comparatively short duration.The model was entirely trained in the open-sourcebig_visioncodebaseon Cloud TPUv5e. However, some of the pretraining datasets remain private.
During training, we partition data, as well as model parameters and optimizer state (Zero-DP style)  across all available devices using JAXwith GSPMD. This fully-sharded data-parallel (FSDP)  sharding strategy is achieved by constructing global arrays and annotating the sharding accordingly, with the XLA compilertaking care of the concrete implementation of the computation and communication between devices.
We measured a model FLOPS utilization (MFU)  of 55%, resulting in 5189 tokens/second/device. Model parameters and optimizer state are kept in float32 to guarantee stable training, but we verified that inference works just as well with bfloat16 model parameters.One training run of the final PaliGemma model using TPUv5e-256 takes slightly less than 3 days for Stage1 and 15h for each Stage2. Stage1 sees slightly less than 350 B tokens, and both Stage2 combined about 90 B tokens. Transfers take between 20min and 10h on TPUv3-32, depending on the task.In order to avoid model brittleness to different image processing details in different frameworks, we randomize the image preprocessing details such as resize method, JPEG encoding, and apply very slightinception_crop.
body
PaliGemma: A versatile 3B VLM for transfer
3Model
2407.07726
4ResultsTable 1:Results (1 random run of 5)  obtained with PaliGemma.
Tasks marked withindicate zero-shot evaluation of the transferred model above. Where numbers depend on server submissions we report standard deviation from validation splits. Highlighted rows indicate resolution sensitive tasks. Per-task details and hyper-parameters are in AppendixBandJ.Task224px448px896pxImage captioningCOCOcap141.9±0.3144.6±0.5-NoCaps121.7±0.3123.6±0.7-COCO-35L (en) 139.2±0.4141.2±0.6-COCO-35L (avg34) 113.7±0.3115.8±0.1-XM3600 (en) 078.0±0.8080.0±0.3-XM3600 (avg35) 041.9±0.0042.4±0.1-Screen2Words117.6±0.7119.6±0.7-TextCaps127.5±1.0153.9±0.5-SciCap162.3±0.7181.5±0.6-WidgetCap136.1±1.4148.4±0.3-Visual question answeringVQAv2083.2±0.4085.6±0.2-MMVP047.3±0.0045.3±0.0-POPE086.0±0.0087.0±0.0-Objaverse Multiview062.7±0.0062.8±0.0-OKVQA063.5±0.3063.2±0.2-AOKVQA-MC076.4±0.4076.9±0.3-AOKVQA-DA061.9±0.6063.2±0.5-GQA065.6±0.4067.0±0.3-xGQA (avg7) 057.3±0.2057.9±0.5-NLVR2090.0±0.2088.9±0.3-MARVL (avg5) 080.6±0.3076.8±0.3-AI2D072.1±0.7073.3±0.7-ScienceQA095.4±0.3095.9±0.2-RSVQA-lr092.6±0.3093.1±0.7-RSVQA-hr (test) 092.6±0.1092.8±0.1-RSVQA-hr (test2) 090.6±0.1090.5±0.2-ChartQA (human) 040.0±0.7054.2±1.1-ChartQA (aug) 074.2±0.2088.5±0.3-VizWizVQA073.7±0.2075.5±0.5-TallyQA (simple) 081.7±0.2084.9±0.1-TallyQA (complex) 069.6±0.1072.3±0.2-CountBenchQA081.9±1.6083.1±2.1-OCR-VQA072.3±0.4074.6±0.4074.9TextVQA055.5±0.2073.2±0.2076.5DocVQA043.7±0.5078.0±0.3084.8InfoVQA028.5±0.4040.5±0.3047.8ST-VQA063.3±0.5081.8±0.3084.4Image segmentationRefCOCO (testA) 075.7±0.1077.9±0.1078.7RefCOCO (testB) 070.7±0.2072.4±0.2073.9RefCOCO+ (testA) 071.9±0.1074.2±0.2076.1RefCOCO+ (testB) 064.5±0.6064.5±0.2066.9RefCOCOg (test) 068.2±0.1071.0±0.1072.7Video inputActivityNet-QA050.8±0.4-±0.0-ActivityNet-CAP034.6±0.6-±0.0-MSRVTT-QA050.1±0.1-±0.0-MSRVTT-CAP070.5±0.9-±0.0-MSVD-QA060.2±0.3-±0.0-VATEX079.7±0.4-±0.0-In order to verify the transferability of PaliGemma to a wide variety of tasks, we transfer the pretrained models on more than 30 academic benchmarks via fine-tuning.
Importantly, none of these tasks or datasets are part of the pretraining data mixture, and their images are explicitly removed from the web-scale pretraining data.
Results are presented in Table1.To select the hyper-parameters for transfer, we first sweep the parameters mentioned in Section3.2.4, starting from the recommended value.
We do not necessarily perform the full cross-product, and we sometimes extend or supersample the range, if it seems promising.
Importantly, we make any such decisions and hyper-parameter choices based on the transfer task’s validation split, and if none is provided, we hold out a small “minival” set from the training data.
Once we found good hyper-parameter values for a task, we re-train using the full training and validation data, and report final test numbers.
Details on tasks, metrics, data splits are in AppendixBand final hyper-parameters in AppendixJ.
In Section6.2we show that a single recommended value for each hyper-parameter without any exploration works almost as well on most tasks.For all but video tasks, we report results on at least two resolutions to provide an impression of which tasks benefit from increased resolution.
We provide many resolution-related ablations in Section5.7.Notably, we have not found any significant benefit from data augmentation.
We simply resize the input images to a square fixed resolution, even for tasks such as RefCOCO segmentation (more on that in Section5.7and AppendixC) .
body
PaliGemma: A versatile 3B VLM for transfer
4Results
2407.07726
5AblationsWe conduct diverse ablations to gain deeper understanding of what matters for training and transferring VLMs.
Unless noted otherwise, all ablations are run with the same setup as the main models, except for making the Stage1 pretraining 10x shorter (i.e.100 M examples seen) , and transfer results are reported on validation sets instead of withheld test-sets.
For each experiment, we present only the salient result summary in the main text, but we provide a full per-task breakdown of results in the Appendix.5.1Multimodal pretraining durationFigure 4:Relative regret of transfers when varying the amount of pretraining during Stage 1. Per task plot in appendixLABEL:sec:app:pt_duration.With its 1 B examples seen, our multimodal pretraining (Stage1)  is on the longer side, similar to BLIP-2, InternVL, QwenVL, Idefics2(all around 1 B) , but unlike ShareGPT4-v, Mini-Gemini, LLaVaand its derivatives (around 1 M) .To the best of our knowledge, the benefits from longer pretraining have not been studied in isolation.
We run pretrainings of various shorter durations, all the way down to completely skipping Stage1, and show the impact in Figure4, with a complete break-down across tasks in the AppendixLABEL:sec:app:pt_duration.
For the case of skipping Stage1, we use the best transfer result when sweeping over three learning-rates for each task.The result shows that shorter training generally hurts, and skipping Stage1 entirely is the worst setting.
Some task are affected significantly, while others only deteriorate a little, highlighting the need for a broad and diverse set of evaluation tasks.
TheM pretraining duration appears to be a good trade-off for ablations: it is 10x shorter while not significantly hurting any task.5.2Causal masking and learning objectiveFigure 5:Learning setup for Stage1.
Left: Where to apply the auto-regressive mask and loss. Middle and right: whether to include a task indicator.We ablate several of our key choices in the pretraining learning objective in Figure5, full per-task breakdown in AppendixLABEL:sec:app:learning.First, we investigate design choices for auto-regressive masking.
PaliGemma uses a prefix-LM strategy which allows full (bi-directional)  attention on the “input” part of the data,i.e.theimageandprefixtokens, see also Figure2.
The motivation is that it allows more tokens to actively participate in the “thinking” process from the start, as the image tokens now can attend to the prefix tokens which represent the query.
This is empirically confirmed in Figure5(left) , where the green bars also include the auto-regressive masking on theprefixtokens, and the orange bars further extend the auto-regressive mask to theimagetokens.
Both sets of bars work, but perform clearly worse than PaliGemma’s prefix-LM setting represented by the blue bar.Second, we apply the next-token-prediction loss on thesuffix(the output)  only.
In principle, it can also be applied on theprefixtokens, once they are auto-regressively masked.
This could provide more learning signal to the model by asking it to “guess the question” for example.
Again, Figure5shows that while it works, doing so clearly reduces average performance.Finally, we eased the multi-task learning by using task-prefixes.
Figure5(middle)  shows that after transfers, this choice of pretraining has no noticeable effect.
However, it would be incorrect to conclude that it has no effect on the model’s training.
Indeed, pretraining validation perplexities on three representative tasks of pretraining are shown in Figure5(right) : when theprefixmakes the task obvious, such as in VQA, a task-prefix has no effect.
For tasks where theprefixdoes not perfectly disambiguate the exact task, however, the model (expectedly)  does become noticeably more uncertain in its predictions without task-prefix.Overall, the prefix-LM with task-prefix and supervision only on thesuffixtokens is an effective VLM pretraining objective.5.3New token initializationFigure 6:Initialization of new tokens matters. Left: distribution of embedding norms at init. Right: Top: initial loss is better with AvgEmb, but this changes after a few thousand steps. Bottom: the model pretrained withinits transfers better to a task heavily using the new tokens.We add new tokens to Gemma’s vocabulary to support PaliGemma’s ability to perform more structured computer vision tasks.
We add 1024 location tokens (<loc0000>to<loc1023>) , which correspond to binned normalized image coordinates and are used in detection, referring expression, and grounded captioning tasks.
We also add 128 VQVAEtokenized single-object mask tokens(<seg000>to<seg127>)  to support referring expression segmentation.This poses the question of how to initialize the embeddings of these new tokens, given all other vocabulary tokens have already been trained as part of Gemma’s pretraining.
One option is to use a standard small Gaussian noise initialization (, blue in Fig6) .
However,argues for matching the average of the pretrained embeddings plus small noise (AvgEmb, mauve in Fig6) .
We compare these strategies in Figure6and see that, while the AvgEmb strategy significantly improves initial perplexity of tasks using the new tokens (top-right zoom-in, step 0) , this gain vanishes after a thousand steps of training.
The standard initialization strategy not only results in significantly better Stage1 perplexities at the end of pretraining (top-right, full plot) , but also results in significantly better transfer of the model to tasks using these tokens, here RefCOCO segmentation MIoU (bottom right) .5.4To freeze or not to freeze?The current common wisdom in VLMsis to keep the image encoder and sometimes the LLM frozen during multimodal pretraining (our Stage1) .
However, inspired by the positive results from CapPaand LocCawhich show that pretraining an image encoder using captioning objectives essentially solves contrastive’s blind spotto relation and localization, we pretrained PaliGemma with no frozen parts.
We now ablate the effect of freezing or tuning various parts of the model during Stage1 in Figure7, full per-task breakdown in AppendixLABEL:sec:app:freeze.
Similar to concurrent works, we find not freezing any part of the model is indeed advantageous.
First, after transfers, there is no difference to keeping the image encoder frozen (left, TT and TF) .
Second, however, the validation perplexity (hence, predictability)  of tasks requiring spatial understanding (right, green)  is significantly improved.Further, we show that all other options that include freezing the language modelare significantly worse.
Finally, resetting (and training, R)  any part of the model hurts performance dramatically, confirming that Stage0 (i.e.leveraging pre-trained components)  is indeed crucial for attaining good results.Figure 7:Training setup for Stage1.
Left: The more is frozen or reset, the more performance deteriorates.
Right: The effect of freezing ViT is most visible in some pretraining perplexities.5.5Connector designThroughout our experiments we use a linear connector to map SigLiP output embeddings to the inputs of Gemma. Given that an MLP connectoris a popular choice in the VLM literature, we also ablate this choice.We consider two connector choices: a linear connector and an MLP (1 hidden layer, with GeLU non-linearity) . We also consider two Stage1 pretraining settings: tune all weights (TT) , or freeze everything but the connector (FF) .When tuning all weights, average transfer score is nearly identical for linear vs MLP, achievingandpoints respectively.
In the “all-frozen” scenario, linear vs MLP achievevs.
Surprisingly, we observe a small performance deterioration with the MLP connector.Overall, we conclude that in our case, the linear connector seems preferable to the MLP connector.5.6Image encoder: with or without?Figure 8:Not using a SigLiP image encoder at all and instead passing a linear projection of raw RGB patches to Gemma works, but is significantly less sample-efficient.Most VLMs follow the setup of having an image encoder, such as CLIP/SigLIP (most works)  or VQGAN (the Chameleon line of work) , to turn the image into soft tokens before passing them to the LLM.
We are aware of only two works that attempt to simplify this overall setup by removing the image encoder entirely and passing raw image patches into a decoder-only LLM, namely Fuyuand EVE.
Unfortunately, the former provides no training details or ablations.
The latter, which is a concurrent work, provides some details about training and various ablations, but with mixed results.Removing the SigLIP encoder in PaliGemma results in a model of the same unified decoder-only architecture.
We run our Stage1 and transfers in this setting.
Since the architecture significantly changed, we also re-tune the learning-rate of Stage1.
Figure8(per-task breakdown in AppendixLABEL:sec:app:enc)  shows that while this architecture still significantly lags behind, the scaling with pretraining duration seems potentially promising.
This is especially noteworthy considering that PaliGemma’s SigLIP encoder has seen 40 B image-text pairs during its Stage0 pre-training, while the Fuyu-style model sees images for the first time in the Stage1 pre-training shown here, and only sees up to 1 B of them in our experiment.
This ablation confirms that such decoder-only VLMs might be a promising future direction towards simpler multimodal models, although they currently still suffer in training efficiency due to not being able to reuse vision components.5.7Image resolutionImage resolution in VLMs is an important topic that has recently received increased attention.
PaliGemma uses a very simple approach to deal with resolution: Stage1 is pretrained at relatively low and economical 224px resolution, and short Stage2 then “upcycles” this checkpoint to higher resolutions (448px and 896px) .
Hence, the final PaliGemma model comes with three different checkpoints for three different resolutions.In this section we justify our approach and the necessity for providing three separate checkpoints, and compare it to a recently popular “windowing” approach.
For the ablation studies, we consider resolutions 224px and 448px, and restrict evaluation to single-image tasks where resolution has significant impact on performance according to Table1, which we call “Resolution-sensitive” tasks.5.7.1Resolution or sequence length?Figure 9:Increasing resolution has two effects: increased information content of the input image, and increased model capacity via sequence length. For tasks that benefit from increased resolution, both of these effects contribute roughly equally to the overall gain.We generally see either neutral or improved performance across tasks when increasing the resolution of the input image (see Table1) .
However, it is unclear whether this improvement comes from the fact that the image has higher resolution and therefore more information, or whether it is thanks to the resulting longer sequence length and thus increased model FLOPs and capacity.
We disentangle these by running Stage2 and transfers at 448 px resolution,but downscaling each image to 224 px before resizing it to 448 px.
Thus, the model gets to see the information content of a low-res (224 px)  image but with the model capacity of the high-res (448 px)  setting.The result in Fig9shows that, for those tasks for which resolution has an effect at all, the reason for the improved performance is split roughly equally between these two causes.
This is true for every individual task and not just an effect of averaging, see AppendixLABEL:sec:app:res_or_seqlen.5.7.2Need resolution-specific checkpoints?Figure 10:Different ways of increasing resolution.
While resolution during transfers works (orange) , it is not the best setup.
Using “windows” works better (mauve,means image resolutionwith window-size) ,
but simply running Stage2 at increased resolution with no tricks works best, justifying the need to provide multiple checkpoints. Per-task breakdown in AppendixLABEL:sec:app:res_window.PaliGemma provides one checkpoint per resolution.
But is this really needed? Could we not provide just a single, maybe high-resolution checkpoint, and adapt it as needed during transfers?Figure10shows clearly that providing either only a 224 px or only a 448 px checkpoint would not be sufficient:
Transferring the 448 px checkpoint at 224 px resolution (first bar, orange)  leads to significantly worse results than using the 224 px checkpoint (second bar, zero) , even though the latter had slightly less pretraining.
Similarly, transferring the 224 px checkpoint at resolution 448 px (third bar, orange) , while improving the results significantly, still lags far behind transferring the checkpoint whose native resolution is 448 px (last bar, green) .Thus, in the absence of flexible-resolution modeling tricks such as FlexiViTor NaViT, we recommend running extended pretraining for increasing resolution (Stage2)  and providing separate checkpoints for all supported resolutions.5.7.3To resize or to window?Another recently common way of increasing input resolution is by “windowing” the models,i.e.applying the same model on windows of the model’s native resolution from the higher-resolution image.
We evaluate the simplest variant of this alternative: the 448 px resolution image is cut into four pieces, each one passed through the SigLIP image encoder separately.
All four sets of 256 image embedding tokens are then concatenated and passed to Gemma.
This is also related to using windowed attention in a ViT taking the full image.
We experimented with adding extra “window ID” position embeddings to indicate which window tokens come from, but this did not significantly change any of the results.The mauve bars in Figure10correspond to windowing settings.
While overall the performance is worse than that of a native 448px model, the windowing approach can be a promising way of transferring a model to a higher resolutionwhen no higher-resolution checkpoint is made available, and when running a Stage2-like continued pretraining is not feasible.Windowing might still seem preferable for speed reasons.
However, we only observed at most a 5% speedup in training across various setups from windowing.
This is explained by Gemma being significantly larger than ViT-So400m, and the Gemma part of the model is unaffected by windowing.5.7.4Stage2 mixture re-weightingFinally, the pretraining mixture changes between Stage1 and Stage2.
While previous PaLI versions change the mixture makeup, for PaliGemma we always use the full set of tasks, only changing their weighting, sampling “resolution-related” tasks (OCR, detection, segmentation)  more frequently at higher resolutions.
As an ablation, we run a Stage2 training with the same mixture ratios as Stage1.
After transfers, this checkpoint is significantly worse on only three tasks (DocVQA, ChartQA, XM3600) , but otherwise within per-task variance (Full results in AppendixLABEL:sec:app:s2_reweight) .Thus, while changing the mixing ratio helped a little, it seems that when the intent is to train a base model for fine-tuning, the precise mixture ratios might not be as important as when training a model intended for sampling zero-shot, where it would significantly affect sampling frequencies.
body
PaliGemma: A versatile 3B VLM for transfer
5Ablations
2407.07726
6TransferabilityTo show that PaliGemma is suitable for transfer under different scenarios, we run experiments that quantify its repeatability, sensitivity to hyper-parameters, and number of transfer examples.6.1Repeatability (variance) Figure 11:For most tasks, the standard deviation of final metric values between re-runs with different seeds is below. This is true both for Stage1 (top, single transfers of three 100M runs) , as well as for transfers (bottom, values from Table1) .In the main results (Table1) , we show the standard deviation across 5 transfer reruns using the best hyper-parameter (listed inJ) .
It is generally very small across most tasks, meaning transfer from the pretrained checkpoint is highly repeatable.
In Figure11we further show that the standard deviation of transferring from three reruns of Stage1 falls within the same range, meaning pretraining itself is also highly repeatable.6.2Transfer hyper-parameter sensitivityTable 2:Relative regret of using the suggested hyper-parameter setting for all tasks instead of performing hyper-parameter search. Per task plot in AppendixLABEL:sec:app:transfer_simple.Rel. Regret#Tasks37All other tasks2ChartQA (human) :3.2%RefCOCO (val) :4.8%2RefCOCOg (val) :5.8%ScienceQA:6.7%2RefCOCO+ (val) :10.7%SciCap:60.5%However one question a reader may have is whether the choice of transfer hyper-parameter is important.
To ablate that we run all tasks at 224px and under a single and simple hyper-parameter setup, which was also highlighted in bold in Section3.2.4: lr=1e-5, bs=256, no dropout, no label smoothing, no weight decay, and not freezing anything.
The only task dependent parameter is the number of epochs for which we use each task’s best one but cap it at 10.The results (Table2)  show that this simplified and single setup works very well for the majority of the tasks. The main exceptions we found were for tasks like RefCOCO and SciCap tasks which seem to benefit significantly from increasing the number of epochs while enabling label smoothing and dropout.6.3Transfer with limited examplesFigure 12:Relative regret of using a limited number of transfer examples. Thick line represents the median. Per task plot in appendixLABEL:sec:app:transfer_lowdata.To analyze how many examples are needed to make PaliGemma solve a new task, we finetune PaliGemma with limited number of examples (64, 256, 1024, 4096) .
We sweep transfer with varying learning rates, epochs and batch size and report the best number without separate minival, to indicate the potential.We run every setting with 5 different seeds, which also affect which examples are used.
We found this important, as finetuning with limited examples exhibits high variance for some tasks (e.g.RefCOCO mIOU varied within 10%-30%) .
As a note, this variance also occurs when repeating with the same examples, but different batch order.
Importantly, seed selection is not overfitting to the metric as the selected model performs equally well in the validation and test splits.But it does allows us to draw conclusions without needing to solve the open problem of making few-example fine-tuning stable.Overall, when comparing the best runs of each hyper-parameter and seed with the results obtained with the full dataset, Figure12shows that it is not necessary to have a transfer dataset in the order of 10 k examples.
The majority of the tasks can reach within 10% of the full-data score when using 4 k examples and 20% when using only 256 examples. In many cases the score with 64 transfer examples are good enough to prototype using PaliGemma for a new application.
body
PaliGemma: A versatile 3B VLM for transfer
6Transferability
2407.07726
7Noteworthy tidbitsWe also briefly discuss several small but interesting discoveries or we made, providing more details on each in the Appendix.Plain resize to square for segmentation.We found simple resize to square ()  to work just as well as popular aspect-ratio preserving zoom and crop augmentations. (AppendixC) Counting: introducing CountBenchQA.Due to skewed number distribution and varying image quality, we found TallyQA lacking in its ability to assess current VLM’s ability to count.
This is why we introduce CountBenchQA, a VLM-ready version of the CountBenchdataset. (AppendixD) .Issues in published WidgetCaps numbers.We found issues in at least three previous work’s evaluation of WidgetCaps, rendering numerical comparisons invalid. (AppendixE) .Image annotations work as well as prompts.Marking the widget to be captioned with a red box in the image gives the same results as indicating it with<loc>tokens in the prompt. (AppendixF) RoPE interpolation unnecessary for upscaling.For Stage2, we tried interpolating the RoPE position indices for the image tokens to preserve their semantics from Stage1. However, we saw no benefit from doing so.Zero-shot generalization to 3D renders.Although never explicitly trained for it, PaliGemma generalizes surprisingly well to 3D renders form Objaverse without fine-tuning. (AppendixG) Our MMVP result is SOTA by a large margin.PaliGemma at 224px achieves 47.3% paired accuracy, while GPT4-V and Gemini achieve 38.7% and 40.7%, respectively, and all other models including LLaVa perform below chance.
body
PaliGemma: A versatile 3B VLM for transfer
7Noteworthy tidbits
2407.07726
8ConclusionPaliGemma is a new, small, open base VLM that shines when transferred to a broad range of tasks.
Our results show that VLMs on the “smaller” side can provide state-of-the-art performance across a wide variety of benchmarks.
We also hope that providing the base model without instruction tuning serves as a useful starting point for further research in instruction tuning, specific applications, and encourages clearer separation of base models and fine-tunes in VLM research.\nobibliography*
body
PaliGemma: A versatile 3B VLM for transfer
8Conclusion
2407.07726
Author ContributionsModel development contributorsCore ContributorsLucas BeyerAndreas SteinerAndré Susano PintoAlexander KolesnikovXiao WangXiaohua ZhaiContributorsDaniel SalzMaxim NeumannIbrahim AlabdulmohsinMichael TschannenEmanuele BugliarelloThomas UnterthinerDaniel KeysersSkanda KoppulaFangyu LiuAdam GrycnerAlexey GritsenkoNeil HoulsbyManoj KumarKeran RongJulian EisenschlosRishabh KabraMatthias BauerMatko BošnjakXi ChenMatthias MindererPaul VoigtlaenderIoana BicaIvana BalazevicJoan PuigcerverPinelopi PapalampidiOlivier HenaffXi XiongRadu SoricutJeremiah HarmsenLeadsXiaohua ZhaiLucas BeyerModel release contributors andgeneral supportPMTris WarkentinGo-to-MarketKat BlackLuiz Gustavo MartinsGlenn CameronRaj GundluruManvinder SinghKaggleMeg RisdalNilay ChauhanNate KeatingNesh DevanathanDocumentationElisa BandyJoe FernandezEthics and SafetyAntonia PatersonJenny BrennanTom EcclesPankil BotadraBen BariachVertex AILav RaiMinwoo ParkDustin LuongDaniel VlasicBo WuWenming YeKerasDivyashree SreepathihalliKiranbir SodhiaGabriel RasskinMatthew WatsonVarun SinghGemma ModelAlek AndreevArmand JoulinSurya BhupatirajuMinh GiangExecutive SponsorsJoelle BarralZoubin Ghahramani
conclusion
PaliGemma: A versatile 3B VLM for transfer
Author Contributions