<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.03320] InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output</title><meta property="og:description" content="We present InternLM-XComposer-2.5 (IXC-2.5), a versatile large-vision language model that supports long-contextual input and output. IXC-2.5 excels in various text-image comprehension and composition applications, achi…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.03320">

<!--Generated on Mon Aug  5 17:51:02 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Pan Zhang<sup id="id32.32.id1" class="ltx_sup"><span id="id32.32.id1.1" class="ltx_text ltx_font_italic">∗1</span></sup>, Xiaoyi Dong<sup id="id33.33.id2" class="ltx_sup"><span id="id33.33.id2.1" class="ltx_text ltx_font_italic">∗1,2</span></sup>, Yuhang Zang<sup id="id34.34.id3" class="ltx_sup"><span id="id34.34.id3.1" class="ltx_text ltx_font_italic">∗1</span></sup>, Yuhang Cao<sup id="id35.35.id4" class="ltx_sup"><span id="id35.35.id4.1" class="ltx_text ltx_font_italic">1</span></sup>, Rui Qian<sup id="id36.36.id5" class="ltx_sup"><span id="id36.36.id5.1" class="ltx_text ltx_font_italic">1,2</span></sup>, Lin Chen<sup id="id37.37.id6" class="ltx_sup"><span id="id37.37.id6.1" class="ltx_text ltx_font_italic">1</span></sup>, Qipeng Guo<sup id="id38.38.id7" class="ltx_sup"><span id="id38.38.id7.1" class="ltx_text ltx_font_italic">1</span></sup>, 
<br class="ltx_break">Haodong Duan<sup id="id39.39.id8" class="ltx_sup"><span id="id39.39.id8.1" class="ltx_text ltx_font_italic">1</span></sup>, Bin Wang<sup id="id40.40.id9" class="ltx_sup"><span id="id40.40.id9.1" class="ltx_text ltx_font_italic">1</span></sup>, Linke Ouyang<sup id="id41.41.id10" class="ltx_sup"><span id="id41.41.id10.1" class="ltx_text ltx_font_italic">1</span></sup>, Songyang Zhang<sup id="id42.42.id11" class="ltx_sup"><span id="id42.42.id11.1" class="ltx_text ltx_font_italic">1</span></sup>, Wenwei Zhang<sup id="id43.43.id12" class="ltx_sup"><span id="id43.43.id12.1" class="ltx_text ltx_font_italic">1</span></sup>, Yining Li<sup id="id44.44.id13" class="ltx_sup"><span id="id44.44.id13.1" class="ltx_text ltx_font_italic">1</span></sup>, 
<br class="ltx_break">Yang Gao<sup id="id45.45.id14" class="ltx_sup"><span id="id45.45.id14.1" class="ltx_text ltx_font_italic">1</span></sup>, Peng Sun<sup id="id46.46.id15" class="ltx_sup"><span id="id46.46.id15.1" class="ltx_text ltx_font_italic">1</span></sup>, Xinyue Zhang<sup id="id47.47.id16" class="ltx_sup"><span id="id47.47.id16.1" class="ltx_text ltx_font_italic">1</span></sup>, Wei Li<sup id="id48.48.id17" class="ltx_sup"><span id="id48.48.id17.1" class="ltx_text ltx_font_italic">1</span></sup>, Jingwen Li<sup id="id49.49.id18" class="ltx_sup"><span id="id49.49.id18.1" class="ltx_text ltx_font_italic">1</span></sup>,
Wenhai Wang<sup id="id50.50.id19" class="ltx_sup"><span id="id50.50.id19.1" class="ltx_text ltx_font_italic">1,2</span></sup>, Hang Yan<sup id="id51.51.id20" class="ltx_sup"><span id="id51.51.id20.1" class="ltx_text ltx_font_italic">1</span></sup>, 
<br class="ltx_break">Conghui He<sup id="id52.52.id21" class="ltx_sup"><span id="id52.52.id21.1" class="ltx_text ltx_font_italic">3</span></sup>, Xingcheng Zhang<sup id="id53.53.id22" class="ltx_sup"><span id="id53.53.id22.1" class="ltx_text ltx_font_italic">3</span></sup>, Kai Chen<sup id="id54.54.id23" class="ltx_sup"><span id="id54.54.id23.1" class="ltx_text ltx_font_italic">1</span></sup>, Jifeng Dai<sup id="id55.55.id24" class="ltx_sup"><span id="id55.55.id24.1" class="ltx_text ltx_font_italic">4,1</span></sup>, Yu Qiao<sup id="id56.56.id25" class="ltx_sup"><span id="id56.56.id25.1" class="ltx_text ltx_font_italic">1</span></sup>, Dahua Lin<sup id="id57.57.id26" class="ltx_sup"><span id="id57.57.id26.1" class="ltx_text ltx_font_italic">1,2</span></sup>, Jiaqi Wang<sup id="id58.58.id27" class="ltx_sup"><span id="id58.58.id27.1" class="ltx_text ltx_font_italic">1,</span></sup><sup id="id59.59.id28" class="ltx_sup">🖂</sup>
<br class="ltx_break"><sup id="id60.60.id29" class="ltx_sup">1</sup>Shanghai Artificial Intelligence Laboratory, <sup id="id61.61.id30" class="ltx_sup">2</sup>The Chinese University of Hong Kong, 
<br class="ltx_break"><sup id="id62.62.id31" class="ltx_sup">3</sup>SenseTime Group, <sup id="id63.63.id32" class="ltx_sup">4</sup>Tsinghua University 
<br class="ltx_break"><span id="id64.64.id33" class="ltx_text ltx_font_typewriter" style="font-size:90%;">internlm@pjlab.org.cn
</span>
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id65.id1" class="ltx_p">We present InternLM-XComposer-2.5 (IXC-2.5), a versatile large-vision language model that supports long-contextual input and output. <span id="id65.id1.1" class="ltx_text ltx_font_bold">IXC-2.5 excels in various text-image comprehension and composition applications, achieving GPT-4V level capabilities with merely 7B LLM backend.</span> Trained with 24K interleaved image-text contexts, it can seamlessly extend to 96K long contexts via RoPE extrapolation. This long-context capability allows IXC-2.5 to excel in tasks requiring extensive input and output contexts. Compared to its previous 2.0 version, InternLM-XComposer-2.5 features three major upgrades in vision-language comprehension: (1) Ultra-High Resolution Understanding, (2) Fine-Grained Video Understanding, and (3) Multi-Turn Multi-Image Dialogue. In addition to comprehension, IXC-2.5 extends to two compelling applications using extra LoRA parameters for text-image composition: (1) Crafting Webpages and (2) Composing High-Quality Text-Image Articles. IXC-2.5 has been evaluated on 28 benchmarks, outperforming existing open-source state-of-the-art models on 16 benchmarks. It also surpasses or competes closely with GPT-4V and Gemini Pro on 16 key tasks. The InternLM-XComposer-2.5 is publicly available at <a target="_blank" href="https://github.com/InternLM/InternLM-XComposer" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/InternLM/InternLM-XComposer</a>.</p>
</div>
<span id="footnotex1" class="ltx_note ltx_role_footnotetext"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">footnotetext: </span>* equal contribution. 🖂 corresponding author.</span></span></span>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<figure id="S1.F1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.03320/assets/figures/ixc2d5_radar.png" id="S1.1.g1" class="ltx_graphics ltx_figure_panel ltx_img_square" width="598" height="578" alt="Refer to caption">
<br class="ltx_break ltx_break">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S1.F1.3.1.1" class="ltx_text" style="font-size:90%;">Figure 1</span>: </span><span id="S1.F1.4.2" class="ltx_text" style="font-size:90%;">Overview of InternLM-XComposer-2.5 (IXC-2.5) performance on benchmarks in different domains, including <span id="S1.F1.4.2.1" class="ltx_text ltx_font_bold" style="color:#C9461C;">Video Benchmarks</span>, <span id="S1.F1.4.2.2" class="ltx_text ltx_font_bold" style="color:#249087;">Structural High-resolution Benchmarks</span>, <span id="S1.F1.4.2.3" class="ltx_text ltx_font_bold" style="color:#314272;">General Visual QA Benchmarks</span>, <span id="S1.F1.4.2.4" class="ltx_text ltx_font_bold" style="color:#0070C0;">Multi-True Multi-Image Benchmark</span>, and <span id="S1.F1.4.2.5" class="ltx_text ltx_font_bold" style="color:#02B0F0;">Webpage Crafting Benchmark</span>.
IXC-2.5 based on InternLM2-7B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib143" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">143</span></a>]</cite> <span id="S1.F1.4.2.6" class="ltx_text ltx_font_bold">matches or even surpasses GPT-4V <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib112" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">112</span></a>]</cite> and Gemini Pro <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib142" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">142</span></a>]</cite> in 15 benchmarks. Please refer to Table <a href="#S3.T3" title="Table 3 ‣ 3.6 Article Composing ‣ 3 Method ‣ InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, <a href="#S3.T4" title="Table 4 ‣ 3.6 Article Composing ‣ 3 Method ‣ InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, <a href="#S3.T5" title="Table 5 ‣ 3.6 Article Composing ‣ 3 Method ‣ InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> for details.</span>
</span></figcaption>
</figure>
<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Recent advancements in Large Language Models (LLMs) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib111" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">111</span></a>, <a href="#bib.bib29" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">29</span></a>, <a href="#bib.bib146" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">146</span></a>, <a href="#bib.bib147" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">147</span></a>, <a href="#bib.bib55" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">55</span></a>, <a href="#bib.bib121" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">121</span></a>]</cite> have sparked interest in the development of Large Vision Language Models (LVLMs) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib112" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">112</span></a>, <a href="#bib.bib41" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">41</span></a>, <a href="#bib.bib183" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">183</span></a>, <a href="#bib.bib31" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">31</span></a>, <a href="#bib.bib173" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">173</span></a>, <a href="#bib.bib84" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">84</span></a>]</cite>. Leading paradigms like GPT-4 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib112" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">112</span></a>]</cite>, Gemini Pro 1.5 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib41" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">41</span></a>]</cite>, and Claude 3 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">3</span></a>]</cite> have achieved considerable success and significantly expanded the range of applications for LLMs. Open-source LVLMs are also being rapidly developed and can compete with proprietary APIs in several benchmarks. However, these open-source models still lag behind closed-source leading paradigms in versatility. They lack the ability to perform diverse vision-language comprehension and composition tasks, largely due to limited diversity in training corpus and challenges in managing long-context input and output.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">To further bridge the gap between proprietary APIs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib112" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">112</span></a>, <a href="#bib.bib41" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">41</span></a>]</cite> and open-sourced Large Vision Language Models, we are introducing InternLM-XComposer-2.5 (IXC-2.5), a versatile LVLM supporting long-contextual input and output with diverse comprehension and composition capacities. IXC-2.5 excels in existing open-sourced LVLMs with two advantages. <span id="S1.p2.1.1" class="ltx_text ltx_font_bold">(1) Versatility</span>: IXC-2.5 supports a wide range of tasks related to comprehension and composition, such as free-form text-image conversation, OCR, video understanding, article composition with illustrations, and webpage crafting. <span id="S1.p2.1.2" class="ltx_text ltx_font_bold">(2) Long-context capabilities in both input and output:</span> It is natively trained with 24K interleaved image-text data, whose context window can be extended to 96K through positional encoding extrapolation <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib94" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">94</span></a>]</cite>, empowering the long-term human-AI interaction and content creation.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">Benefiting from the long contextual capability, compared to its previous 2.0 version <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib33" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">33</span></a>]</cite>, IXC-2.5 has upgraded three comprehension abilities: <span id="S1.p3.1.1" class="ltx_text ltx_font_bold">(1) Ultra-High Resolution Understanding:</span> IXC-2.5 enhances the dynamic resolution solution proposed in IXC2-4KHD <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib34" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">34</span></a>]</cite> with a native 560 × 560 ViT vision encoder, supporting high-resolution images with any aspect ratio. <span id="S1.p3.1.2" class="ltx_text ltx_font_bold">(2) Fine-Grained Video Understanding:</span> IXC-2.5 treats videos as a ultra-high-resolution composite picture consisting of tens to hundreds of frames, allowing it to capture fine details through dense sampling and higher resolution for each frame. <span id="S1.p3.1.3" class="ltx_text ltx_font_bold">(3) Multi-Turn Multi-Image Dialogue:</span> IXC-2.5 supports free-form multi-turn multi-image dialogue, allowing it to naturally interact with humans in multi-round conversations.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">Besides comprehension, IXC-2.5 also supports two notable applications by incorporating extra LoRA parameters for text-image composition:
<span id="S1.p4.1.1" class="ltx_text ltx_font_bold">(1) Crafting Webpages:</span> IXC-2.5 can be readily applied to create webpages by composing source code (HTML, CSS, and JavaScript) following text-image instructions. <span id="S1.p4.1.2" class="ltx_text ltx_font_bold">(2) Composing High-Quality Text-Image Articles:</span> Compared to IXC-2, IXC-2.5 leverages specially designed Chain-of-Thought (CoT) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib153" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">153</span></a>]</cite> and Direct Preference Optimization (DPO) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib124" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">124</span></a>]</cite> techniques to significantly enhance the quality of its written content.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">We evaluated the versatility of InternLM-XComposer-2.5 (IXC-2.5) across a range of twenty-eight benchmarks, including five video benchmarks <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib71" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">71</span></a>, <a href="#bib.bib181" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">181</span></a>, <a href="#bib.bib42" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">42</span></a>, <a href="#bib.bib38" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">38</span></a>, <a href="#bib.bib88" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">88</span></a>]</cite>, nine structural high-resolution benchmarks <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib107" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">107</span></a>, <a href="#bib.bib106" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">106</span></a>, <a href="#bib.bib108" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">108</span></a>, <a href="#bib.bib133" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">133</span></a>, <a href="#bib.bib89" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">89</span></a>, <a href="#bib.bib139" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">139</span></a>, <a href="#bib.bib117" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">117</span></a>, <a href="#bib.bib140" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">140</span></a>, <a href="#bib.bib20" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">20</span></a>]</cite>, twelve general VQA benchmarks <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">18</span></a>, <a href="#bib.bib100" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">100</span></a>, <a href="#bib.bib166" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">166</span></a>, <a href="#bib.bib61" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">61</span></a>, <a href="#bib.bib40" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">40</span></a>, <a href="#bib.bib87" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">87</span></a>, <a href="#bib.bib66" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">66</span></a>, <a href="#bib.bib164" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">164</span></a>, <a href="#bib.bib44" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">44</span></a>, <a href="#bib.bib155" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">155</span></a>]</cite>, one multi-true multi-image benchmark <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib92" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">92</span></a>]</cite>, and one webpage crafting benchmark <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib131" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">131</span></a>]</cite>.
Compared to previous open-source LVLMs, IXC-2.5 achieved state-of-the-art results in 16 out of 28 benchmarks based on InternLM2-7B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib143" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">143</span></a>]</cite> backend. As shown in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, the performance of IXC-2.5 matches or even surpasses proprietary APIs, <em id="S1.p5.1.1" class="ltx_emph ltx_font_italic">e.g</em>.<span id="S1.p5.1.2" class="ltx_text"></span>, GPT-4V <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib112" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">112</span></a>]</cite> and Gemini Pro  <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib41" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">41</span></a>]</cite>, in 16 benchmarks.</p>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p">IXC-2.5 web demo now supports audio input and output using open-source tools <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib123" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">123</span></a>, <a href="#bib.bib179" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">179</span></a>]</cite>. You may try it at <a target="_blank" href="https://huggingface.co/spaces/Willow123/InternLM-XComposer" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://huggingface.co/spaces/Willow123/InternLM-XComposer</a>.</p>
</div>
<figure id="S1.F2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.03320/assets/x1.png" id="S1.2.g1" class="ltx_graphics ltx_figure_panel ltx_img_portrait" width="447" height="636" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S1.F2.3.1.1" class="ltx_text" style="font-size:90%;">Figure 2</span>: </span><span id="S1.F2.4.2" class="ltx_text" style="font-size:90%;">The capabilities of IXC-2.5: Chat on 4K-resolution Image and Multi-Turn Multi-Image Dialogue.
</span></figcaption>
</figure>
<figure id="S1.F3" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.03320/assets/x2.png" id="S1.3.g1" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_portrait" width="447" height="636" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S1.F3.3.1.1" class="ltx_text" style="font-size:90%;">Figure 3</span>: </span><span id="S1.F3.4.2" class="ltx_text" style="font-size:90%;">The capabilities of IXC-2.5: Chat on Video, Resume-to-Homepage and Instruction-aware Webpage Generation.
</span></figcaption>
</figure>
<figure id="S1.F4" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.03320/assets/x3.png" id="S1.4.g1" class="ltx_graphics ltx_figure_panel ltx_img_portrait" width="447" height="636" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S1.F4.3.1.1" class="ltx_text" style="font-size:90%;">Figure 4</span>: </span><span id="S1.F4.4.2" class="ltx_text" style="font-size:90%;">The capabilities of IXC-2.5: Screenshot-to-Code and Article Composing.
</span></figcaption>
</figure>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Works</h2>

<div id="S2.p1" class="ltx_para ltx_noindent">
<p id="S2.p1.1" class="ltx_p"><span id="S2.p1.1.1" class="ltx_text ltx_font_bold">LVLMs for Text-Image Conversation.</span> Large Language Models (LLMs) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib12" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">12</span></a>, <a href="#bib.bib115" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">115</span></a>, <a href="#bib.bib111" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">111</span></a>, <a href="#bib.bib29" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">29</span></a>, <a href="#bib.bib60" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">60</span></a>, <a href="#bib.bib146" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">146</span></a>, <a href="#bib.bib147" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">147</span></a>, <a href="#bib.bib55" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">55</span></a>, <a href="#bib.bib143" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">143</span></a>, <a href="#bib.bib168" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">168</span></a>, <a href="#bib.bib8" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">8</span></a>, <a href="#bib.bib121" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">121</span></a>, <a href="#bib.bib13" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">13</span></a>]</cite> have received considerable attention because of their impressive performance in language comprehension and generation. Large vision-language models (LVLMs) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib112" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">112</span></a>, <a href="#bib.bib24" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">24</span></a>, <a href="#bib.bib22" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">22</span></a>, <a href="#bib.bib23" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">23</span></a>, <a href="#bib.bib35" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">35</span></a>, <a href="#bib.bib41" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">41</span></a>, <a href="#bib.bib183" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">183</span></a>, <a href="#bib.bib31" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">31</span></a>, <a href="#bib.bib173" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">173</span></a>, <a href="#bib.bib9" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">9</span></a>, <a href="#bib.bib68" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">68</span></a>, <a href="#bib.bib118" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">118</span></a>, <a href="#bib.bib159" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">159</span></a>, <a href="#bib.bib5" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">5</span></a>, <a href="#bib.bib34" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">34</span></a>, <a href="#bib.bib78" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">78</span></a>]</cite> have been developed by integrating LLMs with vision encoders <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib122" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">122</span></a>, <a href="#bib.bib170" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">170</span></a>, <a href="#bib.bib138" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">138</span></a>, <a href="#bib.bib169" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">169</span></a>, <a href="#bib.bib113" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">113</span></a>, <a href="#bib.bib167" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">167</span></a>, <a href="#bib.bib91" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">91</span></a>, <a href="#bib.bib25" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">25</span></a>, <a href="#bib.bib17" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">17</span></a>, <a href="#bib.bib79" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">79</span></a>, <a href="#bib.bib6" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">6</span></a>, <a href="#bib.bib150" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">150</span></a>, <a href="#bib.bib33" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">33</span></a>, <a href="#bib.bib14" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">14</span></a>, <a href="#bib.bib93" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">93</span></a>, <a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">26</span></a>, <a href="#bib.bib176" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">176</span></a>]</cite> to extend the ability to understand vision content, enabling the application of text-image conversation. Most existing LVLMs are trained for single-image multi-round conversations, while some works <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib2" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">2</span></a>, <a href="#bib.bib6" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">6</span></a>, <a href="#bib.bib178" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">178</span></a>, <a href="#bib.bib136" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">136</span></a>, <a href="#bib.bib78" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">78</span></a>, <a href="#bib.bib56" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">56</span></a>]</cite> have the ability to understand multi-image inputs. However, IXC-2.5 focuses on providing a free-form long-contextual multi-turn multi-image interaction experience <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib92" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">92</span></a>, <a href="#bib.bib86" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">86</span></a>, <a href="#bib.bib103" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">103</span></a>]</cite>, which has not been addressed yet.</p>
</div>
<div id="S2.p2" class="ltx_para ltx_noindent">
<p id="S2.p2.1" class="ltx_p"><span id="S2.p2.1.1" class="ltx_text ltx_font_bold">LVLMs for High Resolution Images Analysis.</span> Understanding high-resolution images has significant potential applications such as OCR and document/chart analysis, which is attracting increased attention in the LVLMs area. In recent works, there are two main strategies to enable high-resolution understanding: (1) High-resolution (HR) visual encoders <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib102" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">102</span></a>, <a href="#bib.bib151" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">151</span></a>, <a href="#bib.bib47" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">47</span></a>, <a href="#bib.bib74" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">74</span></a>, <a href="#bib.bib141" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">141</span></a>, <a href="#bib.bib177" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">177</span></a>]</cite> directly support higher resolution images. (2) Patchification: A high-resolution image is cropped into patches <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib76" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">76</span></a>, <a href="#bib.bib90" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">90</span></a>, <a href="#bib.bib157" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">157</span></a>, <a href="#bib.bib158" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">158</span></a>, <a href="#bib.bib49" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">49</span></a>, <a href="#bib.bib79" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">79</span></a>, <a href="#bib.bib67" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">67</span></a>, <a href="#bib.bib79" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">79</span></a>, <a href="#bib.bib85" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">85</span></a>, <a href="#bib.bib158" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">158</span></a>, <a href="#bib.bib34" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">34</span></a>]</cite>. Each patch is processed with a low resolution vision encoder, <em id="S2.p2.1.2" class="ltx_emph ltx_font_italic">e.g</em>.<span id="S2.p2.1.3" class="ltx_text"></span>, CLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib122" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">122</span></a>]</cite> and visual embeddings of patches are further concatenated as inputs for LLM backends. IXC-4KHD <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib34" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">34</span></a>]</cite> scales the supported resolution of open-source LVLMs into 4K and beyond for the first time. IXC-2.5 combines both solutions with a vision encoder trained with a resolution of 560x560 and a dynamic resolution solution proposed in IXC2-4KHD <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib34" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">34</span></a>]</cite>, resulting in further improvements.</p>
</div>
<div id="S2.p3" class="ltx_para ltx_noindent">
<p id="S2.p3.1" class="ltx_p"><span id="S2.p3.1.1" class="ltx_text ltx_font_bold">LVLMs for Video Understanding.</span> In addition to image understanding, the LVLMs area has also witnessed emerging efforts in video analysis <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib70" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">70</span></a>, <a href="#bib.bib110" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">110</span></a>, <a href="#bib.bib88" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">88</span></a>, <a href="#bib.bib37" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">37</span></a>, <a href="#bib.bib39" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">39</span></a>, <a href="#bib.bib134" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">134</span></a>, <a href="#bib.bib135" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">135</span></a>]</cite>. To handle complex video inputs, existing works use sparse sampling or temporal pooling <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib77" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">77</span></a>, <a href="#bib.bib104" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">104</span></a>, <a href="#bib.bib101" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">101</span></a>, <a href="#bib.bib52" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">52</span></a>, <a href="#bib.bib162" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">162</span></a>]</cite>, compressed video tokens <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib69" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">69</span></a>, <a href="#bib.bib172" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">172</span></a>, <a href="#bib.bib57" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">57</span></a>, <a href="#bib.bib154" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">154</span></a>, <a href="#bib.bib73" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">73</span></a>, <a href="#bib.bib120" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">120</span></a>]</cite>, memory banks <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib134" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">134</span></a>, <a href="#bib.bib45" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">45</span></a>, <a href="#bib.bib135" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">135</span></a>]</cite>, and language as a bridge <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib59" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">59</span></a>, <a href="#bib.bib54" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">54</span></a>, <a href="#bib.bib171" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">171</span></a>]</cite> for video understanding. Apart from these video-specific designs, video analysis can also be formulated to understand a high-resolution composite picture consisting of sampled video frames <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib63" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">63</span></a>, <a href="#bib.bib156" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">156</span></a>, <a href="#bib.bib174" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">174</span></a>]</cite>. Benefiting from the ability to comprehend ultra-high-resolution images and long context, IXC-2.5 exhibits strong performance on various video benchmarks for LVLMs.</p>
</div>
<div id="S2.p4" class="ltx_para ltx_noindent">
<p id="S2.p4.1" class="ltx_p"><span id="S2.p4.1.1" class="ltx_text ltx_font_bold">Webpage Generation.</span> Pix2Code <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">10</span></a>]</cite> presents an end-to-end solution for UI-to-code transformation leveraging CNNs and RNNs. This approach contends with the challenges posed by intricate visual encoding and extensive text decoding when applied to real-world UIs. In the sphere of recent advancements, works such as Sightseer <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib64" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">64</span></a>]</cite>, DCGen <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib148" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">148</span></a>]</cite>, and Design2Code <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib131" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">131</span></a>]</cite> have employed large vision-language models trained on synthetic screenshot-HTML paired datasets like WebSight v0.1 or v0.2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib64" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">64</span></a>]</cite> to facilitate HTML code generation. Nevertheless, the synthesized web page datasets have been critiqued for their simplicity and lack of diversity. These studies generally concentrate on the screenshot/sketch-to-code task. In contrast, our IXC-2.5 model extends these capabilities to include screenshot-to-code, instruction-aware webpage generation, and resume-to-homepage tasks. IXC-2.5 is trained using a combination of high-quality synthesized and real-world web data. Furthermore, IXC-2.5 is proficient in generating JavaScript code, thereby enabling the development of interactive front-end webpages.</p>
</div>
<div id="S2.p5" class="ltx_para ltx_noindent">
<p id="S2.p5.1" class="ltx_p"><span id="S2.p5.1.1" class="ltx_text ltx_font_bold">Preference Alignment.</span> Reinforcement Learning from Human Feedback (RLHF) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib115" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">115</span></a>]</cite> and Reinforcement Learning from AI Feedback (RLAIF) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib7" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">7</span></a>]</cite> have shown great promise in aligning LLMs across various domains, including improving logical reasoning and generating helpful and harmless outputs. The typical approach involves training a reward model using human or AI preference data and fine-tuning the LLM to maximize the expected reward function with optimization algorithms like Proximal Policy Optimization (PPO) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib126" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">126</span></a>]</cite>. Alternatively, Direct Preference Optimization (DPO) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib124" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">124</span></a>]</cite> and the following works <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib116" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">116</span></a>, <a href="#bib.bib36" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">36</span></a>]</cite> have emerged as leading methods that implicitly represent the reward score and eliminate the need for a separate reward model. Building on the success of RLHF and RLAIF in LLMs, recent studies have successfully extended RLHF/RLAIF algorithms for multimodal LVLMs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib72" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">72</span></a>, <a href="#bib.bib163" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">163</span></a>, <a href="#bib.bib182" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">182</span></a>, <a href="#bib.bib180" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">180</span></a>, <a href="#bib.bib119" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">119</span></a>]</cite> to reduce hallucination. In this work, we investigate the application of preference alignment techniques to the text-image article composition task, with a focus on generating high-quality and stable response results.</p>
</div>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Method</h2>

<figure id="S3.F5" class="ltx_figure"><img src="/html/2407.03320/assets/x4.png" id="S3.F5.g1" class="ltx_graphics ltx_centering ltx_img_portrait" width="461" height="695" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F5.3.1.1" class="ltx_text" style="font-size:90%;">Figure 5</span>: </span><span id="S3.F5.4.2" class="ltx_text ltx_font_bold" style="font-size:90%;">Framework<span id="S3.F5.4.2.1" class="ltx_text ltx_font_medium"> of IXC-2.5 that supports the multi-modal inputs, including text, single/multiple images, and videos. </span></span></figcaption>
</figure>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Model Architecture</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">The model architecture of InternLM-XComposer-2.5 (IXC-2.5 in the following for simplicity) mainly follows the design of InternLM-XComposer2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib33" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">33</span></a>]</cite> and InternLM-XComposer2-4KHD <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib34" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">34</span></a>]</cite> (IXC2 and IXC2-4KHD for simplicity), including a light-weight Vision Encoder OpenAI ViT-L/14 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib122" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">122</span></a>]</cite>, Large Language Model InternLM2-7B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib13" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">13</span></a>]</cite>, and Partial LoRA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib33" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">33</span></a>]</cite> for efficient alignment. We recommend the readers to the IXC2 and IXC2-4KHD papers for more details.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Multi-modal Input</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">Our IXC-2.5 supports diverse input modalities, including text, single/multiple images, and videos. As showin in Figure <a href="#S3.F5" title="Figure 5 ‣ 3 Method ‣ InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, a Unified Dynamic Image Partition strategy is adopted for both videos and multiple images with any resolutions and aspect ratios.</p>
</div>
<div id="S3.SS2.p2" class="ltx_para ltx_noindent">
<p id="S3.SS2.p2.2" class="ltx_p"><span id="S3.SS2.p2.2.1" class="ltx_text ltx_font_bold">Image Processing.</span>
We mainly follow the Dynamic Image Partition and Global-Local Format design used in IXC2-4KHD <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib34" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">34</span></a>]</cite> with a few modifications. For the vision encoder, we reuse the ViT of <math id="S3.SS2.p2.1.m1.1" class="ltx_Math" alttext="490\times 490" display="inline"><semantics id="S3.SS2.p2.1.m1.1a"><mrow id="S3.SS2.p2.1.m1.1.1" xref="S3.SS2.p2.1.m1.1.1.cmml"><mn id="S3.SS2.p2.1.m1.1.1.2" xref="S3.SS2.p2.1.m1.1.1.2.cmml">490</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p2.1.m1.1.1.1" xref="S3.SS2.p2.1.m1.1.1.1.cmml">×</mo><mn id="S3.SS2.p2.1.m1.1.1.3" xref="S3.SS2.p2.1.m1.1.1.3.cmml">490</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.1.m1.1b"><apply id="S3.SS2.p2.1.m1.1.1.cmml" xref="S3.SS2.p2.1.m1.1.1"><times id="S3.SS2.p2.1.m1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.1.1.1"></times><cn type="integer" id="S3.SS2.p2.1.m1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.1.1.2">490</cn><cn type="integer" id="S3.SS2.p2.1.m1.1.1.3.cmml" xref="S3.SS2.p2.1.m1.1.1.3">490</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.1.m1.1c">490\times 490</annotation></semantics></math> resolution used in IXC2 and further increase its resolution to <math id="S3.SS2.p2.2.m2.1" class="ltx_Math" alttext="560\times 560" display="inline"><semantics id="S3.SS2.p2.2.m2.1a"><mrow id="S3.SS2.p2.2.m2.1.1" xref="S3.SS2.p2.2.m2.1.1.cmml"><mn id="S3.SS2.p2.2.m2.1.1.2" xref="S3.SS2.p2.2.m2.1.1.2.cmml">560</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p2.2.m2.1.1.1" xref="S3.SS2.p2.2.m2.1.1.1.cmml">×</mo><mn id="S3.SS2.p2.2.m2.1.1.3" xref="S3.SS2.p2.2.m2.1.1.3.cmml">560</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.2.m2.1b"><apply id="S3.SS2.p2.2.m2.1.1.cmml" xref="S3.SS2.p2.2.m2.1.1"><times id="S3.SS2.p2.2.m2.1.1.1.cmml" xref="S3.SS2.p2.2.m2.1.1.1"></times><cn type="integer" id="S3.SS2.p2.2.m2.1.1.2.cmml" xref="S3.SS2.p2.2.m2.1.1.2">560</cn><cn type="integer" id="S3.SS2.p2.2.m2.1.1.3.cmml" xref="S3.SS2.p2.2.m2.1.1.3">560</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.2.m2.1c">560\times 560</annotation></semantics></math>, so that each sub-image has 400 tokens.</p>
</div>
<div id="S3.SS2.p3" class="ltx_para">
<p id="S3.SS2.p3.5" class="ltx_p">For the high-resolution strategy, we unify the different strategies used in the IXC-4KHD into a scaled identity strategy. Given a maximum partition number <math id="S3.SS2.p3.1.m1.1" class="ltx_Math" alttext="\mathcal{H}" display="inline"><semantics id="S3.SS2.p3.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p3.1.m1.1.1" xref="S3.SS2.p3.1.m1.1.1.cmml">ℋ</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.1.m1.1b"><ci id="S3.SS2.p3.1.m1.1.1.cmml" xref="S3.SS2.p3.1.m1.1.1">ℋ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.1.m1.1c">\mathcal{H}</annotation></semantics></math>, the image <math id="S3.SS2.p3.2.m2.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S3.SS2.p3.2.m2.1a"><mi id="S3.SS2.p3.2.m2.1.1" xref="S3.SS2.p3.2.m2.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.2.m2.1b"><ci id="S3.SS2.p3.2.m2.1.1.cmml" xref="S3.SS2.p3.2.m2.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.2.m2.1c">x</annotation></semantics></math> with size <math id="S3.SS2.p3.3.m3.2" class="ltx_Math" alttext="[h,w]" display="inline"><semantics id="S3.SS2.p3.3.m3.2a"><mrow id="S3.SS2.p3.3.m3.2.3.2" xref="S3.SS2.p3.3.m3.2.3.1.cmml"><mo stretchy="false" id="S3.SS2.p3.3.m3.2.3.2.1" xref="S3.SS2.p3.3.m3.2.3.1.cmml">[</mo><mi id="S3.SS2.p3.3.m3.1.1" xref="S3.SS2.p3.3.m3.1.1.cmml">h</mi><mo id="S3.SS2.p3.3.m3.2.3.2.2" xref="S3.SS2.p3.3.m3.2.3.1.cmml">,</mo><mi id="S3.SS2.p3.3.m3.2.2" xref="S3.SS2.p3.3.m3.2.2.cmml">w</mi><mo stretchy="false" id="S3.SS2.p3.3.m3.2.3.2.3" xref="S3.SS2.p3.3.m3.2.3.1.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.3.m3.2b"><interval closure="closed" id="S3.SS2.p3.3.m3.2.3.1.cmml" xref="S3.SS2.p3.3.m3.2.3.2"><ci id="S3.SS2.p3.3.m3.1.1.cmml" xref="S3.SS2.p3.3.m3.1.1">ℎ</ci><ci id="S3.SS2.p3.3.m3.2.2.cmml" xref="S3.SS2.p3.3.m3.2.2">𝑤</ci></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.3.m3.2c">[h,w]</annotation></semantics></math> is resized and padded to the new image <math id="S3.SS2.p3.4.m4.1" class="ltx_Math" alttext="\hat{x}" display="inline"><semantics id="S3.SS2.p3.4.m4.1a"><mover accent="true" id="S3.SS2.p3.4.m4.1.1" xref="S3.SS2.p3.4.m4.1.1.cmml"><mi id="S3.SS2.p3.4.m4.1.1.2" xref="S3.SS2.p3.4.m4.1.1.2.cmml">x</mi><mo id="S3.SS2.p3.4.m4.1.1.1" xref="S3.SS2.p3.4.m4.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.4.m4.1b"><apply id="S3.SS2.p3.4.m4.1.1.cmml" xref="S3.SS2.p3.4.m4.1.1"><ci id="S3.SS2.p3.4.m4.1.1.1.cmml" xref="S3.SS2.p3.4.m4.1.1.1">^</ci><ci id="S3.SS2.p3.4.m4.1.1.2.cmml" xref="S3.SS2.p3.4.m4.1.1.2">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.4.m4.1c">\hat{x}</annotation></semantics></math> with size <math id="S3.SS2.p3.5.m5.2" class="ltx_Math" alttext="[p_{h}\times 560,p_{w}\times 560]" display="inline"><semantics id="S3.SS2.p3.5.m5.2a"><mrow id="S3.SS2.p3.5.m5.2.2.2" xref="S3.SS2.p3.5.m5.2.2.3.cmml"><mo stretchy="false" id="S3.SS2.p3.5.m5.2.2.2.3" xref="S3.SS2.p3.5.m5.2.2.3.cmml">[</mo><mrow id="S3.SS2.p3.5.m5.1.1.1.1" xref="S3.SS2.p3.5.m5.1.1.1.1.cmml"><msub id="S3.SS2.p3.5.m5.1.1.1.1.2" xref="S3.SS2.p3.5.m5.1.1.1.1.2.cmml"><mi id="S3.SS2.p3.5.m5.1.1.1.1.2.2" xref="S3.SS2.p3.5.m5.1.1.1.1.2.2.cmml">p</mi><mi id="S3.SS2.p3.5.m5.1.1.1.1.2.3" xref="S3.SS2.p3.5.m5.1.1.1.1.2.3.cmml">h</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p3.5.m5.1.1.1.1.1" xref="S3.SS2.p3.5.m5.1.1.1.1.1.cmml">×</mo><mn id="S3.SS2.p3.5.m5.1.1.1.1.3" xref="S3.SS2.p3.5.m5.1.1.1.1.3.cmml">560</mn></mrow><mo id="S3.SS2.p3.5.m5.2.2.2.4" xref="S3.SS2.p3.5.m5.2.2.3.cmml">,</mo><mrow id="S3.SS2.p3.5.m5.2.2.2.2" xref="S3.SS2.p3.5.m5.2.2.2.2.cmml"><msub id="S3.SS2.p3.5.m5.2.2.2.2.2" xref="S3.SS2.p3.5.m5.2.2.2.2.2.cmml"><mi id="S3.SS2.p3.5.m5.2.2.2.2.2.2" xref="S3.SS2.p3.5.m5.2.2.2.2.2.2.cmml">p</mi><mi id="S3.SS2.p3.5.m5.2.2.2.2.2.3" xref="S3.SS2.p3.5.m5.2.2.2.2.2.3.cmml">w</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p3.5.m5.2.2.2.2.1" xref="S3.SS2.p3.5.m5.2.2.2.2.1.cmml">×</mo><mn id="S3.SS2.p3.5.m5.2.2.2.2.3" xref="S3.SS2.p3.5.m5.2.2.2.2.3.cmml">560</mn></mrow><mo stretchy="false" id="S3.SS2.p3.5.m5.2.2.2.5" xref="S3.SS2.p3.5.m5.2.2.3.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.5.m5.2b"><interval closure="closed" id="S3.SS2.p3.5.m5.2.2.3.cmml" xref="S3.SS2.p3.5.m5.2.2.2"><apply id="S3.SS2.p3.5.m5.1.1.1.1.cmml" xref="S3.SS2.p3.5.m5.1.1.1.1"><times id="S3.SS2.p3.5.m5.1.1.1.1.1.cmml" xref="S3.SS2.p3.5.m5.1.1.1.1.1"></times><apply id="S3.SS2.p3.5.m5.1.1.1.1.2.cmml" xref="S3.SS2.p3.5.m5.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p3.5.m5.1.1.1.1.2.1.cmml" xref="S3.SS2.p3.5.m5.1.1.1.1.2">subscript</csymbol><ci id="S3.SS2.p3.5.m5.1.1.1.1.2.2.cmml" xref="S3.SS2.p3.5.m5.1.1.1.1.2.2">𝑝</ci><ci id="S3.SS2.p3.5.m5.1.1.1.1.2.3.cmml" xref="S3.SS2.p3.5.m5.1.1.1.1.2.3">ℎ</ci></apply><cn type="integer" id="S3.SS2.p3.5.m5.1.1.1.1.3.cmml" xref="S3.SS2.p3.5.m5.1.1.1.1.3">560</cn></apply><apply id="S3.SS2.p3.5.m5.2.2.2.2.cmml" xref="S3.SS2.p3.5.m5.2.2.2.2"><times id="S3.SS2.p3.5.m5.2.2.2.2.1.cmml" xref="S3.SS2.p3.5.m5.2.2.2.2.1"></times><apply id="S3.SS2.p3.5.m5.2.2.2.2.2.cmml" xref="S3.SS2.p3.5.m5.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.p3.5.m5.2.2.2.2.2.1.cmml" xref="S3.SS2.p3.5.m5.2.2.2.2.2">subscript</csymbol><ci id="S3.SS2.p3.5.m5.2.2.2.2.2.2.cmml" xref="S3.SS2.p3.5.m5.2.2.2.2.2.2">𝑝</ci><ci id="S3.SS2.p3.5.m5.2.2.2.2.2.3.cmml" xref="S3.SS2.p3.5.m5.2.2.2.2.2.3">𝑤</ci></apply><cn type="integer" id="S3.SS2.p3.5.m5.2.2.2.2.3.cmml" xref="S3.SS2.p3.5.m5.2.2.2.2.3">560</cn></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.5.m5.2c">[p_{h}\times 560,p_{w}\times 560]</annotation></semantics></math>. This process is subject to the following constraints:</p>
<table id="S5.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E1.m1.1" class="ltx_Math" alttext="\displaystyle p_{w1}" display="inline"><semantics id="S3.E1.m1.1a"><msub id="S3.E1.m1.1.1" xref="S3.E1.m1.1.1.cmml"><mi id="S3.E1.m1.1.1.2" xref="S3.E1.m1.1.1.2.cmml">p</mi><mrow id="S3.E1.m1.1.1.3" xref="S3.E1.m1.1.1.3.cmml"><mi id="S3.E1.m1.1.1.3.2" xref="S3.E1.m1.1.1.3.2.cmml">w</mi><mo lspace="0em" rspace="0em" id="S3.E1.m1.1.1.3.1" xref="S3.E1.m1.1.1.3.1.cmml">​</mo><mn id="S3.E1.m1.1.1.3.3" xref="S3.E1.m1.1.1.3.3.cmml">1</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.E1.m1.1b"><apply id="S3.E1.m1.1.1.cmml" xref="S3.E1.m1.1.1"><csymbol cd="ambiguous" id="S3.E1.m1.1.1.1.cmml" xref="S3.E1.m1.1.1">subscript</csymbol><ci id="S3.E1.m1.1.1.2.cmml" xref="S3.E1.m1.1.1.2">𝑝</ci><apply id="S3.E1.m1.1.1.3.cmml" xref="S3.E1.m1.1.1.3"><times id="S3.E1.m1.1.1.3.1.cmml" xref="S3.E1.m1.1.1.3.1"></times><ci id="S3.E1.m1.1.1.3.2.cmml" xref="S3.E1.m1.1.1.3.2">𝑤</ci><cn type="integer" id="S3.E1.m1.1.1.3.3.cmml" xref="S3.E1.m1.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E1.m1.1c">\displaystyle p_{w1}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E1.m2.1" class="ltx_math_unparsed" alttext="\displaystyle\times\lceil p_{w1}\times h/w\rceil\leq\mathcal{H};" display="inline"><semantics id="S3.E1.m2.1a"><mrow id="S3.E1.m2.1b"><mo rspace="0.222em" id="S3.E1.m2.1.1">×</mo><mrow id="S3.E1.m2.1.2"><mo stretchy="false" id="S3.E1.m2.1.2.1">⌈</mo><msub id="S3.E1.m2.1.2.2"><mi id="S3.E1.m2.1.2.2.2">p</mi><mrow id="S3.E1.m2.1.2.2.3"><mi id="S3.E1.m2.1.2.2.3.2">w</mi><mo lspace="0em" rspace="0em" id="S3.E1.m2.1.2.2.3.1">​</mo><mn id="S3.E1.m2.1.2.2.3.3">1</mn></mrow></msub><mo lspace="0.222em" rspace="0.222em" id="S3.E1.m2.1.2.3">×</mo><mi id="S3.E1.m2.1.2.4">h</mi><mo id="S3.E1.m2.1.2.5">/</mo><mi id="S3.E1.m2.1.2.6">w</mi><mo stretchy="false" id="S3.E1.m2.1.2.7">⌉</mo></mrow><mo id="S3.E1.m2.1.3">≤</mo><mi class="ltx_font_mathcaligraphic" id="S3.E1.m2.1.4">ℋ</mi><mo id="S3.E1.m2.1.5">;</mo></mrow><annotation encoding="application/x-tex" id="S3.E1.m2.1c">\displaystyle\times\lceil p_{w1}\times h/w\rceil\leq\mathcal{H};</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
<tbody id="S3.E2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E2.m1.1" class="ltx_Math" alttext="\displaystyle p_{w2}" display="inline"><semantics id="S3.E2.m1.1a"><msub id="S3.E2.m1.1.1" xref="S3.E2.m1.1.1.cmml"><mi id="S3.E2.m1.1.1.2" xref="S3.E2.m1.1.1.2.cmml">p</mi><mrow id="S3.E2.m1.1.1.3" xref="S3.E2.m1.1.1.3.cmml"><mi id="S3.E2.m1.1.1.3.2" xref="S3.E2.m1.1.1.3.2.cmml">w</mi><mo lspace="0em" rspace="0em" id="S3.E2.m1.1.1.3.1" xref="S3.E2.m1.1.1.3.1.cmml">​</mo><mn id="S3.E2.m1.1.1.3.3" xref="S3.E2.m1.1.1.3.3.cmml">2</mn></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.E2.m1.1b"><apply id="S3.E2.m1.1.1.cmml" xref="S3.E2.m1.1.1"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.1.cmml" xref="S3.E2.m1.1.1">subscript</csymbol><ci id="S3.E2.m1.1.1.2.cmml" xref="S3.E2.m1.1.1.2">𝑝</ci><apply id="S3.E2.m1.1.1.3.cmml" xref="S3.E2.m1.1.1.3"><times id="S3.E2.m1.1.1.3.1.cmml" xref="S3.E2.m1.1.1.3.1"></times><ci id="S3.E2.m1.1.1.3.2.cmml" xref="S3.E2.m1.1.1.3.2">𝑤</ci><cn type="integer" id="S3.E2.m1.1.1.3.3.cmml" xref="S3.E2.m1.1.1.3.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E2.m1.1c">\displaystyle p_{w2}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E2.m2.1" class="ltx_Math" alttext="\displaystyle=\lceil w*s/560\rceil\;" display="inline"><semantics id="S3.E2.m2.1a"><mrow id="S3.E2.m2.1.1" xref="S3.E2.m2.1.1.cmml"><mi id="S3.E2.m2.1.1.3" xref="S3.E2.m2.1.1.3.cmml"></mi><mo id="S3.E2.m2.1.1.2" xref="S3.E2.m2.1.1.2.cmml">=</mo><mrow id="S3.E2.m2.1.1.1.1" xref="S3.E2.m2.1.1.1.2.cmml"><mo stretchy="false" id="S3.E2.m2.1.1.1.1.2" xref="S3.E2.m2.1.1.1.2.1.cmml">⌈</mo><mrow id="S3.E2.m2.1.1.1.1.1" xref="S3.E2.m2.1.1.1.1.1.cmml"><mrow id="S3.E2.m2.1.1.1.1.1.2" xref="S3.E2.m2.1.1.1.1.1.2.cmml"><mi id="S3.E2.m2.1.1.1.1.1.2.2" xref="S3.E2.m2.1.1.1.1.1.2.2.cmml">w</mi><mo lspace="0.222em" rspace="0.222em" id="S3.E2.m2.1.1.1.1.1.2.1" xref="S3.E2.m2.1.1.1.1.1.2.1.cmml">∗</mo><mi id="S3.E2.m2.1.1.1.1.1.2.3" xref="S3.E2.m2.1.1.1.1.1.2.3.cmml">s</mi></mrow><mo id="S3.E2.m2.1.1.1.1.1.1" xref="S3.E2.m2.1.1.1.1.1.1.cmml">/</mo><mn id="S3.E2.m2.1.1.1.1.1.3" xref="S3.E2.m2.1.1.1.1.1.3.cmml">560</mn></mrow><mo stretchy="false" id="S3.E2.m2.1.1.1.1.3" xref="S3.E2.m2.1.1.1.2.1.cmml">⌉</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E2.m2.1b"><apply id="S3.E2.m2.1.1.cmml" xref="S3.E2.m2.1.1"><eq id="S3.E2.m2.1.1.2.cmml" xref="S3.E2.m2.1.1.2"></eq><csymbol cd="latexml" id="S3.E2.m2.1.1.3.cmml" xref="S3.E2.m2.1.1.3">absent</csymbol><apply id="S3.E2.m2.1.1.1.2.cmml" xref="S3.E2.m2.1.1.1.1"><ceiling id="S3.E2.m2.1.1.1.2.1.cmml" xref="S3.E2.m2.1.1.1.1.2"></ceiling><apply id="S3.E2.m2.1.1.1.1.1.cmml" xref="S3.E2.m2.1.1.1.1.1"><divide id="S3.E2.m2.1.1.1.1.1.1.cmml" xref="S3.E2.m2.1.1.1.1.1.1"></divide><apply id="S3.E2.m2.1.1.1.1.1.2.cmml" xref="S3.E2.m2.1.1.1.1.1.2"><times id="S3.E2.m2.1.1.1.1.1.2.1.cmml" xref="S3.E2.m2.1.1.1.1.1.2.1"></times><ci id="S3.E2.m2.1.1.1.1.1.2.2.cmml" xref="S3.E2.m2.1.1.1.1.1.2.2">𝑤</ci><ci id="S3.E2.m2.1.1.1.1.1.2.3.cmml" xref="S3.E2.m2.1.1.1.1.1.2.3">𝑠</ci></apply><cn type="integer" id="S3.E2.m2.1.1.1.1.1.3.cmml" xref="S3.E2.m2.1.1.1.1.1.3">560</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E2.m2.1c">\displaystyle=\lceil w*s/560\rceil\;</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
<tbody id="S3.E3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E3.m1.1" class="ltx_Math" alttext="\displaystyle p_{w}" display="inline"><semantics id="S3.E3.m1.1a"><msub id="S3.E3.m1.1.1" xref="S3.E3.m1.1.1.cmml"><mi id="S3.E3.m1.1.1.2" xref="S3.E3.m1.1.1.2.cmml">p</mi><mi id="S3.E3.m1.1.1.3" xref="S3.E3.m1.1.1.3.cmml">w</mi></msub><annotation-xml encoding="MathML-Content" id="S3.E3.m1.1b"><apply id="S3.E3.m1.1.1.cmml" xref="S3.E3.m1.1.1"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.cmml" xref="S3.E3.m1.1.1">subscript</csymbol><ci id="S3.E3.m1.1.1.2.cmml" xref="S3.E3.m1.1.1.2">𝑝</ci><ci id="S3.E3.m1.1.1.3.cmml" xref="S3.E3.m1.1.1.3">𝑤</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m1.1c">\displaystyle p_{w}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E3.m2.2" class="ltx_Math" alttext="\displaystyle=min(p_{w1},p_{w2})\;\;;p_{h}=\lceil p_{w}\times h/w\rceil" display="inline"><semantics id="S3.E3.m2.2a"><mrow id="S3.E3.m2.2.2.2" xref="S3.E3.m2.2.2.3.cmml"><mrow id="S3.E3.m2.1.1.1.1" xref="S3.E3.m2.1.1.1.1.cmml"><mi id="S3.E3.m2.1.1.1.1.4" xref="S3.E3.m2.1.1.1.1.4.cmml"></mi><mo id="S3.E3.m2.1.1.1.1.3" xref="S3.E3.m2.1.1.1.1.3.cmml">=</mo><mrow id="S3.E3.m2.1.1.1.1.2" xref="S3.E3.m2.1.1.1.1.2.cmml"><mi id="S3.E3.m2.1.1.1.1.2.4" xref="S3.E3.m2.1.1.1.1.2.4.cmml">m</mi><mo lspace="0em" rspace="0em" id="S3.E3.m2.1.1.1.1.2.3" xref="S3.E3.m2.1.1.1.1.2.3.cmml">​</mo><mi id="S3.E3.m2.1.1.1.1.2.5" xref="S3.E3.m2.1.1.1.1.2.5.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.E3.m2.1.1.1.1.2.3a" xref="S3.E3.m2.1.1.1.1.2.3.cmml">​</mo><mi id="S3.E3.m2.1.1.1.1.2.6" xref="S3.E3.m2.1.1.1.1.2.6.cmml">n</mi><mo lspace="0em" rspace="0em" id="S3.E3.m2.1.1.1.1.2.3b" xref="S3.E3.m2.1.1.1.1.2.3.cmml">​</mo><mrow id="S3.E3.m2.1.1.1.1.2.2.2" xref="S3.E3.m2.1.1.1.1.2.2.3.cmml"><mo stretchy="false" id="S3.E3.m2.1.1.1.1.2.2.2.3" xref="S3.E3.m2.1.1.1.1.2.2.3.cmml">(</mo><msub id="S3.E3.m2.1.1.1.1.1.1.1.1" xref="S3.E3.m2.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E3.m2.1.1.1.1.1.1.1.1.2" xref="S3.E3.m2.1.1.1.1.1.1.1.1.2.cmml">p</mi><mrow id="S3.E3.m2.1.1.1.1.1.1.1.1.3" xref="S3.E3.m2.1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.E3.m2.1.1.1.1.1.1.1.1.3.2" xref="S3.E3.m2.1.1.1.1.1.1.1.1.3.2.cmml">w</mi><mo lspace="0em" rspace="0em" id="S3.E3.m2.1.1.1.1.1.1.1.1.3.1" xref="S3.E3.m2.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mn id="S3.E3.m2.1.1.1.1.1.1.1.1.3.3" xref="S3.E3.m2.1.1.1.1.1.1.1.1.3.3.cmml">1</mn></mrow></msub><mo id="S3.E3.m2.1.1.1.1.2.2.2.4" xref="S3.E3.m2.1.1.1.1.2.2.3.cmml">,</mo><msub id="S3.E3.m2.1.1.1.1.2.2.2.2" xref="S3.E3.m2.1.1.1.1.2.2.2.2.cmml"><mi id="S3.E3.m2.1.1.1.1.2.2.2.2.2" xref="S3.E3.m2.1.1.1.1.2.2.2.2.2.cmml">p</mi><mrow id="S3.E3.m2.1.1.1.1.2.2.2.2.3" xref="S3.E3.m2.1.1.1.1.2.2.2.2.3.cmml"><mi id="S3.E3.m2.1.1.1.1.2.2.2.2.3.2" xref="S3.E3.m2.1.1.1.1.2.2.2.2.3.2.cmml">w</mi><mo lspace="0em" rspace="0em" id="S3.E3.m2.1.1.1.1.2.2.2.2.3.1" xref="S3.E3.m2.1.1.1.1.2.2.2.2.3.1.cmml">​</mo><mn id="S3.E3.m2.1.1.1.1.2.2.2.2.3.3" xref="S3.E3.m2.1.1.1.1.2.2.2.2.3.3.cmml">2</mn></mrow></msub><mo rspace="0.560em" stretchy="false" id="S3.E3.m2.1.1.1.1.2.2.2.5" xref="S3.E3.m2.1.1.1.1.2.2.3.cmml">)</mo></mrow></mrow></mrow><mo id="S3.E3.m2.2.2.2.3" xref="S3.E3.m2.2.2.3a.cmml">;</mo><mrow id="S3.E3.m2.2.2.2.2" xref="S3.E3.m2.2.2.2.2.cmml"><msub id="S3.E3.m2.2.2.2.2.3" xref="S3.E3.m2.2.2.2.2.3.cmml"><mi id="S3.E3.m2.2.2.2.2.3.2" xref="S3.E3.m2.2.2.2.2.3.2.cmml">p</mi><mi id="S3.E3.m2.2.2.2.2.3.3" xref="S3.E3.m2.2.2.2.2.3.3.cmml">h</mi></msub><mo id="S3.E3.m2.2.2.2.2.2" xref="S3.E3.m2.2.2.2.2.2.cmml">=</mo><mrow id="S3.E3.m2.2.2.2.2.1.1" xref="S3.E3.m2.2.2.2.2.1.2.cmml"><mo stretchy="false" id="S3.E3.m2.2.2.2.2.1.1.2" xref="S3.E3.m2.2.2.2.2.1.2.1.cmml">⌈</mo><mrow id="S3.E3.m2.2.2.2.2.1.1.1" xref="S3.E3.m2.2.2.2.2.1.1.1.cmml"><mrow id="S3.E3.m2.2.2.2.2.1.1.1.2" xref="S3.E3.m2.2.2.2.2.1.1.1.2.cmml"><msub id="S3.E3.m2.2.2.2.2.1.1.1.2.2" xref="S3.E3.m2.2.2.2.2.1.1.1.2.2.cmml"><mi id="S3.E3.m2.2.2.2.2.1.1.1.2.2.2" xref="S3.E3.m2.2.2.2.2.1.1.1.2.2.2.cmml">p</mi><mi id="S3.E3.m2.2.2.2.2.1.1.1.2.2.3" xref="S3.E3.m2.2.2.2.2.1.1.1.2.2.3.cmml">w</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S3.E3.m2.2.2.2.2.1.1.1.2.1" xref="S3.E3.m2.2.2.2.2.1.1.1.2.1.cmml">×</mo><mi id="S3.E3.m2.2.2.2.2.1.1.1.2.3" xref="S3.E3.m2.2.2.2.2.1.1.1.2.3.cmml">h</mi></mrow><mo id="S3.E3.m2.2.2.2.2.1.1.1.1" xref="S3.E3.m2.2.2.2.2.1.1.1.1.cmml">/</mo><mi id="S3.E3.m2.2.2.2.2.1.1.1.3" xref="S3.E3.m2.2.2.2.2.1.1.1.3.cmml">w</mi></mrow><mo stretchy="false" id="S3.E3.m2.2.2.2.2.1.1.3" xref="S3.E3.m2.2.2.2.2.1.2.1.cmml">⌉</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E3.m2.2b"><apply id="S3.E3.m2.2.2.3.cmml" xref="S3.E3.m2.2.2.2"><csymbol cd="ambiguous" id="S3.E3.m2.2.2.3a.cmml" xref="S3.E3.m2.2.2.2.3">formulae-sequence</csymbol><apply id="S3.E3.m2.1.1.1.1.cmml" xref="S3.E3.m2.1.1.1.1"><eq id="S3.E3.m2.1.1.1.1.3.cmml" xref="S3.E3.m2.1.1.1.1.3"></eq><csymbol cd="latexml" id="S3.E3.m2.1.1.1.1.4.cmml" xref="S3.E3.m2.1.1.1.1.4">absent</csymbol><apply id="S3.E3.m2.1.1.1.1.2.cmml" xref="S3.E3.m2.1.1.1.1.2"><times id="S3.E3.m2.1.1.1.1.2.3.cmml" xref="S3.E3.m2.1.1.1.1.2.3"></times><ci id="S3.E3.m2.1.1.1.1.2.4.cmml" xref="S3.E3.m2.1.1.1.1.2.4">𝑚</ci><ci id="S3.E3.m2.1.1.1.1.2.5.cmml" xref="S3.E3.m2.1.1.1.1.2.5">𝑖</ci><ci id="S3.E3.m2.1.1.1.1.2.6.cmml" xref="S3.E3.m2.1.1.1.1.2.6">𝑛</ci><interval closure="open" id="S3.E3.m2.1.1.1.1.2.2.3.cmml" xref="S3.E3.m2.1.1.1.1.2.2.2"><apply id="S3.E3.m2.1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m2.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E3.m2.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m2.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E3.m2.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E3.m2.1.1.1.1.1.1.1.1.2">𝑝</ci><apply id="S3.E3.m2.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E3.m2.1.1.1.1.1.1.1.1.3"><times id="S3.E3.m2.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E3.m2.1.1.1.1.1.1.1.1.3.1"></times><ci id="S3.E3.m2.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E3.m2.1.1.1.1.1.1.1.1.3.2">𝑤</ci><cn type="integer" id="S3.E3.m2.1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E3.m2.1.1.1.1.1.1.1.1.3.3">1</cn></apply></apply><apply id="S3.E3.m2.1.1.1.1.2.2.2.2.cmml" xref="S3.E3.m2.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E3.m2.1.1.1.1.2.2.2.2.1.cmml" xref="S3.E3.m2.1.1.1.1.2.2.2.2">subscript</csymbol><ci id="S3.E3.m2.1.1.1.1.2.2.2.2.2.cmml" xref="S3.E3.m2.1.1.1.1.2.2.2.2.2">𝑝</ci><apply id="S3.E3.m2.1.1.1.1.2.2.2.2.3.cmml" xref="S3.E3.m2.1.1.1.1.2.2.2.2.3"><times id="S3.E3.m2.1.1.1.1.2.2.2.2.3.1.cmml" xref="S3.E3.m2.1.1.1.1.2.2.2.2.3.1"></times><ci id="S3.E3.m2.1.1.1.1.2.2.2.2.3.2.cmml" xref="S3.E3.m2.1.1.1.1.2.2.2.2.3.2">𝑤</ci><cn type="integer" id="S3.E3.m2.1.1.1.1.2.2.2.2.3.3.cmml" xref="S3.E3.m2.1.1.1.1.2.2.2.2.3.3">2</cn></apply></apply></interval></apply></apply><apply id="S3.E3.m2.2.2.2.2.cmml" xref="S3.E3.m2.2.2.2.2"><eq id="S3.E3.m2.2.2.2.2.2.cmml" xref="S3.E3.m2.2.2.2.2.2"></eq><apply id="S3.E3.m2.2.2.2.2.3.cmml" xref="S3.E3.m2.2.2.2.2.3"><csymbol cd="ambiguous" id="S3.E3.m2.2.2.2.2.3.1.cmml" xref="S3.E3.m2.2.2.2.2.3">subscript</csymbol><ci id="S3.E3.m2.2.2.2.2.3.2.cmml" xref="S3.E3.m2.2.2.2.2.3.2">𝑝</ci><ci id="S3.E3.m2.2.2.2.2.3.3.cmml" xref="S3.E3.m2.2.2.2.2.3.3">ℎ</ci></apply><apply id="S3.E3.m2.2.2.2.2.1.2.cmml" xref="S3.E3.m2.2.2.2.2.1.1"><ceiling id="S3.E3.m2.2.2.2.2.1.2.1.cmml" xref="S3.E3.m2.2.2.2.2.1.1.2"></ceiling><apply id="S3.E3.m2.2.2.2.2.1.1.1.cmml" xref="S3.E3.m2.2.2.2.2.1.1.1"><divide id="S3.E3.m2.2.2.2.2.1.1.1.1.cmml" xref="S3.E3.m2.2.2.2.2.1.1.1.1"></divide><apply id="S3.E3.m2.2.2.2.2.1.1.1.2.cmml" xref="S3.E3.m2.2.2.2.2.1.1.1.2"><times id="S3.E3.m2.2.2.2.2.1.1.1.2.1.cmml" xref="S3.E3.m2.2.2.2.2.1.1.1.2.1"></times><apply id="S3.E3.m2.2.2.2.2.1.1.1.2.2.cmml" xref="S3.E3.m2.2.2.2.2.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.E3.m2.2.2.2.2.1.1.1.2.2.1.cmml" xref="S3.E3.m2.2.2.2.2.1.1.1.2.2">subscript</csymbol><ci id="S3.E3.m2.2.2.2.2.1.1.1.2.2.2.cmml" xref="S3.E3.m2.2.2.2.2.1.1.1.2.2.2">𝑝</ci><ci id="S3.E3.m2.2.2.2.2.1.1.1.2.2.3.cmml" xref="S3.E3.m2.2.2.2.2.1.1.1.2.2.3">𝑤</ci></apply><ci id="S3.E3.m2.2.2.2.2.1.1.1.2.3.cmml" xref="S3.E3.m2.2.2.2.2.1.1.1.2.3">ℎ</ci></apply><ci id="S3.E3.m2.2.2.2.2.1.1.1.3.cmml" xref="S3.E3.m2.2.2.2.2.1.1.1.3">𝑤</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m2.2c">\displaystyle=min(p_{w1},p_{w2})\;\;;p_{h}=\lceil p_{w}\times h/w\rceil</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S3.SS2.p3.8" class="ltx_p">where <math id="S3.SS2.p3.6.m1.1" class="ltx_Math" alttext="s" display="inline"><semantics id="S3.SS2.p3.6.m1.1a"><mi id="S3.SS2.p3.6.m1.1.1" xref="S3.SS2.p3.6.m1.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.6.m1.1b"><ci id="S3.SS2.p3.6.m1.1.1.cmml" xref="S3.SS2.p3.6.m1.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.6.m1.1c">s</annotation></semantics></math> is the scale factor, <math id="S3.SS2.p3.7.m2.1" class="ltx_Math" alttext="p_{w}" display="inline"><semantics id="S3.SS2.p3.7.m2.1a"><msub id="S3.SS2.p3.7.m2.1.1" xref="S3.SS2.p3.7.m2.1.1.cmml"><mi id="S3.SS2.p3.7.m2.1.1.2" xref="S3.SS2.p3.7.m2.1.1.2.cmml">p</mi><mi id="S3.SS2.p3.7.m2.1.1.3" xref="S3.SS2.p3.7.m2.1.1.3.cmml">w</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.7.m2.1b"><apply id="S3.SS2.p3.7.m2.1.1.cmml" xref="S3.SS2.p3.7.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.p3.7.m2.1.1.1.cmml" xref="S3.SS2.p3.7.m2.1.1">subscript</csymbol><ci id="S3.SS2.p3.7.m2.1.1.2.cmml" xref="S3.SS2.p3.7.m2.1.1.2">𝑝</ci><ci id="S3.SS2.p3.7.m2.1.1.3.cmml" xref="S3.SS2.p3.7.m2.1.1.3">𝑤</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.7.m2.1c">p_{w}</annotation></semantics></math> and <math id="S3.SS2.p3.8.m3.1" class="ltx_Math" alttext="p_{h}" display="inline"><semantics id="S3.SS2.p3.8.m3.1a"><msub id="S3.SS2.p3.8.m3.1.1" xref="S3.SS2.p3.8.m3.1.1.cmml"><mi id="S3.SS2.p3.8.m3.1.1.2" xref="S3.SS2.p3.8.m3.1.1.2.cmml">p</mi><mi id="S3.SS2.p3.8.m3.1.1.3" xref="S3.SS2.p3.8.m3.1.1.3.cmml">h</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.8.m3.1b"><apply id="S3.SS2.p3.8.m3.1.1.cmml" xref="S3.SS2.p3.8.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.p3.8.m3.1.1.1.cmml" xref="S3.SS2.p3.8.m3.1.1">subscript</csymbol><ci id="S3.SS2.p3.8.m3.1.1.2.cmml" xref="S3.SS2.p3.8.m3.1.1.2">𝑝</ci><ci id="S3.SS2.p3.8.m3.1.1.3.cmml" xref="S3.SS2.p3.8.m3.1.1.3">ℎ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.8.m3.1c">p_{h}</annotation></semantics></math> represent the number of patches in each row and column, respectively.</p>
</div>
<div id="S3.SS2.p4" class="ltx_para">
<p id="S3.SS2.p4.3" class="ltx_p">For multi-image input, we assign an index to each image like <math id="S3.SS2.p4.1.m1.1" class="ltx_Math" alttext="&lt;" display="inline"><semantics id="S3.SS2.p4.1.m1.1a"><mo id="S3.SS2.p4.1.m1.1.1" xref="S3.SS2.p4.1.m1.1.1.cmml">&lt;</mo><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.1.m1.1b"><lt id="S3.SS2.p4.1.m1.1.1.cmml" xref="S3.SS2.p4.1.m1.1.1"></lt></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.1.m1.1c">&lt;</annotation></semantics></math><span id="S3.SS2.p4.3.1" class="ltx_text">IMAGE</span> i<math id="S3.SS2.p4.2.m2.1" class="ltx_Math" alttext="&gt;" display="inline"><semantics id="S3.SS2.p4.2.m2.1a"><mo id="S3.SS2.p4.2.m2.1.1" xref="S3.SS2.p4.2.m2.1.1.cmml">&gt;</mo><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.2.m2.1b"><gt id="S3.SS2.p4.2.m2.1.1.cmml" xref="S3.SS2.p4.2.m2.1.1"></gt></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.2.m2.1c">&gt;</annotation></semantics></math>, <math id="S3.SS2.p4.3.m3.4" class="ltx_Math" alttext="i\in\{1,2,3,\ldots\}" display="inline"><semantics id="S3.SS2.p4.3.m3.4a"><mrow id="S3.SS2.p4.3.m3.4.5" xref="S3.SS2.p4.3.m3.4.5.cmml"><mi id="S3.SS2.p4.3.m3.4.5.2" xref="S3.SS2.p4.3.m3.4.5.2.cmml">i</mi><mo id="S3.SS2.p4.3.m3.4.5.1" xref="S3.SS2.p4.3.m3.4.5.1.cmml">∈</mo><mrow id="S3.SS2.p4.3.m3.4.5.3.2" xref="S3.SS2.p4.3.m3.4.5.3.1.cmml"><mo stretchy="false" id="S3.SS2.p4.3.m3.4.5.3.2.1" xref="S3.SS2.p4.3.m3.4.5.3.1.cmml">{</mo><mn id="S3.SS2.p4.3.m3.1.1" xref="S3.SS2.p4.3.m3.1.1.cmml">1</mn><mo id="S3.SS2.p4.3.m3.4.5.3.2.2" xref="S3.SS2.p4.3.m3.4.5.3.1.cmml">,</mo><mn id="S3.SS2.p4.3.m3.2.2" xref="S3.SS2.p4.3.m3.2.2.cmml">2</mn><mo id="S3.SS2.p4.3.m3.4.5.3.2.3" xref="S3.SS2.p4.3.m3.4.5.3.1.cmml">,</mo><mn id="S3.SS2.p4.3.m3.3.3" xref="S3.SS2.p4.3.m3.3.3.cmml">3</mn><mo id="S3.SS2.p4.3.m3.4.5.3.2.4" xref="S3.SS2.p4.3.m3.4.5.3.1.cmml">,</mo><mi mathvariant="normal" id="S3.SS2.p4.3.m3.4.4" xref="S3.SS2.p4.3.m3.4.4.cmml">…</mi><mo stretchy="false" id="S3.SS2.p4.3.m3.4.5.3.2.5" xref="S3.SS2.p4.3.m3.4.5.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.3.m3.4b"><apply id="S3.SS2.p4.3.m3.4.5.cmml" xref="S3.SS2.p4.3.m3.4.5"><in id="S3.SS2.p4.3.m3.4.5.1.cmml" xref="S3.SS2.p4.3.m3.4.5.1"></in><ci id="S3.SS2.p4.3.m3.4.5.2.cmml" xref="S3.SS2.p4.3.m3.4.5.2">𝑖</ci><set id="S3.SS2.p4.3.m3.4.5.3.1.cmml" xref="S3.SS2.p4.3.m3.4.5.3.2"><cn type="integer" id="S3.SS2.p4.3.m3.1.1.cmml" xref="S3.SS2.p4.3.m3.1.1">1</cn><cn type="integer" id="S3.SS2.p4.3.m3.2.2.cmml" xref="S3.SS2.p4.3.m3.2.2">2</cn><cn type="integer" id="S3.SS2.p4.3.m3.3.3.cmml" xref="S3.SS2.p4.3.m3.3.3">3</cn><ci id="S3.SS2.p4.3.m3.4.4.cmml" xref="S3.SS2.p4.3.m3.4.4">…</ci></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.3.m3.4c">i\in\{1,2,3,\ldots\}</annotation></semantics></math> and format the image and text in an interleaved format.</p>
</div>
<div id="S3.SS2.p5" class="ltx_para ltx_noindent">
<p id="S3.SS2.p5.1" class="ltx_p"><span id="S3.SS2.p5.1.1" class="ltx_text ltx_font_bold">Video Processing.</span>
We sample frames from the given video and concatenate them along the short side of the frame, leading to a high-resolution image. The frame index is also written in the image to provide the temporal relation.</p>
</div>
<div id="S3.SS2.p6" class="ltx_para ltx_noindent">
<p id="S3.SS2.p6.1" class="ltx_p"><span id="S3.SS2.p6.1.1" class="ltx_text ltx_font_bold">Audio Processing.</span>
IXC-2.5 web demo supports audio input and output using open-source tools. For audio input, we employ Whisper <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib123" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">123</span></a>]</cite> to transcribe audio into text. For audio output, we utilize MeloTTS <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib179" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">179</span></a>]</cite> to convert the text back into audio.</p>
</div>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Pre-training</h3>

<figure id="S3.T1" class="ltx_table">
<table id="S3.T1.2" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.T1.2.1.1" class="ltx_tr">
<th id="S3.T1.2.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S3.T1.2.1.1.1.1" class="ltx_text" style="font-size:80%;">Task</span></th>
<th id="S3.T1.2.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S3.T1.2.1.1.2.1" class="ltx_text" style="font-size:80%;">Dataset</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.T1.2.2.1" class="ltx_tr">
<th id="S3.T1.2.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S3.T1.2.2.1.1.1" class="ltx_text" style="font-size:80%;">General Semantic Alignment</span></th>
<td id="S3.T1.2.2.1.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S3.T1.2.2.1.2.1" class="ltx_text" style="font-size:80%;">ShareGPT4V-PT </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T1.2.2.1.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib17" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">17</span></a><span id="S3.T1.2.2.1.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T1.2.2.1.2.4" class="ltx_text" style="font-size:80%;">, COCO </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T1.2.2.1.2.5.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib21" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">21</span></a><span id="S3.T1.2.2.1.2.6.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T1.2.2.1.2.7" class="ltx_text" style="font-size:80%;">, Nocaps </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T1.2.2.1.2.8.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">1</span></a><span id="S3.T1.2.2.1.2.9.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T1.2.2.1.2.10" class="ltx_text" style="font-size:80%;">, TextCaps </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T1.2.2.1.2.11.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib132" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">132</span></a><span id="S3.T1.2.2.1.2.12.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T1.2.2.1.2.13" class="ltx_text" style="font-size:80%;">, LAION </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T1.2.2.1.2.14.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib125" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">125</span></a><span id="S3.T1.2.2.1.2.15.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T1.2.2.1.2.16" class="ltx_text" style="font-size:80%;">, SBU </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T1.2.2.1.2.17.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib114" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">114</span></a><span id="S3.T1.2.2.1.2.18.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T1.2.2.1.2.19" class="ltx_text" style="font-size:80%;">, CC 3M </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T1.2.2.1.2.20.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib129" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">129</span></a><span id="S3.T1.2.2.1.2.21.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T1.2.2.1.2.22" class="ltx_text" style="font-size:80%;"> ALLaVA </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T1.2.2.1.2.23.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib15" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">15</span></a><span id="S3.T1.2.2.1.2.24.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
</tr>
<tr id="S3.T1.2.3.2" class="ltx_tr">
<th id="S3.T1.2.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S3.T1.2.3.2.1.1" class="ltx_text" style="font-size:80%;">World Knowledge Alignment</span></th>
<td id="S3.T1.2.3.2.2" class="ltx_td ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S3.T1.2.3.2.2.1" class="ltx_text" style="font-size:80%;">Concept Data </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T1.2.3.2.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib173" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">173</span></a><span id="S3.T1.2.3.2.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
</tr>
<tr id="S3.T1.2.4.3" class="ltx_tr">
<th id="S3.T1.2.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S3.T1.2.4.3.1.1" class="ltx_text" style="font-size:80%;">Vision Capability Enhancement</span></th>
<td id="S3.T1.2.4.3.2" class="ltx_td ltx_align_left ltx_border_bb" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S3.T1.2.4.3.2.1" class="ltx_text" style="font-size:80%;">WanJuan </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T1.2.4.3.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib46" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">46</span></a><span id="S3.T1.2.4.3.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T1.2.4.3.2.4" class="ltx_text" style="font-size:80%;">, Flicker</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T1.2.4.3.2.5.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib160" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">160</span></a><span id="S3.T1.2.4.3.2.6.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T1.2.4.3.2.7" class="ltx_text" style="font-size:80%;">, MMC-Inst</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T1.2.4.3.2.8.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib82" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">82</span></a><span id="S3.T1.2.4.3.2.9.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T1.2.4.3.2.10" class="ltx_text" style="font-size:80%;">, RCTW-17</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T1.2.4.3.2.11.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib130" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">130</span></a><span id="S3.T1.2.4.3.2.12.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T1.2.4.3.2.13" class="ltx_text" style="font-size:80%;">, CTW</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T1.2.4.3.2.14.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib165" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">165</span></a><span id="S3.T1.2.4.3.2.15.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T1.2.4.3.2.16" class="ltx_text" style="font-size:80%;">, LSVT</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T1.2.4.3.2.17.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib137" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">137</span></a><span id="S3.T1.2.4.3.2.18.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T1.2.4.3.2.19" class="ltx_text" style="font-size:80%;">, ReCTs</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T1.2.4.3.2.20.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib175" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">175</span></a><span id="S3.T1.2.4.3.2.21.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T1.2.4.3.2.22" class="ltx_text" style="font-size:80%;">, ArT</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T1.2.4.3.2.23.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib28" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">28</span></a><span id="S3.T1.2.4.3.2.24.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_table"><span id="S3.T1.6.1.1" class="ltx_text" style="font-size:113%;">Table 1</span>: </span><span id="S3.T1.7.2" class="ltx_text ltx_font_bold" style="font-size:113%;">Datasets used for Pre-Training<span id="S3.T1.7.2.1" class="ltx_text ltx_font_medium">. The data are collected from diverse sources for the three objectives. </span></span></figcaption>
</figure>
<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">During the pre-training phase, the LLM (InternLM2-7B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib143" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">143</span></a>]</cite>) is frozen while both the vision encoder and Partial LoRA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib33" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">33</span></a>]</cite> are fine-tuned to align the visual tokens with the LLM.
The data used for pre-training is shown in Table <a href="#S3.T1" title="Table 1 ‣ 3.3 Pre-training ‣ 3 Method ‣ InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.</p>
</div>
<div id="S3.SS3.p2" class="ltx_para">
<p id="S3.SS3.p2.7" class="ltx_p">In practice, we employ the CLIP ViT-L-14-490 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib122" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">122</span></a>]</cite> from IXC2 as the vision encoder and further increase its resolution to <math id="S3.SS3.p2.1.m1.1" class="ltx_Math" alttext="560\times 560" display="inline"><semantics id="S3.SS3.p2.1.m1.1a"><mrow id="S3.SS3.p2.1.m1.1.1" xref="S3.SS3.p2.1.m1.1.1.cmml"><mn id="S3.SS3.p2.1.m1.1.1.2" xref="S3.SS3.p2.1.m1.1.1.2.cmml">560</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS3.p2.1.m1.1.1.1" xref="S3.SS3.p2.1.m1.1.1.1.cmml">×</mo><mn id="S3.SS3.p2.1.m1.1.1.3" xref="S3.SS3.p2.1.m1.1.1.3.cmml">560</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.1.m1.1b"><apply id="S3.SS3.p2.1.m1.1.1.cmml" xref="S3.SS3.p2.1.m1.1.1"><times id="S3.SS3.p2.1.m1.1.1.1.cmml" xref="S3.SS3.p2.1.m1.1.1.1"></times><cn type="integer" id="S3.SS3.p2.1.m1.1.1.2.cmml" xref="S3.SS3.p2.1.m1.1.1.2">560</cn><cn type="integer" id="S3.SS3.p2.1.m1.1.1.3.cmml" xref="S3.SS3.p2.1.m1.1.1.3">560</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.1.m1.1c">560\times 560</annotation></semantics></math>. For the Unified Dynamic Image Partition strategy <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib34" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">34</span></a>]</cite>, we set the maximum number <math id="S3.SS3.p2.2.m2.1" class="ltx_Math" alttext="\mathcal{H}=12" display="inline"><semantics id="S3.SS3.p2.2.m2.1a"><mrow id="S3.SS3.p2.2.m2.1.1" xref="S3.SS3.p2.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.2.m2.1.1.2" xref="S3.SS3.p2.2.m2.1.1.2.cmml">ℋ</mi><mo id="S3.SS3.p2.2.m2.1.1.1" xref="S3.SS3.p2.2.m2.1.1.1.cmml">=</mo><mn id="S3.SS3.p2.2.m2.1.1.3" xref="S3.SS3.p2.2.m2.1.1.3.cmml">12</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.2.m2.1b"><apply id="S3.SS3.p2.2.m2.1.1.cmml" xref="S3.SS3.p2.2.m2.1.1"><eq id="S3.SS3.p2.2.m2.1.1.1.cmml" xref="S3.SS3.p2.2.m2.1.1.1"></eq><ci id="S3.SS3.p2.2.m2.1.1.2.cmml" xref="S3.SS3.p2.2.m2.1.1.2">ℋ</ci><cn type="integer" id="S3.SS3.p2.2.m2.1.1.3.cmml" xref="S3.SS3.p2.2.m2.1.1.3">12</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.2.m2.1c">\mathcal{H}=12</annotation></semantics></math> for the pertaining.
For the Partial LoRA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib33" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">33</span></a>]</cite>, we set a rank of <math id="S3.SS3.p2.3.m3.1" class="ltx_Math" alttext="256" display="inline"><semantics id="S3.SS3.p2.3.m3.1a"><mn id="S3.SS3.p2.3.m3.1.1" xref="S3.SS3.p2.3.m3.1.1.cmml">256</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.3.m3.1b"><cn type="integer" id="S3.SS3.p2.3.m3.1.1.cmml" xref="S3.SS3.p2.3.m3.1.1">256</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.3.m3.1c">256</annotation></semantics></math> for all the linear layers in the LLM decoder block.
Our training process involves a batch size of 4096 and spans across 2 epochs. The learning rate linearly increases to <math id="S3.SS3.p2.4.m4.1" class="ltx_Math" alttext="2\times 10^{-4}" display="inline"><semantics id="S3.SS3.p2.4.m4.1a"><mrow id="S3.SS3.p2.4.m4.1.1" xref="S3.SS3.p2.4.m4.1.1.cmml"><mn id="S3.SS3.p2.4.m4.1.1.2" xref="S3.SS3.p2.4.m4.1.1.2.cmml">2</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS3.p2.4.m4.1.1.1" xref="S3.SS3.p2.4.m4.1.1.1.cmml">×</mo><msup id="S3.SS3.p2.4.m4.1.1.3" xref="S3.SS3.p2.4.m4.1.1.3.cmml"><mn id="S3.SS3.p2.4.m4.1.1.3.2" xref="S3.SS3.p2.4.m4.1.1.3.2.cmml">10</mn><mrow id="S3.SS3.p2.4.m4.1.1.3.3" xref="S3.SS3.p2.4.m4.1.1.3.3.cmml"><mo id="S3.SS3.p2.4.m4.1.1.3.3a" xref="S3.SS3.p2.4.m4.1.1.3.3.cmml">−</mo><mn id="S3.SS3.p2.4.m4.1.1.3.3.2" xref="S3.SS3.p2.4.m4.1.1.3.3.2.cmml">4</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.4.m4.1b"><apply id="S3.SS3.p2.4.m4.1.1.cmml" xref="S3.SS3.p2.4.m4.1.1"><times id="S3.SS3.p2.4.m4.1.1.1.cmml" xref="S3.SS3.p2.4.m4.1.1.1"></times><cn type="integer" id="S3.SS3.p2.4.m4.1.1.2.cmml" xref="S3.SS3.p2.4.m4.1.1.2">2</cn><apply id="S3.SS3.p2.4.m4.1.1.3.cmml" xref="S3.SS3.p2.4.m4.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.1.1.3.1.cmml" xref="S3.SS3.p2.4.m4.1.1.3">superscript</csymbol><cn type="integer" id="S3.SS3.p2.4.m4.1.1.3.2.cmml" xref="S3.SS3.p2.4.m4.1.1.3.2">10</cn><apply id="S3.SS3.p2.4.m4.1.1.3.3.cmml" xref="S3.SS3.p2.4.m4.1.1.3.3"><minus id="S3.SS3.p2.4.m4.1.1.3.3.1.cmml" xref="S3.SS3.p2.4.m4.1.1.3.3"></minus><cn type="integer" id="S3.SS3.p2.4.m4.1.1.3.3.2.cmml" xref="S3.SS3.p2.4.m4.1.1.3.3.2">4</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.4.m4.1c">2\times 10^{-4}</annotation></semantics></math> within the first <math id="S3.SS3.p2.5.m5.1" class="ltx_Math" alttext="1\%" display="inline"><semantics id="S3.SS3.p2.5.m5.1a"><mrow id="S3.SS3.p2.5.m5.1.1" xref="S3.SS3.p2.5.m5.1.1.cmml"><mn id="S3.SS3.p2.5.m5.1.1.2" xref="S3.SS3.p2.5.m5.1.1.2.cmml">1</mn><mo id="S3.SS3.p2.5.m5.1.1.1" xref="S3.SS3.p2.5.m5.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.5.m5.1b"><apply id="S3.SS3.p2.5.m5.1.1.cmml" xref="S3.SS3.p2.5.m5.1.1"><csymbol cd="latexml" id="S3.SS3.p2.5.m5.1.1.1.cmml" xref="S3.SS3.p2.5.m5.1.1.1">percent</csymbol><cn type="integer" id="S3.SS3.p2.5.m5.1.1.2.cmml" xref="S3.SS3.p2.5.m5.1.1.2">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.5.m5.1c">1\%</annotation></semantics></math> of the training steps. Following this, it decreases to <math id="S3.SS3.p2.6.m6.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S3.SS3.p2.6.m6.1a"><mn id="S3.SS3.p2.6.m6.1.1" xref="S3.SS3.p2.6.m6.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.6.m6.1b"><cn type="integer" id="S3.SS3.p2.6.m6.1.1.cmml" xref="S3.SS3.p2.6.m6.1.1">0</cn></annotation-xml></semantics></math> according to a cosine decay strategy.
To preserve the original knowledge of the vision encoder, we apply a layer-wise learning rate (LLDR) decay strategy <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib33" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">33</span></a>]</cite>, and the decay factor is set to <math id="S3.SS3.p2.7.m7.1" class="ltx_Math" alttext="0.90" display="inline"><semantics id="S3.SS3.p2.7.m7.1a"><mn id="S3.SS3.p2.7.m7.1.1" xref="S3.SS3.p2.7.m7.1.1.cmml">0.90</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.7.m7.1b"><cn type="float" id="S3.SS3.p2.7.m7.1.1.cmml" xref="S3.SS3.p2.7.m7.1.1">0.90</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.7.m7.1c">0.90</annotation></semantics></math>.</p>
</div>
</section>
<section id="S3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.4 </span>Supervised Fine-tuning</h3>

<figure id="S3.T2" class="ltx_table">
<table id="S3.T2.2" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.T2.2.1.1" class="ltx_tr">
<th id="S3.T2.2.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S3.T2.2.1.1.1.1" class="ltx_text" style="font-size:80%;">Task</span></th>
<th id="S3.T2.2.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S3.T2.2.1.1.2.1" class="ltx_text" style="font-size:80%;">Dataset</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.T2.2.2.1" class="ltx_tr">
<th id="S3.T2.2.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S3.T2.2.2.1.1.1" class="ltx_text" style="font-size:80%;">Caption</span></th>
<td id="S3.T2.2.2.1.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S3.T2.2.2.1.2.1" class="ltx_text" style="font-size:80%;">ShareGPT4V </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.2.1.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib17" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">17</span></a><span id="S3.T2.2.2.1.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.2.1.2.4" class="ltx_text" style="font-size:80%;">, COCO </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.2.1.2.5.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib21" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">21</span></a><span id="S3.T2.2.2.1.2.6.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.2.1.2.7" class="ltx_text" style="font-size:80%;">, Nocaps </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.2.1.2.8.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib1" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">1</span></a><span id="S3.T2.2.2.1.2.9.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
</tr>
<tr id="S3.T2.2.3.2" class="ltx_tr">
<th id="S3.T2.2.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S3.T2.2.3.2.1.1" class="ltx_text" style="font-size:80%;">General QA</span></th>
<td id="S3.T2.2.3.2.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S3.T2.2.3.2.2.1" class="ltx_text" style="font-size:80%;">VQAv2 </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.3.2.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib4" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">4</span></a><span id="S3.T2.2.3.2.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.3.2.2.4" class="ltx_text" style="font-size:80%;">, GQA </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.3.2.2.5.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib53" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">53</span></a><span id="S3.T2.2.3.2.2.6.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.3.2.2.7" class="ltx_text" style="font-size:80%;">, OK-VQA </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.3.2.2.8.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib105" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">105</span></a><span id="S3.T2.2.3.2.2.9.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
</tr>
<tr id="S3.T2.2.4.3" class="ltx_tr">
<th id="S3.T2.2.4.3.1" class="ltx_td ltx_th ltx_th_row" style="padding-left:2.8pt;padding-right:2.8pt;"></th>
<td id="S3.T2.2.4.3.2" class="ltx_td ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S3.T2.2.4.3.2.1" class="ltx_text" style="font-size:80%;">VD </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.4.3.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib32" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">32</span></a><span id="S3.T2.2.4.3.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.4.3.2.4" class="ltx_text" style="font-size:80%;">, RD </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.4.3.2.5.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib16" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">16</span></a><span id="S3.T2.2.4.3.2.6.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.4.3.2.7" class="ltx_text" style="font-size:80%;">, VSR </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.4.3.2.8.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib81" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">81</span></a><span id="S3.T2.2.4.3.2.9.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.4.3.2.10" class="ltx_text" style="font-size:80%;">, ALLaVA-QA </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.4.3.2.11.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib15" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">15</span></a><span id="S3.T2.2.4.3.2.12.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
</tr>
<tr id="S3.T2.2.5.4" class="ltx_tr">
<th id="S3.T2.2.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S3.T2.2.5.4.1.1" class="ltx_text" style="font-size:80%;">Multi-Turn QA</span></th>
<td id="S3.T2.2.5.4.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S3.T2.2.5.4.2.1" class="ltx_text" style="font-size:80%;">MMDU </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.5.4.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib92" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">92</span></a><span id="S3.T2.2.5.4.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
</tr>
<tr id="S3.T2.2.6.5" class="ltx_tr">
<th id="S3.T2.2.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S3.T2.2.6.5.1.1" class="ltx_text" style="font-size:80%;">Science QA</span></th>
<td id="S3.T2.2.6.5.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S3.T2.2.6.5.2.1" class="ltx_text" style="font-size:80%;">AI2D </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.6.5.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib61" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">61</span></a><span id="S3.T2.2.6.5.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.6.5.2.4" class="ltx_text" style="font-size:80%;">, SQA </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.6.5.2.5.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib98" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">98</span></a><span id="S3.T2.2.6.5.2.6.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.6.5.2.7" class="ltx_text" style="font-size:80%;">, TQA </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.6.5.2.8.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib62" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">62</span></a><span id="S3.T2.2.6.5.2.9.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.6.5.2.10" class="ltx_text" style="font-size:80%;">, IconQA </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.6.5.2.11.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib97" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">97</span></a><span id="S3.T2.2.6.5.2.12.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
</tr>
<tr id="S3.T2.2.7.6" class="ltx_tr">
<th id="S3.T2.2.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S3.T2.2.7.6.1.1" class="ltx_text" style="font-size:80%;">Chart QA</span></th>
<td id="S3.T2.2.7.6.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S3.T2.2.7.6.2.1" class="ltx_text" style="font-size:80%;">DVQA </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.7.6.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib58" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">58</span></a><span id="S3.T2.2.7.6.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.7.6.2.4" class="ltx_text" style="font-size:80%;">, ChartQA </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.7.6.2.5.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib106" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">106</span></a><span id="S3.T2.2.7.6.2.6.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.7.6.2.7" class="ltx_text" style="font-size:80%;">, ChartQA-AUG </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.7.6.2.8.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib106" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">106</span></a><span id="S3.T2.2.7.6.2.9.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
</tr>
<tr id="S3.T2.2.8.7" class="ltx_tr">
<th id="S3.T2.2.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S3.T2.2.8.7.1.1" class="ltx_text" style="font-size:80%;">Math QA</span></th>
<td id="S3.T2.2.8.7.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S3.T2.2.8.7.2.1" class="ltx_text" style="font-size:80%;">MathQA </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.8.7.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib161" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">161</span></a><span id="S3.T2.2.8.7.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.8.7.2.4" class="ltx_text" style="font-size:80%;">, Geometry3K </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.8.7.2.5.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib96" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">96</span></a><span id="S3.T2.2.8.7.2.6.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.8.7.2.7" class="ltx_text" style="font-size:80%;">, TabMWP </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.8.7.2.8.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib99" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">99</span></a><span id="S3.T2.2.8.7.2.9.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.8.7.2.10" class="ltx_text" style="font-size:80%;">,</span>
</td>
</tr>
<tr id="S3.T2.2.9.8" class="ltx_tr">
<th id="S3.T2.2.9.8.1" class="ltx_td ltx_th ltx_th_row" style="padding-left:2.8pt;padding-right:2.8pt;"></th>
<td id="S3.T2.2.9.8.2" class="ltx_td ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S3.T2.2.9.8.2.1" class="ltx_text" style="font-size:80%;">CLEVR-MATH </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.9.8.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib80" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">80</span></a><span id="S3.T2.2.9.8.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.9.8.2.4" class="ltx_text" style="font-size:80%;">, Super </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.9.8.2.5.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib75" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">75</span></a><span id="S3.T2.2.9.8.2.6.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
</tr>
<tr id="S3.T2.2.10.9" class="ltx_tr">
<th id="S3.T2.2.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S3.T2.2.10.9.1.1" class="ltx_text" style="font-size:80%;">World Knowledge QA</span></th>
<td id="S3.T2.2.10.9.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S3.T2.2.10.9.2.1" class="ltx_text" style="font-size:80%;">A-OKVQA </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.10.9.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib127" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">127</span></a><span id="S3.T2.2.10.9.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.10.9.2.4" class="ltx_text" style="font-size:80%;">, KVQA </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.10.9.2.5.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib128" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">128</span></a><span id="S3.T2.2.10.9.2.6.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.10.9.2.7" class="ltx_text" style="font-size:80%;">, ViQuAE </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.10.9.2.8.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib65" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">65</span></a><span id="S3.T2.2.10.9.2.9.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
</tr>
<tr id="S3.T2.2.11.10" class="ltx_tr">
<th id="S3.T2.2.11.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S3.T2.2.11.10.1.1" class="ltx_text" style="font-size:80%;">OCR QA</span></th>
<td id="S3.T2.2.11.10.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S3.T2.2.11.10.2.1" class="ltx_text" style="font-size:80%;">TextVQA </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.11.10.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib133" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">133</span></a><span id="S3.T2.2.11.10.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.11.10.2.4" class="ltx_text" style="font-size:80%;">, OCR-VQA </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.11.10.2.5.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib109" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">109</span></a><span id="S3.T2.2.11.10.2.6.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.11.10.2.7" class="ltx_text" style="font-size:80%;">, ST-VQA </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.11.10.2.8.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib11" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">11</span></a><span id="S3.T2.2.11.10.2.9.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
</tr>
<tr id="S3.T2.2.12.11" class="ltx_tr">
<th id="S3.T2.2.12.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S3.T2.2.12.11.1.1" class="ltx_text" style="font-size:80%;">HD-OCR QA</span></th>
<td id="S3.T2.2.12.11.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S3.T2.2.12.11.2.1" class="ltx_text" style="font-size:80%;">InfoVQA</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.12.11.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib108" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">108</span></a><span id="S3.T2.2.12.11.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.12.11.2.4" class="ltx_text" style="font-size:80%;">, DocVQA </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.12.11.2.5.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib107" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">107</span></a><span id="S3.T2.2.12.11.2.6.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.12.11.2.7" class="ltx_text" style="font-size:80%;">, TabFact </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.12.11.2.8.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib20" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">20</span></a><span id="S3.T2.2.12.11.2.9.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.12.11.2.10" class="ltx_text" style="font-size:80%;">,</span>
</td>
</tr>
<tr id="S3.T2.2.13.12" class="ltx_tr">
<th id="S3.T2.2.13.12.1" class="ltx_td ltx_th ltx_th_row" style="padding-left:2.8pt;padding-right:2.8pt;"></th>
<td id="S3.T2.2.13.12.2" class="ltx_td ltx_align_left" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S3.T2.2.13.12.2.1" class="ltx_text" style="font-size:80%;">WTQ </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.13.12.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib117" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">117</span></a><span id="S3.T2.2.13.12.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.13.12.2.4" class="ltx_text" style="font-size:80%;">, DeepForm </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.13.12.2.5.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib139" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">139</span></a><span id="S3.T2.2.13.12.2.6.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.13.12.2.7" class="ltx_text" style="font-size:80%;">, Visual MRC </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.13.12.2.8.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib140" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">140</span></a><span id="S3.T2.2.13.12.2.9.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
</tr>
<tr id="S3.T2.2.14.13" class="ltx_tr">
<th id="S3.T2.2.14.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S3.T2.2.14.13.1.1" class="ltx_text" style="font-size:80%;">Video</span></th>
<td id="S3.T2.2.14.13.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S3.T2.2.14.13.2.1" class="ltx_text" style="font-size:80%;">ShareGPT4Video </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.14.13.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib19" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">19</span></a><span id="S3.T2.2.14.13.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.14.13.2.4" class="ltx_text" style="font-size:80%;">, ActivityNet </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.14.13.2.5.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib37" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">37</span></a><span id="S3.T2.2.14.13.2.6.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
</tr>
<tr id="S3.T2.2.15.14" class="ltx_tr">
<th id="S3.T2.2.15.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;"><span id="S3.T2.2.15.14.1.1" class="ltx_text" style="font-size:80%;">Conversation</span></th>
<td id="S3.T2.2.15.14.2" class="ltx_td ltx_align_left ltx_border_t" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S3.T2.2.15.14.2.1" class="ltx_text" style="font-size:80%;">LLaVA-150k </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.15.14.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib84" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">84</span></a><span id="S3.T2.2.15.14.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.15.14.2.4" class="ltx_text" style="font-size:80%;">, LVIS-Instruct4V </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.15.14.2.5.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib149" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">149</span></a><span id="S3.T2.2.15.14.2.6.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
</tr>
<tr id="S3.T2.2.16.15" class="ltx_tr">
<th id="S3.T2.2.16.15.1" class="ltx_td ltx_th ltx_th_row ltx_border_bb" style="padding-left:2.8pt;padding-right:2.8pt;"></th>
<td id="S3.T2.2.16.15.2" class="ltx_td ltx_align_left ltx_border_bb" style="padding-left:2.8pt;padding-right:2.8pt;">
<span id="S3.T2.2.16.15.2.1" class="ltx_text" style="font-size:80%;">ShareGPT-en&amp;zh </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.16.15.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib27" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">27</span></a><span id="S3.T2.2.16.15.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite><span id="S3.T2.2.16.15.2.4" class="ltx_text" style="font-size:80%;">, InternLM-Chat </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T2.2.16.15.2.5.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib143" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">143</span></a><span id="S3.T2.2.16.15.2.6.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_table"><span id="S3.T2.6.1.1" class="ltx_text" style="font-size:113%;">Table 2</span>: </span><span id="S3.T2.7.2" class="ltx_text ltx_font_bold" style="font-size:113%;">Datasets used for Supervised Fine-Tuning<span id="S3.T2.7.2.1" class="ltx_text ltx_font_medium">. We collect data from diverse sources to empower the model with different capabilities. </span></span></figcaption>
</figure>
<div id="S3.SS4.p1" class="ltx_para">
<p id="S3.SS4.p1.5" class="ltx_p">We fine-tune the model with data listed in Table <a href="#S3.T2" title="Table 2 ‣ 3.4 Supervised Fine-tuning ‣ 3 Method ‣ InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>. The maximum number <math id="S3.SS4.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{H}" display="inline"><semantics id="S3.SS4.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS4.p1.1.m1.1.1" xref="S3.SS4.p1.1.m1.1.1.cmml">ℋ</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.1.m1.1b"><ci id="S3.SS4.p1.1.m1.1.1.cmml" xref="S3.SS4.p1.1.m1.1.1">ℋ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.1.m1.1c">\mathcal{H}</annotation></semantics></math> of the Unified Dynamic Image Partition strategy is <math id="S3.SS4.p1.2.m2.1" class="ltx_Math" alttext="24" display="inline"><semantics id="S3.SS4.p1.2.m2.1a"><mn id="S3.SS4.p1.2.m2.1.1" xref="S3.SS4.p1.2.m2.1.1.cmml">24</mn><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.2.m2.1b"><cn type="integer" id="S3.SS4.p1.2.m2.1.1.cmml" xref="S3.SS4.p1.2.m2.1.1">24</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.2.m2.1c">24</annotation></semantics></math> to handle extremely large images and videos. For video datasets, the IXC-2.5 is trained with large images concatenated by at most 64 frames.
The largest training context is set to a 24,000 context window size, where the MMDU <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib92" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">92</span></a>]</cite> dataset can achieve this limitation.
In practice, we jointly train all the components with a batch size of 2048 over 4000 steps. Data from multiple sources are sampled in a weighted manner, with the weights based on the number of data from each source.
The maximum learning rate is set to <math id="S3.SS4.p1.3.m3.1" class="ltx_Math" alttext="5\times 10^{-5}" display="inline"><semantics id="S3.SS4.p1.3.m3.1a"><mrow id="S3.SS4.p1.3.m3.1.1" xref="S3.SS4.p1.3.m3.1.1.cmml"><mn id="S3.SS4.p1.3.m3.1.1.2" xref="S3.SS4.p1.3.m3.1.1.2.cmml">5</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS4.p1.3.m3.1.1.1" xref="S3.SS4.p1.3.m3.1.1.1.cmml">×</mo><msup id="S3.SS4.p1.3.m3.1.1.3" xref="S3.SS4.p1.3.m3.1.1.3.cmml"><mn id="S3.SS4.p1.3.m3.1.1.3.2" xref="S3.SS4.p1.3.m3.1.1.3.2.cmml">10</mn><mrow id="S3.SS4.p1.3.m3.1.1.3.3" xref="S3.SS4.p1.3.m3.1.1.3.3.cmml"><mo id="S3.SS4.p1.3.m3.1.1.3.3a" xref="S3.SS4.p1.3.m3.1.1.3.3.cmml">−</mo><mn id="S3.SS4.p1.3.m3.1.1.3.3.2" xref="S3.SS4.p1.3.m3.1.1.3.3.2.cmml">5</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.3.m3.1b"><apply id="S3.SS4.p1.3.m3.1.1.cmml" xref="S3.SS4.p1.3.m3.1.1"><times id="S3.SS4.p1.3.m3.1.1.1.cmml" xref="S3.SS4.p1.3.m3.1.1.1"></times><cn type="integer" id="S3.SS4.p1.3.m3.1.1.2.cmml" xref="S3.SS4.p1.3.m3.1.1.2">5</cn><apply id="S3.SS4.p1.3.m3.1.1.3.cmml" xref="S3.SS4.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="S3.SS4.p1.3.m3.1.1.3.1.cmml" xref="S3.SS4.p1.3.m3.1.1.3">superscript</csymbol><cn type="integer" id="S3.SS4.p1.3.m3.1.1.3.2.cmml" xref="S3.SS4.p1.3.m3.1.1.3.2">10</cn><apply id="S3.SS4.p1.3.m3.1.1.3.3.cmml" xref="S3.SS4.p1.3.m3.1.1.3.3"><minus id="S3.SS4.p1.3.m3.1.1.3.3.1.cmml" xref="S3.SS4.p1.3.m3.1.1.3.3"></minus><cn type="integer" id="S3.SS4.p1.3.m3.1.1.3.3.2.cmml" xref="S3.SS4.p1.3.m3.1.1.3.3.2">5</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.3.m3.1c">5\times 10^{-5}</annotation></semantics></math>, and each component has its own unique learning strategy.
For the vision encoder, we set the LLDR to <math id="S3.SS4.p1.4.m4.1" class="ltx_Math" alttext="0.9" display="inline"><semantics id="S3.SS4.p1.4.m4.1a"><mn id="S3.SS4.p1.4.m4.1.1" xref="S3.SS4.p1.4.m4.1.1.cmml">0.9</mn><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.4.m4.1b"><cn type="float" id="S3.SS4.p1.4.m4.1.1.cmml" xref="S3.SS4.p1.4.m4.1.1">0.9</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.4.m4.1c">0.9</annotation></semantics></math>, which aligns with the pretraining strategy.
For the LLM, we employ a fixed learning rate scale factor of <math id="S3.SS4.p1.5.m5.1" class="ltx_Math" alttext="0.2" display="inline"><semantics id="S3.SS4.p1.5.m5.1a"><mn id="S3.SS4.p1.5.m5.1.1" xref="S3.SS4.p1.5.m5.1.1.cmml">0.2</mn><annotation-xml encoding="MathML-Content" id="S3.SS4.p1.5.m5.1b"><cn type="float" id="S3.SS4.p1.5.m5.1.1.cmml" xref="S3.SS4.p1.5.m5.1.1">0.2</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p1.5.m5.1c">0.2</annotation></semantics></math>. This slows down the update of the LLM, achieving a balance between preserving its original capabilities and aligning it with vision knowledge.</p>
</div>
</section>
<section id="S3.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.5 </span>Webpage Generation</h3>

<div id="S3.SS5.p1" class="ltx_para">
<p id="S3.SS5.p1.1" class="ltx_p">We enhance the capabilities of the IXC-2.5 to include automated webpage generation. Specifically, the IXC-2.5 is now equipped to autonomously construct web pages, utilizing HTML, CSS, and JavaScript, based on input in the form of a visual screenshot, a set of free-form instructions, or a resume document. Current open-source general-purpose large language models frequently demonstrate suboptimal performance in generating HTML and CSS relative to their proficiency in natural language generation. To address this limitation, we propose training the screenshot-to-code task using extensive datasets from WebSight v0.1/v0.2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib64" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">64</span></a>]</cite>, and Stack v2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib95" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">95</span></a>]</cite>. Subsequently, we fine-tune the model with a smaller, meticulously crafted dataset consisting of instruction-aware webpage generation and personal page generation examples.</p>
</div>
<div id="S3.SS5.p2" class="ltx_para ltx_noindent">
<p id="S3.SS5.p2.1" class="ltx_p"><span id="S3.SS5.p2.1.1" class="ltx_text ltx_font_bold">Screenshot-to-code.</span> In addition to the WebSight <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib64" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">64</span></a>]</cite> datasets, we preprocess the HTML and CSS code from the Stack v2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib95" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">95</span></a>]</cite> dataset to facilitate screenshot-to-code training. Initially, we combine the CSS and HTML code into a single file. Subsequently, we remove all comments, JavaScript code, and external links. Furthermore, we eliminate any CSS styles that are not referenced by the HTML code. We convert all files into screenshots, subsequently discarding those that did not render successfully. The remaining screenshots are then processed using the IXC2-4KHD <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib34" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">34</span></a>]</cite> model to assess the quality of the web pages. Following the exclusion of low-quality web pages, we retained a final set of three remaining about 250,000 high-quality web pages.</p>
</div>
<div id="S3.SS5.p3" class="ltx_para">
<p id="S3.SS5.p3.3" class="ltx_p">We conduct training on the LoRA model utilizing the three aforementioned datasets. The LoRA rank is set to 512. The training protocol employs a batch size of 512 and is executed over a single epoch. Initially, the learning rate is incremented linearly to <math id="S3.SS5.p3.1.m1.1" class="ltx_Math" alttext="1\times 10^{-4}" display="inline"><semantics id="S3.SS5.p3.1.m1.1a"><mrow id="S3.SS5.p3.1.m1.1.1" xref="S3.SS5.p3.1.m1.1.1.cmml"><mn id="S3.SS5.p3.1.m1.1.1.2" xref="S3.SS5.p3.1.m1.1.1.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS5.p3.1.m1.1.1.1" xref="S3.SS5.p3.1.m1.1.1.1.cmml">×</mo><msup id="S3.SS5.p3.1.m1.1.1.3" xref="S3.SS5.p3.1.m1.1.1.3.cmml"><mn id="S3.SS5.p3.1.m1.1.1.3.2" xref="S3.SS5.p3.1.m1.1.1.3.2.cmml">10</mn><mrow id="S3.SS5.p3.1.m1.1.1.3.3" xref="S3.SS5.p3.1.m1.1.1.3.3.cmml"><mo id="S3.SS5.p3.1.m1.1.1.3.3a" xref="S3.SS5.p3.1.m1.1.1.3.3.cmml">−</mo><mn id="S3.SS5.p3.1.m1.1.1.3.3.2" xref="S3.SS5.p3.1.m1.1.1.3.3.2.cmml">4</mn></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS5.p3.1.m1.1b"><apply id="S3.SS5.p3.1.m1.1.1.cmml" xref="S3.SS5.p3.1.m1.1.1"><times id="S3.SS5.p3.1.m1.1.1.1.cmml" xref="S3.SS5.p3.1.m1.1.1.1"></times><cn type="integer" id="S3.SS5.p3.1.m1.1.1.2.cmml" xref="S3.SS5.p3.1.m1.1.1.2">1</cn><apply id="S3.SS5.p3.1.m1.1.1.3.cmml" xref="S3.SS5.p3.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.SS5.p3.1.m1.1.1.3.1.cmml" xref="S3.SS5.p3.1.m1.1.1.3">superscript</csymbol><cn type="integer" id="S3.SS5.p3.1.m1.1.1.3.2.cmml" xref="S3.SS5.p3.1.m1.1.1.3.2">10</cn><apply id="S3.SS5.p3.1.m1.1.1.3.3.cmml" xref="S3.SS5.p3.1.m1.1.1.3.3"><minus id="S3.SS5.p3.1.m1.1.1.3.3.1.cmml" xref="S3.SS5.p3.1.m1.1.1.3.3"></minus><cn type="integer" id="S3.SS5.p3.1.m1.1.1.3.3.2.cmml" xref="S3.SS5.p3.1.m1.1.1.3.3.2">4</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p3.1.m1.1c">1\times 10^{-4}</annotation></semantics></math> within the first <math id="S3.SS5.p3.2.m2.1" class="ltx_Math" alttext="1\%" display="inline"><semantics id="S3.SS5.p3.2.m2.1a"><mrow id="S3.SS5.p3.2.m2.1.1" xref="S3.SS5.p3.2.m2.1.1.cmml"><mn id="S3.SS5.p3.2.m2.1.1.2" xref="S3.SS5.p3.2.m2.1.1.2.cmml">1</mn><mo id="S3.SS5.p3.2.m2.1.1.1" xref="S3.SS5.p3.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS5.p3.2.m2.1b"><apply id="S3.SS5.p3.2.m2.1.1.cmml" xref="S3.SS5.p3.2.m2.1.1"><csymbol cd="latexml" id="S3.SS5.p3.2.m2.1.1.1.cmml" xref="S3.SS5.p3.2.m2.1.1.1">percent</csymbol><cn type="integer" id="S3.SS5.p3.2.m2.1.1.2.cmml" xref="S3.SS5.p3.2.m2.1.1.2">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS5.p3.2.m2.1c">1\%</annotation></semantics></math> of the training iterations. Subsequently, the learning rate decreases to <math id="S3.SS5.p3.3.m3.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S3.SS5.p3.3.m3.1a"><mn id="S3.SS5.p3.3.m3.1.1" xref="S3.SS5.p3.3.m3.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S3.SS5.p3.3.m3.1b"><cn type="integer" id="S3.SS5.p3.3.m3.1.1.cmml" xref="S3.SS5.p3.3.m3.1.1">0</cn></annotation-xml></semantics></math> following a cosine decay schedule.</p>
</div>
<div id="S3.SS5.p4" class="ltx_para ltx_noindent">
<p id="S3.SS5.p4.1" class="ltx_p"><span id="S3.SS5.p4.1.1" class="ltx_text ltx_font_bold">Instruction-aware Webpage Generation.</span> A pivotal attribute of large language models lies in their capability to adhere to human instructions. To facilitate web page generation based on freeform instructions, we propose constructing data through querying closed-source large language models. Specifically, we utilize GPT-4 to generate diverse instructions and concepts for web page creation, encompassing elements such as type, style, and layout. Subsequently, these instructions are harnessed to query Claude-3-sonnet <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">3</span></a>]</cite> for the actual web page generation process. This approach results in 18,000 high-quality, instruction-aware samples. Additionally, we employ Tailwind CSS instead of traditional CSS, given its succinct nature.</p>
</div>
<div id="S3.SS5.p5" class="ltx_para ltx_noindent">
<p id="S3.SS5.p5.1" class="ltx_p"><span id="S3.SS5.p5.1.1" class="ltx_text ltx_font_bold">Resume-to-homepage.</span> In addition to instruction-aware webpage generation, we introduce a more practical task. Specifically, given a resume, the model is designed to generate a personal homepage. This homepage not only encapsulates the information present in the resume but also presents it with a well-structured and visually appealing format, improving both content organization and aesthetic layout. To generate corresponding datasets, we propose an idea-resume-homepage data generation pipeline. Initially, we leverage GPT-4 to produce resume ideas tailored for diverse personas, such as researchers, students, and engineers. GPT-4 is tasked with generating these resumes in markdown format based on the provided ideas. Upon obtaining the generated resumes, we then prompt Claude-3-sonnet <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib3" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">3</span></a>]</cite> to create corresponding homepages from these resumes. To enhance the interactivity of these webpages, Claude-3-sonnet is also utilized to generate JavaScript events based on the HTML code. In total, we have constructed a dataset comprising 2,000 samples.</p>
</div>
<div id="S3.SS5.p6" class="ltx_para">
<p id="S3.SS5.p6.1" class="ltx_p">Upon constructing the dataset for instruction-aware webpage generation and resume-to-homepage, we subsequently fine-tuned the LoRA model for 10 epochs. All other experimental settings were maintained consistent with those employed during the screenshot-to-code training phase.</p>
</div>
</section>
<section id="S3.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.6 </span>Article Composing</h3>

<div id="S3.SS6.p1" class="ltx_para">
<p id="S3.SS6.p1.1" class="ltx_p">Generating high-quality text-image articles (<em id="S3.SS6.p1.1.1" class="ltx_emph ltx_font_italic">e.g</em>.<span id="S3.SS6.p1.1.2" class="ltx_text"></span>, poetry, novels, short stories, and essays) is a crucial capability for AI assistants, with various applications in daily life, including education and entertainment.
Building upon the IXC-2.5 SFT model <math id="S3.SS6.p1.1.m1.1" class="ltx_Math" alttext="\pi" display="inline"><semantics id="S3.SS6.p1.1.m1.1a"><mi id="S3.SS6.p1.1.m1.1.1" xref="S3.SS6.p1.1.m1.1.1.cmml">π</mi><annotation-xml encoding="MathML-Content" id="S3.SS6.p1.1.m1.1b"><ci id="S3.SS6.p1.1.m1.1.1.cmml" xref="S3.SS6.p1.1.m1.1.1">𝜋</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p1.1.m1.1c">\pi</annotation></semantics></math> in Section <a href="#S3.SS4" title="3.4 Supervised Fine-tuning ‣ 3 Method ‣ InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.4</span></a>, we enhance creative writing capabilities for generating high-quality text-image articles.
However, collecting high-quality text-image articles is a rare and expensive endeavor.
Direct fine-tuning on scarce instruction data can lead to unstable responses from LVLMs in most cases.
To overcome these challenges, we propose a scalable pipeline that integrates supervised fine-tuning, reward modeling, preference data collection, and DPO alignment for high-quality and stable article generation.</p>
</div>
<div id="S3.SS6.p2" class="ltx_para ltx_noindent">
<p id="S3.SS6.p2.6" class="ltx_p"><span id="S3.SS6.p2.6.1" class="ltx_text ltx_font_bold">Supervised Fine-tuning.</span>
We begin with the SFT model <math id="S3.SS6.p2.1.m1.1" class="ltx_Math" alttext="\pi" display="inline"><semantics id="S3.SS6.p2.1.m1.1a"><mi id="S3.SS6.p2.1.m1.1.1" xref="S3.SS6.p2.1.m1.1.1.cmml">π</mi><annotation-xml encoding="MathML-Content" id="S3.SS6.p2.1.m1.1b"><ci id="S3.SS6.p2.1.m1.1.1.cmml" xref="S3.SS6.p2.1.m1.1.1">𝜋</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p2.1.m1.1c">\pi</annotation></semantics></math> (Section  <a href="#S3.SS4" title="3.4 Supervised Fine-tuning ‣ 3 Method ‣ InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.4</span></a>) and a collection of 5,000 instruction tuning data samples <math id="S3.SS6.p2.2.m2.1" class="ltx_Math" alttext="\mathcal{D}" display="inline"><semantics id="S3.SS6.p2.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS6.p2.2.m2.1.1" xref="S3.SS6.p2.2.m2.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S3.SS6.p2.2.m2.1b"><ci id="S3.SS6.p2.2.m2.1.1.cmml" xref="S3.SS6.p2.2.m2.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p2.2.m2.1c">\mathcal{D}</annotation></semantics></math> from IXC2 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib33" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">33</span></a>]</cite>, focused on article writing.
Due to the limited scale of the instruction data, we use the SFT model to rewrite the original prompts using the Chain-of-Thought (CoT) technique <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib152" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">152</span></a>]</cite>, generating step-by-step prompts to supplement the instruction tuning data as augmented data <math id="S3.SS6.p2.3.m3.1" class="ltx_Math" alttext="\mathcal{D}^{*}" display="inline"><semantics id="S3.SS6.p2.3.m3.1a"><msup id="S3.SS6.p2.3.m3.1.1" xref="S3.SS6.p2.3.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS6.p2.3.m3.1.1.2" xref="S3.SS6.p2.3.m3.1.1.2.cmml">𝒟</mi><mo id="S3.SS6.p2.3.m3.1.1.3" xref="S3.SS6.p2.3.m3.1.1.3.cmml">∗</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SS6.p2.3.m3.1b"><apply id="S3.SS6.p2.3.m3.1.1.cmml" xref="S3.SS6.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS6.p2.3.m3.1.1.1.cmml" xref="S3.SS6.p2.3.m3.1.1">superscript</csymbol><ci id="S3.SS6.p2.3.m3.1.1.2.cmml" xref="S3.SS6.p2.3.m3.1.1.2">𝒟</ci><times id="S3.SS6.p2.3.m3.1.1.3.cmml" xref="S3.SS6.p2.3.m3.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p2.3.m3.1c">\mathcal{D}^{*}</annotation></semantics></math>.
We observe that the SFT model is more effective in generating long-form responses when using these augmented prompts.
We then train the initial model <math id="S3.SS6.p2.4.m4.1" class="ltx_Math" alttext="\pi" display="inline"><semantics id="S3.SS6.p2.4.m4.1a"><mi id="S3.SS6.p2.4.m4.1.1" xref="S3.SS6.p2.4.m4.1.1.cmml">π</mi><annotation-xml encoding="MathML-Content" id="S3.SS6.p2.4.m4.1b"><ci id="S3.SS6.p2.4.m4.1.1.cmml" xref="S3.SS6.p2.4.m4.1.1">𝜋</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p2.4.m4.1c">\pi</annotation></semantics></math> on the augmented instruction tuning data via LoRA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib51" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">51</span></a>]</cite> with the rank of <math id="S3.SS6.p2.5.m5.1" class="ltx_Math" alttext="256" display="inline"><semantics id="S3.SS6.p2.5.m5.1a"><mn id="S3.SS6.p2.5.m5.1.1" xref="S3.SS6.p2.5.m5.1.1.cmml">256</mn><annotation-xml encoding="MathML-Content" id="S3.SS6.p2.5.m5.1b"><cn type="integer" id="S3.SS6.p2.5.m5.1.1.cmml" xref="S3.SS6.p2.5.m5.1.1">256</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p2.5.m5.1c">256</annotation></semantics></math> and get the model <math id="S3.SS6.p2.6.m6.1" class="ltx_Math" alttext="\pi_{\text{ref}}" display="inline"><semantics id="S3.SS6.p2.6.m6.1a"><msub id="S3.SS6.p2.6.m6.1.1" xref="S3.SS6.p2.6.m6.1.1.cmml"><mi id="S3.SS6.p2.6.m6.1.1.2" xref="S3.SS6.p2.6.m6.1.1.2.cmml">π</mi><mtext id="S3.SS6.p2.6.m6.1.1.3" xref="S3.SS6.p2.6.m6.1.1.3a.cmml">ref</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS6.p2.6.m6.1b"><apply id="S3.SS6.p2.6.m6.1.1.cmml" xref="S3.SS6.p2.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS6.p2.6.m6.1.1.1.cmml" xref="S3.SS6.p2.6.m6.1.1">subscript</csymbol><ci id="S3.SS6.p2.6.m6.1.1.2.cmml" xref="S3.SS6.p2.6.m6.1.1.2">𝜋</ci><ci id="S3.SS6.p2.6.m6.1.1.3a.cmml" xref="S3.SS6.p2.6.m6.1.1.3"><mtext mathsize="70%" id="S3.SS6.p2.6.m6.1.1.3.cmml" xref="S3.SS6.p2.6.m6.1.1.3">ref</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p2.6.m6.1c">\pi_{\text{ref}}</annotation></semantics></math> to establish a starting point of our alignment pipeline.</p>
</div>
<figure id="S3.T3" class="ltx_table">
<div id="S3.T3.2" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:32.5pt;vertical-align:-0.2pt;"><span class="ltx_transformed_inner" style="transform:translate(-870.6pt,64.8pt) scale(0.19938819118433,0.19938819118433) ;">
<table id="S3.T3.2.2" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S3.T3.2.2.2" class="ltx_tr">
<th id="S3.T3.2.2.2.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_th ltx_th_row ltx_border_r ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"></th>
<td id="S3.T3.2.2.2.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;" rowspan="2"><span id="S3.T3.2.2.2.4.1" class="ltx_text" style="font-size:80%;">MVBench</span></td>
<td id="S3.T3.2.2.2.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;" rowspan="2"><span id="S3.T3.2.2.2.5.1" class="ltx_text" style="font-size:80%;">MLVU</span></td>
<td id="S3.T3.2.2.2.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.2.6.1" class="ltx_text" style="font-size:80%;">MME</span></td>
<td id="S3.T3.1.1.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T3.1.1.1.1.1" class="ltx_text" style="font-size:80%;">MMB</span><sup id="S3.T3.1.1.1.1.2" class="ltx_sup"><span id="S3.T3.1.1.1.1.2.1" class="ltx_text ltx_font_italic" style="font-size:80%;">∗1</span></sup>
</td>
<td id="S3.T3.2.2.2.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T3.2.2.2.2.1" class="ltx_text" style="font-size:80%;">Temp</span><sup id="S3.T3.2.2.2.2.2" class="ltx_sup"><span id="S3.T3.2.2.2.2.2.1" class="ltx_text ltx_font_italic" style="font-size:80%;">∗2</span></sup>
</td>
<td id="S3.T3.2.2.2.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.2.7.1" class="ltx_text" style="font-size:80%;">Doc</span></td>
<td id="S3.T3.2.2.2.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.2.8.1" class="ltx_text" style="font-size:80%;">Chart</span></td>
<td id="S3.T3.2.2.2.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.2.9.1" class="ltx_text" style="font-size:80%;">Info</span></td>
<td id="S3.T3.2.2.2.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.2.10.1" class="ltx_text" style="font-size:80%;">Text</span></td>
<td id="S3.T3.2.2.2.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.2.11.1" class="ltx_text" style="font-size:80%;">OCR</span></td>
<td id="S3.T3.2.2.2.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;" rowspan="2"><span id="S3.T3.2.2.2.12.1" class="ltx_text" style="font-size:80%;">WTQ</span></td>
<td id="S3.T3.2.2.2.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.2.13.1" class="ltx_text" style="font-size:80%;">Deep</span></td>
<td id="S3.T3.2.2.2.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.2.14.1" class="ltx_text" style="font-size:80%;">Visual</span></td>
<td id="S3.T3.2.2.2.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.2.15.1" class="ltx_text" style="font-size:80%;">Tab</span></td>
</tr>
<tr id="S3.T3.2.2.3.1" class="ltx_tr">
<th id="S3.T3.2.2.3.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_th ltx_th_row ltx_border_r" style="padding-left:0.6pt;padding-right:0.6pt;"></th>
<td id="S3.T3.2.2.3.1.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.3.1.2.1" class="ltx_text" style="font-size:80%;">Video</span></td>
<td id="S3.T3.2.2.3.1.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.3.1.3.1" class="ltx_text" style="font-size:80%;">Video</span></td>
<td id="S3.T3.2.2.3.1.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.3.1.4.1" class="ltx_text" style="font-size:80%;">Compass</span></td>
<td id="S3.T3.2.2.3.1.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.3.1.5.1" class="ltx_text" style="font-size:80%;">VQA</span></td>
<td id="S3.T3.2.2.3.1.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.3.1.6.1" class="ltx_text" style="font-size:80%;">QA</span></td>
<td id="S3.T3.2.2.3.1.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.3.1.7.1" class="ltx_text" style="font-size:80%;">VQA</span></td>
<td id="S3.T3.2.2.3.1.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.3.1.8.1" class="ltx_text" style="font-size:80%;">VQA</span></td>
<td id="S3.T3.2.2.3.1.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.3.1.9.1" class="ltx_text" style="font-size:80%;">Bench</span></td>
<td id="S3.T3.2.2.3.1.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.3.1.10.1" class="ltx_text" style="font-size:80%;">Form</span></td>
<td id="S3.T3.2.2.3.1.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.3.1.11.1" class="ltx_text" style="font-size:80%;">MRC</span></td>
<td id="S3.T3.2.2.3.1.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.3.1.12.1" class="ltx_text" style="font-size:80%;">Fact</span></td>
</tr>
<tr id="S3.T3.2.2.4.2" class="ltx_tr">
<th id="S3.T3.2.2.4.2.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.4.2.1.1" class="ltx_text" style="font-size:80%;">Open-Source</span></th>
<td id="S3.T3.2.2.4.2.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.4.2.2.1" class="ltx_text" style="font-size:80%;">VideoChat</span></td>
<td id="S3.T3.2.2.4.2.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.4.2.3.1" class="ltx_text" style="font-size:80%;">InternVL</span></td>
<td id="S3.T3.2.2.4.2.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.4.2.4.1" class="ltx_text" style="font-size:80%;">LIVA</span></td>
<td id="S3.T3.2.2.4.2.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.4.2.5.1" class="ltx_text" style="font-size:80%;">InternVL</span></td>
<td id="S3.T3.2.2.4.2.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.4.2.6.1" class="ltx_text" style="font-size:80%;">Qwen-VL</span></td>
<td id="S3.T3.2.2.4.2.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.4.2.7.1" class="ltx_text" style="font-size:80%;">InternVL</span></td>
<td id="S3.T3.2.2.4.2.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.4.2.8.1" class="ltx_text" style="font-size:80%;">InternVL</span></td>
<td id="S3.T3.2.2.4.2.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.4.2.9.1" class="ltx_text" style="font-size:80%;">InternVL</span></td>
<td id="S3.T3.2.2.4.2.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.4.2.10.1" class="ltx_text" style="font-size:80%;">InternVL</span></td>
<td id="S3.T3.2.2.4.2.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.4.2.11.1" class="ltx_text" style="font-size:80%;">GLM-4v</span></td>
<td id="S3.T3.2.2.4.2.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.4.2.12.1" class="ltx_text" style="font-size:80%;">DocOwl</span></td>
<td id="S3.T3.2.2.4.2.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.4.2.13.1" class="ltx_text" style="font-size:80%;">DocOwl</span></td>
<td id="S3.T3.2.2.4.2.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.4.2.14.1" class="ltx_text" style="font-size:80%;">DocOwl</span></td>
<td id="S3.T3.2.2.4.2.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.4.2.15.1" class="ltx_text" style="font-size:80%;">DocOwl</span></td>
</tr>
<tr id="S3.T3.2.2.5.3" class="ltx_tr">
<th id="S3.T3.2.2.5.3.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_r" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.5.3.1.1" class="ltx_text" style="font-size:80%;">Previous SOTA</span></th>
<td id="S3.T3.2.2.5.3.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T3.2.2.5.3.2.1" class="ltx_text" style="font-size:80%;">2-7B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T3.2.2.5.3.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib71" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">71</span></a><span id="S3.T3.2.2.5.3.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T3.2.2.5.3.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T3.2.2.5.3.3.1" class="ltx_text" style="font-size:80%;">1.5-26B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T3.2.2.5.3.3.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">26</span></a><span id="S3.T3.2.2.5.3.3.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T3.2.2.5.3.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T3.2.2.5.3.4.1" class="ltx_text" style="font-size:80%;">34B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T3.2.2.5.3.4.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib78" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">78</span></a><span id="S3.T3.2.2.5.3.4.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T3.2.2.5.3.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T3.2.2.5.3.5.1" class="ltx_text" style="font-size:80%;">1.5-26B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T3.2.2.5.3.5.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">26</span></a><span id="S3.T3.2.2.5.3.5.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T3.2.2.5.3.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T3.2.2.5.3.6.1" class="ltx_text" style="font-size:80%;">7B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T3.2.2.5.3.6.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib6" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">6</span></a><span id="S3.T3.2.2.5.3.6.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T3.2.2.5.3.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T3.2.2.5.3.7.1" class="ltx_text" style="font-size:80%;">1.5-26B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T3.2.2.5.3.7.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">26</span></a><span id="S3.T3.2.2.5.3.7.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T3.2.2.5.3.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T3.2.2.5.3.8.1" class="ltx_text" style="font-size:80%;">1.5-26B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T3.2.2.5.3.8.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">26</span></a><span id="S3.T3.2.2.5.3.8.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T3.2.2.5.3.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T3.2.2.5.3.9.1" class="ltx_text" style="font-size:80%;">1.5-26B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T3.2.2.5.3.9.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">26</span></a><span id="S3.T3.2.2.5.3.9.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T3.2.2.5.3.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T3.2.2.5.3.10.1" class="ltx_text" style="font-size:80%;">1.5-26B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T3.2.2.5.3.10.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">26</span></a><span id="S3.T3.2.2.5.3.10.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T3.2.2.5.3.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T3.2.2.5.3.11.1" class="ltx_text" style="font-size:80%;">9B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T3.2.2.5.3.11.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib43" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">43</span></a><span id="S3.T3.2.2.5.3.11.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T3.2.2.5.3.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T3.2.2.5.3.12.1" class="ltx_text" style="font-size:80%;">1.5-8B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T3.2.2.5.3.12.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib50" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">50</span></a><span id="S3.T3.2.2.5.3.12.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T3.2.2.5.3.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T3.2.2.5.3.13.1" class="ltx_text" style="font-size:80%;">1.5-8B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T3.2.2.5.3.13.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib50" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">50</span></a><span id="S3.T3.2.2.5.3.13.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T3.2.2.5.3.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T3.2.2.5.3.14.1" class="ltx_text" style="font-size:80%;">1.5-8B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T3.2.2.5.3.14.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib50" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">50</span></a><span id="S3.T3.2.2.5.3.14.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T3.2.2.5.3.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T3.2.2.5.3.15.1" class="ltx_text" style="font-size:80%;">1.5-8B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T3.2.2.5.3.15.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib50" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">50</span></a><span id="S3.T3.2.2.5.3.15.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
</tr>
<tr id="S3.T3.2.2.6.4" class="ltx_tr">
<th id="S3.T3.2.2.6.4.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_r" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.6.4.1.1" class="ltx_text" style="font-size:80%;">Performance</span></th>
<td id="S3.T3.2.2.6.4.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.6.4.2.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">60.4</span></td>
<td id="S3.T3.2.2.6.4.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.6.4.3.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">50.4</span></td>
<td id="S3.T3.2.2.6.4.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.6.4.4.1" class="ltx_text" style="font-size:80%;">59.0</span></td>
<td id="S3.T3.2.2.6.4.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.6.4.5.1" class="ltx_text" style="font-size:80%;">42.0</span></td>
<td id="S3.T3.2.2.6.4.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.6.4.6.1" class="ltx_text" style="font-size:80%;">58.4</span></td>
<td id="S3.T3.2.2.6.4.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.6.4.7.1" class="ltx_text ltx_font_bold" style="font-size:80%;">90.9</span></td>
<td id="S3.T3.2.2.6.4.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.6.4.8.1" class="ltx_text ltx_font_bold" style="font-size:80%;">83.8</span></td>
<td id="S3.T3.2.2.6.4.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.6.4.9.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">72.5</span></td>
<td id="S3.T3.2.2.6.4.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.6.4.10.1" class="ltx_text ltx_font_bold" style="font-size:80%;">80.6</span></td>
<td id="S3.T3.2.2.6.4.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.6.4.11.1" class="ltx_text ltx_font_bold" style="font-size:80%;">77.6</span></td>
<td id="S3.T3.2.2.6.4.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.6.4.12.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">40.6</span></td>
<td id="S3.T3.2.2.6.4.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.6.4.13.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">68.8</span></td>
<td id="S3.T3.2.2.6.4.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.6.4.14.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">246.4</span></td>
<td id="S3.T3.2.2.6.4.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.6.4.15.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">80.2</span></td>
</tr>
<tr id="S3.T3.2.2.7.5" class="ltx_tr">
<th id="S3.T3.2.2.7.5.1" class="ltx_td ltx_nopad_l ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;" colspan="11"><span id="S3.T3.2.2.7.5.1.1" class="ltx_text ltx_font_italic" style="font-size:80%;">Closed-source API</span></th>
<td id="S3.T3.2.2.7.5.2" class="ltx_td ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"></td>
<td id="S3.T3.2.2.7.5.3" class="ltx_td ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"></td>
<td id="S3.T3.2.2.7.5.4" class="ltx_td ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"></td>
<td id="S3.T3.2.2.7.5.5" class="ltx_td ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"></td>
</tr>
<tr id="S3.T3.2.2.8.6" class="ltx_tr">
<th id="S3.T3.2.2.8.6.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_r" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T3.2.2.8.6.1.1" class="ltx_text" style="font-size:80%;">GPT-4V </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T3.2.2.8.6.1.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib112" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">112</span></a><span id="S3.T3.2.2.8.6.1.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</th>
<td id="S3.T3.2.2.8.6.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.8.6.2.1" class="ltx_text" style="font-size:80%;">43.5</span></td>
<td id="S3.T3.2.2.8.6.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.8.6.3.1" class="ltx_text" style="font-size:80%;">49.2</span></td>
<td id="S3.T3.2.2.8.6.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.8.6.4.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">59.9</span></td>
<td id="S3.T3.2.2.8.6.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.8.6.5.1" class="ltx_text ltx_font_bold" style="font-size:80%;">56.0</span></td>
<td id="S3.T3.2.2.8.6.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.8.6.6.1" class="ltx_text" style="font-size:80%;">—</span></td>
<td id="S3.T3.2.2.8.6.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.8.6.7.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">88.4</span></td>
<td id="S3.T3.2.2.8.6.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.8.6.8.1" class="ltx_text" style="font-size:80%;">78.5</span></td>
<td id="S3.T3.2.2.8.6.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.8.6.9.1" class="ltx_text" style="font-size:80%;">75.1</span></td>
<td id="S3.T3.2.2.8.6.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.8.6.10.1" class="ltx_text" style="font-size:80%;">78.0</span></td>
<td id="S3.T3.2.2.8.6.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.8.6.11.1" class="ltx_text" style="font-size:80%;">51.6</span></td>
<td id="S3.T3.2.2.8.6.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.8.6.12.1" class="ltx_text" style="font-size:80%;">—</span></td>
<td id="S3.T3.2.2.8.6.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.8.6.13.1" class="ltx_text" style="font-size:80%;">—</span></td>
<td id="S3.T3.2.2.8.6.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.8.6.14.1" class="ltx_text" style="font-size:80%;">—</span></td>
<td id="S3.T3.2.2.8.6.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.8.6.15.1" class="ltx_text" style="font-size:80%;">—</span></td>
</tr>
<tr id="S3.T3.2.2.9.7" class="ltx_tr">
<th id="S3.T3.2.2.9.7.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_r" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T3.2.2.9.7.1.1" class="ltx_text" style="font-size:80%;">Gemini-Pro </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T3.2.2.9.7.1.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib142" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">142</span></a><span id="S3.T3.2.2.9.7.1.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</th>
<td id="S3.T3.2.2.9.7.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.9.7.2.1" class="ltx_text" style="font-size:80%;">—</span></td>
<td id="S3.T3.2.2.9.7.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.9.7.3.1" class="ltx_text" style="font-size:80%;">—</span></td>
<td id="S3.T3.2.2.9.7.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.9.7.4.1" class="ltx_text ltx_font_bold" style="font-size:80%;">75.0</span></td>
<td id="S3.T3.2.2.9.7.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.9.7.5.1" class="ltx_text" style="font-size:80%;">49.3</span></td>
<td id="S3.T3.2.2.9.7.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_r" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.9.7.6.1" class="ltx_text ltx_font_bold" style="font-size:80%;">70.6</span></td>
<td id="S3.T3.2.2.9.7.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.9.7.7.1" class="ltx_text" style="font-size:80%;">88.1</span></td>
<td id="S3.T3.2.2.9.7.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.9.7.8.1" class="ltx_text" style="font-size:80%;">74.1</span></td>
<td id="S3.T3.2.2.9.7.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.9.7.9.1" class="ltx_text ltx_font_bold" style="font-size:80%;">75.2</span></td>
<td id="S3.T3.2.2.9.7.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.9.7.10.1" class="ltx_text" style="font-size:80%;">74.6</span></td>
<td id="S3.T3.2.2.9.7.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.9.7.11.1" class="ltx_text" style="font-size:80%;">68.0</span></td>
<td id="S3.T3.2.2.9.7.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.9.7.12.1" class="ltx_text" style="font-size:80%;">—</span></td>
<td id="S3.T3.2.2.9.7.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.9.7.13.1" class="ltx_text" style="font-size:80%;">—</span></td>
<td id="S3.T3.2.2.9.7.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.9.7.14.1" class="ltx_text" style="font-size:80%;">—</span></td>
<td id="S3.T3.2.2.9.7.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.9.7.15.1" class="ltx_text" style="font-size:80%;">—</span></td>
</tr>
<tr id="S3.T3.2.2.10.8" class="ltx_tr" style="background-color:#F2F3F5;">
<th id="S3.T3.2.2.10.8.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.10.8.1.1" class="ltx_text" style="font-size:80%;background-color:#F2F3F5;">IXC-2.5-7B</span></th>
<td id="S3.T3.2.2.10.8.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.10.8.2.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#F2F3F5;">69.1</span></td>
<td id="S3.T3.2.2.10.8.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.10.8.3.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#F2F3F5;">58.8</span></td>
<td id="S3.T3.2.2.10.8.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.10.8.4.1" class="ltx_text" style="font-size:80%;background-color:#F2F3F5;">55.8</span></td>
<td id="S3.T3.2.2.10.8.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.10.8.5.1" class="ltx_text" style="font-size:80%;background-color:#F2F3F5;">46.9</span></td>
<td id="S3.T3.2.2.10.8.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_r ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.10.8.6.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;background-color:#F2F3F5;">67.1</span></td>
<td id="S3.T3.2.2.10.8.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.10.8.7.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#F2F3F5;">90.9</span></td>
<td id="S3.T3.2.2.10.8.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.10.8.8.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;background-color:#F2F3F5;">82.2</span></td>
<td id="S3.T3.2.2.10.8.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.10.8.9.1" class="ltx_text" style="font-size:80%;background-color:#F2F3F5;">69.9</span></td>
<td id="S3.T3.2.2.10.8.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.10.8.10.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;background-color:#F2F3F5;">78.2</span></td>
<td id="S3.T3.2.2.10.8.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.10.8.11.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;background-color:#F2F3F5;">69.0</span></td>
<td id="S3.T3.2.2.10.8.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.10.8.12.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#F2F3F5;">53.6</span></td>
<td id="S3.T3.2.2.10.8.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.10.8.13.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#F2F3F5;">71.2</span></td>
<td id="S3.T3.2.2.10.8.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.10.8.14.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#F2F3F5;">307.5</span></td>
<td id="S3.T3.2.2.10.8.15" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T3.2.2.10.8.15.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#F2F3F5;">85.2</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_table"><span id="S3.T3.26.5.1" class="ltx_text" style="font-size:113%;">Table 3</span>: </span><span id="S3.T3.10.4" class="ltx_text" style="font-size:113%;">Comparison with closed-source APIs and previous open-source SOTAs <span id="S3.T3.10.4.1" class="ltx_text ltx_font_bold">on Video Benchmarks and Structural High-resolution Benchmarks.</span> The best results are <span id="S3.T3.10.4.2" class="ltx_text ltx_font_bold">bold</span> and the second-best results are <span id="S3.T3.10.4.3" class="ltx_text ltx_framed ltx_framed_underline">underlined</span>. <sup id="S3.T3.10.4.4" class="ltx_sup"><span id="S3.T3.10.4.4.1" class="ltx_text ltx_font_italic">∗1</span></sup> We scale the score from <math id="S3.T3.8.2.m2.1" class="ltx_Math" alttext="0\sim 3" display="inline"><semantics id="S3.T3.8.2.m2.1b"><mrow id="S3.T3.8.2.m2.1.1" xref="S3.T3.8.2.m2.1.1.cmml"><mn id="S3.T3.8.2.m2.1.1.2" xref="S3.T3.8.2.m2.1.1.2.cmml">0</mn><mo id="S3.T3.8.2.m2.1.1.1" xref="S3.T3.8.2.m2.1.1.1.cmml">∼</mo><mn id="S3.T3.8.2.m2.1.1.3" xref="S3.T3.8.2.m2.1.1.3.cmml">3</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.T3.8.2.m2.1c"><apply id="S3.T3.8.2.m2.1.1.cmml" xref="S3.T3.8.2.m2.1.1"><csymbol cd="latexml" id="S3.T3.8.2.m2.1.1.1.cmml" xref="S3.T3.8.2.m2.1.1.1">similar-to</csymbol><cn type="integer" id="S3.T3.8.2.m2.1.1.2.cmml" xref="S3.T3.8.2.m2.1.1.2">0</cn><cn type="integer" id="S3.T3.8.2.m2.1.1.3.cmml" xref="S3.T3.8.2.m2.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T3.8.2.m2.1d">0\sim 3</annotation></semantics></math> to <math id="S3.T3.9.3.m3.1" class="ltx_Math" alttext="0\sim 100" display="inline"><semantics id="S3.T3.9.3.m3.1b"><mrow id="S3.T3.9.3.m3.1.1" xref="S3.T3.9.3.m3.1.1.cmml"><mn id="S3.T3.9.3.m3.1.1.2" xref="S3.T3.9.3.m3.1.1.2.cmml">0</mn><mo id="S3.T3.9.3.m3.1.1.1" xref="S3.T3.9.3.m3.1.1.1.cmml">∼</mo><mn id="S3.T3.9.3.m3.1.1.3" xref="S3.T3.9.3.m3.1.1.3.cmml">100</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.T3.9.3.m3.1c"><apply id="S3.T3.9.3.m3.1.1.cmml" xref="S3.T3.9.3.m3.1.1"><csymbol cd="latexml" id="S3.T3.9.3.m3.1.1.1.cmml" xref="S3.T3.9.3.m3.1.1.1">similar-to</csymbol><cn type="integer" id="S3.T3.9.3.m3.1.1.2.cmml" xref="S3.T3.9.3.m3.1.1.2">0</cn><cn type="integer" id="S3.T3.9.3.m3.1.1.3.cmml" xref="S3.T3.9.3.m3.1.1.3">100</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.T3.9.3.m3.1d">0\sim 100</annotation></semantics></math> for easier understanding. <sup id="S3.T3.10.4.5" class="ltx_sup"><span id="S3.T3.10.4.5.1" class="ltx_text ltx_font_italic">∗2</span></sup> We report the determinism part (MCQA, Y/N, Caption Match) of TempCompass as the evaluation using GPT-3.5 is not stable.</span></figcaption>
</figure>
<figure id="S3.T4" class="ltx_table">
<div id="S3.T4.3" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:35pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-675.7pt,54.5pt) scale(0.242931629101041,0.242931629101041) ;">
<table id="S3.T4.3.3" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S3.T4.3.3.3" class="ltx_tr">
<th id="S3.T4.3.3.3.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_th ltx_th_row ltx_border_r ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"></th>
<th id="S3.T4.3.3.3.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.3.5.1" class="ltx_text" style="font-size:80%;">MMDU</span></th>
<td id="S3.T4.3.3.3.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.3.6.1" class="ltx_text" style="font-size:80%;">MMStar</span></td>
<td id="S3.T4.3.3.3.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.3.7.1" class="ltx_text" style="font-size:80%;">RealWQA</span></td>
<td id="S3.T4.3.3.3.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.3.8.1" class="ltx_text" style="font-size:80%;">MathVista</span></td>
<td id="S3.T4.3.3.3.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.3.9.1" class="ltx_text" style="font-size:80%;">AI2D</span></td>
<td id="S3.T4.3.3.3.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.3.10.1" class="ltx_text" style="font-size:80%;">MMMU</span></td>
<td id="S3.T4.3.3.3.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.3.11.1" class="ltx_text" style="font-size:80%;">MME</span></td>
<td id="S3.T4.3.3.3.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.3.12.1" class="ltx_text" style="font-size:80%;">MMB</span></td>
<td id="S3.T4.1.1.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T4.1.1.1.1.1" class="ltx_text" style="font-size:80%;">MMB</span><sub id="S3.T4.1.1.1.1.2" class="ltx_sub"><span id="S3.T4.1.1.1.1.2.1" class="ltx_text ltx_font_italic" style="font-size:80%;">CN</span></sub>
</td>
<td id="S3.T4.2.2.2.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T4.2.2.2.2.1" class="ltx_text" style="font-size:80%;">MMB</span><sub id="S3.T4.2.2.2.2.2" class="ltx_sub"><span id="S3.T4.2.2.2.2.2.1" class="ltx_text ltx_font_italic" style="font-size:80%;">1.1</span></sub>
</td>
<td id="S3.T4.3.3.3.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T4.3.3.3.3.1" class="ltx_text" style="font-size:80%;">SEED</span><sup id="S3.T4.3.3.3.3.2" class="ltx_sup"><span id="S3.T4.3.3.3.3.2.1" class="ltx_text ltx_font_italic" style="font-size:80%;">I</span></sup>
</td>
<td id="S3.T4.3.3.3.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.3.13.1" class="ltx_text" style="font-size:80%;">MM-Vet</span></td>
<td id="S3.T4.3.3.3.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.3.14.1" class="ltx_text" style="font-size:80%;">HallB</span></td>
</tr>
<tr id="S3.T4.3.3.4.1" class="ltx_tr">
<th id="S3.T4.3.3.4.1.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_r ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.4.1.1.1" class="ltx_text" style="font-size:80%;">Open-Source</span></th>
<th id="S3.T4.3.3.4.1.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.4.1.2.1" class="ltx_text" style="font-size:80%;">LLaVa1.6</span></th>
<td id="S3.T4.3.3.4.1.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.4.1.3.1" class="ltx_text" style="font-size:80%;">InternVL</span></td>
<td id="S3.T4.3.3.4.1.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.4.1.4.1" class="ltx_text" style="font-size:80%;">WeMM</span></td>
<td id="S3.T4.3.3.4.1.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.4.1.5.1" class="ltx_text" style="font-size:80%;">WeMM</span></td>
<td id="S3.T4.3.3.4.1.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.4.1.6.1" class="ltx_text" style="font-size:80%;">InternVL</span></td>
<td id="S3.T4.3.3.4.1.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.4.1.7.1" class="ltx_text" style="font-size:80%;">360VL</span></td>
<td id="S3.T4.3.3.4.1.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.4.1.8.1" class="ltx_text" style="font-size:80%;">InternVL</span></td>
<td id="S3.T4.3.3.4.1.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.4.1.9.1" class="ltx_text" style="font-size:80%;">InternVL1.5</span></td>
<td id="S3.T4.3.3.4.1.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.4.1.10.1" class="ltx_text" style="font-size:80%;">InternVL1.5</span></td>
<td id="S3.T4.3.3.4.1.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.4.1.11.1" class="ltx_text" style="font-size:80%;">InternVL1.5</span></td>
<td id="S3.T4.3.3.4.1.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.4.1.12.1" class="ltx_text" style="font-size:80%;">WeMM</span></td>
<td id="S3.T4.3.3.4.1.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.4.1.13.1" class="ltx_text" style="font-size:80%;">GLM-4v</span></td>
<td id="S3.T4.3.3.4.1.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.4.1.14.1" class="ltx_text" style="font-size:80%;">WeMM</span></td>
</tr>
<tr id="S3.T4.3.3.5.2" class="ltx_tr">
<th id="S3.T4.3.3.5.2.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_r" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.5.2.1.1" class="ltx_text" style="font-size:80%;">Previous SOTA</span></th>
<th id="S3.T4.3.3.5.2.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_row ltx_border_r" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T4.3.3.5.2.2.1" class="ltx_text" style="font-size:80%;">8B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T4.3.3.5.2.2.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib83" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">83</span></a><span id="S3.T4.3.3.5.2.2.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</th>
<td id="S3.T4.3.3.5.2.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T4.3.3.5.2.3.1" class="ltx_text" style="font-size:80%;">1.5-26B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T4.3.3.5.2.3.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">26</span></a><span id="S3.T4.3.3.5.2.3.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T4.3.3.5.2.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T4.3.3.5.2.4.1" class="ltx_text" style="font-size:80%;">8B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T4.3.3.5.2.4.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib145" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">145</span></a><span id="S3.T4.3.3.5.2.4.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T4.3.3.5.2.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T4.3.3.5.2.5.1" class="ltx_text" style="font-size:80%;">8B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T4.3.3.5.2.5.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib145" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">145</span></a><span id="S3.T4.3.3.5.2.5.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T4.3.3.5.2.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T4.3.3.5.2.6.1" class="ltx_text" style="font-size:80%;">1.5-26B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T4.3.3.5.2.6.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">26</span></a><span id="S3.T4.3.3.5.2.6.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T4.3.3.5.2.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T4.3.3.5.2.7.1" class="ltx_text" style="font-size:80%;">70B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T4.3.3.5.2.7.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib144" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">144</span></a><span id="S3.T4.3.3.5.2.7.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T4.3.3.5.2.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T4.3.3.5.2.8.1" class="ltx_text" style="font-size:80%;">1.5-26B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T4.3.3.5.2.8.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">26</span></a><span id="S3.T4.3.3.5.2.8.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T4.3.3.5.2.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T4.3.3.5.2.9.1" class="ltx_text" style="font-size:80%;">1.5-26B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T4.3.3.5.2.9.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">26</span></a><span id="S3.T4.3.3.5.2.9.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T4.3.3.5.2.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T4.3.3.5.2.10.1" class="ltx_text" style="font-size:80%;">1.5-26B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T4.3.3.5.2.10.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">26</span></a><span id="S3.T4.3.3.5.2.10.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T4.3.3.5.2.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T4.3.3.5.2.11.1" class="ltx_text" style="font-size:80%;">1.5-26B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T4.3.3.5.2.11.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib26" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">26</span></a><span id="S3.T4.3.3.5.2.11.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T4.3.3.5.2.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T4.3.3.5.2.12.1" class="ltx_text" style="font-size:80%;">8B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T4.3.3.5.2.12.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib145" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">145</span></a><span id="S3.T4.3.3.5.2.12.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T4.3.3.5.2.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T4.3.3.5.2.13.1" class="ltx_text" style="font-size:80%;">14B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T4.3.3.5.2.13.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib43" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">43</span></a><span id="S3.T4.3.3.5.2.13.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
<td id="S3.T4.3.3.5.2.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T4.3.3.5.2.14.1" class="ltx_text" style="font-size:80%;">8B</span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T4.3.3.5.2.14.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib145" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">145</span></a><span id="S3.T4.3.3.5.2.14.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</td>
</tr>
<tr id="S3.T4.3.3.6.3" class="ltx_tr">
<th id="S3.T4.3.3.6.3.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_r" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.6.3.1.1" class="ltx_text" style="font-size:80%;">Performance</span></th>
<th id="S3.T4.3.3.6.3.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_row ltx_border_r" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.6.3.2.1" class="ltx_text" style="font-size:80%;">42.8</span></th>
<td id="S3.T4.3.3.6.3.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.6.3.3.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">57.1</span></td>
<td id="S3.T4.3.3.6.3.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.6.3.4.1" class="ltx_text ltx_font_bold" style="font-size:80%;">68.1</span></td>
<td id="S3.T4.3.3.6.3.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.6.3.5.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">54.9</span></td>
<td id="S3.T4.3.3.6.3.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.6.3.6.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">80.6</span></td>
<td id="S3.T4.3.3.6.3.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.6.3.7.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">53.4</span></td>
<td id="S3.T4.3.3.6.3.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.6.3.8.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">2,189.6</span></td>
<td id="S3.T4.3.3.6.3.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.6.3.9.1" class="ltx_text ltx_font_bold" style="font-size:80%;">82.3</span></td>
<td id="S3.T4.3.3.6.3.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.6.3.10.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">80.7</span></td>
<td id="S3.T4.3.3.6.3.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.6.3.11.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">79.7</span></td>
<td id="S3.T4.3.3.6.3.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.6.3.12.1" class="ltx_text ltx_font_bold" style="font-size:80%;">75.9</span></td>
<td id="S3.T4.3.3.6.3.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.6.3.13.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">58.0</span></td>
<td id="S3.T4.3.3.6.3.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.6.3.14.1" class="ltx_text ltx_font_bold" style="font-size:80%;">47.5</span></td>
</tr>
<tr id="S3.T4.3.3.7.4" class="ltx_tr">
<th id="S3.T4.3.3.7.4.1" class="ltx_td ltx_nopad_l ltx_align_left ltx_th ltx_th_row ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;" colspan="11"><span id="S3.T4.3.3.7.4.1.1" class="ltx_text ltx_font_italic" style="font-size:80%;">Closed-source API</span></th>
<td id="S3.T4.3.3.7.4.2" class="ltx_td ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"></td>
<td id="S3.T4.3.3.7.4.3" class="ltx_td ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"></td>
<td id="S3.T4.3.3.7.4.4" class="ltx_td ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"></td>
</tr>
<tr id="S3.T4.3.3.8.5" class="ltx_tr">
<th id="S3.T4.3.3.8.5.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_r" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T4.3.3.8.5.1.1" class="ltx_text" style="font-size:80%;">GPT-4V </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T4.3.3.8.5.1.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib112" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">112</span></a><span id="S3.T4.3.3.8.5.1.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</th>
<th id="S3.T4.3.3.8.5.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_row ltx_border_r" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.8.5.2.1" class="ltx_text" style="font-size:80%;">66.3</span></th>
<td id="S3.T4.3.3.8.5.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.8.5.3.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">57.1</span></td>
<td id="S3.T4.3.3.8.5.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.8.5.4.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">68.0</span></td>
<td id="S3.T4.3.3.8.5.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.8.5.5.1" class="ltx_text" style="font-size:80%;">47.8</span></td>
<td id="S3.T4.3.3.8.5.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.8.5.6.1" class="ltx_text" style="font-size:80%;">75.5</span></td>
<td id="S3.T4.3.3.8.5.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.8.5.7.1" class="ltx_text ltx_font_bold" style="font-size:80%;">56.8</span></td>
<td id="S3.T4.3.3.8.5.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.8.5.8.1" class="ltx_text" style="font-size:80%;">1,926.5</span></td>
<td id="S3.T4.3.3.8.5.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.8.5.9.1" class="ltx_text" style="font-size:80%;">81.3</span></td>
<td id="S3.T4.3.3.8.5.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.8.5.10.1" class="ltx_text" style="font-size:80%;">80.2</span></td>
<td id="S3.T4.3.3.8.5.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.8.5.11.1" class="ltx_text ltx_font_bold" style="font-size:80%;">79.8</span></td>
<td id="S3.T4.3.3.8.5.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.8.5.12.1" class="ltx_text" style="font-size:80%;">69.1</span></td>
<td id="S3.T4.3.3.8.5.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.8.5.13.1" class="ltx_text" style="font-size:80%;">56.8</span></td>
<td id="S3.T4.3.3.8.5.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.8.5.14.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">46.5</span></td>
</tr>
<tr id="S3.T4.3.3.9.6" class="ltx_tr">
<th id="S3.T4.3.3.9.6.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_r" style="padding-left:0.6pt;padding-right:0.6pt;">
<span id="S3.T4.3.3.9.6.1.1" class="ltx_text" style="font-size:80%;">Gemini-Pro </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T4.3.3.9.6.1.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib142" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">142</span></a><span id="S3.T4.3.3.9.6.1.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</th>
<th id="S3.T4.3.3.9.6.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_row ltx_border_r" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.9.6.2.1" class="ltx_text" style="font-size:80%;">—</span></th>
<td id="S3.T4.3.3.9.6.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.9.6.3.1" class="ltx_text" style="font-size:80%;">42.6</span></td>
<td id="S3.T4.3.3.9.6.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.9.6.4.1" class="ltx_text" style="font-size:80%;">64.1</span></td>
<td id="S3.T4.3.3.9.6.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.9.6.5.1" class="ltx_text" style="font-size:80%;">45.8</span></td>
<td id="S3.T4.3.3.9.6.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.9.6.6.1" class="ltx_text" style="font-size:80%;">70.2</span></td>
<td id="S3.T4.3.3.9.6.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.9.6.7.1" class="ltx_text" style="font-size:80%;">47.9</span></td>
<td id="S3.T4.3.3.9.6.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.9.6.8.1" class="ltx_text" style="font-size:80%;">1,933.3</span></td>
<td id="S3.T4.3.3.9.6.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.9.6.9.1" class="ltx_text" style="font-size:80%;">73.9</span></td>
<td id="S3.T4.3.3.9.6.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.9.6.10.1" class="ltx_text" style="font-size:80%;">74.3</span></td>
<td id="S3.T4.3.3.9.6.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.9.6.11.1" class="ltx_text" style="font-size:80%;">73.9</span></td>
<td id="S3.T4.3.3.9.6.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.9.6.12.1" class="ltx_text" style="font-size:80%;">70.7</span></td>
<td id="S3.T4.3.3.9.6.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.9.6.13.1" class="ltx_text ltx_font_bold" style="font-size:80%;">59.2</span></td>
<td id="S3.T4.3.3.9.6.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.9.6.14.1" class="ltx_text" style="font-size:80%;">45.2</span></td>
</tr>
<tr id="S3.T4.3.3.10.7" class="ltx_tr" style="background-color:#F2F3F5;">
<th id="S3.T4.3.3.10.7.1" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.10.7.1.1" class="ltx_text" style="font-size:80%;background-color:#F2F3F5;">IXC-2.5-7B</span></th>
<th id="S3.T4.3.3.10.7.2" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_th ltx_th_row ltx_border_bb ltx_border_r ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.10.7.2.1" class="ltx_text" style="font-size:80%;background-color:#F2F3F5;">56.6</span></th>
<td id="S3.T4.3.3.10.7.3" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.10.7.3.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#F2F3F5;">59.9</span></td>
<td id="S3.T4.3.3.10.7.4" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.10.7.4.1" class="ltx_text" style="font-size:80%;background-color:#F2F3F5;">67.8</span></td>
<td id="S3.T4.3.3.10.7.5" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.10.7.5.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#F2F3F5;">63.8</span></td>
<td id="S3.T4.3.3.10.7.6" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.10.7.6.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#F2F3F5;">81.5</span></td>
<td id="S3.T4.3.3.10.7.7" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.10.7.7.1" class="ltx_text" style="font-size:80%;background-color:#F2F3F5;">42.9</span></td>
<td id="S3.T4.3.3.10.7.8" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.10.7.8.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#F2F3F5;">2,229.0</span></td>
<td id="S3.T4.3.3.10.7.9" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.10.7.9.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;background-color:#F2F3F5;">82.2</span></td>
<td id="S3.T4.3.3.10.7.10" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.10.7.10.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#F2F3F5;">80.8</span></td>
<td id="S3.T4.3.3.10.7.11" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.10.7.11.1" class="ltx_text" style="font-size:80%;background-color:#F2F3F5;">79.4</span></td>
<td id="S3.T4.3.3.10.7.12" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.10.7.12.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;background-color:#F2F3F5;">75.4</span></td>
<td id="S3.T4.3.3.10.7.13" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.10.7.13.1" class="ltx_text" style="font-size:80%;background-color:#F2F3F5;">51.7</span></td>
<td id="S3.T4.3.3.10.7.14" class="ltx_td ltx_nopad_l ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t" style="padding-left:0.6pt;padding-right:0.6pt;"><span id="S3.T4.3.3.10.7.14.1" class="ltx_text" style="font-size:80%;background-color:#F2F3F5;">42.4</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_table"><span id="S3.T4.13.1.1" class="ltx_text" style="font-size:113%;">Table 4</span>: </span><span id="S3.T4.14.2" class="ltx_text" style="font-size:113%;">Comparison with closed-source APIs and previous open-source SOTAs <span id="S3.T4.14.2.1" class="ltx_text ltx_font_bold">on Multi-Turn Multi-Image Dialog and General Visual QA Benchmarks.</span> The best results are <span id="S3.T4.14.2.2" class="ltx_text ltx_font_bold">bold</span> and the second-best results are <span id="S3.T4.14.2.3" class="ltx_text ltx_framed ltx_framed_underline">underlined</span>.</span></figcaption>
</figure>
<div id="S3.SS6.p3" class="ltx_para ltx_noindent">
<p id="S3.SS6.p3.9" class="ltx_p"><span id="S3.SS6.p3.9.1" class="ltx_text ltx_font_bold">Preference Data Collection.</span>
We use the fine-tuned model <math id="S3.SS6.p3.1.m1.1" class="ltx_Math" alttext="\pi_{\text{ref}}" display="inline"><semantics id="S3.SS6.p3.1.m1.1a"><msub id="S3.SS6.p3.1.m1.1.1" xref="S3.SS6.p3.1.m1.1.1.cmml"><mi id="S3.SS6.p3.1.m1.1.1.2" xref="S3.SS6.p3.1.m1.1.1.2.cmml">π</mi><mtext id="S3.SS6.p3.1.m1.1.1.3" xref="S3.SS6.p3.1.m1.1.1.3a.cmml">ref</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS6.p3.1.m1.1b"><apply id="S3.SS6.p3.1.m1.1.1.cmml" xref="S3.SS6.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS6.p3.1.m1.1.1.1.cmml" xref="S3.SS6.p3.1.m1.1.1">subscript</csymbol><ci id="S3.SS6.p3.1.m1.1.1.2.cmml" xref="S3.SS6.p3.1.m1.1.1.2">𝜋</ci><ci id="S3.SS6.p3.1.m1.1.1.3a.cmml" xref="S3.SS6.p3.1.m1.1.1.3"><mtext mathsize="70%" id="S3.SS6.p3.1.m1.1.1.3.cmml" xref="S3.SS6.p3.1.m1.1.1.3">ref</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p3.1.m1.1c">\pi_{\text{ref}}</annotation></semantics></math> to generate diverse responses for each prompt in the augmented instruction tuning data <math id="S3.SS6.p3.2.m2.1" class="ltx_Math" alttext="\mathcal{D}^{*}" display="inline"><semantics id="S3.SS6.p3.2.m2.1a"><msup id="S3.SS6.p3.2.m2.1.1" xref="S3.SS6.p3.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS6.p3.2.m2.1.1.2" xref="S3.SS6.p3.2.m2.1.1.2.cmml">𝒟</mi><mo id="S3.SS6.p3.2.m2.1.1.3" xref="S3.SS6.p3.2.m2.1.1.3.cmml">∗</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SS6.p3.2.m2.1b"><apply id="S3.SS6.p3.2.m2.1.1.cmml" xref="S3.SS6.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS6.p3.2.m2.1.1.1.cmml" xref="S3.SS6.p3.2.m2.1.1">superscript</csymbol><ci id="S3.SS6.p3.2.m2.1.1.2.cmml" xref="S3.SS6.p3.2.m2.1.1.2">𝒟</ci><times id="S3.SS6.p3.2.m2.1.1.3.cmml" xref="S3.SS6.p3.2.m2.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p3.2.m2.1c">\mathcal{D}^{*}</annotation></semantics></math>, using different random seeds. This yields a collection of 80,000 prompt-response pairs. Next, we employ the GPT-4o model to label 2,000 responses with chosen or rejected decisions and give the reasons, which serve as our reward modeling data. We then train a reward model <math id="S3.SS6.p3.3.m3.1" class="ltx_Math" alttext="\pi_{\text{rm}}" display="inline"><semantics id="S3.SS6.p3.3.m3.1a"><msub id="S3.SS6.p3.3.m3.1.1" xref="S3.SS6.p3.3.m3.1.1.cmml"><mi id="S3.SS6.p3.3.m3.1.1.2" xref="S3.SS6.p3.3.m3.1.1.2.cmml">π</mi><mtext id="S3.SS6.p3.3.m3.1.1.3" xref="S3.SS6.p3.3.m3.1.1.3a.cmml">rm</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS6.p3.3.m3.1b"><apply id="S3.SS6.p3.3.m3.1.1.cmml" xref="S3.SS6.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS6.p3.3.m3.1.1.1.cmml" xref="S3.SS6.p3.3.m3.1.1">subscript</csymbol><ci id="S3.SS6.p3.3.m3.1.1.2.cmml" xref="S3.SS6.p3.3.m3.1.1.2">𝜋</ci><ci id="S3.SS6.p3.3.m3.1.1.3a.cmml" xref="S3.SS6.p3.3.m3.1.1.3"><mtext mathsize="70%" id="S3.SS6.p3.3.m3.1.1.3.cmml" xref="S3.SS6.p3.3.m3.1.1.3">rm</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p3.3.m3.1c">\pi_{\text{rm}}</annotation></semantics></math>, sharing the same architecture of <math id="S3.SS6.p3.4.m4.1" class="ltx_Math" alttext="\pi_{\text{ref}}" display="inline"><semantics id="S3.SS6.p3.4.m4.1a"><msub id="S3.SS6.p3.4.m4.1.1" xref="S3.SS6.p3.4.m4.1.1.cmml"><mi id="S3.SS6.p3.4.m4.1.1.2" xref="S3.SS6.p3.4.m4.1.1.2.cmml">π</mi><mtext id="S3.SS6.p3.4.m4.1.1.3" xref="S3.SS6.p3.4.m4.1.1.3a.cmml">ref</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS6.p3.4.m4.1b"><apply id="S3.SS6.p3.4.m4.1.1.cmml" xref="S3.SS6.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS6.p3.4.m4.1.1.1.cmml" xref="S3.SS6.p3.4.m4.1.1">subscript</csymbol><ci id="S3.SS6.p3.4.m4.1.1.2.cmml" xref="S3.SS6.p3.4.m4.1.1.2">𝜋</ci><ci id="S3.SS6.p3.4.m4.1.1.3a.cmml" xref="S3.SS6.p3.4.m4.1.1.3"><mtext mathsize="70%" id="S3.SS6.p3.4.m4.1.1.3.cmml" xref="S3.SS6.p3.4.m4.1.1.3">ref</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p3.4.m4.1c">\pi_{\text{ref}}</annotation></semantics></math>, on the reward modeling data. The reward model is used to make the chosen or rejected prediction on the remaining prompt-response pairs. These selected responses are then used to construct the pair data <math id="S3.SS6.p3.5.m5.3" class="ltx_Math" alttext="\mathcal{D}^{p}=\{x,y_{w},y_{l}\}" display="inline"><semantics id="S3.SS6.p3.5.m5.3a"><mrow id="S3.SS6.p3.5.m5.3.3" xref="S3.SS6.p3.5.m5.3.3.cmml"><msup id="S3.SS6.p3.5.m5.3.3.4" xref="S3.SS6.p3.5.m5.3.3.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS6.p3.5.m5.3.3.4.2" xref="S3.SS6.p3.5.m5.3.3.4.2.cmml">𝒟</mi><mi id="S3.SS6.p3.5.m5.3.3.4.3" xref="S3.SS6.p3.5.m5.3.3.4.3.cmml">p</mi></msup><mo id="S3.SS6.p3.5.m5.3.3.3" xref="S3.SS6.p3.5.m5.3.3.3.cmml">=</mo><mrow id="S3.SS6.p3.5.m5.3.3.2.2" xref="S3.SS6.p3.5.m5.3.3.2.3.cmml"><mo stretchy="false" id="S3.SS6.p3.5.m5.3.3.2.2.3" xref="S3.SS6.p3.5.m5.3.3.2.3.cmml">{</mo><mi id="S3.SS6.p3.5.m5.1.1" xref="S3.SS6.p3.5.m5.1.1.cmml">x</mi><mo id="S3.SS6.p3.5.m5.3.3.2.2.4" xref="S3.SS6.p3.5.m5.3.3.2.3.cmml">,</mo><msub id="S3.SS6.p3.5.m5.2.2.1.1.1" xref="S3.SS6.p3.5.m5.2.2.1.1.1.cmml"><mi id="S3.SS6.p3.5.m5.2.2.1.1.1.2" xref="S3.SS6.p3.5.m5.2.2.1.1.1.2.cmml">y</mi><mi id="S3.SS6.p3.5.m5.2.2.1.1.1.3" xref="S3.SS6.p3.5.m5.2.2.1.1.1.3.cmml">w</mi></msub><mo id="S3.SS6.p3.5.m5.3.3.2.2.5" xref="S3.SS6.p3.5.m5.3.3.2.3.cmml">,</mo><msub id="S3.SS6.p3.5.m5.3.3.2.2.2" xref="S3.SS6.p3.5.m5.3.3.2.2.2.cmml"><mi id="S3.SS6.p3.5.m5.3.3.2.2.2.2" xref="S3.SS6.p3.5.m5.3.3.2.2.2.2.cmml">y</mi><mi id="S3.SS6.p3.5.m5.3.3.2.2.2.3" xref="S3.SS6.p3.5.m5.3.3.2.2.2.3.cmml">l</mi></msub><mo stretchy="false" id="S3.SS6.p3.5.m5.3.3.2.2.6" xref="S3.SS6.p3.5.m5.3.3.2.3.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS6.p3.5.m5.3b"><apply id="S3.SS6.p3.5.m5.3.3.cmml" xref="S3.SS6.p3.5.m5.3.3"><eq id="S3.SS6.p3.5.m5.3.3.3.cmml" xref="S3.SS6.p3.5.m5.3.3.3"></eq><apply id="S3.SS6.p3.5.m5.3.3.4.cmml" xref="S3.SS6.p3.5.m5.3.3.4"><csymbol cd="ambiguous" id="S3.SS6.p3.5.m5.3.3.4.1.cmml" xref="S3.SS6.p3.5.m5.3.3.4">superscript</csymbol><ci id="S3.SS6.p3.5.m5.3.3.4.2.cmml" xref="S3.SS6.p3.5.m5.3.3.4.2">𝒟</ci><ci id="S3.SS6.p3.5.m5.3.3.4.3.cmml" xref="S3.SS6.p3.5.m5.3.3.4.3">𝑝</ci></apply><set id="S3.SS6.p3.5.m5.3.3.2.3.cmml" xref="S3.SS6.p3.5.m5.3.3.2.2"><ci id="S3.SS6.p3.5.m5.1.1.cmml" xref="S3.SS6.p3.5.m5.1.1">𝑥</ci><apply id="S3.SS6.p3.5.m5.2.2.1.1.1.cmml" xref="S3.SS6.p3.5.m5.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS6.p3.5.m5.2.2.1.1.1.1.cmml" xref="S3.SS6.p3.5.m5.2.2.1.1.1">subscript</csymbol><ci id="S3.SS6.p3.5.m5.2.2.1.1.1.2.cmml" xref="S3.SS6.p3.5.m5.2.2.1.1.1.2">𝑦</ci><ci id="S3.SS6.p3.5.m5.2.2.1.1.1.3.cmml" xref="S3.SS6.p3.5.m5.2.2.1.1.1.3">𝑤</ci></apply><apply id="S3.SS6.p3.5.m5.3.3.2.2.2.cmml" xref="S3.SS6.p3.5.m5.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.SS6.p3.5.m5.3.3.2.2.2.1.cmml" xref="S3.SS6.p3.5.m5.3.3.2.2.2">subscript</csymbol><ci id="S3.SS6.p3.5.m5.3.3.2.2.2.2.cmml" xref="S3.SS6.p3.5.m5.3.3.2.2.2.2">𝑦</ci><ci id="S3.SS6.p3.5.m5.3.3.2.2.2.3.cmml" xref="S3.SS6.p3.5.m5.3.3.2.2.2.3">𝑙</ci></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p3.5.m5.3c">\mathcal{D}^{p}=\{x,y_{w},y_{l}\}</annotation></semantics></math>, while <math id="S3.SS6.p3.6.m6.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S3.SS6.p3.6.m6.1a"><mi id="S3.SS6.p3.6.m6.1.1" xref="S3.SS6.p3.6.m6.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S3.SS6.p3.6.m6.1b"><ci id="S3.SS6.p3.6.m6.1.1.cmml" xref="S3.SS6.p3.6.m6.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p3.6.m6.1c">x</annotation></semantics></math>, <math id="S3.SS6.p3.7.m7.1" class="ltx_Math" alttext="y_{w}" display="inline"><semantics id="S3.SS6.p3.7.m7.1a"><msub id="S3.SS6.p3.7.m7.1.1" xref="S3.SS6.p3.7.m7.1.1.cmml"><mi id="S3.SS6.p3.7.m7.1.1.2" xref="S3.SS6.p3.7.m7.1.1.2.cmml">y</mi><mi id="S3.SS6.p3.7.m7.1.1.3" xref="S3.SS6.p3.7.m7.1.1.3.cmml">w</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS6.p3.7.m7.1b"><apply id="S3.SS6.p3.7.m7.1.1.cmml" xref="S3.SS6.p3.7.m7.1.1"><csymbol cd="ambiguous" id="S3.SS6.p3.7.m7.1.1.1.cmml" xref="S3.SS6.p3.7.m7.1.1">subscript</csymbol><ci id="S3.SS6.p3.7.m7.1.1.2.cmml" xref="S3.SS6.p3.7.m7.1.1.2">𝑦</ci><ci id="S3.SS6.p3.7.m7.1.1.3.cmml" xref="S3.SS6.p3.7.m7.1.1.3">𝑤</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p3.7.m7.1c">y_{w}</annotation></semantics></math> and <math id="S3.SS6.p3.8.m8.1" class="ltx_Math" alttext="y_{l}" display="inline"><semantics id="S3.SS6.p3.8.m8.1a"><msub id="S3.SS6.p3.8.m8.1.1" xref="S3.SS6.p3.8.m8.1.1.cmml"><mi id="S3.SS6.p3.8.m8.1.1.2" xref="S3.SS6.p3.8.m8.1.1.2.cmml">y</mi><mi id="S3.SS6.p3.8.m8.1.1.3" xref="S3.SS6.p3.8.m8.1.1.3.cmml">l</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS6.p3.8.m8.1b"><apply id="S3.SS6.p3.8.m8.1.1.cmml" xref="S3.SS6.p3.8.m8.1.1"><csymbol cd="ambiguous" id="S3.SS6.p3.8.m8.1.1.1.cmml" xref="S3.SS6.p3.8.m8.1.1">subscript</csymbol><ci id="S3.SS6.p3.8.m8.1.1.2.cmml" xref="S3.SS6.p3.8.m8.1.1.2">𝑦</ci><ci id="S3.SS6.p3.8.m8.1.1.3.cmml" xref="S3.SS6.p3.8.m8.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p3.8.m8.1c">y_{l}</annotation></semantics></math> refer to the prompt, chosen response and rejected response, respectively. Ultimately, we obtain a total of 30,000 preference data <math id="S3.SS6.p3.9.m9.1" class="ltx_Math" alttext="\mathcal{D}^{p}" display="inline"><semantics id="S3.SS6.p3.9.m9.1a"><msup id="S3.SS6.p3.9.m9.1.1" xref="S3.SS6.p3.9.m9.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS6.p3.9.m9.1.1.2" xref="S3.SS6.p3.9.m9.1.1.2.cmml">𝒟</mi><mi id="S3.SS6.p3.9.m9.1.1.3" xref="S3.SS6.p3.9.m9.1.1.3.cmml">p</mi></msup><annotation-xml encoding="MathML-Content" id="S3.SS6.p3.9.m9.1b"><apply id="S3.SS6.p3.9.m9.1.1.cmml" xref="S3.SS6.p3.9.m9.1.1"><csymbol cd="ambiguous" id="S3.SS6.p3.9.m9.1.1.1.cmml" xref="S3.SS6.p3.9.m9.1.1">superscript</csymbol><ci id="S3.SS6.p3.9.m9.1.1.2.cmml" xref="S3.SS6.p3.9.m9.1.1.2">𝒟</ci><ci id="S3.SS6.p3.9.m9.1.1.3.cmml" xref="S3.SS6.p3.9.m9.1.1.3">𝑝</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p3.9.m9.1c">\mathcal{D}^{p}</annotation></semantics></math> for DPO <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib124" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">124</span></a>]</cite> alignment.</p>
</div>
<div id="S3.SS6.p4" class="ltx_para ltx_noindent">
<p id="S3.SS6.p4.2" class="ltx_p"><span id="S3.SS6.p4.2.1" class="ltx_text ltx_font_bold">DPO Alignment.</span>
We use the DPO algorithm to update the SFT model <math id="S3.SS6.p4.1.m1.1" class="ltx_Math" alttext="\pi_{\text{ref}}" display="inline"><semantics id="S3.SS6.p4.1.m1.1a"><msub id="S3.SS6.p4.1.m1.1.1" xref="S3.SS6.p4.1.m1.1.1.cmml"><mi id="S3.SS6.p4.1.m1.1.1.2" xref="S3.SS6.p4.1.m1.1.1.2.cmml">π</mi><mtext id="S3.SS6.p4.1.m1.1.1.3" xref="S3.SS6.p4.1.m1.1.1.3a.cmml">ref</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS6.p4.1.m1.1b"><apply id="S3.SS6.p4.1.m1.1.1.cmml" xref="S3.SS6.p4.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS6.p4.1.m1.1.1.1.cmml" xref="S3.SS6.p4.1.m1.1.1">subscript</csymbol><ci id="S3.SS6.p4.1.m1.1.1.2.cmml" xref="S3.SS6.p4.1.m1.1.1.2">𝜋</ci><ci id="S3.SS6.p4.1.m1.1.1.3a.cmml" xref="S3.SS6.p4.1.m1.1.1.3"><mtext mathsize="70%" id="S3.SS6.p4.1.m1.1.1.3.cmml" xref="S3.SS6.p4.1.m1.1.1.3">ref</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p4.1.m1.1c">\pi_{\text{ref}}</annotation></semantics></math> on target policy from the preference data <math id="S3.SS6.p4.2.m2.1" class="ltx_Math" alttext="\mathcal{D}^{p}" display="inline"><semantics id="S3.SS6.p4.2.m2.1a"><msup id="S3.SS6.p4.2.m2.1.1" xref="S3.SS6.p4.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS6.p4.2.m2.1.1.2" xref="S3.SS6.p4.2.m2.1.1.2.cmml">𝒟</mi><mi id="S3.SS6.p4.2.m2.1.1.3" xref="S3.SS6.p4.2.m2.1.1.3.cmml">p</mi></msup><annotation-xml encoding="MathML-Content" id="S3.SS6.p4.2.m2.1b"><apply id="S3.SS6.p4.2.m2.1.1.cmml" xref="S3.SS6.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS6.p4.2.m2.1.1.1.cmml" xref="S3.SS6.p4.2.m2.1.1">superscript</csymbol><ci id="S3.SS6.p4.2.m2.1.1.2.cmml" xref="S3.SS6.p4.2.m2.1.1.2">𝒟</ci><ci id="S3.SS6.p4.2.m2.1.1.3.cmml" xref="S3.SS6.p4.2.m2.1.1.3">𝑝</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p4.2.m2.1c">\mathcal{D}^{p}</annotation></semantics></math>:</p>
<table id="S3.E4" class="ltx_equationgroup ltx_eqn_table">
<tbody>
<tr id="S3.E4X" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E4X.2.1.1.m1.5" class="ltx_Math" alttext="\displaystyle\mathcal{L_{\text{DPO}}}(\pi_{\theta},\pi_{\text{ref}})=\mathbb{E}_{x,y_{w},y_{l}\sim\mathcal{D}^{p}}" display="inline"><semantics id="S3.E4X.2.1.1.m1.5a"><mrow id="S3.E4X.2.1.1.m1.5.5" xref="S3.E4X.2.1.1.m1.5.5.cmml"><mrow id="S3.E4X.2.1.1.m1.5.5.2" xref="S3.E4X.2.1.1.m1.5.5.2.cmml"><msub id="S3.E4X.2.1.1.m1.5.5.2.4" xref="S3.E4X.2.1.1.m1.5.5.2.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E4X.2.1.1.m1.5.5.2.4.2" xref="S3.E4X.2.1.1.m1.5.5.2.4.2.cmml">ℒ</mi><mtext id="S3.E4X.2.1.1.m1.5.5.2.4.3" xref="S3.E4X.2.1.1.m1.5.5.2.4.3a.cmml">DPO</mtext></msub><mo lspace="0em" rspace="0em" id="S3.E4X.2.1.1.m1.5.5.2.3" xref="S3.E4X.2.1.1.m1.5.5.2.3.cmml">​</mo><mrow id="S3.E4X.2.1.1.m1.5.5.2.2.2" xref="S3.E4X.2.1.1.m1.5.5.2.2.3.cmml"><mo stretchy="false" id="S3.E4X.2.1.1.m1.5.5.2.2.2.3" xref="S3.E4X.2.1.1.m1.5.5.2.2.3.cmml">(</mo><msub id="S3.E4X.2.1.1.m1.4.4.1.1.1.1" xref="S3.E4X.2.1.1.m1.4.4.1.1.1.1.cmml"><mi id="S3.E4X.2.1.1.m1.4.4.1.1.1.1.2" xref="S3.E4X.2.1.1.m1.4.4.1.1.1.1.2.cmml">π</mi><mi id="S3.E4X.2.1.1.m1.4.4.1.1.1.1.3" xref="S3.E4X.2.1.1.m1.4.4.1.1.1.1.3.cmml">θ</mi></msub><mo id="S3.E4X.2.1.1.m1.5.5.2.2.2.4" xref="S3.E4X.2.1.1.m1.5.5.2.2.3.cmml">,</mo><msub id="S3.E4X.2.1.1.m1.5.5.2.2.2.2" xref="S3.E4X.2.1.1.m1.5.5.2.2.2.2.cmml"><mi id="S3.E4X.2.1.1.m1.5.5.2.2.2.2.2" xref="S3.E4X.2.1.1.m1.5.5.2.2.2.2.2.cmml">π</mi><mtext id="S3.E4X.2.1.1.m1.5.5.2.2.2.2.3" xref="S3.E4X.2.1.1.m1.5.5.2.2.2.2.3a.cmml">ref</mtext></msub><mo stretchy="false" id="S3.E4X.2.1.1.m1.5.5.2.2.2.5" xref="S3.E4X.2.1.1.m1.5.5.2.2.3.cmml">)</mo></mrow></mrow><mo id="S3.E4X.2.1.1.m1.5.5.3" xref="S3.E4X.2.1.1.m1.5.5.3.cmml">=</mo><msub id="S3.E4X.2.1.1.m1.5.5.4" xref="S3.E4X.2.1.1.m1.5.5.4.cmml"><mi id="S3.E4X.2.1.1.m1.5.5.4.2" xref="S3.E4X.2.1.1.m1.5.5.4.2.cmml">𝔼</mi><mrow id="S3.E4X.2.1.1.m1.3.3.3" xref="S3.E4X.2.1.1.m1.3.3.3.cmml"><mrow id="S3.E4X.2.1.1.m1.3.3.3.3.2" xref="S3.E4X.2.1.1.m1.3.3.3.3.3.cmml"><mi id="S3.E4X.2.1.1.m1.1.1.1.1" xref="S3.E4X.2.1.1.m1.1.1.1.1.cmml">x</mi><mo id="S3.E4X.2.1.1.m1.3.3.3.3.2.3" xref="S3.E4X.2.1.1.m1.3.3.3.3.3.cmml">,</mo><msub id="S3.E4X.2.1.1.m1.2.2.2.2.1.1" xref="S3.E4X.2.1.1.m1.2.2.2.2.1.1.cmml"><mi id="S3.E4X.2.1.1.m1.2.2.2.2.1.1.2" xref="S3.E4X.2.1.1.m1.2.2.2.2.1.1.2.cmml">y</mi><mi id="S3.E4X.2.1.1.m1.2.2.2.2.1.1.3" xref="S3.E4X.2.1.1.m1.2.2.2.2.1.1.3.cmml">w</mi></msub><mo id="S3.E4X.2.1.1.m1.3.3.3.3.2.4" xref="S3.E4X.2.1.1.m1.3.3.3.3.3.cmml">,</mo><msub id="S3.E4X.2.1.1.m1.3.3.3.3.2.2" xref="S3.E4X.2.1.1.m1.3.3.3.3.2.2.cmml"><mi id="S3.E4X.2.1.1.m1.3.3.3.3.2.2.2" xref="S3.E4X.2.1.1.m1.3.3.3.3.2.2.2.cmml">y</mi><mi id="S3.E4X.2.1.1.m1.3.3.3.3.2.2.3" xref="S3.E4X.2.1.1.m1.3.3.3.3.2.2.3.cmml">l</mi></msub></mrow><mo id="S3.E4X.2.1.1.m1.3.3.3.4" xref="S3.E4X.2.1.1.m1.3.3.3.4.cmml">∼</mo><msup id="S3.E4X.2.1.1.m1.3.3.3.5" xref="S3.E4X.2.1.1.m1.3.3.3.5.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E4X.2.1.1.m1.3.3.3.5.2" xref="S3.E4X.2.1.1.m1.3.3.3.5.2.cmml">𝒟</mi><mi id="S3.E4X.2.1.1.m1.3.3.3.5.3" xref="S3.E4X.2.1.1.m1.3.3.3.5.3.cmml">p</mi></msup></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.E4X.2.1.1.m1.5b"><apply id="S3.E4X.2.1.1.m1.5.5.cmml" xref="S3.E4X.2.1.1.m1.5.5"><eq id="S3.E4X.2.1.1.m1.5.5.3.cmml" xref="S3.E4X.2.1.1.m1.5.5.3"></eq><apply id="S3.E4X.2.1.1.m1.5.5.2.cmml" xref="S3.E4X.2.1.1.m1.5.5.2"><times id="S3.E4X.2.1.1.m1.5.5.2.3.cmml" xref="S3.E4X.2.1.1.m1.5.5.2.3"></times><apply id="S3.E4X.2.1.1.m1.5.5.2.4.cmml" xref="S3.E4X.2.1.1.m1.5.5.2.4"><csymbol cd="ambiguous" id="S3.E4X.2.1.1.m1.5.5.2.4.1.cmml" xref="S3.E4X.2.1.1.m1.5.5.2.4">subscript</csymbol><ci id="S3.E4X.2.1.1.m1.5.5.2.4.2.cmml" xref="S3.E4X.2.1.1.m1.5.5.2.4.2">ℒ</ci><ci id="S3.E4X.2.1.1.m1.5.5.2.4.3a.cmml" xref="S3.E4X.2.1.1.m1.5.5.2.4.3"><mtext mathsize="70%" id="S3.E4X.2.1.1.m1.5.5.2.4.3.cmml" xref="S3.E4X.2.1.1.m1.5.5.2.4.3">DPO</mtext></ci></apply><interval closure="open" id="S3.E4X.2.1.1.m1.5.5.2.2.3.cmml" xref="S3.E4X.2.1.1.m1.5.5.2.2.2"><apply id="S3.E4X.2.1.1.m1.4.4.1.1.1.1.cmml" xref="S3.E4X.2.1.1.m1.4.4.1.1.1.1"><csymbol cd="ambiguous" id="S3.E4X.2.1.1.m1.4.4.1.1.1.1.1.cmml" xref="S3.E4X.2.1.1.m1.4.4.1.1.1.1">subscript</csymbol><ci id="S3.E4X.2.1.1.m1.4.4.1.1.1.1.2.cmml" xref="S3.E4X.2.1.1.m1.4.4.1.1.1.1.2">𝜋</ci><ci id="S3.E4X.2.1.1.m1.4.4.1.1.1.1.3.cmml" xref="S3.E4X.2.1.1.m1.4.4.1.1.1.1.3">𝜃</ci></apply><apply id="S3.E4X.2.1.1.m1.5.5.2.2.2.2.cmml" xref="S3.E4X.2.1.1.m1.5.5.2.2.2.2"><csymbol cd="ambiguous" id="S3.E4X.2.1.1.m1.5.5.2.2.2.2.1.cmml" xref="S3.E4X.2.1.1.m1.5.5.2.2.2.2">subscript</csymbol><ci id="S3.E4X.2.1.1.m1.5.5.2.2.2.2.2.cmml" xref="S3.E4X.2.1.1.m1.5.5.2.2.2.2.2">𝜋</ci><ci id="S3.E4X.2.1.1.m1.5.5.2.2.2.2.3a.cmml" xref="S3.E4X.2.1.1.m1.5.5.2.2.2.2.3"><mtext mathsize="70%" id="S3.E4X.2.1.1.m1.5.5.2.2.2.2.3.cmml" xref="S3.E4X.2.1.1.m1.5.5.2.2.2.2.3">ref</mtext></ci></apply></interval></apply><apply id="S3.E4X.2.1.1.m1.5.5.4.cmml" xref="S3.E4X.2.1.1.m1.5.5.4"><csymbol cd="ambiguous" id="S3.E4X.2.1.1.m1.5.5.4.1.cmml" xref="S3.E4X.2.1.1.m1.5.5.4">subscript</csymbol><ci id="S3.E4X.2.1.1.m1.5.5.4.2.cmml" xref="S3.E4X.2.1.1.m1.5.5.4.2">𝔼</ci><apply id="S3.E4X.2.1.1.m1.3.3.3.cmml" xref="S3.E4X.2.1.1.m1.3.3.3"><csymbol cd="latexml" id="S3.E4X.2.1.1.m1.3.3.3.4.cmml" xref="S3.E4X.2.1.1.m1.3.3.3.4">similar-to</csymbol><list id="S3.E4X.2.1.1.m1.3.3.3.3.3.cmml" xref="S3.E4X.2.1.1.m1.3.3.3.3.2"><ci id="S3.E4X.2.1.1.m1.1.1.1.1.cmml" xref="S3.E4X.2.1.1.m1.1.1.1.1">𝑥</ci><apply id="S3.E4X.2.1.1.m1.2.2.2.2.1.1.cmml" xref="S3.E4X.2.1.1.m1.2.2.2.2.1.1"><csymbol cd="ambiguous" id="S3.E4X.2.1.1.m1.2.2.2.2.1.1.1.cmml" xref="S3.E4X.2.1.1.m1.2.2.2.2.1.1">subscript</csymbol><ci id="S3.E4X.2.1.1.m1.2.2.2.2.1.1.2.cmml" xref="S3.E4X.2.1.1.m1.2.2.2.2.1.1.2">𝑦</ci><ci id="S3.E4X.2.1.1.m1.2.2.2.2.1.1.3.cmml" xref="S3.E4X.2.1.1.m1.2.2.2.2.1.1.3">𝑤</ci></apply><apply id="S3.E4X.2.1.1.m1.3.3.3.3.2.2.cmml" xref="S3.E4X.2.1.1.m1.3.3.3.3.2.2"><csymbol cd="ambiguous" id="S3.E4X.2.1.1.m1.3.3.3.3.2.2.1.cmml" xref="S3.E4X.2.1.1.m1.3.3.3.3.2.2">subscript</csymbol><ci id="S3.E4X.2.1.1.m1.3.3.3.3.2.2.2.cmml" xref="S3.E4X.2.1.1.m1.3.3.3.3.2.2.2">𝑦</ci><ci id="S3.E4X.2.1.1.m1.3.3.3.3.2.2.3.cmml" xref="S3.E4X.2.1.1.m1.3.3.3.3.2.2.3">𝑙</ci></apply></list><apply id="S3.E4X.2.1.1.m1.3.3.3.5.cmml" xref="S3.E4X.2.1.1.m1.3.3.3.5"><csymbol cd="ambiguous" id="S3.E4X.2.1.1.m1.3.3.3.5.1.cmml" xref="S3.E4X.2.1.1.m1.3.3.3.5">superscript</csymbol><ci id="S3.E4X.2.1.1.m1.3.3.3.5.2.cmml" xref="S3.E4X.2.1.1.m1.3.3.3.5.2">𝒟</ci><ci id="S3.E4X.2.1.1.m1.3.3.3.5.3.cmml" xref="S3.E4X.2.1.1.m1.3.3.3.5.3">𝑝</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E4X.2.1.1.m1.5c">\displaystyle\mathcal{L_{\text{DPO}}}(\pi_{\theta},\pi_{\text{ref}})=\mathbb{E}_{x,y_{w},y_{l}\sim\mathcal{D}^{p}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="2" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equationgroup ltx_align_right">(4)</span></td>
</tr>
<tr id="S3.E4Xa" class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E4Xa.2.1.1.m1.7" class="ltx_Math" alttext="\displaystyle[-\log\sigma(\beta\log(\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)})-\beta\log(\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}))]." display="inline"><semantics id="S3.E4Xa.2.1.1.m1.7a"><mrow id="S3.E4Xa.2.1.1.m1.7.7.1"><mrow id="S3.E4Xa.2.1.1.m1.7.7.1.1.1" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.2.cmml"><mo stretchy="false" id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.2" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.2.1.cmml">[</mo><mrow id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.cmml"><mo rspace="0.167em" id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1a" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.cmml">−</mo><mrow id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.cmml"><mrow id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.3" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.3.cmml"><mi id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.3.1" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.3a" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.3.cmml">⁡</mo><mi id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.3.2" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.3.2.cmml">σ</mi></mrow><mo lspace="0em" rspace="0em" id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.2" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.2" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.cmml"><mrow id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.cmml"><mi id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.2" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.2.cmml">β</mi><mo lspace="0.167em" rspace="0em" id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.1" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.1.cmml">​</mo><mrow id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.3.2" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.3.1.cmml"><mi id="S3.E4Xa.2.1.1.m1.5.5" xref="S3.E4Xa.2.1.1.m1.5.5.cmml">log</mi><mo id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.3.2a" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.3.1.cmml">⁡</mo><mrow id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.3.2.1" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.3.1.cmml"><mo stretchy="false" id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.3.2.1.1" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.3.1.cmml">(</mo><mstyle displaystyle="true" id="S3.E4Xa.2.1.1.m1.2.2" xref="S3.E4Xa.2.1.1.m1.2.2.cmml"><mfrac id="S3.E4Xa.2.1.1.m1.2.2a" xref="S3.E4Xa.2.1.1.m1.2.2.cmml"><mrow id="S3.E4Xa.2.1.1.m1.1.1.1" xref="S3.E4Xa.2.1.1.m1.1.1.1.cmml"><msub id="S3.E4Xa.2.1.1.m1.1.1.1.3" xref="S3.E4Xa.2.1.1.m1.1.1.1.3.cmml"><mi id="S3.E4Xa.2.1.1.m1.1.1.1.3.2" xref="S3.E4Xa.2.1.1.m1.1.1.1.3.2.cmml">π</mi><mi id="S3.E4Xa.2.1.1.m1.1.1.1.3.3" xref="S3.E4Xa.2.1.1.m1.1.1.1.3.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.E4Xa.2.1.1.m1.1.1.1.2" xref="S3.E4Xa.2.1.1.m1.1.1.1.2.cmml">​</mo><mrow id="S3.E4Xa.2.1.1.m1.1.1.1.1.1" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.2" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.cmml"><msub id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.2" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.2.cmml"><mi id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.2.2" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.2.2.cmml">y</mi><mi id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.2.3" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.2.3.cmml">w</mi></msub><mo fence="false" id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.cmml">|</mo><mi id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.cmml">x</mi></mrow><mo stretchy="false" id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.3" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mrow id="S3.E4Xa.2.1.1.m1.2.2.2" xref="S3.E4Xa.2.1.1.m1.2.2.2.cmml"><msub id="S3.E4Xa.2.1.1.m1.2.2.2.3" xref="S3.E4Xa.2.1.1.m1.2.2.2.3.cmml"><mi id="S3.E4Xa.2.1.1.m1.2.2.2.3.2" xref="S3.E4Xa.2.1.1.m1.2.2.2.3.2.cmml">π</mi><mtext id="S3.E4Xa.2.1.1.m1.2.2.2.3.3" xref="S3.E4Xa.2.1.1.m1.2.2.2.3.3a.cmml">ref</mtext></msub><mo lspace="0em" rspace="0em" id="S3.E4Xa.2.1.1.m1.2.2.2.2" xref="S3.E4Xa.2.1.1.m1.2.2.2.2.cmml">​</mo><mrow id="S3.E4Xa.2.1.1.m1.2.2.2.1.1" xref="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.cmml"><mo stretchy="false" id="S3.E4Xa.2.1.1.m1.2.2.2.1.1.2" xref="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.cmml">(</mo><mrow id="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1" xref="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.cmml"><msub id="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.2" xref="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.2.cmml"><mi id="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.2.2" xref="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.2.2.cmml">y</mi><mi id="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.2.3" xref="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.2.3.cmml">w</mi></msub><mo fence="false" id="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.1" xref="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.1.cmml">|</mo><mi id="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.3" xref="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.3.cmml">x</mi></mrow><mo stretchy="false" id="S3.E4Xa.2.1.1.m1.2.2.2.1.1.3" xref="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.cmml">)</mo></mrow></mrow></mfrac></mstyle><mo stretchy="false" id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.3.2.1.2" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.3.1.cmml">)</mo></mrow></mrow></mrow><mo id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.1" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.1.cmml">−</mo><mrow id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.2" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.2.cmml">β</mi><mo lspace="0.167em" rspace="0em" id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.1" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mrow id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.3.2" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.3.1.cmml"><mi id="S3.E4Xa.2.1.1.m1.6.6" xref="S3.E4Xa.2.1.1.m1.6.6.cmml">log</mi><mo id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.3.2a" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.3.1.cmml">⁡</mo><mrow id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.3.2.1" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.3.1.cmml"><mo stretchy="false" id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.3.2.1.1" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.3.1.cmml">(</mo><mstyle displaystyle="true" id="S3.E4Xa.2.1.1.m1.4.4" xref="S3.E4Xa.2.1.1.m1.4.4.cmml"><mfrac id="S3.E4Xa.2.1.1.m1.4.4a" xref="S3.E4Xa.2.1.1.m1.4.4.cmml"><mrow id="S3.E4Xa.2.1.1.m1.3.3.1" xref="S3.E4Xa.2.1.1.m1.3.3.1.cmml"><msub id="S3.E4Xa.2.1.1.m1.3.3.1.3" xref="S3.E4Xa.2.1.1.m1.3.3.1.3.cmml"><mi id="S3.E4Xa.2.1.1.m1.3.3.1.3.2" xref="S3.E4Xa.2.1.1.m1.3.3.1.3.2.cmml">π</mi><mi id="S3.E4Xa.2.1.1.m1.3.3.1.3.3" xref="S3.E4Xa.2.1.1.m1.3.3.1.3.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.E4Xa.2.1.1.m1.3.3.1.2" xref="S3.E4Xa.2.1.1.m1.3.3.1.2.cmml">​</mo><mrow id="S3.E4Xa.2.1.1.m1.3.3.1.1.1" xref="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.cmml"><mo stretchy="false" id="S3.E4Xa.2.1.1.m1.3.3.1.1.1.2" xref="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.cmml">(</mo><mrow id="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1" xref="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.cmml"><msub id="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.2" xref="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.2.cmml"><mi id="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.2.2" xref="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.2.2.cmml">y</mi><mi id="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.2.3" xref="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.2.3.cmml">l</mi></msub><mo fence="false" id="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.1" xref="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.1.cmml">|</mo><mi id="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.3" xref="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.3.cmml">x</mi></mrow><mo stretchy="false" id="S3.E4Xa.2.1.1.m1.3.3.1.1.1.3" xref="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.cmml">)</mo></mrow></mrow><mrow id="S3.E4Xa.2.1.1.m1.4.4.2" xref="S3.E4Xa.2.1.1.m1.4.4.2.cmml"><msub id="S3.E4Xa.2.1.1.m1.4.4.2.3" xref="S3.E4Xa.2.1.1.m1.4.4.2.3.cmml"><mi id="S3.E4Xa.2.1.1.m1.4.4.2.3.2" xref="S3.E4Xa.2.1.1.m1.4.4.2.3.2.cmml">π</mi><mtext id="S3.E4Xa.2.1.1.m1.4.4.2.3.3" xref="S3.E4Xa.2.1.1.m1.4.4.2.3.3a.cmml">ref</mtext></msub><mo lspace="0em" rspace="0em" id="S3.E4Xa.2.1.1.m1.4.4.2.2" xref="S3.E4Xa.2.1.1.m1.4.4.2.2.cmml">​</mo><mrow id="S3.E4Xa.2.1.1.m1.4.4.2.1.1" xref="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.cmml"><mo stretchy="false" id="S3.E4Xa.2.1.1.m1.4.4.2.1.1.2" xref="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.cmml">(</mo><mrow id="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1" xref="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.cmml"><msub id="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.2" xref="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.2.cmml"><mi id="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.2.2" xref="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.2.2.cmml">y</mi><mi id="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.2.3" xref="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.2.3.cmml">l</mi></msub><mo fence="false" id="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.1" xref="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.1.cmml">|</mo><mi id="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.3" xref="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.3.cmml">x</mi></mrow><mo stretchy="false" id="S3.E4Xa.2.1.1.m1.4.4.2.1.1.3" xref="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.cmml">)</mo></mrow></mrow></mfrac></mstyle><mo stretchy="false" id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.3.2.1.2" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.3.1.cmml">)</mo></mrow></mrow></mrow></mrow><mo stretchy="false" id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.3" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.3" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.2.1.cmml">]</mo></mrow><mo lspace="0em" id="S3.E4Xa.2.1.1.m1.7.7.1.2">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E4Xa.2.1.1.m1.7b"><apply id="S3.E4Xa.2.1.1.m1.7.7.1.1.2.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1"><csymbol cd="latexml" id="S3.E4Xa.2.1.1.m1.7.7.1.1.2.1.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.2">delimited-[]</csymbol><apply id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1"><minus id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.2.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1"></minus><apply id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1"><times id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.2.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.2"></times><apply id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.3.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.3"><log id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.3.1.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.3.1"></log><ci id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.3.2.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.3.2">𝜎</ci></apply><apply id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1"><minus id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.1"></minus><apply id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2"><times id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.1"></times><ci id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.2">𝛽</ci><apply id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.3.1.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.2.3.2"><log id="S3.E4Xa.2.1.1.m1.5.5.cmml" xref="S3.E4Xa.2.1.1.m1.5.5"></log><apply id="S3.E4Xa.2.1.1.m1.2.2.cmml" xref="S3.E4Xa.2.1.1.m1.2.2"><divide id="S3.E4Xa.2.1.1.m1.2.2.3.cmml" xref="S3.E4Xa.2.1.1.m1.2.2"></divide><apply id="S3.E4Xa.2.1.1.m1.1.1.1.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1"><times id="S3.E4Xa.2.1.1.m1.1.1.1.2.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.2"></times><apply id="S3.E4Xa.2.1.1.m1.1.1.1.3.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4Xa.2.1.1.m1.1.1.1.3.1.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.3">subscript</csymbol><ci id="S3.E4Xa.2.1.1.m1.1.1.1.3.2.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.3.2">𝜋</ci><ci id="S3.E4Xa.2.1.1.m1.1.1.1.3.3.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.3.3">𝜃</ci></apply><apply id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1"><csymbol cd="latexml" id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.1">conditional</csymbol><apply id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.2.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.2.1.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.2.2.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.2.2">𝑦</ci><ci id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.2.3.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.2.3">𝑤</ci></apply><ci id="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3.cmml" xref="S3.E4Xa.2.1.1.m1.1.1.1.1.1.1.3">𝑥</ci></apply></apply><apply id="S3.E4Xa.2.1.1.m1.2.2.2.cmml" xref="S3.E4Xa.2.1.1.m1.2.2.2"><times id="S3.E4Xa.2.1.1.m1.2.2.2.2.cmml" xref="S3.E4Xa.2.1.1.m1.2.2.2.2"></times><apply id="S3.E4Xa.2.1.1.m1.2.2.2.3.cmml" xref="S3.E4Xa.2.1.1.m1.2.2.2.3"><csymbol cd="ambiguous" id="S3.E4Xa.2.1.1.m1.2.2.2.3.1.cmml" xref="S3.E4Xa.2.1.1.m1.2.2.2.3">subscript</csymbol><ci id="S3.E4Xa.2.1.1.m1.2.2.2.3.2.cmml" xref="S3.E4Xa.2.1.1.m1.2.2.2.3.2">𝜋</ci><ci id="S3.E4Xa.2.1.1.m1.2.2.2.3.3a.cmml" xref="S3.E4Xa.2.1.1.m1.2.2.2.3.3"><mtext mathsize="70%" id="S3.E4Xa.2.1.1.m1.2.2.2.3.3.cmml" xref="S3.E4Xa.2.1.1.m1.2.2.2.3.3">ref</mtext></ci></apply><apply id="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.cmml" xref="S3.E4Xa.2.1.1.m1.2.2.2.1.1"><csymbol cd="latexml" id="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.1.cmml" xref="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.1">conditional</csymbol><apply id="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.2.cmml" xref="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.2"><csymbol cd="ambiguous" id="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.2.1.cmml" xref="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.2">subscript</csymbol><ci id="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.2.2.cmml" xref="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.2.2">𝑦</ci><ci id="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.2.3.cmml" xref="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.2.3">𝑤</ci></apply><ci id="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.3.cmml" xref="S3.E4Xa.2.1.1.m1.2.2.2.1.1.1.3">𝑥</ci></apply></apply></apply></apply></apply><apply id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3"><times id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.1"></times><ci id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.2">𝛽</ci><apply id="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.3.1.cmml" xref="S3.E4Xa.2.1.1.m1.7.7.1.1.1.1.1.1.1.1.3.3.2"><log id="S3.E4Xa.2.1.1.m1.6.6.cmml" xref="S3.E4Xa.2.1.1.m1.6.6"></log><apply id="S3.E4Xa.2.1.1.m1.4.4.cmml" xref="S3.E4Xa.2.1.1.m1.4.4"><divide id="S3.E4Xa.2.1.1.m1.4.4.3.cmml" xref="S3.E4Xa.2.1.1.m1.4.4"></divide><apply id="S3.E4Xa.2.1.1.m1.3.3.1.cmml" xref="S3.E4Xa.2.1.1.m1.3.3.1"><times id="S3.E4Xa.2.1.1.m1.3.3.1.2.cmml" xref="S3.E4Xa.2.1.1.m1.3.3.1.2"></times><apply id="S3.E4Xa.2.1.1.m1.3.3.1.3.cmml" xref="S3.E4Xa.2.1.1.m1.3.3.1.3"><csymbol cd="ambiguous" id="S3.E4Xa.2.1.1.m1.3.3.1.3.1.cmml" xref="S3.E4Xa.2.1.1.m1.3.3.1.3">subscript</csymbol><ci id="S3.E4Xa.2.1.1.m1.3.3.1.3.2.cmml" xref="S3.E4Xa.2.1.1.m1.3.3.1.3.2">𝜋</ci><ci id="S3.E4Xa.2.1.1.m1.3.3.1.3.3.cmml" xref="S3.E4Xa.2.1.1.m1.3.3.1.3.3">𝜃</ci></apply><apply id="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.cmml" xref="S3.E4Xa.2.1.1.m1.3.3.1.1.1"><csymbol cd="latexml" id="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.1.cmml" xref="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.1">conditional</csymbol><apply id="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.2.cmml" xref="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.2.1.cmml" xref="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.2">subscript</csymbol><ci id="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.2.2.cmml" xref="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.2.2">𝑦</ci><ci id="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.2.3.cmml" xref="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.2.3">𝑙</ci></apply><ci id="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.3.cmml" xref="S3.E4Xa.2.1.1.m1.3.3.1.1.1.1.3">𝑥</ci></apply></apply><apply id="S3.E4Xa.2.1.1.m1.4.4.2.cmml" xref="S3.E4Xa.2.1.1.m1.4.4.2"><times id="S3.E4Xa.2.1.1.m1.4.4.2.2.cmml" xref="S3.E4Xa.2.1.1.m1.4.4.2.2"></times><apply id="S3.E4Xa.2.1.1.m1.4.4.2.3.cmml" xref="S3.E4Xa.2.1.1.m1.4.4.2.3"><csymbol cd="ambiguous" id="S3.E4Xa.2.1.1.m1.4.4.2.3.1.cmml" xref="S3.E4Xa.2.1.1.m1.4.4.2.3">subscript</csymbol><ci id="S3.E4Xa.2.1.1.m1.4.4.2.3.2.cmml" xref="S3.E4Xa.2.1.1.m1.4.4.2.3.2">𝜋</ci><ci id="S3.E4Xa.2.1.1.m1.4.4.2.3.3a.cmml" xref="S3.E4Xa.2.1.1.m1.4.4.2.3.3"><mtext mathsize="70%" id="S3.E4Xa.2.1.1.m1.4.4.2.3.3.cmml" xref="S3.E4Xa.2.1.1.m1.4.4.2.3.3">ref</mtext></ci></apply><apply id="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.cmml" xref="S3.E4Xa.2.1.1.m1.4.4.2.1.1"><csymbol cd="latexml" id="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.1.cmml" xref="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.1">conditional</csymbol><apply id="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.2.cmml" xref="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.2"><csymbol cd="ambiguous" id="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.2.1.cmml" xref="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.2">subscript</csymbol><ci id="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.2.2.cmml" xref="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.2.2">𝑦</ci><ci id="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.2.3.cmml" xref="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.2.3">𝑙</ci></apply><ci id="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.3.cmml" xref="S3.E4Xa.2.1.1.m1.4.4.2.1.1.1.3">𝑥</ci></apply></apply></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E4Xa.2.1.1.m1.7c">\displaystyle[-\log\sigma(\beta\log(\frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)})-\beta\log(\frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}))].</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr>
</tbody>
</table>
<p id="S3.SS6.p4.6" class="ltx_p">In practice, we use LoRA with a rank of <math id="S3.SS6.p4.3.m1.1" class="ltx_Math" alttext="256" display="inline"><semantics id="S3.SS6.p4.3.m1.1a"><mn id="S3.SS6.p4.3.m1.1.1" xref="S3.SS6.p4.3.m1.1.1.cmml">256</mn><annotation-xml encoding="MathML-Content" id="S3.SS6.p4.3.m1.1b"><cn type="integer" id="S3.SS6.p4.3.m1.1.1.cmml" xref="S3.SS6.p4.3.m1.1.1">256</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p4.3.m1.1c">256</annotation></semantics></math> to get the DPO model <math id="S3.SS6.p4.4.m2.1" class="ltx_Math" alttext="\pi_{\theta}" display="inline"><semantics id="S3.SS6.p4.4.m2.1a"><msub id="S3.SS6.p4.4.m2.1.1" xref="S3.SS6.p4.4.m2.1.1.cmml"><mi id="S3.SS6.p4.4.m2.1.1.2" xref="S3.SS6.p4.4.m2.1.1.2.cmml">π</mi><mi id="S3.SS6.p4.4.m2.1.1.3" xref="S3.SS6.p4.4.m2.1.1.3.cmml">θ</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS6.p4.4.m2.1b"><apply id="S3.SS6.p4.4.m2.1.1.cmml" xref="S3.SS6.p4.4.m2.1.1"><csymbol cd="ambiguous" id="S3.SS6.p4.4.m2.1.1.1.cmml" xref="S3.SS6.p4.4.m2.1.1">subscript</csymbol><ci id="S3.SS6.p4.4.m2.1.1.2.cmml" xref="S3.SS6.p4.4.m2.1.1.2">𝜋</ci><ci id="S3.SS6.p4.4.m2.1.1.3.cmml" xref="S3.SS6.p4.4.m2.1.1.3">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p4.4.m2.1c">\pi_{\theta}</annotation></semantics></math>. We observe that our model tends to prioritize minimizing the likelihood of dis-preferred responses <math id="S3.SS6.p4.5.m3.1" class="ltx_Math" alttext="y_{l}" display="inline"><semantics id="S3.SS6.p4.5.m3.1a"><msub id="S3.SS6.p4.5.m3.1.1" xref="S3.SS6.p4.5.m3.1.1.cmml"><mi id="S3.SS6.p4.5.m3.1.1.2" xref="S3.SS6.p4.5.m3.1.1.2.cmml">y</mi><mi id="S3.SS6.p4.5.m3.1.1.3" xref="S3.SS6.p4.5.m3.1.1.3.cmml">l</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS6.p4.5.m3.1b"><apply id="S3.SS6.p4.5.m3.1.1.cmml" xref="S3.SS6.p4.5.m3.1.1"><csymbol cd="ambiguous" id="S3.SS6.p4.5.m3.1.1.1.cmml" xref="S3.SS6.p4.5.m3.1.1">subscript</csymbol><ci id="S3.SS6.p4.5.m3.1.1.2.cmml" xref="S3.SS6.p4.5.m3.1.1.2">𝑦</ci><ci id="S3.SS6.p4.5.m3.1.1.3.cmml" xref="S3.SS6.p4.5.m3.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p4.5.m3.1c">y_{l}</annotation></semantics></math> over maximizing the likelihood of preferred responses <math id="S3.SS6.p4.6.m4.1" class="ltx_Math" alttext="y_{w}" display="inline"><semantics id="S3.SS6.p4.6.m4.1a"><msub id="S3.SS6.p4.6.m4.1.1" xref="S3.SS6.p4.6.m4.1.1.cmml"><mi id="S3.SS6.p4.6.m4.1.1.2" xref="S3.SS6.p4.6.m4.1.1.2.cmml">y</mi><mi id="S3.SS6.p4.6.m4.1.1.3" xref="S3.SS6.p4.6.m4.1.1.3.cmml">w</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS6.p4.6.m4.1b"><apply id="S3.SS6.p4.6.m4.1.1.cmml" xref="S3.SS6.p4.6.m4.1.1"><csymbol cd="ambiguous" id="S3.SS6.p4.6.m4.1.1.1.cmml" xref="S3.SS6.p4.6.m4.1.1">subscript</csymbol><ci id="S3.SS6.p4.6.m4.1.1.2.cmml" xref="S3.SS6.p4.6.m4.1.1.2">𝑦</ci><ci id="S3.SS6.p4.6.m4.1.1.3.cmml" xref="S3.SS6.p4.6.m4.1.1.3">𝑤</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS6.p4.6.m4.1c">y_{w}</annotation></semantics></math> to avoid generating inappropriate or low-quality content.</p>
</div>
<div id="S3.SS6.p5" class="ltx_para">
<p id="S3.SS6.p5.1" class="ltx_p">In summary, our scalable pipeline consists of three primary components. First, we address the challenge of limited instruction tuning data by re-writing original prompts into augmented prompts. Next, we generate diverse responses using different random seeds, enabling the exploration of various creative possibilities. Finally, we apply the DPO algorithm to the chosen and rejected responses to refine our model’s performance. Through our pipeline, our model is capable of generating high-quality articles.</p>
</div>
<figure id="S3.T5" class="ltx_table">
<table id="S3.T5.2" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S3.T5.2.1.1" class="ltx_tr">
<th id="S3.T5.2.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_r ltx_border_tt"></th>
<td id="S3.T5.2.1.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="S3.T5.2.1.1.2.1" class="ltx_text" style="font-size:80%;">Block-Match</span></td>
<td id="S3.T5.2.1.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="S3.T5.2.1.1.3.1" class="ltx_text" style="font-size:80%;">Text</span></td>
<td id="S3.T5.2.1.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="S3.T5.2.1.1.4.1" class="ltx_text" style="font-size:80%;">Position</span></td>
<td id="S3.T5.2.1.1.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="S3.T5.2.1.1.5.1" class="ltx_text" style="font-size:80%;">Color</span></td>
<td id="S3.T5.2.1.1.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_tt"><span id="S3.T5.2.1.1.6.1" class="ltx_text" style="font-size:80%;">CLIP</span></td>
<td id="S3.T5.2.1.1.7" class="ltx_td ltx_align_center ltx_border_tt"><span id="S3.T5.2.1.1.7.1" class="ltx_text" style="font-size:80%;">Average</span></td>
</tr>
<tr id="S3.T5.2.2.2" class="ltx_tr">
<th id="S3.T5.2.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" colspan="7"><span id="S3.T5.2.2.2.1.1" class="ltx_text ltx_font_italic" style="font-size:80%;">Closed-source API</span></th>
</tr>
<tr id="S3.T5.2.3.3" class="ltx_tr">
<th id="S3.T5.2.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">
<span id="S3.T5.2.3.3.1.1" class="ltx_text" style="font-size:80%;">GPT-4V </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T5.2.3.3.1.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib112" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">112</span></a><span id="S3.T5.2.3.3.1.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</th>
<td id="S3.T5.2.3.3.2" class="ltx_td ltx_align_center"><span id="S3.T5.2.3.3.2.1" class="ltx_text ltx_font_bold" style="font-size:80%;">85.8</span></td>
<td id="S3.T5.2.3.3.3" class="ltx_td ltx_align_center"><span id="S3.T5.2.3.3.3.1" class="ltx_text ltx_font_bold" style="font-size:80%;">97.4</span></td>
<td id="S3.T5.2.3.3.4" class="ltx_td ltx_align_center"><span id="S3.T5.2.3.3.4.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">80.5</span></td>
<td id="S3.T5.2.3.3.5" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T5.2.3.3.5.1" class="ltx_text" style="font-size:80%;">73.3</span></td>
<td id="S3.T5.2.3.3.6" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T5.2.3.3.6.1" class="ltx_text ltx_font_bold" style="font-size:80%;">86.9</span></td>
<td id="S3.T5.2.3.3.7" class="ltx_td ltx_align_center"><span id="S3.T5.2.3.3.7.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">84.8</span></td>
</tr>
<tr id="S3.T5.2.4.4" class="ltx_tr">
<th id="S3.T5.2.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">
<span id="S3.T5.2.4.4.1.1" class="ltx_text" style="font-size:80%;">Gemini-Pro </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T5.2.4.4.1.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib142" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">142</span></a><span id="S3.T5.2.4.4.1.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</th>
<td id="S3.T5.2.4.4.2" class="ltx_td ltx_align_center"><span id="S3.T5.2.4.4.2.1" class="ltx_text" style="font-size:80%;">80.2</span></td>
<td id="S3.T5.2.4.4.3" class="ltx_td ltx_align_center"><span id="S3.T5.2.4.4.3.1" class="ltx_text" style="font-size:80%;">94.6</span></td>
<td id="S3.T5.2.4.4.4" class="ltx_td ltx_align_center"><span id="S3.T5.2.4.4.4.1" class="ltx_text" style="font-size:80%;">72.3</span></td>
<td id="S3.T5.2.4.4.5" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T5.2.4.4.5.1" class="ltx_text" style="font-size:80%;">66.2</span></td>
<td id="S3.T5.2.4.4.6" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T5.2.4.4.6.1" class="ltx_text" style="font-size:80%;">83.9</span></td>
<td id="S3.T5.2.4.4.7" class="ltx_td ltx_align_center"><span id="S3.T5.2.4.4.7.1" class="ltx_text" style="font-size:80%;">79.4</span></td>
</tr>
<tr id="S3.T5.2.5.5" class="ltx_tr">
<th id="S3.T5.2.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" colspan="7"><span id="S3.T5.2.5.5.1.1" class="ltx_text" style="font-size:80%;">Open-source</span></th>
</tr>
<tr id="S3.T5.2.6.6" class="ltx_tr">
<th id="S3.T5.2.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">
<span id="S3.T5.2.6.6.1.1" class="ltx_text" style="font-size:80%;">WebSight VLM-8B </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T5.2.6.6.1.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib64" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">64</span></a><span id="S3.T5.2.6.6.1.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</th>
<td id="S3.T5.2.6.6.2" class="ltx_td ltx_align_center"><span id="S3.T5.2.6.6.2.1" class="ltx_text" style="font-size:80%;">55.9</span></td>
<td id="S3.T5.2.6.6.3" class="ltx_td ltx_align_center"><span id="S3.T5.2.6.6.3.1" class="ltx_text" style="font-size:80%;">86.6</span></td>
<td id="S3.T5.2.6.6.4" class="ltx_td ltx_align_center"><span id="S3.T5.2.6.6.4.1" class="ltx_text" style="font-size:80%;">77.3</span></td>
<td id="S3.T5.2.6.6.5" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T5.2.6.6.5.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">79.4</span></td>
<td id="S3.T5.2.6.6.6" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T5.2.6.6.6.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">86.5</span></td>
<td id="S3.T5.2.6.6.7" class="ltx_td ltx_align_center"><span id="S3.T5.2.6.6.7.1" class="ltx_text" style="font-size:80%;">77.1</span></td>
</tr>
<tr id="S3.T5.2.7.7" class="ltx_tr">
<th id="S3.T5.2.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">
<span id="S3.T5.2.7.7.1.1" class="ltx_text" style="font-size:80%;">CogAgent-Chat-18B </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T5.2.7.7.1.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib48" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">48</span></a><span id="S3.T5.2.7.7.1.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</th>
<td id="S3.T5.2.7.7.2" class="ltx_td ltx_align_center"><span id="S3.T5.2.7.7.2.1" class="ltx_text" style="font-size:80%;">7.1</span></td>
<td id="S3.T5.2.7.7.3" class="ltx_td ltx_align_center"><span id="S3.T5.2.7.7.3.1" class="ltx_text" style="font-size:80%;">18.1</span></td>
<td id="S3.T5.2.7.7.4" class="ltx_td ltx_align_center"><span id="S3.T5.2.7.7.4.1" class="ltx_text" style="font-size:80%;">13.3</span></td>
<td id="S3.T5.2.7.7.5" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T5.2.7.7.5.1" class="ltx_text" style="font-size:80%;">13.0</span></td>
<td id="S3.T5.2.7.7.6" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T5.2.7.7.6.1" class="ltx_text" style="font-size:80%;">75.5</span></td>
<td id="S3.T5.2.7.7.7" class="ltx_td ltx_align_center"><span id="S3.T5.2.7.7.7.1" class="ltx_text" style="font-size:80%;">25.4</span></td>
</tr>
<tr id="S3.T5.2.8.8" class="ltx_tr">
<th id="S3.T5.2.8.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">
<span id="S3.T5.2.8.8.1.1" class="ltx_text" style="font-size:80%;">Design2Code-18B </span><cite class="ltx_cite ltx_citemacro_cite"><span id="S3.T5.2.8.8.1.2.1" class="ltx_text" style="font-size:80%;">[</span><a href="#bib.bib131" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">131</span></a><span id="S3.T5.2.8.8.1.3.2" class="ltx_text" style="font-size:80%;">]</span></cite>
</th>
<td id="S3.T5.2.8.8.2" class="ltx_td ltx_align_center"><span id="S3.T5.2.8.8.2.1" class="ltx_text" style="font-size:80%;">78.5</span></td>
<td id="S3.T5.2.8.8.3" class="ltx_td ltx_align_center"><span id="S3.T5.2.8.8.3.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;">96.4</span></td>
<td id="S3.T5.2.8.8.4" class="ltx_td ltx_align_center"><span id="S3.T5.2.8.8.4.1" class="ltx_text" style="font-size:80%;">74.3</span></td>
<td id="S3.T5.2.8.8.5" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T5.2.8.8.5.1" class="ltx_text" style="font-size:80%;">67.0</span></td>
<td id="S3.T5.2.8.8.6" class="ltx_td ltx_align_center ltx_border_r"><span id="S3.T5.2.8.8.6.1" class="ltx_text" style="font-size:80%;">85.8</span></td>
<td id="S3.T5.2.8.8.7" class="ltx_td ltx_align_center"><span id="S3.T5.2.8.8.7.1" class="ltx_text" style="font-size:80%;">80.4</span></td>
</tr>
<tr id="S3.T5.2.9.9" class="ltx_tr" style="background-color:#F2F3F5;">
<th id="S3.T5.2.9.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r ltx_border_t"><span id="S3.T5.2.9.9.1.1" class="ltx_text" style="font-size:80%;background-color:#F2F3F5;">IXC-2.5-7B</span></th>
<td id="S3.T5.2.9.9.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S3.T5.2.9.9.2.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;background-color:#F2F3F5;">81.9</span></td>
<td id="S3.T5.2.9.9.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S3.T5.2.9.9.3.1" class="ltx_text" style="font-size:80%;background-color:#F2F3F5;">95.6</span></td>
<td id="S3.T5.2.9.9.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S3.T5.2.9.9.4.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#F2F3F5;">80.9</span></td>
<td id="S3.T5.2.9.9.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t"><span id="S3.T5.2.9.9.5.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#F2F3F5;">80.8</span></td>
<td id="S3.T5.2.9.9.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t"><span id="S3.T5.2.9.9.6.1" class="ltx_text ltx_framed ltx_framed_underline" style="font-size:80%;background-color:#F2F3F5;">86.5</span></td>
<td id="S3.T5.2.9.9.7" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S3.T5.2.9.9.7.1" class="ltx_text ltx_font_bold" style="font-size:80%;background-color:#F2F3F5;">85.1</span></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering" style="font-size:80%;"><span class="ltx_tag ltx_tag_table"><span id="S3.T5.11.1.1" class="ltx_text" style="font-size:113%;">Table 5</span>: </span><span id="S3.T5.12.2" class="ltx_text" style="font-size:113%;">Screenshot-to-code. Comparison with closed-source APIs and open-source models on <span id="S3.T5.12.2.1" class="ltx_text ltx_font_bold">Design2Code benchmark</span>. The best results are <span id="S3.T5.12.2.2" class="ltx_text ltx_font_bold">bold</span> and the second-best results are <span id="S3.T5.12.2.3" class="ltx_text ltx_framed ltx_framed_underline">underlined</span>.</span></figcaption>
</figure>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experiments</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">In this section, we validate the benchmark performance of our InternLM-XComposer-2.5 (IXC-2.5) after supervised fine-tuning.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>LVLM Benchmark results.</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.3" class="ltx_p">In Table <a href="#S3.T3" title="Table 3 ‣ 3.6 Article Composing ‣ 3 Method ‣ InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> and Table <a href="#S3.T4" title="Table 4 ‣ 3.6 Article Composing ‣ 3 Method ‣ InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, we compare our IXC-2.5 on a list of benchmarks with both closed-source APIs and SOTA open-source LVLMs (with comparable model size). Here we report video understanding results on MVBench <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib71" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">71</span></a>]</cite>, MLVU <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib181" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">181</span></a>]</cite>, MME-Video <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib42" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">42</span></a>]</cite>, MMBench-Video <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib38" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">38</span></a>]</cite>, TempCompass <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib88" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">88</span></a>]</cite>.
For Structural High-resolution understanding, we report results on DocVQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib107" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">107</span></a>]</cite>, ChartQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib106" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">106</span></a>]</cite>, InfographicVQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib108" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">108</span></a>]</cite>, TextVQA <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib133" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">133</span></a>]</cite>, OCRBench <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib89" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">89</span></a>]</cite>, DeepForm <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib139" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">139</span></a>]</cite>, WikiTableQuestion (WTQ) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib117" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">117</span></a>]</cite>, Visual MRC <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib140" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">140</span></a>]</cite>, and TabFact <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib20" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">20</span></a>]</cite>.
For general visual question answering, we report results on MMStar <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">18</span></a>]</cite>, RealWorldQA<cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib155" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">155</span></a>]</cite>,
MathVista <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib100" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">100</span></a>]</cite>, MMMU <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib166" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">166</span></a>]</cite>, AI2D <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib61" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">61</span></a>]</cite>, MME <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib40" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">40</span></a>]</cite>, MMBench (MMB)  <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib87" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">87</span></a>]</cite>, MMBench-Chinese (MMB<sup id="S4.SS1.p1.3.1" class="ltx_sup"><span id="S4.SS1.p1.3.1.1" class="ltx_text ltx_font_italic">CN</span></sup>) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib87" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">87</span></a>]</cite>, MMBench-v1.1 (MMB<sup id="S4.SS1.p1.3.2" class="ltx_sup"><span id="S4.SS1.p1.3.2.1" class="ltx_text ltx_font_italic">v1.1</span></sup>) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib87" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">87</span></a>]</cite>, SEED-Bench Image Part (SEED<sup id="S4.SS1.p1.3.3" class="ltx_sup"><span id="S4.SS1.p1.3.3.1" class="ltx_text ltx_font_italic">I</span></sup>)<cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib66" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">66</span></a>]</cite>, MM-Vet <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib164" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">164</span></a>]</cite>, HallusionBench (HallB) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">44</span></a>]</cite>. For Multi-True Multi-Image dialogue, we evaluate IXC-2.5 on MMDU <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib92" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">92</span></a>]</cite> benchmark. For webpage crafting, we report a subtask screenshot-to-code <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib131" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">131</span></a>]</cite> since benchmarks for others are not available in the community.</p>
</div>
<div id="S4.SS1.p2" class="ltx_para">
<p id="S4.SS1.p2.1" class="ltx_p">The evaluation is mainly conducted on the OpenCompass VLMEvalKit <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib30" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">30</span></a>]</cite> for the unified reproduction of the results.</p>
</div>
<div id="S4.SS1.p3" class="ltx_para ltx_noindent">
<p id="S4.SS1.p3.2" class="ltx_p"><span id="S4.SS1.p3.2.1" class="ltx_text ltx_font_bold">Comparison on Video Understanding Benchmarks.</span>
As demonstrated in Table <a href="#S3.T3" title="Table 3 ‣ 3.6 Article Composing ‣ 3 Method ‣ InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, IXC-2.5 exhibits competitive performance on fine-grained video understanding tasks, outperforming open-source models on 4 of the 5 benchmarks and being on par with Closed-Source APIs. For example, IXC-2.5 reaches 69.1 on the MVBench, <math id="S4.SS1.p3.1.m1.1" class="ltx_Math" alttext="+8.7\%" display="inline"><semantics id="S4.SS1.p3.1.m1.1a"><mrow id="S4.SS1.p3.1.m1.1.1" xref="S4.SS1.p3.1.m1.1.1.cmml"><mo id="S4.SS1.p3.1.m1.1.1a" xref="S4.SS1.p3.1.m1.1.1.cmml">+</mo><mrow id="S4.SS1.p3.1.m1.1.1.2" xref="S4.SS1.p3.1.m1.1.1.2.cmml"><mn id="S4.SS1.p3.1.m1.1.1.2.2" xref="S4.SS1.p3.1.m1.1.1.2.2.cmml">8.7</mn><mo id="S4.SS1.p3.1.m1.1.1.2.1" xref="S4.SS1.p3.1.m1.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p3.1.m1.1b"><apply id="S4.SS1.p3.1.m1.1.1.cmml" xref="S4.SS1.p3.1.m1.1.1"><plus id="S4.SS1.p3.1.m1.1.1.1.cmml" xref="S4.SS1.p3.1.m1.1.1"></plus><apply id="S4.SS1.p3.1.m1.1.1.2.cmml" xref="S4.SS1.p3.1.m1.1.1.2"><csymbol cd="latexml" id="S4.SS1.p3.1.m1.1.1.2.1.cmml" xref="S4.SS1.p3.1.m1.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS1.p3.1.m1.1.1.2.2.cmml" xref="S4.SS1.p3.1.m1.1.1.2.2">8.7</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p3.1.m1.1c">+8.7\%</annotation></semantics></math> higher than the previous SOTA method VideoChat2-7B and outperforms GPT-4V with <math id="S4.SS1.p3.2.m2.1" class="ltx_Math" alttext="+25.6\%" display="inline"><semantics id="S4.SS1.p3.2.m2.1a"><mrow id="S4.SS1.p3.2.m2.1.1" xref="S4.SS1.p3.2.m2.1.1.cmml"><mo id="S4.SS1.p3.2.m2.1.1a" xref="S4.SS1.p3.2.m2.1.1.cmml">+</mo><mrow id="S4.SS1.p3.2.m2.1.1.2" xref="S4.SS1.p3.2.m2.1.1.2.cmml"><mn id="S4.SS1.p3.2.m2.1.1.2.2" xref="S4.SS1.p3.2.m2.1.1.2.2.cmml">25.6</mn><mo id="S4.SS1.p3.2.m2.1.1.2.1" xref="S4.SS1.p3.2.m2.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p3.2.m2.1b"><apply id="S4.SS1.p3.2.m2.1.1.cmml" xref="S4.SS1.p3.2.m2.1.1"><plus id="S4.SS1.p3.2.m2.1.1.1.cmml" xref="S4.SS1.p3.2.m2.1.1"></plus><apply id="S4.SS1.p3.2.m2.1.1.2.cmml" xref="S4.SS1.p3.2.m2.1.1.2"><csymbol cd="latexml" id="S4.SS1.p3.2.m2.1.1.2.1.cmml" xref="S4.SS1.p3.2.m2.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS1.p3.2.m2.1.1.2.2.cmml" xref="S4.SS1.p3.2.m2.1.1.2.2">25.6</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p3.2.m2.1c">+25.6\%</annotation></semantics></math>. For the recent challenging MMBench-Video, IXC-2.5 reaches the SOTA performance on open-source models and performs close to Gemini-Pro.</p>
</div>
<div id="S4.SS1.p4" class="ltx_para ltx_noindent">
<p id="S4.SS1.p4.5" class="ltx_p"><span id="S4.SS1.p4.5.1" class="ltx_text ltx_font_bold">Comparison on Structural High-resolution Benchmarks.</span>
Benefiting from the unified image partition strategy, IXC-2.5 could handle diverse kinds of images. Table <a href="#S3.T3" title="Table 3 ‣ 3.6 Article Composing ‣ 3 Method ‣ InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> reports its performance on several structural high-resolution benchmarks. IXC-2.5 with only 7B parameters performs on par with the current large open-source LVLMs and close-source APIs. For example, IXC-2.5 gets <math id="S4.SS1.p4.1.m1.1" class="ltx_Math" alttext="90.9\%" display="inline"><semantics id="S4.SS1.p4.1.m1.1a"><mrow id="S4.SS1.p4.1.m1.1.1" xref="S4.SS1.p4.1.m1.1.1.cmml"><mn id="S4.SS1.p4.1.m1.1.1.2" xref="S4.SS1.p4.1.m1.1.1.2.cmml">90.9</mn><mo id="S4.SS1.p4.1.m1.1.1.1" xref="S4.SS1.p4.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p4.1.m1.1b"><apply id="S4.SS1.p4.1.m1.1.1.cmml" xref="S4.SS1.p4.1.m1.1.1"><csymbol cd="latexml" id="S4.SS1.p4.1.m1.1.1.1.cmml" xref="S4.SS1.p4.1.m1.1.1.1">percent</csymbol><cn type="float" id="S4.SS1.p4.1.m1.1.1.2.cmml" xref="S4.SS1.p4.1.m1.1.1.2">90.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p4.1.m1.1c">90.9\%</annotation></semantics></math> on the DocVQA test set, the same as InternVL-1.5 which has nearly <math id="S4.SS1.p4.2.m2.1" class="ltx_math_unparsed" alttext="4\times" display="inline"><semantics id="S4.SS1.p4.2.m2.1a"><mrow id="S4.SS1.p4.2.m2.1b"><mn id="S4.SS1.p4.2.m2.1.1">4</mn><mo lspace="0.222em" id="S4.SS1.p4.2.m2.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S4.SS1.p4.2.m2.1c">4\times</annotation></semantics></math> parameters. For the highly structured form and table understanding tasks, IXC-2.5 outperforms DocOwl 1.5-8B with <math id="S4.SS1.p4.3.m3.1" class="ltx_Math" alttext="+13.0\%" display="inline"><semantics id="S4.SS1.p4.3.m3.1a"><mrow id="S4.SS1.p4.3.m3.1.1" xref="S4.SS1.p4.3.m3.1.1.cmml"><mo id="S4.SS1.p4.3.m3.1.1a" xref="S4.SS1.p4.3.m3.1.1.cmml">+</mo><mrow id="S4.SS1.p4.3.m3.1.1.2" xref="S4.SS1.p4.3.m3.1.1.2.cmml"><mn id="S4.SS1.p4.3.m3.1.1.2.2" xref="S4.SS1.p4.3.m3.1.1.2.2.cmml">13.0</mn><mo id="S4.SS1.p4.3.m3.1.1.2.1" xref="S4.SS1.p4.3.m3.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p4.3.m3.1b"><apply id="S4.SS1.p4.3.m3.1.1.cmml" xref="S4.SS1.p4.3.m3.1.1"><plus id="S4.SS1.p4.3.m3.1.1.1.cmml" xref="S4.SS1.p4.3.m3.1.1"></plus><apply id="S4.SS1.p4.3.m3.1.1.2.cmml" xref="S4.SS1.p4.3.m3.1.1.2"><csymbol cd="latexml" id="S4.SS1.p4.3.m3.1.1.2.1.cmml" xref="S4.SS1.p4.3.m3.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS1.p4.3.m3.1.1.2.2.cmml" xref="S4.SS1.p4.3.m3.1.1.2.2">13.0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p4.3.m3.1c">+13.0\%</annotation></semantics></math>, <math id="S4.SS1.p4.4.m4.1" class="ltx_Math" alttext="+2.4\%" display="inline"><semantics id="S4.SS1.p4.4.m4.1a"><mrow id="S4.SS1.p4.4.m4.1.1" xref="S4.SS1.p4.4.m4.1.1.cmml"><mo id="S4.SS1.p4.4.m4.1.1a" xref="S4.SS1.p4.4.m4.1.1.cmml">+</mo><mrow id="S4.SS1.p4.4.m4.1.1.2" xref="S4.SS1.p4.4.m4.1.1.2.cmml"><mn id="S4.SS1.p4.4.m4.1.1.2.2" xref="S4.SS1.p4.4.m4.1.1.2.2.cmml">2.4</mn><mo id="S4.SS1.p4.4.m4.1.1.2.1" xref="S4.SS1.p4.4.m4.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p4.4.m4.1b"><apply id="S4.SS1.p4.4.m4.1.1.cmml" xref="S4.SS1.p4.4.m4.1.1"><plus id="S4.SS1.p4.4.m4.1.1.1.cmml" xref="S4.SS1.p4.4.m4.1.1"></plus><apply id="S4.SS1.p4.4.m4.1.1.2.cmml" xref="S4.SS1.p4.4.m4.1.1.2"><csymbol cd="latexml" id="S4.SS1.p4.4.m4.1.1.2.1.cmml" xref="S4.SS1.p4.4.m4.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS1.p4.4.m4.1.1.2.2.cmml" xref="S4.SS1.p4.4.m4.1.1.2.2">2.4</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p4.4.m4.1c">+2.4\%</annotation></semantics></math>, <math id="S4.SS1.p4.5.m5.1" class="ltx_Math" alttext="+5.0\%" display="inline"><semantics id="S4.SS1.p4.5.m5.1a"><mrow id="S4.SS1.p4.5.m5.1.1" xref="S4.SS1.p4.5.m5.1.1.cmml"><mo id="S4.SS1.p4.5.m5.1.1a" xref="S4.SS1.p4.5.m5.1.1.cmml">+</mo><mrow id="S4.SS1.p4.5.m5.1.1.2" xref="S4.SS1.p4.5.m5.1.1.2.cmml"><mn id="S4.SS1.p4.5.m5.1.1.2.2" xref="S4.SS1.p4.5.m5.1.1.2.2.cmml">5.0</mn><mo id="S4.SS1.p4.5.m5.1.1.2.1" xref="S4.SS1.p4.5.m5.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p4.5.m5.1b"><apply id="S4.SS1.p4.5.m5.1.1.cmml" xref="S4.SS1.p4.5.m5.1.1"><plus id="S4.SS1.p4.5.m5.1.1.1.cmml" xref="S4.SS1.p4.5.m5.1.1"></plus><apply id="S4.SS1.p4.5.m5.1.1.2.cmml" xref="S4.SS1.p4.5.m5.1.1.2"><csymbol cd="latexml" id="S4.SS1.p4.5.m5.1.1.2.1.cmml" xref="S4.SS1.p4.5.m5.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS1.p4.5.m5.1.1.2.2.cmml" xref="S4.SS1.p4.5.m5.1.1.2.2">5.0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p4.5.m5.1c">+5.0\%</annotation></semantics></math> on WikiTableQuestion, DeepForm and TableFace respectively.</p>
</div>
<div id="S4.SS1.p5" class="ltx_para ltx_noindent">
<p id="S4.SS1.p5.1" class="ltx_p"><span id="S4.SS1.p5.1.1" class="ltx_text ltx_font_bold">Comparison on Multi-Image Multi-Turn Benchmarks.</span>
IXC-2.5 is capable of taking multiple images as input and conducting multi-round free-form dialogue based on them. We evaluate it quantitatively on the newly proposed MMDU benchmark <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib92" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">92</span></a>]</cite>. As shown in Table <a href="#S3.T4" title="Table 4 ‣ 3.6 Article Composing ‣ 3 Method ‣ InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, the IXC-2.5 model demonstrates superior performance, outperforming the previous SOTA open-source model by a significant margin of 13.8%. This notable improvement highlights the effectiveness of our approach in advancing the capabilities of multi-image and multi-turn understanding.</p>
</div>
<div id="S4.SS1.p6" class="ltx_para ltx_noindent">
<p id="S4.SS1.p6.1" class="ltx_p"><span id="S4.SS1.p6.1.1" class="ltx_text ltx_font_bold">Comparison on General Visual QA Benchmarks.</span>
IXC-2.5 is designed as a general LVLM to handle diverse multi-modal tasks. Here we report its performance on general visual QA benchmarks. As shown in Table <a href="#S3.T4" title="Table 4 ‣ 3.6 Article Composing ‣ 3 Method ‣ InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, the IXC-2.5 shows superb performance on these benchmarks and on par with current large open-source LVLMs and closed-source APIs. For example, IXC-2.5 gets <math id="S4.SS1.p6.1.m1.1" class="ltx_Math" alttext="59.9\%" display="inline"><semantics id="S4.SS1.p6.1.m1.1a"><mrow id="S4.SS1.p6.1.m1.1.1" xref="S4.SS1.p6.1.m1.1.1.cmml"><mn id="S4.SS1.p6.1.m1.1.1.2" xref="S4.SS1.p6.1.m1.1.1.2.cmml">59.9</mn><mo id="S4.SS1.p6.1.m1.1.1.1" xref="S4.SS1.p6.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p6.1.m1.1b"><apply id="S4.SS1.p6.1.m1.1.1.cmml" xref="S4.SS1.p6.1.m1.1.1"><csymbol cd="latexml" id="S4.SS1.p6.1.m1.1.1.1.cmml" xref="S4.SS1.p6.1.m1.1.1.1">percent</csymbol><cn type="float" id="S4.SS1.p6.1.m1.1.1.2.cmml" xref="S4.SS1.p6.1.m1.1.1.2">59.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p6.1.m1.1c">59.9\%</annotation></semantics></math> on the challenging MMStar and outperforms GPT-4V and Gemini-Pro. On the RealWorldQA, IXC-2.5 also performs better than Gemini-Pro and close to GPT-4V.</p>
</div>
<div id="S4.SS1.p7" class="ltx_para ltx_noindent">
<p id="S4.SS1.p7.1" class="ltx_p"><span id="S4.SS1.p7.1.1" class="ltx_text ltx_font_bold">Comparison on Screenshot-to-code Benchmark.</span> Table <a href="#S3.T5" title="Table 5 ‣ 3.6 Article Composing ‣ 3 Method ‣ InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> presents the comparison results on the Design2Code <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib131" title="" class="ltx_ref"><span class="ltx_text" style="font-size:90%;">131</span></a>]</cite> benchmark that assesses the ability to translate visual design into code implementation. Our IXC-2.5 even surpasses the GPT-4v on average performance, which highlights the potential of IXC-2.5 to excel in bridging the gap between visual design and code implementation.</p>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Conclusion</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">We have introduced InternLM-XComposer-2.5 (IXC-2.5), a cutting-edge Large Vision-Language Model (LVLM) boasting long-contextual input and output capabilities that enable advanced features such as ultra-high resolution image understanding, fine-grained video understanding, multi-turn multi-image dialogue, webpage generation, and article composing.
Our comprehensive experiments demonstrate that IXC-2.5 achieves competitive performance, remarkably, with a relatively modest 7B Large Language Model (LLM) backend.</p>
</div>
<div id="S5.p2" class="ltx_para">
<p id="S5.p2.1" class="ltx_p">Our model sets out a promising research direction that can extend to a more contextual multi-modal environment, including long-context video understanding (<em id="S5.p2.1.1" class="ltx_emph ltx_font_italic">e.g</em>.<span id="S5.p2.1.2" class="ltx_text"></span>, long movies) and long-context interaction history, to better assist humans in real-world applications.</p>
</div>
<div id="S5.p3" class="ltx_para ltx_noindent">
<p id="S5.p3.1" class="ltx_p"><span id="S5.p3.1.1" class="ltx_text ltx_font_bold">Acknowledgements:</span> We deeply express our gratitude to Prof. Chao Zhang from Tsinghua University for suggestions about audio models and tools.


<span id="S5.p3.1.2" class="ltx_text" style="font-size:90%;"></span></p>
<div class="ltx_pagination ltx_role_newpage"></div>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography" style="font-size:90%;">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib1.5.5.1" class="ltx_text" style="font-size:90%;">Agrawal et al. [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib1.7.1" class="ltx_text" style="font-size:90%;">
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark
Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib1.8.1" class="ltx_text" style="font-size:90%;">Nocaps: Novel object captioning at scale.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib1.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib1.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the IEEE/CVF International Conference on
Computer Vision (ICCV)</em><span id="bib.bib1.11.3" class="ltx_text" style="font-size:90%;">, 2019.
</span>
</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib2.5.5.1" class="ltx_text" style="font-size:90%;">Alayrac et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib2.7.1" class="ltx_text" style="font-size:90%;">
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr,
Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds,
Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina
Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew
Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo
Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib2.8.1" class="ltx_text" style="font-size:90%;">Flamingo: a visual language model for few-shot learning, 2022.
</span>
</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib3.4.4.1" class="ltx_text" style="font-size:90%;">Anthropic [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib3.6.1" class="ltx_text" style="font-size:90%;">
Anthropic.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib3.7.1" class="ltx_text" style="font-size:90%;">Claude 3 haiku: our fastest model yet, 2024.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib3.8.1" class="ltx_text" style="font-size:90%;">Available at: </span><a target="_blank" href="https://www.anthropic.com/news/claude-3-haiku" title="" class="ltx_ref ltx_url ltx_font_typewriter" style="font-size:90%;">https://www.anthropic.com/news/claude-3-haiku</a><span id="bib.bib3.9.2" class="ltx_text" style="font-size:90%;">.
</span>
</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib4.5.5.1" class="ltx_text" style="font-size:90%;">Antol et al. [2015]</span></span>
<span class="ltx_bibblock"><span id="bib.bib4.7.1" class="ltx_text" style="font-size:90%;">
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra,
C. Lawrence Zitnick, and Devi Parikh.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib4.8.1" class="ltx_text" style="font-size:90%;">VQA: Visual question answering.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib4.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib4.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the IEEE/CVF International Conference on
Computer Vision (ICCV)</em><span id="bib.bib4.11.3" class="ltx_text" style="font-size:90%;">, 2015.
</span>
</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib5.5.5.1" class="ltx_text" style="font-size:90%;">Awadalla et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib5.7.1" class="ltx_text" style="font-size:90%;">
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu,
Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev,
Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig
Schmidt.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib5.8.1" class="ltx_text" style="font-size:90%;">Openflamingo: An open-source framework for training large
autoregressive vision-language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib5.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv.org</em><span id="bib.bib5.10.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib6.5.5.1" class="ltx_text" style="font-size:90%;">Bai et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib6.7.1" class="ltx_text" style="font-size:90%;">
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang
Lin, Chang Zhou, and Jingren Zhou.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib6.8.1" class="ltx_text" style="font-size:90%;">Qwen-VL: A frontier large vision-language model with versatile
abilities.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib6.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv.org</em><span id="bib.bib6.10.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib7.5.5.1" class="ltx_text" style="font-size:90%;">Bai et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib7.7.1" class="ltx_text" style="font-size:90%;">
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion,
Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon,
et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib7.8.1" class="ltx_text" style="font-size:90%;">Constitutional AI: Harmlessness from ai feedback.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib7.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2212.08073</em><span id="bib.bib7.10.2" class="ltx_text" style="font-size:90%;">, 2022.
</span>
</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib8.4.4.1" class="ltx_text" style="font-size:90%;">Baichuan [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib8.6.1" class="ltx_text" style="font-size:90%;">
Baichuan.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib8.7.1" class="ltx_text" style="font-size:90%;">Baichuan 2: Open large-scale language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib8.8.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv.org</em><span id="bib.bib8.9.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib9.5.5.1" class="ltx_text" style="font-size:90%;">Bavishi et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib9.7.1" class="ltx_text" style="font-size:90%;">
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena,
Arushi Somani, and Sağnak Taşırlar.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib9.8.1" class="ltx_text" style="font-size:90%;">Introducing our multimodal models, 2023.
</span>
</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib10.4.4.1" class="ltx_text" style="font-size:90%;">Beltramelli [2018]</span></span>
<span class="ltx_bibblock"><span id="bib.bib10.6.1" class="ltx_text" style="font-size:90%;">
Tony Beltramelli.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib10.7.1" class="ltx_text" style="font-size:90%;">pix2code: Generating code from a graphical user interface screenshot.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib10.8.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib10.9.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the ACM SIGCHI symposium on engineering
interactive computing systems</em><span id="bib.bib10.10.3" class="ltx_text" style="font-size:90%;">, 2018.
</span>
</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib11.5.5.1" class="ltx_text" style="font-size:90%;">Biten et al. [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib11.7.1" class="ltx_text" style="font-size:90%;">
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol,
Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib11.8.1" class="ltx_text" style="font-size:90%;">Scene text visual question answering.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib11.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib11.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">ICCV</em><span id="bib.bib11.11.3" class="ltx_text" style="font-size:90%;">, 2019.
</span>
</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib12.5.5.1" class="ltx_text" style="font-size:90%;">Brown et al. [2020]</span></span>
<span class="ltx_bibblock"><span id="bib.bib12.7.1" class="ltx_text" style="font-size:90%;">
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib12.8.1" class="ltx_text" style="font-size:90%;">Language models are few-shot learners.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib12.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Advances in Neural Information Processing Systems (NeurIPS)</em><span id="bib.bib12.10.2" class="ltx_text" style="font-size:90%;">,
33:1877–1901, 2020.
</span>
</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib13.5.5.1" class="ltx_text" style="font-size:90%;">Cai et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib13.7.1" class="ltx_text" style="font-size:90%;">
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen,
Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei,
Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo,
Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin,
Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining
Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran
Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang
Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin
Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang,
Guoteng Wang, Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang,
Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang
Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu,
Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo
Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue
Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming
Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib13.8.1" class="ltx_text" style="font-size:90%;">Internlm2 technical report.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib13.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2403.17297</em><span id="bib.bib13.10.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib14.5.5.1" class="ltx_text" style="font-size:90%;">Cao et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib14.7.1" class="ltx_text" style="font-size:90%;">
Yuhang Cao, Pan Zhang, Xiaoyi Dong, Dahua Lin, and Jiaqi Wang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib14.8.1" class="ltx_text" style="font-size:90%;">DualFocus: Integrating macro and micro perspectives in multi-modal
large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib14.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2402.14767</em><span id="bib.bib14.10.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib15.5.5.1" class="ltx_text" style="font-size:90%;">Chen et al. [2024a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib15.7.1" class="ltx_text" style="font-size:90%;">
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi
Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib15.8.1" class="ltx_text" style="font-size:90%;">ALLaVA harnessing gpt4v-synthesized data for a lite vision-language
model.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib15.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2402.11684</em><span id="bib.bib15.10.2" class="ltx_text" style="font-size:90%;">, 2024a.
</span>
</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib16.5.5.1" class="ltx_text" style="font-size:90%;">Chen et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib16.7.1" class="ltx_text" style="font-size:90%;">
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib16.8.1" class="ltx_text" style="font-size:90%;">Shikra: Unleashing multimodal llm’s referential dialogue magic.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib16.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv.org</em><span id="bib.bib16.10.2" class="ltx_text" style="font-size:90%;">, 2023a.
</span>
</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib17.5.5.1" class="ltx_text" style="font-size:90%;">Chen et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib17.7.1" class="ltx_text" style="font-size:90%;">
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao,
and Dahua Lin.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib17.8.1" class="ltx_text" style="font-size:90%;">Sharegpt4v: Improving large multi-modal models with better captions.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib17.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2311.12793</em><span id="bib.bib17.10.2" class="ltx_text" style="font-size:90%;">, 2023b.
</span>
</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib18.5.5.1" class="ltx_text" style="font-size:90%;">Chen et al. [2024b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib18.7.1" class="ltx_text" style="font-size:90%;">
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong
Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib18.8.1" class="ltx_text" style="font-size:90%;">Are we on the right way for evaluating large vision-language models?
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib18.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2403.20330</em><span id="bib.bib18.10.2" class="ltx_text" style="font-size:90%;">, 2024b.
</span>
</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib19.5.5.1" class="ltx_text" style="font-size:90%;">Chen et al. [2024c]</span></span>
<span class="ltx_bibblock"><span id="bib.bib19.7.1" class="ltx_text" style="font-size:90%;">
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui
Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib19.8.1" class="ltx_text" style="font-size:90%;">ShareGPT4Video: Improving video understanding and generation with
better captions.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib19.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2406.04325</em><span id="bib.bib19.10.2" class="ltx_text" style="font-size:90%;">, 2024c.
</span>
</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib20.5.5.1" class="ltx_text" style="font-size:90%;">Chen et al. [2020]</span></span>
<span class="ltx_bibblock"><span id="bib.bib20.7.1" class="ltx_text" style="font-size:90%;">
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li,
Xiyou Zhou, and William Yang Wang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib20.8.1" class="ltx_text" style="font-size:90%;">TabFact: A large-scale dataset for table-based fact verification.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib20.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib20.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the International Conference on Learning
Representations (ICLR)</em><span id="bib.bib20.11.3" class="ltx_text" style="font-size:90%;">, 2020.
</span>
</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib21.5.5.1" class="ltx_text" style="font-size:90%;">Chen et al. [2015]</span></span>
<span class="ltx_bibblock"><span id="bib.bib21.7.1" class="ltx_text" style="font-size:90%;">
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr
Dollar, and C. Lawrence Zitnick.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib21.8.1" class="ltx_text" style="font-size:90%;">Microsoft coco captions: Data collection and evaluation server, 2015.
</span>
</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib22.5.5.1" class="ltx_text" style="font-size:90%;">Chen et al. [2023c]</span></span>
<span class="ltx_bibblock"><span id="bib.bib22.7.1" class="ltx_text" style="font-size:90%;">
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo,
Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, Siamak
Shakeri, Mostafa Dehghani, Daniel Salz, Mario Lucic, Michael Tschannen, Arsha
Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang, Ceslee Montgomery, Paulina
Pietrzyk, Marvin Ritter, AJ Piergiovanni, Matthias Minderer, Filip Pavetic,
Austin Waters, Gang Li, Ibrahim Alabdulmohsin, Lucas Beyer, Julien Amelot,
Kenton Lee, Andreas Peter Steiner, Yang Li, Daniel Keysers, Anurag Arnab,
Yuanzhong Xu, Keran Rong, Alexander Kolesnikov, Mojtaba Seyedhosseini, Anelia
Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib22.8.1" class="ltx_text" style="font-size:90%;">Pali-x: On scaling up a multilingual vision and language model,
2023c.
</span>
</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib23.5.5.1" class="ltx_text" style="font-size:90%;">Chen et al. [2023d]</span></span>
<span class="ltx_bibblock"><span id="bib.bib23.7.1" class="ltx_text" style="font-size:90%;">
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul
Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr
Padlewski, Daniel Salz, Xi Xiong, Daniel Vlasic, Filip Pavetic, Keran Rong,
Tianli Yu, Daniel Keysers, Xiaohua Zhai, and Radu Soricut.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib23.8.1" class="ltx_text" style="font-size:90%;">Pali-3 vision language models: Smaller, faster, stronger,
2023d.
</span>
</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib24.5.5.1" class="ltx_text" style="font-size:90%;">Chen et al. [2023e]</span></span>
<span class="ltx_bibblock"><span id="bib.bib24.7.1" class="ltx_text" style="font-size:90%;">
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski,
Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer,
Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari,
Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo,
Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas
Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib24.8.1" class="ltx_text" style="font-size:90%;">Pali: A jointly-scaled multilingual language-image model,
2023e.
</span>
</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib25.5.5.1" class="ltx_text" style="font-size:90%;">Chen et al. [2023f]</span></span>
<span class="ltx_bibblock"><span id="bib.bib25.7.1" class="ltx_text" style="font-size:90%;">
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong,
Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and
Jifeng Dai.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib25.8.1" class="ltx_text" style="font-size:90%;">Internvl: Scaling up vision foundation models and aligning for
generic visual-linguistic tasks.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib25.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2312.14238</em><span id="bib.bib25.10.2" class="ltx_text" style="font-size:90%;">, 2023f.
</span>
</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib26.5.5.1" class="ltx_text" style="font-size:90%;">Chen et al. [2024d]</span></span>
<span class="ltx_bibblock"><span id="bib.bib26.7.1" class="ltx_text" style="font-size:90%;">
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen
Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang
Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, Bin Wang,
Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen,
Xiangchao Yan, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao,
Jifeng Dai, and Wenhai Wang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib26.8.1" class="ltx_text" style="font-size:90%;">How far are we to gpt-4v? closing the gap to commercial multimodal
models with open-source suites, 2024d.
</span>
</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib27.5.5.1" class="ltx_text" style="font-size:90%;">Chiang et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib27.7.1" class="ltx_text" style="font-size:90%;">
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin
Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and
Eric P. Xing.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib27.8.1" class="ltx_text" style="font-size:90%;">Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt
quality, 2023.
</span>
</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib28.5.5.1" class="ltx_text" style="font-size:90%;">Chng et al. [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib28.7.1" class="ltx_text" style="font-size:90%;">
Chee Kheng Chng, Yuliang Liu, Yipeng Sun, Chun Chet Ng, Canjie Luo, Zihan Ni,
ChuanMing Fang, Shuaitao Zhang, Junyu Han, Errui Ding, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib28.8.1" class="ltx_text" style="font-size:90%;">Icdar2019 robust reading challenge on arbitrary-shaped text-rrc-art.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib28.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib28.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">International Conference on Document Analysis and
Recognition (ICDAR)</em><span id="bib.bib28.11.3" class="ltx_text" style="font-size:90%;">, 2019.
</span>
</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib29.5.5.1" class="ltx_text" style="font-size:90%;">Chowdhery et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib29.7.1" class="ltx_text" style="font-size:90%;">
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra,
Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian
Gehrmann, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib29.8.1" class="ltx_text" style="font-size:90%;">Palm: Scaling language modeling with pathways.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib29.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv.org</em><span id="bib.bib29.10.2" class="ltx_text" style="font-size:90%;">, 2022.
</span>
</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib30.4.4.1" class="ltx_text" style="font-size:90%;">Contributors [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib30.6.1" class="ltx_text" style="font-size:90%;">
OpenCompass Contributors.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib30.7.1" class="ltx_text" style="font-size:90%;">Opencompass: A universal evaluation platform for foundation models.
</span>
</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/open-compass/opencompass" title="" class="ltx_ref ltx_url ltx_font_typewriter" style="font-size:90%;">https://github.com/open-compass/opencompass</a><span id="bib.bib30.8.1" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib31.5.5.1" class="ltx_text" style="font-size:90%;">Dai et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib31.7.1" class="ltx_text" style="font-size:90%;">
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao,
Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib31.8.1" class="ltx_text" style="font-size:90%;">Instructblip: Towards general-purpose vision-language models with
instruction tuning, 2023.
</span>
</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib32.5.5.1" class="ltx_text" style="font-size:90%;">Das et al. [2017]</span></span>
<span class="ltx_bibblock"><span id="bib.bib32.7.1" class="ltx_text" style="font-size:90%;">
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav,
José M.F. Moura, Devi Parikh, and Dhruv Batra.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib32.8.1" class="ltx_text" style="font-size:90%;">Visual Dialog.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib32.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib32.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR)</em><span id="bib.bib32.11.3" class="ltx_text" style="font-size:90%;">, 2017.
</span>
</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib33.5.5.1" class="ltx_text" style="font-size:90%;">Dong et al. [2024a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib33.7.1" class="ltx_text" style="font-size:90%;">
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin
Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang
Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He,
Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib33.8.1" class="ltx_text" style="font-size:90%;">Internlm-xcomposer2: Mastering free-form text-image composition and
comprehension in vision-language large model.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib33.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2401.16420</em><span id="bib.bib33.10.2" class="ltx_text" style="font-size:90%;">, 2024a.
</span>
</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib34.5.5.1" class="ltx_text" style="font-size:90%;">Dong et al. [2024b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib34.7.1" class="ltx_text" style="font-size:90%;">
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang,
Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao,
Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui
He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, and Jiaqi Wang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib34.8.1" class="ltx_text" style="font-size:90%;">Internlm-xcomposer2-4khd: A pioneering large vision-language model
handling resolutions from 336 pixels to 4k hd.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib34.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2404.06512</em><span id="bib.bib34.10.2" class="ltx_text" style="font-size:90%;">, 2024b.
</span>
</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib35.5.5.1" class="ltx_text" style="font-size:90%;">Driess et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib35.7.1" class="ltx_text" style="font-size:90%;">
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery,
Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong
Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine,
Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng,
Igor Mordatch, and Pete Florence.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib35.8.1" class="ltx_text" style="font-size:90%;">Palm-e: An embodied multimodal language model.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib35.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib35.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2303.03378</em><span id="bib.bib35.11.3" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib36.5.5.1" class="ltx_text" style="font-size:90%;">Ethayarajh et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib36.7.1" class="ltx_text" style="font-size:90%;">
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib36.8.1" class="ltx_text" style="font-size:90%;">KTO: Model alignment as prospect theoretic optimization.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib36.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2402.01306</em><span id="bib.bib36.10.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib37.4.4.1" class="ltx_text" style="font-size:90%;">Fabian Caba Heilbron and Niebles [2015]</span></span>
<span class="ltx_bibblock"><span id="bib.bib37.6.1" class="ltx_text" style="font-size:90%;">
Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib37.7.1" class="ltx_text" style="font-size:90%;">ActivityNet: A large-scale video benchmark for human activity
understanding.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib37.8.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib37.9.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR)</em><span id="bib.bib37.10.3" class="ltx_text" style="font-size:90%;">, 2015.
</span>
</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib38.5.5.1" class="ltx_text" style="font-size:90%;">Fang et al. [2024a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib38.7.1" class="ltx_text" style="font-size:90%;">
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and
Kai Chen.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib38.8.1" class="ltx_text" style="font-size:90%;">MMBench-Video: A long-form multi-shot benchmark for holistic video
understanding.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib38.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2406.14515</em><span id="bib.bib38.10.2" class="ltx_text" style="font-size:90%;">, 2024a.
</span>
</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib39.5.5.1" class="ltx_text" style="font-size:90%;">Fang et al. [2024b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib39.7.1" class="ltx_text" style="font-size:90%;">
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and
Kai Chen.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib39.8.1" class="ltx_text" style="font-size:90%;">Mmbench-video: A long-form multi-shot benchmark for holistic video
understanding, 2024b.
</span>
</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib40.5.5.1" class="ltx_text" style="font-size:90%;">Fu et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib40.7.1" class="ltx_text" style="font-size:90%;">
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin,
Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong
Ji.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib40.8.1" class="ltx_text" style="font-size:90%;">Mme: A comprehensive evaluation benchmark for multimodal large
language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib40.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2306.13394</em><span id="bib.bib40.10.2" class="ltx_text" style="font-size:90%;">, 2023a.
</span>
</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib41.5.5.1" class="ltx_text" style="font-size:90%;">Fu et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib41.7.1" class="ltx_text" style="font-size:90%;">
Chaoyou Fu, Renrui Zhang, Zihan Wang, Yubo Huang, Zhengye Zhang, Longtian Qiu,
Gaoxiang Ye, Yunhang Shen, Mengdan Zhang, Peixian Chen, Sirui Zhao, Shaohui
Lin, Deqiang Jiang, Di Yin, Peng Gao, Ke Li, Hongsheng Li, and Xing Sun.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib41.8.1" class="ltx_text" style="font-size:90%;">A challenger to gpt-4v? early explorations of gemini in visual
expertise.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib41.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2312.12436</em><span id="bib.bib41.10.2" class="ltx_text" style="font-size:90%;">, 2023b.
</span>
</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib42.5.5.1" class="ltx_text" style="font-size:90%;">Fu et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib42.7.1" class="ltx_text" style="font-size:90%;">
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan
Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib42.8.1" class="ltx_text" style="font-size:90%;">Video-MME: The first-ever comprehensive evaluation benchmark of
multi-modal llms in video analysis.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib42.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2405.21075</em><span id="bib.bib42.10.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib43.5.5.1" class="ltx_text" style="font-size:90%;">GLM et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib43.7.1" class="ltx_text" style="font-size:90%;">
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas,
Guanyu Feng, Hanlin Zhao, Hanyu Lai, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib43.8.1" class="ltx_text" style="font-size:90%;">ChatGLM: A family of large language models from glm-130b to glm-4
all tools.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib43.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2406.12793</em><span id="bib.bib43.10.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib44.5.5.1" class="ltx_text" style="font-size:90%;">Guan et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib44.7.1" class="ltx_text" style="font-size:90%;">
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun
Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi
Zhou.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib44.8.1" class="ltx_text" style="font-size:90%;">Hallusionbench: An advanced diagnostic suite for entangled language
hallucination &amp; visual illusion in large vision-language models, 2023.
</span>
</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib45.5.5.1" class="ltx_text" style="font-size:90%;">He et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib45.7.1" class="ltx_text" style="font-size:90%;">
Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah,
Abhinav Shrivastava, and Ser-Nam Lim.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib45.8.1" class="ltx_text" style="font-size:90%;">Ma-lmm: Memory-augmented large multimodal model for long-term video
understanding.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib45.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2404.05726</em><span id="bib.bib45.10.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib46.5.5.1" class="ltx_text" style="font-size:90%;">He et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib46.7.1" class="ltx_text" style="font-size:90%;">
Conghui He, Zhenjiang Jin, Chaoxi Xu, Jiantao Qiu, Bin Wang, Wei Li, Hang Yan,
Jiaqi Wang, and Da Lin.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib46.8.1" class="ltx_text" style="font-size:90%;">Wanjuan: A comprehensive multimodal dataset for advancing english and
chinese large models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib46.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ArXiv</em><span id="bib.bib46.10.2" class="ltx_text" style="font-size:90%;">, abs/2308.10755, 2023.
</span>
</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib47.5.5.1" class="ltx_text" style="font-size:90%;">Hong et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib47.7.1" class="ltx_text" style="font-size:90%;">
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan
Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib47.8.1" class="ltx_text" style="font-size:90%;">Cogagent: A visual language model for gui agents.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib47.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2312.08914</em><span id="bib.bib47.10.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib48.5.5.1" class="ltx_text" style="font-size:90%;">Hong et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib48.7.1" class="ltx_text" style="font-size:90%;">
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan
Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib48.8.1" class="ltx_text" style="font-size:90%;">CogAgent: A visual language model for gui agents.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib48.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib48.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR)</em><span id="bib.bib48.11.3" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib49.5.5.1" class="ltx_text" style="font-size:90%;">Hu et al. [2024a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib49.7.1" class="ltx_text" style="font-size:90%;">
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji
Zhang, Qin Jin, Fei Huang, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib49.8.1" class="ltx_text" style="font-size:90%;">mplug-docowl 1.5: Unified structure learning for ocr-free document
understanding.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib49.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2403.12895</em><span id="bib.bib49.10.2" class="ltx_text" style="font-size:90%;">, 2024a.
</span>
</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib50.5.5.1" class="ltx_text" style="font-size:90%;">Hu et al. [2024b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib50.7.1" class="ltx_text" style="font-size:90%;">
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan, Liang Zhang, Bo Zhang, Chen Li, Ji
Zhang, Qin Jin, Fei Huang, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib50.8.1" class="ltx_text" style="font-size:90%;">mPLUG-DocOwl 1.5: Unified structure learning for ocr-free document
understanding.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib50.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2403.12895</em><span id="bib.bib50.10.2" class="ltx_text" style="font-size:90%;">, 2024b.
</span>
</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib51.5.5.1" class="ltx_text" style="font-size:90%;">Hu et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib51.7.1" class="ltx_text" style="font-size:90%;">
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean
Wang, Lu Wang, and Weizhu Chen.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib51.8.1" class="ltx_text" style="font-size:90%;">LoRA: Low-rank adaptation of large language models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib51.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib51.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the International Conference on Learning
Representations (ICLR)</em><span id="bib.bib51.11.3" class="ltx_text" style="font-size:90%;">, 2022.
</span>
</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib52.5.5.1" class="ltx_text" style="font-size:90%;">Huang et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib52.7.1" class="ltx_text" style="font-size:90%;">
Suyuan Huang, Haoxin Zhang, Yan Gao, Yao Hu, and Zengchang Qin.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib52.8.1" class="ltx_text" style="font-size:90%;">From image to video, what do we need in multimodal llms?
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib52.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2404.11865</em><span id="bib.bib52.10.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib53.4.4.1" class="ltx_text" style="font-size:90%;">Hudson and Manning [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib53.6.1" class="ltx_text" style="font-size:90%;">
Drew A Hudson and Christopher D Manning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib53.7.1" class="ltx_text" style="font-size:90%;">Gqa: A new dataset for real-world visual reasoning and compositional
question answering.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib53.8.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Conference on Computer Vision and Pattern Recognition (CVPR)</em><span id="bib.bib53.9.2" class="ltx_text" style="font-size:90%;">,
2019.
</span>
</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib54.5.5.1" class="ltx_text" style="font-size:90%;">Islam et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib54.7.1" class="ltx_text" style="font-size:90%;">
Md Mohaiminul Islam, Ngan Ho, Xitong Yang, Tushar Nagarajan, Lorenzo Torresani,
and Gedas Bertasius.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib54.8.1" class="ltx_text" style="font-size:90%;">Video recap: Recursive captioning of hour-long videos.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib54.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2402.13250</em><span id="bib.bib54.10.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib55.5.5.1" class="ltx_text" style="font-size:90%;">Jiang et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib55.7.1" class="ltx_text" style="font-size:90%;">
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford,
Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel,
Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux,
Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,
and William El Sayed.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib55.8.1" class="ltx_text" style="font-size:90%;">Mistral 7b, 2023.
</span>
</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib56.5.5.1" class="ltx_text" style="font-size:90%;">Jiang et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib56.7.1" class="ltx_text" style="font-size:90%;">
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib56.8.1" class="ltx_text" style="font-size:90%;">Mantis: Interleaved multi-image instruction tuning, 2024.
</span>
</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib57.5.5.1" class="ltx_text" style="font-size:90%;">Jin et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib57.7.1" class="ltx_text" style="font-size:90%;">
Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib57.8.1" class="ltx_text" style="font-size:90%;">Chat-univi: Unified visual representation empowers large language
models with image and video understanding.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib57.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2311.08046</em><span id="bib.bib57.10.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib58.5.5.1" class="ltx_text" style="font-size:90%;">Kafle et al. [2018]</span></span>
<span class="ltx_bibblock"><span id="bib.bib58.7.1" class="ltx_text" style="font-size:90%;">
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib58.8.1" class="ltx_text" style="font-size:90%;">DVQA: Understanding data visualizations via question answering.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib58.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib58.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR)</em><span id="bib.bib58.11.3" class="ltx_text" style="font-size:90%;">, 2018.
</span>
</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib59.5.5.1" class="ltx_text" style="font-size:90%;">Kahatapitiya et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib59.7.1" class="ltx_text" style="font-size:90%;">
Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S Ryoo.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib59.8.1" class="ltx_text" style="font-size:90%;">Language repository for long video understanding.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib59.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2403.14622</em><span id="bib.bib59.10.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib60.5.5.1" class="ltx_text" style="font-size:90%;">Kaplan et al. [2020]</span></span>
<span class="ltx_bibblock"><span id="bib.bib60.7.1" class="ltx_text" style="font-size:90%;">
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon
Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib60.8.1" class="ltx_text" style="font-size:90%;">Scaling laws for neural language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib60.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2001.08361</em><span id="bib.bib60.10.2" class="ltx_text" style="font-size:90%;">, 2020.
</span>
</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib61.5.5.1" class="ltx_text" style="font-size:90%;">Kembhavi et al. [2016]</span></span>
<span class="ltx_bibblock"><span id="bib.bib61.7.1" class="ltx_text" style="font-size:90%;">
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi,
and Ali Farhadi.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib61.8.1" class="ltx_text" style="font-size:90%;">A diagram is worth a dozen images.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib61.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib61.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the European Conference on Computer Vision
(ECCV)</em><span id="bib.bib61.11.3" class="ltx_text" style="font-size:90%;">, 2016.
</span>
</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib62.5.5.1" class="ltx_text" style="font-size:90%;">Kembhavi et al. [2017]</span></span>
<span class="ltx_bibblock"><span id="bib.bib62.7.1" class="ltx_text" style="font-size:90%;">
Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi,
and Hannaneh Hajishirzi.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib62.8.1" class="ltx_text" style="font-size:90%;">Are you smarter than a sixth grader? textbook question answering for
multimodal machine comprehension.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib62.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib62.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR)</em><span id="bib.bib62.11.3" class="ltx_text" style="font-size:90%;">, 2017.
</span>
</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib63.5.5.1" class="ltx_text" style="font-size:90%;">Kim et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib63.7.1" class="ltx_text" style="font-size:90%;">
Wonkyun Kim, Changin Choi, Wonseok Lee, and Wonjong Rhee.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib63.8.1" class="ltx_text" style="font-size:90%;">An image grid can be worth a video: Zero-shot video question
answering using a vlm, 2024.
</span>
</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib64.5.5.1" class="ltx_text" style="font-size:90%;">Laurençon et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib64.7.1" class="ltx_text" style="font-size:90%;">
Hugo Laurençon, Léo Tronchon, and Victor Sanh.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib64.8.1" class="ltx_text" style="font-size:90%;">Unlocking the conversion of web screenshots into html code with the
websight dataset.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib64.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2403.09029</em><span id="bib.bib64.10.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib65" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib65.5.5.1" class="ltx_text" style="font-size:90%;">Lerner et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib65.7.1" class="ltx_text" style="font-size:90%;">
Paul Lerner, Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric
Besançon, José G Moreno, and Jesús Lovón Melgarejo.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib65.8.1" class="ltx_text" style="font-size:90%;">Viquae, a dataset for knowledge-based visual question answering about
named entities.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib65.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib65.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the 45th International ACM SIGIR Conference
on Research and Development in Information Retrieval</em><span id="bib.bib65.11.3" class="ltx_text" style="font-size:90%;">, pages 3108–3120,
2022.
</span>
</span>
</li>
<li id="bib.bib66" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib66.5.5.1" class="ltx_text" style="font-size:90%;">Li et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib66.7.1" class="ltx_text" style="font-size:90%;">
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib66.8.1" class="ltx_text" style="font-size:90%;">Seed-bench: Benchmarking multimodal llms with generative
comprehension, 2023a.
</span>
</span>
</li>
<li id="bib.bib67" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib67.5.5.1" class="ltx_text" style="font-size:90%;">Li et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib67.7.1" class="ltx_text" style="font-size:90%;">
Bo Li, Peiyuan Zhang, Jingkang Yang, Yuanhan Zhang, Fanyi Pu, and Ziwei Liu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib67.8.1" class="ltx_text" style="font-size:90%;">Otterhd: A high-resolution multi-modality model, 2023b.
</span>
</span>
</li>
<li id="bib.bib68" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib68.5.5.1" class="ltx_text" style="font-size:90%;">Li et al. [2023c]</span></span>
<span class="ltx_bibblock"><span id="bib.bib68.7.1" class="ltx_text" style="font-size:90%;">
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib68.8.1" class="ltx_text" style="font-size:90%;">Otter: A multi-modal model with in-context instruction tuning.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib68.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv.org</em><span id="bib.bib68.10.2" class="ltx_text" style="font-size:90%;">, 2023c.
</span>
</span>
</li>
<li id="bib.bib69" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib69.5.5.1" class="ltx_text" style="font-size:90%;">Li et al. [2023d]</span></span>
<span class="ltx_bibblock"><span id="bib.bib69.7.1" class="ltx_text" style="font-size:90%;">
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang,
Limin Wang, and Yu Qiao.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib69.8.1" class="ltx_text" style="font-size:90%;">Videochat: Chat-centric video understanding.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib69.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2305.06355</em><span id="bib.bib69.10.2" class="ltx_text" style="font-size:90%;">, 2023d.
</span>
</span>
</li>
<li id="bib.bib70" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib70.5.5.1" class="ltx_text" style="font-size:90%;">Li et al. [2023e]</span></span>
<span class="ltx_bibblock"><span id="bib.bib70.7.1" class="ltx_text" style="font-size:90%;">
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan
Xu, Guo Chen, Ping Luo, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib70.8.1" class="ltx_text" style="font-size:90%;">Mvbench: A comprehensive multi-modal video understanding benchmark.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib70.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2311.17005</em><span id="bib.bib70.10.2" class="ltx_text" style="font-size:90%;">, 2023e.
</span>
</span>
</li>
<li id="bib.bib71" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib71.5.5.1" class="ltx_text" style="font-size:90%;">Li et al. [2024a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib71.7.1" class="ltx_text" style="font-size:90%;">
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan
Xu, Guo Chen, Ping Luo, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib71.8.1" class="ltx_text" style="font-size:90%;">Mvbench: A comprehensive multi-modal video understanding benchmark.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib71.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib71.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR)</em><span id="bib.bib71.11.3" class="ltx_text" style="font-size:90%;">, 2024a.
</span>
</span>
</li>
<li id="bib.bib72" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib72.5.5.1" class="ltx_text" style="font-size:90%;">Li et al. [2023f]</span></span>
<span class="ltx_bibblock"><span id="bib.bib72.7.1" class="ltx_text" style="font-size:90%;">
Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng
Yang, Benyou Wang, and Lingpeng Kong.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib72.8.1" class="ltx_text" style="font-size:90%;">Silkie: Preference distillation for large visual language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib72.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2312.10665</em><span id="bib.bib72.10.2" class="ltx_text" style="font-size:90%;">, 2023f.
</span>
</span>
</li>
<li id="bib.bib73" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib73.5.5.1" class="ltx_text" style="font-size:90%;">Li et al. [2023g]</span></span>
<span class="ltx_bibblock"><span id="bib.bib73.7.1" class="ltx_text" style="font-size:90%;">
Yanwei Li, Chengyao Wang, and Jiaya Jia.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib73.8.1" class="ltx_text" style="font-size:90%;">Llama-vid: An image is worth 2 tokens in large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib73.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2311.17043</em><span id="bib.bib73.10.2" class="ltx_text" style="font-size:90%;">, 2023g.
</span>
</span>
</li>
<li id="bib.bib74" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib74.5.5.1" class="ltx_text" style="font-size:90%;">Li et al. [2024b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib74.7.1" class="ltx_text" style="font-size:90%;">
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang
Chu, Shaoteng Liu, and Jiaya Jia.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib74.8.1" class="ltx_text" style="font-size:90%;">Mini-Gemini: Mining the potential of multi-modality vision language
models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib74.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2403.18814</em><span id="bib.bib74.10.2" class="ltx_text" style="font-size:90%;">, 2024b.
</span>
</span>
</li>
<li id="bib.bib75" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib75.5.5.1" class="ltx_text" style="font-size:90%;">Li et al. [2023h]</span></span>
<span class="ltx_bibblock"><span id="bib.bib75.7.1" class="ltx_text" style="font-size:90%;">
Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma,
Benjamin Van Durme, and Alan L Yuille.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib75.8.1" class="ltx_text" style="font-size:90%;">Super-clevr: A virtual benchmark to diagnose domain robustness in
visual reasoning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib75.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib75.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR)</em><span id="bib.bib75.11.3" class="ltx_text" style="font-size:90%;">, 2023h.
</span>
</span>
</li>
<li id="bib.bib76" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib76.5.5.1" class="ltx_text" style="font-size:90%;">Li et al. [2023i]</span></span>
<span class="ltx_bibblock"><span id="bib.bib76.7.1" class="ltx_text" style="font-size:90%;">
Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun,
Yuliang Liu, and Xiang Bai.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib76.8.1" class="ltx_text" style="font-size:90%;">Monkey: Image resolution and text label are important things for
large multi-modal models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib76.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2311.06607</em><span id="bib.bib76.10.2" class="ltx_text" style="font-size:90%;">, 2023i.
</span>
</span>
</li>
<li id="bib.bib77" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib77.5.5.1" class="ltx_text" style="font-size:90%;">Lin et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib77.7.1" class="ltx_text" style="font-size:90%;">
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib77.8.1" class="ltx_text" style="font-size:90%;">Video-llava: Learning united visual representation by alignment
before projection.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib77.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2311.10122</em><span id="bib.bib77.10.2" class="ltx_text" style="font-size:90%;">, 2023a.
</span>
</span>
</li>
<li id="bib.bib78" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib78.5.5.1" class="ltx_text" style="font-size:90%;">Lin et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib78.7.1" class="ltx_text" style="font-size:90%;">
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao,
Jan Kautz, Mohammad Shoeybi, and Song Han.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib78.8.1" class="ltx_text" style="font-size:90%;">Vila: On pre-training for visual language models, 2024.
</span>
</span>
</li>
<li id="bib.bib79" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib79.5.5.1" class="ltx_text" style="font-size:90%;">Lin et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib79.7.1" class="ltx_text" style="font-size:90%;">
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu,
Chen Lin, Wenqi Shao, Keqin Chen, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib79.8.1" class="ltx_text" style="font-size:90%;">Sphinx: The joint mixing of weights, tasks, and visual embeddings for
multi-modal large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib79.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2311.07575</em><span id="bib.bib79.10.2" class="ltx_text" style="font-size:90%;">, 2023b.
</span>
</span>
</li>
<li id="bib.bib80" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib80.4.4.1" class="ltx_text" style="font-size:90%;">Lindström and Abraham [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib80.6.1" class="ltx_text" style="font-size:90%;">
Adam Dahlgren Lindström and Savitha Sam Abraham.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib80.7.1" class="ltx_text" style="font-size:90%;">Clevr-math: A dataset for compositional language, visual and
mathematical reasoning.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib80.8.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2208.05358</em><span id="bib.bib80.9.2" class="ltx_text" style="font-size:90%;">, 2022.
</span>
</span>
</li>
<li id="bib.bib81" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib81.5.5.1" class="ltx_text" style="font-size:90%;">Liu et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib81.7.1" class="ltx_text" style="font-size:90%;">
Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib81.8.1" class="ltx_text" style="font-size:90%;">Visual spatial reasoning.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib81.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Transactions of the Association for Computational Linguistics
(TACL)</em><span id="bib.bib81.10.2" class="ltx_text" style="font-size:90%;">, 2023a.
</span>
</span>
</li>
<li id="bib.bib82" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib82.5.5.1" class="ltx_text" style="font-size:90%;">Liu et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib82.7.1" class="ltx_text" style="font-size:90%;">
Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo
Cho, Yaser Yacoob, and Dong Yu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib82.8.1" class="ltx_text" style="font-size:90%;">Mmc: Advancing multimodal chart understanding with large-scale
instruction tuning.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib82.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2311.10774</em><span id="bib.bib82.10.2" class="ltx_text" style="font-size:90%;">, 2023b.
</span>
</span>
</li>
<li id="bib.bib83" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib83.5.5.1" class="ltx_text" style="font-size:90%;">Liu et al. [2023c]</span></span>
<span class="ltx_bibblock"><span id="bib.bib83.7.1" class="ltx_text" style="font-size:90%;">
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib83.8.1" class="ltx_text" style="font-size:90%;">Improved baselines with visual instruction tuning.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib83.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2310.03744</em><span id="bib.bib83.10.2" class="ltx_text" style="font-size:90%;">, 2023c.
</span>
</span>
</li>
<li id="bib.bib84" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib84.5.5.1" class="ltx_text" style="font-size:90%;">Liu et al. [2023d]</span></span>
<span class="ltx_bibblock"><span id="bib.bib84.7.1" class="ltx_text" style="font-size:90%;">
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib84.8.1" class="ltx_text" style="font-size:90%;">Visual instruction tuning.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib84.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv.org</em><span id="bib.bib84.10.2" class="ltx_text" style="font-size:90%;">, 2023d.
</span>
</span>
</li>
<li id="bib.bib85" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib85.5.5.1" class="ltx_text" style="font-size:90%;">Liu et al. [2024a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib85.7.1" class="ltx_text" style="font-size:90%;">
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and
Yong Jae Lee.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib85.8.1" class="ltx_text" style="font-size:90%;">Llava-next: Improved reasoning, ocr, and world knowledge,
2024a.
</span>
</span>
</li>
<li id="bib.bib86" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib86.5.5.1" class="ltx_text" style="font-size:90%;">Liu et al. [2024b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib86.7.1" class="ltx_text" style="font-size:90%;">
Shuo Liu, Kaining Ying, Hao Zhang, Yue Yang, Yuqi Lin, Tianle Zhang, Chuanhao
Li, Yu Qiao, Ping Luo, Wenqi Shao, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib86.8.1" class="ltx_text" style="font-size:90%;">Convbench: A multi-turn conversation evaluation benchmark with
hierarchical capability for large vision-language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib86.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2403.20194</em><span id="bib.bib86.10.2" class="ltx_text" style="font-size:90%;">, 2024b.
</span>
</span>
</li>
<li id="bib.bib87" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib87.5.5.1" class="ltx_text" style="font-size:90%;">Liu et al. [2023e]</span></span>
<span class="ltx_bibblock"><span id="bib.bib87.7.1" class="ltx_text" style="font-size:90%;">
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhnag, Wangbo Zhao, Yike
Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib87.8.1" class="ltx_text" style="font-size:90%;">Mmbench: Is your multi-modal model an all-around player?
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib87.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv:2307.06281</em><span id="bib.bib87.10.2" class="ltx_text" style="font-size:90%;">, 2023e.
</span>
</span>
</li>
<li id="bib.bib88" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib88.5.5.1" class="ltx_text" style="font-size:90%;">Liu et al. [2024c]</span></span>
<span class="ltx_bibblock"><span id="bib.bib88.7.1" class="ltx_text" style="font-size:90%;">
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo
Chen, Xu Sun, and Lu Hou.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib88.8.1" class="ltx_text" style="font-size:90%;">TempCompass: Do video llms really understand videos?
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib88.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2403.00476</em><span id="bib.bib88.10.2" class="ltx_text" style="font-size:90%;">, 2024c.
</span>
</span>
</li>
<li id="bib.bib89" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib89.5.5.1" class="ltx_text" style="font-size:90%;">Liu et al. [2024d]</span></span>
<span class="ltx_bibblock"><span id="bib.bib89.7.1" class="ltx_text" style="font-size:90%;">
Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng lin Liu,
Lianwen Jin, and Xiang Bai.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib89.8.1" class="ltx_text" style="font-size:90%;">On the hidden mystery of ocr in large multimodal models,
2024d.
</span>
</span>
</li>
<li id="bib.bib90" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib90.5.5.1" class="ltx_text" style="font-size:90%;">Liu et al. [2024e]</span></span>
<span class="ltx_bibblock"><span id="bib.bib90.7.1" class="ltx_text" style="font-size:90%;">
Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang
Bai.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib90.8.1" class="ltx_text" style="font-size:90%;">Textmonkey: An ocr-free large multimodal model for understanding
document.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib90.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2403.04473</em><span id="bib.bib90.10.2" class="ltx_text" style="font-size:90%;">, 2024e.
</span>
</span>
</li>
<li id="bib.bib91" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib91.5.5.1" class="ltx_text" style="font-size:90%;">Liu et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib91.7.1" class="ltx_text" style="font-size:90%;">
Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell,
and Saining Xie.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib91.8.1" class="ltx_text" style="font-size:90%;">A convnet for the 2020s, 2022.
</span>
</span>
</li>
<li id="bib.bib92" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib92.5.5.1" class="ltx_text" style="font-size:90%;">Liu et al. [2024f]</span></span>
<span class="ltx_bibblock"><span id="bib.bib92.7.1" class="ltx_text" style="font-size:90%;">
Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, Zijian
Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib92.8.1" class="ltx_text" style="font-size:90%;">MMDU: A multi-turn multi-image dialog understanding benchmark and
instruction-tuning dataset for lvlms.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib92.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2406.11833</em><span id="bib.bib92.10.2" class="ltx_text" style="font-size:90%;">, 2024f.
</span>
</span>
</li>
<li id="bib.bib93" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib93.5.5.1" class="ltx_text" style="font-size:90%;">Liu et al. [2024g]</span></span>
<span class="ltx_bibblock"><span id="bib.bib93.7.1" class="ltx_text" style="font-size:90%;">
Ziyu Liu, Zeyi Sun, Yuhang Zang, Wei Li, Pan Zhang, Xiaoyi Dong, Yuanjun Xiong,
Dahua Lin, and Jiaqi Wang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib93.8.1" class="ltx_text" style="font-size:90%;">RAR: Retrieving and ranking augmented mllms for visual recognition.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib93.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2403.13805</em><span id="bib.bib93.10.2" class="ltx_text" style="font-size:90%;">, 2024g.
</span>
</span>
</li>
<li id="bib.bib94" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib94.4.4.1" class="ltx_text" style="font-size:90%;">LocalLLaMA [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib94.6.1" class="ltx_text" style="font-size:90%;">
LocalLLaMA.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib94.7.1" class="ltx_text" style="font-size:90%;">Dynamically scaled rope further increases performance of long context
llama with zero fine-tuning, 2023.
</span>
</span>
</li>
<li id="bib.bib95" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib95.5.5.1" class="ltx_text" style="font-size:90%;">Lozhkov et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib95.7.1" class="ltx_text" style="font-size:90%;">
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel
Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang
Wei, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib95.8.1" class="ltx_text" style="font-size:90%;">Starcoder 2 and the stack v2: The next generation.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib95.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2402.19173</em><span id="bib.bib95.10.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib96" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib96.5.5.1" class="ltx_text" style="font-size:90%;">Lu et al. [2021a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib96.7.1" class="ltx_text" style="font-size:90%;">
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and
Song-Chun Zhu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib96.8.1" class="ltx_text" style="font-size:90%;">Inter-gps: Interpretable geometry problem solving with formal
language and symbolic reasoning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib96.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib96.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">The 59th Annual Meeting of the Association for Computational
Linguistics (ACL)</em><span id="bib.bib96.11.3" class="ltx_text" style="font-size:90%;">, 2021a.
</span>
</span>
</li>
<li id="bib.bib97" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib97.5.5.1" class="ltx_text" style="font-size:90%;">Lu et al. [2021b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib97.7.1" class="ltx_text" style="font-size:90%;">
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu,
Xiaodan Liang, and Song-Chun Zhu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib97.8.1" class="ltx_text" style="font-size:90%;">Iconqa: A new benchmark for abstract diagram understanding and visual
language reasoning.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib97.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2110.13214</em><span id="bib.bib97.10.2" class="ltx_text" style="font-size:90%;">, 2021b.
</span>
</span>
</li>
<li id="bib.bib98" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib98.5.5.1" class="ltx_text" style="font-size:90%;">Lu et al. [2022a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib98.7.1" class="ltx_text" style="font-size:90%;">
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu,
Oyvind Tafjord, Peter Clark, and Ashwin Kalyan.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib98.8.1" class="ltx_text" style="font-size:90%;">Learn to explain: Multimodal reasoning via thought chains for science
question answering.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib98.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib98.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Advances in Neural Information Processing Systems
(NeurIPS)</em><span id="bib.bib98.11.3" class="ltx_text" style="font-size:90%;">, 2022a.
</span>
</span>
</li>
<li id="bib.bib99" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib99.5.5.1" class="ltx_text" style="font-size:90%;">Lu et al. [2022b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib99.7.1" class="ltx_text" style="font-size:90%;">
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay
Rajpurohit, Peter Clark, and Ashwin Kalyan.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib99.8.1" class="ltx_text" style="font-size:90%;">Dynamic prompt learning via policy gradient for semi-structured
mathematical reasoning.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib99.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2209.14610</em><span id="bib.bib99.10.2" class="ltx_text" style="font-size:90%;">, 2022b.
</span>
</span>
</li>
<li id="bib.bib100" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib100.5.5.1" class="ltx_text" style="font-size:90%;">Lu et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib100.7.1" class="ltx_text" style="font-size:90%;">
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh
Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib100.8.1" class="ltx_text" style="font-size:90%;">Mathvista: Evaluating mathematical reasoning of foundation models in
visual contexts.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib100.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib100.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">International Conference on Learning Representations
(ICLR)</em><span id="bib.bib100.11.3" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib101" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib101.5.5.1" class="ltx_text" style="font-size:90%;">Luo et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib101.7.1" class="ltx_text" style="font-size:90%;">
Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao
Wang, and Zhongyu Wei.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib101.8.1" class="ltx_text" style="font-size:90%;">Valley: Video assistant with large language model enhanced ability.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib101.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2306.07207</em><span id="bib.bib101.10.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib102" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib102.5.5.1" class="ltx_text" style="font-size:90%;">Lv et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib102.7.1" class="ltx_text" style="font-size:90%;">
Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang,
Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang,
Cha Zhang, and Furu Wei.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib102.8.1" class="ltx_text" style="font-size:90%;">Kosmos-2.5: A multimodal literate model, 2023.
</span>
</span>
</li>
<li id="bib.bib103" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib103.5.5.1" class="ltx_text" style="font-size:90%;">Ma et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib103.7.1" class="ltx_text" style="font-size:90%;">
Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan
Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Jiang Yu-Gang Pan, Liangming,
Jiaqi Wang, Yixin Cao, and Aixin Sun.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib103.8.1" class="ltx_text" style="font-size:90%;">MMLongBench-Doc: Benchmarking long-context document understanding
with visualizations.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib103.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2407.01523</em><span id="bib.bib103.10.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib104" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib104.5.5.1" class="ltx_text" style="font-size:90%;">Maaz et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib104.7.1" class="ltx_text" style="font-size:90%;">
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib104.8.1" class="ltx_text" style="font-size:90%;">Video-chatgpt: Towards detailed video understanding via large vision
and language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib104.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2306.05424</em><span id="bib.bib104.10.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib105" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib105.5.5.1" class="ltx_text" style="font-size:90%;">Marino et al. [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib105.7.1" class="ltx_text" style="font-size:90%;">
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib105.8.1" class="ltx_text" style="font-size:90%;">Ok-vqa: A visual question answering benchmark requiring external
knowledge.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib105.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib105.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR)</em><span id="bib.bib105.11.3" class="ltx_text" style="font-size:90%;">, pages 3195–3204, 2019.
</span>
</span>
</li>
<li id="bib.bib106" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib106.5.5.1" class="ltx_text" style="font-size:90%;">Masry et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib106.7.1" class="ltx_text" style="font-size:90%;">
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib106.8.1" class="ltx_text" style="font-size:90%;">Chartqa: A benchmark for question answering about charts with visual
and logical reasoning.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib106.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2203.10244</em><span id="bib.bib106.10.2" class="ltx_text" style="font-size:90%;">, 2022.
</span>
</span>
</li>
<li id="bib.bib107" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib107.5.5.1" class="ltx_text" style="font-size:90%;">Mathew et al. [2021]</span></span>
<span class="ltx_bibblock"><span id="bib.bib107.7.1" class="ltx_text" style="font-size:90%;">
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib107.8.1" class="ltx_text" style="font-size:90%;">Docvqa: A dataset for vqa on document images.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib107.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib107.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proc. of the IEEE Winter Conference on Applications of
Computer Vision (WACV)</em><span id="bib.bib107.11.3" class="ltx_text" style="font-size:90%;">, 2021.
</span>
</span>
</li>
<li id="bib.bib108" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib108.5.5.1" class="ltx_text" style="font-size:90%;">Mathew et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib108.7.1" class="ltx_text" style="font-size:90%;">
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest
Valveny, and CV Jawahar.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib108.8.1" class="ltx_text" style="font-size:90%;">Infographicvqa.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib108.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib108.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proc. of the IEEE Winter Conference on Applications of
Computer Vision (WACV)</em><span id="bib.bib108.11.3" class="ltx_text" style="font-size:90%;">, 2022.
</span>
</span>
</li>
<li id="bib.bib109" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib109.5.5.1" class="ltx_text" style="font-size:90%;">Mishra et al. [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib109.7.1" class="ltx_text" style="font-size:90%;">
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib109.8.1" class="ltx_text" style="font-size:90%;">OCR-VQA: Visual question answering by reading text in images.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib109.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib109.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">International Conference on Document Analysis and
Recognition (ICDAR)</em><span id="bib.bib109.11.3" class="ltx_text" style="font-size:90%;">, 2019.
</span>
</span>
</li>
<li id="bib.bib110" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib110.5.5.1" class="ltx_text" style="font-size:90%;">Ning et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib110.7.1" class="ltx_text" style="font-size:90%;">
Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and
Li Yuan.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib110.8.1" class="ltx_text" style="font-size:90%;">Video-bench: A comprehensive benchmark and toolkit for evaluating
video-based large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib110.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2311.16103</em><span id="bib.bib110.10.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib111" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib111.4.4.1" class="ltx_text" style="font-size:90%;">OpenAI [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib111.6.1" class="ltx_text" style="font-size:90%;">
OpenAI.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib111.7.1" class="ltx_text" style="font-size:90%;">Chatgpt.
</span>
</span>
<span class="ltx_bibblock"><a target="_blank" href="https://openai.com/blog/chatgpt" title="" class="ltx_ref ltx_url ltx_font_typewriter" style="font-size:90%;">https://openai.com/blog/chatgpt</a><span id="bib.bib111.8.1" class="ltx_text" style="font-size:90%;">, 2022.
</span>
</span>
</li>
<li id="bib.bib112" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib112.4.4.1" class="ltx_text" style="font-size:90%;">OpenAI [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib112.6.1" class="ltx_text" style="font-size:90%;">
OpenAI.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib112.7.1" class="ltx_text" style="font-size:90%;">Gpt-4 technical report, 2023.
</span>
</span>
</li>
<li id="bib.bib113" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib113.5.5.1" class="ltx_text" style="font-size:90%;">Oquab et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib113.7.1" class="ltx_text" style="font-size:90%;">
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec,
Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin
El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes,
Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel
Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin,
and Piotr Bojanowski.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib113.8.1" class="ltx_text" style="font-size:90%;">Dinov2: Learning robust visual features without supervision, 2024.
</span>
</span>
</li>
<li id="bib.bib114" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib114.5.5.1" class="ltx_text" style="font-size:90%;">Ordonez et al. [2011]</span></span>
<span class="ltx_bibblock"><span id="bib.bib114.7.1" class="ltx_text" style="font-size:90%;">
Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib114.8.1" class="ltx_text" style="font-size:90%;">Im2text: Describing images using 1 million captioned photographs.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib114.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib114.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Advances in Neural Information Processing Systems
(NeurIPS)</em><span id="bib.bib114.11.3" class="ltx_text" style="font-size:90%;">, 2011.
</span>
</span>
</li>
<li id="bib.bib115" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib115.5.5.1" class="ltx_text" style="font-size:90%;">Ouyang et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib115.7.1" class="ltx_text" style="font-size:90%;">
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela
Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib115.8.1" class="ltx_text" style="font-size:90%;">Training language models to follow instructions with human feedback.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib115.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Advances in Neural Information Processing Systems (NeurIPS)</em><span id="bib.bib115.10.2" class="ltx_text" style="font-size:90%;">,
2022.
</span>
</span>
</li>
<li id="bib.bib116" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib116.5.5.1" class="ltx_text" style="font-size:90%;">Pal et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib116.7.1" class="ltx_text" style="font-size:90%;">
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and
Colin White.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib116.8.1" class="ltx_text" style="font-size:90%;">Smaug: Fixing failure modes of preference optimisation with
dpo-positive.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib116.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2402.13228</em><span id="bib.bib116.10.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib117" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib117.4.4.1" class="ltx_text" style="font-size:90%;">Pasupat and Liang [2015]</span></span>
<span class="ltx_bibblock"><span id="bib.bib117.6.1" class="ltx_text" style="font-size:90%;">
Panupong Pasupat and Percy Liang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib117.7.1" class="ltx_text" style="font-size:90%;">Compositional semantic parsing on semi-structured tables.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib117.8.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib117.9.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">The Annual Meeting of the Association for Computational
Linguistics (ACL)</em><span id="bib.bib117.10.3" class="ltx_text" style="font-size:90%;">, 2015.
</span>
</span>
</li>
<li id="bib.bib118" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib118.5.5.1" class="ltx_text" style="font-size:90%;">Peng et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib118.7.1" class="ltx_text" style="font-size:90%;">
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and
Furu Wei.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib118.8.1" class="ltx_text" style="font-size:90%;">Kosmos-2: Grounding multimodal large language models to the world.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib118.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv.org</em><span id="bib.bib118.10.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib119" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib119.5.5.1" class="ltx_text" style="font-size:90%;">Pi et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib119.7.1" class="ltx_text" style="font-size:90%;">
Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong
Zhang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib119.8.1" class="ltx_text" style="font-size:90%;">Strengthening multimodal large language model with bootstrapped
preference optimization.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib119.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2403.08730</em><span id="bib.bib119.10.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib120" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib120.5.5.1" class="ltx_text" style="font-size:90%;">Qian et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib120.7.1" class="ltx_text" style="font-size:90%;">
Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and
Jiaqi Wang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib120.8.1" class="ltx_text" style="font-size:90%;">Streaming long video understanding with large language models, 2024.
</span>
</span>
</li>
<li id="bib.bib121" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib121.4.4.1" class="ltx_text" style="font-size:90%;">Qwen [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib121.6.1" class="ltx_text" style="font-size:90%;">
Qwen.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib121.7.1" class="ltx_text" style="font-size:90%;">Introducing Qwen-7B: Open foundation and human-aligned models (of
the state-of-the-arts), 2023.
</span>
</span>
</li>
<li id="bib.bib122" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib122.5.5.1" class="ltx_text" style="font-size:90%;">Radford et al. [2021]</span></span>
<span class="ltx_bibblock"><span id="bib.bib122.7.1" class="ltx_text" style="font-size:90%;">
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib122.8.1" class="ltx_text" style="font-size:90%;">Learning transferable visual models from natural language
supervision.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib122.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib122.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the International Conference on Machine
learning (ICML)</em><span id="bib.bib122.11.3" class="ltx_text" style="font-size:90%;">, 2021.
</span>
</span>
</li>
<li id="bib.bib123" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib123.5.5.1" class="ltx_text" style="font-size:90%;">Radford et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib123.7.1" class="ltx_text" style="font-size:90%;">
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and
Ilya Sutskever.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib123.8.1" class="ltx_text" style="font-size:90%;">Robust speech recognition via large-scale weak supervision. arxiv
2022.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib123.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2212.04356</em><span id="bib.bib123.10.2" class="ltx_text" style="font-size:90%;">, 10, 2022.
</span>
</span>
</li>
<li id="bib.bib124" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib124.5.5.1" class="ltx_text" style="font-size:90%;">Rafailov et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib124.7.1" class="ltx_text" style="font-size:90%;">
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano
Ermon, and Chelsea Finn.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib124.8.1" class="ltx_text" style="font-size:90%;">Direct preference optimization: Your language model is secretly a
reward model.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib124.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib124.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Advances in Neural Information Processing Systems
(NeurIPS)</em><span id="bib.bib124.11.3" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib125" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib125.5.5.1" class="ltx_text" style="font-size:90%;">Schuhmann et al. [2021]</span></span>
<span class="ltx_bibblock"><span id="bib.bib125.7.1" class="ltx_text" style="font-size:90%;">
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk,
Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran
Komatsuzaki.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib125.8.1" class="ltx_text" style="font-size:90%;">Laion-400m: Open dataset of clip-filtered 400 million image-text
pairs.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib125.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2111.02114</em><span id="bib.bib125.10.2" class="ltx_text" style="font-size:90%;">, 2021.
</span>
</span>
</li>
<li id="bib.bib126" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib126.5.5.1" class="ltx_text" style="font-size:90%;">Schulman et al. [2017]</span></span>
<span class="ltx_bibblock"><span id="bib.bib126.7.1" class="ltx_text" style="font-size:90%;">
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib126.8.1" class="ltx_text" style="font-size:90%;">Proximal policy optimization algorithms.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib126.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:1707.06347</em><span id="bib.bib126.10.2" class="ltx_text" style="font-size:90%;">, 2017.
</span>
</span>
</li>
<li id="bib.bib127" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib127.5.5.1" class="ltx_text" style="font-size:90%;">Schwenk et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib127.7.1" class="ltx_text" style="font-size:90%;">
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and
Roozbeh Mottaghi.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib127.8.1" class="ltx_text" style="font-size:90%;">A-okvqa: A benchmark for visual question answering using world
knowledge.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib127.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib127.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the European Conference on Computer Vision
(ECCV)</em><span id="bib.bib127.11.3" class="ltx_text" style="font-size:90%;">, 2022.
</span>
</span>
</li>
<li id="bib.bib128" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib128.5.5.1" class="ltx_text" style="font-size:90%;">Shah et al. [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib128.7.1" class="ltx_text" style="font-size:90%;">
Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib128.8.1" class="ltx_text" style="font-size:90%;">Kvqa: Knowledge-aware visual question answering.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib128.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib128.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the Conference on Artificial Intelligence
(AAAI)</em><span id="bib.bib128.11.3" class="ltx_text" style="font-size:90%;">, 2019.
</span>
</span>
</li>
<li id="bib.bib129" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib129.5.5.1" class="ltx_text" style="font-size:90%;">Sharma et al. [2018]</span></span>
<span class="ltx_bibblock"><span id="bib.bib129.7.1" class="ltx_text" style="font-size:90%;">
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib129.8.1" class="ltx_text" style="font-size:90%;">Conceptual captions: A cleaned, hypernymed, image alt-text dataset
for automatic image captioning.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib129.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib129.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">The Annual Meeting of the Association for Computational
Linguistics (ACL)</em><span id="bib.bib129.11.3" class="ltx_text" style="font-size:90%;">, 2018.
</span>
</span>
</li>
<li id="bib.bib130" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib130.5.5.1" class="ltx_text" style="font-size:90%;">Shi et al. [2017]</span></span>
<span class="ltx_bibblock"><span id="bib.bib130.7.1" class="ltx_text" style="font-size:90%;">
Baoguang Shi, Cong Yao, Minghui Liao, Mingkun Yang, Pei Xu, Linyan Cui, Serge
Belongie, Shijian Lu, and Xiang Bai.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib130.8.1" class="ltx_text" style="font-size:90%;">Icdar2017 competition on reading chinese text in the wild (rctw-17).
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib130.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib130.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">International Conference on Document Analysis and
Recognition (ICDAR)</em><span id="bib.bib130.11.3" class="ltx_text" style="font-size:90%;">, 2017.
</span>
</span>
</li>
<li id="bib.bib131" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib131.5.5.1" class="ltx_text" style="font-size:90%;">Si et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib131.7.1" class="ltx_text" style="font-size:90%;">
Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib131.8.1" class="ltx_text" style="font-size:90%;">Design2code: How far are we from automating front-end engineering?
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib131.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2403.03163</em><span id="bib.bib131.10.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib132" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib132.5.5.1" class="ltx_text" style="font-size:90%;">Sidorov et al. [2020]</span></span>
<span class="ltx_bibblock"><span id="bib.bib132.7.1" class="ltx_text" style="font-size:90%;">
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib132.8.1" class="ltx_text" style="font-size:90%;">Textcaps: a dataset for image captioning with reading comprehension.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib132.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib132.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the European Conference on Computer Vision
(ECCV)</em><span id="bib.bib132.11.3" class="ltx_text" style="font-size:90%;">, 2020.
</span>
</span>
</li>
<li id="bib.bib133" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib133.5.5.1" class="ltx_text" style="font-size:90%;">Singh et al. [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib133.7.1" class="ltx_text" style="font-size:90%;">
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv
Batra, Devi Parikh, and Marcus Rohrbach.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib133.8.1" class="ltx_text" style="font-size:90%;">Towards vqa models that can read.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib133.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib133.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR)</em><span id="bib.bib133.11.3" class="ltx_text" style="font-size:90%;">, 2019.
</span>
</span>
</li>
<li id="bib.bib134" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib134.5.5.1" class="ltx_text" style="font-size:90%;">Song et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib134.7.1" class="ltx_text" style="font-size:90%;">
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang
Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib134.8.1" class="ltx_text" style="font-size:90%;">Moviechat: From dense token to sparse memory for long video
understanding.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib134.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2307.16449</em><span id="bib.bib134.10.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib135" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib135.5.5.1" class="ltx_text" style="font-size:90%;">Song et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib135.7.1" class="ltx_text" style="font-size:90%;">
Enxin Song, Wenhao Chai, Tian Ye, Jenq-Neng Hwang, Xi Li, and Gaoang Wang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib135.8.1" class="ltx_text" style="font-size:90%;">Moviechat+: Question-aware sparse memory for long video question
answering.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib135.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2404.17176</em><span id="bib.bib135.10.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib136" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib136.5.5.1" class="ltx_text" style="font-size:90%;">Sun et al. [2024a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib136.7.1" class="ltx_text" style="font-size:90%;">
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo,
Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib136.8.1" class="ltx_text" style="font-size:90%;">Generative multimodal models are in-context learners,
2024a.
</span>
</span>
</li>
<li id="bib.bib137" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib137.5.5.1" class="ltx_text" style="font-size:90%;">Sun et al. [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib137.7.1" class="ltx_text" style="font-size:90%;">
Yipeng Sun, Zihan Ni, Chee-Kheng Chng, Yuliang Liu, Canjie Luo, Chun Chet Ng,
Junyu Han, Errui Ding, Jingtuo Liu, Dimosthenis Karatzas, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib137.8.1" class="ltx_text" style="font-size:90%;">Icdar 2019 competition on large-scale street view text with partial
labeling-rrc-lsvt.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib137.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib137.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">International Conference on Document Analysis and
Recognition (ICDAR)</em><span id="bib.bib137.11.3" class="ltx_text" style="font-size:90%;">, 2019.
</span>
</span>
</li>
<li id="bib.bib138" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib138.5.5.1" class="ltx_text" style="font-size:90%;">Sun et al. [2024b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib138.7.1" class="ltx_text" style="font-size:90%;">
Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong,
Dahua Lin, and Jiaqi Wang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib138.8.1" class="ltx_text" style="font-size:90%;">Alpha-CLIP: A clip model focusing on wherever you want.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib138.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib138.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR)</em><span id="bib.bib138.11.3" class="ltx_text" style="font-size:90%;">, 2024b.
</span>
</span>
</li>
<li id="bib.bib139" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib139.4.4.1" class="ltx_text" style="font-size:90%;">Svetlichnaya [2020]</span></span>
<span class="ltx_bibblock"><span id="bib.bib139.6.1" class="ltx_text" style="font-size:90%;">
S. Svetlichnaya.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib139.7.1" class="ltx_text" style="font-size:90%;">DeepForm: Understand structured documents at scale., 2020.
</span>
</span>
</li>
<li id="bib.bib140" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib140.5.5.1" class="ltx_text" style="font-size:90%;">Tanaka et al. [2021]</span></span>
<span class="ltx_bibblock"><span id="bib.bib140.7.1" class="ltx_text" style="font-size:90%;">
Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib140.8.1" class="ltx_text" style="font-size:90%;">VisualMRC: Machine reading comprehension on document images.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib140.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib140.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the Conference on Artificial Intelligence
(AAAI)</em><span id="bib.bib140.11.3" class="ltx_text" style="font-size:90%;">, 2021.
</span>
</span>
</li>
<li id="bib.bib141" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib141.5.5.1" class="ltx_text" style="font-size:90%;">Tang et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib141.7.1" class="ltx_text" style="font-size:90%;">
Jingqun Tang, Chunhui Lin, Zhen Zhao, Shu Wei, Binghong Wu, Qi Liu, Hao Feng,
Yang Li, Siqi Wang, Lei Liao, Wei Shi, Yuliang Liu, Hao Liu, Yuan Xie, Xiang
Bai, and Can Huang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib141.8.1" class="ltx_text" style="font-size:90%;">Textsquare: Scaling up text-centric visual instruction tuning, 2024.
</span>
</span>
</li>
<li id="bib.bib142" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib142.4.4.1" class="ltx_text" style="font-size:90%;">Team [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib142.6.1" class="ltx_text" style="font-size:90%;">
Gemini Team.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib142.7.1" class="ltx_text" style="font-size:90%;">Gemini: A family of highly capable multimodal models,
2023a.
</span>
</span>
</li>
<li id="bib.bib143" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib143.4.4.1" class="ltx_text" style="font-size:90%;">Team [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib143.6.1" class="ltx_text" style="font-size:90%;">
InternLM Team.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib143.7.1" class="ltx_text" style="font-size:90%;">Internlm: A multilingual language model with progressively enhanced
capabilities.
</span>
</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/InternLM/InternLM" title="" class="ltx_ref ltx_url ltx_font_typewriter" style="font-size:90%;">https://github.com/InternLM/InternLM</a><span id="bib.bib143.8.1" class="ltx_text" style="font-size:90%;">, 2023b.
</span>
</span>
</li>
<li id="bib.bib144" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib144.4.4.1" class="ltx_text" style="font-size:90%;">Team [2024a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib144.6.1" class="ltx_text" style="font-size:90%;">
360VL Team.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib144.7.1" class="ltx_text" style="font-size:90%;">360vl, 2024a.
</span>
</span>
</li>
<li id="bib.bib145" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib145.4.4.1" class="ltx_text" style="font-size:90%;">Team [2024b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib145.6.1" class="ltx_text" style="font-size:90%;">
WeMM Team.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib145.7.1" class="ltx_text" style="font-size:90%;">Wemm, 2024b.
</span>
</span>
</li>
<li id="bib.bib146" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib146.5.5.1" class="ltx_text" style="font-size:90%;">Touvron et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib146.7.1" class="ltx_text" style="font-size:90%;">
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne
Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric
Hambro, Faisal Azhar, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib146.8.1" class="ltx_text" style="font-size:90%;">Llama: Open and efficient foundation language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib146.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv.org</em><span id="bib.bib146.10.2" class="ltx_text" style="font-size:90%;">, 2023a.
</span>
</span>
</li>
<li id="bib.bib147" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib147.5.5.1" class="ltx_text" style="font-size:90%;">Touvron et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib147.7.1" class="ltx_text" style="font-size:90%;">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine
Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale,
et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib147.8.1" class="ltx_text" style="font-size:90%;">Llama 2: Open foundation and fine-tuned chat models,
2023b.
</span>
</span>
</li>
<li id="bib.bib148" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib148.5.5.1" class="ltx_text" style="font-size:90%;">Wan et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib148.7.1" class="ltx_text" style="font-size:90%;">
Yuxuan Wan, Chaozheng Wang, Yi Dong, Wenxuan Wang, Shuqing Li, Yintong Huo, and
Michael R Lyu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib148.8.1" class="ltx_text" style="font-size:90%;">Automatically generating ui code from screenshot: A
divide-and-conquer-based approach.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib148.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2406.16386</em><span id="bib.bib148.10.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib149" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib149.5.5.1" class="ltx_text" style="font-size:90%;">Wang et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib149.7.1" class="ltx_text" style="font-size:90%;">
Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib149.8.1" class="ltx_text" style="font-size:90%;">To see is to believe: Prompting gpt-4v for better visual instruction
tuning.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib149.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2311.07574</em><span id="bib.bib149.10.2" class="ltx_text" style="font-size:90%;">, 2023a.
</span>
</span>
</li>
<li id="bib.bib150" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib150.5.5.1" class="ltx_text" style="font-size:90%;">Wang et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib150.7.1" class="ltx_text" style="font-size:90%;">
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji,
Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao
Dong, Ming Ding, and Jie Tang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib150.8.1" class="ltx_text" style="font-size:90%;">Cogvlm: Visual expert for pretrained language models,
2023b.
</span>
</span>
</li>
<li id="bib.bib151" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib151.5.5.1" class="ltx_text" style="font-size:90%;">Wei et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib151.7.1" class="ltx_text" style="font-size:90%;">
Haoran Wei, Lingyu Kong, Jinyue Chen, Liang Zhao, Zheng Ge, Jinrong Yang,
Jianjian Sun, Chunrui Han, and Xiangyu Zhang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib151.8.1" class="ltx_text" style="font-size:90%;">Vary: Scaling up the vision vocabulary for large vision-language
models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib151.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2312.06109</em><span id="bib.bib151.10.2" class="ltx_text" style="font-size:90%;">, 2023a.
</span>
</span>
</li>
<li id="bib.bib152" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib152.5.5.1" class="ltx_text" style="font-size:90%;">Wei et al. [2022]</span></span>
<span class="ltx_bibblock"><span id="bib.bib152.7.1" class="ltx_text" style="font-size:90%;">
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V
Le, Denny Zhou, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib152.8.1" class="ltx_text" style="font-size:90%;">Chain-of-thought prompting elicits reasoning in large language
models.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib152.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib152.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Advances in Neural Information Processing Systems
(NeurIPS)</em><span id="bib.bib152.11.3" class="ltx_text" style="font-size:90%;">, 2022.
</span>
</span>
</li>
<li id="bib.bib153" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib153.5.5.1" class="ltx_text" style="font-size:90%;">Wei et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib153.7.1" class="ltx_text" style="font-size:90%;">
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia,
Ed Chi, Quoc Le, and Denny Zhou.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib153.8.1" class="ltx_text" style="font-size:90%;">Chain-of-thought prompting elicits reasoning in large language
models, 2023b.
</span>
</span>
</li>
<li id="bib.bib154" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib154.5.5.1" class="ltx_text" style="font-size:90%;">Weng et al. [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib154.7.1" class="ltx_text" style="font-size:90%;">
Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib154.8.1" class="ltx_text" style="font-size:90%;">Longvlm: Efficient long video understanding via large language
models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib154.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2404.03384</em><span id="bib.bib154.10.2" class="ltx_text" style="font-size:90%;">, 2024.
</span>
</span>
</li>
<li id="bib.bib155" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib155.4.4.1" class="ltx_text" style="font-size:90%;">XAI [2024]</span></span>
<span class="ltx_bibblock"><span id="bib.bib155.6.1" class="ltx_text" style="font-size:90%;">
XAI.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib155.7.1" class="ltx_text" style="font-size:90%;">Grok-1.5 vision preview.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib155.8.1" class="ltx_text" style="font-size:90%;">2024.
</span>
</span>
</li>
<li id="bib.bib156" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib156.5.5.1" class="ltx_text" style="font-size:90%;">Xu et al. [2024a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib156.7.1" class="ltx_text" style="font-size:90%;">
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib156.8.1" class="ltx_text" style="font-size:90%;">Pllava : Parameter-free llava extension from images to videos for
video dense captioning, 2024a.
</span>
</span>
</li>
<li id="bib.bib157" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib157.5.5.1" class="ltx_text" style="font-size:90%;">Xu et al. [2024b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib157.7.1" class="ltx_text" style="font-size:90%;">
Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng
Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib157.8.1" class="ltx_text" style="font-size:90%;">Llava-uhd: an lmm perceiving any aspect ratio and high-resolution
images.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib157.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2403.11703</em><span id="bib.bib157.10.2" class="ltx_text" style="font-size:90%;">, 2024b.
</span>
</span>
</li>
<li id="bib.bib158" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib158.5.5.1" class="ltx_text" style="font-size:90%;">Ye et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib158.7.1" class="ltx_text" style="font-size:90%;">
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li,
Junfeng Tian, Qi Qian, Ji Zhang, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib158.8.1" class="ltx_text" style="font-size:90%;">Ureader: Universal ocr-free visually-situated language understanding
with multimodal large language model.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib158.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2310.05126</em><span id="bib.bib158.10.2" class="ltx_text" style="font-size:90%;">, 2023a.
</span>
</span>
</li>
<li id="bib.bib159" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib159.5.5.1" class="ltx_text" style="font-size:90%;">Ye et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib159.7.1" class="ltx_text" style="font-size:90%;">
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang
Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib159.8.1" class="ltx_text" style="font-size:90%;">mplug-owl: Modularization empowers large language models with
multimodality.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib159.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv.org</em><span id="bib.bib159.10.2" class="ltx_text" style="font-size:90%;">, 2023b.
</span>
</span>
</li>
<li id="bib.bib160" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib160.5.5.1" class="ltx_text" style="font-size:90%;">Young et al. [2014]</span></span>
<span class="ltx_bibblock"><span id="bib.bib160.7.1" class="ltx_text" style="font-size:90%;">
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib160.8.1" class="ltx_text" style="font-size:90%;">From image descriptions to visual denotations: New similarity metrics
for semantic inference over event descriptions.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib160.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Transactions of the Association for Computational Linguistics
(TACL)</em><span id="bib.bib160.10.2" class="ltx_text" style="font-size:90%;">, 2014.
</span>
</span>
</li>
<li id="bib.bib161" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib161.5.5.1" class="ltx_text" style="font-size:90%;">Yu et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib161.7.1" class="ltx_text" style="font-size:90%;">
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang,
James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib161.8.1" class="ltx_text" style="font-size:90%;">Metamath: Bootstrap your own mathematical questions for large
language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib161.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2309.12284</em><span id="bib.bib161.10.2" class="ltx_text" style="font-size:90%;">, 2023a.
</span>
</span>
</li>
<li id="bib.bib162" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib162.5.5.1" class="ltx_text" style="font-size:90%;">Yu et al. [2024a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib162.7.1" class="ltx_text" style="font-size:90%;">
Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib162.8.1" class="ltx_text" style="font-size:90%;">Self-chained image-language model for video localization and question
answering.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib162.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib162.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Advances in Neural Information Processing Systems
(NeurIPS)</em><span id="bib.bib162.11.3" class="ltx_text" style="font-size:90%;">, 2024a.
</span>
</span>
</li>
<li id="bib.bib163" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib163.5.5.1" class="ltx_text" style="font-size:90%;">Yu et al. [2024b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib163.7.1" class="ltx_text" style="font-size:90%;">
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu,
Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib163.8.1" class="ltx_text" style="font-size:90%;">RLHF-V: Towards trustworthy mllms via behavior alignment from
fine-grained correctional human feedback.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib163.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib163.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR)</em><span id="bib.bib163.11.3" class="ltx_text" style="font-size:90%;">, 2024b.
</span>
</span>
</li>
<li id="bib.bib164" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib164.5.5.1" class="ltx_text" style="font-size:90%;">Yu et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib164.7.1" class="ltx_text" style="font-size:90%;">
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu,
Xinchao Wang, and Lijuan Wang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib164.8.1" class="ltx_text" style="font-size:90%;">Mm-vet: Evaluating large multimodal models for integrated
capabilities.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib164.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2308.02490</em><span id="bib.bib164.10.2" class="ltx_text" style="font-size:90%;">, 2023b.
</span>
</span>
</li>
<li id="bib.bib165" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib165.5.5.1" class="ltx_text" style="font-size:90%;">Yuan et al. [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib165.7.1" class="ltx_text" style="font-size:90%;">
Tai-Ling Yuan, Zhe Zhu, Kun Xu, Cheng-Jun Li, Tai-Jiang Mu, and Shi-Min
Hu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib165.8.1" class="ltx_text" style="font-size:90%;">A large chinese text dataset in the wild.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib165.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Journal of Computer Science and Technology</em><span id="bib.bib165.10.2" class="ltx_text" style="font-size:90%;">, 34(3):509–521, 2019.
</span>
</span>
</li>
<li id="bib.bib166" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib166.5.5.1" class="ltx_text" style="font-size:90%;">Yue et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib166.7.1" class="ltx_text" style="font-size:90%;">
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel
Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin
Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao
Huang, Huan Sun, Yu Su, and Wenhu Chen.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib166.8.1" class="ltx_text" style="font-size:90%;">Mmmu: A massive multi-discipline multimodal understanding and
reasoning benchmark for expert agi.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib166.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2311.16502</em><span id="bib.bib166.10.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib167" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib167.5.5.1" class="ltx_text" style="font-size:90%;">Zang et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib167.7.1" class="ltx_text" style="font-size:90%;">
Yuhang Zang, Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib167.8.1" class="ltx_text" style="font-size:90%;">Contextual object detection with multimodal large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib167.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2305.18279</em><span id="bib.bib167.10.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib168" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib168.5.5.1" class="ltx_text" style="font-size:90%;">Zeng et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib168.7.1" class="ltx_text" style="font-size:90%;">
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi
Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue,
Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie
Tang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib168.8.1" class="ltx_text" style="font-size:90%;">GLM-130b: An open bilingual pre-trained model.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib168.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib168.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">Proceedings of the International Conference on Learning
Representations (ICLR)</em><span id="bib.bib168.11.3" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
<li id="bib.bib169" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib169.5.5.1" class="ltx_text" style="font-size:90%;">Zhai et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib169.7.1" class="ltx_text" style="font-size:90%;">
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib169.8.1" class="ltx_text" style="font-size:90%;">Sigmoid loss for language image pre-training, 2023.
</span>
</span>
</li>
<li id="bib.bib170" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib170.5.5.1" class="ltx_text" style="font-size:90%;">Zhang et al. [2024a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib170.7.1" class="ltx_text" style="font-size:90%;">
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib170.8.1" class="ltx_text" style="font-size:90%;">Long-CLIP: Unlocking the long-text capability of clip.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib170.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2403.15378</em><span id="bib.bib170.10.2" class="ltx_text" style="font-size:90%;">, 2024a.
</span>
</span>
</li>
<li id="bib.bib171" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib171.5.5.1" class="ltx_text" style="font-size:90%;">Zhang et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib171.7.1" class="ltx_text" style="font-size:90%;">
Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal,
and Gedas Bertasius.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib171.8.1" class="ltx_text" style="font-size:90%;">A simple llm framework for long-range video question-answering.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib171.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2312.17235</em><span id="bib.bib171.10.2" class="ltx_text" style="font-size:90%;">, 2023a.
</span>
</span>
</li>
<li id="bib.bib172" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib172.5.5.1" class="ltx_text" style="font-size:90%;">Zhang et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib172.7.1" class="ltx_text" style="font-size:90%;">
Hang Zhang, Xin Li, and Lidong Bing.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib172.8.1" class="ltx_text" style="font-size:90%;">Video-llama: An instruction-tuned audio-visual language model for
video understanding.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib172.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2306.02858</em><span id="bib.bib172.10.2" class="ltx_text" style="font-size:90%;">, 2023b.
</span>
</span>
</li>
<li id="bib.bib173" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib173.5.5.1" class="ltx_text" style="font-size:90%;">Zhang et al. [2023c]</span></span>
<span class="ltx_bibblock"><span id="bib.bib173.7.1" class="ltx_text" style="font-size:90%;">
Pan Zhang, Xiaoyi Dong Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan
Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Hang Yan, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib173.8.1" class="ltx_text" style="font-size:90%;">Internlm-xcomposer: A vision-language large model for advanced
text-image comprehension and composition.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib173.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2309.15112</em><span id="bib.bib173.10.2" class="ltx_text" style="font-size:90%;">, 2023c.
</span>
</span>
</li>
<li id="bib.bib174" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib174.5.5.1" class="ltx_text" style="font-size:90%;">Zhang et al. [2024b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib174.7.1" class="ltx_text" style="font-size:90%;">
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan
Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib174.8.1" class="ltx_text" style="font-size:90%;">Long context transfer from language to vision.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib174.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2406.16852</em><span id="bib.bib174.10.2" class="ltx_text" style="font-size:90%;">, 2024b.
</span>
</span>
</li>
<li id="bib.bib175" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib175.5.5.1" class="ltx_text" style="font-size:90%;">Zhang et al. [2019]</span></span>
<span class="ltx_bibblock"><span id="bib.bib175.7.1" class="ltx_text" style="font-size:90%;">
Rui Zhang, Yongsheng Zhou, Qianyi Jiang, Qi Song, Nan Li, Kai Zhou, Lei Wang,
Dong Wang, Minghui Liao, Mingkun Yang, et al.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib175.8.1" class="ltx_text" style="font-size:90%;">Icdar 2019 robust reading challenge on reading chinese text on
signboard.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib175.9.1" class="ltx_text" style="font-size:90%;">In </span><em id="bib.bib175.10.2" class="ltx_emph ltx_font_italic" style="font-size:90%;">International Conference on Document Analysis and
Recognition (ICDAR)</em><span id="bib.bib175.11.3" class="ltx_text" style="font-size:90%;">, 2019.
</span>
</span>
</li>
<li id="bib.bib176" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib176.5.5.1" class="ltx_text" style="font-size:90%;">Zhang et al. [2024c]</span></span>
<span class="ltx_bibblock"><span id="bib.bib176.7.1" class="ltx_text" style="font-size:90%;">
Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji,
Chen Change Loy, and Shuicheng Yan.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib176.8.1" class="ltx_text" style="font-size:90%;">Omg-llava: Bridging image-level, object-level, pixel-level reasoning
and understanding, 2024c.
</span>
</span>
</li>
<li id="bib.bib177" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib177.5.5.1" class="ltx_text" style="font-size:90%;">Zhang et al. [2024d]</span></span>
<span class="ltx_bibblock"><span id="bib.bib177.7.1" class="ltx_text" style="font-size:90%;">
Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and
Rong Jin.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib177.8.1" class="ltx_text" style="font-size:90%;">Beyond llava-hd: Diving into high-resolution large multimodal models,
2024d.
</span>
</span>
</li>
<li id="bib.bib178" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib178.5.5.1" class="ltx_text" style="font-size:90%;">Zhao et al. [2023a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib178.7.1" class="ltx_text" style="font-size:90%;">
Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan
Liu, Sheng Wang, Wenjuan Han, and Baobao Chang.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib178.8.1" class="ltx_text" style="font-size:90%;">Mmicl: Empowering vision-language model with multi-modal in-context
learning.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib178.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv.org</em><span id="bib.bib178.10.2" class="ltx_text" style="font-size:90%;">, 2023a.
</span>
</span>
</li>
<li id="bib.bib179" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib179.5.5.1" class="ltx_text" style="font-size:90%;">Zhao et al. [2023b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib179.7.1" class="ltx_text" style="font-size:90%;">
Wenliang Zhao, Xumin Yu, and Zengyi Qin.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib179.8.1" class="ltx_text" style="font-size:90%;">Melotts: High-quality multi-lingual multi-accent text-to-speech,
2023b.
</span>
</span>
</li>
<li id="bib.bib180" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib180.5.5.1" class="ltx_text" style="font-size:90%;">Zhao et al. [2023c]</span></span>
<span class="ltx_bibblock"><span id="bib.bib180.7.1" class="ltx_text" style="font-size:90%;">
Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib180.8.1" class="ltx_text" style="font-size:90%;">Beyond hallucinations: Enhancing lvlms through hallucination-aware
direct preference optimization.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib180.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2311.16839</em><span id="bib.bib180.10.2" class="ltx_text" style="font-size:90%;">, 2023c.
</span>
</span>
</li>
<li id="bib.bib181" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib181.5.5.1" class="ltx_text" style="font-size:90%;">Zhou et al. [2024a]</span></span>
<span class="ltx_bibblock"><span id="bib.bib181.7.1" class="ltx_text" style="font-size:90%;">
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong,
Bo Zhang, Tiejun Huang, and Zheng Liu.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib181.8.1" class="ltx_text" style="font-size:90%;">MLVU: A comprehensive benchmark for multi-task long video
understanding.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib181.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2406.04264</em><span id="bib.bib181.10.2" class="ltx_text" style="font-size:90%;">, 2024a.
</span>
</span>
</li>
<li id="bib.bib182" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib182.5.5.1" class="ltx_text" style="font-size:90%;">Zhou et al. [2024b]</span></span>
<span class="ltx_bibblock"><span id="bib.bib182.7.1" class="ltx_text" style="font-size:90%;">
Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib182.8.1" class="ltx_text" style="font-size:90%;">Aligning modalities in vision large language models via preference
fine-tuning.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib182.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv preprint arXiv:2402.11411</em><span id="bib.bib182.10.2" class="ltx_text" style="font-size:90%;">, 2024b.
</span>
</span>
</li>
<li id="bib.bib183" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem"><span id="bib.bib183.5.5.1" class="ltx_text" style="font-size:90%;">Zhu et al. [2023]</span></span>
<span class="ltx_bibblock"><span id="bib.bib183.7.1" class="ltx_text" style="font-size:90%;">
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.
</span>
</span>
<span class="ltx_bibblock"><span id="bib.bib183.8.1" class="ltx_text" style="font-size:90%;">Minigpt-4: Enhancing vision-language understanding with advanced
large language models.
</span>
</span>
<span class="ltx_bibblock"><em id="bib.bib183.9.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">arXiv.org</em><span id="bib.bib183.10.2" class="ltx_text" style="font-size:90%;">, 2023.
</span>
</span>
</li>
</ul>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.03319" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.03320" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.03320">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.03320" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.03321" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 17:51:02 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.03320
We present InternLM-XComposer-2.5 (IXC-2.5) , a versatile large-vision language model that supports long-contextual input and output.IXC-2.5 excels in various text-image comprehension and composition applications, achieving GPT-4V level capabilities with merely 7B LLM backend.Trained with 24K interleaved image-text contexts, it can seamlessly extend to 96K long contexts via RoPE extrapolation. This long-context capability allows IXC-2.5 to excel in tasks requiring extensive input and output contexts. Compared to its previous 2.0 version, InternLM-XComposer-2.5 features three major upgrades in vision-language comprehension: (1)  Ultra-High Resolution Understanding, (2)  Fine-Grained Video Understanding, and (3)  Multi-Turn Multi-Image Dialogue. In addition to comprehension, IXC-2.5 extends to two compelling applications using extra LoRA parameters for text-image composition: (1)  Crafting Webpages and (2)  Composing High-Quality Text-Image Articles. IXC-2.5 has been evaluated on 28 benchmarks, outperforming existing open-source state-of-the-art models on 16 benchmarks. It also surpasses or competes closely with GPT-4V and Gemini Pro on 16 key tasks. The InternLM-XComposer-2.5 is publicly available athttps://github.com/InternLM/InternLM-XComposer.
abstract
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
Abstract
2407.03320
1IntroductionFigure 1:Overview of InternLM-XComposer-2.5 (IXC-2.5)  performance on benchmarks in different domains, includingVideo Benchmarks,Structural High-resolution Benchmarks,General Visual QA Benchmarks,Multi-True Multi-Image Benchmark, andWebpage Crafting Benchmark.
IXC-2.5 based on InternLM2-7Bmatches or even surpasses GPT-4Vand Gemini Proin 15 benchmarks. Please refer to Table3,4,5for details.Recent advancements in Large Language Models (LLMs) have sparked interest in the development of Large Vision Language Models (LVLMs) . Leading paradigms like GPT-4, Gemini Pro 1.5, and Claude 3have achieved considerable success and significantly expanded the range of applications for LLMs. Open-source LVLMs are also being rapidly developed and can compete with proprietary APIs in several benchmarks. However, these open-source models still lag behind closed-source leading paradigms in versatility. They lack the ability to perform diverse vision-language comprehension and composition tasks, largely due to limited diversity in training corpus and challenges in managing long-context input and output.To further bridge the gap between proprietary APIsand open-sourced Large Vision Language Models, we are introducing InternLM-XComposer-2.5 (IXC-2.5) , a versatile LVLM supporting long-contextual input and output with diverse comprehension and composition capacities. IXC-2.5 excels in existing open-sourced LVLMs with two advantages.(1)  Versatility: IXC-2.5 supports a wide range of tasks related to comprehension and composition, such as free-form text-image conversation, OCR, video understanding, article composition with illustrations, and webpage crafting.(2)  Long-context capabilities in both input and output:It is natively trained with 24K interleaved image-text data, whose context window can be extended to 96K through positional encoding extrapolation, empowering the long-term human-AI interaction and content creation.Benefiting from the long contextual capability, compared to its previous 2.0 version, IXC-2.5 has upgraded three comprehension abilities:(1)  Ultra-High Resolution Understanding:IXC-2.5 enhances the dynamic resolution solution proposed in IXC2-4KHDwith a native 560 × 560 ViT vision encoder, supporting high-resolution images with any aspect ratio.(2)  Fine-Grained Video Understanding:IXC-2.5 treats videos as a ultra-high-resolution composite picture consisting of tens to hundreds of frames, allowing it to capture fine details through dense sampling and higher resolution for each frame.(3)  Multi-Turn Multi-Image Dialogue:IXC-2.5 supports free-form multi-turn multi-image dialogue, allowing it to naturally interact with humans in multi-round conversations.Besides comprehension, IXC-2.5 also supports two notable applications by incorporating extra LoRA parameters for text-image composition:(1)  Crafting Webpages:IXC-2.5 can be readily applied to create webpages by composing source code (HTML, CSS, and JavaScript)  following text-image instructions.(2)  Composing High-Quality Text-Image Articles:Compared to IXC-2, IXC-2.5 leverages specially designed Chain-of-Thought (CoT) and Direct Preference Optimization (DPO) techniques to significantly enhance the quality of its written content.We evaluated the versatility of InternLM-XComposer-2.5 (IXC-2.5)  across a range of twenty-eight benchmarks, including five video benchmarks, nine structural high-resolution benchmarks, twelve general VQA benchmarks, one multi-true multi-image benchmark, and one webpage crafting benchmark.
Compared to previous open-source LVLMs, IXC-2.5 achieved state-of-the-art results in 16 out of 28 benchmarks based on InternLM2-7Bbackend. As shown in Figure1, the performance of IXC-2.5 matches or even surpasses proprietary APIs,e.g., GPT-4Vand Gemini Pro, in 16 benchmarks.IXC-2.5 web demo now supports audio input and output using open-source tools. You may try it athttps://huggingface.co/spaces/Willow123/InternLM-XComposer.Figure 2:The capabilities of IXC-2.5: Chat on 4K-resolution Image and Multi-Turn Multi-Image Dialogue.Figure 3:The capabilities of IXC-2.5: Chat on Video, Resume-to-Homepage and Instruction-aware Webpage Generation.Figure 4:The capabilities of IXC-2.5: Screenshot-to-Code and Article Composing.
introduction
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
1Introduction
2407.03320
2Related WorksLVLMs for Text-Image Conversation.Large Language Models (LLMs) have received considerable attention because of their impressive performance in language comprehension and generation. Large vision-language models (LVLMs) have been developed by integrating LLMs with vision encodersto extend the ability to understand vision content, enabling the application of text-image conversation. Most existing LVLMs are trained for single-image multi-round conversations, while some workshave the ability to understand multi-image inputs. However, IXC-2.5 focuses on providing a free-form long-contextual multi-turn multi-image interaction experience, which has not been addressed yet.LVLMs for High Resolution Images Analysis.Understanding high-resolution images has significant potential applications such as OCR and document/chart analysis, which is attracting increased attention in the LVLMs area. In recent works, there are two main strategies to enable high-resolution understanding: (1)  High-resolution (HR)  visual encodersdirectly support higher resolution images. (2)  Patchification: A high-resolution image is cropped into patches. Each patch is processed with a low resolution vision encoder,e.g., CLIPand visual embeddings of patches are further concatenated as inputs for LLM backends. IXC-4KHDscales the supported resolution of open-source LVLMs into 4K and beyond for the first time. IXC-2.5 combines both solutions with a vision encoder trained with a resolution of 560x560 and a dynamic resolution solution proposed in IXC2-4KHD, resulting in further improvements.LVLMs for Video Understanding.In addition to image understanding, the LVLMs area has also witnessed emerging efforts in video analysis. To handle complex video inputs, existing works use sparse sampling or temporal pooling, compressed video tokens, memory banks, and language as a bridgefor video understanding. Apart from these video-specific designs, video analysis can also be formulated to understand a high-resolution composite picture consisting of sampled video frames. Benefiting from the ability to comprehend ultra-high-resolution images and long context, IXC-2.5 exhibits strong performance on various video benchmarks for LVLMs.Webpage Generation.Pix2Codepresents an end-to-end solution for UI-to-code transformation leveraging CNNs and RNNs. This approach contends with the challenges posed by intricate visual encoding and extensive text decoding when applied to real-world UIs. In the sphere of recent advancements, works such as Sightseer, DCGen, and Design2Codehave employed large vision-language models trained on synthetic screenshot-HTML paired datasets like WebSight v0.1 or v0.2to facilitate HTML code generation. Nevertheless, the synthesized web page datasets have been critiqued for their simplicity and lack of diversity. These studies generally concentrate on the screenshot/sketch-to-code task. In contrast, our IXC-2.5 model extends these capabilities to include screenshot-to-code, instruction-aware webpage generation, and resume-to-homepage tasks. IXC-2.5 is trained using a combination of high-quality synthesized and real-world web data. Furthermore, IXC-2.5 is proficient in generating JavaScript code, thereby enabling the development of interactive front-end webpages.Preference Alignment.Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from AI Feedback (RLAIF) have shown great promise in aligning LLMs across various domains, including improving logical reasoning and generating helpful and harmless outputs. The typical approach involves training a reward model using human or AI preference data and fine-tuning the LLM to maximize the expected reward function with optimization algorithms like Proximal Policy Optimization (PPO) . Alternatively, Direct Preference Optimization (DPO) and the following workshave emerged as leading methods that implicitly represent the reward score and eliminate the need for a separate reward model. Building on the success of RLHF and RLAIF in LLMs, recent studies have successfully extended RLHF/RLAIF algorithms for multimodal LVLMsto reduce hallucination. In this work, we investigate the application of preference alignment techniques to the text-image article composition task, with a focus on generating high-quality and stable response results.
body
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
2Related Works
2407.03320
3MethodFigure 5:Frameworkof IXC-2.5 that supports the multi-modal inputs, including text, single/multiple images, and videos.3.1Model ArchitectureThe model architecture of InternLM-XComposer-2.5 (IXC-2.5 in the following for simplicity)  mainly follows the design of InternLM-XComposer2and InternLM-XComposer2-4KHD(IXC2 and IXC2-4KHD for simplicity) , including a light-weight Vision Encoder OpenAI ViT-L/14, Large Language Model InternLM2-7B, and Partial LoRAfor efficient alignment. We recommend the readers to the IXC2 and IXC2-4KHD papers for more details.3.2Multi-modal InputOur IXC-2.5 supports diverse input modalities, including text, single/multiple images, and videos. As showin in Figure5, a Unified Dynamic Image Partition strategy is adopted for both videos and multiple images with any resolutions and aspect ratios.Image Processing.We mainly follow the Dynamic Image Partition and Global-Local Format design used in IXC2-4KHDwith a few modifications. For the vision encoder, we reuse the ViT ofresolution used in IXC2 and further increase its resolution to, so that each sub-image has 400 tokens.For the high-resolution strategy, we unify the different strategies used in the IXC-4KHD into a scaled identity strategy. Given a maximum partition number, the imagewith sizeis resized and padded to the new imagewith size. This process is subject to the following constraints:(1) (2) (3) whereis the scale factor,andrepresent the number of patches in each row and column, respectively.For multi-image input, we assign an index to each image likeIMAGEi,and format the image and text in an interleaved format.Video Processing.We sample frames from the given video and concatenate them along the short side of the frame, leading to a high-resolution image. The frame index is also written in the image to provide the temporal relation.Audio Processing.IXC-2.5 web demo supports audio input and output using open-source tools. For audio input, we employ Whisperto transcribe audio into text. For audio output, we utilize MeloTTSto convert the text back into audio.3.3Pre-trainingTaskDatasetGeneral Semantic AlignmentShareGPT4V-PT, COCO, Nocaps, TextCaps, LAION, SBU, CC 3MALLaVAWorld Knowledge AlignmentConcept DataVision Capability EnhancementWanJuan, Flicker, MMC-Inst, RCTW-17, CTW, LSVT, ReCTs, ArTTable 1:Datasets used for Pre-Training. The data are collected from diverse sources for the three objectives.During the pre-training phase, the LLM (InternLM2-7B)  is frozen while both the vision encoder and Partial LoRAare fine-tuned to align the visual tokens with the LLM.
The data used for pre-training is shown in Table1.In practice, we employ the CLIP ViT-L-14-490from IXC2 as the vision encoder and further increase its resolution to. For the Unified Dynamic Image Partition strategy, we set the maximum numberfor the pertaining.
For the Partial LoRA, we set a rank offor all the linear layers in the LLM decoder block.
Our training process involves a batch size of 4096 and spans across 2 epochs. The learning rate linearly increases towithin the firstof the training steps. Following this, it decreases toaccording to a cosine decay strategy.
To preserve the original knowledge of the vision encoder, we apply a layer-wise learning rate (LLDR)  decay strategy, and the decay factor is set to.3.4Supervised Fine-tuningTaskDatasetCaptionShareGPT4V, COCO, NocapsGeneral QAVQAv2, GQA, OK-VQAVD, RD, VSR, ALLaVA-QAMulti-Turn QAMMDUScience QAAI2D, SQA, TQA, IconQAChart QADVQA, ChartQA, ChartQA-AUGMath QAMathQA, Geometry3K, TabMWP,CLEVR-MATH, SuperWorld Knowledge QAA-OKVQA, KVQA, ViQuAEOCR QATextVQA, OCR-VQA, ST-VQAHD-OCR QAInfoVQA, DocVQA, TabFact,WTQ, DeepForm, Visual MRCVideoShareGPT4Video, ActivityNetConversationLLaVA-150k, LVIS-Instruct4VShareGPT-en&zh, InternLM-ChatTable 2:Datasets used for Supervised Fine-Tuning. We collect data from diverse sources to empower the model with different capabilities.We fine-tune the model with data listed in Table2. The maximum numberof the Unified Dynamic Image Partition strategy isto handle extremely large images and videos. For video datasets, the IXC-2.5 is trained with large images concatenated by at most 64 frames.
The largest training context is set to a 24,000 context window size, where the MMDUdataset can achieve this limitation.
In practice, we jointly train all the components with a batch size of 2048 over 4000 steps. Data from multiple sources are sampled in a weighted manner, with the weights based on the number of data from each source.
The maximum learning rate is set to, and each component has its own unique learning strategy.
For the vision encoder, we set the LLDR to, which aligns with the pretraining strategy.
For the LLM, we employ a fixed learning rate scale factor of. This slows down the update of the LLM, achieving a balance between preserving its original capabilities and aligning it with vision knowledge.3.5Webpage GenerationWe enhance the capabilities of the IXC-2.5 to include automated webpage generation. Specifically, the IXC-2.5 is now equipped to autonomously construct web pages, utilizing HTML, CSS, and JavaScript, based on input in the form of a visual screenshot, a set of free-form instructions, or a resume document. Current open-source general-purpose large language models frequently demonstrate suboptimal performance in generating HTML and CSS relative to their proficiency in natural language generation. To address this limitation, we propose training the screenshot-to-code task using extensive datasets from WebSight v0.1/v0.2, and Stack v2. Subsequently, we fine-tune the model with a smaller, meticulously crafted dataset consisting of instruction-aware webpage generation and personal page generation examples.Screenshot-to-code.In addition to the WebSightdatasets, we preprocess the HTML and CSS code from the Stack v2dataset to facilitate screenshot-to-code training. Initially, we combine the CSS and HTML code into a single file. Subsequently, we remove all comments, JavaScript code, and external links. Furthermore, we eliminate any CSS styles that are not referenced by the HTML code. We convert all files into screenshots, subsequently discarding those that did not render successfully. The remaining screenshots are then processed using the IXC2-4KHDmodel to assess the quality of the web pages. Following the exclusion of low-quality web pages, we retained a final set of three remaining about 250,000 high-quality web pages.We conduct training on the LoRA model utilizing the three aforementioned datasets. The LoRA rank is set to 512. The training protocol employs a batch size of 512 and is executed over a single epoch. Initially, the learning rate is incremented linearly towithin the firstof the training iterations. Subsequently, the learning rate decreases tofollowing a cosine decay schedule.Instruction-aware Webpage Generation.A pivotal attribute of large language models lies in their capability to adhere to human instructions. To facilitate web page generation based on freeform instructions, we propose constructing data through querying closed-source large language models. Specifically, we utilize GPT-4 to generate diverse instructions and concepts for web page creation, encompassing elements such as type, style, and layout. Subsequently, these instructions are harnessed to query Claude-3-sonnetfor the actual web page generation process. This approach results in 18,000 high-quality, instruction-aware samples. Additionally, we employ Tailwind CSS instead of traditional CSS, given its succinct nature.Resume-to-homepage.In addition to instruction-aware webpage generation, we introduce a more practical task. Specifically, given a resume, the model is designed to generate a personal homepage. This homepage not only encapsulates the information present in the resume but also presents it with a well-structured and visually appealing format, improving both content organization and aesthetic layout. To generate corresponding datasets, we propose an idea-resume-homepage data generation pipeline. Initially, we leverage GPT-4 to produce resume ideas tailored for diverse personas, such as researchers, students, and engineers. GPT-4 is tasked with generating these resumes in markdown format based on the provided ideas. Upon obtaining the generated resumes, we then prompt Claude-3-sonnetto create corresponding homepages from these resumes. To enhance the interactivity of these webpages, Claude-3-sonnet is also utilized to generate JavaScript events based on the HTML code. In total, we have constructed a dataset comprising 2,000 samples.Upon constructing the dataset for instruction-aware webpage generation and resume-to-homepage, we subsequently fine-tuned the LoRA model for 10 epochs. All other experimental settings were maintained consistent with those employed during the screenshot-to-code training phase.3.6Article ComposingGenerating high-quality text-image articles (e.g., poetry, novels, short stories, and essays)  is a crucial capability for AI assistants, with various applications in daily life, including education and entertainment.
Building upon the IXC-2.5 SFT modelin Section3.4, we enhance creative writing capabilities for generating high-quality text-image articles.
However, collecting high-quality text-image articles is a rare and expensive endeavor.
Direct fine-tuning on scarce instruction data can lead to unstable responses from LVLMs in most cases.
To overcome these challenges, we propose a scalable pipeline that integrates supervised fine-tuning, reward modeling, preference data collection, and DPO alignment for high-quality and stable article generation.Supervised Fine-tuning.We begin with the SFT model(Section3.4)  and a collection of 5,000 instruction tuning data samplesfrom IXC2, focused on article writing.
Due to the limited scale of the instruction data, we use the SFT model to rewrite the original prompts using the Chain-of-Thought (CoT)  technique, generating step-by-step prompts to supplement the instruction tuning data as augmented data.
We observe that the SFT model is more effective in generating long-form responses when using these augmented prompts.
We then train the initial modelon the augmented instruction tuning data via LoRAwith the rank ofand get the modelto establish a starting point of our alignment pipeline.MVBenchMLVUMMEMMB∗1Temp∗2DocChartInfoTextOCRWTQDeepVisualTabVideoVideoCompassVQAQAVQAVQABenchFormMRCFactOpen-SourceVideoChatInternVLLIVAInternVLQwen-VLInternVLInternVLInternVLInternVLGLM-4vDocOwlDocOwlDocOwlDocOwlPrevious SOTA2-7B1.5-26B34B1.5-26B7B1.5-26B1.5-26B1.5-26B1.5-26B9B1.5-8B1.5-8B1.5-8B1.5-8BPerformance60.450.459.042.058.490.983.872.580.677.640.668.8246.480.2Closed-source APIGPT-4V43.549.259.956.0—88.478.575.178.051.6————Gemini-Pro——75.049.370.688.174.175.274.668.0————IXC-2.5-7B69.158.855.846.967.190.982.269.978.269.053.671.2307.585.2Table 3:Comparison with closed-source APIs and previous open-source SOTAson Video Benchmarks and Structural High-resolution Benchmarks.The best results areboldand the second-best results areunderlined.∗1We scale the score fromtofor easier understanding.∗2We report the determinism part (MCQA, Y/N, Caption Match)  of TempCompass as the evaluation using GPT-3.5 is not stable.MMDUMMStarRealWQAMathVistaAI2DMMMUMMEMMBMMBCNMMB1.1SEEDIMM-VetHallBOpen-SourceLLaVa1.6InternVLWeMMWeMMInternVL360VLInternVLInternVL1.5InternVL1.5InternVL1.5WeMMGLM-4vWeMMPrevious SOTA8B1.5-26B8B8B1.5-26B70B1.5-26B1.5-26B1.5-26B1.5-26B8B14B8BPerformance42.857.168.154.980.653.42,189.682.380.779.775.958.047.5Closed-source APIGPT-4V66.357.168.047.875.556.81,926.581.380.279.869.156.846.5Gemini-Pro—42.664.145.870.247.91,933.373.974.373.970.759.245.2IXC-2.5-7B56.659.967.863.881.542.92,229.082.280.879.475.451.742.4Table 4:Comparison with closed-source APIs and previous open-source SOTAson Multi-Turn Multi-Image Dialog and General Visual QA Benchmarks.The best results areboldand the second-best results areunderlined.Preference Data Collection.We use the fine-tuned modelto generate diverse responses for each prompt in the augmented instruction tuning data, using different random seeds. This yields a collection of 80,000 prompt-response pairs. Next, we employ the GPT-4o model to label 2,000 responses with chosen or rejected decisions and give the reasons, which serve as our reward modeling data. We then train a reward model, sharing the same architecture of, on the reward modeling data. The reward model is used to make the chosen or rejected prediction on the remaining prompt-response pairs. These selected responses are then used to construct the pair data, while,andrefer to the prompt, chosen response and rejected response, respectively. Ultimately, we obtain a total of 30,000 preference datafor DPOalignment.DPO Alignment.We use the DPO algorithm to update the SFT modelon target policy from the preference data:(4) In practice, we use LoRA with a rank ofto get the DPO model. We observe that our model tends to prioritize minimizing the likelihood of dis-preferred responsesover maximizing the likelihood of preferred responsesto avoid generating inappropriate or low-quality content.In summary, our scalable pipeline consists of three primary components. First, we address the challenge of limited instruction tuning data by re-writing original prompts into augmented prompts. Next, we generate diverse responses using different random seeds, enabling the exploration of various creative possibilities. Finally, we apply the DPO algorithm to the chosen and rejected responses to refine our model’s performance. Through our pipeline, our model is capable of generating high-quality articles.Block-MatchTextPositionColorCLIPAverageClosed-source APIGPT-4V85.897.480.573.386.984.8Gemini-Pro80.294.672.366.283.979.4Open-sourceWebSight VLM-8B55.986.677.379.486.577.1CogAgent-Chat-18B7.118.113.313.075.525.4Design2Code-18B78.596.474.367.085.880.4IXC-2.5-7B81.995.680.980.886.585.1Table 5:Screenshot-to-code. Comparison with closed-source APIs and open-source models onDesign2Code benchmark. The best results areboldand the second-best results areunderlined.
body
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
3Method
2407.03320
4ExperimentsIn this section, we validate the benchmark performance of our InternLM-XComposer-2.5 (IXC-2.5)  after supervised fine-tuning.4.1LVLM Benchmark results.In Table3and Table4, we compare our IXC-2.5 on a list of benchmarks with both closed-source APIs and SOTA open-source LVLMs (with comparable model size) . Here we report video understanding results on MVBench, MLVU, MME-Video, MMBench-Video, TempCompass.
For Structural High-resolution understanding, we report results on DocVQA, ChartQA, InfographicVQA, TextVQA, OCRBench, DeepForm, WikiTableQuestion (WTQ) , Visual MRC, and TabFact.
For general visual question answering, we report results on MMStar, RealWorldQA,
MathVista, MMMU, AI2D, MME, MMBench (MMB) , MMBench-Chinese (MMBCN) , MMBench-v1.1 (MMBv1.1) , SEED-Bench Image Part (SEEDI) , MM-Vet, HallusionBench (HallB) . For Multi-True Multi-Image dialogue, we evaluate IXC-2.5 on MMDUbenchmark. For webpage crafting, we report a subtask screenshot-to-codesince benchmarks for others are not available in the community.The evaluation is mainly conducted on the OpenCompass VLMEvalKitfor the unified reproduction of the results.Comparison on Video Understanding Benchmarks.As demonstrated in Table3, IXC-2.5 exhibits competitive performance on fine-grained video understanding tasks, outperforming open-source models on 4 of the 5 benchmarks and being on par with Closed-Source APIs. For example, IXC-2.5 reaches 69.1 on the MVBench,higher than the previous SOTA method VideoChat2-7B and outperforms GPT-4V with. For the recent challenging MMBench-Video, IXC-2.5 reaches the SOTA performance on open-source models and performs close to Gemini-Pro.Comparison on Structural High-resolution Benchmarks.Benefiting from the unified image partition strategy, IXC-2.5 could handle diverse kinds of images. Table3reports its performance on several structural high-resolution benchmarks. IXC-2.5 with only 7B parameters performs on par with the current large open-source LVLMs and close-source APIs. For example, IXC-2.5 getson the DocVQA test set, the same as InternVL-1.5 which has nearlyparameters. For the highly structured form and table understanding tasks, IXC-2.5 outperforms DocOwl 1.5-8B with,,on WikiTableQuestion, DeepForm and TableFace respectively.Comparison on Multi-Image Multi-Turn Benchmarks.IXC-2.5 is capable of taking multiple images as input and conducting multi-round free-form dialogue based on them. We evaluate it quantitatively on the newly proposed MMDU benchmark. As shown in Table4, the IXC-2.5 model demonstrates superior performance, outperforming the previous SOTA open-source model by a significant margin of 13.8%. This notable improvement highlights the effectiveness of our approach in advancing the capabilities of multi-image and multi-turn understanding.Comparison on General Visual QA Benchmarks.IXC-2.5 is designed as a general LVLM to handle diverse multi-modal tasks. Here we report its performance on general visual QA benchmarks. As shown in Table4, the IXC-2.5 shows superb performance on these benchmarks and on par with current large open-source LVLMs and closed-source APIs. For example, IXC-2.5 getson the challenging MMStar and outperforms GPT-4V and Gemini-Pro. On the RealWorldQA, IXC-2.5 also performs better than Gemini-Pro and close to GPT-4V.Comparison on Screenshot-to-code Benchmark.Table5presents the comparison results on the Design2Codebenchmark that assesses the ability to translate visual design into code implementation. Our IXC-2.5 even surpasses the GPT-4v on average performance, which highlights the potential of IXC-2.5 to excel in bridging the gap between visual design and code implementation.
body
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
4Experiments
2407.03320
5ConclusionWe have introduced InternLM-XComposer-2.5 (IXC-2.5) , a cutting-edge Large Vision-Language Model (LVLM)  boasting long-contextual input and output capabilities that enable advanced features such as ultra-high resolution image understanding, fine-grained video understanding, multi-turn multi-image dialogue, webpage generation, and article composing.
Our comprehensive experiments demonstrate that IXC-2.5 achieves competitive performance, remarkably, with a relatively modest 7B Large Language Model (LLM)  backend.Our model sets out a promising research direction that can extend to a more contextual multi-modal environment, including long-context video understanding (e.g., long movies)  and long-context interaction history, to better assist humans in real-world applications.Acknowledgements:We deeply express our gratitude to Prof. Chao Zhang from Tsinghua University for suggestions about audio models and tools.
conclusion
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
5Conclusion