<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2406.17711] Data curation via joint example selection further accelerates multimodal learning</title><meta property="og:description" content="Data curation is an essential component of large-scale pretraining.
In this work, we demonstrate that jointly
selecting
batches of data is more effective for learning than selecting examples independently. Multimodal c…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Data curation via joint example selection further accelerates multimodal learning">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Data curation via joint example selection further accelerates multimodal learning">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2406.17711">

<!--Generated on Fri Jul  5 21:40:30 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Data curation via joint example selection 
<br class="ltx_break">further accelerates multimodal learning</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Talfan Evans<sup id="id6.4.id1" class="ltx_sup">∗</sup>  Nikhil Parthasarathy<sup id="id7.5.id2" class="ltx_sup">∗</sup>  Hamza Merzic  Olivier J. Hénaff<sup id="id8.6.id3" class="ltx_sup">∗</sup> 
<br class="ltx_break"> 
<br class="ltx_break">Google DeepMind, London, UK
<br class="ltx_break">
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id5.2" class="ltx_p">Data curation is an essential component of large-scale pretraining.
In this work, we demonstrate that jointly
selecting
<span id="id5.2.1" class="ltx_text ltx_font_italic">batches</span> of data is more effective for learning than selecting examples independently. Multimodal contrastive objectives expose the dependencies between data and thus naturally yield criteria for measuring the <span id="id5.2.2" class="ltx_text ltx_font_italic">joint learnability</span> of a batch. We derive a simple and tractable algorithm for selecting such batches, which significantly accelerate training beyond individually-prioritized data points.
As performance improves by selecting from larger super-batches, we also leverage recent advances in model approximation to reduce the associated computational overhead. As a result, our approach—multimodal contrastive learning with joint example selection (JEST)—surpasses state-of-the-art models with up to 13<math id="id4.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="id4.1.m1.1a"><mo id="id4.1.m1.1.1" xref="id4.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="id4.1.m1.1b"><times id="id4.1.m1.1.1.cmml" xref="id4.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="id4.1.m1.1c">\times</annotation></semantics></math> fewer iterations and 10<math id="id5.2.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="id5.2.m2.1a"><mo id="id5.2.m2.1.1" xref="id5.2.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="id5.2.m2.1b"><times id="id5.2.m2.1.1.cmml" xref="id5.2.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="id5.2.m2.1c">\times</annotation></semantics></math> less computation. Essential to the performance of JEST is the ability to steer the data selection process towards the distribution of smaller, well-curated datasets via pretrained reference models, exposing the level of data curation as a new dimension for neural scaling laws.</p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para ltx_noindent">
<p id="S1.p1.1" class="ltx_p">Data quality is an essential driver of performance for large-scale pretraining.
Whether in language <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib19" title="" class="ltx_ref">19</a>]</cite>, vision <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>, or multimodal modeling <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib1" title="" class="ltx_ref">1</a>, <a href="#bib.bib22" title="" class="ltx_ref">22</a>, <a href="#bib.bib32" title="" class="ltx_ref">32</a>]</cite>, training on well-curated datasets has
consistently demonstrated that strong performance can be achieved with significantly less data.
However, current data pipelines rely heavily on manual curation, which is difficult and expensive to scale. In contrast, model-based data curation <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib31" title="" class="ltx_ref">31</a>, <a href="#bib.bib33" title="" class="ltx_ref">33</a>]</cite>, which uses features of the model being trained to select high quality data, holds promise for improving the slow, power-law scaling of large-scale pretraining across modalities, both in theory <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib47" title="" class="ltx_ref">47</a>]</cite> and in practice <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>.</p>
</div>
<div id="S1.p2" class="ltx_para ltx_noindent">
<p id="S1.p2.1" class="ltx_p">Existing methods apply curation at the level of individual data points <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib12" title="" class="ltx_ref">12</a>, <a href="#bib.bib42" title="" class="ltx_ref">42</a>]</cite>.
Yet the quality of a batch is also a function of its composition, in addition to the summed quality of its data points considered independently.
In computer vision, hard negatives (i.e. clusters of points which lie close to one another but contain different labels) have been found to provide a more effective learning signal than trivially solvable ones <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>, <a href="#bib.bib21" title="" class="ltx_ref">21</a>, <a href="#bib.bib34" title="" class="ltx_ref">34</a>, <a href="#bib.bib41" title="" class="ltx_ref">41</a>, <a href="#bib.bib45" title="" class="ltx_ref">45</a>, <a href="#bib.bib50" title="" class="ltx_ref">50</a>, <a href="#bib.bib53" title="" class="ltx_ref">53</a>]</cite>.
In this work we seek to generalize this notion by asking whether model-based data-selection criteria applied to batches of data can accelerate learning beyond what is possible by selecting examples independently.</p>
</div>
<div id="S1.p3" class="ltx_para ltx_noindent">
<p id="S1.p3.1" class="ltx_p">In multimodal learning, contrastive objectives directly expose the interactions between examples in a batch. We therefore derive a simple and tractable algorithm for joint example selection—JEST—which efficiently selects relevant ‘sub-batches’ of data from much larger ‘super-batches’ given their model-based scores.
When scoring batches with a pretrained reference model (i.e. <span id="S1.p3.1.1" class="ltx_text ltx_font_italic">easy-reference</span>), JEST accelerates learning relative to uniform batch selection, significantly improving on independent example selection using the same reference (as in CLIPScore <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>]</cite>).
When scoring batches according to their <span id="S1.p3.1.2" class="ltx_text ltx_font_italic">learnability</span>, which also takes into account the online model loss <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib33" title="" class="ltx_ref">33</a>]</cite>, JEST improves further, matching the performance of state-of-the-art models <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite> with up to 13<math id="S1.p3.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S1.p3.1.m1.1a"><mo id="S1.p3.1.m1.1.1" xref="S1.p3.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.p3.1.m1.1b"><times id="S1.p3.1.m1.1.1.cmml" xref="S1.p3.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.p3.1.m1.1c">\times</annotation></semantics></math> fewer training iterations.</p>
</div>
<div id="S1.p4" class="ltx_para ltx_noindent">
<p id="S1.p4.2" class="ltx_p">Discovering highly learnable batches requires sifting through much larger super-batches of raw data.
We make learnability scoring of large batches tractable by leveraging recent advances in online model approximation,
which reduce computation while still providing useful predictions <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib4" title="" class="ltx_ref">4</a>, <a href="#bib.bib29" title="" class="ltx_ref">29</a>, <a href="#bib.bib55" title="" class="ltx_ref">55</a>]</cite>. By training a single model at multiple resolutions in parallel, we efficiently apply the model for scoring large super-batches, find their most learnable sub-batch, and spend more valuable computation for learning on them. Thanks to savings in both learning and example scoring, we reduce the overhead of scoring from 133% to 10% additional FLOPs while maintaining significant gains in training efficiency. This approximate scoring framework—Flexi-JEST—produces state-of-the-art models with 11<math id="S1.p4.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S1.p4.1.m1.1a"><mo id="S1.p4.1.m1.1.1" xref="S1.p4.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.p4.1.m1.1b"><times id="S1.p4.1.m1.1.1.cmml" xref="S1.p4.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.1.m1.1c">\times</annotation></semantics></math> fewer iterations <span id="S1.p4.2.1" class="ltx_text ltx_font_italic">and</span> 10<math id="S1.p4.2.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S1.p4.2.m2.1a"><mo id="S1.p4.2.m2.1.1" xref="S1.p4.2.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.p4.2.m2.1b"><times id="S1.p4.2.m2.1.1.cmml" xref="S1.p4.2.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.2.m2.1c">\times</annotation></semantics></math> fewer FLOPs.</p>
</div>
<div id="S1.p5" class="ltx_para ltx_noindent">
<p id="S1.p5.1" class="ltx_p">Finally, we find that central to the performance of our framework is the ability to steer the curation process towards the distribution of smaller, well-curated datasets. This occurs naturally with the model-based selection criteria we consider through the concept of a pretrained reference model, which prioritizes examples
that most resemble the data it was trained on. Crucially, we find this process to enable strong <span id="S1.p5.1.1" class="ltx_text ltx_font_italic">data quality bootstrapping</span>: a reference model trained on a small curated dataset can effectively guide the curation of a much larger dataset, allowing the training of a model which strongly surpasses the quality of the reference model on many downstream tasks.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2406.17711/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="147" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>
<span id="S1.F1.9.1" class="ltx_text ltx_font_bold">Joint Example Selection accelerates multimodal pretraining.</span> Our JEST/JEST++ methods bootstrap from small, strongly curated datasets (Webli-curated/Webli-curated++) to actively curate web-scale datasets. Flexi-JEST++ uses variable patch sizing to reduce the cost of curation. <span id="S1.F1.10.2" class="ltx_text ltx_font_bold">Left</span>: Training with JEST matches the performance of the uniform 40B SigLIP baseline with up to 13<math id="S1.F1.3.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S1.F1.3.m1.1b"><mo id="S1.F1.3.m1.1.1" xref="S1.F1.3.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.F1.3.m1.1c"><times id="S1.F1.3.m1.1.1.cmml" xref="S1.F1.3.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.3.m1.1d">\times</annotation></semantics></math> fewer iterations. <span id="S1.F1.11.3" class="ltx_text ltx_font_bold">Middle</span>: Even when accounting for the cost of scoring, our best variant is almost 10<math id="S1.F1.4.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S1.F1.4.m2.1b"><mo id="S1.F1.4.m2.1.1" xref="S1.F1.4.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.F1.4.m2.1c"><times id="S1.F1.4.m2.1.1.cmml" xref="S1.F1.4.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.4.m2.1d">\times</annotation></semantics></math> more FLOP efficient. <span id="S1.F1.12.4" class="ltx_text ltx_font_bold">Right</span>: Comparison of JEST++/FlexiJEST++ (green) to prior methods (grey). Average accuracy is computed across 8 downstream tasks (left, middle; see Table <a href="#A1.T5" title="Table 5 ‣ A.7 Comparing approximation methods ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>), or ImageNet and COCO (right).  </figcaption>
</figure>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<div id="S2.p1" class="ltx_para ltx_noindent">
<p id="S2.p1.1" class="ltx_p"><span id="S2.p1.1.1" class="ltx_text ltx_font_bold">Offline curation: example-level data pruning.</span> Methods for collecting and filtering large-scale noisy image-text data initially focused on the quality of the textual captions <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>, <a href="#bib.bib9" title="" class="ltx_ref">9</a>, <a href="#bib.bib24" title="" class="ltx_ref">24</a>]</cite>, and proximity to high-quality reference datasets <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>, <a href="#bib.bib17" title="" class="ltx_ref">17</a>, <a href="#bib.bib52" title="" class="ltx_ref">52</a>]</cite>. Instead, model-based
filtering approaches use pretrained models (such as CLIP <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib38" title="" class="ltx_ref">38</a>]</cite> and BLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib27" title="" class="ltx_ref">27</a>]</cite>) as evaluation metrics for curating data via image-text alignment <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>, <a href="#bib.bib22" title="" class="ltx_ref">22</a>, <a href="#bib.bib32" title="" class="ltx_ref">32</a>]</cite>. Critically, all of these methods are applied independently across examples, which fails to account for the relevance of dependencies across examples in a batch.</p>
</div>
<div id="S2.p2" class="ltx_para ltx_noindent">
<p id="S2.p2.1" class="ltx_p"><span id="S2.p2.1.1" class="ltx_text ltx_font_bold">Offline curation: cluster-level data pruning.</span>
Other methods such as semantic redundancy reduction <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib1" title="" class="ltx_ref">1</a>, <a href="#bib.bib2" title="" class="ltx_ref">2</a>, <a href="#bib.bib47" title="" class="ltx_ref">47</a>]</cite> or core-set selection <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib7" title="" class="ltx_ref">7</a>, <a href="#bib.bib20" title="" class="ltx_ref">20</a>]</cite> have proposed to curate based on the marginal importance of data points given other data points in their vicinity. However these methods are based on a heuristic that is decoupled from the training objective. In contrast, our method enables joint-example selection that is specifically tailored to accelerating the contrastive pretraining objective function.</p>
</div>
<div id="S2.p3" class="ltx_para ltx_noindent">
<p id="S2.p3.1" class="ltx_p"><span id="S2.p3.1.1" class="ltx_text ltx_font_bold">Online data curation with model-based scoring.</span> Pre-filtering using the curation procedures described above can lead to large increases in data quality. However, fixed curation strategies do not take into account that the relevance of a training example can change over the course of learning, limiting their utility at scale <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite>. These concerns are addressed by online data curation methods <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>, <a href="#bib.bib30" title="" class="ltx_ref">30</a>, <a href="#bib.bib31" title="" class="ltx_ref">31</a>, <a href="#bib.bib33" title="" class="ltx_ref">33</a>]</cite>, which identify high-quality examples <span id="S2.p3.1.2" class="ltx_text ltx_font_italic">not yet learned by the model</span>. Our work generalizes these by applying model-based criteria to batch-level (rather than example-level) losses, and selecting data accordingly.</p>
</div>
<div id="S2.p4" class="ltx_para ltx_noindent">
<p id="S2.p4.1" class="ltx_p"><span id="S2.p4.1.1" class="ltx_text ltx_font_bold">Hard negative mining.</span> A long literature has described the efficiency gains afforded by choosing the right set of negative examples in classical metric-learning <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib5" title="" class="ltx_ref">5</a>, <a href="#bib.bib21" title="" class="ltx_ref">21</a>, <a href="#bib.bib34" title="" class="ltx_ref">34</a>, <a href="#bib.bib45" title="" class="ltx_ref">45</a>, <a href="#bib.bib50" title="" class="ltx_ref">50</a>, <a href="#bib.bib53" title="" class="ltx_ref">53</a>]</cite> as well as modern contrastive learning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib41" title="" class="ltx_ref">41</a>, <a href="#bib.bib49" title="" class="ltx_ref">49</a>]</cite>.
We generalize hard negative mining in two ways: 1)
we jointly mine for both positive and negative pairs by selecting entire batches and 2) we explore prioritizing <span id="S2.p4.1.2" class="ltx_text ltx_font_italic">learnable</span> negatives, which are hard for the learner but easy for a pretrained model.</p>
</div>
<div id="S2.p5" class="ltx_para ltx_noindent">
<p id="S2.p5.1" class="ltx_p"><span id="S2.p5.1.1" class="ltx_text ltx_font_bold">Model approximation.</span> Several works have demonstrated that smaller models can be used as proxies for much larger models for data selection <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib12" title="" class="ltx_ref">12</a>, <a href="#bib.bib15" title="" class="ltx_ref">15</a>, <a href="#bib.bib51" title="" class="ltx_ref">51</a>]</cite>. However, several techniques have recently been developed that allow inference-time trade-offs between computation and performance, allowing smaller models to be “embedded” without the need for separate training. For Vision Transformers <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib14" title="" class="ltx_ref">14</a>]</cite>, dropping patches <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite> or layers <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib55" title="" class="ltx_ref">55</a>]</cite>, or reducing token resolution <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib4" title="" class="ltx_ref">4</a>]</cite> produce characteristic trade-offs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib28" title="" class="ltx_ref">28</a>]</cite>. Our work is the first to use these techniques in the context of online data selection.</p>
</div>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Methods</h2>

<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Model-based batch-selection criteria</h3>

<div id="S3.SS1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS1.p1.3" class="ltx_p">We refer to the model which we are interested in training as the <span id="S3.SS1.p1.3.1" class="ltx_text ltx_font_italic">learner</span>. Assuming we have a “super-batch” <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{D}" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><ci id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">\mathcal{D}</annotation></semantics></math> (of size <math id="S3.SS1.p1.2.m2.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S3.SS1.p1.2.m2.1a"><mi id="S3.SS1.p1.2.m2.1.1" xref="S3.SS1.p1.2.m2.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m2.1b"><ci id="S3.SS1.p1.2.m2.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m2.1c">B</annotation></semantics></math>) examples to learn from, we wish to extract a sub-batch <math id="S3.SS1.p1.3.m3.5" class="ltx_Math" alttext="\mathcal{B}=\{{\bm{x}}_{i},i\in[1,...,b]\}\subset\mathcal{D}" display="inline"><semantics id="S3.SS1.p1.3.m3.5a"><mrow id="S3.SS1.p1.3.m3.5.5" xref="S3.SS1.p1.3.m3.5.5.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.3.m3.5.5.3" xref="S3.SS1.p1.3.m3.5.5.3.cmml">ℬ</mi><mo id="S3.SS1.p1.3.m3.5.5.4" xref="S3.SS1.p1.3.m3.5.5.4.cmml">=</mo><mrow id="S3.SS1.p1.3.m3.5.5.1.1" xref="S3.SS1.p1.3.m3.5.5.1.2.cmml"><mo stretchy="false" id="S3.SS1.p1.3.m3.5.5.1.1.2" xref="S3.SS1.p1.3.m3.5.5.1.2.cmml">{</mo><mrow id="S3.SS1.p1.3.m3.5.5.1.1.1" xref="S3.SS1.p1.3.m3.5.5.1.1.1.cmml"><mrow id="S3.SS1.p1.3.m3.5.5.1.1.1.1.1" xref="S3.SS1.p1.3.m3.5.5.1.1.1.1.2.cmml"><msub id="S3.SS1.p1.3.m3.5.5.1.1.1.1.1.1" xref="S3.SS1.p1.3.m3.5.5.1.1.1.1.1.1.cmml"><mi id="S3.SS1.p1.3.m3.5.5.1.1.1.1.1.1.2" xref="S3.SS1.p1.3.m3.5.5.1.1.1.1.1.1.2.cmml">𝒙</mi><mi id="S3.SS1.p1.3.m3.5.5.1.1.1.1.1.1.3" xref="S3.SS1.p1.3.m3.5.5.1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.SS1.p1.3.m3.5.5.1.1.1.1.1.2" xref="S3.SS1.p1.3.m3.5.5.1.1.1.1.2.cmml">,</mo><mi id="S3.SS1.p1.3.m3.4.4" xref="S3.SS1.p1.3.m3.4.4.cmml">i</mi></mrow><mo id="S3.SS1.p1.3.m3.5.5.1.1.1.2" xref="S3.SS1.p1.3.m3.5.5.1.1.1.2.cmml">∈</mo><mrow id="S3.SS1.p1.3.m3.5.5.1.1.1.3.2" xref="S3.SS1.p1.3.m3.5.5.1.1.1.3.1.cmml"><mo stretchy="false" id="S3.SS1.p1.3.m3.5.5.1.1.1.3.2.1" xref="S3.SS1.p1.3.m3.5.5.1.1.1.3.1.cmml">[</mo><mn id="S3.SS1.p1.3.m3.1.1" xref="S3.SS1.p1.3.m3.1.1.cmml">1</mn><mo id="S3.SS1.p1.3.m3.5.5.1.1.1.3.2.2" xref="S3.SS1.p1.3.m3.5.5.1.1.1.3.1.cmml">,</mo><mi mathvariant="normal" id="S3.SS1.p1.3.m3.2.2" xref="S3.SS1.p1.3.m3.2.2.cmml">…</mi><mo id="S3.SS1.p1.3.m3.5.5.1.1.1.3.2.3" xref="S3.SS1.p1.3.m3.5.5.1.1.1.3.1.cmml">,</mo><mi id="S3.SS1.p1.3.m3.3.3" xref="S3.SS1.p1.3.m3.3.3.cmml">b</mi><mo stretchy="false" id="S3.SS1.p1.3.m3.5.5.1.1.1.3.2.4" xref="S3.SS1.p1.3.m3.5.5.1.1.1.3.1.cmml">]</mo></mrow></mrow><mo stretchy="false" id="S3.SS1.p1.3.m3.5.5.1.1.3" xref="S3.SS1.p1.3.m3.5.5.1.2.cmml">}</mo></mrow><mo id="S3.SS1.p1.3.m3.5.5.5" xref="S3.SS1.p1.3.m3.5.5.5.cmml">⊂</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.3.m3.5.5.6" xref="S3.SS1.p1.3.m3.5.5.6.cmml">𝒟</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.3.m3.5b"><apply id="S3.SS1.p1.3.m3.5.5.cmml" xref="S3.SS1.p1.3.m3.5.5"><and id="S3.SS1.p1.3.m3.5.5a.cmml" xref="S3.SS1.p1.3.m3.5.5"></and><apply id="S3.SS1.p1.3.m3.5.5b.cmml" xref="S3.SS1.p1.3.m3.5.5"><eq id="S3.SS1.p1.3.m3.5.5.4.cmml" xref="S3.SS1.p1.3.m3.5.5.4"></eq><ci id="S3.SS1.p1.3.m3.5.5.3.cmml" xref="S3.SS1.p1.3.m3.5.5.3">ℬ</ci><set id="S3.SS1.p1.3.m3.5.5.1.2.cmml" xref="S3.SS1.p1.3.m3.5.5.1.1"><apply id="S3.SS1.p1.3.m3.5.5.1.1.1.cmml" xref="S3.SS1.p1.3.m3.5.5.1.1.1"><in id="S3.SS1.p1.3.m3.5.5.1.1.1.2.cmml" xref="S3.SS1.p1.3.m3.5.5.1.1.1.2"></in><list id="S3.SS1.p1.3.m3.5.5.1.1.1.1.2.cmml" xref="S3.SS1.p1.3.m3.5.5.1.1.1.1.1"><apply id="S3.SS1.p1.3.m3.5.5.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.3.m3.5.5.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.3.m3.5.5.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.3.m3.5.5.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS1.p1.3.m3.5.5.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.3.m3.5.5.1.1.1.1.1.1.2">𝒙</ci><ci id="S3.SS1.p1.3.m3.5.5.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p1.3.m3.5.5.1.1.1.1.1.1.3">𝑖</ci></apply><ci id="S3.SS1.p1.3.m3.4.4.cmml" xref="S3.SS1.p1.3.m3.4.4">𝑖</ci></list><list id="S3.SS1.p1.3.m3.5.5.1.1.1.3.1.cmml" xref="S3.SS1.p1.3.m3.5.5.1.1.1.3.2"><cn type="integer" id="S3.SS1.p1.3.m3.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1">1</cn><ci id="S3.SS1.p1.3.m3.2.2.cmml" xref="S3.SS1.p1.3.m3.2.2">…</ci><ci id="S3.SS1.p1.3.m3.3.3.cmml" xref="S3.SS1.p1.3.m3.3.3">𝑏</ci></list></apply></set></apply><apply id="S3.SS1.p1.3.m3.5.5c.cmml" xref="S3.SS1.p1.3.m3.5.5"><subset id="S3.SS1.p1.3.m3.5.5.5.cmml" xref="S3.SS1.p1.3.m3.5.5.5"></subset><share href="#S3.SS1.p1.3.m3.5.5.1.cmml" id="S3.SS1.p1.3.m3.5.5d.cmml" xref="S3.SS1.p1.3.m3.5.5"></share><ci id="S3.SS1.p1.3.m3.5.5.6.cmml" xref="S3.SS1.p1.3.m3.5.5.6">𝒟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.3.m3.5c">\mathcal{B}=\{{\bm{x}}_{i},i\in[1,...,b]\}\subset\mathcal{D}</annotation></semantics></math> that is maximally relevant for learning. Prioritized sampling <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib31" title="" class="ltx_ref">31</a>, <a href="#bib.bib43" title="" class="ltx_ref">43</a>]</cite> performs this by scoring individual examples, then sampling in proportion to these scores. In this work we instead score entire sub-batches, and sample according to these batch-level scores. We consider model-based scoring functions, which use the losses from the learner model and/or pretrained <span id="S3.SS1.p1.3.2" class="ltx_text ltx_font_italic">reference</span> models.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para ltx_noindent">
<p id="S3.SS1.p2.3" class="ltx_p"><span id="S3.SS1.p2.3.1" class="ltx_text ltx_font_bold">Hard learner.</span> An intuitive heuristic would be to prioritize batches <math id="S3.SS1.p2.1.m1.1" class="ltx_Math" alttext="\mathcal{B}" display="inline"><semantics id="S3.SS1.p2.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p2.1.m1.1.1" xref="S3.SS1.p2.1.m1.1.1.cmml">ℬ</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.m1.1b"><ci id="S3.SS1.p2.1.m1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1">ℬ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.m1.1c">\mathcal{B}</annotation></semantics></math> that have a high loss under the learner with parameters <math id="S3.SS1.p2.2.m2.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="S3.SS1.p2.2.m2.1a"><mi id="S3.SS1.p2.2.m2.1.1" xref="S3.SS1.p2.2.m2.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.2.m2.1b"><ci id="S3.SS1.p2.2.m2.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.2.m2.1c">\theta</annotation></semantics></math>: <math id="S3.SS1.p2.3.m3.2" class="ltx_Math" alttext="s^{\textrm{hard}}(\mathcal{B}|\theta)=\ell(\mathcal{B}|\theta)" display="inline"><semantics id="S3.SS1.p2.3.m3.2a"><mrow id="S3.SS1.p2.3.m3.2.2" xref="S3.SS1.p2.3.m3.2.2.cmml"><mrow id="S3.SS1.p2.3.m3.1.1.1" xref="S3.SS1.p2.3.m3.1.1.1.cmml"><msup id="S3.SS1.p2.3.m3.1.1.1.3" xref="S3.SS1.p2.3.m3.1.1.1.3.cmml"><mi id="S3.SS1.p2.3.m3.1.1.1.3.2" xref="S3.SS1.p2.3.m3.1.1.1.3.2.cmml">s</mi><mtext id="S3.SS1.p2.3.m3.1.1.1.3.3" xref="S3.SS1.p2.3.m3.1.1.1.3.3a.cmml">hard</mtext></msup><mo lspace="0em" rspace="0em" id="S3.SS1.p2.3.m3.1.1.1.2" xref="S3.SS1.p2.3.m3.1.1.1.2.cmml">​</mo><mrow id="S3.SS1.p2.3.m3.1.1.1.1.1" xref="S3.SS1.p2.3.m3.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p2.3.m3.1.1.1.1.1.2" xref="S3.SS1.p2.3.m3.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.p2.3.m3.1.1.1.1.1.1" xref="S3.SS1.p2.3.m3.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p2.3.m3.1.1.1.1.1.1.2" xref="S3.SS1.p2.3.m3.1.1.1.1.1.1.2.cmml">ℬ</mi><mo fence="false" id="S3.SS1.p2.3.m3.1.1.1.1.1.1.1" xref="S3.SS1.p2.3.m3.1.1.1.1.1.1.1.cmml">|</mo><mi id="S3.SS1.p2.3.m3.1.1.1.1.1.1.3" xref="S3.SS1.p2.3.m3.1.1.1.1.1.1.3.cmml">θ</mi></mrow><mo stretchy="false" id="S3.SS1.p2.3.m3.1.1.1.1.1.3" xref="S3.SS1.p2.3.m3.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS1.p2.3.m3.2.2.3" xref="S3.SS1.p2.3.m3.2.2.3.cmml">=</mo><mrow id="S3.SS1.p2.3.m3.2.2.2" xref="S3.SS1.p2.3.m3.2.2.2.cmml"><mi mathvariant="normal" id="S3.SS1.p2.3.m3.2.2.2.3" xref="S3.SS1.p2.3.m3.2.2.2.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p2.3.m3.2.2.2.2" xref="S3.SS1.p2.3.m3.2.2.2.2.cmml">​</mo><mrow id="S3.SS1.p2.3.m3.2.2.2.1.1" xref="S3.SS1.p2.3.m3.2.2.2.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p2.3.m3.2.2.2.1.1.2" xref="S3.SS1.p2.3.m3.2.2.2.1.1.1.cmml">(</mo><mrow id="S3.SS1.p2.3.m3.2.2.2.1.1.1" xref="S3.SS1.p2.3.m3.2.2.2.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p2.3.m3.2.2.2.1.1.1.2" xref="S3.SS1.p2.3.m3.2.2.2.1.1.1.2.cmml">ℬ</mi><mo fence="false" id="S3.SS1.p2.3.m3.2.2.2.1.1.1.1" xref="S3.SS1.p2.3.m3.2.2.2.1.1.1.1.cmml">|</mo><mi id="S3.SS1.p2.3.m3.2.2.2.1.1.1.3" xref="S3.SS1.p2.3.m3.2.2.2.1.1.1.3.cmml">θ</mi></mrow><mo stretchy="false" id="S3.SS1.p2.3.m3.2.2.2.1.1.3" xref="S3.SS1.p2.3.m3.2.2.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.3.m3.2b"><apply id="S3.SS1.p2.3.m3.2.2.cmml" xref="S3.SS1.p2.3.m3.2.2"><eq id="S3.SS1.p2.3.m3.2.2.3.cmml" xref="S3.SS1.p2.3.m3.2.2.3"></eq><apply id="S3.SS1.p2.3.m3.1.1.1.cmml" xref="S3.SS1.p2.3.m3.1.1.1"><times id="S3.SS1.p2.3.m3.1.1.1.2.cmml" xref="S3.SS1.p2.3.m3.1.1.1.2"></times><apply id="S3.SS1.p2.3.m3.1.1.1.3.cmml" xref="S3.SS1.p2.3.m3.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p2.3.m3.1.1.1.3.1.cmml" xref="S3.SS1.p2.3.m3.1.1.1.3">superscript</csymbol><ci id="S3.SS1.p2.3.m3.1.1.1.3.2.cmml" xref="S3.SS1.p2.3.m3.1.1.1.3.2">𝑠</ci><ci id="S3.SS1.p2.3.m3.1.1.1.3.3a.cmml" xref="S3.SS1.p2.3.m3.1.1.1.3.3"><mtext mathsize="70%" id="S3.SS1.p2.3.m3.1.1.1.3.3.cmml" xref="S3.SS1.p2.3.m3.1.1.1.3.3">hard</mtext></ci></apply><apply id="S3.SS1.p2.3.m3.1.1.1.1.1.1.cmml" xref="S3.SS1.p2.3.m3.1.1.1.1.1"><csymbol cd="latexml" id="S3.SS1.p2.3.m3.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p2.3.m3.1.1.1.1.1.1.1">conditional</csymbol><ci id="S3.SS1.p2.3.m3.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p2.3.m3.1.1.1.1.1.1.2">ℬ</ci><ci id="S3.SS1.p2.3.m3.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p2.3.m3.1.1.1.1.1.1.3">𝜃</ci></apply></apply><apply id="S3.SS1.p2.3.m3.2.2.2.cmml" xref="S3.SS1.p2.3.m3.2.2.2"><times id="S3.SS1.p2.3.m3.2.2.2.2.cmml" xref="S3.SS1.p2.3.m3.2.2.2.2"></times><ci id="S3.SS1.p2.3.m3.2.2.2.3.cmml" xref="S3.SS1.p2.3.m3.2.2.2.3">ℓ</ci><apply id="S3.SS1.p2.3.m3.2.2.2.1.1.1.cmml" xref="S3.SS1.p2.3.m3.2.2.2.1.1"><csymbol cd="latexml" id="S3.SS1.p2.3.m3.2.2.2.1.1.1.1.cmml" xref="S3.SS1.p2.3.m3.2.2.2.1.1.1.1">conditional</csymbol><ci id="S3.SS1.p2.3.m3.2.2.2.1.1.1.2.cmml" xref="S3.SS1.p2.3.m3.2.2.2.1.1.1.2">ℬ</ci><ci id="S3.SS1.p2.3.m3.2.2.2.1.1.1.3.cmml" xref="S3.SS1.p2.3.m3.2.2.2.1.1.1.3">𝜃</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.3.m3.2c">s^{\textrm{hard}}(\mathcal{B}|\theta)=\ell(\mathcal{B}|\theta)</annotation></semantics></math>, which has the desirable property of discarding trivial data. This heuristic has been proven to work for small, clean datasets <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib36" title="" class="ltx_ref">36</a>, <a href="#bib.bib47" title="" class="ltx_ref">47</a>]</cite> but tends to do more harm than good for larger, less curated datasets <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite> since it will also up-sample noisy examples.</p>
</div>
<div id="S3.SS1.p3" class="ltx_para ltx_noindent">
<p id="S3.SS1.p3.2" class="ltx_p"><span id="S3.SS1.p3.2.1" class="ltx_text ltx_font_bold">Easy reference.</span> In contrast, one could also choose to up-sample data that is “easy” (has low loss) for a pretrained <span id="S3.SS1.p3.2.2" class="ltx_text ltx_font_italic">reference</span> model with parameters <math id="S3.SS1.p3.1.m1.1" class="ltx_Math" alttext="\theta^{*}" display="inline"><semantics id="S3.SS1.p3.1.m1.1a"><msup id="S3.SS1.p3.1.m1.1.1" xref="S3.SS1.p3.1.m1.1.1.cmml"><mi id="S3.SS1.p3.1.m1.1.1.2" xref="S3.SS1.p3.1.m1.1.1.2.cmml">θ</mi><mo id="S3.SS1.p3.1.m1.1.1.3" xref="S3.SS1.p3.1.m1.1.1.3.cmml">∗</mo></msup><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.1.m1.1b"><apply id="S3.SS1.p3.1.m1.1.1.cmml" xref="S3.SS1.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p3.1.m1.1.1.1.cmml" xref="S3.SS1.p3.1.m1.1.1">superscript</csymbol><ci id="S3.SS1.p3.1.m1.1.1.2.cmml" xref="S3.SS1.p3.1.m1.1.1.2">𝜃</ci><times id="S3.SS1.p3.1.m1.1.1.3.cmml" xref="S3.SS1.p3.1.m1.1.1.3"></times></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.1.m1.1c">\theta^{*}</annotation></semantics></math>: <math id="S3.SS1.p3.2.m2.2" class="ltx_Math" alttext="s^{\textrm{easy}}(\mathcal{B}|\theta^{*})=-\ell(\mathcal{B}|\theta^{*})" display="inline"><semantics id="S3.SS1.p3.2.m2.2a"><mrow id="S3.SS1.p3.2.m2.2.2" xref="S3.SS1.p3.2.m2.2.2.cmml"><mrow id="S3.SS1.p3.2.m2.1.1.1" xref="S3.SS1.p3.2.m2.1.1.1.cmml"><msup id="S3.SS1.p3.2.m2.1.1.1.3" xref="S3.SS1.p3.2.m2.1.1.1.3.cmml"><mi id="S3.SS1.p3.2.m2.1.1.1.3.2" xref="S3.SS1.p3.2.m2.1.1.1.3.2.cmml">s</mi><mtext id="S3.SS1.p3.2.m2.1.1.1.3.3" xref="S3.SS1.p3.2.m2.1.1.1.3.3a.cmml">easy</mtext></msup><mo lspace="0em" rspace="0em" id="S3.SS1.p3.2.m2.1.1.1.2" xref="S3.SS1.p3.2.m2.1.1.1.2.cmml">​</mo><mrow id="S3.SS1.p3.2.m2.1.1.1.1.1" xref="S3.SS1.p3.2.m2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p3.2.m2.1.1.1.1.1.2" xref="S3.SS1.p3.2.m2.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.p3.2.m2.1.1.1.1.1.1" xref="S3.SS1.p3.2.m2.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p3.2.m2.1.1.1.1.1.1.2" xref="S3.SS1.p3.2.m2.1.1.1.1.1.1.2.cmml">ℬ</mi><mo fence="false" id="S3.SS1.p3.2.m2.1.1.1.1.1.1.1" xref="S3.SS1.p3.2.m2.1.1.1.1.1.1.1.cmml">|</mo><msup id="S3.SS1.p3.2.m2.1.1.1.1.1.1.3" xref="S3.SS1.p3.2.m2.1.1.1.1.1.1.3.cmml"><mi id="S3.SS1.p3.2.m2.1.1.1.1.1.1.3.2" xref="S3.SS1.p3.2.m2.1.1.1.1.1.1.3.2.cmml">θ</mi><mo id="S3.SS1.p3.2.m2.1.1.1.1.1.1.3.3" xref="S3.SS1.p3.2.m2.1.1.1.1.1.1.3.3.cmml">∗</mo></msup></mrow><mo stretchy="false" id="S3.SS1.p3.2.m2.1.1.1.1.1.3" xref="S3.SS1.p3.2.m2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS1.p3.2.m2.2.2.3" xref="S3.SS1.p3.2.m2.2.2.3.cmml">=</mo><mrow id="S3.SS1.p3.2.m2.2.2.2" xref="S3.SS1.p3.2.m2.2.2.2.cmml"><mo id="S3.SS1.p3.2.m2.2.2.2a" xref="S3.SS1.p3.2.m2.2.2.2.cmml">−</mo><mrow id="S3.SS1.p3.2.m2.2.2.2.1" xref="S3.SS1.p3.2.m2.2.2.2.1.cmml"><mi mathvariant="normal" id="S3.SS1.p3.2.m2.2.2.2.1.3" xref="S3.SS1.p3.2.m2.2.2.2.1.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p3.2.m2.2.2.2.1.2" xref="S3.SS1.p3.2.m2.2.2.2.1.2.cmml">​</mo><mrow id="S3.SS1.p3.2.m2.2.2.2.1.1.1" xref="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p3.2.m2.2.2.2.1.1.1.2" xref="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.p3.2.m2.2.2.2.1.1.1.1" xref="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.2" xref="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.2.cmml">ℬ</mi><mo fence="false" id="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.1" xref="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.1.cmml">|</mo><msup id="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.3" xref="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.3.cmml"><mi id="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.3.2" xref="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.3.2.cmml">θ</mi><mo id="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.3.3" xref="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.3.3.cmml">∗</mo></msup></mrow><mo stretchy="false" id="S3.SS1.p3.2.m2.2.2.2.1.1.1.3" xref="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.2.m2.2b"><apply id="S3.SS1.p3.2.m2.2.2.cmml" xref="S3.SS1.p3.2.m2.2.2"><eq id="S3.SS1.p3.2.m2.2.2.3.cmml" xref="S3.SS1.p3.2.m2.2.2.3"></eq><apply id="S3.SS1.p3.2.m2.1.1.1.cmml" xref="S3.SS1.p3.2.m2.1.1.1"><times id="S3.SS1.p3.2.m2.1.1.1.2.cmml" xref="S3.SS1.p3.2.m2.1.1.1.2"></times><apply id="S3.SS1.p3.2.m2.1.1.1.3.cmml" xref="S3.SS1.p3.2.m2.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p3.2.m2.1.1.1.3.1.cmml" xref="S3.SS1.p3.2.m2.1.1.1.3">superscript</csymbol><ci id="S3.SS1.p3.2.m2.1.1.1.3.2.cmml" xref="S3.SS1.p3.2.m2.1.1.1.3.2">𝑠</ci><ci id="S3.SS1.p3.2.m2.1.1.1.3.3a.cmml" xref="S3.SS1.p3.2.m2.1.1.1.3.3"><mtext mathsize="70%" id="S3.SS1.p3.2.m2.1.1.1.3.3.cmml" xref="S3.SS1.p3.2.m2.1.1.1.3.3">easy</mtext></ci></apply><apply id="S3.SS1.p3.2.m2.1.1.1.1.1.1.cmml" xref="S3.SS1.p3.2.m2.1.1.1.1.1"><csymbol cd="latexml" id="S3.SS1.p3.2.m2.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p3.2.m2.1.1.1.1.1.1.1">conditional</csymbol><ci id="S3.SS1.p3.2.m2.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p3.2.m2.1.1.1.1.1.1.2">ℬ</ci><apply id="S3.SS1.p3.2.m2.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p3.2.m2.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p3.2.m2.1.1.1.1.1.1.3.1.cmml" xref="S3.SS1.p3.2.m2.1.1.1.1.1.1.3">superscript</csymbol><ci id="S3.SS1.p3.2.m2.1.1.1.1.1.1.3.2.cmml" xref="S3.SS1.p3.2.m2.1.1.1.1.1.1.3.2">𝜃</ci><times id="S3.SS1.p3.2.m2.1.1.1.1.1.1.3.3.cmml" xref="S3.SS1.p3.2.m2.1.1.1.1.1.1.3.3"></times></apply></apply></apply><apply id="S3.SS1.p3.2.m2.2.2.2.cmml" xref="S3.SS1.p3.2.m2.2.2.2"><minus id="S3.SS1.p3.2.m2.2.2.2.2.cmml" xref="S3.SS1.p3.2.m2.2.2.2"></minus><apply id="S3.SS1.p3.2.m2.2.2.2.1.cmml" xref="S3.SS1.p3.2.m2.2.2.2.1"><times id="S3.SS1.p3.2.m2.2.2.2.1.2.cmml" xref="S3.SS1.p3.2.m2.2.2.2.1.2"></times><ci id="S3.SS1.p3.2.m2.2.2.2.1.3.cmml" xref="S3.SS1.p3.2.m2.2.2.2.1.3">ℓ</ci><apply id="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.cmml" xref="S3.SS1.p3.2.m2.2.2.2.1.1.1"><csymbol cd="latexml" id="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.1.cmml" xref="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.1">conditional</csymbol><ci id="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.2.cmml" xref="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.2">ℬ</ci><apply id="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.3.cmml" xref="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.3.1.cmml" xref="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.3">superscript</csymbol><ci id="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.3.2.cmml" xref="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.3.2">𝜃</ci><times id="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.3.3.cmml" xref="S3.SS1.p3.2.m2.2.2.2.1.1.1.1.3.3"></times></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.2.m2.2c">s^{\textrm{easy}}(\mathcal{B}|\theta^{*})=-\ell(\mathcal{B}|\theta^{*})</annotation></semantics></math>. This <span id="S3.SS1.p3.2.3" class="ltx_text ltx_font_italic">easy reference</span> heuristic has been used successfully in multimodal learning to identify high-quality examples <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib22" title="" class="ltx_ref">22</a>, <a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite>, but does not reflect the current state of the learner and can therefore be overly dependent on the choice of reference model <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite> and not scale to large compute budgets <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite>.</p>
</div>
<div id="S3.SS1.p4" class="ltx_para ltx_noindent">
<p id="S3.SS1.p4.1" class="ltx_p"><span id="S3.SS1.p4.1.1" class="ltx_text ltx_font_bold">Learnability.</span> Finally, <cite class="ltx_cite ltx_citemacro_citet">Mindermann et al. [<a href="#bib.bib33" title="" class="ltx_ref">33</a>]</cite> propose to combine these scores, prioritizing with the difference of losses: <math id="S3.SS1.p4.1.m1.6" class="ltx_Math" alttext="s^{\textrm{learn}}(\mathcal{B}|\theta,\theta^{*})=s^{\textrm{hard}}(\mathcal{B}|\theta)+s^{\textrm{easy}}(\mathcal{B}|\theta^{*})=\ell(\mathcal{B}|\theta)-\ell(\mathcal{B}|\theta^{*})" display="inline"><semantics id="S3.SS1.p4.1.m1.6a"><mrow id="S3.SS1.p4.1.m1.6.6" xref="S3.SS1.p4.1.m1.6.6.cmml"><mrow id="S3.SS1.p4.1.m1.2.2.1" xref="S3.SS1.p4.1.m1.2.2.1.cmml"><msup id="S3.SS1.p4.1.m1.2.2.1.3" xref="S3.SS1.p4.1.m1.2.2.1.3.cmml"><mi id="S3.SS1.p4.1.m1.2.2.1.3.2" xref="S3.SS1.p4.1.m1.2.2.1.3.2.cmml">s</mi><mtext id="S3.SS1.p4.1.m1.2.2.1.3.3" xref="S3.SS1.p4.1.m1.2.2.1.3.3a.cmml">learn</mtext></msup><mo lspace="0em" rspace="0em" id="S3.SS1.p4.1.m1.2.2.1.2" xref="S3.SS1.p4.1.m1.2.2.1.2.cmml">​</mo><mrow id="S3.SS1.p4.1.m1.2.2.1.1.1" xref="S3.SS1.p4.1.m1.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p4.1.m1.2.2.1.1.1.2" xref="S3.SS1.p4.1.m1.2.2.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.p4.1.m1.2.2.1.1.1.1" xref="S3.SS1.p4.1.m1.2.2.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p4.1.m1.2.2.1.1.1.1.3" xref="S3.SS1.p4.1.m1.2.2.1.1.1.1.3.cmml">ℬ</mi><mo fence="false" id="S3.SS1.p4.1.m1.2.2.1.1.1.1.2" xref="S3.SS1.p4.1.m1.2.2.1.1.1.1.2.cmml">|</mo><mrow id="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.1" xref="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.2.cmml"><mi id="S3.SS1.p4.1.m1.1.1" xref="S3.SS1.p4.1.m1.1.1.cmml">θ</mi><mo id="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.1.2" xref="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.2.cmml">,</mo><msup id="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.1.1" xref="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.1.1.cmml"><mi id="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.1.1.2" xref="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.1.1.2.cmml">θ</mi><mo id="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.1.1.3" xref="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.1.1.3.cmml">∗</mo></msup></mrow></mrow><mo stretchy="false" id="S3.SS1.p4.1.m1.2.2.1.1.1.3" xref="S3.SS1.p4.1.m1.2.2.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS1.p4.1.m1.6.6.7" xref="S3.SS1.p4.1.m1.6.6.7.cmml">=</mo><mrow id="S3.SS1.p4.1.m1.4.4.3" xref="S3.SS1.p4.1.m1.4.4.3.cmml"><mrow id="S3.SS1.p4.1.m1.3.3.2.1" xref="S3.SS1.p4.1.m1.3.3.2.1.cmml"><msup id="S3.SS1.p4.1.m1.3.3.2.1.3" xref="S3.SS1.p4.1.m1.3.3.2.1.3.cmml"><mi id="S3.SS1.p4.1.m1.3.3.2.1.3.2" xref="S3.SS1.p4.1.m1.3.3.2.1.3.2.cmml">s</mi><mtext id="S3.SS1.p4.1.m1.3.3.2.1.3.3" xref="S3.SS1.p4.1.m1.3.3.2.1.3.3a.cmml">hard</mtext></msup><mo lspace="0em" rspace="0em" id="S3.SS1.p4.1.m1.3.3.2.1.2" xref="S3.SS1.p4.1.m1.3.3.2.1.2.cmml">​</mo><mrow id="S3.SS1.p4.1.m1.3.3.2.1.1.1" xref="S3.SS1.p4.1.m1.3.3.2.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p4.1.m1.3.3.2.1.1.1.2" xref="S3.SS1.p4.1.m1.3.3.2.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.p4.1.m1.3.3.2.1.1.1.1" xref="S3.SS1.p4.1.m1.3.3.2.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p4.1.m1.3.3.2.1.1.1.1.2" xref="S3.SS1.p4.1.m1.3.3.2.1.1.1.1.2.cmml">ℬ</mi><mo fence="false" id="S3.SS1.p4.1.m1.3.3.2.1.1.1.1.1" xref="S3.SS1.p4.1.m1.3.3.2.1.1.1.1.1.cmml">|</mo><mi id="S3.SS1.p4.1.m1.3.3.2.1.1.1.1.3" xref="S3.SS1.p4.1.m1.3.3.2.1.1.1.1.3.cmml">θ</mi></mrow><mo stretchy="false" id="S3.SS1.p4.1.m1.3.3.2.1.1.1.3" xref="S3.SS1.p4.1.m1.3.3.2.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS1.p4.1.m1.4.4.3.3" xref="S3.SS1.p4.1.m1.4.4.3.3.cmml">+</mo><mrow id="S3.SS1.p4.1.m1.4.4.3.2" xref="S3.SS1.p4.1.m1.4.4.3.2.cmml"><msup id="S3.SS1.p4.1.m1.4.4.3.2.3" xref="S3.SS1.p4.1.m1.4.4.3.2.3.cmml"><mi id="S3.SS1.p4.1.m1.4.4.3.2.3.2" xref="S3.SS1.p4.1.m1.4.4.3.2.3.2.cmml">s</mi><mtext id="S3.SS1.p4.1.m1.4.4.3.2.3.3" xref="S3.SS1.p4.1.m1.4.4.3.2.3.3a.cmml">easy</mtext></msup><mo lspace="0em" rspace="0em" id="S3.SS1.p4.1.m1.4.4.3.2.2" xref="S3.SS1.p4.1.m1.4.4.3.2.2.cmml">​</mo><mrow id="S3.SS1.p4.1.m1.4.4.3.2.1.1" xref="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p4.1.m1.4.4.3.2.1.1.2" xref="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.cmml">(</mo><mrow id="S3.SS1.p4.1.m1.4.4.3.2.1.1.1" xref="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.2" xref="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.2.cmml">ℬ</mi><mo fence="false" id="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.1" xref="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.1.cmml">|</mo><msup id="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.3" xref="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.3.cmml"><mi id="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.3.2" xref="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.3.2.cmml">θ</mi><mo id="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.3.3" xref="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.3.3.cmml">∗</mo></msup></mrow><mo stretchy="false" id="S3.SS1.p4.1.m1.4.4.3.2.1.1.3" xref="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo id="S3.SS1.p4.1.m1.6.6.8" xref="S3.SS1.p4.1.m1.6.6.8.cmml">=</mo><mrow id="S3.SS1.p4.1.m1.6.6.5" xref="S3.SS1.p4.1.m1.6.6.5.cmml"><mrow id="S3.SS1.p4.1.m1.5.5.4.1" xref="S3.SS1.p4.1.m1.5.5.4.1.cmml"><mi mathvariant="normal" id="S3.SS1.p4.1.m1.5.5.4.1.3" xref="S3.SS1.p4.1.m1.5.5.4.1.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p4.1.m1.5.5.4.1.2" xref="S3.SS1.p4.1.m1.5.5.4.1.2.cmml">​</mo><mrow id="S3.SS1.p4.1.m1.5.5.4.1.1.1" xref="S3.SS1.p4.1.m1.5.5.4.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p4.1.m1.5.5.4.1.1.1.2" xref="S3.SS1.p4.1.m1.5.5.4.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.p4.1.m1.5.5.4.1.1.1.1" xref="S3.SS1.p4.1.m1.5.5.4.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p4.1.m1.5.5.4.1.1.1.1.2" xref="S3.SS1.p4.1.m1.5.5.4.1.1.1.1.2.cmml">ℬ</mi><mo fence="false" id="S3.SS1.p4.1.m1.5.5.4.1.1.1.1.1" xref="S3.SS1.p4.1.m1.5.5.4.1.1.1.1.1.cmml">|</mo><mi id="S3.SS1.p4.1.m1.5.5.4.1.1.1.1.3" xref="S3.SS1.p4.1.m1.5.5.4.1.1.1.1.3.cmml">θ</mi></mrow><mo stretchy="false" id="S3.SS1.p4.1.m1.5.5.4.1.1.1.3" xref="S3.SS1.p4.1.m1.5.5.4.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS1.p4.1.m1.6.6.5.3" xref="S3.SS1.p4.1.m1.6.6.5.3.cmml">−</mo><mrow id="S3.SS1.p4.1.m1.6.6.5.2" xref="S3.SS1.p4.1.m1.6.6.5.2.cmml"><mi mathvariant="normal" id="S3.SS1.p4.1.m1.6.6.5.2.3" xref="S3.SS1.p4.1.m1.6.6.5.2.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p4.1.m1.6.6.5.2.2" xref="S3.SS1.p4.1.m1.6.6.5.2.2.cmml">​</mo><mrow id="S3.SS1.p4.1.m1.6.6.5.2.1.1" xref="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p4.1.m1.6.6.5.2.1.1.2" xref="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.cmml">(</mo><mrow id="S3.SS1.p4.1.m1.6.6.5.2.1.1.1" xref="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.2" xref="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.2.cmml">ℬ</mi><mo fence="false" id="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.1" xref="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.1.cmml">|</mo><msup id="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.3" xref="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.3.cmml"><mi id="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.3.2" xref="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.3.2.cmml">θ</mi><mo id="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.3.3" xref="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.3.3.cmml">∗</mo></msup></mrow><mo stretchy="false" id="S3.SS1.p4.1.m1.6.6.5.2.1.1.3" xref="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.1.m1.6b"><apply id="S3.SS1.p4.1.m1.6.6.cmml" xref="S3.SS1.p4.1.m1.6.6"><and id="S3.SS1.p4.1.m1.6.6a.cmml" xref="S3.SS1.p4.1.m1.6.6"></and><apply id="S3.SS1.p4.1.m1.6.6b.cmml" xref="S3.SS1.p4.1.m1.6.6"><eq id="S3.SS1.p4.1.m1.6.6.7.cmml" xref="S3.SS1.p4.1.m1.6.6.7"></eq><apply id="S3.SS1.p4.1.m1.2.2.1.cmml" xref="S3.SS1.p4.1.m1.2.2.1"><times id="S3.SS1.p4.1.m1.2.2.1.2.cmml" xref="S3.SS1.p4.1.m1.2.2.1.2"></times><apply id="S3.SS1.p4.1.m1.2.2.1.3.cmml" xref="S3.SS1.p4.1.m1.2.2.1.3"><csymbol cd="ambiguous" id="S3.SS1.p4.1.m1.2.2.1.3.1.cmml" xref="S3.SS1.p4.1.m1.2.2.1.3">superscript</csymbol><ci id="S3.SS1.p4.1.m1.2.2.1.3.2.cmml" xref="S3.SS1.p4.1.m1.2.2.1.3.2">𝑠</ci><ci id="S3.SS1.p4.1.m1.2.2.1.3.3a.cmml" xref="S3.SS1.p4.1.m1.2.2.1.3.3"><mtext mathsize="70%" id="S3.SS1.p4.1.m1.2.2.1.3.3.cmml" xref="S3.SS1.p4.1.m1.2.2.1.3.3">learn</mtext></ci></apply><apply id="S3.SS1.p4.1.m1.2.2.1.1.1.1.cmml" xref="S3.SS1.p4.1.m1.2.2.1.1.1"><csymbol cd="latexml" id="S3.SS1.p4.1.m1.2.2.1.1.1.1.2.cmml" xref="S3.SS1.p4.1.m1.2.2.1.1.1.1.2">conditional</csymbol><ci id="S3.SS1.p4.1.m1.2.2.1.1.1.1.3.cmml" xref="S3.SS1.p4.1.m1.2.2.1.1.1.1.3">ℬ</ci><list id="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.2.cmml" xref="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.1"><ci id="S3.SS1.p4.1.m1.1.1.cmml" xref="S3.SS1.p4.1.m1.1.1">𝜃</ci><apply id="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.1.1">superscript</csymbol><ci id="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.1.1.2">𝜃</ci><times id="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p4.1.m1.2.2.1.1.1.1.1.1.1.3"></times></apply></list></apply></apply><apply id="S3.SS1.p4.1.m1.4.4.3.cmml" xref="S3.SS1.p4.1.m1.4.4.3"><plus id="S3.SS1.p4.1.m1.4.4.3.3.cmml" xref="S3.SS1.p4.1.m1.4.4.3.3"></plus><apply id="S3.SS1.p4.1.m1.3.3.2.1.cmml" xref="S3.SS1.p4.1.m1.3.3.2.1"><times id="S3.SS1.p4.1.m1.3.3.2.1.2.cmml" xref="S3.SS1.p4.1.m1.3.3.2.1.2"></times><apply id="S3.SS1.p4.1.m1.3.3.2.1.3.cmml" xref="S3.SS1.p4.1.m1.3.3.2.1.3"><csymbol cd="ambiguous" id="S3.SS1.p4.1.m1.3.3.2.1.3.1.cmml" xref="S3.SS1.p4.1.m1.3.3.2.1.3">superscript</csymbol><ci id="S3.SS1.p4.1.m1.3.3.2.1.3.2.cmml" xref="S3.SS1.p4.1.m1.3.3.2.1.3.2">𝑠</ci><ci id="S3.SS1.p4.1.m1.3.3.2.1.3.3a.cmml" xref="S3.SS1.p4.1.m1.3.3.2.1.3.3"><mtext mathsize="70%" id="S3.SS1.p4.1.m1.3.3.2.1.3.3.cmml" xref="S3.SS1.p4.1.m1.3.3.2.1.3.3">hard</mtext></ci></apply><apply id="S3.SS1.p4.1.m1.3.3.2.1.1.1.1.cmml" xref="S3.SS1.p4.1.m1.3.3.2.1.1.1"><csymbol cd="latexml" id="S3.SS1.p4.1.m1.3.3.2.1.1.1.1.1.cmml" xref="S3.SS1.p4.1.m1.3.3.2.1.1.1.1.1">conditional</csymbol><ci id="S3.SS1.p4.1.m1.3.3.2.1.1.1.1.2.cmml" xref="S3.SS1.p4.1.m1.3.3.2.1.1.1.1.2">ℬ</ci><ci id="S3.SS1.p4.1.m1.3.3.2.1.1.1.1.3.cmml" xref="S3.SS1.p4.1.m1.3.3.2.1.1.1.1.3">𝜃</ci></apply></apply><apply id="S3.SS1.p4.1.m1.4.4.3.2.cmml" xref="S3.SS1.p4.1.m1.4.4.3.2"><times id="S3.SS1.p4.1.m1.4.4.3.2.2.cmml" xref="S3.SS1.p4.1.m1.4.4.3.2.2"></times><apply id="S3.SS1.p4.1.m1.4.4.3.2.3.cmml" xref="S3.SS1.p4.1.m1.4.4.3.2.3"><csymbol cd="ambiguous" id="S3.SS1.p4.1.m1.4.4.3.2.3.1.cmml" xref="S3.SS1.p4.1.m1.4.4.3.2.3">superscript</csymbol><ci id="S3.SS1.p4.1.m1.4.4.3.2.3.2.cmml" xref="S3.SS1.p4.1.m1.4.4.3.2.3.2">𝑠</ci><ci id="S3.SS1.p4.1.m1.4.4.3.2.3.3a.cmml" xref="S3.SS1.p4.1.m1.4.4.3.2.3.3"><mtext mathsize="70%" id="S3.SS1.p4.1.m1.4.4.3.2.3.3.cmml" xref="S3.SS1.p4.1.m1.4.4.3.2.3.3">easy</mtext></ci></apply><apply id="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.cmml" xref="S3.SS1.p4.1.m1.4.4.3.2.1.1"><csymbol cd="latexml" id="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.1.cmml" xref="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.1">conditional</csymbol><ci id="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.2.cmml" xref="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.2">ℬ</ci><apply id="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.3.cmml" xref="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.3.1.cmml" xref="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.3">superscript</csymbol><ci id="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.3.2.cmml" xref="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.3.2">𝜃</ci><times id="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.3.3.cmml" xref="S3.SS1.p4.1.m1.4.4.3.2.1.1.1.3.3"></times></apply></apply></apply></apply></apply><apply id="S3.SS1.p4.1.m1.6.6c.cmml" xref="S3.SS1.p4.1.m1.6.6"><eq id="S3.SS1.p4.1.m1.6.6.8.cmml" xref="S3.SS1.p4.1.m1.6.6.8"></eq><share href="#S3.SS1.p4.1.m1.4.4.3.cmml" id="S3.SS1.p4.1.m1.6.6d.cmml" xref="S3.SS1.p4.1.m1.6.6"></share><apply id="S3.SS1.p4.1.m1.6.6.5.cmml" xref="S3.SS1.p4.1.m1.6.6.5"><minus id="S3.SS1.p4.1.m1.6.6.5.3.cmml" xref="S3.SS1.p4.1.m1.6.6.5.3"></minus><apply id="S3.SS1.p4.1.m1.5.5.4.1.cmml" xref="S3.SS1.p4.1.m1.5.5.4.1"><times id="S3.SS1.p4.1.m1.5.5.4.1.2.cmml" xref="S3.SS1.p4.1.m1.5.5.4.1.2"></times><ci id="S3.SS1.p4.1.m1.5.5.4.1.3.cmml" xref="S3.SS1.p4.1.m1.5.5.4.1.3">ℓ</ci><apply id="S3.SS1.p4.1.m1.5.5.4.1.1.1.1.cmml" xref="S3.SS1.p4.1.m1.5.5.4.1.1.1"><csymbol cd="latexml" id="S3.SS1.p4.1.m1.5.5.4.1.1.1.1.1.cmml" xref="S3.SS1.p4.1.m1.5.5.4.1.1.1.1.1">conditional</csymbol><ci id="S3.SS1.p4.1.m1.5.5.4.1.1.1.1.2.cmml" xref="S3.SS1.p4.1.m1.5.5.4.1.1.1.1.2">ℬ</ci><ci id="S3.SS1.p4.1.m1.5.5.4.1.1.1.1.3.cmml" xref="S3.SS1.p4.1.m1.5.5.4.1.1.1.1.3">𝜃</ci></apply></apply><apply id="S3.SS1.p4.1.m1.6.6.5.2.cmml" xref="S3.SS1.p4.1.m1.6.6.5.2"><times id="S3.SS1.p4.1.m1.6.6.5.2.2.cmml" xref="S3.SS1.p4.1.m1.6.6.5.2.2"></times><ci id="S3.SS1.p4.1.m1.6.6.5.2.3.cmml" xref="S3.SS1.p4.1.m1.6.6.5.2.3">ℓ</ci><apply id="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.cmml" xref="S3.SS1.p4.1.m1.6.6.5.2.1.1"><csymbol cd="latexml" id="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.1.cmml" xref="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.1">conditional</csymbol><ci id="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.2.cmml" xref="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.2">ℬ</ci><apply id="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.3.cmml" xref="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.3.1.cmml" xref="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.3">superscript</csymbol><ci id="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.3.2.cmml" xref="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.3.2">𝜃</ci><times id="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.3.3.cmml" xref="S3.SS1.p4.1.m1.6.6.5.2.1.1.1.3.3"></times></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.1.m1.6c">s^{\textrm{learn}}(\mathcal{B}|\theta,\theta^{*})=s^{\textrm{hard}}(\mathcal{B}|\theta)+s^{\textrm{easy}}(\mathcal{B}|\theta^{*})=\ell(\mathcal{B}|\theta)-\ell(\mathcal{B}|\theta^{*})</annotation></semantics></math>. This heuristic, which we refer to as <span id="S3.SS1.p4.1.2" class="ltx_text ltx_font_italic">learnability</span> scoring throughout, has the advantage of up-sampling data that is both unlearned and learnable, and has been shown to accelerate large-scale learning even when prioritizing individual examples in isolation <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>. In this work, we therefore mainly consider <span id="S3.SS1.p4.1.3" class="ltx_text ltx_font_italic">learnability</span> scoring but for completeness also provide ablations with <span id="S3.SS1.p4.1.4" class="ltx_text ltx_font_italic">easy reference</span> scoring.</p>
</div>
<div id="S3.SS1.p5" class="ltx_para ltx_noindent">
<p id="S3.SS1.p5.2" class="ltx_p">The ratio of the “sub-batch” and “super-batch” sizes defines the <span id="S3.SS1.p5.2.1" class="ltx_text ltx_font_italic">filtering ratio</span> <math id="S3.SS1.p5.1.m1.1" class="ltx_Math" alttext="f=1-b/B" display="inline"><semantics id="S3.SS1.p5.1.m1.1a"><mrow id="S3.SS1.p5.1.m1.1.1" xref="S3.SS1.p5.1.m1.1.1.cmml"><mi id="S3.SS1.p5.1.m1.1.1.2" xref="S3.SS1.p5.1.m1.1.1.2.cmml">f</mi><mo id="S3.SS1.p5.1.m1.1.1.1" xref="S3.SS1.p5.1.m1.1.1.1.cmml">=</mo><mrow id="S3.SS1.p5.1.m1.1.1.3" xref="S3.SS1.p5.1.m1.1.1.3.cmml"><mn id="S3.SS1.p5.1.m1.1.1.3.2" xref="S3.SS1.p5.1.m1.1.1.3.2.cmml">1</mn><mo id="S3.SS1.p5.1.m1.1.1.3.1" xref="S3.SS1.p5.1.m1.1.1.3.1.cmml">−</mo><mrow id="S3.SS1.p5.1.m1.1.1.3.3" xref="S3.SS1.p5.1.m1.1.1.3.3.cmml"><mi id="S3.SS1.p5.1.m1.1.1.3.3.2" xref="S3.SS1.p5.1.m1.1.1.3.3.2.cmml">b</mi><mo id="S3.SS1.p5.1.m1.1.1.3.3.1" xref="S3.SS1.p5.1.m1.1.1.3.3.1.cmml">/</mo><mi id="S3.SS1.p5.1.m1.1.1.3.3.3" xref="S3.SS1.p5.1.m1.1.1.3.3.3.cmml">B</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p5.1.m1.1b"><apply id="S3.SS1.p5.1.m1.1.1.cmml" xref="S3.SS1.p5.1.m1.1.1"><eq id="S3.SS1.p5.1.m1.1.1.1.cmml" xref="S3.SS1.p5.1.m1.1.1.1"></eq><ci id="S3.SS1.p5.1.m1.1.1.2.cmml" xref="S3.SS1.p5.1.m1.1.1.2">𝑓</ci><apply id="S3.SS1.p5.1.m1.1.1.3.cmml" xref="S3.SS1.p5.1.m1.1.1.3"><minus id="S3.SS1.p5.1.m1.1.1.3.1.cmml" xref="S3.SS1.p5.1.m1.1.1.3.1"></minus><cn type="integer" id="S3.SS1.p5.1.m1.1.1.3.2.cmml" xref="S3.SS1.p5.1.m1.1.1.3.2">1</cn><apply id="S3.SS1.p5.1.m1.1.1.3.3.cmml" xref="S3.SS1.p5.1.m1.1.1.3.3"><divide id="S3.SS1.p5.1.m1.1.1.3.3.1.cmml" xref="S3.SS1.p5.1.m1.1.1.3.3.1"></divide><ci id="S3.SS1.p5.1.m1.1.1.3.3.2.cmml" xref="S3.SS1.p5.1.m1.1.1.3.3.2">𝑏</ci><ci id="S3.SS1.p5.1.m1.1.1.3.3.3.cmml" xref="S3.SS1.p5.1.m1.1.1.3.3.3">𝐵</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p5.1.m1.1c">f=1-b/B</annotation></semantics></math>, i.e. the proportion of data discarded at each iteration. For a given learner batch size <math id="S3.SS1.p5.2.m2.1" class="ltx_Math" alttext="b" display="inline"><semantics id="S3.SS1.p5.2.m2.1a"><mi id="S3.SS1.p5.2.m2.1.1" xref="S3.SS1.p5.2.m2.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p5.2.m2.1b"><ci id="S3.SS1.p5.2.m2.1.1.cmml" xref="S3.SS1.p5.2.m2.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p5.2.m2.1c">b</annotation></semantics></math>, higher filtering ratios increase the cost of scoring as they require more inference passes on the super-batch.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Joint example selection (JEST) for multimodal learning</h3>

<div id="S3.SS2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS2.p1.2" class="ltx_p"><span id="S3.SS2.p1.2.1" class="ltx_text ltx_font_bold">Multimodal learning losses.</span> Given the availability of internet-scale datasets of paired images and text, multimodal learning has become the default means of training visual representations. Contrastive learning aims to maximize the alignment of these two modalities for paired examples, while minimizing the alignment of unpaired examples.
Both sigmoid- <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite> and softmax-contrastive <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib38" title="" class="ltx_ref">38</a>]</cite>
losses achieve this with a batch-level loss <math id="S3.SS2.p1.1.m1.4" class="ltx_Math" alttext="\ell(\mathcal{B}|\theta)=\frac{1}{b}\sum_{i=1}^{b}\ell({\bm{x}}_{i}|\theta,\mathcal{B})" display="inline"><semantics id="S3.SS2.p1.1.m1.4a"><mrow id="S3.SS2.p1.1.m1.4.4" xref="S3.SS2.p1.1.m1.4.4.cmml"><mrow id="S3.SS2.p1.1.m1.3.3.1" xref="S3.SS2.p1.1.m1.3.3.1.cmml"><mi mathvariant="normal" id="S3.SS2.p1.1.m1.3.3.1.3" xref="S3.SS2.p1.1.m1.3.3.1.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.1.m1.3.3.1.2" xref="S3.SS2.p1.1.m1.3.3.1.2.cmml">​</mo><mrow id="S3.SS2.p1.1.m1.3.3.1.1.1" xref="S3.SS2.p1.1.m1.3.3.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p1.1.m1.3.3.1.1.1.2" xref="S3.SS2.p1.1.m1.3.3.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p1.1.m1.3.3.1.1.1.1" xref="S3.SS2.p1.1.m1.3.3.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.1.m1.3.3.1.1.1.1.2" xref="S3.SS2.p1.1.m1.3.3.1.1.1.1.2.cmml">ℬ</mi><mo fence="false" id="S3.SS2.p1.1.m1.3.3.1.1.1.1.1" xref="S3.SS2.p1.1.m1.3.3.1.1.1.1.1.cmml">|</mo><mi id="S3.SS2.p1.1.m1.3.3.1.1.1.1.3" xref="S3.SS2.p1.1.m1.3.3.1.1.1.1.3.cmml">θ</mi></mrow><mo stretchy="false" id="S3.SS2.p1.1.m1.3.3.1.1.1.3" xref="S3.SS2.p1.1.m1.3.3.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS2.p1.1.m1.4.4.3" xref="S3.SS2.p1.1.m1.4.4.3.cmml">=</mo><mrow id="S3.SS2.p1.1.m1.4.4.2" xref="S3.SS2.p1.1.m1.4.4.2.cmml"><mfrac id="S3.SS2.p1.1.m1.4.4.2.3" xref="S3.SS2.p1.1.m1.4.4.2.3.cmml"><mn id="S3.SS2.p1.1.m1.4.4.2.3.2" xref="S3.SS2.p1.1.m1.4.4.2.3.2.cmml">1</mn><mi id="S3.SS2.p1.1.m1.4.4.2.3.3" xref="S3.SS2.p1.1.m1.4.4.2.3.3.cmml">b</mi></mfrac><mo lspace="0em" rspace="0em" id="S3.SS2.p1.1.m1.4.4.2.2" xref="S3.SS2.p1.1.m1.4.4.2.2.cmml">​</mo><mrow id="S3.SS2.p1.1.m1.4.4.2.1" xref="S3.SS2.p1.1.m1.4.4.2.1.cmml"><msubsup id="S3.SS2.p1.1.m1.4.4.2.1.2" xref="S3.SS2.p1.1.m1.4.4.2.1.2.cmml"><mo id="S3.SS2.p1.1.m1.4.4.2.1.2.2.2" xref="S3.SS2.p1.1.m1.4.4.2.1.2.2.2.cmml">∑</mo><mrow id="S3.SS2.p1.1.m1.4.4.2.1.2.2.3" xref="S3.SS2.p1.1.m1.4.4.2.1.2.2.3.cmml"><mi id="S3.SS2.p1.1.m1.4.4.2.1.2.2.3.2" xref="S3.SS2.p1.1.m1.4.4.2.1.2.2.3.2.cmml">i</mi><mo id="S3.SS2.p1.1.m1.4.4.2.1.2.2.3.1" xref="S3.SS2.p1.1.m1.4.4.2.1.2.2.3.1.cmml">=</mo><mn id="S3.SS2.p1.1.m1.4.4.2.1.2.2.3.3" xref="S3.SS2.p1.1.m1.4.4.2.1.2.2.3.3.cmml">1</mn></mrow><mi id="S3.SS2.p1.1.m1.4.4.2.1.2.3" xref="S3.SS2.p1.1.m1.4.4.2.1.2.3.cmml">b</mi></msubsup><mrow id="S3.SS2.p1.1.m1.4.4.2.1.1" xref="S3.SS2.p1.1.m1.4.4.2.1.1.cmml"><mi mathvariant="normal" id="S3.SS2.p1.1.m1.4.4.2.1.1.3" xref="S3.SS2.p1.1.m1.4.4.2.1.1.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.1.m1.4.4.2.1.1.2" xref="S3.SS2.p1.1.m1.4.4.2.1.1.2.cmml">​</mo><mrow id="S3.SS2.p1.1.m1.4.4.2.1.1.1.1" xref="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.2" xref="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1" xref="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.cmml"><msub id="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.2" xref="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.2.cmml"><mi id="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.2.2" xref="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.2.2.cmml">𝒙</mi><mi id="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.2.3" xref="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.2.3.cmml">i</mi></msub><mo fence="false" id="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.1" xref="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.1.cmml">|</mo><mrow id="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.3.2" xref="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.3.1.cmml"><mi id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml">θ</mi><mo id="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.3.2.1" xref="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.3.1.cmml">,</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.1.m1.2.2" xref="S3.SS2.p1.1.m1.2.2.cmml">ℬ</mi></mrow></mrow><mo stretchy="false" id="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.3" xref="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.4b"><apply id="S3.SS2.p1.1.m1.4.4.cmml" xref="S3.SS2.p1.1.m1.4.4"><eq id="S3.SS2.p1.1.m1.4.4.3.cmml" xref="S3.SS2.p1.1.m1.4.4.3"></eq><apply id="S3.SS2.p1.1.m1.3.3.1.cmml" xref="S3.SS2.p1.1.m1.3.3.1"><times id="S3.SS2.p1.1.m1.3.3.1.2.cmml" xref="S3.SS2.p1.1.m1.3.3.1.2"></times><ci id="S3.SS2.p1.1.m1.3.3.1.3.cmml" xref="S3.SS2.p1.1.m1.3.3.1.3">ℓ</ci><apply id="S3.SS2.p1.1.m1.3.3.1.1.1.1.cmml" xref="S3.SS2.p1.1.m1.3.3.1.1.1"><csymbol cd="latexml" id="S3.SS2.p1.1.m1.3.3.1.1.1.1.1.cmml" xref="S3.SS2.p1.1.m1.3.3.1.1.1.1.1">conditional</csymbol><ci id="S3.SS2.p1.1.m1.3.3.1.1.1.1.2.cmml" xref="S3.SS2.p1.1.m1.3.3.1.1.1.1.2">ℬ</ci><ci id="S3.SS2.p1.1.m1.3.3.1.1.1.1.3.cmml" xref="S3.SS2.p1.1.m1.3.3.1.1.1.1.3">𝜃</ci></apply></apply><apply id="S3.SS2.p1.1.m1.4.4.2.cmml" xref="S3.SS2.p1.1.m1.4.4.2"><times id="S3.SS2.p1.1.m1.4.4.2.2.cmml" xref="S3.SS2.p1.1.m1.4.4.2.2"></times><apply id="S3.SS2.p1.1.m1.4.4.2.3.cmml" xref="S3.SS2.p1.1.m1.4.4.2.3"><divide id="S3.SS2.p1.1.m1.4.4.2.3.1.cmml" xref="S3.SS2.p1.1.m1.4.4.2.3"></divide><cn type="integer" id="S3.SS2.p1.1.m1.4.4.2.3.2.cmml" xref="S3.SS2.p1.1.m1.4.4.2.3.2">1</cn><ci id="S3.SS2.p1.1.m1.4.4.2.3.3.cmml" xref="S3.SS2.p1.1.m1.4.4.2.3.3">𝑏</ci></apply><apply id="S3.SS2.p1.1.m1.4.4.2.1.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1"><apply id="S3.SS2.p1.1.m1.4.4.2.1.2.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.2"><csymbol cd="ambiguous" id="S3.SS2.p1.1.m1.4.4.2.1.2.1.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.2">superscript</csymbol><apply id="S3.SS2.p1.1.m1.4.4.2.1.2.2.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.2"><csymbol cd="ambiguous" id="S3.SS2.p1.1.m1.4.4.2.1.2.2.1.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.2">subscript</csymbol><sum id="S3.SS2.p1.1.m1.4.4.2.1.2.2.2.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.2.2.2"></sum><apply id="S3.SS2.p1.1.m1.4.4.2.1.2.2.3.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.2.2.3"><eq id="S3.SS2.p1.1.m1.4.4.2.1.2.2.3.1.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.2.2.3.1"></eq><ci id="S3.SS2.p1.1.m1.4.4.2.1.2.2.3.2.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.2.2.3.2">𝑖</ci><cn type="integer" id="S3.SS2.p1.1.m1.4.4.2.1.2.2.3.3.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.2.2.3.3">1</cn></apply></apply><ci id="S3.SS2.p1.1.m1.4.4.2.1.2.3.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.2.3">𝑏</ci></apply><apply id="S3.SS2.p1.1.m1.4.4.2.1.1.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.1"><times id="S3.SS2.p1.1.m1.4.4.2.1.1.2.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.1.2"></times><ci id="S3.SS2.p1.1.m1.4.4.2.1.1.3.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.1.3">ℓ</ci><apply id="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.1.1.1"><csymbol cd="latexml" id="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.1">conditional</csymbol><apply id="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.2.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.2.1.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.2">subscript</csymbol><ci id="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.2.2.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.2.2">𝒙</ci><ci id="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.2.3.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.2.3">𝑖</ci></apply><list id="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.3.1.cmml" xref="S3.SS2.p1.1.m1.4.4.2.1.1.1.1.1.3.2"><ci id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1">𝜃</ci><ci id="S3.SS2.p1.1.m1.2.2.cmml" xref="S3.SS2.p1.1.m1.2.2">ℬ</ci></list></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.4c">\ell(\mathcal{B}|\theta)=\frac{1}{b}\sum_{i=1}^{b}\ell({\bm{x}}_{i}|\theta,\mathcal{B})</annotation></semantics></math>, where the conditional loss <math id="S3.SS2.p1.2.m2.3" class="ltx_Math" alttext="\ell({\bm{x}}_{i}|\theta,\mathcal{B})" display="inline"><semantics id="S3.SS2.p1.2.m2.3a"><mrow id="S3.SS2.p1.2.m2.3.3" xref="S3.SS2.p1.2.m2.3.3.cmml"><mi mathvariant="normal" id="S3.SS2.p1.2.m2.3.3.3" xref="S3.SS2.p1.2.m2.3.3.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.3.3.2" xref="S3.SS2.p1.2.m2.3.3.2.cmml">​</mo><mrow id="S3.SS2.p1.2.m2.3.3.1.1" xref="S3.SS2.p1.2.m2.3.3.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p1.2.m2.3.3.1.1.2" xref="S3.SS2.p1.2.m2.3.3.1.1.1.cmml">(</mo><mrow id="S3.SS2.p1.2.m2.3.3.1.1.1" xref="S3.SS2.p1.2.m2.3.3.1.1.1.cmml"><msub id="S3.SS2.p1.2.m2.3.3.1.1.1.2" xref="S3.SS2.p1.2.m2.3.3.1.1.1.2.cmml"><mi id="S3.SS2.p1.2.m2.3.3.1.1.1.2.2" xref="S3.SS2.p1.2.m2.3.3.1.1.1.2.2.cmml">𝒙</mi><mi id="S3.SS2.p1.2.m2.3.3.1.1.1.2.3" xref="S3.SS2.p1.2.m2.3.3.1.1.1.2.3.cmml">i</mi></msub><mo fence="false" id="S3.SS2.p1.2.m2.3.3.1.1.1.1" xref="S3.SS2.p1.2.m2.3.3.1.1.1.1.cmml">|</mo><mrow id="S3.SS2.p1.2.m2.3.3.1.1.1.3.2" xref="S3.SS2.p1.2.m2.3.3.1.1.1.3.1.cmml"><mi id="S3.SS2.p1.2.m2.1.1" xref="S3.SS2.p1.2.m2.1.1.cmml">θ</mi><mo id="S3.SS2.p1.2.m2.3.3.1.1.1.3.2.1" xref="S3.SS2.p1.2.m2.3.3.1.1.1.3.1.cmml">,</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.2.m2.2.2" xref="S3.SS2.p1.2.m2.2.2.cmml">ℬ</mi></mrow></mrow><mo stretchy="false" id="S3.SS2.p1.2.m2.3.3.1.1.3" xref="S3.SS2.p1.2.m2.3.3.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.2.m2.3b"><apply id="S3.SS2.p1.2.m2.3.3.cmml" xref="S3.SS2.p1.2.m2.3.3"><times id="S3.SS2.p1.2.m2.3.3.2.cmml" xref="S3.SS2.p1.2.m2.3.3.2"></times><ci id="S3.SS2.p1.2.m2.3.3.3.cmml" xref="S3.SS2.p1.2.m2.3.3.3">ℓ</ci><apply id="S3.SS2.p1.2.m2.3.3.1.1.1.cmml" xref="S3.SS2.p1.2.m2.3.3.1.1"><csymbol cd="latexml" id="S3.SS2.p1.2.m2.3.3.1.1.1.1.cmml" xref="S3.SS2.p1.2.m2.3.3.1.1.1.1">conditional</csymbol><apply id="S3.SS2.p1.2.m2.3.3.1.1.1.2.cmml" xref="S3.SS2.p1.2.m2.3.3.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m2.3.3.1.1.1.2.1.cmml" xref="S3.SS2.p1.2.m2.3.3.1.1.1.2">subscript</csymbol><ci id="S3.SS2.p1.2.m2.3.3.1.1.1.2.2.cmml" xref="S3.SS2.p1.2.m2.3.3.1.1.1.2.2">𝒙</ci><ci id="S3.SS2.p1.2.m2.3.3.1.1.1.2.3.cmml" xref="S3.SS2.p1.2.m2.3.3.1.1.1.2.3">𝑖</ci></apply><list id="S3.SS2.p1.2.m2.3.3.1.1.1.3.1.cmml" xref="S3.SS2.p1.2.m2.3.3.1.1.1.3.2"><ci id="S3.SS2.p1.2.m2.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1">𝜃</ci><ci id="S3.SS2.p1.2.m2.2.2.cmml" xref="S3.SS2.p1.2.m2.2.2">ℬ</ci></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.2.m2.3c">\ell({\bm{x}}_{i}|\theta,\mathcal{B})</annotation></semantics></math> can use a sigmoid or softmax contrast function (see Appendix Equations <a href="#A1.E1" title="In A.6 Contrastive loss ablations ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> and <a href="#A1.E2" title="In A.6 Contrastive loss ablations ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>).
Since <cite class="ltx_cite ltx_citemacro_citet">Zhai et al. [<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite> demonstrate the sigmoid-contrastive loss to be a more scalable alternative to the softmax-contrastive one, we adopt it by default. Nevertheless, we show in Appendix <a href="#A1.SS6" title="A.6 Contrastive loss ablations ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.6</span></a> that our results also hold when using the softmax-contrastive loss.</p>
</div>
<div id="S3.SS2.p2" class="ltx_para ltx_noindent">
<p id="S3.SS2.p2.9" class="ltx_p"><span id="S3.SS2.p2.9.1" class="ltx_text ltx_font_bold">Joint example selection.</span>
Because the contrastive loss of a batch decomposes into a sum of conditional losses, the <span id="S3.SS2.p2.9.2" class="ltx_text ltx_font_italic">joint learnability</span> of the batch <math id="S3.SS2.p2.1.m1.12" class="ltx_Math" alttext="s(\mathcal{B}|\theta,\theta^{*})\triangleq\ell(\mathcal{B}|\theta)-\ell(\mathcal{B}|\theta^{*})=\frac{1}{b}\sum_{i=1}^{b}\ell({\bm{x}}_{i}|\theta,\mathcal{B})-\ell({\bm{x}}_{i}|\theta^{*},\mathcal{B})=\frac{1}{b}\sum_{i=1}^{b}s({\bm{x}}|\theta,\theta^{*},\mathcal{B})" display="inline"><semantics id="S3.SS2.p2.1.m1.12a"><mrow id="S3.SS2.p2.1.m1.12.12" xref="S3.SS2.p2.1.m1.12.12.cmml"><mrow id="S3.SS2.p2.1.m1.7.7.1" xref="S3.SS2.p2.1.m1.7.7.1.cmml"><mi id="S3.SS2.p2.1.m1.7.7.1.3" xref="S3.SS2.p2.1.m1.7.7.1.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.1.m1.7.7.1.2" xref="S3.SS2.p2.1.m1.7.7.1.2.cmml">​</mo><mrow id="S3.SS2.p2.1.m1.7.7.1.1.1" xref="S3.SS2.p2.1.m1.7.7.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p2.1.m1.7.7.1.1.1.2" xref="S3.SS2.p2.1.m1.7.7.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p2.1.m1.7.7.1.1.1.1" xref="S3.SS2.p2.1.m1.7.7.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.1.m1.7.7.1.1.1.1.3" xref="S3.SS2.p2.1.m1.7.7.1.1.1.1.3.cmml">ℬ</mi><mo fence="false" id="S3.SS2.p2.1.m1.7.7.1.1.1.1.2" xref="S3.SS2.p2.1.m1.7.7.1.1.1.1.2.cmml">|</mo><mrow id="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.1" xref="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.2.cmml"><mi id="S3.SS2.p2.1.m1.1.1" xref="S3.SS2.p2.1.m1.1.1.cmml">θ</mi><mo id="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.1.2" xref="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.2.cmml">,</mo><msup id="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.1.1" xref="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.1.1.cmml"><mi id="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.1.1.2" xref="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.1.1.2.cmml">θ</mi><mo id="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.1.1.3" xref="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.1.1.3.cmml">∗</mo></msup></mrow></mrow><mo stretchy="false" id="S3.SS2.p2.1.m1.7.7.1.1.1.3" xref="S3.SS2.p2.1.m1.7.7.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS2.p2.1.m1.12.12.8" xref="S3.SS2.p2.1.m1.12.12.8.cmml">≜</mo><mrow id="S3.SS2.p2.1.m1.9.9.3" xref="S3.SS2.p2.1.m1.9.9.3.cmml"><mrow id="S3.SS2.p2.1.m1.8.8.2.1" xref="S3.SS2.p2.1.m1.8.8.2.1.cmml"><mi mathvariant="normal" id="S3.SS2.p2.1.m1.8.8.2.1.3" xref="S3.SS2.p2.1.m1.8.8.2.1.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.1.m1.8.8.2.1.2" xref="S3.SS2.p2.1.m1.8.8.2.1.2.cmml">​</mo><mrow id="S3.SS2.p2.1.m1.8.8.2.1.1.1" xref="S3.SS2.p2.1.m1.8.8.2.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p2.1.m1.8.8.2.1.1.1.2" xref="S3.SS2.p2.1.m1.8.8.2.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p2.1.m1.8.8.2.1.1.1.1" xref="S3.SS2.p2.1.m1.8.8.2.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.1.m1.8.8.2.1.1.1.1.2" xref="S3.SS2.p2.1.m1.8.8.2.1.1.1.1.2.cmml">ℬ</mi><mo fence="false" id="S3.SS2.p2.1.m1.8.8.2.1.1.1.1.1" xref="S3.SS2.p2.1.m1.8.8.2.1.1.1.1.1.cmml">|</mo><mi id="S3.SS2.p2.1.m1.8.8.2.1.1.1.1.3" xref="S3.SS2.p2.1.m1.8.8.2.1.1.1.1.3.cmml">θ</mi></mrow><mo stretchy="false" id="S3.SS2.p2.1.m1.8.8.2.1.1.1.3" xref="S3.SS2.p2.1.m1.8.8.2.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS2.p2.1.m1.9.9.3.3" xref="S3.SS2.p2.1.m1.9.9.3.3.cmml">−</mo><mrow id="S3.SS2.p2.1.m1.9.9.3.2" xref="S3.SS2.p2.1.m1.9.9.3.2.cmml"><mi mathvariant="normal" id="S3.SS2.p2.1.m1.9.9.3.2.3" xref="S3.SS2.p2.1.m1.9.9.3.2.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.1.m1.9.9.3.2.2" xref="S3.SS2.p2.1.m1.9.9.3.2.2.cmml">​</mo><mrow id="S3.SS2.p2.1.m1.9.9.3.2.1.1" xref="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p2.1.m1.9.9.3.2.1.1.2" xref="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.cmml">(</mo><mrow id="S3.SS2.p2.1.m1.9.9.3.2.1.1.1" xref="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.2" xref="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.2.cmml">ℬ</mi><mo fence="false" id="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.1" xref="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.1.cmml">|</mo><msup id="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.3" xref="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.3.cmml"><mi id="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.3.2" xref="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.3.2.cmml">θ</mi><mo id="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.3.3" xref="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.3.3.cmml">∗</mo></msup></mrow><mo stretchy="false" id="S3.SS2.p2.1.m1.9.9.3.2.1.1.3" xref="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo id="S3.SS2.p2.1.m1.12.12.9" xref="S3.SS2.p2.1.m1.12.12.9.cmml">=</mo><mrow id="S3.SS2.p2.1.m1.11.11.5" xref="S3.SS2.p2.1.m1.11.11.5.cmml"><mrow id="S3.SS2.p2.1.m1.10.10.4.1" xref="S3.SS2.p2.1.m1.10.10.4.1.cmml"><mfrac id="S3.SS2.p2.1.m1.10.10.4.1.3" xref="S3.SS2.p2.1.m1.10.10.4.1.3.cmml"><mn id="S3.SS2.p2.1.m1.10.10.4.1.3.2" xref="S3.SS2.p2.1.m1.10.10.4.1.3.2.cmml">1</mn><mi id="S3.SS2.p2.1.m1.10.10.4.1.3.3" xref="S3.SS2.p2.1.m1.10.10.4.1.3.3.cmml">b</mi></mfrac><mo lspace="0em" rspace="0em" id="S3.SS2.p2.1.m1.10.10.4.1.2" xref="S3.SS2.p2.1.m1.10.10.4.1.2.cmml">​</mo><mrow id="S3.SS2.p2.1.m1.10.10.4.1.1" xref="S3.SS2.p2.1.m1.10.10.4.1.1.cmml"><msubsup id="S3.SS2.p2.1.m1.10.10.4.1.1.2" xref="S3.SS2.p2.1.m1.10.10.4.1.1.2.cmml"><mo id="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.2" xref="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.2.cmml">∑</mo><mrow id="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.3" xref="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.3.cmml"><mi id="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.3.2" xref="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.3.2.cmml">i</mi><mo id="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.3.1" xref="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.3.1.cmml">=</mo><mn id="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.3.3" xref="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="S3.SS2.p2.1.m1.10.10.4.1.1.2.3" xref="S3.SS2.p2.1.m1.10.10.4.1.1.2.3.cmml">b</mi></msubsup><mrow id="S3.SS2.p2.1.m1.10.10.4.1.1.1" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.cmml"><mi mathvariant="normal" id="S3.SS2.p2.1.m1.10.10.4.1.1.1.3" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.1.m1.10.10.4.1.1.1.2" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.2.cmml">​</mo><mrow id="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.2" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.cmml"><msub id="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.2" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.2.cmml"><mi id="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.2.2" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.2.2.cmml">𝒙</mi><mi id="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.2.3" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo fence="false" id="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.1" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.1.cmml">|</mo><mrow id="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.3.2" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.3.1.cmml"><mi id="S3.SS2.p2.1.m1.2.2" xref="S3.SS2.p2.1.m1.2.2.cmml">θ</mi><mo id="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.3.2.1" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.3.1.cmml">,</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.1.m1.3.3" xref="S3.SS2.p2.1.m1.3.3.cmml">ℬ</mi></mrow></mrow><mo stretchy="false" id="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.3" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><mo id="S3.SS2.p2.1.m1.11.11.5.3" xref="S3.SS2.p2.1.m1.11.11.5.3.cmml">−</mo><mrow id="S3.SS2.p2.1.m1.11.11.5.2" xref="S3.SS2.p2.1.m1.11.11.5.2.cmml"><mi mathvariant="normal" id="S3.SS2.p2.1.m1.11.11.5.2.3" xref="S3.SS2.p2.1.m1.11.11.5.2.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.1.m1.11.11.5.2.2" xref="S3.SS2.p2.1.m1.11.11.5.2.2.cmml">​</mo><mrow id="S3.SS2.p2.1.m1.11.11.5.2.1.1" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p2.1.m1.11.11.5.2.1.1.2" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.cmml">(</mo><mrow id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.cmml"><msub id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.3" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.3.cmml"><mi id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.3.2" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.3.2.cmml">𝒙</mi><mi id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.3.3" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.3.3.cmml">i</mi></msub><mo fence="false" id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.2" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.2.cmml">|</mo><mrow id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.1" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.2.cmml"><msup id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.1.1" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.1.1.cmml"><mi id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.1.1.2" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.1.1.2.cmml">θ</mi><mo id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.1.1.3" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.1.1.3.cmml">∗</mo></msup><mo id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.1.2" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.2.cmml">,</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.1.m1.4.4" xref="S3.SS2.p2.1.m1.4.4.cmml">ℬ</mi></mrow></mrow><mo stretchy="false" id="S3.SS2.p2.1.m1.11.11.5.2.1.1.3" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo id="S3.SS2.p2.1.m1.12.12.10" xref="S3.SS2.p2.1.m1.12.12.10.cmml">=</mo><mrow id="S3.SS2.p2.1.m1.12.12.6" xref="S3.SS2.p2.1.m1.12.12.6.cmml"><mfrac id="S3.SS2.p2.1.m1.12.12.6.3" xref="S3.SS2.p2.1.m1.12.12.6.3.cmml"><mn id="S3.SS2.p2.1.m1.12.12.6.3.2" xref="S3.SS2.p2.1.m1.12.12.6.3.2.cmml">1</mn><mi id="S3.SS2.p2.1.m1.12.12.6.3.3" xref="S3.SS2.p2.1.m1.12.12.6.3.3.cmml">b</mi></mfrac><mo lspace="0em" rspace="0em" id="S3.SS2.p2.1.m1.12.12.6.2" xref="S3.SS2.p2.1.m1.12.12.6.2.cmml">​</mo><mrow id="S3.SS2.p2.1.m1.12.12.6.1" xref="S3.SS2.p2.1.m1.12.12.6.1.cmml"><msubsup id="S3.SS2.p2.1.m1.12.12.6.1.2" xref="S3.SS2.p2.1.m1.12.12.6.1.2.cmml"><mo id="S3.SS2.p2.1.m1.12.12.6.1.2.2.2" xref="S3.SS2.p2.1.m1.12.12.6.1.2.2.2.cmml">∑</mo><mrow id="S3.SS2.p2.1.m1.12.12.6.1.2.2.3" xref="S3.SS2.p2.1.m1.12.12.6.1.2.2.3.cmml"><mi id="S3.SS2.p2.1.m1.12.12.6.1.2.2.3.2" xref="S3.SS2.p2.1.m1.12.12.6.1.2.2.3.2.cmml">i</mi><mo id="S3.SS2.p2.1.m1.12.12.6.1.2.2.3.1" xref="S3.SS2.p2.1.m1.12.12.6.1.2.2.3.1.cmml">=</mo><mn id="S3.SS2.p2.1.m1.12.12.6.1.2.2.3.3" xref="S3.SS2.p2.1.m1.12.12.6.1.2.2.3.3.cmml">1</mn></mrow><mi id="S3.SS2.p2.1.m1.12.12.6.1.2.3" xref="S3.SS2.p2.1.m1.12.12.6.1.2.3.cmml">b</mi></msubsup><mrow id="S3.SS2.p2.1.m1.12.12.6.1.1" xref="S3.SS2.p2.1.m1.12.12.6.1.1.cmml"><mi id="S3.SS2.p2.1.m1.12.12.6.1.1.3" xref="S3.SS2.p2.1.m1.12.12.6.1.1.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.1.m1.12.12.6.1.1.2" xref="S3.SS2.p2.1.m1.12.12.6.1.1.2.cmml">​</mo><mrow id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.2" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.cmml"><mi id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.3" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.3.cmml">𝒙</mi><mo fence="false" id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.2" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.2.cmml">|</mo><mrow id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.1" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.2.cmml"><mi id="S3.SS2.p2.1.m1.5.5" xref="S3.SS2.p2.1.m1.5.5.cmml">θ</mi><mo id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.1.2" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.2.cmml">,</mo><msup id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.1.1" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.1.1.cmml"><mi id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.1.1.2" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.1.1.2.cmml">θ</mi><mo id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.1.1.3" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.1.1.3.cmml">∗</mo></msup><mo id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.1.3" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.2.cmml">,</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.1.m1.6.6" xref="S3.SS2.p2.1.m1.6.6.cmml">ℬ</mi></mrow></mrow><mo stretchy="false" id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.3" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.1.m1.12b"><apply id="S3.SS2.p2.1.m1.12.12.cmml" xref="S3.SS2.p2.1.m1.12.12"><and id="S3.SS2.p2.1.m1.12.12a.cmml" xref="S3.SS2.p2.1.m1.12.12"></and><apply id="S3.SS2.p2.1.m1.12.12b.cmml" xref="S3.SS2.p2.1.m1.12.12"><ci id="S3.SS2.p2.1.m1.12.12.8.cmml" xref="S3.SS2.p2.1.m1.12.12.8">≜</ci><apply id="S3.SS2.p2.1.m1.7.7.1.cmml" xref="S3.SS2.p2.1.m1.7.7.1"><times id="S3.SS2.p2.1.m1.7.7.1.2.cmml" xref="S3.SS2.p2.1.m1.7.7.1.2"></times><ci id="S3.SS2.p2.1.m1.7.7.1.3.cmml" xref="S3.SS2.p2.1.m1.7.7.1.3">𝑠</ci><apply id="S3.SS2.p2.1.m1.7.7.1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.7.7.1.1.1"><csymbol cd="latexml" id="S3.SS2.p2.1.m1.7.7.1.1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.7.7.1.1.1.1.2">conditional</csymbol><ci id="S3.SS2.p2.1.m1.7.7.1.1.1.1.3.cmml" xref="S3.SS2.p2.1.m1.7.7.1.1.1.1.3">ℬ</ci><list id="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.1"><ci id="S3.SS2.p2.1.m1.1.1.cmml" xref="S3.SS2.p2.1.m1.1.1">𝜃</ci><apply id="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.1.1">superscript</csymbol><ci id="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.1.1.2">𝜃</ci><times id="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p2.1.m1.7.7.1.1.1.1.1.1.1.3"></times></apply></list></apply></apply><apply id="S3.SS2.p2.1.m1.9.9.3.cmml" xref="S3.SS2.p2.1.m1.9.9.3"><minus id="S3.SS2.p2.1.m1.9.9.3.3.cmml" xref="S3.SS2.p2.1.m1.9.9.3.3"></minus><apply id="S3.SS2.p2.1.m1.8.8.2.1.cmml" xref="S3.SS2.p2.1.m1.8.8.2.1"><times id="S3.SS2.p2.1.m1.8.8.2.1.2.cmml" xref="S3.SS2.p2.1.m1.8.8.2.1.2"></times><ci id="S3.SS2.p2.1.m1.8.8.2.1.3.cmml" xref="S3.SS2.p2.1.m1.8.8.2.1.3">ℓ</ci><apply id="S3.SS2.p2.1.m1.8.8.2.1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.8.8.2.1.1.1"><csymbol cd="latexml" id="S3.SS2.p2.1.m1.8.8.2.1.1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.8.8.2.1.1.1.1.1">conditional</csymbol><ci id="S3.SS2.p2.1.m1.8.8.2.1.1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.8.8.2.1.1.1.1.2">ℬ</ci><ci id="S3.SS2.p2.1.m1.8.8.2.1.1.1.1.3.cmml" xref="S3.SS2.p2.1.m1.8.8.2.1.1.1.1.3">𝜃</ci></apply></apply><apply id="S3.SS2.p2.1.m1.9.9.3.2.cmml" xref="S3.SS2.p2.1.m1.9.9.3.2"><times id="S3.SS2.p2.1.m1.9.9.3.2.2.cmml" xref="S3.SS2.p2.1.m1.9.9.3.2.2"></times><ci id="S3.SS2.p2.1.m1.9.9.3.2.3.cmml" xref="S3.SS2.p2.1.m1.9.9.3.2.3">ℓ</ci><apply id="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.cmml" xref="S3.SS2.p2.1.m1.9.9.3.2.1.1"><csymbol cd="latexml" id="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.1">conditional</csymbol><ci id="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.2">ℬ</ci><apply id="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.3.cmml" xref="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.3.1.cmml" xref="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.3">superscript</csymbol><ci id="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.3.2.cmml" xref="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.3.2">𝜃</ci><times id="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.3.3.cmml" xref="S3.SS2.p2.1.m1.9.9.3.2.1.1.1.3.3"></times></apply></apply></apply></apply></apply><apply id="S3.SS2.p2.1.m1.12.12c.cmml" xref="S3.SS2.p2.1.m1.12.12"><eq id="S3.SS2.p2.1.m1.12.12.9.cmml" xref="S3.SS2.p2.1.m1.12.12.9"></eq><share href="#S3.SS2.p2.1.m1.9.9.3.cmml" id="S3.SS2.p2.1.m1.12.12d.cmml" xref="S3.SS2.p2.1.m1.12.12"></share><apply id="S3.SS2.p2.1.m1.11.11.5.cmml" xref="S3.SS2.p2.1.m1.11.11.5"><minus id="S3.SS2.p2.1.m1.11.11.5.3.cmml" xref="S3.SS2.p2.1.m1.11.11.5.3"></minus><apply id="S3.SS2.p2.1.m1.10.10.4.1.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1"><times id="S3.SS2.p2.1.m1.10.10.4.1.2.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.2"></times><apply id="S3.SS2.p2.1.m1.10.10.4.1.3.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.3"><divide id="S3.SS2.p2.1.m1.10.10.4.1.3.1.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.3"></divide><cn type="integer" id="S3.SS2.p2.1.m1.10.10.4.1.3.2.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.3.2">1</cn><ci id="S3.SS2.p2.1.m1.10.10.4.1.3.3.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.3.3">𝑏</ci></apply><apply id="S3.SS2.p2.1.m1.10.10.4.1.1.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1"><apply id="S3.SS2.p2.1.m1.10.10.4.1.1.2.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p2.1.m1.10.10.4.1.1.2.1.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.2">superscript</csymbol><apply id="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.1.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.2">subscript</csymbol><sum id="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.2.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.2"></sum><apply id="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.3.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.3"><eq id="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.3.1.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.3.1"></eq><ci id="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.3.2.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.3.2">𝑖</ci><cn type="integer" id="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.3.3.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.2.2.3.3">1</cn></apply></apply><ci id="S3.SS2.p2.1.m1.10.10.4.1.1.2.3.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.2.3">𝑏</ci></apply><apply id="S3.SS2.p2.1.m1.10.10.4.1.1.1.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1"><times id="S3.SS2.p2.1.m1.10.10.4.1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.2"></times><ci id="S3.SS2.p2.1.m1.10.10.4.1.1.1.3.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.3">ℓ</ci><apply id="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1"><csymbol cd="latexml" id="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.1">conditional</csymbol><apply id="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.2.1.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.2.2.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.2.2">𝒙</ci><ci id="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.2.3.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.2.3">𝑖</ci></apply><list id="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.3.1.cmml" xref="S3.SS2.p2.1.m1.10.10.4.1.1.1.1.1.1.3.2"><ci id="S3.SS2.p2.1.m1.2.2.cmml" xref="S3.SS2.p2.1.m1.2.2">𝜃</ci><ci id="S3.SS2.p2.1.m1.3.3.cmml" xref="S3.SS2.p2.1.m1.3.3">ℬ</ci></list></apply></apply></apply></apply><apply id="S3.SS2.p2.1.m1.11.11.5.2.cmml" xref="S3.SS2.p2.1.m1.11.11.5.2"><times id="S3.SS2.p2.1.m1.11.11.5.2.2.cmml" xref="S3.SS2.p2.1.m1.11.11.5.2.2"></times><ci id="S3.SS2.p2.1.m1.11.11.5.2.3.cmml" xref="S3.SS2.p2.1.m1.11.11.5.2.3">ℓ</ci><apply id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.cmml" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1"><csymbol cd="latexml" id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.2">conditional</csymbol><apply id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.3.cmml" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.3.1.cmml" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.3">subscript</csymbol><ci id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.3.2.cmml" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.3.2">𝒙</ci><ci id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.3.3.cmml" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.3.3">𝑖</ci></apply><list id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.1"><apply id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.1.1">superscript</csymbol><ci id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.1.1.2">𝜃</ci><times id="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p2.1.m1.11.11.5.2.1.1.1.1.1.1.3"></times></apply><ci id="S3.SS2.p2.1.m1.4.4.cmml" xref="S3.SS2.p2.1.m1.4.4">ℬ</ci></list></apply></apply></apply></apply><apply id="S3.SS2.p2.1.m1.12.12e.cmml" xref="S3.SS2.p2.1.m1.12.12"><eq id="S3.SS2.p2.1.m1.12.12.10.cmml" xref="S3.SS2.p2.1.m1.12.12.10"></eq><share href="#S3.SS2.p2.1.m1.11.11.5.cmml" id="S3.SS2.p2.1.m1.12.12f.cmml" xref="S3.SS2.p2.1.m1.12.12"></share><apply id="S3.SS2.p2.1.m1.12.12.6.cmml" xref="S3.SS2.p2.1.m1.12.12.6"><times id="S3.SS2.p2.1.m1.12.12.6.2.cmml" xref="S3.SS2.p2.1.m1.12.12.6.2"></times><apply id="S3.SS2.p2.1.m1.12.12.6.3.cmml" xref="S3.SS2.p2.1.m1.12.12.6.3"><divide id="S3.SS2.p2.1.m1.12.12.6.3.1.cmml" xref="S3.SS2.p2.1.m1.12.12.6.3"></divide><cn type="integer" id="S3.SS2.p2.1.m1.12.12.6.3.2.cmml" xref="S3.SS2.p2.1.m1.12.12.6.3.2">1</cn><ci id="S3.SS2.p2.1.m1.12.12.6.3.3.cmml" xref="S3.SS2.p2.1.m1.12.12.6.3.3">𝑏</ci></apply><apply id="S3.SS2.p2.1.m1.12.12.6.1.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1"><apply id="S3.SS2.p2.1.m1.12.12.6.1.2.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.2"><csymbol cd="ambiguous" id="S3.SS2.p2.1.m1.12.12.6.1.2.1.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.2">superscript</csymbol><apply id="S3.SS2.p2.1.m1.12.12.6.1.2.2.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.2"><csymbol cd="ambiguous" id="S3.SS2.p2.1.m1.12.12.6.1.2.2.1.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.2">subscript</csymbol><sum id="S3.SS2.p2.1.m1.12.12.6.1.2.2.2.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.2.2.2"></sum><apply id="S3.SS2.p2.1.m1.12.12.6.1.2.2.3.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.2.2.3"><eq id="S3.SS2.p2.1.m1.12.12.6.1.2.2.3.1.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.2.2.3.1"></eq><ci id="S3.SS2.p2.1.m1.12.12.6.1.2.2.3.2.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.2.2.3.2">𝑖</ci><cn type="integer" id="S3.SS2.p2.1.m1.12.12.6.1.2.2.3.3.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.2.2.3.3">1</cn></apply></apply><ci id="S3.SS2.p2.1.m1.12.12.6.1.2.3.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.2.3">𝑏</ci></apply><apply id="S3.SS2.p2.1.m1.12.12.6.1.1.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.1"><times id="S3.SS2.p2.1.m1.12.12.6.1.1.2.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.1.2"></times><ci id="S3.SS2.p2.1.m1.12.12.6.1.1.3.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.1.3">𝑠</ci><apply id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1"><csymbol cd="latexml" id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.2">conditional</csymbol><ci id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.3.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.3">𝒙</ci><list id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.1"><ci id="S3.SS2.p2.1.m1.5.5.cmml" xref="S3.SS2.p2.1.m1.5.5">𝜃</ci><apply id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.1.1">superscript</csymbol><ci id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.1.1.2">𝜃</ci><times id="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p2.1.m1.12.12.6.1.1.1.1.1.1.1.1.3"></times></apply><ci id="S3.SS2.p2.1.m1.6.6.cmml" xref="S3.SS2.p2.1.m1.6.6">ℬ</ci></list></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.1.m1.12c">s(\mathcal{B}|\theta,\theta^{*})\triangleq\ell(\mathcal{B}|\theta)-\ell(\mathcal{B}|\theta^{*})=\frac{1}{b}\sum_{i=1}^{b}\ell({\bm{x}}_{i}|\theta,\mathcal{B})-\ell({\bm{x}}_{i}|\theta^{*},\mathcal{B})=\frac{1}{b}\sum_{i=1}^{b}s({\bm{x}}|\theta,\theta^{*},\mathcal{B})</annotation></semantics></math>
also decomposes into a sum of <span id="S3.SS2.p2.9.3" class="ltx_text ltx_font_italic">conditional learnabilities</span> <math id="S3.SS2.p2.2.m2.3" class="ltx_Math" alttext="s({\bm{x}}|\theta,\theta^{*},\mathcal{B})" display="inline"><semantics id="S3.SS2.p2.2.m2.3a"><mrow id="S3.SS2.p2.2.m2.3.3" xref="S3.SS2.p2.2.m2.3.3.cmml"><mi id="S3.SS2.p2.2.m2.3.3.3" xref="S3.SS2.p2.2.m2.3.3.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.2.m2.3.3.2" xref="S3.SS2.p2.2.m2.3.3.2.cmml">​</mo><mrow id="S3.SS2.p2.2.m2.3.3.1.1" xref="S3.SS2.p2.2.m2.3.3.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p2.2.m2.3.3.1.1.2" xref="S3.SS2.p2.2.m2.3.3.1.1.1.cmml">(</mo><mrow id="S3.SS2.p2.2.m2.3.3.1.1.1" xref="S3.SS2.p2.2.m2.3.3.1.1.1.cmml"><mi id="S3.SS2.p2.2.m2.3.3.1.1.1.3" xref="S3.SS2.p2.2.m2.3.3.1.1.1.3.cmml">𝒙</mi><mo fence="false" id="S3.SS2.p2.2.m2.3.3.1.1.1.2" xref="S3.SS2.p2.2.m2.3.3.1.1.1.2.cmml">|</mo><mrow id="S3.SS2.p2.2.m2.3.3.1.1.1.1.1" xref="S3.SS2.p2.2.m2.3.3.1.1.1.1.2.cmml"><mi id="S3.SS2.p2.2.m2.1.1" xref="S3.SS2.p2.2.m2.1.1.cmml">θ</mi><mo id="S3.SS2.p2.2.m2.3.3.1.1.1.1.1.2" xref="S3.SS2.p2.2.m2.3.3.1.1.1.1.2.cmml">,</mo><msup id="S3.SS2.p2.2.m2.3.3.1.1.1.1.1.1" xref="S3.SS2.p2.2.m2.3.3.1.1.1.1.1.1.cmml"><mi id="S3.SS2.p2.2.m2.3.3.1.1.1.1.1.1.2" xref="S3.SS2.p2.2.m2.3.3.1.1.1.1.1.1.2.cmml">θ</mi><mo id="S3.SS2.p2.2.m2.3.3.1.1.1.1.1.1.3" xref="S3.SS2.p2.2.m2.3.3.1.1.1.1.1.1.3.cmml">∗</mo></msup><mo id="S3.SS2.p2.2.m2.3.3.1.1.1.1.1.3" xref="S3.SS2.p2.2.m2.3.3.1.1.1.1.2.cmml">,</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.2.m2.2.2" xref="S3.SS2.p2.2.m2.2.2.cmml">ℬ</mi></mrow></mrow><mo stretchy="false" id="S3.SS2.p2.2.m2.3.3.1.1.3" xref="S3.SS2.p2.2.m2.3.3.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.2.m2.3b"><apply id="S3.SS2.p2.2.m2.3.3.cmml" xref="S3.SS2.p2.2.m2.3.3"><times id="S3.SS2.p2.2.m2.3.3.2.cmml" xref="S3.SS2.p2.2.m2.3.3.2"></times><ci id="S3.SS2.p2.2.m2.3.3.3.cmml" xref="S3.SS2.p2.2.m2.3.3.3">𝑠</ci><apply id="S3.SS2.p2.2.m2.3.3.1.1.1.cmml" xref="S3.SS2.p2.2.m2.3.3.1.1"><csymbol cd="latexml" id="S3.SS2.p2.2.m2.3.3.1.1.1.2.cmml" xref="S3.SS2.p2.2.m2.3.3.1.1.1.2">conditional</csymbol><ci id="S3.SS2.p2.2.m2.3.3.1.1.1.3.cmml" xref="S3.SS2.p2.2.m2.3.3.1.1.1.3">𝒙</ci><list id="S3.SS2.p2.2.m2.3.3.1.1.1.1.2.cmml" xref="S3.SS2.p2.2.m2.3.3.1.1.1.1.1"><ci id="S3.SS2.p2.2.m2.1.1.cmml" xref="S3.SS2.p2.2.m2.1.1">𝜃</ci><apply id="S3.SS2.p2.2.m2.3.3.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.2.m2.3.3.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.2.m2.3.3.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.2.m2.3.3.1.1.1.1.1.1">superscript</csymbol><ci id="S3.SS2.p2.2.m2.3.3.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.2.m2.3.3.1.1.1.1.1.1.2">𝜃</ci><times id="S3.SS2.p2.2.m2.3.3.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p2.2.m2.3.3.1.1.1.1.1.1.3"></times></apply><ci id="S3.SS2.p2.2.m2.2.2.cmml" xref="S3.SS2.p2.2.m2.2.2">ℬ</ci></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.2.m2.3c">s({\bm{x}}|\theta,\theta^{*},\mathcal{B})</annotation></semantics></math> of each example given other examples in the batch.
We wish to sample batches in proportion to their joint learnability, i.e. <math id="S3.SS2.p2.3.m3.4" class="ltx_Math" alttext="p(\{X_{k}\}=\mathcal{B})\propto\exp(s(\mathcal{B}|\theta,\theta^{*}))" display="inline"><semantics id="S3.SS2.p2.3.m3.4a"><mrow id="S3.SS2.p2.3.m3.4.4" xref="S3.SS2.p2.3.m3.4.4.cmml"><mrow id="S3.SS2.p2.3.m3.3.3.1" xref="S3.SS2.p2.3.m3.3.3.1.cmml"><mi id="S3.SS2.p2.3.m3.3.3.1.3" xref="S3.SS2.p2.3.m3.3.3.1.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.3.m3.3.3.1.2" xref="S3.SS2.p2.3.m3.3.3.1.2.cmml">​</mo><mrow id="S3.SS2.p2.3.m3.3.3.1.1.1" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p2.3.m3.3.3.1.1.1.2" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p2.3.m3.3.3.1.1.1.1" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.cmml"><mrow id="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.1" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.1.2" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.2.cmml">{</mo><msub id="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.1.1" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.1.1.cmml"><mi id="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.1.1.2" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.1.1.2.cmml">X</mi><mi id="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.1.1.3" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.1.1.3.cmml">k</mi></msub><mo stretchy="false" id="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.1.3" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.2.cmml">}</mo></mrow><mo id="S3.SS2.p2.3.m3.3.3.1.1.1.1.2" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.2.cmml">=</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.3.m3.3.3.1.1.1.1.3" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.3.cmml">ℬ</mi></mrow><mo stretchy="false" id="S3.SS2.p2.3.m3.3.3.1.1.1.3" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS2.p2.3.m3.4.4.3" xref="S3.SS2.p2.3.m3.4.4.3.cmml">∝</mo><mrow id="S3.SS2.p2.3.m3.4.4.2.1" xref="S3.SS2.p2.3.m3.4.4.2.2.cmml"><mi id="S3.SS2.p2.3.m3.2.2" xref="S3.SS2.p2.3.m3.2.2.cmml">exp</mi><mo id="S3.SS2.p2.3.m3.4.4.2.1a" xref="S3.SS2.p2.3.m3.4.4.2.2.cmml">⁡</mo><mrow id="S3.SS2.p2.3.m3.4.4.2.1.1" xref="S3.SS2.p2.3.m3.4.4.2.2.cmml"><mo stretchy="false" id="S3.SS2.p2.3.m3.4.4.2.1.1.2" xref="S3.SS2.p2.3.m3.4.4.2.2.cmml">(</mo><mrow id="S3.SS2.p2.3.m3.4.4.2.1.1.1" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.cmml"><mi id="S3.SS2.p2.3.m3.4.4.2.1.1.1.3" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.3.m3.4.4.2.1.1.1.2" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.2.cmml">​</mo><mrow id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.2" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.3" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.3.cmml">ℬ</mi><mo fence="false" id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.2" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.2.cmml">|</mo><mrow id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.1" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.2.cmml"><mi id="S3.SS2.p2.3.m3.1.1" xref="S3.SS2.p2.3.m3.1.1.cmml">θ</mi><mo id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.1.2" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.2.cmml">,</mo><msup id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.1.1" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.1.1.2" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.1.1.2.cmml">θ</mi><mo id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.1.1.3" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.1.1.3.cmml">∗</mo></msup></mrow></mrow><mo stretchy="false" id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.3" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S3.SS2.p2.3.m3.4.4.2.1.1.3" xref="S3.SS2.p2.3.m3.4.4.2.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.3.m3.4b"><apply id="S3.SS2.p2.3.m3.4.4.cmml" xref="S3.SS2.p2.3.m3.4.4"><csymbol cd="latexml" id="S3.SS2.p2.3.m3.4.4.3.cmml" xref="S3.SS2.p2.3.m3.4.4.3">proportional-to</csymbol><apply id="S3.SS2.p2.3.m3.3.3.1.cmml" xref="S3.SS2.p2.3.m3.3.3.1"><times id="S3.SS2.p2.3.m3.3.3.1.2.cmml" xref="S3.SS2.p2.3.m3.3.3.1.2"></times><ci id="S3.SS2.p2.3.m3.3.3.1.3.cmml" xref="S3.SS2.p2.3.m3.3.3.1.3">𝑝</ci><apply id="S3.SS2.p2.3.m3.3.3.1.1.1.1.cmml" xref="S3.SS2.p2.3.m3.3.3.1.1.1"><eq id="S3.SS2.p2.3.m3.3.3.1.1.1.1.2.cmml" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.2"></eq><set id="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.1"><apply id="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.1.1.2">𝑋</ci><ci id="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.1.1.1.3">𝑘</ci></apply></set><ci id="S3.SS2.p2.3.m3.3.3.1.1.1.1.3.cmml" xref="S3.SS2.p2.3.m3.3.3.1.1.1.1.3">ℬ</ci></apply></apply><apply id="S3.SS2.p2.3.m3.4.4.2.2.cmml" xref="S3.SS2.p2.3.m3.4.4.2.1"><exp id="S3.SS2.p2.3.m3.2.2.cmml" xref="S3.SS2.p2.3.m3.2.2"></exp><apply id="S3.SS2.p2.3.m3.4.4.2.1.1.1.cmml" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1"><times id="S3.SS2.p2.3.m3.4.4.2.1.1.1.2.cmml" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.2"></times><ci id="S3.SS2.p2.3.m3.4.4.2.1.1.1.3.cmml" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.3">𝑠</ci><apply id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1"><csymbol cd="latexml" id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.2">conditional</csymbol><ci id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.3">ℬ</ci><list id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.1"><ci id="S3.SS2.p2.3.m3.1.1.cmml" xref="S3.SS2.p2.3.m3.1.1">𝜃</ci><apply id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.1.1">superscript</csymbol><ci id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.1.1.2">𝜃</ci><times id="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p2.3.m3.4.4.2.1.1.1.1.1.1.1.1.1.3"></times></apply></list></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.3.m3.4c">p(\{X_{k}\}=\mathcal{B})\propto\exp(s(\mathcal{B}|\theta,\theta^{*}))</annotation></semantics></math>,
which is enabled by a sequential approach inspired by
blocked Gibbs sampling (see Algorithm <a href="#algorithm1" title="Algorithm 1 ‣ 3.2 Joint example selection (JEST) for multimodal learning ‣ 3 Methods ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>). Given a subset of examples <math id="S3.SS2.p2.4.m4.1" class="ltx_Math" alttext="\mathcal{B}_{n}" display="inline"><semantics id="S3.SS2.p2.4.m4.1a"><msub id="S3.SS2.p2.4.m4.1.1" xref="S3.SS2.p2.4.m4.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.4.m4.1.1.2" xref="S3.SS2.p2.4.m4.1.1.2.cmml">ℬ</mi><mi id="S3.SS2.p2.4.m4.1.1.3" xref="S3.SS2.p2.4.m4.1.1.3.cmml">n</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.4.m4.1b"><apply id="S3.SS2.p2.4.m4.1.1.cmml" xref="S3.SS2.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.4.m4.1.1.1.cmml" xref="S3.SS2.p2.4.m4.1.1">subscript</csymbol><ci id="S3.SS2.p2.4.m4.1.1.2.cmml" xref="S3.SS2.p2.4.m4.1.1.2">ℬ</ci><ci id="S3.SS2.p2.4.m4.1.1.3.cmml" xref="S3.SS2.p2.4.m4.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.4.m4.1c">\mathcal{B}_{n}</annotation></semantics></math> already included in the batch at iteration <math id="S3.SS2.p2.5.m5.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S3.SS2.p2.5.m5.1a"><mi id="S3.SS2.p2.5.m5.1.1" xref="S3.SS2.p2.5.m5.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.5.m5.1b"><ci id="S3.SS2.p2.5.m5.1.1.cmml" xref="S3.SS2.p2.5.m5.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.5.m5.1c">n</annotation></semantics></math>, we compute the conditional learnability of remaining candidate examples <math id="S3.SS2.p2.6.m6.1" class="ltx_Math" alttext="{\bm{x}}_{i}" display="inline"><semantics id="S3.SS2.p2.6.m6.1a"><msub id="S3.SS2.p2.6.m6.1.1" xref="S3.SS2.p2.6.m6.1.1.cmml"><mi id="S3.SS2.p2.6.m6.1.1.2" xref="S3.SS2.p2.6.m6.1.1.2.cmml">𝒙</mi><mi id="S3.SS2.p2.6.m6.1.1.3" xref="S3.SS2.p2.6.m6.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.6.m6.1b"><apply id="S3.SS2.p2.6.m6.1.1.cmml" xref="S3.SS2.p2.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.6.m6.1.1.1.cmml" xref="S3.SS2.p2.6.m6.1.1">subscript</csymbol><ci id="S3.SS2.p2.6.m6.1.1.2.cmml" xref="S3.SS2.p2.6.m6.1.1.2">𝒙</ci><ci id="S3.SS2.p2.6.m6.1.1.3.cmml" xref="S3.SS2.p2.6.m6.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.6.m6.1c">{\bm{x}}_{i}</annotation></semantics></math> with <math id="S3.SS2.p2.7.m7.2" class="ltx_Math" alttext="s({\bm{x}}_{i}|\theta,\theta^{*},\mathcal{B}_{n})" display="inline"><semantics id="S3.SS2.p2.7.m7.2a"><mrow id="S3.SS2.p2.7.m7.2.2" xref="S3.SS2.p2.7.m7.2.2.cmml"><mi id="S3.SS2.p2.7.m7.2.2.3" xref="S3.SS2.p2.7.m7.2.2.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.7.m7.2.2.2" xref="S3.SS2.p2.7.m7.2.2.2.cmml">​</mo><mrow id="S3.SS2.p2.7.m7.2.2.1.1" xref="S3.SS2.p2.7.m7.2.2.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p2.7.m7.2.2.1.1.2" xref="S3.SS2.p2.7.m7.2.2.1.1.1.cmml">(</mo><mrow id="S3.SS2.p2.7.m7.2.2.1.1.1" xref="S3.SS2.p2.7.m7.2.2.1.1.1.cmml"><msub id="S3.SS2.p2.7.m7.2.2.1.1.1.4" xref="S3.SS2.p2.7.m7.2.2.1.1.1.4.cmml"><mi id="S3.SS2.p2.7.m7.2.2.1.1.1.4.2" xref="S3.SS2.p2.7.m7.2.2.1.1.1.4.2.cmml">𝒙</mi><mi id="S3.SS2.p2.7.m7.2.2.1.1.1.4.3" xref="S3.SS2.p2.7.m7.2.2.1.1.1.4.3.cmml">i</mi></msub><mo fence="false" id="S3.SS2.p2.7.m7.2.2.1.1.1.3" xref="S3.SS2.p2.7.m7.2.2.1.1.1.3.cmml">|</mo><mrow id="S3.SS2.p2.7.m7.2.2.1.1.1.2.2" xref="S3.SS2.p2.7.m7.2.2.1.1.1.2.3.cmml"><mi id="S3.SS2.p2.7.m7.1.1" xref="S3.SS2.p2.7.m7.1.1.cmml">θ</mi><mo id="S3.SS2.p2.7.m7.2.2.1.1.1.2.2.3" xref="S3.SS2.p2.7.m7.2.2.1.1.1.2.3.cmml">,</mo><msup id="S3.SS2.p2.7.m7.2.2.1.1.1.1.1.1" xref="S3.SS2.p2.7.m7.2.2.1.1.1.1.1.1.cmml"><mi id="S3.SS2.p2.7.m7.2.2.1.1.1.1.1.1.2" xref="S3.SS2.p2.7.m7.2.2.1.1.1.1.1.1.2.cmml">θ</mi><mo id="S3.SS2.p2.7.m7.2.2.1.1.1.1.1.1.3" xref="S3.SS2.p2.7.m7.2.2.1.1.1.1.1.1.3.cmml">∗</mo></msup><mo id="S3.SS2.p2.7.m7.2.2.1.1.1.2.2.4" xref="S3.SS2.p2.7.m7.2.2.1.1.1.2.3.cmml">,</mo><msub id="S3.SS2.p2.7.m7.2.2.1.1.1.2.2.2" xref="S3.SS2.p2.7.m7.2.2.1.1.1.2.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.7.m7.2.2.1.1.1.2.2.2.2" xref="S3.SS2.p2.7.m7.2.2.1.1.1.2.2.2.2.cmml">ℬ</mi><mi id="S3.SS2.p2.7.m7.2.2.1.1.1.2.2.2.3" xref="S3.SS2.p2.7.m7.2.2.1.1.1.2.2.2.3.cmml">n</mi></msub></mrow></mrow><mo stretchy="false" id="S3.SS2.p2.7.m7.2.2.1.1.3" xref="S3.SS2.p2.7.m7.2.2.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.7.m7.2b"><apply id="S3.SS2.p2.7.m7.2.2.cmml" xref="S3.SS2.p2.7.m7.2.2"><times id="S3.SS2.p2.7.m7.2.2.2.cmml" xref="S3.SS2.p2.7.m7.2.2.2"></times><ci id="S3.SS2.p2.7.m7.2.2.3.cmml" xref="S3.SS2.p2.7.m7.2.2.3">𝑠</ci><apply id="S3.SS2.p2.7.m7.2.2.1.1.1.cmml" xref="S3.SS2.p2.7.m7.2.2.1.1"><csymbol cd="latexml" id="S3.SS2.p2.7.m7.2.2.1.1.1.3.cmml" xref="S3.SS2.p2.7.m7.2.2.1.1.1.3">conditional</csymbol><apply id="S3.SS2.p2.7.m7.2.2.1.1.1.4.cmml" xref="S3.SS2.p2.7.m7.2.2.1.1.1.4"><csymbol cd="ambiguous" id="S3.SS2.p2.7.m7.2.2.1.1.1.4.1.cmml" xref="S3.SS2.p2.7.m7.2.2.1.1.1.4">subscript</csymbol><ci id="S3.SS2.p2.7.m7.2.2.1.1.1.4.2.cmml" xref="S3.SS2.p2.7.m7.2.2.1.1.1.4.2">𝒙</ci><ci id="S3.SS2.p2.7.m7.2.2.1.1.1.4.3.cmml" xref="S3.SS2.p2.7.m7.2.2.1.1.1.4.3">𝑖</ci></apply><list id="S3.SS2.p2.7.m7.2.2.1.1.1.2.3.cmml" xref="S3.SS2.p2.7.m7.2.2.1.1.1.2.2"><ci id="S3.SS2.p2.7.m7.1.1.cmml" xref="S3.SS2.p2.7.m7.1.1">𝜃</ci><apply id="S3.SS2.p2.7.m7.2.2.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.7.m7.2.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.7.m7.2.2.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.7.m7.2.2.1.1.1.1.1.1">superscript</csymbol><ci id="S3.SS2.p2.7.m7.2.2.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.7.m7.2.2.1.1.1.1.1.1.2">𝜃</ci><times id="S3.SS2.p2.7.m7.2.2.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p2.7.m7.2.2.1.1.1.1.1.1.3"></times></apply><apply id="S3.SS2.p2.7.m7.2.2.1.1.1.2.2.2.cmml" xref="S3.SS2.p2.7.m7.2.2.1.1.1.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.p2.7.m7.2.2.1.1.1.2.2.2.1.cmml" xref="S3.SS2.p2.7.m7.2.2.1.1.1.2.2.2">subscript</csymbol><ci id="S3.SS2.p2.7.m7.2.2.1.1.1.2.2.2.2.cmml" xref="S3.SS2.p2.7.m7.2.2.1.1.1.2.2.2.2">ℬ</ci><ci id="S3.SS2.p2.7.m7.2.2.1.1.1.2.2.2.3.cmml" xref="S3.SS2.p2.7.m7.2.2.1.1.1.2.2.2.3">𝑛</ci></apply></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.7.m7.2c">s({\bm{x}}_{i}|\theta,\theta^{*},\mathcal{B}_{n})</annotation></semantics></math>, and sample a new chunk of examples <math id="S3.SS2.p2.8.m8.1" class="ltx_Math" alttext="\{X_{k}\}" display="inline"><semantics id="S3.SS2.p2.8.m8.1a"><mrow id="S3.SS2.p2.8.m8.1.1.1" xref="S3.SS2.p2.8.m8.1.1.2.cmml"><mo stretchy="false" id="S3.SS2.p2.8.m8.1.1.1.2" xref="S3.SS2.p2.8.m8.1.1.2.cmml">{</mo><msub id="S3.SS2.p2.8.m8.1.1.1.1" xref="S3.SS2.p2.8.m8.1.1.1.1.cmml"><mi id="S3.SS2.p2.8.m8.1.1.1.1.2" xref="S3.SS2.p2.8.m8.1.1.1.1.2.cmml">X</mi><mi id="S3.SS2.p2.8.m8.1.1.1.1.3" xref="S3.SS2.p2.8.m8.1.1.1.1.3.cmml">k</mi></msub><mo stretchy="false" id="S3.SS2.p2.8.m8.1.1.1.3" xref="S3.SS2.p2.8.m8.1.1.2.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.8.m8.1b"><set id="S3.SS2.p2.8.m8.1.1.2.cmml" xref="S3.SS2.p2.8.m8.1.1.1"><apply id="S3.SS2.p2.8.m8.1.1.1.1.cmml" xref="S3.SS2.p2.8.m8.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.8.m8.1.1.1.1.1.cmml" xref="S3.SS2.p2.8.m8.1.1.1.1">subscript</csymbol><ci id="S3.SS2.p2.8.m8.1.1.1.1.2.cmml" xref="S3.SS2.p2.8.m8.1.1.1.1.2">𝑋</ci><ci id="S3.SS2.p2.8.m8.1.1.1.1.3.cmml" xref="S3.SS2.p2.8.m8.1.1.1.1.3">𝑘</ci></apply></set></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.8.m8.1c">\{X_{k}\}</annotation></semantics></math> independently and without replacement according to these probabilities: <math id="S3.SS2.p2.9.m9.4" class="ltx_Math" alttext="p(X_{k}={\bm{x}}_{i})\propto\exp(s({\bm{x}}_{i}|\theta,\theta^{*},\mathcal{B}_{n}))" display="inline"><semantics id="S3.SS2.p2.9.m9.4a"><mrow id="S3.SS2.p2.9.m9.4.4" xref="S3.SS2.p2.9.m9.4.4.cmml"><mrow id="S3.SS2.p2.9.m9.3.3.1" xref="S3.SS2.p2.9.m9.3.3.1.cmml"><mi id="S3.SS2.p2.9.m9.3.3.1.3" xref="S3.SS2.p2.9.m9.3.3.1.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.9.m9.3.3.1.2" xref="S3.SS2.p2.9.m9.3.3.1.2.cmml">​</mo><mrow id="S3.SS2.p2.9.m9.3.3.1.1.1" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p2.9.m9.3.3.1.1.1.2" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p2.9.m9.3.3.1.1.1.1" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.cmml"><msub id="S3.SS2.p2.9.m9.3.3.1.1.1.1.2" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.2.cmml"><mi id="S3.SS2.p2.9.m9.3.3.1.1.1.1.2.2" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.2.2.cmml">X</mi><mi id="S3.SS2.p2.9.m9.3.3.1.1.1.1.2.3" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.2.3.cmml">k</mi></msub><mo id="S3.SS2.p2.9.m9.3.3.1.1.1.1.1" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.1.cmml">=</mo><msub id="S3.SS2.p2.9.m9.3.3.1.1.1.1.3" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.3.cmml"><mi id="S3.SS2.p2.9.m9.3.3.1.1.1.1.3.2" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.3.2.cmml">𝒙</mi><mi id="S3.SS2.p2.9.m9.3.3.1.1.1.1.3.3" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.3.3.cmml">i</mi></msub></mrow><mo stretchy="false" id="S3.SS2.p2.9.m9.3.3.1.1.1.3" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS2.p2.9.m9.4.4.3" xref="S3.SS2.p2.9.m9.4.4.3.cmml">∝</mo><mrow id="S3.SS2.p2.9.m9.4.4.2.1" xref="S3.SS2.p2.9.m9.4.4.2.2.cmml"><mi id="S3.SS2.p2.9.m9.2.2" xref="S3.SS2.p2.9.m9.2.2.cmml">exp</mi><mo id="S3.SS2.p2.9.m9.4.4.2.1a" xref="S3.SS2.p2.9.m9.4.4.2.2.cmml">⁡</mo><mrow id="S3.SS2.p2.9.m9.4.4.2.1.1" xref="S3.SS2.p2.9.m9.4.4.2.2.cmml"><mo stretchy="false" id="S3.SS2.p2.9.m9.4.4.2.1.1.2" xref="S3.SS2.p2.9.m9.4.4.2.2.cmml">(</mo><mrow id="S3.SS2.p2.9.m9.4.4.2.1.1.1" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.cmml"><mi id="S3.SS2.p2.9.m9.4.4.2.1.1.1.3" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.9.m9.4.4.2.1.1.1.2" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.2.cmml">​</mo><mrow id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.2" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.cmml"><msub id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.4" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.4.cmml"><mi id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.4.2" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.4.2.cmml">𝒙</mi><mi id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.4.3" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.4.3.cmml">i</mi></msub><mo fence="false" id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.3" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.3.cmml">|</mo><mrow id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.2" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.3.cmml"><mi id="S3.SS2.p2.9.m9.1.1" xref="S3.SS2.p2.9.m9.1.1.cmml">θ</mi><mo id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.2.3" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.3.cmml">,</mo><msup id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.1.1.1" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.1.1.1.2" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.1.1.1.2.cmml">θ</mi><mo id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.1.1.1.3" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.1.1.1.3.cmml">∗</mo></msup><mo id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.2.4" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.3.cmml">,</mo><msub id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.2.2" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.2.2.2" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.2.2.2.cmml">ℬ</mi><mi id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.2.2.3" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.2.2.3.cmml">n</mi></msub></mrow></mrow><mo stretchy="false" id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.3" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S3.SS2.p2.9.m9.4.4.2.1.1.3" xref="S3.SS2.p2.9.m9.4.4.2.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.9.m9.4b"><apply id="S3.SS2.p2.9.m9.4.4.cmml" xref="S3.SS2.p2.9.m9.4.4"><csymbol cd="latexml" id="S3.SS2.p2.9.m9.4.4.3.cmml" xref="S3.SS2.p2.9.m9.4.4.3">proportional-to</csymbol><apply id="S3.SS2.p2.9.m9.3.3.1.cmml" xref="S3.SS2.p2.9.m9.3.3.1"><times id="S3.SS2.p2.9.m9.3.3.1.2.cmml" xref="S3.SS2.p2.9.m9.3.3.1.2"></times><ci id="S3.SS2.p2.9.m9.3.3.1.3.cmml" xref="S3.SS2.p2.9.m9.3.3.1.3">𝑝</ci><apply id="S3.SS2.p2.9.m9.3.3.1.1.1.1.cmml" xref="S3.SS2.p2.9.m9.3.3.1.1.1"><eq id="S3.SS2.p2.9.m9.3.3.1.1.1.1.1.cmml" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.1"></eq><apply id="S3.SS2.p2.9.m9.3.3.1.1.1.1.2.cmml" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p2.9.m9.3.3.1.1.1.1.2.1.cmml" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.2">subscript</csymbol><ci id="S3.SS2.p2.9.m9.3.3.1.1.1.1.2.2.cmml" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.2.2">𝑋</ci><ci id="S3.SS2.p2.9.m9.3.3.1.1.1.1.2.3.cmml" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.2.3">𝑘</ci></apply><apply id="S3.SS2.p2.9.m9.3.3.1.1.1.1.3.cmml" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p2.9.m9.3.3.1.1.1.1.3.1.cmml" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.3">subscript</csymbol><ci id="S3.SS2.p2.9.m9.3.3.1.1.1.1.3.2.cmml" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.3.2">𝒙</ci><ci id="S3.SS2.p2.9.m9.3.3.1.1.1.1.3.3.cmml" xref="S3.SS2.p2.9.m9.3.3.1.1.1.1.3.3">𝑖</ci></apply></apply></apply><apply id="S3.SS2.p2.9.m9.4.4.2.2.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1"><exp id="S3.SS2.p2.9.m9.2.2.cmml" xref="S3.SS2.p2.9.m9.2.2"></exp><apply id="S3.SS2.p2.9.m9.4.4.2.1.1.1.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1"><times id="S3.SS2.p2.9.m9.4.4.2.1.1.1.2.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.2"></times><ci id="S3.SS2.p2.9.m9.4.4.2.1.1.1.3.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.3">𝑠</ci><apply id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1"><csymbol cd="latexml" id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.3">conditional</csymbol><apply id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.4.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.4.1.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.4">subscript</csymbol><ci id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.4.2.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.4.2">𝒙</ci><ci id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.4.3.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.4.3">𝑖</ci></apply><list id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.3.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.2"><ci id="S3.SS2.p2.9.m9.1.1.cmml" xref="S3.SS2.p2.9.m9.1.1">𝜃</ci><apply id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.1.1.1">superscript</csymbol><ci id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.1.1.1.2">𝜃</ci><times id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.1.1.1.3"></times></apply><apply id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.2.2.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.2.2.1.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.2.2">subscript</csymbol><ci id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.2.2.2.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.2.2.2">ℬ</ci><ci id="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.2.2.3.cmml" xref="S3.SS2.p2.9.m9.4.4.2.1.1.1.1.1.1.2.2.2.3">𝑛</ci></apply></list></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.9.m9.4c">p(X_{k}={\bm{x}}_{i})\propto\exp(s({\bm{x}}_{i}|\theta,\theta^{*},\mathcal{B}_{n}))</annotation></semantics></math>.</p>
</div>
<div id="S3.SS2.p3" class="ltx_para ltx_noindent">
<p id="S3.SS2.p3.8" class="ltx_p">We update the batch by appending this chunk to the previous subset: <math id="S3.SS2.p3.1.m1.1" class="ltx_Math" alttext="\mathcal{B}_{n+1}=\mathcal{B}_{n}\cup\{X_{k}\}" display="inline"><semantics id="S3.SS2.p3.1.m1.1a"><mrow id="S3.SS2.p3.1.m1.1.1" xref="S3.SS2.p3.1.m1.1.1.cmml"><msub id="S3.SS2.p3.1.m1.1.1.3" xref="S3.SS2.p3.1.m1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p3.1.m1.1.1.3.2" xref="S3.SS2.p3.1.m1.1.1.3.2.cmml">ℬ</mi><mrow id="S3.SS2.p3.1.m1.1.1.3.3" xref="S3.SS2.p3.1.m1.1.1.3.3.cmml"><mi id="S3.SS2.p3.1.m1.1.1.3.3.2" xref="S3.SS2.p3.1.m1.1.1.3.3.2.cmml">n</mi><mo id="S3.SS2.p3.1.m1.1.1.3.3.1" xref="S3.SS2.p3.1.m1.1.1.3.3.1.cmml">+</mo><mn id="S3.SS2.p3.1.m1.1.1.3.3.3" xref="S3.SS2.p3.1.m1.1.1.3.3.3.cmml">1</mn></mrow></msub><mo id="S3.SS2.p3.1.m1.1.1.2" xref="S3.SS2.p3.1.m1.1.1.2.cmml">=</mo><mrow id="S3.SS2.p3.1.m1.1.1.1" xref="S3.SS2.p3.1.m1.1.1.1.cmml"><msub id="S3.SS2.p3.1.m1.1.1.1.3" xref="S3.SS2.p3.1.m1.1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p3.1.m1.1.1.1.3.2" xref="S3.SS2.p3.1.m1.1.1.1.3.2.cmml">ℬ</mi><mi id="S3.SS2.p3.1.m1.1.1.1.3.3" xref="S3.SS2.p3.1.m1.1.1.1.3.3.cmml">n</mi></msub><mo id="S3.SS2.p3.1.m1.1.1.1.2" xref="S3.SS2.p3.1.m1.1.1.1.2.cmml">∪</mo><mrow id="S3.SS2.p3.1.m1.1.1.1.1.1" xref="S3.SS2.p3.1.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS2.p3.1.m1.1.1.1.1.1.2" xref="S3.SS2.p3.1.m1.1.1.1.1.2.cmml">{</mo><msub id="S3.SS2.p3.1.m1.1.1.1.1.1.1" xref="S3.SS2.p3.1.m1.1.1.1.1.1.1.cmml"><mi id="S3.SS2.p3.1.m1.1.1.1.1.1.1.2" xref="S3.SS2.p3.1.m1.1.1.1.1.1.1.2.cmml">X</mi><mi id="S3.SS2.p3.1.m1.1.1.1.1.1.1.3" xref="S3.SS2.p3.1.m1.1.1.1.1.1.1.3.cmml">k</mi></msub><mo stretchy="false" id="S3.SS2.p3.1.m1.1.1.1.1.1.3" xref="S3.SS2.p3.1.m1.1.1.1.1.2.cmml">}</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.1.m1.1b"><apply id="S3.SS2.p3.1.m1.1.1.cmml" xref="S3.SS2.p3.1.m1.1.1"><eq id="S3.SS2.p3.1.m1.1.1.2.cmml" xref="S3.SS2.p3.1.m1.1.1.2"></eq><apply id="S3.SS2.p3.1.m1.1.1.3.cmml" xref="S3.SS2.p3.1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p3.1.m1.1.1.3.1.cmml" xref="S3.SS2.p3.1.m1.1.1.3">subscript</csymbol><ci id="S3.SS2.p3.1.m1.1.1.3.2.cmml" xref="S3.SS2.p3.1.m1.1.1.3.2">ℬ</ci><apply id="S3.SS2.p3.1.m1.1.1.3.3.cmml" xref="S3.SS2.p3.1.m1.1.1.3.3"><plus id="S3.SS2.p3.1.m1.1.1.3.3.1.cmml" xref="S3.SS2.p3.1.m1.1.1.3.3.1"></plus><ci id="S3.SS2.p3.1.m1.1.1.3.3.2.cmml" xref="S3.SS2.p3.1.m1.1.1.3.3.2">𝑛</ci><cn type="integer" id="S3.SS2.p3.1.m1.1.1.3.3.3.cmml" xref="S3.SS2.p3.1.m1.1.1.3.3.3">1</cn></apply></apply><apply id="S3.SS2.p3.1.m1.1.1.1.cmml" xref="S3.SS2.p3.1.m1.1.1.1"><union id="S3.SS2.p3.1.m1.1.1.1.2.cmml" xref="S3.SS2.p3.1.m1.1.1.1.2"></union><apply id="S3.SS2.p3.1.m1.1.1.1.3.cmml" xref="S3.SS2.p3.1.m1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p3.1.m1.1.1.1.3.1.cmml" xref="S3.SS2.p3.1.m1.1.1.1.3">subscript</csymbol><ci id="S3.SS2.p3.1.m1.1.1.1.3.2.cmml" xref="S3.SS2.p3.1.m1.1.1.1.3.2">ℬ</ci><ci id="S3.SS2.p3.1.m1.1.1.1.3.3.cmml" xref="S3.SS2.p3.1.m1.1.1.1.3.3">𝑛</ci></apply><set id="S3.SS2.p3.1.m1.1.1.1.1.2.cmml" xref="S3.SS2.p3.1.m1.1.1.1.1.1"><apply id="S3.SS2.p3.1.m1.1.1.1.1.1.1.cmml" xref="S3.SS2.p3.1.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p3.1.m1.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p3.1.m1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS2.p3.1.m1.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p3.1.m1.1.1.1.1.1.1.2">𝑋</ci><ci id="S3.SS2.p3.1.m1.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p3.1.m1.1.1.1.1.1.1.3">𝑘</ci></apply></set></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.1.m1.1c">\mathcal{B}_{n+1}=\mathcal{B}_{n}\cup\{X_{k}\}</annotation></semantics></math>, and iterate until <math id="S3.SS2.p3.2.m2.1" class="ltx_Math" alttext="n=N" display="inline"><semantics id="S3.SS2.p3.2.m2.1a"><mrow id="S3.SS2.p3.2.m2.1.1" xref="S3.SS2.p3.2.m2.1.1.cmml"><mi id="S3.SS2.p3.2.m2.1.1.2" xref="S3.SS2.p3.2.m2.1.1.2.cmml">n</mi><mo id="S3.SS2.p3.2.m2.1.1.1" xref="S3.SS2.p3.2.m2.1.1.1.cmml">=</mo><mi id="S3.SS2.p3.2.m2.1.1.3" xref="S3.SS2.p3.2.m2.1.1.3.cmml">N</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.2.m2.1b"><apply id="S3.SS2.p3.2.m2.1.1.cmml" xref="S3.SS2.p3.2.m2.1.1"><eq id="S3.SS2.p3.2.m2.1.1.1.cmml" xref="S3.SS2.p3.2.m2.1.1.1"></eq><ci id="S3.SS2.p3.2.m2.1.1.2.cmml" xref="S3.SS2.p3.2.m2.1.1.2">𝑛</ci><ci id="S3.SS2.p3.2.m2.1.1.3.cmml" xref="S3.SS2.p3.2.m2.1.1.3">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.2.m2.1c">n=N</annotation></semantics></math>, the number of chunks. The first chunk <math id="S3.SS2.p3.3.m3.1" class="ltx_Math" alttext="\mathcal{B}_{1}" display="inline"><semantics id="S3.SS2.p3.3.m3.1a"><msub id="S3.SS2.p3.3.m3.1.1" xref="S3.SS2.p3.3.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p3.3.m3.1.1.2" xref="S3.SS2.p3.3.m3.1.1.2.cmml">ℬ</mi><mn id="S3.SS2.p3.3.m3.1.1.3" xref="S3.SS2.p3.3.m3.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.3.m3.1b"><apply id="S3.SS2.p3.3.m3.1.1.cmml" xref="S3.SS2.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.p3.3.m3.1.1.1.cmml" xref="S3.SS2.p3.3.m3.1.1">subscript</csymbol><ci id="S3.SS2.p3.3.m3.1.1.2.cmml" xref="S3.SS2.p3.3.m3.1.1.2">ℬ</ci><cn type="integer" id="S3.SS2.p3.3.m3.1.1.3.cmml" xref="S3.SS2.p3.3.m3.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.3.m3.1c">\mathcal{B}_{1}</annotation></semantics></math> is sampled using unconditional learnability (i.e. self-similarity only) <math id="S3.SS2.p3.4.m4.2" class="ltx_math_unparsed" alttext="s({\bm{x}}_{i}|\theta,\theta^{*},\varnothing))=\ell({\bm{x}}_{i}|\theta,\varnothing)-\ell({\bm{x}}_{i}|\theta^{*},\varnothing)" display="inline"><semantics id="S3.SS2.p3.4.m4.2a"><mrow id="S3.SS2.p3.4.m4.2b"><mi id="S3.SS2.p3.4.m4.2.3">s</mi><mrow id="S3.SS2.p3.4.m4.2.4"><mo stretchy="false" id="S3.SS2.p3.4.m4.2.4.1">(</mo><msub id="S3.SS2.p3.4.m4.2.4.2"><mi id="S3.SS2.p3.4.m4.2.4.2.2">𝒙</mi><mi id="S3.SS2.p3.4.m4.2.4.2.3">i</mi></msub><mo fence="false" rspace="0.167em" stretchy="false" id="S3.SS2.p3.4.m4.2.4.3">|</mo><mi id="S3.SS2.p3.4.m4.1.1">θ</mi><mo id="S3.SS2.p3.4.m4.2.4.4">,</mo><msup id="S3.SS2.p3.4.m4.2.4.5"><mi id="S3.SS2.p3.4.m4.2.4.5.2">θ</mi><mo id="S3.SS2.p3.4.m4.2.4.5.3">∗</mo></msup><mo id="S3.SS2.p3.4.m4.2.4.6">,</mo><mi mathvariant="normal" id="S3.SS2.p3.4.m4.2.2">∅</mi><mo stretchy="false" id="S3.SS2.p3.4.m4.2.4.7">)</mo></mrow><mo stretchy="false" id="S3.SS2.p3.4.m4.2.5">)</mo><mo id="S3.SS2.p3.4.m4.2.6">=</mo><mi mathvariant="normal" id="S3.SS2.p3.4.m4.2.7">ℓ</mi><mo stretchy="false" id="S3.SS2.p3.4.m4.2.8">(</mo><msub id="S3.SS2.p3.4.m4.2.9"><mi id="S3.SS2.p3.4.m4.2.9.2">𝒙</mi><mi id="S3.SS2.p3.4.m4.2.9.3">i</mi></msub><mo fence="false" rspace="0.167em" stretchy="false" id="S3.SS2.p3.4.m4.2.10">|</mo><mi id="S3.SS2.p3.4.m4.2.11">θ</mi><mo id="S3.SS2.p3.4.m4.2.12">,</mo><mi mathvariant="normal" id="S3.SS2.p3.4.m4.2.13">∅</mi><mo stretchy="false" id="S3.SS2.p3.4.m4.2.14">)</mo><mo id="S3.SS2.p3.4.m4.2.15">−</mo><mi mathvariant="normal" id="S3.SS2.p3.4.m4.2.16">ℓ</mi><mo stretchy="false" id="S3.SS2.p3.4.m4.2.17">(</mo><msub id="S3.SS2.p3.4.m4.2.18"><mi id="S3.SS2.p3.4.m4.2.18.2">𝒙</mi><mi id="S3.SS2.p3.4.m4.2.18.3">i</mi></msub><mo fence="false" rspace="0.167em" stretchy="false" id="S3.SS2.p3.4.m4.2.19">|</mo><msup id="S3.SS2.p3.4.m4.2.20"><mi id="S3.SS2.p3.4.m4.2.20.2">θ</mi><mo id="S3.SS2.p3.4.m4.2.20.3">∗</mo></msup><mo id="S3.SS2.p3.4.m4.2.21">,</mo><mi mathvariant="normal" id="S3.SS2.p3.4.m4.2.22">∅</mi><mo stretchy="false" id="S3.SS2.p3.4.m4.2.23">)</mo></mrow><annotation encoding="application/x-tex" id="S3.SS2.p3.4.m4.2c">s({\bm{x}}_{i}|\theta,\theta^{*},\varnothing))=\ell({\bm{x}}_{i}|\theta,\varnothing)-\ell({\bm{x}}_{i}|\theta^{*},\varnothing)</annotation></semantics></math> where the unconditional losses are computed as <math id="S3.SS2.p3.5.m5.3" class="ltx_Math" alttext="\ell({\bm{x}}_{i}|\theta,\varnothing)=-\alpha{\bm{z}}^{\textrm{im}}_{i}{\cdot}{\bm{z}}^{\textrm{txt}}_{i}" display="inline"><semantics id="S3.SS2.p3.5.m5.3a"><mrow id="S3.SS2.p3.5.m5.3.3" xref="S3.SS2.p3.5.m5.3.3.cmml"><mrow id="S3.SS2.p3.5.m5.3.3.1" xref="S3.SS2.p3.5.m5.3.3.1.cmml"><mi mathvariant="normal" id="S3.SS2.p3.5.m5.3.3.1.3" xref="S3.SS2.p3.5.m5.3.3.1.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p3.5.m5.3.3.1.2" xref="S3.SS2.p3.5.m5.3.3.1.2.cmml">​</mo><mrow id="S3.SS2.p3.5.m5.3.3.1.1.1" xref="S3.SS2.p3.5.m5.3.3.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p3.5.m5.3.3.1.1.1.2" xref="S3.SS2.p3.5.m5.3.3.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p3.5.m5.3.3.1.1.1.1" xref="S3.SS2.p3.5.m5.3.3.1.1.1.1.cmml"><msub id="S3.SS2.p3.5.m5.3.3.1.1.1.1.2" xref="S3.SS2.p3.5.m5.3.3.1.1.1.1.2.cmml"><mi id="S3.SS2.p3.5.m5.3.3.1.1.1.1.2.2" xref="S3.SS2.p3.5.m5.3.3.1.1.1.1.2.2.cmml">𝒙</mi><mi id="S3.SS2.p3.5.m5.3.3.1.1.1.1.2.3" xref="S3.SS2.p3.5.m5.3.3.1.1.1.1.2.3.cmml">i</mi></msub><mo fence="false" id="S3.SS2.p3.5.m5.3.3.1.1.1.1.1" xref="S3.SS2.p3.5.m5.3.3.1.1.1.1.1.cmml">|</mo><mrow id="S3.SS2.p3.5.m5.3.3.1.1.1.1.3.2" xref="S3.SS2.p3.5.m5.3.3.1.1.1.1.3.1.cmml"><mi id="S3.SS2.p3.5.m5.1.1" xref="S3.SS2.p3.5.m5.1.1.cmml">θ</mi><mo id="S3.SS2.p3.5.m5.3.3.1.1.1.1.3.2.1" xref="S3.SS2.p3.5.m5.3.3.1.1.1.1.3.1.cmml">,</mo><mi mathvariant="normal" id="S3.SS2.p3.5.m5.2.2" xref="S3.SS2.p3.5.m5.2.2.cmml">∅</mi></mrow></mrow><mo stretchy="false" id="S3.SS2.p3.5.m5.3.3.1.1.1.3" xref="S3.SS2.p3.5.m5.3.3.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS2.p3.5.m5.3.3.2" xref="S3.SS2.p3.5.m5.3.3.2.cmml">=</mo><mrow id="S3.SS2.p3.5.m5.3.3.3" xref="S3.SS2.p3.5.m5.3.3.3.cmml"><mo id="S3.SS2.p3.5.m5.3.3.3a" xref="S3.SS2.p3.5.m5.3.3.3.cmml">−</mo><mrow id="S3.SS2.p3.5.m5.3.3.3.2" xref="S3.SS2.p3.5.m5.3.3.3.2.cmml"><mrow id="S3.SS2.p3.5.m5.3.3.3.2.2" xref="S3.SS2.p3.5.m5.3.3.3.2.2.cmml"><mi id="S3.SS2.p3.5.m5.3.3.3.2.2.2" xref="S3.SS2.p3.5.m5.3.3.3.2.2.2.cmml">α</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p3.5.m5.3.3.3.2.2.1" xref="S3.SS2.p3.5.m5.3.3.3.2.2.1.cmml">​</mo><msubsup id="S3.SS2.p3.5.m5.3.3.3.2.2.3" xref="S3.SS2.p3.5.m5.3.3.3.2.2.3.cmml"><mi id="S3.SS2.p3.5.m5.3.3.3.2.2.3.2.2" xref="S3.SS2.p3.5.m5.3.3.3.2.2.3.2.2.cmml">𝒛</mi><mi id="S3.SS2.p3.5.m5.3.3.3.2.2.3.3" xref="S3.SS2.p3.5.m5.3.3.3.2.2.3.3.cmml">i</mi><mtext id="S3.SS2.p3.5.m5.3.3.3.2.2.3.2.3" xref="S3.SS2.p3.5.m5.3.3.3.2.2.3.2.3a.cmml">im</mtext></msubsup></mrow><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p3.5.m5.3.3.3.2.1" xref="S3.SS2.p3.5.m5.3.3.3.2.1.cmml">⋅</mo><msubsup id="S3.SS2.p3.5.m5.3.3.3.2.3" xref="S3.SS2.p3.5.m5.3.3.3.2.3.cmml"><mi id="S3.SS2.p3.5.m5.3.3.3.2.3.2.2" xref="S3.SS2.p3.5.m5.3.3.3.2.3.2.2.cmml">𝒛</mi><mi id="S3.SS2.p3.5.m5.3.3.3.2.3.3" xref="S3.SS2.p3.5.m5.3.3.3.2.3.3.cmml">i</mi><mtext id="S3.SS2.p3.5.m5.3.3.3.2.3.2.3" xref="S3.SS2.p3.5.m5.3.3.3.2.3.2.3a.cmml">txt</mtext></msubsup></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.5.m5.3b"><apply id="S3.SS2.p3.5.m5.3.3.cmml" xref="S3.SS2.p3.5.m5.3.3"><eq id="S3.SS2.p3.5.m5.3.3.2.cmml" xref="S3.SS2.p3.5.m5.3.3.2"></eq><apply id="S3.SS2.p3.5.m5.3.3.1.cmml" xref="S3.SS2.p3.5.m5.3.3.1"><times id="S3.SS2.p3.5.m5.3.3.1.2.cmml" xref="S3.SS2.p3.5.m5.3.3.1.2"></times><ci id="S3.SS2.p3.5.m5.3.3.1.3.cmml" xref="S3.SS2.p3.5.m5.3.3.1.3">ℓ</ci><apply id="S3.SS2.p3.5.m5.3.3.1.1.1.1.cmml" xref="S3.SS2.p3.5.m5.3.3.1.1.1"><csymbol cd="latexml" id="S3.SS2.p3.5.m5.3.3.1.1.1.1.1.cmml" xref="S3.SS2.p3.5.m5.3.3.1.1.1.1.1">conditional</csymbol><apply id="S3.SS2.p3.5.m5.3.3.1.1.1.1.2.cmml" xref="S3.SS2.p3.5.m5.3.3.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p3.5.m5.3.3.1.1.1.1.2.1.cmml" xref="S3.SS2.p3.5.m5.3.3.1.1.1.1.2">subscript</csymbol><ci id="S3.SS2.p3.5.m5.3.3.1.1.1.1.2.2.cmml" xref="S3.SS2.p3.5.m5.3.3.1.1.1.1.2.2">𝒙</ci><ci id="S3.SS2.p3.5.m5.3.3.1.1.1.1.2.3.cmml" xref="S3.SS2.p3.5.m5.3.3.1.1.1.1.2.3">𝑖</ci></apply><list id="S3.SS2.p3.5.m5.3.3.1.1.1.1.3.1.cmml" xref="S3.SS2.p3.5.m5.3.3.1.1.1.1.3.2"><ci id="S3.SS2.p3.5.m5.1.1.cmml" xref="S3.SS2.p3.5.m5.1.1">𝜃</ci><emptyset id="S3.SS2.p3.5.m5.2.2.cmml" xref="S3.SS2.p3.5.m5.2.2"></emptyset></list></apply></apply><apply id="S3.SS2.p3.5.m5.3.3.3.cmml" xref="S3.SS2.p3.5.m5.3.3.3"><minus id="S3.SS2.p3.5.m5.3.3.3.1.cmml" xref="S3.SS2.p3.5.m5.3.3.3"></minus><apply id="S3.SS2.p3.5.m5.3.3.3.2.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2"><ci id="S3.SS2.p3.5.m5.3.3.3.2.1.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.1">⋅</ci><apply id="S3.SS2.p3.5.m5.3.3.3.2.2.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.2"><times id="S3.SS2.p3.5.m5.3.3.3.2.2.1.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.2.1"></times><ci id="S3.SS2.p3.5.m5.3.3.3.2.2.2.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.2.2">𝛼</ci><apply id="S3.SS2.p3.5.m5.3.3.3.2.2.3.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.2.3"><csymbol cd="ambiguous" id="S3.SS2.p3.5.m5.3.3.3.2.2.3.1.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.2.3">subscript</csymbol><apply id="S3.SS2.p3.5.m5.3.3.3.2.2.3.2.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.2.3"><csymbol cd="ambiguous" id="S3.SS2.p3.5.m5.3.3.3.2.2.3.2.1.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.2.3">superscript</csymbol><ci id="S3.SS2.p3.5.m5.3.3.3.2.2.3.2.2.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.2.3.2.2">𝒛</ci><ci id="S3.SS2.p3.5.m5.3.3.3.2.2.3.2.3a.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.2.3.2.3"><mtext mathsize="70%" id="S3.SS2.p3.5.m5.3.3.3.2.2.3.2.3.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.2.3.2.3">im</mtext></ci></apply><ci id="S3.SS2.p3.5.m5.3.3.3.2.2.3.3.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.2.3.3">𝑖</ci></apply></apply><apply id="S3.SS2.p3.5.m5.3.3.3.2.3.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.3"><csymbol cd="ambiguous" id="S3.SS2.p3.5.m5.3.3.3.2.3.1.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.3">subscript</csymbol><apply id="S3.SS2.p3.5.m5.3.3.3.2.3.2.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.3"><csymbol cd="ambiguous" id="S3.SS2.p3.5.m5.3.3.3.2.3.2.1.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.3">superscript</csymbol><ci id="S3.SS2.p3.5.m5.3.3.3.2.3.2.2.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.3.2.2">𝒛</ci><ci id="S3.SS2.p3.5.m5.3.3.3.2.3.2.3a.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.3.2.3"><mtext mathsize="70%" id="S3.SS2.p3.5.m5.3.3.3.2.3.2.3.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.3.2.3">txt</mtext></ci></apply><ci id="S3.SS2.p3.5.m5.3.3.3.2.3.3.cmml" xref="S3.SS2.p3.5.m5.3.3.3.2.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.5.m5.3c">\ell({\bm{x}}_{i}|\theta,\varnothing)=-\alpha{\bm{z}}^{\textrm{im}}_{i}{\cdot}{\bm{z}}^{\textrm{txt}}_{i}</annotation></semantics></math> for the softmax-contrastive loss and <math id="S3.SS2.p3.6.m6.6" class="ltx_Math" alttext="\ell({\bm{x}}_{i}|\theta,\varnothing)=\log\left[1+\exp(-\alpha{\bm{z}}^{\textrm{im}}_{i}{\cdot}{\bm{z}}^{\textrm{txt}}_{i}+\beta)\right]" display="inline"><semantics id="S3.SS2.p3.6.m6.6a"><mrow id="S3.SS2.p3.6.m6.6.6" xref="S3.SS2.p3.6.m6.6.6.cmml"><mrow id="S3.SS2.p3.6.m6.5.5.1" xref="S3.SS2.p3.6.m6.5.5.1.cmml"><mi mathvariant="normal" id="S3.SS2.p3.6.m6.5.5.1.3" xref="S3.SS2.p3.6.m6.5.5.1.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p3.6.m6.5.5.1.2" xref="S3.SS2.p3.6.m6.5.5.1.2.cmml">​</mo><mrow id="S3.SS2.p3.6.m6.5.5.1.1.1" xref="S3.SS2.p3.6.m6.5.5.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p3.6.m6.5.5.1.1.1.2" xref="S3.SS2.p3.6.m6.5.5.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p3.6.m6.5.5.1.1.1.1" xref="S3.SS2.p3.6.m6.5.5.1.1.1.1.cmml"><msub id="S3.SS2.p3.6.m6.5.5.1.1.1.1.2" xref="S3.SS2.p3.6.m6.5.5.1.1.1.1.2.cmml"><mi id="S3.SS2.p3.6.m6.5.5.1.1.1.1.2.2" xref="S3.SS2.p3.6.m6.5.5.1.1.1.1.2.2.cmml">𝒙</mi><mi id="S3.SS2.p3.6.m6.5.5.1.1.1.1.2.3" xref="S3.SS2.p3.6.m6.5.5.1.1.1.1.2.3.cmml">i</mi></msub><mo fence="false" id="S3.SS2.p3.6.m6.5.5.1.1.1.1.1" xref="S3.SS2.p3.6.m6.5.5.1.1.1.1.1.cmml">|</mo><mrow id="S3.SS2.p3.6.m6.5.5.1.1.1.1.3.2" xref="S3.SS2.p3.6.m6.5.5.1.1.1.1.3.1.cmml"><mi id="S3.SS2.p3.6.m6.1.1" xref="S3.SS2.p3.6.m6.1.1.cmml">θ</mi><mo id="S3.SS2.p3.6.m6.5.5.1.1.1.1.3.2.1" xref="S3.SS2.p3.6.m6.5.5.1.1.1.1.3.1.cmml">,</mo><mi mathvariant="normal" id="S3.SS2.p3.6.m6.2.2" xref="S3.SS2.p3.6.m6.2.2.cmml">∅</mi></mrow></mrow><mo stretchy="false" id="S3.SS2.p3.6.m6.5.5.1.1.1.3" xref="S3.SS2.p3.6.m6.5.5.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS2.p3.6.m6.6.6.3" xref="S3.SS2.p3.6.m6.6.6.3.cmml">=</mo><mrow id="S3.SS2.p3.6.m6.6.6.2.1" xref="S3.SS2.p3.6.m6.6.6.2.2.cmml"><mi id="S3.SS2.p3.6.m6.4.4" xref="S3.SS2.p3.6.m6.4.4.cmml">log</mi><mo id="S3.SS2.p3.6.m6.6.6.2.1a" xref="S3.SS2.p3.6.m6.6.6.2.2.cmml">⁡</mo><mrow id="S3.SS2.p3.6.m6.6.6.2.1.1" xref="S3.SS2.p3.6.m6.6.6.2.2.cmml"><mo id="S3.SS2.p3.6.m6.6.6.2.1.1.2" xref="S3.SS2.p3.6.m6.6.6.2.2.cmml">[</mo><mrow id="S3.SS2.p3.6.m6.6.6.2.1.1.1" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.cmml"><mn id="S3.SS2.p3.6.m6.6.6.2.1.1.1.3" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.3.cmml">1</mn><mo id="S3.SS2.p3.6.m6.6.6.2.1.1.1.2" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.2.cmml">+</mo><mrow id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.2.cmml"><mi id="S3.SS2.p3.6.m6.3.3" xref="S3.SS2.p3.6.m6.3.3.cmml">exp</mi><mo id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1a" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.2.cmml">⁡</mo><mrow id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.2" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.2.cmml">(</mo><mrow id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.cmml"><mrow id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.cmml"><mo id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2a" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.cmml">−</mo><mrow id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.cmml"><mrow id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.cmml"><mi id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.2" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.2.cmml">α</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.1" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.1.cmml">​</mo><msubsup id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.cmml"><mi id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.2.2" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.2.2.cmml">𝒛</mi><mi id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.3" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.3.cmml">i</mi><mtext id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.2.3" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.2.3a.cmml">im</mtext></msubsup></mrow><mo lspace="0.222em" rspace="0.222em" id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.1" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.1.cmml">⋅</mo><msubsup id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.cmml"><mi id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.2.2" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.2.2.cmml">𝒛</mi><mi id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.3" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.3.cmml">i</mi><mtext id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.2.3" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.2.3a.cmml">txt</mtext></msubsup></mrow></mrow><mo id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.1" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.1.cmml">+</mo><mi id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.3" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.3.cmml">β</mi></mrow><mo stretchy="false" id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.3" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.2.cmml">)</mo></mrow></mrow></mrow><mo id="S3.SS2.p3.6.m6.6.6.2.1.1.3" xref="S3.SS2.p3.6.m6.6.6.2.2.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.6.m6.6b"><apply id="S3.SS2.p3.6.m6.6.6.cmml" xref="S3.SS2.p3.6.m6.6.6"><eq id="S3.SS2.p3.6.m6.6.6.3.cmml" xref="S3.SS2.p3.6.m6.6.6.3"></eq><apply id="S3.SS2.p3.6.m6.5.5.1.cmml" xref="S3.SS2.p3.6.m6.5.5.1"><times id="S3.SS2.p3.6.m6.5.5.1.2.cmml" xref="S3.SS2.p3.6.m6.5.5.1.2"></times><ci id="S3.SS2.p3.6.m6.5.5.1.3.cmml" xref="S3.SS2.p3.6.m6.5.5.1.3">ℓ</ci><apply id="S3.SS2.p3.6.m6.5.5.1.1.1.1.cmml" xref="S3.SS2.p3.6.m6.5.5.1.1.1"><csymbol cd="latexml" id="S3.SS2.p3.6.m6.5.5.1.1.1.1.1.cmml" xref="S3.SS2.p3.6.m6.5.5.1.1.1.1.1">conditional</csymbol><apply id="S3.SS2.p3.6.m6.5.5.1.1.1.1.2.cmml" xref="S3.SS2.p3.6.m6.5.5.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.p3.6.m6.5.5.1.1.1.1.2.1.cmml" xref="S3.SS2.p3.6.m6.5.5.1.1.1.1.2">subscript</csymbol><ci id="S3.SS2.p3.6.m6.5.5.1.1.1.1.2.2.cmml" xref="S3.SS2.p3.6.m6.5.5.1.1.1.1.2.2">𝒙</ci><ci id="S3.SS2.p3.6.m6.5.5.1.1.1.1.2.3.cmml" xref="S3.SS2.p3.6.m6.5.5.1.1.1.1.2.3">𝑖</ci></apply><list id="S3.SS2.p3.6.m6.5.5.1.1.1.1.3.1.cmml" xref="S3.SS2.p3.6.m6.5.5.1.1.1.1.3.2"><ci id="S3.SS2.p3.6.m6.1.1.cmml" xref="S3.SS2.p3.6.m6.1.1">𝜃</ci><emptyset id="S3.SS2.p3.6.m6.2.2.cmml" xref="S3.SS2.p3.6.m6.2.2"></emptyset></list></apply></apply><apply id="S3.SS2.p3.6.m6.6.6.2.2.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1"><log id="S3.SS2.p3.6.m6.4.4.cmml" xref="S3.SS2.p3.6.m6.4.4"></log><apply id="S3.SS2.p3.6.m6.6.6.2.1.1.1.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1"><plus id="S3.SS2.p3.6.m6.6.6.2.1.1.1.2.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.2"></plus><cn type="integer" id="S3.SS2.p3.6.m6.6.6.2.1.1.1.3.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.3">1</cn><apply id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.2.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1"><exp id="S3.SS2.p3.6.m6.3.3.cmml" xref="S3.SS2.p3.6.m6.3.3"></exp><apply id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1"><plus id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.1"></plus><apply id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2"><minus id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.1.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2"></minus><apply id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2"><ci id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.1.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.1">⋅</ci><apply id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2"><times id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.1.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.1"></times><ci id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.2.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.2">𝛼</ci><apply id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3"><csymbol cd="ambiguous" id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.1.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3">subscript</csymbol><apply id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.2.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3"><csymbol cd="ambiguous" id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.2.1.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3">superscript</csymbol><ci id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.2.2.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.2.2">𝒛</ci><ci id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.2.3a.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.2.3"><mtext mathsize="70%" id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.2.3.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.2.3">im</mtext></ci></apply><ci id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.3.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.2.3.3">𝑖</ci></apply></apply><apply id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3"><csymbol cd="ambiguous" id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.1.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3">subscript</csymbol><apply id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.2.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3"><csymbol cd="ambiguous" id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.2.1.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3">superscript</csymbol><ci id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.2.2.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.2.2">𝒛</ci><ci id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.2.3a.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.2.3"><mtext mathsize="70%" id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.2.3.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.2.3">txt</mtext></ci></apply><ci id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.3.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.2.2.3.3">𝑖</ci></apply></apply></apply><ci id="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p3.6.m6.6.6.2.1.1.1.1.1.1.1.3">𝛽</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.6.m6.6c">\ell({\bm{x}}_{i}|\theta,\varnothing)=\log\left[1+\exp(-\alpha{\bm{z}}^{\textrm{im}}_{i}{\cdot}{\bm{z}}^{\textrm{txt}}_{i}+\beta)\right]</annotation></semantics></math> for the sigmoid-contrastive loss.
We find that a relatively small number of chunks (<math id="S3.SS2.p3.7.m7.1" class="ltx_Math" alttext="N=16" display="inline"><semantics id="S3.SS2.p3.7.m7.1a"><mrow id="S3.SS2.p3.7.m7.1.1" xref="S3.SS2.p3.7.m7.1.1.cmml"><mi id="S3.SS2.p3.7.m7.1.1.2" xref="S3.SS2.p3.7.m7.1.1.2.cmml">N</mi><mo id="S3.SS2.p3.7.m7.1.1.1" xref="S3.SS2.p3.7.m7.1.1.1.cmml">=</mo><mn id="S3.SS2.p3.7.m7.1.1.3" xref="S3.SS2.p3.7.m7.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.7.m7.1b"><apply id="S3.SS2.p3.7.m7.1.1.cmml" xref="S3.SS2.p3.7.m7.1.1"><eq id="S3.SS2.p3.7.m7.1.1.1.cmml" xref="S3.SS2.p3.7.m7.1.1.1"></eq><ci id="S3.SS2.p3.7.m7.1.1.2.cmml" xref="S3.SS2.p3.7.m7.1.1.2">𝑁</ci><cn type="integer" id="S3.SS2.p3.7.m7.1.1.3.cmml" xref="S3.SS2.p3.7.m7.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.7.m7.1c">N=16</annotation></semantics></math>, sampling <math id="S3.SS2.p3.8.m8.1" class="ltx_Math" alttext="b/N" display="inline"><semantics id="S3.SS2.p3.8.m8.1a"><mrow id="S3.SS2.p3.8.m8.1.1" xref="S3.SS2.p3.8.m8.1.1.cmml"><mi id="S3.SS2.p3.8.m8.1.1.2" xref="S3.SS2.p3.8.m8.1.1.2.cmml">b</mi><mo id="S3.SS2.p3.8.m8.1.1.1" xref="S3.SS2.p3.8.m8.1.1.1.cmml">/</mo><mi id="S3.SS2.p3.8.m8.1.1.3" xref="S3.SS2.p3.8.m8.1.1.3.cmml">N</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.8.m8.1b"><apply id="S3.SS2.p3.8.m8.1.1.cmml" xref="S3.SS2.p3.8.m8.1.1"><divide id="S3.SS2.p3.8.m8.1.1.1.cmml" xref="S3.SS2.p3.8.m8.1.1.1"></divide><ci id="S3.SS2.p3.8.m8.1.1.2.cmml" xref="S3.SS2.p3.8.m8.1.1.2">𝑏</ci><ci id="S3.SS2.p3.8.m8.1.1.3.cmml" xref="S3.SS2.p3.8.m8.1.1.3">𝑁</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.8.m8.1c">b/N</annotation></semantics></math> = 2,048 examples independently at each iteration) is sufficient to recover batches with very high learnability (see Section <a href="#S4.SS1" title="4.1 Joint example selection yields learnable batches ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1</span></a>, Algorithm <a href="#algorithm1" title="Algorithm 1 ‣ 3.2 Joint example selection (JEST) for multimodal learning ‣ 3 Methods ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>).</p>
</div>
<figure id="algorithm1" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="algorithm1.2.1.1" class="ltx_text ltx_font_bold">Algorithm 1</span> </span> Joint example selection: sigmoid loss</figcaption>
<div id="algorithm1.3" class="ltx_listing ltx_lst_language_python ltx_lstlisting ltx_listing" style="background-color:#F5F5FA;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,ZGVmIGpvaW50bHlfc2FtcGxlX2JhdGNoKGxlYXJuZXJfbG9zcywgcmVmX2xvc3MsIG5fY2h1bmtzPTE2LCBmaWx0ZXJfcmF0aW89MC44LCBtZXRob2Q9ImxlYXJuYWJpbGl0eSIpOgogIHNjb3JlcyA9IGxlYXJuZXJfbG9zcyAtIHJlZl9sb3NzIGlmIG1ldGhvZCA9PSAibGVhcm5hYmlsaXR5IiBlbHNlIC0gcmVmX2xvc3MKICBuX2ltYWdlcyA9IHNjb3Jlcy5zaGFwZVswXSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAjIHNjb3Jlcy5zaGFwZSA9IFtCLCBCXQogIG5fZHJhd3MgPSBpbnQobl9pbWFnZXMgKiAoMSAtIGZpbHRlcl9yYXRpbykgLyBuX2NodW5rcykgICMgU2l6ZSBvZiBlYWNoIGNodW5rLgogIGxvZ2l0c19paSA9IG5wLmRpYWcoc2NvcmVzKSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgU2VsZi1zaW1pbGFyaXR5IHNjb3Jlcy4KICBpbmRzID0gcmFuZG9tLmNob2ljZShsb2dpdHNfaWksIG5fZHJhd3MpICAgICAgICAgICAgICAgICAjIFNhbXBsZSBmaXJzdCBjaHVuay4KCiAgZm9yIF8gaW4gcmFuZ2Uobl9jaHVua3MgLSAxKToKICAgIGlzX3NhbXBsZWQgPSBucC5leWUobl9pbWFnZXMpW2luZHNdLnN1bShheGlzPTApICMgQmluYXJ5IGluZGljYXRvciBvZiBjdXJyZW50IHNhbXBsZXMgW25faW1hZ2VzLF0uCiAgICBsb2dpdHNfaWogPSAoc2NvcmVzICogaXNfc2FtcGxlZC5yZXNoYXBlKG5faW1hZ2VzLCAxKSkuc3VtKGF4aXM9MCkgIyBOZWdhdGl2ZSB0ZXJtcyBpaiBbbl9pbWFnZXMsXS4KICAgIGxvZ2l0c19qaSA9IChzY29yZXMgKiBpc19zYW1wbGVkLnJlc2hhcGUoMSwgbl9pbWFnZXMpKS5zdW0oYXhpcz0xKSAjIE5lZ2F0aXZlIHRlcm1zIGppIFtuX2ltYWdlcyxdLgogICAgbG9naXRzID0gbG9naXRzX2lpICsgbG9naXRzX2lqICsgbG9naXRzX2ppICAgICAgIyBDb25kaXRpb25hbCBsZWFybmFiaWxpdHkgZ2l2ZW4gcGFzdCBzYW1wbGVzLgogICAgbG9naXRzID0gbG9naXRzIC0gaXNfc2FtcGxlZCAqIDFlOCAgICAgICAgICAgICAgIyBBdm9pZCBzYW1wbGluZyB3aXRoIHJlcGxhY2VtZW50LgogICAgbmV3X2luZHMgPSByYW5kb20uY2hvaWNlKG5faW1hZ2VzLCBuX2RyYXdzLCBwPW5wLmV4cChsb2dpdHMpKQogICAgaW5kcyA9IG5wLmNvbmNhdGVuYXRlKChpbmRzLCBuZXdfaW5kcykpICAgICAgICAgIyBFeHBhbmQgdGhlIGFycmF5IG9mIGluZGljZXMgc2FtcGxlZC4KICByZXR1cm4gaW5kcyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgR2F0aGVyIGFuZCByZXR1cm4gc3Vic2V0IGluZGljZXMu" download="">⬇</a></div>
<div id="lstnumberx1" class="ltx_listingline">
<span id="lstnumberx1.1" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">def</span><span id="lstnumberx1.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx1.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">jointly_sample_batch</span><span id="lstnumberx1.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx1.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">learner_loss</span><span id="lstnumberx1.6" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx1.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx1.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ref_loss</span><span id="lstnumberx1.9" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx1.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx1.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_chunks</span><span id="lstnumberx1.12" class="ltx_text ltx_font_typewriter">=16,</span><span id="lstnumberx1.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx1.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">filter_ratio</span><span id="lstnumberx1.15" class="ltx_text ltx_font_typewriter">=0.8,</span><span id="lstnumberx1.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx1.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">method</span><span id="lstnumberx1.18" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx1.19" class="ltx_text ltx_lst_string ltx_font_typewriter">"learnability"</span><span id="lstnumberx1.20" class="ltx_text ltx_font_typewriter">):</span>
</div>
<div id="lstnumberx2" class="ltx_listingline">
<span id="lstnumberx2.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx2.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">scores</span><span id="lstnumberx2.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx2.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">learner_loss</span><span id="lstnumberx2.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.8" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx2.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ref_loss</span><span id="lstnumberx2.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.12" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">if</span><span id="lstnumberx2.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">method</span><span id="lstnumberx2.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.16" class="ltx_text ltx_font_typewriter">==</span><span id="lstnumberx2.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.18" class="ltx_text ltx_lst_string ltx_font_typewriter">"learnability"</span><span id="lstnumberx2.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.20" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">else</span><span id="lstnumberx2.21" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.22" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx2.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx2.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ref_loss</span>
</div>
<div id="lstnumberx3" class="ltx_listingline">
<span id="lstnumberx3.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx3.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_images</span><span id="lstnumberx3.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx3.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx3.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">scores</span><span id="lstnumberx3.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx3.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">shape</span><span id="lstnumberx3.9" class="ltx_text ltx_font_typewriter">[0]</span><span id="lstnumberx3.10" class="ltx_text ltx_lst_space ltx_font_typewriter">                               </span><span id="lstnumberx3.11" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx3.11.1" class="ltx_text ltx_lst_space"> </span>scores.shape<span id="lstnumberx3.11.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx3.11.3" class="ltx_text ltx_lst_space"> </span>[B,<span id="lstnumberx3.11.4" class="ltx_text ltx_lst_space"> </span>B]</span>
</div>
<div id="lstnumberx4" class="ltx_listingline">
<span id="lstnumberx4.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx4.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_draws</span><span id="lstnumberx4.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx4.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.6" class="ltx_text ltx_lst_keyword ltx_lst_keywords2 ltx_font_typewriter" style="color:#408080;">int</span><span id="lstnumberx4.7" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx4.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_images</span><span id="lstnumberx4.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.10" class="ltx_text ltx_font_typewriter">*</span><span id="lstnumberx4.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.12" class="ltx_text ltx_font_typewriter">(1</span><span id="lstnumberx4.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.14" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx4.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">filter_ratio</span><span id="lstnumberx4.17" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx4.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.19" class="ltx_text ltx_font_typewriter">/</span><span id="lstnumberx4.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx4.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_chunks</span><span id="lstnumberx4.22" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx4.23" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx4.24" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx4.24.1" class="ltx_text ltx_lst_space"> </span>Size<span id="lstnumberx4.24.2" class="ltx_text ltx_lst_space"> </span>of<span id="lstnumberx4.24.3" class="ltx_text ltx_lst_space"> </span>each<span id="lstnumberx4.24.4" class="ltx_text ltx_lst_space"> </span>chunk.</span>
</div>
<div id="lstnumberx5" class="ltx_listingline">
<span id="lstnumberx5.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx5.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_ii</span><span id="lstnumberx5.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx5.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx5.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx5.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">np</span><span id="lstnumberx5.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx5.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">diag</span><span id="lstnumberx5.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx5.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">scores</span><span id="lstnumberx5.11" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx5.12" class="ltx_text ltx_lst_space ltx_font_typewriter">                              </span><span id="lstnumberx5.13" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx5.13.1" class="ltx_text ltx_lst_space"> </span>Self-similarity<span id="lstnumberx5.13.2" class="ltx_text ltx_lst_space"> </span>scores.</span>
</div>
<div id="lstnumberx6" class="ltx_listingline">
<span id="lstnumberx6.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx6.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">inds</span><span id="lstnumberx6.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx6.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx6.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx6.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">random</span><span id="lstnumberx6.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx6.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">choice</span><span id="lstnumberx6.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx6.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_ii</span><span id="lstnumberx6.11" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx6.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx6.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_draws</span><span id="lstnumberx6.14" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx6.15" class="ltx_text ltx_lst_space ltx_font_typewriter">                 </span><span id="lstnumberx6.16" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx6.16.1" class="ltx_text ltx_lst_space"> </span>Sample<span id="lstnumberx6.16.2" class="ltx_text ltx_lst_space"> </span>first<span id="lstnumberx6.16.3" class="ltx_text ltx_lst_space"> </span>chunk.</span>
</div>
<div id="lstnumberx7" class="ltx_listingline">
</div>
<div id="lstnumberx8" class="ltx_listingline">
<span id="lstnumberx8.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx8.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">for</span><span id="lstnumberx8.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">_</span><span id="lstnumberx8.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.6" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">in</span><span id="lstnumberx8.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.8" class="ltx_text ltx_lst_keyword ltx_lst_keywords2 ltx_font_typewriter" style="color:#408080;">range</span><span id="lstnumberx8.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx8.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_chunks</span><span id="lstnumberx8.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.12" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx8.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx8.14" class="ltx_text ltx_font_typewriter">1):</span>
</div>
<div id="lstnumberx9" class="ltx_listingline">
<span id="lstnumberx9.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx9.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx9.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx9.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx9.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx9.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">np</span><span id="lstnumberx9.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx9.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">eye</span><span id="lstnumberx9.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx9.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_images</span><span id="lstnumberx9.11" class="ltx_text ltx_font_typewriter">)[</span><span id="lstnumberx9.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">inds</span><span id="lstnumberx9.13" class="ltx_text ltx_font_typewriter">].</span><span id="lstnumberx9.14" class="ltx_text ltx_lst_keyword ltx_lst_keywords2 ltx_font_typewriter" style="color:#408080;">sum</span><span id="lstnumberx9.15" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx9.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">axis</span><span id="lstnumberx9.17" class="ltx_text ltx_font_typewriter">=0)</span><span id="lstnumberx9.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx9.19" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx9.19.1" class="ltx_text ltx_lst_space"> </span>Binary<span id="lstnumberx9.19.2" class="ltx_text ltx_lst_space"> </span>indicator<span id="lstnumberx9.19.3" class="ltx_text ltx_lst_space"> </span>of<span id="lstnumberx9.19.4" class="ltx_text ltx_lst_space"> </span>current<span id="lstnumberx9.19.5" class="ltx_text ltx_lst_space"> </span>samples<span id="lstnumberx9.19.6" class="ltx_text ltx_lst_space"> </span>[n_images,].</span>
</div>
<div id="lstnumberx10" class="ltx_listingline">
<span id="lstnumberx10.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx10.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_ij</span><span id="lstnumberx10.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx10.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx10.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx10.6" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx10.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">scores</span><span id="lstnumberx10.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx10.9" class="ltx_text ltx_font_typewriter">*</span><span id="lstnumberx10.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx10.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx10.12" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx10.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">reshape</span><span id="lstnumberx10.14" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx10.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_images</span><span id="lstnumberx10.16" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx10.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx10.18" class="ltx_text ltx_font_typewriter">1)).</span><span id="lstnumberx10.19" class="ltx_text ltx_lst_keyword ltx_lst_keywords2 ltx_font_typewriter" style="color:#408080;">sum</span><span id="lstnumberx10.20" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx10.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">axis</span><span id="lstnumberx10.22" class="ltx_text ltx_font_typewriter">=0)</span><span id="lstnumberx10.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx10.24" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx10.24.1" class="ltx_text ltx_lst_space"> </span>Negative<span id="lstnumberx10.24.2" class="ltx_text ltx_lst_space"> </span>terms<span id="lstnumberx10.24.3" class="ltx_text ltx_lst_space"> </span>ij<span id="lstnumberx10.24.4" class="ltx_text ltx_lst_space"> </span>[n_images,].</span>
</div>
<div id="lstnumberx11" class="ltx_listingline">
<span id="lstnumberx11.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx11.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_ji</span><span id="lstnumberx11.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx11.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.6" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx11.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">scores</span><span id="lstnumberx11.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.9" class="ltx_text ltx_font_typewriter">*</span><span id="lstnumberx11.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx11.12" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx11.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">reshape</span><span id="lstnumberx11.14" class="ltx_text ltx_font_typewriter">(1,</span><span id="lstnumberx11.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_images</span><span id="lstnumberx11.17" class="ltx_text ltx_font_typewriter">)).</span><span id="lstnumberx11.18" class="ltx_text ltx_lst_keyword ltx_lst_keywords2 ltx_font_typewriter" style="color:#408080;">sum</span><span id="lstnumberx11.19" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx11.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">axis</span><span id="lstnumberx11.21" class="ltx_text ltx_font_typewriter">=1)</span><span id="lstnumberx11.22" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx11.23" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx11.23.1" class="ltx_text ltx_lst_space"> </span>Negative<span id="lstnumberx11.23.2" class="ltx_text ltx_lst_space"> </span>terms<span id="lstnumberx11.23.3" class="ltx_text ltx_lst_space"> </span>ji<span id="lstnumberx11.23.4" class="ltx_text ltx_lst_space"> </span>[n_images,].</span>
</div>
<div id="lstnumberx12" class="ltx_listingline">
<span id="lstnumberx12.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx12.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits</span><span id="lstnumberx12.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx12.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx12.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx12.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_ii</span><span id="lstnumberx12.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx12.8" class="ltx_text ltx_font_typewriter">+</span><span id="lstnumberx12.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx12.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_ij</span><span id="lstnumberx12.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx12.12" class="ltx_text ltx_font_typewriter">+</span><span id="lstnumberx12.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx12.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_ji</span><span id="lstnumberx12.15" class="ltx_text ltx_lst_space ltx_font_typewriter">      </span><span id="lstnumberx12.16" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx12.16.1" class="ltx_text ltx_lst_space"> </span>Conditional<span id="lstnumberx12.16.2" class="ltx_text ltx_lst_space"> </span>learnability<span id="lstnumberx12.16.3" class="ltx_text ltx_lst_space"> </span>given<span id="lstnumberx12.16.4" class="ltx_text ltx_lst_space"> </span>past<span id="lstnumberx12.16.5" class="ltx_text ltx_lst_space"> </span>samples.</span>
</div>
<div id="lstnumberx13" class="ltx_listingline">
<span id="lstnumberx13.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx13.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits</span><span id="lstnumberx13.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx13.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx13.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx13.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits</span><span id="lstnumberx13.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx13.8" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx13.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx13.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx13.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx13.12" class="ltx_text ltx_font_typewriter">*</span><span id="lstnumberx13.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx13.14" class="ltx_text ltx_font_typewriter">1</span><span id="lstnumberx13.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">e8</span><span id="lstnumberx13.16" class="ltx_text ltx_lst_space ltx_font_typewriter">              </span><span id="lstnumberx13.17" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx13.17.1" class="ltx_text ltx_lst_space"> </span>Avoid<span id="lstnumberx13.17.2" class="ltx_text ltx_lst_space"> </span>sampling<span id="lstnumberx13.17.3" class="ltx_text ltx_lst_space"> </span>with<span id="lstnumberx13.17.4" class="ltx_text ltx_lst_space"> </span>replacement.</span>
</div>
<div id="lstnumberx14" class="ltx_listingline">
<span id="lstnumberx14.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx14.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">new_inds</span><span id="lstnumberx14.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx14.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">random</span><span id="lstnumberx14.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx14.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">choice</span><span id="lstnumberx14.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx14.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_images</span><span id="lstnumberx14.11" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx14.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_draws</span><span id="lstnumberx14.14" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx14.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx14.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">p</span><span id="lstnumberx14.17" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx14.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">np</span><span id="lstnumberx14.19" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx14.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">exp</span><span id="lstnumberx14.21" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx14.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits</span><span id="lstnumberx14.23" class="ltx_text ltx_font_typewriter">))</span>
</div>
<div id="lstnumberx15" class="ltx_listingline">
<span id="lstnumberx15.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx15.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">inds</span><span id="lstnumberx15.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx15.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx15.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx15.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">np</span><span id="lstnumberx15.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx15.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">concatenate</span><span id="lstnumberx15.9" class="ltx_text ltx_font_typewriter">((</span><span id="lstnumberx15.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">inds</span><span id="lstnumberx15.11" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx15.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx15.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">new_inds</span><span id="lstnumberx15.14" class="ltx_text ltx_font_typewriter">))</span><span id="lstnumberx15.15" class="ltx_text ltx_lst_space ltx_font_typewriter">         </span><span id="lstnumberx15.16" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx15.16.1" class="ltx_text ltx_lst_space"> </span>Expand<span id="lstnumberx15.16.2" class="ltx_text ltx_lst_space"> </span>the<span id="lstnumberx15.16.3" class="ltx_text ltx_lst_space"> </span>array<span id="lstnumberx15.16.4" class="ltx_text ltx_lst_space"> </span>of<span id="lstnumberx15.16.5" class="ltx_text ltx_lst_space"> </span>indices<span id="lstnumberx15.16.6" class="ltx_text ltx_lst_space"> </span>sampled.</span>
</div>
<div id="lstnumberx16" class="ltx_listingline">
<span id="lstnumberx16.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx16.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">return</span><span id="lstnumberx16.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx16.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">inds</span><span id="lstnumberx16.5" class="ltx_text ltx_lst_space ltx_font_typewriter">                                       </span><span id="lstnumberx16.6" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx16.6.1" class="ltx_text ltx_lst_space"> </span>Gather<span id="lstnumberx16.6.2" class="ltx_text ltx_lst_space"> </span>and<span id="lstnumberx16.6.3" class="ltx_text ltx_lst_space"> </span>return<span id="lstnumberx16.6.4" class="ltx_text ltx_lst_space"> </span>subset<span id="lstnumberx16.6.5" class="ltx_text ltx_lst_space"> </span>indices.</span>
</div>
</div>
</figure>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Efficient scoring and multi-resolution training</h3>

<div id="S3.SS3.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.p1.2" class="ltx_p"><span id="S3.SS3.p1.2.1" class="ltx_text ltx_font_bold">Efficient scoring with online model approximation.</span>
Scoring large super-batches increases the cost per iteration, lowering the efficiency gains in terms of total FLOPs.
While <cite class="ltx_cite ltx_citemacro_citet">Evans et al. [<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite> required additional small, proxy models to efficiently score data on behalf of larger learners, we remove this requirement by using online model approximation.
We only approximate the image encoding since this accounts for the bulk of the cost of each inference pass <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite>.
For this we adopt the FlexiViT architecture <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib4" title="" class="ltx_ref">4</a>]</cite>, which lowers the image resolution while minimally degrading performance (see Figure <a href="#A1.F9" title="Figure 9 ‣ A.7 Comparing approximation methods ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a> for a comparison to patch dropping <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite>).
In our experiments, we evaluate the super-batch with 32<math id="S3.SS3.p1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S3.SS3.p1.1.m1.1a"><mo id="S3.SS3.p1.1.m1.1.1" xref="S3.SS3.p1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.1.m1.1b"><times id="S3.SS3.p1.1.m1.1.1.cmml" xref="S3.SS3.p1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.1.m1.1c">\times</annotation></semantics></math>32-pixel patches,
which gives a 72% reduction in FLOPs and 67% reduction in wall-clock time vs. full-resolution scoring at patch size 16<math id="S3.SS3.p1.2.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S3.SS3.p1.2.m2.1a"><mo id="S3.SS3.p1.2.m2.1.1" xref="S3.SS3.p1.2.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.2.m2.1b"><times id="S3.SS3.p1.2.m2.1.1.cmml" xref="S3.SS3.p1.2.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.2.m2.1c">\times</annotation></semantics></math>16 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite> (see Section <a href="#A1.SS4" title="A.4 FLOP calculations for JEST / Flexi-JEST ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.4</span></a>).</p>
</div>
<div id="S3.SS3.p2" class="ltx_para ltx_noindent">
<p id="S3.SS3.p2.11" class="ltx_p"><span id="S3.SS3.p2.11.1" class="ltx_text ltx_font_bold">Multi-resolution training.</span> While we want to score examples at low resolution (i.e. with large patches), at test-time we wish to evaluate the model at full resolution (i.e. with small patches). To enable both resolutions of computation, we simply train at both resolutions. Specifically, given a sub-batch <math id="S3.SS3.p2.1.m1.1" class="ltx_Math" alttext="\mathcal{B}" display="inline"><semantics id="S3.SS3.p2.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.1.m1.1.1" xref="S3.SS3.p2.1.m1.1.1.cmml">ℬ</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.1.m1.1b"><ci id="S3.SS3.p2.1.m1.1.1.cmml" xref="S3.SS3.p2.1.m1.1.1">ℬ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.1.m1.1c">\mathcal{B}</annotation></semantics></math> for learning, we randomly split it into two halves, <math id="S3.SS3.p2.2.m2.1" class="ltx_Math" alttext="\mathcal{B}^{lo}" display="inline"><semantics id="S3.SS3.p2.2.m2.1a"><msup id="S3.SS3.p2.2.m2.1.1" xref="S3.SS3.p2.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.2.m2.1.1.2" xref="S3.SS3.p2.2.m2.1.1.2.cmml">ℬ</mi><mrow id="S3.SS3.p2.2.m2.1.1.3" xref="S3.SS3.p2.2.m2.1.1.3.cmml"><mi id="S3.SS3.p2.2.m2.1.1.3.2" xref="S3.SS3.p2.2.m2.1.1.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.2.m2.1.1.3.1" xref="S3.SS3.p2.2.m2.1.1.3.1.cmml">​</mo><mi id="S3.SS3.p2.2.m2.1.1.3.3" xref="S3.SS3.p2.2.m2.1.1.3.3.cmml">o</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.2.m2.1b"><apply id="S3.SS3.p2.2.m2.1.1.cmml" xref="S3.SS3.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.2.m2.1.1.1.cmml" xref="S3.SS3.p2.2.m2.1.1">superscript</csymbol><ci id="S3.SS3.p2.2.m2.1.1.2.cmml" xref="S3.SS3.p2.2.m2.1.1.2">ℬ</ci><apply id="S3.SS3.p2.2.m2.1.1.3.cmml" xref="S3.SS3.p2.2.m2.1.1.3"><times id="S3.SS3.p2.2.m2.1.1.3.1.cmml" xref="S3.SS3.p2.2.m2.1.1.3.1"></times><ci id="S3.SS3.p2.2.m2.1.1.3.2.cmml" xref="S3.SS3.p2.2.m2.1.1.3.2">𝑙</ci><ci id="S3.SS3.p2.2.m2.1.1.3.3.cmml" xref="S3.SS3.p2.2.m2.1.1.3.3">𝑜</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.2.m2.1c">\mathcal{B}^{lo}</annotation></semantics></math> and <math id="S3.SS3.p2.3.m3.1" class="ltx_Math" alttext="\mathcal{B}^{hi}" display="inline"><semantics id="S3.SS3.p2.3.m3.1a"><msup id="S3.SS3.p2.3.m3.1.1" xref="S3.SS3.p2.3.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.3.m3.1.1.2" xref="S3.SS3.p2.3.m3.1.1.2.cmml">ℬ</mi><mrow id="S3.SS3.p2.3.m3.1.1.3" xref="S3.SS3.p2.3.m3.1.1.3.cmml"><mi id="S3.SS3.p2.3.m3.1.1.3.2" xref="S3.SS3.p2.3.m3.1.1.3.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.3.m3.1.1.3.1" xref="S3.SS3.p2.3.m3.1.1.3.1.cmml">​</mo><mi id="S3.SS3.p2.3.m3.1.1.3.3" xref="S3.SS3.p2.3.m3.1.1.3.3.cmml">i</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.3.m3.1b"><apply id="S3.SS3.p2.3.m3.1.1.cmml" xref="S3.SS3.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.3.m3.1.1.1.cmml" xref="S3.SS3.p2.3.m3.1.1">superscript</csymbol><ci id="S3.SS3.p2.3.m3.1.1.2.cmml" xref="S3.SS3.p2.3.m3.1.1.2">ℬ</ci><apply id="S3.SS3.p2.3.m3.1.1.3.cmml" xref="S3.SS3.p2.3.m3.1.1.3"><times id="S3.SS3.p2.3.m3.1.1.3.1.cmml" xref="S3.SS3.p2.3.m3.1.1.3.1"></times><ci id="S3.SS3.p2.3.m3.1.1.3.2.cmml" xref="S3.SS3.p2.3.m3.1.1.3.2">ℎ</ci><ci id="S3.SS3.p2.3.m3.1.1.3.3.cmml" xref="S3.SS3.p2.3.m3.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.3.m3.1c">\mathcal{B}^{hi}</annotation></semantics></math>, and encode each half with a different resolution: <math id="S3.SS3.p2.4.m4.10" class="ltx_Math" alttext="\mathcal{Z}^{lo}=\{f^{\textrm{im}}({\bm{x}};\theta,p=32),{\bm{x}}\in\mathcal{B}^{lo}\},\mathcal{Z}^{hi}=\{f^{\textrm{im}}({\bm{x}};\theta,p=16),{\bm{x}}\in\mathcal{B}^{hi}\}" display="inline"><semantics id="S3.SS3.p2.4.m4.10a"><mrow id="S3.SS3.p2.4.m4.10.10.2" xref="S3.SS3.p2.4.m4.10.10.3.cmml"><mrow id="S3.SS3.p2.4.m4.9.9.1.1" xref="S3.SS3.p2.4.m4.9.9.1.1.cmml"><msup id="S3.SS3.p2.4.m4.9.9.1.1.3" xref="S3.SS3.p2.4.m4.9.9.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.4.m4.9.9.1.1.3.2" xref="S3.SS3.p2.4.m4.9.9.1.1.3.2.cmml">𝒵</mi><mrow id="S3.SS3.p2.4.m4.9.9.1.1.3.3" xref="S3.SS3.p2.4.m4.9.9.1.1.3.3.cmml"><mi id="S3.SS3.p2.4.m4.9.9.1.1.3.3.2" xref="S3.SS3.p2.4.m4.9.9.1.1.3.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.4.m4.9.9.1.1.3.3.1" xref="S3.SS3.p2.4.m4.9.9.1.1.3.3.1.cmml">​</mo><mi id="S3.SS3.p2.4.m4.9.9.1.1.3.3.3" xref="S3.SS3.p2.4.m4.9.9.1.1.3.3.3.cmml">o</mi></mrow></msup><mo id="S3.SS3.p2.4.m4.9.9.1.1.2" xref="S3.SS3.p2.4.m4.9.9.1.1.2.cmml">=</mo><mrow id="S3.SS3.p2.4.m4.9.9.1.1.1.1" xref="S3.SS3.p2.4.m4.9.9.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS3.p2.4.m4.9.9.1.1.1.1.2" xref="S3.SS3.p2.4.m4.9.9.1.1.1.2.cmml">{</mo><mrow id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.cmml"><mrow id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.2.cmml"><mrow id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.cmml"><msup id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.3" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.3.2" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.3.2.cmml">f</mi><mtext id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.3.3" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.3.3a.cmml">im</mtext></msup><mo lspace="0em" rspace="0em" id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.2" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.2" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.cmml"><mrow id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml"><mi id="S3.SS3.p2.4.m4.1.1" xref="S3.SS3.p2.4.m4.1.1.cmml">𝒙</mi><mo id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.2.2.1" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml">;</mo><mi id="S3.SS3.p2.4.m4.2.2" xref="S3.SS3.p2.4.m4.2.2.cmml">θ</mi><mo id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.2.2.2" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml">,</mo><mi id="S3.SS3.p2.4.m4.3.3" xref="S3.SS3.p2.4.m4.3.3.cmml">p</mi></mrow><mo id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.1.cmml">=</mo><mn id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.3.cmml">32</mn></mrow><mo stretchy="false" id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.3" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.2" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.2.cmml">,</mo><mi id="S3.SS3.p2.4.m4.4.4" xref="S3.SS3.p2.4.m4.4.4.cmml">𝒙</mi></mrow><mo id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.2" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.2.cmml">∈</mo><msup id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.2" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.2.cmml">ℬ</mi><mrow id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.3" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.3.cmml"><mi id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.3.2" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.3.1" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.3.3" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.3.3.cmml">o</mi></mrow></msup></mrow><mo stretchy="false" id="S3.SS3.p2.4.m4.9.9.1.1.1.1.3" xref="S3.SS3.p2.4.m4.9.9.1.1.1.2.cmml">}</mo></mrow></mrow><mo id="S3.SS3.p2.4.m4.10.10.2.3" xref="S3.SS3.p2.4.m4.10.10.3a.cmml">,</mo><mrow id="S3.SS3.p2.4.m4.10.10.2.2" xref="S3.SS3.p2.4.m4.10.10.2.2.cmml"><msup id="S3.SS3.p2.4.m4.10.10.2.2.3" xref="S3.SS3.p2.4.m4.10.10.2.2.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.4.m4.10.10.2.2.3.2" xref="S3.SS3.p2.4.m4.10.10.2.2.3.2.cmml">𝒵</mi><mrow id="S3.SS3.p2.4.m4.10.10.2.2.3.3" xref="S3.SS3.p2.4.m4.10.10.2.2.3.3.cmml"><mi id="S3.SS3.p2.4.m4.10.10.2.2.3.3.2" xref="S3.SS3.p2.4.m4.10.10.2.2.3.3.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.4.m4.10.10.2.2.3.3.1" xref="S3.SS3.p2.4.m4.10.10.2.2.3.3.1.cmml">​</mo><mi id="S3.SS3.p2.4.m4.10.10.2.2.3.3.3" xref="S3.SS3.p2.4.m4.10.10.2.2.3.3.3.cmml">i</mi></mrow></msup><mo id="S3.SS3.p2.4.m4.10.10.2.2.2" xref="S3.SS3.p2.4.m4.10.10.2.2.2.cmml">=</mo><mrow id="S3.SS3.p2.4.m4.10.10.2.2.1.1" xref="S3.SS3.p2.4.m4.10.10.2.2.1.2.cmml"><mo stretchy="false" id="S3.SS3.p2.4.m4.10.10.2.2.1.1.2" xref="S3.SS3.p2.4.m4.10.10.2.2.1.2.cmml">{</mo><mrow id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.cmml"><mrow id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.2.cmml"><mrow id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.cmml"><msup id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.3" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.3.cmml"><mi id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.3.2" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.3.2.cmml">f</mi><mtext id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.3.3" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.3.3a.cmml">im</mtext></msup><mo lspace="0em" rspace="0em" id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.2" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.2" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.cmml"><mrow id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.2.2" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.2.1.cmml"><mi id="S3.SS3.p2.4.m4.5.5" xref="S3.SS3.p2.4.m4.5.5.cmml">𝒙</mi><mo id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.2.2.1" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.2.1.cmml">;</mo><mi id="S3.SS3.p2.4.m4.6.6" xref="S3.SS3.p2.4.m4.6.6.cmml">θ</mi><mo id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.2.2.2" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.2.1.cmml">,</mo><mi id="S3.SS3.p2.4.m4.7.7" xref="S3.SS3.p2.4.m4.7.7.cmml">p</mi></mrow><mo id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.1" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.1.cmml">=</mo><mn id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.3" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.3.cmml">16</mn></mrow><mo stretchy="false" id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.3" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.2" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.2.cmml">,</mo><mi id="S3.SS3.p2.4.m4.8.8" xref="S3.SS3.p2.4.m4.8.8.cmml">𝒙</mi></mrow><mo id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.2" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.2.cmml">∈</mo><msup id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.2" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.2.cmml">ℬ</mi><mrow id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.3" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.3.cmml"><mi id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.3.2" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.3.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.3.1" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.3.1.cmml">​</mo><mi id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.3.3" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.3.3.cmml">i</mi></mrow></msup></mrow><mo stretchy="false" id="S3.SS3.p2.4.m4.10.10.2.2.1.1.3" xref="S3.SS3.p2.4.m4.10.10.2.2.1.2.cmml">}</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.4.m4.10b"><apply id="S3.SS3.p2.4.m4.10.10.3.cmml" xref="S3.SS3.p2.4.m4.10.10.2"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.10.10.3a.cmml" xref="S3.SS3.p2.4.m4.10.10.2.3">formulae-sequence</csymbol><apply id="S3.SS3.p2.4.m4.9.9.1.1.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1"><eq id="S3.SS3.p2.4.m4.9.9.1.1.2.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.2"></eq><apply id="S3.SS3.p2.4.m4.9.9.1.1.3.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.9.9.1.1.3.1.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.3">superscript</csymbol><ci id="S3.SS3.p2.4.m4.9.9.1.1.3.2.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.3.2">𝒵</ci><apply id="S3.SS3.p2.4.m4.9.9.1.1.3.3.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.3.3"><times id="S3.SS3.p2.4.m4.9.9.1.1.3.3.1.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.3.3.1"></times><ci id="S3.SS3.p2.4.m4.9.9.1.1.3.3.2.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.3.3.2">𝑙</ci><ci id="S3.SS3.p2.4.m4.9.9.1.1.3.3.3.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.3.3.3">𝑜</ci></apply></apply><set id="S3.SS3.p2.4.m4.9.9.1.1.1.2.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1"><apply id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1"><in id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.2.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.2"></in><list id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.2.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1"><apply id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1"><times id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.2"></times><apply id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.3">superscript</csymbol><ci id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.3.2">𝑓</ci><ci id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.3.3a.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.3.3"><mtext mathsize="70%" id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.3.3">im</mtext></ci></apply><apply id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1"><eq id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.1"></eq><list id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.2.2"><ci id="S3.SS3.p2.4.m4.1.1.cmml" xref="S3.SS3.p2.4.m4.1.1">𝒙</ci><ci id="S3.SS3.p2.4.m4.2.2.cmml" xref="S3.SS3.p2.4.m4.2.2">𝜃</ci><ci id="S3.SS3.p2.4.m4.3.3.cmml" xref="S3.SS3.p2.4.m4.3.3">𝑝</ci></list><cn type="integer" id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.1.1.1.1.1.1.3">32</cn></apply></apply><ci id="S3.SS3.p2.4.m4.4.4.cmml" xref="S3.SS3.p2.4.m4.4.4">𝒙</ci></list><apply id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.1.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3">superscript</csymbol><ci id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.2.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.2">ℬ</ci><apply id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.3.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.3"><times id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.3.1.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.3.1"></times><ci id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.3.2.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.3.2">𝑙</ci><ci id="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.3.3.cmml" xref="S3.SS3.p2.4.m4.9.9.1.1.1.1.1.3.3.3">𝑜</ci></apply></apply></apply></set></apply><apply id="S3.SS3.p2.4.m4.10.10.2.2.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2"><eq id="S3.SS3.p2.4.m4.10.10.2.2.2.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.2"></eq><apply id="S3.SS3.p2.4.m4.10.10.2.2.3.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.3"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.10.10.2.2.3.1.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.3">superscript</csymbol><ci id="S3.SS3.p2.4.m4.10.10.2.2.3.2.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.3.2">𝒵</ci><apply id="S3.SS3.p2.4.m4.10.10.2.2.3.3.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.3.3"><times id="S3.SS3.p2.4.m4.10.10.2.2.3.3.1.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.3.3.1"></times><ci id="S3.SS3.p2.4.m4.10.10.2.2.3.3.2.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.3.3.2">ℎ</ci><ci id="S3.SS3.p2.4.m4.10.10.2.2.3.3.3.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.3.3.3">𝑖</ci></apply></apply><set id="S3.SS3.p2.4.m4.10.10.2.2.1.2.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1"><apply id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1"><in id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.2.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.2"></in><list id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.2.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1"><apply id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1"><times id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.2.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.2"></times><apply id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.3.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.3.1.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.3">superscript</csymbol><ci id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.3.2.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.3.2">𝑓</ci><ci id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.3.3a.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.3.3"><mtext mathsize="70%" id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.3.3.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.3.3">im</mtext></ci></apply><apply id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1"><eq id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.1"></eq><list id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.2.2"><ci id="S3.SS3.p2.4.m4.5.5.cmml" xref="S3.SS3.p2.4.m4.5.5">𝒙</ci><ci id="S3.SS3.p2.4.m4.6.6.cmml" xref="S3.SS3.p2.4.m4.6.6">𝜃</ci><ci id="S3.SS3.p2.4.m4.7.7.cmml" xref="S3.SS3.p2.4.m4.7.7">𝑝</ci></list><cn type="integer" id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.1.1.1.1.1.1.3">16</cn></apply></apply><ci id="S3.SS3.p2.4.m4.8.8.cmml" xref="S3.SS3.p2.4.m4.8.8">𝒙</ci></list><apply id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.1.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3">superscript</csymbol><ci id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.2.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.2">ℬ</ci><apply id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.3.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.3"><times id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.3.1.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.3.1"></times><ci id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.3.2.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.3.2">ℎ</ci><ci id="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.3.3.cmml" xref="S3.SS3.p2.4.m4.10.10.2.2.1.1.1.3.3.3">𝑖</ci></apply></apply></apply></set></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.4.m4.10c">\mathcal{Z}^{lo}=\{f^{\textrm{im}}({\bm{x}};\theta,p=32),{\bm{x}}\in\mathcal{B}^{lo}\},\mathcal{Z}^{hi}=\{f^{\textrm{im}}({\bm{x}};\theta,p=16),{\bm{x}}\in\mathcal{B}^{hi}\}</annotation></semantics></math>.
These images embeddings are then concatenated together as <math id="S3.SS3.p2.5.m5.1" class="ltx_Math" alttext="\mathcal{Z}=\mathcal{Z}^{lo}\cup\mathcal{Z}^{hi}" display="inline"><semantics id="S3.SS3.p2.5.m5.1a"><mrow id="S3.SS3.p2.5.m5.1.1" xref="S3.SS3.p2.5.m5.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.5.m5.1.1.2" xref="S3.SS3.p2.5.m5.1.1.2.cmml">𝒵</mi><mo id="S3.SS3.p2.5.m5.1.1.1" xref="S3.SS3.p2.5.m5.1.1.1.cmml">=</mo><mrow id="S3.SS3.p2.5.m5.1.1.3" xref="S3.SS3.p2.5.m5.1.1.3.cmml"><msup id="S3.SS3.p2.5.m5.1.1.3.2" xref="S3.SS3.p2.5.m5.1.1.3.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.5.m5.1.1.3.2.2" xref="S3.SS3.p2.5.m5.1.1.3.2.2.cmml">𝒵</mi><mrow id="S3.SS3.p2.5.m5.1.1.3.2.3" xref="S3.SS3.p2.5.m5.1.1.3.2.3.cmml"><mi id="S3.SS3.p2.5.m5.1.1.3.2.3.2" xref="S3.SS3.p2.5.m5.1.1.3.2.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.5.m5.1.1.3.2.3.1" xref="S3.SS3.p2.5.m5.1.1.3.2.3.1.cmml">​</mo><mi id="S3.SS3.p2.5.m5.1.1.3.2.3.3" xref="S3.SS3.p2.5.m5.1.1.3.2.3.3.cmml">o</mi></mrow></msup><mo id="S3.SS3.p2.5.m5.1.1.3.1" xref="S3.SS3.p2.5.m5.1.1.3.1.cmml">∪</mo><msup id="S3.SS3.p2.5.m5.1.1.3.3" xref="S3.SS3.p2.5.m5.1.1.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.5.m5.1.1.3.3.2" xref="S3.SS3.p2.5.m5.1.1.3.3.2.cmml">𝒵</mi><mrow id="S3.SS3.p2.5.m5.1.1.3.3.3" xref="S3.SS3.p2.5.m5.1.1.3.3.3.cmml"><mi id="S3.SS3.p2.5.m5.1.1.3.3.3.2" xref="S3.SS3.p2.5.m5.1.1.3.3.3.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.5.m5.1.1.3.3.3.1" xref="S3.SS3.p2.5.m5.1.1.3.3.3.1.cmml">​</mo><mi id="S3.SS3.p2.5.m5.1.1.3.3.3.3" xref="S3.SS3.p2.5.m5.1.1.3.3.3.3.cmml">i</mi></mrow></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.5.m5.1b"><apply id="S3.SS3.p2.5.m5.1.1.cmml" xref="S3.SS3.p2.5.m5.1.1"><eq id="S3.SS3.p2.5.m5.1.1.1.cmml" xref="S3.SS3.p2.5.m5.1.1.1"></eq><ci id="S3.SS3.p2.5.m5.1.1.2.cmml" xref="S3.SS3.p2.5.m5.1.1.2">𝒵</ci><apply id="S3.SS3.p2.5.m5.1.1.3.cmml" xref="S3.SS3.p2.5.m5.1.1.3"><union id="S3.SS3.p2.5.m5.1.1.3.1.cmml" xref="S3.SS3.p2.5.m5.1.1.3.1"></union><apply id="S3.SS3.p2.5.m5.1.1.3.2.cmml" xref="S3.SS3.p2.5.m5.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS3.p2.5.m5.1.1.3.2.1.cmml" xref="S3.SS3.p2.5.m5.1.1.3.2">superscript</csymbol><ci id="S3.SS3.p2.5.m5.1.1.3.2.2.cmml" xref="S3.SS3.p2.5.m5.1.1.3.2.2">𝒵</ci><apply id="S3.SS3.p2.5.m5.1.1.3.2.3.cmml" xref="S3.SS3.p2.5.m5.1.1.3.2.3"><times id="S3.SS3.p2.5.m5.1.1.3.2.3.1.cmml" xref="S3.SS3.p2.5.m5.1.1.3.2.3.1"></times><ci id="S3.SS3.p2.5.m5.1.1.3.2.3.2.cmml" xref="S3.SS3.p2.5.m5.1.1.3.2.3.2">𝑙</ci><ci id="S3.SS3.p2.5.m5.1.1.3.2.3.3.cmml" xref="S3.SS3.p2.5.m5.1.1.3.2.3.3">𝑜</ci></apply></apply><apply id="S3.SS3.p2.5.m5.1.1.3.3.cmml" xref="S3.SS3.p2.5.m5.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS3.p2.5.m5.1.1.3.3.1.cmml" xref="S3.SS3.p2.5.m5.1.1.3.3">superscript</csymbol><ci id="S3.SS3.p2.5.m5.1.1.3.3.2.cmml" xref="S3.SS3.p2.5.m5.1.1.3.3.2">𝒵</ci><apply id="S3.SS3.p2.5.m5.1.1.3.3.3.cmml" xref="S3.SS3.p2.5.m5.1.1.3.3.3"><times id="S3.SS3.p2.5.m5.1.1.3.3.3.1.cmml" xref="S3.SS3.p2.5.m5.1.1.3.3.3.1"></times><ci id="S3.SS3.p2.5.m5.1.1.3.3.3.2.cmml" xref="S3.SS3.p2.5.m5.1.1.3.3.3.2">ℎ</ci><ci id="S3.SS3.p2.5.m5.1.1.3.3.3.3.cmml" xref="S3.SS3.p2.5.m5.1.1.3.3.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.5.m5.1c">\mathcal{Z}=\mathcal{Z}^{lo}\cup\mathcal{Z}^{hi}</annotation></semantics></math> and the rest of training proceeds as usual. In addition to allowing for efficient scoring, multi-resolution training itself yields a gain in efficiency: since <math id="S3.SS3.p2.6.m6.1" class="ltx_Math" alttext="\mathcal{B}^{lo}" display="inline"><semantics id="S3.SS3.p2.6.m6.1a"><msup id="S3.SS3.p2.6.m6.1.1" xref="S3.SS3.p2.6.m6.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.6.m6.1.1.2" xref="S3.SS3.p2.6.m6.1.1.2.cmml">ℬ</mi><mrow id="S3.SS3.p2.6.m6.1.1.3" xref="S3.SS3.p2.6.m6.1.1.3.cmml"><mi id="S3.SS3.p2.6.m6.1.1.3.2" xref="S3.SS3.p2.6.m6.1.1.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.6.m6.1.1.3.1" xref="S3.SS3.p2.6.m6.1.1.3.1.cmml">​</mo><mi id="S3.SS3.p2.6.m6.1.1.3.3" xref="S3.SS3.p2.6.m6.1.1.3.3.cmml">o</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.6.m6.1b"><apply id="S3.SS3.p2.6.m6.1.1.cmml" xref="S3.SS3.p2.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.6.m6.1.1.1.cmml" xref="S3.SS3.p2.6.m6.1.1">superscript</csymbol><ci id="S3.SS3.p2.6.m6.1.1.2.cmml" xref="S3.SS3.p2.6.m6.1.1.2">ℬ</ci><apply id="S3.SS3.p2.6.m6.1.1.3.cmml" xref="S3.SS3.p2.6.m6.1.1.3"><times id="S3.SS3.p2.6.m6.1.1.3.1.cmml" xref="S3.SS3.p2.6.m6.1.1.3.1"></times><ci id="S3.SS3.p2.6.m6.1.1.3.2.cmml" xref="S3.SS3.p2.6.m6.1.1.3.2">𝑙</ci><ci id="S3.SS3.p2.6.m6.1.1.3.3.cmml" xref="S3.SS3.p2.6.m6.1.1.3.3">𝑜</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.6.m6.1c">\mathcal{B}^{lo}</annotation></semantics></math> is processed with 4<math id="S3.SS3.p2.7.m7.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S3.SS3.p2.7.m7.1a"><mo id="S3.SS3.p2.7.m7.1.1" xref="S3.SS3.p2.7.m7.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.7.m7.1b"><times id="S3.SS3.p2.7.m7.1.1.cmml" xref="S3.SS3.p2.7.m7.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.7.m7.1c">\times</annotation></semantics></math> fewer tokens, it also benefits from close to a 4<math id="S3.SS3.p2.8.m8.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S3.SS3.p2.8.m8.1a"><mo id="S3.SS3.p2.8.m8.1.1" xref="S3.SS3.p2.8.m8.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.8.m8.1b"><times id="S3.SS3.p2.8.m8.1.1.cmml" xref="S3.SS3.p2.8.m8.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.8.m8.1c">\times</annotation></semantics></math> speed-up. If <math id="S3.SS3.p2.9.m9.1" class="ltx_Math" alttext="\mathcal{B}^{lo}" display="inline"><semantics id="S3.SS3.p2.9.m9.1a"><msup id="S3.SS3.p2.9.m9.1.1" xref="S3.SS3.p2.9.m9.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.9.m9.1.1.2" xref="S3.SS3.p2.9.m9.1.1.2.cmml">ℬ</mi><mrow id="S3.SS3.p2.9.m9.1.1.3" xref="S3.SS3.p2.9.m9.1.1.3.cmml"><mi id="S3.SS3.p2.9.m9.1.1.3.2" xref="S3.SS3.p2.9.m9.1.1.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.9.m9.1.1.3.1" xref="S3.SS3.p2.9.m9.1.1.3.1.cmml">​</mo><mi id="S3.SS3.p2.9.m9.1.1.3.3" xref="S3.SS3.p2.9.m9.1.1.3.3.cmml">o</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.9.m9.1b"><apply id="S3.SS3.p2.9.m9.1.1.cmml" xref="S3.SS3.p2.9.m9.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.9.m9.1.1.1.cmml" xref="S3.SS3.p2.9.m9.1.1">superscript</csymbol><ci id="S3.SS3.p2.9.m9.1.1.2.cmml" xref="S3.SS3.p2.9.m9.1.1.2">ℬ</ci><apply id="S3.SS3.p2.9.m9.1.1.3.cmml" xref="S3.SS3.p2.9.m9.1.1.3"><times id="S3.SS3.p2.9.m9.1.1.3.1.cmml" xref="S3.SS3.p2.9.m9.1.1.3.1"></times><ci id="S3.SS3.p2.9.m9.1.1.3.2.cmml" xref="S3.SS3.p2.9.m9.1.1.3.2">𝑙</ci><ci id="S3.SS3.p2.9.m9.1.1.3.3.cmml" xref="S3.SS3.p2.9.m9.1.1.3.3">𝑜</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.9.m9.1c">\mathcal{B}^{lo}</annotation></semantics></math> and <math id="S3.SS3.p2.10.m10.1" class="ltx_Math" alttext="\mathcal{B}^{hi}" display="inline"><semantics id="S3.SS3.p2.10.m10.1a"><msup id="S3.SS3.p2.10.m10.1.1" xref="S3.SS3.p2.10.m10.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.10.m10.1.1.2" xref="S3.SS3.p2.10.m10.1.1.2.cmml">ℬ</mi><mrow id="S3.SS3.p2.10.m10.1.1.3" xref="S3.SS3.p2.10.m10.1.1.3.cmml"><mi id="S3.SS3.p2.10.m10.1.1.3.2" xref="S3.SS3.p2.10.m10.1.1.3.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.10.m10.1.1.3.1" xref="S3.SS3.p2.10.m10.1.1.3.1.cmml">​</mo><mi id="S3.SS3.p2.10.m10.1.1.3.3" xref="S3.SS3.p2.10.m10.1.1.3.3.cmml">i</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.10.m10.1b"><apply id="S3.SS3.p2.10.m10.1.1.cmml" xref="S3.SS3.p2.10.m10.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.10.m10.1.1.1.cmml" xref="S3.SS3.p2.10.m10.1.1">superscript</csymbol><ci id="S3.SS3.p2.10.m10.1.1.2.cmml" xref="S3.SS3.p2.10.m10.1.1.2">ℬ</ci><apply id="S3.SS3.p2.10.m10.1.1.3.cmml" xref="S3.SS3.p2.10.m10.1.1.3"><times id="S3.SS3.p2.10.m10.1.1.3.1.cmml" xref="S3.SS3.p2.10.m10.1.1.3.1"></times><ci id="S3.SS3.p2.10.m10.1.1.3.2.cmml" xref="S3.SS3.p2.10.m10.1.1.3.2">ℎ</ci><ci id="S3.SS3.p2.10.m10.1.1.3.3.cmml" xref="S3.SS3.p2.10.m10.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.10.m10.1c">\mathcal{B}^{hi}</annotation></semantics></math> each account for half of the batch, the cost of multi-resolution training on <math id="S3.SS3.p2.11.m11.1" class="ltx_Math" alttext="\mathcal{B}" display="inline"><semantics id="S3.SS3.p2.11.m11.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.11.m11.1.1" xref="S3.SS3.p2.11.m11.1.1.cmml">ℬ</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.11.m11.1b"><ci id="S3.SS3.p2.11.m11.1.1.cmml" xref="S3.SS3.p2.11.m11.1.1">ℬ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.11.m11.1c">\mathcal{B}</annotation></semantics></math> is 64% of the FLOPs and 67% of the time of full-resolution training. Pseudocode for the full-resolution JEST and multi-resolution Flexi-JEST implementation is detailed in Algorithm <a href="#algorithm2" title="Algorithm 2 ‣ A.1 Training configuration ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
<figure id="S3.F2" class="ltx_figure"><img src="/html/2406.17711/assets/x2.png" id="S3.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="157" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span><span id="S3.F2.4.1" class="ltx_text ltx_font_bold">Joint example selection yields more learnable batches. Left:</span> the learnability of a batch is highly structured and non-diagonal. <span id="S3.F2.5.2" class="ltx_text ltx_font_bold">Middle: </span> Joint example selection quickly discovers sub-batches with high learnability, on-par with brute-force Gibbs sampling. <span id="S3.F2.6.3" class="ltx_text ltx_font_bold">Right: </span> the learnability of sampled batches improves with higher filtering ratios (i.e. selecting from larger super-batches). </figcaption>
</figure>
<div id="S3.SS3.p3" class="ltx_para ltx_noindent">
<p id="S3.SS3.p3.1" class="ltx_p"><span id="S3.SS3.p3.1.1" class="ltx_text ltx_font_bold">Training datasets.</span>

We train the learner model in all JEST experiments on the WebLI dataset <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>]</cite>, specifically
a billion-scale subset of English image-text pairs loosely filtered with image-text alignment <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>, using the <span id="S3.SS3.p3.1.2" class="ltx_text ltx_font_typewriter">big_vision</span> codebase <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib3" title="" class="ltx_ref">3</a>]</cite>. To train reference models, we use smaller high-quality datasets. JEST/Flexi-JEST reference models are trained on a strongly filtered 100M scale subset of WebLI filtered for high text and image quality and image-text alignment, which we refer to as “WebLI-curated”. We additionally explore
<span id="S3.SS3.p3.1.3" class="ltx_text ltx_font_italic">scaling data curation</span> (JEST++/FlexiJEST++) using reference models trained on “WebLI-curated++” which adds approximately 600M additional web-scraped image-text pairs
filtered with the same strong curation pipeline. Unless specified otherwise, the average performance we report uses 4 canonical benchmarks for multimodal transfer: ImageNet 0-Shot and 10-Shot classification and COCO image-to-text and text-to-image top-1 retrieval.

</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experiments</h2>

<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Joint example selection yields learnable batches</h3>

<div id="S4.SS1.p1" class="ltx_para ltx_noindent">
<p id="S4.SS1.p1.1" class="ltx_p">We start by evaluating the efficacy of joint example selection (JEST) for selecting learnable batches. To gain an intuition for our method, we start by visualizing the learnability matrix (i.e. the difference in loss between learner and reference models, for all pairs of examples in the batch).
JEST is designed to sample sub-matrices of examples in proportion to their summed learnability. Since the matrix is strongly non-diagonal (Figure <a href="#S3.F2" title="Figure 2 ‣ 3.3 Efficient scoring and multi-resolution training ‣ 3 Methods ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, left), independent selection will clearly be sub-optimal.</p>
</div>
<div id="S4.SS1.p2" class="ltx_para ltx_noindent">
<p id="S4.SS1.p2.1" class="ltx_p">With a small number of iterations (corresponding to populating the batch with <math id="S4.SS1.p2.1.m1.1" class="ltx_Math" alttext="N=16" display="inline"><semantics id="S4.SS1.p2.1.m1.1a"><mrow id="S4.SS1.p2.1.m1.1.1" xref="S4.SS1.p2.1.m1.1.1.cmml"><mi id="S4.SS1.p2.1.m1.1.1.2" xref="S4.SS1.p2.1.m1.1.1.2.cmml">N</mi><mo id="S4.SS1.p2.1.m1.1.1.1" xref="S4.SS1.p2.1.m1.1.1.1.cmml">=</mo><mn id="S4.SS1.p2.1.m1.1.1.3" xref="S4.SS1.p2.1.m1.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.1.m1.1b"><apply id="S4.SS1.p2.1.m1.1.1.cmml" xref="S4.SS1.p2.1.m1.1.1"><eq id="S4.SS1.p2.1.m1.1.1.1.cmml" xref="S4.SS1.p2.1.m1.1.1.1"></eq><ci id="S4.SS1.p2.1.m1.1.1.2.cmml" xref="S4.SS1.p2.1.m1.1.1.2">𝑁</ci><cn type="integer" id="S4.SS1.p2.1.m1.1.1.3.cmml" xref="S4.SS1.p2.1.m1.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.1.m1.1c">N=16</annotation></semantics></math> chunks), we find the learnability of the sub-batch to quickly increase, matching the learnability of batches extracted by brute-force Gibbs sampling requiring thousands of iterations (Figure <a href="#S3.F2" title="Figure 2 ‣ 3.3 Efficient scoring and multi-resolution training ‣ 3 Methods ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, middle).</p>
</div>
<div id="S4.SS1.p3" class="ltx_para ltx_noindent">
<p id="S4.SS1.p3.1" class="ltx_p">For filtering ratios of 0.5, 0.8, and 0.9, we select sub-batches of 32,768 examples from super-batches of size 65,536, 163,840 and 327,680 respectively. In Figure <a href="#S3.F2" title="Figure 2 ‣ 3.3 Efficient scoring and multi-resolution training ‣ 3 Methods ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, right, we find that the learnability of the sub-batch increases with larger filtering ratios.
In summary, our joint example selection (JEST) algorithm is an effective and efficient means of selecting highly learnable batches during training.</p>
</div>
<figure id="S4.F3" class="ltx_figure"><img src="/html/2406.17711/assets/Figures/fig_3_spi_single_color.png" id="S4.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="225" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span><span id="S4.F3.13.1" class="ltx_text ltx_font_bold">Joint example selection accelerates multimodal learning. Left:</span> training on the most learnable sub-batch selected from super-batches that are 2<math id="S4.F3.4.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.F3.4.m1.1b"><mo id="S4.F3.4.m1.1.1" xref="S4.F3.4.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.F3.4.m1.1c"><times id="S4.F3.4.m1.1.1.cmml" xref="S4.F3.4.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.4.m1.1d">\times</annotation></semantics></math>, 5<math id="S4.F3.5.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.F3.5.m2.1b"><mo id="S4.F3.5.m2.1.1" xref="S4.F3.5.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.F3.5.m2.1c"><times id="S4.F3.5.m2.1.1.cmml" xref="S4.F3.5.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.5.m2.1d">\times</annotation></semantics></math>, or 10<math id="S4.F3.6.m3.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.F3.6.m3.1b"><mo id="S4.F3.6.m3.1.1" xref="S4.F3.6.m3.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.F3.6.m3.1c"><times id="S4.F3.6.m3.1.1.cmml" xref="S4.F3.6.m3.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.F3.6.m3.1d">\times</annotation></semantics></math> larger significantly accelerates multimodal learning. <span id="S4.F3.14.2" class="ltx_text ltx_font_bold">Middle: </span> Jointly prioritizing <span id="S4.F3.15.3" class="ltx_text ltx_font_italic">learnable batches</span> yields significantly better results than simply prioritizing individual examples. <span id="S4.F3.16.4" class="ltx_text ltx_font_bold">Right: </span> joint examples selection also improves <span id="S4.F3.17.5" class="ltx_text ltx_font_italic">easy reference</span> prioritization, although <span id="S4.F3.18.6" class="ltx_text ltx_font_italic">learnability</span> scales better with more aggressive filtering.

</figcaption>
</figure>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Joint example selection accelerates multimodal learning</h3>

<div id="S4.SS2.p1" class="ltx_para ltx_noindent">
<p id="S4.SS2.p1.1" class="ltx_p">We now investigate the effect of training on more learnable batches, as selected by our JEST algorithm. All runs use a reference model trained on WebLI-curated, a ViT-B/16 and Bert-B image-text dual encoder, 3 billion training examples, and the sigmoid-contrastive loss. Figure <a href="#S4.F3" title="Figure 3 ‣ 4.1 Joint example selection yields learnable batches ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> (left) shows the average performance on multiple downstream tasks (ImageNet 0-Shot/10-Shot accuracy and COCO image-to-text/text-to-image retrieval) over the course of training. We find that JEST
significantly accelerates learning,
reaching the final performance of the 3B-uniform baseline after only 2B, 1B, and 0.67B training examples, when using filtering ratios of 50%, 80%, and 90% respectively. At larger filtering ratios we observe similar training instabilities to those observed for larger batch sizes <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>, necessitating a modification to stabilize the Adam optimizer (<math id="S4.SS2.p1.1.m1.1" class="ltx_Math" alttext="\beta_{2}=0.95" display="inline"><semantics id="S4.SS2.p1.1.m1.1a"><mrow id="S4.SS2.p1.1.m1.1.1" xref="S4.SS2.p1.1.m1.1.1.cmml"><msub id="S4.SS2.p1.1.m1.1.1.2" xref="S4.SS2.p1.1.m1.1.1.2.cmml"><mi id="S4.SS2.p1.1.m1.1.1.2.2" xref="S4.SS2.p1.1.m1.1.1.2.2.cmml">β</mi><mn id="S4.SS2.p1.1.m1.1.1.2.3" xref="S4.SS2.p1.1.m1.1.1.2.3.cmml">2</mn></msub><mo id="S4.SS2.p1.1.m1.1.1.1" xref="S4.SS2.p1.1.m1.1.1.1.cmml">=</mo><mn id="S4.SS2.p1.1.m1.1.1.3" xref="S4.SS2.p1.1.m1.1.1.3.cmml">0.95</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.1.m1.1b"><apply id="S4.SS2.p1.1.m1.1.1.cmml" xref="S4.SS2.p1.1.m1.1.1"><eq id="S4.SS2.p1.1.m1.1.1.1.cmml" xref="S4.SS2.p1.1.m1.1.1.1"></eq><apply id="S4.SS2.p1.1.m1.1.1.2.cmml" xref="S4.SS2.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.SS2.p1.1.m1.1.1.2.1.cmml" xref="S4.SS2.p1.1.m1.1.1.2">subscript</csymbol><ci id="S4.SS2.p1.1.m1.1.1.2.2.cmml" xref="S4.SS2.p1.1.m1.1.1.2.2">𝛽</ci><cn type="integer" id="S4.SS2.p1.1.m1.1.1.2.3.cmml" xref="S4.SS2.p1.1.m1.1.1.2.3">2</cn></apply><cn type="float" id="S4.SS2.p1.1.m1.1.1.3.cmml" xref="S4.SS2.p1.1.m1.1.1.3">0.95</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.1.m1.1c">\beta_{2}=0.95</annotation></semantics></math>) and suggesting that data curation with JEST can be thought of as increasing the effective batch size (Appendix <a href="#A1.SS2" title="A.2 Optimisation robustness ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.2</span></a>, <a href="#A1.SS3" title="A.3 Effects of varying the training batch size ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.3</span></a>).</p>
</div>
<div id="S4.SS2.p2" class="ltx_para ltx_noindent">
<p id="S4.SS2.p2.1" class="ltx_p">In terms of final performance, JEST also delivers significant gains of up to
6%
when filtering 90% of data (Figure <a href="#S4.F3" title="Figure 3 ‣ 4.1 Joint example selection yields learnable batches ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, middle, blue curve). Notably, this scaling behavior is absent from previous selection methods based on independent prioritization of individual examples (Figure <a href="#S4.F3" title="Figure 3 ‣ 4.1 Joint example selection yields learnable batches ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, middle, orange curve). Finally, we assess whether
JEST also improves prioritization criteria other than learnability. Figure <a href="#S4.F3" title="Figure 3 ‣ 4.1 Joint example selection yields learnable batches ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, right, shows the performance of models with <span id="S4.SS2.p2.1.1" class="ltx_text ltx_font_italic">easy-reference</span> prioritization, for varying filtering ratios. Consistent with learnability-based prioritization, JEST strongly outperforms independent example selection, particularly for high filtering ratios (where independent example selection leads to a regression in performance). Prioritising data with the highest loss produced smaller gains and degrades more quickly as we filter more data (Figure <a href="#A1.F10" title="Figure 10 ‣ A.7 Comparing approximation methods ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a>). Since learnability-based JEST yielded the best scaling behavior
we retain this criterion for subsequent experiments.</p>
</div>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Synergies between multi-resolution training and online batch selection</h3>

<div id="S4.SS3.p1" class="ltx_para ltx_noindent">
<p id="S4.SS3.p1.2" class="ltx_p">Joint example selection
with <span id="S4.SS3.p1.2.1" class="ltx_text ltx_font_italic">learnability</span> scores becomes more efficient as larger fractions of each batch are filtered.
However, the cost of scoring
results in a significant overhead:
filtering 80% of the super-batch results in 4<math id="S4.SS3.p1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS3.p1.1.m1.1a"><mo id="S4.SS3.p1.1.m1.1.1" xref="S4.SS3.p1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.1.m1.1b"><times id="S4.SS3.p1.1.m1.1.1.cmml" xref="S4.SS3.p1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.1.m1.1c">\times</annotation></semantics></math> more FLOPs per iteration than IID training, or 2.3<math id="S4.SS3.p1.2.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS3.p1.2.m2.1a"><mo id="S4.SS3.p1.2.m2.1.1" xref="S4.SS3.p1.2.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.2.m2.1b"><times id="S4.SS3.p1.2.m2.1.1.cmml" xref="S4.SS3.p1.2.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.2.m2.1c">\times</annotation></semantics></math> when caching the reference-model scores (Appendix <a href="#A1.SS4" title="A.4 FLOP calculations for JEST / Flexi-JEST ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.4</span></a>).
Although JEST is significantly more efficient in terms of training iterations (hereinafter ‘training efficiency’), the additional scoring FLOPs reduce its compute efficiency relative to the IID baseline
(Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, left vs. right). We therefore also investigated a compute efficient variant, Flexi-JEST, which uses multi-resolution training and low-resolution scoring to
reduce the total overhead to only 10% vs. the baseline
(Figure <a href="#S4.F4" title="Figure 4 ‣ 4.3 Synergies between multi-resolution training and online batch selection ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, left; see Section <a href="#A1.SS4" title="A.4 FLOP calculations for JEST / Flexi-JEST ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.4</span></a>).</p>
</div>
<div id="S4.SS3.p2" class="ltx_para ltx_noindent">
<p id="S4.SS3.p2.2" class="ltx_p">What is the effect of these approximations on performance? As might be expected, the per-iteration performance of Flexi-JEST decreases relative to the JEST, although still produces significant speed-ups over IID (Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, left; Figure <a href="#S4.F4" title="Figure 4 ‣ 4.3 Synergies between multi-resolution training and online batch selection ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, middle). However, the decrease in per-iteration performance is more than favorable when accounting for the decrease in total FLOPS: our best Flexi-JEST model produces the same average performance as a 40B Siglip run with 9.9<math id="S4.SS3.p2.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS3.p2.1.m1.1a"><mo id="S4.SS3.p2.1.m1.1.1" xref="S4.SS3.p2.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.1.m1.1b"><times id="S4.SS3.p2.1.m1.1.1.cmml" xref="S4.SS3.p2.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.1.m1.1c">\times</annotation></semantics></math> fewer FLOPs, and  2<math id="S4.SS3.p2.2.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS3.p2.2.m2.1a"><mo id="S4.SS3.p2.2.m2.1.1" xref="S4.SS3.p2.2.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.2.m2.1b"><times id="S4.SS3.p2.2.m2.1.1.cmml" xref="S4.SS3.p2.2.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.2.m2.1c">\times</annotation></semantics></math> fewer than full-resolution JEST (Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, right; Figure <a href="#S4.F4" title="Figure 4 ‣ 4.3 Synergies between multi-resolution training and online batch selection ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, middle).</p>
</div>
<figure id="S4.F4" class="ltx_figure"><img src="/html/2406.17711/assets/x3.png" id="S4.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="161" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span><span id="S4.F4.5.1" class="ltx_text ltx_font_bold">Efficient scoring and multi-resolution training </span>.
<span id="S4.F4.6.2" class="ltx_text ltx_font_bold">Left:</span>
In scoring large super-batches with the learner and reference models, JEST incurs a large computational cost per iteration. By caching the fixed reference model scores in the dataset, this overhead can be cut in half.
Efficient scoring and multi-resolution training further reduce this to be comparable to standard IID training.
<span id="S4.F4.7.3" class="ltx_text ltx_font_bold">Middle: </span> Flexi-JEST improves the total FLOP-efficiency of JEST over standard IID training. <span id="S4.F4.8.4" class="ltx_text ltx_font_bold">Right: </span> Multi-resolution training improves FlexiJEST more than standard IID training.
Without multi-resolution training (left-most point) Flexi-JEST underperforms the IID baseline (due to an untrained approximate model), but quickly improves with even a small amount of co-training (25%). 
</figcaption>
</figure>
<div id="S4.SS3.p3" class="ltx_para ltx_noindent">
<p id="S4.SS3.p3.3" class="ltx_p">What is the relative contribution of efficient scoring and multi-resolution training in Flexi-JEST?
We conducted an ablation where we varied the fraction of the selected batch trained at full and low resolution (i.e. the relative sizes of <math id="S4.SS3.p3.1.m1.1" class="ltx_Math" alttext="\mathcal{B}^{hi}" display="inline"><semantics id="S4.SS3.p3.1.m1.1a"><msup id="S4.SS3.p3.1.m1.1.1" xref="S4.SS3.p3.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS3.p3.1.m1.1.1.2" xref="S4.SS3.p3.1.m1.1.1.2.cmml">ℬ</mi><mrow id="S4.SS3.p3.1.m1.1.1.3" xref="S4.SS3.p3.1.m1.1.1.3.cmml"><mi id="S4.SS3.p3.1.m1.1.1.3.2" xref="S4.SS3.p3.1.m1.1.1.3.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="S4.SS3.p3.1.m1.1.1.3.1" xref="S4.SS3.p3.1.m1.1.1.3.1.cmml">​</mo><mi id="S4.SS3.p3.1.m1.1.1.3.3" xref="S4.SS3.p3.1.m1.1.1.3.3.cmml">i</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.1.m1.1b"><apply id="S4.SS3.p3.1.m1.1.1.cmml" xref="S4.SS3.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS3.p3.1.m1.1.1.1.cmml" xref="S4.SS3.p3.1.m1.1.1">superscript</csymbol><ci id="S4.SS3.p3.1.m1.1.1.2.cmml" xref="S4.SS3.p3.1.m1.1.1.2">ℬ</ci><apply id="S4.SS3.p3.1.m1.1.1.3.cmml" xref="S4.SS3.p3.1.m1.1.1.3"><times id="S4.SS3.p3.1.m1.1.1.3.1.cmml" xref="S4.SS3.p3.1.m1.1.1.3.1"></times><ci id="S4.SS3.p3.1.m1.1.1.3.2.cmml" xref="S4.SS3.p3.1.m1.1.1.3.2">ℎ</ci><ci id="S4.SS3.p3.1.m1.1.1.3.3.cmml" xref="S4.SS3.p3.1.m1.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.1.m1.1c">\mathcal{B}^{hi}</annotation></semantics></math> and <math id="S4.SS3.p3.2.m2.1" class="ltx_Math" alttext="\mathcal{B}^{lo}" display="inline"><semantics id="S4.SS3.p3.2.m2.1a"><msup id="S4.SS3.p3.2.m2.1.1" xref="S4.SS3.p3.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS3.p3.2.m2.1.1.2" xref="S4.SS3.p3.2.m2.1.1.2.cmml">ℬ</mi><mrow id="S4.SS3.p3.2.m2.1.1.3" xref="S4.SS3.p3.2.m2.1.1.3.cmml"><mi id="S4.SS3.p3.2.m2.1.1.3.2" xref="S4.SS3.p3.2.m2.1.1.3.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="S4.SS3.p3.2.m2.1.1.3.1" xref="S4.SS3.p3.2.m2.1.1.3.1.cmml">​</mo><mi id="S4.SS3.p3.2.m2.1.1.3.3" xref="S4.SS3.p3.2.m2.1.1.3.3.cmml">o</mi></mrow></msup><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.2.m2.1b"><apply id="S4.SS3.p3.2.m2.1.1.cmml" xref="S4.SS3.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS3.p3.2.m2.1.1.1.cmml" xref="S4.SS3.p3.2.m2.1.1">superscript</csymbol><ci id="S4.SS3.p3.2.m2.1.1.2.cmml" xref="S4.SS3.p3.2.m2.1.1.2">ℬ</ci><apply id="S4.SS3.p3.2.m2.1.1.3.cmml" xref="S4.SS3.p3.2.m2.1.1.3"><times id="S4.SS3.p3.2.m2.1.1.3.1.cmml" xref="S4.SS3.p3.2.m2.1.1.3.1"></times><ci id="S4.SS3.p3.2.m2.1.1.3.2.cmml" xref="S4.SS3.p3.2.m2.1.1.3.2">𝑙</ci><ci id="S4.SS3.p3.2.m2.1.1.3.3.cmml" xref="S4.SS3.p3.2.m2.1.1.3.3">𝑜</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.2.m2.1c">\mathcal{B}^{lo}</annotation></semantics></math>; see Methods).
We ensure the learner spends the same FLOPs by increasing the number of training iterations as we send more data to the approximate model, since the FLOPs per iteration decrease in this case (see Section <a href="#A1.SS7" title="A.7 Comparing approximation methods ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.7</span></a> for details). Figure <a href="#S4.F4" title="Figure 4 ‣ 4.3 Synergies between multi-resolution training and online batch selection ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> (right) shows that the IID baseline performance increases with larger fractions of data sent to the approximate model, consistent with a growing literature on the FLOP-efficiency of approximate training <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib4" title="" class="ltx_ref">4</a>, <a href="#bib.bib29" title="" class="ltx_ref">29</a>, <a href="#bib.bib13" title="" class="ltx_ref">13</a>, <a href="#bib.bib40" title="" class="ltx_ref">40</a>]</cite>.
Nevertheless, Flexi-JEST significantly outperforms the multi-resolution baseline as long as the low-res model trains on enough data to align with the learner
(e.g. <math id="S4.SS3.p3.3.m3.1" class="ltx_Math" alttext="\geq" display="inline"><semantics id="S4.SS3.p3.3.m3.1a"><mo id="S4.SS3.p3.3.m3.1.1" xref="S4.SS3.p3.3.m3.1.1.cmml">≥</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.p3.3.m3.1b"><geq id="S4.SS3.p3.3.m3.1.1.cmml" xref="S4.SS3.p3.3.m3.1.1"></geq></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p3.3.m3.1c">\geq</annotation></semantics></math> 25% data). These experiments demonstrate a synergy between multi-resolution training and joint example selection, as the former yields efficient and accurate scoring capabilities for accelerating the latter. </p>
</div>
<div id="S4.SS3.p4" class="ltx_para ltx_noindent">
<p id="S4.SS3.p4.1" class="ltx_p">Our results also point to a pareto front of data curation strategies. If maximizing training speed or training efficiency is desirable at the expense of computation, the full-resolution JEST method produces up to a 13<math id="S4.SS3.p4.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS3.p4.1.m1.1a"><mo id="S4.SS3.p4.1.m1.1.1" xref="S4.SS3.p4.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.p4.1.m1.1b"><times id="S4.SS3.p4.1.m1.1.1.cmml" xref="S4.SS3.p4.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p4.1.m1.1c">\times</annotation></semantics></math> speed up relative to a comparable IID training run. If FLOPs should be minimized at the expense of training efficiency, Flexi-JEST produces the most favorable trade-off. We note that the scoring of the next batch can be implemented on separate devices, in parallel with training,
potentially further reducing the additional wall-clock time.
</p>
</div>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>Joint examples selection enables strong data-quality bootstrapping</h3>

<div id="S4.SS4.p1" class="ltx_para ltx_noindent">
<p id="S4.SS4.p1.1" class="ltx_p">At the heart of learnability-based scoring is a reference model trained on a small, curated dataset of our choosing.
How does JEST performance vary as a function of different curation strategies that trade off quality vs. quantity? Furthermore, do improvements in JEST training correlate with the performance of the reference models or are these metrics decoupled?</p>
</div>
<div id="S4.SS4.p2" class="ltx_para ltx_noindent">
<p id="S4.SS4.p2.1" class="ltx_p"><span id="S4.SS4.p2.1.1" class="ltx_text ltx_font_bold">Understanding quality vs. quantity trade-offs</span>. We explore three scales of curation, each being a subset of the original WebLI dataset:
<span id="S4.SS4.p2.1.2" class="ltx_text ltx_font_italic">weak</span> (billion-scale) curation
with image-text alignment (ITA) filters,
<span id="S4.SS4.p2.1.3" class="ltx_text ltx_font_italic">moderate</span> (300M scale) curation with either ITA filters or text-quality (TQ) filters, and <span id="S4.SS4.p2.1.4" class="ltx_text ltx_font_italic">strong</span> (100M scale) curation with a combination of TQ, ITA, and additional image-quality (aesthetic) filters.
Throughout, we refer to this strongly curated subset as “WebLI-curated”.
</p>
</div>
<div id="S4.SS4.p3" class="ltx_para ltx_noindent">
<p id="S4.SS4.p3.1" class="ltx_p">We train standard SigLIP encoders on these four WebLI subsets for 10 epochs each, and use them as reference models for JEST training on the full WebLI dataset. Across curation methods, reference model performance and JEST performance appear to be decoupled (or even anti-correlated; Figure <a href="#S4.F5" title="Figure 5 ‣ 4.4 Joint examples selection enables strong data-quality bootstrapping ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, left), consistent with previous findings for fixed data curation <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite>.
Whereas increasing curation (and decreasing dataset size) yields weaker models, when used as reference models for JEST pretraining they have the opposite effect: JEST with a strongly-curated reference benefits from a 2.7% improvement, moderate a 1.5% improvement, and weak a 0.3% improvement.</p>
</div>
<figure id="S4.F5" class="ltx_figure"><img src="/html/2406.17711/assets/Figures/fig_6_curation_study_square.png" id="S4.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="538" height="274" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span><span id="S4.F5.5.1" class="ltx_text ltx_font_bold">Scaling strong data curation improves JEST performance.</span> <span id="S4.F5.6.2" class="ltx_text ltx_font_bold">Left:</span> We compare JEST performance vs. reference model performance (relative to the uniform baseline) for 4 curation types: ‘weak’ curation with image-text alignment (ITA), ‘moderate’ curation with ITA or text-quality (TQ), and ‘strong’ curation (using a combination of TQ, ITA, and additional image-quality (IQ). <span id="S4.F5.7.3" class="ltx_text ltx_font_bold">Right:</span> We use our best reference dataset (TQ+ITA+IQ) and evaluate JEST vs. reference performance varying the number of examples seen during reference pretraining. There is a strong correlation between additional reference training and JEST performance that saturates after 1B examples seen. By scaling strong data curation to a 600M dataset, this saturation is broken as both reference model and JEST performance improve for the 1B <span id="S4.F5.8.4" class="ltx_text ltx_font_italic">and</span> 2B reference training.

</figcaption>
</figure>
<div id="S4.SS4.p4" class="ltx_para ltx_noindent">
<p id="S4.SS4.p4.1" class="ltx_p"><span id="S4.SS4.p4.1.1" class="ltx_text ltx_font_bold">Scaling data curation</span>.
We hypothesized that
the general decoupling between reference model performance and JEST performance might simply be explained by the dataset size limits imposed by data curation.
To understand this effect, we trained 5 reference models on WebLI-curated while varying the total examples seen (250M to 3B). In this context, Figure <a href="#S4.F5" title="Figure 5 ‣ 4.4 Joint examples selection enables strong data-quality bootstrapping ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> (right) shows a striking correlation between improved reference models and better JEST pretraining. This suggests that the “decoupling” phenomenon can be mostly attributed to the saturation of the reference model as a result of the reductions in dataset size following curation.</p>
</div>
<div id="S4.SS4.p5" class="ltx_para ltx_noindent">
<p id="S4.SS4.p5.1" class="ltx_p">We note that the correlation in Figure <a href="#S4.F5" title="Figure 5 ‣ 4.4 Joint examples selection enables strong data-quality bootstrapping ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> (right) starts to break down when the dataset is saturated, i.e. after 10 epochs or 1B examples seen.
These results suggest that JEST would benefit further from scaling data curation of the reference datasets. To test this, we grew WebLI-curated to approximately 600M examples sourced from an expanded set of image-text pairs. At this scale, however, it is difficult to satisfy the stringent TQ/ITA criteria of WebLI-curated. Therefore, for all image-text pairs that did not meet the original ITA threshold, we re-captioned the images with high-quality synthetic captions based on the PaLI model family <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>, <a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite>—we denote this dataset as “WebLI-curated++”. We find that this scaled dataset allows us to break the 2B saturation point for “WebLI-curated” as both reference model and JEST performance (Figure <a href="#S4.F5" title="Figure 5 ‣ 4.4 Joint examples selection enables strong data-quality bootstrapping ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, Right: <math id="S4.SS4.p5.1.m1.1" class="ltx_Math" alttext="\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\star}" display="inline"><semantics id="S4.SS4.p5.1.m1.1a"><mo class="ltx_mathvariant_bold" mathcolor="#00FFFF" mathvariant="bold" id="S4.SS4.p5.1.m1.1.1" xref="S4.SS4.p5.1.m1.1.1.cmml">⋆</mo><annotation-xml encoding="MathML-Content" id="S4.SS4.p5.1.m1.1b"><ci id="S4.SS4.p5.1.m1.1.1.cmml" xref="S4.SS4.p5.1.m1.1.1">bold-⋆</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.p5.1.m1.1c">\color[rgb]{0,1,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,1}\pgfsys@color@cmyk@stroke{1}{0}{0}{0}\pgfsys@color@cmyk@fill{1}{0}{0}{0}\bm{\star}</annotation></semantics></math>) improves significantly. We therefore use WebLI-curated++ for our best models, JEST++ and FlexiJEST++.</p>
</div>
<div id="S4.SS4.p6" class="ltx_para ltx_noindent">
<p id="S4.SS4.p6.1" class="ltx_p">Given that scaling data curation with WebLI-curated++ strongly improves reference model performance, we asked whether pretraining on the original WebLI dataset is necessary at all. However when evaluating the reference model’s performance <span id="S4.SS4.p6.1.1" class="ltx_text ltx_font_italic">across datasets</span> we find it to be very imbalanced: while it outperforms WebLI pretraining on 2 downstream tasks, it significantly underperforms on 6 others, as well as on average (Table <a href="#A1.T5" title="Table 5 ‣ A.7 Comparing approximation methods ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>). In contrast, JEST++ pretraining on WebLI yields a <span id="S4.SS4.p6.1.2" class="ltx_text ltx_font_italic">generalist</span> foundation model that outperforms the baseline on 6 of 8 benchmarks, as well as on average.</p>
</div>
<figure id="S4.T1" class="ltx_table">
<table id="S4.T1.5" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T1.5.6.1" class="ltx_tr">
<th id="S4.T1.5.6.1.1" class="ltx_td ltx_th ltx_th_column"></th>
<th id="S4.T1.5.6.1.2" class="ltx_td ltx_th ltx_th_column"></th>
<th id="S4.T1.5.6.1.3" class="ltx_td ltx_th ltx_th_column"></th>
<th id="S4.T1.5.6.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column" colspan="2">FLOPs %</th>
<th id="S4.T1.5.6.1.5" class="ltx_td ltx_th ltx_th_column"></th>
<th id="S4.T1.5.6.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column" colspan="2">ImageNet-1K</th>
<th id="S4.T1.5.6.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column" colspan="2">COCO</th>
</tr>
<tr id="S4.T1.1.1" class="ltx_tr">
<th id="S4.T1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column">Method</th>
<th id="S4.T1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">Variant</th>
<th id="S4.T1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column"># Train</th>
<th id="S4.T1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Per Iter.</th>
<th id="S4.T1.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t"><span id="S4.T1.1.1.6.1" class="ltx_text ltx_font_bold">Total</span></th>
<th id="S4.T1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column">
<span id="S4.T1.1.1.1.1" class="ltx_text ltx_font_bold">Mean</span> <math id="S4.T1.1.1.1.m1.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="S4.T1.1.1.1.m1.1a"><mi mathvariant="normal" id="S4.T1.1.1.1.m1.1.1" xref="S4.T1.1.1.1.m1.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="S4.T1.1.1.1.m1.1b"><ci id="S4.T1.1.1.1.m1.1.1.cmml" xref="S4.T1.1.1.1.m1.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.1.1.1.m1.1c">\Delta</annotation></semantics></math>
</th>
<th id="S4.T1.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">10-S</th>
<th id="S4.T1.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">ZS</th>
<th id="S4.T1.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">I2T</th>
<th id="S4.T1.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">T2I</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T1.5.7.1" class="ltx_tr">
<td id="S4.T1.5.7.1.1" class="ltx_td ltx_align_left ltx_border_t">CLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib38" title="" class="ltx_ref">38</a>]</cite>
</td>
<td id="S4.T1.5.7.1.2" class="ltx_td ltx_align_center ltx_border_t">B</td>
<td id="S4.T1.5.7.1.3" class="ltx_td ltx_align_center ltx_border_t">13B</td>
<td id="S4.T1.5.7.1.4" class="ltx_td ltx_align_center ltx_border_t">100</td>
<td id="S4.T1.5.7.1.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">  32</td>
<td id="S4.T1.5.7.1.6" class="ltx_td ltx_align_center ltx_border_t">– 11.8</td>
<td id="S4.T1.5.7.1.7" class="ltx_td ltx_border_t"></td>
<td id="S4.T1.5.7.1.8" class="ltx_td ltx_align_center ltx_border_t">68.3</td>
<td id="S4.T1.5.7.1.9" class="ltx_td ltx_align_center ltx_border_t">52.4</td>
<td id="S4.T1.5.7.1.10" class="ltx_td ltx_align_center ltx_border_t">33.1</td>
</tr>
<tr id="S4.T1.5.8.2" class="ltx_tr">
<td id="S4.T1.5.8.2.1" class="ltx_td ltx_align_left">EVA-CLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib48" title="" class="ltx_ref">48</a>]</cite>
</td>
<td id="S4.T1.5.8.2.2" class="ltx_td ltx_align_center">B</td>
<td id="S4.T1.5.8.2.3" class="ltx_td ltx_align_center">  8B</td>
<td id="S4.T1.5.8.2.4" class="ltx_td ltx_align_center">100</td>
<td id="S4.T1.5.8.2.5" class="ltx_td ltx_align_center ltx_border_r">  20</td>
<td id="S4.T1.5.8.2.6" class="ltx_td ltx_align_center">  – 4.6</td>
<td id="S4.T1.5.8.2.7" class="ltx_td"></td>
<td id="S4.T1.5.8.2.8" class="ltx_td ltx_align_center">74.7</td>
<td id="S4.T1.5.8.2.9" class="ltx_td ltx_align_center">58.7</td>
<td id="S4.T1.5.8.2.10" class="ltx_td ltx_align_center">42.2</td>
</tr>
<tr id="S4.T1.5.9.3" class="ltx_tr">
<td id="S4.T1.5.9.3.1" class="ltx_td ltx_align_left">OpenCLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib23" title="" class="ltx_ref">23</a>]</cite>
</td>
<td id="S4.T1.5.9.3.2" class="ltx_td ltx_align_center">B</td>
<td id="S4.T1.5.9.3.3" class="ltx_td ltx_align_center">34B</td>
<td id="S4.T1.5.9.3.4" class="ltx_td ltx_align_center">100</td>
<td id="S4.T1.5.9.3.5" class="ltx_td ltx_align_center ltx_border_r">  85</td>
<td id="S4.T1.5.9.3.6" class="ltx_td ltx_align_center">  – 5.8</td>
<td id="S4.T1.5.9.3.7" class="ltx_td"></td>
<td id="S4.T1.5.9.3.8" class="ltx_td ltx_align_center">70.2</td>
<td id="S4.T1.5.9.3.9" class="ltx_td ltx_align_center">59.4</td>
<td id="S4.T1.5.9.3.10" class="ltx_td ltx_align_center">42.3</td>
</tr>
<tr id="S4.T1.5.10.4" class="ltx_tr">
<td id="S4.T1.5.10.4.1" class="ltx_td ltx_align_left">LessIsMore <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib8" title="" class="ltx_ref">8</a>]</cite>
</td>
<td id="S4.T1.5.10.4.2" class="ltx_td ltx_align_center">B</td>
<td id="S4.T1.5.10.4.3" class="ltx_td ltx_align_center">11B</td>
<td id="S4.T1.5.10.4.4" class="ltx_td ltx_align_center">100</td>
<td id="S4.T1.5.10.4.5" class="ltx_td ltx_align_center ltx_border_r">  28</td>
<td id="S4.T1.5.10.4.6" class="ltx_td ltx_align_center">  – 5.9</td>
<td id="S4.T1.5.10.4.7" class="ltx_td"></td>
<td id="S4.T1.5.10.4.8" class="ltx_td ltx_align_center">70.8</td>
<td id="S4.T1.5.10.4.9" class="ltx_td ltx_align_center">58.3</td>
<td id="S4.T1.5.10.4.10" class="ltx_td ltx_align_center">42.5</td>
</tr>
<tr id="S4.T1.2.2" class="ltx_tr">
<td id="S4.T1.2.2.2" class="ltx_td ltx_align_left">SILC-S <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib35" title="" class="ltx_ref">35</a>]</cite>
</td>
<td id="S4.T1.2.2.3" class="ltx_td ltx_align_center">B</td>
<td id="S4.T1.2.2.4" class="ltx_td ltx_align_center">20B</td>
<td id="S4.T1.2.2.5" class="ltx_td ltx_align_center">380</td>
<td id="S4.T1.2.2.6" class="ltx_td ltx_align_center ltx_border_r">190</td>
<td id="S4.T1.2.2.1" class="ltx_td ltx_align_center"> <math id="S4.T1.2.2.1.m1.1" class="ltx_Math" alttext="+" display="inline"><semantics id="S4.T1.2.2.1.m1.1a"><mo id="S4.T1.2.2.1.m1.1.1" xref="S4.T1.2.2.1.m1.1.1.cmml">+</mo><annotation-xml encoding="MathML-Content" id="S4.T1.2.2.1.m1.1b"><plus id="S4.T1.2.2.1.m1.1.1.cmml" xref="S4.T1.2.2.1.m1.1.1"></plus></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.2.2.1.m1.1c">+</annotation></semantics></math> 0.2</td>
<td id="S4.T1.2.2.7" class="ltx_td ltx_align_center">68.9</td>
<td id="S4.T1.2.2.8" class="ltx_td ltx_align_center">76.6</td>
<td id="S4.T1.2.2.9" class="ltx_td ltx_align_center">66.2</td>
<td id="S4.T1.2.2.10" class="ltx_td ltx_align_center">48.7</td>
</tr>
<tr id="S4.T1.5.11.5" class="ltx_tr">
<td id="S4.T1.5.11.5.1" class="ltx_td ltx_align_left">SigLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>
</td>
<td id="S4.T1.5.11.5.2" class="ltx_td ltx_align_center">B</td>
<td id="S4.T1.5.11.5.3" class="ltx_td ltx_align_center">40B</td>
<td id="S4.T1.5.11.5.4" class="ltx_td ltx_align_center">100</td>
<td id="S4.T1.5.11.5.5" class="ltx_td ltx_align_center ltx_border_r">100</td>
<td id="S4.T1.5.11.5.6" class="ltx_td ltx_align_center">    0.0</td>
<td id="S4.T1.5.11.5.7" class="ltx_td ltx_align_center">70.3</td>
<td id="S4.T1.5.11.5.8" class="ltx_td ltx_align_center">76.7</td>
<td id="S4.T1.5.11.5.9" class="ltx_td ltx_align_center">65.2</td>
<td id="S4.T1.5.11.5.10" class="ltx_td ltx_align_center">47.4</td>
</tr>
<tr id="S4.T1.3.3" class="ltx_tr" style="background-color:#E6E6FF;">
<td id="S4.T1.3.3.2" class="ltx_td ltx_align_left"><span id="S4.T1.3.3.2.1" class="ltx_text" style="background-color:#E6E6FF;">JEST++</span></td>
<td id="S4.T1.3.3.3" class="ltx_td ltx_align_center"><span id="S4.T1.3.3.3.1" class="ltx_text" style="background-color:#E6E6FF;">B</span></td>
<td id="S4.T1.3.3.4" class="ltx_td ltx_align_center"><span id="S4.T1.3.3.4.1" class="ltx_text" style="background-color:#E6E6FF;">  4B</span></td>
<td id="S4.T1.3.3.5" class="ltx_td ltx_align_center"><span id="S4.T1.3.3.5.1" class="ltx_text" style="background-color:#E6E6FF;">233</span></td>
<td id="S4.T1.3.3.6" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T1.3.3.6.1" class="ltx_text" style="background-color:#E6E6FF;">  23</span></td>
<td id="S4.T1.3.3.1" class="ltx_td ltx_align_center"><span id="S4.T1.3.3.1.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;"> <math id="S4.T1.3.3.1.1.m1.1" class="ltx_Math" alttext="+" display="inline"><semantics id="S4.T1.3.3.1.1.m1.1a"><mo mathbackground="#E6E6FF" id="S4.T1.3.3.1.1.m1.1.1" xref="S4.T1.3.3.1.1.m1.1.1.cmml">+</mo><annotation-xml encoding="MathML-Content" id="S4.T1.3.3.1.1.m1.1b"><plus id="S4.T1.3.3.1.1.m1.1.1.cmml" xref="S4.T1.3.3.1.1.m1.1.1"></plus></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.3.3.1.1.m1.1c">+</annotation></semantics></math> 2.8</span></td>
<td id="S4.T1.3.3.7" class="ltx_td ltx_align_center"><span id="S4.T1.3.3.7.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;">70.3</span></td>
<td id="S4.T1.3.3.8" class="ltx_td ltx_align_center"><span id="S4.T1.3.3.8.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;">76.9</span></td>
<td id="S4.T1.3.3.9" class="ltx_td ltx_align_center"><span id="S4.T1.3.3.9.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;">70.3</span></td>
<td id="S4.T1.3.3.10" class="ltx_td ltx_align_center"><span id="S4.T1.3.3.10.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;">53.3</span></td>
</tr>
<tr id="S4.T1.4.4" class="ltx_tr" style="background-color:#E6E6FF;">
<td id="S4.T1.4.4.2" class="ltx_td ltx_align_left"><span id="S4.T1.4.4.2.1" class="ltx_text" style="background-color:#E6E6FF;">Flexi-JEST++</span></td>
<td id="S4.T1.4.4.3" class="ltx_td ltx_align_center"><span id="S4.T1.4.4.3.1" class="ltx_text" style="background-color:#E6E6FF;">B</span></td>
<td id="S4.T1.4.4.4" class="ltx_td ltx_align_center"><span id="S4.T1.4.4.4.1" class="ltx_text" style="background-color:#E6E6FF;">  4B</span></td>
<td id="S4.T1.4.4.5" class="ltx_td ltx_align_center"><span id="S4.T1.4.4.5.1" class="ltx_text" style="background-color:#E6E6FF;">110</span></td>
<td id="S4.T1.4.4.6" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T1.4.4.6.1" class="ltx_text" style="background-color:#E6E6FF;">  <span id="S4.T1.4.4.6.1.1" class="ltx_text ltx_font_bold">11</span></span></td>
<td id="S4.T1.4.4.1" class="ltx_td ltx_align_center"><span id="S4.T1.4.4.1.1" class="ltx_text" style="background-color:#E6E6FF;"> <math id="S4.T1.4.4.1.1.m1.1" class="ltx_Math" alttext="+" display="inline"><semantics id="S4.T1.4.4.1.1.m1.1a"><mo mathbackground="#E6E6FF" id="S4.T1.4.4.1.1.m1.1.1" xref="S4.T1.4.4.1.1.m1.1.1.cmml">+</mo><annotation-xml encoding="MathML-Content" id="S4.T1.4.4.1.1.m1.1b"><plus id="S4.T1.4.4.1.1.m1.1.1.cmml" xref="S4.T1.4.4.1.1.m1.1.1"></plus></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.4.4.1.1.m1.1c">+</annotation></semantics></math> 0.9</span></td>
<td id="S4.T1.4.4.7" class="ltx_td ltx_align_center"><span id="S4.T1.4.4.7.1" class="ltx_text" style="background-color:#E6E6FF;">68.2</span></td>
<td id="S4.T1.4.4.8" class="ltx_td ltx_align_center"><span id="S4.T1.4.4.8.1" class="ltx_text" style="background-color:#E6E6FF;">75.8</span></td>
<td id="S4.T1.4.4.9" class="ltx_td ltx_align_center"><span id="S4.T1.4.4.9.1" class="ltx_text" style="background-color:#E6E6FF;">68.0</span></td>
<td id="S4.T1.4.4.10" class="ltx_td ltx_align_center"><span id="S4.T1.4.4.10.1" class="ltx_text" style="background-color:#E6E6FF;">51.2</span></td>
</tr>
<tr id="S4.T1.5.12.6" class="ltx_tr">
<td id="S4.T1.5.12.6.1" class="ltx_td ltx_align_left">CLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib38" title="" class="ltx_ref">38</a>]</cite>
</td>
<td id="S4.T1.5.12.6.2" class="ltx_td ltx_align_center">L</td>
<td id="S4.T1.5.12.6.3" class="ltx_td ltx_align_center">13B</td>
<td id="S4.T1.5.12.6.4" class="ltx_td ltx_align_center">100</td>
<td id="S4.T1.5.12.6.5" class="ltx_td ltx_align_center ltx_border_r">  32</td>
<td id="S4.T1.5.12.6.6" class="ltx_td ltx_align_center">– 11.0</td>
<td id="S4.T1.5.12.6.7" class="ltx_td"></td>
<td id="S4.T1.5.12.6.8" class="ltx_td ltx_align_center">75.5</td>
<td id="S4.T1.5.12.6.9" class="ltx_td ltx_align_center">56.3</td>
<td id="S4.T1.5.12.6.10" class="ltx_td ltx_align_center">36.5</td>
</tr>
<tr id="S4.T1.5.13.7" class="ltx_tr">
<td id="S4.T1.5.13.7.1" class="ltx_td ltx_align_left">EVA-CLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib48" title="" class="ltx_ref">48</a>]</cite>
</td>
<td id="S4.T1.5.13.7.2" class="ltx_td ltx_align_center">L</td>
<td id="S4.T1.5.13.7.3" class="ltx_td ltx_align_center">   4B</td>
<td id="S4.T1.5.13.7.4" class="ltx_td ltx_align_center">100</td>
<td id="S4.T1.5.13.7.5" class="ltx_td ltx_align_center ltx_border_r">  10</td>
<td id="S4.T1.5.13.7.6" class="ltx_td ltx_align_center">  – 3.4</td>
<td id="S4.T1.5.13.7.7" class="ltx_td"></td>
<td id="S4.T1.5.13.7.8" class="ltx_td ltx_align_center">79.8</td>
<td id="S4.T1.5.13.7.9" class="ltx_td ltx_align_center">63.7</td>
<td id="S4.T1.5.13.7.10" class="ltx_td ltx_align_center">47.5</td>
</tr>
<tr id="S4.T1.5.14.8" class="ltx_tr">
<td id="S4.T1.5.14.8.1" class="ltx_td ltx_align_left">OpenCLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib23" title="" class="ltx_ref">23</a>]</cite>
</td>
<td id="S4.T1.5.14.8.2" class="ltx_td ltx_align_center">L</td>
<td id="S4.T1.5.14.8.3" class="ltx_td ltx_align_center">32B</td>
<td id="S4.T1.5.14.8.4" class="ltx_td ltx_align_center">100</td>
<td id="S4.T1.5.14.8.5" class="ltx_td ltx_align_center ltx_border_r">  80</td>
<td id="S4.T1.5.14.8.6" class="ltx_td ltx_align_center">  – 6.3</td>
<td id="S4.T1.5.14.8.7" class="ltx_td"></td>
<td id="S4.T1.5.14.8.8" class="ltx_td ltx_align_center">74.0</td>
<td id="S4.T1.5.14.8.9" class="ltx_td ltx_align_center">62.1</td>
<td id="S4.T1.5.14.8.10" class="ltx_td ltx_align_center">46.1</td>
</tr>
<tr id="S4.T1.5.15.9" class="ltx_tr">
<td id="S4.T1.5.15.9.1" class="ltx_td ltx_align_left">SigLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>
</td>
<td id="S4.T1.5.15.9.2" class="ltx_td ltx_align_center">L</td>
<td id="S4.T1.5.15.9.3" class="ltx_td ltx_align_center">40B</td>
<td id="S4.T1.5.15.9.4" class="ltx_td ltx_align_center">100</td>
<td id="S4.T1.5.15.9.5" class="ltx_td ltx_align_center ltx_border_r">100</td>
<td id="S4.T1.5.15.9.6" class="ltx_td ltx_align_center">    0.0</td>
<td id="S4.T1.5.15.9.7" class="ltx_td ltx_align_center">77.1</td>
<td id="S4.T1.5.15.9.8" class="ltx_td ltx_align_center">80.5</td>
<td id="S4.T1.5.15.9.9" class="ltx_td ltx_align_center">69.5</td>
<td id="S4.T1.5.15.9.10" class="ltx_td ltx_align_center">51.2</td>
</tr>
<tr id="S4.T1.5.5" class="ltx_tr" style="background-color:#E6E6FF;">
<td id="S4.T1.5.5.2" class="ltx_td ltx_align_left"><span id="S4.T1.5.5.2.1" class="ltx_text" style="background-color:#E6E6FF;">JEST++</span></td>
<td id="S4.T1.5.5.3" class="ltx_td ltx_align_center"><span id="S4.T1.5.5.3.1" class="ltx_text" style="background-color:#E6E6FF;">L</span></td>
<td id="S4.T1.5.5.4" class="ltx_td ltx_align_center"><span id="S4.T1.5.5.4.1" class="ltx_text" style="background-color:#E6E6FF;">  4B</span></td>
<td id="S4.T1.5.5.5" class="ltx_td ltx_align_center"><span id="S4.T1.5.5.5.1" class="ltx_text" style="background-color:#E6E6FF;">233</span></td>
<td id="S4.T1.5.5.6" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T1.5.5.6.1" class="ltx_text" style="background-color:#E6E6FF;">  <span id="S4.T1.5.5.6.1.1" class="ltx_text ltx_font_bold">23</span></span></td>
<td id="S4.T1.5.5.1" class="ltx_td ltx_align_center"><span id="S4.T1.5.5.1.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;"> <math id="S4.T1.5.5.1.1.m1.1" class="ltx_Math" alttext="+" display="inline"><semantics id="S4.T1.5.5.1.1.m1.1a"><mo mathbackground="#E6E6FF" id="S4.T1.5.5.1.1.m1.1.1" xref="S4.T1.5.5.1.1.m1.1.1.cmml">+</mo><annotation-xml encoding="MathML-Content" id="S4.T1.5.5.1.1.m1.1b"><plus id="S4.T1.5.5.1.1.m1.1.1.cmml" xref="S4.T1.5.5.1.1.m1.1.1"></plus></annotation-xml><annotation encoding="application/x-tex" id="S4.T1.5.5.1.1.m1.1c">+</annotation></semantics></math> 1.8</span></td>
<td id="S4.T1.5.5.7" class="ltx_td ltx_align_center"><span id="S4.T1.5.5.7.1" class="ltx_text" style="background-color:#E6E6FF;">75.5</span></td>
<td id="S4.T1.5.5.8" class="ltx_td ltx_align_center"><span id="S4.T1.5.5.8.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;">80.5</span></td>
<td id="S4.T1.5.5.9" class="ltx_td ltx_align_center"><span id="S4.T1.5.5.9.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;">71.1</span></td>
<td id="S4.T1.5.5.10" class="ltx_td ltx_align_center"><span id="S4.T1.5.5.10.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;">54.8</span></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>
<span id="S4.T1.7.1" class="ltx_text ltx_font_bold">Comparison to prior art.</span> FLOP % are measured relative to SigLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>. Mean denotes the average performance over all metrics. “Per Iter.” denotes FLOPs per iteration.
</figcaption>
</figure>
</section>
<section id="S4.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.5 </span>Comparison to prior art</h3>

<div id="S4.SS5.p1" class="ltx_para ltx_noindent">
<p id="S4.SS5.p1.3" class="ltx_p">We now compare to prior art, including the state-of-art SigLIP model trained for 40 billion examples <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite> as well as recent strong CLIP variants.
Table <a href="#S4.T1" title="Table 1 ‣ 4.4 Joint examples selection enables strong data-quality bootstrapping ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> shows that our most training-efficient model, JEST++, sets a new state-of-the-art on both ImageNet and COCO all while using 10<math id="S4.SS5.p1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS5.p1.1.m1.1a"><mo id="S4.SS5.p1.1.m1.1.1" xref="S4.SS5.p1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS5.p1.1.m1.1b"><times id="S4.SS5.p1.1.m1.1.1.cmml" xref="S4.SS5.p1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p1.1.m1.1c">\times</annotation></semantics></math> fewer iterations and 4<math id="S4.SS5.p1.2.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS5.p1.2.m2.1a"><mo id="S4.SS5.p1.2.m2.1.1" xref="S4.SS5.p1.2.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS5.p1.2.m2.1b"><times id="S4.SS5.p1.2.m2.1.1.cmml" xref="S4.SS5.p1.2.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p1.2.m2.1c">\times</annotation></semantics></math> less compute. On COCO in particular, JEST++ improves the previous state of the art by over 5%. Our most compute-efficient model, Flexi-JEST++, also surpasses the previous SoTA on average, while using 9<math id="S4.SS5.p1.3.m3.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS5.p1.3.m3.1a"><mo id="S4.SS5.p1.3.m3.1.1" xref="S4.SS5.p1.3.m3.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS5.p1.3.m3.1b"><times id="S4.SS5.p1.3.m3.1.1.cmml" xref="S4.SS5.p1.3.m3.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p1.3.m3.1c">\times</annotation></semantics></math> less compute. Training JEST for longer furthered these gains (see Appendix Table <a href="#A1.T4" title="Table 4 ‣ A.7 Comparing approximation methods ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>).</p>
</div>
<div id="S4.SS5.p2" class="ltx_para ltx_noindent">
<p id="S4.SS5.p2.1" class="ltx_p">Our results also scale gracefully with model size. Training with a ViT-L learner and ViT-L reference trained on the same WebLI-curated++ dataset,
JEST++
continues to yield strongly accelerated learning,
matching the SigLIP ViT-L 40B baseline with only 4B examples seen (Table <a href="#S4.T1" title="Table 1 ‣ 4.4 Joint examples selection enables strong data-quality bootstrapping ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, bottom).</p>
</div>
<div id="S4.SS5.p3" class="ltx_para ltx_noindent">
<p id="S4.SS5.p3.1" class="ltx_p">Finally, we apply JEST++ for pretraining on the publicly available LAION-2B dataset <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib44" title="" class="ltx_ref">44</a>]</cite>. We follow the standard practice of removing unsafe image-text pairs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib46" title="" class="ltx_ref">46</a>]</cite>, but do not otherwise pre-filter the dataset. JEST++ strongly surpasses previous methods for offline data curation, despite using 4<math id="S4.SS5.p3.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S4.SS5.p3.1.m1.1a"><mo id="S4.SS5.p3.1.m1.1.1" xref="S4.SS5.p3.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S4.SS5.p3.1.m1.1b"><times id="S4.SS5.p3.1.m1.1.1.cmml" xref="S4.SS5.p3.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S4.SS5.p3.1.m1.1c">\times</annotation></semantics></math> fewer training examples than the previous state-of-the-art (Table <a href="#S4.T2" title="Table 2 ‣ 4.5 Comparison to prior art ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>). With this training budget, SigLIP pretraining severely under-performs all methods, further highlighting the benefits of JEST.</p>
</div>
<div id="S4.SS5.p4" class="ltx_para ltx_noindent">
<div id="S4.SS5.p4.1" class="ltx_block ltx_minipage ltx_align_middle" style="width:229.8pt;">
<table id="S4.SS5.p4.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.SS5.p4.1.1.1.1" class="ltx_tr">
<th id="S4.SS5.p4.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row">Method</th>
<th id="S4.SS5.p4.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column"># Train</th>
<th id="S4.SS5.p4.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">IN1K ZS</th>
<th id="S4.SS5.p4.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column">COCO</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.SS5.p4.1.1.2.1" class="ltx_tr">
<th id="S4.SS5.p4.1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">LAION-440M <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>]</cite>
</th>
<td id="S4.SS5.p4.1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">12.8B</td>
<td id="S4.SS5.p4.1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">64.1</td>
<td id="S4.SS5.p4.1.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t">48.1</td>
</tr>
<tr id="S4.SS5.p4.1.1.3.2" class="ltx_tr">
<th id="S4.SS5.p4.1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">SemDeDup <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib1" title="" class="ltx_ref">1</a>]</cite>
</th>
<td id="S4.SS5.p4.1.1.3.2.2" class="ltx_td ltx_align_center">8.8B</td>
<td id="S4.SS5.p4.1.1.3.2.3" class="ltx_td ltx_align_center">64.3</td>
<td id="S4.SS5.p4.1.1.3.2.4" class="ltx_td ltx_align_center">48.9</td>
</tr>
<tr id="S4.SS5.p4.1.1.4.3" class="ltx_tr">
<th id="S4.SS5.p4.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">DBP <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib2" title="" class="ltx_ref">2</a>]</cite>
</th>
<td id="S4.SS5.p4.1.1.4.3.2" class="ltx_td ltx_align_center">5.3B</td>
<td id="S4.SS5.p4.1.1.4.3.3" class="ltx_td ltx_align_center">65.5</td>
<td id="S4.SS5.p4.1.1.4.3.4" class="ltx_td ltx_align_center">48.4</td>
</tr>
<tr id="S4.SS5.p4.1.1.5.4" class="ltx_tr">
<th id="S4.SS5.p4.1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">DBP <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib2" title="" class="ltx_ref">2</a>]</cite>
</th>
<td id="S4.SS5.p4.1.1.5.4.2" class="ltx_td ltx_align_center">3.6B</td>
<td id="S4.SS5.p4.1.1.5.4.3" class="ltx_td ltx_align_center">64.1</td>
<td id="S4.SS5.p4.1.1.5.4.4" class="ltx_td ltx_align_center">45.7</td>
</tr>
<tr id="S4.SS5.p4.1.1.6.5" class="ltx_tr">
<th id="S4.SS5.p4.1.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">SigLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>
</th>
<td id="S4.SS5.p4.1.1.6.5.2" class="ltx_td ltx_align_center">1.3B</td>
<td id="S4.SS5.p4.1.1.6.5.3" class="ltx_td ltx_align_center">57.2</td>
<td id="S4.SS5.p4.1.1.6.5.4" class="ltx_td ltx_align_center">43.3</td>
</tr>
<tr id="S4.SS5.p4.1.1.7.6" class="ltx_tr" style="background-color:#E6E6FF;">
<th id="S4.SS5.p4.1.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.SS5.p4.1.1.7.6.1.1" class="ltx_text" style="background-color:#E6E6FF;">JEST++</span></th>
<td id="S4.SS5.p4.1.1.7.6.2" class="ltx_td ltx_align_center"><span id="S4.SS5.p4.1.1.7.6.2.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;">1.3B</span></td>
<td id="S4.SS5.p4.1.1.7.6.3" class="ltx_td ltx_align_center"><span id="S4.SS5.p4.1.1.7.6.3.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;">66.8</span></td>
<td id="S4.SS5.p4.1.1.7.6.4" class="ltx_td ltx_align_center"><span id="S4.SS5.p4.1.1.7.6.4.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;">54.8</span></td>
</tr>
</tbody>
</table>
</div>
<div id="S4.SS5.p4.2" class="ltx_logical-block ltx_minipage ltx_align_middle" style="width:203.8pt;">
<figure id="S4.T2" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 2: </span><span id="S4.T2.2.1" class="ltx_text ltx_font_bold">Comparison to LAION pretraining.</span> JEST++ strongly surpasses prior art while requiring significantly fewer training iterations. COCO performance denotes the average of image-to-text and text-to-image retrieval.</figcaption>
</figure>
</div>
</div>
</section>
<section id="S4.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.6 </span>Simplifying data curation</h3>

<div id="S4.SS6.p1" class="ltx_para ltx_noindent">
<p id="S4.SS6.p1.1" class="ltx_p">The WebLI dataset <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>]</cite> used for our pretraining experiments has already been filtered for high image-text alignment. As shown in Table <a href="#S4.T3" title="Table 3 ‣ 4.6 Simplifying data curation ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, this offline curation is critical for strong IID training performance. For JEST++ however, this pre-filtering is redundant as performance does not degrade when training on the unfiltered WebLI, alleviating the need for foundation datasets.</p>
</div>
<div id="S4.SS6.p2" class="ltx_para ltx_noindent">
<table id="S4.SS6.p2.1" class="ltx_tabular ltx_minipage ltx_align_middle" style="width:216.8pt;">
<thead class="ltx_thead">
<tr id="S4.SS6.p2.1.1.1" class="ltx_tr">
<th id="S4.SS6.p2.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row">Method</th>
<th id="S4.SS6.p2.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column">Filtered Data?</th>
<th id="S4.SS6.p2.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">IN1K ZS</th>
<th id="S4.SS6.p2.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column">COCO</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.SS6.p2.1.2.1" class="ltx_tr">
<th id="S4.SS6.p2.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="2"><span id="S4.SS6.p2.1.2.1.1.1" class="ltx_text">IID</span></th>
<td id="S4.SS6.p2.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">✓</td>
<td id="S4.SS6.p2.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">73.6</td>
<td id="S4.SS6.p2.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t">52.5</td>
</tr>
<tr id="S4.SS6.p2.1.3.2" class="ltx_tr">
<td id="S4.SS6.p2.1.3.2.1" class="ltx_td ltx_align_center">✗</td>
<td id="S4.SS6.p2.1.3.2.2" class="ltx_td ltx_align_center" style="background-color:#F6BFBF;"><span id="S4.SS6.p2.1.3.2.2.1" class="ltx_text" style="background-color:#F6BFBF;">69.4</span></td>
<td id="S4.SS6.p2.1.3.2.3" class="ltx_td ltx_align_center" style="background-color:#F9D1D1;"><span id="S4.SS6.p2.1.3.2.3.1" class="ltx_text" style="background-color:#F9D1D1;">49.5</span></td>
</tr>
<tr id="S4.SS6.p2.1.4.3" class="ltx_tr">
<th id="S4.SS6.p2.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" rowspan="2"><span id="S4.SS6.p2.1.4.3.1.1" class="ltx_text">JEST++</span></th>
<td id="S4.SS6.p2.1.4.3.2" class="ltx_td ltx_align_center ltx_border_t">✓</td>
<td id="S4.SS6.p2.1.4.3.3" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#CDEBCD;"><span id="S4.SS6.p2.1.4.3.3.1" class="ltx_text" style="background-color:#CDEBCD;">76.9</span></td>
<td id="S4.SS6.p2.1.4.3.4" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#71C671;"><span id="S4.SS6.p2.1.4.3.4.1" class="ltx_text" style="background-color:#71C671;">61.8</span></td>
</tr>
<tr id="S4.SS6.p2.1.5.4" class="ltx_tr">
<td id="S4.SS6.p2.1.5.4.1" class="ltx_td ltx_align_center">✗</td>
<td id="S4.SS6.p2.1.5.4.2" class="ltx_td ltx_align_center" style="background-color:#D0ECD0;"><span id="S4.SS6.p2.1.5.4.2.1" class="ltx_text" style="background-color:#D0ECD0;">76.7</span></td>
<td id="S4.SS6.p2.1.5.4.3" class="ltx_td ltx_align_center" style="background-color:#71C671;"><span id="S4.SS6.p2.1.5.4.3.1" class="ltx_text" style="background-color:#71C671;">61.8</span></td>
</tr>
</tbody>
</table>
<div id="S4.SS6.p2.2" class="ltx_logical-block ltx_minipage ltx_align_middle" style="width:216.8pt;">
<figure id="S4.T3" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 3: </span><span id="S4.T3.2.1" class="ltx_text ltx_font_bold">Simplifying data curation.</span> All models are trained for 4B examples seen. Performance for JEST++ is nearly identical for pre-training on raw (uncurated) vs. filtered data. Color gradients measured relative to IID training on filtered WebLI.</figcaption>
</figure>
</div>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Discussion</h2>

<div id="S5.p1" class="ltx_para ltx_noindent">
<p id="S5.p1.2" class="ltx_p">We proposed a method—JEST—for jointly selecting the most learnable batches of data, which significantly accelerates large-scale multimodal learning, surpassing the previous state-of-the-art with up to 10<math id="S5.p1.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S5.p1.1.m1.1a"><mo id="S5.p1.1.m1.1.1" xref="S5.p1.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S5.p1.1.m1.1b"><times id="S5.p1.1.m1.1.1.cmml" xref="S5.p1.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S5.p1.1.m1.1c">\times</annotation></semantics></math> fewer FLOPs and 13<math id="S5.p1.2.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S5.p1.2.m2.1a"><mo id="S5.p1.2.m2.1.1" xref="S5.p1.2.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S5.p1.2.m2.1b"><times id="S5.p1.2.m2.1.1.cmml" xref="S5.p1.2.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S5.p1.2.m2.1c">\times</annotation></semantics></math> fewer examples. In particular, our experiments point to the strong potential for “data quality bootstrapping”, using small curated datasets to guide learning on much larger, uncurated ones.</p>
</div>
<div id="S5.p2" class="ltx_para ltx_noindent">
<p id="S5.p2.1" class="ltx_p">Recent work has shown that static dataset filtering, without knowledge of downstream training, can ultimately limit performance <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite>. Our results demonstrate that
useful batches, which must be constructed online, improve pretraining efficiency beyond individually selected examples. These findings therefore advocate for <span id="S5.p2.1.1" class="ltx_text ltx_font_italic">foundation distributions</span>—either through pre-scored datasets with <span id="S5.p2.1.2" class="ltx_text ltx_font_italic">easy-reference</span> JEST, or dynamically adjusted to the demands of the model with <span id="S5.p2.1.3" class="ltx_text ltx_font_italic">learnability</span> JEST—as a more general and effective replacement to generic foundation datasets.</p>
</div>
<div id="S5.p3" class="ltx_para ltx_noindent">
<p id="S5.p3.1" class="ltx_p"><span id="S5.p3.1.1" class="ltx_text ltx_font_bold">Limitations.</span> While our method has accelerated multimodal learning of canonical downstream tasks, it has relied on small, well-curated reference datasets which specify the distribution to prioritize within much larger uncurated data. We would therefore encourage future work exploring the inference of reference datasets from the set of downstream tasks of interest.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Abbas et al. [2023]</span>
<span class="ltx_bibblock">
Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos.

</span>
<span class="ltx_bibblock">Semdedup: Data-efficient learning at web-scale through semantic
deduplication.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2303.09540</em>, 2023.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Abbas et al. [2024]</span>
<span class="ltx_bibblock">
Amro Abbas, Evgenia Rusak, Kushal Tirumala, Wieland Brendel, Kamalika
Chaudhuri, and Ari S Morcos.

</span>
<span class="ltx_bibblock">Effective pruning of web-scale datasets based on complexity of
concept clusters.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2401.04578</em>, 2024.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Beyer et al. [2022]</span>
<span class="ltx_bibblock">
Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov.

</span>
<span class="ltx_bibblock">Big vision.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/google-research/big_vision" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/google-research/big_vision</a>, 2022.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Beyer et al. [2023]</span>
<span class="ltx_bibblock">
Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon
Kornblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim
Alabdulmohsin, and Filip Pavetic.

</span>
<span class="ltx_bibblock">Flexivit: One model for all patch sizes.

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition</em>, pages 14496–14506, 2023.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bucher et al. [2016]</span>
<span class="ltx_bibblock">
Maxime Bucher, Stéphane Herbin, and Frédéric Jurie.

</span>
<span class="ltx_bibblock">Hard negative mining for metric learning based zero-shot
classification.

</span>
<span class="ltx_bibblock">In <em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">Computer Vision–ECCV 2016 Workshops: Amsterdam, The
Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14</em>, pages
524–531. Springer, 2016.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Byeon et al. [2022]</span>
<span class="ltx_bibblock">
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and
Saehoon Kim.

</span>
<span class="ltx_bibblock">Coyo-700m: Image-text pair dataset.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">Coyo-700m: Image-text pair dataset</em>, 2022.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Campbell and Broderick [2018]</span>
<span class="ltx_bibblock">
Trevor Campbell and Tamara Broderick.

</span>
<span class="ltx_bibblock">Bayesian coreset construction via greedy iterative geodesic ascent.

</span>
<span class="ltx_bibblock">In <em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pages
698–706. PMLR, 2018.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cao et al. [2023]</span>
<span class="ltx_bibblock">
Liangliang Cao, Bowen Zhang, Chen Chen, Yinfei Yang, Xianzhi Du, Wencong Zhang,
Zhiyun Lu, and Yantao Zheng.

</span>
<span class="ltx_bibblock">Less is more: Removing text-regions improves clip training efficiency
and robustness.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.05095</em>, 2023.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Changpinyo et al. [2021]</span>
<span class="ltx_bibblock">
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.

</span>
<span class="ltx_bibblock">Conceptual 12m: Pushing web-scale image-text pre-training to
recognize long-tail visual concepts.

</span>
<span class="ltx_bibblock">In <em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition</em>, pages 3558–3568, 2021.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. [2022]</span>
<span class="ltx_bibblock">
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski,
Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer,
et al.

</span>
<span class="ltx_bibblock">Pali: A jointly-scaled multilingual language-image model.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2209.06794</em>, 2022.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. [2023]</span>
<span class="ltx_bibblock">
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul
Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr
Padlewski, et al.

</span>
<span class="ltx_bibblock">Pali-3 vision language models: Smaller, faster, stronger.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.09199</em>, 2023.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Coleman et al. [2019]</span>
<span class="ltx_bibblock">
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter
Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia.

</span>
<span class="ltx_bibblock">Selection via proxy: Efficient data selection for deep learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1906.11829</em>, 2019.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dehghani et al. [2024]</span>
<span class="ltx_bibblock">
Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias
Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos,
Ibrahim M Alabdulmohsin, et al.

</span>
<span class="ltx_bibblock">Patch n’pack: Navit, a vision transformer for any aspect ratio and
resolution.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dosovitskiy et al. [2021]</span>
<span class="ltx_bibblock">
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, et al.

</span>
<span class="ltx_bibblock">An image is worth 16x16 words: Transformers for image recognition at
scale.

</span>
<span class="ltx_bibblock">In <em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2021.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Evans et al. [2023]</span>
<span class="ltx_bibblock">
Talfan Evans, Shreya Pathak, Hamza Merzic, Jonathan Schwarz, Ryutaro Tanno, and
Olivier J Henaff.

</span>
<span class="ltx_bibblock">Bad students make great teachers: Active learning accelerates
large-scale visual understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2312.05328</em>, 2023.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fang et al. [2023]</span>
<span class="ltx_bibblock">
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev,
and Vaishaal Shankar.

</span>
<span class="ltx_bibblock">Data filtering networks.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2309.17425</em>, 2023.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gadre et al. [2023]</span>
<span class="ltx_bibblock">
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios
Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu
Zhang, et al.

</span>
<span class="ltx_bibblock">Datacomp: In search of the next generation of multimodal datasets.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2304.14108</em>, 2023.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Goyal et al. [2024]</span>
<span class="ltx_bibblock">
Sachin Goyal, Pratyush Maini, Zachary C Lipton, Aditi Raghunathan, and J Zico
Kolter.

</span>
<span class="ltx_bibblock">Scaling laws for data filtering–data curation cannot be compute
agnostic.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2404.07177</em>, 2024.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gunasekar et al. [2023]</span>
<span class="ltx_bibblock">
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie
Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo
de Rosa, Olli Saarikivi, et al.

</span>
<span class="ltx_bibblock">Textbooks are all you need.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2306.11644</em>, 2023.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Har-Peled and Mazumdar [2004]</span>
<span class="ltx_bibblock">
Sariel Har-Peled and Soham Mazumdar.

</span>
<span class="ltx_bibblock">On coresets for k-means and k-median clustering.

</span>
<span class="ltx_bibblock">In <em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Proceedings of the thirty-sixth annual ACM symposium on
Theory of computing</em>, pages 291–300, 2004.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Harwood et al. [2017]</span>
<span class="ltx_bibblock">
Ben Harwood, Vijay Kumar BG, Gustavo Carneiro, Ian Reid, and Tom Drummond.

</span>
<span class="ltx_bibblock">Smart mining for deep metric learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE international conference on computer
vision</em>, pages 2821–2829, 2017.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hessel et al. [2021]</span>
<span class="ltx_bibblock">
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.

</span>
<span class="ltx_bibblock">Clipscore: A reference-free evaluation metric for image captioning.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2104.08718</em>, 2021.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ilharco et al. [2021]</span>
<span class="ltx_bibblock">
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas
Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John
Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.

</span>
<span class="ltx_bibblock">Openclip, July 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://doi.org/10.5281/zenodo.5143773" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.5281/zenodo.5143773</a>.

</span>
<span class="ltx_bibblock">If you use this software, please cite it as below.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jia et al. [2021]</span>
<span class="ltx_bibblock">
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le,
Yun-Hsuan Sung, Zhen Li, and Tom Duerig.

</span>
<span class="ltx_bibblock">Scaling up visual and vision-language representation learning with
noisy text supervision.

</span>
<span class="ltx_bibblock">In <em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">International conference on machine learning</em>, pages
4904–4916. PMLR, 2021.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jouppi et al. [2017]</span>
<span class="ltx_bibblock">
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal,
Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al.

</span>
<span class="ltx_bibblock">In-datacenter performance analysis of a tensor processing unit.

</span>
<span class="ltx_bibblock">In <em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 44th annual international symposium on
computer architecture</em>, pages 1–12, 2017.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kudo and Richardson [2018]</span>
<span class="ltx_bibblock">
Taku Kudo and John Richardson.

</span>
<span class="ltx_bibblock">Sentencepiece: A simple and language independent subword tokenizer
and detokenizer for neural text processing.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1808.06226</em>, 2018.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. [2022]</span>
<span class="ltx_bibblock">
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.

</span>
<span class="ltx_bibblock">Blip: Bootstrapping language-image pre-training for unified
vision-language understanding and generation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">International conference on machine learning</em>, pages
12888–12900. PMLR, 2022.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. [2024]</span>
<span class="ltx_bibblock">
Xianhang Li, Zeyu Wang, and Cihang Xie.

</span>
<span class="ltx_bibblock">An inverse scaling law for clip training.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. [2023]</span>
<span class="ltx_bibblock">
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He.

</span>
<span class="ltx_bibblock">Scaling language-image pre-training via masking, 2023.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin et al. [2024]</span>
<span class="ltx_bibblock">
Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen
Lin, Yujiu Yang, Jian Jiao, Nan Duan, et al.

</span>
<span class="ltx_bibblock">Rho-1: Not all tokens are what you need.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2404.07965</em>, 2024.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Loshchilov and Hutter [2015]</span>
<span class="ltx_bibblock">
Ilya Loshchilov and Frank Hutter.

</span>
<span class="ltx_bibblock">Online batch selection for faster training of neural networks.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1511.06343</em>, 2015.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mahmoud et al. [2023]</span>
<span class="ltx_bibblock">
Anas Mahmoud, Mostafa Elhoushi, Amro Abbas, Yu Yang, Newsha Ardalani, Hugh
Leather, and Ari Morcos.

</span>
<span class="ltx_bibblock">Sieve: Multimodal dataset pruning using image captioning models.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.02110</em>, 2023.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mindermann et al. [2022]</span>
<span class="ltx_bibblock">
Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas
Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot,
Sebastian Farquhar, et al.

</span>
<span class="ltx_bibblock">Prioritized training on points that are learnable, worth learning,
and not yet learnt.

</span>
<span class="ltx_bibblock">In <em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pages
15630–15649. PMLR, 2022.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mishchuk et al. [2017]</span>
<span class="ltx_bibblock">
Anastasiia Mishchuk, Dmytro Mishkin, Filip Radenovic, and Jiri Matas.

</span>
<span class="ltx_bibblock">Working hard to know your neighbor’s margins: Local descriptor
learning loss.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 30, 2017.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Naeem et al. [2023]</span>
<span class="ltx_bibblock">
Muhammad Ferjad Naeem, Yongqin Xian, Xiaohua Zhai, Lukas Hoyer, Luc Van Gool,
and Federico Tombari.

</span>
<span class="ltx_bibblock">Silc: Improving vision language pretraining with self-distillation,
2023.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Paul et al. [2021]</span>
<span class="ltx_bibblock">
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite.

</span>
<span class="ltx_bibblock">Deep learning on a data diet: Finding important examples early in
training.

</span>
<span class="ltx_bibblock"><em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
34:20596–20607, 2021.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radenovic et al. [2023]</span>
<span class="ltx_bibblock">
Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon
Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, and Dhruv Mahajan.

</span>
<span class="ltx_bibblock">Filtering, distillation, and hard negatives for vision-language
pre-training.

</span>
<span class="ltx_bibblock">In <em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition</em>, pages 6967–6977, 2023.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. [2021]</span>
<span class="ltx_bibblock">
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et al.

</span>
<span class="ltx_bibblock">Learning transferable visual models from natural language
supervision.

</span>
<span class="ltx_bibblock">In <em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, 2021.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raffel et al. [2020]</span>
<span class="ltx_bibblock">
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael
Matena, Yanqi Zhou, Wei Li, and Peter J Liu.

</span>
<span class="ltx_bibblock">Exploring the limits of transfer learning with a unified text-to-text
transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">Journal of machine learning research</em>, 21(140):1–67, 2020.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raposo et al. [2024]</span>
<span class="ltx_bibblock">
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway
Humphreys, and Adam Santoro.

</span>
<span class="ltx_bibblock">Mixture-of-depths: Dynamically allocating compute in
transformer-based language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2404.02258</em>, 2024.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Robinson et al. [2020]</span>
<span class="ltx_bibblock">
Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka.

</span>
<span class="ltx_bibblock">Contrastive learning with hard negative samples.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2010.04592</em>, 2020.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sachdeva et al. [2024]</span>
<span class="ltx_bibblock">
Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong,
Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng.

</span>
<span class="ltx_bibblock">How to train data-efficient llms.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.09668</em>, 2024.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schaul et al. [2015]</span>
<span class="ltx_bibblock">
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver.

</span>
<span class="ltx_bibblock">Prioritized experience replay.

</span>
<span class="ltx_bibblock"><em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1511.05952</em>, 2015.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schuhmann et al. [2022]</span>
<span class="ltx_bibblock">
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross
Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell
Wortsman, et al.

</span>
<span class="ltx_bibblock">Laion-5b: An open large-scale dataset for training next generation
image-text models.

</span>
<span class="ltx_bibblock"><em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
35:25278–25294, 2022.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Simo-Serra et al. [2015]</span>
<span class="ltx_bibblock">
Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, Iasonas Kokkinos, Pascal Fua, and
Francesc Moreno-Noguer.

</span>
<span class="ltx_bibblock">Discriminative learning of deep convolutional feature point
descriptors.

</span>
<span class="ltx_bibblock">In <em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE international conference on computer
vision</em>, pages 118–126, 2015.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Singer et al. [2022]</span>
<span class="ltx_bibblock">
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan
Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.

</span>
<span class="ltx_bibblock">Make-a-video: Text-to-video generation without text-video data.

</span>
<span class="ltx_bibblock"><em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2209.14792</em>, 2022.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sorscher et al. [2022]</span>
<span class="ltx_bibblock">
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos.

</span>
<span class="ltx_bibblock">Beyond neural scaling laws: beating power law scaling via data
pruning.

</span>
<span class="ltx_bibblock"><em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
35:19523–19536, 2022.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sun et al. [2023]</span>
<span class="ltx_bibblock">
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao.

</span>
<span class="ltx_bibblock">Eva-clip: Improved training techniques for clip at scale.

</span>
<span class="ltx_bibblock"><em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2303.15389</em>, 2023.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tian et al. [2021]</span>
<span class="ltx_bibblock">
Yonglong Tian, Olivier J Henaff, and Aäron van den Oord.

</span>
<span class="ltx_bibblock">Divide and contrast: Self-supervised learning from uncurated data.

</span>
<span class="ltx_bibblock">In <em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF International Conference on
Computer Vision</em>, 2021.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wu et al. [2017]</span>
<span class="ltx_bibblock">
Chao-Yuan Wu, R Manmatha, Alexander J Smola, and Philipp Krahenbuhl.

</span>
<span class="ltx_bibblock">Sampling matters in deep embedding learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE international conference on computer
vision</em>, pages 2840–2848, 2017.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xie et al. [2023]</span>
<span class="ltx_bibblock">
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy
Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu.

</span>
<span class="ltx_bibblock">Doremi: Optimizing data mixtures speeds up language model
pretraining, 2023.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. [2023]</span>
<span class="ltx_bibblock">
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu
Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph
Feichtenhofer.

</span>
<span class="ltx_bibblock">Demystifying clip data.

</span>
<span class="ltx_bibblock"><em id="bib.bib52.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2309.16671</em>, 2023.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xuan et al. [2020]</span>
<span class="ltx_bibblock">
Hong Xuan, Abby Stylianou, Xiaotong Liu, and Robert Pless.

</span>
<span class="ltx_bibblock">Hard negative examples are hard, but useful.

</span>
<span class="ltx_bibblock">In <em id="bib.bib53.1.1" class="ltx_emph ltx_font_italic">Computer Vision–ECCV 2020: 16th European Conference,
Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16</em>, pages 126–142.
Springer, 2020.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhai et al. [2023]</span>
<span class="ltx_bibblock">
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.

</span>
<span class="ltx_bibblock">Sigmoid loss for language image pre-training.

</span>
<span class="ltx_bibblock"><em id="bib.bib54.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2303.15343</em>, 2023.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang and He [2020]</span>
<span class="ltx_bibblock">
Minjia Zhang and Yuxiong He.

</span>
<span class="ltx_bibblock">Accelerating training of transformer-based language models with
progressive layer dropping.

</span>
<span class="ltx_bibblock"><em id="bib.bib55.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
33:14011–14023, 2020.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Appendix</h2>

<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>Training configuration</h3>

<div id="A1.SS1.p1" class="ltx_para ltx_noindent">
<p id="A1.SS1.p1.1" class="ltx_p">Our default training configuration follows that of SigLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>, with a ViT-B/16 and Bert-B image-text dual encoder, training on WebLI for 3 billion examples with a batch size of 32k and the sigmoid-contrastive loss. The vision encoder takes images resized to (256 x 256) and the text-encoder tokenizes text with the sentencepiece tokenizer <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib26" title="" class="ltx_ref">26</a>]</cite> trained on the English C4 dataset <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib39" title="" class="ltx_ref">39</a>]</cite>. We crop the text to the first 64 tokens. The initial learning rate is 0.001, warmed up linearly during the first 1% of training, followed by cosine decay. We use a weight decay 0.0001, gradient clipping to a maximum norm of 1.0, and the Adam optimizer with <math id="A1.SS1.p1.1.m1.1" class="ltx_Math" alttext="\beta_{2}=0.95" display="inline"><semantics id="A1.SS1.p1.1.m1.1a"><mrow id="A1.SS1.p1.1.m1.1.1" xref="A1.SS1.p1.1.m1.1.1.cmml"><msub id="A1.SS1.p1.1.m1.1.1.2" xref="A1.SS1.p1.1.m1.1.1.2.cmml"><mi id="A1.SS1.p1.1.m1.1.1.2.2" xref="A1.SS1.p1.1.m1.1.1.2.2.cmml">β</mi><mn id="A1.SS1.p1.1.m1.1.1.2.3" xref="A1.SS1.p1.1.m1.1.1.2.3.cmml">2</mn></msub><mo id="A1.SS1.p1.1.m1.1.1.1" xref="A1.SS1.p1.1.m1.1.1.1.cmml">=</mo><mn id="A1.SS1.p1.1.m1.1.1.3" xref="A1.SS1.p1.1.m1.1.1.3.cmml">0.95</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS1.p1.1.m1.1b"><apply id="A1.SS1.p1.1.m1.1.1.cmml" xref="A1.SS1.p1.1.m1.1.1"><eq id="A1.SS1.p1.1.m1.1.1.1.cmml" xref="A1.SS1.p1.1.m1.1.1.1"></eq><apply id="A1.SS1.p1.1.m1.1.1.2.cmml" xref="A1.SS1.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="A1.SS1.p1.1.m1.1.1.2.1.cmml" xref="A1.SS1.p1.1.m1.1.1.2">subscript</csymbol><ci id="A1.SS1.p1.1.m1.1.1.2.2.cmml" xref="A1.SS1.p1.1.m1.1.1.2.2">𝛽</ci><cn type="integer" id="A1.SS1.p1.1.m1.1.1.2.3.cmml" xref="A1.SS1.p1.1.m1.1.1.2.3">2</cn></apply><cn type="float" id="A1.SS1.p1.1.m1.1.1.3.cmml" xref="A1.SS1.p1.1.m1.1.1.3">0.95</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS1.p1.1.m1.1c">\beta_{2}=0.95</annotation></semantics></math>. We split training across 256 TPUv5e chips.</p>
</div>
<div id="A1.SS1.p2" class="ltx_para ltx_noindent">
<p id="A1.SS1.p2.1" class="ltx_p">For the LAION experiments in Table. <a href="#S4.T2" title="Table 2 ‣ 4.5 Comparison to prior art ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, we use an architecture matched to the prior art (ViT-B/32 vision encoder and resized image inputs to (224 x 224)). Otherwise, we use the same training settings as above. We note that this batch size is comparable to that used in the prior art and we find similar results using the softmax-contrastive loss instead of the sigmoid-contrastive loss.</p>
</div>
<figure id="algorithm2" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="algorithm2.2.1.1" class="ltx_text ltx_font_bold">Algorithm 2</span> </span> Pseudocode for JEST / Flexi-JEST</figcaption>
<div id="algorithm2.3" class="ltx_listing ltx_lst_language_python ltx_lstlisting ltx_listing" style="background-color:#F5F5FA;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,CmNmZyA9IENvbmZpZ0RpY3QoCiAgICBuX2NodW5rcz0xNiwKICAgIGZpbHRlcl9yYXRpbz0wLjgsCiAgICBtZXRob2Q9ImxlYXJuYWJpbGl0eSIsCiAgICBtZXRob2Q9Implc3QiLCAgIyBvciAiZmxleGktamVzdCIKICAgIHNvZnRtYXhfc2NvcmVfZ2Fpbj0xMDAuMCwKICAgIGxvc3NfdHlwZSA9ICJzaWdtb2lkIiwKKQoKZGVmIHNpZ21vaWRfbmxsKHBhcmFtcywgZW1iZWRzKToKICB6aW1nLCB6dHh0ID0gZW1iZWRzCiAgbG9naXRzID0gbnAuZG90KHppbWcsIHp0eHQuVCkgICMgW0IsIEJdCiAgbG9naXRzID0gbG9naXRzICogcGFyYW1zWyJhbHBoYSJdICsgcGFyYW1zWyJiZXRhIl0KICBleWUgPSBucC5leWUoemltZy5zaGFwZVswXSkKICBtMV9kaWFnMSA9IC1ucC5vbmVzX2xpa2UobG9naXRzKSArIDIgKiBleWUKCiAgbmxsX21hdCA9IC1sb2dfc2lnbW9pZChtMV9kaWFnMSAqIGxvZ2l0cykKICBubGwgPSBucC5zdW0obmxsX21hdCwgYXhpcz0tMSkubWVhbigpCgogIHJldHVybiBubGwsIG5sbF9tYXQgICMgWyxdLCBbQiwgQl0KCmRlZiBnZXRfc2NvcmVzX3NpZ21vaWQoZW1iZWRzLCBlbWJlZHNfcmVmLCBwYXJhbXMsIHBhcmFtc19yZWYpOgogIF8sIG5sbF9tb2QgPSBzaWdtb2lkX25sbChwYXJhbXMsIGVtYmVkcykgICMgW0IsIEJdCiAgXywgbmxsX3JlZiA9IHNpZ21vaWRfbmxsKHBhcmFtc19yZWYsIGVtYmVkc19yZWYpICAjIFtCLCBCXQogIGlmIGNmZy5zY29yaW5nID09ICJsZWFybmFiaWxpdHkiOgogICAgc2NvcmVzID0gbmxsX21vZCAtIG5sbF9yZWYKICBlbGlmIGNmZy5zY29yaW5nID09ICJlYXN5X3JlZiI6CiAgICBzY29yZXMgPSAtIG5sbF9yZWYKICByZXR1cm4gc2NvcmVzICogY2ZnLnNvZnRtYXhfc2NvcmVfZ2FpbgoKZGVmIGxvc3NfZm4ocGFyYW1zLCBwYXJhbXNfcmVmLCBiYXRjaCk6CiAgaW1hZ2VzLCB0ZXh0cyA9IGJhdGNoCiAgYXBwcm94ID0gVHJ1ZSBpZiBjZmcubWV0aG9kID09ICJmbGV4aS1qZXN0IiBlbHNlIEZhbHNlCgogICMgU2NvcmUgYW5kIHN1Yi1zYW1wbGUgdGhlIGluaXRpYWwgc3VwZXItYmF0Y2gKICBlbWJlZHMgPSBtb2RlbC5mb3J3YXJkKGltYWdlcywgdGV4dHMsIHBhcmFtcywgYXBwcm94PWFwcHJveCkgICMgWzVCLCBEXQogIGVtYmVkc19yZWYgPSBiYXRjaFsiZW1iZWRzX3JlZiJdICAjIFByZS1jYWNoZWQgaW4gZGF0YXNldAogIGlmIGNmZy5sb3NzX3R5cGUgPT0gInNpZ21vaWQiOgogICAgICBzY29yZXMgPSBnZXRfc2NvcmVzX3NpZ21vaWQoZW1iZWRzLCBlbWJlZHNfcmVmLCBwYXJhbXMsIHBhcmFtc19yZWYpICAjIEdldCBzY29yZXMKICAgICAgaW5kcyA9IGpvaW50bHlfc2FtcGxlX2JhdGNoKHNjb3JlcywgY2ZnLm5fY2h1bmtzLCBjZmcuZmlsdGVyX3JhdGlvLCBjZmcubGVhcm5hYmlsaXR5KQogIGVsaWYgY2ZnLmxvc3NfdHlwZSA9PSAic29mdG1heCI6CiAgICAgIGluZHMgPSBqb2ludGx5X3NhbXBsZV9iYXRjaF9zb2Z0bWF4KGVtYmVkc19yZWYsIGVtYmVkcywgcGFyYW1zX3JlZiwgcGFyYW1zLCBjZmcubl9jaHVua3MsIGNmZy5maWx0ZXJfcmF0aW8pICMgZm9yIHNvZnRtYXggbG9zcywgc2NvcmVzIGFyZSByZS1jb21wdXRlZCBpbiB0aGUgaXRlcmF0aXZlIHNhbXBsaW5nLgoKICBpbWFnZXMsIHRleHRzID0gc3RvcF9ncmFkKGltYWdlc1tpbmRzXSksIHN0b3BfZ3JhZCh0ZXh0c1tpbmRzXSkgICMgW0IsIC4uLl0KCiAgIyBTcGxpdCBiYXRjaCBmb3IgY28tdHJhaW5pbmcKICBpbWFnZXNfZnVsbCwgaW1hZ2VzX2FwcHJveCA9IGltYWdlc1s6OjJdLCBpbWFnZXNbMTo6Ml0gICMgW0IvMiwgLi4uXSwgW0IvMiwgLi4uXQogIHRleHRzX2Z1bGwsIHRleHRzX2FwcHJveCA9IHRleHRzWzo6Ml0sIHRleHRzWzE6OjJdICAjIFtCLzIsIC4uLl0sIFtCLzIsIC4uLl0KCiAgIyBDb21wdXRlIG92ZXJhbGwgbG9zcwogIGVtYmVkc19mdWxsID0gbW9kZWwuZm9yd2FyZChpbWFnZXNfZnVsbCwgdGV4dHNfZnVsbCwgcGFyYW1zLCBhcHByb3g9RmFsc2UpICAjIFtCLzIsIERdLCBbQi8yLCBEXQogIGVtYmVkc19hcHByb3ggPSBtb2RlbC5mb3J3YXJkKGltYWdlc19hcHByb3gsIHRleHRzX2FwcHJveCwgcGFyYW1zLCBhcHByb3g9YXBwcm94KSAgIyBbQi8yLCBEXSwgW0IvMiwgRF0KICB6aW1nID0gbnAuY29uY2F0ZW5hdGUoW2VtYmVkc19mdWxsWzBdLCBlbWJlZHNfYXBwcm94WzBdXSwgYXhpcz0wKQogIHp0eHQgPSBucC5jb25jYXRlbmF0ZShbZW1iZWRzX2Z1bGxbMV0sIGVtYmVkc19hcHByb3hbMV1dLCBheGlzPTApCiAgaWYgbG9zc190eXBlID09ICJzaWdtb2lkIjoKICAgICAgbG9zcywgXyA9IHNpZ21vaWRfbmxsKHBhcmFtcywgKHppbWcsIHp0eHQpKQogIGVsaWYgbG9zc190eXBlID09ICJzb2Z0bWF4IjoKICAgICAgbG9zcywgXywgXyA9IHNvZnRtYXhfbmxsKHBhcmFtcywgKHppbWcsIHp0eHQpLCBpc19zYW1wbGVkPU5vbmUpCgogIHJldHVybiBsb3NzCg==" download="">⬇</a></div>
<div id="lstnumberx17" class="ltx_listingline">
</div>
<div id="lstnumberx18" class="ltx_listingline">
<span id="lstnumberx18.1" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cfg</span><span id="lstnumberx18.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.3" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx18.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx18.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ConfigDict</span><span id="lstnumberx18.6" class="ltx_text ltx_font_typewriter">(</span>
</div>
<div id="lstnumberx19" class="ltx_listingline">
<span id="lstnumberx19.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx19.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_chunks</span><span id="lstnumberx19.3" class="ltx_text ltx_font_typewriter">=16,</span>
</div>
<div id="lstnumberx20" class="ltx_listingline">
<span id="lstnumberx20.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx20.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">filter_ratio</span><span id="lstnumberx20.3" class="ltx_text ltx_font_typewriter">=0.8,</span>
</div>
<div id="lstnumberx21" class="ltx_listingline">
<span id="lstnumberx21.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx21.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">method</span><span id="lstnumberx21.3" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx21.4" class="ltx_text ltx_lst_string ltx_font_typewriter">"learnability"</span><span id="lstnumberx21.5" class="ltx_text ltx_font_typewriter">,</span>
</div>
<div id="lstnumberx22" class="ltx_listingline">
<span id="lstnumberx22.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx22.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">method</span><span id="lstnumberx22.3" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx22.4" class="ltx_text ltx_lst_string ltx_font_typewriter">"jest"</span><span id="lstnumberx22.5" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx22.6" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx22.7" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx22.7.1" class="ltx_text ltx_lst_space"> </span>or<span id="lstnumberx22.7.2" class="ltx_text ltx_lst_space"> </span>"flexi-jest"</span>
</div>
<div id="lstnumberx23" class="ltx_listingline">
<span id="lstnumberx23.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx23.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">softmax_score_gain</span><span id="lstnumberx23.3" class="ltx_text ltx_font_typewriter">=100.0,</span>
</div>
<div id="lstnumberx24" class="ltx_listingline">
<span id="lstnumberx24.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx24.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">loss_type</span><span id="lstnumberx24.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx24.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx24.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx24.6" class="ltx_text ltx_lst_string ltx_font_typewriter">"sigmoid"</span><span id="lstnumberx24.7" class="ltx_text ltx_font_typewriter">,</span>
</div>
<div id="lstnumberx25" class="ltx_listingline">
<span id="lstnumberx25.1" class="ltx_text ltx_font_typewriter">)</span>
</div>
<div id="lstnumberx26" class="ltx_listingline">
</div>
<div id="lstnumberx27" class="ltx_listingline">
<span id="lstnumberx27.1" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">def</span><span id="lstnumberx27.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx27.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">sigmoid_nll</span><span id="lstnumberx27.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx27.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params</span><span id="lstnumberx27.6" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx27.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx27.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds</span><span id="lstnumberx27.9" class="ltx_text ltx_font_typewriter">):</span>
</div>
<div id="lstnumberx28" class="ltx_listingline">
<span id="lstnumberx28.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx28.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">zimg</span><span id="lstnumberx28.3" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx28.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx28.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ztxt</span><span id="lstnumberx28.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx28.7" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx28.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx28.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds</span>
</div>
<div id="lstnumberx29" class="ltx_listingline">
<span id="lstnumberx29.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx29.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits</span><span id="lstnumberx29.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx29.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx29.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx29.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">np</span><span id="lstnumberx29.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx29.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">dot</span><span id="lstnumberx29.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx29.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">zimg</span><span id="lstnumberx29.11" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx29.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx29.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ztxt</span><span id="lstnumberx29.14" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx29.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">T</span><span id="lstnumberx29.16" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx29.17" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx29.18" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx29.18.1" class="ltx_text ltx_lst_space"> </span>[B,<span id="lstnumberx29.18.2" class="ltx_text ltx_lst_space"> </span>B]</span>
</div>
<div id="lstnumberx30" class="ltx_listingline">
<span id="lstnumberx30.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx30.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits</span><span id="lstnumberx30.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx30.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx30.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx30.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits</span><span id="lstnumberx30.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx30.8" class="ltx_text ltx_font_typewriter">*</span><span id="lstnumberx30.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx30.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params</span><span id="lstnumberx30.11" class="ltx_text ltx_font_typewriter">[</span><span id="lstnumberx30.12" class="ltx_text ltx_lst_string ltx_font_typewriter">"alpha"</span><span id="lstnumberx30.13" class="ltx_text ltx_font_typewriter">]</span><span id="lstnumberx30.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx30.15" class="ltx_text ltx_font_typewriter">+</span><span id="lstnumberx30.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx30.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params</span><span id="lstnumberx30.18" class="ltx_text ltx_font_typewriter">[</span><span id="lstnumberx30.19" class="ltx_text ltx_lst_string ltx_font_typewriter">"beta"</span><span id="lstnumberx30.20" class="ltx_text ltx_font_typewriter">]</span>
</div>
<div id="lstnumberx31" class="ltx_listingline">
<span id="lstnumberx31.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx31.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">eye</span><span id="lstnumberx31.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx31.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx31.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx31.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">np</span><span id="lstnumberx31.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx31.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">eye</span><span id="lstnumberx31.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx31.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">zimg</span><span id="lstnumberx31.11" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx31.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">shape</span><span id="lstnumberx31.13" class="ltx_text ltx_font_typewriter">[0])</span>
</div>
<div id="lstnumberx32" class="ltx_listingline">
<span id="lstnumberx32.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx32.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">m1_diag1</span><span id="lstnumberx32.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx32.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx32.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx32.6" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx32.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">np</span><span id="lstnumberx32.8" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx32.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ones_like</span><span id="lstnumberx32.10" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx32.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits</span><span id="lstnumberx32.12" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx32.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx32.14" class="ltx_text ltx_font_typewriter">+</span><span id="lstnumberx32.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx32.16" class="ltx_text ltx_font_typewriter">2</span><span id="lstnumberx32.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx32.18" class="ltx_text ltx_font_typewriter">*</span><span id="lstnumberx32.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx32.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">eye</span>
</div>
<div id="lstnumberx33" class="ltx_listingline">
</div>
<div id="lstnumberx34" class="ltx_listingline">
<span id="lstnumberx34.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx34.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nll_mat</span><span id="lstnumberx34.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx34.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx34.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx34.6" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx34.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">log_sigmoid</span><span id="lstnumberx34.8" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx34.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">m1_diag1</span><span id="lstnumberx34.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx34.11" class="ltx_text ltx_font_typewriter">*</span><span id="lstnumberx34.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx34.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits</span><span id="lstnumberx34.14" class="ltx_text ltx_font_typewriter">)</span>
</div>
<div id="lstnumberx35" class="ltx_listingline">
<span id="lstnumberx35.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx35.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nll</span><span id="lstnumberx35.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx35.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx35.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx35.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">np</span><span id="lstnumberx35.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx35.8" class="ltx_text ltx_lst_keyword ltx_lst_keywords2 ltx_font_typewriter" style="color:#408080;">sum</span><span id="lstnumberx35.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx35.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nll_mat</span><span id="lstnumberx35.11" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx35.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx35.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">axis</span><span id="lstnumberx35.14" class="ltx_text ltx_font_typewriter">=-1).</span><span id="lstnumberx35.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">mean</span><span id="lstnumberx35.16" class="ltx_text ltx_font_typewriter">()</span>
</div>
<div id="lstnumberx36" class="ltx_listingline">
</div>
<div id="lstnumberx37" class="ltx_listingline">
<span id="lstnumberx37.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx37.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">return</span><span id="lstnumberx37.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx37.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nll</span><span id="lstnumberx37.5" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx37.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx37.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nll_mat</span><span id="lstnumberx37.8" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx37.9" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx37.9.1" class="ltx_text ltx_lst_space"> </span>[,],<span id="lstnumberx37.9.2" class="ltx_text ltx_lst_space"> </span>[B,<span id="lstnumberx37.9.3" class="ltx_text ltx_lst_space"> </span>B]</span>
</div>
<div id="lstnumberx38" class="ltx_listingline">
</div>
<div id="lstnumberx39" class="ltx_listingline">
<span id="lstnumberx39.1" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">def</span><span id="lstnumberx39.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx39.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">get_scores_sigmoid</span><span id="lstnumberx39.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx39.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds</span><span id="lstnumberx39.6" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx39.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx39.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds_ref</span><span id="lstnumberx39.9" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx39.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx39.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params</span><span id="lstnumberx39.12" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx39.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx39.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params_ref</span><span id="lstnumberx39.15" class="ltx_text ltx_font_typewriter">):</span>
</div>
<div id="lstnumberx40" class="ltx_listingline">
<span id="lstnumberx40.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx40.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">_</span><span id="lstnumberx40.3" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx40.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx40.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nll_mod</span><span id="lstnumberx40.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx40.7" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx40.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx40.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">sigmoid_nll</span><span id="lstnumberx40.10" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx40.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params</span><span id="lstnumberx40.12" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx40.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx40.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds</span><span id="lstnumberx40.15" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx40.16" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx40.17" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx40.17.1" class="ltx_text ltx_lst_space"> </span>[B,<span id="lstnumberx40.17.2" class="ltx_text ltx_lst_space"> </span>B]</span>
</div>
<div id="lstnumberx41" class="ltx_listingline">
<span id="lstnumberx41.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx41.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">_</span><span id="lstnumberx41.3" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx41.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx41.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nll_ref</span><span id="lstnumberx41.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx41.7" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx41.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx41.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">sigmoid_nll</span><span id="lstnumberx41.10" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx41.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params_ref</span><span id="lstnumberx41.12" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx41.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx41.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds_ref</span><span id="lstnumberx41.15" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx41.16" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx41.17" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx41.17.1" class="ltx_text ltx_lst_space"> </span>[B,<span id="lstnumberx41.17.2" class="ltx_text ltx_lst_space"> </span>B]</span>
</div>
<div id="lstnumberx42" class="ltx_listingline">
<span id="lstnumberx42.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx42.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">if</span><span id="lstnumberx42.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cfg</span><span id="lstnumberx42.5" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx42.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">scoring</span><span id="lstnumberx42.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.8" class="ltx_text ltx_font_typewriter">==</span><span id="lstnumberx42.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx42.10" class="ltx_text ltx_lst_string ltx_font_typewriter">"learnability"</span><span id="lstnumberx42.11" class="ltx_text ltx_font_typewriter">:</span>
</div>
<div id="lstnumberx43" class="ltx_listingline">
<span id="lstnumberx43.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx43.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">scores</span><span id="lstnumberx43.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx43.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx43.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx43.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nll_mod</span><span id="lstnumberx43.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx43.8" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx43.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx43.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nll_ref</span>
</div>
<div id="lstnumberx44" class="ltx_listingline">
<span id="lstnumberx44.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx44.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">elif</span><span id="lstnumberx44.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx44.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cfg</span><span id="lstnumberx44.5" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx44.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">scoring</span><span id="lstnumberx44.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx44.8" class="ltx_text ltx_font_typewriter">==</span><span id="lstnumberx44.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx44.10" class="ltx_text ltx_lst_string ltx_font_typewriter">"easy_ref"</span><span id="lstnumberx44.11" class="ltx_text ltx_font_typewriter">:</span>
</div>
<div id="lstnumberx45" class="ltx_listingline">
<span id="lstnumberx45.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx45.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">scores</span><span id="lstnumberx45.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx45.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx45.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx45.6" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx45.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx45.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nll_ref</span>
</div>
<div id="lstnumberx46" class="ltx_listingline">
<span id="lstnumberx46.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx46.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">return</span><span id="lstnumberx46.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx46.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">scores</span><span id="lstnumberx46.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx46.6" class="ltx_text ltx_font_typewriter">*</span><span id="lstnumberx46.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx46.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cfg</span><span id="lstnumberx46.9" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx46.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">softmax_score_gain</span>
</div>
<div id="lstnumberx47" class="ltx_listingline">
</div>
<div id="lstnumberx48" class="ltx_listingline">
<span id="lstnumberx48.1" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">def</span><span id="lstnumberx48.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx48.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">loss_fn</span><span id="lstnumberx48.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx48.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params</span><span id="lstnumberx48.6" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx48.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx48.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params_ref</span><span id="lstnumberx48.9" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx48.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx48.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">batch</span><span id="lstnumberx48.12" class="ltx_text ltx_font_typewriter">):</span>
</div>
<div id="lstnumberx49" class="ltx_listingline">
<span id="lstnumberx49.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx49.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">images</span><span id="lstnumberx49.3" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx49.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx49.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">texts</span><span id="lstnumberx49.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx49.7" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx49.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx49.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">batch</span>
</div>
<div id="lstnumberx50" class="ltx_listingline">
<span id="lstnumberx50.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx50.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">approx</span><span id="lstnumberx50.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx50.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx50.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx50.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">True</span><span id="lstnumberx50.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx50.8" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">if</span><span id="lstnumberx50.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx50.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cfg</span><span id="lstnumberx50.11" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx50.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">method</span><span id="lstnumberx50.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx50.14" class="ltx_text ltx_font_typewriter">==</span><span id="lstnumberx50.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx50.16" class="ltx_text ltx_lst_string ltx_font_typewriter">"flexi-jest"</span><span id="lstnumberx50.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx50.18" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">else</span><span id="lstnumberx50.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx50.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">False</span>
</div>
<div id="lstnumberx51" class="ltx_listingline">
</div>
<div id="lstnumberx52" class="ltx_listingline">
<span id="lstnumberx52.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx52.2" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx52.2.1" class="ltx_text ltx_lst_space"> </span>Score<span id="lstnumberx52.2.2" class="ltx_text ltx_lst_space"> </span>and<span id="lstnumberx52.2.3" class="ltx_text ltx_lst_space"> </span>sub-sample<span id="lstnumberx52.2.4" class="ltx_text ltx_lst_space"> </span>the<span id="lstnumberx52.2.5" class="ltx_text ltx_lst_space"> </span>initial<span id="lstnumberx52.2.6" class="ltx_text ltx_lst_space"> </span>super-batch</span>
</div>
<div id="lstnumberx53" class="ltx_listingline">
<span id="lstnumberx53.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx53.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds</span><span id="lstnumberx53.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx53.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx53.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx53.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">model</span><span id="lstnumberx53.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx53.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">forward</span><span id="lstnumberx53.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx53.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">images</span><span id="lstnumberx53.11" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx53.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx53.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">texts</span><span id="lstnumberx53.14" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx53.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx53.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params</span><span id="lstnumberx53.17" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx53.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx53.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">approx</span><span id="lstnumberx53.20" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx53.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">approx</span><span id="lstnumberx53.22" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx53.23" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx53.24" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx53.24.1" class="ltx_text ltx_lst_space"> </span>[5B,<span id="lstnumberx53.24.2" class="ltx_text ltx_lst_space"> </span>D]</span>
</div>
<div id="lstnumberx54" class="ltx_listingline">
<span id="lstnumberx54.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx54.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds_ref</span><span id="lstnumberx54.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx54.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx54.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx54.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">batch</span><span id="lstnumberx54.7" class="ltx_text ltx_font_typewriter">[</span><span id="lstnumberx54.8" class="ltx_text ltx_lst_string ltx_font_typewriter">"embeds_ref"</span><span id="lstnumberx54.9" class="ltx_text ltx_font_typewriter">]</span><span id="lstnumberx54.10" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx54.11" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx54.11.1" class="ltx_text ltx_lst_space"> </span>Pre-cached<span id="lstnumberx54.11.2" class="ltx_text ltx_lst_space"> </span>in<span id="lstnumberx54.11.3" class="ltx_text ltx_lst_space"> </span>dataset</span>
</div>
<div id="lstnumberx55" class="ltx_listingline">
<span id="lstnumberx55.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx55.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">if</span><span id="lstnumberx55.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx55.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cfg</span><span id="lstnumberx55.5" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx55.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">loss_type</span><span id="lstnumberx55.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx55.8" class="ltx_text ltx_font_typewriter">==</span><span id="lstnumberx55.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx55.10" class="ltx_text ltx_lst_string ltx_font_typewriter">"sigmoid"</span><span id="lstnumberx55.11" class="ltx_text ltx_font_typewriter">:</span>
</div>
<div id="lstnumberx56" class="ltx_listingline">
<span id="lstnumberx56.1" class="ltx_text ltx_lst_space ltx_font_typewriter">      </span><span id="lstnumberx56.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">scores</span><span id="lstnumberx56.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx56.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">get_scores_sigmoid</span><span id="lstnumberx56.7" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx56.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds</span><span id="lstnumberx56.9" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx56.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds_ref</span><span id="lstnumberx56.12" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx56.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params</span><span id="lstnumberx56.15" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx56.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx56.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params_ref</span><span id="lstnumberx56.18" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx56.19" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx56.20" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx56.20.1" class="ltx_text ltx_lst_space"> </span>Get<span id="lstnumberx56.20.2" class="ltx_text ltx_lst_space"> </span>scores</span>
</div>
<div id="lstnumberx57" class="ltx_listingline">
<span id="lstnumberx57.1" class="ltx_text ltx_lst_space ltx_font_typewriter">      </span><span id="lstnumberx57.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">inds</span><span id="lstnumberx57.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx57.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">jointly_sample_batch</span><span id="lstnumberx57.7" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx57.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">scores</span><span id="lstnumberx57.9" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx57.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cfg</span><span id="lstnumberx57.12" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx57.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_chunks</span><span id="lstnumberx57.14" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx57.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cfg</span><span id="lstnumberx57.17" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx57.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">filter_ratio</span><span id="lstnumberx57.19" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx57.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx57.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cfg</span><span id="lstnumberx57.22" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx57.23" class="ltx_text ltx_lst_identifier ltx_font_typewriter">learnability</span><span id="lstnumberx57.24" class="ltx_text ltx_font_typewriter">)</span>
</div>
<div id="lstnumberx58" class="ltx_listingline">
<span id="lstnumberx58.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx58.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">elif</span><span id="lstnumberx58.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx58.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cfg</span><span id="lstnumberx58.5" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx58.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">loss_type</span><span id="lstnumberx58.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx58.8" class="ltx_text ltx_font_typewriter">==</span><span id="lstnumberx58.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx58.10" class="ltx_text ltx_lst_string ltx_font_typewriter">"softmax"</span><span id="lstnumberx58.11" class="ltx_text ltx_font_typewriter">:</span>
</div>
<div id="lstnumberx59" class="ltx_listingline">
<span id="lstnumberx59.1" class="ltx_text ltx_lst_space ltx_font_typewriter">      </span><span id="lstnumberx59.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">inds</span><span id="lstnumberx59.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx59.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx59.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx59.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">jointly_sample_batch_softmax</span><span id="lstnumberx59.7" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx59.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds_ref</span><span id="lstnumberx59.9" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx59.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx59.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds</span><span id="lstnumberx59.12" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx59.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx59.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params_ref</span><span id="lstnumberx59.15" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx59.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx59.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params</span><span id="lstnumberx59.18" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx59.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx59.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cfg</span><span id="lstnumberx59.21" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx59.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_chunks</span><span id="lstnumberx59.23" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx59.24" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx59.25" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cfg</span><span id="lstnumberx59.26" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx59.27" class="ltx_text ltx_lst_identifier ltx_font_typewriter">filter_ratio</span><span id="lstnumberx59.28" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx59.29" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx59.30" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx59.30.1" class="ltx_text ltx_lst_space"> </span>for<span id="lstnumberx59.30.2" class="ltx_text ltx_lst_space"> </span>softmax<span id="lstnumberx59.30.3" class="ltx_text ltx_lst_space"> </span>loss,<span id="lstnumberx59.30.4" class="ltx_text ltx_lst_space"> </span>scores<span id="lstnumberx59.30.5" class="ltx_text ltx_lst_space"> </span>are<span id="lstnumberx59.30.6" class="ltx_text ltx_lst_space"> </span>re-computed<span id="lstnumberx59.30.7" class="ltx_text ltx_lst_space"> </span>in<span id="lstnumberx59.30.8" class="ltx_text ltx_lst_space"> </span>the<span id="lstnumberx59.30.9" class="ltx_text ltx_lst_space"> </span>iterative<span id="lstnumberx59.30.10" class="ltx_text ltx_lst_space"> </span>sampling.</span>
</div>
<div id="lstnumberx60" class="ltx_listingline">
</div>
<div id="lstnumberx61" class="ltx_listingline">
<span id="lstnumberx61.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx61.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">images</span><span id="lstnumberx61.3" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx61.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx61.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">texts</span><span id="lstnumberx61.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx61.7" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx61.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx61.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">stop_grad</span><span id="lstnumberx61.10" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx61.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">images</span><span id="lstnumberx61.12" class="ltx_text ltx_font_typewriter">[</span><span id="lstnumberx61.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">inds</span><span id="lstnumberx61.14" class="ltx_text ltx_font_typewriter">]),</span><span id="lstnumberx61.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx61.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">stop_grad</span><span id="lstnumberx61.17" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx61.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">texts</span><span id="lstnumberx61.19" class="ltx_text ltx_font_typewriter">[</span><span id="lstnumberx61.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">inds</span><span id="lstnumberx61.21" class="ltx_text ltx_font_typewriter">])</span><span id="lstnumberx61.22" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx61.23" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx61.23.1" class="ltx_text ltx_lst_space"> </span>[B,<span id="lstnumberx61.23.2" class="ltx_text ltx_lst_space"> </span>...]</span>
</div>
<div id="lstnumberx62" class="ltx_listingline">
</div>
<div id="lstnumberx63" class="ltx_listingline">
<span id="lstnumberx63.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx63.2" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx63.2.1" class="ltx_text ltx_lst_space"> </span>Split<span id="lstnumberx63.2.2" class="ltx_text ltx_lst_space"> </span>batch<span id="lstnumberx63.2.3" class="ltx_text ltx_lst_space"> </span>for<span id="lstnumberx63.2.4" class="ltx_text ltx_lst_space"> </span>co-training</span>
</div>
<div id="lstnumberx64" class="ltx_listingline">
<span id="lstnumberx64.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx64.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">images_full</span><span id="lstnumberx64.3" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx64.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">images_approx</span><span id="lstnumberx64.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.7" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx64.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">images</span><span id="lstnumberx64.10" class="ltx_text ltx_font_typewriter">[::2],</span><span id="lstnumberx64.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx64.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">images</span><span id="lstnumberx64.13" class="ltx_text ltx_font_typewriter">[1::2]</span><span id="lstnumberx64.14" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx64.15" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx64.15.1" class="ltx_text ltx_lst_space"> </span>[B/2,<span id="lstnumberx64.15.2" class="ltx_text ltx_lst_space"> </span>...],<span id="lstnumberx64.15.3" class="ltx_text ltx_lst_space"> </span>[B/2,<span id="lstnumberx64.15.4" class="ltx_text ltx_lst_space"> </span>...]</span>
</div>
<div id="lstnumberx65" class="ltx_listingline">
<span id="lstnumberx65.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx65.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">texts_full</span><span id="lstnumberx65.3" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx65.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">texts_approx</span><span id="lstnumberx65.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.7" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx65.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">texts</span><span id="lstnumberx65.10" class="ltx_text ltx_font_typewriter">[::2],</span><span id="lstnumberx65.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx65.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">texts</span><span id="lstnumberx65.13" class="ltx_text ltx_font_typewriter">[1::2]</span><span id="lstnumberx65.14" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx65.15" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx65.15.1" class="ltx_text ltx_lst_space"> </span>[B/2,<span id="lstnumberx65.15.2" class="ltx_text ltx_lst_space"> </span>...],<span id="lstnumberx65.15.3" class="ltx_text ltx_lst_space"> </span>[B/2,<span id="lstnumberx65.15.4" class="ltx_text ltx_lst_space"> </span>...]</span>
</div>
<div id="lstnumberx66" class="ltx_listingline">
</div>
<div id="lstnumberx67" class="ltx_listingline">
<span id="lstnumberx67.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx67.2" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx67.2.1" class="ltx_text ltx_lst_space"> </span>Compute<span id="lstnumberx67.2.2" class="ltx_text ltx_lst_space"> </span>overall<span id="lstnumberx67.2.3" class="ltx_text ltx_lst_space"> </span>loss</span>
</div>
<div id="lstnumberx68" class="ltx_listingline">
<span id="lstnumberx68.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx68.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds_full</span><span id="lstnumberx68.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx68.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">model</span><span id="lstnumberx68.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx68.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">forward</span><span id="lstnumberx68.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx68.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">images_full</span><span id="lstnumberx68.11" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx68.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">texts_full</span><span id="lstnumberx68.14" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx68.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params</span><span id="lstnumberx68.17" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx68.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx68.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">approx</span><span id="lstnumberx68.20" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx68.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">False</span><span id="lstnumberx68.22" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx68.23" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx68.24" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx68.24.1" class="ltx_text ltx_lst_space"> </span>[B/2,<span id="lstnumberx68.24.2" class="ltx_text ltx_lst_space"> </span>D],<span id="lstnumberx68.24.3" class="ltx_text ltx_lst_space"> </span>[B/2,<span id="lstnumberx68.24.4" class="ltx_text ltx_lst_space"> </span>D]</span>
</div>
<div id="lstnumberx69" class="ltx_listingline">
<span id="lstnumberx69.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx69.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds_approx</span><span id="lstnumberx69.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx69.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">model</span><span id="lstnumberx69.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx69.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">forward</span><span id="lstnumberx69.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx69.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">images_approx</span><span id="lstnumberx69.11" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx69.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">texts_approx</span><span id="lstnumberx69.14" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx69.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params</span><span id="lstnumberx69.17" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx69.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx69.19" class="ltx_text ltx_lst_identifier ltx_font_typewriter">approx</span><span id="lstnumberx69.20" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx69.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">approx</span><span id="lstnumberx69.22" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx69.23" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx69.24" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx69.24.1" class="ltx_text ltx_lst_space"> </span>[B/2,<span id="lstnumberx69.24.2" class="ltx_text ltx_lst_space"> </span>D],<span id="lstnumberx69.24.3" class="ltx_text ltx_lst_space"> </span>[B/2,<span id="lstnumberx69.24.4" class="ltx_text ltx_lst_space"> </span>D]</span>
</div>
<div id="lstnumberx70" class="ltx_listingline">
<span id="lstnumberx70.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx70.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">zimg</span><span id="lstnumberx70.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx70.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx70.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx70.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">np</span><span id="lstnumberx70.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx70.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">concatenate</span><span id="lstnumberx70.9" class="ltx_text ltx_font_typewriter">([</span><span id="lstnumberx70.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds_full</span><span id="lstnumberx70.11" class="ltx_text ltx_font_typewriter">[0],</span><span id="lstnumberx70.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx70.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds_approx</span><span id="lstnumberx70.14" class="ltx_text ltx_font_typewriter">[0]],</span><span id="lstnumberx70.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx70.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">axis</span><span id="lstnumberx70.17" class="ltx_text ltx_font_typewriter">=0)</span>
</div>
<div id="lstnumberx71" class="ltx_listingline">
<span id="lstnumberx71.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx71.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ztxt</span><span id="lstnumberx71.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx71.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx71.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx71.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">np</span><span id="lstnumberx71.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx71.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">concatenate</span><span id="lstnumberx71.9" class="ltx_text ltx_font_typewriter">([</span><span id="lstnumberx71.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds_full</span><span id="lstnumberx71.11" class="ltx_text ltx_font_typewriter">[1],</span><span id="lstnumberx71.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx71.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds_approx</span><span id="lstnumberx71.14" class="ltx_text ltx_font_typewriter">[1]],</span><span id="lstnumberx71.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx71.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">axis</span><span id="lstnumberx71.17" class="ltx_text ltx_font_typewriter">=0)</span>
</div>
<div id="lstnumberx72" class="ltx_listingline">
<span id="lstnumberx72.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx72.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">if</span><span id="lstnumberx72.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx72.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">loss_type</span><span id="lstnumberx72.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx72.6" class="ltx_text ltx_font_typewriter">==</span><span id="lstnumberx72.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx72.8" class="ltx_text ltx_lst_string ltx_font_typewriter">"sigmoid"</span><span id="lstnumberx72.9" class="ltx_text ltx_font_typewriter">:</span>
</div>
<div id="lstnumberx73" class="ltx_listingline">
<span id="lstnumberx73.1" class="ltx_text ltx_lst_space ltx_font_typewriter">      </span><span id="lstnumberx73.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">loss</span><span id="lstnumberx73.3" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx73.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx73.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">_</span><span id="lstnumberx73.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx73.7" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx73.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx73.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">sigmoid_nll</span><span id="lstnumberx73.10" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx73.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params</span><span id="lstnumberx73.12" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx73.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx73.14" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx73.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">zimg</span><span id="lstnumberx73.16" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx73.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx73.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ztxt</span><span id="lstnumberx73.19" class="ltx_text ltx_font_typewriter">))</span>
</div>
<div id="lstnumberx74" class="ltx_listingline">
<span id="lstnumberx74.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx74.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">elif</span><span id="lstnumberx74.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx74.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">loss_type</span><span id="lstnumberx74.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx74.6" class="ltx_text ltx_font_typewriter">==</span><span id="lstnumberx74.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx74.8" class="ltx_text ltx_lst_string ltx_font_typewriter">"softmax"</span><span id="lstnumberx74.9" class="ltx_text ltx_font_typewriter">:</span>
</div>
<div id="lstnumberx75" class="ltx_listingline">
<span id="lstnumberx75.1" class="ltx_text ltx_lst_space ltx_font_typewriter">      </span><span id="lstnumberx75.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">loss</span><span id="lstnumberx75.3" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx75.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx75.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">_</span><span id="lstnumberx75.6" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx75.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx75.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">_</span><span id="lstnumberx75.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx75.10" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx75.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx75.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">softmax_nll</span><span id="lstnumberx75.13" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx75.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params</span><span id="lstnumberx75.15" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx75.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx75.17" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx75.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">zimg</span><span id="lstnumberx75.19" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx75.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx75.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ztxt</span><span id="lstnumberx75.22" class="ltx_text ltx_font_typewriter">),</span><span id="lstnumberx75.23" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx75.24" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx75.25" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx75.26" class="ltx_text ltx_lst_identifier ltx_font_typewriter">None</span><span id="lstnumberx75.27" class="ltx_text ltx_font_typewriter">)</span>
</div>
<div id="lstnumberx76" class="ltx_listingline">
</div>
<div id="lstnumberx77" class="ltx_listingline">
<span id="lstnumberx77.1" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx77.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">return</span><span id="lstnumberx77.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx77.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">loss</span>
</div>
</div>
</figure>
</section>
<section id="A1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.2 </span>Optimisation robustness</h3>

<div id="A1.SS2.p1" class="ltx_para ltx_noindent">
<p id="A1.SS2.p1.2" class="ltx_p">Our initial experiments used the default ADAM optimizer parameters. At a filtering ratio of 50%, we observed good gains over IID as described previously. Increasing the filtering ratio to 80% however did not produce further gains. Inspecting the training curves showed training instabilities (Figure <a href="#A1.F6" title="Figure 6 ‣ A.2 Optimisation robustness ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>, Left). In the original SigLIP paper <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>, the authors found similar instabilities when increasing the batch size, but found that setting <math id="A1.SS2.p1.1.m1.1" class="ltx_Math" alttext="\beta_{2}=0.95" display="inline"><semantics id="A1.SS2.p1.1.m1.1a"><mrow id="A1.SS2.p1.1.m1.1.1" xref="A1.SS2.p1.1.m1.1.1.cmml"><msub id="A1.SS2.p1.1.m1.1.1.2" xref="A1.SS2.p1.1.m1.1.1.2.cmml"><mi id="A1.SS2.p1.1.m1.1.1.2.2" xref="A1.SS2.p1.1.m1.1.1.2.2.cmml">β</mi><mn id="A1.SS2.p1.1.m1.1.1.2.3" xref="A1.SS2.p1.1.m1.1.1.2.3.cmml">2</mn></msub><mo id="A1.SS2.p1.1.m1.1.1.1" xref="A1.SS2.p1.1.m1.1.1.1.cmml">=</mo><mn id="A1.SS2.p1.1.m1.1.1.3" xref="A1.SS2.p1.1.m1.1.1.3.cmml">0.95</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.1.m1.1b"><apply id="A1.SS2.p1.1.m1.1.1.cmml" xref="A1.SS2.p1.1.m1.1.1"><eq id="A1.SS2.p1.1.m1.1.1.1.cmml" xref="A1.SS2.p1.1.m1.1.1.1"></eq><apply id="A1.SS2.p1.1.m1.1.1.2.cmml" xref="A1.SS2.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="A1.SS2.p1.1.m1.1.1.2.1.cmml" xref="A1.SS2.p1.1.m1.1.1.2">subscript</csymbol><ci id="A1.SS2.p1.1.m1.1.1.2.2.cmml" xref="A1.SS2.p1.1.m1.1.1.2.2">𝛽</ci><cn type="integer" id="A1.SS2.p1.1.m1.1.1.2.3.cmml" xref="A1.SS2.p1.1.m1.1.1.2.3">2</cn></apply><cn type="float" id="A1.SS2.p1.1.m1.1.1.3.cmml" xref="A1.SS2.p1.1.m1.1.1.3">0.95</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.1.m1.1c">\beta_{2}=0.95</annotation></semantics></math> (from <math id="A1.SS2.p1.2.m2.1" class="ltx_Math" alttext="\beta_{2}=0.999" display="inline"><semantics id="A1.SS2.p1.2.m2.1a"><mrow id="A1.SS2.p1.2.m2.1.1" xref="A1.SS2.p1.2.m2.1.1.cmml"><msub id="A1.SS2.p1.2.m2.1.1.2" xref="A1.SS2.p1.2.m2.1.1.2.cmml"><mi id="A1.SS2.p1.2.m2.1.1.2.2" xref="A1.SS2.p1.2.m2.1.1.2.2.cmml">β</mi><mn id="A1.SS2.p1.2.m2.1.1.2.3" xref="A1.SS2.p1.2.m2.1.1.2.3.cmml">2</mn></msub><mo id="A1.SS2.p1.2.m2.1.1.1" xref="A1.SS2.p1.2.m2.1.1.1.cmml">=</mo><mn id="A1.SS2.p1.2.m2.1.1.3" xref="A1.SS2.p1.2.m2.1.1.3.cmml">0.999</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.2.m2.1b"><apply id="A1.SS2.p1.2.m2.1.1.cmml" xref="A1.SS2.p1.2.m2.1.1"><eq id="A1.SS2.p1.2.m2.1.1.1.cmml" xref="A1.SS2.p1.2.m2.1.1.1"></eq><apply id="A1.SS2.p1.2.m2.1.1.2.cmml" xref="A1.SS2.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="A1.SS2.p1.2.m2.1.1.2.1.cmml" xref="A1.SS2.p1.2.m2.1.1.2">subscript</csymbol><ci id="A1.SS2.p1.2.m2.1.1.2.2.cmml" xref="A1.SS2.p1.2.m2.1.1.2.2">𝛽</ci><cn type="integer" id="A1.SS2.p1.2.m2.1.1.2.3.cmml" xref="A1.SS2.p1.2.m2.1.1.2.3">2</cn></apply><cn type="float" id="A1.SS2.p1.2.m2.1.1.3.cmml" xref="A1.SS2.p1.2.m2.1.1.3">0.999</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.2.m2.1c">\beta_{2}=0.999</annotation></semantics></math>) ameliorated the problem.</p>
</div>
<div id="A1.SS2.p2" class="ltx_para ltx_noindent">
<p id="A1.SS2.p2.1" class="ltx_p">We find the same behaviour (Figure <a href="#A1.F6" title="Figure 6 ‣ A.2 Optimisation robustness ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>). Although the filtering ratio does not directly affect the training batch size (which is constant in our experiments), this finding suggests that filtering for salient data has the effect of increasing the <span id="A1.SS2.p2.1.1" class="ltx_text ltx_font_italic">effective</span> batch size. Although this suggests an importance sampling interpretation, applying a re-weighting to the selected data decreased performance in our experiments. We leave this for further work - it is possible that further performance could be extracted from optimizer tuning at higher filtering ratios.</p>
</div>
<figure id="A1.F6" class="ltx_figure"><img src="/html/2406.17711/assets/x4.png" id="A1.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="147" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span><span id="A1.F6.7.1" class="ltx_text ltx_font_bold">Aggressive data prioritization requires robust optimization to perform well. Left:</span> With standard optimization settings (e.g. <math id="A1.F6.3.m1.1" class="ltx_Math" alttext="\beta_{2}=0.999" display="inline"><semantics id="A1.F6.3.m1.1b"><mrow id="A1.F6.3.m1.1.1" xref="A1.F6.3.m1.1.1.cmml"><msub id="A1.F6.3.m1.1.1.2" xref="A1.F6.3.m1.1.1.2.cmml"><mi id="A1.F6.3.m1.1.1.2.2" xref="A1.F6.3.m1.1.1.2.2.cmml">β</mi><mn id="A1.F6.3.m1.1.1.2.3" xref="A1.F6.3.m1.1.1.2.3.cmml">2</mn></msub><mo id="A1.F6.3.m1.1.1.1" xref="A1.F6.3.m1.1.1.1.cmml">=</mo><mn id="A1.F6.3.m1.1.1.3" xref="A1.F6.3.m1.1.1.3.cmml">0.999</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.F6.3.m1.1c"><apply id="A1.F6.3.m1.1.1.cmml" xref="A1.F6.3.m1.1.1"><eq id="A1.F6.3.m1.1.1.1.cmml" xref="A1.F6.3.m1.1.1.1"></eq><apply id="A1.F6.3.m1.1.1.2.cmml" xref="A1.F6.3.m1.1.1.2"><csymbol cd="ambiguous" id="A1.F6.3.m1.1.1.2.1.cmml" xref="A1.F6.3.m1.1.1.2">subscript</csymbol><ci id="A1.F6.3.m1.1.1.2.2.cmml" xref="A1.F6.3.m1.1.1.2.2">𝛽</ci><cn type="integer" id="A1.F6.3.m1.1.1.2.3.cmml" xref="A1.F6.3.m1.1.1.2.3">2</cn></apply><cn type="float" id="A1.F6.3.m1.1.1.3.cmml" xref="A1.F6.3.m1.1.1.3">0.999</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F6.3.m1.1d">\beta_{2}=0.999</annotation></semantics></math> in the Adam optimizer), aggressive data prioritization leads to instabilities (spikes) in the optimization process. Setting <math id="A1.F6.4.m2.1" class="ltx_Math" alttext="\beta_{2}" display="inline"><semantics id="A1.F6.4.m2.1b"><msub id="A1.F6.4.m2.1.1" xref="A1.F6.4.m2.1.1.cmml"><mi id="A1.F6.4.m2.1.1.2" xref="A1.F6.4.m2.1.1.2.cmml">β</mi><mn id="A1.F6.4.m2.1.1.3" xref="A1.F6.4.m2.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A1.F6.4.m2.1c"><apply id="A1.F6.4.m2.1.1.cmml" xref="A1.F6.4.m2.1.1"><csymbol cd="ambiguous" id="A1.F6.4.m2.1.1.1.cmml" xref="A1.F6.4.m2.1.1">subscript</csymbol><ci id="A1.F6.4.m2.1.1.2.cmml" xref="A1.F6.4.m2.1.1.2">𝛽</ci><cn type="integer" id="A1.F6.4.m2.1.1.3.cmml" xref="A1.F6.4.m2.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F6.4.m2.1d">\beta_{2}</annotation></semantics></math> to the more stable value of 0.95 remediates this instability. <span id="A1.F6.8.2" class="ltx_text ltx_font_bold">Middle, Right: </span> Stable optimization and strong prioritization together yield large improvement gains.
</figcaption>
</figure>
</section>
<section id="A1.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.3 </span>Effects of varying the training batch size</h3>

<div id="A1.SS3.p1" class="ltx_para ltx_noindent">
<p id="A1.SS3.p1.1" class="ltx_p">It is well known that performance improvements saturate with increased training batch size. In <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>, increasing the batch size beyond 32K was found to actually decrease performance, even after adjusting the <math id="A1.SS3.p1.1.m1.1" class="ltx_Math" alttext="\beta_{2}" display="inline"><semantics id="A1.SS3.p1.1.m1.1a"><msub id="A1.SS3.p1.1.m1.1.1" xref="A1.SS3.p1.1.m1.1.1.cmml"><mi id="A1.SS3.p1.1.m1.1.1.2" xref="A1.SS3.p1.1.m1.1.1.2.cmml">β</mi><mn id="A1.SS3.p1.1.m1.1.1.3" xref="A1.SS3.p1.1.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A1.SS3.p1.1.m1.1b"><apply id="A1.SS3.p1.1.m1.1.1.cmml" xref="A1.SS3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A1.SS3.p1.1.m1.1.1.1.cmml" xref="A1.SS3.p1.1.m1.1.1">subscript</csymbol><ci id="A1.SS3.p1.1.m1.1.1.2.cmml" xref="A1.SS3.p1.1.m1.1.1.2">𝛽</ci><cn type="integer" id="A1.SS3.p1.1.m1.1.1.3.cmml" xref="A1.SS3.p1.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.p1.1.m1.1c">\beta_{2}</annotation></semantics></math> parameter. In our experiments, we use 32K batch size throughout.</p>
</div>
<div id="A1.SS3.p2" class="ltx_para ltx_noindent">
<p id="A1.SS3.p2.1" class="ltx_p">The observation that filtering larger amounts of data produced the same loss spikes as observed by <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite> suggests that the training batches selected by JEST might correspond to a much larger <span id="A1.SS3.p2.1.1" class="ltx_text ltx_font_italic">effective</span> batch size. To investigate, we conducted an ablation in which we instead fixed the super-batch size and progressively decreased the training batch size (i.e. changing the filtering ratio by decreasing the amount of training data, instead of increasing the size of the super-batch as done throughout the paper).</p>
</div>
<div id="A1.SS3.p3" class="ltx_para ltx_noindent">
<p id="A1.SS3.p3.1" class="ltx_p">The results in Figure <a href="#A1.F7" title="Figure 7 ‣ A.3 Effects of varying the training batch size ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> demonstrate that, as we decrease the batch size (increase the proportion of data filtered) for a fixed super-batch size of 160K, the performance drops predictably for IID training (Left) but decreases much more slowly for JEST training (Middle / Right). Notably, for a halving of the batch size from 32K (corresponding to our f=80% experiments throughout) to 16K (filtering 90%), there was no noticeable performance drop.</p>
</div>
<div id="A1.SS3.p4" class="ltx_para ltx_noindent">
<p id="A1.SS3.p4.1" class="ltx_p">These results suggest that, at 32K training batch size, our experiments might be already operating at close to the optimal <span id="A1.SS3.p4.1.1" class="ltx_text ltx_font_italic">effective</span> batch size. We did not conduct further ablations, but it is possible that a more favourable FLOP improvement could be achieved by simultaneously increasing the super-batch size and decreasing the training batch size.</p>
</div>
<div id="A1.SS3.p5" class="ltx_para ltx_noindent">
<p id="A1.SS3.p5.1" class="ltx_p">These results suggest an importance sampling interpretation of learnability scoring - assuming the “True” mini-batch gradient is given by the expectation of the gradients from IID samples from the data, JEST is sampling only the data that contributes most to that expectation. This suggests that most of the gradient information can be reconstructed from a small number of data points. Although JEST does not explicitly sample based on the magnitude of the gradients of the data, it was demonstrated previously via a simple Taylor expansion argument that the two are equivalent in the case where learnability scores are small <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>.</p>
</div>
<figure id="A1.F7" class="ltx_figure"><img src="/html/2406.17711/assets/x5.png" id="A1.F7.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="147" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 7: </span><span id="A1.F7.2.1" class="ltx_text ltx_font_bold">Effective Batch Size Experiments</span> Right: Difference in average performance between JEST++ and IID training as a function of training batch size. Instead of increasing the filtering ratio by increasing the super-batch size as done in previous experiments, we instead fix the super-batch size and reduce the training batch size. There was no noticeable drop from halving the training batch size, suggesting that further efficiency gains might be achieved by training on less data in addition to filtering from a larger pool.</figcaption>
</figure>
</section>
<section id="A1.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.4 </span>FLOP calculations for JEST / Flexi-JEST</h3>

<div id="A1.SS4.p1" class="ltx_para ltx_noindent">
<p id="A1.SS4.p1.2" class="ltx_p">We assume that training on a single data point cost approximately <math id="A1.SS4.p1.1.m1.1" class="ltx_Math" alttext="C_{\text{IID}}=3F" display="inline"><semantics id="A1.SS4.p1.1.m1.1a"><mrow id="A1.SS4.p1.1.m1.1.1" xref="A1.SS4.p1.1.m1.1.1.cmml"><msub id="A1.SS4.p1.1.m1.1.1.2" xref="A1.SS4.p1.1.m1.1.1.2.cmml"><mi id="A1.SS4.p1.1.m1.1.1.2.2" xref="A1.SS4.p1.1.m1.1.1.2.2.cmml">C</mi><mtext id="A1.SS4.p1.1.m1.1.1.2.3" xref="A1.SS4.p1.1.m1.1.1.2.3a.cmml">IID</mtext></msub><mo id="A1.SS4.p1.1.m1.1.1.1" xref="A1.SS4.p1.1.m1.1.1.1.cmml">=</mo><mrow id="A1.SS4.p1.1.m1.1.1.3" xref="A1.SS4.p1.1.m1.1.1.3.cmml"><mn id="A1.SS4.p1.1.m1.1.1.3.2" xref="A1.SS4.p1.1.m1.1.1.3.2.cmml">3</mn><mo lspace="0em" rspace="0em" id="A1.SS4.p1.1.m1.1.1.3.1" xref="A1.SS4.p1.1.m1.1.1.3.1.cmml">​</mo><mi id="A1.SS4.p1.1.m1.1.1.3.3" xref="A1.SS4.p1.1.m1.1.1.3.3.cmml">F</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p1.1.m1.1b"><apply id="A1.SS4.p1.1.m1.1.1.cmml" xref="A1.SS4.p1.1.m1.1.1"><eq id="A1.SS4.p1.1.m1.1.1.1.cmml" xref="A1.SS4.p1.1.m1.1.1.1"></eq><apply id="A1.SS4.p1.1.m1.1.1.2.cmml" xref="A1.SS4.p1.1.m1.1.1.2"><csymbol cd="ambiguous" id="A1.SS4.p1.1.m1.1.1.2.1.cmml" xref="A1.SS4.p1.1.m1.1.1.2">subscript</csymbol><ci id="A1.SS4.p1.1.m1.1.1.2.2.cmml" xref="A1.SS4.p1.1.m1.1.1.2.2">𝐶</ci><ci id="A1.SS4.p1.1.m1.1.1.2.3a.cmml" xref="A1.SS4.p1.1.m1.1.1.2.3"><mtext mathsize="70%" id="A1.SS4.p1.1.m1.1.1.2.3.cmml" xref="A1.SS4.p1.1.m1.1.1.2.3">IID</mtext></ci></apply><apply id="A1.SS4.p1.1.m1.1.1.3.cmml" xref="A1.SS4.p1.1.m1.1.1.3"><times id="A1.SS4.p1.1.m1.1.1.3.1.cmml" xref="A1.SS4.p1.1.m1.1.1.3.1"></times><cn type="integer" id="A1.SS4.p1.1.m1.1.1.3.2.cmml" xref="A1.SS4.p1.1.m1.1.1.3.2">3</cn><ci id="A1.SS4.p1.1.m1.1.1.3.3.cmml" xref="A1.SS4.p1.1.m1.1.1.3.3">𝐹</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p1.1.m1.1c">C_{\text{IID}}=3F</annotation></semantics></math> forward passes <math id="A1.SS4.p1.2.m2.1" class="ltx_Math" alttext="F" display="inline"><semantics id="A1.SS4.p1.2.m2.1a"><mi id="A1.SS4.p1.2.m2.1.1" xref="A1.SS4.p1.2.m2.1.1.cmml">F</mi><annotation-xml encoding="MathML-Content" id="A1.SS4.p1.2.m2.1b"><ci id="A1.SS4.p1.2.m2.1.1.cmml" xref="A1.SS4.p1.2.m2.1.1">𝐹</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p1.2.m2.1c">F</annotation></semantics></math> of the learner model <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib25" title="" class="ltx_ref">25</a>]</cite>. The cost for a single JEST update can therefore be computed as:</p>
</div>
<div id="A1.SS4.p2" class="ltx_para ltx_noindent">
<table id="A1.Ex1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.Ex1.m1.1" class="ltx_Math" alttext="C_{\text{JEST}}=3F+FB/b-F=F(2+B/b)" display="block"><semantics id="A1.Ex1.m1.1a"><mrow id="A1.Ex1.m1.1.1" xref="A1.Ex1.m1.1.1.cmml"><msub id="A1.Ex1.m1.1.1.3" xref="A1.Ex1.m1.1.1.3.cmml"><mi id="A1.Ex1.m1.1.1.3.2" xref="A1.Ex1.m1.1.1.3.2.cmml">C</mi><mtext id="A1.Ex1.m1.1.1.3.3" xref="A1.Ex1.m1.1.1.3.3a.cmml">JEST</mtext></msub><mo id="A1.Ex1.m1.1.1.4" xref="A1.Ex1.m1.1.1.4.cmml">=</mo><mrow id="A1.Ex1.m1.1.1.5" xref="A1.Ex1.m1.1.1.5.cmml"><mrow id="A1.Ex1.m1.1.1.5.2" xref="A1.Ex1.m1.1.1.5.2.cmml"><mrow id="A1.Ex1.m1.1.1.5.2.2" xref="A1.Ex1.m1.1.1.5.2.2.cmml"><mn id="A1.Ex1.m1.1.1.5.2.2.2" xref="A1.Ex1.m1.1.1.5.2.2.2.cmml">3</mn><mo lspace="0em" rspace="0em" id="A1.Ex1.m1.1.1.5.2.2.1" xref="A1.Ex1.m1.1.1.5.2.2.1.cmml">​</mo><mi id="A1.Ex1.m1.1.1.5.2.2.3" xref="A1.Ex1.m1.1.1.5.2.2.3.cmml">F</mi></mrow><mo id="A1.Ex1.m1.1.1.5.2.1" xref="A1.Ex1.m1.1.1.5.2.1.cmml">+</mo><mrow id="A1.Ex1.m1.1.1.5.2.3" xref="A1.Ex1.m1.1.1.5.2.3.cmml"><mrow id="A1.Ex1.m1.1.1.5.2.3.2" xref="A1.Ex1.m1.1.1.5.2.3.2.cmml"><mi id="A1.Ex1.m1.1.1.5.2.3.2.2" xref="A1.Ex1.m1.1.1.5.2.3.2.2.cmml">F</mi><mo lspace="0em" rspace="0em" id="A1.Ex1.m1.1.1.5.2.3.2.1" xref="A1.Ex1.m1.1.1.5.2.3.2.1.cmml">​</mo><mi id="A1.Ex1.m1.1.1.5.2.3.2.3" xref="A1.Ex1.m1.1.1.5.2.3.2.3.cmml">B</mi></mrow><mo id="A1.Ex1.m1.1.1.5.2.3.1" xref="A1.Ex1.m1.1.1.5.2.3.1.cmml">/</mo><mi id="A1.Ex1.m1.1.1.5.2.3.3" xref="A1.Ex1.m1.1.1.5.2.3.3.cmml">b</mi></mrow></mrow><mo id="A1.Ex1.m1.1.1.5.1" xref="A1.Ex1.m1.1.1.5.1.cmml">−</mo><mi id="A1.Ex1.m1.1.1.5.3" xref="A1.Ex1.m1.1.1.5.3.cmml">F</mi></mrow><mo id="A1.Ex1.m1.1.1.6" xref="A1.Ex1.m1.1.1.6.cmml">=</mo><mrow id="A1.Ex1.m1.1.1.1" xref="A1.Ex1.m1.1.1.1.cmml"><mi id="A1.Ex1.m1.1.1.1.3" xref="A1.Ex1.m1.1.1.1.3.cmml">F</mi><mo lspace="0em" rspace="0em" id="A1.Ex1.m1.1.1.1.2" xref="A1.Ex1.m1.1.1.1.2.cmml">​</mo><mrow id="A1.Ex1.m1.1.1.1.1.1" xref="A1.Ex1.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.Ex1.m1.1.1.1.1.1.2" xref="A1.Ex1.m1.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.Ex1.m1.1.1.1.1.1.1" xref="A1.Ex1.m1.1.1.1.1.1.1.cmml"><mn id="A1.Ex1.m1.1.1.1.1.1.1.2" xref="A1.Ex1.m1.1.1.1.1.1.1.2.cmml">2</mn><mo id="A1.Ex1.m1.1.1.1.1.1.1.1" xref="A1.Ex1.m1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="A1.Ex1.m1.1.1.1.1.1.1.3" xref="A1.Ex1.m1.1.1.1.1.1.1.3.cmml"><mi id="A1.Ex1.m1.1.1.1.1.1.1.3.2" xref="A1.Ex1.m1.1.1.1.1.1.1.3.2.cmml">B</mi><mo id="A1.Ex1.m1.1.1.1.1.1.1.3.1" xref="A1.Ex1.m1.1.1.1.1.1.1.3.1.cmml">/</mo><mi id="A1.Ex1.m1.1.1.1.1.1.1.3.3" xref="A1.Ex1.m1.1.1.1.1.1.1.3.3.cmml">b</mi></mrow></mrow><mo stretchy="false" id="A1.Ex1.m1.1.1.1.1.1.3" xref="A1.Ex1.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex1.m1.1b"><apply id="A1.Ex1.m1.1.1.cmml" xref="A1.Ex1.m1.1.1"><and id="A1.Ex1.m1.1.1a.cmml" xref="A1.Ex1.m1.1.1"></and><apply id="A1.Ex1.m1.1.1b.cmml" xref="A1.Ex1.m1.1.1"><eq id="A1.Ex1.m1.1.1.4.cmml" xref="A1.Ex1.m1.1.1.4"></eq><apply id="A1.Ex1.m1.1.1.3.cmml" xref="A1.Ex1.m1.1.1.3"><csymbol cd="ambiguous" id="A1.Ex1.m1.1.1.3.1.cmml" xref="A1.Ex1.m1.1.1.3">subscript</csymbol><ci id="A1.Ex1.m1.1.1.3.2.cmml" xref="A1.Ex1.m1.1.1.3.2">𝐶</ci><ci id="A1.Ex1.m1.1.1.3.3a.cmml" xref="A1.Ex1.m1.1.1.3.3"><mtext mathsize="70%" id="A1.Ex1.m1.1.1.3.3.cmml" xref="A1.Ex1.m1.1.1.3.3">JEST</mtext></ci></apply><apply id="A1.Ex1.m1.1.1.5.cmml" xref="A1.Ex1.m1.1.1.5"><minus id="A1.Ex1.m1.1.1.5.1.cmml" xref="A1.Ex1.m1.1.1.5.1"></minus><apply id="A1.Ex1.m1.1.1.5.2.cmml" xref="A1.Ex1.m1.1.1.5.2"><plus id="A1.Ex1.m1.1.1.5.2.1.cmml" xref="A1.Ex1.m1.1.1.5.2.1"></plus><apply id="A1.Ex1.m1.1.1.5.2.2.cmml" xref="A1.Ex1.m1.1.1.5.2.2"><times id="A1.Ex1.m1.1.1.5.2.2.1.cmml" xref="A1.Ex1.m1.1.1.5.2.2.1"></times><cn type="integer" id="A1.Ex1.m1.1.1.5.2.2.2.cmml" xref="A1.Ex1.m1.1.1.5.2.2.2">3</cn><ci id="A1.Ex1.m1.1.1.5.2.2.3.cmml" xref="A1.Ex1.m1.1.1.5.2.2.3">𝐹</ci></apply><apply id="A1.Ex1.m1.1.1.5.2.3.cmml" xref="A1.Ex1.m1.1.1.5.2.3"><divide id="A1.Ex1.m1.1.1.5.2.3.1.cmml" xref="A1.Ex1.m1.1.1.5.2.3.1"></divide><apply id="A1.Ex1.m1.1.1.5.2.3.2.cmml" xref="A1.Ex1.m1.1.1.5.2.3.2"><times id="A1.Ex1.m1.1.1.5.2.3.2.1.cmml" xref="A1.Ex1.m1.1.1.5.2.3.2.1"></times><ci id="A1.Ex1.m1.1.1.5.2.3.2.2.cmml" xref="A1.Ex1.m1.1.1.5.2.3.2.2">𝐹</ci><ci id="A1.Ex1.m1.1.1.5.2.3.2.3.cmml" xref="A1.Ex1.m1.1.1.5.2.3.2.3">𝐵</ci></apply><ci id="A1.Ex1.m1.1.1.5.2.3.3.cmml" xref="A1.Ex1.m1.1.1.5.2.3.3">𝑏</ci></apply></apply><ci id="A1.Ex1.m1.1.1.5.3.cmml" xref="A1.Ex1.m1.1.1.5.3">𝐹</ci></apply></apply><apply id="A1.Ex1.m1.1.1c.cmml" xref="A1.Ex1.m1.1.1"><eq id="A1.Ex1.m1.1.1.6.cmml" xref="A1.Ex1.m1.1.1.6"></eq><share href="#A1.Ex1.m1.1.1.5.cmml" id="A1.Ex1.m1.1.1d.cmml" xref="A1.Ex1.m1.1.1"></share><apply id="A1.Ex1.m1.1.1.1.cmml" xref="A1.Ex1.m1.1.1.1"><times id="A1.Ex1.m1.1.1.1.2.cmml" xref="A1.Ex1.m1.1.1.1.2"></times><ci id="A1.Ex1.m1.1.1.1.3.cmml" xref="A1.Ex1.m1.1.1.1.3">𝐹</ci><apply id="A1.Ex1.m1.1.1.1.1.1.1.cmml" xref="A1.Ex1.m1.1.1.1.1.1"><plus id="A1.Ex1.m1.1.1.1.1.1.1.1.cmml" xref="A1.Ex1.m1.1.1.1.1.1.1.1"></plus><cn type="integer" id="A1.Ex1.m1.1.1.1.1.1.1.2.cmml" xref="A1.Ex1.m1.1.1.1.1.1.1.2">2</cn><apply id="A1.Ex1.m1.1.1.1.1.1.1.3.cmml" xref="A1.Ex1.m1.1.1.1.1.1.1.3"><divide id="A1.Ex1.m1.1.1.1.1.1.1.3.1.cmml" xref="A1.Ex1.m1.1.1.1.1.1.1.3.1"></divide><ci id="A1.Ex1.m1.1.1.1.1.1.1.3.2.cmml" xref="A1.Ex1.m1.1.1.1.1.1.1.3.2">𝐵</ci><ci id="A1.Ex1.m1.1.1.1.1.1.1.3.3.cmml" xref="A1.Ex1.m1.1.1.1.1.1.1.3.3">𝑏</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex1.m1.1c">C_{\text{JEST}}=3F+FB/b-F=F(2+B/b)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="A1.SS4.p3" class="ltx_para ltx_noindent">
<p id="A1.SS4.p3.5" class="ltx_p">where <math id="A1.SS4.p3.1.m1.1" class="ltx_Math" alttext="B" display="inline"><semantics id="A1.SS4.p3.1.m1.1a"><mi id="A1.SS4.p3.1.m1.1.1" xref="A1.SS4.p3.1.m1.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="A1.SS4.p3.1.m1.1b"><ci id="A1.SS4.p3.1.m1.1.1.cmml" xref="A1.SS4.p3.1.m1.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p3.1.m1.1c">B</annotation></semantics></math> and <math id="A1.SS4.p3.2.m2.1" class="ltx_Math" alttext="b" display="inline"><semantics id="A1.SS4.p3.2.m2.1a"><mi id="A1.SS4.p3.2.m2.1.1" xref="A1.SS4.p3.2.m2.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="A1.SS4.p3.2.m2.1b"><ci id="A1.SS4.p3.2.m2.1.1.cmml" xref="A1.SS4.p3.2.m2.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p3.2.m2.1c">b</annotation></semantics></math> are the super-batch and sub-batch sizes respectively and <math id="A1.SS4.p3.3.m3.1" class="ltx_Math" alttext="f=1-b/B" display="inline"><semantics id="A1.SS4.p3.3.m3.1a"><mrow id="A1.SS4.p3.3.m3.1.1" xref="A1.SS4.p3.3.m3.1.1.cmml"><mi id="A1.SS4.p3.3.m3.1.1.2" xref="A1.SS4.p3.3.m3.1.1.2.cmml">f</mi><mo id="A1.SS4.p3.3.m3.1.1.1" xref="A1.SS4.p3.3.m3.1.1.1.cmml">=</mo><mrow id="A1.SS4.p3.3.m3.1.1.3" xref="A1.SS4.p3.3.m3.1.1.3.cmml"><mn id="A1.SS4.p3.3.m3.1.1.3.2" xref="A1.SS4.p3.3.m3.1.1.3.2.cmml">1</mn><mo id="A1.SS4.p3.3.m3.1.1.3.1" xref="A1.SS4.p3.3.m3.1.1.3.1.cmml">−</mo><mrow id="A1.SS4.p3.3.m3.1.1.3.3" xref="A1.SS4.p3.3.m3.1.1.3.3.cmml"><mi id="A1.SS4.p3.3.m3.1.1.3.3.2" xref="A1.SS4.p3.3.m3.1.1.3.3.2.cmml">b</mi><mo id="A1.SS4.p3.3.m3.1.1.3.3.1" xref="A1.SS4.p3.3.m3.1.1.3.3.1.cmml">/</mo><mi id="A1.SS4.p3.3.m3.1.1.3.3.3" xref="A1.SS4.p3.3.m3.1.1.3.3.3.cmml">B</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p3.3.m3.1b"><apply id="A1.SS4.p3.3.m3.1.1.cmml" xref="A1.SS4.p3.3.m3.1.1"><eq id="A1.SS4.p3.3.m3.1.1.1.cmml" xref="A1.SS4.p3.3.m3.1.1.1"></eq><ci id="A1.SS4.p3.3.m3.1.1.2.cmml" xref="A1.SS4.p3.3.m3.1.1.2">𝑓</ci><apply id="A1.SS4.p3.3.m3.1.1.3.cmml" xref="A1.SS4.p3.3.m3.1.1.3"><minus id="A1.SS4.p3.3.m3.1.1.3.1.cmml" xref="A1.SS4.p3.3.m3.1.1.3.1"></minus><cn type="integer" id="A1.SS4.p3.3.m3.1.1.3.2.cmml" xref="A1.SS4.p3.3.m3.1.1.3.2">1</cn><apply id="A1.SS4.p3.3.m3.1.1.3.3.cmml" xref="A1.SS4.p3.3.m3.1.1.3.3"><divide id="A1.SS4.p3.3.m3.1.1.3.3.1.cmml" xref="A1.SS4.p3.3.m3.1.1.3.3.1"></divide><ci id="A1.SS4.p3.3.m3.1.1.3.3.2.cmml" xref="A1.SS4.p3.3.m3.1.1.3.3.2">𝑏</ci><ci id="A1.SS4.p3.3.m3.1.1.3.3.3.cmml" xref="A1.SS4.p3.3.m3.1.1.3.3.3">𝐵</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p3.3.m3.1c">f=1-b/B</annotation></semantics></math>. The base JEST method does not use approximations on the learner, which allows us to cache the forward pass during scoring and re-use it for the gradient computation. Relative to an IID update, the cost of a single JEST iteration at a filtering ratio <math id="A1.SS4.p3.4.m4.1" class="ltx_Math" alttext="f=0.8" display="inline"><semantics id="A1.SS4.p3.4.m4.1a"><mrow id="A1.SS4.p3.4.m4.1.1" xref="A1.SS4.p3.4.m4.1.1.cmml"><mi id="A1.SS4.p3.4.m4.1.1.2" xref="A1.SS4.p3.4.m4.1.1.2.cmml">f</mi><mo id="A1.SS4.p3.4.m4.1.1.1" xref="A1.SS4.p3.4.m4.1.1.1.cmml">=</mo><mn id="A1.SS4.p3.4.m4.1.1.3" xref="A1.SS4.p3.4.m4.1.1.3.cmml">0.8</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p3.4.m4.1b"><apply id="A1.SS4.p3.4.m4.1.1.cmml" xref="A1.SS4.p3.4.m4.1.1"><eq id="A1.SS4.p3.4.m4.1.1.1.cmml" xref="A1.SS4.p3.4.m4.1.1.1"></eq><ci id="A1.SS4.p3.4.m4.1.1.2.cmml" xref="A1.SS4.p3.4.m4.1.1.2">𝑓</ci><cn type="float" id="A1.SS4.p3.4.m4.1.1.3.cmml" xref="A1.SS4.p3.4.m4.1.1.3">0.8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p3.4.m4.1c">f=0.8</annotation></semantics></math> comes out as <math id="A1.SS4.p3.5.m5.1" class="ltx_Math" alttext="\alpha_{\text{JEST}}=7/3=2.33" display="inline"><semantics id="A1.SS4.p3.5.m5.1a"><mrow id="A1.SS4.p3.5.m5.1.1" xref="A1.SS4.p3.5.m5.1.1.cmml"><msub id="A1.SS4.p3.5.m5.1.1.2" xref="A1.SS4.p3.5.m5.1.1.2.cmml"><mi id="A1.SS4.p3.5.m5.1.1.2.2" xref="A1.SS4.p3.5.m5.1.1.2.2.cmml">α</mi><mtext id="A1.SS4.p3.5.m5.1.1.2.3" xref="A1.SS4.p3.5.m5.1.1.2.3a.cmml">JEST</mtext></msub><mo id="A1.SS4.p3.5.m5.1.1.3" xref="A1.SS4.p3.5.m5.1.1.3.cmml">=</mo><mrow id="A1.SS4.p3.5.m5.1.1.4" xref="A1.SS4.p3.5.m5.1.1.4.cmml"><mn id="A1.SS4.p3.5.m5.1.1.4.2" xref="A1.SS4.p3.5.m5.1.1.4.2.cmml">7</mn><mo id="A1.SS4.p3.5.m5.1.1.4.1" xref="A1.SS4.p3.5.m5.1.1.4.1.cmml">/</mo><mn id="A1.SS4.p3.5.m5.1.1.4.3" xref="A1.SS4.p3.5.m5.1.1.4.3.cmml">3</mn></mrow><mo id="A1.SS4.p3.5.m5.1.1.5" xref="A1.SS4.p3.5.m5.1.1.5.cmml">=</mo><mn id="A1.SS4.p3.5.m5.1.1.6" xref="A1.SS4.p3.5.m5.1.1.6.cmml">2.33</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p3.5.m5.1b"><apply id="A1.SS4.p3.5.m5.1.1.cmml" xref="A1.SS4.p3.5.m5.1.1"><and id="A1.SS4.p3.5.m5.1.1a.cmml" xref="A1.SS4.p3.5.m5.1.1"></and><apply id="A1.SS4.p3.5.m5.1.1b.cmml" xref="A1.SS4.p3.5.m5.1.1"><eq id="A1.SS4.p3.5.m5.1.1.3.cmml" xref="A1.SS4.p3.5.m5.1.1.3"></eq><apply id="A1.SS4.p3.5.m5.1.1.2.cmml" xref="A1.SS4.p3.5.m5.1.1.2"><csymbol cd="ambiguous" id="A1.SS4.p3.5.m5.1.1.2.1.cmml" xref="A1.SS4.p3.5.m5.1.1.2">subscript</csymbol><ci id="A1.SS4.p3.5.m5.1.1.2.2.cmml" xref="A1.SS4.p3.5.m5.1.1.2.2">𝛼</ci><ci id="A1.SS4.p3.5.m5.1.1.2.3a.cmml" xref="A1.SS4.p3.5.m5.1.1.2.3"><mtext mathsize="70%" id="A1.SS4.p3.5.m5.1.1.2.3.cmml" xref="A1.SS4.p3.5.m5.1.1.2.3">JEST</mtext></ci></apply><apply id="A1.SS4.p3.5.m5.1.1.4.cmml" xref="A1.SS4.p3.5.m5.1.1.4"><divide id="A1.SS4.p3.5.m5.1.1.4.1.cmml" xref="A1.SS4.p3.5.m5.1.1.4.1"></divide><cn type="integer" id="A1.SS4.p3.5.m5.1.1.4.2.cmml" xref="A1.SS4.p3.5.m5.1.1.4.2">7</cn><cn type="integer" id="A1.SS4.p3.5.m5.1.1.4.3.cmml" xref="A1.SS4.p3.5.m5.1.1.4.3">3</cn></apply></apply><apply id="A1.SS4.p3.5.m5.1.1c.cmml" xref="A1.SS4.p3.5.m5.1.1"><eq id="A1.SS4.p3.5.m5.1.1.5.cmml" xref="A1.SS4.p3.5.m5.1.1.5"></eq><share href="#A1.SS4.p3.5.m5.1.1.4.cmml" id="A1.SS4.p3.5.m5.1.1d.cmml" xref="A1.SS4.p3.5.m5.1.1"></share><cn type="float" id="A1.SS4.p3.5.m5.1.1.6.cmml" xref="A1.SS4.p3.5.m5.1.1.6">2.33</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p3.5.m5.1c">\alpha_{\text{JEST}}=7/3=2.33</annotation></semantics></math>. Flexi-JEST uses two approximations. Firstly, we split the training batch 50:50 between the full:approximate learner, effectively parallelising the method of <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite> and reducing the per-data point cost of training. Secondly, we approximate the learner when performing scoring, which reduces the cost of scoring. The overall cost for a single Flexi-JEST update can therefore be computed as:</p>
</div>
<div id="A1.SS4.p4" class="ltx_para ltx_noindent">
<table id="A1.Ex2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.Ex2.m1.1" class="ltx_Math" alttext="C_{\text{Flexi-JEST}}=3F(0.5+0.5A)+AFB/b" display="block"><semantics id="A1.Ex2.m1.1a"><mrow id="A1.Ex2.m1.1.1" xref="A1.Ex2.m1.1.1.cmml"><msub id="A1.Ex2.m1.1.1.3" xref="A1.Ex2.m1.1.1.3.cmml"><mi id="A1.Ex2.m1.1.1.3.2" xref="A1.Ex2.m1.1.1.3.2.cmml">C</mi><mtext id="A1.Ex2.m1.1.1.3.3" xref="A1.Ex2.m1.1.1.3.3a.cmml">Flexi-JEST</mtext></msub><mo id="A1.Ex2.m1.1.1.2" xref="A1.Ex2.m1.1.1.2.cmml">=</mo><mrow id="A1.Ex2.m1.1.1.1" xref="A1.Ex2.m1.1.1.1.cmml"><mrow id="A1.Ex2.m1.1.1.1.1" xref="A1.Ex2.m1.1.1.1.1.cmml"><mn id="A1.Ex2.m1.1.1.1.1.3" xref="A1.Ex2.m1.1.1.1.1.3.cmml">3</mn><mo lspace="0em" rspace="0em" id="A1.Ex2.m1.1.1.1.1.2" xref="A1.Ex2.m1.1.1.1.1.2.cmml">​</mo><mi id="A1.Ex2.m1.1.1.1.1.4" xref="A1.Ex2.m1.1.1.1.1.4.cmml">F</mi><mo lspace="0em" rspace="0em" id="A1.Ex2.m1.1.1.1.1.2a" xref="A1.Ex2.m1.1.1.1.1.2.cmml">​</mo><mrow id="A1.Ex2.m1.1.1.1.1.1.1" xref="A1.Ex2.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.Ex2.m1.1.1.1.1.1.1.2" xref="A1.Ex2.m1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.Ex2.m1.1.1.1.1.1.1.1" xref="A1.Ex2.m1.1.1.1.1.1.1.1.cmml"><mn id="A1.Ex2.m1.1.1.1.1.1.1.1.2" xref="A1.Ex2.m1.1.1.1.1.1.1.1.2.cmml">0.5</mn><mo id="A1.Ex2.m1.1.1.1.1.1.1.1.1" xref="A1.Ex2.m1.1.1.1.1.1.1.1.1.cmml">+</mo><mrow id="A1.Ex2.m1.1.1.1.1.1.1.1.3" xref="A1.Ex2.m1.1.1.1.1.1.1.1.3.cmml"><mn id="A1.Ex2.m1.1.1.1.1.1.1.1.3.2" xref="A1.Ex2.m1.1.1.1.1.1.1.1.3.2.cmml">0.5</mn><mo lspace="0em" rspace="0em" id="A1.Ex2.m1.1.1.1.1.1.1.1.3.1" xref="A1.Ex2.m1.1.1.1.1.1.1.1.3.1.cmml">​</mo><mi id="A1.Ex2.m1.1.1.1.1.1.1.1.3.3" xref="A1.Ex2.m1.1.1.1.1.1.1.1.3.3.cmml">A</mi></mrow></mrow><mo stretchy="false" id="A1.Ex2.m1.1.1.1.1.1.1.3" xref="A1.Ex2.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A1.Ex2.m1.1.1.1.2" xref="A1.Ex2.m1.1.1.1.2.cmml">+</mo><mrow id="A1.Ex2.m1.1.1.1.3" xref="A1.Ex2.m1.1.1.1.3.cmml"><mrow id="A1.Ex2.m1.1.1.1.3.2" xref="A1.Ex2.m1.1.1.1.3.2.cmml"><mi id="A1.Ex2.m1.1.1.1.3.2.2" xref="A1.Ex2.m1.1.1.1.3.2.2.cmml">A</mi><mo lspace="0em" rspace="0em" id="A1.Ex2.m1.1.1.1.3.2.1" xref="A1.Ex2.m1.1.1.1.3.2.1.cmml">​</mo><mi id="A1.Ex2.m1.1.1.1.3.2.3" xref="A1.Ex2.m1.1.1.1.3.2.3.cmml">F</mi><mo lspace="0em" rspace="0em" id="A1.Ex2.m1.1.1.1.3.2.1a" xref="A1.Ex2.m1.1.1.1.3.2.1.cmml">​</mo><mi id="A1.Ex2.m1.1.1.1.3.2.4" xref="A1.Ex2.m1.1.1.1.3.2.4.cmml">B</mi></mrow><mo id="A1.Ex2.m1.1.1.1.3.1" xref="A1.Ex2.m1.1.1.1.3.1.cmml">/</mo><mi id="A1.Ex2.m1.1.1.1.3.3" xref="A1.Ex2.m1.1.1.1.3.3.cmml">b</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex2.m1.1b"><apply id="A1.Ex2.m1.1.1.cmml" xref="A1.Ex2.m1.1.1"><eq id="A1.Ex2.m1.1.1.2.cmml" xref="A1.Ex2.m1.1.1.2"></eq><apply id="A1.Ex2.m1.1.1.3.cmml" xref="A1.Ex2.m1.1.1.3"><csymbol cd="ambiguous" id="A1.Ex2.m1.1.1.3.1.cmml" xref="A1.Ex2.m1.1.1.3">subscript</csymbol><ci id="A1.Ex2.m1.1.1.3.2.cmml" xref="A1.Ex2.m1.1.1.3.2">𝐶</ci><ci id="A1.Ex2.m1.1.1.3.3a.cmml" xref="A1.Ex2.m1.1.1.3.3"><mtext mathsize="70%" id="A1.Ex2.m1.1.1.3.3.cmml" xref="A1.Ex2.m1.1.1.3.3">Flexi-JEST</mtext></ci></apply><apply id="A1.Ex2.m1.1.1.1.cmml" xref="A1.Ex2.m1.1.1.1"><plus id="A1.Ex2.m1.1.1.1.2.cmml" xref="A1.Ex2.m1.1.1.1.2"></plus><apply id="A1.Ex2.m1.1.1.1.1.cmml" xref="A1.Ex2.m1.1.1.1.1"><times id="A1.Ex2.m1.1.1.1.1.2.cmml" xref="A1.Ex2.m1.1.1.1.1.2"></times><cn type="integer" id="A1.Ex2.m1.1.1.1.1.3.cmml" xref="A1.Ex2.m1.1.1.1.1.3">3</cn><ci id="A1.Ex2.m1.1.1.1.1.4.cmml" xref="A1.Ex2.m1.1.1.1.1.4">𝐹</ci><apply id="A1.Ex2.m1.1.1.1.1.1.1.1.cmml" xref="A1.Ex2.m1.1.1.1.1.1.1"><plus id="A1.Ex2.m1.1.1.1.1.1.1.1.1.cmml" xref="A1.Ex2.m1.1.1.1.1.1.1.1.1"></plus><cn type="float" id="A1.Ex2.m1.1.1.1.1.1.1.1.2.cmml" xref="A1.Ex2.m1.1.1.1.1.1.1.1.2">0.5</cn><apply id="A1.Ex2.m1.1.1.1.1.1.1.1.3.cmml" xref="A1.Ex2.m1.1.1.1.1.1.1.1.3"><times id="A1.Ex2.m1.1.1.1.1.1.1.1.3.1.cmml" xref="A1.Ex2.m1.1.1.1.1.1.1.1.3.1"></times><cn type="float" id="A1.Ex2.m1.1.1.1.1.1.1.1.3.2.cmml" xref="A1.Ex2.m1.1.1.1.1.1.1.1.3.2">0.5</cn><ci id="A1.Ex2.m1.1.1.1.1.1.1.1.3.3.cmml" xref="A1.Ex2.m1.1.1.1.1.1.1.1.3.3">𝐴</ci></apply></apply></apply><apply id="A1.Ex2.m1.1.1.1.3.cmml" xref="A1.Ex2.m1.1.1.1.3"><divide id="A1.Ex2.m1.1.1.1.3.1.cmml" xref="A1.Ex2.m1.1.1.1.3.1"></divide><apply id="A1.Ex2.m1.1.1.1.3.2.cmml" xref="A1.Ex2.m1.1.1.1.3.2"><times id="A1.Ex2.m1.1.1.1.3.2.1.cmml" xref="A1.Ex2.m1.1.1.1.3.2.1"></times><ci id="A1.Ex2.m1.1.1.1.3.2.2.cmml" xref="A1.Ex2.m1.1.1.1.3.2.2">𝐴</ci><ci id="A1.Ex2.m1.1.1.1.3.2.3.cmml" xref="A1.Ex2.m1.1.1.1.3.2.3">𝐹</ci><ci id="A1.Ex2.m1.1.1.1.3.2.4.cmml" xref="A1.Ex2.m1.1.1.1.3.2.4">𝐵</ci></apply><ci id="A1.Ex2.m1.1.1.1.3.3.cmml" xref="A1.Ex2.m1.1.1.1.3.3">𝑏</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex2.m1.1c">C_{\text{Flexi-JEST}}=3F(0.5+0.5A)+AFB/b</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="A1.SS4.p5" class="ltx_para ltx_noindent">
<p id="A1.SS4.p5.8" class="ltx_p">where <math id="A1.SS4.p5.1.m1.1" class="ltx_Math" alttext="A" display="inline"><semantics id="A1.SS4.p5.1.m1.1a"><mi id="A1.SS4.p5.1.m1.1.1" xref="A1.SS4.p5.1.m1.1.1.cmml">A</mi><annotation-xml encoding="MathML-Content" id="A1.SS4.p5.1.m1.1b"><ci id="A1.SS4.p5.1.m1.1.1.cmml" xref="A1.SS4.p5.1.m1.1.1">𝐴</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p5.1.m1.1c">A</annotation></semantics></math> is the FLOP reduction factor resulting from model approximation (i.e. increasing the patch size, see <a href="#S3.SS3" title="3.3 Efficient scoring and multi-resolution training ‣ 3 Methods ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>). Note that we can no longer cache the forward pass from scoring since it is computed with an approximate version of the learner. Relative to an IID update, the cost of a single Flexi-JEST iteration at a filtering ratio <math id="A1.SS4.p5.2.m2.1" class="ltx_Math" alttext="f=0.8" display="inline"><semantics id="A1.SS4.p5.2.m2.1a"><mrow id="A1.SS4.p5.2.m2.1.1" xref="A1.SS4.p5.2.m2.1.1.cmml"><mi id="A1.SS4.p5.2.m2.1.1.2" xref="A1.SS4.p5.2.m2.1.1.2.cmml">f</mi><mo id="A1.SS4.p5.2.m2.1.1.1" xref="A1.SS4.p5.2.m2.1.1.1.cmml">=</mo><mn id="A1.SS4.p5.2.m2.1.1.3" xref="A1.SS4.p5.2.m2.1.1.3.cmml">0.8</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p5.2.m2.1b"><apply id="A1.SS4.p5.2.m2.1.1.cmml" xref="A1.SS4.p5.2.m2.1.1"><eq id="A1.SS4.p5.2.m2.1.1.1.cmml" xref="A1.SS4.p5.2.m2.1.1.1"></eq><ci id="A1.SS4.p5.2.m2.1.1.2.cmml" xref="A1.SS4.p5.2.m2.1.1.2">𝑓</ci><cn type="float" id="A1.SS4.p5.2.m2.1.1.3.cmml" xref="A1.SS4.p5.2.m2.1.1.3">0.8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p5.2.m2.1c">f=0.8</annotation></semantics></math> and approximation factor <math id="A1.SS4.p5.3.m3.1" class="ltx_Math" alttext="A=0.25" display="inline"><semantics id="A1.SS4.p5.3.m3.1a"><mrow id="A1.SS4.p5.3.m3.1.1" xref="A1.SS4.p5.3.m3.1.1.cmml"><mi id="A1.SS4.p5.3.m3.1.1.2" xref="A1.SS4.p5.3.m3.1.1.2.cmml">A</mi><mo id="A1.SS4.p5.3.m3.1.1.1" xref="A1.SS4.p5.3.m3.1.1.1.cmml">=</mo><mn id="A1.SS4.p5.3.m3.1.1.3" xref="A1.SS4.p5.3.m3.1.1.3.cmml">0.25</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p5.3.m3.1b"><apply id="A1.SS4.p5.3.m3.1.1.cmml" xref="A1.SS4.p5.3.m3.1.1"><eq id="A1.SS4.p5.3.m3.1.1.1.cmml" xref="A1.SS4.p5.3.m3.1.1.1"></eq><ci id="A1.SS4.p5.3.m3.1.1.2.cmml" xref="A1.SS4.p5.3.m3.1.1.2">𝐴</ci><cn type="float" id="A1.SS4.p5.3.m3.1.1.3.cmml" xref="A1.SS4.p5.3.m3.1.1.3">0.25</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p5.3.m3.1c">A=0.25</annotation></semantics></math> comes out as <math id="A1.SS4.p5.4.m4.1" class="ltx_Math" alttext="\alpha_{\text{Flexi-JEST}}=1.04" display="inline"><semantics id="A1.SS4.p5.4.m4.1a"><mrow id="A1.SS4.p5.4.m4.1.1" xref="A1.SS4.p5.4.m4.1.1.cmml"><msub id="A1.SS4.p5.4.m4.1.1.2" xref="A1.SS4.p5.4.m4.1.1.2.cmml"><mi id="A1.SS4.p5.4.m4.1.1.2.2" xref="A1.SS4.p5.4.m4.1.1.2.2.cmml">α</mi><mtext id="A1.SS4.p5.4.m4.1.1.2.3" xref="A1.SS4.p5.4.m4.1.1.2.3a.cmml">Flexi-JEST</mtext></msub><mo id="A1.SS4.p5.4.m4.1.1.1" xref="A1.SS4.p5.4.m4.1.1.1.cmml">=</mo><mn id="A1.SS4.p5.4.m4.1.1.3" xref="A1.SS4.p5.4.m4.1.1.3.cmml">1.04</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p5.4.m4.1b"><apply id="A1.SS4.p5.4.m4.1.1.cmml" xref="A1.SS4.p5.4.m4.1.1"><eq id="A1.SS4.p5.4.m4.1.1.1.cmml" xref="A1.SS4.p5.4.m4.1.1.1"></eq><apply id="A1.SS4.p5.4.m4.1.1.2.cmml" xref="A1.SS4.p5.4.m4.1.1.2"><csymbol cd="ambiguous" id="A1.SS4.p5.4.m4.1.1.2.1.cmml" xref="A1.SS4.p5.4.m4.1.1.2">subscript</csymbol><ci id="A1.SS4.p5.4.m4.1.1.2.2.cmml" xref="A1.SS4.p5.4.m4.1.1.2.2">𝛼</ci><ci id="A1.SS4.p5.4.m4.1.1.2.3a.cmml" xref="A1.SS4.p5.4.m4.1.1.2.3"><mtext mathsize="70%" id="A1.SS4.p5.4.m4.1.1.2.3.cmml" xref="A1.SS4.p5.4.m4.1.1.2.3">Flexi-JEST</mtext></ci></apply><cn type="float" id="A1.SS4.p5.4.m4.1.1.3.cmml" xref="A1.SS4.p5.4.m4.1.1.3">1.04</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p5.4.m4.1c">\alpha_{\text{Flexi-JEST}}=1.04</annotation></semantics></math>. In practice, <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib29" title="" class="ltx_ref">29</a>]</cite> estimated the FLOP reduction from a doubling in patch size as closer to <math id="A1.SS4.p5.5.m5.1" class="ltx_Math" alttext="A=0.28" display="inline"><semantics id="A1.SS4.p5.5.m5.1a"><mrow id="A1.SS4.p5.5.m5.1.1" xref="A1.SS4.p5.5.m5.1.1.cmml"><mi id="A1.SS4.p5.5.m5.1.1.2" xref="A1.SS4.p5.5.m5.1.1.2.cmml">A</mi><mo id="A1.SS4.p5.5.m5.1.1.1" xref="A1.SS4.p5.5.m5.1.1.1.cmml">=</mo><mn id="A1.SS4.p5.5.m5.1.1.3" xref="A1.SS4.p5.5.m5.1.1.3.cmml">0.28</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p5.5.m5.1b"><apply id="A1.SS4.p5.5.m5.1.1.cmml" xref="A1.SS4.p5.5.m5.1.1"><eq id="A1.SS4.p5.5.m5.1.1.1.cmml" xref="A1.SS4.p5.5.m5.1.1.1"></eq><ci id="A1.SS4.p5.5.m5.1.1.2.cmml" xref="A1.SS4.p5.5.m5.1.1.2">𝐴</ci><cn type="float" id="A1.SS4.p5.5.m5.1.1.3.cmml" xref="A1.SS4.p5.5.m5.1.1.3">0.28</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p5.5.m5.1c">A=0.28</annotation></semantics></math>, which is slightly higher than the <math id="A1.SS4.p5.6.m6.1" class="ltx_Math" alttext="A=0.25" display="inline"><semantics id="A1.SS4.p5.6.m6.1a"><mrow id="A1.SS4.p5.6.m6.1.1" xref="A1.SS4.p5.6.m6.1.1.cmml"><mi id="A1.SS4.p5.6.m6.1.1.2" xref="A1.SS4.p5.6.m6.1.1.2.cmml">A</mi><mo id="A1.SS4.p5.6.m6.1.1.1" xref="A1.SS4.p5.6.m6.1.1.1.cmml">=</mo><mn id="A1.SS4.p5.6.m6.1.1.3" xref="A1.SS4.p5.6.m6.1.1.3.cmml">0.25</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p5.6.m6.1b"><apply id="A1.SS4.p5.6.m6.1.1.cmml" xref="A1.SS4.p5.6.m6.1.1"><eq id="A1.SS4.p5.6.m6.1.1.1.cmml" xref="A1.SS4.p5.6.m6.1.1.1"></eq><ci id="A1.SS4.p5.6.m6.1.1.2.cmml" xref="A1.SS4.p5.6.m6.1.1.2">𝐴</ci><cn type="float" id="A1.SS4.p5.6.m6.1.1.3.cmml" xref="A1.SS4.p5.6.m6.1.1.3">0.25</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p5.6.m6.1c">A=0.25</annotation></semantics></math> expected by reducing the number of patches by <math id="A1.SS4.p5.7.m7.1" class="ltx_Math" alttext="0.5^{2}" display="inline"><semantics id="A1.SS4.p5.7.m7.1a"><msup id="A1.SS4.p5.7.m7.1.1" xref="A1.SS4.p5.7.m7.1.1.cmml"><mn id="A1.SS4.p5.7.m7.1.1.2" xref="A1.SS4.p5.7.m7.1.1.2.cmml">0.5</mn><mn id="A1.SS4.p5.7.m7.1.1.3" xref="A1.SS4.p5.7.m7.1.1.3.cmml">2</mn></msup><annotation-xml encoding="MathML-Content" id="A1.SS4.p5.7.m7.1b"><apply id="A1.SS4.p5.7.m7.1.1.cmml" xref="A1.SS4.p5.7.m7.1.1"><csymbol cd="ambiguous" id="A1.SS4.p5.7.m7.1.1.1.cmml" xref="A1.SS4.p5.7.m7.1.1">superscript</csymbol><cn type="float" id="A1.SS4.p5.7.m7.1.1.2.cmml" xref="A1.SS4.p5.7.m7.1.1.2">0.5</cn><cn type="integer" id="A1.SS4.p5.7.m7.1.1.3.cmml" xref="A1.SS4.p5.7.m7.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p5.7.m7.1c">0.5^{2}</annotation></semantics></math>. We use this more conservative calculation (<math id="A1.SS4.p5.8.m8.1" class="ltx_Math" alttext="\alpha_{\text{Flexi-JEST}}=1.10" display="inline"><semantics id="A1.SS4.p5.8.m8.1a"><mrow id="A1.SS4.p5.8.m8.1.1" xref="A1.SS4.p5.8.m8.1.1.cmml"><msub id="A1.SS4.p5.8.m8.1.1.2" xref="A1.SS4.p5.8.m8.1.1.2.cmml"><mi id="A1.SS4.p5.8.m8.1.1.2.2" xref="A1.SS4.p5.8.m8.1.1.2.2.cmml">α</mi><mtext id="A1.SS4.p5.8.m8.1.1.2.3" xref="A1.SS4.p5.8.m8.1.1.2.3a.cmml">Flexi-JEST</mtext></msub><mo id="A1.SS4.p5.8.m8.1.1.1" xref="A1.SS4.p5.8.m8.1.1.1.cmml">=</mo><mn id="A1.SS4.p5.8.m8.1.1.3" xref="A1.SS4.p5.8.m8.1.1.3.cmml">1.10</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p5.8.m8.1b"><apply id="A1.SS4.p5.8.m8.1.1.cmml" xref="A1.SS4.p5.8.m8.1.1"><eq id="A1.SS4.p5.8.m8.1.1.1.cmml" xref="A1.SS4.p5.8.m8.1.1.1"></eq><apply id="A1.SS4.p5.8.m8.1.1.2.cmml" xref="A1.SS4.p5.8.m8.1.1.2"><csymbol cd="ambiguous" id="A1.SS4.p5.8.m8.1.1.2.1.cmml" xref="A1.SS4.p5.8.m8.1.1.2">subscript</csymbol><ci id="A1.SS4.p5.8.m8.1.1.2.2.cmml" xref="A1.SS4.p5.8.m8.1.1.2.2">𝛼</ci><ci id="A1.SS4.p5.8.m8.1.1.2.3a.cmml" xref="A1.SS4.p5.8.m8.1.1.2.3"><mtext mathsize="70%" id="A1.SS4.p5.8.m8.1.1.2.3.cmml" xref="A1.SS4.p5.8.m8.1.1.2.3">Flexi-JEST</mtext></ci></apply><cn type="float" id="A1.SS4.p5.8.m8.1.1.3.cmml" xref="A1.SS4.p5.8.m8.1.1.3">1.10</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p5.8.m8.1c">\alpha_{\text{Flexi-JEST}}=1.10</annotation></semantics></math>) throughout.</p>
</div>
</section>
<section id="A1.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.5 </span>Caching reference model scores</h3>

<div id="A1.SS5.p1" class="ltx_para ltx_noindent">
<p id="A1.SS5.p1.1" class="ltx_p">Since the reference model is pretrained and fixed, its scores do not vary over the course of a training run and can be cached within the dataset. For independent example selection we only need to store the scalar scores. However, since data is not likely to be sampled from the training set in the same order in which it is initially scored (e.g. if the batch size varies), the batch composition is unknown ahead of training, which will affect the computation of the scores.</p>
</div>
<div id="A1.SS5.p2" class="ltx_para ltx_noindent">
<p id="A1.SS5.p2.1" class="ltx_p">To amortize the cost of reference model scoring across training runs for joint example selection, we therefore instead store the <span id="A1.SS5.p2.1.1" class="ltx_text ltx_font_italic">embeddings</span> from the reference mode. For a ViT-B/16, the embeddings (text and image) are of size 768, which are considerably smaller but not negligible in comparison to the raw data points. Given these embeddings, the super-batch contrastive matrix can be recomputed before sub-sampling to obtain the sub-batch.</p>
</div>
</section>
<section id="A1.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.6 </span>Contrastive loss ablations</h3>

<div id="A1.SS6.p1" class="ltx_para ltx_noindent">
<p id="A1.SS6.p1.1" class="ltx_p">Contrastive learning maximizes the alignment of image and text modalities for paired examples, while minimizing the alignment of unpaired examples, with batch-level losses <math id="A1.SS6.p1.1.m1.4" class="ltx_Math" alttext="\ell(\mathcal{B}|\theta)=\frac{1}{b}\sum_{i=1}^{b}\ell({\bm{x}}_{i}|\theta,\mathcal{B})" display="inline"><semantics id="A1.SS6.p1.1.m1.4a"><mrow id="A1.SS6.p1.1.m1.4.4" xref="A1.SS6.p1.1.m1.4.4.cmml"><mrow id="A1.SS6.p1.1.m1.3.3.1" xref="A1.SS6.p1.1.m1.3.3.1.cmml"><mi mathvariant="normal" id="A1.SS6.p1.1.m1.3.3.1.3" xref="A1.SS6.p1.1.m1.3.3.1.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="A1.SS6.p1.1.m1.3.3.1.2" xref="A1.SS6.p1.1.m1.3.3.1.2.cmml">​</mo><mrow id="A1.SS6.p1.1.m1.3.3.1.1.1" xref="A1.SS6.p1.1.m1.3.3.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS6.p1.1.m1.3.3.1.1.1.2" xref="A1.SS6.p1.1.m1.3.3.1.1.1.1.cmml">(</mo><mrow id="A1.SS6.p1.1.m1.3.3.1.1.1.1" xref="A1.SS6.p1.1.m1.3.3.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.SS6.p1.1.m1.3.3.1.1.1.1.2" xref="A1.SS6.p1.1.m1.3.3.1.1.1.1.2.cmml">ℬ</mi><mo fence="false" id="A1.SS6.p1.1.m1.3.3.1.1.1.1.1" xref="A1.SS6.p1.1.m1.3.3.1.1.1.1.1.cmml">|</mo><mi id="A1.SS6.p1.1.m1.3.3.1.1.1.1.3" xref="A1.SS6.p1.1.m1.3.3.1.1.1.1.3.cmml">θ</mi></mrow><mo stretchy="false" id="A1.SS6.p1.1.m1.3.3.1.1.1.3" xref="A1.SS6.p1.1.m1.3.3.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A1.SS6.p1.1.m1.4.4.3" xref="A1.SS6.p1.1.m1.4.4.3.cmml">=</mo><mrow id="A1.SS6.p1.1.m1.4.4.2" xref="A1.SS6.p1.1.m1.4.4.2.cmml"><mfrac id="A1.SS6.p1.1.m1.4.4.2.3" xref="A1.SS6.p1.1.m1.4.4.2.3.cmml"><mn id="A1.SS6.p1.1.m1.4.4.2.3.2" xref="A1.SS6.p1.1.m1.4.4.2.3.2.cmml">1</mn><mi id="A1.SS6.p1.1.m1.4.4.2.3.3" xref="A1.SS6.p1.1.m1.4.4.2.3.3.cmml">b</mi></mfrac><mo lspace="0em" rspace="0em" id="A1.SS6.p1.1.m1.4.4.2.2" xref="A1.SS6.p1.1.m1.4.4.2.2.cmml">​</mo><mrow id="A1.SS6.p1.1.m1.4.4.2.1" xref="A1.SS6.p1.1.m1.4.4.2.1.cmml"><msubsup id="A1.SS6.p1.1.m1.4.4.2.1.2" xref="A1.SS6.p1.1.m1.4.4.2.1.2.cmml"><mo id="A1.SS6.p1.1.m1.4.4.2.1.2.2.2" xref="A1.SS6.p1.1.m1.4.4.2.1.2.2.2.cmml">∑</mo><mrow id="A1.SS6.p1.1.m1.4.4.2.1.2.2.3" xref="A1.SS6.p1.1.m1.4.4.2.1.2.2.3.cmml"><mi id="A1.SS6.p1.1.m1.4.4.2.1.2.2.3.2" xref="A1.SS6.p1.1.m1.4.4.2.1.2.2.3.2.cmml">i</mi><mo id="A1.SS6.p1.1.m1.4.4.2.1.2.2.3.1" xref="A1.SS6.p1.1.m1.4.4.2.1.2.2.3.1.cmml">=</mo><mn id="A1.SS6.p1.1.m1.4.4.2.1.2.2.3.3" xref="A1.SS6.p1.1.m1.4.4.2.1.2.2.3.3.cmml">1</mn></mrow><mi id="A1.SS6.p1.1.m1.4.4.2.1.2.3" xref="A1.SS6.p1.1.m1.4.4.2.1.2.3.cmml">b</mi></msubsup><mrow id="A1.SS6.p1.1.m1.4.4.2.1.1" xref="A1.SS6.p1.1.m1.4.4.2.1.1.cmml"><mi mathvariant="normal" id="A1.SS6.p1.1.m1.4.4.2.1.1.3" xref="A1.SS6.p1.1.m1.4.4.2.1.1.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="A1.SS6.p1.1.m1.4.4.2.1.1.2" xref="A1.SS6.p1.1.m1.4.4.2.1.1.2.cmml">​</mo><mrow id="A1.SS6.p1.1.m1.4.4.2.1.1.1.1" xref="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.2" xref="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.cmml">(</mo><mrow id="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1" xref="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.cmml"><msub id="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.2" xref="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.2.cmml"><mi id="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.2.2" xref="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.2.2.cmml">𝒙</mi><mi id="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.2.3" xref="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.2.3.cmml">i</mi></msub><mo fence="false" id="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.1" xref="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.1.cmml">|</mo><mrow id="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.3.2" xref="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.3.1.cmml"><mi id="A1.SS6.p1.1.m1.1.1" xref="A1.SS6.p1.1.m1.1.1.cmml">θ</mi><mo id="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.3.2.1" xref="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.3.1.cmml">,</mo><mi class="ltx_font_mathcaligraphic" id="A1.SS6.p1.1.m1.2.2" xref="A1.SS6.p1.1.m1.2.2.cmml">ℬ</mi></mrow></mrow><mo stretchy="false" id="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.3" xref="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS6.p1.1.m1.4b"><apply id="A1.SS6.p1.1.m1.4.4.cmml" xref="A1.SS6.p1.1.m1.4.4"><eq id="A1.SS6.p1.1.m1.4.4.3.cmml" xref="A1.SS6.p1.1.m1.4.4.3"></eq><apply id="A1.SS6.p1.1.m1.3.3.1.cmml" xref="A1.SS6.p1.1.m1.3.3.1"><times id="A1.SS6.p1.1.m1.3.3.1.2.cmml" xref="A1.SS6.p1.1.m1.3.3.1.2"></times><ci id="A1.SS6.p1.1.m1.3.3.1.3.cmml" xref="A1.SS6.p1.1.m1.3.3.1.3">ℓ</ci><apply id="A1.SS6.p1.1.m1.3.3.1.1.1.1.cmml" xref="A1.SS6.p1.1.m1.3.3.1.1.1"><csymbol cd="latexml" id="A1.SS6.p1.1.m1.3.3.1.1.1.1.1.cmml" xref="A1.SS6.p1.1.m1.3.3.1.1.1.1.1">conditional</csymbol><ci id="A1.SS6.p1.1.m1.3.3.1.1.1.1.2.cmml" xref="A1.SS6.p1.1.m1.3.3.1.1.1.1.2">ℬ</ci><ci id="A1.SS6.p1.1.m1.3.3.1.1.1.1.3.cmml" xref="A1.SS6.p1.1.m1.3.3.1.1.1.1.3">𝜃</ci></apply></apply><apply id="A1.SS6.p1.1.m1.4.4.2.cmml" xref="A1.SS6.p1.1.m1.4.4.2"><times id="A1.SS6.p1.1.m1.4.4.2.2.cmml" xref="A1.SS6.p1.1.m1.4.4.2.2"></times><apply id="A1.SS6.p1.1.m1.4.4.2.3.cmml" xref="A1.SS6.p1.1.m1.4.4.2.3"><divide id="A1.SS6.p1.1.m1.4.4.2.3.1.cmml" xref="A1.SS6.p1.1.m1.4.4.2.3"></divide><cn type="integer" id="A1.SS6.p1.1.m1.4.4.2.3.2.cmml" xref="A1.SS6.p1.1.m1.4.4.2.3.2">1</cn><ci id="A1.SS6.p1.1.m1.4.4.2.3.3.cmml" xref="A1.SS6.p1.1.m1.4.4.2.3.3">𝑏</ci></apply><apply id="A1.SS6.p1.1.m1.4.4.2.1.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1"><apply id="A1.SS6.p1.1.m1.4.4.2.1.2.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.2"><csymbol cd="ambiguous" id="A1.SS6.p1.1.m1.4.4.2.1.2.1.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.2">superscript</csymbol><apply id="A1.SS6.p1.1.m1.4.4.2.1.2.2.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.2"><csymbol cd="ambiguous" id="A1.SS6.p1.1.m1.4.4.2.1.2.2.1.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.2">subscript</csymbol><sum id="A1.SS6.p1.1.m1.4.4.2.1.2.2.2.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.2.2.2"></sum><apply id="A1.SS6.p1.1.m1.4.4.2.1.2.2.3.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.2.2.3"><eq id="A1.SS6.p1.1.m1.4.4.2.1.2.2.3.1.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.2.2.3.1"></eq><ci id="A1.SS6.p1.1.m1.4.4.2.1.2.2.3.2.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.2.2.3.2">𝑖</ci><cn type="integer" id="A1.SS6.p1.1.m1.4.4.2.1.2.2.3.3.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.2.2.3.3">1</cn></apply></apply><ci id="A1.SS6.p1.1.m1.4.4.2.1.2.3.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.2.3">𝑏</ci></apply><apply id="A1.SS6.p1.1.m1.4.4.2.1.1.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.1"><times id="A1.SS6.p1.1.m1.4.4.2.1.1.2.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.1.2"></times><ci id="A1.SS6.p1.1.m1.4.4.2.1.1.3.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.1.3">ℓ</ci><apply id="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.1.1.1"><csymbol cd="latexml" id="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.1.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.1">conditional</csymbol><apply id="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.2.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.2.1.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.2">subscript</csymbol><ci id="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.2.2.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.2.2">𝒙</ci><ci id="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.2.3.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.2.3">𝑖</ci></apply><list id="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.3.1.cmml" xref="A1.SS6.p1.1.m1.4.4.2.1.1.1.1.1.3.2"><ci id="A1.SS6.p1.1.m1.1.1.cmml" xref="A1.SS6.p1.1.m1.1.1">𝜃</ci><ci id="A1.SS6.p1.1.m1.2.2.cmml" xref="A1.SS6.p1.1.m1.2.2">ℬ</ci></list></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS6.p1.1.m1.4c">\ell(\mathcal{B}|\theta)=\frac{1}{b}\sum_{i=1}^{b}\ell({\bm{x}}_{i}|\theta,\mathcal{B})</annotation></semantics></math>.</p>
</div>
<div id="A1.SS6.p2" class="ltx_para ltx_noindent">
<p id="A1.SS6.p2.3" class="ltx_p">Each data point <math id="A1.SS6.p2.1.m1.1" class="ltx_Math" alttext="{\bm{x}}_{i}" display="inline"><semantics id="A1.SS6.p2.1.m1.1a"><msub id="A1.SS6.p2.1.m1.1.1" xref="A1.SS6.p2.1.m1.1.1.cmml"><mi id="A1.SS6.p2.1.m1.1.1.2" xref="A1.SS6.p2.1.m1.1.1.2.cmml">𝒙</mi><mi id="A1.SS6.p2.1.m1.1.1.3" xref="A1.SS6.p2.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="A1.SS6.p2.1.m1.1b"><apply id="A1.SS6.p2.1.m1.1.1.cmml" xref="A1.SS6.p2.1.m1.1.1"><csymbol cd="ambiguous" id="A1.SS6.p2.1.m1.1.1.1.cmml" xref="A1.SS6.p2.1.m1.1.1">subscript</csymbol><ci id="A1.SS6.p2.1.m1.1.1.2.cmml" xref="A1.SS6.p2.1.m1.1.1.2">𝒙</ci><ci id="A1.SS6.p2.1.m1.1.1.3.cmml" xref="A1.SS6.p2.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS6.p2.1.m1.1c">{\bm{x}}_{i}</annotation></semantics></math> is comprised of an image and associated text which are embedded with their respective encoders as
<math id="A1.SS6.p2.2.m2.2" class="ltx_Math" alttext="{\bm{z}}^{\textrm{im}}_{i}=f^{\textrm{im}}({\bm{x}}_{i};\theta)" display="inline"><semantics id="A1.SS6.p2.2.m2.2a"><mrow id="A1.SS6.p2.2.m2.2.2" xref="A1.SS6.p2.2.m2.2.2.cmml"><msubsup id="A1.SS6.p2.2.m2.2.2.3" xref="A1.SS6.p2.2.m2.2.2.3.cmml"><mi id="A1.SS6.p2.2.m2.2.2.3.2.2" xref="A1.SS6.p2.2.m2.2.2.3.2.2.cmml">𝒛</mi><mi id="A1.SS6.p2.2.m2.2.2.3.3" xref="A1.SS6.p2.2.m2.2.2.3.3.cmml">i</mi><mtext id="A1.SS6.p2.2.m2.2.2.3.2.3" xref="A1.SS6.p2.2.m2.2.2.3.2.3a.cmml">im</mtext></msubsup><mo id="A1.SS6.p2.2.m2.2.2.2" xref="A1.SS6.p2.2.m2.2.2.2.cmml">=</mo><mrow id="A1.SS6.p2.2.m2.2.2.1" xref="A1.SS6.p2.2.m2.2.2.1.cmml"><msup id="A1.SS6.p2.2.m2.2.2.1.3" xref="A1.SS6.p2.2.m2.2.2.1.3.cmml"><mi id="A1.SS6.p2.2.m2.2.2.1.3.2" xref="A1.SS6.p2.2.m2.2.2.1.3.2.cmml">f</mi><mtext id="A1.SS6.p2.2.m2.2.2.1.3.3" xref="A1.SS6.p2.2.m2.2.2.1.3.3a.cmml">im</mtext></msup><mo lspace="0em" rspace="0em" id="A1.SS6.p2.2.m2.2.2.1.2" xref="A1.SS6.p2.2.m2.2.2.1.2.cmml">​</mo><mrow id="A1.SS6.p2.2.m2.2.2.1.1.1" xref="A1.SS6.p2.2.m2.2.2.1.1.2.cmml"><mo stretchy="false" id="A1.SS6.p2.2.m2.2.2.1.1.1.2" xref="A1.SS6.p2.2.m2.2.2.1.1.2.cmml">(</mo><msub id="A1.SS6.p2.2.m2.2.2.1.1.1.1" xref="A1.SS6.p2.2.m2.2.2.1.1.1.1.cmml"><mi id="A1.SS6.p2.2.m2.2.2.1.1.1.1.2" xref="A1.SS6.p2.2.m2.2.2.1.1.1.1.2.cmml">𝒙</mi><mi id="A1.SS6.p2.2.m2.2.2.1.1.1.1.3" xref="A1.SS6.p2.2.m2.2.2.1.1.1.1.3.cmml">i</mi></msub><mo id="A1.SS6.p2.2.m2.2.2.1.1.1.3" xref="A1.SS6.p2.2.m2.2.2.1.1.2.cmml">;</mo><mi id="A1.SS6.p2.2.m2.1.1" xref="A1.SS6.p2.2.m2.1.1.cmml">θ</mi><mo stretchy="false" id="A1.SS6.p2.2.m2.2.2.1.1.1.4" xref="A1.SS6.p2.2.m2.2.2.1.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS6.p2.2.m2.2b"><apply id="A1.SS6.p2.2.m2.2.2.cmml" xref="A1.SS6.p2.2.m2.2.2"><eq id="A1.SS6.p2.2.m2.2.2.2.cmml" xref="A1.SS6.p2.2.m2.2.2.2"></eq><apply id="A1.SS6.p2.2.m2.2.2.3.cmml" xref="A1.SS6.p2.2.m2.2.2.3"><csymbol cd="ambiguous" id="A1.SS6.p2.2.m2.2.2.3.1.cmml" xref="A1.SS6.p2.2.m2.2.2.3">subscript</csymbol><apply id="A1.SS6.p2.2.m2.2.2.3.2.cmml" xref="A1.SS6.p2.2.m2.2.2.3"><csymbol cd="ambiguous" id="A1.SS6.p2.2.m2.2.2.3.2.1.cmml" xref="A1.SS6.p2.2.m2.2.2.3">superscript</csymbol><ci id="A1.SS6.p2.2.m2.2.2.3.2.2.cmml" xref="A1.SS6.p2.2.m2.2.2.3.2.2">𝒛</ci><ci id="A1.SS6.p2.2.m2.2.2.3.2.3a.cmml" xref="A1.SS6.p2.2.m2.2.2.3.2.3"><mtext mathsize="70%" id="A1.SS6.p2.2.m2.2.2.3.2.3.cmml" xref="A1.SS6.p2.2.m2.2.2.3.2.3">im</mtext></ci></apply><ci id="A1.SS6.p2.2.m2.2.2.3.3.cmml" xref="A1.SS6.p2.2.m2.2.2.3.3">𝑖</ci></apply><apply id="A1.SS6.p2.2.m2.2.2.1.cmml" xref="A1.SS6.p2.2.m2.2.2.1"><times id="A1.SS6.p2.2.m2.2.2.1.2.cmml" xref="A1.SS6.p2.2.m2.2.2.1.2"></times><apply id="A1.SS6.p2.2.m2.2.2.1.3.cmml" xref="A1.SS6.p2.2.m2.2.2.1.3"><csymbol cd="ambiguous" id="A1.SS6.p2.2.m2.2.2.1.3.1.cmml" xref="A1.SS6.p2.2.m2.2.2.1.3">superscript</csymbol><ci id="A1.SS6.p2.2.m2.2.2.1.3.2.cmml" xref="A1.SS6.p2.2.m2.2.2.1.3.2">𝑓</ci><ci id="A1.SS6.p2.2.m2.2.2.1.3.3a.cmml" xref="A1.SS6.p2.2.m2.2.2.1.3.3"><mtext mathsize="70%" id="A1.SS6.p2.2.m2.2.2.1.3.3.cmml" xref="A1.SS6.p2.2.m2.2.2.1.3.3">im</mtext></ci></apply><list id="A1.SS6.p2.2.m2.2.2.1.1.2.cmml" xref="A1.SS6.p2.2.m2.2.2.1.1.1"><apply id="A1.SS6.p2.2.m2.2.2.1.1.1.1.cmml" xref="A1.SS6.p2.2.m2.2.2.1.1.1.1"><csymbol cd="ambiguous" id="A1.SS6.p2.2.m2.2.2.1.1.1.1.1.cmml" xref="A1.SS6.p2.2.m2.2.2.1.1.1.1">subscript</csymbol><ci id="A1.SS6.p2.2.m2.2.2.1.1.1.1.2.cmml" xref="A1.SS6.p2.2.m2.2.2.1.1.1.1.2">𝒙</ci><ci id="A1.SS6.p2.2.m2.2.2.1.1.1.1.3.cmml" xref="A1.SS6.p2.2.m2.2.2.1.1.1.1.3">𝑖</ci></apply><ci id="A1.SS6.p2.2.m2.1.1.cmml" xref="A1.SS6.p2.2.m2.1.1">𝜃</ci></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS6.p2.2.m2.2c">{\bm{z}}^{\textrm{im}}_{i}=f^{\textrm{im}}({\bm{x}}_{i};\theta)</annotation></semantics></math> and <math id="A1.SS6.p2.3.m3.2" class="ltx_Math" alttext="{\bm{z}}^{\textrm{txt}}_{i}=f^{\textrm{txt}}({\bm{x}}_{i};\theta)" display="inline"><semantics id="A1.SS6.p2.3.m3.2a"><mrow id="A1.SS6.p2.3.m3.2.2" xref="A1.SS6.p2.3.m3.2.2.cmml"><msubsup id="A1.SS6.p2.3.m3.2.2.3" xref="A1.SS6.p2.3.m3.2.2.3.cmml"><mi id="A1.SS6.p2.3.m3.2.2.3.2.2" xref="A1.SS6.p2.3.m3.2.2.3.2.2.cmml">𝒛</mi><mi id="A1.SS6.p2.3.m3.2.2.3.3" xref="A1.SS6.p2.3.m3.2.2.3.3.cmml">i</mi><mtext id="A1.SS6.p2.3.m3.2.2.3.2.3" xref="A1.SS6.p2.3.m3.2.2.3.2.3a.cmml">txt</mtext></msubsup><mo id="A1.SS6.p2.3.m3.2.2.2" xref="A1.SS6.p2.3.m3.2.2.2.cmml">=</mo><mrow id="A1.SS6.p2.3.m3.2.2.1" xref="A1.SS6.p2.3.m3.2.2.1.cmml"><msup id="A1.SS6.p2.3.m3.2.2.1.3" xref="A1.SS6.p2.3.m3.2.2.1.3.cmml"><mi id="A1.SS6.p2.3.m3.2.2.1.3.2" xref="A1.SS6.p2.3.m3.2.2.1.3.2.cmml">f</mi><mtext id="A1.SS6.p2.3.m3.2.2.1.3.3" xref="A1.SS6.p2.3.m3.2.2.1.3.3a.cmml">txt</mtext></msup><mo lspace="0em" rspace="0em" id="A1.SS6.p2.3.m3.2.2.1.2" xref="A1.SS6.p2.3.m3.2.2.1.2.cmml">​</mo><mrow id="A1.SS6.p2.3.m3.2.2.1.1.1" xref="A1.SS6.p2.3.m3.2.2.1.1.2.cmml"><mo stretchy="false" id="A1.SS6.p2.3.m3.2.2.1.1.1.2" xref="A1.SS6.p2.3.m3.2.2.1.1.2.cmml">(</mo><msub id="A1.SS6.p2.3.m3.2.2.1.1.1.1" xref="A1.SS6.p2.3.m3.2.2.1.1.1.1.cmml"><mi id="A1.SS6.p2.3.m3.2.2.1.1.1.1.2" xref="A1.SS6.p2.3.m3.2.2.1.1.1.1.2.cmml">𝒙</mi><mi id="A1.SS6.p2.3.m3.2.2.1.1.1.1.3" xref="A1.SS6.p2.3.m3.2.2.1.1.1.1.3.cmml">i</mi></msub><mo id="A1.SS6.p2.3.m3.2.2.1.1.1.3" xref="A1.SS6.p2.3.m3.2.2.1.1.2.cmml">;</mo><mi id="A1.SS6.p2.3.m3.1.1" xref="A1.SS6.p2.3.m3.1.1.cmml">θ</mi><mo stretchy="false" id="A1.SS6.p2.3.m3.2.2.1.1.1.4" xref="A1.SS6.p2.3.m3.2.2.1.1.2.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS6.p2.3.m3.2b"><apply id="A1.SS6.p2.3.m3.2.2.cmml" xref="A1.SS6.p2.3.m3.2.2"><eq id="A1.SS6.p2.3.m3.2.2.2.cmml" xref="A1.SS6.p2.3.m3.2.2.2"></eq><apply id="A1.SS6.p2.3.m3.2.2.3.cmml" xref="A1.SS6.p2.3.m3.2.2.3"><csymbol cd="ambiguous" id="A1.SS6.p2.3.m3.2.2.3.1.cmml" xref="A1.SS6.p2.3.m3.2.2.3">subscript</csymbol><apply id="A1.SS6.p2.3.m3.2.2.3.2.cmml" xref="A1.SS6.p2.3.m3.2.2.3"><csymbol cd="ambiguous" id="A1.SS6.p2.3.m3.2.2.3.2.1.cmml" xref="A1.SS6.p2.3.m3.2.2.3">superscript</csymbol><ci id="A1.SS6.p2.3.m3.2.2.3.2.2.cmml" xref="A1.SS6.p2.3.m3.2.2.3.2.2">𝒛</ci><ci id="A1.SS6.p2.3.m3.2.2.3.2.3a.cmml" xref="A1.SS6.p2.3.m3.2.2.3.2.3"><mtext mathsize="70%" id="A1.SS6.p2.3.m3.2.2.3.2.3.cmml" xref="A1.SS6.p2.3.m3.2.2.3.2.3">txt</mtext></ci></apply><ci id="A1.SS6.p2.3.m3.2.2.3.3.cmml" xref="A1.SS6.p2.3.m3.2.2.3.3">𝑖</ci></apply><apply id="A1.SS6.p2.3.m3.2.2.1.cmml" xref="A1.SS6.p2.3.m3.2.2.1"><times id="A1.SS6.p2.3.m3.2.2.1.2.cmml" xref="A1.SS6.p2.3.m3.2.2.1.2"></times><apply id="A1.SS6.p2.3.m3.2.2.1.3.cmml" xref="A1.SS6.p2.3.m3.2.2.1.3"><csymbol cd="ambiguous" id="A1.SS6.p2.3.m3.2.2.1.3.1.cmml" xref="A1.SS6.p2.3.m3.2.2.1.3">superscript</csymbol><ci id="A1.SS6.p2.3.m3.2.2.1.3.2.cmml" xref="A1.SS6.p2.3.m3.2.2.1.3.2">𝑓</ci><ci id="A1.SS6.p2.3.m3.2.2.1.3.3a.cmml" xref="A1.SS6.p2.3.m3.2.2.1.3.3"><mtext mathsize="70%" id="A1.SS6.p2.3.m3.2.2.1.3.3.cmml" xref="A1.SS6.p2.3.m3.2.2.1.3.3">txt</mtext></ci></apply><list id="A1.SS6.p2.3.m3.2.2.1.1.2.cmml" xref="A1.SS6.p2.3.m3.2.2.1.1.1"><apply id="A1.SS6.p2.3.m3.2.2.1.1.1.1.cmml" xref="A1.SS6.p2.3.m3.2.2.1.1.1.1"><csymbol cd="ambiguous" id="A1.SS6.p2.3.m3.2.2.1.1.1.1.1.cmml" xref="A1.SS6.p2.3.m3.2.2.1.1.1.1">subscript</csymbol><ci id="A1.SS6.p2.3.m3.2.2.1.1.1.1.2.cmml" xref="A1.SS6.p2.3.m3.2.2.1.1.1.1.2">𝒙</ci><ci id="A1.SS6.p2.3.m3.2.2.1.1.1.1.3.cmml" xref="A1.SS6.p2.3.m3.2.2.1.1.1.1.3">𝑖</ci></apply><ci id="A1.SS6.p2.3.m3.1.1.cmml" xref="A1.SS6.p2.3.m3.1.1">𝜃</ci></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS6.p2.3.m3.2c">{\bm{z}}^{\textrm{txt}}_{i}=f^{\textrm{txt}}({\bm{x}}_{i};\theta)</annotation></semantics></math>.</p>
</div>
<div id="A1.SS6.p3" class="ltx_para ltx_noindent">
<p id="A1.SS6.p3.1" class="ltx_p">In softmax-contrastive learning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib38" title="" class="ltx_ref">38</a>]</cite>, the conditional loss is</p>
<table id="A1.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E1.m1.12" class="ltx_Math" alttext="\ell({\bm{x}}_{i}|\theta,\mathcal{B})=-\frac{1}{2}\left(\log\frac{\exp(\alpha{\bm{z}}^{\textrm{im}}_{i}{\cdot}{\bm{z}}^{\textrm{txt}}_{i})}{\sum_{j}\exp(\alpha{\bm{z}}^{\textrm{im}}_{i}{\cdot}{\bm{z}}^{\textrm{txt}}_{j})}+\log\frac{\exp(\alpha{\bm{z}}^{\textrm{im}}_{i}{\cdot}{\bm{z}}^{\textrm{txt}}_{i})}{\sum_{j}\exp(\alpha{\bm{z}}^{\textrm{txt}}_{i}{\cdot}{\bm{z}}^{\textrm{im}}_{j})}\right)" display="block"><semantics id="A1.E1.m1.12a"><mrow id="A1.E1.m1.12.12" xref="A1.E1.m1.12.12.cmml"><mrow id="A1.E1.m1.11.11.1" xref="A1.E1.m1.11.11.1.cmml"><mi mathvariant="normal" id="A1.E1.m1.11.11.1.3" xref="A1.E1.m1.11.11.1.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="A1.E1.m1.11.11.1.2" xref="A1.E1.m1.11.11.1.2.cmml">​</mo><mrow id="A1.E1.m1.11.11.1.1.1" xref="A1.E1.m1.11.11.1.1.1.1.cmml"><mo stretchy="false" id="A1.E1.m1.11.11.1.1.1.2" xref="A1.E1.m1.11.11.1.1.1.1.cmml">(</mo><mrow id="A1.E1.m1.11.11.1.1.1.1" xref="A1.E1.m1.11.11.1.1.1.1.cmml"><msub id="A1.E1.m1.11.11.1.1.1.1.2" xref="A1.E1.m1.11.11.1.1.1.1.2.cmml"><mi id="A1.E1.m1.11.11.1.1.1.1.2.2" xref="A1.E1.m1.11.11.1.1.1.1.2.2.cmml">𝒙</mi><mi id="A1.E1.m1.11.11.1.1.1.1.2.3" xref="A1.E1.m1.11.11.1.1.1.1.2.3.cmml">i</mi></msub><mo fence="false" id="A1.E1.m1.11.11.1.1.1.1.1" xref="A1.E1.m1.11.11.1.1.1.1.1.cmml">|</mo><mrow id="A1.E1.m1.11.11.1.1.1.1.3.2" xref="A1.E1.m1.11.11.1.1.1.1.3.1.cmml"><mi id="A1.E1.m1.9.9" xref="A1.E1.m1.9.9.cmml">θ</mi><mo id="A1.E1.m1.11.11.1.1.1.1.3.2.1" xref="A1.E1.m1.11.11.1.1.1.1.3.1.cmml">,</mo><mi class="ltx_font_mathcaligraphic" id="A1.E1.m1.10.10" xref="A1.E1.m1.10.10.cmml">ℬ</mi></mrow></mrow><mo stretchy="false" id="A1.E1.m1.11.11.1.1.1.3" xref="A1.E1.m1.11.11.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A1.E1.m1.12.12.3" xref="A1.E1.m1.12.12.3.cmml">=</mo><mrow id="A1.E1.m1.12.12.2" xref="A1.E1.m1.12.12.2.cmml"><mo id="A1.E1.m1.12.12.2a" xref="A1.E1.m1.12.12.2.cmml">−</mo><mrow id="A1.E1.m1.12.12.2.1" xref="A1.E1.m1.12.12.2.1.cmml"><mfrac id="A1.E1.m1.12.12.2.1.3" xref="A1.E1.m1.12.12.2.1.3.cmml"><mn id="A1.E1.m1.12.12.2.1.3.2" xref="A1.E1.m1.12.12.2.1.3.2.cmml">1</mn><mn id="A1.E1.m1.12.12.2.1.3.3" xref="A1.E1.m1.12.12.2.1.3.3.cmml">2</mn></mfrac><mo lspace="0em" rspace="0em" id="A1.E1.m1.12.12.2.1.2" xref="A1.E1.m1.12.12.2.1.2.cmml">​</mo><mrow id="A1.E1.m1.12.12.2.1.1.1" xref="A1.E1.m1.12.12.2.1.1.1.1.cmml"><mo id="A1.E1.m1.12.12.2.1.1.1.2" xref="A1.E1.m1.12.12.2.1.1.1.1.cmml">(</mo><mrow id="A1.E1.m1.12.12.2.1.1.1.1" xref="A1.E1.m1.12.12.2.1.1.1.1.cmml"><mrow id="A1.E1.m1.12.12.2.1.1.1.1.2" xref="A1.E1.m1.12.12.2.1.1.1.1.2.cmml"><mi id="A1.E1.m1.12.12.2.1.1.1.1.2.1" xref="A1.E1.m1.12.12.2.1.1.1.1.2.1.cmml">log</mi><mo lspace="0.167em" id="A1.E1.m1.12.12.2.1.1.1.1.2a" xref="A1.E1.m1.12.12.2.1.1.1.1.2.cmml">⁡</mo><mfrac id="A1.E1.m1.4.4" xref="A1.E1.m1.4.4.cmml"><mrow id="A1.E1.m1.2.2.2.2" xref="A1.E1.m1.2.2.2.3.cmml"><mi id="A1.E1.m1.1.1.1.1" xref="A1.E1.m1.1.1.1.1.cmml">exp</mi><mo id="A1.E1.m1.2.2.2.2a" xref="A1.E1.m1.2.2.2.3.cmml">⁡</mo><mrow id="A1.E1.m1.2.2.2.2.1" xref="A1.E1.m1.2.2.2.3.cmml"><mo stretchy="false" id="A1.E1.m1.2.2.2.2.1.2" xref="A1.E1.m1.2.2.2.3.cmml">(</mo><mrow id="A1.E1.m1.2.2.2.2.1.1" xref="A1.E1.m1.2.2.2.2.1.1.cmml"><mrow id="A1.E1.m1.2.2.2.2.1.1.2" xref="A1.E1.m1.2.2.2.2.1.1.2.cmml"><mi id="A1.E1.m1.2.2.2.2.1.1.2.2" xref="A1.E1.m1.2.2.2.2.1.1.2.2.cmml">α</mi><mo lspace="0em" rspace="0em" id="A1.E1.m1.2.2.2.2.1.1.2.1" xref="A1.E1.m1.2.2.2.2.1.1.2.1.cmml">​</mo><msubsup id="A1.E1.m1.2.2.2.2.1.1.2.3" xref="A1.E1.m1.2.2.2.2.1.1.2.3.cmml"><mi id="A1.E1.m1.2.2.2.2.1.1.2.3.2.2" xref="A1.E1.m1.2.2.2.2.1.1.2.3.2.2.cmml">𝒛</mi><mi id="A1.E1.m1.2.2.2.2.1.1.2.3.3" xref="A1.E1.m1.2.2.2.2.1.1.2.3.3.cmml">i</mi><mtext id="A1.E1.m1.2.2.2.2.1.1.2.3.2.3" xref="A1.E1.m1.2.2.2.2.1.1.2.3.2.3a.cmml">im</mtext></msubsup></mrow><mo lspace="0.222em" rspace="0.222em" id="A1.E1.m1.2.2.2.2.1.1.1" xref="A1.E1.m1.2.2.2.2.1.1.1.cmml">⋅</mo><msubsup id="A1.E1.m1.2.2.2.2.1.1.3" xref="A1.E1.m1.2.2.2.2.1.1.3.cmml"><mi id="A1.E1.m1.2.2.2.2.1.1.3.2.2" xref="A1.E1.m1.2.2.2.2.1.1.3.2.2.cmml">𝒛</mi><mi id="A1.E1.m1.2.2.2.2.1.1.3.3" xref="A1.E1.m1.2.2.2.2.1.1.3.3.cmml">i</mi><mtext id="A1.E1.m1.2.2.2.2.1.1.3.2.3" xref="A1.E1.m1.2.2.2.2.1.1.3.2.3a.cmml">txt</mtext></msubsup></mrow><mo stretchy="false" id="A1.E1.m1.2.2.2.2.1.3" xref="A1.E1.m1.2.2.2.3.cmml">)</mo></mrow></mrow><mrow id="A1.E1.m1.4.4.4" xref="A1.E1.m1.4.4.4.cmml"><msub id="A1.E1.m1.4.4.4.3" xref="A1.E1.m1.4.4.4.3.cmml"><mo id="A1.E1.m1.4.4.4.3.2" xref="A1.E1.m1.4.4.4.3.2.cmml">∑</mo><mi id="A1.E1.m1.4.4.4.3.3" xref="A1.E1.m1.4.4.4.3.3.cmml">j</mi></msub><mrow id="A1.E1.m1.4.4.4.2.1" xref="A1.E1.m1.4.4.4.2.2.cmml"><mi id="A1.E1.m1.3.3.3.1" xref="A1.E1.m1.3.3.3.1.cmml">exp</mi><mo id="A1.E1.m1.4.4.4.2.1a" xref="A1.E1.m1.4.4.4.2.2.cmml">⁡</mo><mrow id="A1.E1.m1.4.4.4.2.1.1" xref="A1.E1.m1.4.4.4.2.2.cmml"><mo stretchy="false" id="A1.E1.m1.4.4.4.2.1.1.2" xref="A1.E1.m1.4.4.4.2.2.cmml">(</mo><mrow id="A1.E1.m1.4.4.4.2.1.1.1" xref="A1.E1.m1.4.4.4.2.1.1.1.cmml"><mrow id="A1.E1.m1.4.4.4.2.1.1.1.2" xref="A1.E1.m1.4.4.4.2.1.1.1.2.cmml"><mi id="A1.E1.m1.4.4.4.2.1.1.1.2.2" xref="A1.E1.m1.4.4.4.2.1.1.1.2.2.cmml">α</mi><mo lspace="0em" rspace="0em" id="A1.E1.m1.4.4.4.2.1.1.1.2.1" xref="A1.E1.m1.4.4.4.2.1.1.1.2.1.cmml">​</mo><msubsup id="A1.E1.m1.4.4.4.2.1.1.1.2.3" xref="A1.E1.m1.4.4.4.2.1.1.1.2.3.cmml"><mi id="A1.E1.m1.4.4.4.2.1.1.1.2.3.2.2" xref="A1.E1.m1.4.4.4.2.1.1.1.2.3.2.2.cmml">𝒛</mi><mi id="A1.E1.m1.4.4.4.2.1.1.1.2.3.3" xref="A1.E1.m1.4.4.4.2.1.1.1.2.3.3.cmml">i</mi><mtext id="A1.E1.m1.4.4.4.2.1.1.1.2.3.2.3" xref="A1.E1.m1.4.4.4.2.1.1.1.2.3.2.3a.cmml">im</mtext></msubsup></mrow><mo lspace="0.222em" rspace="0.222em" id="A1.E1.m1.4.4.4.2.1.1.1.1" xref="A1.E1.m1.4.4.4.2.1.1.1.1.cmml">⋅</mo><msubsup id="A1.E1.m1.4.4.4.2.1.1.1.3" xref="A1.E1.m1.4.4.4.2.1.1.1.3.cmml"><mi id="A1.E1.m1.4.4.4.2.1.1.1.3.2.2" xref="A1.E1.m1.4.4.4.2.1.1.1.3.2.2.cmml">𝒛</mi><mi id="A1.E1.m1.4.4.4.2.1.1.1.3.3" xref="A1.E1.m1.4.4.4.2.1.1.1.3.3.cmml">j</mi><mtext id="A1.E1.m1.4.4.4.2.1.1.1.3.2.3" xref="A1.E1.m1.4.4.4.2.1.1.1.3.2.3a.cmml">txt</mtext></msubsup></mrow><mo stretchy="false" id="A1.E1.m1.4.4.4.2.1.1.3" xref="A1.E1.m1.4.4.4.2.2.cmml">)</mo></mrow></mrow></mrow></mfrac></mrow><mo id="A1.E1.m1.12.12.2.1.1.1.1.1" xref="A1.E1.m1.12.12.2.1.1.1.1.1.cmml">+</mo><mrow id="A1.E1.m1.12.12.2.1.1.1.1.3" xref="A1.E1.m1.12.12.2.1.1.1.1.3.cmml"><mi id="A1.E1.m1.12.12.2.1.1.1.1.3.1" xref="A1.E1.m1.12.12.2.1.1.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="A1.E1.m1.12.12.2.1.1.1.1.3a" xref="A1.E1.m1.12.12.2.1.1.1.1.3.cmml">⁡</mo><mfrac id="A1.E1.m1.8.8" xref="A1.E1.m1.8.8.cmml"><mrow id="A1.E1.m1.6.6.2.2" xref="A1.E1.m1.6.6.2.3.cmml"><mi id="A1.E1.m1.5.5.1.1" xref="A1.E1.m1.5.5.1.1.cmml">exp</mi><mo id="A1.E1.m1.6.6.2.2a" xref="A1.E1.m1.6.6.2.3.cmml">⁡</mo><mrow id="A1.E1.m1.6.6.2.2.1" xref="A1.E1.m1.6.6.2.3.cmml"><mo stretchy="false" id="A1.E1.m1.6.6.2.2.1.2" xref="A1.E1.m1.6.6.2.3.cmml">(</mo><mrow id="A1.E1.m1.6.6.2.2.1.1" xref="A1.E1.m1.6.6.2.2.1.1.cmml"><mrow id="A1.E1.m1.6.6.2.2.1.1.2" xref="A1.E1.m1.6.6.2.2.1.1.2.cmml"><mi id="A1.E1.m1.6.6.2.2.1.1.2.2" xref="A1.E1.m1.6.6.2.2.1.1.2.2.cmml">α</mi><mo lspace="0em" rspace="0em" id="A1.E1.m1.6.6.2.2.1.1.2.1" xref="A1.E1.m1.6.6.2.2.1.1.2.1.cmml">​</mo><msubsup id="A1.E1.m1.6.6.2.2.1.1.2.3" xref="A1.E1.m1.6.6.2.2.1.1.2.3.cmml"><mi id="A1.E1.m1.6.6.2.2.1.1.2.3.2.2" xref="A1.E1.m1.6.6.2.2.1.1.2.3.2.2.cmml">𝒛</mi><mi id="A1.E1.m1.6.6.2.2.1.1.2.3.3" xref="A1.E1.m1.6.6.2.2.1.1.2.3.3.cmml">i</mi><mtext id="A1.E1.m1.6.6.2.2.1.1.2.3.2.3" xref="A1.E1.m1.6.6.2.2.1.1.2.3.2.3a.cmml">im</mtext></msubsup></mrow><mo lspace="0.222em" rspace="0.222em" id="A1.E1.m1.6.6.2.2.1.1.1" xref="A1.E1.m1.6.6.2.2.1.1.1.cmml">⋅</mo><msubsup id="A1.E1.m1.6.6.2.2.1.1.3" xref="A1.E1.m1.6.6.2.2.1.1.3.cmml"><mi id="A1.E1.m1.6.6.2.2.1.1.3.2.2" xref="A1.E1.m1.6.6.2.2.1.1.3.2.2.cmml">𝒛</mi><mi id="A1.E1.m1.6.6.2.2.1.1.3.3" xref="A1.E1.m1.6.6.2.2.1.1.3.3.cmml">i</mi><mtext id="A1.E1.m1.6.6.2.2.1.1.3.2.3" xref="A1.E1.m1.6.6.2.2.1.1.3.2.3a.cmml">txt</mtext></msubsup></mrow><mo stretchy="false" id="A1.E1.m1.6.6.2.2.1.3" xref="A1.E1.m1.6.6.2.3.cmml">)</mo></mrow></mrow><mrow id="A1.E1.m1.8.8.4" xref="A1.E1.m1.8.8.4.cmml"><msub id="A1.E1.m1.8.8.4.3" xref="A1.E1.m1.8.8.4.3.cmml"><mo id="A1.E1.m1.8.8.4.3.2" xref="A1.E1.m1.8.8.4.3.2.cmml">∑</mo><mi id="A1.E1.m1.8.8.4.3.3" xref="A1.E1.m1.8.8.4.3.3.cmml">j</mi></msub><mrow id="A1.E1.m1.8.8.4.2.1" xref="A1.E1.m1.8.8.4.2.2.cmml"><mi id="A1.E1.m1.7.7.3.1" xref="A1.E1.m1.7.7.3.1.cmml">exp</mi><mo id="A1.E1.m1.8.8.4.2.1a" xref="A1.E1.m1.8.8.4.2.2.cmml">⁡</mo><mrow id="A1.E1.m1.8.8.4.2.1.1" xref="A1.E1.m1.8.8.4.2.2.cmml"><mo stretchy="false" id="A1.E1.m1.8.8.4.2.1.1.2" xref="A1.E1.m1.8.8.4.2.2.cmml">(</mo><mrow id="A1.E1.m1.8.8.4.2.1.1.1" xref="A1.E1.m1.8.8.4.2.1.1.1.cmml"><mrow id="A1.E1.m1.8.8.4.2.1.1.1.2" xref="A1.E1.m1.8.8.4.2.1.1.1.2.cmml"><mi id="A1.E1.m1.8.8.4.2.1.1.1.2.2" xref="A1.E1.m1.8.8.4.2.1.1.1.2.2.cmml">α</mi><mo lspace="0em" rspace="0em" id="A1.E1.m1.8.8.4.2.1.1.1.2.1" xref="A1.E1.m1.8.8.4.2.1.1.1.2.1.cmml">​</mo><msubsup id="A1.E1.m1.8.8.4.2.1.1.1.2.3" xref="A1.E1.m1.8.8.4.2.1.1.1.2.3.cmml"><mi id="A1.E1.m1.8.8.4.2.1.1.1.2.3.2.2" xref="A1.E1.m1.8.8.4.2.1.1.1.2.3.2.2.cmml">𝒛</mi><mi id="A1.E1.m1.8.8.4.2.1.1.1.2.3.3" xref="A1.E1.m1.8.8.4.2.1.1.1.2.3.3.cmml">i</mi><mtext id="A1.E1.m1.8.8.4.2.1.1.1.2.3.2.3" xref="A1.E1.m1.8.8.4.2.1.1.1.2.3.2.3a.cmml">txt</mtext></msubsup></mrow><mo lspace="0.222em" rspace="0.222em" id="A1.E1.m1.8.8.4.2.1.1.1.1" xref="A1.E1.m1.8.8.4.2.1.1.1.1.cmml">⋅</mo><msubsup id="A1.E1.m1.8.8.4.2.1.1.1.3" xref="A1.E1.m1.8.8.4.2.1.1.1.3.cmml"><mi id="A1.E1.m1.8.8.4.2.1.1.1.3.2.2" xref="A1.E1.m1.8.8.4.2.1.1.1.3.2.2.cmml">𝒛</mi><mi id="A1.E1.m1.8.8.4.2.1.1.1.3.3" xref="A1.E1.m1.8.8.4.2.1.1.1.3.3.cmml">j</mi><mtext id="A1.E1.m1.8.8.4.2.1.1.1.3.2.3" xref="A1.E1.m1.8.8.4.2.1.1.1.3.2.3a.cmml">im</mtext></msubsup></mrow><mo stretchy="false" id="A1.E1.m1.8.8.4.2.1.1.3" xref="A1.E1.m1.8.8.4.2.2.cmml">)</mo></mrow></mrow></mrow></mfrac></mrow></mrow><mo id="A1.E1.m1.12.12.2.1.1.1.3" xref="A1.E1.m1.12.12.2.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.E1.m1.12b"><apply id="A1.E1.m1.12.12.cmml" xref="A1.E1.m1.12.12"><eq id="A1.E1.m1.12.12.3.cmml" xref="A1.E1.m1.12.12.3"></eq><apply id="A1.E1.m1.11.11.1.cmml" xref="A1.E1.m1.11.11.1"><times id="A1.E1.m1.11.11.1.2.cmml" xref="A1.E1.m1.11.11.1.2"></times><ci id="A1.E1.m1.11.11.1.3.cmml" xref="A1.E1.m1.11.11.1.3">ℓ</ci><apply id="A1.E1.m1.11.11.1.1.1.1.cmml" xref="A1.E1.m1.11.11.1.1.1"><csymbol cd="latexml" id="A1.E1.m1.11.11.1.1.1.1.1.cmml" xref="A1.E1.m1.11.11.1.1.1.1.1">conditional</csymbol><apply id="A1.E1.m1.11.11.1.1.1.1.2.cmml" xref="A1.E1.m1.11.11.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.E1.m1.11.11.1.1.1.1.2.1.cmml" xref="A1.E1.m1.11.11.1.1.1.1.2">subscript</csymbol><ci id="A1.E1.m1.11.11.1.1.1.1.2.2.cmml" xref="A1.E1.m1.11.11.1.1.1.1.2.2">𝒙</ci><ci id="A1.E1.m1.11.11.1.1.1.1.2.3.cmml" xref="A1.E1.m1.11.11.1.1.1.1.2.3">𝑖</ci></apply><list id="A1.E1.m1.11.11.1.1.1.1.3.1.cmml" xref="A1.E1.m1.11.11.1.1.1.1.3.2"><ci id="A1.E1.m1.9.9.cmml" xref="A1.E1.m1.9.9">𝜃</ci><ci id="A1.E1.m1.10.10.cmml" xref="A1.E1.m1.10.10">ℬ</ci></list></apply></apply><apply id="A1.E1.m1.12.12.2.cmml" xref="A1.E1.m1.12.12.2"><minus id="A1.E1.m1.12.12.2.2.cmml" xref="A1.E1.m1.12.12.2"></minus><apply id="A1.E1.m1.12.12.2.1.cmml" xref="A1.E1.m1.12.12.2.1"><times id="A1.E1.m1.12.12.2.1.2.cmml" xref="A1.E1.m1.12.12.2.1.2"></times><apply id="A1.E1.m1.12.12.2.1.3.cmml" xref="A1.E1.m1.12.12.2.1.3"><divide id="A1.E1.m1.12.12.2.1.3.1.cmml" xref="A1.E1.m1.12.12.2.1.3"></divide><cn type="integer" id="A1.E1.m1.12.12.2.1.3.2.cmml" xref="A1.E1.m1.12.12.2.1.3.2">1</cn><cn type="integer" id="A1.E1.m1.12.12.2.1.3.3.cmml" xref="A1.E1.m1.12.12.2.1.3.3">2</cn></apply><apply id="A1.E1.m1.12.12.2.1.1.1.1.cmml" xref="A1.E1.m1.12.12.2.1.1.1"><plus id="A1.E1.m1.12.12.2.1.1.1.1.1.cmml" xref="A1.E1.m1.12.12.2.1.1.1.1.1"></plus><apply id="A1.E1.m1.12.12.2.1.1.1.1.2.cmml" xref="A1.E1.m1.12.12.2.1.1.1.1.2"><log id="A1.E1.m1.12.12.2.1.1.1.1.2.1.cmml" xref="A1.E1.m1.12.12.2.1.1.1.1.2.1"></log><apply id="A1.E1.m1.4.4.cmml" xref="A1.E1.m1.4.4"><divide id="A1.E1.m1.4.4.5.cmml" xref="A1.E1.m1.4.4"></divide><apply id="A1.E1.m1.2.2.2.3.cmml" xref="A1.E1.m1.2.2.2.2"><exp id="A1.E1.m1.1.1.1.1.cmml" xref="A1.E1.m1.1.1.1.1"></exp><apply id="A1.E1.m1.2.2.2.2.1.1.cmml" xref="A1.E1.m1.2.2.2.2.1.1"><ci id="A1.E1.m1.2.2.2.2.1.1.1.cmml" xref="A1.E1.m1.2.2.2.2.1.1.1">⋅</ci><apply id="A1.E1.m1.2.2.2.2.1.1.2.cmml" xref="A1.E1.m1.2.2.2.2.1.1.2"><times id="A1.E1.m1.2.2.2.2.1.1.2.1.cmml" xref="A1.E1.m1.2.2.2.2.1.1.2.1"></times><ci id="A1.E1.m1.2.2.2.2.1.1.2.2.cmml" xref="A1.E1.m1.2.2.2.2.1.1.2.2">𝛼</ci><apply id="A1.E1.m1.2.2.2.2.1.1.2.3.cmml" xref="A1.E1.m1.2.2.2.2.1.1.2.3"><csymbol cd="ambiguous" id="A1.E1.m1.2.2.2.2.1.1.2.3.1.cmml" xref="A1.E1.m1.2.2.2.2.1.1.2.3">subscript</csymbol><apply id="A1.E1.m1.2.2.2.2.1.1.2.3.2.cmml" xref="A1.E1.m1.2.2.2.2.1.1.2.3"><csymbol cd="ambiguous" id="A1.E1.m1.2.2.2.2.1.1.2.3.2.1.cmml" xref="A1.E1.m1.2.2.2.2.1.1.2.3">superscript</csymbol><ci id="A1.E1.m1.2.2.2.2.1.1.2.3.2.2.cmml" xref="A1.E1.m1.2.2.2.2.1.1.2.3.2.2">𝒛</ci><ci id="A1.E1.m1.2.2.2.2.1.1.2.3.2.3a.cmml" xref="A1.E1.m1.2.2.2.2.1.1.2.3.2.3"><mtext mathsize="70%" id="A1.E1.m1.2.2.2.2.1.1.2.3.2.3.cmml" xref="A1.E1.m1.2.2.2.2.1.1.2.3.2.3">im</mtext></ci></apply><ci id="A1.E1.m1.2.2.2.2.1.1.2.3.3.cmml" xref="A1.E1.m1.2.2.2.2.1.1.2.3.3">𝑖</ci></apply></apply><apply id="A1.E1.m1.2.2.2.2.1.1.3.cmml" xref="A1.E1.m1.2.2.2.2.1.1.3"><csymbol cd="ambiguous" id="A1.E1.m1.2.2.2.2.1.1.3.1.cmml" xref="A1.E1.m1.2.2.2.2.1.1.3">subscript</csymbol><apply id="A1.E1.m1.2.2.2.2.1.1.3.2.cmml" xref="A1.E1.m1.2.2.2.2.1.1.3"><csymbol cd="ambiguous" id="A1.E1.m1.2.2.2.2.1.1.3.2.1.cmml" xref="A1.E1.m1.2.2.2.2.1.1.3">superscript</csymbol><ci id="A1.E1.m1.2.2.2.2.1.1.3.2.2.cmml" xref="A1.E1.m1.2.2.2.2.1.1.3.2.2">𝒛</ci><ci id="A1.E1.m1.2.2.2.2.1.1.3.2.3a.cmml" xref="A1.E1.m1.2.2.2.2.1.1.3.2.3"><mtext mathsize="70%" id="A1.E1.m1.2.2.2.2.1.1.3.2.3.cmml" xref="A1.E1.m1.2.2.2.2.1.1.3.2.3">txt</mtext></ci></apply><ci id="A1.E1.m1.2.2.2.2.1.1.3.3.cmml" xref="A1.E1.m1.2.2.2.2.1.1.3.3">𝑖</ci></apply></apply></apply><apply id="A1.E1.m1.4.4.4.cmml" xref="A1.E1.m1.4.4.4"><apply id="A1.E1.m1.4.4.4.3.cmml" xref="A1.E1.m1.4.4.4.3"><csymbol cd="ambiguous" id="A1.E1.m1.4.4.4.3.1.cmml" xref="A1.E1.m1.4.4.4.3">subscript</csymbol><sum id="A1.E1.m1.4.4.4.3.2.cmml" xref="A1.E1.m1.4.4.4.3.2"></sum><ci id="A1.E1.m1.4.4.4.3.3.cmml" xref="A1.E1.m1.4.4.4.3.3">𝑗</ci></apply><apply id="A1.E1.m1.4.4.4.2.2.cmml" xref="A1.E1.m1.4.4.4.2.1"><exp id="A1.E1.m1.3.3.3.1.cmml" xref="A1.E1.m1.3.3.3.1"></exp><apply id="A1.E1.m1.4.4.4.2.1.1.1.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1"><ci id="A1.E1.m1.4.4.4.2.1.1.1.1.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.1">⋅</ci><apply id="A1.E1.m1.4.4.4.2.1.1.1.2.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.2"><times id="A1.E1.m1.4.4.4.2.1.1.1.2.1.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.2.1"></times><ci id="A1.E1.m1.4.4.4.2.1.1.1.2.2.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.2.2">𝛼</ci><apply id="A1.E1.m1.4.4.4.2.1.1.1.2.3.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.2.3"><csymbol cd="ambiguous" id="A1.E1.m1.4.4.4.2.1.1.1.2.3.1.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.2.3">subscript</csymbol><apply id="A1.E1.m1.4.4.4.2.1.1.1.2.3.2.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.2.3"><csymbol cd="ambiguous" id="A1.E1.m1.4.4.4.2.1.1.1.2.3.2.1.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.2.3">superscript</csymbol><ci id="A1.E1.m1.4.4.4.2.1.1.1.2.3.2.2.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.2.3.2.2">𝒛</ci><ci id="A1.E1.m1.4.4.4.2.1.1.1.2.3.2.3a.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.2.3.2.3"><mtext mathsize="70%" id="A1.E1.m1.4.4.4.2.1.1.1.2.3.2.3.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.2.3.2.3">im</mtext></ci></apply><ci id="A1.E1.m1.4.4.4.2.1.1.1.2.3.3.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.2.3.3">𝑖</ci></apply></apply><apply id="A1.E1.m1.4.4.4.2.1.1.1.3.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.3"><csymbol cd="ambiguous" id="A1.E1.m1.4.4.4.2.1.1.1.3.1.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.3">subscript</csymbol><apply id="A1.E1.m1.4.4.4.2.1.1.1.3.2.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.3"><csymbol cd="ambiguous" id="A1.E1.m1.4.4.4.2.1.1.1.3.2.1.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.3">superscript</csymbol><ci id="A1.E1.m1.4.4.4.2.1.1.1.3.2.2.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.3.2.2">𝒛</ci><ci id="A1.E1.m1.4.4.4.2.1.1.1.3.2.3a.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.3.2.3"><mtext mathsize="70%" id="A1.E1.m1.4.4.4.2.1.1.1.3.2.3.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.3.2.3">txt</mtext></ci></apply><ci id="A1.E1.m1.4.4.4.2.1.1.1.3.3.cmml" xref="A1.E1.m1.4.4.4.2.1.1.1.3.3">𝑗</ci></apply></apply></apply></apply></apply></apply><apply id="A1.E1.m1.12.12.2.1.1.1.1.3.cmml" xref="A1.E1.m1.12.12.2.1.1.1.1.3"><log id="A1.E1.m1.12.12.2.1.1.1.1.3.1.cmml" xref="A1.E1.m1.12.12.2.1.1.1.1.3.1"></log><apply id="A1.E1.m1.8.8.cmml" xref="A1.E1.m1.8.8"><divide id="A1.E1.m1.8.8.5.cmml" xref="A1.E1.m1.8.8"></divide><apply id="A1.E1.m1.6.6.2.3.cmml" xref="A1.E1.m1.6.6.2.2"><exp id="A1.E1.m1.5.5.1.1.cmml" xref="A1.E1.m1.5.5.1.1"></exp><apply id="A1.E1.m1.6.6.2.2.1.1.cmml" xref="A1.E1.m1.6.6.2.2.1.1"><ci id="A1.E1.m1.6.6.2.2.1.1.1.cmml" xref="A1.E1.m1.6.6.2.2.1.1.1">⋅</ci><apply id="A1.E1.m1.6.6.2.2.1.1.2.cmml" xref="A1.E1.m1.6.6.2.2.1.1.2"><times id="A1.E1.m1.6.6.2.2.1.1.2.1.cmml" xref="A1.E1.m1.6.6.2.2.1.1.2.1"></times><ci id="A1.E1.m1.6.6.2.2.1.1.2.2.cmml" xref="A1.E1.m1.6.6.2.2.1.1.2.2">𝛼</ci><apply id="A1.E1.m1.6.6.2.2.1.1.2.3.cmml" xref="A1.E1.m1.6.6.2.2.1.1.2.3"><csymbol cd="ambiguous" id="A1.E1.m1.6.6.2.2.1.1.2.3.1.cmml" xref="A1.E1.m1.6.6.2.2.1.1.2.3">subscript</csymbol><apply id="A1.E1.m1.6.6.2.2.1.1.2.3.2.cmml" xref="A1.E1.m1.6.6.2.2.1.1.2.3"><csymbol cd="ambiguous" id="A1.E1.m1.6.6.2.2.1.1.2.3.2.1.cmml" xref="A1.E1.m1.6.6.2.2.1.1.2.3">superscript</csymbol><ci id="A1.E1.m1.6.6.2.2.1.1.2.3.2.2.cmml" xref="A1.E1.m1.6.6.2.2.1.1.2.3.2.2">𝒛</ci><ci id="A1.E1.m1.6.6.2.2.1.1.2.3.2.3a.cmml" xref="A1.E1.m1.6.6.2.2.1.1.2.3.2.3"><mtext mathsize="70%" id="A1.E1.m1.6.6.2.2.1.1.2.3.2.3.cmml" xref="A1.E1.m1.6.6.2.2.1.1.2.3.2.3">im</mtext></ci></apply><ci id="A1.E1.m1.6.6.2.2.1.1.2.3.3.cmml" xref="A1.E1.m1.6.6.2.2.1.1.2.3.3">𝑖</ci></apply></apply><apply id="A1.E1.m1.6.6.2.2.1.1.3.cmml" xref="A1.E1.m1.6.6.2.2.1.1.3"><csymbol cd="ambiguous" id="A1.E1.m1.6.6.2.2.1.1.3.1.cmml" xref="A1.E1.m1.6.6.2.2.1.1.3">subscript</csymbol><apply id="A1.E1.m1.6.6.2.2.1.1.3.2.cmml" xref="A1.E1.m1.6.6.2.2.1.1.3"><csymbol cd="ambiguous" id="A1.E1.m1.6.6.2.2.1.1.3.2.1.cmml" xref="A1.E1.m1.6.6.2.2.1.1.3">superscript</csymbol><ci id="A1.E1.m1.6.6.2.2.1.1.3.2.2.cmml" xref="A1.E1.m1.6.6.2.2.1.1.3.2.2">𝒛</ci><ci id="A1.E1.m1.6.6.2.2.1.1.3.2.3a.cmml" xref="A1.E1.m1.6.6.2.2.1.1.3.2.3"><mtext mathsize="70%" id="A1.E1.m1.6.6.2.2.1.1.3.2.3.cmml" xref="A1.E1.m1.6.6.2.2.1.1.3.2.3">txt</mtext></ci></apply><ci id="A1.E1.m1.6.6.2.2.1.1.3.3.cmml" xref="A1.E1.m1.6.6.2.2.1.1.3.3">𝑖</ci></apply></apply></apply><apply id="A1.E1.m1.8.8.4.cmml" xref="A1.E1.m1.8.8.4"><apply id="A1.E1.m1.8.8.4.3.cmml" xref="A1.E1.m1.8.8.4.3"><csymbol cd="ambiguous" id="A1.E1.m1.8.8.4.3.1.cmml" xref="A1.E1.m1.8.8.4.3">subscript</csymbol><sum id="A1.E1.m1.8.8.4.3.2.cmml" xref="A1.E1.m1.8.8.4.3.2"></sum><ci id="A1.E1.m1.8.8.4.3.3.cmml" xref="A1.E1.m1.8.8.4.3.3">𝑗</ci></apply><apply id="A1.E1.m1.8.8.4.2.2.cmml" xref="A1.E1.m1.8.8.4.2.1"><exp id="A1.E1.m1.7.7.3.1.cmml" xref="A1.E1.m1.7.7.3.1"></exp><apply id="A1.E1.m1.8.8.4.2.1.1.1.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1"><ci id="A1.E1.m1.8.8.4.2.1.1.1.1.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.1">⋅</ci><apply id="A1.E1.m1.8.8.4.2.1.1.1.2.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.2"><times id="A1.E1.m1.8.8.4.2.1.1.1.2.1.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.2.1"></times><ci id="A1.E1.m1.8.8.4.2.1.1.1.2.2.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.2.2">𝛼</ci><apply id="A1.E1.m1.8.8.4.2.1.1.1.2.3.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.2.3"><csymbol cd="ambiguous" id="A1.E1.m1.8.8.4.2.1.1.1.2.3.1.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.2.3">subscript</csymbol><apply id="A1.E1.m1.8.8.4.2.1.1.1.2.3.2.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.2.3"><csymbol cd="ambiguous" id="A1.E1.m1.8.8.4.2.1.1.1.2.3.2.1.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.2.3">superscript</csymbol><ci id="A1.E1.m1.8.8.4.2.1.1.1.2.3.2.2.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.2.3.2.2">𝒛</ci><ci id="A1.E1.m1.8.8.4.2.1.1.1.2.3.2.3a.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.2.3.2.3"><mtext mathsize="70%" id="A1.E1.m1.8.8.4.2.1.1.1.2.3.2.3.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.2.3.2.3">txt</mtext></ci></apply><ci id="A1.E1.m1.8.8.4.2.1.1.1.2.3.3.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.2.3.3">𝑖</ci></apply></apply><apply id="A1.E1.m1.8.8.4.2.1.1.1.3.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.3"><csymbol cd="ambiguous" id="A1.E1.m1.8.8.4.2.1.1.1.3.1.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.3">subscript</csymbol><apply id="A1.E1.m1.8.8.4.2.1.1.1.3.2.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.3"><csymbol cd="ambiguous" id="A1.E1.m1.8.8.4.2.1.1.1.3.2.1.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.3">superscript</csymbol><ci id="A1.E1.m1.8.8.4.2.1.1.1.3.2.2.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.3.2.2">𝒛</ci><ci id="A1.E1.m1.8.8.4.2.1.1.1.3.2.3a.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.3.2.3"><mtext mathsize="70%" id="A1.E1.m1.8.8.4.2.1.1.1.3.2.3.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.3.2.3">im</mtext></ci></apply><ci id="A1.E1.m1.8.8.4.2.1.1.1.3.3.cmml" xref="A1.E1.m1.8.8.4.2.1.1.1.3.3">𝑗</ci></apply></apply></apply></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E1.m1.12c">\ell({\bm{x}}_{i}|\theta,\mathcal{B})=-\frac{1}{2}\left(\log\frac{\exp(\alpha{\bm{z}}^{\textrm{im}}_{i}{\cdot}{\bm{z}}^{\textrm{txt}}_{i})}{\sum_{j}\exp(\alpha{\bm{z}}^{\textrm{im}}_{i}{\cdot}{\bm{z}}^{\textrm{txt}}_{j})}+\log\frac{\exp(\alpha{\bm{z}}^{\textrm{im}}_{i}{\cdot}{\bm{z}}^{\textrm{txt}}_{i})}{\sum_{j}\exp(\alpha{\bm{z}}^{\textrm{txt}}_{i}{\cdot}{\bm{z}}^{\textrm{im}}_{j})}\right)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="A1.SS6.p3.2" class="ltx_p">whereas in sigmoid-contrastive learning <cite class="ltx_cite ltx_citemacro_citep">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>, the conditional loss is</p>
<table id="A1.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.E2.m1.7" class="ltx_Math" alttext="\ell({\bm{x}}_{i}|\theta,\mathcal{B})=\log\left[1+\exp(-\alpha{\bm{z}}^{\textrm{im}}_{i}{\cdot}{\bm{z}}^{\textrm{txt}}_{i}+\beta)\right]+\sum_{j\neq i}\log\left[1+\exp(\alpha{\bm{z}}^{\textrm{im}}_{i}{\cdot}{\bm{z}}^{\textrm{txt}}_{j}-\beta)\right]." display="block"><semantics id="A1.E2.m1.7a"><mrow id="A1.E2.m1.7.7.1" xref="A1.E2.m1.7.7.1.1.cmml"><mrow id="A1.E2.m1.7.7.1.1" xref="A1.E2.m1.7.7.1.1.cmml"><mrow id="A1.E2.m1.7.7.1.1.1" xref="A1.E2.m1.7.7.1.1.1.cmml"><mi mathvariant="normal" id="A1.E2.m1.7.7.1.1.1.3" xref="A1.E2.m1.7.7.1.1.1.3.cmml">ℓ</mi><mo lspace="0em" rspace="0em" id="A1.E2.m1.7.7.1.1.1.2" xref="A1.E2.m1.7.7.1.1.1.2.cmml">​</mo><mrow id="A1.E2.m1.7.7.1.1.1.1.1" xref="A1.E2.m1.7.7.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A1.E2.m1.7.7.1.1.1.1.1.2" xref="A1.E2.m1.7.7.1.1.1.1.1.1.cmml">(</mo><mrow id="A1.E2.m1.7.7.1.1.1.1.1.1" xref="A1.E2.m1.7.7.1.1.1.1.1.1.cmml"><msub id="A1.E2.m1.7.7.1.1.1.1.1.1.2" xref="A1.E2.m1.7.7.1.1.1.1.1.1.2.cmml"><mi id="A1.E2.m1.7.7.1.1.1.1.1.1.2.2" xref="A1.E2.m1.7.7.1.1.1.1.1.1.2.2.cmml">𝒙</mi><mi id="A1.E2.m1.7.7.1.1.1.1.1.1.2.3" xref="A1.E2.m1.7.7.1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo fence="false" id="A1.E2.m1.7.7.1.1.1.1.1.1.1" xref="A1.E2.m1.7.7.1.1.1.1.1.1.1.cmml">|</mo><mrow id="A1.E2.m1.7.7.1.1.1.1.1.1.3.2" xref="A1.E2.m1.7.7.1.1.1.1.1.1.3.1.cmml"><mi id="A1.E2.m1.1.1" xref="A1.E2.m1.1.1.cmml">θ</mi><mo id="A1.E2.m1.7.7.1.1.1.1.1.1.3.2.1" xref="A1.E2.m1.7.7.1.1.1.1.1.1.3.1.cmml">,</mo><mi class="ltx_font_mathcaligraphic" id="A1.E2.m1.2.2" xref="A1.E2.m1.2.2.cmml">ℬ</mi></mrow></mrow><mo stretchy="false" id="A1.E2.m1.7.7.1.1.1.1.1.3" xref="A1.E2.m1.7.7.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A1.E2.m1.7.7.1.1.4" xref="A1.E2.m1.7.7.1.1.4.cmml">=</mo><mrow id="A1.E2.m1.7.7.1.1.3" xref="A1.E2.m1.7.7.1.1.3.cmml"><mrow id="A1.E2.m1.7.7.1.1.2.1.1" xref="A1.E2.m1.7.7.1.1.2.1.2.cmml"><mi id="A1.E2.m1.4.4" xref="A1.E2.m1.4.4.cmml">log</mi><mo id="A1.E2.m1.7.7.1.1.2.1.1a" xref="A1.E2.m1.7.7.1.1.2.1.2.cmml">⁡</mo><mrow id="A1.E2.m1.7.7.1.1.2.1.1.1" xref="A1.E2.m1.7.7.1.1.2.1.2.cmml"><mo id="A1.E2.m1.7.7.1.1.2.1.1.1.2" xref="A1.E2.m1.7.7.1.1.2.1.2.cmml">[</mo><mrow id="A1.E2.m1.7.7.1.1.2.1.1.1.1" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.cmml"><mn id="A1.E2.m1.7.7.1.1.2.1.1.1.1.3" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.3.cmml">1</mn><mo id="A1.E2.m1.7.7.1.1.2.1.1.1.1.2" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.2.cmml">+</mo><mrow id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.2.cmml"><mi id="A1.E2.m1.3.3" xref="A1.E2.m1.3.3.cmml">exp</mi><mo id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1a" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.2.cmml">⁡</mo><mrow id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.2" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.2.cmml">(</mo><mrow id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.cmml"><mrow id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.cmml"><mo id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2a" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.cmml">−</mo><mrow id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.cmml"><mrow id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.cmml"><mi id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.2" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.2.cmml">α</mi><mo lspace="0em" rspace="0em" id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.1" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.1.cmml">​</mo><msubsup id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.cmml"><mi id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.2.2" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.2.2.cmml">𝒛</mi><mi id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.3" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.3.cmml">i</mi><mtext id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.2.3" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.2.3a.cmml">im</mtext></msubsup></mrow><mo lspace="0.222em" rspace="0.222em" id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.1" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.1.cmml">⋅</mo><msubsup id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.cmml"><mi id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.2.2" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.2.2.cmml">𝒛</mi><mi id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.3" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.3.cmml">i</mi><mtext id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.2.3" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.2.3a.cmml">txt</mtext></msubsup></mrow></mrow><mo id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.1" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.1.cmml">+</mo><mi id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.3" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.3.cmml">β</mi></mrow><mo stretchy="false" id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.3" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.2.cmml">)</mo></mrow></mrow></mrow><mo id="A1.E2.m1.7.7.1.1.2.1.1.1.3" xref="A1.E2.m1.7.7.1.1.2.1.2.cmml">]</mo></mrow></mrow><mo rspace="0.055em" id="A1.E2.m1.7.7.1.1.3.3" xref="A1.E2.m1.7.7.1.1.3.3.cmml">+</mo><mrow id="A1.E2.m1.7.7.1.1.3.2" xref="A1.E2.m1.7.7.1.1.3.2.cmml"><munder id="A1.E2.m1.7.7.1.1.3.2.2" xref="A1.E2.m1.7.7.1.1.3.2.2.cmml"><mo movablelimits="false" id="A1.E2.m1.7.7.1.1.3.2.2.2" xref="A1.E2.m1.7.7.1.1.3.2.2.2.cmml">∑</mo><mrow id="A1.E2.m1.7.7.1.1.3.2.2.3" xref="A1.E2.m1.7.7.1.1.3.2.2.3.cmml"><mi id="A1.E2.m1.7.7.1.1.3.2.2.3.2" xref="A1.E2.m1.7.7.1.1.3.2.2.3.2.cmml">j</mi><mo id="A1.E2.m1.7.7.1.1.3.2.2.3.1" xref="A1.E2.m1.7.7.1.1.3.2.2.3.1.cmml">≠</mo><mi id="A1.E2.m1.7.7.1.1.3.2.2.3.3" xref="A1.E2.m1.7.7.1.1.3.2.2.3.3.cmml">i</mi></mrow></munder><mrow id="A1.E2.m1.7.7.1.1.3.2.1.1" xref="A1.E2.m1.7.7.1.1.3.2.1.2.cmml"><mi id="A1.E2.m1.6.6" xref="A1.E2.m1.6.6.cmml">log</mi><mo id="A1.E2.m1.7.7.1.1.3.2.1.1a" xref="A1.E2.m1.7.7.1.1.3.2.1.2.cmml">⁡</mo><mrow id="A1.E2.m1.7.7.1.1.3.2.1.1.1" xref="A1.E2.m1.7.7.1.1.3.2.1.2.cmml"><mo id="A1.E2.m1.7.7.1.1.3.2.1.1.1.2" xref="A1.E2.m1.7.7.1.1.3.2.1.2.cmml">[</mo><mrow id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.cmml"><mn id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.3" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.3.cmml">1</mn><mo id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.2" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.2.cmml">+</mo><mrow id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.2.cmml"><mi id="A1.E2.m1.5.5" xref="A1.E2.m1.5.5.cmml">exp</mi><mo id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1a" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.2.cmml">⁡</mo><mrow id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.2" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.2.cmml">(</mo><mrow id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.cmml"><mrow id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.cmml"><mrow id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.cmml"><mi id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.2" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.2.cmml">α</mi><mo lspace="0em" rspace="0em" id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.1" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.1.cmml">​</mo><msubsup id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.cmml"><mi id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.2.2" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.2.2.cmml">𝒛</mi><mi id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.3" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.3.cmml">i</mi><mtext id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.2.3" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.2.3a.cmml">im</mtext></msubsup></mrow><mo lspace="0.222em" rspace="0.222em" id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.1" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.1.cmml">⋅</mo><msubsup id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.cmml"><mi id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.2.2" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.2.2.cmml">𝒛</mi><mi id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.3" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.3.cmml">j</mi><mtext id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.2.3" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.2.3a.cmml">txt</mtext></msubsup></mrow><mo id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.1" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.1.cmml">−</mo><mi id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.3" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.3.cmml">β</mi></mrow><mo stretchy="false" id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.3" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.2.cmml">)</mo></mrow></mrow></mrow><mo id="A1.E2.m1.7.7.1.1.3.2.1.1.1.3" xref="A1.E2.m1.7.7.1.1.3.2.1.2.cmml">]</mo></mrow></mrow></mrow></mrow></mrow><mo lspace="0em" id="A1.E2.m1.7.7.1.2" xref="A1.E2.m1.7.7.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.E2.m1.7b"><apply id="A1.E2.m1.7.7.1.1.cmml" xref="A1.E2.m1.7.7.1"><eq id="A1.E2.m1.7.7.1.1.4.cmml" xref="A1.E2.m1.7.7.1.1.4"></eq><apply id="A1.E2.m1.7.7.1.1.1.cmml" xref="A1.E2.m1.7.7.1.1.1"><times id="A1.E2.m1.7.7.1.1.1.2.cmml" xref="A1.E2.m1.7.7.1.1.1.2"></times><ci id="A1.E2.m1.7.7.1.1.1.3.cmml" xref="A1.E2.m1.7.7.1.1.1.3">ℓ</ci><apply id="A1.E2.m1.7.7.1.1.1.1.1.1.cmml" xref="A1.E2.m1.7.7.1.1.1.1.1"><csymbol cd="latexml" id="A1.E2.m1.7.7.1.1.1.1.1.1.1.cmml" xref="A1.E2.m1.7.7.1.1.1.1.1.1.1">conditional</csymbol><apply id="A1.E2.m1.7.7.1.1.1.1.1.1.2.cmml" xref="A1.E2.m1.7.7.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A1.E2.m1.7.7.1.1.1.1.1.1.2.1.cmml" xref="A1.E2.m1.7.7.1.1.1.1.1.1.2">subscript</csymbol><ci id="A1.E2.m1.7.7.1.1.1.1.1.1.2.2.cmml" xref="A1.E2.m1.7.7.1.1.1.1.1.1.2.2">𝒙</ci><ci id="A1.E2.m1.7.7.1.1.1.1.1.1.2.3.cmml" xref="A1.E2.m1.7.7.1.1.1.1.1.1.2.3">𝑖</ci></apply><list id="A1.E2.m1.7.7.1.1.1.1.1.1.3.1.cmml" xref="A1.E2.m1.7.7.1.1.1.1.1.1.3.2"><ci id="A1.E2.m1.1.1.cmml" xref="A1.E2.m1.1.1">𝜃</ci><ci id="A1.E2.m1.2.2.cmml" xref="A1.E2.m1.2.2">ℬ</ci></list></apply></apply><apply id="A1.E2.m1.7.7.1.1.3.cmml" xref="A1.E2.m1.7.7.1.1.3"><plus id="A1.E2.m1.7.7.1.1.3.3.cmml" xref="A1.E2.m1.7.7.1.1.3.3"></plus><apply id="A1.E2.m1.7.7.1.1.2.1.2.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1"><log id="A1.E2.m1.4.4.cmml" xref="A1.E2.m1.4.4"></log><apply id="A1.E2.m1.7.7.1.1.2.1.1.1.1.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1"><plus id="A1.E2.m1.7.7.1.1.2.1.1.1.1.2.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.2"></plus><cn type="integer" id="A1.E2.m1.7.7.1.1.2.1.1.1.1.3.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.3">1</cn><apply id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.2.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1"><exp id="A1.E2.m1.3.3.cmml" xref="A1.E2.m1.3.3"></exp><apply id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1"><plus id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.1.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.1"></plus><apply id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2"><minus id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.1.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2"></minus><apply id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2"><ci id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.1.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.1">⋅</ci><apply id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2"><times id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.1.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.1"></times><ci id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.2.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.2">𝛼</ci><apply id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3"><csymbol cd="ambiguous" id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.1.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3">subscript</csymbol><apply id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.2.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3"><csymbol cd="ambiguous" id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.2.1.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3">superscript</csymbol><ci id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.2.2.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.2.2">𝒛</ci><ci id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.2.3a.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.2.3"><mtext mathsize="70%" id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.2.3.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.2.3">im</mtext></ci></apply><ci id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.3.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.2.3.3">𝑖</ci></apply></apply><apply id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3"><csymbol cd="ambiguous" id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.1.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3">subscript</csymbol><apply id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.2.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3"><csymbol cd="ambiguous" id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.2.1.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3">superscript</csymbol><ci id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.2.2.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.2.2">𝒛</ci><ci id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.2.3a.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.2.3"><mtext mathsize="70%" id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.2.3.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.2.3">txt</mtext></ci></apply><ci id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.3.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.2.2.3.3">𝑖</ci></apply></apply></apply><ci id="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.3.cmml" xref="A1.E2.m1.7.7.1.1.2.1.1.1.1.1.1.1.1.3">𝛽</ci></apply></apply></apply></apply><apply id="A1.E2.m1.7.7.1.1.3.2.cmml" xref="A1.E2.m1.7.7.1.1.3.2"><apply id="A1.E2.m1.7.7.1.1.3.2.2.cmml" xref="A1.E2.m1.7.7.1.1.3.2.2"><csymbol cd="ambiguous" id="A1.E2.m1.7.7.1.1.3.2.2.1.cmml" xref="A1.E2.m1.7.7.1.1.3.2.2">subscript</csymbol><sum id="A1.E2.m1.7.7.1.1.3.2.2.2.cmml" xref="A1.E2.m1.7.7.1.1.3.2.2.2"></sum><apply id="A1.E2.m1.7.7.1.1.3.2.2.3.cmml" xref="A1.E2.m1.7.7.1.1.3.2.2.3"><neq id="A1.E2.m1.7.7.1.1.3.2.2.3.1.cmml" xref="A1.E2.m1.7.7.1.1.3.2.2.3.1"></neq><ci id="A1.E2.m1.7.7.1.1.3.2.2.3.2.cmml" xref="A1.E2.m1.7.7.1.1.3.2.2.3.2">𝑗</ci><ci id="A1.E2.m1.7.7.1.1.3.2.2.3.3.cmml" xref="A1.E2.m1.7.7.1.1.3.2.2.3.3">𝑖</ci></apply></apply><apply id="A1.E2.m1.7.7.1.1.3.2.1.2.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1"><log id="A1.E2.m1.6.6.cmml" xref="A1.E2.m1.6.6"></log><apply id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1"><plus id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.2.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.2"></plus><cn type="integer" id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.3.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.3">1</cn><apply id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.2.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1"><exp id="A1.E2.m1.5.5.cmml" xref="A1.E2.m1.5.5"></exp><apply id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1"><minus id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.1.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.1"></minus><apply id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2"><ci id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.1.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.1">⋅</ci><apply id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2"><times id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.1.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.1"></times><ci id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.2.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.2">𝛼</ci><apply id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3"><csymbol cd="ambiguous" id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.1.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3">subscript</csymbol><apply id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.2.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3"><csymbol cd="ambiguous" id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.2.1.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3">superscript</csymbol><ci id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.2.2.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.2.2">𝒛</ci><ci id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.2.3a.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.2.3"><mtext mathsize="70%" id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.2.3.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.2.3">im</mtext></ci></apply><ci id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.3.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.2.3.3">𝑖</ci></apply></apply><apply id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.1.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3">subscript</csymbol><apply id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.2.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3"><csymbol cd="ambiguous" id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.2.1.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3">superscript</csymbol><ci id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.2.2.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.2.2">𝒛</ci><ci id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.2.3a.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.2.3"><mtext mathsize="70%" id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.2.3.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.2.3">txt</mtext></ci></apply><ci id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.3.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.2.3.3">𝑗</ci></apply></apply><ci id="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.3.cmml" xref="A1.E2.m1.7.7.1.1.3.2.1.1.1.1.1.1.1.1.3">𝛽</ci></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.E2.m1.7c">\ell({\bm{x}}_{i}|\theta,\mathcal{B})=\log\left[1+\exp(-\alpha{\bm{z}}^{\textrm{im}}_{i}{\cdot}{\bm{z}}^{\textrm{txt}}_{i}+\beta)\right]+\sum_{j\neq i}\log\left[1+\exp(\alpha{\bm{z}}^{\textrm{im}}_{i}{\cdot}{\bm{z}}^{\textrm{txt}}_{j}-\beta)\right].</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
</div>
<figure id="A1.F8" class="ltx_figure"><img src="/html/2406.17711/assets/Figures/fig_6_softmax_final.png" id="A1.F8.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="479" height="261" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 8: </span><span id="A1.F8.4.1" class="ltx_text ltx_font_bold">JEST is robust to the choice of contrastive loss (softmax vs. sigmoid).</span> We confirm the robustness of JEST to variations of contrastive learning by testing with the standard softmax-based loss. <span id="A1.F8.5.2" class="ltx_text ltx_font_bold">Left:</span> Similar to the SigLIP version, softmax contrastive learning is accelerated by JEST and benefits from increasing filtering ratios. <span id="A1.F8.6.3" class="ltx_text ltx_font_bold">Right:</span> Compared to the SigLIP loss, the uniform sampling softmax baseline is slightly worse, but relative improvements from JEST are maintained.</figcaption>
</figure>
<div id="A1.SS6.p4" class="ltx_para ltx_noindent">
<p id="A1.SS6.p4.1" class="ltx_p">Although we leverage the sigmoid pairwise contrastive loss (SigLIP) formulation for our main results, a natural question is whether JEST benefits the standard softmax contrastive learning in Eq. <a href="#A1.E1" title="In A.6 Contrastive loss ablations ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. Due to the formulation of the loss, the JEST algorithm is slightly different from the sigmoid version. We detail the joint-example selection algorithm in Alg. <a href="#algorithm3" title="Algorithm 3 ‣ A.6 Contrastive loss ablations ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, but the main training loop remains the same as in Alg. <a href="#algorithm2" title="Algorithm 2 ‣ A.1 Training configuration ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>. We note that unlike the sigmoid version, the softmax loss requires re-computing the softmax during the iterative conditional sampling. This leads to inefficiencies relative to the SigLIP formulation, especially when the batch is split over a large number of devices.</p>
</div>
<figure id="algorithm3" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="algorithm3.2.1.1" class="ltx_text ltx_font_bold">Algorithm 3</span> </span> Joint example selection: softmax loss</figcaption>
<div id="algorithm3.3" class="ltx_listing ltx_lst_language_python ltx_lstlisting ltx_listing" style="background-color:#F5F5FA;">
<div class="ltx_listing_data"><a href="data:text/plain;base64,ZGVmIHNvZnRtYXhfbmVnKGxvZ2l0cywgaXNfc2FtcGxlZD1Ob25lLCBheGlzPS0xKToKICAgICMgZHVyaW5nIHRoZSBpdGVyYXRpdmUgc2FtcGxpbmcsIHdlIG5lZWQgdG8gbWFzayBvdXQgbmVnYXRpdmVzIHRoYXQgaGF2ZSBhbHJlYWR5IGJlZW4gc2VsZWN0ZWQKICAgIGlmIGlzX3NhbXBsZWQgaXMgbm90IE5vbmU6CiAgICAgICAgbG9naXRzX25lZyA9IGxvZ2l0cyAtICgxLi0gaXNfc2FtcGxlZCkqMWU4CiAgICBlbHNlOgogICAgICAgIGxvZ2l0c19uZWcgPSBsb2dpdHMKICAgIGxvZ2l0c19uZWcgPSBubi5sb2dzdW1leHAobG9naXRzX25lZywgYXhpcz1heGlzKSAjIENvbXB1dGUgdGhlIHNvZnRtYXggbmVnYXRpdmUgdGVybQogICAgcmV0dXJuIGxvZ2l0c19uZWcKCmRlZiBzb2Z0bWF4X25sbChwYXJhbXMsIGVtYmVkcywgaXNfc2FtcGxlZD1Ob25lKToKICAgIHppbWcsIHp0eHQgPSBlbWJlZHMKICAgIGxvZ2l0c19tYXQgPSBucC5kb3QoemltZywgenR4dC5UKSAqIHBhcmFtc1sidCJdCiAgICBpZiBpc19zYW1wbGVkIGlzIG5vdCBOb25lOgogICAgICAgIGlzX3NhbXBsZWRfMCA9IGlzX3NhbXBsZWQucmVzaGFwZShsb2dpdHNfbWF0LnNoYXBlWzBdLCAxKQogICAgICAgIGlzX3NhbXBsZWRfMSA9IGlzX3NhbXBsZWQucmVzaGFwZSgxLCBsb2dpdHNfbWF0LnNoYXBlWzBdKQogICAgZWxzZToKICAgICAgICBpc19zYW1wbGVkXzAgPSBOb25lCiAgICAgICAgaXNfc2FtcGxlZF8xID0gTm9uZQoKICAgIGxvZ2l0c19paiA9IHNvZnRtYXhfbmVnKGxvZ2l0c19tYXQsIGlzX3NhbXBsZWRfMCwgYXhpcz0wKSAjIGNhbGN1bGF0ZSBuZWdhdGl2ZSBzb2Z0bWF4IHRlcm0gZm9yIGltYWdlIHRvIHRleHQgaGFsZiBvZiBsb3NzCiAgICBsb2dpdHNfamkgPSBzb2Z0bWF4X25lZyhsb2dpdHNfbWF0LCBpc19zYW1wbGVkXzEsIGF4aXM9MSkgIyBjYWxjdWxhdGUgbmVnYXRpdmUgc29mdG1heCB0ZXJtIGZvciB0ZXh0IHRvIGltYWdlIGhhbGYgb2YgbG9zcwogICAgbG9zc18wID0gLShqbnAuZGlhZyhsb2dpdHNfbWF0KSAtIGxvZ2l0c19paikKICAgIGxvc3NfMSA9IC0oam5wLmRpYWcobG9naXRzX21hdCkgLSBsb2dpdHNfamkpCiAgICBuZWdfbG9naXRzID0gMC41Kihsb2dpdHNfaWogKyBsb2dpdHNfamkpCgogICAgbCA9IGpucC5tZWFuKDAuNSoobG9zc18wICsgbG9zc18xKSkKICAgIHJldHVybiBsLCBuZWdfbG9naXRzLCAtbG9naXRzX21hdAoKCmRlZiBqb2ludGx5X3NhbXBsZV9iYXRjaF9zb2Z0bWF4KGVtYmVkc19yZWYsIGVtYmVkc19sZWFybmVyLCBwYXJhbXNfcmVmLCBwYXJhbXNfbGVhcm5lciwgbl9jaHVua3M9MTYsIGZpbHRlcl9yYXRpbz0wLjgpOgogICAgXywgXywgbGVhcm5lcl9sb2dpdHMgPSBzb2Z0bWF4X25sbChwYXJhbXNfbGVhcm5lciwgZW1iZWRzX2xlYXJuZXIsIGlzX3NhbXBsZWQ9Tm9uZSkKICAgIF8sIF8sIHJlZl9sb2dpdHMgPSBzb2ZtdGF4X25sbChwYXJhbXNfbGVhcm5lciwgZW1iZWRzX2xlYXJuZXIsIGlzX3NhbXBsZWQ9Tm9uZSkKICAgIHNjb3JlcyA9IChsZWFybmVyX2xvZ2l0cyAtIHJlZl9sb2dpdHMpKmNmZy5zb2Z0bWF4X3Njb3JlX2dhaW4KICAgIG5faW1hZ2VzID0gc2NvcmVzLnNoYXBlWzBdICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgc2NvcmVzLnNoYXBlID0gW0IsIEJdCiAgICBuX2RyYXdzID0gaW50KG5faW1hZ2VzICogKDEgLSBmaWx0ZXJfcmF0aW8pIC8gbl9jaHVua3MpICAjIFNpemUgb2YgZWFjaCBjaHVuay4KICAgIGxvZ2l0c19paSA9IG5wLmRpYWcoc2NvcmVzKSAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgU2VsZi1zaW1pbGFyaXR5IHNjb3Jlcy4KICAgIGluZHMgPSByYW5kb20uY2hvaWNlKGxvZ2l0c19paSwgbl9kcmF3cykKICAgICMgU2FtcGxlIGZpcnN0IGNodW5rLgoKICAgIGZvciBfIGluIHJhbmdlKG5fY2h1bmtzIC0gMSk6CiAgICAgICAgaXNfc2FtcGxlZCA9IG5wLmV5ZShuX2ltYWdlcylbaW5kc10uc3VtKGF4aXM9MCkgIyBCaW5hcnkgaW5kaWNhdG9yIG9mIGN1cnJlbnQgc2FtcGxlcyBbbl9pbWFnZXMsXS4KICAgICAgICBfLCBsZWFybmVyX2xvZ2l0c19uLCBfID0gc29mdG1heF9ubGwocGFyYW1zX2xlYXJuZXIsIGVtYmVkc19sZWFybmVyLCBpc19zYW1wbGVkPWlzX3NhbXBsZWQpCiAgICAgICAgXywgcmVmX2xvZ2l0c19uLCBfID0gc29mdG1heF9ubGwocGFyYW1zX3JlZiwgZW1iZWRzX3JlZiwgaXNfc2FtcGxlZD1pc19zYW1wbGVkKQogICAgICAgIHJob19zY29yZXNfbiA9IChsZWFybmVyX2xvZ2l0c19uIC0gcmVmX2xvZ2l0c19uKSpjZmcuc29mdG1heF9zY29yZV9nYWluCiAgICAgICAgbG9naXRzID0gbG9naXRzX2lpICsgcmhvX3Njb3Jlc19uICAgICAgIyBDb25kaXRpb25hbCBsZWFybmFiaWxpdHkgZ2l2ZW4gcGFzdCBzYW1wbGVzLgogICAgICAgIGxvZ2l0cyA9IGxvZ2l0cyAtIGlzX3NhbXBsZWQgKiAxZTggICAgICAgICAgICAgICMgQXZvaWQgc2FtcGxpbmcgd2l0aCByZXBsYWNlbWVudC4KICAgICAgICBuZXdfaW5kcyA9IHJhbmRvbS5jaG9pY2Uobl9pbWFnZXMsIG5fZHJhd3MsIHA9bnAuZXhwKGxvZ2l0cykpCiAgICAgICAgaW5kcyA9IG5wLmNvbmNhdGVuYXRlKChpbmRzLCBuZXdfaW5kcykpICAgICAgICAgIyBFeHBhbmQgdGhlIGFycmF5IG9mIGluZGljZXMgc2FtcGxlZC4KICAgICAgICByZXR1cm4gaW5kcyAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICMgR2F0aGVyIGFuZCByZXR1cm4gc3Vic2V0IGluZGljZXMuCgo=" download="">⬇</a></div>
<div id="lstnumberx78" class="ltx_listingline">
<span id="lstnumberx78.1" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">def</span><span id="lstnumberx78.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx78.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">softmax_neg</span><span id="lstnumberx78.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx78.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits</span><span id="lstnumberx78.6" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx78.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx78.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx78.9" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx78.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">None</span><span id="lstnumberx78.11" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx78.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx78.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">axis</span><span id="lstnumberx78.14" class="ltx_text ltx_font_typewriter">=-1):</span>
</div>
<div id="lstnumberx79" class="ltx_listingline">
<span id="lstnumberx79.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx79.2" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx79.2.1" class="ltx_text ltx_lst_space"> </span>during<span id="lstnumberx79.2.2" class="ltx_text ltx_lst_space"> </span>the<span id="lstnumberx79.2.3" class="ltx_text ltx_lst_space"> </span>iterative<span id="lstnumberx79.2.4" class="ltx_text ltx_lst_space"> </span>sampling,<span id="lstnumberx79.2.5" class="ltx_text ltx_lst_space"> </span>we<span id="lstnumberx79.2.6" class="ltx_text ltx_lst_space"> </span>need<span id="lstnumberx79.2.7" class="ltx_text ltx_lst_space"> </span>to<span id="lstnumberx79.2.8" class="ltx_text ltx_lst_space"> </span>mask<span id="lstnumberx79.2.9" class="ltx_text ltx_lst_space"> </span>out<span id="lstnumberx79.2.10" class="ltx_text ltx_lst_space"> </span>negatives<span id="lstnumberx79.2.11" class="ltx_text ltx_lst_space"> </span>that<span id="lstnumberx79.2.12" class="ltx_text ltx_lst_space"> </span>have<span id="lstnumberx79.2.13" class="ltx_text ltx_lst_space"> </span>already<span id="lstnumberx79.2.14" class="ltx_text ltx_lst_space"> </span>been<span id="lstnumberx79.2.15" class="ltx_text ltx_lst_space"> </span>selected</span>
</div>
<div id="lstnumberx80" class="ltx_listingline">
<span id="lstnumberx80.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx80.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">if</span><span id="lstnumberx80.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx80.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx80.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx80.6" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">is</span><span id="lstnumberx80.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx80.8" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">not</span><span id="lstnumberx80.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx80.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">None</span><span id="lstnumberx80.11" class="ltx_text ltx_font_typewriter">:</span>
</div>
<div id="lstnumberx81" class="ltx_listingline">
<span id="lstnumberx81.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx81.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_neg</span><span id="lstnumberx81.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx81.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx81.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx81.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits</span><span id="lstnumberx81.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx81.8" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx81.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx81.10" class="ltx_text ltx_font_typewriter">(1.-</span><span id="lstnumberx81.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx81.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx81.13" class="ltx_text ltx_font_typewriter">)*1</span><span id="lstnumberx81.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">e8</span>
</div>
<div id="lstnumberx82" class="ltx_listingline">
<span id="lstnumberx82.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx82.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">else</span><span id="lstnumberx82.3" class="ltx_text ltx_font_typewriter">:</span>
</div>
<div id="lstnumberx83" class="ltx_listingline">
<span id="lstnumberx83.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx83.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_neg</span><span id="lstnumberx83.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx83.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx83.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx83.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits</span>
</div>
<div id="lstnumberx84" class="ltx_listingline">
<span id="lstnumberx84.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx84.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_neg</span><span id="lstnumberx84.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx84.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">nn</span><span id="lstnumberx84.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx84.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logsumexp</span><span id="lstnumberx84.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx84.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_neg</span><span id="lstnumberx84.11" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx84.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">axis</span><span id="lstnumberx84.14" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx84.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">axis</span><span id="lstnumberx84.16" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx84.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx84.18" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx84.18.1" class="ltx_text ltx_lst_space"> </span>Compute<span id="lstnumberx84.18.2" class="ltx_text ltx_lst_space"> </span>the<span id="lstnumberx84.18.3" class="ltx_text ltx_lst_space"> </span>softmax<span id="lstnumberx84.18.4" class="ltx_text ltx_lst_space"> </span>negative<span id="lstnumberx84.18.5" class="ltx_text ltx_lst_space"> </span>term</span>
</div>
<div id="lstnumberx85" class="ltx_listingline">
<span id="lstnumberx85.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx85.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">return</span><span id="lstnumberx85.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx85.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_neg</span>
</div>
<div id="lstnumberx86" class="ltx_listingline">
</div>
<div id="lstnumberx87" class="ltx_listingline">
<span id="lstnumberx87.1" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">def</span><span id="lstnumberx87.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx87.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">softmax_nll</span><span id="lstnumberx87.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx87.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params</span><span id="lstnumberx87.6" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx87.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx87.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds</span><span id="lstnumberx87.9" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx87.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx87.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx87.12" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx87.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">None</span><span id="lstnumberx87.14" class="ltx_text ltx_font_typewriter">):</span>
</div>
<div id="lstnumberx88" class="ltx_listingline">
<span id="lstnumberx88.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx88.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">zimg</span><span id="lstnumberx88.3" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx88.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx88.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ztxt</span><span id="lstnumberx88.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx88.7" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx88.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx88.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds</span>
</div>
<div id="lstnumberx89" class="ltx_listingline">
<span id="lstnumberx89.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx89.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_mat</span><span id="lstnumberx89.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx89.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx89.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx89.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">np</span><span id="lstnumberx89.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx89.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">dot</span><span id="lstnumberx89.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx89.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">zimg</span><span id="lstnumberx89.11" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx89.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx89.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ztxt</span><span id="lstnumberx89.14" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx89.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">T</span><span id="lstnumberx89.16" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx89.17" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx89.18" class="ltx_text ltx_font_typewriter">*</span><span id="lstnumberx89.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx89.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params</span><span id="lstnumberx89.21" class="ltx_text ltx_font_typewriter">[</span><span id="lstnumberx89.22" class="ltx_text ltx_lst_string ltx_font_typewriter">"t"</span><span id="lstnumberx89.23" class="ltx_text ltx_font_typewriter">]</span>
</div>
<div id="lstnumberx90" class="ltx_listingline">
<span id="lstnumberx90.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx90.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">if</span><span id="lstnumberx90.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx90.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx90.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx90.6" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">is</span><span id="lstnumberx90.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx90.8" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">not</span><span id="lstnumberx90.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx90.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">None</span><span id="lstnumberx90.11" class="ltx_text ltx_font_typewriter">:</span>
</div>
<div id="lstnumberx91" class="ltx_listingline">
<span id="lstnumberx91.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx91.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled_0</span><span id="lstnumberx91.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx91.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx91.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx91.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx91.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx91.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">reshape</span><span id="lstnumberx91.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx91.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_mat</span><span id="lstnumberx91.11" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx91.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">shape</span><span id="lstnumberx91.13" class="ltx_text ltx_font_typewriter">[0],</span><span id="lstnumberx91.14" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx91.15" class="ltx_text ltx_font_typewriter">1)</span>
</div>
<div id="lstnumberx92" class="ltx_listingline">
<span id="lstnumberx92.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx92.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled_1</span><span id="lstnumberx92.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx92.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx92.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx92.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx92.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx92.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">reshape</span><span id="lstnumberx92.9" class="ltx_text ltx_font_typewriter">(1,</span><span id="lstnumberx92.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx92.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_mat</span><span id="lstnumberx92.12" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx92.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">shape</span><span id="lstnumberx92.14" class="ltx_text ltx_font_typewriter">[0])</span>
</div>
<div id="lstnumberx93" class="ltx_listingline">
<span id="lstnumberx93.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx93.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">else</span><span id="lstnumberx93.3" class="ltx_text ltx_font_typewriter">:</span>
</div>
<div id="lstnumberx94" class="ltx_listingline">
<span id="lstnumberx94.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx94.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled_0</span><span id="lstnumberx94.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx94.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx94.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">None</span>
</div>
<div id="lstnumberx95" class="ltx_listingline">
<span id="lstnumberx95.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx95.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled_1</span><span id="lstnumberx95.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx95.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx95.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">None</span>
</div>
<div id="lstnumberx96" class="ltx_listingline">
</div>
<div id="lstnumberx97" class="ltx_listingline">
<span id="lstnumberx97.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx97.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_ij</span><span id="lstnumberx97.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx97.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">softmax_neg</span><span id="lstnumberx97.7" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx97.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_mat</span><span id="lstnumberx97.9" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx97.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled_0</span><span id="lstnumberx97.12" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx97.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">axis</span><span id="lstnumberx97.15" class="ltx_text ltx_font_typewriter">=0)</span><span id="lstnumberx97.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx97.17" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx97.17.1" class="ltx_text ltx_lst_space"> </span>calculate<span id="lstnumberx97.17.2" class="ltx_text ltx_lst_space"> </span>negative<span id="lstnumberx97.17.3" class="ltx_text ltx_lst_space"> </span>softmax<span id="lstnumberx97.17.4" class="ltx_text ltx_lst_space"> </span>term<span id="lstnumberx97.17.5" class="ltx_text ltx_lst_space"> </span>for<span id="lstnumberx97.17.6" class="ltx_text ltx_lst_space"> </span>image<span id="lstnumberx97.17.7" class="ltx_text ltx_lst_space"> </span>to<span id="lstnumberx97.17.8" class="ltx_text ltx_lst_space"> </span>text<span id="lstnumberx97.17.9" class="ltx_text ltx_lst_space"> </span>half<span id="lstnumberx97.17.10" class="ltx_text ltx_lst_space"> </span>of<span id="lstnumberx97.17.11" class="ltx_text ltx_lst_space"> </span>loss</span>
</div>
<div id="lstnumberx98" class="ltx_listingline">
<span id="lstnumberx98.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx98.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_ji</span><span id="lstnumberx98.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx98.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx98.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx98.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">softmax_neg</span><span id="lstnumberx98.7" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx98.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_mat</span><span id="lstnumberx98.9" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx98.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx98.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled_1</span><span id="lstnumberx98.12" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx98.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx98.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">axis</span><span id="lstnumberx98.15" class="ltx_text ltx_font_typewriter">=1)</span><span id="lstnumberx98.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx98.17" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx98.17.1" class="ltx_text ltx_lst_space"> </span>calculate<span id="lstnumberx98.17.2" class="ltx_text ltx_lst_space"> </span>negative<span id="lstnumberx98.17.3" class="ltx_text ltx_lst_space"> </span>softmax<span id="lstnumberx98.17.4" class="ltx_text ltx_lst_space"> </span>term<span id="lstnumberx98.17.5" class="ltx_text ltx_lst_space"> </span>for<span id="lstnumberx98.17.6" class="ltx_text ltx_lst_space"> </span>text<span id="lstnumberx98.17.7" class="ltx_text ltx_lst_space"> </span>to<span id="lstnumberx98.17.8" class="ltx_text ltx_lst_space"> </span>image<span id="lstnumberx98.17.9" class="ltx_text ltx_lst_space"> </span>half<span id="lstnumberx98.17.10" class="ltx_text ltx_lst_space"> </span>of<span id="lstnumberx98.17.11" class="ltx_text ltx_lst_space"> </span>loss</span>
</div>
<div id="lstnumberx99" class="ltx_listingline">
<span id="lstnumberx99.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx99.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">loss_0</span><span id="lstnumberx99.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx99.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.6" class="ltx_text ltx_font_typewriter">-(</span><span id="lstnumberx99.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">jnp</span><span id="lstnumberx99.8" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx99.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">diag</span><span id="lstnumberx99.10" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx99.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_mat</span><span id="lstnumberx99.12" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx99.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.14" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx99.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx99.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_ij</span><span id="lstnumberx99.17" class="ltx_text ltx_font_typewriter">)</span>
</div>
<div id="lstnumberx100" class="ltx_listingline">
<span id="lstnumberx100.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx100.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">loss_1</span><span id="lstnumberx100.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx100.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx100.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx100.6" class="ltx_text ltx_font_typewriter">-(</span><span id="lstnumberx100.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">jnp</span><span id="lstnumberx100.8" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx100.9" class="ltx_text ltx_lst_identifier ltx_font_typewriter">diag</span><span id="lstnumberx100.10" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx100.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_mat</span><span id="lstnumberx100.12" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx100.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx100.14" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx100.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx100.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_ji</span><span id="lstnumberx100.17" class="ltx_text ltx_font_typewriter">)</span>
</div>
<div id="lstnumberx101" class="ltx_listingline">
<span id="lstnumberx101.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx101.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">neg_logits</span><span id="lstnumberx101.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx101.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx101.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx101.6" class="ltx_text ltx_font_typewriter">0.5*(</span><span id="lstnumberx101.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_ij</span><span id="lstnumberx101.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx101.9" class="ltx_text ltx_font_typewriter">+</span><span id="lstnumberx101.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx101.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_ji</span><span id="lstnumberx101.12" class="ltx_text ltx_font_typewriter">)</span>
</div>
<div id="lstnumberx102" class="ltx_listingline">
</div>
<div id="lstnumberx103" class="ltx_listingline">
<span id="lstnumberx103.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx103.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">l</span><span id="lstnumberx103.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx103.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx103.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx103.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">jnp</span><span id="lstnumberx103.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx103.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">mean</span><span id="lstnumberx103.9" class="ltx_text ltx_font_typewriter">(0.5*(</span><span id="lstnumberx103.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">loss_0</span><span id="lstnumberx103.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx103.12" class="ltx_text ltx_font_typewriter">+</span><span id="lstnumberx103.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx103.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">loss_1</span><span id="lstnumberx103.15" class="ltx_text ltx_font_typewriter">))</span>
</div>
<div id="lstnumberx104" class="ltx_listingline">
<span id="lstnumberx104.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx104.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">return</span><span id="lstnumberx104.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">l</span><span id="lstnumberx104.5" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx104.6" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">neg_logits</span><span id="lstnumberx104.8" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx104.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx104.10" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx104.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_mat</span>
</div>
<div id="lstnumberx105" class="ltx_listingline">
</div>
<div id="lstnumberx106" class="ltx_listingline">
</div>
<div id="lstnumberx107" class="ltx_listingline">
<span id="lstnumberx107.1" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">def</span><span id="lstnumberx107.2" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx107.3" class="ltx_text ltx_lst_identifier ltx_font_typewriter">jointly_sample_batch_softmax</span><span id="lstnumberx107.4" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx107.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds_ref</span><span id="lstnumberx107.6" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx107.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx107.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds_learner</span><span id="lstnumberx107.9" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx107.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx107.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params_ref</span><span id="lstnumberx107.12" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx107.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx107.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params_learner</span><span id="lstnumberx107.15" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx107.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx107.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_chunks</span><span id="lstnumberx107.18" class="ltx_text ltx_font_typewriter">=16,</span><span id="lstnumberx107.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx107.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">filter_ratio</span><span id="lstnumberx107.21" class="ltx_text ltx_font_typewriter">=0.8):</span>
</div>
<div id="lstnumberx108" class="ltx_listingline">
<span id="lstnumberx108.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx108.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">_</span><span id="lstnumberx108.3" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx108.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx108.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">_</span><span id="lstnumberx108.6" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx108.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx108.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">learner_logits</span><span id="lstnumberx108.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx108.10" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx108.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx108.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">softmax_nll</span><span id="lstnumberx108.13" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx108.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params_learner</span><span id="lstnumberx108.15" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx108.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx108.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds_learner</span><span id="lstnumberx108.18" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx108.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx108.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx108.21" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx108.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">None</span><span id="lstnumberx108.23" class="ltx_text ltx_font_typewriter">)</span>
</div>
<div id="lstnumberx109" class="ltx_listingline">
<span id="lstnumberx109.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx109.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">_</span><span id="lstnumberx109.3" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx109.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">_</span><span id="lstnumberx109.6" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx109.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ref_logits</span><span id="lstnumberx109.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.10" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx109.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">sofmtax_nll</span><span id="lstnumberx109.13" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx109.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params_learner</span><span id="lstnumberx109.15" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx109.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds_learner</span><span id="lstnumberx109.18" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx109.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx109.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx109.21" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx109.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">None</span><span id="lstnumberx109.23" class="ltx_text ltx_font_typewriter">)</span>
</div>
<div id="lstnumberx110" class="ltx_listingline">
<span id="lstnumberx110.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx110.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">scores</span><span id="lstnumberx110.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx110.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx110.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx110.6" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx110.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">learner_logits</span><span id="lstnumberx110.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx110.9" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx110.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx110.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ref_logits</span><span id="lstnumberx110.12" class="ltx_text ltx_font_typewriter">)*</span><span id="lstnumberx110.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cfg</span><span id="lstnumberx110.14" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx110.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">softmax_score_gain</span>
</div>
<div id="lstnumberx111" class="ltx_listingline">
<span id="lstnumberx111.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx111.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_images</span><span id="lstnumberx111.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx111.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx111.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx111.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">scores</span><span id="lstnumberx111.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx111.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">shape</span><span id="lstnumberx111.9" class="ltx_text ltx_font_typewriter">[0]</span><span id="lstnumberx111.10" class="ltx_text ltx_lst_space ltx_font_typewriter">                               </span><span id="lstnumberx111.11" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx111.11.1" class="ltx_text ltx_lst_space"> </span>scores.shape<span id="lstnumberx111.11.2" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx111.11.3" class="ltx_text ltx_lst_space"> </span>[B,<span id="lstnumberx111.11.4" class="ltx_text ltx_lst_space"> </span>B]</span>
</div>
<div id="lstnumberx112" class="ltx_listingline">
<span id="lstnumberx112.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx112.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_draws</span><span id="lstnumberx112.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx112.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx112.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx112.6" class="ltx_text ltx_lst_keyword ltx_lst_keywords2 ltx_font_typewriter" style="color:#408080;">int</span><span id="lstnumberx112.7" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx112.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_images</span><span id="lstnumberx112.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx112.10" class="ltx_text ltx_font_typewriter">*</span><span id="lstnumberx112.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx112.12" class="ltx_text ltx_font_typewriter">(1</span><span id="lstnumberx112.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx112.14" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx112.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx112.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">filter_ratio</span><span id="lstnumberx112.17" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx112.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx112.19" class="ltx_text ltx_font_typewriter">/</span><span id="lstnumberx112.20" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx112.21" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_chunks</span><span id="lstnumberx112.22" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx112.23" class="ltx_text ltx_lst_space ltx_font_typewriter">  </span><span id="lstnumberx112.24" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx112.24.1" class="ltx_text ltx_lst_space"> </span>Size<span id="lstnumberx112.24.2" class="ltx_text ltx_lst_space"> </span>of<span id="lstnumberx112.24.3" class="ltx_text ltx_lst_space"> </span>each<span id="lstnumberx112.24.4" class="ltx_text ltx_lst_space"> </span>chunk.</span>
</div>
<div id="lstnumberx113" class="ltx_listingline">
<span id="lstnumberx113.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx113.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_ii</span><span id="lstnumberx113.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx113.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx113.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx113.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">np</span><span id="lstnumberx113.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx113.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">diag</span><span id="lstnumberx113.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx113.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">scores</span><span id="lstnumberx113.11" class="ltx_text ltx_font_typewriter">)</span><span id="lstnumberx113.12" class="ltx_text ltx_lst_space ltx_font_typewriter">                              </span><span id="lstnumberx113.13" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx113.13.1" class="ltx_text ltx_lst_space"> </span>Self-similarity<span id="lstnumberx113.13.2" class="ltx_text ltx_lst_space"> </span>scores.</span>
</div>
<div id="lstnumberx114" class="ltx_listingline">
<span id="lstnumberx114.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx114.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">inds</span><span id="lstnumberx114.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx114.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx114.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx114.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">random</span><span id="lstnumberx114.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx114.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">choice</span><span id="lstnumberx114.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx114.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_ii</span><span id="lstnumberx114.11" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx114.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx114.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_draws</span><span id="lstnumberx114.14" class="ltx_text ltx_font_typewriter">)</span>
</div>
<div id="lstnumberx115" class="ltx_listingline">
<span id="lstnumberx115.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx115.2" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx115.2.1" class="ltx_text ltx_lst_space"> </span>Sample<span id="lstnumberx115.2.2" class="ltx_text ltx_lst_space"> </span>first<span id="lstnumberx115.2.3" class="ltx_text ltx_lst_space"> </span>chunk.</span>
</div>
<div id="lstnumberx116" class="ltx_listingline">
</div>
<div id="lstnumberx117" class="ltx_listingline">
<span id="lstnumberx117.1" class="ltx_text ltx_lst_space ltx_font_typewriter">    </span><span id="lstnumberx117.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">for</span><span id="lstnumberx117.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">_</span><span id="lstnumberx117.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.6" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">in</span><span id="lstnumberx117.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.8" class="ltx_text ltx_lst_keyword ltx_lst_keywords2 ltx_font_typewriter" style="color:#408080;">range</span><span id="lstnumberx117.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx117.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_chunks</span><span id="lstnumberx117.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.12" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx117.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx117.14" class="ltx_text ltx_font_typewriter">1):</span>
</div>
<div id="lstnumberx118" class="ltx_listingline">
<span id="lstnumberx118.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx118.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx118.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx118.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">np</span><span id="lstnumberx118.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx118.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">eye</span><span id="lstnumberx118.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx118.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_images</span><span id="lstnumberx118.11" class="ltx_text ltx_font_typewriter">)[</span><span id="lstnumberx118.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">inds</span><span id="lstnumberx118.13" class="ltx_text ltx_font_typewriter">].</span><span id="lstnumberx118.14" class="ltx_text ltx_lst_keyword ltx_lst_keywords2 ltx_font_typewriter" style="color:#408080;">sum</span><span id="lstnumberx118.15" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx118.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">axis</span><span id="lstnumberx118.17" class="ltx_text ltx_font_typewriter">=0)</span><span id="lstnumberx118.18" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx118.19" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx118.19.1" class="ltx_text ltx_lst_space"> </span>Binary<span id="lstnumberx118.19.2" class="ltx_text ltx_lst_space"> </span>indicator<span id="lstnumberx118.19.3" class="ltx_text ltx_lst_space"> </span>of<span id="lstnumberx118.19.4" class="ltx_text ltx_lst_space"> </span>current<span id="lstnumberx118.19.5" class="ltx_text ltx_lst_space"> </span>samples<span id="lstnumberx118.19.6" class="ltx_text ltx_lst_space"> </span>[n_images,].</span>
</div>
<div id="lstnumberx119" class="ltx_listingline">
<span id="lstnumberx119.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx119.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">_</span><span id="lstnumberx119.3" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx119.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">learner_logits_n</span><span id="lstnumberx119.6" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx119.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">_</span><span id="lstnumberx119.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.10" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx119.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">softmax_nll</span><span id="lstnumberx119.13" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx119.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params_learner</span><span id="lstnumberx119.15" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx119.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds_learner</span><span id="lstnumberx119.18" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx119.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx119.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx119.21" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx119.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx119.23" class="ltx_text ltx_font_typewriter">)</span>
</div>
<div id="lstnumberx120" class="ltx_listingline">
<span id="lstnumberx120.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx120.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">_</span><span id="lstnumberx120.3" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx120.4" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.5" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ref_logits_n</span><span id="lstnumberx120.6" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx120.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">_</span><span id="lstnumberx120.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.10" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx120.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.12" class="ltx_text ltx_lst_identifier ltx_font_typewriter">softmax_nll</span><span id="lstnumberx120.13" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx120.14" class="ltx_text ltx_lst_identifier ltx_font_typewriter">params_ref</span><span id="lstnumberx120.15" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx120.16" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.17" class="ltx_text ltx_lst_identifier ltx_font_typewriter">embeds_ref</span><span id="lstnumberx120.18" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx120.19" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx120.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx120.21" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx120.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx120.23" class="ltx_text ltx_font_typewriter">)</span>
</div>
<div id="lstnumberx121" class="ltx_listingline">
<span id="lstnumberx121.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx121.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">rho_scores_n</span><span id="lstnumberx121.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx121.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.6" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx121.7" class="ltx_text ltx_lst_identifier ltx_font_typewriter">learner_logits_n</span><span id="lstnumberx121.8" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.9" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx121.10" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx121.11" class="ltx_text ltx_lst_identifier ltx_font_typewriter">ref_logits_n</span><span id="lstnumberx121.12" class="ltx_text ltx_font_typewriter">)*</span><span id="lstnumberx121.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">cfg</span><span id="lstnumberx121.14" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx121.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">softmax_score_gain</span>
</div>
<div id="lstnumberx122" class="ltx_listingline">
<span id="lstnumberx122.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx122.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits</span><span id="lstnumberx122.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx122.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx122.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx122.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits_ii</span><span id="lstnumberx122.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx122.8" class="ltx_text ltx_font_typewriter">+</span><span id="lstnumberx122.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx122.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">rho_scores_n</span><span id="lstnumberx122.11" class="ltx_text ltx_lst_space ltx_font_typewriter">      </span><span id="lstnumberx122.12" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx122.12.1" class="ltx_text ltx_lst_space"> </span>Conditional<span id="lstnumberx122.12.2" class="ltx_text ltx_lst_space"> </span>learnability<span id="lstnumberx122.12.3" class="ltx_text ltx_lst_space"> </span>given<span id="lstnumberx122.12.4" class="ltx_text ltx_lst_space"> </span>past<span id="lstnumberx122.12.5" class="ltx_text ltx_lst_space"> </span>samples.</span>
</div>
<div id="lstnumberx123" class="ltx_listingline">
<span id="lstnumberx123.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx123.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits</span><span id="lstnumberx123.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx123.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits</span><span id="lstnumberx123.7" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.8" class="ltx_text ltx_font_typewriter">-</span><span id="lstnumberx123.9" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">is_sampled</span><span id="lstnumberx123.11" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.12" class="ltx_text ltx_font_typewriter">*</span><span id="lstnumberx123.13" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx123.14" class="ltx_text ltx_font_typewriter">1</span><span id="lstnumberx123.15" class="ltx_text ltx_lst_identifier ltx_font_typewriter">e8</span><span id="lstnumberx123.16" class="ltx_text ltx_lst_space ltx_font_typewriter">              </span><span id="lstnumberx123.17" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx123.17.1" class="ltx_text ltx_lst_space"> </span>Avoid<span id="lstnumberx123.17.2" class="ltx_text ltx_lst_space"> </span>sampling<span id="lstnumberx123.17.3" class="ltx_text ltx_lst_space"> </span>with<span id="lstnumberx123.17.4" class="ltx_text ltx_lst_space"> </span>replacement.</span>
</div>
<div id="lstnumberx124" class="ltx_listingline">
<span id="lstnumberx124.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx124.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">new_inds</span><span id="lstnumberx124.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx124.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx124.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx124.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">random</span><span id="lstnumberx124.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx124.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">choice</span><span id="lstnumberx124.9" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx124.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_images</span><span id="lstnumberx124.11" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx124.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx124.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">n_draws</span><span id="lstnumberx124.14" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx124.15" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx124.16" class="ltx_text ltx_lst_identifier ltx_font_typewriter">p</span><span id="lstnumberx124.17" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx124.18" class="ltx_text ltx_lst_identifier ltx_font_typewriter">np</span><span id="lstnumberx124.19" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx124.20" class="ltx_text ltx_lst_identifier ltx_font_typewriter">exp</span><span id="lstnumberx124.21" class="ltx_text ltx_font_typewriter">(</span><span id="lstnumberx124.22" class="ltx_text ltx_lst_identifier ltx_font_typewriter">logits</span><span id="lstnumberx124.23" class="ltx_text ltx_font_typewriter">))</span>
</div>
<div id="lstnumberx125" class="ltx_listingline">
<span id="lstnumberx125.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx125.2" class="ltx_text ltx_lst_identifier ltx_font_typewriter">inds</span><span id="lstnumberx125.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx125.4" class="ltx_text ltx_font_typewriter">=</span><span id="lstnumberx125.5" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx125.6" class="ltx_text ltx_lst_identifier ltx_font_typewriter">np</span><span id="lstnumberx125.7" class="ltx_text ltx_font_typewriter">.</span><span id="lstnumberx125.8" class="ltx_text ltx_lst_identifier ltx_font_typewriter">concatenate</span><span id="lstnumberx125.9" class="ltx_text ltx_font_typewriter">((</span><span id="lstnumberx125.10" class="ltx_text ltx_lst_identifier ltx_font_typewriter">inds</span><span id="lstnumberx125.11" class="ltx_text ltx_font_typewriter">,</span><span id="lstnumberx125.12" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx125.13" class="ltx_text ltx_lst_identifier ltx_font_typewriter">new_inds</span><span id="lstnumberx125.14" class="ltx_text ltx_font_typewriter">))</span><span id="lstnumberx125.15" class="ltx_text ltx_lst_space ltx_font_typewriter">         </span><span id="lstnumberx125.16" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx125.16.1" class="ltx_text ltx_lst_space"> </span>Expand<span id="lstnumberx125.16.2" class="ltx_text ltx_lst_space"> </span>the<span id="lstnumberx125.16.3" class="ltx_text ltx_lst_space"> </span>array<span id="lstnumberx125.16.4" class="ltx_text ltx_lst_space"> </span>of<span id="lstnumberx125.16.5" class="ltx_text ltx_lst_space"> </span>indices<span id="lstnumberx125.16.6" class="ltx_text ltx_lst_space"> </span>sampled.</span>
</div>
<div id="lstnumberx126" class="ltx_listingline">
<span id="lstnumberx126.1" class="ltx_text ltx_lst_space ltx_font_typewriter">        </span><span id="lstnumberx126.2" class="ltx_text ltx_lst_keyword ltx_font_typewriter" style="color:#408080;">return</span><span id="lstnumberx126.3" class="ltx_text ltx_lst_space ltx_font_typewriter"> </span><span id="lstnumberx126.4" class="ltx_text ltx_lst_identifier ltx_font_typewriter">inds</span><span id="lstnumberx126.5" class="ltx_text ltx_lst_space ltx_font_typewriter">                                       </span><span id="lstnumberx126.6" class="ltx_text ltx_lst_comment ltx_font_typewriter" style="color:#408080;">#<span id="lstnumberx126.6.1" class="ltx_text ltx_lst_space"> </span>Gather<span id="lstnumberx126.6.2" class="ltx_text ltx_lst_space"> </span>and<span id="lstnumberx126.6.3" class="ltx_text ltx_lst_space"> </span>return<span id="lstnumberx126.6.4" class="ltx_text ltx_lst_space"> </span>subset<span id="lstnumberx126.6.5" class="ltx_text ltx_lst_space"> </span>indices.</span>
</div>
</div>
</figure>
<div id="A1.SS6.p5" class="ltx_para ltx_noindent">
<p id="A1.SS6.p5.1" class="ltx_p">Nevertheless, in Fig. <a href="#A1.F8" title="Figure 8 ‣ A.6 Contrastive loss ablations ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>, we show that JEST is indeed robust to the choice of contrastive loss. In the right panel, we see that the gains over the baseline softmax are comparable to the gains for the SigLIP loss. However, due to the degradation in the softmax baseline relative to the baseline SigLIP, the combination of JEST with SigLIP is preferred.</p>
</div>
</section>
<section id="A1.SS7" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.7 </span>Comparing approximation methods</h3>

<div id="A1.SS7.p1" class="ltx_para ltx_noindent">
<p id="A1.SS7.p1.7" class="ltx_p">We compared two canonical strategies for online model approximation. Both ablations are conducted at <math id="A1.SS7.p1.1.m1.1" class="ltx_Math" alttext="\sim" display="inline"><semantics id="A1.SS7.p1.1.m1.1a"><mo id="A1.SS7.p1.1.m1.1.1" xref="A1.SS7.p1.1.m1.1.1.cmml">∼</mo><annotation-xml encoding="MathML-Content" id="A1.SS7.p1.1.m1.1b"><csymbol cd="latexml" id="A1.SS7.p1.1.m1.1.1.cmml" xref="A1.SS7.p1.1.m1.1.1">similar-to</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A1.SS7.p1.1.m1.1c">\sim</annotation></semantics></math>75% FLOP reduction by either dropping 75% of patches or doubling the patch size (see Main Section <a href="#S3" title="3 Methods ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>). We vary the proportion of data used for approximate and full-resolution training (<math id="A1.SS7.p1.2.m2.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="A1.SS7.p1.2.m2.1a"><mi id="A1.SS7.p1.2.m2.1.1" xref="A1.SS7.p1.2.m2.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="A1.SS7.p1.2.m2.1b"><ci id="A1.SS7.p1.2.m2.1.1.cmml" xref="A1.SS7.p1.2.m2.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS7.p1.2.m2.1c">\lambda</annotation></semantics></math> and <math id="A1.SS7.p1.3.m3.1" class="ltx_Math" alttext="1-\lambda" display="inline"><semantics id="A1.SS7.p1.3.m3.1a"><mrow id="A1.SS7.p1.3.m3.1.1" xref="A1.SS7.p1.3.m3.1.1.cmml"><mn id="A1.SS7.p1.3.m3.1.1.2" xref="A1.SS7.p1.3.m3.1.1.2.cmml">1</mn><mo id="A1.SS7.p1.3.m3.1.1.1" xref="A1.SS7.p1.3.m3.1.1.1.cmml">−</mo><mi id="A1.SS7.p1.3.m3.1.1.3" xref="A1.SS7.p1.3.m3.1.1.3.cmml">λ</mi></mrow><annotation-xml encoding="MathML-Content" id="A1.SS7.p1.3.m3.1b"><apply id="A1.SS7.p1.3.m3.1.1.cmml" xref="A1.SS7.p1.3.m3.1.1"><minus id="A1.SS7.p1.3.m3.1.1.1.cmml" xref="A1.SS7.p1.3.m3.1.1.1"></minus><cn type="integer" id="A1.SS7.p1.3.m3.1.1.2.cmml" xref="A1.SS7.p1.3.m3.1.1.2">1</cn><ci id="A1.SS7.p1.3.m3.1.1.3.cmml" xref="A1.SS7.p1.3.m3.1.1.3">𝜆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS7.p1.3.m3.1c">1-\lambda</annotation></semantics></math>, respectively), keeping the total number of FLOPs used by the learner the same. Since the cost of one training iteration is proportional <math id="A1.SS7.p1.4.m4.1" class="ltx_Math" alttext="0.25\lambda+1-\lambda" display="inline"><semantics id="A1.SS7.p1.4.m4.1a"><mrow id="A1.SS7.p1.4.m4.1.1" xref="A1.SS7.p1.4.m4.1.1.cmml"><mrow id="A1.SS7.p1.4.m4.1.1.2" xref="A1.SS7.p1.4.m4.1.1.2.cmml"><mrow id="A1.SS7.p1.4.m4.1.1.2.2" xref="A1.SS7.p1.4.m4.1.1.2.2.cmml"><mn id="A1.SS7.p1.4.m4.1.1.2.2.2" xref="A1.SS7.p1.4.m4.1.1.2.2.2.cmml">0.25</mn><mo lspace="0em" rspace="0em" id="A1.SS7.p1.4.m4.1.1.2.2.1" xref="A1.SS7.p1.4.m4.1.1.2.2.1.cmml">​</mo><mi id="A1.SS7.p1.4.m4.1.1.2.2.3" xref="A1.SS7.p1.4.m4.1.1.2.2.3.cmml">λ</mi></mrow><mo id="A1.SS7.p1.4.m4.1.1.2.1" xref="A1.SS7.p1.4.m4.1.1.2.1.cmml">+</mo><mn id="A1.SS7.p1.4.m4.1.1.2.3" xref="A1.SS7.p1.4.m4.1.1.2.3.cmml">1</mn></mrow><mo id="A1.SS7.p1.4.m4.1.1.1" xref="A1.SS7.p1.4.m4.1.1.1.cmml">−</mo><mi id="A1.SS7.p1.4.m4.1.1.3" xref="A1.SS7.p1.4.m4.1.1.3.cmml">λ</mi></mrow><annotation-xml encoding="MathML-Content" id="A1.SS7.p1.4.m4.1b"><apply id="A1.SS7.p1.4.m4.1.1.cmml" xref="A1.SS7.p1.4.m4.1.1"><minus id="A1.SS7.p1.4.m4.1.1.1.cmml" xref="A1.SS7.p1.4.m4.1.1.1"></minus><apply id="A1.SS7.p1.4.m4.1.1.2.cmml" xref="A1.SS7.p1.4.m4.1.1.2"><plus id="A1.SS7.p1.4.m4.1.1.2.1.cmml" xref="A1.SS7.p1.4.m4.1.1.2.1"></plus><apply id="A1.SS7.p1.4.m4.1.1.2.2.cmml" xref="A1.SS7.p1.4.m4.1.1.2.2"><times id="A1.SS7.p1.4.m4.1.1.2.2.1.cmml" xref="A1.SS7.p1.4.m4.1.1.2.2.1"></times><cn type="float" id="A1.SS7.p1.4.m4.1.1.2.2.2.cmml" xref="A1.SS7.p1.4.m4.1.1.2.2.2">0.25</cn><ci id="A1.SS7.p1.4.m4.1.1.2.2.3.cmml" xref="A1.SS7.p1.4.m4.1.1.2.2.3">𝜆</ci></apply><cn type="integer" id="A1.SS7.p1.4.m4.1.1.2.3.cmml" xref="A1.SS7.p1.4.m4.1.1.2.3">1</cn></apply><ci id="A1.SS7.p1.4.m4.1.1.3.cmml" xref="A1.SS7.p1.4.m4.1.1.3">𝜆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS7.p1.4.m4.1c">0.25\lambda+1-\lambda</annotation></semantics></math>, we divide the number of training iterations by this factor to keep the training budget constant with respect to <math id="A1.SS7.p1.5.m5.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="A1.SS7.p1.5.m5.1a"><mi id="A1.SS7.p1.5.m5.1.1" xref="A1.SS7.p1.5.m5.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="A1.SS7.p1.5.m5.1b"><ci id="A1.SS7.p1.5.m5.1.1.cmml" xref="A1.SS7.p1.5.m5.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS7.p1.5.m5.1c">\lambda</annotation></semantics></math>. For <math id="A1.SS7.p1.6.m6.5" class="ltx_Math" alttext="\lambda\in[0.0,0.25,0.5,0.75,0.95]" display="inline"><semantics id="A1.SS7.p1.6.m6.5a"><mrow id="A1.SS7.p1.6.m6.5.6" xref="A1.SS7.p1.6.m6.5.6.cmml"><mi id="A1.SS7.p1.6.m6.5.6.2" xref="A1.SS7.p1.6.m6.5.6.2.cmml">λ</mi><mo id="A1.SS7.p1.6.m6.5.6.1" xref="A1.SS7.p1.6.m6.5.6.1.cmml">∈</mo><mrow id="A1.SS7.p1.6.m6.5.6.3.2" xref="A1.SS7.p1.6.m6.5.6.3.1.cmml"><mo stretchy="false" id="A1.SS7.p1.6.m6.5.6.3.2.1" xref="A1.SS7.p1.6.m6.5.6.3.1.cmml">[</mo><mn id="A1.SS7.p1.6.m6.1.1" xref="A1.SS7.p1.6.m6.1.1.cmml">0.0</mn><mo id="A1.SS7.p1.6.m6.5.6.3.2.2" xref="A1.SS7.p1.6.m6.5.6.3.1.cmml">,</mo><mn id="A1.SS7.p1.6.m6.2.2" xref="A1.SS7.p1.6.m6.2.2.cmml">0.25</mn><mo id="A1.SS7.p1.6.m6.5.6.3.2.3" xref="A1.SS7.p1.6.m6.5.6.3.1.cmml">,</mo><mn id="A1.SS7.p1.6.m6.3.3" xref="A1.SS7.p1.6.m6.3.3.cmml">0.5</mn><mo id="A1.SS7.p1.6.m6.5.6.3.2.4" xref="A1.SS7.p1.6.m6.5.6.3.1.cmml">,</mo><mn id="A1.SS7.p1.6.m6.4.4" xref="A1.SS7.p1.6.m6.4.4.cmml">0.75</mn><mo id="A1.SS7.p1.6.m6.5.6.3.2.5" xref="A1.SS7.p1.6.m6.5.6.3.1.cmml">,</mo><mn id="A1.SS7.p1.6.m6.5.5" xref="A1.SS7.p1.6.m6.5.5.cmml">0.95</mn><mo stretchy="false" id="A1.SS7.p1.6.m6.5.6.3.2.6" xref="A1.SS7.p1.6.m6.5.6.3.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.SS7.p1.6.m6.5b"><apply id="A1.SS7.p1.6.m6.5.6.cmml" xref="A1.SS7.p1.6.m6.5.6"><in id="A1.SS7.p1.6.m6.5.6.1.cmml" xref="A1.SS7.p1.6.m6.5.6.1"></in><ci id="A1.SS7.p1.6.m6.5.6.2.cmml" xref="A1.SS7.p1.6.m6.5.6.2">𝜆</ci><list id="A1.SS7.p1.6.m6.5.6.3.1.cmml" xref="A1.SS7.p1.6.m6.5.6.3.2"><cn type="float" id="A1.SS7.p1.6.m6.1.1.cmml" xref="A1.SS7.p1.6.m6.1.1">0.0</cn><cn type="float" id="A1.SS7.p1.6.m6.2.2.cmml" xref="A1.SS7.p1.6.m6.2.2">0.25</cn><cn type="float" id="A1.SS7.p1.6.m6.3.3.cmml" xref="A1.SS7.p1.6.m6.3.3">0.5</cn><cn type="float" id="A1.SS7.p1.6.m6.4.4.cmml" xref="A1.SS7.p1.6.m6.4.4">0.75</cn><cn type="float" id="A1.SS7.p1.6.m6.5.5.cmml" xref="A1.SS7.p1.6.m6.5.5">0.95</cn></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS7.p1.6.m6.5c">\lambda\in[0.0,0.25,0.5,0.75,0.95]</annotation></semantics></math>, this results in training budgets of <math id="A1.SS7.p1.7.m7.5" class="ltx_Math" alttext="[3,3.69,4.8,6.86,10.43]" display="inline"><semantics id="A1.SS7.p1.7.m7.5a"><mrow id="A1.SS7.p1.7.m7.5.6.2" xref="A1.SS7.p1.7.m7.5.6.1.cmml"><mo stretchy="false" id="A1.SS7.p1.7.m7.5.6.2.1" xref="A1.SS7.p1.7.m7.5.6.1.cmml">[</mo><mn id="A1.SS7.p1.7.m7.1.1" xref="A1.SS7.p1.7.m7.1.1.cmml">3</mn><mo id="A1.SS7.p1.7.m7.5.6.2.2" xref="A1.SS7.p1.7.m7.5.6.1.cmml">,</mo><mn id="A1.SS7.p1.7.m7.2.2" xref="A1.SS7.p1.7.m7.2.2.cmml">3.69</mn><mo id="A1.SS7.p1.7.m7.5.6.2.3" xref="A1.SS7.p1.7.m7.5.6.1.cmml">,</mo><mn id="A1.SS7.p1.7.m7.3.3" xref="A1.SS7.p1.7.m7.3.3.cmml">4.8</mn><mo id="A1.SS7.p1.7.m7.5.6.2.4" xref="A1.SS7.p1.7.m7.5.6.1.cmml">,</mo><mn id="A1.SS7.p1.7.m7.4.4" xref="A1.SS7.p1.7.m7.4.4.cmml">6.86</mn><mo id="A1.SS7.p1.7.m7.5.6.2.5" xref="A1.SS7.p1.7.m7.5.6.1.cmml">,</mo><mn id="A1.SS7.p1.7.m7.5.5" xref="A1.SS7.p1.7.m7.5.5.cmml">10.43</mn><mo stretchy="false" id="A1.SS7.p1.7.m7.5.6.2.6" xref="A1.SS7.p1.7.m7.5.6.1.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.SS7.p1.7.m7.5b"><list id="A1.SS7.p1.7.m7.5.6.1.cmml" xref="A1.SS7.p1.7.m7.5.6.2"><cn type="integer" id="A1.SS7.p1.7.m7.1.1.cmml" xref="A1.SS7.p1.7.m7.1.1">3</cn><cn type="float" id="A1.SS7.p1.7.m7.2.2.cmml" xref="A1.SS7.p1.7.m7.2.2">3.69</cn><cn type="float" id="A1.SS7.p1.7.m7.3.3.cmml" xref="A1.SS7.p1.7.m7.3.3">4.8</cn><cn type="float" id="A1.SS7.p1.7.m7.4.4.cmml" xref="A1.SS7.p1.7.m7.4.4">6.86</cn><cn type="float" id="A1.SS7.p1.7.m7.5.5.cmml" xref="A1.SS7.p1.7.m7.5.5">10.43</cn></list></annotation-xml><annotation encoding="application/x-tex" id="A1.SS7.p1.7.m7.5c">[3,3.69,4.8,6.86,10.43]</annotation></semantics></math> billion examples seen.</p>
</div>
<div id="A1.SS7.p2" class="ltx_para ltx_noindent">
<p id="A1.SS7.p2.1" class="ltx_p">Our results (Figure <a href="#A1.F9" title="Figure 9 ‣ A.7 Comparing approximation methods ‣ Appendix A Appendix ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>) demonstrate that downscaling data by decreasing the resolution effects a much more favourable trade-off than dropping a subset of patches. Both methods perform differently out-of-distribution, but only FlexiViT benefits significantly from co-training. We adopt this strategy throughout the paper.</p>
</div>
<figure id="A1.F9" class="ltx_figure"><img src="/html/2406.17711/assets/x6.png" id="A1.F9.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="392" height="205" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 9: </span><span id="A1.F9.4.1" class="ltx_text ltx_font_bold">Comparing model approximation strategies.</span> Comparing Flexi-JEST and PatchDrop-JEST at a 50% filtering proportion. On the x-axis, we vary the proportion of data used to co-train the scoring model. Green curve shows JEST runs, blue curves shows equivalent IID run without data curation. All runs conducted at isoFLOP, see Fig. <a href="#S4.F4" title="Figure 4 ‣ 4.3 Synergies between multi-resolution training and online batch selection ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>. <span id="A1.F9.5.2" class="ltx_text ltx_font_bold">Left: </span> FlexiViT scoring performs badly 0-shot out of distribution inference, but quickly recovers with co-training. <span id="A1.F9.6.3" class="ltx_text ltx_font_bold">Right: </span> Patch dropping is more robust to 0-shot inference, but doesn’t benefit as much from co-training.</figcaption>
</figure>
<figure id="A1.F10" class="ltx_figure"><img src="/html/2406.17711/assets/x7.png" id="A1.F10.g1" class="ltx_graphics ltx_centering ltx_img_square" width="230" height="210" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 10: </span><span id="A1.F10.5.1" class="ltx_text ltx_font_bold">Prioritizing data that is difficult for the learner</span> Compared to <span id="A1.F10.6.2" class="ltx_text ltx_font_italic">easy-reference</span> and <span id="A1.F10.7.3" class="ltx_text ltx_font_italic">learnability</span> scoring (Figure <a href="#S3.F2" title="Figure 2 ‣ 3.3 Efficient scoring and multi-resolution training ‣ 3 Methods ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>) prioritising data with high learner loss results in small gains at a filtering ratio of 50%, but quickly degrades as we filter larger amounts of data. Joint example selection exacerbates the effect at larger filtering ratios, aligning with the interpretation that <span id="A1.F10.8.4" class="ltx_text ltx_font_italic">hard-learner</span> prioritisation prioritises sampling noise in the data.
</figcaption>
</figure>
<figure id="A1.T4" class="ltx_table">
<table id="A1.T4.7" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T4.7.8.1" class="ltx_tr">
<th id="A1.T4.7.8.1.1" class="ltx_td ltx_th ltx_th_column"></th>
<th id="A1.T4.7.8.1.2" class="ltx_td ltx_th ltx_th_column"></th>
<th id="A1.T4.7.8.1.3" class="ltx_td ltx_th ltx_th_column"></th>
<th id="A1.T4.7.8.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column" colspan="2">FLOPs %</th>
<th id="A1.T4.7.8.1.5" class="ltx_td ltx_th ltx_th_column"></th>
<th id="A1.T4.7.8.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column" colspan="2">ImageNet-1k</th>
<th id="A1.T4.7.8.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column" colspan="2">COCO</th>
</tr>
<tr id="A1.T4.1.1" class="ltx_tr">
<th id="A1.T4.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column">Method</th>
<th id="A1.T4.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">Variant</th>
<th id="A1.T4.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column"># Train</th>
<th id="A1.T4.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">Per Iter.</th>
<th id="A1.T4.1.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_t"><span id="A1.T4.1.1.6.1" class="ltx_text ltx_font_bold">Total</span></th>
<th id="A1.T4.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column">
<span id="A1.T4.1.1.1.1" class="ltx_text ltx_font_bold">Mean</span> <math id="A1.T4.1.1.1.m1.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="A1.T4.1.1.1.m1.1a"><mi mathvariant="normal" id="A1.T4.1.1.1.m1.1.1" xref="A1.T4.1.1.1.m1.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="A1.T4.1.1.1.m1.1b"><ci id="A1.T4.1.1.1.m1.1.1.cmml" xref="A1.T4.1.1.1.m1.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.T4.1.1.1.m1.1c">\Delta</annotation></semantics></math>
</th>
<th id="A1.T4.1.1.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">10-S</th>
<th id="A1.T4.1.1.8" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">ZS</th>
<th id="A1.T4.1.1.9" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">I2T</th>
<th id="A1.T4.1.1.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">T2I</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T4.7.9.1" class="ltx_tr">
<td id="A1.T4.7.9.1.1" class="ltx_td ltx_align_left ltx_border_t">SigLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>
</td>
<td id="A1.T4.7.9.1.2" class="ltx_td ltx_align_center ltx_border_t">B</td>
<td id="A1.T4.7.9.1.3" class="ltx_td ltx_align_center ltx_border_t">40B</td>
<td id="A1.T4.7.9.1.4" class="ltx_td ltx_align_center ltx_border_t">100</td>
<td id="A1.T4.7.9.1.5" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">100</td>
<td id="A1.T4.7.9.1.6" class="ltx_td ltx_align_center ltx_border_t">    0.0</td>
<td id="A1.T4.7.9.1.7" class="ltx_td ltx_align_center ltx_border_t">70.3</td>
<td id="A1.T4.7.9.1.8" class="ltx_td ltx_align_center ltx_border_t">76.7</td>
<td id="A1.T4.7.9.1.9" class="ltx_td ltx_align_center ltx_border_t">65.2</td>
<td id="A1.T4.7.9.1.10" class="ltx_td ltx_align_center ltx_border_t">47.4</td>
</tr>
<tr id="A1.T4.7.10.2" class="ltx_tr" style="background-color:#E6E6FF;">
<td id="A1.T4.7.10.2.1" class="ltx_td ltx_align_left"><span id="A1.T4.7.10.2.1.1" class="ltx_text" style="background-color:#E6E6FF;">JEST</span></td>
<td id="A1.T4.7.10.2.2" class="ltx_td ltx_align_center"><span id="A1.T4.7.10.2.2.1" class="ltx_text" style="background-color:#E6E6FF;">B</span></td>
<td id="A1.T4.7.10.2.3" class="ltx_td ltx_align_center"><span id="A1.T4.7.10.2.3.1" class="ltx_text" style="background-color:#E6E6FF;">  5B</span></td>
<td id="A1.T4.7.10.2.4" class="ltx_td ltx_align_center"><span id="A1.T4.7.10.2.4.1" class="ltx_text" style="background-color:#E6E6FF;">233</span></td>
<td id="A1.T4.7.10.2.5" class="ltx_td ltx_align_center ltx_border_r"><span id="A1.T4.7.10.2.5.1" class="ltx_text" style="background-color:#E6E6FF;">29</span></td>
<td id="A1.T4.7.10.2.6" class="ltx_td ltx_align_center"><span id="A1.T4.7.10.2.6.1" class="ltx_text" style="background-color:#E6E6FF;"> – 0.5</span></td>
<td id="A1.T4.7.10.2.7" class="ltx_td ltx_align_center"><span id="A1.T4.7.10.2.7.1" class="ltx_text" style="background-color:#E6E6FF;">68.2</span></td>
<td id="A1.T4.7.10.2.8" class="ltx_td ltx_align_center"><span id="A1.T4.7.10.2.8.1" class="ltx_text" style="background-color:#E6E6FF;">75.5</span></td>
<td id="A1.T4.7.10.2.9" class="ltx_td ltx_align_center"><span id="A1.T4.7.10.2.9.1" class="ltx_text" style="background-color:#E6E6FF;">66.1</span></td>
<td id="A1.T4.7.10.2.10" class="ltx_td ltx_align_center"><span id="A1.T4.7.10.2.10.1" class="ltx_text" style="background-color:#E6E6FF;">47.9</span></td>
</tr>
<tr id="A1.T4.2.2" class="ltx_tr" style="background-color:#E6E6FF;">
<td id="A1.T4.2.2.2" class="ltx_td ltx_align_left"><span id="A1.T4.2.2.2.1" class="ltx_text" style="background-color:#E6E6FF;">Flexi-JEST</span></td>
<td id="A1.T4.2.2.3" class="ltx_td ltx_align_center"><span id="A1.T4.2.2.3.1" class="ltx_text" style="background-color:#E6E6FF;">B</span></td>
<td id="A1.T4.2.2.4" class="ltx_td ltx_align_center"><span id="A1.T4.2.2.4.1" class="ltx_text" style="background-color:#E6E6FF;">13B</span></td>
<td id="A1.T4.2.2.5" class="ltx_td ltx_align_center"><span id="A1.T4.2.2.5.1" class="ltx_text" style="background-color:#E6E6FF;">110</span></td>
<td id="A1.T4.2.2.6" class="ltx_td ltx_align_center ltx_border_r"><span id="A1.T4.2.2.6.1" class="ltx_text" style="background-color:#E6E6FF;">36</span></td>
<td id="A1.T4.2.2.1" class="ltx_td ltx_align_center"><span id="A1.T4.2.2.1.1" class="ltx_text" style="background-color:#E6E6FF;"> <math id="A1.T4.2.2.1.1.m1.1" class="ltx_Math" alttext="+" display="inline"><semantics id="A1.T4.2.2.1.1.m1.1a"><mo mathbackground="#E6E6FF" id="A1.T4.2.2.1.1.m1.1.1" xref="A1.T4.2.2.1.1.m1.1.1.cmml">+</mo><annotation-xml encoding="MathML-Content" id="A1.T4.2.2.1.1.m1.1b"><plus id="A1.T4.2.2.1.1.m1.1.1.cmml" xref="A1.T4.2.2.1.1.m1.1.1"></plus></annotation-xml><annotation encoding="application/x-tex" id="A1.T4.2.2.1.1.m1.1c">+</annotation></semantics></math> 1.2</span></td>
<td id="A1.T4.2.2.7" class="ltx_td ltx_align_center"><span id="A1.T4.2.2.7.1" class="ltx_text" style="background-color:#E6E6FF;">69.4</span></td>
<td id="A1.T4.2.2.8" class="ltx_td ltx_align_center"><span id="A1.T4.2.2.8.1" class="ltx_text" style="background-color:#E6E6FF;">76.6</span></td>
<td id="A1.T4.2.2.9" class="ltx_td ltx_align_center"><span id="A1.T4.2.2.9.1" class="ltx_text" style="background-color:#E6E6FF;">68.2</span></td>
<td id="A1.T4.2.2.10" class="ltx_td ltx_align_center"><span id="A1.T4.2.2.10.1" class="ltx_text" style="background-color:#E6E6FF;">50.2</span></td>
</tr>
<tr id="A1.T4.3.3" class="ltx_tr" style="background-color:#E6E6FF;">
<td id="A1.T4.3.3.2" class="ltx_td ltx_align_left"><span id="A1.T4.3.3.2.1" class="ltx_text" style="background-color:#E6E6FF;">JEST++</span></td>
<td id="A1.T4.3.3.3" class="ltx_td ltx_align_center"><span id="A1.T4.3.3.3.1" class="ltx_text" style="background-color:#E6E6FF;">B</span></td>
<td id="A1.T4.3.3.4" class="ltx_td ltx_align_center"><span id="A1.T4.3.3.4.1" class="ltx_text" style="background-color:#E6E6FF;">  4B</span></td>
<td id="A1.T4.3.3.5" class="ltx_td ltx_align_center"><span id="A1.T4.3.3.5.1" class="ltx_text" style="background-color:#E6E6FF;">233</span></td>
<td id="A1.T4.3.3.6" class="ltx_td ltx_align_center ltx_border_r"><span id="A1.T4.3.3.6.1" class="ltx_text" style="background-color:#E6E6FF;">  23</span></td>
<td id="A1.T4.3.3.1" class="ltx_td ltx_align_center"><span id="A1.T4.3.3.1.1" class="ltx_text" style="background-color:#E6E6FF;"> <math id="A1.T4.3.3.1.1.m1.1" class="ltx_Math" alttext="+" display="inline"><semantics id="A1.T4.3.3.1.1.m1.1a"><mo mathbackground="#E6E6FF" id="A1.T4.3.3.1.1.m1.1.1" xref="A1.T4.3.3.1.1.m1.1.1.cmml">+</mo><annotation-xml encoding="MathML-Content" id="A1.T4.3.3.1.1.m1.1b"><plus id="A1.T4.3.3.1.1.m1.1.1.cmml" xref="A1.T4.3.3.1.1.m1.1.1"></plus></annotation-xml><annotation encoding="application/x-tex" id="A1.T4.3.3.1.1.m1.1c">+</annotation></semantics></math> 2.8</span></td>
<td id="A1.T4.3.3.7" class="ltx_td ltx_align_center"><span id="A1.T4.3.3.7.1" class="ltx_text" style="background-color:#E6E6FF;">70.3</span></td>
<td id="A1.T4.3.3.8" class="ltx_td ltx_align_center"><span id="A1.T4.3.3.8.1" class="ltx_text" style="background-color:#E6E6FF;">76.9</span></td>
<td id="A1.T4.3.3.9" class="ltx_td ltx_align_center"><span id="A1.T4.3.3.9.1" class="ltx_text" style="background-color:#E6E6FF;">70.3</span></td>
<td id="A1.T4.3.3.10" class="ltx_td ltx_align_center"><span id="A1.T4.3.3.10.1" class="ltx_text" style="background-color:#E6E6FF;">53.3</span></td>
</tr>
<tr id="A1.T4.4.4" class="ltx_tr" style="background-color:#E6E6FF;">
<td id="A1.T4.4.4.2" class="ltx_td ltx_align_left"><span id="A1.T4.4.4.2.1" class="ltx_text" style="background-color:#E6E6FF;">Flexi-JEST++</span></td>
<td id="A1.T4.4.4.3" class="ltx_td ltx_align_center"><span id="A1.T4.4.4.3.1" class="ltx_text" style="background-color:#E6E6FF;">B</span></td>
<td id="A1.T4.4.4.4" class="ltx_td ltx_align_center"><span id="A1.T4.4.4.4.1" class="ltx_text" style="background-color:#E6E6FF;">  4B</span></td>
<td id="A1.T4.4.4.5" class="ltx_td ltx_align_center"><span id="A1.T4.4.4.5.1" class="ltx_text" style="background-color:#E6E6FF;">110</span></td>
<td id="A1.T4.4.4.6" class="ltx_td ltx_align_center ltx_border_r"><span id="A1.T4.4.4.6.1" class="ltx_text" style="background-color:#E6E6FF;">  <span id="A1.T4.4.4.6.1.1" class="ltx_text ltx_font_bold">11</span></span></td>
<td id="A1.T4.4.4.1" class="ltx_td ltx_align_center"><span id="A1.T4.4.4.1.1" class="ltx_text" style="background-color:#E6E6FF;"> <math id="A1.T4.4.4.1.1.m1.1" class="ltx_Math" alttext="+" display="inline"><semantics id="A1.T4.4.4.1.1.m1.1a"><mo mathbackground="#E6E6FF" id="A1.T4.4.4.1.1.m1.1.1" xref="A1.T4.4.4.1.1.m1.1.1.cmml">+</mo><annotation-xml encoding="MathML-Content" id="A1.T4.4.4.1.1.m1.1b"><plus id="A1.T4.4.4.1.1.m1.1.1.cmml" xref="A1.T4.4.4.1.1.m1.1.1"></plus></annotation-xml><annotation encoding="application/x-tex" id="A1.T4.4.4.1.1.m1.1c">+</annotation></semantics></math> 0.9</span></td>
<td id="A1.T4.4.4.7" class="ltx_td ltx_align_center"><span id="A1.T4.4.4.7.1" class="ltx_text" style="background-color:#E6E6FF;">68.2</span></td>
<td id="A1.T4.4.4.8" class="ltx_td ltx_align_center"><span id="A1.T4.4.4.8.1" class="ltx_text" style="background-color:#E6E6FF;">75.8</span></td>
<td id="A1.T4.4.4.9" class="ltx_td ltx_align_center"><span id="A1.T4.4.4.9.1" class="ltx_text" style="background-color:#E6E6FF;">68.0</span></td>
<td id="A1.T4.4.4.10" class="ltx_td ltx_align_center"><span id="A1.T4.4.4.10.1" class="ltx_text" style="background-color:#E6E6FF;">51.2</span></td>
</tr>
<tr id="A1.T4.5.5" class="ltx_tr" style="background-color:#E6E6FF;">
<td id="A1.T4.5.5.2" class="ltx_td ltx_align_left"><span id="A1.T4.5.5.2.1" class="ltx_text" style="background-color:#E6E6FF;">JEST++</span></td>
<td id="A1.T4.5.5.3" class="ltx_td ltx_align_center"><span id="A1.T4.5.5.3.1" class="ltx_text" style="background-color:#E6E6FF;">B</span></td>
<td id="A1.T4.5.5.4" class="ltx_td ltx_align_center"><span id="A1.T4.5.5.4.1" class="ltx_text" style="background-color:#E6E6FF;">10B</span></td>
<td id="A1.T4.5.5.5" class="ltx_td ltx_align_center"><span id="A1.T4.5.5.5.1" class="ltx_text" style="background-color:#E6E6FF;">233</span></td>
<td id="A1.T4.5.5.6" class="ltx_td ltx_align_center ltx_border_r"><span id="A1.T4.5.5.6.1" class="ltx_text" style="background-color:#E6E6FF;">58</span></td>
<td id="A1.T4.5.5.1" class="ltx_td ltx_align_center"><span id="A1.T4.5.5.1.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;"> <math id="A1.T4.5.5.1.1.m1.1" class="ltx_Math" alttext="+" display="inline"><semantics id="A1.T4.5.5.1.1.m1.1a"><mo mathbackground="#E6E6FF" id="A1.T4.5.5.1.1.m1.1.1" xref="A1.T4.5.5.1.1.m1.1.1.cmml">+</mo><annotation-xml encoding="MathML-Content" id="A1.T4.5.5.1.1.m1.1b"><plus id="A1.T4.5.5.1.1.m1.1.1.cmml" xref="A1.T4.5.5.1.1.m1.1.1"></plus></annotation-xml><annotation encoding="application/x-tex" id="A1.T4.5.5.1.1.m1.1c">+</annotation></semantics></math> 4.0</span></td>
<td id="A1.T4.5.5.7" class="ltx_td ltx_align_center"><span id="A1.T4.5.5.7.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;">72.3</span></td>
<td id="A1.T4.5.5.8" class="ltx_td ltx_align_center"><span id="A1.T4.5.5.8.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;">77.6</span></td>
<td id="A1.T4.5.5.9" class="ltx_td ltx_align_center"><span id="A1.T4.5.5.9.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;">71.8</span></td>
<td id="A1.T4.5.5.10" class="ltx_td ltx_align_center"><span id="A1.T4.5.5.10.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;">53.9</span></td>
</tr>
<tr id="A1.T4.6.6" class="ltx_tr" style="background-color:#E6E6FF;">
<td id="A1.T4.6.6.2" class="ltx_td ltx_align_left"><span id="A1.T4.6.6.2.1" class="ltx_text" style="background-color:#E6E6FF;">Flexi-JEST++</span></td>
<td id="A1.T4.6.6.3" class="ltx_td ltx_align_center"><span id="A1.T4.6.6.3.1" class="ltx_text" style="background-color:#E6E6FF;">B</span></td>
<td id="A1.T4.6.6.4" class="ltx_td ltx_align_center"><span id="A1.T4.6.6.4.1" class="ltx_text" style="background-color:#E6E6FF;">10B</span></td>
<td id="A1.T4.6.6.5" class="ltx_td ltx_align_center"><span id="A1.T4.6.6.5.1" class="ltx_text" style="background-color:#E6E6FF;">110</span></td>
<td id="A1.T4.6.6.6" class="ltx_td ltx_align_center ltx_border_r"><span id="A1.T4.6.6.6.1" class="ltx_text" style="background-color:#E6E6FF;">28</span></td>
<td id="A1.T4.6.6.1" class="ltx_td ltx_align_center"><span id="A1.T4.6.6.1.1" class="ltx_text" style="background-color:#E6E6FF;"> <math id="A1.T4.6.6.1.1.m1.1" class="ltx_Math" alttext="+" display="inline"><semantics id="A1.T4.6.6.1.1.m1.1a"><mo mathbackground="#E6E6FF" id="A1.T4.6.6.1.1.m1.1.1" xref="A1.T4.6.6.1.1.m1.1.1.cmml">+</mo><annotation-xml encoding="MathML-Content" id="A1.T4.6.6.1.1.m1.1b"><plus id="A1.T4.6.6.1.1.m1.1.1.cmml" xref="A1.T4.6.6.1.1.m1.1.1"></plus></annotation-xml><annotation encoding="application/x-tex" id="A1.T4.6.6.1.1.m1.1c">+</annotation></semantics></math> 3.1</span></td>
<td id="A1.T4.6.6.7" class="ltx_td ltx_align_center"><span id="A1.T4.6.6.7.1" class="ltx_text" style="background-color:#E6E6FF;">71.1</span></td>
<td id="A1.T4.6.6.8" class="ltx_td ltx_align_center"><span id="A1.T4.6.6.8.1" class="ltx_text" style="background-color:#E6E6FF;">77.2</span></td>
<td id="A1.T4.6.6.9" class="ltx_td ltx_align_center"><span id="A1.T4.6.6.9.1" class="ltx_text" style="background-color:#E6E6FF;">70.2</span></td>
<td id="A1.T4.6.6.10" class="ltx_td ltx_align_center"><span id="A1.T4.6.6.10.1" class="ltx_text" style="background-color:#E6E6FF;">53.3</span></td>
</tr>
<tr id="A1.T4.7.11.3" class="ltx_tr">
<td id="A1.T4.7.11.3.1" class="ltx_td ltx_align_left">SigLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>
</td>
<td id="A1.T4.7.11.3.2" class="ltx_td ltx_align_center">L</td>
<td id="A1.T4.7.11.3.3" class="ltx_td ltx_align_center">40B</td>
<td id="A1.T4.7.11.3.4" class="ltx_td ltx_align_center">100</td>
<td id="A1.T4.7.11.3.5" class="ltx_td ltx_align_center ltx_border_r">100</td>
<td id="A1.T4.7.11.3.6" class="ltx_td ltx_align_center">    0.0</td>
<td id="A1.T4.7.11.3.7" class="ltx_td ltx_align_center">77.1</td>
<td id="A1.T4.7.11.3.8" class="ltx_td ltx_align_center">80.5</td>
<td id="A1.T4.7.11.3.9" class="ltx_td ltx_align_center">69.5</td>
<td id="A1.T4.7.11.3.10" class="ltx_td ltx_align_center">51.2</td>
</tr>
<tr id="A1.T4.7.7" class="ltx_tr" style="background-color:#E6E6FF;">
<td id="A1.T4.7.7.2" class="ltx_td ltx_align_left"><span id="A1.T4.7.7.2.1" class="ltx_text" style="background-color:#E6E6FF;">JEST++</span></td>
<td id="A1.T4.7.7.3" class="ltx_td ltx_align_center"><span id="A1.T4.7.7.3.1" class="ltx_text" style="background-color:#E6E6FF;">L</span></td>
<td id="A1.T4.7.7.4" class="ltx_td ltx_align_center"><span id="A1.T4.7.7.4.1" class="ltx_text" style="background-color:#E6E6FF;">  4B</span></td>
<td id="A1.T4.7.7.5" class="ltx_td ltx_align_center"><span id="A1.T4.7.7.5.1" class="ltx_text" style="background-color:#E6E6FF;">233</span></td>
<td id="A1.T4.7.7.6" class="ltx_td ltx_align_center ltx_border_r"><span id="A1.T4.7.7.6.1" class="ltx_text" style="background-color:#E6E6FF;">  <span id="A1.T4.7.7.6.1.1" class="ltx_text ltx_font_bold">23</span></span></td>
<td id="A1.T4.7.7.1" class="ltx_td ltx_align_center"><span id="A1.T4.7.7.1.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;"> <math id="A1.T4.7.7.1.1.m1.1" class="ltx_Math" alttext="+" display="inline"><semantics id="A1.T4.7.7.1.1.m1.1a"><mo mathbackground="#E6E6FF" id="A1.T4.7.7.1.1.m1.1.1" xref="A1.T4.7.7.1.1.m1.1.1.cmml">+</mo><annotation-xml encoding="MathML-Content" id="A1.T4.7.7.1.1.m1.1b"><plus id="A1.T4.7.7.1.1.m1.1.1.cmml" xref="A1.T4.7.7.1.1.m1.1.1"></plus></annotation-xml><annotation encoding="application/x-tex" id="A1.T4.7.7.1.1.m1.1c">+</annotation></semantics></math> 1.8</span></td>
<td id="A1.T4.7.7.7" class="ltx_td ltx_align_center"><span id="A1.T4.7.7.7.1" class="ltx_text" style="background-color:#E6E6FF;">75.5</span></td>
<td id="A1.T4.7.7.8" class="ltx_td ltx_align_center"><span id="A1.T4.7.7.8.1" class="ltx_text" style="background-color:#E6E6FF;">80.4</span></td>
<td id="A1.T4.7.7.9" class="ltx_td ltx_align_center"><span id="A1.T4.7.7.9.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;">71.1</span></td>
<td id="A1.T4.7.7.10" class="ltx_td ltx_align_center"><span id="A1.T4.7.7.10.1" class="ltx_text ltx_font_bold" style="background-color:#E6E6FF;">54.8</span></td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>
<span id="A1.T4.9.1" class="ltx_text ltx_font_bold">JEST continues to improve with longer training runs.</span> FLOP % is measured relative to SigLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>. Mean denotes the average performance over all metrics. “Per Iter.” denotes FLOPs per iteration. 10B training runs of both JEST++ and FlexiJEST++ continue to improve over the 4B results presented in main Table <a href="#S4.T1" title="Table 1 ‣ 4.4 Joint examples selection enables strong data-quality bootstrapping ‣ 4 Experiments ‣ Data curation via joint example selection further accelerates multimodal learning" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>. JEST and Flexi-JEST, which use the WebLi-curated reference dataset both perform strongly on a per-FLOP basis, with Flexi-JEST also outperforming the SigLIP 40B baseilne on average.
</figcaption>
</figure>
<figure id="A1.T5" class="ltx_table">
<div id="A1.T5.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:75.4pt;vertical-align:-0.8pt;"><span class="ltx_transformed_inner" style="transform:translate(-44.8pt,7.7pt) scale(0.828835886515039,0.828835886515039) ;">
<table id="A1.T5.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T5.1.1.1.1" class="ltx_tr">
<th id="A1.T5.1.1.1.1.1" class="ltx_td ltx_th ltx_th_column ltx_th_row"></th>
<th id="A1.T5.1.1.1.1.2" class="ltx_td ltx_th ltx_th_column ltx_th_row ltx_border_r"></th>
<th id="A1.T5.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column" colspan="2">ImageNet-1k</th>
<th id="A1.T5.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column" colspan="2">COCO</th>
<th id="A1.T5.1.1.1.1.5" class="ltx_td ltx_th ltx_th_column"></th>
<th id="A1.T5.1.1.1.1.6" class="ltx_td ltx_th ltx_th_column"></th>
<th id="A1.T5.1.1.1.1.7" class="ltx_td ltx_th ltx_th_column"></th>
<th id="A1.T5.1.1.1.1.8" class="ltx_td ltx_th ltx_th_column ltx_border_r"></th>
<th id="A1.T5.1.1.1.1.9" class="ltx_td ltx_th ltx_th_column"></th>
</tr>
<tr id="A1.T5.1.1.2.2" class="ltx_tr">
<th id="A1.T5.1.1.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row">Method</th>
<th id="A1.T5.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r"># Train</th>
<th id="A1.T5.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">10-S</th>
<th id="A1.T5.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">ZS</th>
<th id="A1.T5.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">I2T</th>
<th id="A1.T5.1.1.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">T2I</th>
<th id="A1.T5.1.1.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column">Birds</th>
<th id="A1.T5.1.1.2.2.8" class="ltx_td ltx_align_center ltx_th ltx_th_column">Caltech</th>
<th id="A1.T5.1.1.2.2.9" class="ltx_td ltx_align_center ltx_th ltx_th_column">Cars</th>
<th id="A1.T5.1.1.2.2.10" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">Pets</th>
<th id="A1.T5.1.1.2.2.11" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="A1.T5.1.1.2.2.11.1" class="ltx_text ltx_font_bold">Mean</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T5.1.1.3.1" class="ltx_tr">
<th id="A1.T5.1.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t"><span id="A1.T5.1.1.3.1.1.1" class="ltx_text">SigLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite></span></th>
<th id="A1.T5.1.1.3.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">40B</th>
<td id="A1.T5.1.1.3.1.3" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T5.1.1.3.1.3.1" class="ltx_text ltx_font_bold" style="background-color:#FFFFFF;">70.3</span></td>
<td id="A1.T5.1.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T5.1.1.3.1.4.1" class="ltx_text" style="background-color:#FFFFFF;">76.7</span></td>
<td id="A1.T5.1.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T5.1.1.3.1.5.1" class="ltx_text" style="background-color:#FFFFFF;">65.2</span></td>
<td id="A1.T5.1.1.3.1.6" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T5.1.1.3.1.6.1" class="ltx_text" style="background-color:#FFFFFF;">47.4</span></td>
<td id="A1.T5.1.1.3.1.7" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T5.1.1.3.1.7.1" class="ltx_text" style="background-color:#FFFFFF;">75.5</span></td>
<td id="A1.T5.1.1.3.1.8" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T5.1.1.3.1.8.1" class="ltx_text" style="background-color:#FFFFFF;">92.8</span></td>
<td id="A1.T5.1.1.3.1.9" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T5.1.1.3.1.9.1" class="ltx_text" style="background-color:#FFFFFF;">92.0</span></td>
<td id="A1.T5.1.1.3.1.10" class="ltx_td ltx_align_center ltx_border_r ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T5.1.1.3.1.10.1" class="ltx_text" style="background-color:#FFFFFF;">91.2</span></td>
<td id="A1.T5.1.1.3.1.11" class="ltx_td ltx_align_center ltx_border_t" style="background-color:#FFFFFF;"><span id="A1.T5.1.1.3.1.11.1" class="ltx_text" style="background-color:#FFFFFF;">76.4</span></td>
</tr>
<tr id="A1.T5.1.1.4.2" class="ltx_tr">
<th id="A1.T5.1.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A1.T5.1.1.4.2.1.1" class="ltx_text">WebLI Curated++ Ref.</span></th>
<th id="A1.T5.1.1.4.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">  5B</th>
<td id="A1.T5.1.1.4.2.3" class="ltx_td ltx_align_center" style="background-color:#E64040;"><span id="A1.T5.1.1.4.2.3.1" class="ltx_text" style="background-color:#E64040;">60.6</span></td>
<td id="A1.T5.1.1.4.2.4" class="ltx_td ltx_align_center" style="background-color:#E64040;"><span id="A1.T5.1.1.4.2.4.1" class="ltx_text" style="background-color:#E64040;">67.1</span></td>
<td id="A1.T5.1.1.4.2.5" class="ltx_td ltx_align_center" style="background-color:#40B340;"><span id="A1.T5.1.1.4.2.5.1" class="ltx_text ltx_font_bold" style="background-color:#40B340;">74.2</span></td>
<td id="A1.T5.1.1.4.2.6" class="ltx_td ltx_align_center" style="background-color:#40B340;"><span id="A1.T5.1.1.4.2.6.1" class="ltx_text ltx_font_bold" style="background-color:#40B340;">56.8</span></td>
<td id="A1.T5.1.1.4.2.7" class="ltx_td ltx_align_center" style="background-color:#E64040;"><span id="A1.T5.1.1.4.2.7.1" class="ltx_text" style="background-color:#E64040;">65.4</span></td>
<td id="A1.T5.1.1.4.2.8" class="ltx_td ltx_align_center" style="background-color:#F3A6A6;"><span id="A1.T5.1.1.4.2.8.1" class="ltx_text" style="background-color:#F3A6A6;">89.9</span></td>
<td id="A1.T5.1.1.4.2.9" class="ltx_td ltx_align_center" style="background-color:#E64040;"><span id="A1.T5.1.1.4.2.9.1" class="ltx_text" style="background-color:#E64040;">64.1</span></td>
<td id="A1.T5.1.1.4.2.10" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#E64040;"><span id="A1.T5.1.1.4.2.10.1" class="ltx_text" style="background-color:#E64040;">83.0</span></td>
<td id="A1.T5.1.1.4.2.11" class="ltx_td ltx_align_center" style="background-color:#E64141;"><span id="A1.T5.1.1.4.2.11.1" class="ltx_text" style="background-color:#E64141;">70.2</span></td>
</tr>
<tr id="A1.T5.1.1.5.3" class="ltx_tr">
<th id="A1.T5.1.1.5.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A1.T5.1.1.5.3.1.1" class="ltx_text">JEST++</span></th>
<th id="A1.T5.1.1.5.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">  4B</th>
<td id="A1.T5.1.1.5.3.3" class="ltx_td ltx_align_center" style="background-color:#FFFFFF;"><span id="A1.T5.1.1.5.3.3.1" class="ltx_text ltx_font_bold" style="background-color:#FFFFFF;">70.3</span></td>
<td id="A1.T5.1.1.5.3.4" class="ltx_td ltx_align_center" style="background-color:#F9FDF9;"><span id="A1.T5.1.1.5.3.4.1" class="ltx_text ltx_font_bold" style="background-color:#F9FDF9;">76.9</span></td>
<td id="A1.T5.1.1.5.3.5" class="ltx_td ltx_align_center" style="background-color:#63C163;"><span id="A1.T5.1.1.5.3.5.1" class="ltx_text" style="background-color:#63C163;">70.3</span></td>
<td id="A1.T5.1.1.5.3.6" class="ltx_td ltx_align_center" style="background-color:#4AB74A;"><span id="A1.T5.1.1.5.3.6.1" class="ltx_text" style="background-color:#4AB74A;">53.3</span></td>
<td id="A1.T5.1.1.5.3.7" class="ltx_td ltx_align_center" style="background-color:#69C369;"><span id="A1.T5.1.1.5.3.7.1" class="ltx_text ltx_font_bold" style="background-color:#69C369;">80.4</span></td>
<td id="A1.T5.1.1.5.3.8" class="ltx_td ltx_align_center" style="background-color:#FADADA;"><span id="A1.T5.1.1.5.3.8.1" class="ltx_text" style="background-color:#FADADA;">91.6</span></td>
<td id="A1.T5.1.1.5.3.9" class="ltx_td ltx_align_center" style="background-color:#F5B3B3;"><span id="A1.T5.1.1.5.3.9.1" class="ltx_text" style="background-color:#F5B3B3;">89.5</span></td>
<td id="A1.T5.1.1.5.3.10" class="ltx_td ltx_align_center ltx_border_r" style="background-color:#C5E8C5;"><span id="A1.T5.1.1.5.3.10.1" class="ltx_text ltx_font_bold" style="background-color:#C5E8C5;">93.1</span></td>
<td id="A1.T5.1.1.5.3.11" class="ltx_td ltx_align_center" style="background-color:#C8E9C8;"><span id="A1.T5.1.1.5.3.11.1" class="ltx_text ltx_font_bold" style="background-color:#C8E9C8;">78.2</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span><span id="A1.T5.3.1" class="ltx_text ltx_font_bold">JEST++ efficiently leverages specialist reference models to train generalist foundation models.</span> Colour gradients are measured relative to SigLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>]</cite>. Mean denotes the average performance over all metrics. WebLI-curated++ reference performance is highly specialized- improving (over the SigLIP baseline) on only 2 out of the 8 benchmarks and severely under-performing on the rest. On the other hand JEST++ sees far more general improvements across benchmarks (8% better than the reference model on average).
</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2406.17710" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2406.17711" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2406.17711">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2406.17711" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2406.17712" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Fri Jul  5 21:40:30 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2406.17711
Data curation is an essential component of large-scale pretraining.
In this work, we demonstrate that jointly
selectingbatchesof data is more effective for learning than selecting examples independently. Multimodal contrastive objectives expose the dependencies between data and thus naturally yield criteria for measuring thejoint learnabilityof a batch. We derive a simple and tractable algorithm for selecting such batches, which significantly accelerate training beyond individually-prioritized data points.
As performance improves by selecting from larger super-batches, we also leverage recent advances in model approximation to reduce the associated computational overhead. As a result, our approach—multimodal contrastive learning with joint example selection (JEST) —surpasses state-of-the-art models with up to 13fewer iterations and 10less computation. Essential to the performance of JEST is the ability to steer the data selection process towards the distribution of smaller, well-curated datasets via pretrained reference models, exposing the level of data curation as a new dimension for neural scaling laws.
abstract
Data curation via joint example selectionfurther accelerates multimodal learning
Abstract
2406.17711
1IntroductionData quality is an essential driver of performance for large-scale pretraining.
Whether in language, vision, or multimodal modeling, training on well-curated datasets has
consistently demonstrated that strong performance can be achieved with significantly less data.
However, current data pipelines rely heavily on manual curation, which is difficult and expensive to scale. In contrast, model-based data curation, which uses features of the model being trained to select high quality data, holds promise for improving the slow, power-law scaling of large-scale pretraining across modalities, both in theoryand in practice.Existing methods apply curation at the level of individual data points.
Yet the quality of a batch is also a function of its composition, in addition to the summed quality of its data points considered independently.
In computer vision, hard negatives (i.e. clusters of points which lie close to one another but contain different labels)  have been found to provide a more effective learning signal than trivially solvable ones.
In this work we seek to generalize this notion by asking whether model-based data-selection criteria applied to batches of data can accelerate learning beyond what is possible by selecting examples independently.In multimodal learning, contrastive objectives directly expose the interactions between examples in a batch. We therefore derive a simple and tractable algorithm for joint example selection—JEST—which efficiently selects relevant ‘sub-batches’ of data from much larger ‘super-batches’ given their model-based scores.
When scoring batches with a pretrained reference model (i.e.easy-reference) , JEST accelerates learning relative to uniform batch selection, significantly improving on independent example selection using the same reference (as in CLIPScore) .
When scoring batches according to theirlearnability, which also takes into account the online model loss, JEST improves further, matching the performance of state-of-the-art modelswith up to 13fewer training iterations.Discovering highly learnable batches requires sifting through much larger super-batches of raw data.
We make learnability scoring of large batches tractable by leveraging recent advances in online model approximation,
which reduce computation while still providing useful predictions. By training a single model at multiple resolutions in parallel, we efficiently apply the model for scoring large super-batches, find their most learnable sub-batch, and spend more valuable computation for learning on them. Thanks to savings in both learning and example scoring, we reduce the overhead of scoring from 133% to 10% additional FLOPs while maintaining significant gains in training efficiency. This approximate scoring framework—Flexi-JEST—produces state-of-the-art models with 11fewer iterationsand10fewer FLOPs.Finally, we find that central to the performance of our framework is the ability to steer the curation process towards the distribution of smaller, well-curated datasets. This occurs naturally with the model-based selection criteria we consider through the concept of a pretrained reference model, which prioritizes examples
that most resemble the data it was trained on. Crucially, we find this process to enable strongdata quality bootstrapping: a reference model trained on a small curated dataset can effectively guide the curation of a much larger dataset, allowing the training of a model which strongly surpasses the quality of the reference model on many downstream tasks.Figure 1:Joint Example Selection accelerates multimodal pretraining.Our JEST/JEST++ methods bootstrap from small, strongly curated datasets (Webli-curated/Webli-curated++)  to actively curate web-scale datasets. Flexi-JEST++ uses variable patch sizing to reduce the cost of curation.Left: Training with JEST matches the performance of the uniform 40B SigLIP baseline with up to 13fewer iterations.Middle: Even when accounting for the cost of scoring, our best variant is almost 10more FLOP efficient.Right: Comparison of JEST++/FlexiJEST++ (green)  to prior methods (grey) . Average accuracy is computed across 8 downstream tasks (left, middle; see Table5) , or ImageNet and COCO (right) .
introduction
Data curation via joint example selectionfurther accelerates multimodal learning
1Introduction
2406.17711
2Related WorkOffline curation: example-level data pruning.Methods for collecting and filtering large-scale noisy image-text data initially focused on the quality of the textual captions, and proximity to high-quality reference datasets. Instead, model-based
filtering approaches use pretrained models (such as CLIPand BLIP)  as evaluation metrics for curating data via image-text alignment. Critically, all of these methods are applied independently across examples, which fails to account for the relevance of dependencies across examples in a batch.Offline curation: cluster-level data pruning.Other methods such as semantic redundancy reductionor core-set selectionhave proposed to curate based on the marginal importance of data points given other data points in their vicinity. However these methods are based on a heuristic that is decoupled from the training objective. In contrast, our method enables joint-example selection that is specifically tailored to accelerating the contrastive pretraining objective function.Online data curation with model-based scoring.Pre-filtering using the curation procedures described above can lead to large increases in data quality. However, fixed curation strategies do not take into account that the relevance of a training example can change over the course of learning, limiting their utility at scale. These concerns are addressed by online data curation methods, which identify high-quality examplesnot yet learned by the model. Our work generalizes these by applying model-based criteria to batch-level (rather than example-level)  losses, and selecting data accordingly.Hard negative mining.A long literature has described the efficiency gains afforded by choosing the right set of negative examples in classical metric-learningas well as modern contrastive learning.
We generalize hard negative mining in two ways: 1) 
we jointly mine for both positive and negative pairs by selecting entire batches and 2)  we explore prioritizinglearnablenegatives, which are hard for the learner but easy for a pretrained model.Model approximation.Several works have demonstrated that smaller models can be used as proxies for much larger models for data selection. However, several techniques have recently been developed that allow inference-time trade-offs between computation and performance, allowing smaller models to be “embedded” without the need for separate training. For Vision Transformers, dropping patchesor layers, or reducing token resolutionproduce characteristic trade-offs. Our work is the first to use these techniques in the context of online data selection.
body
Data curation via joint example selectionfurther accelerates multimodal learning
2Related Work
2406.17711
3Methods3.1Model-based batch-selection criteriaWe refer to the model which we are interested in training as thelearner. Assuming we have a “super-batch”(of size)  examples to learn from, we wish to extract a sub-batchthat is maximally relevant for learning. Prioritized samplingperforms this by scoring individual examples, then sampling in proportion to these scores. In this work we instead score entire sub-batches, and sample according to these batch-level scores. We consider model-based scoring functions, which use the losses from the learner model and/or pretrainedreferencemodels.Hard learner.An intuitive heuristic would be to prioritize batchesthat have a high loss under the learner with parameters:, which has the desirable property of discarding trivial data. This heuristic has been proven to work for small, clean datasetsbut tends to do more harm than good for larger, less curated datasetssince it will also up-sample noisy examples.Easy reference.In contrast, one could also choose to up-sample data that is “easy” (has low loss)  for a pretrainedreferencemodel with parameters:. Thiseasy referenceheuristic has been used successfully in multimodal learning to identify high-quality examples, but does not reflect the current state of the learner and can therefore be overly dependent on the choice of reference modeland not scale to large compute budgets.Learnability.Finally,propose to combine these scores, prioritizing with the difference of losses:. This heuristic, which we refer to aslearnabilityscoring throughout, has the advantage of up-sampling data that is both unlearned and learnable, and has been shown to accelerate large-scale learning even when prioritizing individual examples in isolation. In this work, we therefore mainly considerlearnabilityscoring but for completeness also provide ablations witheasy referencescoring.The ratio of the “sub-batch” and “super-batch” sizes defines thefiltering ratio, i.e. the proportion of data discarded at each iteration. For a given learner batch size, higher filtering ratios increase the cost of scoring as they require more inference passes on the super-batch.3.2Joint example selection (JEST)  for multimodal learningMultimodal learning losses.Given the availability of internet-scale datasets of paired images and text, multimodal learning has become the default means of training visual representations. Contrastive learning aims to maximize the alignment of these two modalities for paired examples, while minimizing the alignment of unpaired examples.
Both sigmoid-and softmax-contrastivelosses achieve this with a batch-level loss, where the conditional losscan use a sigmoid or softmax contrast function (see Appendix Equations1and2) .
Sincedemonstrate the sigmoid-contrastive loss to be a more scalable alternative to the softmax-contrastive one, we adopt it by default. Nevertheless, we show in AppendixA.6that our results also hold when using the softmax-contrastive loss.Joint example selection.Because the contrastive loss of a batch decomposes into a sum of conditional losses, thejoint learnabilityof the batchalso decomposes into a sum ofconditional learnabilitiesof each example given other examples in the batch.
We wish to sample batches in proportion to their joint learnability, i.e.,
which is enabled by a sequential approach inspired by
blocked Gibbs sampling (see Algorithm1) . Given a subset of examplesalready included in the batch at iteration, we compute the conditional learnability of remaining candidate exampleswith, and sample a new chunk of examplesindependently and without replacement according to these probabilities:.We update the batch by appending this chunk to the previous subset:, and iterate until, the number of chunks. The first chunkis sampled using unconditional learnability (i.e. self-similarity only) where the unconditional losses are computed asfor the softmax-contrastive loss andfor the sigmoid-contrastive loss.
We find that a relatively small number of chunks (, sampling= 2,048 examples independently at each iteration)  is sufficient to recover batches with very high learnability (see Section4.1, Algorithm1) .Algorithm 1Joint example selection: sigmoid loss⬇defjointly_sample_batch(learner_loss,ref_loss,n_chunks=16,filter_ratio=0.8,method="learnability") :scores=learner_loss-ref_lossifmethod=="learnability"else-ref_lossn_images=scores.shape[0]#scores.shape=[B,B]n_draws=int(n_images*(1-filter_ratio) /n_chunks) #Sizeofeachchunk.logits_ii=np.diag(scores) #Self-similarityscores.inds=random.choice(logits_ii,n_draws) #Samplefirstchunk.for_inrange(n_chunks-1) :is_sampled=np.eye(n_images) [inds].sum(axis=0) #Binaryindicatorofcurrentsamples[n_images,].logits_ij=(scores*is_sampled.reshape(n_images,1) ) .sum(axis=0) #Negativetermsij[n_images,].logits_ji=(scores*is_sampled.reshape(1,n_images) ) .sum(axis=1) #Negativetermsji[n_images,].logits=logits_ii+logits_ij+logits_ji#Conditionallearnabilitygivenpastsamples.logits=logits-is_sampled*1e8#Avoidsamplingwithreplacement.new_inds=random.choice(n_images,n_draws,p=np.exp(logits) ) inds=np.concatenate((inds,new_inds) ) #Expandthearrayofindicessampled.returninds#Gatherandreturnsubsetindices.3.3Efficient scoring and multi-resolution trainingEfficient scoring with online model approximation.Scoring large super-batches increases the cost per iteration, lowering the efficiency gains in terms of total FLOPs.
Whilerequired additional small, proxy models to efficiently score data on behalf of larger learners, we remove this requirement by using online model approximation.
We only approximate the image encoding since this accounts for the bulk of the cost of each inference pass.
For this we adopt the FlexiViT architecture, which lowers the image resolution while minimally degrading performance (see Figure9for a comparison to patch dropping) .
In our experiments, we evaluate the super-batch with 3232-pixel patches,
which gives a 72% reduction in FLOPs and 67% reduction in wall-clock time vs. full-resolution scoring at patch size 1616(see SectionA.4) .Multi-resolution training.While we want to score examples at low resolution (i.e. with large patches) , at test-time we wish to evaluate the model at full resolution (i.e. with small patches) . To enable both resolutions of computation, we simply train at both resolutions. Specifically, given a sub-batchfor learning, we randomly split it into two halves,and, and encode each half with a different resolution:.
These images embeddings are then concatenated together asand the rest of training proceeds as usual. In addition to allowing for efficient scoring, multi-resolution training itself yields a gain in efficiency: sinceis processed with 4fewer tokens, it also benefits from close to a 4speed-up. Ifandeach account for half of the batch, the cost of multi-resolution training onis 64% of the FLOPs and 67% of the time of full-resolution training. Pseudocode for the full-resolution JEST and multi-resolution Flexi-JEST implementation is detailed in Algorithm2.Figure 2:Joint example selection yields more learnable batches. Left:the learnability of a batch is highly structured and non-diagonal.Middle:Joint example selection quickly discovers sub-batches with high learnability, on-par with brute-force Gibbs sampling.Right:the learnability of sampled batches improves with higher filtering ratios (i.e. selecting from larger super-batches) .Training datasets.We train the learner model in all JEST experiments on the WebLI dataset, specifically
a billion-scale subset of English image-text pairs loosely filtered with image-text alignment, using thebig_visioncodebase. To train reference models, we use smaller high-quality datasets. JEST/Flexi-JEST reference models are trained on a strongly filtered 100M scale subset of WebLI filtered for high text and image quality and image-text alignment, which we refer to as “WebLI-curated”. We additionally explorescaling data curation(JEST++/FlexiJEST++)  using reference models trained on “WebLI-curated++” which adds approximately 600M additional web-scraped image-text pairs
filtered with the same strong curation pipeline. Unless specified otherwise, the average performance we report uses 4 canonical benchmarks for multimodal transfer: ImageNet 0-Shot and 10-Shot classification and COCO image-to-text and text-to-image top-1 retrieval.
body
Data curation via joint example selectionfurther accelerates multimodal learning
3Methods
2406.17711
4Experiments4.1Joint example selection yields learnable batchesWe start by evaluating the efficacy of joint example selection (JEST)  for selecting learnable batches. To gain an intuition for our method, we start by visualizing the learnability matrix (i.e. the difference in loss between learner and reference models, for all pairs of examples in the batch) .
JEST is designed to sample sub-matrices of examples in proportion to their summed learnability. Since the matrix is strongly non-diagonal (Figure2, left) , independent selection will clearly be sub-optimal.With a small number of iterations (corresponding to populating the batch withchunks) , we find the learnability of the sub-batch to quickly increase, matching the learnability of batches extracted by brute-force Gibbs sampling requiring thousands of iterations (Figure2, middle) .For filtering ratios of 0.5, 0.8, and 0.9, we select sub-batches of 32,768 examples from super-batches of size 65,536, 163,840 and 327,680 respectively. In Figure2, right, we find that the learnability of the sub-batch increases with larger filtering ratios.
In summary, our joint example selection (JEST)  algorithm is an effective and efficient means of selecting highly learnable batches during training.Figure 3:Joint example selection accelerates multimodal learning. Left:training on the most learnable sub-batch selected from super-batches that are 2, 5, or 10larger significantly accelerates multimodal learning.Middle:Jointly prioritizinglearnable batchesyields significantly better results than simply prioritizing individual examples.Right:joint examples selection also improveseasy referenceprioritization, althoughlearnabilityscales better with more aggressive filtering.4.2Joint example selection accelerates multimodal learningWe now investigate the effect of training on more learnable batches, as selected by our JEST algorithm. All runs use a reference model trained on WebLI-curated, a ViT-B/16 and Bert-B image-text dual encoder, 3 billion training examples, and the sigmoid-contrastive loss. Figure3(left)  shows the average performance on multiple downstream tasks (ImageNet 0-Shot/10-Shot accuracy and COCO image-to-text/text-to-image retrieval)  over the course of training. We find that JEST
significantly accelerates learning,
reaching the final performance of the 3B-uniform baseline after only 2B, 1B, and 0.67B training examples, when using filtering ratios of 50%, 80%, and 90% respectively. At larger filtering ratios we observe similar training instabilities to those observed for larger batch sizes, necessitating a modification to stabilize the Adam optimizer ()  and suggesting that data curation with JEST can be thought of as increasing the effective batch size (AppendixA.2,A.3) .In terms of final performance, JEST also delivers significant gains of up to
6%
when filtering 90% of data (Figure3, middle, blue curve) . Notably, this scaling behavior is absent from previous selection methods based on independent prioritization of individual examples (Figure3, middle, orange curve) . Finally, we assess whether
JEST also improves prioritization criteria other than learnability. Figure3, right, shows the performance of models witheasy-referenceprioritization, for varying filtering ratios. Consistent with learnability-based prioritization, JEST strongly outperforms independent example selection, particularly for high filtering ratios (where independent example selection leads to a regression in performance) . Prioritising data with the highest loss produced smaller gains and degrades more quickly as we filter more data (Figure10) . Since learnability-based JEST yielded the best scaling behavior
we retain this criterion for subsequent experiments.4.3Synergies between multi-resolution training and online batch selectionJoint example selection
withlearnabilityscores becomes more efficient as larger fractions of each batch are filtered.
However, the cost of scoring
results in a significant overhead:
filtering 80% of the super-batch results in 4more FLOPs per iteration than IID training, or 2.3when caching the reference-model scores (AppendixA.4) .
Although JEST is significantly more efficient in terms of training iterations (hereinafter ‘training efficiency’) , the additional scoring FLOPs reduce its compute efficiency relative to the IID baseline
(Figure1, left vs. right) . We therefore also investigated a compute efficient variant, Flexi-JEST, which uses multi-resolution training and low-resolution scoring to
reduce the total overhead to only 10% vs. the baseline
(Figure4, left; see SectionA.4) .What is the effect of these approximations on performance? As might be expected, the per-iteration performance of Flexi-JEST decreases relative to the JEST, although still produces significant speed-ups over IID (Figure1, left; Figure4, middle) . However, the decrease in per-iteration performance is more than favorable when accounting for the decrease in total FLOPS: our best Flexi-JEST model produces the same average performance as a 40B Siglip run with 9.9fewer FLOPs, and  2fewer than full-resolution JEST (Figure1, right; Figure4, middle) .Figure 4:Efficient scoring and multi-resolution training.Left:In scoring large super-batches with the learner and reference models, JEST incurs a large computational cost per iteration. By caching the fixed reference model scores in the dataset, this overhead can be cut in half.
Efficient scoring and multi-resolution training further reduce this to be comparable to standard IID training.Middle:Flexi-JEST improves the total FLOP-efficiency of JEST over standard IID training.Right:Multi-resolution training improves FlexiJEST more than standard IID training.
Without multi-resolution training (left-most point)  Flexi-JEST underperforms the IID baseline (due to an untrained approximate model) , but quickly improves with even a small amount of co-training (25%) .What is the relative contribution of efficient scoring and multi-resolution training in Flexi-JEST?
We conducted an ablation where we varied the fraction of the selected batch trained at full and low resolution (i.e. the relative sizes ofand; see Methods) .
We ensure the learner spends the same FLOPs by increasing the number of training iterations as we send more data to the approximate model, since the FLOPs per iteration decrease in this case (see SectionA.7for details) . Figure4(right)  shows that the IID baseline performance increases with larger fractions of data sent to the approximate model, consistent with a growing literature on the FLOP-efficiency of approximate training.
Nevertheless, Flexi-JEST significantly outperforms the multi-resolution baseline as long as the low-res model trains on enough data to align with the learner
(e.g.25% data) . These experiments demonstrate a synergy between multi-resolution training and joint example selection, as the former yields efficient and accurate scoring capabilities for accelerating the latter.Our results also point to a pareto front of data curation strategies. If maximizing training speed or training efficiency is desirable at the expense of computation, the full-resolution JEST method produces up to a 13speed up relative to a comparable IID training run. If FLOPs should be minimized at the expense of training efficiency, Flexi-JEST produces the most favorable trade-off. We note that the scoring of the next batch can be implemented on separate devices, in parallel with training,
potentially further reducing the additional wall-clock time.4.4Joint examples selection enables strong data-quality bootstrappingAt the heart of learnability-based scoring is a reference model trained on a small, curated dataset of our choosing.
How does JEST performance vary as a function of different curation strategies that trade off quality vs. quantity? Furthermore, do improvements in JEST training correlate with the performance of the reference models or are these metrics decoupled?Understanding quality vs. quantity trade-offs. We explore three scales of curation, each being a subset of the original WebLI dataset:weak(billion-scale)  curation
with image-text alignment (ITA)  filters,moderate(300M scale)  curation with either ITA filters or text-quality (TQ)  filters, andstrong(100M scale)  curation with a combination of TQ, ITA, and additional image-quality (aesthetic)  filters.
Throughout, we refer to this strongly curated subset as “WebLI-curated”.We train standard SigLIP encoders on these four WebLI subsets for 10 epochs each, and use them as reference models for JEST training on the full WebLI dataset. Across curation methods, reference model performance and JEST performance appear to be decoupled (or even anti-correlated; Figure5, left) , consistent with previous findings for fixed data curation.
Whereas increasing curation (and decreasing dataset size)  yields weaker models, when used as reference models for JEST pretraining they have the opposite effect: JEST with a strongly-curated reference benefits from a 2.7% improvement, moderate a 1.5% improvement, and weak a 0.3% improvement.Figure 5:Scaling strong data curation improves JEST performance.Left:We compare JEST performance vs. reference model performance (relative to the uniform baseline)  for 4 curation types: ‘weak’ curation with image-text alignment (ITA) , ‘moderate’ curation with ITA or text-quality (TQ) , and ‘strong’ curation (using a combination of TQ, ITA, and additional image-quality (IQ) .Right:We use our best reference dataset (TQ+ITA+IQ)  and evaluate JEST vs. reference performance varying the number of examples seen during reference pretraining. There is a strong correlation between additional reference training and JEST performance that saturates after 1B examples seen. By scaling strong data curation to a 600M dataset, this saturation is broken as both reference model and JEST performance improve for the 1Band2B reference training.Scaling data curation.
We hypothesized that
the general decoupling between reference model performance and JEST performance might simply be explained by the dataset size limits imposed by data curation.
To understand this effect, we trained 5 reference models on WebLI-curated while varying the total examples seen (250M to 3B) . In this context, Figure5(right)  shows a striking correlation between improved reference models and better JEST pretraining. This suggests that the “decoupling” phenomenon can be mostly attributed to the saturation of the reference model as a result of the reductions in dataset size following curation.We note that the correlation in Figure5(right)  starts to break down when the dataset is saturated, i.e. after 10 epochs or 1B examples seen.
These results suggest that JEST would benefit further from scaling data curation of the reference datasets. To test this, we grew WebLI-curated to approximately 600M examples sourced from an expanded set of image-text pairs. At this scale, however, it is difficult to satisfy the stringent TQ/ITA criteria of WebLI-curated. Therefore, for all image-text pairs that did not meet the original ITA threshold, we re-captioned the images with high-quality synthetic captions based on the PaLI model family—we denote this dataset as “WebLI-curated++”. We find that this scaled dataset allows us to break the 2B saturation point for “WebLI-curated” as both reference model and JEST performance (Figure5, Right:)  improves significantly. We therefore use WebLI-curated++ for our best models, JEST++ and FlexiJEST++.Given that scaling data curation with WebLI-curated++ strongly improves reference model performance, we asked whether pretraining on the original WebLI dataset is necessary at all. However when evaluating the reference model’s performanceacross datasetswe find it to be very imbalanced: while it outperforms WebLI pretraining on 2 downstream tasks, it significantly underperforms on 6 others, as well as on average (Table5) . In contrast, JEST++ pretraining on WebLI yields ageneralistfoundation model that outperforms the baseline on 6 of 8 benchmarks, as well as on average.FLOPs %ImageNet-1KCOCOMethodVariant# TrainPer Iter.TotalMean10-SZSI2TT2ICLIPB13B10032– 11.868.352.433.1EVA-CLIPB8B10020– 4.674.758.742.2OpenCLIPB34B10085– 5.870.259.442.3LessIsMoreB11B10028– 5.970.858.342.5SILC-SB20B3801900.268.976.666.248.7SigLIPB40B1001000.070.376.765.247.4JEST++B4B233232.870.376.970.353.3Flexi-JEST++B4B110110.968.275.868.051.2CLIPL13B10032– 11.075.556.336.5EVA-CLIPL4B10010– 3.479.863.747.5OpenCLIPL32B10080– 6.374.062.146.1SigLIPL40B1001000.077.180.569.551.2JEST++L4B233231.875.580.571.154.8Table 1:Comparison to prior art.FLOP % are measured relative to SigLIP. Mean denotes the average performance over all metrics. “Per Iter.” denotes FLOPs per iteration.4.5Comparison to prior artWe now compare to prior art, including the state-of-art SigLIP model trained for 40 billion examplesas well as recent strong CLIP variants.
Table1shows that our most training-efficient model, JEST++, sets a new state-of-the-art on both ImageNet and COCO all while using 10fewer iterations and 4less compute. On COCO in particular, JEST++ improves the previous state of the art by over 5%. Our most compute-efficient model, Flexi-JEST++, also surpasses the previous SoTA on average, while using 9less compute. Training JEST for longer furthered these gains (see Appendix Table4) .Our results also scale gracefully with model size. Training with a ViT-L learner and ViT-L reference trained on the same WebLI-curated++ dataset,
JEST++
continues to yield strongly accelerated learning,
matching the SigLIP ViT-L 40B baseline with only 4B examples seen (Table1, bottom) .Finally, we apply JEST++ for pretraining on the publicly available LAION-2B dataset. We follow the standard practice of removing unsafe image-text pairs, but do not otherwise pre-filter the dataset. JEST++ strongly surpasses previous methods for offline data curation, despite using 4fewer training examples than the previous state-of-the-art (Table2) . With this training budget, SigLIP pretraining severely under-performs all methods, further highlighting the benefits of JEST.Method# TrainIN1K ZSCOCOLAION-440M12.8B64.148.1SemDeDup8.8B64.348.9DBP5.3B65.548.4DBP3.6B64.145.7SigLIP1.3B57.243.3JEST++1.3B66.854.8Table 2:Comparison to LAION pretraining.JEST++ strongly surpasses prior art while requiring significantly fewer training iterations. COCO performance denotes the average of image-to-text and text-to-image retrieval.4.6Simplifying data curationThe WebLI datasetused for our pretraining experiments has already been filtered for high image-text alignment. As shown in Table3, this offline curation is critical for strong IID training performance. For JEST++ however, this pre-filtering is redundant as performance does not degrade when training on the unfiltered WebLI, alleviating the need for foundation datasets.MethodFiltered Data?IN1K ZSCOCOIID✓73.652.5✗69.449.5JEST++✓76.961.8✗76.761.8Table 3:Simplifying data curation.All models are trained for 4B examples seen. Performance for JEST++ is nearly identical for pre-training on raw (uncurated)  vs. filtered data. Color gradients measured relative to IID training on filtered WebLI.
body
Data curation via joint example selectionfurther accelerates multimodal learning
4Experiments
2406.17711
5DiscussionWe proposed a method—JEST—for jointly selecting the most learnable batches of data, which significantly accelerates large-scale multimodal learning, surpassing the previous state-of-the-art with up to 10fewer FLOPs and 13fewer examples. In particular, our experiments point to the strong potential for “data quality bootstrapping”, using small curated datasets to guide learning on much larger, uncurated ones.Recent work has shown that static dataset filtering, without knowledge of downstream training, can ultimately limit performance. Our results demonstrate that
useful batches, which must be constructed online, improve pretraining efficiency beyond individually selected examples. These findings therefore advocate forfoundation distributions—either through pre-scored datasets witheasy-referenceJEST, or dynamically adjusted to the demands of the model withlearnabilityJEST—as a more general and effective replacement to generic foundation datasets.Limitations.While our method has accelerated multimodal learning of canonical downstream tasks, it has relied on small, well-curated reference datasets which specify the distribution to prioritize within much larger uncurated data. We would therefore encourage future work exploring the inference of reference datasets from the set of downstream tasks of interest.
conclusion
Data curation via joint example selectionfurther accelerates multimodal learning
5Discussion