<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2406.05184] The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better</title><meta property="og:description" content="Generative text-to-image models enable us to synthesize unlimited amounts of images in a controllable manner, spurring many recent efforts to train vision models with synthetic data.
However, every synthetic image ulti…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2406.05184">

<!--Generated on Sat Jul  6 00:26:36 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">The Unmet Promise of Synthetic Training Images:
<br class="ltx_break">Using Retrieved Real Images Performs Better</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">
Scott Geng<sup id="id10.10.id1" class="ltx_sup"><span id="id10.10.id1.1" class="ltx_text ltx_font_italic">♣</span></sup>    
Cheng-Yu Hsieh<sup id="id11.11.id2" class="ltx_sup"><span id="id11.11.id2.1" class="ltx_text ltx_font_italic">♣</span></sup>    
Vivek Ramanujan<sup id="id12.12.id3" class="ltx_sup">♣</sup>    
Matthew Wallingford<sup id="id13.13.id4" class="ltx_sup">♣</sup> 
<br class="ltx_break"><span id="id8.8.4" class="ltx_text ltx_font_bold">
Chun-Liang Li<sup id="id8.8.4.1" class="ltx_sup"><span id="id8.8.4.1.1" class="ltx_text ltx_font_medium ltx_font_italic">♣</span></sup>      
Pang Wei Koh  <sup id="id8.8.4.2" class="ltx_sup"><span id="id8.8.4.2.1" class="ltx_text ltx_font_medium ltx_font_italic">♣♠</span></sup>      
Ranjay Krishna<span id="footnotex1" class="ltx_note ltx_role_footnotemark"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">footnotemark: </span></span></span></span>  <sup id="id8.8.4.3" class="ltx_sup"><span id="id8.8.4.3.1" class="ltx_text ltx_font_medium ltx_font_italic">♣♠</span></sup>
<br class="ltx_break">
<br class="ltx_break"><sup id="id8.8.4.4" class="ltx_sup"><span id="id8.8.4.4.1" class="ltx_text ltx_font_medium">♣</span></sup></span>University of Washington  
<sup id="id14.14.id5" class="ltx_sup">♠</sup>Allen Institute for AI 
<br class="ltx_break"><span id="id15.15.id6" class="ltx_text ltx_font_typewriter">sgeng@cs.washington.edu</span>
</span><span class="ltx_author_notes"><span id="id16.16.id1" class="ltx_text ltx_font_bold">Equal advising.</span></span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id17.id1" class="ltx_p">Generative text-to-image models enable us to synthesize unlimited amounts of images in a controllable manner, spurring many recent efforts to train vision models with synthetic data.
However, every synthetic image ultimately originates from the upstream data used to train the generator. What additional value does the intermediate generator provide over directly training on relevant parts of the upstream data?
Grounding this question in the setting of image classification, we compare finetuning on task-relevant, targeted synthetic data generated by Stable Diffusion—a generative model trained on the LAION-2B dataset—against finetuning on targeted real images retrieved directly from LAION-2B. We show that while synthetic data can benefit some downstream tasks, it is universally matched or outperformed by real data from the simple retrieval baseline.
Our analysis suggests that this underperformance is partially due to generator artifacts and inaccurate task-relevant visual details in the synthetic images.
Overall, we argue that retrieval is a critical baseline to consider when training with synthetic data—a baseline that current methods do not yet surpass. We release code, data, and models at <a target="_blank" href="https://github.com/scottgeng00/unmet-promise/" title="" class="ltx_ref ltx_href ltx_font_typewriter">https://github.com/scottgeng00/unmet-promise</a>.</p>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">The success of modern machine learning systems fundamentally relies on the quantity <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib33" title="" class="ltx_ref">33</a>, <a href="#bib.bib10" title="" class="ltx_ref">10</a>, <a href="#bib.bib28" title="" class="ltx_ref">28</a>, <a href="#bib.bib55" title="" class="ltx_ref">55</a>, <a href="#bib.bib50" title="" class="ltx_ref">50</a>, <a href="#bib.bib63" title="" class="ltx_ref">63</a>]</cite>, quality <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib20" title="" class="ltx_ref">20</a>, <a href="#bib.bib70" title="" class="ltx_ref">70</a>, <a href="#bib.bib44" title="" class="ltx_ref">44</a>, <a href="#bib.bib43" title="" class="ltx_ref">43</a>, <a href="#bib.bib36" title="" class="ltx_ref">36</a>]</cite>, and distribution <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib17" title="" class="ltx_ref">17</a>, <a href="#bib.bib22" title="" class="ltx_ref">22</a>, <a href="#bib.bib60" title="" class="ltx_ref">60</a>, <a href="#bib.bib68" title="" class="ltx_ref">68</a>, <a href="#bib.bib9" title="" class="ltx_ref">9</a>]</cite> of the data they are trained on. However, acquiring large amounts of quality data remains challenging, due to the sheer cost of data collection and annotation. As demand for training data continues to rise, the field is actively exploring approaches to automatically curate data at scale <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib20" title="" class="ltx_ref">20</a>, <a href="#bib.bib2" title="" class="ltx_ref">2</a>, <a href="#bib.bib18" title="" class="ltx_ref">18</a>]</cite>. One burgeoning approach is to source <span id="S1.p1.1.1" class="ltx_text ltx_font_italic">synthetic training data</span> from conditional generative models. Generative models enable data to be tailored to specific requirements and generated at scale, presenting a promising alternative to the challenges of real data curation.
Recent work highlights this potential: for example, in natural language processing (NLP), researchers prompt strong proprietary language models to cheaply synthesize large-scale datasets for instruction tuning <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib66" title="" class="ltx_ref">66</a>, <a href="#bib.bib29" title="" class="ltx_ref">29</a>, <a href="#bib.bib59" title="" class="ltx_ref">59</a>]</cite>.</p>
</div>
<figure id="S1.F1.fig1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_middle" style="width:156.1pt;">
<div id="S1.F1.1" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_middle" style="width:268.8pt;">
<img src="/html/2406.05184/assets/x1.png" id="S1.F1.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="273" alt="Refer to caption">
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>
Given an upstream dataset of general real image-text pairs, we aim to curate a <span id="S1.F1.fig1.5.1" class="ltx_text ltx_font_italic">targeted</span> dataset to train a learner on some target task. We can either (1) <span id="S1.F1.fig1.6.2" class="ltx_text ltx_font_bold" style="color:#0174B2;">retrieve targeted real images</span> directly from the upstream dataset, or we can (2) first train an intermediate generative model and then <span id="S1.F1.fig1.7.3" class="ltx_text ltx_font_bold" style="color:#DF680B;">synthesize targeted synthetic images</span>. By comparing these two approaches, our paper seeks to measure what value training on generated synthetic data adds.
</figcaption>
</figure>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">Analogously, in computer vision—the focus of our research—many recent works train models on synthetic images from modern text-to-image generators, aiming to achieve state-of-the-art visual recognition performance  <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib54" title="" class="ltx_ref">54</a>, <a href="#bib.bib61" title="" class="ltx_ref">61</a>, <a href="#bib.bib3" title="" class="ltx_ref">3</a>, <a href="#bib.bib23" title="" class="ltx_ref">23</a>, <a href="#bib.bib24" title="" class="ltx_ref">24</a>]</cite>.
For example, SynCLR <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite> cleverly prompts Stable Diffusion for synthetic images tailored to pre-specified downstream image recognition domains; they find that a CLIP-like model trained from scratch on the resulting <span id="S1.p2.1.1" class="ltx_text ltx_font_italic">targeted</span> synthetic images can outperform CLIP trained on LAION-2B, a significantly larger untargeted dataset of real images. This result is quite surprising.
Stable Diffusion is <span id="S1.p2.1.2" class="ltx_text ltx_font_italic">also</span> trained on LAION-2B, so by the data processing inequality, the synthetic images it generates cannot contain any additional information over the images in LAION-2B. Yet, training on these derivative synthetic images appears to outperform training directly on LAION-2B. How do we make sense of these additional gains?</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">In this paper, we argue that the performance gained by training on generated synthetic images needs to be contextualized against a critical baseline often missing in prior work: training on real images from the generative model’s
pretraining data.
In particular, prior work has often compared <span id="S1.p3.1.1" class="ltx_text ltx_font_italic">task-targeted</span> synthetic images to <span id="S1.p3.1.2" class="ltx_text ltx_font_italic">general, untargeted</span> real images, thereby entangling the effects of training on synthetic versus real images with the effects of targeted versus general data collection. However, these variables are not intrinsically conflated. Any generative model we use to synthesize images fundamentally derives from its upstream training data. Instead of using that upstream data to train an intermediate generative model and synthesize targeted synthetic images, we can alternatively seek to directly retrieve targeted real images from the upstream source (Figure <a href="#S1.F1.fig1" title="Figure 1 ‣ 1 Introduction ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>). By comparing synthetic training data against this retrieval baseline, we isolate the value added by the generative model.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">We formalize our study under the ubiquitous problem of task adaptation, where we seek to curate task-targeted images to finetune a pretrained vision model.
We empirically compare training on targeted synthetic images generated from Stable Diffusion 1.5—a text-to-image model trained on the upstream LAION-2B dataset—against training on targeted real images retrieved from LAION-2B itself. We perform hundreds of experiment runs across an order of magnitude of data scales on five visual recognition tasks (ImageNet <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib14" title="" class="ltx_ref">14</a>]</cite>, Describable Textures (DTD) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib12" title="" class="ltx_ref">12</a>]</cite>, FGVC-Aircraft <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib40" title="" class="ltx_ref">40</a>]</cite>, StanfordCars <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib35" title="" class="ltx_ref">35</a>]</cite>, and Oxford Flowers102 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib45" title="" class="ltx_ref">45</a>]</cite>) where training on synthetic data has shown promise <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>, <a href="#bib.bib23" title="" class="ltx_ref">23</a>]</cite>.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">Together, we find that <span id="S1.p5.1.1" class="ltx_text ltx_font_bold">training on targeted real data retrieved from a generative model’s upstream training dataset outperforms training on synthetic data from the generative model.</span> For example, while training on targeted synthetic images can improve downstream accuracy by up to 7.1% (absolute) on its best-case benchmark (FGVC-Aircraft), training on targeted real images helps even further, boosting accuracy by a massive <math id="S1.p5.1.m1.1" class="ltx_Math" alttext="17.7\%" display="inline"><semantics id="S1.p5.1.m1.1a"><mrow id="S1.p5.1.m1.1.1" xref="S1.p5.1.m1.1.1.cmml"><mn id="S1.p5.1.m1.1.1.2" xref="S1.p5.1.m1.1.1.2.cmml">17.7</mn><mo id="S1.p5.1.m1.1.1.1" xref="S1.p5.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S1.p5.1.m1.1b"><apply id="S1.p5.1.m1.1.1.cmml" xref="S1.p5.1.m1.1.1"><csymbol cd="latexml" id="S1.p5.1.m1.1.1.1.cmml" xref="S1.p5.1.m1.1.1.1">percent</csymbol><cn type="float" id="S1.p5.1.m1.1.1.2.cmml" xref="S1.p5.1.m1.1.1.2">17.7</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.p5.1.m1.1c">17.7\%</annotation></semantics></math>. On other benchmarks, such as ImageNet, we find that training on synthetic images can sometimes <span id="S1.p5.1.2" class="ltx_text ltx_font_italic">hurt</span> performance even when training on real data improves it.
We further show that these findings hold across several different versions of Stable Diffusion, as well as when we train on a mix of synthetic and real data. Our analysis suggests that the consistent underperformance of models trained on synthetic images is partially due to low-level generator artifacts in the synthetic images (e.g., blurs), and partially because synthetic images may distort high-level class-specific visual details that real images preserve.</p>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p">Overall, we conclude that retrieval is a critical baseline to consider when evaluating the true added utility of generated synthetic training data.
Our goal is not to make normative claims about whether synthetic data will ever surpass this standard, but to contribute a simple baseline to aim for, and a clean set of experiments to explicitly measure progress towards surpassing it. For instance, by conceptualizing retrieval from a generator’s training data as a strong alternative to synthesizing data, a natural future direction for improving synthetic training data is to synthesize image compositions that are explicitly absent from the generator’s upstream training set. Images generated in this manner may offer unique value beyond what can be retrieved from the training data.
Finally, in settings where the upstream dataset of a generator is unavailable altogether (e.g., due to privacy concerns, due to proprietary data, or due to download bandwidth restrictions), the retrieval baseline is unrealizable by assumption; synthetic data therefore retains strong utility for distilling knowledge from generative models and for privacy preservation. We release all code, models, and over 1TB of generated images to guide future work (<a target="_blank" href="https://github.com/scottgeng00/unmet-promise/" title="" class="ltx_ref ltx_href ltx_font_typewriter">https://github.com/scottgeng00/unmet-promise</a>).</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<section id="S2.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Learning from synthetic data.</h4>

<div id="S2.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px1.p1.1" class="ltx_p">Synthetic data has been widely explored in the context of many machine learning problems <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib66" title="" class="ltx_ref">66</a>, <a href="#bib.bib29" title="" class="ltx_ref">29</a>, <a href="#bib.bib25" title="" class="ltx_ref">25</a>, <a href="#bib.bib32" title="" class="ltx_ref">32</a>, <a href="#bib.bib59" title="" class="ltx_ref">59</a>, <a href="#bib.bib56" title="" class="ltx_ref">56</a>, <a href="#bib.bib5" title="" class="ltx_ref">5</a>, <a href="#bib.bib39" title="" class="ltx_ref">39</a>]</cite>.
For example, in NLP, synthetic data generated from strong large language models <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib46" title="" class="ltx_ref">46</a>, <a href="#bib.bib11" title="" class="ltx_ref">11</a>]</cite> has been used to distill instruction-following behavior <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib66" title="" class="ltx_ref">66</a>]</cite> and task-specific knowledge <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib30" title="" class="ltx_ref">30</a>]</cite> into smaller models. In computer vision, prior works have sought to use synthetic data to improve the state-of-the-art across a breadth of visual tasks, such as object detection <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib47" title="" class="ltx_ref">47</a>, <a href="#bib.bib31" title="" class="ltx_ref">31</a>]</cite>, semantic segmentation <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib53" title="" class="ltx_ref">53</a>, <a href="#bib.bib51" title="" class="ltx_ref">51</a>, <a href="#bib.bib8" title="" class="ltx_ref">8</a>]</cite>, and optical flow <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib15" title="" class="ltx_ref">15</a>]</cite>. Traditionally, this synthetic training data has been sourced from expert-crafted simulation and rendering pipelines <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib48" title="" class="ltx_ref">48</a>, <a href="#bib.bib15" title="" class="ltx_ref">15</a>, <a href="#bib.bib51" title="" class="ltx_ref">51</a>, <a href="#bib.bib8" title="" class="ltx_ref">8</a>, <a href="#bib.bib53" title="" class="ltx_ref">53</a>, <a href="#bib.bib31" title="" class="ltx_ref">31</a>]</cite>.
Recent advances in text-to-image synthesis via diffusion models <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib57" title="" class="ltx_ref">57</a>, <a href="#bib.bib27" title="" class="ltx_ref">27</a>, <a href="#bib.bib52" title="" class="ltx_ref">52</a>]</cite> are changing this paradigm, inspiring a new wave of works that seek to train visual models on synthetic data algorithmically sampled from modern generative models <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib24" title="" class="ltx_ref">24</a>, <a href="#bib.bib61" title="" class="ltx_ref">61</a>, <a href="#bib.bib54" title="" class="ltx_ref">54</a>, <a href="#bib.bib23" title="" class="ltx_ref">23</a>, <a href="#bib.bib72" title="" class="ltx_ref">72</a>]</cite>. This structural shift in the source of synthetic images from <span id="S2.SS0.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_italic">expert-supervised programmatic simulation</span> to <span id="S2.SS0.SSS0.Px1.p1.1.2" class="ltx_text ltx_font_italic">a learned generator</span> that itself derives supervision from upstream data raises a critical question: does the intermediate step of training a generator and sampling synthetic data provide any gains over simply training on the upstream data directly? Our work formalizes and empirically grounds this question, contributing experiments and baselines to rigorously measure the benefits of training on modern data-derived synthetic data.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Adapting pretrained vision models.</h4>

<div id="S2.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px2.p1.1" class="ltx_p">Large-scale pretrained image models such as CLIP <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib49" title="" class="ltx_ref">49</a>, <a href="#bib.bib10" title="" class="ltx_ref">10</a>]</cite> offer transferable visual features that benefit a wide range of downstream vision tasks. It is now common practice to use pretrained models as a starting point when deploying downstream task-specific models instead of training them from scratch <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib67" title="" class="ltx_ref">67</a>, <a href="#bib.bib64" title="" class="ltx_ref">64</a>]</cite>. From an algorithmic perspective, many methods have been proposed to adapt CLIP models to downstream tasks, each with varying trade-offs <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib69" title="" class="ltx_ref">69</a>, <a href="#bib.bib71" title="" class="ltx_ref">71</a>, <a href="#bib.bib41" title="" class="ltx_ref">41</a>, <a href="#bib.bib21" title="" class="ltx_ref">21</a>, <a href="#bib.bib4" title="" class="ltx_ref">4</a>, <a href="#bib.bib67" title="" class="ltx_ref">67</a>]</cite>. We choose to study simple full-finetuning, centering our work on the <span id="S2.SS0.SSS0.Px2.p1.1.1" class="ltx_text ltx_font_italic">data</span> we adapt on as opposed to the algorithm. In particular, the quality and relevance of adaptation data has a crucial impact on downstream task performance; distribution shifts at inference time can significantly hurt performance <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib34" title="" class="ltx_ref">34</a>]</cite>. Acquiring task-targeted data thus remains an active area of research <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib38" title="" class="ltx_ref">38</a>, <a href="#bib.bib65" title="" class="ltx_ref">65</a>, <a href="#bib.bib25" title="" class="ltx_ref">25</a>]</cite>. Most related to our work is <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib38" title="" class="ltx_ref">38</a>, <a href="#bib.bib65" title="" class="ltx_ref">65</a>]</cite>, who also employ retrieval as a technique for collecting task targeted-data. Our work builds upon these methods to construct baselines for systematically measuring the true added utility of model-generated synthetic training images.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Retrieval as a baseline for synthetic data.</h4>

<div id="S2.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px3.p1.1" class="ltx_p">Recent studies <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib72" title="" class="ltx_ref">72</a>, <a href="#bib.bib7" title="" class="ltx_ref">7</a>]</cite> similarly use retrieval from generative model training data as a baseline for synthetic training images. For example, <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib72" title="" class="ltx_ref">72</a>]</cite> adapts Stable Diffusion with real medical images and then generates synthetic medical images to train a ResNet from scratch; the resulting ResNet outperforms a ResNet trained on LAION-retrieved medical images. Consistent with our work, <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib72" title="" class="ltx_ref">72</a>, <a href="#bib.bib7" title="" class="ltx_ref">7</a>]</cite> find that in the low-data regime of general image recognition tasks (e.g., subsampled ImageNet-1K, STL-10), augmenting real datasets with LAION-retrieved images outperforms augmenting with Stable Diffusion generated images when training ResNets from scratch. <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib7" title="" class="ltx_ref">7</a>]</cite> further finds that this performance gap in the low-data regime persists across many synthetic image generation methods, including when they finetune Stable Diffusion with task-specific real data. In the medium-data regime (e.g. full ImageNet-1k), <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib7" title="" class="ltx_ref">7</a>]</cite> finds augmenting with synthetic data matches but does not outperform augmenting with retrieved data. Our work generalizes the shared motivation of measuring the utility of synthetic training images to the modern data-rich regime and a wide range of visual classification tasks. We center our experiments on the scaling behavior of synthetic versus retrieved data for finetuning a web-scale pretrained CLIP model on general image recognition (ImageNet-1K), fine-grained recognition (FGVC-Aircraft, Flowers102, StanfordCars), and abstract concepts (DTD). We show that retrieved data often strictly outperforms synthetic data even at large data scales. We further contribute analyses to unpack the consistently observed performance gap between synthetic and retrieved real training images.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Problem Setting and Method</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.9" class="ltx_p">Given a large dataset <math id="S3.p1.1.m1.1" class="ltx_Math" alttext="{\mathcal{D}}" display="inline"><semantics id="S3.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S3.p1.1.m1.1.1" xref="S3.p1.1.m1.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S3.p1.1.m1.1b"><ci id="S3.p1.1.m1.1.1.cmml" xref="S3.p1.1.m1.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.1.m1.1c">{\mathcal{D}}</annotation></semantics></math> of general real image-text pairs and a downstream visual classification task specified as a set of text class names <math id="S3.p1.2.m2.1" class="ltx_Math" alttext="{\mathcal{C}}" display="inline"><semantics id="S3.p1.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S3.p1.2.m2.1.1" xref="S3.p1.2.m2.1.1.cmml">𝒞</mi><annotation-xml encoding="MathML-Content" id="S3.p1.2.m2.1b"><ci id="S3.p1.2.m2.1.1.cmml" xref="S3.p1.2.m2.1.1">𝒞</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.2.m2.1c">{\mathcal{C}}</annotation></semantics></math>, we aim to algorithmically curate a <span id="S3.p1.3.1" class="ltx_text ltx_font_bold">targeted adaptation dataset <math id="S3.p1.3.1.m1.1" class="ltx_Math" alttext="{\mathcal{D}_{\mathcal{C}}}" display="inline"><semantics id="S3.p1.3.1.m1.1a"><msub id="S3.p1.3.1.m1.1.1" xref="S3.p1.3.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.p1.3.1.m1.1.1.2" xref="S3.p1.3.1.m1.1.1.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S3.p1.3.1.m1.1.1.3" xref="S3.p1.3.1.m1.1.1.3.cmml">𝒞</mi></msub><annotation-xml encoding="MathML-Content" id="S3.p1.3.1.m1.1b"><apply id="S3.p1.3.1.m1.1.1.cmml" xref="S3.p1.3.1.m1.1.1"><csymbol cd="ambiguous" id="S3.p1.3.1.m1.1.1.1.cmml" xref="S3.p1.3.1.m1.1.1">subscript</csymbol><ci id="S3.p1.3.1.m1.1.1.2.cmml" xref="S3.p1.3.1.m1.1.1.2">𝒟</ci><ci id="S3.p1.3.1.m1.1.1.3.cmml" xref="S3.p1.3.1.m1.1.1.3">𝒞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.3.1.m1.1c">{\mathcal{D}_{\mathcal{C}}}</annotation></semantics></math></span> of images <math id="S3.p1.4.m3.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S3.p1.4.m3.1a"><msub id="S3.p1.4.m3.1.1" xref="S3.p1.4.m3.1.1.cmml"><mi id="S3.p1.4.m3.1.1.2" xref="S3.p1.4.m3.1.1.2.cmml">x</mi><mi id="S3.p1.4.m3.1.1.3" xref="S3.p1.4.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.p1.4.m3.1b"><apply id="S3.p1.4.m3.1.1.cmml" xref="S3.p1.4.m3.1.1"><csymbol cd="ambiguous" id="S3.p1.4.m3.1.1.1.cmml" xref="S3.p1.4.m3.1.1">subscript</csymbol><ci id="S3.p1.4.m3.1.1.2.cmml" xref="S3.p1.4.m3.1.1.2">𝑥</ci><ci id="S3.p1.4.m3.1.1.3.cmml" xref="S3.p1.4.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.4.m3.1c">x_{i}</annotation></semantics></math> and one-hot class labels <math id="S3.p1.5.m4.1" class="ltx_Math" alttext="y_{i}" display="inline"><semantics id="S3.p1.5.m4.1a"><msub id="S3.p1.5.m4.1.1" xref="S3.p1.5.m4.1.1.cmml"><mi id="S3.p1.5.m4.1.1.2" xref="S3.p1.5.m4.1.1.2.cmml">y</mi><mi id="S3.p1.5.m4.1.1.3" xref="S3.p1.5.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.p1.5.m4.1b"><apply id="S3.p1.5.m4.1.1.cmml" xref="S3.p1.5.m4.1.1"><csymbol cd="ambiguous" id="S3.p1.5.m4.1.1.1.cmml" xref="S3.p1.5.m4.1.1">subscript</csymbol><ci id="S3.p1.5.m4.1.1.2.cmml" xref="S3.p1.5.m4.1.1.2">𝑦</ci><ci id="S3.p1.5.m4.1.1.3.cmml" xref="S3.p1.5.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.5.m4.1c">y_{i}</annotation></semantics></math> to finetune and improve a pretrained vision model’s performance on the downstream task. We compare two high-level approaches for sourcing this targeted data, shown in Figure <a href="#S1.F1.fig1" title="Figure 1 ‣ 1 Introduction ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>: (1) we <span id="S3.p1.9.2" class="ltx_text ltx_font_bold">retrieve targeted real images</span> directly from <math id="S3.p1.6.m5.1" class="ltx_Math" alttext="{\mathcal{D}}" display="inline"><semantics id="S3.p1.6.m5.1a"><mi class="ltx_font_mathcaligraphic" id="S3.p1.6.m5.1.1" xref="S3.p1.6.m5.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S3.p1.6.m5.1b"><ci id="S3.p1.6.m5.1.1.cmml" xref="S3.p1.6.m5.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.6.m5.1c">{\mathcal{D}}</annotation></semantics></math>, forming a targeted retrieved dataset <math id="S3.p1.7.m6.1" class="ltx_Math" alttext="{\mathcal{D}_{\mathcal{C}}^{(\text{retrieved})}}\subset{\mathcal{D}}" display="inline"><semantics id="S3.p1.7.m6.1a"><mrow id="S3.p1.7.m6.1.2" xref="S3.p1.7.m6.1.2.cmml"><msubsup id="S3.p1.7.m6.1.2.2" xref="S3.p1.7.m6.1.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.p1.7.m6.1.2.2.2.2" xref="S3.p1.7.m6.1.2.2.2.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S3.p1.7.m6.1.2.2.2.3" xref="S3.p1.7.m6.1.2.2.2.3.cmml">𝒞</mi><mrow id="S3.p1.7.m6.1.1.1.3" xref="S3.p1.7.m6.1.1.1.1a.cmml"><mo stretchy="false" id="S3.p1.7.m6.1.1.1.3.1" xref="S3.p1.7.m6.1.1.1.1a.cmml">(</mo><mtext id="S3.p1.7.m6.1.1.1.1" xref="S3.p1.7.m6.1.1.1.1.cmml">retrieved</mtext><mo stretchy="false" id="S3.p1.7.m6.1.1.1.3.2" xref="S3.p1.7.m6.1.1.1.1a.cmml">)</mo></mrow></msubsup><mo id="S3.p1.7.m6.1.2.1" xref="S3.p1.7.m6.1.2.1.cmml">⊂</mo><mi class="ltx_font_mathcaligraphic" id="S3.p1.7.m6.1.2.3" xref="S3.p1.7.m6.1.2.3.cmml">𝒟</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.p1.7.m6.1b"><apply id="S3.p1.7.m6.1.2.cmml" xref="S3.p1.7.m6.1.2"><subset id="S3.p1.7.m6.1.2.1.cmml" xref="S3.p1.7.m6.1.2.1"></subset><apply id="S3.p1.7.m6.1.2.2.cmml" xref="S3.p1.7.m6.1.2.2"><csymbol cd="ambiguous" id="S3.p1.7.m6.1.2.2.1.cmml" xref="S3.p1.7.m6.1.2.2">superscript</csymbol><apply id="S3.p1.7.m6.1.2.2.2.cmml" xref="S3.p1.7.m6.1.2.2"><csymbol cd="ambiguous" id="S3.p1.7.m6.1.2.2.2.1.cmml" xref="S3.p1.7.m6.1.2.2">subscript</csymbol><ci id="S3.p1.7.m6.1.2.2.2.2.cmml" xref="S3.p1.7.m6.1.2.2.2.2">𝒟</ci><ci id="S3.p1.7.m6.1.2.2.2.3.cmml" xref="S3.p1.7.m6.1.2.2.2.3">𝒞</ci></apply><ci id="S3.p1.7.m6.1.1.1.1a.cmml" xref="S3.p1.7.m6.1.1.1.3"><mtext mathsize="70%" id="S3.p1.7.m6.1.1.1.1.cmml" xref="S3.p1.7.m6.1.1.1.1">retrieved</mtext></ci></apply><ci id="S3.p1.7.m6.1.2.3.cmml" xref="S3.p1.7.m6.1.2.3">𝒟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.7.m6.1c">{\mathcal{D}_{\mathcal{C}}^{(\text{retrieved})}}\subset{\mathcal{D}}</annotation></semantics></math>.
Alternatively, (2) we <span id="S3.p1.9.3" class="ltx_text ltx_font_bold">generate targeted synthetic images</span> by prompting an intermediate text-to-image generative model trained on <math id="S3.p1.8.m7.1" class="ltx_Math" alttext="{\mathcal{D}}" display="inline"><semantics id="S3.p1.8.m7.1a"><mi class="ltx_font_mathcaligraphic" id="S3.p1.8.m7.1.1" xref="S3.p1.8.m7.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S3.p1.8.m7.1b"><ci id="S3.p1.8.m7.1.1.cmml" xref="S3.p1.8.m7.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.8.m7.1c">{\mathcal{D}}</annotation></semantics></math>, forming a targeted synthetic dataset <math id="S3.p1.9.m8.1" class="ltx_Math" alttext="{\mathcal{D}_{\mathcal{C}}^{(\text{synthetic})}}" display="inline"><semantics id="S3.p1.9.m8.1a"><msubsup id="S3.p1.9.m8.1.2" xref="S3.p1.9.m8.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.p1.9.m8.1.2.2.2" xref="S3.p1.9.m8.1.2.2.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S3.p1.9.m8.1.2.2.3" xref="S3.p1.9.m8.1.2.2.3.cmml">𝒞</mi><mrow id="S3.p1.9.m8.1.1.1.3" xref="S3.p1.9.m8.1.1.1.1a.cmml"><mo stretchy="false" id="S3.p1.9.m8.1.1.1.3.1" xref="S3.p1.9.m8.1.1.1.1a.cmml">(</mo><mtext id="S3.p1.9.m8.1.1.1.1" xref="S3.p1.9.m8.1.1.1.1.cmml">synthetic</mtext><mo stretchy="false" id="S3.p1.9.m8.1.1.1.3.2" xref="S3.p1.9.m8.1.1.1.1a.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S3.p1.9.m8.1b"><apply id="S3.p1.9.m8.1.2.cmml" xref="S3.p1.9.m8.1.2"><csymbol cd="ambiguous" id="S3.p1.9.m8.1.2.1.cmml" xref="S3.p1.9.m8.1.2">superscript</csymbol><apply id="S3.p1.9.m8.1.2.2.cmml" xref="S3.p1.9.m8.1.2"><csymbol cd="ambiguous" id="S3.p1.9.m8.1.2.2.1.cmml" xref="S3.p1.9.m8.1.2">subscript</csymbol><ci id="S3.p1.9.m8.1.2.2.2.cmml" xref="S3.p1.9.m8.1.2.2.2">𝒟</ci><ci id="S3.p1.9.m8.1.2.2.3.cmml" xref="S3.p1.9.m8.1.2.2.3">𝒞</ci></apply><ci id="S3.p1.9.m8.1.1.1.1a.cmml" xref="S3.p1.9.m8.1.1.1.3"><mtext mathsize="70%" id="S3.p1.9.m8.1.1.1.1.cmml" xref="S3.p1.9.m8.1.1.1.1">synthetic</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p1.9.m8.1c">{\mathcal{D}_{\mathcal{C}}^{(\text{synthetic})}}</annotation></semantics></math>. We detail each approach below.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Sourcing data by generating synthetic images</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.9" class="ltx_p">We follow SynCLR <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite>, a representative method for curating synthetic training data from off-the-shelf text-to-image models.
Given the set of visual class names <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="{\mathcal{C}}" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml">𝒞</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><ci id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">𝒞</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">{\mathcal{C}}</annotation></semantics></math>, we first synthesize a large corpus of corresponding image captions by prompting a large language model (details in Appendix <a href="#A2.SS1" title="B.1 Sourcing data by generating synthetic images ‣ Appendix B Details in Methodology ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B.1</span></a>). For example, if the class name <math id="S3.SS1.p1.2.m2.1" class="ltx_Math" alttext="c\in{\mathcal{C}}" display="inline"><semantics id="S3.SS1.p1.2.m2.1a"><mrow id="S3.SS1.p1.2.m2.1.1" xref="S3.SS1.p1.2.m2.1.1.cmml"><mi id="S3.SS1.p1.2.m2.1.1.2" xref="S3.SS1.p1.2.m2.1.1.2.cmml">c</mi><mo id="S3.SS1.p1.2.m2.1.1.1" xref="S3.SS1.p1.2.m2.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.2.m2.1.1.3" xref="S3.SS1.p1.2.m2.1.1.3.cmml">𝒞</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m2.1b"><apply id="S3.SS1.p1.2.m2.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1"><in id="S3.SS1.p1.2.m2.1.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1.1"></in><ci id="S3.SS1.p1.2.m2.1.1.2.cmml" xref="S3.SS1.p1.2.m2.1.1.2">𝑐</ci><ci id="S3.SS1.p1.2.m2.1.1.3.cmml" xref="S3.SS1.p1.2.m2.1.1.3">𝒞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m2.1c">c\in{\mathcal{C}}</annotation></semantics></math> is “rose,” then a generated caption might be “a close-up of a pink rose in bloom.”
We then use these captions as input for a text-to-image generator <math id="S3.SS1.p1.3.m3.1" class="ltx_Math" alttext="G" display="inline"><semantics id="S3.SS1.p1.3.m3.1a"><mi id="S3.SS1.p1.3.m3.1.1" xref="S3.SS1.p1.3.m3.1.1.cmml">G</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.3.m3.1b"><ci id="S3.SS1.p1.3.m3.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1">𝐺</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.3.m3.1c">G</annotation></semantics></math> trained on the upstream data <math id="S3.SS1.p1.4.m4.1" class="ltx_Math" alttext="{\mathcal{D}}" display="inline"><semantics id="S3.SS1.p1.4.m4.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.4.m4.1.1" xref="S3.SS1.p1.4.m4.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.4.m4.1b"><ci id="S3.SS1.p1.4.m4.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.4.m4.1c">{\mathcal{D}}</annotation></semantics></math>, yielding a large set of synthesized images <math id="S3.SS1.p1.5.m5.1" class="ltx_Math" alttext="\widetilde{x_{i}}" display="inline"><semantics id="S3.SS1.p1.5.m5.1a"><mover accent="true" id="S3.SS1.p1.5.m5.1.1" xref="S3.SS1.p1.5.m5.1.1.cmml"><msub id="S3.SS1.p1.5.m5.1.1.2" xref="S3.SS1.p1.5.m5.1.1.2.cmml"><mi id="S3.SS1.p1.5.m5.1.1.2.2" xref="S3.SS1.p1.5.m5.1.1.2.2.cmml">x</mi><mi id="S3.SS1.p1.5.m5.1.1.2.3" xref="S3.SS1.p1.5.m5.1.1.2.3.cmml">i</mi></msub><mo id="S3.SS1.p1.5.m5.1.1.1" xref="S3.SS1.p1.5.m5.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.5.m5.1b"><apply id="S3.SS1.p1.5.m5.1.1.cmml" xref="S3.SS1.p1.5.m5.1.1"><ci id="S3.SS1.p1.5.m5.1.1.1.cmml" xref="S3.SS1.p1.5.m5.1.1.1">~</ci><apply id="S3.SS1.p1.5.m5.1.1.2.cmml" xref="S3.SS1.p1.5.m5.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p1.5.m5.1.1.2.1.cmml" xref="S3.SS1.p1.5.m5.1.1.2">subscript</csymbol><ci id="S3.SS1.p1.5.m5.1.1.2.2.cmml" xref="S3.SS1.p1.5.m5.1.1.2.2">𝑥</ci><ci id="S3.SS1.p1.5.m5.1.1.2.3.cmml" xref="S3.SS1.p1.5.m5.1.1.2.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.5.m5.1c">\widetilde{x_{i}}</annotation></semantics></math>. Each image is assigned a class label <math id="S3.SS1.p1.6.m6.1" class="ltx_Math" alttext="y_{i}" display="inline"><semantics id="S3.SS1.p1.6.m6.1a"><msub id="S3.SS1.p1.6.m6.1.1" xref="S3.SS1.p1.6.m6.1.1.cmml"><mi id="S3.SS1.p1.6.m6.1.1.2" xref="S3.SS1.p1.6.m6.1.1.2.cmml">y</mi><mi id="S3.SS1.p1.6.m6.1.1.3" xref="S3.SS1.p1.6.m6.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.6.m6.1b"><apply id="S3.SS1.p1.6.m6.1.1.cmml" xref="S3.SS1.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.6.m6.1.1.1.cmml" xref="S3.SS1.p1.6.m6.1.1">subscript</csymbol><ci id="S3.SS1.p1.6.m6.1.1.2.cmml" xref="S3.SS1.p1.6.m6.1.1.2">𝑦</ci><ci id="S3.SS1.p1.6.m6.1.1.3.cmml" xref="S3.SS1.p1.6.m6.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.6.m6.1c">y_{i}</annotation></semantics></math> based on the class name <math id="S3.SS1.p1.7.m7.1" class="ltx_Math" alttext="c" display="inline"><semantics id="S3.SS1.p1.7.m7.1a"><mi id="S3.SS1.p1.7.m7.1.1" xref="S3.SS1.p1.7.m7.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.7.m7.1b"><ci id="S3.SS1.p1.7.m7.1.1.cmml" xref="S3.SS1.p1.7.m7.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.7.m7.1c">c</annotation></semantics></math> used to synthesize its caption. These synthetic images and labels <math id="S3.SS1.p1.8.m8.2" class="ltx_Math" alttext="\{(\widetilde{x_{i}},y_{i})\}" display="inline"><semantics id="S3.SS1.p1.8.m8.2a"><mrow id="S3.SS1.p1.8.m8.2.2.1" xref="S3.SS1.p1.8.m8.2.2.2.cmml"><mo stretchy="false" id="S3.SS1.p1.8.m8.2.2.1.2" xref="S3.SS1.p1.8.m8.2.2.2.cmml">{</mo><mrow id="S3.SS1.p1.8.m8.2.2.1.1.1" xref="S3.SS1.p1.8.m8.2.2.1.1.2.cmml"><mo stretchy="false" id="S3.SS1.p1.8.m8.2.2.1.1.1.2" xref="S3.SS1.p1.8.m8.2.2.1.1.2.cmml">(</mo><mover accent="true" id="S3.SS1.p1.8.m8.1.1" xref="S3.SS1.p1.8.m8.1.1.cmml"><msub id="S3.SS1.p1.8.m8.1.1.2" xref="S3.SS1.p1.8.m8.1.1.2.cmml"><mi id="S3.SS1.p1.8.m8.1.1.2.2" xref="S3.SS1.p1.8.m8.1.1.2.2.cmml">x</mi><mi id="S3.SS1.p1.8.m8.1.1.2.3" xref="S3.SS1.p1.8.m8.1.1.2.3.cmml">i</mi></msub><mo id="S3.SS1.p1.8.m8.1.1.1" xref="S3.SS1.p1.8.m8.1.1.1.cmml">~</mo></mover><mo id="S3.SS1.p1.8.m8.2.2.1.1.1.3" xref="S3.SS1.p1.8.m8.2.2.1.1.2.cmml">,</mo><msub id="S3.SS1.p1.8.m8.2.2.1.1.1.1" xref="S3.SS1.p1.8.m8.2.2.1.1.1.1.cmml"><mi id="S3.SS1.p1.8.m8.2.2.1.1.1.1.2" xref="S3.SS1.p1.8.m8.2.2.1.1.1.1.2.cmml">y</mi><mi id="S3.SS1.p1.8.m8.2.2.1.1.1.1.3" xref="S3.SS1.p1.8.m8.2.2.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS1.p1.8.m8.2.2.1.1.1.4" xref="S3.SS1.p1.8.m8.2.2.1.1.2.cmml">)</mo></mrow><mo stretchy="false" id="S3.SS1.p1.8.m8.2.2.1.3" xref="S3.SS1.p1.8.m8.2.2.2.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.8.m8.2b"><set id="S3.SS1.p1.8.m8.2.2.2.cmml" xref="S3.SS1.p1.8.m8.2.2.1"><interval closure="open" id="S3.SS1.p1.8.m8.2.2.1.1.2.cmml" xref="S3.SS1.p1.8.m8.2.2.1.1.1"><apply id="S3.SS1.p1.8.m8.1.1.cmml" xref="S3.SS1.p1.8.m8.1.1"><ci id="S3.SS1.p1.8.m8.1.1.1.cmml" xref="S3.SS1.p1.8.m8.1.1.1">~</ci><apply id="S3.SS1.p1.8.m8.1.1.2.cmml" xref="S3.SS1.p1.8.m8.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p1.8.m8.1.1.2.1.cmml" xref="S3.SS1.p1.8.m8.1.1.2">subscript</csymbol><ci id="S3.SS1.p1.8.m8.1.1.2.2.cmml" xref="S3.SS1.p1.8.m8.1.1.2.2">𝑥</ci><ci id="S3.SS1.p1.8.m8.1.1.2.3.cmml" xref="S3.SS1.p1.8.m8.1.1.2.3">𝑖</ci></apply></apply><apply id="S3.SS1.p1.8.m8.2.2.1.1.1.1.cmml" xref="S3.SS1.p1.8.m8.2.2.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.8.m8.2.2.1.1.1.1.1.cmml" xref="S3.SS1.p1.8.m8.2.2.1.1.1.1">subscript</csymbol><ci id="S3.SS1.p1.8.m8.2.2.1.1.1.1.2.cmml" xref="S3.SS1.p1.8.m8.2.2.1.1.1.1.2">𝑦</ci><ci id="S3.SS1.p1.8.m8.2.2.1.1.1.1.3.cmml" xref="S3.SS1.p1.8.m8.2.2.1.1.1.1.3">𝑖</ci></apply></interval></set></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.8.m8.2c">\{(\widetilde{x_{i}},y_{i})\}</annotation></semantics></math> form our curated dataset <math id="S3.SS1.p1.9.m9.1" class="ltx_Math" alttext="{\mathcal{D}_{\mathcal{C}}^{(\text{synthetic})}}" display="inline"><semantics id="S3.SS1.p1.9.m9.1a"><msubsup id="S3.SS1.p1.9.m9.1.2" xref="S3.SS1.p1.9.m9.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.9.m9.1.2.2.2" xref="S3.SS1.p1.9.m9.1.2.2.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.9.m9.1.2.2.3" xref="S3.SS1.p1.9.m9.1.2.2.3.cmml">𝒞</mi><mrow id="S3.SS1.p1.9.m9.1.1.1.3" xref="S3.SS1.p1.9.m9.1.1.1.1a.cmml"><mo stretchy="false" id="S3.SS1.p1.9.m9.1.1.1.3.1" xref="S3.SS1.p1.9.m9.1.1.1.1a.cmml">(</mo><mtext id="S3.SS1.p1.9.m9.1.1.1.1" xref="S3.SS1.p1.9.m9.1.1.1.1.cmml">synthetic</mtext><mo stretchy="false" id="S3.SS1.p1.9.m9.1.1.1.3.2" xref="S3.SS1.p1.9.m9.1.1.1.1a.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.9.m9.1b"><apply id="S3.SS1.p1.9.m9.1.2.cmml" xref="S3.SS1.p1.9.m9.1.2"><csymbol cd="ambiguous" id="S3.SS1.p1.9.m9.1.2.1.cmml" xref="S3.SS1.p1.9.m9.1.2">superscript</csymbol><apply id="S3.SS1.p1.9.m9.1.2.2.cmml" xref="S3.SS1.p1.9.m9.1.2"><csymbol cd="ambiguous" id="S3.SS1.p1.9.m9.1.2.2.1.cmml" xref="S3.SS1.p1.9.m9.1.2">subscript</csymbol><ci id="S3.SS1.p1.9.m9.1.2.2.2.cmml" xref="S3.SS1.p1.9.m9.1.2.2.2">𝒟</ci><ci id="S3.SS1.p1.9.m9.1.2.2.3.cmml" xref="S3.SS1.p1.9.m9.1.2.2.3">𝒞</ci></apply><ci id="S3.SS1.p1.9.m9.1.1.1.1a.cmml" xref="S3.SS1.p1.9.m9.1.1.1.3"><mtext mathsize="70%" id="S3.SS1.p1.9.m9.1.1.1.1.cmml" xref="S3.SS1.p1.9.m9.1.1.1.1">synthetic</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.9.m9.1c">{\mathcal{D}_{\mathcal{C}}^{(\text{synthetic})}}</annotation></semantics></math>.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Sourcing data by retrieving real images</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.4" class="ltx_p">Rather than querying a generator trained on an upstream dataset <math id="S3.SS2.p1.1.m1.1" class="ltx_Math" alttext="{\mathcal{D}}" display="inline"><semantics id="S3.SS2.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.1b"><ci id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.1c">{\mathcal{D}}</annotation></semantics></math>, we can directly train on parts of <math id="S3.SS2.p1.2.m2.1" class="ltx_Math" alttext="{\mathcal{D}}" display="inline"><semantics id="S3.SS2.p1.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.2.m2.1.1" xref="S3.SS2.p1.2.m2.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.2.m2.1b"><ci id="S3.SS2.p1.2.m2.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.2.m2.1c">{\mathcal{D}}</annotation></semantics></math> itself by retrieving relevant data. <math id="S3.SS2.p1.3.m3.1" class="ltx_Math" alttext="{\mathcal{D}}" display="inline"><semantics id="S3.SS2.p1.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.3.m3.1.1" xref="S3.SS2.p1.3.m3.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.3.m3.1b"><ci id="S3.SS2.p1.3.m3.1.1.cmml" xref="S3.SS2.p1.3.m3.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.3.m3.1c">{\mathcal{D}}</annotation></semantics></math> consists of image-text pairs <math id="S3.SS2.p1.4.m4.2" class="ltx_Math" alttext="(x_{i},t_{i})" display="inline"><semantics id="S3.SS2.p1.4.m4.2a"><mrow id="S3.SS2.p1.4.m4.2.2.2" xref="S3.SS2.p1.4.m4.2.2.3.cmml"><mo stretchy="false" id="S3.SS2.p1.4.m4.2.2.2.3" xref="S3.SS2.p1.4.m4.2.2.3.cmml">(</mo><msub id="S3.SS2.p1.4.m4.1.1.1.1" xref="S3.SS2.p1.4.m4.1.1.1.1.cmml"><mi id="S3.SS2.p1.4.m4.1.1.1.1.2" xref="S3.SS2.p1.4.m4.1.1.1.1.2.cmml">x</mi><mi id="S3.SS2.p1.4.m4.1.1.1.1.3" xref="S3.SS2.p1.4.m4.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.SS2.p1.4.m4.2.2.2.4" xref="S3.SS2.p1.4.m4.2.2.3.cmml">,</mo><msub id="S3.SS2.p1.4.m4.2.2.2.2" xref="S3.SS2.p1.4.m4.2.2.2.2.cmml"><mi id="S3.SS2.p1.4.m4.2.2.2.2.2" xref="S3.SS2.p1.4.m4.2.2.2.2.2.cmml">t</mi><mi id="S3.SS2.p1.4.m4.2.2.2.2.3" xref="S3.SS2.p1.4.m4.2.2.2.2.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS2.p1.4.m4.2.2.2.5" xref="S3.SS2.p1.4.m4.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.4.m4.2b"><interval closure="open" id="S3.SS2.p1.4.m4.2.2.3.cmml" xref="S3.SS2.p1.4.m4.2.2.2"><apply id="S3.SS2.p1.4.m4.1.1.1.1.cmml" xref="S3.SS2.p1.4.m4.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.4.m4.1.1.1.1.1.cmml" xref="S3.SS2.p1.4.m4.1.1.1.1">subscript</csymbol><ci id="S3.SS2.p1.4.m4.1.1.1.1.2.cmml" xref="S3.SS2.p1.4.m4.1.1.1.1.2">𝑥</ci><ci id="S3.SS2.p1.4.m4.1.1.1.1.3.cmml" xref="S3.SS2.p1.4.m4.1.1.1.1.3">𝑖</ci></apply><apply id="S3.SS2.p1.4.m4.2.2.2.2.cmml" xref="S3.SS2.p1.4.m4.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.p1.4.m4.2.2.2.2.1.cmml" xref="S3.SS2.p1.4.m4.2.2.2.2">subscript</csymbol><ci id="S3.SS2.p1.4.m4.2.2.2.2.2.cmml" xref="S3.SS2.p1.4.m4.2.2.2.2.2">𝑡</ci><ci id="S3.SS2.p1.4.m4.2.2.2.2.3.cmml" xref="S3.SS2.p1.4.m4.2.2.2.2.3">𝑖</ci></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.4.m4.2c">(x_{i},t_{i})</annotation></semantics></math>. To retrieve relevant pairs, we consider two strategies. We additionally deduplicate all retrieved images with respect to our evaluation datasets following <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib20" title="" class="ltx_ref">20</a>]</cite> to minimize test set leakage. We apply NSFW filtering <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib55" title="" class="ltx_ref">55</a>]</cite>.</p>
</div>
<section id="S3.SS2.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Strategy 1: hard substring matching.</h4>

<div id="S3.SS2.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS2.SSS0.Px1.p1.4" class="ltx_p">Inspired by <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib65" title="" class="ltx_ref">65</a>]</cite>, we retrieve the set <math id="S3.SS2.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="{\mathcal{D}_{\mathcal{C}}^{(\text{retrieved})}}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.1.m1.1a"><msubsup id="S3.SS2.SSS0.Px1.p1.1.m1.1.2" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.SSS0.Px1.p1.1.m1.1.2.2.2" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.2.2.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS2.SSS0.Px1.p1.1.m1.1.2.2.3" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.2.2.3.cmml">𝒞</mi><mrow id="S3.SS2.SSS0.Px1.p1.1.m1.1.1.1.3" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.1.1.1a.cmml"><mo stretchy="false" id="S3.SS2.SSS0.Px1.p1.1.m1.1.1.1.3.1" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.1.1.1a.cmml">(</mo><mtext id="S3.SS2.SSS0.Px1.p1.1.m1.1.1.1.1" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.1.1.1.cmml">retrieved</mtext><mo stretchy="false" id="S3.SS2.SSS0.Px1.p1.1.m1.1.1.1.3.2" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.1.1.1a.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.1.m1.1b"><apply id="S3.SS2.SSS0.Px1.p1.1.m1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.1.m1.1.2.1.cmml" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.2">superscript</csymbol><apply id="S3.SS2.SSS0.Px1.p1.1.m1.1.2.2.cmml" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.1.m1.1.2.2.1.cmml" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.1.m1.1.2.2.2.cmml" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.2.2.2">𝒟</ci><ci id="S3.SS2.SSS0.Px1.p1.1.m1.1.2.2.3.cmml" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.2.2.3">𝒞</ci></apply><ci id="S3.SS2.SSS0.Px1.p1.1.m1.1.1.1.1a.cmml" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.1.1.3"><mtext mathsize="70%" id="S3.SS2.SSS0.Px1.p1.1.m1.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.1.m1.1.1.1.1">retrieved</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.1.m1.1c">{\mathcal{D}_{\mathcal{C}}^{(\text{retrieved})}}</annotation></semantics></math> of all images <math id="S3.SS2.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.2.m2.1a"><msub id="S3.SS2.SSS0.Px1.p1.2.m2.1.1" xref="S3.SS2.SSS0.Px1.p1.2.m2.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p1.2.m2.1.1.2" xref="S3.SS2.SSS0.Px1.p1.2.m2.1.1.2.cmml">x</mi><mi id="S3.SS2.SSS0.Px1.p1.2.m2.1.1.3" xref="S3.SS2.SSS0.Px1.p1.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.2.m2.1b"><apply id="S3.SS2.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.2.m2.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.2.m2.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.1.1.2">𝑥</ci><ci id="S3.SS2.SSS0.Px1.p1.2.m2.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p1.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.2.m2.1c">x_{i}</annotation></semantics></math> whose corresponding caption <math id="S3.SS2.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="t_{i}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.3.m3.1a"><msub id="S3.SS2.SSS0.Px1.p1.3.m3.1.1" xref="S3.SS2.SSS0.Px1.p1.3.m3.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p1.3.m3.1.1.2" xref="S3.SS2.SSS0.Px1.p1.3.m3.1.1.2.cmml">t</mi><mi id="S3.SS2.SSS0.Px1.p1.3.m3.1.1.3" xref="S3.SS2.SSS0.Px1.p1.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.3.m3.1b"><apply id="S3.SS2.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.3.m3.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.3.m3.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.3.m3.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.3.m3.1.1.2">𝑡</ci><ci id="S3.SS2.SSS0.Px1.p1.3.m3.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p1.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.3.m3.1c">t_{i}</annotation></semantics></math> contains at least one target class name <math id="S3.SS2.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="c\in{\mathcal{C}}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.4.m4.1a"><mrow id="S3.SS2.SSS0.Px1.p1.4.m4.1.1" xref="S3.SS2.SSS0.Px1.p1.4.m4.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p1.4.m4.1.1.2" xref="S3.SS2.SSS0.Px1.p1.4.m4.1.1.2.cmml">c</mi><mo id="S3.SS2.SSS0.Px1.p1.4.m4.1.1.1" xref="S3.SS2.SSS0.Px1.p1.4.m4.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS2.SSS0.Px1.p1.4.m4.1.1.3" xref="S3.SS2.SSS0.Px1.p1.4.m4.1.1.3.cmml">𝒞</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.4.m4.1b"><apply id="S3.SS2.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.4.m4.1.1"><in id="S3.SS2.SSS0.Px1.p1.4.m4.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.4.m4.1.1.1"></in><ci id="S3.SS2.SSS0.Px1.p1.4.m4.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.4.m4.1.1.2">𝑐</ci><ci id="S3.SS2.SSS0.Px1.p1.4.m4.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p1.4.m4.1.1.3">𝒞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.4.m4.1c">c\in{\mathcal{C}}</annotation></semantics></math> as a substring:</p>
<table id="A5.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.Ex1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.Ex1.m1.4" class="ltx_Math" alttext="\displaystyle{\mathcal{D}_{\mathcal{C}}^{(\text{retrieved})}}=\left\{(x_{i},y_{i})\,\,\colon\,(x_{i},t_{i})\in{\mathcal{D}}\textrm{ such that some class $c\in{\mathcal{C}}$ is a substring of $t_{i}$}\right\}." display="inline"><semantics id="S3.Ex1.m1.4a"><mrow id="S3.Ex1.m1.4.4.1" xref="S3.Ex1.m1.4.4.1.1.cmml"><mrow id="S3.Ex1.m1.4.4.1.1" xref="S3.Ex1.m1.4.4.1.1.cmml"><msubsup id="S3.Ex1.m1.4.4.1.1.4" xref="S3.Ex1.m1.4.4.1.1.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.Ex1.m1.4.4.1.1.4.2.2" xref="S3.Ex1.m1.4.4.1.1.4.2.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S3.Ex1.m1.4.4.1.1.4.2.3" xref="S3.Ex1.m1.4.4.1.1.4.2.3.cmml">𝒞</mi><mrow id="S3.Ex1.m1.3.3.1.3" xref="S3.Ex1.m1.3.3.1.1a.cmml"><mo stretchy="false" id="S3.Ex1.m1.3.3.1.3.1" xref="S3.Ex1.m1.3.3.1.1a.cmml">(</mo><mtext id="S3.Ex1.m1.3.3.1.1" xref="S3.Ex1.m1.3.3.1.1.cmml">retrieved</mtext><mo stretchy="false" id="S3.Ex1.m1.3.3.1.3.2" xref="S3.Ex1.m1.3.3.1.1a.cmml">)</mo></mrow></msubsup><mo id="S3.Ex1.m1.4.4.1.1.3" xref="S3.Ex1.m1.4.4.1.1.3.cmml">=</mo><mrow id="S3.Ex1.m1.4.4.1.1.2.2" xref="S3.Ex1.m1.4.4.1.1.2.3.cmml"><mo id="S3.Ex1.m1.4.4.1.1.2.2.3" xref="S3.Ex1.m1.4.4.1.1.2.3.1.cmml">{</mo><mrow id="S3.Ex1.m1.4.4.1.1.1.1.1.2" xref="S3.Ex1.m1.4.4.1.1.1.1.1.3.cmml"><mo stretchy="false" id="S3.Ex1.m1.4.4.1.1.1.1.1.2.3" xref="S3.Ex1.m1.4.4.1.1.1.1.1.3.cmml">(</mo><msub id="S3.Ex1.m1.4.4.1.1.1.1.1.1.1" xref="S3.Ex1.m1.4.4.1.1.1.1.1.1.1.cmml"><mi id="S3.Ex1.m1.4.4.1.1.1.1.1.1.1.2" xref="S3.Ex1.m1.4.4.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S3.Ex1.m1.4.4.1.1.1.1.1.1.1.3" xref="S3.Ex1.m1.4.4.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.Ex1.m1.4.4.1.1.1.1.1.2.4" xref="S3.Ex1.m1.4.4.1.1.1.1.1.3.cmml">,</mo><msub id="S3.Ex1.m1.4.4.1.1.1.1.1.2.2" xref="S3.Ex1.m1.4.4.1.1.1.1.1.2.2.cmml"><mi id="S3.Ex1.m1.4.4.1.1.1.1.1.2.2.2" xref="S3.Ex1.m1.4.4.1.1.1.1.1.2.2.2.cmml">y</mi><mi id="S3.Ex1.m1.4.4.1.1.1.1.1.2.2.3" xref="S3.Ex1.m1.4.4.1.1.1.1.1.2.2.3.cmml">i</mi></msub><mo rspace="0.608em" stretchy="false" id="S3.Ex1.m1.4.4.1.1.1.1.1.2.5" xref="S3.Ex1.m1.4.4.1.1.1.1.1.3.cmml">)</mo></mrow><mo rspace="0.448em" id="S3.Ex1.m1.4.4.1.1.2.2.4" xref="S3.Ex1.m1.4.4.1.1.2.3.1.cmml">:</mo><mrow id="S3.Ex1.m1.4.4.1.1.2.2.2" xref="S3.Ex1.m1.4.4.1.1.2.2.2.cmml"><mrow id="S3.Ex1.m1.4.4.1.1.2.2.2.2.2" xref="S3.Ex1.m1.4.4.1.1.2.2.2.2.3.cmml"><mo stretchy="false" id="S3.Ex1.m1.4.4.1.1.2.2.2.2.2.3" xref="S3.Ex1.m1.4.4.1.1.2.2.2.2.3.cmml">(</mo><msub id="S3.Ex1.m1.4.4.1.1.2.2.2.1.1.1" xref="S3.Ex1.m1.4.4.1.1.2.2.2.1.1.1.cmml"><mi id="S3.Ex1.m1.4.4.1.1.2.2.2.1.1.1.2" xref="S3.Ex1.m1.4.4.1.1.2.2.2.1.1.1.2.cmml">x</mi><mi id="S3.Ex1.m1.4.4.1.1.2.2.2.1.1.1.3" xref="S3.Ex1.m1.4.4.1.1.2.2.2.1.1.1.3.cmml">i</mi></msub><mo id="S3.Ex1.m1.4.4.1.1.2.2.2.2.2.4" xref="S3.Ex1.m1.4.4.1.1.2.2.2.2.3.cmml">,</mo><msub id="S3.Ex1.m1.4.4.1.1.2.2.2.2.2.2" xref="S3.Ex1.m1.4.4.1.1.2.2.2.2.2.2.cmml"><mi id="S3.Ex1.m1.4.4.1.1.2.2.2.2.2.2.2" xref="S3.Ex1.m1.4.4.1.1.2.2.2.2.2.2.2.cmml">t</mi><mi id="S3.Ex1.m1.4.4.1.1.2.2.2.2.2.2.3" xref="S3.Ex1.m1.4.4.1.1.2.2.2.2.2.2.3.cmml">i</mi></msub><mo stretchy="false" id="S3.Ex1.m1.4.4.1.1.2.2.2.2.2.5" xref="S3.Ex1.m1.4.4.1.1.2.2.2.2.3.cmml">)</mo></mrow><mo id="S3.Ex1.m1.4.4.1.1.2.2.2.3" xref="S3.Ex1.m1.4.4.1.1.2.2.2.3.cmml">∈</mo><mrow id="S3.Ex1.m1.4.4.1.1.2.2.2.4" xref="S3.Ex1.m1.4.4.1.1.2.2.2.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.Ex1.m1.4.4.1.1.2.2.2.4.2" xref="S3.Ex1.m1.4.4.1.1.2.2.2.4.2.cmml">𝒟</mi><mo lspace="0em" rspace="0em" id="S3.Ex1.m1.4.4.1.1.2.2.2.4.1" xref="S3.Ex1.m1.4.4.1.1.2.2.2.4.1.cmml">​</mo><mrow id="S3.Ex1.m1.2.2.2a" xref="S3.Ex1.m1.2.2.2ac.cmml"><mtext id="S3.Ex1.m1.2.2.2aa" xref="S3.Ex1.m1.2.2.2ac.cmml"> such that some class </mtext><mrow id="S3.Ex1.m1.1.1.1.m1.1.1" xref="S3.Ex1.m1.1.1.1.m1.1.1.cmml"><mi id="S3.Ex1.m1.1.1.1.m1.1.1.2" xref="S3.Ex1.m1.1.1.1.m1.1.1.2.cmml">c</mi><mo id="S3.Ex1.m1.1.1.1.m1.1.1.1" xref="S3.Ex1.m1.1.1.1.m1.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.Ex1.m1.1.1.1.m1.1.1.3" xref="S3.Ex1.m1.1.1.1.m1.1.1.3.cmml">𝒞</mi></mrow><mtext id="S3.Ex1.m1.2.2.2ab" xref="S3.Ex1.m1.2.2.2ac.cmml"> is a substring of </mtext><msub id="S3.Ex1.m1.2.2.2.m2.1.1" xref="S3.Ex1.m1.2.2.2.m2.1.1.cmml"><mi id="S3.Ex1.m1.2.2.2.m2.1.1.2" xref="S3.Ex1.m1.2.2.2.m2.1.1.2.cmml">t</mi><mi id="S3.Ex1.m1.2.2.2.m2.1.1.3" xref="S3.Ex1.m1.2.2.2.m2.1.1.3.cmml">i</mi></msub></mrow></mrow></mrow><mo id="S3.Ex1.m1.4.4.1.1.2.2.5" xref="S3.Ex1.m1.4.4.1.1.2.3.1.cmml">}</mo></mrow></mrow><mo lspace="0em" id="S3.Ex1.m1.4.4.1.2" xref="S3.Ex1.m1.4.4.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.Ex1.m1.4b"><apply id="S3.Ex1.m1.4.4.1.1.cmml" xref="S3.Ex1.m1.4.4.1"><eq id="S3.Ex1.m1.4.4.1.1.3.cmml" xref="S3.Ex1.m1.4.4.1.1.3"></eq><apply id="S3.Ex1.m1.4.4.1.1.4.cmml" xref="S3.Ex1.m1.4.4.1.1.4"><csymbol cd="ambiguous" id="S3.Ex1.m1.4.4.1.1.4.1.cmml" xref="S3.Ex1.m1.4.4.1.1.4">superscript</csymbol><apply id="S3.Ex1.m1.4.4.1.1.4.2.cmml" xref="S3.Ex1.m1.4.4.1.1.4"><csymbol cd="ambiguous" id="S3.Ex1.m1.4.4.1.1.4.2.1.cmml" xref="S3.Ex1.m1.4.4.1.1.4">subscript</csymbol><ci id="S3.Ex1.m1.4.4.1.1.4.2.2.cmml" xref="S3.Ex1.m1.4.4.1.1.4.2.2">𝒟</ci><ci id="S3.Ex1.m1.4.4.1.1.4.2.3.cmml" xref="S3.Ex1.m1.4.4.1.1.4.2.3">𝒞</ci></apply><ci id="S3.Ex1.m1.3.3.1.1a.cmml" xref="S3.Ex1.m1.3.3.1.3"><mtext mathsize="70%" id="S3.Ex1.m1.3.3.1.1.cmml" xref="S3.Ex1.m1.3.3.1.1">retrieved</mtext></ci></apply><apply id="S3.Ex1.m1.4.4.1.1.2.3.cmml" xref="S3.Ex1.m1.4.4.1.1.2.2"><csymbol cd="latexml" id="S3.Ex1.m1.4.4.1.1.2.3.1.cmml" xref="S3.Ex1.m1.4.4.1.1.2.2.3">conditional-set</csymbol><interval closure="open" id="S3.Ex1.m1.4.4.1.1.1.1.1.3.cmml" xref="S3.Ex1.m1.4.4.1.1.1.1.1.2"><apply id="S3.Ex1.m1.4.4.1.1.1.1.1.1.1.cmml" xref="S3.Ex1.m1.4.4.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.Ex1.m1.4.4.1.1.1.1.1.1.1.1.cmml" xref="S3.Ex1.m1.4.4.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.Ex1.m1.4.4.1.1.1.1.1.1.1.2.cmml" xref="S3.Ex1.m1.4.4.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S3.Ex1.m1.4.4.1.1.1.1.1.1.1.3.cmml" xref="S3.Ex1.m1.4.4.1.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="S3.Ex1.m1.4.4.1.1.1.1.1.2.2.cmml" xref="S3.Ex1.m1.4.4.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.Ex1.m1.4.4.1.1.1.1.1.2.2.1.cmml" xref="S3.Ex1.m1.4.4.1.1.1.1.1.2.2">subscript</csymbol><ci id="S3.Ex1.m1.4.4.1.1.1.1.1.2.2.2.cmml" xref="S3.Ex1.m1.4.4.1.1.1.1.1.2.2.2">𝑦</ci><ci id="S3.Ex1.m1.4.4.1.1.1.1.1.2.2.3.cmml" xref="S3.Ex1.m1.4.4.1.1.1.1.1.2.2.3">𝑖</ci></apply></interval><apply id="S3.Ex1.m1.4.4.1.1.2.2.2.cmml" xref="S3.Ex1.m1.4.4.1.1.2.2.2"><in id="S3.Ex1.m1.4.4.1.1.2.2.2.3.cmml" xref="S3.Ex1.m1.4.4.1.1.2.2.2.3"></in><interval closure="open" id="S3.Ex1.m1.4.4.1.1.2.2.2.2.3.cmml" xref="S3.Ex1.m1.4.4.1.1.2.2.2.2.2"><apply id="S3.Ex1.m1.4.4.1.1.2.2.2.1.1.1.cmml" xref="S3.Ex1.m1.4.4.1.1.2.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.Ex1.m1.4.4.1.1.2.2.2.1.1.1.1.cmml" xref="S3.Ex1.m1.4.4.1.1.2.2.2.1.1.1">subscript</csymbol><ci id="S3.Ex1.m1.4.4.1.1.2.2.2.1.1.1.2.cmml" xref="S3.Ex1.m1.4.4.1.1.2.2.2.1.1.1.2">𝑥</ci><ci id="S3.Ex1.m1.4.4.1.1.2.2.2.1.1.1.3.cmml" xref="S3.Ex1.m1.4.4.1.1.2.2.2.1.1.1.3">𝑖</ci></apply><apply id="S3.Ex1.m1.4.4.1.1.2.2.2.2.2.2.cmml" xref="S3.Ex1.m1.4.4.1.1.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.Ex1.m1.4.4.1.1.2.2.2.2.2.2.1.cmml" xref="S3.Ex1.m1.4.4.1.1.2.2.2.2.2.2">subscript</csymbol><ci id="S3.Ex1.m1.4.4.1.1.2.2.2.2.2.2.2.cmml" xref="S3.Ex1.m1.4.4.1.1.2.2.2.2.2.2.2">𝑡</ci><ci id="S3.Ex1.m1.4.4.1.1.2.2.2.2.2.2.3.cmml" xref="S3.Ex1.m1.4.4.1.1.2.2.2.2.2.2.3">𝑖</ci></apply></interval><apply id="S3.Ex1.m1.4.4.1.1.2.2.2.4.cmml" xref="S3.Ex1.m1.4.4.1.1.2.2.2.4"><times id="S3.Ex1.m1.4.4.1.1.2.2.2.4.1.cmml" xref="S3.Ex1.m1.4.4.1.1.2.2.2.4.1"></times><ci id="S3.Ex1.m1.4.4.1.1.2.2.2.4.2.cmml" xref="S3.Ex1.m1.4.4.1.1.2.2.2.4.2">𝒟</ci><ci id="S3.Ex1.m1.2.2.2ac.cmml" xref="S3.Ex1.m1.2.2.2a"><mrow id="S3.Ex1.m1.2.2.2a.cmml" xref="S3.Ex1.m1.2.2.2a"><mtext id="S3.Ex1.m1.2.2.2aa.cmml" xref="S3.Ex1.m1.2.2.2a"> such that some class </mtext><mrow id="S3.Ex1.m1.1.1.1.m1.1.1.cmml" xref="S3.Ex1.m1.1.1.1.m1.1.1"><mi id="S3.Ex1.m1.1.1.1.m1.1.1.2.cmml" xref="S3.Ex1.m1.1.1.1.m1.1.1.2">c</mi><mo id="S3.Ex1.m1.1.1.1.m1.1.1.1.cmml" xref="S3.Ex1.m1.1.1.1.m1.1.1.1">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.Ex1.m1.1.1.1.m1.1.1.3.cmml" xref="S3.Ex1.m1.1.1.1.m1.1.1.3">𝒞</mi></mrow><mtext id="S3.Ex1.m1.2.2.2ab.cmml" xref="S3.Ex1.m1.2.2.2a"> is a substring of </mtext><msub id="S3.Ex1.m1.2.2.2.m2.1.1.cmml" xref="S3.Ex1.m1.2.2.2.m2.1.1"><mi id="S3.Ex1.m1.2.2.2.m2.1.1.2.cmml" xref="S3.Ex1.m1.2.2.2.m2.1.1.2">t</mi><mi id="S3.Ex1.m1.2.2.2.m2.1.1.3.cmml" xref="S3.Ex1.m1.2.2.2.m2.1.1.3">i</mi></msub></mrow></ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.Ex1.m1.4c">\displaystyle{\mathcal{D}_{\mathcal{C}}^{(\text{retrieved})}}=\left\{(x_{i},y_{i})\,\,\colon\,(x_{i},t_{i})\in{\mathcal{D}}\textrm{ such that some class $c\in{\mathcal{C}}$ is a substring of $t_{i}$}\right\}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S3.SS2.SSS0.Px1.p1.11" class="ltx_p">Here, label <math id="S3.SS2.SSS0.Px1.p1.5.m1.1" class="ltx_Math" alttext="y_{i}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.5.m1.1a"><msub id="S3.SS2.SSS0.Px1.p1.5.m1.1.1" xref="S3.SS2.SSS0.Px1.p1.5.m1.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p1.5.m1.1.1.2" xref="S3.SS2.SSS0.Px1.p1.5.m1.1.1.2.cmml">y</mi><mi id="S3.SS2.SSS0.Px1.p1.5.m1.1.1.3" xref="S3.SS2.SSS0.Px1.p1.5.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.5.m1.1b"><apply id="S3.SS2.SSS0.Px1.p1.5.m1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.5.m1.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m1.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.5.m1.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m1.1.1.2">𝑦</ci><ci id="S3.SS2.SSS0.Px1.p1.5.m1.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p1.5.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.5.m1.1c">y_{i}</annotation></semantics></math> is assigned based on the class <math id="S3.SS2.SSS0.Px1.p1.6.m2.1" class="ltx_Math" alttext="c" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.6.m2.1a"><mi id="S3.SS2.SSS0.Px1.p1.6.m2.1.1" xref="S3.SS2.SSS0.Px1.p1.6.m2.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.6.m2.1b"><ci id="S3.SS2.SSS0.Px1.p1.6.m2.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.6.m2.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.6.m2.1c">c</annotation></semantics></math> contained in <math id="S3.SS2.SSS0.Px1.p1.7.m3.1" class="ltx_Math" alttext="t_{i}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.7.m3.1a"><msub id="S3.SS2.SSS0.Px1.p1.7.m3.1.1" xref="S3.SS2.SSS0.Px1.p1.7.m3.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p1.7.m3.1.1.2" xref="S3.SS2.SSS0.Px1.p1.7.m3.1.1.2.cmml">t</mi><mi id="S3.SS2.SSS0.Px1.p1.7.m3.1.1.3" xref="S3.SS2.SSS0.Px1.p1.7.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.7.m3.1b"><apply id="S3.SS2.SSS0.Px1.p1.7.m3.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.7.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.7.m3.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.7.m3.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.7.m3.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.7.m3.1.1.2">𝑡</ci><ci id="S3.SS2.SSS0.Px1.p1.7.m3.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p1.7.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.7.m3.1c">t_{i}</annotation></semantics></math>. If an image-text pair <math id="S3.SS2.SSS0.Px1.p1.8.m4.2" class="ltx_Math" alttext="(x_{i},t_{i})\in{\mathcal{D}}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.8.m4.2a"><mrow id="S3.SS2.SSS0.Px1.p1.8.m4.2.2" xref="S3.SS2.SSS0.Px1.p1.8.m4.2.2.cmml"><mrow id="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2" xref="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.3.cmml"><mo stretchy="false" id="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2.3" xref="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.3.cmml">(</mo><msub id="S3.SS2.SSS0.Px1.p1.8.m4.1.1.1.1.1" xref="S3.SS2.SSS0.Px1.p1.8.m4.1.1.1.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p1.8.m4.1.1.1.1.1.2" xref="S3.SS2.SSS0.Px1.p1.8.m4.1.1.1.1.1.2.cmml">x</mi><mi id="S3.SS2.SSS0.Px1.p1.8.m4.1.1.1.1.1.3" xref="S3.SS2.SSS0.Px1.p1.8.m4.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2.4" xref="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.3.cmml">,</mo><msub id="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2.2" xref="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2.2.cmml"><mi id="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2.2.2" xref="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2.2.2.cmml">t</mi><mi id="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2.2.3" xref="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2.2.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2.5" xref="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.3.cmml">)</mo></mrow><mo id="S3.SS2.SSS0.Px1.p1.8.m4.2.2.3" xref="S3.SS2.SSS0.Px1.p1.8.m4.2.2.3.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS2.SSS0.Px1.p1.8.m4.2.2.4" xref="S3.SS2.SSS0.Px1.p1.8.m4.2.2.4.cmml">𝒟</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.8.m4.2b"><apply id="S3.SS2.SSS0.Px1.p1.8.m4.2.2.cmml" xref="S3.SS2.SSS0.Px1.p1.8.m4.2.2"><in id="S3.SS2.SSS0.Px1.p1.8.m4.2.2.3.cmml" xref="S3.SS2.SSS0.Px1.p1.8.m4.2.2.3"></in><interval closure="open" id="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.3.cmml" xref="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2"><apply id="S3.SS2.SSS0.Px1.p1.8.m4.1.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.8.m4.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.8.m4.1.1.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.8.m4.1.1.1.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.8.m4.1.1.1.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.8.m4.1.1.1.1.1.2">𝑥</ci><ci id="S3.SS2.SSS0.Px1.p1.8.m4.1.1.1.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p1.8.m4.1.1.1.1.1.3">𝑖</ci></apply><apply id="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2.2.cmml" xref="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2.2.1.cmml" xref="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2.2.2.cmml" xref="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2.2.2">𝑡</ci><ci id="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2.2.3.cmml" xref="S3.SS2.SSS0.Px1.p1.8.m4.2.2.2.2.2.3">𝑖</ci></apply></interval><ci id="S3.SS2.SSS0.Px1.p1.8.m4.2.2.4.cmml" xref="S3.SS2.SSS0.Px1.p1.8.m4.2.2.4">𝒟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.8.m4.2c">(x_{i},t_{i})\in{\mathcal{D}}</annotation></semantics></math> has text <math id="S3.SS2.SSS0.Px1.p1.9.m5.1" class="ltx_Math" alttext="t_{i}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.9.m5.1a"><msub id="S3.SS2.SSS0.Px1.p1.9.m5.1.1" xref="S3.SS2.SSS0.Px1.p1.9.m5.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p1.9.m5.1.1.2" xref="S3.SS2.SSS0.Px1.p1.9.m5.1.1.2.cmml">t</mi><mi id="S3.SS2.SSS0.Px1.p1.9.m5.1.1.3" xref="S3.SS2.SSS0.Px1.p1.9.m5.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.9.m5.1b"><apply id="S3.SS2.SSS0.Px1.p1.9.m5.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.9.m5.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.9.m5.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.9.m5.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.9.m5.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.9.m5.1.1.2">𝑡</ci><ci id="S3.SS2.SSS0.Px1.p1.9.m5.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p1.9.m5.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.9.m5.1c">t_{i}</annotation></semantics></math> containing multiple class names <math id="S3.SS2.SSS0.Px1.p1.10.m6.2" class="ltx_Math" alttext="c,c^{\prime}\in{\mathcal{C}}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.10.m6.2a"><mrow id="S3.SS2.SSS0.Px1.p1.10.m6.2.2" xref="S3.SS2.SSS0.Px1.p1.10.m6.2.2.cmml"><mrow id="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.1" xref="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.2.cmml"><mi id="S3.SS2.SSS0.Px1.p1.10.m6.1.1" xref="S3.SS2.SSS0.Px1.p1.10.m6.1.1.cmml">c</mi><mo id="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.1.2" xref="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.2.cmml">,</mo><msup id="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.1.1" xref="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.1.1.2" xref="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.1.1.2.cmml">c</mi><mo id="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.1.1.3" xref="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.1.1.3.cmml">′</mo></msup></mrow><mo id="S3.SS2.SSS0.Px1.p1.10.m6.2.2.2" xref="S3.SS2.SSS0.Px1.p1.10.m6.2.2.2.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS2.SSS0.Px1.p1.10.m6.2.2.3" xref="S3.SS2.SSS0.Px1.p1.10.m6.2.2.3.cmml">𝒞</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.10.m6.2b"><apply id="S3.SS2.SSS0.Px1.p1.10.m6.2.2.cmml" xref="S3.SS2.SSS0.Px1.p1.10.m6.2.2"><in id="S3.SS2.SSS0.Px1.p1.10.m6.2.2.2.cmml" xref="S3.SS2.SSS0.Px1.p1.10.m6.2.2.2"></in><list id="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.1"><ci id="S3.SS2.SSS0.Px1.p1.10.m6.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.10.m6.1.1">𝑐</ci><apply id="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.1.1">superscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.1.1.2">𝑐</ci><ci id="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p1.10.m6.2.2.1.1.1.3">′</ci></apply></list><ci id="S3.SS2.SSS0.Px1.p1.10.m6.2.2.3.cmml" xref="S3.SS2.SSS0.Px1.p1.10.m6.2.2.3">𝒞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.10.m6.2c">c,c^{\prime}\in{\mathcal{C}}</annotation></semantics></math>, then we simply retrieve <math id="S3.SS2.SSS0.Px1.p1.11.m7.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S3.SS2.SSS0.Px1.p1.11.m7.1a"><msub id="S3.SS2.SSS0.Px1.p1.11.m7.1.1" xref="S3.SS2.SSS0.Px1.p1.11.m7.1.1.cmml"><mi id="S3.SS2.SSS0.Px1.p1.11.m7.1.1.2" xref="S3.SS2.SSS0.Px1.p1.11.m7.1.1.2.cmml">x</mi><mi id="S3.SS2.SSS0.Px1.p1.11.m7.1.1.3" xref="S3.SS2.SSS0.Px1.p1.11.m7.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px1.p1.11.m7.1b"><apply id="S3.SS2.SSS0.Px1.p1.11.m7.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.11.m7.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px1.p1.11.m7.1.1.1.cmml" xref="S3.SS2.SSS0.Px1.p1.11.m7.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px1.p1.11.m7.1.1.2.cmml" xref="S3.SS2.SSS0.Px1.p1.11.m7.1.1.2">𝑥</ci><ci id="S3.SS2.SSS0.Px1.p1.11.m7.1.1.3.cmml" xref="S3.SS2.SSS0.Px1.p1.11.m7.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px1.p1.11.m7.1c">x_{i}</annotation></semantics></math> multiple times and assign each instance a different label, once for each unique matched class name.</p>
</div>
</section>
<section id="S3.SS2.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Strategy 2: semantic <math id="S3.SS2.SSS0.Px2.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS2.SSS0.Px2.1.m1.1b"><mi id="S3.SS2.SSS0.Px2.1.m1.1.1" xref="S3.SS2.SSS0.Px2.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.1.m1.1c"><ci id="S3.SS2.SSS0.Px2.1.m1.1.1.cmml" xref="S3.SS2.SSS0.Px2.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.1.m1.1d">k</annotation></semantics></math>-NN retrieval.</h4>

<div id="S3.SS2.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS2.SSS0.Px2.p1.13" class="ltx_p">Hard substring matching is simple and effective when the target visual concepts <math id="S3.SS2.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="c\in{\mathcal{C}}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.1.m1.1a"><mrow id="S3.SS2.SSS0.Px2.p1.1.m1.1.1" xref="S3.SS2.SSS0.Px2.p1.1.m1.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.1.m1.1.1.2" xref="S3.SS2.SSS0.Px2.p1.1.m1.1.1.2.cmml">c</mi><mo id="S3.SS2.SSS0.Px2.p1.1.m1.1.1.1" xref="S3.SS2.SSS0.Px2.p1.1.m1.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS2.SSS0.Px2.p1.1.m1.1.1.3" xref="S3.SS2.SSS0.Px2.p1.1.m1.1.1.3.cmml">𝒞</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.1.m1.1b"><apply id="S3.SS2.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.1.m1.1.1"><in id="S3.SS2.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.1.m1.1.1.1"></in><ci id="S3.SS2.SSS0.Px2.p1.1.m1.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.1.m1.1.1.2">𝑐</ci><ci id="S3.SS2.SSS0.Px2.p1.1.m1.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.1.m1.1.1.3">𝒞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.1.m1.1c">c\in{\mathcal{C}}</annotation></semantics></math> are concrete entities that are likely to be described in text captions (e.g., <math id="S3.SS2.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="c=" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.2.m2.1a"><mrow id="S3.SS2.SSS0.Px2.p1.2.m2.1.1" xref="S3.SS2.SSS0.Px2.p1.2.m2.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.2.m2.1.1.2" xref="S3.SS2.SSS0.Px2.p1.2.m2.1.1.2.cmml">c</mi><mo id="S3.SS2.SSS0.Px2.p1.2.m2.1.1.1" xref="S3.SS2.SSS0.Px2.p1.2.m2.1.1.1.cmml">=</mo><mi id="S3.SS2.SSS0.Px2.p1.2.m2.1.1.3" xref="S3.SS2.SSS0.Px2.p1.2.m2.1.1.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.2.m2.1b"><apply id="S3.SS2.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.2.m2.1.1"><eq id="S3.SS2.SSS0.Px2.p1.2.m2.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.2.m2.1.1.1"></eq><ci id="S3.SS2.SSS0.Px2.p1.2.m2.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.2.m2.1.1.2">𝑐</ci><csymbol cd="latexml" id="S3.SS2.SSS0.Px2.p1.2.m2.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.2.m2.1.1.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.2.m2.1c">c=</annotation></semantics></math> “fire lily”), but may be less effective when the concepts are abstract (e.g., <math id="S3.SS2.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="c=" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.3.m3.1a"><mrow id="S3.SS2.SSS0.Px2.p1.3.m3.1.1" xref="S3.SS2.SSS0.Px2.p1.3.m3.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.3.m3.1.1.2" xref="S3.SS2.SSS0.Px2.p1.3.m3.1.1.2.cmml">c</mi><mo id="S3.SS2.SSS0.Px2.p1.3.m3.1.1.1" xref="S3.SS2.SSS0.Px2.p1.3.m3.1.1.1.cmml">=</mo><mi id="S3.SS2.SSS0.Px2.p1.3.m3.1.1.3" xref="S3.SS2.SSS0.Px2.p1.3.m3.1.1.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.3.m3.1b"><apply id="S3.SS2.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.3.m3.1.1"><eq id="S3.SS2.SSS0.Px2.p1.3.m3.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.3.m3.1.1.1"></eq><ci id="S3.SS2.SSS0.Px2.p1.3.m3.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.3.m3.1.1.2">𝑐</ci><csymbol cd="latexml" id="S3.SS2.SSS0.Px2.p1.3.m3.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.3.m3.1.1.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.3.m3.1c">c=</annotation></semantics></math> “lined texture”). Thus, we also consider semantic (soft) retrieval via CLIP image-text embedding space similarity<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>We perform semantic retrieval using precomputed LAION-2B embeddings from OpenAI CLIP ViT-L/14 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib49" title="" class="ltx_ref">49</a>]</cite>, the same model Stable Diffusion uses to embed text prompts during generation.</span></span></span>. We convert each target class name <math id="S3.SS2.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="c\in{\mathcal{C}}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.4.m4.1a"><mrow id="S3.SS2.SSS0.Px2.p1.4.m4.1.1" xref="S3.SS2.SSS0.Px2.p1.4.m4.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.4.m4.1.1.2" xref="S3.SS2.SSS0.Px2.p1.4.m4.1.1.2.cmml">c</mi><mo id="S3.SS2.SSS0.Px2.p1.4.m4.1.1.1" xref="S3.SS2.SSS0.Px2.p1.4.m4.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS2.SSS0.Px2.p1.4.m4.1.1.3" xref="S3.SS2.SSS0.Px2.p1.4.m4.1.1.3.cmml">𝒞</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.4.m4.1b"><apply id="S3.SS2.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.4.m4.1.1"><in id="S3.SS2.SSS0.Px2.p1.4.m4.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.4.m4.1.1.1"></in><ci id="S3.SS2.SSS0.Px2.p1.4.m4.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.4.m4.1.1.2">𝑐</ci><ci id="S3.SS2.SSS0.Px2.p1.4.m4.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.4.m4.1.1.3">𝒞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.4.m4.1c">c\in{\mathcal{C}}</annotation></semantics></math> into a set of natural language search queries <math id="S3.SS2.SSS0.Px2.p1.5.m5.1" class="ltx_Math" alttext="Q_{c}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.5.m5.1a"><msub id="S3.SS2.SSS0.Px2.p1.5.m5.1.1" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.2" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.2.cmml">Q</mi><mi id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.3" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.5.m5.1b"><apply id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.2">𝑄</ci><ci id="S3.SS2.SSS0.Px2.p1.5.m5.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.5.m5.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.5.m5.1c">Q_{c}</annotation></semantics></math> based on the templates from the original CLIP paper <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib49" title="" class="ltx_ref">49</a>]</cite>.
For each query <math id="S3.SS2.SSS0.Px2.p1.6.m6.1" class="ltx_Math" alttext="q_{c}\in Q_{c}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.6.m6.1a"><mrow id="S3.SS2.SSS0.Px2.p1.6.m6.1.1" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1.cmml"><msub id="S3.SS2.SSS0.Px2.p1.6.m6.1.1.2" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1.2.cmml"><mi id="S3.SS2.SSS0.Px2.p1.6.m6.1.1.2.2" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1.2.2.cmml">q</mi><mi id="S3.SS2.SSS0.Px2.p1.6.m6.1.1.2.3" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1.2.3.cmml">c</mi></msub><mo id="S3.SS2.SSS0.Px2.p1.6.m6.1.1.1" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1.1.cmml">∈</mo><msub id="S3.SS2.SSS0.Px2.p1.6.m6.1.1.3" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1.3.cmml"><mi id="S3.SS2.SSS0.Px2.p1.6.m6.1.1.3.2" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1.3.2.cmml">Q</mi><mi id="S3.SS2.SSS0.Px2.p1.6.m6.1.1.3.3" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1.3.3.cmml">c</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.6.m6.1b"><apply id="S3.SS2.SSS0.Px2.p1.6.m6.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1"><in id="S3.SS2.SSS0.Px2.p1.6.m6.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1.1"></in><apply id="S3.SS2.SSS0.Px2.p1.6.m6.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.6.m6.1.1.2.1.cmml" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.6.m6.1.1.2.2.cmml" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1.2.2">𝑞</ci><ci id="S3.SS2.SSS0.Px2.p1.6.m6.1.1.2.3.cmml" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1.2.3">𝑐</ci></apply><apply id="S3.SS2.SSS0.Px2.p1.6.m6.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.6.m6.1.1.3.1.cmml" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.6.m6.1.1.3.2.cmml" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1.3.2">𝑄</ci><ci id="S3.SS2.SSS0.Px2.p1.6.m6.1.1.3.3.cmml" xref="S3.SS2.SSS0.Px2.p1.6.m6.1.1.3.3">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.6.m6.1c">q_{c}\in Q_{c}</annotation></semantics></math>, we use approximate <math id="S3.SS2.SSS0.Px2.p1.7.m7.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.7.m7.1a"><mi id="S3.SS2.SSS0.Px2.p1.7.m7.1.1" xref="S3.SS2.SSS0.Px2.p1.7.m7.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.7.m7.1b"><ci id="S3.SS2.SSS0.Px2.p1.7.m7.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.7.m7.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.7.m7.1c">k</annotation></semantics></math>-NN search <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite> to retrieve the set <math id="S3.SS2.SSS0.Px2.p1.8.m8.1" class="ltx_Math" alttext="S_{q_{c}}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.8.m8.1a"><msub id="S3.SS2.SSS0.Px2.p1.8.m8.1.1" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.2" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.2.cmml">S</mi><msub id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.cmml"><mi id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.2" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.2.cmml">q</mi><mi id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.3" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.3.cmml">c</mi></msub></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.8.m8.1b"><apply id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.2">𝑆</ci><apply id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.1.cmml" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.2.cmml" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.2">𝑞</ci><ci id="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.3.cmml" xref="S3.SS2.SSS0.Px2.p1.8.m8.1.1.3.3">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.8.m8.1c">S_{q_{c}}</annotation></semantics></math> of <math id="S3.SS2.SSS0.Px2.p1.9.m9.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.9.m9.1a"><mi id="S3.SS2.SSS0.Px2.p1.9.m9.1.1" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.9.m9.1b"><ci id="S3.SS2.SSS0.Px2.p1.9.m9.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.9.m9.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.9.m9.1c">k</annotation></semantics></math>-nearest image-text pairs <math id="S3.SS2.SSS0.Px2.p1.10.m10.2" class="ltx_Math" alttext="(x_{i},t_{i})\in{\mathcal{D}}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.10.m10.2a"><mrow id="S3.SS2.SSS0.Px2.p1.10.m10.2.2" xref="S3.SS2.SSS0.Px2.p1.10.m10.2.2.cmml"><mrow id="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2" xref="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.3.cmml"><mo stretchy="false" id="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2.3" xref="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.3.cmml">(</mo><msub id="S3.SS2.SSS0.Px2.p1.10.m10.1.1.1.1.1" xref="S3.SS2.SSS0.Px2.p1.10.m10.1.1.1.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.10.m10.1.1.1.1.1.2" xref="S3.SS2.SSS0.Px2.p1.10.m10.1.1.1.1.1.2.cmml">x</mi><mi id="S3.SS2.SSS0.Px2.p1.10.m10.1.1.1.1.1.3" xref="S3.SS2.SSS0.Px2.p1.10.m10.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2.4" xref="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.3.cmml">,</mo><msub id="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2.2" xref="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2.2.cmml"><mi id="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2.2.2" xref="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2.2.2.cmml">t</mi><mi id="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2.2.3" xref="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2.2.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2.5" xref="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.3.cmml">)</mo></mrow><mo id="S3.SS2.SSS0.Px2.p1.10.m10.2.2.3" xref="S3.SS2.SSS0.Px2.p1.10.m10.2.2.3.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS2.SSS0.Px2.p1.10.m10.2.2.4" xref="S3.SS2.SSS0.Px2.p1.10.m10.2.2.4.cmml">𝒟</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.10.m10.2b"><apply id="S3.SS2.SSS0.Px2.p1.10.m10.2.2.cmml" xref="S3.SS2.SSS0.Px2.p1.10.m10.2.2"><in id="S3.SS2.SSS0.Px2.p1.10.m10.2.2.3.cmml" xref="S3.SS2.SSS0.Px2.p1.10.m10.2.2.3"></in><interval closure="open" id="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.3.cmml" xref="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2"><apply id="S3.SS2.SSS0.Px2.p1.10.m10.1.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.10.m10.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.10.m10.1.1.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.10.m10.1.1.1.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.10.m10.1.1.1.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.10.m10.1.1.1.1.1.2">𝑥</ci><ci id="S3.SS2.SSS0.Px2.p1.10.m10.1.1.1.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.10.m10.1.1.1.1.1.3">𝑖</ci></apply><apply id="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2.2.cmml" xref="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2.2.1.cmml" xref="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2.2.2.cmml" xref="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2.2.2">𝑡</ci><ci id="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2.2.3.cmml" xref="S3.SS2.SSS0.Px2.p1.10.m10.2.2.2.2.2.3">𝑖</ci></apply></interval><ci id="S3.SS2.SSS0.Px2.p1.10.m10.2.2.4.cmml" xref="S3.SS2.SSS0.Px2.p1.10.m10.2.2.4">𝒟</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.10.m10.2c">(x_{i},t_{i})\in{\mathcal{D}}</annotation></semantics></math> by CLIP similarity between the query <math id="S3.SS2.SSS0.Px2.p1.11.m11.1" class="ltx_Math" alttext="q_{c}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.11.m11.1a"><msub id="S3.SS2.SSS0.Px2.p1.11.m11.1.1" xref="S3.SS2.SSS0.Px2.p1.11.m11.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.11.m11.1.1.2" xref="S3.SS2.SSS0.Px2.p1.11.m11.1.1.2.cmml">q</mi><mi id="S3.SS2.SSS0.Px2.p1.11.m11.1.1.3" xref="S3.SS2.SSS0.Px2.p1.11.m11.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.11.m11.1b"><apply id="S3.SS2.SSS0.Px2.p1.11.m11.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.11.m11.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.11.m11.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.11.m11.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.11.m11.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.11.m11.1.1.2">𝑞</ci><ci id="S3.SS2.SSS0.Px2.p1.11.m11.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.11.m11.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.11.m11.1c">q_{c}</annotation></semantics></math> and <span id="S3.SS2.SSS0.Px2.p1.13.1" class="ltx_text ltx_font_italic">either</span> the image <math id="S3.SS2.SSS0.Px2.p1.12.m12.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.12.m12.1a"><msub id="S3.SS2.SSS0.Px2.p1.12.m12.1.1" xref="S3.SS2.SSS0.Px2.p1.12.m12.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.12.m12.1.1.2" xref="S3.SS2.SSS0.Px2.p1.12.m12.1.1.2.cmml">x</mi><mi id="S3.SS2.SSS0.Px2.p1.12.m12.1.1.3" xref="S3.SS2.SSS0.Px2.p1.12.m12.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.12.m12.1b"><apply id="S3.SS2.SSS0.Px2.p1.12.m12.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.12.m12.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.12.m12.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.12.m12.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.12.m12.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.12.m12.1.1.2">𝑥</ci><ci id="S3.SS2.SSS0.Px2.p1.12.m12.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.12.m12.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.12.m12.1c">x_{i}</annotation></semantics></math> or the text <math id="S3.SS2.SSS0.Px2.p1.13.m13.1" class="ltx_Math" alttext="t_{i}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.13.m13.1a"><msub id="S3.SS2.SSS0.Px2.p1.13.m13.1.1" xref="S3.SS2.SSS0.Px2.p1.13.m13.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.13.m13.1.1.2" xref="S3.SS2.SSS0.Px2.p1.13.m13.1.1.2.cmml">t</mi><mi id="S3.SS2.SSS0.Px2.p1.13.m13.1.1.3" xref="S3.SS2.SSS0.Px2.p1.13.m13.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.13.m13.1b"><apply id="S3.SS2.SSS0.Px2.p1.13.m13.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.13.m13.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.13.m13.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.13.m13.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.13.m13.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.13.m13.1.1.2">𝑡</ci><ci id="S3.SS2.SSS0.Px2.p1.13.m13.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.13.m13.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.13.m13.1c">t_{i}</annotation></semantics></math>:</p>
<table id="A5.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.Ex2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.Ex2.m1.5" class="ltx_Math" alttext="\displaystyle S_{q_{c}}=\Bigl{\{}\operatorname*{arg\,top-k}_{(x_{i},t_{i})\in D}\text{CLIP}(x_{i},q_{c})\Bigr{\}}\cup\Bigl{\{}\operatorname*{arg\,top-k}_{(x_{i},t_{i})\in D}\text{CLIP}(t_{i},q_{c})\Bigr{\}}." display="inline"><semantics id="S3.Ex2.m1.5a"><mrow id="S3.Ex2.m1.5.5.1" xref="S3.Ex2.m1.5.5.1.1.cmml"><mrow id="S3.Ex2.m1.5.5.1.1" xref="S3.Ex2.m1.5.5.1.1.cmml"><msub id="S3.Ex2.m1.5.5.1.1.4" xref="S3.Ex2.m1.5.5.1.1.4.cmml"><mi id="S3.Ex2.m1.5.5.1.1.4.2" xref="S3.Ex2.m1.5.5.1.1.4.2.cmml">S</mi><msub id="S3.Ex2.m1.5.5.1.1.4.3" xref="S3.Ex2.m1.5.5.1.1.4.3.cmml"><mi id="S3.Ex2.m1.5.5.1.1.4.3.2" xref="S3.Ex2.m1.5.5.1.1.4.3.2.cmml">q</mi><mi id="S3.Ex2.m1.5.5.1.1.4.3.3" xref="S3.Ex2.m1.5.5.1.1.4.3.3.cmml">c</mi></msub></msub><mo id="S3.Ex2.m1.5.5.1.1.3" xref="S3.Ex2.m1.5.5.1.1.3.cmml">=</mo><mrow id="S3.Ex2.m1.5.5.1.1.2" xref="S3.Ex2.m1.5.5.1.1.2.cmml"><mrow id="S3.Ex2.m1.5.5.1.1.1.1.1" xref="S3.Ex2.m1.5.5.1.1.1.1.2.cmml"><mo maxsize="160%" minsize="160%" id="S3.Ex2.m1.5.5.1.1.1.1.1.2" xref="S3.Ex2.m1.5.5.1.1.1.1.2.cmml">{</mo><mrow id="S3.Ex2.m1.5.5.1.1.1.1.1.1" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.cmml"><mrow id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.cmml"><munder id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.cmml"><mrow id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.cmml"><mrow id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.2" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.2.cmml"><mi id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.2.2" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.2.2.cmml">arg</mi><mo lspace="0.170em" rspace="0em" id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.2.1" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.2.1.cmml">​</mo><mi id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.2.3" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.2.3.cmml">top</mi></mrow><mo id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.1" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.1.cmml">−</mo><mi mathvariant="normal" id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.3" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.3.cmml">k</mi></mrow><mrow id="S3.Ex2.m1.2.2.2" xref="S3.Ex2.m1.2.2.2.cmml"><mrow id="S3.Ex2.m1.2.2.2.2.2" xref="S3.Ex2.m1.2.2.2.2.3.cmml"><mo stretchy="false" id="S3.Ex2.m1.2.2.2.2.2.3" xref="S3.Ex2.m1.2.2.2.2.3.cmml">(</mo><msub id="S3.Ex2.m1.1.1.1.1.1.1" xref="S3.Ex2.m1.1.1.1.1.1.1.cmml"><mi id="S3.Ex2.m1.1.1.1.1.1.1.2" xref="S3.Ex2.m1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S3.Ex2.m1.1.1.1.1.1.1.3" xref="S3.Ex2.m1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.Ex2.m1.2.2.2.2.2.4" xref="S3.Ex2.m1.2.2.2.2.3.cmml">,</mo><msub id="S3.Ex2.m1.2.2.2.2.2.2" xref="S3.Ex2.m1.2.2.2.2.2.2.cmml"><mi id="S3.Ex2.m1.2.2.2.2.2.2.2" xref="S3.Ex2.m1.2.2.2.2.2.2.2.cmml">t</mi><mi id="S3.Ex2.m1.2.2.2.2.2.2.3" xref="S3.Ex2.m1.2.2.2.2.2.2.3.cmml">i</mi></msub><mo stretchy="false" id="S3.Ex2.m1.2.2.2.2.2.5" xref="S3.Ex2.m1.2.2.2.2.3.cmml">)</mo></mrow><mo id="S3.Ex2.m1.2.2.2.3" xref="S3.Ex2.m1.2.2.2.3.cmml">∈</mo><mi id="S3.Ex2.m1.2.2.2.4" xref="S3.Ex2.m1.2.2.2.4.cmml">D</mi></mrow></munder><mo id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4a" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.cmml">⁡</mo><mtext id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.2" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.2a.cmml">CLIP</mtext></mrow><mo lspace="0em" rspace="0em" id="S3.Ex2.m1.5.5.1.1.1.1.1.1.3" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.3.cmml">​</mo><mrow id="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.3.cmml"><mo stretchy="false" id="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2.3" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.3.cmml">(</mo><msub id="S3.Ex2.m1.5.5.1.1.1.1.1.1.1.1.1" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.Ex2.m1.5.5.1.1.1.1.1.1.1.1.1.2" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S3.Ex2.m1.5.5.1.1.1.1.1.1.1.1.1.3" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2.4" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.3.cmml">,</mo><msub id="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2.2" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2.2.cmml"><mi id="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2.2.2" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2.2.2.cmml">q</mi><mi id="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2.2.3" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2.2.3.cmml">c</mi></msub><mo stretchy="false" id="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2.5" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.3.cmml">)</mo></mrow></mrow><mo maxsize="160%" minsize="160%" id="S3.Ex2.m1.5.5.1.1.1.1.1.3" xref="S3.Ex2.m1.5.5.1.1.1.1.2.cmml">}</mo></mrow><mo id="S3.Ex2.m1.5.5.1.1.2.3" xref="S3.Ex2.m1.5.5.1.1.2.3.cmml">∪</mo><mrow id="S3.Ex2.m1.5.5.1.1.2.2.1" xref="S3.Ex2.m1.5.5.1.1.2.2.2.cmml"><mo maxsize="160%" minsize="160%" id="S3.Ex2.m1.5.5.1.1.2.2.1.2" xref="S3.Ex2.m1.5.5.1.1.2.2.2.cmml">{</mo><mrow id="S3.Ex2.m1.5.5.1.1.2.2.1.1" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.cmml"><mrow id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.cmml"><munder id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.cmml"><mrow id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.cmml"><mrow id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.2" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.2.cmml"><mi id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.2.2" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.2.2.cmml">arg</mi><mo lspace="0.170em" rspace="0em" id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.2.1" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.2.1.cmml">​</mo><mi id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.2.3" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.2.3.cmml">top</mi></mrow><mo id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.1" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.1.cmml">−</mo><mi mathvariant="normal" id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.3" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.3.cmml">k</mi></mrow><mrow id="S3.Ex2.m1.4.4.2" xref="S3.Ex2.m1.4.4.2.cmml"><mrow id="S3.Ex2.m1.4.4.2.2.2" xref="S3.Ex2.m1.4.4.2.2.3.cmml"><mo stretchy="false" id="S3.Ex2.m1.4.4.2.2.2.3" xref="S3.Ex2.m1.4.4.2.2.3.cmml">(</mo><msub id="S3.Ex2.m1.3.3.1.1.1.1" xref="S3.Ex2.m1.3.3.1.1.1.1.cmml"><mi id="S3.Ex2.m1.3.3.1.1.1.1.2" xref="S3.Ex2.m1.3.3.1.1.1.1.2.cmml">x</mi><mi id="S3.Ex2.m1.3.3.1.1.1.1.3" xref="S3.Ex2.m1.3.3.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.Ex2.m1.4.4.2.2.2.4" xref="S3.Ex2.m1.4.4.2.2.3.cmml">,</mo><msub id="S3.Ex2.m1.4.4.2.2.2.2" xref="S3.Ex2.m1.4.4.2.2.2.2.cmml"><mi id="S3.Ex2.m1.4.4.2.2.2.2.2" xref="S3.Ex2.m1.4.4.2.2.2.2.2.cmml">t</mi><mi id="S3.Ex2.m1.4.4.2.2.2.2.3" xref="S3.Ex2.m1.4.4.2.2.2.2.3.cmml">i</mi></msub><mo stretchy="false" id="S3.Ex2.m1.4.4.2.2.2.5" xref="S3.Ex2.m1.4.4.2.2.3.cmml">)</mo></mrow><mo id="S3.Ex2.m1.4.4.2.3" xref="S3.Ex2.m1.4.4.2.3.cmml">∈</mo><mi id="S3.Ex2.m1.4.4.2.4" xref="S3.Ex2.m1.4.4.2.4.cmml">D</mi></mrow></munder><mo id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4a" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.cmml">⁡</mo><mtext id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.2" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.2a.cmml">CLIP</mtext></mrow><mo lspace="0em" rspace="0em" id="S3.Ex2.m1.5.5.1.1.2.2.1.1.3" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.3.cmml">​</mo><mrow id="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.3.cmml"><mo stretchy="false" id="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2.3" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.3.cmml">(</mo><msub id="S3.Ex2.m1.5.5.1.1.2.2.1.1.1.1.1" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.1.1.1.cmml"><mi id="S3.Ex2.m1.5.5.1.1.2.2.1.1.1.1.1.2" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.1.1.1.2.cmml">t</mi><mi id="S3.Ex2.m1.5.5.1.1.2.2.1.1.1.1.1.3" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2.4" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.3.cmml">,</mo><msub id="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2.2" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2.2.cmml"><mi id="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2.2.2" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2.2.2.cmml">q</mi><mi id="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2.2.3" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2.2.3.cmml">c</mi></msub><mo stretchy="false" id="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2.5" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.3.cmml">)</mo></mrow></mrow><mo maxsize="160%" minsize="160%" id="S3.Ex2.m1.5.5.1.1.2.2.1.3" xref="S3.Ex2.m1.5.5.1.1.2.2.2.cmml">}</mo></mrow></mrow></mrow><mo lspace="0em" id="S3.Ex2.m1.5.5.1.2" xref="S3.Ex2.m1.5.5.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.Ex2.m1.5b"><apply id="S3.Ex2.m1.5.5.1.1.cmml" xref="S3.Ex2.m1.5.5.1"><eq id="S3.Ex2.m1.5.5.1.1.3.cmml" xref="S3.Ex2.m1.5.5.1.1.3"></eq><apply id="S3.Ex2.m1.5.5.1.1.4.cmml" xref="S3.Ex2.m1.5.5.1.1.4"><csymbol cd="ambiguous" id="S3.Ex2.m1.5.5.1.1.4.1.cmml" xref="S3.Ex2.m1.5.5.1.1.4">subscript</csymbol><ci id="S3.Ex2.m1.5.5.1.1.4.2.cmml" xref="S3.Ex2.m1.5.5.1.1.4.2">𝑆</ci><apply id="S3.Ex2.m1.5.5.1.1.4.3.cmml" xref="S3.Ex2.m1.5.5.1.1.4.3"><csymbol cd="ambiguous" id="S3.Ex2.m1.5.5.1.1.4.3.1.cmml" xref="S3.Ex2.m1.5.5.1.1.4.3">subscript</csymbol><ci id="S3.Ex2.m1.5.5.1.1.4.3.2.cmml" xref="S3.Ex2.m1.5.5.1.1.4.3.2">𝑞</ci><ci id="S3.Ex2.m1.5.5.1.1.4.3.3.cmml" xref="S3.Ex2.m1.5.5.1.1.4.3.3">𝑐</ci></apply></apply><apply id="S3.Ex2.m1.5.5.1.1.2.cmml" xref="S3.Ex2.m1.5.5.1.1.2"><union id="S3.Ex2.m1.5.5.1.1.2.3.cmml" xref="S3.Ex2.m1.5.5.1.1.2.3"></union><set id="S3.Ex2.m1.5.5.1.1.1.1.2.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1"><apply id="S3.Ex2.m1.5.5.1.1.1.1.1.1.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1"><times id="S3.Ex2.m1.5.5.1.1.1.1.1.1.3.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.3"></times><apply id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4"><apply id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1"><csymbol cd="ambiguous" id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.1.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1">subscript</csymbol><apply id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2"><minus id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.1.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.1"></minus><apply id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.2.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.2"><times id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.2.1.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.2.1"></times><ci id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.2.2.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.2.2">arg</ci><ci id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.2.3.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.2.3">top</ci></apply><ci id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.3.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.1.2.3">k</ci></apply><apply id="S3.Ex2.m1.2.2.2.cmml" xref="S3.Ex2.m1.2.2.2"><in id="S3.Ex2.m1.2.2.2.3.cmml" xref="S3.Ex2.m1.2.2.2.3"></in><interval closure="open" id="S3.Ex2.m1.2.2.2.2.3.cmml" xref="S3.Ex2.m1.2.2.2.2.2"><apply id="S3.Ex2.m1.1.1.1.1.1.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.Ex2.m1.1.1.1.1.1.1.1.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.Ex2.m1.1.1.1.1.1.1.2.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.2">𝑥</ci><ci id="S3.Ex2.m1.1.1.1.1.1.1.3.cmml" xref="S3.Ex2.m1.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="S3.Ex2.m1.2.2.2.2.2.2.cmml" xref="S3.Ex2.m1.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.Ex2.m1.2.2.2.2.2.2.1.cmml" xref="S3.Ex2.m1.2.2.2.2.2.2">subscript</csymbol><ci id="S3.Ex2.m1.2.2.2.2.2.2.2.cmml" xref="S3.Ex2.m1.2.2.2.2.2.2.2">𝑡</ci><ci id="S3.Ex2.m1.2.2.2.2.2.2.3.cmml" xref="S3.Ex2.m1.2.2.2.2.2.2.3">𝑖</ci></apply></interval><ci id="S3.Ex2.m1.2.2.2.4.cmml" xref="S3.Ex2.m1.2.2.2.4">𝐷</ci></apply></apply><ci id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.2a.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.2"><mtext id="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.2.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.4.2">CLIP</mtext></ci></apply><interval closure="open" id="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.3.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2"><apply id="S3.Ex2.m1.5.5.1.1.1.1.1.1.1.1.1.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.Ex2.m1.5.5.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.Ex2.m1.5.5.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S3.Ex2.m1.5.5.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2.2.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2.2.1.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2.2">subscript</csymbol><ci id="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2.2.2.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2.2.2">𝑞</ci><ci id="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2.2.3.cmml" xref="S3.Ex2.m1.5.5.1.1.1.1.1.1.2.2.2.3">𝑐</ci></apply></interval></apply></set><set id="S3.Ex2.m1.5.5.1.1.2.2.2.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1"><apply id="S3.Ex2.m1.5.5.1.1.2.2.1.1.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1"><times id="S3.Ex2.m1.5.5.1.1.2.2.1.1.3.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.3"></times><apply id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4"><apply id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1"><csymbol cd="ambiguous" id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.1.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1">subscript</csymbol><apply id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2"><minus id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.1.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.1"></minus><apply id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.2.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.2"><times id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.2.1.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.2.1"></times><ci id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.2.2.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.2.2">arg</ci><ci id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.2.3.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.2.3">top</ci></apply><ci id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.3.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.1.2.3">k</ci></apply><apply id="S3.Ex2.m1.4.4.2.cmml" xref="S3.Ex2.m1.4.4.2"><in id="S3.Ex2.m1.4.4.2.3.cmml" xref="S3.Ex2.m1.4.4.2.3"></in><interval closure="open" id="S3.Ex2.m1.4.4.2.2.3.cmml" xref="S3.Ex2.m1.4.4.2.2.2"><apply id="S3.Ex2.m1.3.3.1.1.1.1.cmml" xref="S3.Ex2.m1.3.3.1.1.1.1"><csymbol cd="ambiguous" id="S3.Ex2.m1.3.3.1.1.1.1.1.cmml" xref="S3.Ex2.m1.3.3.1.1.1.1">subscript</csymbol><ci id="S3.Ex2.m1.3.3.1.1.1.1.2.cmml" xref="S3.Ex2.m1.3.3.1.1.1.1.2">𝑥</ci><ci id="S3.Ex2.m1.3.3.1.1.1.1.3.cmml" xref="S3.Ex2.m1.3.3.1.1.1.1.3">𝑖</ci></apply><apply id="S3.Ex2.m1.4.4.2.2.2.2.cmml" xref="S3.Ex2.m1.4.4.2.2.2.2"><csymbol cd="ambiguous" id="S3.Ex2.m1.4.4.2.2.2.2.1.cmml" xref="S3.Ex2.m1.4.4.2.2.2.2">subscript</csymbol><ci id="S3.Ex2.m1.4.4.2.2.2.2.2.cmml" xref="S3.Ex2.m1.4.4.2.2.2.2.2">𝑡</ci><ci id="S3.Ex2.m1.4.4.2.2.2.2.3.cmml" xref="S3.Ex2.m1.4.4.2.2.2.2.3">𝑖</ci></apply></interval><ci id="S3.Ex2.m1.4.4.2.4.cmml" xref="S3.Ex2.m1.4.4.2.4">𝐷</ci></apply></apply><ci id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.2a.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.2"><mtext id="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.2.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.4.2">CLIP</mtext></ci></apply><interval closure="open" id="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.3.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2"><apply id="S3.Ex2.m1.5.5.1.1.2.2.1.1.1.1.1.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.Ex2.m1.5.5.1.1.2.2.1.1.1.1.1.1.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.1.1.1">subscript</csymbol><ci id="S3.Ex2.m1.5.5.1.1.2.2.1.1.1.1.1.2.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.1.1.1.2">𝑡</ci><ci id="S3.Ex2.m1.5.5.1.1.2.2.1.1.1.1.1.3.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.1.1.1.3">𝑖</ci></apply><apply id="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2.2.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2.2"><csymbol cd="ambiguous" id="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2.2.1.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2.2">subscript</csymbol><ci id="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2.2.2.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2.2.2">𝑞</ci><ci id="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2.2.3.cmml" xref="S3.Ex2.m1.5.5.1.1.2.2.1.1.2.2.2.3">𝑐</ci></apply></interval></apply></set></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.Ex2.m1.5c">\displaystyle S_{q_{c}}=\Bigl{\{}\operatorname*{arg\,top-k}_{(x_{i},t_{i})\in D}\text{CLIP}(x_{i},q_{c})\Bigr{\}}\cup\Bigl{\{}\operatorname*{arg\,top-k}_{(x_{i},t_{i})\in D}\text{CLIP}(t_{i},q_{c})\Bigr{\}}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S3.SS2.SSS0.Px2.p1.19" class="ltx_p">We assign each image-text pair <math id="S3.SS2.SSS0.Px2.p1.14.m1.2" class="ltx_Math" alttext="(x_{i},t_{i})\in S_{q_{c}}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.14.m1.2a"><mrow id="S3.SS2.SSS0.Px2.p1.14.m1.2.2" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.cmml"><mrow id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.3.cmml"><mo stretchy="false" id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2.3" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.3.cmml">(</mo><msub id="S3.SS2.SSS0.Px2.p1.14.m1.1.1.1.1.1" xref="S3.SS2.SSS0.Px2.p1.14.m1.1.1.1.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.14.m1.1.1.1.1.1.2" xref="S3.SS2.SSS0.Px2.p1.14.m1.1.1.1.1.1.2.cmml">x</mi><mi id="S3.SS2.SSS0.Px2.p1.14.m1.1.1.1.1.1.3" xref="S3.SS2.SSS0.Px2.p1.14.m1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2.4" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.3.cmml">,</mo><msub id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2.2" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2.2.cmml"><mi id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2.2.2" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2.2.2.cmml">t</mi><mi id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2.2.3" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2.2.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2.5" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.3.cmml">)</mo></mrow><mo id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.3" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.3.cmml">∈</mo><msub id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.cmml"><mi id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.2" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.2.cmml">S</mi><msub id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.3" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.3.cmml"><mi id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.3.2" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.3.2.cmml">q</mi><mi id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.3.3" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.3.3.cmml">c</mi></msub></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.14.m1.2b"><apply id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.cmml" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2"><in id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.3.cmml" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.3"></in><interval closure="open" id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.3.cmml" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2"><apply id="S3.SS2.SSS0.Px2.p1.14.m1.1.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.14.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.14.m1.1.1.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.14.m1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.14.m1.1.1.1.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.14.m1.1.1.1.1.1.2">𝑥</ci><ci id="S3.SS2.SSS0.Px2.p1.14.m1.1.1.1.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.14.m1.1.1.1.1.1.3">𝑖</ci></apply><apply id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2.2.cmml" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2.2.1.cmml" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2.2.2.cmml" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2.2.2">𝑡</ci><ci id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2.2.3.cmml" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.2.2.2.3">𝑖</ci></apply></interval><apply id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.cmml" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.1.cmml" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.2.cmml" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.2">𝑆</ci><apply id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.3.cmml" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.3.1.cmml" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.3.2.cmml" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.3.2">𝑞</ci><ci id="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.3.3.cmml" xref="S3.SS2.SSS0.Px2.p1.14.m1.2.2.4.3.3">𝑐</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.14.m1.2c">(x_{i},t_{i})\in S_{q_{c}}</annotation></semantics></math> a class label <math id="S3.SS2.SSS0.Px2.p1.15.m2.1" class="ltx_Math" alttext="y_{i}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.15.m2.1a"><msub id="S3.SS2.SSS0.Px2.p1.15.m2.1.1" xref="S3.SS2.SSS0.Px2.p1.15.m2.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.15.m2.1.1.2" xref="S3.SS2.SSS0.Px2.p1.15.m2.1.1.2.cmml">y</mi><mi id="S3.SS2.SSS0.Px2.p1.15.m2.1.1.3" xref="S3.SS2.SSS0.Px2.p1.15.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.15.m2.1b"><apply id="S3.SS2.SSS0.Px2.p1.15.m2.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.15.m2.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.15.m2.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.15.m2.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.15.m2.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.15.m2.1.1.2">𝑦</ci><ci id="S3.SS2.SSS0.Px2.p1.15.m2.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.15.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.15.m2.1c">y_{i}</annotation></semantics></math> based on the class name in query <math id="S3.SS2.SSS0.Px2.p1.16.m3.1" class="ltx_Math" alttext="q_{c}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.16.m3.1a"><msub id="S3.SS2.SSS0.Px2.p1.16.m3.1.1" xref="S3.SS2.SSS0.Px2.p1.16.m3.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.16.m3.1.1.2" xref="S3.SS2.SSS0.Px2.p1.16.m3.1.1.2.cmml">q</mi><mi id="S3.SS2.SSS0.Px2.p1.16.m3.1.1.3" xref="S3.SS2.SSS0.Px2.p1.16.m3.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.16.m3.1b"><apply id="S3.SS2.SSS0.Px2.p1.16.m3.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.16.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.16.m3.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.16.m3.1.1">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.16.m3.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.16.m3.1.1.2">𝑞</ci><ci id="S3.SS2.SSS0.Px2.p1.16.m3.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.16.m3.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.16.m3.1c">q_{c}</annotation></semantics></math>. We form the targeted dataset <math id="S3.SS2.SSS0.Px2.p1.17.m4.1" class="ltx_Math" alttext="{\mathcal{D}_{\mathcal{C}}^{(\text{retrieved})}}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.17.m4.1a"><msubsup id="S3.SS2.SSS0.Px2.p1.17.m4.1.2" xref="S3.SS2.SSS0.Px2.p1.17.m4.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.SSS0.Px2.p1.17.m4.1.2.2.2" xref="S3.SS2.SSS0.Px2.p1.17.m4.1.2.2.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS2.SSS0.Px2.p1.17.m4.1.2.2.3" xref="S3.SS2.SSS0.Px2.p1.17.m4.1.2.2.3.cmml">𝒞</mi><mrow id="S3.SS2.SSS0.Px2.p1.17.m4.1.1.1.3" xref="S3.SS2.SSS0.Px2.p1.17.m4.1.1.1.1a.cmml"><mo stretchy="false" id="S3.SS2.SSS0.Px2.p1.17.m4.1.1.1.3.1" xref="S3.SS2.SSS0.Px2.p1.17.m4.1.1.1.1a.cmml">(</mo><mtext id="S3.SS2.SSS0.Px2.p1.17.m4.1.1.1.1" xref="S3.SS2.SSS0.Px2.p1.17.m4.1.1.1.1.cmml">retrieved</mtext><mo stretchy="false" id="S3.SS2.SSS0.Px2.p1.17.m4.1.1.1.3.2" xref="S3.SS2.SSS0.Px2.p1.17.m4.1.1.1.1a.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.17.m4.1b"><apply id="S3.SS2.SSS0.Px2.p1.17.m4.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.17.m4.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.17.m4.1.2.1.cmml" xref="S3.SS2.SSS0.Px2.p1.17.m4.1.2">superscript</csymbol><apply id="S3.SS2.SSS0.Px2.p1.17.m4.1.2.2.cmml" xref="S3.SS2.SSS0.Px2.p1.17.m4.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.17.m4.1.2.2.1.cmml" xref="S3.SS2.SSS0.Px2.p1.17.m4.1.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.17.m4.1.2.2.2.cmml" xref="S3.SS2.SSS0.Px2.p1.17.m4.1.2.2.2">𝒟</ci><ci id="S3.SS2.SSS0.Px2.p1.17.m4.1.2.2.3.cmml" xref="S3.SS2.SSS0.Px2.p1.17.m4.1.2.2.3">𝒞</ci></apply><ci id="S3.SS2.SSS0.Px2.p1.17.m4.1.1.1.1a.cmml" xref="S3.SS2.SSS0.Px2.p1.17.m4.1.1.1.3"><mtext mathsize="70%" id="S3.SS2.SSS0.Px2.p1.17.m4.1.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.17.m4.1.1.1.1">retrieved</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.17.m4.1c">{\mathcal{D}_{\mathcal{C}}^{(\text{retrieved})}}</annotation></semantics></math> by unioning over all queries <math id="S3.SS2.SSS0.Px2.p1.18.m5.1" class="ltx_Math" alttext="q_{c}\in Q_{c}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.18.m5.1a"><mrow id="S3.SS2.SSS0.Px2.p1.18.m5.1.1" xref="S3.SS2.SSS0.Px2.p1.18.m5.1.1.cmml"><msub id="S3.SS2.SSS0.Px2.p1.18.m5.1.1.2" xref="S3.SS2.SSS0.Px2.p1.18.m5.1.1.2.cmml"><mi id="S3.SS2.SSS0.Px2.p1.18.m5.1.1.2.2" xref="S3.SS2.SSS0.Px2.p1.18.m5.1.1.2.2.cmml">q</mi><mi id="S3.SS2.SSS0.Px2.p1.18.m5.1.1.2.3" xref="S3.SS2.SSS0.Px2.p1.18.m5.1.1.2.3.cmml">c</mi></msub><mo id="S3.SS2.SSS0.Px2.p1.18.m5.1.1.1" xref="S3.SS2.SSS0.Px2.p1.18.m5.1.1.1.cmml">∈</mo><msub id="S3.SS2.SSS0.Px2.p1.18.m5.1.1.3" xref="S3.SS2.SSS0.Px2.p1.18.m5.1.1.3.cmml"><mi id="S3.SS2.SSS0.Px2.p1.18.m5.1.1.3.2" xref="S3.SS2.SSS0.Px2.p1.18.m5.1.1.3.2.cmml">Q</mi><mi id="S3.SS2.SSS0.Px2.p1.18.m5.1.1.3.3" xref="S3.SS2.SSS0.Px2.p1.18.m5.1.1.3.3.cmml">c</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.18.m5.1b"><apply id="S3.SS2.SSS0.Px2.p1.18.m5.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.18.m5.1.1"><in id="S3.SS2.SSS0.Px2.p1.18.m5.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.18.m5.1.1.1"></in><apply id="S3.SS2.SSS0.Px2.p1.18.m5.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.18.m5.1.1.2"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.18.m5.1.1.2.1.cmml" xref="S3.SS2.SSS0.Px2.p1.18.m5.1.1.2">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.18.m5.1.1.2.2.cmml" xref="S3.SS2.SSS0.Px2.p1.18.m5.1.1.2.2">𝑞</ci><ci id="S3.SS2.SSS0.Px2.p1.18.m5.1.1.2.3.cmml" xref="S3.SS2.SSS0.Px2.p1.18.m5.1.1.2.3">𝑐</ci></apply><apply id="S3.SS2.SSS0.Px2.p1.18.m5.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.18.m5.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.SSS0.Px2.p1.18.m5.1.1.3.1.cmml" xref="S3.SS2.SSS0.Px2.p1.18.m5.1.1.3">subscript</csymbol><ci id="S3.SS2.SSS0.Px2.p1.18.m5.1.1.3.2.cmml" xref="S3.SS2.SSS0.Px2.p1.18.m5.1.1.3.2">𝑄</ci><ci id="S3.SS2.SSS0.Px2.p1.18.m5.1.1.3.3.cmml" xref="S3.SS2.SSS0.Px2.p1.18.m5.1.1.3.3">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.18.m5.1c">q_{c}\in Q_{c}</annotation></semantics></math> and all classes <math id="S3.SS2.SSS0.Px2.p1.19.m6.1" class="ltx_Math" alttext="c\in{\mathcal{C}}" display="inline"><semantics id="S3.SS2.SSS0.Px2.p1.19.m6.1a"><mrow id="S3.SS2.SSS0.Px2.p1.19.m6.1.1" xref="S3.SS2.SSS0.Px2.p1.19.m6.1.1.cmml"><mi id="S3.SS2.SSS0.Px2.p1.19.m6.1.1.2" xref="S3.SS2.SSS0.Px2.p1.19.m6.1.1.2.cmml">c</mi><mo id="S3.SS2.SSS0.Px2.p1.19.m6.1.1.1" xref="S3.SS2.SSS0.Px2.p1.19.m6.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS2.SSS0.Px2.p1.19.m6.1.1.3" xref="S3.SS2.SSS0.Px2.p1.19.m6.1.1.3.cmml">𝒞</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS0.Px2.p1.19.m6.1b"><apply id="S3.SS2.SSS0.Px2.p1.19.m6.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.19.m6.1.1"><in id="S3.SS2.SSS0.Px2.p1.19.m6.1.1.1.cmml" xref="S3.SS2.SSS0.Px2.p1.19.m6.1.1.1"></in><ci id="S3.SS2.SSS0.Px2.p1.19.m6.1.1.2.cmml" xref="S3.SS2.SSS0.Px2.p1.19.m6.1.1.2">𝑐</ci><ci id="S3.SS2.SSS0.Px2.p1.19.m6.1.1.3.cmml" xref="S3.SS2.SSS0.Px2.p1.19.m6.1.1.3">𝒞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS0.Px2.p1.19.m6.1c">c\in{\mathcal{C}}</annotation></semantics></math>:</p>
<table id="A5.EGx3" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.Ex3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.Ex3.m1.2" class="ltx_Math" alttext="\displaystyle{\mathcal{D}_{\mathcal{C}}^{(\text{retrieved})}}=\bigcup_{c\in{\mathcal{C}}}\bigcup_{q_{c}\in Q_{c}}\left\{(x_{i},y_{i})\colon(x_{i},t_{i},y_{i})\in S_{q_{c}}\right\}." display="inline"><semantics id="S3.Ex3.m1.2a"><mrow id="S3.Ex3.m1.2.2.1" xref="S3.Ex3.m1.2.2.1.1.cmml"><mrow id="S3.Ex3.m1.2.2.1.1" xref="S3.Ex3.m1.2.2.1.1.cmml"><msubsup id="S3.Ex3.m1.2.2.1.1.4" xref="S3.Ex3.m1.2.2.1.1.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.Ex3.m1.2.2.1.1.4.2.2" xref="S3.Ex3.m1.2.2.1.1.4.2.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S3.Ex3.m1.2.2.1.1.4.2.3" xref="S3.Ex3.m1.2.2.1.1.4.2.3.cmml">𝒞</mi><mrow id="S3.Ex3.m1.1.1.1.3" xref="S3.Ex3.m1.1.1.1.1a.cmml"><mo stretchy="false" id="S3.Ex3.m1.1.1.1.3.1" xref="S3.Ex3.m1.1.1.1.1a.cmml">(</mo><mtext id="S3.Ex3.m1.1.1.1.1" xref="S3.Ex3.m1.1.1.1.1.cmml">retrieved</mtext><mo stretchy="false" id="S3.Ex3.m1.1.1.1.3.2" xref="S3.Ex3.m1.1.1.1.1a.cmml">)</mo></mrow></msubsup><mo id="S3.Ex3.m1.2.2.1.1.3" xref="S3.Ex3.m1.2.2.1.1.3.cmml">=</mo><mrow id="S3.Ex3.m1.2.2.1.1.2" xref="S3.Ex3.m1.2.2.1.1.2.cmml"><mstyle displaystyle="true" id="S3.Ex3.m1.2.2.1.1.2.3" xref="S3.Ex3.m1.2.2.1.1.2.3.cmml"><munder id="S3.Ex3.m1.2.2.1.1.2.3a" xref="S3.Ex3.m1.2.2.1.1.2.3.cmml"><mo movablelimits="false" id="S3.Ex3.m1.2.2.1.1.2.3.2" xref="S3.Ex3.m1.2.2.1.1.2.3.2.cmml">⋃</mo><mrow id="S3.Ex3.m1.2.2.1.1.2.3.3" xref="S3.Ex3.m1.2.2.1.1.2.3.3.cmml"><mi id="S3.Ex3.m1.2.2.1.1.2.3.3.2" xref="S3.Ex3.m1.2.2.1.1.2.3.3.2.cmml">c</mi><mo id="S3.Ex3.m1.2.2.1.1.2.3.3.1" xref="S3.Ex3.m1.2.2.1.1.2.3.3.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S3.Ex3.m1.2.2.1.1.2.3.3.3" xref="S3.Ex3.m1.2.2.1.1.2.3.3.3.cmml">𝒞</mi></mrow></munder></mstyle><mrow id="S3.Ex3.m1.2.2.1.1.2.2" xref="S3.Ex3.m1.2.2.1.1.2.2.cmml"><mstyle displaystyle="true" id="S3.Ex3.m1.2.2.1.1.2.2.3" xref="S3.Ex3.m1.2.2.1.1.2.2.3.cmml"><munder id="S3.Ex3.m1.2.2.1.1.2.2.3a" xref="S3.Ex3.m1.2.2.1.1.2.2.3.cmml"><mo movablelimits="false" id="S3.Ex3.m1.2.2.1.1.2.2.3.2" xref="S3.Ex3.m1.2.2.1.1.2.2.3.2.cmml">⋃</mo><mrow id="S3.Ex3.m1.2.2.1.1.2.2.3.3" xref="S3.Ex3.m1.2.2.1.1.2.2.3.3.cmml"><msub id="S3.Ex3.m1.2.2.1.1.2.2.3.3.2" xref="S3.Ex3.m1.2.2.1.1.2.2.3.3.2.cmml"><mi id="S3.Ex3.m1.2.2.1.1.2.2.3.3.2.2" xref="S3.Ex3.m1.2.2.1.1.2.2.3.3.2.2.cmml">q</mi><mi id="S3.Ex3.m1.2.2.1.1.2.2.3.3.2.3" xref="S3.Ex3.m1.2.2.1.1.2.2.3.3.2.3.cmml">c</mi></msub><mo id="S3.Ex3.m1.2.2.1.1.2.2.3.3.1" xref="S3.Ex3.m1.2.2.1.1.2.2.3.3.1.cmml">∈</mo><msub id="S3.Ex3.m1.2.2.1.1.2.2.3.3.3" xref="S3.Ex3.m1.2.2.1.1.2.2.3.3.3.cmml"><mi id="S3.Ex3.m1.2.2.1.1.2.2.3.3.3.2" xref="S3.Ex3.m1.2.2.1.1.2.2.3.3.3.2.cmml">Q</mi><mi id="S3.Ex3.m1.2.2.1.1.2.2.3.3.3.3" xref="S3.Ex3.m1.2.2.1.1.2.2.3.3.3.3.cmml">c</mi></msub></mrow></munder></mstyle><mrow id="S3.Ex3.m1.2.2.1.1.2.2.2.2" xref="S3.Ex3.m1.2.2.1.1.2.2.2.3.cmml"><mo id="S3.Ex3.m1.2.2.1.1.2.2.2.2.3" xref="S3.Ex3.m1.2.2.1.1.2.2.2.3.1.cmml">{</mo><mrow id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.3.cmml"><mo stretchy="false" id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2.3" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.3.cmml">(</mo><msub id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.1.1" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.1.1.2" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.1.1.3" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2.4" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.3.cmml">,</mo><msub id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2.2" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2.2.cmml"><mi id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2.2.2" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2.2.2.cmml">y</mi><mi id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2.2.3" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2.2.3.cmml">i</mi></msub><mo rspace="0.278em" stretchy="false" id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2.5" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.3.cmml">)</mo></mrow><mo rspace="0.278em" id="S3.Ex3.m1.2.2.1.1.2.2.2.2.4" xref="S3.Ex3.m1.2.2.1.1.2.2.2.3.1.cmml">:</mo><mrow id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.cmml"><mrow id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.4.cmml"><mo stretchy="false" id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3.4" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.4.cmml">(</mo><msub id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.1.1.1" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.1.1.1.cmml"><mi id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.1.1.1.2" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.1.1.1.2.cmml">x</mi><mi id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.1.1.1.3" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.1.1.1.3.cmml">i</mi></msub><mo id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3.5" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.4.cmml">,</mo><msub id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.2.2.2" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.2.2.2.cmml"><mi id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.2.2.2.2" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.2.2.2.2.cmml">t</mi><mi id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.2.2.2.3" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.2.2.2.3.cmml">i</mi></msub><mo id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3.6" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.4.cmml">,</mo><msub id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3.3" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3.3.cmml"><mi id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3.3.2" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3.3.2.cmml">y</mi><mi id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3.3.3" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3.3.3.cmml">i</mi></msub><mo stretchy="false" id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3.7" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.4.cmml">)</mo></mrow><mo id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.4" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.4.cmml">∈</mo><msub id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.cmml"><mi id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.2" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.2.cmml">S</mi><msub id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.3" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.3.cmml"><mi id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.3.2" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.3.2.cmml">q</mi><mi id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.3.3" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.3.3.cmml">c</mi></msub></msub></mrow><mo id="S3.Ex3.m1.2.2.1.1.2.2.2.2.5" xref="S3.Ex3.m1.2.2.1.1.2.2.2.3.1.cmml">}</mo></mrow></mrow></mrow></mrow><mo lspace="0em" id="S3.Ex3.m1.2.2.1.2" xref="S3.Ex3.m1.2.2.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.Ex3.m1.2b"><apply id="S3.Ex3.m1.2.2.1.1.cmml" xref="S3.Ex3.m1.2.2.1"><eq id="S3.Ex3.m1.2.2.1.1.3.cmml" xref="S3.Ex3.m1.2.2.1.1.3"></eq><apply id="S3.Ex3.m1.2.2.1.1.4.cmml" xref="S3.Ex3.m1.2.2.1.1.4"><csymbol cd="ambiguous" id="S3.Ex3.m1.2.2.1.1.4.1.cmml" xref="S3.Ex3.m1.2.2.1.1.4">superscript</csymbol><apply id="S3.Ex3.m1.2.2.1.1.4.2.cmml" xref="S3.Ex3.m1.2.2.1.1.4"><csymbol cd="ambiguous" id="S3.Ex3.m1.2.2.1.1.4.2.1.cmml" xref="S3.Ex3.m1.2.2.1.1.4">subscript</csymbol><ci id="S3.Ex3.m1.2.2.1.1.4.2.2.cmml" xref="S3.Ex3.m1.2.2.1.1.4.2.2">𝒟</ci><ci id="S3.Ex3.m1.2.2.1.1.4.2.3.cmml" xref="S3.Ex3.m1.2.2.1.1.4.2.3">𝒞</ci></apply><ci id="S3.Ex3.m1.1.1.1.1a.cmml" xref="S3.Ex3.m1.1.1.1.3"><mtext mathsize="70%" id="S3.Ex3.m1.1.1.1.1.cmml" xref="S3.Ex3.m1.1.1.1.1">retrieved</mtext></ci></apply><apply id="S3.Ex3.m1.2.2.1.1.2.cmml" xref="S3.Ex3.m1.2.2.1.1.2"><apply id="S3.Ex3.m1.2.2.1.1.2.3.cmml" xref="S3.Ex3.m1.2.2.1.1.2.3"><csymbol cd="ambiguous" id="S3.Ex3.m1.2.2.1.1.2.3.1.cmml" xref="S3.Ex3.m1.2.2.1.1.2.3">subscript</csymbol><union id="S3.Ex3.m1.2.2.1.1.2.3.2.cmml" xref="S3.Ex3.m1.2.2.1.1.2.3.2"></union><apply id="S3.Ex3.m1.2.2.1.1.2.3.3.cmml" xref="S3.Ex3.m1.2.2.1.1.2.3.3"><in id="S3.Ex3.m1.2.2.1.1.2.3.3.1.cmml" xref="S3.Ex3.m1.2.2.1.1.2.3.3.1"></in><ci id="S3.Ex3.m1.2.2.1.1.2.3.3.2.cmml" xref="S3.Ex3.m1.2.2.1.1.2.3.3.2">𝑐</ci><ci id="S3.Ex3.m1.2.2.1.1.2.3.3.3.cmml" xref="S3.Ex3.m1.2.2.1.1.2.3.3.3">𝒞</ci></apply></apply><apply id="S3.Ex3.m1.2.2.1.1.2.2.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2"><apply id="S3.Ex3.m1.2.2.1.1.2.2.3.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.3"><csymbol cd="ambiguous" id="S3.Ex3.m1.2.2.1.1.2.2.3.1.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.3">subscript</csymbol><union id="S3.Ex3.m1.2.2.1.1.2.2.3.2.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.3.2"></union><apply id="S3.Ex3.m1.2.2.1.1.2.2.3.3.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.3.3"><in id="S3.Ex3.m1.2.2.1.1.2.2.3.3.1.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.3.3.1"></in><apply id="S3.Ex3.m1.2.2.1.1.2.2.3.3.2.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.3.3.2"><csymbol cd="ambiguous" id="S3.Ex3.m1.2.2.1.1.2.2.3.3.2.1.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.3.3.2">subscript</csymbol><ci id="S3.Ex3.m1.2.2.1.1.2.2.3.3.2.2.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.3.3.2.2">𝑞</ci><ci id="S3.Ex3.m1.2.2.1.1.2.2.3.3.2.3.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.3.3.2.3">𝑐</ci></apply><apply id="S3.Ex3.m1.2.2.1.1.2.2.3.3.3.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.3.3.3"><csymbol cd="ambiguous" id="S3.Ex3.m1.2.2.1.1.2.2.3.3.3.1.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.3.3.3">subscript</csymbol><ci id="S3.Ex3.m1.2.2.1.1.2.2.3.3.3.2.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.3.3.3.2">𝑄</ci><ci id="S3.Ex3.m1.2.2.1.1.2.2.3.3.3.3.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.3.3.3.3">𝑐</ci></apply></apply></apply><apply id="S3.Ex3.m1.2.2.1.1.2.2.2.3.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2"><csymbol cd="latexml" id="S3.Ex3.m1.2.2.1.1.2.2.2.3.1.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.3">conditional-set</csymbol><interval closure="open" id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.3.cmml" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2"><apply id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.1.1.cmml" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2.2.cmml" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2.2.1.cmml" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2.2">subscript</csymbol><ci id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2.2.2.cmml" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2.2.2">𝑦</ci><ci id="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2.2.3.cmml" xref="S3.Ex3.m1.2.2.1.1.1.1.1.1.1.2.2.3">𝑖</ci></apply></interval><apply id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2"><in id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.4.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.4"></in><vector id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.4.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3"><apply id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.1.1.1.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.1.1.1.1.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.1.1.1">subscript</csymbol><ci id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.1.1.1.2.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.1.1.1.2">𝑥</ci><ci id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.1.1.1.3.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.1.1.1.3">𝑖</ci></apply><apply id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.2.2.2.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.2.2.2.1.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.2.2.2">subscript</csymbol><ci id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.2.2.2.2.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.2.2.2.2">𝑡</ci><ci id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.2.2.2.3.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.2.2.2.3">𝑖</ci></apply><apply id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3.3.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3.3"><csymbol cd="ambiguous" id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3.3.1.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3.3">subscript</csymbol><ci id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3.3.2.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3.3.2">𝑦</ci><ci id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3.3.3.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.3.3.3.3">𝑖</ci></apply></vector><apply id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5"><csymbol cd="ambiguous" id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.1.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5">subscript</csymbol><ci id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.2.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.2">𝑆</ci><apply id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.3.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.3"><csymbol cd="ambiguous" id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.3.1.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.3">subscript</csymbol><ci id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.3.2.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.3.2">𝑞</ci><ci id="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.3.3.cmml" xref="S3.Ex3.m1.2.2.1.1.2.2.2.2.2.5.3.3">𝑐</ci></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.Ex3.m1.2c">\displaystyle{\mathcal{D}_{\mathcal{C}}^{(\text{retrieved})}}=\bigcup_{c\in{\mathcal{C}}}\bigcup_{q_{c}\in Q_{c}}\left\{(x_{i},y_{i})\colon(x_{i},t_{i},y_{i})\in S_{q_{c}}\right\}.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
</section>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Additional data filtering and postprocessing</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.3" class="ltx_p">Data filtering has been shown to improve training performance for both real and synthetic data and is widely used in practice <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib20" title="" class="ltx_ref">20</a>, <a href="#bib.bib24" title="" class="ltx_ref">24</a>]</cite>. We filter both our synthetic and retrieved datasets. Given a curated dataset <math id="S3.SS3.p1.1.m1.1" class="ltx_Math" alttext="{\mathcal{D}_{\mathcal{C}}}" display="inline"><semantics id="S3.SS3.p1.1.m1.1a"><msub id="S3.SS3.p1.1.m1.1.1" xref="S3.SS3.p1.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p1.1.m1.1.1.2" xref="S3.SS3.p1.1.m1.1.1.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p1.1.m1.1.1.3" xref="S3.SS3.p1.1.m1.1.1.3.cmml">𝒞</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.1.m1.1b"><apply id="S3.SS3.p1.1.m1.1.1.cmml" xref="S3.SS3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p1.1.m1.1.1.1.cmml" xref="S3.SS3.p1.1.m1.1.1">subscript</csymbol><ci id="S3.SS3.p1.1.m1.1.1.2.cmml" xref="S3.SS3.p1.1.m1.1.1.2">𝒟</ci><ci id="S3.SS3.p1.1.m1.1.1.3.cmml" xref="S3.SS3.p1.1.m1.1.1.3">𝒞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.1.m1.1c">{\mathcal{D}_{\mathcal{C}}}</annotation></semantics></math>, we compute the CLIP similarity of each <math id="S3.SS3.p1.2.m2.1" class="ltx_Math" alttext="x_{i}\in{\mathcal{D}_{\mathcal{C}}}" display="inline"><semantics id="S3.SS3.p1.2.m2.1a"><mrow id="S3.SS3.p1.2.m2.1.1" xref="S3.SS3.p1.2.m2.1.1.cmml"><msub id="S3.SS3.p1.2.m2.1.1.2" xref="S3.SS3.p1.2.m2.1.1.2.cmml"><mi id="S3.SS3.p1.2.m2.1.1.2.2" xref="S3.SS3.p1.2.m2.1.1.2.2.cmml">x</mi><mi id="S3.SS3.p1.2.m2.1.1.2.3" xref="S3.SS3.p1.2.m2.1.1.2.3.cmml">i</mi></msub><mo id="S3.SS3.p1.2.m2.1.1.1" xref="S3.SS3.p1.2.m2.1.1.1.cmml">∈</mo><msub id="S3.SS3.p1.2.m2.1.1.3" xref="S3.SS3.p1.2.m2.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p1.2.m2.1.1.3.2" xref="S3.SS3.p1.2.m2.1.1.3.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p1.2.m2.1.1.3.3" xref="S3.SS3.p1.2.m2.1.1.3.3.cmml">𝒞</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.2.m2.1b"><apply id="S3.SS3.p1.2.m2.1.1.cmml" xref="S3.SS3.p1.2.m2.1.1"><in id="S3.SS3.p1.2.m2.1.1.1.cmml" xref="S3.SS3.p1.2.m2.1.1.1"></in><apply id="S3.SS3.p1.2.m2.1.1.2.cmml" xref="S3.SS3.p1.2.m2.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p1.2.m2.1.1.2.1.cmml" xref="S3.SS3.p1.2.m2.1.1.2">subscript</csymbol><ci id="S3.SS3.p1.2.m2.1.1.2.2.cmml" xref="S3.SS3.p1.2.m2.1.1.2.2">𝑥</ci><ci id="S3.SS3.p1.2.m2.1.1.2.3.cmml" xref="S3.SS3.p1.2.m2.1.1.2.3">𝑖</ci></apply><apply id="S3.SS3.p1.2.m2.1.1.3.cmml" xref="S3.SS3.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p1.2.m2.1.1.3.1.cmml" xref="S3.SS3.p1.2.m2.1.1.3">subscript</csymbol><ci id="S3.SS3.p1.2.m2.1.1.3.2.cmml" xref="S3.SS3.p1.2.m2.1.1.3.2">𝒟</ci><ci id="S3.SS3.p1.2.m2.1.1.3.3.cmml" xref="S3.SS3.p1.2.m2.1.1.3.3">𝒞</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.2.m2.1c">x_{i}\in{\mathcal{D}_{\mathcal{C}}}</annotation></semantics></math> with text corresponding to its assigned label <math id="S3.SS3.p1.3.m3.1" class="ltx_Math" alttext="y_{i}" display="inline"><semantics id="S3.SS3.p1.3.m3.1a"><msub id="S3.SS3.p1.3.m3.1.1" xref="S3.SS3.p1.3.m3.1.1.cmml"><mi id="S3.SS3.p1.3.m3.1.1.2" xref="S3.SS3.p1.3.m3.1.1.2.cmml">y</mi><mi id="S3.SS3.p1.3.m3.1.1.3" xref="S3.SS3.p1.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p1.3.m3.1b"><apply id="S3.SS3.p1.3.m3.1.1.cmml" xref="S3.SS3.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS3.p1.3.m3.1.1.1.cmml" xref="S3.SS3.p1.3.m3.1.1">subscript</csymbol><ci id="S3.SS3.p1.3.m3.1.1.2.cmml" xref="S3.SS3.p1.3.m3.1.1.2">𝑦</ci><ci id="S3.SS3.p1.3.m3.1.1.3.cmml" xref="S3.SS3.p1.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p1.3.m3.1c">y_{i}</annotation></semantics></math> (e.g., “a photo of {class name}”), constructed using the CLIP zero-shot classification templates <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib49" title="" class="ltx_ref">49</a>]</cite>. When there are multiple templates for a given class, we aggregate by taking the maximum similarity across templates. We keep the top 30% of images by aggregate similarity. Intuitively, filtering helps remove generated and retrieved images with class-misaligned content. For example, an image labeled “dog” but without any dogs present (i.e., due to retrieval or generation errors) would receive a lower CLIP similarity score and thus likely be discarded.</p>
</div>
<div id="S3.SS3.p2" class="ltx_para">
<p id="S3.SS3.p2.5" class="ltx_p">Our synthetic adaptation datasets <math id="S3.SS3.p2.1.m1.1" class="ltx_Math" alttext="{\mathcal{D}_{\mathcal{C}}^{(\text{synthetic})}}" display="inline"><semantics id="S3.SS3.p2.1.m1.1a"><msubsup id="S3.SS3.p2.1.m1.1.2" xref="S3.SS3.p2.1.m1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.1.m1.1.2.2.2" xref="S3.SS3.p2.1.m1.1.2.2.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.1.m1.1.2.2.3" xref="S3.SS3.p2.1.m1.1.2.2.3.cmml">𝒞</mi><mrow id="S3.SS3.p2.1.m1.1.1.1.3" xref="S3.SS3.p2.1.m1.1.1.1.1a.cmml"><mo stretchy="false" id="S3.SS3.p2.1.m1.1.1.1.3.1" xref="S3.SS3.p2.1.m1.1.1.1.1a.cmml">(</mo><mtext id="S3.SS3.p2.1.m1.1.1.1.1" xref="S3.SS3.p2.1.m1.1.1.1.1.cmml">synthetic</mtext><mo stretchy="false" id="S3.SS3.p2.1.m1.1.1.1.3.2" xref="S3.SS3.p2.1.m1.1.1.1.1a.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.1.m1.1b"><apply id="S3.SS3.p2.1.m1.1.2.cmml" xref="S3.SS3.p2.1.m1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p2.1.m1.1.2.1.cmml" xref="S3.SS3.p2.1.m1.1.2">superscript</csymbol><apply id="S3.SS3.p2.1.m1.1.2.2.cmml" xref="S3.SS3.p2.1.m1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p2.1.m1.1.2.2.1.cmml" xref="S3.SS3.p2.1.m1.1.2">subscript</csymbol><ci id="S3.SS3.p2.1.m1.1.2.2.2.cmml" xref="S3.SS3.p2.1.m1.1.2.2.2">𝒟</ci><ci id="S3.SS3.p2.1.m1.1.2.2.3.cmml" xref="S3.SS3.p2.1.m1.1.2.2.3">𝒞</ci></apply><ci id="S3.SS3.p2.1.m1.1.1.1.1a.cmml" xref="S3.SS3.p2.1.m1.1.1.1.3"><mtext mathsize="70%" id="S3.SS3.p2.1.m1.1.1.1.1.cmml" xref="S3.SS3.p2.1.m1.1.1.1.1">synthetic</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.1.m1.1c">{\mathcal{D}_{\mathcal{C}}^{(\text{synthetic})}}</annotation></semantics></math> are class-balanced by construction (i.e., we uniformly generate images for each class). We further postprocess the retrieved adaptation datasets <math id="S3.SS3.p2.2.m2.1" class="ltx_Math" alttext="{\mathcal{D}_{\mathcal{C}}^{(\text{retrieved})}}" display="inline"><semantics id="S3.SS3.p2.2.m2.1a"><msubsup id="S3.SS3.p2.2.m2.1.2" xref="S3.SS3.p2.2.m2.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.2.m2.1.2.2.2" xref="S3.SS3.p2.2.m2.1.2.2.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.2.m2.1.2.2.3" xref="S3.SS3.p2.2.m2.1.2.2.3.cmml">𝒞</mi><mrow id="S3.SS3.p2.2.m2.1.1.1.3" xref="S3.SS3.p2.2.m2.1.1.1.1a.cmml"><mo stretchy="false" id="S3.SS3.p2.2.m2.1.1.1.3.1" xref="S3.SS3.p2.2.m2.1.1.1.1a.cmml">(</mo><mtext id="S3.SS3.p2.2.m2.1.1.1.1" xref="S3.SS3.p2.2.m2.1.1.1.1.cmml">retrieved</mtext><mo stretchy="false" id="S3.SS3.p2.2.m2.1.1.1.3.2" xref="S3.SS3.p2.2.m2.1.1.1.1a.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.2.m2.1b"><apply id="S3.SS3.p2.2.m2.1.2.cmml" xref="S3.SS3.p2.2.m2.1.2"><csymbol cd="ambiguous" id="S3.SS3.p2.2.m2.1.2.1.cmml" xref="S3.SS3.p2.2.m2.1.2">superscript</csymbol><apply id="S3.SS3.p2.2.m2.1.2.2.cmml" xref="S3.SS3.p2.2.m2.1.2"><csymbol cd="ambiguous" id="S3.SS3.p2.2.m2.1.2.2.1.cmml" xref="S3.SS3.p2.2.m2.1.2">subscript</csymbol><ci id="S3.SS3.p2.2.m2.1.2.2.2.cmml" xref="S3.SS3.p2.2.m2.1.2.2.2">𝒟</ci><ci id="S3.SS3.p2.2.m2.1.2.2.3.cmml" xref="S3.SS3.p2.2.m2.1.2.2.3">𝒞</ci></apply><ci id="S3.SS3.p2.2.m2.1.1.1.1a.cmml" xref="S3.SS3.p2.2.m2.1.1.1.3"><mtext mathsize="70%" id="S3.SS3.p2.2.m2.1.1.1.1.cmml" xref="S3.SS3.p2.2.m2.1.1.1.1">retrieved</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.2.m2.1c">{\mathcal{D}_{\mathcal{C}}^{(\text{retrieved})}}</annotation></semantics></math> to improve class balancing by manually fixing a global threshold <math id="S3.SS3.p2.3.m3.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S3.SS3.p2.3.m3.1a"><mi id="S3.SS3.p2.3.m3.1.1" xref="S3.SS3.p2.3.m3.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.3.m3.1b"><ci id="S3.SS3.p2.3.m3.1.1.cmml" xref="S3.SS3.p2.3.m3.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.3.m3.1c">M</annotation></semantics></math> and truncating the dataset such that each class label <math id="S3.SS3.p2.4.m4.1" class="ltx_Math" alttext="y_{i}" display="inline"><semantics id="S3.SS3.p2.4.m4.1a"><msub id="S3.SS3.p2.4.m4.1.1" xref="S3.SS3.p2.4.m4.1.1.cmml"><mi id="S3.SS3.p2.4.m4.1.1.2" xref="S3.SS3.p2.4.m4.1.1.2.cmml">y</mi><mi id="S3.SS3.p2.4.m4.1.1.3" xref="S3.SS3.p2.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.4.m4.1b"><apply id="S3.SS3.p2.4.m4.1.1.cmml" xref="S3.SS3.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.1.1.1.cmml" xref="S3.SS3.p2.4.m4.1.1">subscript</csymbol><ci id="S3.SS3.p2.4.m4.1.1.2.cmml" xref="S3.SS3.p2.4.m4.1.1.2">𝑦</ci><ci id="S3.SS3.p2.4.m4.1.1.3.cmml" xref="S3.SS3.p2.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.4.m4.1c">y_{i}</annotation></semantics></math> occurs at most <math id="S3.SS3.p2.5.m5.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S3.SS3.p2.5.m5.1a"><mi id="S3.SS3.p2.5.m5.1.1" xref="S3.SS3.p2.5.m5.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.5.m5.1b"><ci id="S3.SS3.p2.5.m5.1.1.cmml" xref="S3.SS3.p2.5.m5.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.5.m5.1c">M</annotation></semantics></math> times.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Main Experiments</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.2" class="ltx_p">We seek to measure the utility of learning from model-generated synthetic images. Grounding this question empirically, our experiments compare finetuning a pretrained CLIP model on (1) targeted synthetic images <math id="S4.p1.1.m1.1" class="ltx_Math" alttext="{\mathcal{D}_{\mathcal{C}}^{(\text{synthetic})}}" display="inline"><semantics id="S4.p1.1.m1.1a"><msubsup id="S4.p1.1.m1.1.2" xref="S4.p1.1.m1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.p1.1.m1.1.2.2.2" xref="S4.p1.1.m1.1.2.2.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S4.p1.1.m1.1.2.2.3" xref="S4.p1.1.m1.1.2.2.3.cmml">𝒞</mi><mrow id="S4.p1.1.m1.1.1.1.3" xref="S4.p1.1.m1.1.1.1.1a.cmml"><mo stretchy="false" id="S4.p1.1.m1.1.1.1.3.1" xref="S4.p1.1.m1.1.1.1.1a.cmml">(</mo><mtext id="S4.p1.1.m1.1.1.1.1" xref="S4.p1.1.m1.1.1.1.1.cmml">synthetic</mtext><mo stretchy="false" id="S4.p1.1.m1.1.1.1.3.2" xref="S4.p1.1.m1.1.1.1.1a.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S4.p1.1.m1.1b"><apply id="S4.p1.1.m1.1.2.cmml" xref="S4.p1.1.m1.1.2"><csymbol cd="ambiguous" id="S4.p1.1.m1.1.2.1.cmml" xref="S4.p1.1.m1.1.2">superscript</csymbol><apply id="S4.p1.1.m1.1.2.2.cmml" xref="S4.p1.1.m1.1.2"><csymbol cd="ambiguous" id="S4.p1.1.m1.1.2.2.1.cmml" xref="S4.p1.1.m1.1.2">subscript</csymbol><ci id="S4.p1.1.m1.1.2.2.2.cmml" xref="S4.p1.1.m1.1.2.2.2">𝒟</ci><ci id="S4.p1.1.m1.1.2.2.3.cmml" xref="S4.p1.1.m1.1.2.2.3">𝒞</ci></apply><ci id="S4.p1.1.m1.1.1.1.1a.cmml" xref="S4.p1.1.m1.1.1.1.3"><mtext mathsize="70%" id="S4.p1.1.m1.1.1.1.1.cmml" xref="S4.p1.1.m1.1.1.1.1">synthetic</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.1.m1.1c">{\mathcal{D}_{\mathcal{C}}^{(\text{synthetic})}}</annotation></semantics></math> to (2) finetuning on targeted retrieved real images <math id="S4.p1.2.m2.1" class="ltx_Math" alttext="{\mathcal{D}_{\mathcal{C}}^{(\text{retrieved})}}" display="inline"><semantics id="S4.p1.2.m2.1a"><msubsup id="S4.p1.2.m2.1.2" xref="S4.p1.2.m2.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.p1.2.m2.1.2.2.2" xref="S4.p1.2.m2.1.2.2.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S4.p1.2.m2.1.2.2.3" xref="S4.p1.2.m2.1.2.2.3.cmml">𝒞</mi><mrow id="S4.p1.2.m2.1.1.1.3" xref="S4.p1.2.m2.1.1.1.1a.cmml"><mo stretchy="false" id="S4.p1.2.m2.1.1.1.3.1" xref="S4.p1.2.m2.1.1.1.1a.cmml">(</mo><mtext id="S4.p1.2.m2.1.1.1.1" xref="S4.p1.2.m2.1.1.1.1.cmml">retrieved</mtext><mo stretchy="false" id="S4.p1.2.m2.1.1.1.3.2" xref="S4.p1.2.m2.1.1.1.1a.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S4.p1.2.m2.1b"><apply id="S4.p1.2.m2.1.2.cmml" xref="S4.p1.2.m2.1.2"><csymbol cd="ambiguous" id="S4.p1.2.m2.1.2.1.cmml" xref="S4.p1.2.m2.1.2">superscript</csymbol><apply id="S4.p1.2.m2.1.2.2.cmml" xref="S4.p1.2.m2.1.2"><csymbol cd="ambiguous" id="S4.p1.2.m2.1.2.2.1.cmml" xref="S4.p1.2.m2.1.2">subscript</csymbol><ci id="S4.p1.2.m2.1.2.2.2.cmml" xref="S4.p1.2.m2.1.2.2.2">𝒟</ci><ci id="S4.p1.2.m2.1.2.2.3.cmml" xref="S4.p1.2.m2.1.2.2.3">𝒞</ci></apply><ci id="S4.p1.2.m2.1.1.1.1a.cmml" xref="S4.p1.2.m2.1.1.1.3"><mtext mathsize="70%" id="S4.p1.2.m2.1.1.1.1.cmml" xref="S4.p1.2.m2.1.1.1.1">retrieved</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.2.m2.1c">{\mathcal{D}_{\mathcal{C}}^{(\text{retrieved})}}</annotation></semantics></math>.</p>
</div>
<div id="S4.p2" class="ltx_para">
<p id="S4.p2.1" class="ltx_p"><span id="S4.p2.1.1" class="ltx_text ltx_font_bold">Benchmarks.</span> We focus on five downstream tasks where synthetic data has shown promise compared to similar scale <span id="S4.p2.1.2" class="ltx_text ltx_font_italic">untargeted</span> real data <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite>. We select (a) ImageNet-1K <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib14" title="" class="ltx_ref">14</a>]</cite> and Describable Textures (DTD) <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib12" title="" class="ltx_ref">12</a>]</cite> to evaluate recognition performance on broad categories and (b)
FGVC-Aircraft <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib40" title="" class="ltx_ref">40</a>]</cite>, StanfordCars <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib35" title="" class="ltx_ref">35</a>]</cite>, and Oxford Flowers102 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib45" title="" class="ltx_ref">45</a>]</cite> to evaluate performance in fine-grained settings. We use standard pre-defined train, test, and validation splits when available, and otherwise randomly subset the training set to create missing train-validation splits (details in Appendix <a href="#A3.SS2" title="C.2 Details for downstream benchmarks ‣ Appendix C Details in Experimental Setup ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C.2</span></a>).</p>
</div>
<figure id="S4.F2" class="ltx_figure"><img src="/html/2406.05184/assets/x2.png" id="S4.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="216" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>We adapt a pretrained CLIP image encoder (<span id="S4.F2.9.1" class="ltx_text ltx_font_bold" style="color:#D285C2;">dashed purple line</span>) to different downstream image classification tasks, using either (a) targeted synthetic data (<span id="S4.F2.10.2" class="ltx_text ltx_font_bold" style="color:#DF670B;">orange triangles</span>) generated from a Stable Diffusion model trained on LAION-2B or using (b) targeted real data (<span id="S4.F2.11.3" class="ltx_text ltx_font_bold" style="color:#0174B2;">blue circles</span>) directly retrieved from LAION-2B. We measure performance via downstream <span id="S4.F2.12.4" class="ltx_text ltx_font_bold">zero-shot (ZS)</span> and <span id="S4.F2.13.5" class="ltx_text ltx_font_bold">linear probing (LP)</span> accuracy, aggregating results over at least 3 seeds (error bars indicate <math id="S4.F2.2.m1.1" class="ltx_Math" alttext="\pm 1" display="inline"><semantics id="S4.F2.2.m1.1b"><mrow id="S4.F2.2.m1.1.1" xref="S4.F2.2.m1.1.1.cmml"><mo id="S4.F2.2.m1.1.1b" xref="S4.F2.2.m1.1.1.cmml">±</mo><mn id="S4.F2.2.m1.1.1.2" xref="S4.F2.2.m1.1.1.2.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.F2.2.m1.1c"><apply id="S4.F2.2.m1.1.1.cmml" xref="S4.F2.2.m1.1.1"><csymbol cd="latexml" id="S4.F2.2.m1.1.1.1.cmml" xref="S4.F2.2.m1.1.1">plus-or-minus</csymbol><cn type="integer" id="S4.F2.2.m1.1.1.2.cmml" xref="S4.F2.2.m1.1.1.2">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.F2.2.m1.1d">\pm 1</annotation></semantics></math> standard deviation). Overall, while adapting CLIP with targeted synthetic data can sometimes improve performance over an off-the-shelf model, synthetic data is universally outperformed or matched by targeted real data. This gap persists even when we scale the sample size of the synthetic adaptation dataset beyond the maximum amount of (finite) targeted real data considered (<span id="S4.F2.14.6" class="ltx_text ltx_font_bold" style="color:#6B6B6B;">gray shaded regions</span>).

</figcaption>
</figure>
<div id="S4.p3" class="ltx_para">
<p id="S4.p3.1" class="ltx_p"><span id="S4.p3.1.1" class="ltx_text ltx_font_bold">Finetuning data curation.</span> For each downstream benchmark, we first curate an adaptation dataset <math id="S4.p3.1.m1.1" class="ltx_Math" alttext="{\mathcal{D}_{\mathcal{C}}}" display="inline"><semantics id="S4.p3.1.m1.1a"><msub id="S4.p3.1.m1.1.1" xref="S4.p3.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.p3.1.m1.1.1.2" xref="S4.p3.1.m1.1.1.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S4.p3.1.m1.1.1.3" xref="S4.p3.1.m1.1.1.3.cmml">𝒞</mi></msub><annotation-xml encoding="MathML-Content" id="S4.p3.1.m1.1b"><apply id="S4.p3.1.m1.1.1.cmml" xref="S4.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S4.p3.1.m1.1.1.1.cmml" xref="S4.p3.1.m1.1.1">subscript</csymbol><ci id="S4.p3.1.m1.1.1.2.cmml" xref="S4.p3.1.m1.1.1.2">𝒟</ci><ci id="S4.p3.1.m1.1.1.3.cmml" xref="S4.p3.1.m1.1.1.3">𝒞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p3.1.m1.1c">{\mathcal{D}_{\mathcal{C}}}</annotation></semantics></math> (Section <a href="#S3" title="3 Problem Setting and Method ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>) by either (1) generating synthetic images with Stable Diffusion 1.5 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib52" title="" class="ltx_ref">52</a>]</cite>, trained on the LAION-2B dataset <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib55" title="" class="ltx_ref">55</a>]</cite>, or (2) retrieving real images directly from LAION-2B. We treat the choice between our substring-based and semantic retrieval strategies as a hyperparameter, using downstream validation set accuracy to determine the best choice for each benchmark. Hyperparameters for retrieval are detailed in Appendix <a href="#A3.SS1" title="C.1 Retrieval hyperparameters ‣ Appendix C Details in Experimental Setup ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C.1</span></a>.</p>
</div>
<div id="S4.p4" class="ltx_para">
<p id="S4.p4.2" class="ltx_p"><span id="S4.p4.2.1" class="ltx_text ltx_font_bold">Model adaptation and evaluation.</span> We adapt a LAION-2B pretrained CLIP ViT-B/16 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>]</cite> image encoder by finetuning on the curated adaptation dataset <math id="S4.p4.1.m1.1" class="ltx_Math" alttext="{\mathcal{D}_{\mathcal{C}}}" display="inline"><semantics id="S4.p4.1.m1.1a"><msub id="S4.p4.1.m1.1.1" xref="S4.p4.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.p4.1.m1.1.1.2" xref="S4.p4.1.m1.1.1.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S4.p4.1.m1.1.1.3" xref="S4.p4.1.m1.1.1.3.cmml">𝒞</mi></msub><annotation-xml encoding="MathML-Content" id="S4.p4.1.m1.1b"><apply id="S4.p4.1.m1.1.1.cmml" xref="S4.p4.1.m1.1.1"><csymbol cd="ambiguous" id="S4.p4.1.m1.1.1.1.cmml" xref="S4.p4.1.m1.1.1">subscript</csymbol><ci id="S4.p4.1.m1.1.1.2.cmml" xref="S4.p4.1.m1.1.1.2">𝒟</ci><ci id="S4.p4.1.m1.1.1.3.cmml" xref="S4.p4.1.m1.1.1.3">𝒞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.1.m1.1c">{\mathcal{D}_{\mathcal{C}}}</annotation></semantics></math> with a cross-entropy classification loss for a pre-set number of epochs. To elucidate the scaling trends of synthetic and retrieved data, we finetune across an order of magnitude of different adaptation dataset scales, subsampled from the full targeted adaptation dataset <math id="S4.p4.2.m2.1" class="ltx_Math" alttext="{\mathcal{D}_{\mathcal{C}}}" display="inline"><semantics id="S4.p4.2.m2.1a"><msub id="S4.p4.2.m2.1.1" xref="S4.p4.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.p4.2.m2.1.1.2" xref="S4.p4.2.m2.1.1.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S4.p4.2.m2.1.1.3" xref="S4.p4.2.m2.1.1.3.cmml">𝒞</mi></msub><annotation-xml encoding="MathML-Content" id="S4.p4.2.m2.1b"><apply id="S4.p4.2.m2.1.1.cmml" xref="S4.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S4.p4.2.m2.1.1.1.cmml" xref="S4.p4.2.m2.1.1">subscript</csymbol><ci id="S4.p4.2.m2.1.1.2.cmml" xref="S4.p4.2.m2.1.1.2">𝒟</ci><ci id="S4.p4.2.m2.1.1.3.cmml" xref="S4.p4.2.m2.1.1.3">𝒞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p4.2.m2.1c">{\mathcal{D}_{\mathcal{C}}}</annotation></semantics></math>. We report zero-shot (ZS) and linear probing (LP) test set accuracy, using the benchmark train set to train the linear head. For both LP and ZS evaluation, we use the validation set to identify the best epoch and finetuning hyperparameters.
For each data scale, we aggregate accuracy across the results of at least three random seeds, and report the standard deviation due to seed randomness. Additional training and full hyperparameter details are provided in Appendix <a href="#A3.SS3" title="C.3 Details for model adaptation ‣ Appendix C Details in Experimental Setup ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C.3</span></a>.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Main results: synthetic training data lags behind a baseline of retrieved real images</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">We present our main zero-shot and linear probing scaling results in Figure <a href="#S4.F2" title="Figure 2 ‣ 4 Main Experiments ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
<section id="S4.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">At equivalent data scales, finetuning with model-generated synthetic images can help, but is universally matched or outperformed by finetuning directly with images from the generator’s training data.</h4>

<div id="S4.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S4.SS1.SSS0.Px1.p1.2" class="ltx_p">Consistent with prior research <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite>, we find that training with targeted synthetic data can improve an unadapted model. For example, on FGVC-Aircraft—the setting where previous works have found strongest gains—finetuning with 139K Stable-Diffusion generated images improved downstream linear probing accuracy by an average of 3.8 percentage points over an off-the-shelf CLIP model (<math id="S4.SS1.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="64.9\%\rightarrow 68.7\%" display="inline"><semantics id="S4.SS1.SSS0.Px1.p1.1.m1.1a"><mrow id="S4.SS1.SSS0.Px1.p1.1.m1.1.1" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.cmml"><mrow id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.cmml"><mn id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.2" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.2.cmml">64.9</mn><mo id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.1" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.1.cmml">%</mo></mrow><mo stretchy="false" id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.1" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.1.cmml">→</mo><mrow id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.3" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.3.cmml"><mn id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.3.2" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.3.2.cmml">68.7</mn><mo id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.3.1" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p1.1.m1.1b"><apply id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1"><ci id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.1">→</ci><apply id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2"><csymbol cd="latexml" id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.1.cmml" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.2.cmml" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.2.2">64.9</cn></apply><apply id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.3"><csymbol cd="latexml" id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.3.1.cmml" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.3.1">percent</csymbol><cn type="float" id="S4.SS1.SSS0.Px1.p1.1.m1.1.1.3.2.cmml" xref="S4.SS1.SSS0.Px1.p1.1.m1.1.1.3.2">68.7</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p1.1.m1.1c">64.9\%\rightarrow 68.7\%</annotation></semantics></math>); on DTD, training with 110K synthetic images improves zero-shot accuracy by 3.3 points (<math id="S4.SS1.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="56.3\%\rightarrow 59.6\%" display="inline"><semantics id="S4.SS1.SSS0.Px1.p1.2.m2.1a"><mrow id="S4.SS1.SSS0.Px1.p1.2.m2.1.1" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.cmml"><mrow id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.cmml"><mn id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.2" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.2.cmml">56.3</mn><mo id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.1" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.1.cmml">%</mo></mrow><mo stretchy="false" id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.1" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.1.cmml">→</mo><mrow id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.3" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.3.cmml"><mn id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.3.2" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.3.2.cmml">59.6</mn><mo id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.3.1" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p1.2.m2.1b"><apply id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1"><ci id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.1">→</ci><apply id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.cmml" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2"><csymbol cd="latexml" id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.1.cmml" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.2.cmml" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.2.2">56.3</cn></apply><apply id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.3.cmml" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.3"><csymbol cd="latexml" id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.3.1.cmml" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.3.1">percent</csymbol><cn type="float" id="S4.SS1.SSS0.Px1.p1.2.m2.1.1.3.2.cmml" xref="S4.SS1.SSS0.Px1.p1.2.m2.1.1.3.2">59.6</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p1.2.m2.1c">56.3\%\rightarrow 59.6\%</annotation></semantics></math>).</p>
</div>
<div id="S4.SS1.SSS0.Px1.p2" class="ltx_para">
<p id="S4.SS1.SSS0.Px1.p2.1" class="ltx_p">However, the gains from training on synthetic data are consistently matched or surpassed by training on retrieved real data. For instance, on FGVC-Aircraft, finetuning with an equivalent 139K LAION-2B retrieved images boosts performance by a massive 17.8 points (<math id="S4.SS1.SSS0.Px1.p2.1.m1.1" class="ltx_Math" alttext="64.9\%\rightarrow 82.7\%" display="inline"><semantics id="S4.SS1.SSS0.Px1.p2.1.m1.1a"><mrow id="S4.SS1.SSS0.Px1.p2.1.m1.1.1" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.cmml"><mrow id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.2" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.2.cmml"><mn id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.2.2" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.2.2.cmml">64.9</mn><mo id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.2.1" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.2.1.cmml">%</mo></mrow><mo stretchy="false" id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.1" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.1.cmml">→</mo><mrow id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.3" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.3.cmml"><mn id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.3.2" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.3.2.cmml">82.7</mn><mo id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.3.1" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px1.p2.1.m1.1b"><apply id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1"><ci id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.1.cmml" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.1">→</ci><apply id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.2.cmml" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.2"><csymbol cd="latexml" id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.2.1.cmml" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.2.2.cmml" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.2.2">64.9</cn></apply><apply id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.3.cmml" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.3"><csymbol cd="latexml" id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.3.1.cmml" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.3.1">percent</csymbol><cn type="float" id="S4.SS1.SSS0.Px1.p2.1.m1.1.1.3.2.cmml" xref="S4.SS1.SSS0.Px1.p2.1.m1.1.1.3.2">82.7</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px1.p2.1.m1.1c">64.9\%\rightarrow 82.7\%</annotation></semantics></math>). Moreover, adapting with retrieved data can improve performance even when synthetic data does not (e.g., on ImageNet and Flowers102 zero-shot accuracy.) Finally, adapting with synthetic data can sometimes even <span id="S4.SS1.SSS0.Px1.p2.1.1" class="ltx_text ltx_font_italic">hurt</span> performance (ImageNet, StanfordCars, Flowers102 zero-shot), while targeted retrieved data improves or at least does not hurt performance across all settings considered. Given equal amounts of targeted retrieved and synthetic data, retrieved data is the clear winner.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Synthetic data can sometimes decrease the gap with retrieved data given additional scale, but remains behind.</h4>

<div id="S4.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S4.SS1.SSS0.Px2.p1.1" class="ltx_p">The amount of data we can retrieve is fundamentally limited by the finite upstream data pool. For example, even after searching all 2 billion LAION-2B samples for images containing an FGVC-Aircraft class name in the caption, substring-based retrieval returned only 139K targeted images post-filtering. In contrast, it is straightforward to create ever-larger synthetic datasets by simply generating more data.</p>
</div>
<div id="S4.SS1.SSS0.Px2.p2" class="ltx_para">
<p id="S4.SS1.SSS0.Px2.p2.1" class="ltx_p">Scaling the synthetic adaptation dataset size beyond the amount of retrieved data considered (illustrated in the gray-shaded regions of Figure <a href="#S4.F2" title="Figure 2 ‣ 4 Main Experiments ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>), we find that increasing the amount of targeted synthetic data does not always improve performance. For example, on DTD, synthetic data exhibits U-shaped scaling, with performance positively scaling up to 110K synthetic training images, after which performance declines. On ImageNet, Flowers102, and StanfordCars, increasing the synthetic dataset size consistently hurts zero-shot accuracy and has minimal impact on linear probing performance.</p>
</div>
<div id="S4.SS1.SSS0.Px2.p3" class="ltx_para">
<p id="S4.SS1.SSS0.Px2.p3.3" class="ltx_p">On Aircraft, scaling helps; there is a log-linear relationship between the size of the synthetic adaptation dataset and downstream linear probing accuracy (<span id="S4.SS1.SSS0.Px2.p3.3.1" class="ltx_text ltx_font_italic">e.g.</span>, scaling from 139K <math id="S4.SS1.SSS0.Px2.p3.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S4.SS1.SSS0.Px2.p3.1.m1.1a"><mo stretchy="false" id="S4.SS1.SSS0.Px2.p3.1.m1.1.1" xref="S4.SS1.SSS0.Px2.p3.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p3.1.m1.1b"><ci id="S4.SS1.SSS0.Px2.p3.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p3.1.m1.1c">\rightarrow</annotation></semantics></math> 250K synthetic images improves linear probing accuracy from <math id="S4.SS1.SSS0.Px2.p3.2.m2.1" class="ltx_Math" alttext="68.7\%\rightarrow 70.7\%" display="inline"><semantics id="S4.SS1.SSS0.Px2.p3.2.m2.1a"><mrow id="S4.SS1.SSS0.Px2.p3.2.m2.1.1" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.cmml"><mrow id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2.cmml"><mn id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2.2" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2.2.cmml">68.7</mn><mo id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2.1" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2.1.cmml">%</mo></mrow><mo stretchy="false" id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.1" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.1.cmml">→</mo><mrow id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3.cmml"><mn id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3.2" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3.2.cmml">70.7</mn><mo id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3.1" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p3.2.m2.1b"><apply id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1"><ci id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.1">→</ci><apply id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2"><csymbol cd="latexml" id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2.1.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2.1">percent</csymbol><cn type="float" id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2.2.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.2.2">68.7</cn></apply><apply id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3"><csymbol cd="latexml" id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3.1.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3.1">percent</csymbol><cn type="float" id="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3.2.cmml" xref="S4.SS1.SSS0.Px2.p3.2.m2.1.1.3.2">70.7</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p3.2.m2.1c">68.7\%\rightarrow 70.7\%</annotation></semantics></math>). However, synthetic data still lags behind retrieved data: matching the performance of a mere 15K retrieved aircraft images requires scaling the synthetic dataset to 500K images, reflecting a <math id="S4.SS1.SSS0.Px2.p3.3.m3.1" class="ltx_Math" alttext="\sim" display="inline"><semantics id="S4.SS1.SSS0.Px2.p3.3.m3.1a"><mo id="S4.SS1.SSS0.Px2.p3.3.m3.1.1" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1.cmml">∼</mo><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px2.p3.3.m3.1b"><csymbol cd="latexml" id="S4.SS1.SSS0.Px2.p3.3.m3.1.1.cmml" xref="S4.SS1.SSS0.Px2.p3.3.m3.1.1">similar-to</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px2.p3.3.m3.1c">\sim</annotation></semantics></math>33x difference in dataset size and required finetuning compute. Naively extrapolating this ratio, matching the performance of the full 139K retrieved adaptation dataset would require nearly 5M synthetic images after top 30% filtering. We note, however, that synthetic data is unlikely to truly scale infinitely, as synthetic data fundamentally derives from the (finite) training set of our generative model. Still, the performance of synthetic data is likely unsaturated at the 500K scale (<span id="S4.SS1.SSS0.Px2.p3.3.2" class="ltx_text ltx_font_italic">i.e.</span>, accuracy is still trending up); due to compute limitations, studying whether further scaling can outperform retrieved data is left for future work.</p>
</div>
</section>
<section id="S4.SS1.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Synthetic data can improve a model’s task representation without significantly improving the model’s task performance.</h4>

<div id="S4.SS1.SSS0.Px3.p1" class="ltx_para">
<p id="S4.SS1.SSS0.Px3.p1.3" class="ltx_p">Broadly speaking, zero-shot task accuracy measures a model’s ability to directly solve the downstream task, whereas linear probing accuracy measures the quality of the model’s learned <span id="S4.SS1.SSS0.Px3.p1.3.1" class="ltx_text ltx_font_italic">task-relevant representation</span>. We find that even when training on synthetic data improves the model’s representation (i.e., downstream linear probing accuracy), it may not significantly improve the model’s zero-shot accuracy. In contrast, when training on retrieved data improves the model’s representation, zero-shot accuracy also exhibits positive scaling. For example, CLIP adapted with either 15K retrieved images or 500K synthetic images both achieve a similar linear probing accuracy (<math id="S4.SS1.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="\sim 72\%" display="inline"><semantics id="S4.SS1.SSS0.Px3.p1.1.m1.1a"><mrow id="S4.SS1.SSS0.Px3.p1.1.m1.1.1" xref="S4.SS1.SSS0.Px3.p1.1.m1.1.1.cmml"><mi id="S4.SS1.SSS0.Px3.p1.1.m1.1.1.2" xref="S4.SS1.SSS0.Px3.p1.1.m1.1.1.2.cmml"></mi><mo id="S4.SS1.SSS0.Px3.p1.1.m1.1.1.1" xref="S4.SS1.SSS0.Px3.p1.1.m1.1.1.1.cmml">∼</mo><mrow id="S4.SS1.SSS0.Px3.p1.1.m1.1.1.3" xref="S4.SS1.SSS0.Px3.p1.1.m1.1.1.3.cmml"><mn id="S4.SS1.SSS0.Px3.p1.1.m1.1.1.3.2" xref="S4.SS1.SSS0.Px3.p1.1.m1.1.1.3.2.cmml">72</mn><mo id="S4.SS1.SSS0.Px3.p1.1.m1.1.1.3.1" xref="S4.SS1.SSS0.Px3.p1.1.m1.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px3.p1.1.m1.1b"><apply id="S4.SS1.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S4.SS1.SSS0.Px3.p1.1.m1.1.1"><csymbol cd="latexml" id="S4.SS1.SSS0.Px3.p1.1.m1.1.1.1.cmml" xref="S4.SS1.SSS0.Px3.p1.1.m1.1.1.1">similar-to</csymbol><csymbol cd="latexml" id="S4.SS1.SSS0.Px3.p1.1.m1.1.1.2.cmml" xref="S4.SS1.SSS0.Px3.p1.1.m1.1.1.2">absent</csymbol><apply id="S4.SS1.SSS0.Px3.p1.1.m1.1.1.3.cmml" xref="S4.SS1.SSS0.Px3.p1.1.m1.1.1.3"><csymbol cd="latexml" id="S4.SS1.SSS0.Px3.p1.1.m1.1.1.3.1.cmml" xref="S4.SS1.SSS0.Px3.p1.1.m1.1.1.3.1">percent</csymbol><cn type="integer" id="S4.SS1.SSS0.Px3.p1.1.m1.1.1.3.2.cmml" xref="S4.SS1.SSS0.Px3.p1.1.m1.1.1.3.2">72</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px3.p1.1.m1.1c">\sim 72\%</annotation></semantics></math>), yet the model adapted with synthetic data achieves a much worse zero-shot accuracy (<math id="S4.SS1.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="28.9\%" display="inline"><semantics id="S4.SS1.SSS0.Px3.p1.2.m2.1a"><mrow id="S4.SS1.SSS0.Px3.p1.2.m2.1.1" xref="S4.SS1.SSS0.Px3.p1.2.m2.1.1.cmml"><mn id="S4.SS1.SSS0.Px3.p1.2.m2.1.1.2" xref="S4.SS1.SSS0.Px3.p1.2.m2.1.1.2.cmml">28.9</mn><mo id="S4.SS1.SSS0.Px3.p1.2.m2.1.1.1" xref="S4.SS1.SSS0.Px3.p1.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px3.p1.2.m2.1b"><apply id="S4.SS1.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S4.SS1.SSS0.Px3.p1.2.m2.1.1"><csymbol cd="latexml" id="S4.SS1.SSS0.Px3.p1.2.m2.1.1.1.cmml" xref="S4.SS1.SSS0.Px3.p1.2.m2.1.1.1">percent</csymbol><cn type="float" id="S4.SS1.SSS0.Px3.p1.2.m2.1.1.2.cmml" xref="S4.SS1.SSS0.Px3.p1.2.m2.1.1.2">28.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px3.p1.2.m2.1c">28.9\%</annotation></semantics></math> versus <math id="S4.SS1.SSS0.Px3.p1.3.m3.1" class="ltx_Math" alttext="39.5\%" display="inline"><semantics id="S4.SS1.SSS0.Px3.p1.3.m3.1a"><mrow id="S4.SS1.SSS0.Px3.p1.3.m3.1.1" xref="S4.SS1.SSS0.Px3.p1.3.m3.1.1.cmml"><mn id="S4.SS1.SSS0.Px3.p1.3.m3.1.1.2" xref="S4.SS1.SSS0.Px3.p1.3.m3.1.1.2.cmml">39.5</mn><mo id="S4.SS1.SSS0.Px3.p1.3.m3.1.1.1" xref="S4.SS1.SSS0.Px3.p1.3.m3.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS0.Px3.p1.3.m3.1b"><apply id="S4.SS1.SSS0.Px3.p1.3.m3.1.1.cmml" xref="S4.SS1.SSS0.Px3.p1.3.m3.1.1"><csymbol cd="latexml" id="S4.SS1.SSS0.Px3.p1.3.m3.1.1.1.cmml" xref="S4.SS1.SSS0.Px3.p1.3.m3.1.1.1">percent</csymbol><cn type="float" id="S4.SS1.SSS0.Px3.p1.3.m3.1.1.2.cmml" xref="S4.SS1.SSS0.Px3.p1.3.m3.1.1.2">39.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS0.Px3.p1.3.m3.1c">39.5\%</annotation></semantics></math>). We discuss possible reasons for this qualitative discrepancy in model behavior in our analyses below (Section <a href="#S5.SS1" title="5.1 Why does synthetic data lag retrieved real data? ‣ 5 Analysis ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.1</span></a>).</p>
</div>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Analysis</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">In this section, we explore two questions to better understand our main results. First, what factors drive the underperformance of synthetic data? Second, do our findings hold under variations of our experimental setup? We focus our analysis experiments on ImageNet, to understand general image recognition performance, and FGVC-Aircraft, the sole benchmark where synthetic data exhibited strong positive log-linear scaling.</p>
</div>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Why does synthetic data lag retrieved real data?</h3>

<section id="S5.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Qualitative visualizations.</h4>

<div id="S5.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px1.p1.1" class="ltx_p">We visualize a random selection of images from our curated synthetic and retrieved adaptation datasets in Figure <a href="#S5.F3" title="Figure 3 ‣ Qualitative visualizations. ‣ 5.1 Why does synthetic data lag retrieved real data? ‣ 5 Analysis ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>. Compared to retrieved real images, we observe that the synthetic images (1) contain low-level generator artifacts, and (2) differ in visual content distribution, both in terms of semantic details and overall image composition. For example, although the synthetic FGVC-Aircraft adaptation images (top two rows of Figure <a href="#S5.F3" title="Figure 3 ‣ Qualitative visualizations. ‣ 5.1 Why does synthetic data lag retrieved real data? ‣ 5 Analysis ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>) are recognizable as airplanes, the visual content often contains incorrect class-relevant semantic details: a correctly-depicted “<span id="S5.SS1.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_typewriter">Airbus A320</span>” should have one engine per wing and two sets of wheels at its rear, yet our synthetic images often exhibit incorrect engine or wheel configurations. This qualitative discrepancy in visual detail precision may partially explain why training on synthetic data does not improve task zero-shot accuracy; synthetic images do not retain enough class-accurate details to directly teach the model the downstream task. In contrast, training on synthetic images can improve linear probing accuracy, because the synthetic images still broadly look like aircraft and thus may help align the model’s representation to the downstream domain.</p>
</div>
<figure id="S5.F3" class="ltx_figure"><img src="/html/2406.05184/assets/x3.png" id="S5.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="280" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>We visualize <span id="S5.F3.7.1" class="ltx_text ltx_font_bold" style="color:#0174B2;">retrieved real images</span> and <span id="S5.F3.8.2" class="ltx_text ltx_font_bold" style="color:#DF670B;">synthetic images</span> from our targeted adaptation datasets for FGVC-Aircraft (top two rows) and ImageNet-1K (bottom two rows), alongside ground truth images (left column) for reference. Compared to retrieved images, synthetic images often (1) contain generator artifacts (e.g., the blur on the edges of the “<span id="S5.F3.9.3" class="ltx_text ltx_font_typewriter">Cessna 172</span>”, the eyes and mouth of the “<span id="S5.F3.10.4" class="ltx_text ltx_font_typewriter">Tabby Cat</span>”) and also (2) distort class-relevant visual content, such as the engine configuration of a true “<span id="S5.F3.11.5" class="ltx_text ltx_font_typewriter">Airbus A320</span>” (i.e., exactly one engine per wing) and the entire visual appearance of a “<span id="S5.F3.12.6" class="ltx_text ltx_font_typewriter">Flute</span>”. We hypothesize that both factors contribute to synthetic training data’s underperformance versus real training data.

</figcaption>
</figure>
</section>
<section id="S5.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Synthetically perturbing retrieved real images.</h4>

<div id="S5.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S5.SS1.SSS0.Px2.p1.9" class="ltx_p">To disentangle the effect of low-level generator artifacts and visual content differences between synthetic and retrieved real images on downstream model performance, we trained on “hybrid” images that have similar semantic visual content as our retrieved real images but contain generator artifacts like our synthetic images. Following SDEdit <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib42" title="" class="ltx_ref">42</a>]</cite>, we use Stable Diffusion to synthetically perturb our retrieved images to introduce model-specific artifacts present in the synthetic images Stable Diffusion generates. Given a noise strength parameter <math id="S5.SS1.SSS0.Px2.p1.1.m1.2" class="ltx_Math" alttext="\gamma\in[0,1]" display="inline"><semantics id="S5.SS1.SSS0.Px2.p1.1.m1.2a"><mrow id="S5.SS1.SSS0.Px2.p1.1.m1.2.3" xref="S5.SS1.SSS0.Px2.p1.1.m1.2.3.cmml"><mi id="S5.SS1.SSS0.Px2.p1.1.m1.2.3.2" xref="S5.SS1.SSS0.Px2.p1.1.m1.2.3.2.cmml">γ</mi><mo id="S5.SS1.SSS0.Px2.p1.1.m1.2.3.1" xref="S5.SS1.SSS0.Px2.p1.1.m1.2.3.1.cmml">∈</mo><mrow id="S5.SS1.SSS0.Px2.p1.1.m1.2.3.3.2" xref="S5.SS1.SSS0.Px2.p1.1.m1.2.3.3.1.cmml"><mo stretchy="false" id="S5.SS1.SSS0.Px2.p1.1.m1.2.3.3.2.1" xref="S5.SS1.SSS0.Px2.p1.1.m1.2.3.3.1.cmml">[</mo><mn id="S5.SS1.SSS0.Px2.p1.1.m1.1.1" xref="S5.SS1.SSS0.Px2.p1.1.m1.1.1.cmml">0</mn><mo id="S5.SS1.SSS0.Px2.p1.1.m1.2.3.3.2.2" xref="S5.SS1.SSS0.Px2.p1.1.m1.2.3.3.1.cmml">,</mo><mn id="S5.SS1.SSS0.Px2.p1.1.m1.2.2" xref="S5.SS1.SSS0.Px2.p1.1.m1.2.2.cmml">1</mn><mo stretchy="false" id="S5.SS1.SSS0.Px2.p1.1.m1.2.3.3.2.3" xref="S5.SS1.SSS0.Px2.p1.1.m1.2.3.3.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px2.p1.1.m1.2b"><apply id="S5.SS1.SSS0.Px2.p1.1.m1.2.3.cmml" xref="S5.SS1.SSS0.Px2.p1.1.m1.2.3"><in id="S5.SS1.SSS0.Px2.p1.1.m1.2.3.1.cmml" xref="S5.SS1.SSS0.Px2.p1.1.m1.2.3.1"></in><ci id="S5.SS1.SSS0.Px2.p1.1.m1.2.3.2.cmml" xref="S5.SS1.SSS0.Px2.p1.1.m1.2.3.2">𝛾</ci><interval closure="closed" id="S5.SS1.SSS0.Px2.p1.1.m1.2.3.3.1.cmml" xref="S5.SS1.SSS0.Px2.p1.1.m1.2.3.3.2"><cn type="integer" id="S5.SS1.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S5.SS1.SSS0.Px2.p1.1.m1.1.1">0</cn><cn type="integer" id="S5.SS1.SSS0.Px2.p1.1.m1.2.2.cmml" xref="S5.SS1.SSS0.Px2.p1.1.m1.2.2">1</cn></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px2.p1.1.m1.2c">\gamma\in[0,1]</annotation></semantics></math> and a retrieved image <math id="S5.SS1.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="x_{0}" display="inline"><semantics id="S5.SS1.SSS0.Px2.p1.2.m2.1a"><msub id="S5.SS1.SSS0.Px2.p1.2.m2.1.1" xref="S5.SS1.SSS0.Px2.p1.2.m2.1.1.cmml"><mi id="S5.SS1.SSS0.Px2.p1.2.m2.1.1.2" xref="S5.SS1.SSS0.Px2.p1.2.m2.1.1.2.cmml">x</mi><mn id="S5.SS1.SSS0.Px2.p1.2.m2.1.1.3" xref="S5.SS1.SSS0.Px2.p1.2.m2.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px2.p1.2.m2.1b"><apply id="S5.SS1.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S5.SS1.SSS0.Px2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S5.SS1.SSS0.Px2.p1.2.m2.1.1.1.cmml" xref="S5.SS1.SSS0.Px2.p1.2.m2.1.1">subscript</csymbol><ci id="S5.SS1.SSS0.Px2.p1.2.m2.1.1.2.cmml" xref="S5.SS1.SSS0.Px2.p1.2.m2.1.1.2">𝑥</ci><cn type="integer" id="S5.SS1.SSS0.Px2.p1.2.m2.1.1.3.cmml" xref="S5.SS1.SSS0.Px2.p1.2.m2.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px2.p1.2.m2.1c">x_{0}</annotation></semantics></math>, SDEdit adds Gaussian noise to <math id="S5.SS1.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="x_{0}" display="inline"><semantics id="S5.SS1.SSS0.Px2.p1.3.m3.1a"><msub id="S5.SS1.SSS0.Px2.p1.3.m3.1.1" xref="S5.SS1.SSS0.Px2.p1.3.m3.1.1.cmml"><mi id="S5.SS1.SSS0.Px2.p1.3.m3.1.1.2" xref="S5.SS1.SSS0.Px2.p1.3.m3.1.1.2.cmml">x</mi><mn id="S5.SS1.SSS0.Px2.p1.3.m3.1.1.3" xref="S5.SS1.SSS0.Px2.p1.3.m3.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px2.p1.3.m3.1b"><apply id="S5.SS1.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S5.SS1.SSS0.Px2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S5.SS1.SSS0.Px2.p1.3.m3.1.1.1.cmml" xref="S5.SS1.SSS0.Px2.p1.3.m3.1.1">subscript</csymbol><ci id="S5.SS1.SSS0.Px2.p1.3.m3.1.1.2.cmml" xref="S5.SS1.SSS0.Px2.p1.3.m3.1.1.2">𝑥</ci><cn type="integer" id="S5.SS1.SSS0.Px2.p1.3.m3.1.1.3.cmml" xref="S5.SS1.SSS0.Px2.p1.3.m3.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px2.p1.3.m3.1c">x_{0}</annotation></semantics></math> according to timestep <math id="S5.SS1.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="t=\gamma" display="inline"><semantics id="S5.SS1.SSS0.Px2.p1.4.m4.1a"><mrow id="S5.SS1.SSS0.Px2.p1.4.m4.1.1" xref="S5.SS1.SSS0.Px2.p1.4.m4.1.1.cmml"><mi id="S5.SS1.SSS0.Px2.p1.4.m4.1.1.2" xref="S5.SS1.SSS0.Px2.p1.4.m4.1.1.2.cmml">t</mi><mo id="S5.SS1.SSS0.Px2.p1.4.m4.1.1.1" xref="S5.SS1.SSS0.Px2.p1.4.m4.1.1.1.cmml">=</mo><mi id="S5.SS1.SSS0.Px2.p1.4.m4.1.1.3" xref="S5.SS1.SSS0.Px2.p1.4.m4.1.1.3.cmml">γ</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px2.p1.4.m4.1b"><apply id="S5.SS1.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S5.SS1.SSS0.Px2.p1.4.m4.1.1"><eq id="S5.SS1.SSS0.Px2.p1.4.m4.1.1.1.cmml" xref="S5.SS1.SSS0.Px2.p1.4.m4.1.1.1"></eq><ci id="S5.SS1.SSS0.Px2.p1.4.m4.1.1.2.cmml" xref="S5.SS1.SSS0.Px2.p1.4.m4.1.1.2">𝑡</ci><ci id="S5.SS1.SSS0.Px2.p1.4.m4.1.1.3.cmml" xref="S5.SS1.SSS0.Px2.p1.4.m4.1.1.3">𝛾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px2.p1.4.m4.1c">t=\gamma</annotation></semantics></math> of Stable Diffusion’s time-dependent forward process.
We then denoise the noisy image using the same reverse diffusion process as in text-to-image generation, yielding a perturbed image <math id="S5.SS1.SSS0.Px2.p1.5.m5.1" class="ltx_Math" alttext="x^{(\gamma)}" display="inline"><semantics id="S5.SS1.SSS0.Px2.p1.5.m5.1a"><msup id="S5.SS1.SSS0.Px2.p1.5.m5.1.2" xref="S5.SS1.SSS0.Px2.p1.5.m5.1.2.cmml"><mi id="S5.SS1.SSS0.Px2.p1.5.m5.1.2.2" xref="S5.SS1.SSS0.Px2.p1.5.m5.1.2.2.cmml">x</mi><mrow id="S5.SS1.SSS0.Px2.p1.5.m5.1.1.1.3" xref="S5.SS1.SSS0.Px2.p1.5.m5.1.2.cmml"><mo stretchy="false" id="S5.SS1.SSS0.Px2.p1.5.m5.1.1.1.3.1" xref="S5.SS1.SSS0.Px2.p1.5.m5.1.2.cmml">(</mo><mi id="S5.SS1.SSS0.Px2.p1.5.m5.1.1.1.1" xref="S5.SS1.SSS0.Px2.p1.5.m5.1.1.1.1.cmml">γ</mi><mo stretchy="false" id="S5.SS1.SSS0.Px2.p1.5.m5.1.1.1.3.2" xref="S5.SS1.SSS0.Px2.p1.5.m5.1.2.cmml">)</mo></mrow></msup><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px2.p1.5.m5.1b"><apply id="S5.SS1.SSS0.Px2.p1.5.m5.1.2.cmml" xref="S5.SS1.SSS0.Px2.p1.5.m5.1.2"><csymbol cd="ambiguous" id="S5.SS1.SSS0.Px2.p1.5.m5.1.2.1.cmml" xref="S5.SS1.SSS0.Px2.p1.5.m5.1.2">superscript</csymbol><ci id="S5.SS1.SSS0.Px2.p1.5.m5.1.2.2.cmml" xref="S5.SS1.SSS0.Px2.p1.5.m5.1.2.2">𝑥</ci><ci id="S5.SS1.SSS0.Px2.p1.5.m5.1.1.1.1.cmml" xref="S5.SS1.SSS0.Px2.p1.5.m5.1.1.1.1">𝛾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px2.p1.5.m5.1c">x^{(\gamma)}</annotation></semantics></math> that looks semantically like <math id="S5.SS1.SSS0.Px2.p1.6.m6.1" class="ltx_Math" alttext="x_{0}" display="inline"><semantics id="S5.SS1.SSS0.Px2.p1.6.m6.1a"><msub id="S5.SS1.SSS0.Px2.p1.6.m6.1.1" xref="S5.SS1.SSS0.Px2.p1.6.m6.1.1.cmml"><mi id="S5.SS1.SSS0.Px2.p1.6.m6.1.1.2" xref="S5.SS1.SSS0.Px2.p1.6.m6.1.1.2.cmml">x</mi><mn id="S5.SS1.SSS0.Px2.p1.6.m6.1.1.3" xref="S5.SS1.SSS0.Px2.p1.6.m6.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px2.p1.6.m6.1b"><apply id="S5.SS1.SSS0.Px2.p1.6.m6.1.1.cmml" xref="S5.SS1.SSS0.Px2.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S5.SS1.SSS0.Px2.p1.6.m6.1.1.1.cmml" xref="S5.SS1.SSS0.Px2.p1.6.m6.1.1">subscript</csymbol><ci id="S5.SS1.SSS0.Px2.p1.6.m6.1.1.2.cmml" xref="S5.SS1.SSS0.Px2.p1.6.m6.1.1.2">𝑥</ci><cn type="integer" id="S5.SS1.SSS0.Px2.p1.6.m6.1.1.3.cmml" xref="S5.SS1.SSS0.Px2.p1.6.m6.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px2.p1.6.m6.1c">x_{0}</annotation></semantics></math> while also containing Stable Diffusion-specific artifacts. Increasing <math id="S5.SS1.SSS0.Px2.p1.7.m7.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="S5.SS1.SSS0.Px2.p1.7.m7.1a"><mi id="S5.SS1.SSS0.Px2.p1.7.m7.1.1" xref="S5.SS1.SSS0.Px2.p1.7.m7.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px2.p1.7.m7.1b"><ci id="S5.SS1.SSS0.Px2.p1.7.m7.1.1.cmml" xref="S5.SS1.SSS0.Px2.p1.7.m7.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px2.p1.7.m7.1c">\gamma</annotation></semantics></math> increases the amount of Gaussian noise added to <math id="S5.SS1.SSS0.Px2.p1.8.m8.1" class="ltx_Math" alttext="x_{0}" display="inline"><semantics id="S5.SS1.SSS0.Px2.p1.8.m8.1a"><msub id="S5.SS1.SSS0.Px2.p1.8.m8.1.1" xref="S5.SS1.SSS0.Px2.p1.8.m8.1.1.cmml"><mi id="S5.SS1.SSS0.Px2.p1.8.m8.1.1.2" xref="S5.SS1.SSS0.Px2.p1.8.m8.1.1.2.cmml">x</mi><mn id="S5.SS1.SSS0.Px2.p1.8.m8.1.1.3" xref="S5.SS1.SSS0.Px2.p1.8.m8.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px2.p1.8.m8.1b"><apply id="S5.SS1.SSS0.Px2.p1.8.m8.1.1.cmml" xref="S5.SS1.SSS0.Px2.p1.8.m8.1.1"><csymbol cd="ambiguous" id="S5.SS1.SSS0.Px2.p1.8.m8.1.1.1.cmml" xref="S5.SS1.SSS0.Px2.p1.8.m8.1.1">subscript</csymbol><ci id="S5.SS1.SSS0.Px2.p1.8.m8.1.1.2.cmml" xref="S5.SS1.SSS0.Px2.p1.8.m8.1.1.2">𝑥</ci><cn type="integer" id="S5.SS1.SSS0.Px2.p1.8.m8.1.1.3.cmml" xref="S5.SS1.SSS0.Px2.p1.8.m8.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px2.p1.8.m8.1c">x_{0}</annotation></semantics></math>, thereby increasing the severity of visual artifacts introduced in the resulting <math id="S5.SS1.SSS0.Px2.p1.9.m9.1" class="ltx_Math" alttext="x^{(\gamma)}" display="inline"><semantics id="S5.SS1.SSS0.Px2.p1.9.m9.1a"><msup id="S5.SS1.SSS0.Px2.p1.9.m9.1.2" xref="S5.SS1.SSS0.Px2.p1.9.m9.1.2.cmml"><mi id="S5.SS1.SSS0.Px2.p1.9.m9.1.2.2" xref="S5.SS1.SSS0.Px2.p1.9.m9.1.2.2.cmml">x</mi><mrow id="S5.SS1.SSS0.Px2.p1.9.m9.1.1.1.3" xref="S5.SS1.SSS0.Px2.p1.9.m9.1.2.cmml"><mo stretchy="false" id="S5.SS1.SSS0.Px2.p1.9.m9.1.1.1.3.1" xref="S5.SS1.SSS0.Px2.p1.9.m9.1.2.cmml">(</mo><mi id="S5.SS1.SSS0.Px2.p1.9.m9.1.1.1.1" xref="S5.SS1.SSS0.Px2.p1.9.m9.1.1.1.1.cmml">γ</mi><mo stretchy="false" id="S5.SS1.SSS0.Px2.p1.9.m9.1.1.1.3.2" xref="S5.SS1.SSS0.Px2.p1.9.m9.1.2.cmml">)</mo></mrow></msup><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px2.p1.9.m9.1b"><apply id="S5.SS1.SSS0.Px2.p1.9.m9.1.2.cmml" xref="S5.SS1.SSS0.Px2.p1.9.m9.1.2"><csymbol cd="ambiguous" id="S5.SS1.SSS0.Px2.p1.9.m9.1.2.1.cmml" xref="S5.SS1.SSS0.Px2.p1.9.m9.1.2">superscript</csymbol><ci id="S5.SS1.SSS0.Px2.p1.9.m9.1.2.2.cmml" xref="S5.SS1.SSS0.Px2.p1.9.m9.1.2.2">𝑥</ci><ci id="S5.SS1.SSS0.Px2.p1.9.m9.1.1.1.1.cmml" xref="S5.SS1.SSS0.Px2.p1.9.m9.1.1.1.1">𝛾</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px2.p1.9.m9.1c">x^{(\gamma)}</annotation></semantics></math> (see Appendix <a href="#A4.SS1" title="D.1 Synthetically perturbing retrieved images ‣ Appendix D Details in Analysis Experiments ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D.1</span></a> for further details). In pseudocode,</p>
<table id="A5.EGx4" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S5.Ex4"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S5.Ex4.m1.4" class="ltx_Math" alttext="\displaystyle x^{(\gamma)}=\textrm{ StableDiffusion.denoise}(\textrm{StableDiffusion.add\_noise$(x_{0},\gamma)$},\gamma)." display="inline"><semantics id="S5.Ex4.m1.4a"><mrow id="S5.Ex4.m1.4.4.1" xref="S5.Ex4.m1.4.4.1.1.cmml"><mrow id="S5.Ex4.m1.4.4.1.1" xref="S5.Ex4.m1.4.4.1.1.cmml"><msup id="S5.Ex4.m1.4.4.1.1.2" xref="S5.Ex4.m1.4.4.1.1.2.cmml"><mi id="S5.Ex4.m1.4.4.1.1.2.2" xref="S5.Ex4.m1.4.4.1.1.2.2.cmml">x</mi><mrow id="S5.Ex4.m1.2.2.1.3" xref="S5.Ex4.m1.4.4.1.1.2.cmml"><mo stretchy="false" id="S5.Ex4.m1.2.2.1.3.1" xref="S5.Ex4.m1.4.4.1.1.2.cmml">(</mo><mi id="S5.Ex4.m1.2.2.1.1" xref="S5.Ex4.m1.2.2.1.1.cmml">γ</mi><mo stretchy="false" id="S5.Ex4.m1.2.2.1.3.2" xref="S5.Ex4.m1.4.4.1.1.2.cmml">)</mo></mrow></msup><mo id="S5.Ex4.m1.4.4.1.1.1" xref="S5.Ex4.m1.4.4.1.1.1.cmml">=</mo><mrow id="S5.Ex4.m1.4.4.1.1.3" xref="S5.Ex4.m1.4.4.1.1.3.cmml"><mtext id="S5.Ex4.m1.4.4.1.1.3.2" xref="S5.Ex4.m1.4.4.1.1.3.2a.cmml"> StableDiffusion.denoise</mtext><mo lspace="0em" rspace="0em" id="S5.Ex4.m1.4.4.1.1.3.1" xref="S5.Ex4.m1.4.4.1.1.3.1.cmml">​</mo><mrow id="S5.Ex4.m1.4.4.1.1.3.3.2" xref="S5.Ex4.m1.4.4.1.1.3.3.1.cmml"><mo stretchy="false" id="S5.Ex4.m1.4.4.1.1.3.3.2.1" xref="S5.Ex4.m1.4.4.1.1.3.3.1.cmml">(</mo><mrow id="S5.Ex4.m1.1.1.1a" xref="S5.Ex4.m1.1.1.1ab.cmml"><mtext id="S5.Ex4.m1.1.1.1aa" xref="S5.Ex4.m1.1.1.1ab.cmml">StableDiffusion.add_noise</mtext><mrow id="S5.Ex4.m1.1.1.1.m1.2.2.1" xref="S5.Ex4.m1.1.1.1.m1.2.2.1.cmml"><mo stretchy="false" id="S5.Ex4.m1.1.1.1.m1.2.2.1.2" xref="S5.Ex4.m1.1.1.1.m1.2.2.1.cmml">(</mo><msub id="S5.Ex4.m1.1.1.1.m1.2.2.1.1" xref="S5.Ex4.m1.1.1.1.m1.2.2.1.1.cmml"><mi id="S5.Ex4.m1.1.1.1.m1.2.2.1.1.2" xref="S5.Ex4.m1.1.1.1.m1.2.2.1.1.2.cmml">x</mi><mn id="S5.Ex4.m1.1.1.1.m1.2.2.1.1.3" xref="S5.Ex4.m1.1.1.1.m1.2.2.1.1.3.cmml">0</mn></msub><mo id="S5.Ex4.m1.1.1.1.m1.2.2.1.3" xref="S5.Ex4.m1.1.1.1.m1.2.2.1.cmml">,</mo><mi id="S5.Ex4.m1.1.1.1.m1.1.1" xref="S5.Ex4.m1.1.1.1.m1.1.1.cmml">γ</mi><mo stretchy="false" id="S5.Ex4.m1.1.1.1.m1.2.2.1.4" xref="S5.Ex4.m1.1.1.1.m1.2.2.1.cmml">)</mo></mrow></mrow><mo id="S5.Ex4.m1.4.4.1.1.3.3.2.2" xref="S5.Ex4.m1.4.4.1.1.3.3.1.cmml">,</mo><mi id="S5.Ex4.m1.3.3" xref="S5.Ex4.m1.3.3.cmml">γ</mi><mo stretchy="false" id="S5.Ex4.m1.4.4.1.1.3.3.2.3" xref="S5.Ex4.m1.4.4.1.1.3.3.1.cmml">)</mo></mrow></mrow></mrow><mo lspace="0em" id="S5.Ex4.m1.4.4.1.2" xref="S5.Ex4.m1.4.4.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S5.Ex4.m1.4b"><apply id="S5.Ex4.m1.4.4.1.1.cmml" xref="S5.Ex4.m1.4.4.1"><eq id="S5.Ex4.m1.4.4.1.1.1.cmml" xref="S5.Ex4.m1.4.4.1.1.1"></eq><apply id="S5.Ex4.m1.4.4.1.1.2.cmml" xref="S5.Ex4.m1.4.4.1.1.2"><csymbol cd="ambiguous" id="S5.Ex4.m1.4.4.1.1.2.1.cmml" xref="S5.Ex4.m1.4.4.1.1.2">superscript</csymbol><ci id="S5.Ex4.m1.4.4.1.1.2.2.cmml" xref="S5.Ex4.m1.4.4.1.1.2.2">𝑥</ci><ci id="S5.Ex4.m1.2.2.1.1.cmml" xref="S5.Ex4.m1.2.2.1.1">𝛾</ci></apply><apply id="S5.Ex4.m1.4.4.1.1.3.cmml" xref="S5.Ex4.m1.4.4.1.1.3"><times id="S5.Ex4.m1.4.4.1.1.3.1.cmml" xref="S5.Ex4.m1.4.4.1.1.3.1"></times><ci id="S5.Ex4.m1.4.4.1.1.3.2a.cmml" xref="S5.Ex4.m1.4.4.1.1.3.2"><mtext id="S5.Ex4.m1.4.4.1.1.3.2.cmml" xref="S5.Ex4.m1.4.4.1.1.3.2"> StableDiffusion.denoise</mtext></ci><interval closure="open" id="S5.Ex4.m1.4.4.1.1.3.3.1.cmml" xref="S5.Ex4.m1.4.4.1.1.3.3.2"><ci id="S5.Ex4.m1.1.1.1ab.cmml" xref="S5.Ex4.m1.1.1.1a"><mrow id="S5.Ex4.m1.1.1.1a.cmml" xref="S5.Ex4.m1.1.1.1a"><mtext id="S5.Ex4.m1.1.1.1aa.cmml" xref="S5.Ex4.m1.1.1.1a">StableDiffusion.add_noise</mtext><mrow id="S5.Ex4.m1.1.1.1.m1.2.2.1.cmml" xref="S5.Ex4.m1.1.1.1.m1.2.2.1"><mo stretchy="false" id="S5.Ex4.m1.1.1.1.m1.2.2.1.2.cmml" xref="S5.Ex4.m1.1.1.1.m1.2.2.1">(</mo><msub id="S5.Ex4.m1.1.1.1.m1.2.2.1.1.cmml" xref="S5.Ex4.m1.1.1.1.m1.2.2.1.1"><mi id="S5.Ex4.m1.1.1.1.m1.2.2.1.1.2.cmml" xref="S5.Ex4.m1.1.1.1.m1.2.2.1.1.2">x</mi><mn id="S5.Ex4.m1.1.1.1.m1.2.2.1.1.3.cmml" xref="S5.Ex4.m1.1.1.1.m1.2.2.1.1.3">0</mn></msub><mo id="S5.Ex4.m1.1.1.1.m1.2.2.1.3.cmml" xref="S5.Ex4.m1.1.1.1.m1.2.2.1">,</mo><mi id="S5.Ex4.m1.1.1.1.m1.1.1.cmml" xref="S5.Ex4.m1.1.1.1.m1.1.1">γ</mi><mo stretchy="false" id="S5.Ex4.m1.1.1.1.m1.2.2.1.4.cmml" xref="S5.Ex4.m1.1.1.1.m1.2.2.1">)</mo></mrow></mrow></ci><ci id="S5.Ex4.m1.3.3.cmml" xref="S5.Ex4.m1.3.3">𝛾</ci></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.Ex4.m1.4c">\displaystyle x^{(\gamma)}=\textrm{ StableDiffusion.denoise}(\textrm{StableDiffusion.add\_noise$(x_{0},\gamma)$},\gamma).</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S5.SS1.SSS0.Px2.p1.12" class="ltx_p">Starting from the full targeted retrieved adaptation datasets <math id="S5.SS1.SSS0.Px2.p1.10.m1.1" class="ltx_Math" alttext="{\mathcal{D}_{\mathcal{C}}^{(\text{retrieved})}}" display="inline"><semantics id="S5.SS1.SSS0.Px2.p1.10.m1.1a"><msubsup id="S5.SS1.SSS0.Px2.p1.10.m1.1.2" xref="S5.SS1.SSS0.Px2.p1.10.m1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS1.SSS0.Px2.p1.10.m1.1.2.2.2" xref="S5.SS1.SSS0.Px2.p1.10.m1.1.2.2.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S5.SS1.SSS0.Px2.p1.10.m1.1.2.2.3" xref="S5.SS1.SSS0.Px2.p1.10.m1.1.2.2.3.cmml">𝒞</mi><mrow id="S5.SS1.SSS0.Px2.p1.10.m1.1.1.1.3" xref="S5.SS1.SSS0.Px2.p1.10.m1.1.1.1.1a.cmml"><mo stretchy="false" id="S5.SS1.SSS0.Px2.p1.10.m1.1.1.1.3.1" xref="S5.SS1.SSS0.Px2.p1.10.m1.1.1.1.1a.cmml">(</mo><mtext id="S5.SS1.SSS0.Px2.p1.10.m1.1.1.1.1" xref="S5.SS1.SSS0.Px2.p1.10.m1.1.1.1.1.cmml">retrieved</mtext><mo stretchy="false" id="S5.SS1.SSS0.Px2.p1.10.m1.1.1.1.3.2" xref="S5.SS1.SSS0.Px2.p1.10.m1.1.1.1.1a.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px2.p1.10.m1.1b"><apply id="S5.SS1.SSS0.Px2.p1.10.m1.1.2.cmml" xref="S5.SS1.SSS0.Px2.p1.10.m1.1.2"><csymbol cd="ambiguous" id="S5.SS1.SSS0.Px2.p1.10.m1.1.2.1.cmml" xref="S5.SS1.SSS0.Px2.p1.10.m1.1.2">superscript</csymbol><apply id="S5.SS1.SSS0.Px2.p1.10.m1.1.2.2.cmml" xref="S5.SS1.SSS0.Px2.p1.10.m1.1.2"><csymbol cd="ambiguous" id="S5.SS1.SSS0.Px2.p1.10.m1.1.2.2.1.cmml" xref="S5.SS1.SSS0.Px2.p1.10.m1.1.2">subscript</csymbol><ci id="S5.SS1.SSS0.Px2.p1.10.m1.1.2.2.2.cmml" xref="S5.SS1.SSS0.Px2.p1.10.m1.1.2.2.2">𝒟</ci><ci id="S5.SS1.SSS0.Px2.p1.10.m1.1.2.2.3.cmml" xref="S5.SS1.SSS0.Px2.p1.10.m1.1.2.2.3">𝒞</ci></apply><ci id="S5.SS1.SSS0.Px2.p1.10.m1.1.1.1.1a.cmml" xref="S5.SS1.SSS0.Px2.p1.10.m1.1.1.1.3"><mtext mathsize="70%" id="S5.SS1.SSS0.Px2.p1.10.m1.1.1.1.1.cmml" xref="S5.SS1.SSS0.Px2.p1.10.m1.1.1.1.1">retrieved</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px2.p1.10.m1.1c">{\mathcal{D}_{\mathcal{C}}^{(\text{retrieved})}}</annotation></semantics></math> for FGVC-Aircraft and ImageNet, we use SDEdit to introduce generator artifacts into the retrieved real images over a range of <math id="S5.SS1.SSS0.Px2.p1.11.m2.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="S5.SS1.SSS0.Px2.p1.11.m2.1a"><mi id="S5.SS1.SSS0.Px2.p1.11.m2.1.1" xref="S5.SS1.SSS0.Px2.p1.11.m2.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px2.p1.11.m2.1b"><ci id="S5.SS1.SSS0.Px2.p1.11.m2.1.1.cmml" xref="S5.SS1.SSS0.Px2.p1.11.m2.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px2.p1.11.m2.1c">\gamma</annotation></semantics></math> values. We visualize the resulting perturbed images in Figure <a href="#S5.F4" title="Figure 4 ‣ Synthetically perturbing retrieved real images. ‣ 5.1 Why does synthetic data lag retrieved real data? ‣ 5 Analysis ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, and plot the results of training on these perturbed images across <math id="S5.SS1.SSS0.Px2.p1.12.m3.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="S5.SS1.SSS0.Px2.p1.12.m3.1a"><mi id="S5.SS1.SSS0.Px2.p1.12.m3.1.1" xref="S5.SS1.SSS0.Px2.p1.12.m3.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px2.p1.12.m3.1b"><ci id="S5.SS1.SSS0.Px2.p1.12.m3.1.1.cmml" xref="S5.SS1.SSS0.Px2.p1.12.m3.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px2.p1.12.m3.1c">\gamma</annotation></semantics></math> in Figure <a href="#S5.F5" title="Figure 5 ‣ Synthetically perturbing retrieved real images. ‣ 5.1 Why does synthetic data lag retrieved real data? ‣ 5 Analysis ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>.</p>
</div>
<figure id="S5.F4" class="ltx_figure"><img src="/html/2406.05184/assets/x4.png" id="S5.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="136" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>We use Stable Diffusion to synthetically perturb real images according to a noise strength parameter <math id="S5.F4.5.m1.2" class="ltx_Math" alttext="\gamma\in[0,1]" display="inline"><semantics id="S5.F4.5.m1.2b"><mrow id="S5.F4.5.m1.2.3" xref="S5.F4.5.m1.2.3.cmml"><mi id="S5.F4.5.m1.2.3.2" xref="S5.F4.5.m1.2.3.2.cmml">γ</mi><mo id="S5.F4.5.m1.2.3.1" xref="S5.F4.5.m1.2.3.1.cmml">∈</mo><mrow id="S5.F4.5.m1.2.3.3.2" xref="S5.F4.5.m1.2.3.3.1.cmml"><mo stretchy="false" id="S5.F4.5.m1.2.3.3.2.1" xref="S5.F4.5.m1.2.3.3.1.cmml">[</mo><mn id="S5.F4.5.m1.1.1" xref="S5.F4.5.m1.1.1.cmml">0</mn><mo id="S5.F4.5.m1.2.3.3.2.2" xref="S5.F4.5.m1.2.3.3.1.cmml">,</mo><mn id="S5.F4.5.m1.2.2" xref="S5.F4.5.m1.2.2.cmml">1</mn><mo stretchy="false" id="S5.F4.5.m1.2.3.3.2.3" xref="S5.F4.5.m1.2.3.3.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.F4.5.m1.2c"><apply id="S5.F4.5.m1.2.3.cmml" xref="S5.F4.5.m1.2.3"><in id="S5.F4.5.m1.2.3.1.cmml" xref="S5.F4.5.m1.2.3.1"></in><ci id="S5.F4.5.m1.2.3.2.cmml" xref="S5.F4.5.m1.2.3.2">𝛾</ci><interval closure="closed" id="S5.F4.5.m1.2.3.3.1.cmml" xref="S5.F4.5.m1.2.3.3.2"><cn type="integer" id="S5.F4.5.m1.1.1.cmml" xref="S5.F4.5.m1.1.1">0</cn><cn type="integer" id="S5.F4.5.m1.2.2.cmml" xref="S5.F4.5.m1.2.2">1</cn></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.F4.5.m1.2d">\gamma\in[0,1]</annotation></semantics></math>, where larger <math id="S5.F4.6.m2.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="S5.F4.6.m2.1b"><mi id="S5.F4.6.m2.1.1" xref="S5.F4.6.m2.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="S5.F4.6.m2.1c"><ci id="S5.F4.6.m2.1.1.cmml" xref="S5.F4.6.m2.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F4.6.m2.1d">\gamma</annotation></semantics></math> increases the severity of generator-specific artifacts added by the perturbation. When <math id="S5.F4.7.m3.1" class="ltx_Math" alttext="\gamma\geq 0.6" display="inline"><semantics id="S5.F4.7.m3.1b"><mrow id="S5.F4.7.m3.1.1" xref="S5.F4.7.m3.1.1.cmml"><mi id="S5.F4.7.m3.1.1.2" xref="S5.F4.7.m3.1.1.2.cmml">γ</mi><mo id="S5.F4.7.m3.1.1.1" xref="S5.F4.7.m3.1.1.1.cmml">≥</mo><mn id="S5.F4.7.m3.1.1.3" xref="S5.F4.7.m3.1.1.3.cmml">0.6</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.F4.7.m3.1c"><apply id="S5.F4.7.m3.1.1.cmml" xref="S5.F4.7.m3.1.1"><geq id="S5.F4.7.m3.1.1.1.cmml" xref="S5.F4.7.m3.1.1.1"></geq><ci id="S5.F4.7.m3.1.1.2.cmml" xref="S5.F4.7.m3.1.1.2">𝛾</ci><cn type="float" id="S5.F4.7.m3.1.1.3.cmml" xref="S5.F4.7.m3.1.1.3">0.6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.F4.7.m3.1d">\gamma\geq 0.6</annotation></semantics></math>, the introduced artifacts can be strong enough to damage task-relevant visual details for finegrained tasks like FGVC-Aircraft (e.g., the airplane’s engine and rear wheels). For broad tasks like ImageNet, artifacts have a lesser impact on class-relevant details; the “<span id="S5.F4.10.1" class="ltx_text ltx_font_typewriter">Tabby Cat</span>” is recognizable as a cat even after perturbing with high <math id="S5.F4.8.m4.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="S5.F4.8.m4.1b"><mi id="S5.F4.8.m4.1.1" xref="S5.F4.8.m4.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="S5.F4.8.m4.1c"><ci id="S5.F4.8.m4.1.1.cmml" xref="S5.F4.8.m4.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F4.8.m4.1d">\gamma</annotation></semantics></math>.
</figcaption>
</figure>
<figure id="S5.F5" class="ltx_figure"><img src="/html/2406.05184/assets/x5.png" id="S5.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="121" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>We finetune a pretrained CLIP model (<span id="S5.F5.12.1" class="ltx_text ltx_font_bold" style="color:#D285C2;">dashed purple line</span>) on retrieved real images that are synthetically perturbed (<span id="S5.F5.13.2" class="ltx_text ltx_font_bold" style="color:#029E73;">green circles</span>) with Stable Diffusion to introduce generator artifacts. The perturbation strength is controlled by a parameter <math id="S5.F5.4.m1.2" class="ltx_Math" alttext="\gamma\in[0,1]" display="inline"><semantics id="S5.F5.4.m1.2b"><mrow id="S5.F5.4.m1.2.3" xref="S5.F5.4.m1.2.3.cmml"><mi id="S5.F5.4.m1.2.3.2" xref="S5.F5.4.m1.2.3.2.cmml">γ</mi><mo id="S5.F5.4.m1.2.3.1" xref="S5.F5.4.m1.2.3.1.cmml">∈</mo><mrow id="S5.F5.4.m1.2.3.3.2" xref="S5.F5.4.m1.2.3.3.1.cmml"><mo stretchy="false" id="S5.F5.4.m1.2.3.3.2.1" xref="S5.F5.4.m1.2.3.3.1.cmml">[</mo><mn id="S5.F5.4.m1.1.1" xref="S5.F5.4.m1.1.1.cmml">0</mn><mo id="S5.F5.4.m1.2.3.3.2.2" xref="S5.F5.4.m1.2.3.3.1.cmml">,</mo><mn id="S5.F5.4.m1.2.2" xref="S5.F5.4.m1.2.2.cmml">1</mn><mo stretchy="false" id="S5.F5.4.m1.2.3.3.2.3" xref="S5.F5.4.m1.2.3.3.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.F5.4.m1.2c"><apply id="S5.F5.4.m1.2.3.cmml" xref="S5.F5.4.m1.2.3"><in id="S5.F5.4.m1.2.3.1.cmml" xref="S5.F5.4.m1.2.3.1"></in><ci id="S5.F5.4.m1.2.3.2.cmml" xref="S5.F5.4.m1.2.3.2">𝛾</ci><interval closure="closed" id="S5.F5.4.m1.2.3.3.1.cmml" xref="S5.F5.4.m1.2.3.3.2"><cn type="integer" id="S5.F5.4.m1.1.1.cmml" xref="S5.F5.4.m1.1.1">0</cn><cn type="integer" id="S5.F5.4.m1.2.2.cmml" xref="S5.F5.4.m1.2.2">1</cn></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.F5.4.m1.2d">\gamma\in[0,1]</annotation></semantics></math> where larger <math id="S5.F5.5.m2.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="S5.F5.5.m2.1b"><mi id="S5.F5.5.m2.1.1" xref="S5.F5.5.m2.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="S5.F5.5.m2.1c"><ci id="S5.F5.5.m2.1.1.cmml" xref="S5.F5.5.m2.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.F5.5.m2.1d">\gamma</annotation></semantics></math> corresponds to stronger artifacts; within the <span id="S5.F5.14.3" class="ltx_text ltx_font_bold" style="color:#6B6B6B;">gray-shaded region</span>, the artifacts are strong enough to damage class-relevant details. Our results suggest that generator artifacts do contribute to synthetic data’s underperformance—with any amount of artifacts introduced, performance drops below training on retrieved images (<span id="S5.F5.15.4" class="ltx_text ltx_font_bold" style="color:#0174B2;">dashed blue line</span>). Moreover, differences in visual content between synthetic and retrieved images also matter; even with a relatively strong perturbation strength of <math id="S5.F5.6.m3.1" class="ltx_Math" alttext="\gamma=0.5" display="inline"><semantics id="S5.F5.6.m3.1b"><mrow id="S5.F5.6.m3.1.1" xref="S5.F5.6.m3.1.1.cmml"><mi id="S5.F5.6.m3.1.1.2" xref="S5.F5.6.m3.1.1.2.cmml">γ</mi><mo id="S5.F5.6.m3.1.1.1" xref="S5.F5.6.m3.1.1.1.cmml">=</mo><mn id="S5.F5.6.m3.1.1.3" xref="S5.F5.6.m3.1.1.3.cmml">0.5</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.F5.6.m3.1c"><apply id="S5.F5.6.m3.1.1.cmml" xref="S5.F5.6.m3.1.1"><eq id="S5.F5.6.m3.1.1.1.cmml" xref="S5.F5.6.m3.1.1.1"></eq><ci id="S5.F5.6.m3.1.1.2.cmml" xref="S5.F5.6.m3.1.1.2">𝛾</ci><cn type="float" id="S5.F5.6.m3.1.1.3.cmml" xref="S5.F5.6.m3.1.1.3">0.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.F5.6.m3.1d">\gamma=0.5</annotation></semantics></math>, training on artifact-afflicted perturbed images that retain the semantic content of retrieved images outperforms training on synthetic images (<span id="S5.F5.16.5" class="ltx_text ltx_font_bold" style="color:#DF670B;">dashed orange line</span>).
</figcaption>
</figure>
<div id="S5.SS1.SSS0.Px2.p2" class="ltx_para">
<p id="S5.SS1.SSS0.Px2.p2.4" class="ltx_p">Our results suggest three takeaways. First, generator artifacts indeed contribute to the underperformance of synthetic training images, especially for fine-grained classification tasks. On FGVC-Aircraft, any amount of added generator artifacts drops downstream accuracy. Second, the impact of artifacts is relatively lower for broad classification domains such as ImageNet, where downstream performance is not significantly impacted until we perturb with a relatively strong noise strength of <math id="S5.SS1.SSS0.Px2.p2.1.m1.1" class="ltx_Math" alttext="\gamma=0.5" display="inline"><semantics id="S5.SS1.SSS0.Px2.p2.1.m1.1a"><mrow id="S5.SS1.SSS0.Px2.p2.1.m1.1.1" xref="S5.SS1.SSS0.Px2.p2.1.m1.1.1.cmml"><mi id="S5.SS1.SSS0.Px2.p2.1.m1.1.1.2" xref="S5.SS1.SSS0.Px2.p2.1.m1.1.1.2.cmml">γ</mi><mo id="S5.SS1.SSS0.Px2.p2.1.m1.1.1.1" xref="S5.SS1.SSS0.Px2.p2.1.m1.1.1.1.cmml">=</mo><mn id="S5.SS1.SSS0.Px2.p2.1.m1.1.1.3" xref="S5.SS1.SSS0.Px2.p2.1.m1.1.1.3.cmml">0.5</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px2.p2.1.m1.1b"><apply id="S5.SS1.SSS0.Px2.p2.1.m1.1.1.cmml" xref="S5.SS1.SSS0.Px2.p2.1.m1.1.1"><eq id="S5.SS1.SSS0.Px2.p2.1.m1.1.1.1.cmml" xref="S5.SS1.SSS0.Px2.p2.1.m1.1.1.1"></eq><ci id="S5.SS1.SSS0.Px2.p2.1.m1.1.1.2.cmml" xref="S5.SS1.SSS0.Px2.p2.1.m1.1.1.2">𝛾</ci><cn type="float" id="S5.SS1.SSS0.Px2.p2.1.m1.1.1.3.cmml" xref="S5.SS1.SSS0.Px2.p2.1.m1.1.1.3">0.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px2.p2.1.m1.1c">\gamma=0.5</annotation></semantics></math>. Finally, visual content differences between synthetic and retrieved images also play a key role in the performance gap between synthetic and retrieved training data. When we perturb images with strength <math id="S5.SS1.SSS0.Px2.p2.2.m2.1" class="ltx_Math" alttext="\gamma=0.5" display="inline"><semantics id="S5.SS1.SSS0.Px2.p2.2.m2.1a"><mrow id="S5.SS1.SSS0.Px2.p2.2.m2.1.1" xref="S5.SS1.SSS0.Px2.p2.2.m2.1.1.cmml"><mi id="S5.SS1.SSS0.Px2.p2.2.m2.1.1.2" xref="S5.SS1.SSS0.Px2.p2.2.m2.1.1.2.cmml">γ</mi><mo id="S5.SS1.SSS0.Px2.p2.2.m2.1.1.1" xref="S5.SS1.SSS0.Px2.p2.2.m2.1.1.1.cmml">=</mo><mn id="S5.SS1.SSS0.Px2.p2.2.m2.1.1.3" xref="S5.SS1.SSS0.Px2.p2.2.m2.1.1.3.cmml">0.5</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px2.p2.2.m2.1b"><apply id="S5.SS1.SSS0.Px2.p2.2.m2.1.1.cmml" xref="S5.SS1.SSS0.Px2.p2.2.m2.1.1"><eq id="S5.SS1.SSS0.Px2.p2.2.m2.1.1.1.cmml" xref="S5.SS1.SSS0.Px2.p2.2.m2.1.1.1"></eq><ci id="S5.SS1.SSS0.Px2.p2.2.m2.1.1.2.cmml" xref="S5.SS1.SSS0.Px2.p2.2.m2.1.1.2">𝛾</ci><cn type="float" id="S5.SS1.SSS0.Px2.p2.2.m2.1.1.3.cmml" xref="S5.SS1.SSS0.Px2.p2.2.m2.1.1.3">0.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px2.p2.2.m2.1c">\gamma=0.5</annotation></semantics></math>, the resulting images are heavily afflicted with artifacts, but still retain the important class-relevant details of retrieved real images, such as correct airplane engine configurations. Training on <math id="S5.SS1.SSS0.Px2.p2.3.m3.1" class="ltx_Math" alttext="\gamma=0.5" display="inline"><semantics id="S5.SS1.SSS0.Px2.p2.3.m3.1a"><mrow id="S5.SS1.SSS0.Px2.p2.3.m3.1.1" xref="S5.SS1.SSS0.Px2.p2.3.m3.1.1.cmml"><mi id="S5.SS1.SSS0.Px2.p2.3.m3.1.1.2" xref="S5.SS1.SSS0.Px2.p2.3.m3.1.1.2.cmml">γ</mi><mo id="S5.SS1.SSS0.Px2.p2.3.m3.1.1.1" xref="S5.SS1.SSS0.Px2.p2.3.m3.1.1.1.cmml">=</mo><mn id="S5.SS1.SSS0.Px2.p2.3.m3.1.1.3" xref="S5.SS1.SSS0.Px2.p2.3.m3.1.1.3.cmml">0.5</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px2.p2.3.m3.1b"><apply id="S5.SS1.SSS0.Px2.p2.3.m3.1.1.cmml" xref="S5.SS1.SSS0.Px2.p2.3.m3.1.1"><eq id="S5.SS1.SSS0.Px2.p2.3.m3.1.1.1.cmml" xref="S5.SS1.SSS0.Px2.p2.3.m3.1.1.1"></eq><ci id="S5.SS1.SSS0.Px2.p2.3.m3.1.1.2.cmml" xref="S5.SS1.SSS0.Px2.p2.3.m3.1.1.2">𝛾</ci><cn type="float" id="S5.SS1.SSS0.Px2.p2.3.m3.1.1.3.cmml" xref="S5.SS1.SSS0.Px2.p2.3.m3.1.1.3">0.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px2.p2.3.m3.1c">\gamma=0.5</annotation></semantics></math> images significantly outperforms training on synthetic images. Intriguingly, training on aircraft images perturbed beyond the point where class-relevant visual details are damaged <math id="S5.SS1.SSS0.Px2.p2.4.m4.1" class="ltx_Math" alttext="(\gamma\geq 0.6)" display="inline"><semantics id="S5.SS1.SSS0.Px2.p2.4.m4.1a"><mrow id="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1" xref="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.1.cmml"><mo stretchy="false" id="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.2" xref="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.1.cmml">(</mo><mrow id="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.1" xref="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.1.cmml"><mi id="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.1.2" xref="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.1.2.cmml">γ</mi><mo id="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.1.1" xref="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.1.1.cmml">≥</mo><mn id="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.1.3" xref="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.1.3.cmml">0.6</mn></mrow><mo stretchy="false" id="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.3" xref="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.SSS0.Px2.p2.4.m4.1b"><apply id="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.1.cmml" xref="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1"><geq id="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.1.1.cmml" xref="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.1.1"></geq><ci id="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.1.2.cmml" xref="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.1.2">𝛾</ci><cn type="float" id="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.1.3.cmml" xref="S5.SS1.SSS0.Px2.p2.4.m4.1.1.1.1.3">0.6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.SSS0.Px2.p2.4.m4.1c">(\gamma\geq 0.6)</annotation></semantics></math> still outperforms synthetic data; we speculate that this is because these heavily perturbed images still retain the overall image composition of retrieved images.</p>
</div>
</section>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Synthesizing data via another generative model</h3>

<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.1" class="ltx_p">For our main scaling experiments, we generate synthetic image datasets using Stable Diffusion 1.5 to maintain consistency with prior work <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>, <a href="#bib.bib23" title="" class="ltx_ref">23</a>]</cite>. To what degree does our choice of generative model impact our findings? At the time of our study, Stable Diffusion 1.x models are the only modern text-to-image models with open source training data available to retrieve from. Therefore, we focus our study here on Stable Diffusion (SD) versions 1.1, 1.3, and 1.5. Starting from SD v1.1, which is trained on the full LAION-2B dataset, SD v1.3 and v1.5 are derived by further finetuning on high-quality subsets of LAION-2B. This additional finetuning improves image generation fidelity <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib52" title="" class="ltx_ref">52</a>]</cite>, but may lead to the model forgetting parts of the LAION-2B distribution <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib19" title="" class="ltx_ref">19</a>]</cite>.
Following our main experiment setup (Section <a href="#S4" title="4 Main Experiments ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>), we use SD v1.1 and SD v1.3 to generate various-sized targeted synthetic adaptation datasets for ImageNet and FGVC-Aircraft. Results are plotted in Figure <a href="#S5.F6" title="Figure 6 ‣ 5.2 Synthesizing data via another generative model ‣ 5 Analysis ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>. Overall, while training with synthetic data from different generative models yields varying performance, synthetic data from all generative models considered consistently fall short of retrieval.</p>
</div>
<figure id="S5.F6" class="ltx_figure"><img src="/html/2406.05184/assets/x6.png" id="S5.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="123" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>We ablate our choice of generative model used to synthesize images in our main experiments, comparing training on synthetic images from <span id="S5.F6.5.1" class="ltx_text ltx_font_bold" style="color:#DF670B;">Stable Diffusion 1.5 (SD v1.5)</span> to synthetic images from <span id="S5.F6.6.2" class="ltx_text ltx_font_bold" style="color:#029E73;">SD v1.1</span> and <span id="S5.F6.7.3" class="ltx_text ltx_font_bold" style="color:#8C0900;">SD v1.3</span>. Across multiple Stable Diffusion models—the only modern text-to-image generators with open-source training data available—training with images directly retrieved from the generative models’ training dataset (<span id="S5.F6.8.4" class="ltx_text ltx_font_bold" style="color:#0174B2;">LAION-2B</span>) outperforms training with generated images.</figcaption>
</figure>
</section>
<section id="S5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.3 </span>Mixing synthetic and retrieved data</h3>

<div id="S5.SS3.p1" class="ltx_para">
<p id="S5.SS3.p1.2" class="ltx_p">On FGVC-Aircraft, finetuning CLIP with either synthetic or retrieved data alone consistently improves downstream task accuracy. The gains from retrieved data are stronger than the gains from synthetic data across all data scales; however, synthetic data may improve CLIP in ways that is complementary to retrieved data, and thus present orthogonal value. To test this possibility, we measure whether training on a mix of synthetic and retrieved Aircraft adaptation data significantly outperforms training with either alone. Starting from our largest retrieved adaptation dataset <math id="S5.SS3.p1.1.m1.1" class="ltx_Math" alttext="{\mathcal{D}_{\mathcal{C}}^{(\text{retrieved})}}" display="inline"><semantics id="S5.SS3.p1.1.m1.1a"><msubsup id="S5.SS3.p1.1.m1.1.2" xref="S5.SS3.p1.1.m1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS3.p1.1.m1.1.2.2.2" xref="S5.SS3.p1.1.m1.1.2.2.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S5.SS3.p1.1.m1.1.2.2.3" xref="S5.SS3.p1.1.m1.1.2.2.3.cmml">𝒞</mi><mrow id="S5.SS3.p1.1.m1.1.1.1.3" xref="S5.SS3.p1.1.m1.1.1.1.1a.cmml"><mo stretchy="false" id="S5.SS3.p1.1.m1.1.1.1.3.1" xref="S5.SS3.p1.1.m1.1.1.1.1a.cmml">(</mo><mtext id="S5.SS3.p1.1.m1.1.1.1.1" xref="S5.SS3.p1.1.m1.1.1.1.1.cmml">retrieved</mtext><mo stretchy="false" id="S5.SS3.p1.1.m1.1.1.1.3.2" xref="S5.SS3.p1.1.m1.1.1.1.1a.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S5.SS3.p1.1.m1.1b"><apply id="S5.SS3.p1.1.m1.1.2.cmml" xref="S5.SS3.p1.1.m1.1.2"><csymbol cd="ambiguous" id="S5.SS3.p1.1.m1.1.2.1.cmml" xref="S5.SS3.p1.1.m1.1.2">superscript</csymbol><apply id="S5.SS3.p1.1.m1.1.2.2.cmml" xref="S5.SS3.p1.1.m1.1.2"><csymbol cd="ambiguous" id="S5.SS3.p1.1.m1.1.2.2.1.cmml" xref="S5.SS3.p1.1.m1.1.2">subscript</csymbol><ci id="S5.SS3.p1.1.m1.1.2.2.2.cmml" xref="S5.SS3.p1.1.m1.1.2.2.2">𝒟</ci><ci id="S5.SS3.p1.1.m1.1.2.2.3.cmml" xref="S5.SS3.p1.1.m1.1.2.2.3">𝒞</ci></apply><ci id="S5.SS3.p1.1.m1.1.1.1.1a.cmml" xref="S5.SS3.p1.1.m1.1.1.1.3"><mtext mathsize="70%" id="S5.SS3.p1.1.m1.1.1.1.1.cmml" xref="S5.SS3.p1.1.m1.1.1.1.1">retrieved</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p1.1.m1.1c">{\mathcal{D}_{\mathcal{C}}^{(\text{retrieved})}}</annotation></semantics></math> (139K images), we progressively add in increasing amounts of synthetic images from our synthetic adaptation dataset <math id="S5.SS3.p1.2.m2.1" class="ltx_Math" alttext="{\mathcal{D}_{\mathcal{C}}^{(\text{synthetic})}}" display="inline"><semantics id="S5.SS3.p1.2.m2.1a"><msubsup id="S5.SS3.p1.2.m2.1.2" xref="S5.SS3.p1.2.m2.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS3.p1.2.m2.1.2.2.2" xref="S5.SS3.p1.2.m2.1.2.2.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="S5.SS3.p1.2.m2.1.2.2.3" xref="S5.SS3.p1.2.m2.1.2.2.3.cmml">𝒞</mi><mrow id="S5.SS3.p1.2.m2.1.1.1.3" xref="S5.SS3.p1.2.m2.1.1.1.1a.cmml"><mo stretchy="false" id="S5.SS3.p1.2.m2.1.1.1.3.1" xref="S5.SS3.p1.2.m2.1.1.1.1a.cmml">(</mo><mtext id="S5.SS3.p1.2.m2.1.1.1.1" xref="S5.SS3.p1.2.m2.1.1.1.1.cmml">synthetic</mtext><mo stretchy="false" id="S5.SS3.p1.2.m2.1.1.1.3.2" xref="S5.SS3.p1.2.m2.1.1.1.1a.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S5.SS3.p1.2.m2.1b"><apply id="S5.SS3.p1.2.m2.1.2.cmml" xref="S5.SS3.p1.2.m2.1.2"><csymbol cd="ambiguous" id="S5.SS3.p1.2.m2.1.2.1.cmml" xref="S5.SS3.p1.2.m2.1.2">superscript</csymbol><apply id="S5.SS3.p1.2.m2.1.2.2.cmml" xref="S5.SS3.p1.2.m2.1.2"><csymbol cd="ambiguous" id="S5.SS3.p1.2.m2.1.2.2.1.cmml" xref="S5.SS3.p1.2.m2.1.2">subscript</csymbol><ci id="S5.SS3.p1.2.m2.1.2.2.2.cmml" xref="S5.SS3.p1.2.m2.1.2.2.2">𝒟</ci><ci id="S5.SS3.p1.2.m2.1.2.2.3.cmml" xref="S5.SS3.p1.2.m2.1.2.2.3">𝒞</ci></apply><ci id="S5.SS3.p1.2.m2.1.1.1.1a.cmml" xref="S5.SS3.p1.2.m2.1.1.1.3"><mtext mathsize="70%" id="S5.SS3.p1.2.m2.1.1.1.1.cmml" xref="S5.SS3.p1.2.m2.1.1.1.1">synthetic</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS3.p1.2.m2.1c">{\mathcal{D}_{\mathcal{C}}^{(\text{synthetic})}}</annotation></semantics></math> and finetune a pretrained CLIP model with the resulting mix. We plot results in Figure <a href="#S5.F7.fig1" title="Figure 7 ‣ 5.3 Mixing synthetic and retrieved data ‣ 5 Analysis ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>.
We find that training on the mixed images outperforms training on synthetic images alone; however, training on a mix significantly drops performance compared to using retrieved data alone.</p>
</div>
<figure id="S5.F7.fig1" class="ltx_figure ltx_figure_panel ltx_minipage ltx_align_middle" style="width:164.8pt;">
<div id="S5.F7.1" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_middle" style="width:264.5pt;">
<img src="/html/2406.05184/assets/x7.png" id="S5.F7.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="248" alt="Refer to caption">
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>
We finetune a pretrained CLIP model (<span id="S5.F7.fig1.6.1" class="ltx_text ltx_font_bold" style="color:#D285C2;">dashed purple line</span>) on a dataset comprising a mix of targeted synthetic and retrieved real images (<span id="S5.F7.fig1.7.2" class="ltx_text ltx_font_bold" style="color:#029E73;">green squares</span>). The mix is constructed by adding varying amounts of synthetic images to our full retrieved adaptation dataset. Training on the mix outperforms training on synthetic images alone (<span id="S5.F7.fig1.8.3" class="ltx_text ltx_font_bold" style="color:#DF670B;">orange triangles</span>), but is strictly worse than training on retrieved images alone (<span id="S5.F7.fig1.9.4" class="ltx_text ltx_font_bold" style="color:#0174B2;">blue circles</span>).
</figcaption>
</figure>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Discussion</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">Our work sought to answer a key question: given that all model-generated synthetic images derive from the generator’s upstream training data, does training on synthetic images provide value over training on the parts of the upstream real data directly? We contribute a set of rigorous experiments to ground this question empirically, and discover that training on upstream real images collected via our simple retrieval baseline significantly outperforms training on synthetic images. Our initial question is answered negatively. We therefore argue that retrieval is a critical baseline to surpass in order to show value from synthetic training data, and encourage comparison against it in future research.</p>
</div>
<div id="S6.p2" class="ltx_para">
<p id="S6.p2.1" class="ltx_p">Importantly, we do not seek to make normative claims about whether training with synthetic images will ever surpass this baseline—future work may unlock gains that we have not yet found. As a first step, we contribute analyses of why synthetic training images underperform upstream real images, finding that both generator artifacts and semantic errors within synthetic images are key areas for future improvement. Furthermore, given that image retrieval is a strong alternative to image synthesis, a natural next step is to generate image compositions that are explicitly absent from the generator’s upstream training dataset; synthesizing these “missing" images may offer unique value beyond the existing upstream real images. Such an approach leverages the compositional generalization abilities of the generator, which recent research promisingly suggests may be stronger than the compositionality of a discriminative model trained on the same upstream data <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib37" title="" class="ltx_ref">37</a>, <a href="#bib.bib13" title="" class="ltx_ref">13</a>]</cite>.</p>
</div>
<div id="S6.p3" class="ltx_para">
<p id="S6.p3.1" class="ltx_p">Finally, our findings assume access to the generative model’s upstream training data, an assumption that may not always hold. The upstream pool may be proprietary or strictly regulated due to privacy concerns. In such settings, training directly on the upstream data is impossible; synthetic data from a generative model trained on this unavailable upstream data remains an exciting alternative to acquire otherwise inaccessible information.</p>
</div>
<section id="S6.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Limitations.</h4>

<div id="S6.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S6.SS0.SSS0.Px1.p1.1" class="ltx_p">As an empirical study, our compute budget limits the number of experimental variations we consider. Our results are derived from adapting CLIP models with standard full finetuning; we conjecture that our findings generalize to other large-scale pretrained backbones and adaptation methods as well, but we were not able to test this empirically. Moreover, at the time of our work, Stable Diffusion is the only text-to-image model with publicly available training data to retrieve from (i.e. LAION-2B); we do not study other generators trained on other data pools.
Finally, we focus on model accuracy, leaving a comparison of model robustness and fairness from training on synthetic versus real data to future work.</p>
</div>
</section>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Acknowledgments and Disclosure of Funding</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p">We graciously thank (in alphabetical order) Eric Frankel, Jacqueline He, Athena Tsu, and Rui Xin for their helpful comments and feedback. SG is supported by the NSF GRFP.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Achiam et al. [2023]</span>
<span class="ltx_bibblock">
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.

</span>
<span class="ltx_bibblock">Gpt-4 technical report.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2303.08774</em>, 2023.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Albalak et al. [2024]</span>
<span class="ltx_bibblock">
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al.

</span>
<span class="ltx_bibblock">A survey on data selection for language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.16827</em>, 2024.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Azizi et al. [2023]</span>
<span class="ltx_bibblock">
Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet.

</span>
<span class="ltx_bibblock">Synthetic data from diffusion models improves imagenet classification.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2304.08466</em>, 2023.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bahng et al. [2022]</span>
<span class="ltx_bibblock">
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola.

</span>
<span class="ltx_bibblock">Exploring visual prompts for adapting large-scale models.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2203.17274</em>, 2022.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Baradad Jurjo et al. [2021]</span>
<span class="ltx_bibblock">
Manel Baradad Jurjo, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba.

</span>
<span class="ltx_bibblock">Learning to see by looking at noise.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 34:2556–2569, 2021.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Beaumont [2022]</span>
<span class="ltx_bibblock">
Romain Beaumont.

</span>
<span class="ltx_bibblock">Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/rom1504/clip-retrieval" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/rom1504/clip-retrieval</a>, 2022.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Burg et al. [2023]</span>
<span class="ltx_bibblock">
Max F Burg, Florian Wenzel, Dominik Zietlow, Max Horn, Osama Makansi, Francesco Locatello, and Chris Russell.

</span>
<span class="ltx_bibblock">Image retrieval outperforms diffusion models on data augmentation.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">Transactions on Machine Learning Research</em>, 2023.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. [2019]</span>
<span class="ltx_bibblock">
Yuhua Chen, Wen Li, Xiaoran Chen, and Luc Van Gool.

</span>
<span class="ltx_bibblock">Learning semantic segmentation from synthetic data: A geometrically guided input-output adaptation approach.

</span>
<span class="ltx_bibblock">In <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF conference on computer vision and pattern recognition</em>, pages 1841–1850, 2019.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. [2023]</span>
<span class="ltx_bibblock">
Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al.

</span>
<span class="ltx_bibblock">Meditron-70b: Scaling medical pretraining for large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2311.16079</em>, 2023.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cherti et al. [2023]</span>
<span class="ltx_bibblock">
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev.

</span>
<span class="ltx_bibblock">Reproducible scaling laws for contrastive language-image learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition</em>, pages 2818–2829, 2023.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chowdhery et al. [2023]</span>
<span class="ltx_bibblock">
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.

</span>
<span class="ltx_bibblock">Palm: Scaling language modeling with pathways.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Journal of Machine Learning Research</em>, 24(240):1–113, 2023.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cimpoi et al. [2014]</span>
<span class="ltx_bibblock">
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi.

</span>
<span class="ltx_bibblock">Describing textures in the wild.

</span>
<span class="ltx_bibblock">In <em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)</em>, 2014.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Clark and Jaini [2024]</span>
<span class="ltx_bibblock">
Kevin Clark and Priyank Jaini.

</span>
<span class="ltx_bibblock">Text-to-image diffusion models are zero shot classifiers.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Deng et al. [2009]</span>
<span class="ltx_bibblock">
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.

</span>
<span class="ltx_bibblock">Imagenet: A large-scale hierarchical image database.

</span>
<span class="ltx_bibblock">In <em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">2009 IEEE conference on computer vision and pattern recognition</em>, pages 248–255. Ieee, 2009.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dosovitskiy et al. [2015]</span>
<span class="ltx_bibblock">
Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox.

</span>
<span class="ltx_bibblock">Flownet: Learning optical flow with convolutional networks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE international conference on computer vision</em>, pages 2758–2766, 2015.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Douze et al. [2024]</span>
<span class="ltx_bibblock">
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou.

</span>
<span class="ltx_bibblock">The faiss library.

</span>
<span class="ltx_bibblock">2024.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fang et al. [2022]</span>
<span class="ltx_bibblock">
Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt.

</span>
<span class="ltx_bibblock">Data determines distributional robustness in contrastive language image pre-training (clip).

</span>
<span class="ltx_bibblock">In <em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, pages 6216–6234. PMLR, 2022.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fang et al. [2023]</span>
<span class="ltx_bibblock">
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar.

</span>
<span class="ltx_bibblock">Data filtering networks.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2309.17425</em>, 2023.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">French [1999]</span>
<span class="ltx_bibblock">
Robert M French.

</span>
<span class="ltx_bibblock">Catastrophic forgetting in connectionist networks.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">Trends in cognitive sciences</em>, 3(4):128–135, 1999.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gadre et al. [2024]</span>
<span class="ltx_bibblock">
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al.

</span>
<span class="ltx_bibblock">Datacomp: In search of the next generation of multimodal datasets.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Goyal et al. [2023]</span>
<span class="ltx_bibblock">
Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan.

</span>
<span class="ltx_bibblock">Finetune like you pretrain: Improved finetuning of zero-shot vision models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition</em>, pages 19338–19347, 2023.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gururangan et al. [2020]</span>
<span class="ltx_bibblock">
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith.

</span>
<span class="ltx_bibblock">Don’t stop pretraining: Adapt language models to domains and tasks.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2004.10964</em>, 2020.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hammoud et al. [2024]</span>
<span class="ltx_bibblock">
Hasan Abed Al Kader Hammoud, Hani Itani, Fabio Pizzati, Philip Torr, Adel Bibi, and Bernard Ghanem.

</span>
<span class="ltx_bibblock">Synthclip: Are we ready for a fully synthetic clip training?

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.01832</em>, 2024.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">He et al. [2022a]</span>
<span class="ltx_bibblock">
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi.

</span>
<span class="ltx_bibblock">Is synthetic data from generative models ready for image recognition?

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.07574</em>, 2022a.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">He et al. [2022b]</span>
<span class="ltx_bibblock">
Xuanli He, Islam Nassar, Jamie Kiros, Gholamreza Haffari, and Mohammad Norouzi.

</span>
<span class="ltx_bibblock">Generate, annotate, and learn: Nlp with synthetic text.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>, 10:826–842, 2022b.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ho and Salimans [2022]</span>
<span class="ltx_bibblock">
Jonathan Ho and Tim Salimans.

</span>
<span class="ltx_bibblock">Classifier-free diffusion guidance.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2207.12598</em>, 2022.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ho et al. [2020]</span>
<span class="ltx_bibblock">
Jonathan Ho, Ajay Jain, and Pieter Abbeel.

</span>
<span class="ltx_bibblock">Denoising diffusion probabilistic models.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 33:6840–6851, 2020.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hoffmann et al. [2022]</span>
<span class="ltx_bibblock">
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.

</span>
<span class="ltx_bibblock">Training compute-optimal large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2203.15556</em>, 2022.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Honovich et al. [2022]</span>
<span class="ltx_bibblock">
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick.

</span>
<span class="ltx_bibblock">Unnatural instructions: Tuning language models with (almost) no human labor.

</span>
<span class="ltx_bibblock"><em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2212.09689</em>, 2022.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hsieh et al. [2023]</span>
<span class="ltx_bibblock">
Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister.

</span>
<span class="ltx_bibblock">Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes.

</span>
<span class="ltx_bibblock"><em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.02301</em>, 2023.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Johnson-Roberson et al. [2016]</span>
<span class="ltx_bibblock">
Matthew Johnson-Roberson, Charles Barto, Rounak Mehta, Sharath Nittur Sridhar, Karl Rosaen, and Ram Vasudevan.

</span>
<span class="ltx_bibblock">Driving in the matrix: Can virtual worlds replace human-generated annotations for real world tasks?

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1610.01983</em>, 2016.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jung et al. [2023]</span>
<span class="ltx_bibblock">
Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi.

</span>
<span class="ltx_bibblock">Impossible distillation: from low-quality model to high-quality dataset &amp; model for summarization and paraphrasing.

</span>
<span class="ltx_bibblock"><em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.16635</em>, 2023.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kaplan et al. [2020]</span>
<span class="ltx_bibblock">
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.

</span>
<span class="ltx_bibblock">Scaling laws for neural language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2001.08361</em>, 2020.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Koh et al. [2021]</span>
<span class="ltx_bibblock">
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al.

</span>
<span class="ltx_bibblock">Wilds: A benchmark of in-the-wild distribution shifts.

</span>
<span class="ltx_bibblock">In <em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">International conference on machine learning</em>, pages 5637–5664. PMLR, 2021.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Krause et al. [2013]</span>
<span class="ltx_bibblock">
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.

</span>
<span class="ltx_bibblock">3d object representations for fine-grained categorization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE international conference on computer vision workshops</em>, pages 554–561, 2013.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee et al. [2021]</span>
<span class="ltx_bibblock">
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini.

</span>
<span class="ltx_bibblock">Deduplicating training data makes language models better.

</span>
<span class="ltx_bibblock"><em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2107.06499</em>, 2021.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. [2023]</span>
<span class="ltx_bibblock">
Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak.

</span>
<span class="ltx_bibblock">Your diffusion model is secretly a zero-shot classifier.

</span>
<span class="ltx_bibblock">In <em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF International Conference on Computer Vision</em>, pages 2206–2217, 2023.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. [2023]</span>
<span class="ltx_bibblock">
Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jae Lee, and Chunyuan Li.

</span>
<span class="ltx_bibblock">Learning customized visual models with retrieval-augmented knowledge.

</span>
<span class="ltx_bibblock">In <em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition</em>, pages 15148–15158, 2023.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. [2024]</span>
<span class="ltx_bibblock">
Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al.

</span>
<span class="ltx_bibblock">Best practices and lessons learned on synthetic data for language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2404.07503</em>, 2024.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Maji et al. [2013]</span>
<span class="ltx_bibblock">
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.

</span>
<span class="ltx_bibblock">Fine-grained visual classification of aircraft.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1306.5151</em>, 2013.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mao et al. [2022]</span>
<span class="ltx_bibblock">
Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick.

</span>
<span class="ltx_bibblock">Understanding zero-shot adversarial robustness for large-scale models.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2212.07016</em>, 2022.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Meng et al. [2021]</span>
<span class="ltx_bibblock">
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.

</span>
<span class="ltx_bibblock">Sdedit: Guided image synthesis and editing with stochastic differential equations.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2108.01073</em>, 2021.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nguyen et al. [2022]</span>
<span class="ltx_bibblock">
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt.

</span>
<span class="ltx_bibblock">Quality not quantity: On the interaction between dataset design and robustness of clip.

</span>
<span class="ltx_bibblock"><em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 35:21455–21469, 2022.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nguyen et al. [2024]</span>
<span class="ltx_bibblock">
Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt.

</span>
<span class="ltx_bibblock">Improving multimodal datasets with image captioning.

</span>
<span class="ltx_bibblock"><em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nilsback and Zisserman [2008]</span>
<span class="ltx_bibblock">
Maria-Elena Nilsback and Andrew Zisserman.

</span>
<span class="ltx_bibblock">Automated flower classification over a large number of classes.

</span>
<span class="ltx_bibblock">In <em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">2008 Sixth Indian conference on computer vision, graphics &amp; image processing</em>, pages 722–729. IEEE, 2008.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenAI [2022]</span>
<span class="ltx_bibblock">
OpenAI.

</span>
<span class="ltx_bibblock">Chatgpt.

</span>
<span class="ltx_bibblock">2022.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Peng et al. [2015]</span>
<span class="ltx_bibblock">
Xingchao Peng, Baochen Sun, Karim Ali, and Kate Saenko.

</span>
<span class="ltx_bibblock">Learning deep object detectors from 3d models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE international conference on computer vision</em>, pages 1278–1286, 2015.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Peng et al. [2017]</span>
<span class="ltx_bibblock">
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko.

</span>
<span class="ltx_bibblock">Visda: The visual domain adaptation challenge.

</span>
<span class="ltx_bibblock"><em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1710.06924</em>, 2017.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. [2021]</span>
<span class="ltx_bibblock">
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.

</span>
<span class="ltx_bibblock">Learning transferable visual models from natural language supervision.

</span>
<span class="ltx_bibblock">In <em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">International conference on machine learning</em>, pages 8748–8763. PMLR, 2021.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ramanujan et al. [2024]</span>
<span class="ltx_bibblock">
Vivek Ramanujan, Thao Nguyen, Sewoong Oh, Ali Farhadi, and Ludwig Schmidt.

</span>
<span class="ltx_bibblock">On the connection between pre-training data diversity and fine-tuning robustness.

</span>
<span class="ltx_bibblock"><em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Richter et al. [2016]</span>
<span class="ltx_bibblock">
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun.

</span>
<span class="ltx_bibblock">Playing for data: Ground truth from computer games.

</span>
<span class="ltx_bibblock">In <em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14</em>, pages 102–118. Springer, 2016.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rombach et al. [2022]</span>
<span class="ltx_bibblock">
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.

</span>
<span class="ltx_bibblock">High-resolution image synthesis with latent diffusion models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib52.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF conference on computer vision and pattern recognition</em>, pages 10684–10695, 2022.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ros et al. [2016]</span>
<span class="ltx_bibblock">
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez.

</span>
<span class="ltx_bibblock">The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes.

</span>
<span class="ltx_bibblock">In <em id="bib.bib53.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE conference on computer vision and pattern recognition</em>, pages 3234–3243, 2016.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sarıyıldız et al. [2023]</span>
<span class="ltx_bibblock">
Mert Bülent Sarıyıldız, Karteek Alahari, Diane Larlus, and Yannis Kalantidis.

</span>
<span class="ltx_bibblock">Fake it till you make it: Learning transferable representations from synthetic imagenet clones.

</span>
<span class="ltx_bibblock">In <em id="bib.bib54.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition</em>, pages 8011–8021, 2023.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schuhmann et al. [2022]</span>
<span class="ltx_bibblock">
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.

</span>
<span class="ltx_bibblock">Laion-5b: An open large-scale dataset for training next generation image-text models.

</span>
<span class="ltx_bibblock"><em id="bib.bib55.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 35:25278–25294, 2022.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Silver et al. [2017]</span>
<span class="ltx_bibblock">
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.

</span>
<span class="ltx_bibblock">Mastering chess and shogi by self-play with a general reinforcement learning algorithm.

</span>
<span class="ltx_bibblock"><em id="bib.bib56.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1712.01815</em>, 2017.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sohl-Dickstein et al. [2015]</span>
<span class="ltx_bibblock">
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.

</span>
<span class="ltx_bibblock">Deep unsupervised learning using nonequilibrium thermodynamics.

</span>
<span class="ltx_bibblock">In <em id="bib.bib57.1.1" class="ltx_emph ltx_font_italic">International conference on machine learning</em>, pages 2256–2265. PMLR, 2015.

</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Song et al. [2020]</span>
<span class="ltx_bibblock">
Jiaming Song, Chenlin Meng, and Stefano Ermon.

</span>
<span class="ltx_bibblock">Denoising diffusion implicit models.

</span>
<span class="ltx_bibblock"><em id="bib.bib58.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2010.02502</em>, 2020.

</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Taori et al. [2023]</span>
<span class="ltx_bibblock">
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto.

</span>
<span class="ltx_bibblock">Alpaca: A strong, replicable instruction-following model.

</span>
<span class="ltx_bibblock"><em id="bib.bib59.1.1" class="ltx_emph ltx_font_italic">Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html</em>, 3(6):7, 2023.

</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tian et al. [2023]</span>
<span class="ltx_bibblock">
Yonglong Tian, Lijie Fan, Kaifeng Chen, Dina Katabi, Dilip Krishnan, and Phillip Isola.

</span>
<span class="ltx_bibblock">Learning vision from models rivals learning vision from data.

</span>
<span class="ltx_bibblock"><em id="bib.bib60.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2312.17742</em>, 2023.

</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tian et al. [2024]</span>
<span class="ltx_bibblock">
Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan.

</span>
<span class="ltx_bibblock">Stablerep: Synthetic images from text-to-image models make strong visual representation learners.

</span>
<span class="ltx_bibblock"><em id="bib.bib61.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. [2023]</span>
<span class="ltx_bibblock">
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.

</span>
<span class="ltx_bibblock">Llama 2: Open foundation and fine-tuned chat models.

</span>
<span class="ltx_bibblock"><em id="bib.bib62.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2307.09288</em>, 2023.

</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Udandarao et al. [2024]</span>
<span class="ltx_bibblock">
Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip HS Torr, Adel Bibi, Samuel Albanie, and Matthias Bethge.

</span>
<span class="ltx_bibblock">No" zero-shot" without exponential data: Pretraining concept frequency determines multimodal model performance.

</span>
<span class="ltx_bibblock"><em id="bib.bib63.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2404.04125</em>, 2024.

</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vasu et al. [2023]</span>
<span class="ltx_bibblock">
Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, and Oncel Tuzel.

</span>
<span class="ltx_bibblock">Mobileclip: Fast image-text models through multi-modal reinforced training.

</span>
<span class="ltx_bibblock"><em id="bib.bib64.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2311.17049</em>, 2023.

</span>
</li>
<li id="bib.bib65" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wallingford et al. [2024]</span>
<span class="ltx_bibblock">
Matthew Wallingford, Vivek Ramanujan, Alex Fang, Aditya Kusupati, Roozbeh Mottaghi, Aniruddha Kembhavi, Ludwig Schmidt, and Ali Farhadi.

</span>
<span class="ltx_bibblock">Neural priming for sample-efficient adaptation.

</span>
<span class="ltx_bibblock"><em id="bib.bib65.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib66" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. [2022]</span>
<span class="ltx_bibblock">
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi.

</span>
<span class="ltx_bibblock">Self-instruct: Aligning language models with self-generated instructions.

</span>
<span class="ltx_bibblock"><em id="bib.bib66.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2212.10560</em>, 2022.

</span>
</li>
<li id="bib.bib67" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wortsman et al. [2022]</span>
<span class="ltx_bibblock">
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al.

</span>
<span class="ltx_bibblock">Robust fine-tuning of zero-shot models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib67.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE/CVF conference on computer vision and pattern recognition</em>, pages 7959–7971, 2022.

</span>
</li>
<li id="bib.bib68" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. [2023]</span>
<span class="ltx_bibblock">
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer.

</span>
<span class="ltx_bibblock">Demystifying clip data.

</span>
<span class="ltx_bibblock"><em id="bib.bib68.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2309.16671</em>, 2023.

</span>
</li>
<li id="bib.bib69" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. [2021]</span>
<span class="ltx_bibblock">
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li.

</span>
<span class="ltx_bibblock">Tip-adapter: Training-free clip-adapter for better vision-language modeling.

</span>
<span class="ltx_bibblock"><em id="bib.bib69.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2111.03930</em>, 2021.

</span>
</li>
<li id="bib.bib70" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. [2024]</span>
<span class="ltx_bibblock">
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al.

</span>
<span class="ltx_bibblock">Lima: Less is more for alignment.

</span>
<span class="ltx_bibblock"><em id="bib.bib70.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 36, 2024.

</span>
</li>
<li id="bib.bib71" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. [2022]</span>
<span class="ltx_bibblock">
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.

</span>
<span class="ltx_bibblock">Learning to prompt for vision-language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib71.1.1" class="ltx_emph ltx_font_italic">International Journal of Computer Vision</em>, 130(9):2337–2348, 2022.

</span>
</li>
<li id="bib.bib72" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. [2023]</span>
<span class="ltx_bibblock">
Yongchao Zhou, Hshmat Sahak, and Jimmy Ba.

</span>
<span class="ltx_bibblock">Using synthetic data for data augmentation to improve classification accuracy.

</span>
<span class="ltx_bibblock">2023.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Broader Impacts</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.1" class="ltx_p">We release all models and synthetic data from our work to benefit future research. All assets released are meant to be scientific research artifacts. Moreover, all models released are task-specific classification models, limiting potential for misuse. Nonetheless, we do not encourage the use or deployment of our models in practice.</p>
</div>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Details in Methodology</h2>

<section id="A2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.1 </span>Sourcing data by generating synthetic images</h3>

<div id="A2.SS1.p1" class="ltx_para">
<p id="A2.SS1.p1.3" class="ltx_p">Given a set of visual class names <math id="A2.SS1.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{C}" display="inline"><semantics id="A2.SS1.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="A2.SS1.p1.1.m1.1.1" xref="A2.SS1.p1.1.m1.1.1.cmml">𝒞</mi><annotation-xml encoding="MathML-Content" id="A2.SS1.p1.1.m1.1b"><ci id="A2.SS1.p1.1.m1.1.1.cmml" xref="A2.SS1.p1.1.m1.1.1">𝒞</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p1.1.m1.1c">\mathcal{C}</annotation></semantics></math> from our target task, we first synthesize a large corpus of image captions for each class name by prompting a large language model (we use LLamA-2 7B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib62" title="" class="ltx_ref">62</a>]</cite>). For each concept name <math id="A2.SS1.p1.2.m2.1" class="ltx_Math" alttext="c\in\mathcal{C}" display="inline"><semantics id="A2.SS1.p1.2.m2.1a"><mrow id="A2.SS1.p1.2.m2.1.1" xref="A2.SS1.p1.2.m2.1.1.cmml"><mi id="A2.SS1.p1.2.m2.1.1.2" xref="A2.SS1.p1.2.m2.1.1.2.cmml">c</mi><mo id="A2.SS1.p1.2.m2.1.1.1" xref="A2.SS1.p1.2.m2.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="A2.SS1.p1.2.m2.1.1.3" xref="A2.SS1.p1.2.m2.1.1.3.cmml">𝒞</mi></mrow><annotation-xml encoding="MathML-Content" id="A2.SS1.p1.2.m2.1b"><apply id="A2.SS1.p1.2.m2.1.1.cmml" xref="A2.SS1.p1.2.m2.1.1"><in id="A2.SS1.p1.2.m2.1.1.1.cmml" xref="A2.SS1.p1.2.m2.1.1.1"></in><ci id="A2.SS1.p1.2.m2.1.1.2.cmml" xref="A2.SS1.p1.2.m2.1.1.2">𝑐</ci><ci id="A2.SS1.p1.2.m2.1.1.3.cmml" xref="A2.SS1.p1.2.m2.1.1.3">𝒞</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p1.2.m2.1c">c\in\mathcal{C}</annotation></semantics></math>, we use three type of prompts to convert <math id="A2.SS1.p1.3.m3.1" class="ltx_Math" alttext="c" display="inline"><semantics id="A2.SS1.p1.3.m3.1a"><mi id="A2.SS1.p1.3.m3.1.1" xref="A2.SS1.p1.3.m3.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="A2.SS1.p1.3.m3.1b"><ci id="A2.SS1.p1.3.m3.1.1.cmml" xref="A2.SS1.p1.3.m3.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p1.3.m3.1c">c</annotation></semantics></math> into an image caption following <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite>. For the sake of completeness, we detail the prompts here:</p>
</div>
<div id="A2.SS1.p2" class="ltx_para">
<p id="A2.SS1.p2.1" class="ltx_p">1. <math id="A2.SS1.p2.1.m1.1" class="ltx_Math" alttext="c\mapsto" display="inline"><semantics id="A2.SS1.p2.1.m1.1a"><mrow id="A2.SS1.p2.1.m1.1.1" xref="A2.SS1.p2.1.m1.1.1.cmml"><mi id="A2.SS1.p2.1.m1.1.1.2" xref="A2.SS1.p2.1.m1.1.1.2.cmml">c</mi><mo stretchy="false" id="A2.SS1.p2.1.m1.1.1.1" xref="A2.SS1.p2.1.m1.1.1.1.cmml">↦</mo><mi id="A2.SS1.p2.1.m1.1.1.3" xref="A2.SS1.p2.1.m1.1.1.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="A2.SS1.p2.1.m1.1b"><apply id="A2.SS1.p2.1.m1.1.1.cmml" xref="A2.SS1.p2.1.m1.1.1"><csymbol cd="latexml" id="A2.SS1.p2.1.m1.1.1.1.cmml" xref="A2.SS1.p2.1.m1.1.1.1">maps-to</csymbol><ci id="A2.SS1.p2.1.m1.1.1.2.cmml" xref="A2.SS1.p2.1.m1.1.1.2">𝑐</ci><csymbol cd="latexml" id="A2.SS1.p2.1.m1.1.1.3.cmml" xref="A2.SS1.p2.1.m1.1.1.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p2.1.m1.1c">c\mapsto</annotation></semantics></math> caption. We prompt the language model (LM) to directly translate the class name into a caption using a prompt with 3 few-shot in-context examples.</p>
</div>
<div id="A2.SS1.p3" class="ltx_para">
<p id="A2.SS1.p3.5" class="ltx_p">2. <math id="A2.SS1.p3.1.m1.2" class="ltx_Math" alttext="c,background\mapsto" display="inline"><semantics id="A2.SS1.p3.1.m1.2a"><mrow id="A2.SS1.p3.1.m1.2.2" xref="A2.SS1.p3.1.m1.2.2.cmml"><mrow id="A2.SS1.p3.1.m1.2.2.1.1" xref="A2.SS1.p3.1.m1.2.2.1.2.cmml"><mi id="A2.SS1.p3.1.m1.1.1" xref="A2.SS1.p3.1.m1.1.1.cmml">c</mi><mo id="A2.SS1.p3.1.m1.2.2.1.1.2" xref="A2.SS1.p3.1.m1.2.2.1.2.cmml">,</mo><mrow id="A2.SS1.p3.1.m1.2.2.1.1.1" xref="A2.SS1.p3.1.m1.2.2.1.1.1.cmml"><mi id="A2.SS1.p3.1.m1.2.2.1.1.1.2" xref="A2.SS1.p3.1.m1.2.2.1.1.1.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.1.m1.2.2.1.1.1.1" xref="A2.SS1.p3.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p3.1.m1.2.2.1.1.1.3" xref="A2.SS1.p3.1.m1.2.2.1.1.1.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.1.m1.2.2.1.1.1.1a" xref="A2.SS1.p3.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p3.1.m1.2.2.1.1.1.4" xref="A2.SS1.p3.1.m1.2.2.1.1.1.4.cmml">c</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.1.m1.2.2.1.1.1.1b" xref="A2.SS1.p3.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p3.1.m1.2.2.1.1.1.5" xref="A2.SS1.p3.1.m1.2.2.1.1.1.5.cmml">k</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.1.m1.2.2.1.1.1.1c" xref="A2.SS1.p3.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p3.1.m1.2.2.1.1.1.6" xref="A2.SS1.p3.1.m1.2.2.1.1.1.6.cmml">g</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.1.m1.2.2.1.1.1.1d" xref="A2.SS1.p3.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p3.1.m1.2.2.1.1.1.7" xref="A2.SS1.p3.1.m1.2.2.1.1.1.7.cmml">r</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.1.m1.2.2.1.1.1.1e" xref="A2.SS1.p3.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p3.1.m1.2.2.1.1.1.8" xref="A2.SS1.p3.1.m1.2.2.1.1.1.8.cmml">o</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.1.m1.2.2.1.1.1.1f" xref="A2.SS1.p3.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p3.1.m1.2.2.1.1.1.9" xref="A2.SS1.p3.1.m1.2.2.1.1.1.9.cmml">u</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.1.m1.2.2.1.1.1.1g" xref="A2.SS1.p3.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p3.1.m1.2.2.1.1.1.10" xref="A2.SS1.p3.1.m1.2.2.1.1.1.10.cmml">n</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.1.m1.2.2.1.1.1.1h" xref="A2.SS1.p3.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p3.1.m1.2.2.1.1.1.11" xref="A2.SS1.p3.1.m1.2.2.1.1.1.11.cmml">d</mi></mrow></mrow><mo stretchy="false" id="A2.SS1.p3.1.m1.2.2.2" xref="A2.SS1.p3.1.m1.2.2.2.cmml">↦</mo><mi id="A2.SS1.p3.1.m1.2.2.3" xref="A2.SS1.p3.1.m1.2.2.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="A2.SS1.p3.1.m1.2b"><apply id="A2.SS1.p3.1.m1.2.2.cmml" xref="A2.SS1.p3.1.m1.2.2"><csymbol cd="latexml" id="A2.SS1.p3.1.m1.2.2.2.cmml" xref="A2.SS1.p3.1.m1.2.2.2">maps-to</csymbol><list id="A2.SS1.p3.1.m1.2.2.1.2.cmml" xref="A2.SS1.p3.1.m1.2.2.1.1"><ci id="A2.SS1.p3.1.m1.1.1.cmml" xref="A2.SS1.p3.1.m1.1.1">𝑐</ci><apply id="A2.SS1.p3.1.m1.2.2.1.1.1.cmml" xref="A2.SS1.p3.1.m1.2.2.1.1.1"><times id="A2.SS1.p3.1.m1.2.2.1.1.1.1.cmml" xref="A2.SS1.p3.1.m1.2.2.1.1.1.1"></times><ci id="A2.SS1.p3.1.m1.2.2.1.1.1.2.cmml" xref="A2.SS1.p3.1.m1.2.2.1.1.1.2">𝑏</ci><ci id="A2.SS1.p3.1.m1.2.2.1.1.1.3.cmml" xref="A2.SS1.p3.1.m1.2.2.1.1.1.3">𝑎</ci><ci id="A2.SS1.p3.1.m1.2.2.1.1.1.4.cmml" xref="A2.SS1.p3.1.m1.2.2.1.1.1.4">𝑐</ci><ci id="A2.SS1.p3.1.m1.2.2.1.1.1.5.cmml" xref="A2.SS1.p3.1.m1.2.2.1.1.1.5">𝑘</ci><ci id="A2.SS1.p3.1.m1.2.2.1.1.1.6.cmml" xref="A2.SS1.p3.1.m1.2.2.1.1.1.6">𝑔</ci><ci id="A2.SS1.p3.1.m1.2.2.1.1.1.7.cmml" xref="A2.SS1.p3.1.m1.2.2.1.1.1.7">𝑟</ci><ci id="A2.SS1.p3.1.m1.2.2.1.1.1.8.cmml" xref="A2.SS1.p3.1.m1.2.2.1.1.1.8">𝑜</ci><ci id="A2.SS1.p3.1.m1.2.2.1.1.1.9.cmml" xref="A2.SS1.p3.1.m1.2.2.1.1.1.9">𝑢</ci><ci id="A2.SS1.p3.1.m1.2.2.1.1.1.10.cmml" xref="A2.SS1.p3.1.m1.2.2.1.1.1.10">𝑛</ci><ci id="A2.SS1.p3.1.m1.2.2.1.1.1.11.cmml" xref="A2.SS1.p3.1.m1.2.2.1.1.1.11">𝑑</ci></apply></list><csymbol cd="latexml" id="A2.SS1.p3.1.m1.2.2.3.cmml" xref="A2.SS1.p3.1.m1.2.2.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p3.1.m1.2c">c,background\mapsto</annotation></semantics></math> caption. We prompt the LM with an additional background attribute that is randomly sampled from a set that is predetermined based on the domain of <math id="A2.SS1.p3.2.m2.1" class="ltx_Math" alttext="\mathcal{C}" display="inline"><semantics id="A2.SS1.p3.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="A2.SS1.p3.2.m2.1.1" xref="A2.SS1.p3.2.m2.1.1.cmml">𝒞</mi><annotation-xml encoding="MathML-Content" id="A2.SS1.p3.2.m2.1b"><ci id="A2.SS1.p3.2.m2.1.1.cmml" xref="A2.SS1.p3.2.m2.1.1">𝒞</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p3.2.m2.1c">\mathcal{C}</annotation></semantics></math>. For example, if <math id="A2.SS1.p3.3.m3.1" class="ltx_Math" alttext="\mathcal{C}" display="inline"><semantics id="A2.SS1.p3.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="A2.SS1.p3.3.m3.1.1" xref="A2.SS1.p3.3.m3.1.1.cmml">𝒞</mi><annotation-xml encoding="MathML-Content" id="A2.SS1.p3.3.m3.1b"><ci id="A2.SS1.p3.3.m3.1.1.cmml" xref="A2.SS1.p3.3.m3.1.1">𝒞</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p3.3.m3.1c">\mathcal{C}</annotation></semantics></math> contains a list of flower names, then possible background attributes might include “garden," “meadow," or “forest." These background attributes are automatically generated by prompting a strong instruction-tuned language model such as GPT-4 <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib1" title="" class="ltx_ref">1</a>]</cite> with the class names <math id="A2.SS1.p3.4.m4.1" class="ltx_Math" alttext="\mathcal{C}" display="inline"><semantics id="A2.SS1.p3.4.m4.1a"><mi class="ltx_font_mathcaligraphic" id="A2.SS1.p3.4.m4.1.1" xref="A2.SS1.p3.4.m4.1.1.cmml">𝒞</mi><annotation-xml encoding="MathML-Content" id="A2.SS1.p3.4.m4.1b"><ci id="A2.SS1.p3.4.m4.1.1.cmml" xref="A2.SS1.p3.4.m4.1.1">𝒞</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p3.4.m4.1c">\mathcal{C}</annotation></semantics></math>. We provide the LM with 3 in-context examples of <math id="A2.SS1.p3.5.m5.2" class="ltx_Math" alttext="c,background\mapsto" display="inline"><semantics id="A2.SS1.p3.5.m5.2a"><mrow id="A2.SS1.p3.5.m5.2.2" xref="A2.SS1.p3.5.m5.2.2.cmml"><mrow id="A2.SS1.p3.5.m5.2.2.1.1" xref="A2.SS1.p3.5.m5.2.2.1.2.cmml"><mi id="A2.SS1.p3.5.m5.1.1" xref="A2.SS1.p3.5.m5.1.1.cmml">c</mi><mo id="A2.SS1.p3.5.m5.2.2.1.1.2" xref="A2.SS1.p3.5.m5.2.2.1.2.cmml">,</mo><mrow id="A2.SS1.p3.5.m5.2.2.1.1.1" xref="A2.SS1.p3.5.m5.2.2.1.1.1.cmml"><mi id="A2.SS1.p3.5.m5.2.2.1.1.1.2" xref="A2.SS1.p3.5.m5.2.2.1.1.1.2.cmml">b</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.5.m5.2.2.1.1.1.1" xref="A2.SS1.p3.5.m5.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p3.5.m5.2.2.1.1.1.3" xref="A2.SS1.p3.5.m5.2.2.1.1.1.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.5.m5.2.2.1.1.1.1a" xref="A2.SS1.p3.5.m5.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p3.5.m5.2.2.1.1.1.4" xref="A2.SS1.p3.5.m5.2.2.1.1.1.4.cmml">c</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.5.m5.2.2.1.1.1.1b" xref="A2.SS1.p3.5.m5.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p3.5.m5.2.2.1.1.1.5" xref="A2.SS1.p3.5.m5.2.2.1.1.1.5.cmml">k</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.5.m5.2.2.1.1.1.1c" xref="A2.SS1.p3.5.m5.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p3.5.m5.2.2.1.1.1.6" xref="A2.SS1.p3.5.m5.2.2.1.1.1.6.cmml">g</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.5.m5.2.2.1.1.1.1d" xref="A2.SS1.p3.5.m5.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p3.5.m5.2.2.1.1.1.7" xref="A2.SS1.p3.5.m5.2.2.1.1.1.7.cmml">r</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.5.m5.2.2.1.1.1.1e" xref="A2.SS1.p3.5.m5.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p3.5.m5.2.2.1.1.1.8" xref="A2.SS1.p3.5.m5.2.2.1.1.1.8.cmml">o</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.5.m5.2.2.1.1.1.1f" xref="A2.SS1.p3.5.m5.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p3.5.m5.2.2.1.1.1.9" xref="A2.SS1.p3.5.m5.2.2.1.1.1.9.cmml">u</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.5.m5.2.2.1.1.1.1g" xref="A2.SS1.p3.5.m5.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p3.5.m5.2.2.1.1.1.10" xref="A2.SS1.p3.5.m5.2.2.1.1.1.10.cmml">n</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p3.5.m5.2.2.1.1.1.1h" xref="A2.SS1.p3.5.m5.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p3.5.m5.2.2.1.1.1.11" xref="A2.SS1.p3.5.m5.2.2.1.1.1.11.cmml">d</mi></mrow></mrow><mo stretchy="false" id="A2.SS1.p3.5.m5.2.2.2" xref="A2.SS1.p3.5.m5.2.2.2.cmml">↦</mo><mi id="A2.SS1.p3.5.m5.2.2.3" xref="A2.SS1.p3.5.m5.2.2.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="A2.SS1.p3.5.m5.2b"><apply id="A2.SS1.p3.5.m5.2.2.cmml" xref="A2.SS1.p3.5.m5.2.2"><csymbol cd="latexml" id="A2.SS1.p3.5.m5.2.2.2.cmml" xref="A2.SS1.p3.5.m5.2.2.2">maps-to</csymbol><list id="A2.SS1.p3.5.m5.2.2.1.2.cmml" xref="A2.SS1.p3.5.m5.2.2.1.1"><ci id="A2.SS1.p3.5.m5.1.1.cmml" xref="A2.SS1.p3.5.m5.1.1">𝑐</ci><apply id="A2.SS1.p3.5.m5.2.2.1.1.1.cmml" xref="A2.SS1.p3.5.m5.2.2.1.1.1"><times id="A2.SS1.p3.5.m5.2.2.1.1.1.1.cmml" xref="A2.SS1.p3.5.m5.2.2.1.1.1.1"></times><ci id="A2.SS1.p3.5.m5.2.2.1.1.1.2.cmml" xref="A2.SS1.p3.5.m5.2.2.1.1.1.2">𝑏</ci><ci id="A2.SS1.p3.5.m5.2.2.1.1.1.3.cmml" xref="A2.SS1.p3.5.m5.2.2.1.1.1.3">𝑎</ci><ci id="A2.SS1.p3.5.m5.2.2.1.1.1.4.cmml" xref="A2.SS1.p3.5.m5.2.2.1.1.1.4">𝑐</ci><ci id="A2.SS1.p3.5.m5.2.2.1.1.1.5.cmml" xref="A2.SS1.p3.5.m5.2.2.1.1.1.5">𝑘</ci><ci id="A2.SS1.p3.5.m5.2.2.1.1.1.6.cmml" xref="A2.SS1.p3.5.m5.2.2.1.1.1.6">𝑔</ci><ci id="A2.SS1.p3.5.m5.2.2.1.1.1.7.cmml" xref="A2.SS1.p3.5.m5.2.2.1.1.1.7">𝑟</ci><ci id="A2.SS1.p3.5.m5.2.2.1.1.1.8.cmml" xref="A2.SS1.p3.5.m5.2.2.1.1.1.8">𝑜</ci><ci id="A2.SS1.p3.5.m5.2.2.1.1.1.9.cmml" xref="A2.SS1.p3.5.m5.2.2.1.1.1.9">𝑢</ci><ci id="A2.SS1.p3.5.m5.2.2.1.1.1.10.cmml" xref="A2.SS1.p3.5.m5.2.2.1.1.1.10">𝑛</ci><ci id="A2.SS1.p3.5.m5.2.2.1.1.1.11.cmml" xref="A2.SS1.p3.5.m5.2.2.1.1.1.11">𝑑</ci></apply></list><csymbol cd="latexml" id="A2.SS1.p3.5.m5.2.2.3.cmml" xref="A2.SS1.p3.5.m5.2.2.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p3.5.m5.2c">c,background\mapsto</annotation></semantics></math> caption mappings.</p>
</div>
<div id="A2.SS1.p4" class="ltx_para">
<p id="A2.SS1.p4.2" class="ltx_p">3. <math id="A2.SS1.p4.1.m1.2" class="ltx_Math" alttext="c,relation\mapsto" display="inline"><semantics id="A2.SS1.p4.1.m1.2a"><mrow id="A2.SS1.p4.1.m1.2.2" xref="A2.SS1.p4.1.m1.2.2.cmml"><mrow id="A2.SS1.p4.1.m1.2.2.1.1" xref="A2.SS1.p4.1.m1.2.2.1.2.cmml"><mi id="A2.SS1.p4.1.m1.1.1" xref="A2.SS1.p4.1.m1.1.1.cmml">c</mi><mo id="A2.SS1.p4.1.m1.2.2.1.1.2" xref="A2.SS1.p4.1.m1.2.2.1.2.cmml">,</mo><mrow id="A2.SS1.p4.1.m1.2.2.1.1.1" xref="A2.SS1.p4.1.m1.2.2.1.1.1.cmml"><mi id="A2.SS1.p4.1.m1.2.2.1.1.1.2" xref="A2.SS1.p4.1.m1.2.2.1.1.1.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p4.1.m1.2.2.1.1.1.1" xref="A2.SS1.p4.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p4.1.m1.2.2.1.1.1.3" xref="A2.SS1.p4.1.m1.2.2.1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p4.1.m1.2.2.1.1.1.1a" xref="A2.SS1.p4.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p4.1.m1.2.2.1.1.1.4" xref="A2.SS1.p4.1.m1.2.2.1.1.1.4.cmml">l</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p4.1.m1.2.2.1.1.1.1b" xref="A2.SS1.p4.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p4.1.m1.2.2.1.1.1.5" xref="A2.SS1.p4.1.m1.2.2.1.1.1.5.cmml">a</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p4.1.m1.2.2.1.1.1.1c" xref="A2.SS1.p4.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p4.1.m1.2.2.1.1.1.6" xref="A2.SS1.p4.1.m1.2.2.1.1.1.6.cmml">t</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p4.1.m1.2.2.1.1.1.1d" xref="A2.SS1.p4.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p4.1.m1.2.2.1.1.1.7" xref="A2.SS1.p4.1.m1.2.2.1.1.1.7.cmml">i</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p4.1.m1.2.2.1.1.1.1e" xref="A2.SS1.p4.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p4.1.m1.2.2.1.1.1.8" xref="A2.SS1.p4.1.m1.2.2.1.1.1.8.cmml">o</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p4.1.m1.2.2.1.1.1.1f" xref="A2.SS1.p4.1.m1.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p4.1.m1.2.2.1.1.1.9" xref="A2.SS1.p4.1.m1.2.2.1.1.1.9.cmml">n</mi></mrow></mrow><mo stretchy="false" id="A2.SS1.p4.1.m1.2.2.2" xref="A2.SS1.p4.1.m1.2.2.2.cmml">↦</mo><mi id="A2.SS1.p4.1.m1.2.2.3" xref="A2.SS1.p4.1.m1.2.2.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="A2.SS1.p4.1.m1.2b"><apply id="A2.SS1.p4.1.m1.2.2.cmml" xref="A2.SS1.p4.1.m1.2.2"><csymbol cd="latexml" id="A2.SS1.p4.1.m1.2.2.2.cmml" xref="A2.SS1.p4.1.m1.2.2.2">maps-to</csymbol><list id="A2.SS1.p4.1.m1.2.2.1.2.cmml" xref="A2.SS1.p4.1.m1.2.2.1.1"><ci id="A2.SS1.p4.1.m1.1.1.cmml" xref="A2.SS1.p4.1.m1.1.1">𝑐</ci><apply id="A2.SS1.p4.1.m1.2.2.1.1.1.cmml" xref="A2.SS1.p4.1.m1.2.2.1.1.1"><times id="A2.SS1.p4.1.m1.2.2.1.1.1.1.cmml" xref="A2.SS1.p4.1.m1.2.2.1.1.1.1"></times><ci id="A2.SS1.p4.1.m1.2.2.1.1.1.2.cmml" xref="A2.SS1.p4.1.m1.2.2.1.1.1.2">𝑟</ci><ci id="A2.SS1.p4.1.m1.2.2.1.1.1.3.cmml" xref="A2.SS1.p4.1.m1.2.2.1.1.1.3">𝑒</ci><ci id="A2.SS1.p4.1.m1.2.2.1.1.1.4.cmml" xref="A2.SS1.p4.1.m1.2.2.1.1.1.4">𝑙</ci><ci id="A2.SS1.p4.1.m1.2.2.1.1.1.5.cmml" xref="A2.SS1.p4.1.m1.2.2.1.1.1.5">𝑎</ci><ci id="A2.SS1.p4.1.m1.2.2.1.1.1.6.cmml" xref="A2.SS1.p4.1.m1.2.2.1.1.1.6">𝑡</ci><ci id="A2.SS1.p4.1.m1.2.2.1.1.1.7.cmml" xref="A2.SS1.p4.1.m1.2.2.1.1.1.7">𝑖</ci><ci id="A2.SS1.p4.1.m1.2.2.1.1.1.8.cmml" xref="A2.SS1.p4.1.m1.2.2.1.1.1.8">𝑜</ci><ci id="A2.SS1.p4.1.m1.2.2.1.1.1.9.cmml" xref="A2.SS1.p4.1.m1.2.2.1.1.1.9">𝑛</ci></apply></list><csymbol cd="latexml" id="A2.SS1.p4.1.m1.2.2.3.cmml" xref="A2.SS1.p4.1.m1.2.2.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p4.1.m1.2c">c,relation\mapsto</annotation></semantics></math> caption. We prompt with an additional spatial relationship attribute that is sampled from a domain-invariant set of relationships, such as “next to," “below," “besides," etc. We provide 3 in-context examples of <math id="A2.SS1.p4.2.m2.2" class="ltx_Math" alttext="c,relation\mapsto" display="inline"><semantics id="A2.SS1.p4.2.m2.2a"><mrow id="A2.SS1.p4.2.m2.2.2" xref="A2.SS1.p4.2.m2.2.2.cmml"><mrow id="A2.SS1.p4.2.m2.2.2.1.1" xref="A2.SS1.p4.2.m2.2.2.1.2.cmml"><mi id="A2.SS1.p4.2.m2.1.1" xref="A2.SS1.p4.2.m2.1.1.cmml">c</mi><mo id="A2.SS1.p4.2.m2.2.2.1.1.2" xref="A2.SS1.p4.2.m2.2.2.1.2.cmml">,</mo><mrow id="A2.SS1.p4.2.m2.2.2.1.1.1" xref="A2.SS1.p4.2.m2.2.2.1.1.1.cmml"><mi id="A2.SS1.p4.2.m2.2.2.1.1.1.2" xref="A2.SS1.p4.2.m2.2.2.1.1.1.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p4.2.m2.2.2.1.1.1.1" xref="A2.SS1.p4.2.m2.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p4.2.m2.2.2.1.1.1.3" xref="A2.SS1.p4.2.m2.2.2.1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p4.2.m2.2.2.1.1.1.1a" xref="A2.SS1.p4.2.m2.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p4.2.m2.2.2.1.1.1.4" xref="A2.SS1.p4.2.m2.2.2.1.1.1.4.cmml">l</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p4.2.m2.2.2.1.1.1.1b" xref="A2.SS1.p4.2.m2.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p4.2.m2.2.2.1.1.1.5" xref="A2.SS1.p4.2.m2.2.2.1.1.1.5.cmml">a</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p4.2.m2.2.2.1.1.1.1c" xref="A2.SS1.p4.2.m2.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p4.2.m2.2.2.1.1.1.6" xref="A2.SS1.p4.2.m2.2.2.1.1.1.6.cmml">t</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p4.2.m2.2.2.1.1.1.1d" xref="A2.SS1.p4.2.m2.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p4.2.m2.2.2.1.1.1.7" xref="A2.SS1.p4.2.m2.2.2.1.1.1.7.cmml">i</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p4.2.m2.2.2.1.1.1.1e" xref="A2.SS1.p4.2.m2.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p4.2.m2.2.2.1.1.1.8" xref="A2.SS1.p4.2.m2.2.2.1.1.1.8.cmml">o</mi><mo lspace="0em" rspace="0em" id="A2.SS1.p4.2.m2.2.2.1.1.1.1f" xref="A2.SS1.p4.2.m2.2.2.1.1.1.1.cmml">​</mo><mi id="A2.SS1.p4.2.m2.2.2.1.1.1.9" xref="A2.SS1.p4.2.m2.2.2.1.1.1.9.cmml">n</mi></mrow></mrow><mo stretchy="false" id="A2.SS1.p4.2.m2.2.2.2" xref="A2.SS1.p4.2.m2.2.2.2.cmml">↦</mo><mi id="A2.SS1.p4.2.m2.2.2.3" xref="A2.SS1.p4.2.m2.2.2.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="A2.SS1.p4.2.m2.2b"><apply id="A2.SS1.p4.2.m2.2.2.cmml" xref="A2.SS1.p4.2.m2.2.2"><csymbol cd="latexml" id="A2.SS1.p4.2.m2.2.2.2.cmml" xref="A2.SS1.p4.2.m2.2.2.2">maps-to</csymbol><list id="A2.SS1.p4.2.m2.2.2.1.2.cmml" xref="A2.SS1.p4.2.m2.2.2.1.1"><ci id="A2.SS1.p4.2.m2.1.1.cmml" xref="A2.SS1.p4.2.m2.1.1">𝑐</ci><apply id="A2.SS1.p4.2.m2.2.2.1.1.1.cmml" xref="A2.SS1.p4.2.m2.2.2.1.1.1"><times id="A2.SS1.p4.2.m2.2.2.1.1.1.1.cmml" xref="A2.SS1.p4.2.m2.2.2.1.1.1.1"></times><ci id="A2.SS1.p4.2.m2.2.2.1.1.1.2.cmml" xref="A2.SS1.p4.2.m2.2.2.1.1.1.2">𝑟</ci><ci id="A2.SS1.p4.2.m2.2.2.1.1.1.3.cmml" xref="A2.SS1.p4.2.m2.2.2.1.1.1.3">𝑒</ci><ci id="A2.SS1.p4.2.m2.2.2.1.1.1.4.cmml" xref="A2.SS1.p4.2.m2.2.2.1.1.1.4">𝑙</ci><ci id="A2.SS1.p4.2.m2.2.2.1.1.1.5.cmml" xref="A2.SS1.p4.2.m2.2.2.1.1.1.5">𝑎</ci><ci id="A2.SS1.p4.2.m2.2.2.1.1.1.6.cmml" xref="A2.SS1.p4.2.m2.2.2.1.1.1.6">𝑡</ci><ci id="A2.SS1.p4.2.m2.2.2.1.1.1.7.cmml" xref="A2.SS1.p4.2.m2.2.2.1.1.1.7">𝑖</ci><ci id="A2.SS1.p4.2.m2.2.2.1.1.1.8.cmml" xref="A2.SS1.p4.2.m2.2.2.1.1.1.8">𝑜</ci><ci id="A2.SS1.p4.2.m2.2.2.1.1.1.9.cmml" xref="A2.SS1.p4.2.m2.2.2.1.1.1.9">𝑛</ci></apply></list><csymbol cd="latexml" id="A2.SS1.p4.2.m2.2.2.3.cmml" xref="A2.SS1.p4.2.m2.2.2.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p4.2.m2.2c">c,relation\mapsto</annotation></semantics></math> caption mappings.</p>
</div>
<div id="A2.SS1.p5" class="ltx_para ltx_noindent">
<p id="A2.SS1.p5.1" class="ltx_p">Each of these captions are directly used as text input to Stable Diffusion 1.5 to produce our targeted synthetic dataset <math id="A2.SS1.p5.1.m1.1" class="ltx_Math" alttext="{\mathcal{D}_{\mathcal{C}}^{(\text{synthetic})}}" display="inline"><semantics id="A2.SS1.p5.1.m1.1a"><msubsup id="A2.SS1.p5.1.m1.1.2" xref="A2.SS1.p5.1.m1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="A2.SS1.p5.1.m1.1.2.2.2" xref="A2.SS1.p5.1.m1.1.2.2.2.cmml">𝒟</mi><mi class="ltx_font_mathcaligraphic" id="A2.SS1.p5.1.m1.1.2.2.3" xref="A2.SS1.p5.1.m1.1.2.2.3.cmml">𝒞</mi><mrow id="A2.SS1.p5.1.m1.1.1.1.3" xref="A2.SS1.p5.1.m1.1.1.1.1a.cmml"><mo stretchy="false" id="A2.SS1.p5.1.m1.1.1.1.3.1" xref="A2.SS1.p5.1.m1.1.1.1.1a.cmml">(</mo><mtext id="A2.SS1.p5.1.m1.1.1.1.1" xref="A2.SS1.p5.1.m1.1.1.1.1.cmml">synthetic</mtext><mo stretchy="false" id="A2.SS1.p5.1.m1.1.1.1.3.2" xref="A2.SS1.p5.1.m1.1.1.1.1a.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="A2.SS1.p5.1.m1.1b"><apply id="A2.SS1.p5.1.m1.1.2.cmml" xref="A2.SS1.p5.1.m1.1.2"><csymbol cd="ambiguous" id="A2.SS1.p5.1.m1.1.2.1.cmml" xref="A2.SS1.p5.1.m1.1.2">superscript</csymbol><apply id="A2.SS1.p5.1.m1.1.2.2.cmml" xref="A2.SS1.p5.1.m1.1.2"><csymbol cd="ambiguous" id="A2.SS1.p5.1.m1.1.2.2.1.cmml" xref="A2.SS1.p5.1.m1.1.2">subscript</csymbol><ci id="A2.SS1.p5.1.m1.1.2.2.2.cmml" xref="A2.SS1.p5.1.m1.1.2.2.2">𝒟</ci><ci id="A2.SS1.p5.1.m1.1.2.2.3.cmml" xref="A2.SS1.p5.1.m1.1.2.2.3">𝒞</ci></apply><ci id="A2.SS1.p5.1.m1.1.1.1.1a.cmml" xref="A2.SS1.p5.1.m1.1.1.1.3"><mtext mathsize="70%" id="A2.SS1.p5.1.m1.1.1.1.1.cmml" xref="A2.SS1.p5.1.m1.1.1.1.1">synthetic</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.p5.1.m1.1c">{\mathcal{D}_{\mathcal{C}}^{(\text{synthetic})}}</annotation></semantics></math>. When sampling from Stable Diffusion, we denoise for 50 DDIM <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib58" title="" class="ltx_ref">58</a>]</cite> steps starting from Gaussian noise, using a classifier-free guidance <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib26" title="" class="ltx_ref">26</a>]</cite> scale of 2.0.</p>
</div>
<div id="A2.SS1.p6" class="ltx_para">
<p id="A2.SS1.p6.1" class="ltx_p">Generating the images can be computationally expensive; every 1M synthetic images generated (pre-filtering) requires around 12 hours of generation on 64 NVIDIA A100 GPUs. To lower the barrier for future research, we will release our generated synthetic images at <a target="_blank" href="https://github.com/scottgeng00/unmet-promise/" title="" class="ltx_ref ltx_href ltx_font_typewriter">https://github.com/scottgeng00/unmet-promise</a>.</p>
</div>
</section>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Details in Experimental Setup</h2>

<section id="A3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.1 </span>Retrieval hyperparameters</h3>

<div id="A3.SS1.p1" class="ltx_para">
<p id="A3.SS1.p1.6" class="ltx_p">We perform <math id="A3.SS1.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A3.SS1.p1.1.m1.1a"><mi id="A3.SS1.p1.1.m1.1.1" xref="A3.SS1.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A3.SS1.p1.1.m1.1b"><ci id="A3.SS1.p1.1.m1.1.1.cmml" xref="A3.SS1.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.SS1.p1.1.m1.1c">k</annotation></semantics></math>-NN retrieval with <math id="A3.SS1.p1.2.m2.1" class="ltx_Math" alttext="k=2000" display="inline"><semantics id="A3.SS1.p1.2.m2.1a"><mrow id="A3.SS1.p1.2.m2.1.1" xref="A3.SS1.p1.2.m2.1.1.cmml"><mi id="A3.SS1.p1.2.m2.1.1.2" xref="A3.SS1.p1.2.m2.1.1.2.cmml">k</mi><mo id="A3.SS1.p1.2.m2.1.1.1" xref="A3.SS1.p1.2.m2.1.1.1.cmml">=</mo><mn id="A3.SS1.p1.2.m2.1.1.3" xref="A3.SS1.p1.2.m2.1.1.3.cmml">2000</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.SS1.p1.2.m2.1b"><apply id="A3.SS1.p1.2.m2.1.1.cmml" xref="A3.SS1.p1.2.m2.1.1"><eq id="A3.SS1.p1.2.m2.1.1.1.cmml" xref="A3.SS1.p1.2.m2.1.1.1"></eq><ci id="A3.SS1.p1.2.m2.1.1.2.cmml" xref="A3.SS1.p1.2.m2.1.1.2">𝑘</ci><cn type="integer" id="A3.SS1.p1.2.m2.1.1.3.cmml" xref="A3.SS1.p1.2.m2.1.1.3">2000</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS1.p1.2.m2.1c">k=2000</annotation></semantics></math> for every downstream benchmark except ImageNet-1K, where we use <math id="A3.SS1.p1.3.m3.1" class="ltx_Math" alttext="k=500" display="inline"><semantics id="A3.SS1.p1.3.m3.1a"><mrow id="A3.SS1.p1.3.m3.1.1" xref="A3.SS1.p1.3.m3.1.1.cmml"><mi id="A3.SS1.p1.3.m3.1.1.2" xref="A3.SS1.p1.3.m3.1.1.2.cmml">k</mi><mo id="A3.SS1.p1.3.m3.1.1.1" xref="A3.SS1.p1.3.m3.1.1.1.cmml">=</mo><mn id="A3.SS1.p1.3.m3.1.1.3" xref="A3.SS1.p1.3.m3.1.1.3.cmml">500</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.SS1.p1.3.m3.1b"><apply id="A3.SS1.p1.3.m3.1.1.cmml" xref="A3.SS1.p1.3.m3.1.1"><eq id="A3.SS1.p1.3.m3.1.1.1.cmml" xref="A3.SS1.p1.3.m3.1.1.1"></eq><ci id="A3.SS1.p1.3.m3.1.1.2.cmml" xref="A3.SS1.p1.3.m3.1.1.2">𝑘</ci><cn type="integer" id="A3.SS1.p1.3.m3.1.1.3.cmml" xref="A3.SS1.p1.3.m3.1.1.3">500</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS1.p1.3.m3.1c">k=500</annotation></semantics></math>. We picked these values of <math id="A3.SS1.p1.4.m4.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A3.SS1.p1.4.m4.1a"><mi id="A3.SS1.p1.4.m4.1.1" xref="A3.SS1.p1.4.m4.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A3.SS1.p1.4.m4.1b"><ci id="A3.SS1.p1.4.m4.1.1.cmml" xref="A3.SS1.p1.4.m4.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.SS1.p1.4.m4.1c">k</annotation></semantics></math> with a rough target of retrieving 10K images per class. In particular, the original CLIP paper <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib49" title="" class="ltx_ref">49</a>]</cite> has a different set of class template strings for each benchmark; our query sets <math id="A3.SS1.p1.5.m5.1" class="ltx_Math" alttext="Q_{c}" display="inline"><semantics id="A3.SS1.p1.5.m5.1a"><msub id="A3.SS1.p1.5.m5.1.1" xref="A3.SS1.p1.5.m5.1.1.cmml"><mi id="A3.SS1.p1.5.m5.1.1.2" xref="A3.SS1.p1.5.m5.1.1.2.cmml">Q</mi><mi id="A3.SS1.p1.5.m5.1.1.3" xref="A3.SS1.p1.5.m5.1.1.3.cmml">c</mi></msub><annotation-xml encoding="MathML-Content" id="A3.SS1.p1.5.m5.1b"><apply id="A3.SS1.p1.5.m5.1.1.cmml" xref="A3.SS1.p1.5.m5.1.1"><csymbol cd="ambiguous" id="A3.SS1.p1.5.m5.1.1.1.cmml" xref="A3.SS1.p1.5.m5.1.1">subscript</csymbol><ci id="A3.SS1.p1.5.m5.1.1.2.cmml" xref="A3.SS1.p1.5.m5.1.1.2">𝑄</ci><ci id="A3.SS1.p1.5.m5.1.1.3.cmml" xref="A3.SS1.p1.5.m5.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS1.p1.5.m5.1c">Q_{c}</annotation></semantics></math> for each benchmark are differently sized, and our values of <math id="A3.SS1.p1.6.m6.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A3.SS1.p1.6.m6.1a"><mi id="A3.SS1.p1.6.m6.1.1" xref="A3.SS1.p1.6.m6.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A3.SS1.p1.6.m6.1b"><ci id="A3.SS1.p1.6.m6.1.1.cmml" xref="A3.SS1.p1.6.m6.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.SS1.p1.6.m6.1c">k</annotation></semantics></math> vary to reflect that.</p>
</div>
<div id="A3.SS1.p2" class="ltx_para">
<p id="A3.SS1.p2.3" class="ltx_p">For class balancing, we set <math id="A3.SS1.p2.1.m1.1" class="ltx_Math" alttext="M=10000" display="inline"><semantics id="A3.SS1.p2.1.m1.1a"><mrow id="A3.SS1.p2.1.m1.1.1" xref="A3.SS1.p2.1.m1.1.1.cmml"><mi id="A3.SS1.p2.1.m1.1.1.2" xref="A3.SS1.p2.1.m1.1.1.2.cmml">M</mi><mo id="A3.SS1.p2.1.m1.1.1.1" xref="A3.SS1.p2.1.m1.1.1.1.cmml">=</mo><mn id="A3.SS1.p2.1.m1.1.1.3" xref="A3.SS1.p2.1.m1.1.1.3.cmml">10000</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.SS1.p2.1.m1.1b"><apply id="A3.SS1.p2.1.m1.1.1.cmml" xref="A3.SS1.p2.1.m1.1.1"><eq id="A3.SS1.p2.1.m1.1.1.1.cmml" xref="A3.SS1.p2.1.m1.1.1.1"></eq><ci id="A3.SS1.p2.1.m1.1.1.2.cmml" xref="A3.SS1.p2.1.m1.1.1.2">𝑀</ci><cn type="integer" id="A3.SS1.p2.1.m1.1.1.3.cmml" xref="A3.SS1.p2.1.m1.1.1.3">10000</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS1.p2.1.m1.1c">M=10000</annotation></semantics></math>. We do not tune <math id="A3.SS1.p2.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A3.SS1.p2.2.m2.1a"><mi id="A3.SS1.p2.2.m2.1.1" xref="A3.SS1.p2.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A3.SS1.p2.2.m2.1b"><ci id="A3.SS1.p2.2.m2.1.1.cmml" xref="A3.SS1.p2.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.SS1.p2.2.m2.1c">k</annotation></semantics></math> or <math id="A3.SS1.p2.3.m3.1" class="ltx_Math" alttext="M" display="inline"><semantics id="A3.SS1.p2.3.m3.1a"><mi id="A3.SS1.p2.3.m3.1.1" xref="A3.SS1.p2.3.m3.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="A3.SS1.p2.3.m3.1b"><ci id="A3.SS1.p2.3.m3.1.1.cmml" xref="A3.SS1.p2.3.m3.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.SS1.p2.3.m3.1c">M</annotation></semantics></math> in our experiments.</p>
</div>
<div id="A3.SS1.p3" class="ltx_para">
<p id="A3.SS1.p3.1" class="ltx_p">Choosing based on downstream validation set accuracy, we use our substring retrieval strategy for FGVC-Aircraft and Flowers102; we use our semantic retrieval strategy for ImageNet-1K, DTD, and StanfordCars.</p>
</div>
<div id="A3.SS1.p4" class="ltx_para">
<p id="A3.SS1.p4.1" class="ltx_p">We use precomputed <math id="A3.SS1.p4.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A3.SS1.p4.1.m1.1a"><mi id="A3.SS1.p4.1.m1.1.1" xref="A3.SS1.p4.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A3.SS1.p4.1.m1.1b"><ci id="A3.SS1.p4.1.m1.1.1.cmml" xref="A3.SS1.p4.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.SS1.p4.1.m1.1c">k</annotation></semantics></math>-NN search indicies from LAION-2B <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib55" title="" class="ltx_ref">55</a>]</cite> to query against OpenAI CLIP ViT-L/14 image embeddings. No search indicies are available for querying against text embeddings; we construct our own using FAISS <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite>, using the configuration <span id="A3.SS1.p4.1.1" class="ltx_text ltx_font_typewriter">OPQ256 768,IVF131072 HNSW32,PQ256x8</span>. Computing 2 billion OpenAI CLIP ViT-L/14 text embeddings for the captions in LAION-2B took approximately 2 hours on 100 GPUs of varying capacity. Computing the search index from the embeddings took approximately 12 hours on 128 CPU cores.</p>
</div>
</section>
<section id="A3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.2 </span>Details for downstream benchmarks</h3>

<div id="A3.SS2.p1" class="ltx_para">
<p id="A3.SS2.p1.1" class="ltx_p">We use the standard pre-defined train-test-validation splits for FGVC-Aircraft, DTD, and Flowers-102. Standard validation splits are not available for StanfordCars and ImageNet-1K. We construct a train-validation split for StanfordCars by randomly splitting 80% and 20% of the pre-defined training set (respectively) using <span id="A3.SS2.p1.1.1" class="ltx_text ltx_font_typewriter">torch.utils.data.random_split</span> with random seed 42. We construct a validation set for ImageNet-1K by randomly subsampling 50K images from the pre-defined training set using <span id="A3.SS2.p1.1.2" class="ltx_text ltx_font_typewriter">torch.utils.data.random_split</span> with random seed 42.</p>
</div>
</section>
<section id="A3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.3 </span>Details for model adaptation</h3>

<section id="A3.SS3.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Finetuning details.</h4>

<div id="A3.SS3.SSS0.Px1.p1" class="ltx_para">
<p id="A3.SS3.SSS0.Px1.p1.3" class="ltx_p">To finetune CLIP for a specific downstream image classification task, we first initialize a linear readout head <math id="A3.SS3.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="W" display="inline"><semantics id="A3.SS3.SSS0.Px1.p1.1.m1.1a"><mi id="A3.SS3.SSS0.Px1.p1.1.m1.1.1" xref="A3.SS3.SSS0.Px1.p1.1.m1.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="A3.SS3.SSS0.Px1.p1.1.m1.1b"><ci id="A3.SS3.SSS0.Px1.p1.1.m1.1.1.cmml" xref="A3.SS3.SSS0.Px1.p1.1.m1.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.SS3.SSS0.Px1.p1.1.m1.1c">W</annotation></semantics></math> using the weights from the text-based zero-shot CLIP model <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib10" title="" class="ltx_ref">10</a>]</cite>. Concretely, we initialize <math id="A3.SS3.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="W" display="inline"><semantics id="A3.SS3.SSS0.Px1.p1.2.m2.1a"><mi id="A3.SS3.SSS0.Px1.p1.2.m2.1.1" xref="A3.SS3.SSS0.Px1.p1.2.m2.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="A3.SS3.SSS0.Px1.p1.2.m2.1b"><ci id="A3.SS3.SSS0.Px1.p1.2.m2.1.1.cmml" xref="A3.SS3.SSS0.Px1.p1.2.m2.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.SS3.SSS0.Px1.p1.2.m2.1c">W</annotation></semantics></math> using the CLIP text embeddings of the class names for the desired downstream task. We then append the classification head <math id="A3.SS3.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="W" display="inline"><semantics id="A3.SS3.SSS0.Px1.p1.3.m3.1a"><mi id="A3.SS3.SSS0.Px1.p1.3.m3.1.1" xref="A3.SS3.SSS0.Px1.p1.3.m3.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="A3.SS3.SSS0.Px1.p1.3.m3.1b"><ci id="A3.SS3.SSS0.Px1.p1.3.m3.1.1.cmml" xref="A3.SS3.SSS0.Px1.p1.3.m3.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.SS3.SSS0.Px1.p1.3.m3.1c">W</annotation></semantics></math> on top of CLIP’s vision encoder, and train end-to-end using a standard cross entropy classification loss against one-hot labels.</p>
</div>
<div id="A3.SS3.SSS0.Px1.p2" class="ltx_para">
<p id="A3.SS3.SSS0.Px1.p2.1" class="ltx_p">We could alternatively choose to finetune CLIP with a contrastive objective, where each positive pair is a synthetic or retrieved image alongside its corresponding caption. However, we find that cross entropy finetuning performs better across the board, so we use cross entropy finetuning for all experiments in our paper.</p>
</div>
<div id="A3.SS3.SSS0.Px1.p3" class="ltx_para">
<p id="A3.SS3.SSS0.Px1.p3.1" class="ltx_p">A full adaptation dataset scale sweep for a single benchmark and a fixed set of hyperparameters takes approximately 24-36 hours on 2 NVIDIA A40 GPUs.</p>
</div>
</section>
<section id="A3.SS3.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Random seed.</h4>

<div id="A3.SS3.SSS0.Px2.p1" class="ltx_para">
<p id="A3.SS3.SSS0.Px2.p1.1" class="ltx_p">For our main experiments, generator ablations, and data mixing experiments we report results aggregated across at least three random seeds. The random seed is used to (1) seed the training algorithm, and (2) controls adaptation dataset subsampling.</p>
</div>
</section>
<section id="A3.SS3.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Hyperparameter details.</h4>

<div id="A3.SS3.SSS0.Px3.p1" class="ltx_para">
<p id="A3.SS3.SSS0.Px3.p1.3" class="ltx_p">We start with relatively standard hyperparameters from prior work <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib67" title="" class="ltx_ref">67</a>]</cite>, and initially tune them in our setting by finetuning CLIP on a small-scale dataset of retrieved or synthetic images from each downstream benchmark and grid-sweeping by learning rate and batch size. From the hyperparameters we tried at this scale, we find the following work best for both synthetic and retrieved images across all downstream benchmarks:</p>
<ul id="A3.I1" class="ltx_itemize">
<li id="A3.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I1.i1.p1" class="ltx_para">
<p id="A3.I1.i1.p1.1" class="ltx_p">Batch size: 512</p>
</div>
</li>
<li id="A3.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I1.i2.p1" class="ltx_para">
<p id="A3.I1.i2.p1.1" class="ltx_p">Warmup steps: 500</p>
</div>
</li>
<li id="A3.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I1.i3.p1" class="ltx_para">
<p id="A3.I1.i3.p1.1" class="ltx_p">LR schedule: Cosine decay</p>
</div>
</li>
<li id="A3.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A3.I1.i4.p1" class="ltx_para">
<p id="A3.I1.i4.p1.1" class="ltx_p">L2 weight decay: 0.1</p>
</div>
</li>
</ul>
<p id="A3.SS3.SSS0.Px3.p1.2" class="ltx_p">We find that models are sensitive to learning rate; there is no one optimal learning rate across all settings. Thus, for our full-scale experiments, we sweep learning rate across <math id="A3.SS3.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="\{" display="inline"><semantics id="A3.SS3.SSS0.Px3.p1.1.m1.1a"><mo stretchy="false" id="A3.SS3.SSS0.Px3.p1.1.m1.1.1" xref="A3.SS3.SSS0.Px3.p1.1.m1.1.1.cmml">{</mo><annotation-xml encoding="MathML-Content" id="A3.SS3.SSS0.Px3.p1.1.m1.1b"><ci id="A3.SS3.SSS0.Px3.p1.1.m1.1.1.cmml" xref="A3.SS3.SSS0.Px3.p1.1.m1.1.1">{</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.SS3.SSS0.Px3.p1.1.m1.1c">\{</annotation></semantics></math>5e-4, 1e-5, 1e-6<math id="A3.SS3.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="\}" display="inline"><semantics id="A3.SS3.SSS0.Px3.p1.2.m2.1a"><mo stretchy="false" id="A3.SS3.SSS0.Px3.p1.2.m2.1.1" xref="A3.SS3.SSS0.Px3.p1.2.m2.1.1.cmml">}</mo><annotation-xml encoding="MathML-Content" id="A3.SS3.SSS0.Px3.p1.2.m2.1b"><ci id="A3.SS3.SSS0.Px3.p1.2.m2.1.1.cmml" xref="A3.SS3.SSS0.Px3.p1.2.m2.1.1">}</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.SS3.SSS0.Px3.p1.2.m2.1c">\}</annotation></semantics></math>, and select the best learning rate for each downstream benchmark based on validation set accuracy.</p>
</div>
<div id="A3.SS3.SSS0.Px3.p2" class="ltx_para">
<p id="A3.SS3.SSS0.Px3.p2.1" class="ltx_p">We train with an AdamW optimizer, using <math id="A3.SS3.SSS0.Px3.p2.1.m1.2" class="ltx_Math" alttext="\beta_{1}=0.9,\beta_{2}=0.95" display="inline"><semantics id="A3.SS3.SSS0.Px3.p2.1.m1.2a"><mrow id="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2" xref="A3.SS3.SSS0.Px3.p2.1.m1.2.2.3.cmml"><mrow id="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1" xref="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.cmml"><msub id="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.2" xref="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.2.cmml"><mi id="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.2.2" xref="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.2.2.cmml">β</mi><mn id="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.2.3" xref="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.2.3.cmml">1</mn></msub><mo id="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.1" xref="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.1.cmml">=</mo><mn id="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.3" xref="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.3.cmml">0.9</mn></mrow><mo id="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.3" xref="A3.SS3.SSS0.Px3.p2.1.m1.2.2.3a.cmml">,</mo><mrow id="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2" xref="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.cmml"><msub id="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.2" xref="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.2.cmml"><mi id="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.2.2" xref="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.2.2.cmml">β</mi><mn id="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.2.3" xref="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.2.3.cmml">2</mn></msub><mo id="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.1" xref="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.1.cmml">=</mo><mn id="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.3" xref="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.3.cmml">0.95</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.SS3.SSS0.Px3.p2.1.m1.2b"><apply id="A3.SS3.SSS0.Px3.p2.1.m1.2.2.3.cmml" xref="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2"><csymbol cd="ambiguous" id="A3.SS3.SSS0.Px3.p2.1.m1.2.2.3a.cmml" xref="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.3">formulae-sequence</csymbol><apply id="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.cmml" xref="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1"><eq id="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.1.cmml" xref="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.1"></eq><apply id="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.2.cmml" xref="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.2"><csymbol cd="ambiguous" id="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.2.1.cmml" xref="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.2">subscript</csymbol><ci id="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.2.2.cmml" xref="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.2.2">𝛽</ci><cn type="integer" id="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.2.3.cmml" xref="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.2.3">1</cn></apply><cn type="float" id="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.3.cmml" xref="A3.SS3.SSS0.Px3.p2.1.m1.1.1.1.1.3">0.9</cn></apply><apply id="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.cmml" xref="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2"><eq id="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.1.cmml" xref="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.1"></eq><apply id="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.2.cmml" xref="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.2.1.cmml" xref="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.2">subscript</csymbol><ci id="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.2.2.cmml" xref="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.2.2">𝛽</ci><cn type="integer" id="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.2.3.cmml" xref="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.2.3">2</cn></apply><cn type="float" id="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.3.cmml" xref="A3.SS3.SSS0.Px3.p2.1.m1.2.2.2.2.3">0.95</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS3.SSS0.Px3.p2.1.m1.2c">\beta_{1}=0.9,\beta_{2}=0.95</annotation></semantics></math>.</p>
</div>
<div id="A3.SS3.SSS0.Px3.p3" class="ltx_para">
<p id="A3.SS3.SSS0.Px3.p3.1" class="ltx_p">On all benchmarks except ImageNet, we finetune for a fixed 30 epochs. On ImageNet, we train for a fixed 10 epochs to save compute, as we found that validation set accuracy plateaued early on.</p>
</div>
</section>
</section>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Details in Analysis Experiments</h2>

<section id="A4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">D.1 </span>Synthetically perturbing retrieved images</h3>

<div id="A4.SS1.p1" class="ltx_para">
<p id="A4.SS1.p1.2" class="ltx_p">We use SDEdit <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib42" title="" class="ltx_ref">42</a>]</cite> to synthetically perturb the retrieved real images by adding Gaussian noise and then denoising with Stable Diffusion 1.5. We study the perturbation across 10 different choices of the noise scale (i.e. perturbation strength) parameter <math id="A4.SS1.p1.1.m1.10" class="ltx_Math" alttext="\gamma\in\{0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0\}" display="inline"><semantics id="A4.SS1.p1.1.m1.10a"><mrow id="A4.SS1.p1.1.m1.10.11" xref="A4.SS1.p1.1.m1.10.11.cmml"><mi id="A4.SS1.p1.1.m1.10.11.2" xref="A4.SS1.p1.1.m1.10.11.2.cmml">γ</mi><mo id="A4.SS1.p1.1.m1.10.11.1" xref="A4.SS1.p1.1.m1.10.11.1.cmml">∈</mo><mrow id="A4.SS1.p1.1.m1.10.11.3.2" xref="A4.SS1.p1.1.m1.10.11.3.1.cmml"><mo stretchy="false" id="A4.SS1.p1.1.m1.10.11.3.2.1" xref="A4.SS1.p1.1.m1.10.11.3.1.cmml">{</mo><mn id="A4.SS1.p1.1.m1.1.1" xref="A4.SS1.p1.1.m1.1.1.cmml">0.1</mn><mo id="A4.SS1.p1.1.m1.10.11.3.2.2" xref="A4.SS1.p1.1.m1.10.11.3.1.cmml">,</mo><mn id="A4.SS1.p1.1.m1.2.2" xref="A4.SS1.p1.1.m1.2.2.cmml">0.2</mn><mo id="A4.SS1.p1.1.m1.10.11.3.2.3" xref="A4.SS1.p1.1.m1.10.11.3.1.cmml">,</mo><mn id="A4.SS1.p1.1.m1.3.3" xref="A4.SS1.p1.1.m1.3.3.cmml">0.3</mn><mo id="A4.SS1.p1.1.m1.10.11.3.2.4" xref="A4.SS1.p1.1.m1.10.11.3.1.cmml">,</mo><mn id="A4.SS1.p1.1.m1.4.4" xref="A4.SS1.p1.1.m1.4.4.cmml">0.4</mn><mo id="A4.SS1.p1.1.m1.10.11.3.2.5" xref="A4.SS1.p1.1.m1.10.11.3.1.cmml">,</mo><mn id="A4.SS1.p1.1.m1.5.5" xref="A4.SS1.p1.1.m1.5.5.cmml">0.5</mn><mo id="A4.SS1.p1.1.m1.10.11.3.2.6" xref="A4.SS1.p1.1.m1.10.11.3.1.cmml">,</mo><mn id="A4.SS1.p1.1.m1.6.6" xref="A4.SS1.p1.1.m1.6.6.cmml">0.6</mn><mo id="A4.SS1.p1.1.m1.10.11.3.2.7" xref="A4.SS1.p1.1.m1.10.11.3.1.cmml">,</mo><mn id="A4.SS1.p1.1.m1.7.7" xref="A4.SS1.p1.1.m1.7.7.cmml">0.7</mn><mo id="A4.SS1.p1.1.m1.10.11.3.2.8" xref="A4.SS1.p1.1.m1.10.11.3.1.cmml">,</mo><mn id="A4.SS1.p1.1.m1.8.8" xref="A4.SS1.p1.1.m1.8.8.cmml">0.8</mn><mo id="A4.SS1.p1.1.m1.10.11.3.2.9" xref="A4.SS1.p1.1.m1.10.11.3.1.cmml">,</mo><mn id="A4.SS1.p1.1.m1.9.9" xref="A4.SS1.p1.1.m1.9.9.cmml">0.9</mn><mo id="A4.SS1.p1.1.m1.10.11.3.2.10" xref="A4.SS1.p1.1.m1.10.11.3.1.cmml">,</mo><mn id="A4.SS1.p1.1.m1.10.10" xref="A4.SS1.p1.1.m1.10.10.cmml">1.0</mn><mo stretchy="false" id="A4.SS1.p1.1.m1.10.11.3.2.11" xref="A4.SS1.p1.1.m1.10.11.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.SS1.p1.1.m1.10b"><apply id="A4.SS1.p1.1.m1.10.11.cmml" xref="A4.SS1.p1.1.m1.10.11"><in id="A4.SS1.p1.1.m1.10.11.1.cmml" xref="A4.SS1.p1.1.m1.10.11.1"></in><ci id="A4.SS1.p1.1.m1.10.11.2.cmml" xref="A4.SS1.p1.1.m1.10.11.2">𝛾</ci><set id="A4.SS1.p1.1.m1.10.11.3.1.cmml" xref="A4.SS1.p1.1.m1.10.11.3.2"><cn type="float" id="A4.SS1.p1.1.m1.1.1.cmml" xref="A4.SS1.p1.1.m1.1.1">0.1</cn><cn type="float" id="A4.SS1.p1.1.m1.2.2.cmml" xref="A4.SS1.p1.1.m1.2.2">0.2</cn><cn type="float" id="A4.SS1.p1.1.m1.3.3.cmml" xref="A4.SS1.p1.1.m1.3.3">0.3</cn><cn type="float" id="A4.SS1.p1.1.m1.4.4.cmml" xref="A4.SS1.p1.1.m1.4.4">0.4</cn><cn type="float" id="A4.SS1.p1.1.m1.5.5.cmml" xref="A4.SS1.p1.1.m1.5.5">0.5</cn><cn type="float" id="A4.SS1.p1.1.m1.6.6.cmml" xref="A4.SS1.p1.1.m1.6.6">0.6</cn><cn type="float" id="A4.SS1.p1.1.m1.7.7.cmml" xref="A4.SS1.p1.1.m1.7.7">0.7</cn><cn type="float" id="A4.SS1.p1.1.m1.8.8.cmml" xref="A4.SS1.p1.1.m1.8.8">0.8</cn><cn type="float" id="A4.SS1.p1.1.m1.9.9.cmml" xref="A4.SS1.p1.1.m1.9.9">0.9</cn><cn type="float" id="A4.SS1.p1.1.m1.10.10.cmml" xref="A4.SS1.p1.1.m1.10.10">1.0</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.SS1.p1.1.m1.10c">\gamma\in\{0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0\}</annotation></semantics></math>. All other hyperparameters for the perturbation (e.g. guidance scale, noise scheduler) are fixed to the same hyperparameters used to generate synthetic images from scratch (detailed in Appendix <a href="#A2.SS1" title="B.1 Sourcing data by generating synthetic images ‣ Appendix B Details in Methodology ‣ The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Better" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B.1</span></a>) When using SDEdit to perturb a retrieved real image, we additionaly require a text prompt to guide the denoising. We find that using an empty string for the prompt results in incoherent images. We thus use the string “<span id="A4.SS1.p1.2.1" class="ltx_text ltx_font_typewriter">a photo of {classname}</span>”, where <span id="A4.SS1.p1.2.2" class="ltx_text ltx_font_typewriter">{classname}</span> is the label name assigned to the retrieved image being perturbed. Thus, even at noise scale <math id="A4.SS1.p1.2.m2.1" class="ltx_Math" alttext="\gamma=1.0" display="inline"><semantics id="A4.SS1.p1.2.m2.1a"><mrow id="A4.SS1.p1.2.m2.1.1" xref="A4.SS1.p1.2.m2.1.1.cmml"><mi id="A4.SS1.p1.2.m2.1.1.2" xref="A4.SS1.p1.2.m2.1.1.2.cmml">γ</mi><mo id="A4.SS1.p1.2.m2.1.1.1" xref="A4.SS1.p1.2.m2.1.1.1.cmml">=</mo><mn id="A4.SS1.p1.2.m2.1.1.3" xref="A4.SS1.p1.2.m2.1.1.3.cmml">1.0</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.SS1.p1.2.m2.1b"><apply id="A4.SS1.p1.2.m2.1.1.cmml" xref="A4.SS1.p1.2.m2.1.1"><eq id="A4.SS1.p1.2.m2.1.1.1.cmml" xref="A4.SS1.p1.2.m2.1.1.1"></eq><ci id="A4.SS1.p1.2.m2.1.1.2.cmml" xref="A4.SS1.p1.2.m2.1.1.2">𝛾</ci><cn type="float" id="A4.SS1.p1.2.m2.1.1.3.cmml" xref="A4.SS1.p1.2.m2.1.1.3">1.0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.SS1.p1.2.m2.1c">\gamma=1.0</annotation></semantics></math> (i.e., beginning the denoising at the endpoint of Stable Diffusion’s forward diffusion process<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>Even at <math id="footnote2.m1.1" class="ltx_Math" alttext="\gamma=1.0" display="inline"><semantics id="footnote2.m1.1b"><mrow id="footnote2.m1.1.1" xref="footnote2.m1.1.1.cmml"><mi id="footnote2.m1.1.1.2" xref="footnote2.m1.1.1.2.cmml">γ</mi><mo id="footnote2.m1.1.1.1" xref="footnote2.m1.1.1.1.cmml">=</mo><mn id="footnote2.m1.1.1.3" xref="footnote2.m1.1.1.3.cmml">1.0</mn></mrow><annotation-xml encoding="MathML-Content" id="footnote2.m1.1c"><apply id="footnote2.m1.1.1.cmml" xref="footnote2.m1.1.1"><eq id="footnote2.m1.1.1.1.cmml" xref="footnote2.m1.1.1.1"></eq><ci id="footnote2.m1.1.1.2.cmml" xref="footnote2.m1.1.1.2">𝛾</ci><cn type="float" id="footnote2.m1.1.1.3.cmml" xref="footnote2.m1.1.1.3">1.0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote2.m1.1d">\gamma=1.0</annotation></semantics></math>, which maps input images to the endpoint of the forward diffusion process, Stable Diffusion’s noise schedule does not convert the input image into pure isotropic Gaussian noise. This is due to the noise schedule parameters.</span></span></span>), we would not expect model performance from training on perturbed retrieved images to be identical to model performance from training on our synthetically generated images; our synthetically generated images are synthesized using LLM-written prompts that contain richer information.</p>
</div>
</section>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Details in Licensing Information</h2>

<section id="A5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.1 </span>Benchmarks</h3>

<div id="A5.SS1.p1" class="ltx_para">
<p id="A5.SS1.p1.1" class="ltx_p">ImageNet-1K is released under a custom license that specifies non-commercial research use only. Details can be found at <a target="_blank" href="https://www.image-net.org/download.php" title="" class="ltx_ref ltx_href ltx_font_typewriter">https://www.image-net.org/download.php</a>.
Licensing information is unavailable for DTD, Flowers102, FGVC-Aircraft, and StanfordCars; all four datasets are products of academic research and are publicly available online for download.</p>
</div>
</section>
<section id="A5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.2 </span>Models</h3>

<div id="A5.SS2.p1" class="ltx_para">
<p id="A5.SS2.p1.1" class="ltx_p">Stable Diffusion 1.1, 1.3, and 1.5 are all released under a CreativeML OpenRAIL M license. OpenCLIP models and OpenAI CLIP are released under MIT License.</p>
</div>
</section>
<section id="A5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.3 </span>Data</h3>

<div id="A5.SS3.p1" class="ltx_para">
<p id="A5.SS3.p1.1" class="ltx_p">LAION-2B metadata, precomputed embeddings, and <math id="A5.SS3.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A5.SS3.p1.1.m1.1a"><mi id="A5.SS3.p1.1.m1.1.1" xref="A5.SS3.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A5.SS3.p1.1.m1.1b"><ci id="A5.SS3.p1.1.m1.1.1.cmml" xref="A5.SS3.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.SS3.p1.1.m1.1c">k</annotation></semantics></math>-NN search indices are released under CC-BY-4.0 licensing. As a web-scraped dataset, all images pointed to by the URLs retain their original licenses.</p>
</div>
</section>
<section id="A5.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.4 </span>Software</h3>

<div id="A5.SS4.p1" class="ltx_para">
<p id="A5.SS4.p1.1" class="ltx_p">We build off the open source code of <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib52" title="" class="ltx_ref">52</a>, <a href="#bib.bib38" title="" class="ltx_ref">38</a>, <a href="#bib.bib6" title="" class="ltx_ref">6</a>, <a href="#bib.bib16" title="" class="ltx_ref">16</a>, <a href="#bib.bib67" title="" class="ltx_ref">67</a>]</cite>. FAISS <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib16" title="" class="ltx_ref">16</a>]</cite> and clip-retrieval <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib6" title="" class="ltx_ref">6</a>]</cite> are released under MIT license. SynCLR code <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib60" title="" class="ltx_ref">60</a>]</cite> is released under Apache 2.0. Stable Diffusion code <cite class="ltx_cite ltx_citemacro_cite">[<a href="#bib.bib52" title="" class="ltx_ref">52</a>]</cite> is released under CreativeML Open RAIL-M. WiSE-FT (the codebase we build off of for CLIP finetuning) is released under an MIT license.</p>
</div>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2406.05183" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2406.05184" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2406.05184">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2406.05184" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2406.05185" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Sat Jul  6 00:26:36 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2406.05184
Generative text-to-image models enable us to synthesize unlimited amounts of images in a controllable manner, spurring many recent efforts to train vision models with synthetic data.
However, every synthetic image ultimately originates from the upstream data used to train the generator. What additional value does the intermediate generator provide over directly training on relevant parts of the upstream data?
Grounding this question in the setting of image classification, we compare finetuning on task-relevant, targeted synthetic data generated by Stable Diffusion—a generative model trained on the LAION-2B dataset—against finetuning on targeted real images retrieved directly from LAION-2B. We show that while synthetic data can benefit some downstream tasks, it is universally matched or outperformed by real data from the simple retrieval baseline.
Our analysis suggests that this underperformance is partially due to generator artifacts and inaccurate task-relevant visual details in the synthetic images.
Overall, we argue that retrieval is a critical baseline to consider when training with synthetic data—a baseline that current methods do not yet surpass. We release code, data, and models athttps://github.com/scottgeng00/unmet-promise.
abstract
The Unmet Promise of Synthetic Training Images:Using Retrieved Real Images Performs Better
Abstract
2406.05184
1IntroductionThe success of modern machine learning systems fundamentally relies on the quantity, quality, and distributionof the data they are trained on. However, acquiring large amounts of quality data remains challenging, due to the sheer cost of data collection and annotation. As demand for training data continues to rise, the field is actively exploring approaches to automatically curate data at scale. One burgeoning approach is to sourcesynthetic training datafrom conditional generative models. Generative models enable data to be tailored to specific requirements and generated at scale, presenting a promising alternative to the challenges of real data curation.
Recent work highlights this potential: for example, in natural language processing (NLP) , researchers prompt strong proprietary language models to cheaply synthesize large-scale datasets for instruction tuning.Figure 1:Given an upstream dataset of general real image-text pairs, we aim to curate atargeteddataset to train a learner on some target task. We can either (1) retrieve targeted real imagesdirectly from the upstream dataset, or we can (2)  first train an intermediate generative model and thensynthesize targeted synthetic images. By comparing these two approaches, our paper seeks to measure what value training on generated synthetic data adds.Analogously, in computer vision—the focus of our research—many recent works train models on synthetic images from modern text-to-image generators, aiming to achieve state-of-the-art visual recognition performance.
For example, SynCLRcleverly prompts Stable Diffusion for synthetic images tailored to pre-specified downstream image recognition domains; they find that a CLIP-like model trained from scratch on the resultingtargetedsynthetic images can outperform CLIP trained on LAION-2B, a significantly larger untargeted dataset of real images. This result is quite surprising.
Stable Diffusion isalsotrained on LAION-2B, so by the data processing inequality, the synthetic images it generates cannot contain any additional information over the images in LAION-2B. Yet, training on these derivative synthetic images appears to outperform training directly on LAION-2B. How do we make sense of these additional gains?In this paper, we argue that the performance gained by training on generated synthetic images needs to be contextualized against a critical baseline often missing in prior work: training on real images from the generative model’s
pretraining data.
In particular, prior work has often comparedtask-targetedsynthetic images togeneral, untargetedreal images, thereby entangling the effects of training on synthetic versus real images with the effects of targeted versus general data collection. However, these variables are not intrinsically conflated. Any generative model we use to synthesize images fundamentally derives from its upstream training data. Instead of using that upstream data to train an intermediate generative model and synthesize targeted synthetic images, we can alternatively seek to directly retrieve targeted real images from the upstream source (Figure1) . By comparing synthetic training data against this retrieval baseline, we isolate the value added by the generative model.We formalize our study under the ubiquitous problem of task adaptation, where we seek to curate task-targeted images to finetune a pretrained vision model.
We empirically compare training on targeted synthetic images generated from Stable Diffusion 1.5—a text-to-image model trained on the upstream LAION-2B dataset—against training on targeted real images retrieved from LAION-2B itself. We perform hundreds of experiment runs across an order of magnitude of data scales on five visual recognition tasks (ImageNet, Describable Textures (DTD) , FGVC-Aircraft, StanfordCars, and Oxford Flowers102)  where training on synthetic data has shown promise.Together, we find thattraining on targeted real data retrieved from a generative model’s upstream training dataset outperforms training on synthetic data from the generative model.For example, while training on targeted synthetic images can improve downstream accuracy by up to 7.1% (absolute)  on its best-case benchmark (FGVC-Aircraft) , training on targeted real images helps even further, boosting accuracy by a massive. On other benchmarks, such as ImageNet, we find that training on synthetic images can sometimeshurtperformance even when training on real data improves it.
We further show that these findings hold across several different versions of Stable Diffusion, as well as when we train on a mix of synthetic and real data. Our analysis suggests that the consistent underperformance of models trained on synthetic images is partially due to low-level generator artifacts in the synthetic images (e.g., blurs) , and partially because synthetic images may distort high-level class-specific visual details that real images preserve.Overall, we conclude that retrieval is a critical baseline to consider when evaluating the true added utility of generated synthetic training data.
Our goal is not to make normative claims about whether synthetic data will ever surpass this standard, but to contribute a simple baseline to aim for, and a clean set of experiments to explicitly measure progress towards surpassing it. For instance, by conceptualizing retrieval from a generator’s training data as a strong alternative to synthesizing data, a natural future direction for improving synthetic training data is to synthesize image compositions that are explicitly absent from the generator’s upstream training set. Images generated in this manner may offer unique value beyond what can be retrieved from the training data.
Finally, in settings where the upstream dataset of a generator is unavailable altogether (e.g., due to privacy concerns, due to proprietary data, or due to download bandwidth restrictions) , the retrieval baseline is unrealizable by assumption; synthetic data therefore retains strong utility for distilling knowledge from generative models and for privacy preservation. We release all code, models, and over 1TB of generated images to guide future work (https://github.com/scottgeng00/unmet-promise) .
introduction
The Unmet Promise of Synthetic Training Images:Using Retrieved Real Images Performs Better
1Introduction
2406.05184
2Related WorkLearning from synthetic data.Synthetic data has been widely explored in the context of many machine learning problems.
For example, in NLP, synthetic data generated from strong large language modelshas been used to distill instruction-following behaviorand task-specific knowledgeinto smaller models. In computer vision, prior works have sought to use synthetic data to improve the state-of-the-art across a breadth of visual tasks, such as object detection, semantic segmentation, and optical flow. Traditionally, this synthetic training data has been sourced from expert-crafted simulation and rendering pipelines.
Recent advances in text-to-image synthesis via diffusion modelsare changing this paradigm, inspiring a new wave of works that seek to train visual models on synthetic data algorithmically sampled from modern generative models. This structural shift in the source of synthetic images fromexpert-supervised programmatic simulationtoa learned generatorthat itself derives supervision from upstream data raises a critical question: does the intermediate step of training a generator and sampling synthetic data provide any gains over simply training on the upstream data directly? Our work formalizes and empirically grounds this question, contributing experiments and baselines to rigorously measure the benefits of training on modern data-derived synthetic data.Adapting pretrained vision models.Large-scale pretrained image models such as CLIPoffer transferable visual features that benefit a wide range of downstream vision tasks. It is now common practice to use pretrained models as a starting point when deploying downstream task-specific models instead of training them from scratch. From an algorithmic perspective, many methods have been proposed to adapt CLIP models to downstream tasks, each with varying trade-offs. We choose to study simple full-finetuning, centering our work on thedatawe adapt on as opposed to the algorithm. In particular, the quality and relevance of adaptation data has a crucial impact on downstream task performance; distribution shifts at inference time can significantly hurt performance. Acquiring task-targeted data thus remains an active area of research. Most related to our work is, who also employ retrieval as a technique for collecting task targeted-data. Our work builds upon these methods to construct baselines for systematically measuring the true added utility of model-generated synthetic training images.Retrieval as a baseline for synthetic data.Recent studiessimilarly use retrieval from generative model training data as a baseline for synthetic training images. For example,adapts Stable Diffusion with real medical images and then generates synthetic medical images to train a ResNet from scratch; the resulting ResNet outperforms a ResNet trained on LAION-retrieved medical images. Consistent with our work,find that in the low-data regime of general image recognition tasks (e.g., subsampled ImageNet-1K, STL-10) , augmenting real datasets with LAION-retrieved images outperforms augmenting with Stable Diffusion generated images when training ResNets from scratch.further finds that this performance gap in the low-data regime persists across many synthetic image generation methods, including when they finetune Stable Diffusion with task-specific real data. In the medium-data regime (e.g. full ImageNet-1k) ,finds augmenting with synthetic data matches but does not outperform augmenting with retrieved data. Our work generalizes the shared motivation of measuring the utility of synthetic training images to the modern data-rich regime and a wide range of visual classification tasks. We center our experiments on the scaling behavior of synthetic versus retrieved data for finetuning a web-scale pretrained CLIP model on general image recognition (ImageNet-1K) , fine-grained recognition (FGVC-Aircraft, Flowers102, StanfordCars) , and abstract concepts (DTD) . We show that retrieved data often strictly outperforms synthetic data even at large data scales. We further contribute analyses to unpack the consistently observed performance gap between synthetic and retrieved real training images.
body
The Unmet Promise of Synthetic Training Images:Using Retrieved Real Images Performs Better
2Related Work
2406.05184
3Problem Setting and MethodGiven a large datasetof general real image-text pairs and a downstream visual classification task specified as a set of text class names, we aim to algorithmically curate atargeted adaptation datasetof imagesand one-hot class labelsto finetune and improve a pretrained vision model’s performance on the downstream task. We compare two high-level approaches for sourcing this targeted data, shown in Figure1: (1)  weretrieve targeted real imagesdirectly from, forming a targeted retrieved dataset.
Alternatively, (2)  wegenerate targeted synthetic imagesby prompting an intermediate text-to-image generative model trained on, forming a targeted synthetic dataset. We detail each approach below.3.1Sourcing data by generating synthetic imagesWe follow SynCLR, a representative method for curating synthetic training data from off-the-shelf text-to-image models.
Given the set of visual class names, we first synthesize a large corpus of corresponding image captions by prompting a large language model (details in AppendixB.1) . For example, if the class nameis “rose,” then a generated caption might be “a close-up of a pink rose in bloom.”
We then use these captions as input for a text-to-image generatortrained on the upstream data, yielding a large set of synthesized images. Each image is assigned a class labelbased on the class nameused to synthesize its caption. These synthetic images and labelsform our curated dataset.3.2Sourcing data by retrieving real imagesRather than querying a generator trained on an upstream dataset, we can directly train on parts ofitself by retrieving relevant data.consists of image-text pairs. To retrieve relevant pairs, we consider two strategies. We additionally deduplicate all retrieved images with respect to our evaluation datasets followingto minimize test set leakage. We apply NSFW filtering.Strategy 1: hard substring matching.Inspired by, we retrieve the setof all imageswhose corresponding captioncontains at least one target class nameas a substring:Here, labelis assigned based on the classcontained in. If an image-text pairhas textcontaining multiple class names, then we simply retrievemultiple times and assign each instance a different label, once for each unique matched class name.Strategy 2: semantic-NN retrieval.Hard substring matching is simple and effective when the target visual conceptsare concrete entities that are likely to be described in text captions (e.g.,“fire lily”) , but may be less effective when the concepts are abstract (e.g.,“lined texture”) . Thus, we also consider semantic (soft)  retrieval via CLIP image-text embedding space similarity111We perform semantic retrieval using precomputed LAION-2B embeddings from OpenAI CLIP ViT-L/14, the same model Stable Diffusion uses to embed text prompts during generation.. We convert each target class nameinto a set of natural language search queriesbased on the templates from the original CLIP paper.
For each query, we use approximate-NN searchto retrieve the setof-nearest image-text pairsby CLIP similarity between the queryandeitherthe imageor the text:We assign each image-text paira class labelbased on the class name in query. We form the targeted datasetby unioning over all queriesand all classes:3.3Additional data filtering and postprocessingData filtering has been shown to improve training performance for both real and synthetic data and is widely used in practice. We filter both our synthetic and retrieved datasets. Given a curated dataset, we compute the CLIP similarity of eachwith text corresponding to its assigned label(e.g., “a photo of {class name}”) , constructed using the CLIP zero-shot classification templates. When there are multiple templates for a given class, we aggregate by taking the maximum similarity across templates. We keep the top 30% of images by aggregate similarity. Intuitively, filtering helps remove generated and retrieved images with class-misaligned content. For example, an image labeled “dog” but without any dogs present (i.e., due to retrieval or generation errors)  would receive a lower CLIP similarity score and thus likely be discarded.Our synthetic adaptation datasetsare class-balanced by construction (i.e., we uniformly generate images for each class) . We further postprocess the retrieved adaptation datasetsto improve class balancing by manually fixing a global thresholdand truncating the dataset such that each class labeloccurs at mosttimes.
body
The Unmet Promise of Synthetic Training Images:Using Retrieved Real Images Performs Better
3Problem Setting and Method
2406.05184
4Main ExperimentsWe seek to measure the utility of learning from model-generated synthetic images. Grounding this question empirically, our experiments compare finetuning a pretrained CLIP model on (1)  targeted synthetic imagesto (2)  finetuning on targeted retrieved real images.Benchmarks.We focus on five downstream tasks where synthetic data has shown promise compared to similar scaleuntargetedreal data. We select (a)  ImageNet-1Kand Describable Textures (DTD) to evaluate recognition performance on broad categories and (b) 
FGVC-Aircraft, StanfordCars, and Oxford Flowers102to evaluate performance in fine-grained settings. We use standard pre-defined train, test, and validation splits when available, and otherwise randomly subset the training set to create missing train-validation splits (details in AppendixC.2) .Figure 2:We adapt a pretrained CLIP image encoder (dashed purple line)  to different downstream image classification tasks, using either (a)  targeted synthetic data (orange triangles)  generated from a Stable Diffusion model trained on LAION-2B or using (b)  targeted real data (blue circles)  directly retrieved from LAION-2B. We measure performance via downstreamzero-shot (ZS) andlinear probing (LP) accuracy, aggregating results over at least 3 seeds (error bars indicatestandard deviation) . Overall, while adapting CLIP with targeted synthetic data can sometimes improve performance over an off-the-shelf model, synthetic data is universally outperformed or matched by targeted real data. This gap persists even when we scale the sample size of the synthetic adaptation dataset beyond the maximum amount of (finite)  targeted real data considered (gray shaded regions) .Finetuning data curation.For each downstream benchmark, we first curate an adaptation dataset(Section3)  by either (1)  generating synthetic images with Stable Diffusion 1.5, trained on the LAION-2B dataset, or (2)  retrieving real images directly from LAION-2B. We treat the choice between our substring-based and semantic retrieval strategies as a hyperparameter, using downstream validation set accuracy to determine the best choice for each benchmark. Hyperparameters for retrieval are detailed in AppendixC.1.Model adaptation and evaluation.We adapt a LAION-2B pretrained CLIP ViT-B/16image encoder by finetuning on the curated adaptation datasetwith a cross-entropy classification loss for a pre-set number of epochs. To elucidate the scaling trends of synthetic and retrieved data, we finetune across an order of magnitude of different adaptation dataset scales, subsampled from the full targeted adaptation dataset. We report zero-shot (ZS)  and linear probing (LP)  test set accuracy, using the benchmark train set to train the linear head. For both LP and ZS evaluation, we use the validation set to identify the best epoch and finetuning hyperparameters.
For each data scale, we aggregate accuracy across the results of at least three random seeds, and report the standard deviation due to seed randomness. Additional training and full hyperparameter details are provided in AppendixC.3.4.1Main results: synthetic training data lags behind a baseline of retrieved real imagesWe present our main zero-shot and linear probing scaling results in Figure2.At equivalent data scales, finetuning with model-generated synthetic images can help, but is universally matched or outperformed by finetuning directly with images from the generator’s training data.Consistent with prior research, we find that training with targeted synthetic data can improve an unadapted model. For example, on FGVC-Aircraft—the setting where previous works have found strongest gains—finetuning with 139K Stable-Diffusion generated images improved downstream linear probing accuracy by an average of 3.8 percentage points over an off-the-shelf CLIP model () ; on DTD, training with 110K synthetic images improves zero-shot accuracy by 3.3 points () .However, the gains from training on synthetic data are consistently matched or surpassed by training on retrieved real data. For instance, on FGVC-Aircraft, finetuning with an equivalent 139K LAION-2B retrieved images boosts performance by a massive 17.8 points () . Moreover, adapting with retrieved data can improve performance even when synthetic data does not (e.g., on ImageNet and Flowers102 zero-shot accuracy.)  Finally, adapting with synthetic data can sometimes evenhurtperformance (ImageNet, StanfordCars, Flowers102 zero-shot) , while targeted retrieved data improves or at least does not hurt performance across all settings considered. Given equal amounts of targeted retrieved and synthetic data, retrieved data is the clear winner.Synthetic data can sometimes decrease the gap with retrieved data given additional scale, but remains behind.The amount of data we can retrieve is fundamentally limited by the finite upstream data pool. For example, even after searching all 2 billion LAION-2B samples for images containing an FGVC-Aircraft class name in the caption, substring-based retrieval returned only 139K targeted images post-filtering. In contrast, it is straightforward to create ever-larger synthetic datasets by simply generating more data.Scaling the synthetic adaptation dataset size beyond the amount of retrieved data considered (illustrated in the gray-shaded regions of Figure2) , we find that increasing the amount of targeted synthetic data does not always improve performance. For example, on DTD, synthetic data exhibits U-shaped scaling, with performance positively scaling up to 110K synthetic training images, after which performance declines. On ImageNet, Flowers102, and StanfordCars, increasing the synthetic dataset size consistently hurts zero-shot accuracy and has minimal impact on linear probing performance.On Aircraft, scaling helps; there is a log-linear relationship between the size of the synthetic adaptation dataset and downstream linear probing accuracy (e.g., scaling from 139K250K synthetic images improves linear probing accuracy from) . However, synthetic data still lags behind retrieved data: matching the performance of a mere 15K retrieved aircraft images requires scaling the synthetic dataset to 500K images, reflecting a33x difference in dataset size and required finetuning compute. Naively extrapolating this ratio, matching the performance of the full 139K retrieved adaptation dataset would require nearly 5M synthetic images after top 30% filtering. We note, however, that synthetic data is unlikely to truly scale infinitely, as synthetic data fundamentally derives from the (finite)  training set of our generative model. Still, the performance of synthetic data is likely unsaturated at the 500K scale (i.e., accuracy is still trending up) ; due to compute limitations, studying whether further scaling can outperform retrieved data is left for future work.Synthetic data can improve a model’s task representation without significantly improving the model’s task performance.Broadly speaking, zero-shot task accuracy measures a model’s ability to directly solve the downstream task, whereas linear probing accuracy measures the quality of the model’s learnedtask-relevant representation. We find that even when training on synthetic data improves the model’s representation (i.e., downstream linear probing accuracy) , it may not significantly improve the model’s zero-shot accuracy. In contrast, when training on retrieved data improves the model’s representation, zero-shot accuracy also exhibits positive scaling. For example, CLIP adapted with either 15K retrieved images or 500K synthetic images both achieve a similar linear probing accuracy () , yet the model adapted with synthetic data achieves a much worse zero-shot accuracy (versus) . We discuss possible reasons for this qualitative discrepancy in model behavior in our analyses below (Section5.1) .
body
The Unmet Promise of Synthetic Training Images:Using Retrieved Real Images Performs Better
4Main Experiments
2406.05184
5AnalysisIn this section, we explore two questions to better understand our main results. First, what factors drive the underperformance of synthetic data? Second, do our findings hold under variations of our experimental setup? We focus our analysis experiments on ImageNet, to understand general image recognition performance, and FGVC-Aircraft, the sole benchmark where synthetic data exhibited strong positive log-linear scaling.5.1Why does synthetic data lag retrieved real data?Qualitative visualizations.We visualize a random selection of images from our curated synthetic and retrieved adaptation datasets in Figure3. Compared to retrieved real images, we observe that the synthetic images (1)  contain low-level generator artifacts, and (2)  differ in visual content distribution, both in terms of semantic details and overall image composition. For example, although the synthetic FGVC-Aircraft adaptation images (top two rows of Figure3)  are recognizable as airplanes, the visual content often contains incorrect class-relevant semantic details: a correctly-depicted “Airbus A320” should have one engine per wing and two sets of wheels at its rear, yet our synthetic images often exhibit incorrect engine or wheel configurations. This qualitative discrepancy in visual detail precision may partially explain why training on synthetic data does not improve task zero-shot accuracy; synthetic images do not retain enough class-accurate details to directly teach the model the downstream task. In contrast, training on synthetic images can improve linear probing accuracy, because the synthetic images still broadly look like aircraft and thus may help align the model’s representation to the downstream domain.Figure 3:We visualizeretrieved real imagesandsynthetic imagesfrom our targeted adaptation datasets for FGVC-Aircraft (top two rows)  and ImageNet-1K (bottom two rows) , alongside ground truth images (left column)  for reference. Compared to retrieved images, synthetic images often (1)  contain generator artifacts (e.g., the blur on the edges of the “Cessna 172”, the eyes and mouth of the “Tabby Cat”)  and also (2)  distort class-relevant visual content, such as the engine configuration of a true “Airbus A320” (i.e., exactly one engine per wing)  and the entire visual appearance of a “Flute”. We hypothesize that both factors contribute to synthetic training data’s underperformance versus real training data.Synthetically perturbing retrieved real images.To disentangle the effect of low-level generator artifacts and visual content differences between synthetic and retrieved real images on downstream model performance, we trained on “hybrid” images that have similar semantic visual content as our retrieved real images but contain generator artifacts like our synthetic images. Following SDEdit, we use Stable Diffusion to synthetically perturb our retrieved images to introduce model-specific artifacts present in the synthetic images Stable Diffusion generates. Given a noise strength parameterand a retrieved image, SDEdit adds Gaussian noise toaccording to timestepof Stable Diffusion’s time-dependent forward process.
We then denoise the noisy image using the same reverse diffusion process as in text-to-image generation, yielding a perturbed imagethat looks semantically likewhile also containing Stable Diffusion-specific artifacts. Increasingincreases the amount of Gaussian noise added to, thereby increasing the severity of visual artifacts introduced in the resulting(see AppendixD.1for further details) . In pseudocode,Starting from the full targeted retrieved adaptation datasetsfor FGVC-Aircraft and ImageNet, we use SDEdit to introduce generator artifacts into the retrieved real images over a range ofvalues. We visualize the resulting perturbed images in Figure4, and plot the results of training on these perturbed images acrossin Figure5.Figure 4:We use Stable Diffusion to synthetically perturb real images according to a noise strength parameter, where largerincreases the severity of generator-specific artifacts added by the perturbation. When, the introduced artifacts can be strong enough to damage task-relevant visual details for finegrained tasks like FGVC-Aircraft (e.g., the airplane’s engine and rear wheels) . For broad tasks like ImageNet, artifacts have a lesser impact on class-relevant details; the “Tabby Cat” is recognizable as a cat even after perturbing with high.Figure 5:We finetune a pretrained CLIP model (dashed purple line)  on retrieved real images that are synthetically perturbed (green circles)  with Stable Diffusion to introduce generator artifacts. The perturbation strength is controlled by a parameterwhere largercorresponds to stronger artifacts; within thegray-shaded region, the artifacts are strong enough to damage class-relevant details. Our results suggest that generator artifacts do contribute to synthetic data’s underperformance—with any amount of artifacts introduced, performance drops below training on retrieved images (dashed blue line) . Moreover, differences in visual content between synthetic and retrieved images also matter; even with a relatively strong perturbation strength of, training on artifact-afflicted perturbed images that retain the semantic content of retrieved images outperforms training on synthetic images (dashed orange line) .Our results suggest three takeaways. First, generator artifacts indeed contribute to the underperformance of synthetic training images, especially for fine-grained classification tasks. On FGVC-Aircraft, any amount of added generator artifacts drops downstream accuracy. Second, the impact of artifacts is relatively lower for broad classification domains such as ImageNet, where downstream performance is not significantly impacted until we perturb with a relatively strong noise strength of. Finally, visual content differences between synthetic and retrieved images also play a key role in the performance gap between synthetic and retrieved training data. When we perturb images with strength, the resulting images are heavily afflicted with artifacts, but still retain the important class-relevant details of retrieved real images, such as correct airplane engine configurations. Training onimages significantly outperforms training on synthetic images. Intriguingly, training on aircraft images perturbed beyond the point where class-relevant visual details are damagedstill outperforms synthetic data; we speculate that this is because these heavily perturbed images still retain the overall image composition of retrieved images.5.2Synthesizing data via another generative modelFor our main scaling experiments, we generate synthetic image datasets using Stable Diffusion 1.5 to maintain consistency with prior work. To what degree does our choice of generative model impact our findings? At the time of our study, Stable Diffusion 1.x models are the only modern text-to-image models with open source training data available to retrieve from. Therefore, we focus our study here on Stable Diffusion (SD)  versions 1.1, 1.3, and 1.5. Starting from SD v1.1, which is trained on the full LAION-2B dataset, SD v1.3 and v1.5 are derived by further finetuning on high-quality subsets of LAION-2B. This additional finetuning improves image generation fidelity, but may lead to the model forgetting parts of the LAION-2B distribution.
Following our main experiment setup (Section4) , we use SD v1.1 and SD v1.3 to generate various-sized targeted synthetic adaptation datasets for ImageNet and FGVC-Aircraft. Results are plotted in Figure6. Overall, while training with synthetic data from different generative models yields varying performance, synthetic data from all generative models considered consistently fall short of retrieval.Figure 6:We ablate our choice of generative model used to synthesize images in our main experiments, comparing training on synthetic images fromStable Diffusion 1.5 (SD v1.5) to synthetic images fromSD v1.1andSD v1.3. Across multiple Stable Diffusion models—the only modern text-to-image generators with open-source training data available—training with images directly retrieved from the generative models’ training dataset (LAION-2B)  outperforms training with generated images.5.3Mixing synthetic and retrieved dataOn FGVC-Aircraft, finetuning CLIP with either synthetic or retrieved data alone consistently improves downstream task accuracy. The gains from retrieved data are stronger than the gains from synthetic data across all data scales; however, synthetic data may improve CLIP in ways that is complementary to retrieved data, and thus present orthogonal value. To test this possibility, we measure whether training on a mix of synthetic and retrieved Aircraft adaptation data significantly outperforms training with either alone. Starting from our largest retrieved adaptation dataset(139K images) , we progressively add in increasing amounts of synthetic images from our synthetic adaptation datasetand finetune a pretrained CLIP model with the resulting mix. We plot results in Figure7.
We find that training on the mixed images outperforms training on synthetic images alone; however, training on a mix significantly drops performance compared to using retrieved data alone.Figure 7:We finetune a pretrained CLIP model (dashed purple line)  on a dataset comprising a mix of targeted synthetic and retrieved real images (green squares) . The mix is constructed by adding varying amounts of synthetic images to our full retrieved adaptation dataset. Training on the mix outperforms training on synthetic images alone (orange triangles) , but is strictly worse than training on retrieved images alone (blue circles) .
body
The Unmet Promise of Synthetic Training Images:Using Retrieved Real Images Performs Better
5Analysis
2406.05184
6DiscussionOur work sought to answer a key question: given that all model-generated synthetic images derive from the generator’s upstream training data, does training on synthetic images provide value over training on the parts of the upstream real data directly? We contribute a set of rigorous experiments to ground this question empirically, and discover that training on upstream real images collected via our simple retrieval baseline significantly outperforms training on synthetic images. Our initial question is answered negatively. We therefore argue that retrieval is a critical baseline to surpass in order to show value from synthetic training data, and encourage comparison against it in future research.Importantly, we do not seek to make normative claims about whether training with synthetic images will ever surpass this baseline—future work may unlock gains that we have not yet found. As a first step, we contribute analyses of why synthetic training images underperform upstream real images, finding that both generator artifacts and semantic errors within synthetic images are key areas for future improvement. Furthermore, given that image retrieval is a strong alternative to image synthesis, a natural next step is to generate image compositions that are explicitly absent from the generator’s upstream training dataset; synthesizing these “missing" images may offer unique value beyond the existing upstream real images. Such an approach leverages the compositional generalization abilities of the generator, which recent research promisingly suggests may be stronger than the compositionality of a discriminative model trained on the same upstream data.Finally, our findings assume access to the generative model’s upstream training data, an assumption that may not always hold. The upstream pool may be proprietary or strictly regulated due to privacy concerns. In such settings, training directly on the upstream data is impossible; synthetic data from a generative model trained on this unavailable upstream data remains an exciting alternative to acquire otherwise inaccessible information.Limitations.As an empirical study, our compute budget limits the number of experimental variations we consider. Our results are derived from adapting CLIP models with standard full finetuning; we conjecture that our findings generalize to other large-scale pretrained backbones and adaptation methods as well, but we were not able to test this empirically. Moreover, at the time of our work, Stable Diffusion is the only text-to-image model with publicly available training data to retrieve from (i.e. LAION-2B) ; we do not study other generators trained on other data pools.
Finally, we focus on model accuracy, leaving a comparison of model robustness and fairness from training on synthetic versus real data to future work.
body
The Unmet Promise of Synthetic Training Images:Using Retrieved Real Images Performs Better
6Discussion
2406.05184
Acknowledgments and Disclosure of FundingWe graciously thank (in alphabetical order)  Eric Frankel, Jacqueline He, Athena Tsu, and Rui Xin for their helpful comments and feedback. SG is supported by the NSF GRFP.
conclusion
The Unmet Promise of Synthetic Training Images:Using Retrieved Real Images Performs Better
Acknowledgments and Disclosure of Funding