<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2304.07193] DINOv2: Learning Robust Visual Features without Supervision</title><meta property="og:description" content="The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision.
These models could greatly simplify the us…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="DINOv2: Learning Robust Visual Features without Supervision">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="DINOv2: Learning Robust Visual Features without Supervision">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2304.07193">

<!--Generated on Thu Feb 29 14:16:12 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">
DINOv2: Learning Robust Visual Features 
<br class="ltx_break ltx_centering">without Supervision
</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">
Maxime Oquab<sup id="id16.16.id1" class="ltx_sup"><span id="id16.16.id1.1" class="ltx_text ltx_font_italic">∗∗</span></sup>,
Timothée Darcet<sup id="id17.17.id2" class="ltx_sup"><span id="id17.17.id2.1" class="ltx_text ltx_font_italic">∗∗</span></sup>,
Théo Moutakanni<sup id="id18.18.id3" class="ltx_sup"><span id="id18.18.id3.1" class="ltx_text ltx_font_italic">∗∗</span></sup>, 
<br class="ltx_break">Huy V. Vo<sup id="id19.19.id4" class="ltx_sup">∗</sup>,
Marc Szafraniec<sup id="id20.20.id5" class="ltx_sup">∗</sup>,
Vasil Khalidov<sup id="id21.21.id6" class="ltx_sup">∗</sup>,
Pierre Fernandez,
Daniel Haziza, 
<br class="ltx_break">Francisco Massa,
Alaaeldin El-Nouby,
Mahmoud Assran,
Nicolas Ballas,
Wojciech Galuba, 
<br class="ltx_break">Russell Howes,
Po-Yao Huang,
Shang-Wen Li,
Ishan Misra,
Michael Rabbat,
<br class="ltx_break">Vasu Sharma,
Gabriel Synnaeve,
Hu Xu,
Hervé Jegou,
Julien Mairal<sup id="id22.22.id7" class="ltx_sup">1</sup>, 
<br class="ltx_break">Patrick Labatut<sup id="id23.23.id8" class="ltx_sup">∗</sup>,
Armand Joulin<sup id="id24.24.id9" class="ltx_sup">∗</sup>,
Piotr Bojanowski<sup id="id25.25.id10" class="ltx_sup">∗</sup> 
<br class="ltx_break">
<span id="id15.15.5" class="ltx_text ltx_font_italic" style="font-size:90%;">Meta AI Research<span id="id12.12.2.2" class="ltx_text ltx_font_upright">          <sup id="id12.12.2.2.1" class="ltx_sup">1</sup></span>Inria<span id="id15.15.5.5" class="ltx_text ltx_font_upright"> 
<br class="ltx_break">
<sup id="id15.15.5.5.1" class="ltx_sup">∗</sup>core team        <sup id="id15.15.5.5.2" class="ltx_sup"><span id="id15.15.5.5.2.1" class="ltx_text ltx_font_italic">∗∗</span></sup>equal contribution
</span></span>
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id26.id1" class="ltx_p">The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision.
These models could greatly simplify the use of images in any system by producing general-purpose visual features, i.e., features that work across image distributions and tasks without finetuning.
This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources.
We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size.
Most of the technical contributions aim at accelerating and stabilizing the training at scale.
In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature.
In terms of models, we train a ViT model <cite class="ltx_cite ltx_citemacro_citep">(Dosovitskiy et al., <a href="#bib.bib38" title="" class="ltx_ref">2021</a>)</cite> with 1B parameters and distill it into a series of smaller models that surpass the best available general-purpose features, OpenCLIP <cite class="ltx_cite ltx_citemacro_citep">(Ilharco et al., <a href="#bib.bib66" title="" class="ltx_ref">2021</a>)</cite> on most of the benchmarks at image and pixel levels.</p>
</div>
<span id="footnotex1" class="ltx_note ltx_role_footnotetext"><sup class="ltx_note_mark">0</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">0</sup><span class="ltx_note_type">footnotetext: </span>
All the authors are affiliated to Meta, except Julien Mairal who is affiliated to Inria.
Timothée Darcet and Pierre Fernandez have a co-affiliation with Inria.
Théo Moutakanni has a co-affiliation with Université Paris Saclay.
Alaaeldin El-Nouby has a co-affiliation with Inria and ENS-PSL.
Correspondence: {qas, timdarcet, theomoutakanni, ajoulin, bojanowski}@meta.com
</span></span></span>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para ltx_noindent">
<p id="S1.p1.1" class="ltx_p">Learning task-agnostic pretrained representations have become the standard in Natural Language Processing (NLP) <cite class="ltx_cite ltx_citemacro_citep">(Radford et al., <a href="#bib.bib92" title="" class="ltx_ref">2019</a>; Raffel et al., <a href="#bib.bib94" title="" class="ltx_ref">2020</a>; Chowdhery et al., <a href="#bib.bib28" title="" class="ltx_ref">2022</a>; Hoffmann et al., <a href="#bib.bib64" title="" class="ltx_ref">2022</a>; Touvron et al., <a href="#bib.bib115" title="" class="ltx_ref">2023</a>)</cite>.
One can use these features “as they are”, i.e., without fine-tuning, and achieve performances on downstream tasks that are significantly better than those produced by task-specific models <cite class="ltx_cite ltx_citemacro_citep">(Brown et al., <a href="#bib.bib15" title="" class="ltx_ref">2020</a>)</cite>.
This success has been fueled by pretraining on large quantities of raw text using pretext objectives, such as language modeling <cite class="ltx_cite ltx_citemacro_citep">(Radford et al., <a href="#bib.bib91" title="" class="ltx_ref">2017</a>)</cite> or word vectors <cite class="ltx_cite ltx_citemacro_citep">(Devlin et al., <a href="#bib.bib35" title="" class="ltx_ref">2019</a>)</cite>, that require no supervision.</p>
</div>
<div id="S1.p2" class="ltx_para ltx_noindent">
<p id="S1.p2.1" class="ltx_p">Following this paradigm shift in NLP, we expect similar “foundation” models to appear in computer vision <cite class="ltx_cite ltx_citemacro_citep">(Bommasani et al., <a href="#bib.bib13" title="" class="ltx_ref">2021</a>)</cite>.
These models should generate visual features that work out of the box on any task, both at the image level, e.g., image classification, and pixel level, e.g., segmentation.
Most promising efforts towards these foundation models focus on text-guided pretraining, i.e., using a form of textual supervision to guide the training of the features <cite class="ltx_cite ltx_citemacro_citep">(Joulin et al., <a href="#bib.bib69" title="" class="ltx_ref">2016</a>; Mahajan et al., <a href="#bib.bib79" title="" class="ltx_ref">2018</a>; Radford et al., <a href="#bib.bib93" title="" class="ltx_ref">2021</a>)</cite>.
This form of text-guided pretraining limits the information that can be retained about the image since captions only approximate the rich information in images, and complex pixel-level information may not surface with this supervision.
Furthermore, these image encoders require aligned text-image corpora and hence, do not offer the flexibility of their text counterparts, that is, to learn from raw data alone.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2304.07193/assets/new-figure-1.jpg" id="S1.F1.g1" class="ltx_graphics ltx_img_landscape" width="598" height="249" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S1.F1.3.1.1" class="ltx_text" style="font-size:90%;">Figure 1</span>: </span><span id="S1.F1.4.2" class="ltx_text" style="font-size:90%;">
<span id="S1.F1.4.2.1" class="ltx_text ltx_font_bold">Visualization of the first PCA components.</span> We compute a PCA between the patches of the images from the same column (a, b, c and d) and show their first 3 components.
Each component is matched to a different color channel.
Same parts are matched between related images despite changes of pose, style or even objects.
Background is removed by thresholding the first PCA component.
</span></figcaption>
</figure>
<div id="S1.p3" class="ltx_para ltx_noindent">
<p id="S1.p3.1" class="ltx_p">An alternative to text-guided pretraining is self-supervised learning  <cite class="ltx_cite ltx_citemacro_citep">(Caron et al., <a href="#bib.bib16" title="" class="ltx_ref">2018</a>; Chen et al., <a href="#bib.bib21" title="" class="ltx_ref">2020</a>; He et al., <a href="#bib.bib58" title="" class="ltx_ref">2022</a>)</cite> where features are learned from images alone.
These approaches are conceptually closer to pretext tasks such as language modeling and can capture information at the image and pixel level <cite class="ltx_cite ltx_citemacro_citep">(Caron et al., <a href="#bib.bib19" title="" class="ltx_ref">2021</a>)</cite>.
Additionally, the features output by self-supervised models have been shown to exhibit various useful properties, and have enabled enabled a wide variety of applications <cite class="ltx_cite ltx_citemacro_citep">(Amir et al., <a href="#bib.bib1" title="" class="ltx_ref">2022</a>; Tumanyan et al., <a href="#bib.bib116" title="" class="ltx_ref">2022</a>; Ofri-Amar et al., <a href="#bib.bib84" title="" class="ltx_ref">2023</a>; Hamilton et al., <a href="#bib.bib55" title="" class="ltx_ref">2022</a>)</cite>.
However, despite their potential to learn general-purpose features, most of the advances in self-supervised learning were made in the context of pretraining on a small curated dataset, ImageNet-1k <cite class="ltx_cite ltx_citemacro_citep">(Russakovsky et al., <a href="#bib.bib99" title="" class="ltx_ref">2015</a>)</cite>.
Some efforts on scaling these approaches beyond ImageNet-1k have been attempted <cite class="ltx_cite ltx_citemacro_citep">(Caron et al., <a href="#bib.bib17" title="" class="ltx_ref">2019</a>; Goyal et al., <a href="#bib.bib49" title="" class="ltx_ref">2021</a>; <a href="#bib.bib50" title="" class="ltx_ref">2022a</a>)</cite>, but they focused on uncurated datasets, which typically lead to a significant drop in the quality of the features.
This is explained by the lack of control over the data quality and diversity, which are essential to produce good features.</p>
</div>
<div id="S1.p4" class="ltx_para ltx_noindent">
<p id="S1.p4.2" class="ltx_p">In this work, we explore if self-supervised learning has the potential to learn general-purpose visual features if pretrained on a large quantity of curated data.
We revisit existing discriminative self-supervised approaches that learn features at both the image and patch level, such as iBOT <cite class="ltx_cite ltx_citemacro_citep">(Zhou et al., <a href="#bib.bib136" title="" class="ltx_ref">2022a</a>)</cite>, and we reconsider some of their design choices under the lens of a larger dataset.
Most of our technical contributions are tailored toward stabilizing and accelerating discriminative self-supervised learning when scaling in model and data sizes.
These improvements make our approach around 2<math id="S1.p4.1.m1.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S1.p4.1.m1.1a"><mo id="S1.p4.1.m1.1.1" xref="S1.p4.1.m1.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.p4.1.m1.1b"><times id="S1.p4.1.m1.1.1.cmml" xref="S1.p4.1.m1.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.1.m1.1c">\times</annotation></semantics></math> faster and require 3<math id="S1.p4.2.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S1.p4.2.m2.1a"><mo id="S1.p4.2.m2.1.1" xref="S1.p4.2.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S1.p4.2.m2.1b"><times id="S1.p4.2.m2.1.1.cmml" xref="S1.p4.2.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S1.p4.2.m2.1c">\times</annotation></semantics></math> less memory than similar discriminative self-supervised methods, allowing us to leverage longer training with larger batch sizes.</p>
</div>
<div id="S1.p5" class="ltx_para ltx_noindent">
<p id="S1.p5.1" class="ltx_p">Regarding pretraining data, we have built an automatic pipeline to filter and rebalance datasets from an extensive collection of uncurated images.
This pipeline is inspired by pipelines used in NLP <cite class="ltx_cite ltx_citemacro_citep">(Wenzek et al., <a href="#bib.bib124" title="" class="ltx_ref">2020</a>)</cite>, where data similarities are used instead of external metadata and do not require manual annotation.
A major difficulty when dealing with images in the wild is to rebalance concepts and avoid overfitting on a few dominant modes.
In this work, a naive clustering approach works reasonably well to resolve this issue.
We gathered a small but diverse corpus of 142M images to validate our approach.</p>
</div>
<div id="S1.p6" class="ltx_para ltx_noindent">
<p id="S1.p6.1" class="ltx_p">Finally, we provide a variety of pretrained visual models, called DINOv2, trained with different Vision Transformers (ViT) <cite class="ltx_cite ltx_citemacro_citep">(Dosovitskiy et al., <a href="#bib.bib37" title="" class="ltx_ref">2016</a>)</cite> architectures on our data.
We release all the models and the code to retrain DINOv2 on any data.
We validate the quality of DINOv2 on various computer vision benchmarks at both image and pixel levels as we scale them, as summarized in Fig. <a href="#S2.F2" title="Figure 2 ‣ Intra-image self-supervised training. ‣ 2 Related Work ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.
We conclude that self-supervised pretraining alone is a good candidate for learning transferable frozen features that are competitive with the best openly available weakly-supervised models.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<section id="S2.SS0.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Intra-image self-supervised training.</h5>

<div id="S2.SS0.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px1.p1.1" class="ltx_p">A first family of self-supervised methods focuses on pretext tasks built from the image, i.e., extracting a signal from the image to be predicted from the rest of the image.
This idea has become prevalent with the work of <cite class="ltx_cite ltx_citemacro_cite">Doersch et al. (<a href="#bib.bib36" title="" class="ltx_ref">2015</a>)</cite>, where they train by predicting the context of a given patch.
Many other pretext tasks were introduced based on, for example, re-colorizing images <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib133" title="" class="ltx_ref">2016</a>)</cite>, predicting transformations <cite class="ltx_cite ltx_citemacro_citep">(Gidaris et al., <a href="#bib.bib46" title="" class="ltx_ref">2018</a>)</cite>, inpainting <cite class="ltx_cite ltx_citemacro_citep">(Pathak et al., <a href="#bib.bib86" title="" class="ltx_ref">2016</a>)</cite> or patch re-ordering <cite class="ltx_cite ltx_citemacro_citep">(Noroozi &amp; Favaro, <a href="#bib.bib83" title="" class="ltx_ref">2016</a>; Misra &amp; Maaten, <a href="#bib.bib81" title="" class="ltx_ref">2020</a>)</cite>.
Recently, the emergence of patch-based architectures, like ViTs, has led to a revisit of inpainting for pre-training <cite class="ltx_cite ltx_citemacro_citep">(He et al., <a href="#bib.bib58" title="" class="ltx_ref">2022</a>; Bao et al., <a href="#bib.bib6" title="" class="ltx_ref">2021</a>; El-Nouby et al., <a href="#bib.bib41" title="" class="ltx_ref">2021</a>)</cite>, potentially in feature space <cite class="ltx_cite ltx_citemacro_citep">(Assran et al., <a href="#bib.bib4" title="" class="ltx_ref">2023</a>; Baevski et al., <a href="#bib.bib5" title="" class="ltx_ref">2022</a>)</cite>.
Of particular interest, <cite class="ltx_cite ltx_citemacro_cite">He et al. (<a href="#bib.bib58" title="" class="ltx_ref">2022</a>)</cite> show that a masked auto-encoder (MAE) learns features that provide substantial improvements when finetuned on downstream tasks.
This property of MAEs has been further validated on video <cite class="ltx_cite ltx_citemacro_citep">(Tong et al., <a href="#bib.bib112" title="" class="ltx_ref">2022</a>)</cite>, audio <cite class="ltx_cite ltx_citemacro_citep">(Xu et al., <a href="#bib.bib128" title="" class="ltx_ref">2022</a>)</cite>, and across other modalities <cite class="ltx_cite ltx_citemacro_citep">(Girdhar et al., <a href="#bib.bib47" title="" class="ltx_ref">2023</a>)</cite>.
However, their features require supervised finetuning, while our features perform well out of the box.</p>
</div>
<figure id="S2.F2" class="ltx_figure"><img src="/html/2304.07193/assets/x1.png" id="S2.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="361" height="166" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S2.F2.3.1.1" class="ltx_text" style="font-size:90%;">Figure 2</span>: </span><span id="S2.F2.4.2" class="ltx_text" style="font-size:90%;">
<span id="S2.F2.4.2.1" class="ltx_text ltx_font_bold">Evolution of performance when scaling in parameters.</span>
We show performance on eight types of vision tasks, as presented in Sec. <a href="#S7" title="7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>, and average metrics with each type.
Features are extracted from our self-supervised encoders, DINOv2 (dark blue), and we compare them with self-supervised methods (pale orange), as well as weakly-supervised methods (dark pink).
We report the best-performing weakly-supervised model’s performance as a dashed horizontal line.
Our family of models drastically improves over the previous state of the art in self-supervised learning and reaches performance comparable with weakly-supervised features.
See Sec. <a href="#S7" title="7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> for a detailed analysis.

</span></figcaption>
</figure>
</section>
<section id="S2.SS0.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Discriminative self-supervised learning.</h5>

<div id="S2.SS0.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px2.p1.1" class="ltx_p">The second line of work, closer to ours, is using discriminative signals between images or groups of images to learn features.
This family of methods has roots in early deep learning work <cite class="ltx_cite ltx_citemacro_citep">(Hadsell et al., <a href="#bib.bib54" title="" class="ltx_ref">2006</a>)</cite> but became popular with the emergence of instance classification methods <cite class="ltx_cite ltx_citemacro_citep">(Dosovitskiy et al., <a href="#bib.bib37" title="" class="ltx_ref">2016</a>; Bojanowski &amp; Joulin, <a href="#bib.bib12" title="" class="ltx_ref">2017</a>; Wu et al., <a href="#bib.bib126" title="" class="ltx_ref">2018</a>)</cite>.
Several improvements were made based either on instance-level objectives <cite class="ltx_cite ltx_citemacro_citep">(Hénaff et al., <a href="#bib.bib59" title="" class="ltx_ref">2019</a>; He et al., <a href="#bib.bib57" title="" class="ltx_ref">2020</a>; Chen &amp; He, <a href="#bib.bib23" title="" class="ltx_ref">2021</a>; Chen et al., <a href="#bib.bib21" title="" class="ltx_ref">2020</a>; Grill et al., <a href="#bib.bib53" title="" class="ltx_ref">2020</a>; Caron et al., <a href="#bib.bib19" title="" class="ltx_ref">2021</a>)</cite> or clustering <cite class="ltx_cite ltx_citemacro_citep">(Caron et al., <a href="#bib.bib16" title="" class="ltx_ref">2018</a>; Asano et al., <a href="#bib.bib2" title="" class="ltx_ref">2020</a>; Caron et al., <a href="#bib.bib18" title="" class="ltx_ref">2020</a>)</cite>.
These methods provide performant frozen features on standard benchmarks like ImageNet <cite class="ltx_cite ltx_citemacro_citep">(Russakovsky et al., <a href="#bib.bib99" title="" class="ltx_ref">2015</a>)</cite>, but they are hard to scale to larger model sizes <cite class="ltx_cite ltx_citemacro_citep">(Chen et al., <a href="#bib.bib24" title="" class="ltx_ref">2021</a>)</cite>.
In this work, we revisit the training of these approaches in the context of large pretraining datasets and models.
In particular, we build on top of <cite class="ltx_cite ltx_citemacro_cite">Zhou et al. (<a href="#bib.bib136" title="" class="ltx_ref">2022a</a>)</cite> that we find particularly suited for scaling.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Scaling self-supervised pretraining.</h5>

<div id="S2.SS0.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px3.p1.1" class="ltx_p">A growing body of work has focused on the scaling abilities of self-supervised learning in terms of data and model size <cite class="ltx_cite ltx_citemacro_citep">(Caron et al., <a href="#bib.bib17" title="" class="ltx_ref">2019</a>; Goyal et al., <a href="#bib.bib48" title="" class="ltx_ref">2019</a>; Tian et al., <a href="#bib.bib110" title="" class="ltx_ref">2021</a>; Goyal et al., <a href="#bib.bib50" title="" class="ltx_ref">2022a</a>)</cite>.
Most of these works use large quantities of uncurated data to train models without supervision.
They show evidence that discriminative methods scale with data, but because of the poor quality of the pretraining data, most of the results are obtained by finetuning the features.
Of particular interest, <cite class="ltx_cite ltx_citemacro_cite">Goyal et al. (<a href="#bib.bib49" title="" class="ltx_ref">2021</a>)</cite> have also shown that these methods benefit from scaling in model size given enough pretrained data.
This line of work questions the ability of self-supervised methods to work on any data while we focus on producing the best pretrained encoders.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px4" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Automatic data curation.</h5>

<div id="S2.SS0.SSS0.Px4.p1" class="ltx_para ltx_noindent">
<p id="S2.SS0.SSS0.Px4.p1.1" class="ltx_p">Our dataset construction borrows from the image retrieval community <cite class="ltx_cite ltx_citemacro_citep">(Weinzaepfel et al., <a href="#bib.bib122" title="" class="ltx_ref">2021</a>; Radenović et al., <a href="#bib.bib90" title="" class="ltx_ref">2018b</a>; Berman et al., <a href="#bib.bib8" title="" class="ltx_ref">2019</a>; Douze et al., <a href="#bib.bib39" title="" class="ltx_ref">2009</a>; Tolias et al., <a href="#bib.bib111" title="" class="ltx_ref">2016</a>; Revaud et al., <a href="#bib.bib97" title="" class="ltx_ref">2019</a>)</cite>.
In particular, the use of retrieval to augment the training set has been studied in the context of semi-supervised learning <cite class="ltx_cite ltx_citemacro_citep">(Yalniz et al., <a href="#bib.bib129" title="" class="ltx_ref">2019</a>)</cite>.
Similarly, others have used hashtags or other metadata <cite class="ltx_cite ltx_citemacro_citep">(Mahajan et al., <a href="#bib.bib79" title="" class="ltx_ref">2018</a>; Radford et al., <a href="#bib.bib93" title="" class="ltx_ref">2021</a>)</cite> or pretrained vision encoders <cite class="ltx_cite ltx_citemacro_citep">(Schuhmann et al., <a href="#bib.bib101" title="" class="ltx_ref">2021</a>; <a href="#bib.bib102" title="" class="ltx_ref">2022</a>)</cite> to filter uncurated datasets.
Unlike these works, we use no pretrained encoders, metadata nor supervision to filter images and leverage visual similarity between images.
Our approach is inspired by text curation pipelines <cite class="ltx_cite ltx_citemacro_citep">(Wenzek et al., <a href="#bib.bib124" title="" class="ltx_ref">2020</a>)</cite>, where a language model is trained on Wikipedia to score texts extracted from an uncurated source.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Data Processing</h2>

<div id="S3.p1" class="ltx_para ltx_noindent">
<p id="S3.p1.1" class="ltx_p">We assemble our curated LVD-142M dataset by retrieving, from a large pool of uncurated data, images that are close to those in several curated datasets.
We describe below the main components in our data pipeline including the curated/uncurated data sources, the image deduplication step and the retrieval system.
Our pipeline does not require any metadata or text and directly works with images, as shown in Fig. <a href="#S3.F3" title="Figure 3 ‣ 3 Data Processing ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.
We refer the reader to appendix <a href="#A1" title="Appendix A Data Processing ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a> for more details on our approach.</p>
</div>
<figure id="S3.F3" class="ltx_figure"><img src="/html/2304.07193/assets/x2.png" id="S3.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="361" height="111" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F3.3.1.1" class="ltx_text" style="font-size:90%;">Figure 3</span>: </span><span id="S3.F3.4.2" class="ltx_text" style="font-size:90%;">
<span id="S3.F3.4.2.1" class="ltx_text ltx_font_bold">Overview of our data processing pipeline.</span>
Images from curated and uncurated data sources are first mapped to embeddings.
Uncurated images are then deduplicated before being matched to curated images.
The resulting combination augments the initial dataset through a self-supervised retrieval system.
</span></figcaption>
</figure>
<div id="S3.p2" class="ltx_para ltx_noindent">
<p id="S3.p2.1" class="ltx_p"><span id="S3.p2.1.1" class="ltx_text ltx_font_bold">Data sources.</span>
Our selection of curated datasets is detailed in the appendix (Table <a href="#A1.T15" title="Table 15 ‣ A.4 Retrieval ‣ Appendix A Data Processing ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">15</span></a>) and contains ImageNet-22k, the train split of ImageNet-1k, Google Landmarks and several fine-grained datasets.
For the uncurated data source, we collect a raw unfiltered dataset of images from a publicly available repository of crawled web data.
From each web page in the repository, we extract URL links of images from <span id="S3.p2.1.2" class="ltx_text ltx_font_typewriter">&lt;img&gt;</span> tags.
We discard URLs that are unsafe or restricted by domains, and post-process the downloaded images (PCA hash deduplication, NSFW filtering, and blurring identifiable faces).
This results in 1.2B unique images.</p>
</div>
<div id="S3.p3" class="ltx_para ltx_noindent">
<p id="S3.p3.1" class="ltx_p"><span id="S3.p3.1.1" class="ltx_text ltx_font_bold">Deduplication.</span>
We apply the copy detection pipeline of <cite class="ltx_cite ltx_citemacro_cite">Pizzi et al. (<a href="#bib.bib88" title="" class="ltx_ref">2022</a>)</cite> to the uncurated data and remove near-duplicate images.
This reduces redundancy and increases diversity among images.
We also remove near-duplicates of images contained in the test or validation set of any benchmark used in this work.</p>
</div>
<div id="S3.p4" class="ltx_para ltx_noindent">
<p id="S3.p4.4" class="ltx_p"><span id="S3.p4.4.1" class="ltx_text ltx_font_bold">Self-supervised image retrieval.</span>
We build our curated pretraining dataset by retrieving images from our uncurated data source that are close to images in our curated sources.
In order to do this, we first compute an image embedding using a self-supervised ViT-H/16 network pretrained on ImageNet-22k, and use cosine-similarity as a distance measure between images.
Then, we perform k-means clustering of the uncurated data.
Given a query dataset for retrieval, if it is large enough we retrieve <math id="S3.p4.1.m1.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S3.p4.1.m1.1a"><mi id="S3.p4.1.m1.1.1" xref="S3.p4.1.m1.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S3.p4.1.m1.1b"><ci id="S3.p4.1.m1.1.1.cmml" xref="S3.p4.1.m1.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p4.1.m1.1c">N</annotation></semantics></math> (typically 4) nearest neighbors for each query image.
If it is small, we sample <math id="S3.p4.2.m2.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S3.p4.2.m2.1a"><mi id="S3.p4.2.m2.1.1" xref="S3.p4.2.m2.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S3.p4.2.m2.1b"><ci id="S3.p4.2.m2.1.1.cmml" xref="S3.p4.2.m2.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p4.2.m2.1c">M</annotation></semantics></math> images from the cluster corresponding to each query image.
Although visual inspection seemed to indicate good retrieval quality for <math id="S3.p4.3.m3.1" class="ltx_Math" alttext="N" display="inline"><semantics id="S3.p4.3.m3.1a"><mi id="S3.p4.3.m3.1.1" xref="S3.p4.3.m3.1.1.cmml">N</mi><annotation-xml encoding="MathML-Content" id="S3.p4.3.m3.1b"><ci id="S3.p4.3.m3.1.1.cmml" xref="S3.p4.3.m3.1.1">𝑁</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.p4.3.m3.1c">N</annotation></semantics></math> much larger than 4, this leads to more collisions (images that are nearest-neighbor retrievals of multiple queries). We choose <math id="S3.p4.4.m4.1" class="ltx_Math" alttext="N=4" display="inline"><semantics id="S3.p4.4.m4.1a"><mrow id="S3.p4.4.m4.1.1" xref="S3.p4.4.m4.1.1.cmml"><mi id="S3.p4.4.m4.1.1.2" xref="S3.p4.4.m4.1.1.2.cmml">N</mi><mo id="S3.p4.4.m4.1.1.1" xref="S3.p4.4.m4.1.1.1.cmml">=</mo><mn id="S3.p4.4.m4.1.1.3" xref="S3.p4.4.m4.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.p4.4.m4.1b"><apply id="S3.p4.4.m4.1.1.cmml" xref="S3.p4.4.m4.1.1"><eq id="S3.p4.4.m4.1.1.1.cmml" xref="S3.p4.4.m4.1.1.1"></eq><ci id="S3.p4.4.m4.1.1.2.cmml" xref="S3.p4.4.m4.1.1.2">𝑁</ci><cn type="integer" id="S3.p4.4.m4.1.1.3.cmml" xref="S3.p4.4.m4.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.p4.4.m4.1c">N=4</annotation></semantics></math> as it provides a good tradeoff in that sense.</p>
</div>
<div id="S3.p5" class="ltx_para ltx_noindent">
<p id="S3.p5.1" class="ltx_p"><span id="S3.p5.1.1" class="ltx_text ltx_font_bold">Implementation Details.</span>
The deduplication and retrieval stages of our pipeline rely on the Faiss library <cite class="ltx_cite ltx_citemacro_citep">(Johnson et al., <a href="#bib.bib68" title="" class="ltx_ref">2019</a>)</cite> to efficiently index and compute batch searches of nearest embeddings.
In particular, we heavily leverage its support for GPU-accelerated indices, using inverted file indices with product quantization codes <cite class="ltx_cite ltx_citemacro_citep">(Jegou et al., <a href="#bib.bib67" title="" class="ltx_ref">2010</a>)</cite>.
The whole processing is distributed on a compute cluster of 20 nodes equipped with 8 V100-32GB GPUs and takes less than two days to produce the LVD-142M dataset.</p>
</div>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Discriminative Self-supervised Pre-training</h2>

<div id="S4.p1" class="ltx_para ltx_noindent">
<p id="S4.p1.1" class="ltx_p">We learn our features with a discriminative self-supervised method that can be seen as a combination of DINO and iBOT losses with the centering of SwAV <cite class="ltx_cite ltx_citemacro_citep">(Caron et al., <a href="#bib.bib18" title="" class="ltx_ref">2020</a>)</cite>.
We also add a regularizer to spread features and a short high-resolution training phase.
We rapidly introduce each of these approaches, but more details can be found in the related papers, or in our open-sourced code.</p>
</div>
<div id="S4.p2" class="ltx_para ltx_noindent">
<ul id="S4.I1" class="ltx_itemize">
<li id="S4.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I1.i1.p1" class="ltx_para ltx_noindent">
<p id="S4.I1.i1.p1.2" class="ltx_p"><span id="S4.I1.i1.p1.2.1" class="ltx_text ltx_font_bold">Image-level objective <cite class="ltx_cite ltx_citemacro_citep">(Caron et al., <a href="#bib.bib19" title="" class="ltx_ref">2021</a>)</cite>.</span>
We consider the cross-entropy loss between the features extracted from a student and a teacher network.
Both features are coming from the class token of a ViT, obtained from different crops of the same image.

We pass the student class token through the student DINO head.
This head is an MLP model outputting a vector of scores, that we call "prototype scores".
We then apply a softmax to obtain <math id="S4.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="p_{s}" display="inline"><semantics id="S4.I1.i1.p1.1.m1.1a"><msub id="S4.I1.i1.p1.1.m1.1.1" xref="S4.I1.i1.p1.1.m1.1.1.cmml"><mi id="S4.I1.i1.p1.1.m1.1.1.2" xref="S4.I1.i1.p1.1.m1.1.1.2.cmml">p</mi><mi id="S4.I1.i1.p1.1.m1.1.1.3" xref="S4.I1.i1.p1.1.m1.1.1.3.cmml">s</mi></msub><annotation-xml encoding="MathML-Content" id="S4.I1.i1.p1.1.m1.1b"><apply id="S4.I1.i1.p1.1.m1.1.1.cmml" xref="S4.I1.i1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.I1.i1.p1.1.m1.1.1.1.cmml" xref="S4.I1.i1.p1.1.m1.1.1">subscript</csymbol><ci id="S4.I1.i1.p1.1.m1.1.1.2.cmml" xref="S4.I1.i1.p1.1.m1.1.1.2">𝑝</ci><ci id="S4.I1.i1.p1.1.m1.1.1.3.cmml" xref="S4.I1.i1.p1.1.m1.1.1.3">𝑠</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i1.p1.1.m1.1c">p_{s}</annotation></semantics></math>.
Similarly, we apply the teacher DINO head to the teacher class token to obtain teacher prototype scores.
We then apply a softmax followed by a centering with moving average (or a Sinkhorn-Knopp centering as detailed thereafter) to obtain <math id="S4.I1.i1.p1.2.m2.1" class="ltx_Math" alttext="p_{t}" display="inline"><semantics id="S4.I1.i1.p1.2.m2.1a"><msub id="S4.I1.i1.p1.2.m2.1.1" xref="S4.I1.i1.p1.2.m2.1.1.cmml"><mi id="S4.I1.i1.p1.2.m2.1.1.2" xref="S4.I1.i1.p1.2.m2.1.1.2.cmml">p</mi><mi id="S4.I1.i1.p1.2.m2.1.1.3" xref="S4.I1.i1.p1.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S4.I1.i1.p1.2.m2.1b"><apply id="S4.I1.i1.p1.2.m2.1.1.cmml" xref="S4.I1.i1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S4.I1.i1.p1.2.m2.1.1.1.cmml" xref="S4.I1.i1.p1.2.m2.1.1">subscript</csymbol><ci id="S4.I1.i1.p1.2.m2.1.1.2.cmml" xref="S4.I1.i1.p1.2.m2.1.1.2">𝑝</ci><ci id="S4.I1.i1.p1.2.m2.1.1.3.cmml" xref="S4.I1.i1.p1.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i1.p1.2.m2.1c">p_{t}</annotation></semantics></math>.
The DINO loss term corresponds to:</p>
<table id="S4.Ex1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.Ex1.m1.1" class="ltx_Math" alttext="{\mathcal{L}}_{DINO}=-\sum p_{t}\log p_{s}" display="block"><semantics id="S4.Ex1.m1.1a"><mrow id="S4.Ex1.m1.1.1" xref="S4.Ex1.m1.1.1.cmml"><msub id="S4.Ex1.m1.1.1.2" xref="S4.Ex1.m1.1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.Ex1.m1.1.1.2.2" xref="S4.Ex1.m1.1.1.2.2.cmml">ℒ</mi><mrow id="S4.Ex1.m1.1.1.2.3" xref="S4.Ex1.m1.1.1.2.3.cmml"><mi id="S4.Ex1.m1.1.1.2.3.2" xref="S4.Ex1.m1.1.1.2.3.2.cmml">D</mi><mo lspace="0em" rspace="0em" id="S4.Ex1.m1.1.1.2.3.1" xref="S4.Ex1.m1.1.1.2.3.1.cmml">​</mo><mi id="S4.Ex1.m1.1.1.2.3.3" xref="S4.Ex1.m1.1.1.2.3.3.cmml">I</mi><mo lspace="0em" rspace="0em" id="S4.Ex1.m1.1.1.2.3.1a" xref="S4.Ex1.m1.1.1.2.3.1.cmml">​</mo><mi id="S4.Ex1.m1.1.1.2.3.4" xref="S4.Ex1.m1.1.1.2.3.4.cmml">N</mi><mo lspace="0em" rspace="0em" id="S4.Ex1.m1.1.1.2.3.1b" xref="S4.Ex1.m1.1.1.2.3.1.cmml">​</mo><mi id="S4.Ex1.m1.1.1.2.3.5" xref="S4.Ex1.m1.1.1.2.3.5.cmml">O</mi></mrow></msub><mo id="S4.Ex1.m1.1.1.1" xref="S4.Ex1.m1.1.1.1.cmml">=</mo><mrow id="S4.Ex1.m1.1.1.3" xref="S4.Ex1.m1.1.1.3.cmml"><mo id="S4.Ex1.m1.1.1.3a" xref="S4.Ex1.m1.1.1.3.cmml">−</mo><mrow id="S4.Ex1.m1.1.1.3.2" xref="S4.Ex1.m1.1.1.3.2.cmml"><mo movablelimits="false" id="S4.Ex1.m1.1.1.3.2.1" xref="S4.Ex1.m1.1.1.3.2.1.cmml">∑</mo><mrow id="S4.Ex1.m1.1.1.3.2.2" xref="S4.Ex1.m1.1.1.3.2.2.cmml"><msub id="S4.Ex1.m1.1.1.3.2.2.2" xref="S4.Ex1.m1.1.1.3.2.2.2.cmml"><mi id="S4.Ex1.m1.1.1.3.2.2.2.2" xref="S4.Ex1.m1.1.1.3.2.2.2.2.cmml">p</mi><mi id="S4.Ex1.m1.1.1.3.2.2.2.3" xref="S4.Ex1.m1.1.1.3.2.2.2.3.cmml">t</mi></msub><mo lspace="0.167em" rspace="0em" id="S4.Ex1.m1.1.1.3.2.2.1" xref="S4.Ex1.m1.1.1.3.2.2.1.cmml">​</mo><mrow id="S4.Ex1.m1.1.1.3.2.2.3" xref="S4.Ex1.m1.1.1.3.2.2.3.cmml"><mi id="S4.Ex1.m1.1.1.3.2.2.3.1" xref="S4.Ex1.m1.1.1.3.2.2.3.1.cmml">log</mi><mo lspace="0.167em" id="S4.Ex1.m1.1.1.3.2.2.3a" xref="S4.Ex1.m1.1.1.3.2.2.3.cmml">⁡</mo><msub id="S4.Ex1.m1.1.1.3.2.2.3.2" xref="S4.Ex1.m1.1.1.3.2.2.3.2.cmml"><mi id="S4.Ex1.m1.1.1.3.2.2.3.2.2" xref="S4.Ex1.m1.1.1.3.2.2.3.2.2.cmml">p</mi><mi id="S4.Ex1.m1.1.1.3.2.2.3.2.3" xref="S4.Ex1.m1.1.1.3.2.2.3.2.3.cmml">s</mi></msub></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.Ex1.m1.1b"><apply id="S4.Ex1.m1.1.1.cmml" xref="S4.Ex1.m1.1.1"><eq id="S4.Ex1.m1.1.1.1.cmml" xref="S4.Ex1.m1.1.1.1"></eq><apply id="S4.Ex1.m1.1.1.2.cmml" xref="S4.Ex1.m1.1.1.2"><csymbol cd="ambiguous" id="S4.Ex1.m1.1.1.2.1.cmml" xref="S4.Ex1.m1.1.1.2">subscript</csymbol><ci id="S4.Ex1.m1.1.1.2.2.cmml" xref="S4.Ex1.m1.1.1.2.2">ℒ</ci><apply id="S4.Ex1.m1.1.1.2.3.cmml" xref="S4.Ex1.m1.1.1.2.3"><times id="S4.Ex1.m1.1.1.2.3.1.cmml" xref="S4.Ex1.m1.1.1.2.3.1"></times><ci id="S4.Ex1.m1.1.1.2.3.2.cmml" xref="S4.Ex1.m1.1.1.2.3.2">𝐷</ci><ci id="S4.Ex1.m1.1.1.2.3.3.cmml" xref="S4.Ex1.m1.1.1.2.3.3">𝐼</ci><ci id="S4.Ex1.m1.1.1.2.3.4.cmml" xref="S4.Ex1.m1.1.1.2.3.4">𝑁</ci><ci id="S4.Ex1.m1.1.1.2.3.5.cmml" xref="S4.Ex1.m1.1.1.2.3.5">𝑂</ci></apply></apply><apply id="S4.Ex1.m1.1.1.3.cmml" xref="S4.Ex1.m1.1.1.3"><minus id="S4.Ex1.m1.1.1.3.1.cmml" xref="S4.Ex1.m1.1.1.3"></minus><apply id="S4.Ex1.m1.1.1.3.2.cmml" xref="S4.Ex1.m1.1.1.3.2"><sum id="S4.Ex1.m1.1.1.3.2.1.cmml" xref="S4.Ex1.m1.1.1.3.2.1"></sum><apply id="S4.Ex1.m1.1.1.3.2.2.cmml" xref="S4.Ex1.m1.1.1.3.2.2"><times id="S4.Ex1.m1.1.1.3.2.2.1.cmml" xref="S4.Ex1.m1.1.1.3.2.2.1"></times><apply id="S4.Ex1.m1.1.1.3.2.2.2.cmml" xref="S4.Ex1.m1.1.1.3.2.2.2"><csymbol cd="ambiguous" id="S4.Ex1.m1.1.1.3.2.2.2.1.cmml" xref="S4.Ex1.m1.1.1.3.2.2.2">subscript</csymbol><ci id="S4.Ex1.m1.1.1.3.2.2.2.2.cmml" xref="S4.Ex1.m1.1.1.3.2.2.2.2">𝑝</ci><ci id="S4.Ex1.m1.1.1.3.2.2.2.3.cmml" xref="S4.Ex1.m1.1.1.3.2.2.2.3">𝑡</ci></apply><apply id="S4.Ex1.m1.1.1.3.2.2.3.cmml" xref="S4.Ex1.m1.1.1.3.2.2.3"><log id="S4.Ex1.m1.1.1.3.2.2.3.1.cmml" xref="S4.Ex1.m1.1.1.3.2.2.3.1"></log><apply id="S4.Ex1.m1.1.1.3.2.2.3.2.cmml" xref="S4.Ex1.m1.1.1.3.2.2.3.2"><csymbol cd="ambiguous" id="S4.Ex1.m1.1.1.3.2.2.3.2.1.cmml" xref="S4.Ex1.m1.1.1.3.2.2.3.2">subscript</csymbol><ci id="S4.Ex1.m1.1.1.3.2.2.3.2.2.cmml" xref="S4.Ex1.m1.1.1.3.2.2.3.2.2">𝑝</ci><ci id="S4.Ex1.m1.1.1.3.2.2.3.2.3.cmml" xref="S4.Ex1.m1.1.1.3.2.2.3.2.3">𝑠</ci></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Ex1.m1.1c">{\mathcal{L}}_{DINO}=-\sum p_{t}\log p_{s}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="S4.I1.i1.p2" class="ltx_para">
<p id="S4.I1.i1.p2.1" class="ltx_p">We learn the parameters of the student and build the teacher head with an exponential moving average of past iterates <cite class="ltx_cite ltx_citemacro_citep">(He et al., <a href="#bib.bib57" title="" class="ltx_ref">2020</a>)</cite>.</p>
</div>
</li>
<li id="S4.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I1.i2.p1" class="ltx_para ltx_noindent">
<p id="S4.I1.i2.p1.2" class="ltx_p"><span id="S4.I1.i2.p1.2.1" class="ltx_text ltx_font_bold">Patch-level objective <cite class="ltx_cite ltx_citemacro_citep">(Zhou et al., <a href="#bib.bib136" title="" class="ltx_ref">2022a</a>)</cite>.</span>
We randomly mask some of the input patches given to the student, but not to the teacher.

We then apply the student iBOT head to the student mask tokens.
Similarly, we apply the teacher iBOT head to the (visible) teacher patch tokens corresponding to the ones masked in the student.
We then apply the softmax and centering steps as above, and obtain the iBOT loss term:</p>
<table id="S4.Ex2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.Ex2.m1.1" class="ltx_Math" alttext="{\mathcal{L}}_{iBOT}=-\sum_{i}p_{ti}\log p_{si}" display="block"><semantics id="S4.Ex2.m1.1a"><mrow id="S4.Ex2.m1.1.1" xref="S4.Ex2.m1.1.1.cmml"><msub id="S4.Ex2.m1.1.1.2" xref="S4.Ex2.m1.1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.Ex2.m1.1.1.2.2" xref="S4.Ex2.m1.1.1.2.2.cmml">ℒ</mi><mrow id="S4.Ex2.m1.1.1.2.3" xref="S4.Ex2.m1.1.1.2.3.cmml"><mi id="S4.Ex2.m1.1.1.2.3.2" xref="S4.Ex2.m1.1.1.2.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S4.Ex2.m1.1.1.2.3.1" xref="S4.Ex2.m1.1.1.2.3.1.cmml">​</mo><mi id="S4.Ex2.m1.1.1.2.3.3" xref="S4.Ex2.m1.1.1.2.3.3.cmml">B</mi><mo lspace="0em" rspace="0em" id="S4.Ex2.m1.1.1.2.3.1a" xref="S4.Ex2.m1.1.1.2.3.1.cmml">​</mo><mi id="S4.Ex2.m1.1.1.2.3.4" xref="S4.Ex2.m1.1.1.2.3.4.cmml">O</mi><mo lspace="0em" rspace="0em" id="S4.Ex2.m1.1.1.2.3.1b" xref="S4.Ex2.m1.1.1.2.3.1.cmml">​</mo><mi id="S4.Ex2.m1.1.1.2.3.5" xref="S4.Ex2.m1.1.1.2.3.5.cmml">T</mi></mrow></msub><mo id="S4.Ex2.m1.1.1.1" xref="S4.Ex2.m1.1.1.1.cmml">=</mo><mrow id="S4.Ex2.m1.1.1.3" xref="S4.Ex2.m1.1.1.3.cmml"><mo id="S4.Ex2.m1.1.1.3a" xref="S4.Ex2.m1.1.1.3.cmml">−</mo><mrow id="S4.Ex2.m1.1.1.3.2" xref="S4.Ex2.m1.1.1.3.2.cmml"><munder id="S4.Ex2.m1.1.1.3.2.1" xref="S4.Ex2.m1.1.1.3.2.1.cmml"><mo movablelimits="false" id="S4.Ex2.m1.1.1.3.2.1.2" xref="S4.Ex2.m1.1.1.3.2.1.2.cmml">∑</mo><mi id="S4.Ex2.m1.1.1.3.2.1.3" xref="S4.Ex2.m1.1.1.3.2.1.3.cmml">i</mi></munder><mrow id="S4.Ex2.m1.1.1.3.2.2" xref="S4.Ex2.m1.1.1.3.2.2.cmml"><msub id="S4.Ex2.m1.1.1.3.2.2.2" xref="S4.Ex2.m1.1.1.3.2.2.2.cmml"><mi id="S4.Ex2.m1.1.1.3.2.2.2.2" xref="S4.Ex2.m1.1.1.3.2.2.2.2.cmml">p</mi><mrow id="S4.Ex2.m1.1.1.3.2.2.2.3" xref="S4.Ex2.m1.1.1.3.2.2.2.3.cmml"><mi id="S4.Ex2.m1.1.1.3.2.2.2.3.2" xref="S4.Ex2.m1.1.1.3.2.2.2.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S4.Ex2.m1.1.1.3.2.2.2.3.1" xref="S4.Ex2.m1.1.1.3.2.2.2.3.1.cmml">​</mo><mi id="S4.Ex2.m1.1.1.3.2.2.2.3.3" xref="S4.Ex2.m1.1.1.3.2.2.2.3.3.cmml">i</mi></mrow></msub><mo lspace="0.167em" rspace="0em" id="S4.Ex2.m1.1.1.3.2.2.1" xref="S4.Ex2.m1.1.1.3.2.2.1.cmml">​</mo><mrow id="S4.Ex2.m1.1.1.3.2.2.3" xref="S4.Ex2.m1.1.1.3.2.2.3.cmml"><mi id="S4.Ex2.m1.1.1.3.2.2.3.1" xref="S4.Ex2.m1.1.1.3.2.2.3.1.cmml">log</mi><mo lspace="0.167em" id="S4.Ex2.m1.1.1.3.2.2.3a" xref="S4.Ex2.m1.1.1.3.2.2.3.cmml">⁡</mo><msub id="S4.Ex2.m1.1.1.3.2.2.3.2" xref="S4.Ex2.m1.1.1.3.2.2.3.2.cmml"><mi id="S4.Ex2.m1.1.1.3.2.2.3.2.2" xref="S4.Ex2.m1.1.1.3.2.2.3.2.2.cmml">p</mi><mrow id="S4.Ex2.m1.1.1.3.2.2.3.2.3" xref="S4.Ex2.m1.1.1.3.2.2.3.2.3.cmml"><mi id="S4.Ex2.m1.1.1.3.2.2.3.2.3.2" xref="S4.Ex2.m1.1.1.3.2.2.3.2.3.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="S4.Ex2.m1.1.1.3.2.2.3.2.3.1" xref="S4.Ex2.m1.1.1.3.2.2.3.2.3.1.cmml">​</mo><mi id="S4.Ex2.m1.1.1.3.2.2.3.2.3.3" xref="S4.Ex2.m1.1.1.3.2.2.3.2.3.3.cmml">i</mi></mrow></msub></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.Ex2.m1.1b"><apply id="S4.Ex2.m1.1.1.cmml" xref="S4.Ex2.m1.1.1"><eq id="S4.Ex2.m1.1.1.1.cmml" xref="S4.Ex2.m1.1.1.1"></eq><apply id="S4.Ex2.m1.1.1.2.cmml" xref="S4.Ex2.m1.1.1.2"><csymbol cd="ambiguous" id="S4.Ex2.m1.1.1.2.1.cmml" xref="S4.Ex2.m1.1.1.2">subscript</csymbol><ci id="S4.Ex2.m1.1.1.2.2.cmml" xref="S4.Ex2.m1.1.1.2.2">ℒ</ci><apply id="S4.Ex2.m1.1.1.2.3.cmml" xref="S4.Ex2.m1.1.1.2.3"><times id="S4.Ex2.m1.1.1.2.3.1.cmml" xref="S4.Ex2.m1.1.1.2.3.1"></times><ci id="S4.Ex2.m1.1.1.2.3.2.cmml" xref="S4.Ex2.m1.1.1.2.3.2">𝑖</ci><ci id="S4.Ex2.m1.1.1.2.3.3.cmml" xref="S4.Ex2.m1.1.1.2.3.3">𝐵</ci><ci id="S4.Ex2.m1.1.1.2.3.4.cmml" xref="S4.Ex2.m1.1.1.2.3.4">𝑂</ci><ci id="S4.Ex2.m1.1.1.2.3.5.cmml" xref="S4.Ex2.m1.1.1.2.3.5">𝑇</ci></apply></apply><apply id="S4.Ex2.m1.1.1.3.cmml" xref="S4.Ex2.m1.1.1.3"><minus id="S4.Ex2.m1.1.1.3.1.cmml" xref="S4.Ex2.m1.1.1.3"></minus><apply id="S4.Ex2.m1.1.1.3.2.cmml" xref="S4.Ex2.m1.1.1.3.2"><apply id="S4.Ex2.m1.1.1.3.2.1.cmml" xref="S4.Ex2.m1.1.1.3.2.1"><csymbol cd="ambiguous" id="S4.Ex2.m1.1.1.3.2.1.1.cmml" xref="S4.Ex2.m1.1.1.3.2.1">subscript</csymbol><sum id="S4.Ex2.m1.1.1.3.2.1.2.cmml" xref="S4.Ex2.m1.1.1.3.2.1.2"></sum><ci id="S4.Ex2.m1.1.1.3.2.1.3.cmml" xref="S4.Ex2.m1.1.1.3.2.1.3">𝑖</ci></apply><apply id="S4.Ex2.m1.1.1.3.2.2.cmml" xref="S4.Ex2.m1.1.1.3.2.2"><times id="S4.Ex2.m1.1.1.3.2.2.1.cmml" xref="S4.Ex2.m1.1.1.3.2.2.1"></times><apply id="S4.Ex2.m1.1.1.3.2.2.2.cmml" xref="S4.Ex2.m1.1.1.3.2.2.2"><csymbol cd="ambiguous" id="S4.Ex2.m1.1.1.3.2.2.2.1.cmml" xref="S4.Ex2.m1.1.1.3.2.2.2">subscript</csymbol><ci id="S4.Ex2.m1.1.1.3.2.2.2.2.cmml" xref="S4.Ex2.m1.1.1.3.2.2.2.2">𝑝</ci><apply id="S4.Ex2.m1.1.1.3.2.2.2.3.cmml" xref="S4.Ex2.m1.1.1.3.2.2.2.3"><times id="S4.Ex2.m1.1.1.3.2.2.2.3.1.cmml" xref="S4.Ex2.m1.1.1.3.2.2.2.3.1"></times><ci id="S4.Ex2.m1.1.1.3.2.2.2.3.2.cmml" xref="S4.Ex2.m1.1.1.3.2.2.2.3.2">𝑡</ci><ci id="S4.Ex2.m1.1.1.3.2.2.2.3.3.cmml" xref="S4.Ex2.m1.1.1.3.2.2.2.3.3">𝑖</ci></apply></apply><apply id="S4.Ex2.m1.1.1.3.2.2.3.cmml" xref="S4.Ex2.m1.1.1.3.2.2.3"><log id="S4.Ex2.m1.1.1.3.2.2.3.1.cmml" xref="S4.Ex2.m1.1.1.3.2.2.3.1"></log><apply id="S4.Ex2.m1.1.1.3.2.2.3.2.cmml" xref="S4.Ex2.m1.1.1.3.2.2.3.2"><csymbol cd="ambiguous" id="S4.Ex2.m1.1.1.3.2.2.3.2.1.cmml" xref="S4.Ex2.m1.1.1.3.2.2.3.2">subscript</csymbol><ci id="S4.Ex2.m1.1.1.3.2.2.3.2.2.cmml" xref="S4.Ex2.m1.1.1.3.2.2.3.2.2">𝑝</ci><apply id="S4.Ex2.m1.1.1.3.2.2.3.2.3.cmml" xref="S4.Ex2.m1.1.1.3.2.2.3.2.3"><times id="S4.Ex2.m1.1.1.3.2.2.3.2.3.1.cmml" xref="S4.Ex2.m1.1.1.3.2.2.3.2.3.1"></times><ci id="S4.Ex2.m1.1.1.3.2.2.3.2.3.2.cmml" xref="S4.Ex2.m1.1.1.3.2.2.3.2.3.2">𝑠</ci><ci id="S4.Ex2.m1.1.1.3.2.2.3.2.3.3.cmml" xref="S4.Ex2.m1.1.1.3.2.2.3.2.3.3">𝑖</ci></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Ex2.m1.1c">{\mathcal{L}}_{iBOT}=-\sum_{i}p_{ti}\log p_{si}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S4.I1.i2.p1.1" class="ltx_p">,
where <math id="S4.I1.i2.p1.1.m1.1" class="ltx_Math" alttext="i" display="inline"><semantics id="S4.I1.i2.p1.1.m1.1a"><mi id="S4.I1.i2.p1.1.m1.1.1" xref="S4.I1.i2.p1.1.m1.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="S4.I1.i2.p1.1.m1.1b"><ci id="S4.I1.i2.p1.1.m1.1.1.cmml" xref="S4.I1.i2.p1.1.m1.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i2.p1.1.m1.1c">i</annotation></semantics></math> are patch indices for masked tokens.
Similarly to above, we learn the parameters of the student, and build the teacher head through exponential moving average.</p>
</div>
</li>
<li id="S4.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I1.i3.p1" class="ltx_para ltx_noindent">
<p id="S4.I1.i3.p1.1" class="ltx_p"><span id="S4.I1.i3.p1.1.1" class="ltx_text ltx_font_bold">Untying head weights between both objectives.</span>

Both the DINO and the iBOT loss use a learnable MLP projection head.
It is applied to the output tokens and the loss is compute atop.
In <cite class="ltx_cite ltx_citemacro_cite">Zhou et al. (<a href="#bib.bib136" title="" class="ltx_ref">2022a</a>)</cite>, an ablation study shows that sharing parameters between the DINO and iBOT heads leads to better performance.
At scale, we observed that the opposite is true, and we therefore use two separate heads in all our experiments.</p>
</div>
</li>
<li id="S4.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I1.i4.p1" class="ltx_para ltx_noindent">
<p id="S4.I1.i4.p1.1" class="ltx_p"><span id="S4.I1.i4.p1.1.1" class="ltx_text ltx_font_bold">Sinkhorn-Knopp centering <cite class="ltx_cite ltx_citemacro_citep">(Caron et al., <a href="#bib.bib18" title="" class="ltx_ref">2020</a>)</cite>.</span>
<cite class="ltx_cite ltx_citemacro_cite">Ruan et al. (<a href="#bib.bib98" title="" class="ltx_ref">2023</a>)</cite> recommend to replace the teacher softmax-centering step of DINO and iBot by the Sinkhorn-Knopp (SK) batch normalization of SwAV <cite class="ltx_cite ltx_citemacro_citep">(Caron et al., <a href="#bib.bib18" title="" class="ltx_ref">2020</a>)</cite>.
We run the Sinkhorn-Knopp algorithm steps for 3 iterations.
For the student, we apply the softmax normalization.</p>
</div>
</li>
<li id="S4.I1.i5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I1.i5.p1" class="ltx_para ltx_noindent">
<p id="S4.I1.i5.p1.2" class="ltx_p"><span id="S4.I1.i5.p1.2.1" class="ltx_text ltx_font_bold">KoLeo regularizer  <cite class="ltx_cite ltx_citemacro_citep">(Sablayrolles et al., <a href="#bib.bib100" title="" class="ltx_ref">2019</a>)</cite>.</span>
The KoLeo regularizer derives from the Kozachenko-Leonenko differential entropy estimator (see  <cite class="ltx_cite ltx_citemacro_citet">Beirlant et al. (<a href="#bib.bib7" title="" class="ltx_ref">1997</a>); Delattre &amp; Fournier (<a href="#bib.bib33" title="" class="ltx_ref">2017</a>)</cite>) and encourages a uniform span of the features within a batch.
Given a set of <math id="S4.I1.i5.p1.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S4.I1.i5.p1.1.m1.1a"><mi id="S4.I1.i5.p1.1.m1.1.1" xref="S4.I1.i5.p1.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S4.I1.i5.p1.1.m1.1b"><ci id="S4.I1.i5.p1.1.m1.1.1.cmml" xref="S4.I1.i5.p1.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i5.p1.1.m1.1c">n</annotation></semantics></math> vectors <math id="S4.I1.i5.p1.2.m2.3" class="ltx_Math" alttext="(x_{1},\dots,x_{n})" display="inline"><semantics id="S4.I1.i5.p1.2.m2.3a"><mrow id="S4.I1.i5.p1.2.m2.3.3.2" xref="S4.I1.i5.p1.2.m2.3.3.3.cmml"><mo stretchy="false" id="S4.I1.i5.p1.2.m2.3.3.2.3" xref="S4.I1.i5.p1.2.m2.3.3.3.cmml">(</mo><msub id="S4.I1.i5.p1.2.m2.2.2.1.1" xref="S4.I1.i5.p1.2.m2.2.2.1.1.cmml"><mi id="S4.I1.i5.p1.2.m2.2.2.1.1.2" xref="S4.I1.i5.p1.2.m2.2.2.1.1.2.cmml">x</mi><mn id="S4.I1.i5.p1.2.m2.2.2.1.1.3" xref="S4.I1.i5.p1.2.m2.2.2.1.1.3.cmml">1</mn></msub><mo id="S4.I1.i5.p1.2.m2.3.3.2.4" xref="S4.I1.i5.p1.2.m2.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S4.I1.i5.p1.2.m2.1.1" xref="S4.I1.i5.p1.2.m2.1.1.cmml">…</mi><mo id="S4.I1.i5.p1.2.m2.3.3.2.5" xref="S4.I1.i5.p1.2.m2.3.3.3.cmml">,</mo><msub id="S4.I1.i5.p1.2.m2.3.3.2.2" xref="S4.I1.i5.p1.2.m2.3.3.2.2.cmml"><mi id="S4.I1.i5.p1.2.m2.3.3.2.2.2" xref="S4.I1.i5.p1.2.m2.3.3.2.2.2.cmml">x</mi><mi id="S4.I1.i5.p1.2.m2.3.3.2.2.3" xref="S4.I1.i5.p1.2.m2.3.3.2.2.3.cmml">n</mi></msub><mo stretchy="false" id="S4.I1.i5.p1.2.m2.3.3.2.6" xref="S4.I1.i5.p1.2.m2.3.3.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.I1.i5.p1.2.m2.3b"><vector id="S4.I1.i5.p1.2.m2.3.3.3.cmml" xref="S4.I1.i5.p1.2.m2.3.3.2"><apply id="S4.I1.i5.p1.2.m2.2.2.1.1.cmml" xref="S4.I1.i5.p1.2.m2.2.2.1.1"><csymbol cd="ambiguous" id="S4.I1.i5.p1.2.m2.2.2.1.1.1.cmml" xref="S4.I1.i5.p1.2.m2.2.2.1.1">subscript</csymbol><ci id="S4.I1.i5.p1.2.m2.2.2.1.1.2.cmml" xref="S4.I1.i5.p1.2.m2.2.2.1.1.2">𝑥</ci><cn type="integer" id="S4.I1.i5.p1.2.m2.2.2.1.1.3.cmml" xref="S4.I1.i5.p1.2.m2.2.2.1.1.3">1</cn></apply><ci id="S4.I1.i5.p1.2.m2.1.1.cmml" xref="S4.I1.i5.p1.2.m2.1.1">…</ci><apply id="S4.I1.i5.p1.2.m2.3.3.2.2.cmml" xref="S4.I1.i5.p1.2.m2.3.3.2.2"><csymbol cd="ambiguous" id="S4.I1.i5.p1.2.m2.3.3.2.2.1.cmml" xref="S4.I1.i5.p1.2.m2.3.3.2.2">subscript</csymbol><ci id="S4.I1.i5.p1.2.m2.3.3.2.2.2.cmml" xref="S4.I1.i5.p1.2.m2.3.3.2.2.2">𝑥</ci><ci id="S4.I1.i5.p1.2.m2.3.3.2.2.3.cmml" xref="S4.I1.i5.p1.2.m2.3.3.2.2.3">𝑛</ci></apply></vector></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i5.p1.2.m2.3c">(x_{1},\dots,x_{n})</annotation></semantics></math>, it is defined as</p>
<table id="S4.Ex3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.Ex3.m1.4" class="ltx_Math" alttext="{\mathcal{L}}_{\mathrm{koleo}}=-\frac{1}{n}\sum_{i=1}^{n}\log(d_{n,i})," display="block"><semantics id="S4.Ex3.m1.4a"><mrow id="S4.Ex3.m1.4.4.1" xref="S4.Ex3.m1.4.4.1.1.cmml"><mrow id="S4.Ex3.m1.4.4.1.1" xref="S4.Ex3.m1.4.4.1.1.cmml"><msub id="S4.Ex3.m1.4.4.1.1.3" xref="S4.Ex3.m1.4.4.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.Ex3.m1.4.4.1.1.3.2" xref="S4.Ex3.m1.4.4.1.1.3.2.cmml">ℒ</mi><mi id="S4.Ex3.m1.4.4.1.1.3.3" xref="S4.Ex3.m1.4.4.1.1.3.3.cmml">koleo</mi></msub><mo id="S4.Ex3.m1.4.4.1.1.2" xref="S4.Ex3.m1.4.4.1.1.2.cmml">=</mo><mrow id="S4.Ex3.m1.4.4.1.1.1" xref="S4.Ex3.m1.4.4.1.1.1.cmml"><mo id="S4.Ex3.m1.4.4.1.1.1a" xref="S4.Ex3.m1.4.4.1.1.1.cmml">−</mo><mrow id="S4.Ex3.m1.4.4.1.1.1.1" xref="S4.Ex3.m1.4.4.1.1.1.1.cmml"><mfrac id="S4.Ex3.m1.4.4.1.1.1.1.3" xref="S4.Ex3.m1.4.4.1.1.1.1.3.cmml"><mn id="S4.Ex3.m1.4.4.1.1.1.1.3.2" xref="S4.Ex3.m1.4.4.1.1.1.1.3.2.cmml">1</mn><mi id="S4.Ex3.m1.4.4.1.1.1.1.3.3" xref="S4.Ex3.m1.4.4.1.1.1.1.3.3.cmml">n</mi></mfrac><mo lspace="0em" rspace="0em" id="S4.Ex3.m1.4.4.1.1.1.1.2" xref="S4.Ex3.m1.4.4.1.1.1.1.2.cmml">​</mo><mrow id="S4.Ex3.m1.4.4.1.1.1.1.1" xref="S4.Ex3.m1.4.4.1.1.1.1.1.cmml"><munderover id="S4.Ex3.m1.4.4.1.1.1.1.1.2" xref="S4.Ex3.m1.4.4.1.1.1.1.1.2.cmml"><mo movablelimits="false" id="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.2" xref="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.2.cmml">∑</mo><mrow id="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.3" xref="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.3.cmml"><mi id="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.3.2" xref="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.3.2.cmml">i</mi><mo id="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.3.1" xref="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.3.1.cmml">=</mo><mn id="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.3.3" xref="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="S4.Ex3.m1.4.4.1.1.1.1.1.2.3" xref="S4.Ex3.m1.4.4.1.1.1.1.1.2.3.cmml">n</mi></munderover><mrow id="S4.Ex3.m1.4.4.1.1.1.1.1.1.1" xref="S4.Ex3.m1.4.4.1.1.1.1.1.1.2.cmml"><mi id="S4.Ex3.m1.3.3" xref="S4.Ex3.m1.3.3.cmml">log</mi><mo id="S4.Ex3.m1.4.4.1.1.1.1.1.1.1a" xref="S4.Ex3.m1.4.4.1.1.1.1.1.1.2.cmml">⁡</mo><mrow id="S4.Ex3.m1.4.4.1.1.1.1.1.1.1.1" xref="S4.Ex3.m1.4.4.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S4.Ex3.m1.4.4.1.1.1.1.1.1.1.1.2" xref="S4.Ex3.m1.4.4.1.1.1.1.1.1.2.cmml">(</mo><msub id="S4.Ex3.m1.4.4.1.1.1.1.1.1.1.1.1" xref="S4.Ex3.m1.4.4.1.1.1.1.1.1.1.1.1.cmml"><mi id="S4.Ex3.m1.4.4.1.1.1.1.1.1.1.1.1.2" xref="S4.Ex3.m1.4.4.1.1.1.1.1.1.1.1.1.2.cmml">d</mi><mrow id="S4.Ex3.m1.2.2.2.4" xref="S4.Ex3.m1.2.2.2.3.cmml"><mi id="S4.Ex3.m1.1.1.1.1" xref="S4.Ex3.m1.1.1.1.1.cmml">n</mi><mo id="S4.Ex3.m1.2.2.2.4.1" xref="S4.Ex3.m1.2.2.2.3.cmml">,</mo><mi id="S4.Ex3.m1.2.2.2.2" xref="S4.Ex3.m1.2.2.2.2.cmml">i</mi></mrow></msub><mo stretchy="false" id="S4.Ex3.m1.4.4.1.1.1.1.1.1.1.1.3" xref="S4.Ex3.m1.4.4.1.1.1.1.1.1.2.cmml">)</mo></mrow></mrow></mrow></mrow></mrow></mrow><mo id="S4.Ex3.m1.4.4.1.2" xref="S4.Ex3.m1.4.4.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.Ex3.m1.4b"><apply id="S4.Ex3.m1.4.4.1.1.cmml" xref="S4.Ex3.m1.4.4.1"><eq id="S4.Ex3.m1.4.4.1.1.2.cmml" xref="S4.Ex3.m1.4.4.1.1.2"></eq><apply id="S4.Ex3.m1.4.4.1.1.3.cmml" xref="S4.Ex3.m1.4.4.1.1.3"><csymbol cd="ambiguous" id="S4.Ex3.m1.4.4.1.1.3.1.cmml" xref="S4.Ex3.m1.4.4.1.1.3">subscript</csymbol><ci id="S4.Ex3.m1.4.4.1.1.3.2.cmml" xref="S4.Ex3.m1.4.4.1.1.3.2">ℒ</ci><ci id="S4.Ex3.m1.4.4.1.1.3.3.cmml" xref="S4.Ex3.m1.4.4.1.1.3.3">koleo</ci></apply><apply id="S4.Ex3.m1.4.4.1.1.1.cmml" xref="S4.Ex3.m1.4.4.1.1.1"><minus id="S4.Ex3.m1.4.4.1.1.1.2.cmml" xref="S4.Ex3.m1.4.4.1.1.1"></minus><apply id="S4.Ex3.m1.4.4.1.1.1.1.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1"><times id="S4.Ex3.m1.4.4.1.1.1.1.2.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.2"></times><apply id="S4.Ex3.m1.4.4.1.1.1.1.3.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.3"><divide id="S4.Ex3.m1.4.4.1.1.1.1.3.1.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.3"></divide><cn type="integer" id="S4.Ex3.m1.4.4.1.1.1.1.3.2.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.3.2">1</cn><ci id="S4.Ex3.m1.4.4.1.1.1.1.3.3.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.3.3">𝑛</ci></apply><apply id="S4.Ex3.m1.4.4.1.1.1.1.1.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.1"><apply id="S4.Ex3.m1.4.4.1.1.1.1.1.2.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.Ex3.m1.4.4.1.1.1.1.1.2.1.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.1.2">superscript</csymbol><apply id="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.1.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.1.2">subscript</csymbol><sum id="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.2.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.2"></sum><apply id="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.3.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.3"><eq id="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.3.1.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.3.1"></eq><ci id="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.3.2.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.3.2">𝑖</ci><cn type="integer" id="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.3.3.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.1.2.2.3.3">1</cn></apply></apply><ci id="S4.Ex3.m1.4.4.1.1.1.1.1.2.3.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.1.2.3">𝑛</ci></apply><apply id="S4.Ex3.m1.4.4.1.1.1.1.1.1.2.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.1.1.1"><log id="S4.Ex3.m1.3.3.cmml" xref="S4.Ex3.m1.3.3"></log><apply id="S4.Ex3.m1.4.4.1.1.1.1.1.1.1.1.1.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.Ex3.m1.4.4.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S4.Ex3.m1.4.4.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.Ex3.m1.4.4.1.1.1.1.1.1.1.1.1.2">𝑑</ci><list id="S4.Ex3.m1.2.2.2.3.cmml" xref="S4.Ex3.m1.2.2.2.4"><ci id="S4.Ex3.m1.1.1.1.1.cmml" xref="S4.Ex3.m1.1.1.1.1">𝑛</ci><ci id="S4.Ex3.m1.2.2.2.2.cmml" xref="S4.Ex3.m1.2.2.2.2">𝑖</ci></list></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Ex3.m1.4c">{\mathcal{L}}_{\mathrm{koleo}}=-\frac{1}{n}\sum_{i=1}^{n}\log(d_{n,i}),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S4.I1.i5.p1.5" class="ltx_p">where <math id="S4.I1.i5.p1.3.m1.3" class="ltx_Math" alttext="d_{n,i}=\min_{j\neq i}\|x_{i}-x_{j}\|" display="inline"><semantics id="S4.I1.i5.p1.3.m1.3a"><mrow id="S4.I1.i5.p1.3.m1.3.3" xref="S4.I1.i5.p1.3.m1.3.3.cmml"><msub id="S4.I1.i5.p1.3.m1.3.3.3" xref="S4.I1.i5.p1.3.m1.3.3.3.cmml"><mi id="S4.I1.i5.p1.3.m1.3.3.3.2" xref="S4.I1.i5.p1.3.m1.3.3.3.2.cmml">d</mi><mrow id="S4.I1.i5.p1.3.m1.2.2.2.4" xref="S4.I1.i5.p1.3.m1.2.2.2.3.cmml"><mi id="S4.I1.i5.p1.3.m1.1.1.1.1" xref="S4.I1.i5.p1.3.m1.1.1.1.1.cmml">n</mi><mo id="S4.I1.i5.p1.3.m1.2.2.2.4.1" xref="S4.I1.i5.p1.3.m1.2.2.2.3.cmml">,</mo><mi id="S4.I1.i5.p1.3.m1.2.2.2.2" xref="S4.I1.i5.p1.3.m1.2.2.2.2.cmml">i</mi></mrow></msub><mo id="S4.I1.i5.p1.3.m1.3.3.2" xref="S4.I1.i5.p1.3.m1.3.3.2.cmml">=</mo><mrow id="S4.I1.i5.p1.3.m1.3.3.1" xref="S4.I1.i5.p1.3.m1.3.3.1.cmml"><msub id="S4.I1.i5.p1.3.m1.3.3.1.2" xref="S4.I1.i5.p1.3.m1.3.3.1.2.cmml"><mi id="S4.I1.i5.p1.3.m1.3.3.1.2.2" xref="S4.I1.i5.p1.3.m1.3.3.1.2.2.cmml">min</mi><mrow id="S4.I1.i5.p1.3.m1.3.3.1.2.3" xref="S4.I1.i5.p1.3.m1.3.3.1.2.3.cmml"><mi id="S4.I1.i5.p1.3.m1.3.3.1.2.3.2" xref="S4.I1.i5.p1.3.m1.3.3.1.2.3.2.cmml">j</mi><mo id="S4.I1.i5.p1.3.m1.3.3.1.2.3.1" xref="S4.I1.i5.p1.3.m1.3.3.1.2.3.1.cmml">≠</mo><mi id="S4.I1.i5.p1.3.m1.3.3.1.2.3.3" xref="S4.I1.i5.p1.3.m1.3.3.1.2.3.3.cmml">i</mi></mrow></msub><mo id="S4.I1.i5.p1.3.m1.3.3.1a" xref="S4.I1.i5.p1.3.m1.3.3.1.cmml">⁡</mo><mrow id="S4.I1.i5.p1.3.m1.3.3.1.1.1" xref="S4.I1.i5.p1.3.m1.3.3.1.1.2.cmml"><mo stretchy="false" id="S4.I1.i5.p1.3.m1.3.3.1.1.1.2" xref="S4.I1.i5.p1.3.m1.3.3.1.1.2.1.cmml">‖</mo><mrow id="S4.I1.i5.p1.3.m1.3.3.1.1.1.1" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.cmml"><msub id="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.2" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.2.cmml"><mi id="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.2.2" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.2.2.cmml">x</mi><mi id="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.2.3" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.2.3.cmml">i</mi></msub><mo id="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.1" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.1.cmml">−</mo><msub id="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.3" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.3.cmml"><mi id="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.3.2" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.3.2.cmml">x</mi><mi id="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.3.3" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.3.3.cmml">j</mi></msub></mrow><mo stretchy="false" id="S4.I1.i5.p1.3.m1.3.3.1.1.1.3" xref="S4.I1.i5.p1.3.m1.3.3.1.1.2.1.cmml">‖</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.I1.i5.p1.3.m1.3b"><apply id="S4.I1.i5.p1.3.m1.3.3.cmml" xref="S4.I1.i5.p1.3.m1.3.3"><eq id="S4.I1.i5.p1.3.m1.3.3.2.cmml" xref="S4.I1.i5.p1.3.m1.3.3.2"></eq><apply id="S4.I1.i5.p1.3.m1.3.3.3.cmml" xref="S4.I1.i5.p1.3.m1.3.3.3"><csymbol cd="ambiguous" id="S4.I1.i5.p1.3.m1.3.3.3.1.cmml" xref="S4.I1.i5.p1.3.m1.3.3.3">subscript</csymbol><ci id="S4.I1.i5.p1.3.m1.3.3.3.2.cmml" xref="S4.I1.i5.p1.3.m1.3.3.3.2">𝑑</ci><list id="S4.I1.i5.p1.3.m1.2.2.2.3.cmml" xref="S4.I1.i5.p1.3.m1.2.2.2.4"><ci id="S4.I1.i5.p1.3.m1.1.1.1.1.cmml" xref="S4.I1.i5.p1.3.m1.1.1.1.1">𝑛</ci><ci id="S4.I1.i5.p1.3.m1.2.2.2.2.cmml" xref="S4.I1.i5.p1.3.m1.2.2.2.2">𝑖</ci></list></apply><apply id="S4.I1.i5.p1.3.m1.3.3.1.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1"><apply id="S4.I1.i5.p1.3.m1.3.3.1.2.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.2"><csymbol cd="ambiguous" id="S4.I1.i5.p1.3.m1.3.3.1.2.1.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.2">subscript</csymbol><min id="S4.I1.i5.p1.3.m1.3.3.1.2.2.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.2.2"></min><apply id="S4.I1.i5.p1.3.m1.3.3.1.2.3.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.2.3"><neq id="S4.I1.i5.p1.3.m1.3.3.1.2.3.1.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.2.3.1"></neq><ci id="S4.I1.i5.p1.3.m1.3.3.1.2.3.2.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.2.3.2">𝑗</ci><ci id="S4.I1.i5.p1.3.m1.3.3.1.2.3.3.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.2.3.3">𝑖</ci></apply></apply><apply id="S4.I1.i5.p1.3.m1.3.3.1.1.2.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1"><csymbol cd="latexml" id="S4.I1.i5.p1.3.m1.3.3.1.1.2.1.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.2">norm</csymbol><apply id="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.1"><minus id="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.1.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.1"></minus><apply id="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.2.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.2.1.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.2">subscript</csymbol><ci id="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.2.2.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.2.2">𝑥</ci><ci id="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.2.3.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.2.3">𝑖</ci></apply><apply id="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.3.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.3.1.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.3">subscript</csymbol><ci id="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.3.2.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.3.2">𝑥</ci><ci id="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.3.3.cmml" xref="S4.I1.i5.p1.3.m1.3.3.1.1.1.1.3.3">𝑗</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i5.p1.3.m1.3c">d_{n,i}=\min_{j\neq i}\|x_{i}-x_{j}\|</annotation></semantics></math> is the minimum distance between <math id="S4.I1.i5.p1.4.m2.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S4.I1.i5.p1.4.m2.1a"><msub id="S4.I1.i5.p1.4.m2.1.1" xref="S4.I1.i5.p1.4.m2.1.1.cmml"><mi id="S4.I1.i5.p1.4.m2.1.1.2" xref="S4.I1.i5.p1.4.m2.1.1.2.cmml">x</mi><mi id="S4.I1.i5.p1.4.m2.1.1.3" xref="S4.I1.i5.p1.4.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S4.I1.i5.p1.4.m2.1b"><apply id="S4.I1.i5.p1.4.m2.1.1.cmml" xref="S4.I1.i5.p1.4.m2.1.1"><csymbol cd="ambiguous" id="S4.I1.i5.p1.4.m2.1.1.1.cmml" xref="S4.I1.i5.p1.4.m2.1.1">subscript</csymbol><ci id="S4.I1.i5.p1.4.m2.1.1.2.cmml" xref="S4.I1.i5.p1.4.m2.1.1.2">𝑥</ci><ci id="S4.I1.i5.p1.4.m2.1.1.3.cmml" xref="S4.I1.i5.p1.4.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i5.p1.4.m2.1c">x_{i}</annotation></semantics></math> and any other point within the batch.
We also <math id="S4.I1.i5.p1.5.m3.1" class="ltx_Math" alttext="\ell_{2}" display="inline"><semantics id="S4.I1.i5.p1.5.m3.1a"><msub id="S4.I1.i5.p1.5.m3.1.1" xref="S4.I1.i5.p1.5.m3.1.1.cmml"><mi mathvariant="normal" id="S4.I1.i5.p1.5.m3.1.1.2" xref="S4.I1.i5.p1.5.m3.1.1.2.cmml">ℓ</mi><mn id="S4.I1.i5.p1.5.m3.1.1.3" xref="S4.I1.i5.p1.5.m3.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.I1.i5.p1.5.m3.1b"><apply id="S4.I1.i5.p1.5.m3.1.1.cmml" xref="S4.I1.i5.p1.5.m3.1.1"><csymbol cd="ambiguous" id="S4.I1.i5.p1.5.m3.1.1.1.cmml" xref="S4.I1.i5.p1.5.m3.1.1">subscript</csymbol><ci id="S4.I1.i5.p1.5.m3.1.1.2.cmml" xref="S4.I1.i5.p1.5.m3.1.1.2">ℓ</ci><cn type="integer" id="S4.I1.i5.p1.5.m3.1.1.3.cmml" xref="S4.I1.i5.p1.5.m3.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i5.p1.5.m3.1c">\ell_{2}</annotation></semantics></math>-normalize the features before computing this regularizer.</p>
</div>
</li>
<li id="S4.I1.i6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I1.i6.p1" class="ltx_para ltx_noindent">
<p id="S4.I1.i6.p1.1" class="ltx_p"><span id="S4.I1.i6.p1.1.1" class="ltx_text ltx_font_bold">Adapting the resolution <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib113" title="" class="ltx_ref">2019</a>)</cite>.</span>
Increasing image resolution is key to pixel-level downstream tasks such as segmentation or detection, where small objects disappear at low resolutions.
However, training at high resolution is time and memory demanding, and instead, we increase the resolution of images to <math id="S4.I1.i6.p1.1.m1.1" class="ltx_Math" alttext="518\times 518" display="inline"><semantics id="S4.I1.i6.p1.1.m1.1a"><mrow id="S4.I1.i6.p1.1.m1.1.1" xref="S4.I1.i6.p1.1.m1.1.1.cmml"><mn id="S4.I1.i6.p1.1.m1.1.1.2" xref="S4.I1.i6.p1.1.m1.1.1.2.cmml">518</mn><mo lspace="0.222em" rspace="0.222em" id="S4.I1.i6.p1.1.m1.1.1.1" xref="S4.I1.i6.p1.1.m1.1.1.1.cmml">×</mo><mn id="S4.I1.i6.p1.1.m1.1.1.3" xref="S4.I1.i6.p1.1.m1.1.1.3.cmml">518</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.I1.i6.p1.1.m1.1b"><apply id="S4.I1.i6.p1.1.m1.1.1.cmml" xref="S4.I1.i6.p1.1.m1.1.1"><times id="S4.I1.i6.p1.1.m1.1.1.1.cmml" xref="S4.I1.i6.p1.1.m1.1.1.1"></times><cn type="integer" id="S4.I1.i6.p1.1.m1.1.1.2.cmml" xref="S4.I1.i6.p1.1.m1.1.1.2">518</cn><cn type="integer" id="S4.I1.i6.p1.1.m1.1.1.3.cmml" xref="S4.I1.i6.p1.1.m1.1.1.3">518</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.I1.i6.p1.1.m1.1c">518\times 518</annotation></semantics></math> during a short period at the end of pretraining. This is also similar to UniViT training from <cite class="ltx_cite ltx_citemacro_cite">Likhomanenko et al. (<a href="#bib.bib77" title="" class="ltx_ref">2021</a>)</cite> and FlexiViT training from <cite class="ltx_cite ltx_citemacro_cite">Beyer et al. (<a href="#bib.bib10" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
</li>
</ul>
</div>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Efficient implementation</h2>

<div id="S5.p1" class="ltx_para ltx_noindent">
<p id="S5.p1.2" class="ltx_p">We consider several improvements to train models at a larger scale.
We train models on A100 GPUs using PyTorch 2.0.
The code and pretrained models are made available under Apache 2.0 license <span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span><a target="_blank" href="https://github.com/facebookresearch/dinov2" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/facebookresearch/dinov2</a></span></span></span>.
The details of our models are in the appendix, Table <a href="#A2.T17" title="Table 17 ‣ B.1 Unsupervised pre-training ‣ Appendix B Implementation Details ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">17</span></a>. With the same hardware, compared to the iBOT implementation, the DINOv2 code runs around <math id="S5.p1.1.m1.1" class="ltx_math_unparsed" alttext="2\times" display="inline"><semantics id="S5.p1.1.m1.1a"><mrow id="S5.p1.1.m1.1b"><mn id="S5.p1.1.m1.1.1">2</mn><mo lspace="0.222em" id="S5.p1.1.m1.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S5.p1.1.m1.1c">2\times</annotation></semantics></math> faster using only <math id="S5.p1.2.m2.1" class="ltx_Math" alttext="1/3" display="inline"><semantics id="S5.p1.2.m2.1a"><mrow id="S5.p1.2.m2.1.1" xref="S5.p1.2.m2.1.1.cmml"><mn id="S5.p1.2.m2.1.1.2" xref="S5.p1.2.m2.1.1.2.cmml">1</mn><mo id="S5.p1.2.m2.1.1.1" xref="S5.p1.2.m2.1.1.1.cmml">/</mo><mn id="S5.p1.2.m2.1.1.3" xref="S5.p1.2.m2.1.1.3.cmml">3</mn></mrow><annotation-xml encoding="MathML-Content" id="S5.p1.2.m2.1b"><apply id="S5.p1.2.m2.1.1.cmml" xref="S5.p1.2.m2.1.1"><divide id="S5.p1.2.m2.1.1.1.cmml" xref="S5.p1.2.m2.1.1.1"></divide><cn type="integer" id="S5.p1.2.m2.1.1.2.cmml" xref="S5.p1.2.m2.1.1.2">1</cn><cn type="integer" id="S5.p1.2.m2.1.1.3.cmml" xref="S5.p1.2.m2.1.1.3">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.p1.2.m2.1c">1/3</annotation></semantics></math> of the memory.</p>
</div>
<section id="S5.SS0.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Fast and memory-efficient attention.</h5>

<div id="S5.SS0.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S5.SS0.SSS0.Px1.p1.1" class="ltx_p">We implemented our own version of FlashAttention <cite class="ltx_cite ltx_citemacro_citep">(Dao et al., <a href="#bib.bib31" title="" class="ltx_ref">2022</a>)</cite> to improve memory usage and speed on the self-attention layers.
Our version is on par with or better than the original on all cases considered, while covering more use-cases and hardware.
Due to the GPU hardware specifics, the efficiency is best when the embedding dimension per head is a multiple of 64, and the matrix operations are even better when the full embedding dimension is a multiple of 256.
As a consequence, our ViT-g architecture slightly differs from the architecture proposed by <cite class="ltx_cite ltx_citemacro_cite">Zhai et al. (<a href="#bib.bib132" title="" class="ltx_ref">2022</a>)</cite> in order to maximize compute efficiency, and we use an embedding dimension of 1536 with 24 heads (64 dim/head), rather than 1408 with 16 heads (88 dim/head). Our experiments did not show significant differences in final accuracy, and our ViT-g backbone counts 1.1B parameters.</p>
</div>
</section>
<section id="S5.SS0.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Sequence packing.</h5>

<div id="S5.SS0.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S5.SS0.SSS0.Px2.p1.1" class="ltx_p">The DINO algorithm requires forwarding both large crops (at resolution 224) and small crops (resolution 98).
When split into patches, these two groups are represented by token sequences of different lengths and cannot be forwarded together.
In order to accelerate training, we use a trick called "sequence packing," which originates from NLP <cite class="ltx_cite ltx_citemacro_citep">(Krell et al., <a href="#bib.bib72" title="" class="ltx_ref">2022</a>)</cite>.
The idea is simple: we concatenate the sequences we must forward through the transformers into a single long sequence.
We pass this sequence through the transformer blocks as usual.
However, a block-diagonal mask is applied to the self-attention matrix in attention layers, preventing attention between different sequences.
This way, the forward is strictly equivalent to forwarding each sequence separately.
This trick gives us significant compute efficiency gains compared to using separate forward and backward passes, as in prior implementations.
The lower-level components of our setup are available in the xFormers library<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span><a target="_blank" href="https://github.com/facebookresearch/xformers" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/facebookresearch/xformers</a></span></span></span> (<cite class="ltx_cite ltx_citemacro_cite">Lefaudeux et al. (<a href="#bib.bib74" title="" class="ltx_ref">2022</a>)</cite>).</p>
</div>
</section>
<section id="S5.SS0.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Efficient stochastic depth.</h5>

<div id="S5.SS0.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S5.SS0.SSS0.Px3.p1.3" class="ltx_p">We implement an improved version of stochastic depth <cite class="ltx_cite ltx_citemacro_citep">(Huang et al., <a href="#bib.bib65" title="" class="ltx_ref">2016</a>)</cite> that skips the computation of the dropped residuals rather than masking the result.
This saves memory and compute in proportion approximately equal to the drop rate, thanks to specific fused kernels.
With high drop rates (<math id="S5.SS0.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="d=40\%" display="inline"><semantics id="S5.SS0.SSS0.Px3.p1.1.m1.1a"><mrow id="S5.SS0.SSS0.Px3.p1.1.m1.1.1" xref="S5.SS0.SSS0.Px3.p1.1.m1.1.1.cmml"><mi id="S5.SS0.SSS0.Px3.p1.1.m1.1.1.2" xref="S5.SS0.SSS0.Px3.p1.1.m1.1.1.2.cmml">d</mi><mo id="S5.SS0.SSS0.Px3.p1.1.m1.1.1.1" xref="S5.SS0.SSS0.Px3.p1.1.m1.1.1.1.cmml">=</mo><mrow id="S5.SS0.SSS0.Px3.p1.1.m1.1.1.3" xref="S5.SS0.SSS0.Px3.p1.1.m1.1.1.3.cmml"><mn id="S5.SS0.SSS0.Px3.p1.1.m1.1.1.3.2" xref="S5.SS0.SSS0.Px3.p1.1.m1.1.1.3.2.cmml">40</mn><mo id="S5.SS0.SSS0.Px3.p1.1.m1.1.1.3.1" xref="S5.SS0.SSS0.Px3.p1.1.m1.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px3.p1.1.m1.1b"><apply id="S5.SS0.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S5.SS0.SSS0.Px3.p1.1.m1.1.1"><eq id="S5.SS0.SSS0.Px3.p1.1.m1.1.1.1.cmml" xref="S5.SS0.SSS0.Px3.p1.1.m1.1.1.1"></eq><ci id="S5.SS0.SSS0.Px3.p1.1.m1.1.1.2.cmml" xref="S5.SS0.SSS0.Px3.p1.1.m1.1.1.2">𝑑</ci><apply id="S5.SS0.SSS0.Px3.p1.1.m1.1.1.3.cmml" xref="S5.SS0.SSS0.Px3.p1.1.m1.1.1.3"><csymbol cd="latexml" id="S5.SS0.SSS0.Px3.p1.1.m1.1.1.3.1.cmml" xref="S5.SS0.SSS0.Px3.p1.1.m1.1.1.3.1">percent</csymbol><cn type="integer" id="S5.SS0.SSS0.Px3.p1.1.m1.1.1.3.2.cmml" xref="S5.SS0.SSS0.Px3.p1.1.m1.1.1.3.2">40</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px3.p1.1.m1.1c">d=40\%</annotation></semantics></math> in this work), this allows a drastic improvement in compute efficiency and memory usage.
The implementation consists of randomly shuffling the <math id="S5.SS0.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="B" display="inline"><semantics id="S5.SS0.SSS0.Px3.p1.2.m2.1a"><mi id="S5.SS0.SSS0.Px3.p1.2.m2.1.1" xref="S5.SS0.SSS0.Px3.p1.2.m2.1.1.cmml">B</mi><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px3.p1.2.m2.1b"><ci id="S5.SS0.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S5.SS0.SSS0.Px3.p1.2.m2.1.1">𝐵</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px3.p1.2.m2.1c">B</annotation></semantics></math> samples over the batch dimension, and slicing the first <math id="S5.SS0.SSS0.Px3.p1.3.m3.1" class="ltx_Math" alttext="(1-d)\times B" display="inline"><semantics id="S5.SS0.SSS0.Px3.p1.3.m3.1a"><mrow id="S5.SS0.SSS0.Px3.p1.3.m3.1.1" xref="S5.SS0.SSS0.Px3.p1.3.m3.1.1.cmml"><mrow id="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1" xref="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.1.cmml"><mo stretchy="false" id="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.2" xref="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.1.cmml">(</mo><mrow id="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.1" xref="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.1.cmml"><mn id="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.1.2" xref="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.1.2.cmml">1</mn><mo id="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.1.1" xref="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.1.1.cmml">−</mo><mi id="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.1.3" xref="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.1.3.cmml">d</mi></mrow><mo rspace="0.055em" stretchy="false" id="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.3" xref="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.1.cmml">)</mo></mrow><mo rspace="0.222em" id="S5.SS0.SSS0.Px3.p1.3.m3.1.1.2" xref="S5.SS0.SSS0.Px3.p1.3.m3.1.1.2.cmml">×</mo><mi id="S5.SS0.SSS0.Px3.p1.3.m3.1.1.3" xref="S5.SS0.SSS0.Px3.p1.3.m3.1.1.3.cmml">B</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px3.p1.3.m3.1b"><apply id="S5.SS0.SSS0.Px3.p1.3.m3.1.1.cmml" xref="S5.SS0.SSS0.Px3.p1.3.m3.1.1"><times id="S5.SS0.SSS0.Px3.p1.3.m3.1.1.2.cmml" xref="S5.SS0.SSS0.Px3.p1.3.m3.1.1.2"></times><apply id="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1"><minus id="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.1.1.cmml" xref="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.1.1"></minus><cn type="integer" id="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.1.2.cmml" xref="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.1.2">1</cn><ci id="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.1.3.cmml" xref="S5.SS0.SSS0.Px3.p1.3.m3.1.1.1.1.1.3">𝑑</ci></apply><ci id="S5.SS0.SSS0.Px3.p1.3.m3.1.1.3.cmml" xref="S5.SS0.SSS0.Px3.p1.3.m3.1.1.3">𝐵</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px3.p1.3.m3.1c">(1-d)\times B</annotation></semantics></math> samples for the computations in the block.</p>
</div>
</section>
<section id="S5.SS0.SSS0.Px4" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Fully-Sharded Data Parallel (FSDP).</h5>

<div id="S5.SS0.SSS0.Px4.p1" class="ltx_para ltx_noindent">
<p id="S5.SS0.SSS0.Px4.p1.2" class="ltx_p">Minimizing our objective with the AdamW optimizer requires 4 model replicas in float32 precision – student, teacher, optimizer first moments, optimizer second moments.
This sums to <math id="S5.SS0.SSS0.Px4.p1.1.m1.1" class="ltx_Math" alttext="16~{}\mathrm{GB}" display="inline"><semantics id="S5.SS0.SSS0.Px4.p1.1.m1.1a"><mrow id="S5.SS0.SSS0.Px4.p1.1.m1.1.1" xref="S5.SS0.SSS0.Px4.p1.1.m1.1.1.cmml"><mn id="S5.SS0.SSS0.Px4.p1.1.m1.1.1.2" xref="S5.SS0.SSS0.Px4.p1.1.m1.1.1.2.cmml">16</mn><mo lspace="0.330em" rspace="0em" id="S5.SS0.SSS0.Px4.p1.1.m1.1.1.1" xref="S5.SS0.SSS0.Px4.p1.1.m1.1.1.1.cmml">​</mo><mi id="S5.SS0.SSS0.Px4.p1.1.m1.1.1.3" xref="S5.SS0.SSS0.Px4.p1.1.m1.1.1.3.cmml">GB</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px4.p1.1.m1.1b"><apply id="S5.SS0.SSS0.Px4.p1.1.m1.1.1.cmml" xref="S5.SS0.SSS0.Px4.p1.1.m1.1.1"><times id="S5.SS0.SSS0.Px4.p1.1.m1.1.1.1.cmml" xref="S5.SS0.SSS0.Px4.p1.1.m1.1.1.1"></times><cn type="integer" id="S5.SS0.SSS0.Px4.p1.1.m1.1.1.2.cmml" xref="S5.SS0.SSS0.Px4.p1.1.m1.1.1.2">16</cn><ci id="S5.SS0.SSS0.Px4.p1.1.m1.1.1.3.cmml" xref="S5.SS0.SSS0.Px4.p1.1.m1.1.1.3">GB</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px4.p1.1.m1.1c">16~{}\mathrm{GB}</annotation></semantics></math> of memory for a billion-parameter model such as our ViT-g.
In order to reduce this memory footprint per GPU, we split the model replicas across GPUs, i.e., sharding <math id="S5.SS0.SSS0.Px4.p1.2.m2.1" class="ltx_Math" alttext="16~{}\mathrm{GB}" display="inline"><semantics id="S5.SS0.SSS0.Px4.p1.2.m2.1a"><mrow id="S5.SS0.SSS0.Px4.p1.2.m2.1.1" xref="S5.SS0.SSS0.Px4.p1.2.m2.1.1.cmml"><mn id="S5.SS0.SSS0.Px4.p1.2.m2.1.1.2" xref="S5.SS0.SSS0.Px4.p1.2.m2.1.1.2.cmml">16</mn><mo lspace="0.330em" rspace="0em" id="S5.SS0.SSS0.Px4.p1.2.m2.1.1.1" xref="S5.SS0.SSS0.Px4.p1.2.m2.1.1.1.cmml">​</mo><mi id="S5.SS0.SSS0.Px4.p1.2.m2.1.1.3" xref="S5.SS0.SSS0.Px4.p1.2.m2.1.1.3.cmml">GB</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS0.SSS0.Px4.p1.2.m2.1b"><apply id="S5.SS0.SSS0.Px4.p1.2.m2.1.1.cmml" xref="S5.SS0.SSS0.Px4.p1.2.m2.1.1"><times id="S5.SS0.SSS0.Px4.p1.2.m2.1.1.1.cmml" xref="S5.SS0.SSS0.Px4.p1.2.m2.1.1.1"></times><cn type="integer" id="S5.SS0.SSS0.Px4.p1.2.m2.1.1.2.cmml" xref="S5.SS0.SSS0.Px4.p1.2.m2.1.1.2">16</cn><ci id="S5.SS0.SSS0.Px4.p1.2.m2.1.1.3.cmml" xref="S5.SS0.SSS0.Px4.p1.2.m2.1.1.3">GB</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS0.SSS0.Px4.p1.2.m2.1c">16~{}\mathrm{GB}</annotation></semantics></math> across GPUs using the PyTorch implementation of FSDP.
Consequently, the model size is not bounded by the memory of a single GPU but by the total sum of GPU memory across compute nodes.
The Pytorch implementation of FSDP brings a second advantage, which is to save on the cross-GPU communication costs: the weight shards are stored in float32 precision as required by the optimizer, but broadcasting weights and reducing gradients is done in float16 precision for the backbone (MLP heads gradients are reduced in float32 to avoid training instabilities).
This leads to approximately 50% reduction in communication costs compared to the float32 gradient all-reduce operation used in DistributedDataParallel (DDP), which is used in other self-supervised pretraining methods <cite class="ltx_cite ltx_citemacro_citep">(Caron et al., <a href="#bib.bib19" title="" class="ltx_ref">2021</a>; Zhou et al., <a href="#bib.bib136" title="" class="ltx_ref">2022a</a>)</cite>.
As a consequence, the training procedure scales more efficiently than DDP with float16 autocast when scaling the number of GPU nodes.
Overall, Pytorch-FSDP mixed-precision is superior to DDP with autocast in virtually all cases we encountered.</p>
</div>
</section>
<section id="S5.SS0.SSS0.Px5" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Model distillation.</h5>

<div id="S5.SS0.SSS0.Px5.p1" class="ltx_para ltx_noindent">
<p id="S5.SS0.SSS0.Px5.p1.1" class="ltx_p">Most of our technical improvements to the training loop aim at improving the training of large models over large quantities of data.
For smaller models, we distill them from our largest model, the ViT-g, instead of training them from scratch.
Knowledge distillation <cite class="ltx_cite ltx_citemacro_citep">(Hinton et al., <a href="#bib.bib63" title="" class="ltx_ref">2014</a>)</cite> aims at reproducing the output of a large model with a smaller model by minimizing some distance between both outputs for a set of given inputs.
Since our objective function is a form of distillation from the teacher network to the student network, we leverage the same training loop with a few exceptions:
we use a larger model as a frozen teacher, keep a spare EMA of the student that we use as our final model, remove the masking and stochastic depth, and, apply the iBOT loss on the two global crops.
In our ablations, we observe that this approach achieves better performance than training from scratch, even for a ViT-L.
Our distillation method ends up close to the one described by <cite class="ltx_cite ltx_citemacro_cite">Duval et al. (<a href="#bib.bib40" title="" class="ltx_ref">2023</a>)</cite>, except we do not modify the loss terms for distillation and evaluate the EMA of the student.</p>
</div>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Ablation Studies</h2>

<div id="S6.p1" class="ltx_para ltx_noindent">
<p id="S6.p1.1" class="ltx_p">We present a set of ablations to empirically validate different components of our pipeline:
the technical modifications described in Sec. <a href="#S4" title="4 Discriminative Self-supervised Pre-training ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, the pretraining data and the impact of model distillation.
We consider various downstream tasks that are described in Sec. <a href="#S7" title="7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>.</p>
</div>
<section id="S6.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.1 </span>Improved Training Recipe</h3>

<div id="S6.SS1.p1" class="ltx_para ltx_noindent">
<p id="S6.SS1.p1.1" class="ltx_p">Our approach improves over the iBOT method by combining it with several existing components described in Sec. <a href="#S4" title="4 Discriminative Self-supervised Pre-training ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.
To evaluate their importance, we train multiple models where we successively add components to a baseline iBOT model.
We report the Top-1 accuracy on the validation set of ImageNet-1k with a k-NN and a linear probe in Table <a href="#S6.T1" title="Table 1 ‣ 6.1 Improved Training Recipe ‣ 6 Ablation Studies ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.
Generally, we observe that each component improves the performance on either k-NN or linear probing and even both in most cases.
Only LayerScale and Stochastic Depth incur a performance drop in linear probing but significantly improve the training stability in our experience.</p>
</div>
<figure id="S6.T1" class="ltx_table">
<table id="S6.T1.22" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S6.T1.22.23" class="ltx_tr">
<td id="S6.T1.22.23.1" class="ltx_td ltx_border_tt"></td>
<td id="S6.T1.22.23.2" class="ltx_td ltx_align_center ltx_border_tt">INet-1k k-NN</td>
<td id="S6.T1.22.23.3" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">INet-1k linear</td>
</tr>
<tr id="S6.T1.22.24" class="ltx_tr">
<td id="S6.T1.22.24.1" class="ltx_td ltx_align_left ltx_border_t">iBOT</td>
<td id="S6.T1.22.24.2" class="ltx_td ltx_align_left ltx_border_t">72.9</td>
<td id="S6.T1.22.24.3" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">82.3</td>
</tr>
<tr id="S6.T1.2.2" class="ltx_tr">
<td id="S6.T1.2.2.3" class="ltx_td ltx_align_left">+(our reproduction)</td>
<td id="S6.T1.1.1.1" class="ltx_td ltx_align_left">74.5 <math id="S6.T1.1.1.1.m1.1" class="ltx_Math" alttext="\uparrow 1.6" display="inline"><semantics id="S6.T1.1.1.1.m1.1a"><mrow id="S6.T1.1.1.1.m1.1.1" xref="S6.T1.1.1.1.m1.1.1.cmml"><mi id="S6.T1.1.1.1.m1.1.1.2" xref="S6.T1.1.1.1.m1.1.1.2.cmml"></mi><mo mathcolor="#408C40" mathsize="80%" stretchy="false" id="S6.T1.1.1.1.m1.1.1.1" xref="S6.T1.1.1.1.m1.1.1.1.cmml">↑</mo><mn mathcolor="#408C40" mathsize="80%" id="S6.T1.1.1.1.m1.1.1.3" xref="S6.T1.1.1.1.m1.1.1.3.cmml">1.6</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.1.1.1.m1.1b"><apply id="S6.T1.1.1.1.m1.1.1.cmml" xref="S6.T1.1.1.1.m1.1.1"><ci id="S6.T1.1.1.1.m1.1.1.1.cmml" xref="S6.T1.1.1.1.m1.1.1.1">↑</ci><csymbol cd="latexml" id="S6.T1.1.1.1.m1.1.1.2.cmml" xref="S6.T1.1.1.1.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.1.1.1.m1.1.1.3.cmml" xref="S6.T1.1.1.1.m1.1.1.3">1.6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.1.1.1.m1.1c">\uparrow 1.6</annotation></semantics></math>
</td>
<td id="S6.T1.2.2.2" class="ltx_td ltx_nopad_r ltx_align_left">83.2 <math id="S6.T1.2.2.2.m1.1" class="ltx_Math" alttext="\uparrow 0.9" display="inline"><semantics id="S6.T1.2.2.2.m1.1a"><mrow id="S6.T1.2.2.2.m1.1.1" xref="S6.T1.2.2.2.m1.1.1.cmml"><mi id="S6.T1.2.2.2.m1.1.1.2" xref="S6.T1.2.2.2.m1.1.1.2.cmml"></mi><mo mathcolor="#408C40" mathsize="80%" stretchy="false" id="S6.T1.2.2.2.m1.1.1.1" xref="S6.T1.2.2.2.m1.1.1.1.cmml">↑</mo><mn mathcolor="#408C40" mathsize="80%" id="S6.T1.2.2.2.m1.1.1.3" xref="S6.T1.2.2.2.m1.1.1.3.cmml">0.9</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.2.2.2.m1.1b"><apply id="S6.T1.2.2.2.m1.1.1.cmml" xref="S6.T1.2.2.2.m1.1.1"><ci id="S6.T1.2.2.2.m1.1.1.1.cmml" xref="S6.T1.2.2.2.m1.1.1.1">↑</ci><csymbol cd="latexml" id="S6.T1.2.2.2.m1.1.1.2.cmml" xref="S6.T1.2.2.2.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.2.2.2.m1.1.1.3.cmml" xref="S6.T1.2.2.2.m1.1.1.3">0.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.2.2.2.m1.1c">\uparrow 0.9</annotation></semantics></math>
</td>
</tr>
<tr id="S6.T1.4.4" class="ltx_tr">
<td id="S6.T1.4.4.3" class="ltx_td ltx_align_left">+LayerScale, Stochastic Depth</td>
<td id="S6.T1.3.3.1" class="ltx_td ltx_align_left">75.4 <math id="S6.T1.3.3.1.m1.1" class="ltx_Math" alttext="\uparrow 0.9" display="inline"><semantics id="S6.T1.3.3.1.m1.1a"><mrow id="S6.T1.3.3.1.m1.1.1" xref="S6.T1.3.3.1.m1.1.1.cmml"><mi id="S6.T1.3.3.1.m1.1.1.2" xref="S6.T1.3.3.1.m1.1.1.2.cmml"></mi><mo mathcolor="#408C40" mathsize="80%" stretchy="false" id="S6.T1.3.3.1.m1.1.1.1" xref="S6.T1.3.3.1.m1.1.1.1.cmml">↑</mo><mn mathcolor="#408C40" mathsize="80%" id="S6.T1.3.3.1.m1.1.1.3" xref="S6.T1.3.3.1.m1.1.1.3.cmml">0.9</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.3.3.1.m1.1b"><apply id="S6.T1.3.3.1.m1.1.1.cmml" xref="S6.T1.3.3.1.m1.1.1"><ci id="S6.T1.3.3.1.m1.1.1.1.cmml" xref="S6.T1.3.3.1.m1.1.1.1">↑</ci><csymbol cd="latexml" id="S6.T1.3.3.1.m1.1.1.2.cmml" xref="S6.T1.3.3.1.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.3.3.1.m1.1.1.3.cmml" xref="S6.T1.3.3.1.m1.1.1.3">0.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.3.3.1.m1.1c">\uparrow 0.9</annotation></semantics></math>
</td>
<td id="S6.T1.4.4.2" class="ltx_td ltx_nopad_r ltx_align_left">82.0 <math id="S6.T1.4.4.2.m1.1" class="ltx_Math" alttext="\downarrow 1.2" display="inline"><semantics id="S6.T1.4.4.2.m1.1a"><mrow id="S6.T1.4.4.2.m1.1.1" xref="S6.T1.4.4.2.m1.1.1.cmml"><mi id="S6.T1.4.4.2.m1.1.1.2" xref="S6.T1.4.4.2.m1.1.1.2.cmml"></mi><mo mathcolor="#8C4040" mathsize="80%" stretchy="false" id="S6.T1.4.4.2.m1.1.1.1" xref="S6.T1.4.4.2.m1.1.1.1.cmml">↓</mo><mn mathcolor="#8C4040" mathsize="80%" id="S6.T1.4.4.2.m1.1.1.3" xref="S6.T1.4.4.2.m1.1.1.3.cmml">1.2</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.4.4.2.m1.1b"><apply id="S6.T1.4.4.2.m1.1.1.cmml" xref="S6.T1.4.4.2.m1.1.1"><ci id="S6.T1.4.4.2.m1.1.1.1.cmml" xref="S6.T1.4.4.2.m1.1.1.1">↓</ci><csymbol cd="latexml" id="S6.T1.4.4.2.m1.1.1.2.cmml" xref="S6.T1.4.4.2.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.4.4.2.m1.1.1.3.cmml" xref="S6.T1.4.4.2.m1.1.1.3">1.2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.4.4.2.m1.1c">\downarrow 1.2</annotation></semantics></math>
</td>
</tr>
<tr id="S6.T1.6.6" class="ltx_tr">
<td id="S6.T1.6.6.3" class="ltx_td ltx_align_left">+128k prototypes</td>
<td id="S6.T1.5.5.1" class="ltx_td ltx_align_left">76.6 <math id="S6.T1.5.5.1.m1.1" class="ltx_Math" alttext="\uparrow 1.2" display="inline"><semantics id="S6.T1.5.5.1.m1.1a"><mrow id="S6.T1.5.5.1.m1.1.1" xref="S6.T1.5.5.1.m1.1.1.cmml"><mi id="S6.T1.5.5.1.m1.1.1.2" xref="S6.T1.5.5.1.m1.1.1.2.cmml"></mi><mo mathcolor="#408C40" mathsize="80%" stretchy="false" id="S6.T1.5.5.1.m1.1.1.1" xref="S6.T1.5.5.1.m1.1.1.1.cmml">↑</mo><mn mathcolor="#408C40" mathsize="80%" id="S6.T1.5.5.1.m1.1.1.3" xref="S6.T1.5.5.1.m1.1.1.3.cmml">1.2</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.5.5.1.m1.1b"><apply id="S6.T1.5.5.1.m1.1.1.cmml" xref="S6.T1.5.5.1.m1.1.1"><ci id="S6.T1.5.5.1.m1.1.1.1.cmml" xref="S6.T1.5.5.1.m1.1.1.1">↑</ci><csymbol cd="latexml" id="S6.T1.5.5.1.m1.1.1.2.cmml" xref="S6.T1.5.5.1.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.5.5.1.m1.1.1.3.cmml" xref="S6.T1.5.5.1.m1.1.1.3">1.2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.5.5.1.m1.1c">\uparrow 1.2</annotation></semantics></math>
</td>
<td id="S6.T1.6.6.2" class="ltx_td ltx_nopad_r ltx_align_left">81.9 <math id="S6.T1.6.6.2.m1.1" class="ltx_Math" alttext="\downarrow 0.1" display="inline"><semantics id="S6.T1.6.6.2.m1.1a"><mrow id="S6.T1.6.6.2.m1.1.1" xref="S6.T1.6.6.2.m1.1.1.cmml"><mi id="S6.T1.6.6.2.m1.1.1.2" xref="S6.T1.6.6.2.m1.1.1.2.cmml"></mi><mo mathcolor="#8C4040" mathsize="80%" stretchy="false" id="S6.T1.6.6.2.m1.1.1.1" xref="S6.T1.6.6.2.m1.1.1.1.cmml">↓</mo><mn mathcolor="#8C4040" mathsize="80%" id="S6.T1.6.6.2.m1.1.1.3" xref="S6.T1.6.6.2.m1.1.1.3.cmml">0.1</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.6.6.2.m1.1b"><apply id="S6.T1.6.6.2.m1.1.1.cmml" xref="S6.T1.6.6.2.m1.1.1"><ci id="S6.T1.6.6.2.m1.1.1.1.cmml" xref="S6.T1.6.6.2.m1.1.1.1">↓</ci><csymbol cd="latexml" id="S6.T1.6.6.2.m1.1.1.2.cmml" xref="S6.T1.6.6.2.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.6.6.2.m1.1.1.3.cmml" xref="S6.T1.6.6.2.m1.1.1.3">0.1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.6.6.2.m1.1c">\downarrow 0.1</annotation></semantics></math>
</td>
</tr>
<tr id="S6.T1.8.8" class="ltx_tr">
<td id="S6.T1.8.8.3" class="ltx_td ltx_align_left">+KoLeo</td>
<td id="S6.T1.7.7.1" class="ltx_td ltx_align_left">78.9 <math id="S6.T1.7.7.1.m1.1" class="ltx_Math" alttext="\uparrow 2.3" display="inline"><semantics id="S6.T1.7.7.1.m1.1a"><mrow id="S6.T1.7.7.1.m1.1.1" xref="S6.T1.7.7.1.m1.1.1.cmml"><mi id="S6.T1.7.7.1.m1.1.1.2" xref="S6.T1.7.7.1.m1.1.1.2.cmml"></mi><mo mathcolor="#408C40" mathsize="80%" stretchy="false" id="S6.T1.7.7.1.m1.1.1.1" xref="S6.T1.7.7.1.m1.1.1.1.cmml">↑</mo><mn mathcolor="#408C40" mathsize="80%" id="S6.T1.7.7.1.m1.1.1.3" xref="S6.T1.7.7.1.m1.1.1.3.cmml">2.3</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.7.7.1.m1.1b"><apply id="S6.T1.7.7.1.m1.1.1.cmml" xref="S6.T1.7.7.1.m1.1.1"><ci id="S6.T1.7.7.1.m1.1.1.1.cmml" xref="S6.T1.7.7.1.m1.1.1.1">↑</ci><csymbol cd="latexml" id="S6.T1.7.7.1.m1.1.1.2.cmml" xref="S6.T1.7.7.1.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.7.7.1.m1.1.1.3.cmml" xref="S6.T1.7.7.1.m1.1.1.3">2.3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.7.7.1.m1.1c">\uparrow 2.3</annotation></semantics></math>
</td>
<td id="S6.T1.8.8.2" class="ltx_td ltx_nopad_r ltx_align_left">82.5 <math id="S6.T1.8.8.2.m1.1" class="ltx_Math" alttext="\uparrow 0.6" display="inline"><semantics id="S6.T1.8.8.2.m1.1a"><mrow id="S6.T1.8.8.2.m1.1.1" xref="S6.T1.8.8.2.m1.1.1.cmml"><mi id="S6.T1.8.8.2.m1.1.1.2" xref="S6.T1.8.8.2.m1.1.1.2.cmml"></mi><mo mathcolor="#408C40" mathsize="80%" stretchy="false" id="S6.T1.8.8.2.m1.1.1.1" xref="S6.T1.8.8.2.m1.1.1.1.cmml">↑</mo><mn mathcolor="#408C40" mathsize="80%" id="S6.T1.8.8.2.m1.1.1.3" xref="S6.T1.8.8.2.m1.1.1.3.cmml">0.6</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.8.8.2.m1.1b"><apply id="S6.T1.8.8.2.m1.1.1.cmml" xref="S6.T1.8.8.2.m1.1.1"><ci id="S6.T1.8.8.2.m1.1.1.1.cmml" xref="S6.T1.8.8.2.m1.1.1.1">↑</ci><csymbol cd="latexml" id="S6.T1.8.8.2.m1.1.1.2.cmml" xref="S6.T1.8.8.2.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.8.8.2.m1.1.1.3.cmml" xref="S6.T1.8.8.2.m1.1.1.3">0.6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.8.8.2.m1.1c">\uparrow 0.6</annotation></semantics></math>
</td>
</tr>
<tr id="S6.T1.10.10" class="ltx_tr">
<td id="S6.T1.10.10.3" class="ltx_td ltx_align_left">+SwiGLU FFN</td>
<td id="S6.T1.9.9.1" class="ltx_td ltx_align_left">78.7 <math id="S6.T1.9.9.1.m1.1" class="ltx_Math" alttext="\downarrow 0.2" display="inline"><semantics id="S6.T1.9.9.1.m1.1a"><mrow id="S6.T1.9.9.1.m1.1.1" xref="S6.T1.9.9.1.m1.1.1.cmml"><mi id="S6.T1.9.9.1.m1.1.1.2" xref="S6.T1.9.9.1.m1.1.1.2.cmml"></mi><mo mathcolor="#8C4040" mathsize="80%" stretchy="false" id="S6.T1.9.9.1.m1.1.1.1" xref="S6.T1.9.9.1.m1.1.1.1.cmml">↓</mo><mn mathcolor="#8C4040" mathsize="80%" id="S6.T1.9.9.1.m1.1.1.3" xref="S6.T1.9.9.1.m1.1.1.3.cmml">0.2</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.9.9.1.m1.1b"><apply id="S6.T1.9.9.1.m1.1.1.cmml" xref="S6.T1.9.9.1.m1.1.1"><ci id="S6.T1.9.9.1.m1.1.1.1.cmml" xref="S6.T1.9.9.1.m1.1.1.1">↓</ci><csymbol cd="latexml" id="S6.T1.9.9.1.m1.1.1.2.cmml" xref="S6.T1.9.9.1.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.9.9.1.m1.1.1.3.cmml" xref="S6.T1.9.9.1.m1.1.1.3">0.2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.9.9.1.m1.1c">\downarrow 0.2</annotation></semantics></math>
</td>
<td id="S6.T1.10.10.2" class="ltx_td ltx_nopad_r ltx_align_left">83.1 <math id="S6.T1.10.10.2.m1.1" class="ltx_Math" alttext="\uparrow 0.6" display="inline"><semantics id="S6.T1.10.10.2.m1.1a"><mrow id="S6.T1.10.10.2.m1.1.1" xref="S6.T1.10.10.2.m1.1.1.cmml"><mi id="S6.T1.10.10.2.m1.1.1.2" xref="S6.T1.10.10.2.m1.1.1.2.cmml"></mi><mo mathcolor="#408C40" mathsize="80%" stretchy="false" id="S6.T1.10.10.2.m1.1.1.1" xref="S6.T1.10.10.2.m1.1.1.1.cmml">↑</mo><mn mathcolor="#408C40" mathsize="80%" id="S6.T1.10.10.2.m1.1.1.3" xref="S6.T1.10.10.2.m1.1.1.3.cmml">0.6</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.10.10.2.m1.1b"><apply id="S6.T1.10.10.2.m1.1.1.cmml" xref="S6.T1.10.10.2.m1.1.1"><ci id="S6.T1.10.10.2.m1.1.1.1.cmml" xref="S6.T1.10.10.2.m1.1.1.1">↑</ci><csymbol cd="latexml" id="S6.T1.10.10.2.m1.1.1.2.cmml" xref="S6.T1.10.10.2.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.10.10.2.m1.1.1.3.cmml" xref="S6.T1.10.10.2.m1.1.1.3">0.6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.10.10.2.m1.1c">\uparrow 0.6</annotation></semantics></math>
</td>
</tr>
<tr id="S6.T1.12.12" class="ltx_tr">
<td id="S6.T1.12.12.3" class="ltx_td ltx_align_left">+Patch size 14</td>
<td id="S6.T1.11.11.1" class="ltx_td ltx_align_left">78.9 <math id="S6.T1.11.11.1.m1.1" class="ltx_Math" alttext="\uparrow 0.2" display="inline"><semantics id="S6.T1.11.11.1.m1.1a"><mrow id="S6.T1.11.11.1.m1.1.1" xref="S6.T1.11.11.1.m1.1.1.cmml"><mi id="S6.T1.11.11.1.m1.1.1.2" xref="S6.T1.11.11.1.m1.1.1.2.cmml"></mi><mo mathcolor="#408C40" mathsize="80%" stretchy="false" id="S6.T1.11.11.1.m1.1.1.1" xref="S6.T1.11.11.1.m1.1.1.1.cmml">↑</mo><mn mathcolor="#408C40" mathsize="80%" id="S6.T1.11.11.1.m1.1.1.3" xref="S6.T1.11.11.1.m1.1.1.3.cmml">0.2</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.11.11.1.m1.1b"><apply id="S6.T1.11.11.1.m1.1.1.cmml" xref="S6.T1.11.11.1.m1.1.1"><ci id="S6.T1.11.11.1.m1.1.1.1.cmml" xref="S6.T1.11.11.1.m1.1.1.1">↑</ci><csymbol cd="latexml" id="S6.T1.11.11.1.m1.1.1.2.cmml" xref="S6.T1.11.11.1.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.11.11.1.m1.1.1.3.cmml" xref="S6.T1.11.11.1.m1.1.1.3">0.2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.11.11.1.m1.1c">\uparrow 0.2</annotation></semantics></math>
</td>
<td id="S6.T1.12.12.2" class="ltx_td ltx_nopad_r ltx_align_left">83.5 <math id="S6.T1.12.12.2.m1.1" class="ltx_Math" alttext="\uparrow 0.4" display="inline"><semantics id="S6.T1.12.12.2.m1.1a"><mrow id="S6.T1.12.12.2.m1.1.1" xref="S6.T1.12.12.2.m1.1.1.cmml"><mi id="S6.T1.12.12.2.m1.1.1.2" xref="S6.T1.12.12.2.m1.1.1.2.cmml"></mi><mo mathcolor="#408C40" mathsize="80%" stretchy="false" id="S6.T1.12.12.2.m1.1.1.1" xref="S6.T1.12.12.2.m1.1.1.1.cmml">↑</mo><mn mathcolor="#408C40" mathsize="80%" id="S6.T1.12.12.2.m1.1.1.3" xref="S6.T1.12.12.2.m1.1.1.3.cmml">0.4</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.12.12.2.m1.1b"><apply id="S6.T1.12.12.2.m1.1.1.cmml" xref="S6.T1.12.12.2.m1.1.1"><ci id="S6.T1.12.12.2.m1.1.1.1.cmml" xref="S6.T1.12.12.2.m1.1.1.1">↑</ci><csymbol cd="latexml" id="S6.T1.12.12.2.m1.1.1.2.cmml" xref="S6.T1.12.12.2.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.12.12.2.m1.1.1.3.cmml" xref="S6.T1.12.12.2.m1.1.1.3">0.4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.12.12.2.m1.1c">\uparrow 0.4</annotation></semantics></math>
</td>
</tr>
<tr id="S6.T1.14.14" class="ltx_tr">
<td id="S6.T1.14.14.3" class="ltx_td ltx_align_left">+Teacher momentum 0.994</td>
<td id="S6.T1.13.13.1" class="ltx_td ltx_align_left">79.4 <math id="S6.T1.13.13.1.m1.1" class="ltx_Math" alttext="\uparrow 0.5" display="inline"><semantics id="S6.T1.13.13.1.m1.1a"><mrow id="S6.T1.13.13.1.m1.1.1" xref="S6.T1.13.13.1.m1.1.1.cmml"><mi id="S6.T1.13.13.1.m1.1.1.2" xref="S6.T1.13.13.1.m1.1.1.2.cmml"></mi><mo mathcolor="#408C40" mathsize="80%" stretchy="false" id="S6.T1.13.13.1.m1.1.1.1" xref="S6.T1.13.13.1.m1.1.1.1.cmml">↑</mo><mn mathcolor="#408C40" mathsize="80%" id="S6.T1.13.13.1.m1.1.1.3" xref="S6.T1.13.13.1.m1.1.1.3.cmml">0.5</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.13.13.1.m1.1b"><apply id="S6.T1.13.13.1.m1.1.1.cmml" xref="S6.T1.13.13.1.m1.1.1"><ci id="S6.T1.13.13.1.m1.1.1.1.cmml" xref="S6.T1.13.13.1.m1.1.1.1">↑</ci><csymbol cd="latexml" id="S6.T1.13.13.1.m1.1.1.2.cmml" xref="S6.T1.13.13.1.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.13.13.1.m1.1.1.3.cmml" xref="S6.T1.13.13.1.m1.1.1.3">0.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.13.13.1.m1.1c">\uparrow 0.5</annotation></semantics></math>
</td>
<td id="S6.T1.14.14.2" class="ltx_td ltx_nopad_r ltx_align_left">83.6 <math id="S6.T1.14.14.2.m1.1" class="ltx_Math" alttext="\uparrow 0.1" display="inline"><semantics id="S6.T1.14.14.2.m1.1a"><mrow id="S6.T1.14.14.2.m1.1.1" xref="S6.T1.14.14.2.m1.1.1.cmml"><mi id="S6.T1.14.14.2.m1.1.1.2" xref="S6.T1.14.14.2.m1.1.1.2.cmml"></mi><mo mathcolor="#408C40" mathsize="80%" stretchy="false" id="S6.T1.14.14.2.m1.1.1.1" xref="S6.T1.14.14.2.m1.1.1.1.cmml">↑</mo><mn mathcolor="#408C40" mathsize="80%" id="S6.T1.14.14.2.m1.1.1.3" xref="S6.T1.14.14.2.m1.1.1.3.cmml">0.1</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.14.14.2.m1.1b"><apply id="S6.T1.14.14.2.m1.1.1.cmml" xref="S6.T1.14.14.2.m1.1.1"><ci id="S6.T1.14.14.2.m1.1.1.1.cmml" xref="S6.T1.14.14.2.m1.1.1.1">↑</ci><csymbol cd="latexml" id="S6.T1.14.14.2.m1.1.1.2.cmml" xref="S6.T1.14.14.2.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.14.14.2.m1.1.1.3.cmml" xref="S6.T1.14.14.2.m1.1.1.3">0.1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.14.14.2.m1.1c">\uparrow 0.1</annotation></semantics></math>
</td>
</tr>
<tr id="S6.T1.16.16" class="ltx_tr">
<td id="S6.T1.16.16.3" class="ltx_td ltx_align_left">+Tweak warmup schedules</td>
<td id="S6.T1.15.15.1" class="ltx_td ltx_align_left">80.5 <math id="S6.T1.15.15.1.m1.1" class="ltx_Math" alttext="\uparrow 1.1" display="inline"><semantics id="S6.T1.15.15.1.m1.1a"><mrow id="S6.T1.15.15.1.m1.1.1" xref="S6.T1.15.15.1.m1.1.1.cmml"><mi id="S6.T1.15.15.1.m1.1.1.2" xref="S6.T1.15.15.1.m1.1.1.2.cmml"></mi><mo mathcolor="#408C40" mathsize="80%" stretchy="false" id="S6.T1.15.15.1.m1.1.1.1" xref="S6.T1.15.15.1.m1.1.1.1.cmml">↑</mo><mn mathcolor="#408C40" mathsize="80%" id="S6.T1.15.15.1.m1.1.1.3" xref="S6.T1.15.15.1.m1.1.1.3.cmml">1.1</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.15.15.1.m1.1b"><apply id="S6.T1.15.15.1.m1.1.1.cmml" xref="S6.T1.15.15.1.m1.1.1"><ci id="S6.T1.15.15.1.m1.1.1.1.cmml" xref="S6.T1.15.15.1.m1.1.1.1">↑</ci><csymbol cd="latexml" id="S6.T1.15.15.1.m1.1.1.2.cmml" xref="S6.T1.15.15.1.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.15.15.1.m1.1.1.3.cmml" xref="S6.T1.15.15.1.m1.1.1.3">1.1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.15.15.1.m1.1c">\uparrow 1.1</annotation></semantics></math>
</td>
<td id="S6.T1.16.16.2" class="ltx_td ltx_nopad_r ltx_align_left">83.8 <math id="S6.T1.16.16.2.m1.1" class="ltx_Math" alttext="\uparrow 0.2" display="inline"><semantics id="S6.T1.16.16.2.m1.1a"><mrow id="S6.T1.16.16.2.m1.1.1" xref="S6.T1.16.16.2.m1.1.1.cmml"><mi id="S6.T1.16.16.2.m1.1.1.2" xref="S6.T1.16.16.2.m1.1.1.2.cmml"></mi><mo mathcolor="#408C40" mathsize="80%" stretchy="false" id="S6.T1.16.16.2.m1.1.1.1" xref="S6.T1.16.16.2.m1.1.1.1.cmml">↑</mo><mn mathcolor="#408C40" mathsize="80%" id="S6.T1.16.16.2.m1.1.1.3" xref="S6.T1.16.16.2.m1.1.1.3.cmml">0.2</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.16.16.2.m1.1b"><apply id="S6.T1.16.16.2.m1.1.1.cmml" xref="S6.T1.16.16.2.m1.1.1"><ci id="S6.T1.16.16.2.m1.1.1.1.cmml" xref="S6.T1.16.16.2.m1.1.1.1">↑</ci><csymbol cd="latexml" id="S6.T1.16.16.2.m1.1.1.2.cmml" xref="S6.T1.16.16.2.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.16.16.2.m1.1.1.3.cmml" xref="S6.T1.16.16.2.m1.1.1.3">0.2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.16.16.2.m1.1c">\uparrow 0.2</annotation></semantics></math>
</td>
</tr>
<tr id="S6.T1.18.18" class="ltx_tr">
<td id="S6.T1.18.18.3" class="ltx_td ltx_align_left">+Batch size 3k</td>
<td id="S6.T1.17.17.1" class="ltx_td ltx_align_left">81.7 <math id="S6.T1.17.17.1.m1.1" class="ltx_Math" alttext="\uparrow 1.2" display="inline"><semantics id="S6.T1.17.17.1.m1.1a"><mrow id="S6.T1.17.17.1.m1.1.1" xref="S6.T1.17.17.1.m1.1.1.cmml"><mi id="S6.T1.17.17.1.m1.1.1.2" xref="S6.T1.17.17.1.m1.1.1.2.cmml"></mi><mo mathcolor="#408C40" mathsize="80%" stretchy="false" id="S6.T1.17.17.1.m1.1.1.1" xref="S6.T1.17.17.1.m1.1.1.1.cmml">↑</mo><mn mathcolor="#408C40" mathsize="80%" id="S6.T1.17.17.1.m1.1.1.3" xref="S6.T1.17.17.1.m1.1.1.3.cmml">1.2</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.17.17.1.m1.1b"><apply id="S6.T1.17.17.1.m1.1.1.cmml" xref="S6.T1.17.17.1.m1.1.1"><ci id="S6.T1.17.17.1.m1.1.1.1.cmml" xref="S6.T1.17.17.1.m1.1.1.1">↑</ci><csymbol cd="latexml" id="S6.T1.17.17.1.m1.1.1.2.cmml" xref="S6.T1.17.17.1.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.17.17.1.m1.1.1.3.cmml" xref="S6.T1.17.17.1.m1.1.1.3">1.2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.17.17.1.m1.1c">\uparrow 1.2</annotation></semantics></math>
</td>
<td id="S6.T1.18.18.2" class="ltx_td ltx_nopad_r ltx_align_left">84.7 <math id="S6.T1.18.18.2.m1.1" class="ltx_Math" alttext="\uparrow 0.9" display="inline"><semantics id="S6.T1.18.18.2.m1.1a"><mrow id="S6.T1.18.18.2.m1.1.1" xref="S6.T1.18.18.2.m1.1.1.cmml"><mi id="S6.T1.18.18.2.m1.1.1.2" xref="S6.T1.18.18.2.m1.1.1.2.cmml"></mi><mo mathcolor="#408C40" mathsize="80%" stretchy="false" id="S6.T1.18.18.2.m1.1.1.1" xref="S6.T1.18.18.2.m1.1.1.1.cmml">↑</mo><mn mathcolor="#408C40" mathsize="80%" id="S6.T1.18.18.2.m1.1.1.3" xref="S6.T1.18.18.2.m1.1.1.3.cmml">0.9</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.18.18.2.m1.1b"><apply id="S6.T1.18.18.2.m1.1.1.cmml" xref="S6.T1.18.18.2.m1.1.1"><ci id="S6.T1.18.18.2.m1.1.1.1.cmml" xref="S6.T1.18.18.2.m1.1.1.1">↑</ci><csymbol cd="latexml" id="S6.T1.18.18.2.m1.1.1.2.cmml" xref="S6.T1.18.18.2.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.18.18.2.m1.1.1.3.cmml" xref="S6.T1.18.18.2.m1.1.1.3">0.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.18.18.2.m1.1c">\uparrow 0.9</annotation></semantics></math>
</td>
</tr>
<tr id="S6.T1.20.20" class="ltx_tr">
<td id="S6.T1.20.20.3" class="ltx_td ltx_align_left">+Sinkhorn-Knopp</td>
<td id="S6.T1.19.19.1" class="ltx_td ltx_align_left">81.7 <math id="S6.T1.19.19.1.m1.1" class="ltx_Math" alttext="=" display="inline"><semantics id="S6.T1.19.19.1.m1.1a"><mo mathsize="80%" id="S6.T1.19.19.1.m1.1.1" xref="S6.T1.19.19.1.m1.1.1.cmml">=</mo><annotation-xml encoding="MathML-Content" id="S6.T1.19.19.1.m1.1b"><eq id="S6.T1.19.19.1.m1.1.1.cmml" xref="S6.T1.19.19.1.m1.1.1"></eq></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.19.19.1.m1.1c">=</annotation></semantics></math>
</td>
<td id="S6.T1.20.20.2" class="ltx_td ltx_nopad_r ltx_align_left">84.7 <math id="S6.T1.20.20.2.m1.1" class="ltx_Math" alttext="=" display="inline"><semantics id="S6.T1.20.20.2.m1.1a"><mo mathsize="80%" id="S6.T1.20.20.2.m1.1.1" xref="S6.T1.20.20.2.m1.1.1.cmml">=</mo><annotation-xml encoding="MathML-Content" id="S6.T1.20.20.2.m1.1b"><eq id="S6.T1.20.20.2.m1.1.1.cmml" xref="S6.T1.20.20.2.m1.1.1"></eq></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.20.20.2.m1.1c">=</annotation></semantics></math>
</td>
</tr>
<tr id="S6.T1.22.22" class="ltx_tr">
<td id="S6.T1.22.22.3" class="ltx_td ltx_align_left ltx_border_bb">+Untying heads = DINOv2</td>
<td id="S6.T1.21.21.1" class="ltx_td ltx_align_left ltx_border_bb">82.0 <math id="S6.T1.21.21.1.m1.1" class="ltx_Math" alttext="\uparrow 0.3" display="inline"><semantics id="S6.T1.21.21.1.m1.1a"><mrow id="S6.T1.21.21.1.m1.1.1" xref="S6.T1.21.21.1.m1.1.1.cmml"><mi id="S6.T1.21.21.1.m1.1.1.2" xref="S6.T1.21.21.1.m1.1.1.2.cmml"></mi><mo mathcolor="#408C40" mathsize="80%" stretchy="false" id="S6.T1.21.21.1.m1.1.1.1" xref="S6.T1.21.21.1.m1.1.1.1.cmml">↑</mo><mn mathcolor="#408C40" mathsize="80%" id="S6.T1.21.21.1.m1.1.1.3" xref="S6.T1.21.21.1.m1.1.1.3.cmml">0.3</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.21.21.1.m1.1b"><apply id="S6.T1.21.21.1.m1.1.1.cmml" xref="S6.T1.21.21.1.m1.1.1"><ci id="S6.T1.21.21.1.m1.1.1.1.cmml" xref="S6.T1.21.21.1.m1.1.1.1">↑</ci><csymbol cd="latexml" id="S6.T1.21.21.1.m1.1.1.2.cmml" xref="S6.T1.21.21.1.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.21.21.1.m1.1.1.3.cmml" xref="S6.T1.21.21.1.m1.1.1.3">0.3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.21.21.1.m1.1c">\uparrow 0.3</annotation></semantics></math>
</td>
<td id="S6.T1.22.22.2" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">84.5 <math id="S6.T1.22.22.2.m1.1" class="ltx_Math" alttext="\downarrow 0.2" display="inline"><semantics id="S6.T1.22.22.2.m1.1a"><mrow id="S6.T1.22.22.2.m1.1.1" xref="S6.T1.22.22.2.m1.1.1.cmml"><mi id="S6.T1.22.22.2.m1.1.1.2" xref="S6.T1.22.22.2.m1.1.1.2.cmml"></mi><mo mathcolor="#8C4040" mathsize="80%" stretchy="false" id="S6.T1.22.22.2.m1.1.1.1" xref="S6.T1.22.22.2.m1.1.1.1.cmml">↓</mo><mn mathcolor="#8C4040" mathsize="80%" id="S6.T1.22.22.2.m1.1.1.3" xref="S6.T1.22.22.2.m1.1.1.3.cmml">0.2</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T1.22.22.2.m1.1b"><apply id="S6.T1.22.22.2.m1.1.1.cmml" xref="S6.T1.22.22.2.m1.1.1"><ci id="S6.T1.22.22.2.m1.1.1.1.cmml" xref="S6.T1.22.22.2.m1.1.1.1">↓</ci><csymbol cd="latexml" id="S6.T1.22.22.2.m1.1.1.2.cmml" xref="S6.T1.22.22.2.m1.1.1.2">absent</csymbol><cn type="float" id="S6.T1.22.22.2.m1.1.1.3.cmml" xref="S6.T1.22.22.2.m1.1.1.3">0.2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.22.22.2.m1.1c">\downarrow 0.2</annotation></semantics></math>
</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S6.T1.27.2.1" class="ltx_text" style="font-size:90%;">Table 1</span>: </span><span id="S6.T1.24.1" class="ltx_text" style="font-size:90%;">
<span id="S6.T1.24.1.1" class="ltx_text ltx_font_bold">Ablation study of the training differences between iBOT and DINOv2.</span>
We optimize for k-NN performance, as in our experience, the linear probe performance is lower-bounded by the k-NN performance.
Some modifications, like LayerScale and a high Stochastic Depth (rate=<math id="S6.T1.24.1.m1.1" class="ltx_Math" alttext="0.4" display="inline"><semantics id="S6.T1.24.1.m1.1b"><mn id="S6.T1.24.1.m1.1.1" xref="S6.T1.24.1.m1.1.1.cmml">0.4</mn><annotation-xml encoding="MathML-Content" id="S6.T1.24.1.m1.1c"><cn type="float" id="S6.T1.24.1.m1.1.1.cmml" xref="S6.T1.24.1.m1.1.1">0.4</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.24.1.m1.1d">0.4</annotation></semantics></math>), incur a decrease in linear probe performance, but have the benefits of increasing the stability of training by avoiding NaN loss values during training <cite class="ltx_cite ltx_citemacro_citep">(Touvron et al., <a href="#bib.bib114" title="" class="ltx_ref">2022</a>)</cite>.
Overall, these modifications allowed for the next set of improvements to be added. Experiments are run using the ViT-Large architecture on ImageNet-22k.
</span></figcaption>
</figure>
</section>
<section id="S6.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.2 </span>Pretraining Data Source</h3>

<div id="S6.SS2.p1" class="ltx_para ltx_noindent">
<p id="S6.SS2.p1.2" class="ltx_p">The quality of features is directly related to the quality of the pretraining data.
In this experiment, we probe the impact of LVD-142M compared to ImageNet-22k, a commonly used pretraining dataset, or using directly raw and uncurated data.
For the uncurated dataset, we randomly sample <math id="S6.SS2.p1.1.m1.1" class="ltx_Math" alttext="142" display="inline"><semantics id="S6.SS2.p1.1.m1.1a"><mn id="S6.SS2.p1.1.m1.1.1" xref="S6.SS2.p1.1.m1.1.1.cmml">142</mn><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.1.m1.1b"><cn type="integer" id="S6.SS2.p1.1.m1.1.1.cmml" xref="S6.SS2.p1.1.m1.1.1">142</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.1.m1.1c">142</annotation></semantics></math> million images from the same data source as LVD-142M.
We train a ViT-g/14 on each dataset for the same number of iterations.
We also include a variant of ImageNet-22k obtained by removing the synsets of ImageNet-1k (INet-22k <math id="S6.SS2.p1.2.m2.1" class="ltx_Math" alttext="\setminus" display="inline"><semantics id="S6.SS2.p1.2.m2.1a"><mo id="S6.SS2.p1.2.m2.1.1" xref="S6.SS2.p1.2.m2.1.1.cmml">∖</mo><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.2.m2.1b"><setdiff id="S6.SS2.p1.2.m2.1.1.cmml" xref="S6.SS2.p1.2.m2.1.1"></setdiff></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.2.m2.1c">\setminus</annotation></semantics></math> INet-1k) for completeness.
We report the comparisons in Table <a href="#S6.T2" title="Table 2 ‣ 6.2 Pretraining Data Source ‣ 6 Ablation Studies ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
<figure id="S6.T2" class="ltx_table">
<table id="S6.T2.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S6.T2.1.2" class="ltx_tr">
<td id="S6.T2.1.2.1" class="ltx_td ltx_align_left ltx_border_tt">Training Data</td>
<td id="S6.T2.1.2.2" class="ltx_td ltx_border_tt"></td>
<td id="S6.T2.1.2.3" class="ltx_td ltx_align_center ltx_border_tt">INet-1k</td>
<td id="S6.T2.1.2.4" class="ltx_td ltx_align_center ltx_border_tt">Im-A</td>
<td id="S6.T2.1.2.5" class="ltx_td ltx_align_center ltx_border_tt">ADE-20k</td>
<td id="S6.T2.1.2.6" class="ltx_td ltx_align_center ltx_border_tt">Oxford-M</td>
<td id="S6.T2.1.2.7" class="ltx_td ltx_align_center ltx_border_tt">iNat2018</td>
<td id="S6.T2.1.2.8" class="ltx_td ltx_align_center ltx_border_tt">iNat2021</td>
<td id="S6.T2.1.2.9" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_tt">Places205</td>
</tr>
<tr id="S6.T2.1.3" class="ltx_tr">
<td id="S6.T2.1.3.1" class="ltx_td ltx_align_left ltx_border_t">INet-22k</td>
<td id="S6.T2.1.3.2" class="ltx_td ltx_border_t"></td>
<td id="S6.T2.1.3.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T2.1.3.3.1" class="ltx_text ltx_font_bold">85.9</span></td>
<td id="S6.T2.1.3.4" class="ltx_td ltx_align_center ltx_border_t">73.5</td>
<td id="S6.T2.1.3.5" class="ltx_td ltx_align_center ltx_border_t">46.6</td>
<td id="S6.T2.1.3.6" class="ltx_td ltx_align_center ltx_border_t">62.5</td>
<td id="S6.T2.1.3.7" class="ltx_td ltx_align_center ltx_border_t">81.1</td>
<td id="S6.T2.1.3.8" class="ltx_td ltx_align_center ltx_border_t">85.6</td>
<td id="S6.T2.1.3.9" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">67.0</td>
</tr>
<tr id="S6.T2.1.1" class="ltx_tr">
<td id="S6.T2.1.1.1" class="ltx_td ltx_align_left">INet-22k <math id="S6.T2.1.1.1.m1.1" class="ltx_Math" alttext="\setminus" display="inline"><semantics id="S6.T2.1.1.1.m1.1a"><mo id="S6.T2.1.1.1.m1.1.1" xref="S6.T2.1.1.1.m1.1.1.cmml">∖</mo><annotation-xml encoding="MathML-Content" id="S6.T2.1.1.1.m1.1b"><setdiff id="S6.T2.1.1.1.m1.1.1.cmml" xref="S6.T2.1.1.1.m1.1.1"></setdiff></annotation-xml><annotation encoding="application/x-tex" id="S6.T2.1.1.1.m1.1c">\setminus</annotation></semantics></math> INet-1k</td>
<td id="S6.T2.1.1.2" class="ltx_td"></td>
<td id="S6.T2.1.1.3" class="ltx_td ltx_align_center">85.3</td>
<td id="S6.T2.1.1.4" class="ltx_td ltx_align_center">70.3</td>
<td id="S6.T2.1.1.5" class="ltx_td ltx_align_center">46.2</td>
<td id="S6.T2.1.1.6" class="ltx_td ltx_align_center">58.7</td>
<td id="S6.T2.1.1.7" class="ltx_td ltx_align_center">80.1</td>
<td id="S6.T2.1.1.8" class="ltx_td ltx_align_center">85.1</td>
<td id="S6.T2.1.1.9" class="ltx_td ltx_nopad_r ltx_align_center">66.5</td>
</tr>
<tr id="S6.T2.1.4" class="ltx_tr">
<td id="S6.T2.1.4.1" class="ltx_td ltx_align_left">Uncurated data</td>
<td id="S6.T2.1.4.2" class="ltx_td"></td>
<td id="S6.T2.1.4.3" class="ltx_td ltx_align_center">83.3</td>
<td id="S6.T2.1.4.4" class="ltx_td ltx_align_center">59.4</td>
<td id="S6.T2.1.4.5" class="ltx_td ltx_align_center">48.5</td>
<td id="S6.T2.1.4.6" class="ltx_td ltx_align_center">54.3</td>
<td id="S6.T2.1.4.7" class="ltx_td ltx_align_center">68.0</td>
<td id="S6.T2.1.4.8" class="ltx_td ltx_align_center">76.4</td>
<td id="S6.T2.1.4.9" class="ltx_td ltx_nopad_r ltx_align_center">67.2</td>
</tr>
<tr id="S6.T2.1.5" class="ltx_tr">
<td id="S6.T2.1.5.1" class="ltx_td ltx_align_left ltx_border_bb">LVD-142M</td>
<td id="S6.T2.1.5.2" class="ltx_td ltx_border_bb"></td>
<td id="S6.T2.1.5.3" class="ltx_td ltx_align_center ltx_border_bb">85.8</td>
<td id="S6.T2.1.5.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S6.T2.1.5.4.1" class="ltx_text ltx_font_bold">73.9</span></td>
<td id="S6.T2.1.5.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S6.T2.1.5.5.1" class="ltx_text ltx_font_bold">47.7</span></td>
<td id="S6.T2.1.5.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="S6.T2.1.5.6.1" class="ltx_text ltx_font_bold">64.6</span></td>
<td id="S6.T2.1.5.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="S6.T2.1.5.7.1" class="ltx_text ltx_font_bold">82.3</span></td>
<td id="S6.T2.1.5.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="S6.T2.1.5.8.1" class="ltx_text ltx_font_bold">86.4</span></td>
<td id="S6.T2.1.5.9" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S6.T2.1.5.9.1" class="ltx_text ltx_font_bold">67.6</span></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S6.T2.4.1.1" class="ltx_text" style="font-size:90%;">Table 2</span>: </span><span id="S6.T2.5.2" class="ltx_text" style="font-size:90%;">
<span id="S6.T2.5.2.1" class="ltx_text ltx_font_bold">Ablation of the source of pretraining data.</span>
We compare the INet-22k dataset that was used in iBOT to our dataset, LVD-142M.
Each model is trained for the same number of iterations, that is smaller than in our final run, without high-resolution adaptation.
Pretraining on LVD-142M maintains the performance over INet-1k while leading to models that perform better in other domains.
</span></figcaption>
</figure>
<div id="S6.SS2.p2" class="ltx_para ltx_noindent">
<p id="S6.SS2.p2.1" class="ltx_p">The most salient observation is that training on a curated set of images works better on most benchmarks than training on uncurated data.
This confirms the benefit of curating data, even in the case of self-supervised pretraining.
When compared with models trained on ImageNet-22k, training on LVD-142M is also superior on all the benchmarks but ImageNet-1k.
This confirms that training on a more diverse set of images improves the quality of the features in domains that are not covered by ImageNet-22k.
We also see that training on our curated data increases the performances on domains that are not used for the curation process (INaturalist 2018, 2021 and Places205), proving that scale and diversity can benefit unseen domains.
Overall, the conclusion of this ablation is that our dataset provides a good balance of different types of images that leads to the best performance overall.</p>
</div>
</section>
<section id="S6.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.3 </span>Model Size and Data</h3>

<div id="S6.SS3.p1" class="ltx_para ltx_noindent">
<p id="S6.SS3.p1.1" class="ltx_p">We quantify the importance of scaling data with the model size in Fig. <a href="#S6.F4" title="Figure 4 ‣ 6.3 Model Size and Data ‣ 6 Ablation Studies ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.
As the size of models grow, training on LVD-142M becomes more beneficial than training on ImageNet-22k.
For instance, a ViT-g trained on LVD-142M matches the performance on ImageNet-1k of a model trained on ImageNet-22k while significantly outperforming it on the other benchmarks.</p>
</div>
<figure id="S6.F4" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_many"><img src="/html/2304.07193/assets/x3.png" id="S6.F4.g1" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_square" width="45" height="45" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_many"><img src="/html/2304.07193/assets/x4.png" id="S6.F4.g2" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_square" width="45" height="45" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_many"><img src="/html/2304.07193/assets/x5.png" id="S6.F4.g3" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_square" width="45" height="45" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_many"><img src="/html/2304.07193/assets/x6.png" id="S6.F4.g4" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_square" width="45" height="45" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_many"><img src="/html/2304.07193/assets/x7.png" id="S6.F4.g5" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_square" width="45" height="45" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_many"><img src="/html/2304.07193/assets/x8.png" id="S6.F4.g6" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_square" width="45" height="45" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_many"><img src="/html/2304.07193/assets/x9.png" id="S6.F4.g7" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_square" width="45" height="45" alt="Refer to caption"></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S6.F4.3.1.1" class="ltx_text" style="font-size:90%;">Figure 4</span>: </span><span id="S6.F4.4.2" class="ltx_text" style="font-size:90%;">
<span id="S6.F4.4.2.1" class="ltx_text ltx_font_bold">Model scale versus data scale.</span>
Evolution of performance as a function of model size for two different pretraining datasets: ImageNet-22k (14M images) and LVD-142M (142M images).
The ViT-g trained on LVD-142M surpasses the ViT-g trained on ImageNet-22k on most benchmarks.
</span></figcaption>
</figure>
</section>
<section id="S6.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.4 </span>Loss Components</h3>

<div id="S6.SS4.p1" class="ltx_para ltx_noindent">
<p id="S6.SS4.p1.2" class="ltx_p">We validated the proposed technical improvements in Sec. <a href="#S6.SS1" title="6.1 Improved Training Recipe ‣ 6 Ablation Studies ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6.1</span></a> by adding them incrementally.
This section analyzes the performance hit observed if we ablate specific loss terms, starting from our best-performing model.
We ablate the importance of the KoLeo loss and the impact of the masked image modeling term.
For both, we report performance on ImageNet-1k using a linear classifier, ADE-20k segmentation using a linear classifier, and nearest-neighbor image retrieval on Oxford-M.
Table <a href="#S6.T3.st1" title="In Table 3 ‣ 6.4 Loss Components ‣ 6 Ablation Studies ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3(a)</span></a> shows the impact of using the KoLeo loss.
We see that the instance retrieval performance improves by more than <math id="S6.SS4.p1.1.m1.1" class="ltx_Math" alttext="8\%" display="inline"><semantics id="S6.SS4.p1.1.m1.1a"><mrow id="S6.SS4.p1.1.m1.1.1" xref="S6.SS4.p1.1.m1.1.1.cmml"><mn id="S6.SS4.p1.1.m1.1.1.2" xref="S6.SS4.p1.1.m1.1.1.2.cmml">8</mn><mo id="S6.SS4.p1.1.m1.1.1.1" xref="S6.SS4.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS4.p1.1.m1.1b"><apply id="S6.SS4.p1.1.m1.1.1.cmml" xref="S6.SS4.p1.1.m1.1.1"><csymbol cd="latexml" id="S6.SS4.p1.1.m1.1.1.1.cmml" xref="S6.SS4.p1.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S6.SS4.p1.1.m1.1.1.2.cmml" xref="S6.SS4.p1.1.m1.1.1.2">8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS4.p1.1.m1.1c">8\%</annotation></semantics></math>, confirming that this term helps spread features in the output space.
At the same time, the other metrics do not suffer from this regularization.
In Table <a href="#S6.T3.st2" title="In Table 3 ‣ 6.4 Loss Components ‣ 6 Ablation Studies ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3(b)</span></a>, we show the impact of using the masked image modeling term from iBOT.
This term is critical for dense prediction tasks, leading to almost <math id="S6.SS4.p1.2.m2.1" class="ltx_Math" alttext="3\%" display="inline"><semantics id="S6.SS4.p1.2.m2.1a"><mrow id="S6.SS4.p1.2.m2.1.1" xref="S6.SS4.p1.2.m2.1.1.cmml"><mn id="S6.SS4.p1.2.m2.1.1.2" xref="S6.SS4.p1.2.m2.1.1.2.cmml">3</mn><mo id="S6.SS4.p1.2.m2.1.1.1" xref="S6.SS4.p1.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS4.p1.2.m2.1b"><apply id="S6.SS4.p1.2.m2.1.1.cmml" xref="S6.SS4.p1.2.m2.1.1"><csymbol cd="latexml" id="S6.SS4.p1.2.m2.1.1.1.cmml" xref="S6.SS4.p1.2.m2.1.1.1">percent</csymbol><cn type="integer" id="S6.SS4.p1.2.m2.1.1.2.cmml" xref="S6.SS4.p1.2.m2.1.1.2">3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS4.p1.2.m2.1c">3\%</annotation></semantics></math> performance improvement.</p>
</div>
<figure id="S6.T3" class="ltx_table">
<div class="ltx_flex_figure ltx_flex_table">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S6.T3.st1" class="ltx_table ltx_figure_panel">
<div id="S6.T3.st1.2" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:460.4pt;height:104.8pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(111.5pt,-25.4pt) scale(1.94016126160198,1.94016126160198) ;">
<table id="S6.T3.st1.2.1" class="ltx_tabular ltx_align_middle">
<tr id="S6.T3.st1.2.1.1" class="ltx_tr">
<td id="S6.T3.st1.2.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">KoLeo</td>
<td id="S6.T3.st1.2.1.1.2" class="ltx_td ltx_border_tt"></td>
<td id="S6.T3.st1.2.1.1.3" class="ltx_td ltx_align_center ltx_border_tt">INet-1k</td>
<td id="S6.T3.st1.2.1.1.4" class="ltx_td ltx_align_center ltx_border_tt">Im-A</td>
<td id="S6.T3.st1.2.1.1.5" class="ltx_td ltx_align_center ltx_border_tt">ADE-20k</td>
<td id="S6.T3.st1.2.1.1.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_tt">Oxford-M</td>
</tr>
<tr id="S6.T3.st1.2.1.2" class="ltx_tr">
<td id="S6.T3.st1.2.1.2.1" class="ltx_td ltx_align_left ltx_border_t">✕</td>
<td id="S6.T3.st1.2.1.2.2" class="ltx_td ltx_border_t"></td>
<td id="S6.T3.st1.2.1.2.3" class="ltx_td ltx_align_center ltx_border_t">85.3</td>
<td id="S6.T3.st1.2.1.2.4" class="ltx_td ltx_align_center ltx_border_t">70.6</td>
<td id="S6.T3.st1.2.1.2.5" class="ltx_td ltx_align_center ltx_border_t">47.2</td>
<td id="S6.T3.st1.2.1.2.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">55.6</td>
</tr>
<tr id="S6.T3.st1.2.1.3" class="ltx_tr">
<td id="S6.T3.st1.2.1.3.1" class="ltx_td ltx_align_left ltx_border_bb">✓</td>
<td id="S6.T3.st1.2.1.3.2" class="ltx_td ltx_border_bb"></td>
<td id="S6.T3.st1.2.1.3.3" class="ltx_td ltx_align_center ltx_border_bb">85.8</td>
<td id="S6.T3.st1.2.1.3.4" class="ltx_td ltx_align_center ltx_border_bb">72.8</td>
<td id="S6.T3.st1.2.1.3.5" class="ltx_td ltx_align_center ltx_border_bb">47.1</td>
<td id="S6.T3.st1.2.1.3.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">63.9</td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S6.T3.st1.3.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="S6.T3.st1.4.2" class="ltx_text" style="font-size:90%;">Koleo loss</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S6.T3.st2" class="ltx_table ltx_figure_panel">
<div id="S6.T3.st2.2" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:460.4pt;height:107.7pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(114.8pt,-26.9pt) scale(1.99504736499464,1.99504736499464) ;">
<table id="S6.T3.st2.2.1" class="ltx_tabular ltx_align_middle">
<tr id="S6.T3.st2.2.1.1" class="ltx_tr">
<td id="S6.T3.st2.2.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">MIM</td>
<td id="S6.T3.st2.2.1.1.2" class="ltx_td ltx_border_tt"></td>
<td id="S6.T3.st2.2.1.1.3" class="ltx_td ltx_align_center ltx_border_tt">INet-1k</td>
<td id="S6.T3.st2.2.1.1.4" class="ltx_td ltx_align_center ltx_border_tt">Im-A</td>
<td id="S6.T3.st2.2.1.1.5" class="ltx_td ltx_align_center ltx_border_tt">ADE-20k</td>
<td id="S6.T3.st2.2.1.1.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_tt">Oxford-M</td>
</tr>
<tr id="S6.T3.st2.2.1.2" class="ltx_tr">
<td id="S6.T3.st2.2.1.2.1" class="ltx_td ltx_align_left ltx_border_t">✕</td>
<td id="S6.T3.st2.2.1.2.2" class="ltx_td ltx_border_t"></td>
<td id="S6.T3.st2.2.1.2.3" class="ltx_td ltx_align_center ltx_border_t">85.3</td>
<td id="S6.T3.st2.2.1.2.4" class="ltx_td ltx_align_center ltx_border_t">72.0</td>
<td id="S6.T3.st2.2.1.2.5" class="ltx_td ltx_align_center ltx_border_t">44.2</td>
<td id="S6.T3.st2.2.1.2.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">64.3</td>
</tr>
<tr id="S6.T3.st2.2.1.3" class="ltx_tr">
<td id="S6.T3.st2.2.1.3.1" class="ltx_td ltx_align_left ltx_border_bb">✓</td>
<td id="S6.T3.st2.2.1.3.2" class="ltx_td ltx_border_bb"></td>
<td id="S6.T3.st2.2.1.3.3" class="ltx_td ltx_align_center ltx_border_bb">85.8</td>
<td id="S6.T3.st2.2.1.3.4" class="ltx_td ltx_align_center ltx_border_bb">72.8</td>
<td id="S6.T3.st2.2.1.3.5" class="ltx_td ltx_align_center ltx_border_bb">47.1</td>
<td id="S6.T3.st2.2.1.3.6" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">63.9</td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S6.T3.st2.3.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="S6.T3.st2.4.2" class="ltx_text" style="font-size:90%;">MIM objective in iBOT</span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S6.T3.4.1.1" class="ltx_text" style="font-size:90%;">Table 3</span>: </span><span id="S6.T3.5.2" class="ltx_text" style="font-size:90%;">
<span id="S6.T3.5.2.1" class="ltx_text ltx_font_bold">(a)</span> Effect of the KoLeo loss term.
<span id="S6.T3.5.2.2" class="ltx_text ltx_font_bold">(b)</span> Effect of the iBOT Masked Image Modeling (MIM) loss term.
Evaluation performed on ImageNet-{1k,A} (classification with linear probe, accuracy %), ADE-20k (segmentation with linear layer, mIoU) and Oxford-M (image retrieval, mAP).
Each model is trained on the same number of iterations, that is smaller than our final run.
The KoLeo loss term improves nearest-neighbor search tasks (e.g. retrieval), and the MIM loss improves patch-level tasks (e.g. segmentation).
</span></figcaption>
</figure>
</section>
<section id="S6.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.5 </span>Impact of Knowledge Distillation</h3>

<div id="S6.SS5.p1" class="ltx_para ltx_noindent">
<p id="S6.SS5.p1.1" class="ltx_p">For small architectures, we distill larger models instead of training them from scratch.
We use the distillation procedure described in Sec. <a href="#S5.SS0.SSS0.Px5" title="Model distillation. ‣ 5 Efficient implementation ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>.
We evaluate the effectiveness of this approach by comparing a ViT-L/14 trained from scratch with one distilled from a ViT-g/14 over 12 benchmarks in Fig. <a href="#S6.F5" title="Figure 5 ‣ 6.5 Impact of Knowledge Distillation ‣ 6 Ablation Studies ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>.
We also report the performance of the ViT-g/14 used for distillation as a topline.
The distilled model outperforms the one trained from scratch on all 12 benchmarks, validating our pretraining approach for small models.</p>
</div>
<figure id="S6.F5" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S6.F5.sf1" class="ltx_figure ltx_figure_panel"><img src="/html/2304.07193/assets/x10.png" id="S6.F5.sf1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="352" height="284" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S6.F5.sf1.2.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="S6.F5.sf1.3.2" class="ltx_text" style="font-size:90%;">Comparison on individual metrics</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S6.F5.sf2" class="ltx_figure ltx_figure_panel">
<div id="S6.F5.sf2.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:460.4pt;height:220.8pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(80.1pt,-38.4pt) scale(1.53338828183897,1.53338828183897) ;">
<table id="S6.F5.sf2.1.1" class="ltx_tabular ltx_align_middle">
<tr id="S6.F5.sf2.1.1.1" class="ltx_tr">
<td id="S6.F5.sf2.1.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">Arch</td>
<td id="S6.F5.sf2.1.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">Method</td>
<td id="S6.F5.sf2.1.1.1.4" class="ltx_td ltx_border_tt"></td>
<td id="S6.F5.sf2.1.1.1.5" class="ltx_td ltx_align_center ltx_border_tt">INet-1k</td>
<td id="S6.F5.sf2.1.1.1.6" class="ltx_td ltx_align_center ltx_border_tt">Segm.</td>
<td id="S6.F5.sf2.1.1.1.1" class="ltx_td ltx_align_center ltx_border_tt">Depth<math id="S6.F5.sf2.1.1.1.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S6.F5.sf2.1.1.1.1.m1.1a"><mo stretchy="false" id="S6.F5.sf2.1.1.1.1.m1.1.1" xref="S6.F5.sf2.1.1.1.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S6.F5.sf2.1.1.1.1.m1.1b"><ci id="S6.F5.sf2.1.1.1.1.m1.1.1.cmml" xref="S6.F5.sf2.1.1.1.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.F5.sf2.1.1.1.1.m1.1c">\downarrow</annotation></semantics></math>
</td>
<td id="S6.F5.sf2.1.1.1.7" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_tt">Classif.</td>
</tr>
<tr id="S6.F5.sf2.1.1.2" class="ltx_tr">
<td id="S6.F5.sf2.1.1.2.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S6.F5.sf2.1.1.2.1.1" class="ltx_text" style="color:#808080;">ViT-g/14</span></td>
<td id="S6.F5.sf2.1.1.2.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S6.F5.sf2.1.1.2.2.1" class="ltx_text" style="color:#808080;">Scratch</span></td>
<td id="S6.F5.sf2.1.1.2.3" class="ltx_td ltx_border_t"></td>
<td id="S6.F5.sf2.1.1.2.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.F5.sf2.1.1.2.4.1" class="ltx_text" style="color:#808080;">86.5</span></td>
<td id="S6.F5.sf2.1.1.2.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.F5.sf2.1.1.2.5.1" class="ltx_text" style="color:#808080;">73.4</span></td>
<td id="S6.F5.sf2.1.1.2.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.F5.sf2.1.1.2.6.1" class="ltx_text" style="color:#808080;">1.00</span></td>
<td id="S6.F5.sf2.1.1.2.7" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t"><span id="S6.F5.sf2.1.1.2.7.1" class="ltx_text" style="color:#808080;">92.1</span></td>
</tr>
<tr id="S6.F5.sf2.1.1.3" class="ltx_tr">
<td id="S6.F5.sf2.1.1.3.1" class="ltx_td ltx_align_left">ViT-L/14</td>
<td id="S6.F5.sf2.1.1.3.2" class="ltx_td ltx_align_left">Scratch</td>
<td id="S6.F5.sf2.1.1.3.3" class="ltx_td"></td>
<td id="S6.F5.sf2.1.1.3.4" class="ltx_td ltx_align_center">84.5</td>
<td id="S6.F5.sf2.1.1.3.5" class="ltx_td ltx_align_center">72.2</td>
<td id="S6.F5.sf2.1.1.3.6" class="ltx_td ltx_align_center">1.10</td>
<td id="S6.F5.sf2.1.1.3.7" class="ltx_td ltx_nopad_r ltx_align_center">90.2</td>
</tr>
<tr id="S6.F5.sf2.1.1.4" class="ltx_tr">
<td id="S6.F5.sf2.1.1.4.1" class="ltx_td ltx_align_left">ViT-L/14</td>
<td id="S6.F5.sf2.1.1.4.2" class="ltx_td ltx_align_left">Distill</td>
<td id="S6.F5.sf2.1.1.4.3" class="ltx_td"></td>
<td id="S6.F5.sf2.1.1.4.4" class="ltx_td ltx_align_center"><span id="S6.F5.sf2.1.1.4.4.1" class="ltx_text ltx_font_bold">86.3</span></td>
<td id="S6.F5.sf2.1.1.4.5" class="ltx_td ltx_align_center"><span id="S6.F5.sf2.1.1.4.5.1" class="ltx_text ltx_font_bold">73.3</span></td>
<td id="S6.F5.sf2.1.1.4.6" class="ltx_td ltx_align_center"><span id="S6.F5.sf2.1.1.4.6.1" class="ltx_text ltx_font_bold">1.08</span></td>
<td id="S6.F5.sf2.1.1.4.7" class="ltx_td ltx_nopad_r ltx_align_center"><span id="S6.F5.sf2.1.1.4.7.1" class="ltx_text ltx_font_bold">91.2</span></td>
</tr>
<tr id="S6.F5.sf2.1.1.5" class="ltx_tr">
<td id="S6.F5.sf2.1.1.5.1" class="ltx_td ltx_align_left ltx_border_tt ltx_border_tt">Arch</td>
<td id="S6.F5.sf2.1.1.5.2" class="ltx_td ltx_align_left ltx_border_tt ltx_border_tt">Method</td>
<td id="S6.F5.sf2.1.1.5.3" class="ltx_td ltx_border_tt ltx_border_tt"></td>
<td id="S6.F5.sf2.1.1.5.4" class="ltx_td ltx_align_center ltx_border_tt ltx_border_tt">Finegr.</td>
<td id="S6.F5.sf2.1.1.5.5" class="ltx_td ltx_align_center ltx_border_tt ltx_border_tt">Retriev.</td>
<td id="S6.F5.sf2.1.1.5.6" class="ltx_td ltx_align_center ltx_border_tt ltx_border_tt">ARSketch</td>
<td id="S6.F5.sf2.1.1.5.7" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_tt ltx_border_tt">Video</td>
</tr>
<tr id="S6.F5.sf2.1.1.6" class="ltx_tr">
<td id="S6.F5.sf2.1.1.6.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S6.F5.sf2.1.1.6.1.1" class="ltx_text" style="color:#808080;">ViT-g/14</span></td>
<td id="S6.F5.sf2.1.1.6.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S6.F5.sf2.1.1.6.2.1" class="ltx_text" style="color:#808080;">Scratch</span></td>
<td id="S6.F5.sf2.1.1.6.3" class="ltx_td ltx_border_t"></td>
<td id="S6.F5.sf2.1.1.6.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.F5.sf2.1.1.6.4.1" class="ltx_text" style="color:#808080;">78.3</span></td>
<td id="S6.F5.sf2.1.1.6.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.F5.sf2.1.1.6.5.1" class="ltx_text" style="color:#808080;">75.2</span></td>
<td id="S6.F5.sf2.1.1.6.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.F5.sf2.1.1.6.6.1" class="ltx_text" style="color:#808080;">77.0</span></td>
<td id="S6.F5.sf2.1.1.6.7" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t"><span id="S6.F5.sf2.1.1.6.7.1" class="ltx_text" style="color:#808080;">69.3</span></td>
</tr>
<tr id="S6.F5.sf2.1.1.7" class="ltx_tr">
<td id="S6.F5.sf2.1.1.7.1" class="ltx_td ltx_align_left">ViT-L/14</td>
<td id="S6.F5.sf2.1.1.7.2" class="ltx_td ltx_align_left">Scratch</td>
<td id="S6.F5.sf2.1.1.7.3" class="ltx_td"></td>
<td id="S6.F5.sf2.1.1.7.4" class="ltx_td ltx_align_center">75.8</td>
<td id="S6.F5.sf2.1.1.7.5" class="ltx_td ltx_align_center">71.3</td>
<td id="S6.F5.sf2.1.1.7.6" class="ltx_td ltx_align_center">69.5</td>
<td id="S6.F5.sf2.1.1.7.7" class="ltx_td ltx_nopad_r ltx_align_center">67.3</td>
</tr>
<tr id="S6.F5.sf2.1.1.8" class="ltx_tr">
<td id="S6.F5.sf2.1.1.8.1" class="ltx_td ltx_align_left ltx_border_bb">ViT-L/14</td>
<td id="S6.F5.sf2.1.1.8.2" class="ltx_td ltx_align_left ltx_border_bb">Distill</td>
<td id="S6.F5.sf2.1.1.8.3" class="ltx_td ltx_border_bb"></td>
<td id="S6.F5.sf2.1.1.8.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S6.F5.sf2.1.1.8.4.1" class="ltx_text ltx_font_bold">77.6</span></td>
<td id="S6.F5.sf2.1.1.8.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S6.F5.sf2.1.1.8.5.1" class="ltx_text ltx_font_bold">76.3</span></td>
<td id="S6.F5.sf2.1.1.8.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="S6.F5.sf2.1.1.8.6.1" class="ltx_text ltx_font_bold">74.5</span></td>
<td id="S6.F5.sf2.1.1.8.7" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S6.F5.sf2.1.1.8.7.1" class="ltx_text ltx_font_bold">67.5</span></td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S6.F5.sf2.3.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="S6.F5.sf2.4.2" class="ltx_text" style="font-size:90%;">Averaged metrics on 8 vision tasks</span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S6.F5.3.1.1" class="ltx_text" style="font-size:90%;">Figure 5</span>: </span><span id="S6.F5.4.2" class="ltx_text" style="font-size:90%;">
<span id="S6.F5.4.2.1" class="ltx_text ltx_font_bold">Effectiveness of knowledge distillation.</span>
Comparison between a ViT-L trained from scratch or distilled from DINOv2 using ViT-g/14.
For reference, we also report the performance of the ViT-g/14 teacher. We show that a ViT-L model distilled from a frozen ViT-g outperforms a the same model trained from scratch on all benchmarks, sometimes even outperforming the distillation target.
</span></figcaption>
</figure>
</section>
<section id="S6.SS6" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.6 </span>Impact of Resolution</h3>

<div id="S6.SS6.p1" class="ltx_para ltx_noindent">
<p id="S6.SS6.p1.7" class="ltx_p">We measure the impact of changing the resolution during the pretraining on the performance of image and patch-level features.
We consider models trained from scratch using a fixed resolution of either <math id="S6.SS6.p1.1.m1.1" class="ltx_Math" alttext="224\times 224" display="inline"><semantics id="S6.SS6.p1.1.m1.1a"><mrow id="S6.SS6.p1.1.m1.1.1" xref="S6.SS6.p1.1.m1.1.1.cmml"><mn id="S6.SS6.p1.1.m1.1.1.2" xref="S6.SS6.p1.1.m1.1.1.2.cmml">224</mn><mo lspace="0.222em" rspace="0.222em" id="S6.SS6.p1.1.m1.1.1.1" xref="S6.SS6.p1.1.m1.1.1.1.cmml">×</mo><mn id="S6.SS6.p1.1.m1.1.1.3" xref="S6.SS6.p1.1.m1.1.1.3.cmml">224</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.SS6.p1.1.m1.1b"><apply id="S6.SS6.p1.1.m1.1.1.cmml" xref="S6.SS6.p1.1.m1.1.1"><times id="S6.SS6.p1.1.m1.1.1.1.cmml" xref="S6.SS6.p1.1.m1.1.1.1"></times><cn type="integer" id="S6.SS6.p1.1.m1.1.1.2.cmml" xref="S6.SS6.p1.1.m1.1.1.2">224</cn><cn type="integer" id="S6.SS6.p1.1.m1.1.1.3.cmml" xref="S6.SS6.p1.1.m1.1.1.3">224</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS6.p1.1.m1.1c">224\times 224</annotation></semantics></math> or <math id="S6.SS6.p1.2.m2.1" class="ltx_Math" alttext="416\times 416" display="inline"><semantics id="S6.SS6.p1.2.m2.1a"><mrow id="S6.SS6.p1.2.m2.1.1" xref="S6.SS6.p1.2.m2.1.1.cmml"><mn id="S6.SS6.p1.2.m2.1.1.2" xref="S6.SS6.p1.2.m2.1.1.2.cmml">416</mn><mo lspace="0.222em" rspace="0.222em" id="S6.SS6.p1.2.m2.1.1.1" xref="S6.SS6.p1.2.m2.1.1.1.cmml">×</mo><mn id="S6.SS6.p1.2.m2.1.1.3" xref="S6.SS6.p1.2.m2.1.1.3.cmml">416</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.SS6.p1.2.m2.1b"><apply id="S6.SS6.p1.2.m2.1.1.cmml" xref="S6.SS6.p1.2.m2.1.1"><times id="S6.SS6.p1.2.m2.1.1.1.cmml" xref="S6.SS6.p1.2.m2.1.1.1"></times><cn type="integer" id="S6.SS6.p1.2.m2.1.1.2.cmml" xref="S6.SS6.p1.2.m2.1.1.2">416</cn><cn type="integer" id="S6.SS6.p1.2.m2.1.1.3.cmml" xref="S6.SS6.p1.2.m2.1.1.3">416</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS6.p1.2.m2.1c">416\times 416</annotation></semantics></math>, and a model trained from scratch at <math id="S6.SS6.p1.3.m3.1" class="ltx_Math" alttext="224\times 224" display="inline"><semantics id="S6.SS6.p1.3.m3.1a"><mrow id="S6.SS6.p1.3.m3.1.1" xref="S6.SS6.p1.3.m3.1.1.cmml"><mn id="S6.SS6.p1.3.m3.1.1.2" xref="S6.SS6.p1.3.m3.1.1.2.cmml">224</mn><mo lspace="0.222em" rspace="0.222em" id="S6.SS6.p1.3.m3.1.1.1" xref="S6.SS6.p1.3.m3.1.1.1.cmml">×</mo><mn id="S6.SS6.p1.3.m3.1.1.3" xref="S6.SS6.p1.3.m3.1.1.3.cmml">224</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.SS6.p1.3.m3.1b"><apply id="S6.SS6.p1.3.m3.1.1.cmml" xref="S6.SS6.p1.3.m3.1.1"><times id="S6.SS6.p1.3.m3.1.1.1.cmml" xref="S6.SS6.p1.3.m3.1.1.1"></times><cn type="integer" id="S6.SS6.p1.3.m3.1.1.2.cmml" xref="S6.SS6.p1.3.m3.1.1.2">224</cn><cn type="integer" id="S6.SS6.p1.3.m3.1.1.3.cmml" xref="S6.SS6.p1.3.m3.1.1.3">224</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS6.p1.3.m3.1c">224\times 224</annotation></semantics></math>, then resumed for 10k more iterations at <math id="S6.SS6.p1.4.m4.1" class="ltx_Math" alttext="416\times 416" display="inline"><semantics id="S6.SS6.p1.4.m4.1a"><mrow id="S6.SS6.p1.4.m4.1.1" xref="S6.SS6.p1.4.m4.1.1.cmml"><mn id="S6.SS6.p1.4.m4.1.1.2" xref="S6.SS6.p1.4.m4.1.1.2.cmml">416</mn><mo lspace="0.222em" rspace="0.222em" id="S6.SS6.p1.4.m4.1.1.1" xref="S6.SS6.p1.4.m4.1.1.1.cmml">×</mo><mn id="S6.SS6.p1.4.m4.1.1.3" xref="S6.SS6.p1.4.m4.1.1.3.cmml">416</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.SS6.p1.4.m4.1b"><apply id="S6.SS6.p1.4.m4.1.1.cmml" xref="S6.SS6.p1.4.m4.1.1"><times id="S6.SS6.p1.4.m4.1.1.1.cmml" xref="S6.SS6.p1.4.m4.1.1.1"></times><cn type="integer" id="S6.SS6.p1.4.m4.1.1.2.cmml" xref="S6.SS6.p1.4.m4.1.1.2">416</cn><cn type="integer" id="S6.SS6.p1.4.m4.1.1.3.cmml" xref="S6.SS6.p1.4.m4.1.1.3">416</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS6.p1.4.m4.1c">416\times 416</annotation></semantics></math>.
High-resolution training is compute-intensive, so we conduct this ablation on a small setup: a ViT-L/16 trained on ImageNet1k.
In Fig. <a href="#S6.F6" title="Figure 6 ‣ 6.6 Impact of Resolution ‣ 6 Ablation Studies ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>, we report the performance of a linear probe on ImageNet-1k and ADE-20k, evaluated at various resolutions.
The model trained on high-resolution images performs best across resolutions, but this comes at a high cost: training at <math id="S6.SS6.p1.5.m5.1" class="ltx_Math" alttext="416" display="inline"><semantics id="S6.SS6.p1.5.m5.1a"><mn id="S6.SS6.p1.5.m5.1.1" xref="S6.SS6.p1.5.m5.1.1.cmml">416</mn><annotation-xml encoding="MathML-Content" id="S6.SS6.p1.5.m5.1b"><cn type="integer" id="S6.SS6.p1.5.m5.1.1.cmml" xref="S6.SS6.p1.5.m5.1.1">416</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS6.p1.5.m5.1c">416</annotation></semantics></math> is approximately <math id="S6.SS6.p1.6.m6.1" class="ltx_math_unparsed" alttext="3\times{}" display="inline"><semantics id="S6.SS6.p1.6.m6.1a"><mrow id="S6.SS6.p1.6.m6.1b"><mn id="S6.SS6.p1.6.m6.1.1">3</mn><mo lspace="0.222em" id="S6.SS6.p1.6.m6.1.2">×</mo></mrow><annotation encoding="application/x-tex" id="S6.SS6.p1.6.m6.1c">3\times{}</annotation></semantics></math> more compute-intensive than training at <math id="S6.SS6.p1.7.m7.1" class="ltx_Math" alttext="224" display="inline"><semantics id="S6.SS6.p1.7.m7.1a"><mn id="S6.SS6.p1.7.m7.1.1" xref="S6.SS6.p1.7.m7.1.1.cmml">224</mn><annotation-xml encoding="MathML-Content" id="S6.SS6.p1.7.m7.1b"><cn type="integer" id="S6.SS6.p1.7.m7.1.1.cmml" xref="S6.SS6.p1.7.m7.1.1">224</cn></annotation-xml><annotation encoding="application/x-tex" id="S6.SS6.p1.7.m7.1c">224</annotation></semantics></math>.
On the other hand, training at high resolution for only 10k iterations at the end of the training is almost as good and only requiring a fraction of the compute.
As a consequence, we include this step at the end of the training rather than training at a high resolution from scratch.</p>
</div>
<figure id="S6.F6" class="ltx_figure"><img src="/html/2304.07193/assets/x11.png" id="S6.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="250" height="94" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S6.F6.5.2.1" class="ltx_text" style="font-size:90%;">Figure 6</span>: </span><span id="S6.F6.2.1" class="ltx_text" style="font-size:90%;">
<span id="S6.F6.2.1.1" class="ltx_text ltx_font_bold">Role of resolution.</span>
Performance of ViT-L/16 trained on ImageNet-1k at fixed resolution (“224” and “416”) or trained at 224 then 416 for a short duration (“224<math id="S6.F6.2.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S6.F6.2.1.m1.1b"><mo stretchy="false" id="S6.F6.2.1.m1.1.1" xref="S6.F6.2.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S6.F6.2.1.m1.1c"><ci id="S6.F6.2.1.m1.1.1.cmml" xref="S6.F6.2.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.F6.2.1.m1.1d">\rightarrow</annotation></semantics></math>416”).
We train linear classifiers on top of frozen features at different resolutions and report Top-1 accuracy on ImageNet and mIoU on ADE-20k. We observe that performing SSL training at high resolution for a short duration achieve behavior and results close to training at the same high resolution for the full training, at a fraction of the cost.
</span></figcaption>
</figure>
</section>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Results</h2>

<div id="S7.p1" class="ltx_para ltx_noindent">
<p id="S7.p1.1" class="ltx_p">In this section, we present the empirical evaluation of our models on many image understanding tasks.
We evaluate both global and local image representations, on category and instance-level recognition, semantic segmentation, monocular depth prediction, and action recognition.
We detail the list of benchmarks in Appendix <a href="#A3" title="Appendix C List of Datasets used ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>.
The goal of this evaluation is twofold.
First, we show that our self-supervised features outperform the current state of the art by a very large margin.
Second, we show that they match, or surpass the performance of weakly-supervised ones on a substantial number of tasks.</p>
</div>
<div id="S7.p2" class="ltx_para ltx_noindent">
<p id="S7.p2.1" class="ltx_p"><span id="S7.p2.1.1" class="ltx_text ltx_font_bold">Baselines.</span>
In our comparisons, we use two kinds of models as baselines.
We compare to the best performing self-supervised models that are openly available.
First, we run our evaluations for MAE <cite class="ltx_cite ltx_citemacro_citep">(He et al., <a href="#bib.bib58" title="" class="ltx_ref">2022</a>)</cite>, DINO <cite class="ltx_cite ltx_citemacro_citep">(Caron et al., <a href="#bib.bib19" title="" class="ltx_ref">2021</a>)</cite>, SEERv2 <cite class="ltx_cite ltx_citemacro_citep">(Goyal et al., <a href="#bib.bib50" title="" class="ltx_ref">2022a</a>)</cite>, MSN <cite class="ltx_cite ltx_citemacro_citep">(Assran et al., <a href="#bib.bib3" title="" class="ltx_ref">2022</a>)</cite>, EsViT <cite class="ltx_cite ltx_citemacro_citep">(Li et al., <a href="#bib.bib75" title="" class="ltx_ref">2022a</a>)</cite>, Mugs <cite class="ltx_cite ltx_citemacro_citep">(Zhou et al., <a href="#bib.bib137" title="" class="ltx_ref">2022b</a>)</cite> and iBOT <cite class="ltx_cite ltx_citemacro_citep">(Zhou et al., <a href="#bib.bib136" title="" class="ltx_ref">2022a</a>)</cite>.
When several architectural variants were proposed for a given method, we report results for the one that leads to best top-1 accuracy on ImageNet-1k.
Second, we report performance of open-source weakly-supervised models such as CLIP <cite class="ltx_cite ltx_citemacro_citep">(Radford et al., <a href="#bib.bib93" title="" class="ltx_ref">2021</a>)</cite>, OpenCLIP <cite class="ltx_cite ltx_citemacro_citep">(Ilharco et al., <a href="#bib.bib66" title="" class="ltx_ref">2021</a>; Cherti et al., <a href="#bib.bib27" title="" class="ltx_ref">2023</a>)</cite>, and SWAG <cite class="ltx_cite ltx_citemacro_citep">(Singh et al., <a href="#bib.bib105" title="" class="ltx_ref">2022</a>)</cite>.
When evaluating models on ImageNet-1k, we report the performance for each of the aforementioned methods.
For all other evaluations, we report the four best-performing models amongst SSL ones.
Also, for reference, we report the best performing OpenCLIP-G for weakly-supervised ones.</p>
</div>
<section id="S7.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">7.1 </span>ImageNet Classification</h3>

<div id="S7.SS1.p1" class="ltx_para ltx_noindent">
<p id="S7.SS1.p1.1" class="ltx_p">As a first evaluation, we probe the quality of the holistic image representation produced by the model on the ImageNet-1k classification dataset.
We evaluate the quality of features by training a simple classifier over a frozen backbone, and do not perform finetuning of the backbone weights.
Following previous work, we use a linear model for simplicity, ensuring a reproducible evaluation, despite the fact that classes may not be linearly separable.
Because most SSL methods were developped using ImageNet-1k validation performance as a debugging signal, we also report the top-1 accuracy on ImageNet-ReaL and ImageNet-V2.
In order to report this additional validation performance, for all models, we run the evaluation with our code.
We compare our frozen features to the best publicly available SSL features in Table <a href="#S7.T4" title="Table 4 ‣ 7.1 ImageNet Classification ‣ 7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, regardless of architecture or pretraining data.
We see the components proposed in this work lead to a very significant improvement (<math id="S7.SS1.p1.1.m1.1" class="ltx_Math" alttext="+4.2\%" display="inline"><semantics id="S7.SS1.p1.1.m1.1a"><mrow id="S7.SS1.p1.1.m1.1.1" xref="S7.SS1.p1.1.m1.1.1.cmml"><mo id="S7.SS1.p1.1.m1.1.1a" xref="S7.SS1.p1.1.m1.1.1.cmml">+</mo><mrow id="S7.SS1.p1.1.m1.1.1.2" xref="S7.SS1.p1.1.m1.1.1.2.cmml"><mn id="S7.SS1.p1.1.m1.1.1.2.2" xref="S7.SS1.p1.1.m1.1.1.2.2.cmml">4.2</mn><mo id="S7.SS1.p1.1.m1.1.1.2.1" xref="S7.SS1.p1.1.m1.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.p1.1.m1.1b"><apply id="S7.SS1.p1.1.m1.1.1.cmml" xref="S7.SS1.p1.1.m1.1.1"><plus id="S7.SS1.p1.1.m1.1.1.1.cmml" xref="S7.SS1.p1.1.m1.1.1"></plus><apply id="S7.SS1.p1.1.m1.1.1.2.cmml" xref="S7.SS1.p1.1.m1.1.1.2"><csymbol cd="latexml" id="S7.SS1.p1.1.m1.1.1.2.1.cmml" xref="S7.SS1.p1.1.m1.1.1.2.1">percent</csymbol><cn type="float" id="S7.SS1.p1.1.m1.1.1.2.2.cmml" xref="S7.SS1.p1.1.m1.1.1.2.2">4.2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.p1.1.m1.1c">+4.2\%</annotation></semantics></math>) over the previous state of the art (iBOT ViT-L/16 trained on ImageNet-22k) on linear evaluation.
At the same time, we also see that the performance increase on the alternative test sets is larger for our method, indicating stronger generalization. We describe details of our linear evaluation in Appendix <a href="#A2.SS3" title="B.3 Linear probing evaluation ‣ Appendix B Implementation Details ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B.3</span></a>.</p>
</div>
<figure id="S7.T4" class="ltx_table">
<table id="S7.T4.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S7.T4.2.3" class="ltx_tr">
<td id="S7.T4.2.3.1" class="ltx_td ltx_border_tt"></td>
<td id="S7.T4.2.3.2" class="ltx_td ltx_border_tt"></td>
<td id="S7.T4.2.3.3" class="ltx_td ltx_border_tt"></td>
<td id="S7.T4.2.3.4" class="ltx_td ltx_border_tt"></td>
<td id="S7.T4.2.3.5" class="ltx_td ltx_border_tt"></td>
<td id="S7.T4.2.3.6" class="ltx_td ltx_align_center ltx_border_tt">kNN</td>
<td id="S7.T4.2.3.7" class="ltx_td ltx_border_tt"></td>
<td id="S7.T4.2.3.8" class="ltx_td ltx_align_center ltx_border_tt" colspan="3">linear</td>
</tr>
<tr id="S7.T4.2.4" class="ltx_tr">
<td id="S7.T4.2.4.1" class="ltx_td ltx_align_left">Method</td>
<td id="S7.T4.2.4.2" class="ltx_td ltx_align_left">Arch.</td>
<td id="S7.T4.2.4.3" class="ltx_td ltx_align_left">Data</td>
<td id="S7.T4.2.4.4" class="ltx_td ltx_align_center">Text sup.</td>
<td id="S7.T4.2.4.5" class="ltx_td"></td>
<td id="S7.T4.2.4.6" class="ltx_td ltx_align_center ltx_border_t">val</td>
<td id="S7.T4.2.4.7" class="ltx_td"></td>
<td id="S7.T4.2.4.8" class="ltx_td ltx_align_center ltx_border_t">val</td>
<td id="S7.T4.2.4.9" class="ltx_td ltx_align_center ltx_border_t">ReaL</td>
<td id="S7.T4.2.4.10" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">V2</td>
</tr>
<tr id="S7.T4.2.5" class="ltx_tr">
<td id="S7.T4.2.5.1" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t" colspan="10"><span id="S7.T4.2.5.1.1" class="ltx_text ltx_font_bold">Weakly supervised</span></td>
</tr>
<tr id="S7.T4.2.6" class="ltx_tr">
<td id="S7.T4.2.6.1" class="ltx_td ltx_align_left ltx_border_t">CLIP</td>
<td id="S7.T4.2.6.2" class="ltx_td ltx_align_left ltx_border_t">ViT-L/14</td>
<td id="S7.T4.2.6.3" class="ltx_td ltx_align_left ltx_border_t">WIT-400M</td>
<td id="S7.T4.2.6.4" class="ltx_td ltx_align_center ltx_border_t">✓</td>
<td id="S7.T4.2.6.5" class="ltx_td ltx_border_t"></td>
<td id="S7.T4.2.6.6" class="ltx_td ltx_align_center ltx_border_t">79.8</td>
<td id="S7.T4.2.6.7" class="ltx_td ltx_border_t"></td>
<td id="S7.T4.2.6.8" class="ltx_td ltx_align_center ltx_border_t">84.3</td>
<td id="S7.T4.2.6.9" class="ltx_td ltx_align_center ltx_border_t">88.1</td>
<td id="S7.T4.2.6.10" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">75.3</td>
</tr>
<tr id="S7.T4.1.1" class="ltx_tr">
<td id="S7.T4.1.1.2" class="ltx_td ltx_align_left">CLIP</td>
<td id="S7.T4.1.1.1" class="ltx_td ltx_align_left">ViT-L/14<math id="S7.T4.1.1.1.m1.1" class="ltx_Math" alttext="{}_{\text{336}}" display="inline"><semantics id="S7.T4.1.1.1.m1.1a"><msub id="S7.T4.1.1.1.m1.1.1" xref="S7.T4.1.1.1.m1.1.1.cmml"><mi id="S7.T4.1.1.1.m1.1.1a" xref="S7.T4.1.1.1.m1.1.1.cmml"></mi><mtext id="S7.T4.1.1.1.m1.1.1.1" xref="S7.T4.1.1.1.m1.1.1.1a.cmml">336</mtext></msub><annotation-xml encoding="MathML-Content" id="S7.T4.1.1.1.m1.1b"><apply id="S7.T4.1.1.1.m1.1.1.cmml" xref="S7.T4.1.1.1.m1.1.1"><ci id="S7.T4.1.1.1.m1.1.1.1a.cmml" xref="S7.T4.1.1.1.m1.1.1.1"><mtext mathsize="70%" id="S7.T4.1.1.1.m1.1.1.1.cmml" xref="S7.T4.1.1.1.m1.1.1.1">336</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T4.1.1.1.m1.1c">{}_{\text{336}}</annotation></semantics></math>
</td>
<td id="S7.T4.1.1.3" class="ltx_td ltx_align_left">WIT-400M</td>
<td id="S7.T4.1.1.4" class="ltx_td ltx_align_center">✓</td>
<td id="S7.T4.1.1.5" class="ltx_td"></td>
<td id="S7.T4.1.1.6" class="ltx_td ltx_align_center">80.5</td>
<td id="S7.T4.1.1.7" class="ltx_td"></td>
<td id="S7.T4.1.1.8" class="ltx_td ltx_align_center">85.3</td>
<td id="S7.T4.1.1.9" class="ltx_td ltx_align_center">88.8</td>
<td id="S7.T4.1.1.10" class="ltx_td ltx_nopad_r ltx_align_center">75.8</td>
</tr>
<tr id="S7.T4.2.7" class="ltx_tr">
<td id="S7.T4.2.7.1" class="ltx_td ltx_align_left">SWAG</td>
<td id="S7.T4.2.7.2" class="ltx_td ltx_align_left">ViT-H/14</td>
<td id="S7.T4.2.7.3" class="ltx_td ltx_align_left">IG3.6B</td>
<td id="S7.T4.2.7.4" class="ltx_td ltx_align_center">✓</td>
<td id="S7.T4.2.7.5" class="ltx_td"></td>
<td id="S7.T4.2.7.6" class="ltx_td ltx_align_center">82.6</td>
<td id="S7.T4.2.7.7" class="ltx_td"></td>
<td id="S7.T4.2.7.8" class="ltx_td ltx_align_center">85.7</td>
<td id="S7.T4.2.7.9" class="ltx_td ltx_align_center">88.7</td>
<td id="S7.T4.2.7.10" class="ltx_td ltx_nopad_r ltx_align_center">77.6</td>
</tr>
<tr id="S7.T4.2.8" class="ltx_tr">
<td id="S7.T4.2.8.1" class="ltx_td ltx_align_left">OpenCLIP</td>
<td id="S7.T4.2.8.2" class="ltx_td ltx_align_left">ViT-H/14</td>
<td id="S7.T4.2.8.3" class="ltx_td ltx_align_left">LAION-2B</td>
<td id="S7.T4.2.8.4" class="ltx_td ltx_align_center">✓</td>
<td id="S7.T4.2.8.5" class="ltx_td"></td>
<td id="S7.T4.2.8.6" class="ltx_td ltx_align_center">81.7</td>
<td id="S7.T4.2.8.7" class="ltx_td"></td>
<td id="S7.T4.2.8.8" class="ltx_td ltx_align_center">84.4</td>
<td id="S7.T4.2.8.9" class="ltx_td ltx_align_center">88.4</td>
<td id="S7.T4.2.8.10" class="ltx_td ltx_nopad_r ltx_align_center">75.5</td>
</tr>
<tr id="S7.T4.2.9" class="ltx_tr">
<td id="S7.T4.2.9.1" class="ltx_td ltx_align_left">OpenCLIP</td>
<td id="S7.T4.2.9.2" class="ltx_td ltx_align_left">ViT-G/14</td>
<td id="S7.T4.2.9.3" class="ltx_td ltx_align_left">LAION-2B</td>
<td id="S7.T4.2.9.4" class="ltx_td ltx_align_center">✓</td>
<td id="S7.T4.2.9.5" class="ltx_td"></td>
<td id="S7.T4.2.9.6" class="ltx_td ltx_align_center">83.2</td>
<td id="S7.T4.2.9.7" class="ltx_td"></td>
<td id="S7.T4.2.9.8" class="ltx_td ltx_align_center">86.2</td>
<td id="S7.T4.2.9.9" class="ltx_td ltx_align_center">89.4</td>
<td id="S7.T4.2.9.10" class="ltx_td ltx_nopad_r ltx_align_center">77.2</td>
</tr>
<tr id="S7.T4.2.2" class="ltx_tr">
<td id="S7.T4.2.2.2" class="ltx_td ltx_align_left">EVA-CLIP</td>
<td id="S7.T4.2.2.3" class="ltx_td ltx_align_left">ViT-g/14</td>
<td id="S7.T4.2.2.1" class="ltx_td ltx_align_left">custom<sup id="S7.T4.2.2.1.1" class="ltx_sup">∗</sup>
</td>
<td id="S7.T4.2.2.4" class="ltx_td ltx_align_center">✓</td>
<td id="S7.T4.2.2.5" class="ltx_td"></td>
<td id="S7.T4.2.2.6" class="ltx_td ltx_align_center"><span id="S7.T4.2.2.6.1" class="ltx_text ltx_font_bold">83.5</span></td>
<td id="S7.T4.2.2.7" class="ltx_td"></td>
<td id="S7.T4.2.2.8" class="ltx_td ltx_align_center">86.4</td>
<td id="S7.T4.2.2.9" class="ltx_td ltx_align_center">89.3</td>
<td id="S7.T4.2.2.10" class="ltx_td ltx_nopad_r ltx_align_center">77.4</td>
</tr>
<tr id="S7.T4.2.10" class="ltx_tr">
<td id="S7.T4.2.10.1" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t" colspan="10"><span id="S7.T4.2.10.1.1" class="ltx_text ltx_font_bold">Self-supervised</span></td>
</tr>
<tr id="S7.T4.2.11" class="ltx_tr">
<td id="S7.T4.2.11.1" class="ltx_td ltx_align_left ltx_border_t">MAE</td>
<td id="S7.T4.2.11.2" class="ltx_td ltx_align_left ltx_border_t">ViT-H/14</td>
<td id="S7.T4.2.11.3" class="ltx_td ltx_align_left ltx_border_t">INet-1k</td>
<td id="S7.T4.2.11.4" class="ltx_td ltx_align_center ltx_border_t">✕</td>
<td id="S7.T4.2.11.5" class="ltx_td ltx_border_t"></td>
<td id="S7.T4.2.11.6" class="ltx_td ltx_align_center ltx_border_t">49.4</td>
<td id="S7.T4.2.11.7" class="ltx_td ltx_border_t"></td>
<td id="S7.T4.2.11.8" class="ltx_td ltx_align_center ltx_border_t">76.6</td>
<td id="S7.T4.2.11.9" class="ltx_td ltx_align_center ltx_border_t">83.3</td>
<td id="S7.T4.2.11.10" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">64.8</td>
</tr>
<tr id="S7.T4.2.12" class="ltx_tr">
<td id="S7.T4.2.12.1" class="ltx_td ltx_align_left">DINO</td>
<td id="S7.T4.2.12.2" class="ltx_td ltx_align_left">ViT-S/8</td>
<td id="S7.T4.2.12.3" class="ltx_td ltx_align_left">INet-1k</td>
<td id="S7.T4.2.12.4" class="ltx_td ltx_align_center">✕</td>
<td id="S7.T4.2.12.5" class="ltx_td"></td>
<td id="S7.T4.2.12.6" class="ltx_td ltx_align_center">78.6</td>
<td id="S7.T4.2.12.7" class="ltx_td"></td>
<td id="S7.T4.2.12.8" class="ltx_td ltx_align_center">79.2</td>
<td id="S7.T4.2.12.9" class="ltx_td ltx_align_center">85.5</td>
<td id="S7.T4.2.12.10" class="ltx_td ltx_nopad_r ltx_align_center">68.2</td>
</tr>
<tr id="S7.T4.2.13" class="ltx_tr">
<td id="S7.T4.2.13.1" class="ltx_td ltx_align_left">SEERv2</td>
<td id="S7.T4.2.13.2" class="ltx_td ltx_align_left">RG10B</td>
<td id="S7.T4.2.13.3" class="ltx_td ltx_align_left">IG2B</td>
<td id="S7.T4.2.13.4" class="ltx_td ltx_align_center">✕</td>
<td id="S7.T4.2.13.5" class="ltx_td"></td>
<td id="S7.T4.2.13.6" class="ltx_td ltx_align_center">–</td>
<td id="S7.T4.2.13.7" class="ltx_td"></td>
<td id="S7.T4.2.13.8" class="ltx_td ltx_align_center">79.8</td>
<td id="S7.T4.2.13.9" class="ltx_td ltx_align_center">–</td>
<td id="S7.T4.2.13.10" class="ltx_td ltx_nopad_r ltx_align_center">–</td>
</tr>
<tr id="S7.T4.2.14" class="ltx_tr">
<td id="S7.T4.2.14.1" class="ltx_td ltx_align_left">MSN</td>
<td id="S7.T4.2.14.2" class="ltx_td ltx_align_left">ViT-L/7</td>
<td id="S7.T4.2.14.3" class="ltx_td ltx_align_left">INet-1k</td>
<td id="S7.T4.2.14.4" class="ltx_td ltx_align_center">✕</td>
<td id="S7.T4.2.14.5" class="ltx_td"></td>
<td id="S7.T4.2.14.6" class="ltx_td ltx_align_center">79.2</td>
<td id="S7.T4.2.14.7" class="ltx_td"></td>
<td id="S7.T4.2.14.8" class="ltx_td ltx_align_center">80.7</td>
<td id="S7.T4.2.14.9" class="ltx_td ltx_align_center">86.0</td>
<td id="S7.T4.2.14.10" class="ltx_td ltx_nopad_r ltx_align_center">69.7</td>
</tr>
<tr id="S7.T4.2.15" class="ltx_tr">
<td id="S7.T4.2.15.1" class="ltx_td ltx_align_left">EsViT</td>
<td id="S7.T4.2.15.2" class="ltx_td ltx_align_left">Swin-B/W=14</td>
<td id="S7.T4.2.15.3" class="ltx_td ltx_align_left">INet-1k</td>
<td id="S7.T4.2.15.4" class="ltx_td ltx_align_center">✕</td>
<td id="S7.T4.2.15.5" class="ltx_td"></td>
<td id="S7.T4.2.15.6" class="ltx_td ltx_align_center">79.4</td>
<td id="S7.T4.2.15.7" class="ltx_td"></td>
<td id="S7.T4.2.15.8" class="ltx_td ltx_align_center">81.3</td>
<td id="S7.T4.2.15.9" class="ltx_td ltx_align_center">87.0</td>
<td id="S7.T4.2.15.10" class="ltx_td ltx_nopad_r ltx_align_center">70.4</td>
</tr>
<tr id="S7.T4.2.16" class="ltx_tr">
<td id="S7.T4.2.16.1" class="ltx_td ltx_align_left">Mugs</td>
<td id="S7.T4.2.16.2" class="ltx_td ltx_align_left">ViT-L/16</td>
<td id="S7.T4.2.16.3" class="ltx_td ltx_align_left">INet-1k</td>
<td id="S7.T4.2.16.4" class="ltx_td ltx_align_center">✕</td>
<td id="S7.T4.2.16.5" class="ltx_td"></td>
<td id="S7.T4.2.16.6" class="ltx_td ltx_align_center">80.2</td>
<td id="S7.T4.2.16.7" class="ltx_td"></td>
<td id="S7.T4.2.16.8" class="ltx_td ltx_align_center">82.1</td>
<td id="S7.T4.2.16.9" class="ltx_td ltx_align_center">86.9</td>
<td id="S7.T4.2.16.10" class="ltx_td ltx_nopad_r ltx_align_center">70.8</td>
</tr>
<tr id="S7.T4.2.17" class="ltx_tr">
<td id="S7.T4.2.17.1" class="ltx_td ltx_align_left">iBOT</td>
<td id="S7.T4.2.17.2" class="ltx_td ltx_align_left">ViT-L/16</td>
<td id="S7.T4.2.17.3" class="ltx_td ltx_align_left">INet-22k</td>
<td id="S7.T4.2.17.4" class="ltx_td ltx_align_center">✕</td>
<td id="S7.T4.2.17.5" class="ltx_td"></td>
<td id="S7.T4.2.17.6" class="ltx_td ltx_align_center">72.9</td>
<td id="S7.T4.2.17.7" class="ltx_td"></td>
<td id="S7.T4.2.17.8" class="ltx_td ltx_align_center">82.3</td>
<td id="S7.T4.2.17.9" class="ltx_td ltx_align_center">87.5</td>
<td id="S7.T4.2.17.10" class="ltx_td ltx_nopad_r ltx_align_center">72.4</td>
</tr>
<tr id="S7.T4.2.18" class="ltx_tr">
<td id="S7.T4.2.18.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" rowspan="4"><span id="S7.T4.2.18.1.1" class="ltx_text">DINOv2</span></td>
<td id="S7.T4.2.18.2" class="ltx_td ltx_align_left ltx_border_t">ViT-S/14</td>
<td id="S7.T4.2.18.3" class="ltx_td ltx_align_left ltx_border_t">LVD-142M</td>
<td id="S7.T4.2.18.4" class="ltx_td ltx_align_center ltx_border_t">✕</td>
<td id="S7.T4.2.18.5" class="ltx_td ltx_border_t"></td>
<td id="S7.T4.2.18.6" class="ltx_td ltx_align_center ltx_border_t">79.0</td>
<td id="S7.T4.2.18.7" class="ltx_td ltx_border_t"></td>
<td id="S7.T4.2.18.8" class="ltx_td ltx_align_center ltx_border_t">81.1</td>
<td id="S7.T4.2.18.9" class="ltx_td ltx_align_center ltx_border_t">86.6</td>
<td id="S7.T4.2.18.10" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">70.9</td>
</tr>
<tr id="S7.T4.2.19" class="ltx_tr">
<td id="S7.T4.2.19.1" class="ltx_td ltx_align_left">ViT-B/14</td>
<td id="S7.T4.2.19.2" class="ltx_td ltx_align_left">LVD-142M</td>
<td id="S7.T4.2.19.3" class="ltx_td ltx_align_center">✕</td>
<td id="S7.T4.2.19.4" class="ltx_td"></td>
<td id="S7.T4.2.19.5" class="ltx_td ltx_align_center">82.1</td>
<td id="S7.T4.2.19.6" class="ltx_td"></td>
<td id="S7.T4.2.19.7" class="ltx_td ltx_align_center">84.5</td>
<td id="S7.T4.2.19.8" class="ltx_td ltx_align_center">88.3</td>
<td id="S7.T4.2.19.9" class="ltx_td ltx_nopad_r ltx_align_center">75.1</td>
</tr>
<tr id="S7.T4.2.20" class="ltx_tr">
<td id="S7.T4.2.20.1" class="ltx_td ltx_align_left">ViT-L/14</td>
<td id="S7.T4.2.20.2" class="ltx_td ltx_align_left">LVD-142M</td>
<td id="S7.T4.2.20.3" class="ltx_td ltx_align_center">✕</td>
<td id="S7.T4.2.20.4" class="ltx_td"></td>
<td id="S7.T4.2.20.5" class="ltx_td ltx_align_center"><span id="S7.T4.2.20.5.1" class="ltx_text ltx_font_bold">83.5</span></td>
<td id="S7.T4.2.20.6" class="ltx_td"></td>
<td id="S7.T4.2.20.7" class="ltx_td ltx_align_center">86.3</td>
<td id="S7.T4.2.20.8" class="ltx_td ltx_align_center">89.5</td>
<td id="S7.T4.2.20.9" class="ltx_td ltx_nopad_r ltx_align_center">78.0</td>
</tr>
<tr id="S7.T4.2.21" class="ltx_tr">
<td id="S7.T4.2.21.1" class="ltx_td ltx_align_left ltx_border_bb">ViT-g/14</td>
<td id="S7.T4.2.21.2" class="ltx_td ltx_align_left ltx_border_bb">LVD-142M</td>
<td id="S7.T4.2.21.3" class="ltx_td ltx_align_center ltx_border_bb">✕</td>
<td id="S7.T4.2.21.4" class="ltx_td ltx_border_bb"></td>
<td id="S7.T4.2.21.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T4.2.21.5.1" class="ltx_text ltx_font_bold">83.5</span></td>
<td id="S7.T4.2.21.6" class="ltx_td ltx_border_bb"></td>
<td id="S7.T4.2.21.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T4.2.21.7.1" class="ltx_text ltx_font_bold">86.5</span></td>
<td id="S7.T4.2.21.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T4.2.21.8.1" class="ltx_text ltx_font_bold">89.6</span></td>
<td id="S7.T4.2.21.9" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S7.T4.2.21.9.1" class="ltx_text ltx_font_bold">78.4</span></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S7.T4.7.2.1" class="ltx_text" style="font-size:90%;">Table 4</span>: </span><span id="S7.T4.4.1" class="ltx_text" style="font-size:90%;">
<span id="S7.T4.4.1.1" class="ltx_text ltx_font_bold">Linear evaluation on ImageNet-1k of frozen pretrained features.</span>
We report Top-1 accuracy on the validation set for publicly available models trained on public or private data, and with or without text supervision (text sup.). For reference, we also report the kNN performance on the validation set.
We compare across any possible architectures (Arch.), at resolution <math id="S7.T4.4.1.m1.1" class="ltx_Math" alttext="224\times 224" display="inline"><semantics id="S7.T4.4.1.m1.1b"><mrow id="S7.T4.4.1.m1.1.1" xref="S7.T4.4.1.m1.1.1.cmml"><mn id="S7.T4.4.1.m1.1.1.2" xref="S7.T4.4.1.m1.1.1.2.cmml">224</mn><mo lspace="0.222em" rspace="0.222em" id="S7.T4.4.1.m1.1.1.1" xref="S7.T4.4.1.m1.1.1.1.cmml">×</mo><mn id="S7.T4.4.1.m1.1.1.3" xref="S7.T4.4.1.m1.1.1.3.cmml">224</mn></mrow><annotation-xml encoding="MathML-Content" id="S7.T4.4.1.m1.1c"><apply id="S7.T4.4.1.m1.1.1.cmml" xref="S7.T4.4.1.m1.1.1"><times id="S7.T4.4.1.m1.1.1.1.cmml" xref="S7.T4.4.1.m1.1.1.1"></times><cn type="integer" id="S7.T4.4.1.m1.1.1.2.cmml" xref="S7.T4.4.1.m1.1.1.2">224</cn><cn type="integer" id="S7.T4.4.1.m1.1.1.3.cmml" xref="S7.T4.4.1.m1.1.1.3">224</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T4.4.1.m1.1d">224\times 224</annotation></semantics></math> unless stated otherwise.
The dataset used for training EVA-CLIP is a custom mixture, see paper for details <cite class="ltx_cite ltx_citemacro_citep">(Fang et al., <a href="#bib.bib43" title="" class="ltx_ref">2023</a>)</cite>.
</span></figcaption>
</figure>
<section id="S7.SS1.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">How far are we from weakly-supervised models?</h5>

<div id="S7.SS1.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S7.SS1.SSS0.Px1.p1.3" class="ltx_p">We also want to validate that our features are competitive with state-of-the-art open-source weakly supervised models.
To this end, we compare on ImageNet-1k, using the linear evaluation, to three off-the-shelf methods with several architectural variants.
For all models, we run the linear evaluation using our code, after making sure that our numbers match those reported in technical reports and papers.
We show the result of this evaluation in Table <a href="#S7.T4" title="Table 4 ‣ 7.1 ImageNet Classification ‣ 7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.
We see that our backbone, surpases the performance of OpenCLIP with a ViT-G/14 architecture (<math id="S7.SS1.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="+0.3\%" display="inline"><semantics id="S7.SS1.SSS0.Px1.p1.1.m1.1a"><mrow id="S7.SS1.SSS0.Px1.p1.1.m1.1.1" xref="S7.SS1.SSS0.Px1.p1.1.m1.1.1.cmml"><mo id="S7.SS1.SSS0.Px1.p1.1.m1.1.1a" xref="S7.SS1.SSS0.Px1.p1.1.m1.1.1.cmml">+</mo><mrow id="S7.SS1.SSS0.Px1.p1.1.m1.1.1.2" xref="S7.SS1.SSS0.Px1.p1.1.m1.1.1.2.cmml"><mn id="S7.SS1.SSS0.Px1.p1.1.m1.1.1.2.2" xref="S7.SS1.SSS0.Px1.p1.1.m1.1.1.2.2.cmml">0.3</mn><mo id="S7.SS1.SSS0.Px1.p1.1.m1.1.1.2.1" xref="S7.SS1.SSS0.Px1.p1.1.m1.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.SSS0.Px1.p1.1.m1.1b"><apply id="S7.SS1.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S7.SS1.SSS0.Px1.p1.1.m1.1.1"><plus id="S7.SS1.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="S7.SS1.SSS0.Px1.p1.1.m1.1.1"></plus><apply id="S7.SS1.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S7.SS1.SSS0.Px1.p1.1.m1.1.1.2"><csymbol cd="latexml" id="S7.SS1.SSS0.Px1.p1.1.m1.1.1.2.1.cmml" xref="S7.SS1.SSS0.Px1.p1.1.m1.1.1.2.1">percent</csymbol><cn type="float" id="S7.SS1.SSS0.Px1.p1.1.m1.1.1.2.2.cmml" xref="S7.SS1.SSS0.Px1.p1.1.m1.1.1.2.2">0.3</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.SSS0.Px1.p1.1.m1.1c">+0.3\%</annotation></semantics></math>) and EVA-CLIP with a ViT-g/14 (<math id="S7.SS1.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="+0.1\%" display="inline"><semantics id="S7.SS1.SSS0.Px1.p1.2.m2.1a"><mrow id="S7.SS1.SSS0.Px1.p1.2.m2.1.1" xref="S7.SS1.SSS0.Px1.p1.2.m2.1.1.cmml"><mo id="S7.SS1.SSS0.Px1.p1.2.m2.1.1a" xref="S7.SS1.SSS0.Px1.p1.2.m2.1.1.cmml">+</mo><mrow id="S7.SS1.SSS0.Px1.p1.2.m2.1.1.2" xref="S7.SS1.SSS0.Px1.p1.2.m2.1.1.2.cmml"><mn id="S7.SS1.SSS0.Px1.p1.2.m2.1.1.2.2" xref="S7.SS1.SSS0.Px1.p1.2.m2.1.1.2.2.cmml">0.1</mn><mo id="S7.SS1.SSS0.Px1.p1.2.m2.1.1.2.1" xref="S7.SS1.SSS0.Px1.p1.2.m2.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.SSS0.Px1.p1.2.m2.1b"><apply id="S7.SS1.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S7.SS1.SSS0.Px1.p1.2.m2.1.1"><plus id="S7.SS1.SSS0.Px1.p1.2.m2.1.1.1.cmml" xref="S7.SS1.SSS0.Px1.p1.2.m2.1.1"></plus><apply id="S7.SS1.SSS0.Px1.p1.2.m2.1.1.2.cmml" xref="S7.SS1.SSS0.Px1.p1.2.m2.1.1.2"><csymbol cd="latexml" id="S7.SS1.SSS0.Px1.p1.2.m2.1.1.2.1.cmml" xref="S7.SS1.SSS0.Px1.p1.2.m2.1.1.2.1">percent</csymbol><cn type="float" id="S7.SS1.SSS0.Px1.p1.2.m2.1.1.2.2.cmml" xref="S7.SS1.SSS0.Px1.p1.2.m2.1.1.2.2">0.1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.SSS0.Px1.p1.2.m2.1c">+0.1\%</annotation></semantics></math>).
At the same time, we also observe that our performance on the ImageNet-V2 test set is significantly better (<math id="S7.SS1.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="+1.1\%" display="inline"><semantics id="S7.SS1.SSS0.Px1.p1.3.m3.1a"><mrow id="S7.SS1.SSS0.Px1.p1.3.m3.1.1" xref="S7.SS1.SSS0.Px1.p1.3.m3.1.1.cmml"><mo id="S7.SS1.SSS0.Px1.p1.3.m3.1.1a" xref="S7.SS1.SSS0.Px1.p1.3.m3.1.1.cmml">+</mo><mrow id="S7.SS1.SSS0.Px1.p1.3.m3.1.1.2" xref="S7.SS1.SSS0.Px1.p1.3.m3.1.1.2.cmml"><mn id="S7.SS1.SSS0.Px1.p1.3.m3.1.1.2.2" xref="S7.SS1.SSS0.Px1.p1.3.m3.1.1.2.2.cmml">1.1</mn><mo id="S7.SS1.SSS0.Px1.p1.3.m3.1.1.2.1" xref="S7.SS1.SSS0.Px1.p1.3.m3.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.SSS0.Px1.p1.3.m3.1b"><apply id="S7.SS1.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S7.SS1.SSS0.Px1.p1.3.m3.1.1"><plus id="S7.SS1.SSS0.Px1.p1.3.m3.1.1.1.cmml" xref="S7.SS1.SSS0.Px1.p1.3.m3.1.1"></plus><apply id="S7.SS1.SSS0.Px1.p1.3.m3.1.1.2.cmml" xref="S7.SS1.SSS0.Px1.p1.3.m3.1.1.2"><csymbol cd="latexml" id="S7.SS1.SSS0.Px1.p1.3.m3.1.1.2.1.cmml" xref="S7.SS1.SSS0.Px1.p1.3.m3.1.1.2.1">percent</csymbol><cn type="float" id="S7.SS1.SSS0.Px1.p1.3.m3.1.1.2.2.cmml" xref="S7.SS1.SSS0.Px1.p1.3.m3.1.1.2.2">1.1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.SSS0.Px1.p1.3.m3.1c">+1.1\%</annotation></semantics></math> versus EVA-CLIP), indicating better generalization.
For the remainder of this section, we report OpenCLIP-G as a reference for weakly-supervised models.</p>
</div>
</section>
<section id="S7.SS1.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Can we finetune the encoders?</h5>

<div id="S7.SS1.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S7.SS1.SSS0.Px2.p1.6" class="ltx_p">We question if the ability of our models to produce high quality frozen features impact their performance when finetuned with supervision on a specific dataset.
While this is not core to this paper, this experiment is indicative of whether we have involuntarily specialized our models to the setting of linear evaluations of frozen features.
To run this sanity check, we apply the finetuning pipeline from <cite class="ltx_cite ltx_citemacro_citet">Touvron et al. (<a href="#bib.bib114" title="" class="ltx_ref">2022</a>)</cite>, without tweaking hyper-parameters.
In Table <a href="#S7.T5" title="Table 5 ‣ Can we finetune the encoders? ‣ 7.1 ImageNet Classification ‣ 7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, we show that the Top-1 accuracy on the validation set of ImageNet-1k improves by more than <math id="S7.SS1.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="+2\%" display="inline"><semantics id="S7.SS1.SSS0.Px2.p1.1.m1.1a"><mrow id="S7.SS1.SSS0.Px2.p1.1.m1.1.1" xref="S7.SS1.SSS0.Px2.p1.1.m1.1.1.cmml"><mo id="S7.SS1.SSS0.Px2.p1.1.m1.1.1a" xref="S7.SS1.SSS0.Px2.p1.1.m1.1.1.cmml">+</mo><mrow id="S7.SS1.SSS0.Px2.p1.1.m1.1.1.2" xref="S7.SS1.SSS0.Px2.p1.1.m1.1.1.2.cmml"><mn id="S7.SS1.SSS0.Px2.p1.1.m1.1.1.2.2" xref="S7.SS1.SSS0.Px2.p1.1.m1.1.1.2.2.cmml">2</mn><mo id="S7.SS1.SSS0.Px2.p1.1.m1.1.1.2.1" xref="S7.SS1.SSS0.Px2.p1.1.m1.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.SSS0.Px2.p1.1.m1.1b"><apply id="S7.SS1.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S7.SS1.SSS0.Px2.p1.1.m1.1.1"><plus id="S7.SS1.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="S7.SS1.SSS0.Px2.p1.1.m1.1.1"></plus><apply id="S7.SS1.SSS0.Px2.p1.1.m1.1.1.2.cmml" xref="S7.SS1.SSS0.Px2.p1.1.m1.1.1.2"><csymbol cd="latexml" id="S7.SS1.SSS0.Px2.p1.1.m1.1.1.2.1.cmml" xref="S7.SS1.SSS0.Px2.p1.1.m1.1.1.2.1">percent</csymbol><cn type="integer" id="S7.SS1.SSS0.Px2.p1.1.m1.1.1.2.2.cmml" xref="S7.SS1.SSS0.Px2.p1.1.m1.1.1.2.2">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.SSS0.Px2.p1.1.m1.1c">+2\%</annotation></semantics></math> when the backbone is finetuned.
This is true both when using models at resolution <math id="S7.SS1.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="224" display="inline"><semantics id="S7.SS1.SSS0.Px2.p1.2.m2.1a"><mn id="S7.SS1.SSS0.Px2.p1.2.m2.1.1" xref="S7.SS1.SSS0.Px2.p1.2.m2.1.1.cmml">224</mn><annotation-xml encoding="MathML-Content" id="S7.SS1.SSS0.Px2.p1.2.m2.1b"><cn type="integer" id="S7.SS1.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S7.SS1.SSS0.Px2.p1.2.m2.1.1">224</cn></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.SSS0.Px2.p1.2.m2.1c">224</annotation></semantics></math> and <math id="S7.SS1.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="448" display="inline"><semantics id="S7.SS1.SSS0.Px2.p1.3.m3.1a"><mn id="S7.SS1.SSS0.Px2.p1.3.m3.1.1" xref="S7.SS1.SSS0.Px2.p1.3.m3.1.1.cmml">448</mn><annotation-xml encoding="MathML-Content" id="S7.SS1.SSS0.Px2.p1.3.m3.1b"><cn type="integer" id="S7.SS1.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S7.SS1.SSS0.Px2.p1.3.m3.1.1">448</cn></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.SSS0.Px2.p1.3.m3.1c">448</annotation></semantics></math>.
Further gains can be obtained by tuning the hyper-parameters of the finetuning, but this is beyond the goal of this sanity check.
Nonetheless, our best finetuned performance (<math id="S7.SS1.SSS0.Px2.p1.4.m4.1" class="ltx_Math" alttext="88.9\%" display="inline"><semantics id="S7.SS1.SSS0.Px2.p1.4.m4.1a"><mrow id="S7.SS1.SSS0.Px2.p1.4.m4.1.1" xref="S7.SS1.SSS0.Px2.p1.4.m4.1.1.cmml"><mn id="S7.SS1.SSS0.Px2.p1.4.m4.1.1.2" xref="S7.SS1.SSS0.Px2.p1.4.m4.1.1.2.cmml">88.9</mn><mo id="S7.SS1.SSS0.Px2.p1.4.m4.1.1.1" xref="S7.SS1.SSS0.Px2.p1.4.m4.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.SSS0.Px2.p1.4.m4.1b"><apply id="S7.SS1.SSS0.Px2.p1.4.m4.1.1.cmml" xref="S7.SS1.SSS0.Px2.p1.4.m4.1.1"><csymbol cd="latexml" id="S7.SS1.SSS0.Px2.p1.4.m4.1.1.1.cmml" xref="S7.SS1.SSS0.Px2.p1.4.m4.1.1.1">percent</csymbol><cn type="float" id="S7.SS1.SSS0.Px2.p1.4.m4.1.1.2.cmml" xref="S7.SS1.SSS0.Px2.p1.4.m4.1.1.2">88.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.SSS0.Px2.p1.4.m4.1c">88.9\%</annotation></semantics></math>) is only a couple of percent below (<math id="S7.SS1.SSS0.Px2.p1.5.m5.1" class="ltx_Math" alttext="-2.2\%" display="inline"><semantics id="S7.SS1.SSS0.Px2.p1.5.m5.1a"><mrow id="S7.SS1.SSS0.Px2.p1.5.m5.1.1" xref="S7.SS1.SSS0.Px2.p1.5.m5.1.1.cmml"><mo id="S7.SS1.SSS0.Px2.p1.5.m5.1.1a" xref="S7.SS1.SSS0.Px2.p1.5.m5.1.1.cmml">−</mo><mrow id="S7.SS1.SSS0.Px2.p1.5.m5.1.1.2" xref="S7.SS1.SSS0.Px2.p1.5.m5.1.1.2.cmml"><mn id="S7.SS1.SSS0.Px2.p1.5.m5.1.1.2.2" xref="S7.SS1.SSS0.Px2.p1.5.m5.1.1.2.2.cmml">2.2</mn><mo id="S7.SS1.SSS0.Px2.p1.5.m5.1.1.2.1" xref="S7.SS1.SSS0.Px2.p1.5.m5.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.SSS0.Px2.p1.5.m5.1b"><apply id="S7.SS1.SSS0.Px2.p1.5.m5.1.1.cmml" xref="S7.SS1.SSS0.Px2.p1.5.m5.1.1"><minus id="S7.SS1.SSS0.Px2.p1.5.m5.1.1.1.cmml" xref="S7.SS1.SSS0.Px2.p1.5.m5.1.1"></minus><apply id="S7.SS1.SSS0.Px2.p1.5.m5.1.1.2.cmml" xref="S7.SS1.SSS0.Px2.p1.5.m5.1.1.2"><csymbol cd="latexml" id="S7.SS1.SSS0.Px2.p1.5.m5.1.1.2.1.cmml" xref="S7.SS1.SSS0.Px2.p1.5.m5.1.1.2.1">percent</csymbol><cn type="float" id="S7.SS1.SSS0.Px2.p1.5.m5.1.1.2.2.cmml" xref="S7.SS1.SSS0.Px2.p1.5.m5.1.1.2.2">2.2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.SSS0.Px2.p1.5.m5.1c">-2.2\%</annotation></semantics></math>) the absolute state of the arts (<math id="S7.SS1.SSS0.Px2.p1.6.m6.1" class="ltx_Math" alttext="91.1\%" display="inline"><semantics id="S7.SS1.SSS0.Px2.p1.6.m6.1a"><mrow id="S7.SS1.SSS0.Px2.p1.6.m6.1.1" xref="S7.SS1.SSS0.Px2.p1.6.m6.1.1.cmml"><mn id="S7.SS1.SSS0.Px2.p1.6.m6.1.1.2" xref="S7.SS1.SSS0.Px2.p1.6.m6.1.1.2.cmml">91.1</mn><mo id="S7.SS1.SSS0.Px2.p1.6.m6.1.1.1" xref="S7.SS1.SSS0.Px2.p1.6.m6.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.SSS0.Px2.p1.6.m6.1b"><apply id="S7.SS1.SSS0.Px2.p1.6.m6.1.1.cmml" xref="S7.SS1.SSS0.Px2.p1.6.m6.1.1"><csymbol cd="latexml" id="S7.SS1.SSS0.Px2.p1.6.m6.1.1.1.cmml" xref="S7.SS1.SSS0.Px2.p1.6.m6.1.1.1">percent</csymbol><cn type="float" id="S7.SS1.SSS0.Px2.p1.6.m6.1.1.2.cmml" xref="S7.SS1.SSS0.Px2.p1.6.m6.1.1.2">91.1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.SSS0.Px2.p1.6.m6.1c">91.1\%</annotation></semantics></math>), obtained by <cite class="ltx_cite ltx_citemacro_citet">Chen et al. (<a href="#bib.bib22" title="" class="ltx_ref">2023a</a>)</cite>.
As DINOv2 leads to features that are strong in both the linear and finetuning settings, a strong property of our approach is that <span id="S7.SS1.SSS0.Px2.p1.6.1" class="ltx_text ltx_font_italic">finetuning is optional</span>.</p>
</div>
<figure id="S7.T5" class="ltx_table">
<table id="S7.T5.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S7.T5.1.1" class="ltx_tr">
<td id="S7.T5.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">Arch.</td>
<td id="S7.T5.1.1.3" class="ltx_td ltx_align_center ltx_border_tt">Res.</td>
<td id="S7.T5.1.1.4" class="ltx_td ltx_align_center ltx_border_tt">Linear</td>
<td id="S7.T5.1.1.5" class="ltx_td ltx_align_center ltx_border_tt">Finetuned</td>
<td id="S7.T5.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_tt"><math id="S7.T5.1.1.1.m1.1" class="ltx_Math" alttext="\Delta" display="inline"><semantics id="S7.T5.1.1.1.m1.1a"><mi mathvariant="normal" id="S7.T5.1.1.1.m1.1.1" xref="S7.T5.1.1.1.m1.1.1.cmml">Δ</mi><annotation-xml encoding="MathML-Content" id="S7.T5.1.1.1.m1.1b"><ci id="S7.T5.1.1.1.m1.1.1.cmml" xref="S7.T5.1.1.1.m1.1.1">Δ</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.T5.1.1.1.m1.1c">\Delta</annotation></semantics></math></td>
</tr>
<tr id="S7.T5.1.2" class="ltx_tr">
<td id="S7.T5.1.2.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" rowspan="2"><span id="S7.T5.1.2.1.1" class="ltx_text">ViT-g/14</span></td>
<td id="S7.T5.1.2.2" class="ltx_td ltx_align_center ltx_border_t">224</td>
<td id="S7.T5.1.2.3" class="ltx_td ltx_align_center ltx_border_t">86.5</td>
<td id="S7.T5.1.2.4" class="ltx_td ltx_align_center ltx_border_t">88.5</td>
<td id="S7.T5.1.2.5" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">+2.0</td>
</tr>
<tr id="S7.T5.1.3" class="ltx_tr">
<td id="S7.T5.1.3.1" class="ltx_td ltx_align_center ltx_border_bb">448</td>
<td id="S7.T5.1.3.2" class="ltx_td ltx_align_center ltx_border_bb">86.7</td>
<td id="S7.T5.1.3.3" class="ltx_td ltx_align_center ltx_border_bb">88.9</td>
<td id="S7.T5.1.3.4" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">+2.2</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S7.T5.8.3.1" class="ltx_text" style="font-size:90%;">Table 5</span>: </span><span id="S7.T5.5.2" class="ltx_text" style="font-size:90%;">
<span id="S7.T5.5.2.1" class="ltx_text ltx_font_bold">Supervised finetuning on ImageNet-1k.</span>
We use the pipeline of <cite class="ltx_cite ltx_citemacro_cite">Touvron et al. (<a href="#bib.bib114" title="" class="ltx_ref">2022</a>)</cite> to finetune our encoders on ImageNet-1k at resolutions <math id="S7.T5.4.1.m1.1" class="ltx_Math" alttext="224\times 224" display="inline"><semantics id="S7.T5.4.1.m1.1b"><mrow id="S7.T5.4.1.m1.1.1" xref="S7.T5.4.1.m1.1.1.cmml"><mn id="S7.T5.4.1.m1.1.1.2" xref="S7.T5.4.1.m1.1.1.2.cmml">224</mn><mo lspace="0.222em" rspace="0.222em" id="S7.T5.4.1.m1.1.1.1" xref="S7.T5.4.1.m1.1.1.1.cmml">×</mo><mn id="S7.T5.4.1.m1.1.1.3" xref="S7.T5.4.1.m1.1.1.3.cmml">224</mn></mrow><annotation-xml encoding="MathML-Content" id="S7.T5.4.1.m1.1c"><apply id="S7.T5.4.1.m1.1.1.cmml" xref="S7.T5.4.1.m1.1.1"><times id="S7.T5.4.1.m1.1.1.1.cmml" xref="S7.T5.4.1.m1.1.1.1"></times><cn type="integer" id="S7.T5.4.1.m1.1.1.2.cmml" xref="S7.T5.4.1.m1.1.1.2">224</cn><cn type="integer" id="S7.T5.4.1.m1.1.1.3.cmml" xref="S7.T5.4.1.m1.1.1.3">224</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T5.4.1.m1.1d">224\times 224</annotation></semantics></math> or <math id="S7.T5.5.2.m2.1" class="ltx_Math" alttext="448\times 448" display="inline"><semantics id="S7.T5.5.2.m2.1b"><mrow id="S7.T5.5.2.m2.1.1" xref="S7.T5.5.2.m2.1.1.cmml"><mn id="S7.T5.5.2.m2.1.1.2" xref="S7.T5.5.2.m2.1.1.2.cmml">448</mn><mo lspace="0.222em" rspace="0.222em" id="S7.T5.5.2.m2.1.1.1" xref="S7.T5.5.2.m2.1.1.1.cmml">×</mo><mn id="S7.T5.5.2.m2.1.1.3" xref="S7.T5.5.2.m2.1.1.3.cmml">448</mn></mrow><annotation-xml encoding="MathML-Content" id="S7.T5.5.2.m2.1c"><apply id="S7.T5.5.2.m2.1.1.cmml" xref="S7.T5.5.2.m2.1.1"><times id="S7.T5.5.2.m2.1.1.1.cmml" xref="S7.T5.5.2.m2.1.1.1"></times><cn type="integer" id="S7.T5.5.2.m2.1.1.2.cmml" xref="S7.T5.5.2.m2.1.1.2">448</cn><cn type="integer" id="S7.T5.5.2.m2.1.1.3.cmml" xref="S7.T5.5.2.m2.1.1.3">448</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T5.5.2.m2.1d">448\times 448</annotation></semantics></math>. We compare with the accuracy obtained with linear probing and observe only modest improvements with fine-tuning: this suggests that DINOv2 features already perform well out-of-the-box.
</span></figcaption>
</figure>
</section>
<section id="S7.SS1.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Robustness analysis.</h5>

<div id="S7.SS1.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S7.SS1.SSS0.Px3.p1.3" class="ltx_p">To complement our study, and probe the generalization of our features, we evaluate our ImageNet-1k models trained with linear classification heads on domain generalization benchmarks.
We use the best performing linear classifier as described above and simply run inference on those benchmarks.
Please note that most results in the literature are obtained with models that are finetuned end-to-end on ImageNet-1k.
We show the result of this experiment in Table <a href="#S7.T6" title="Table 6 ‣ Robustness analysis. ‣ 7.1 ImageNet Classification ‣ 7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.
When comparing with state-of-the-art SSL methods, our models shows drastically better robustness (<math id="S7.SS1.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="+29.6\%" display="inline"><semantics id="S7.SS1.SSS0.Px3.p1.1.m1.1a"><mrow id="S7.SS1.SSS0.Px3.p1.1.m1.1.1" xref="S7.SS1.SSS0.Px3.p1.1.m1.1.1.cmml"><mo id="S7.SS1.SSS0.Px3.p1.1.m1.1.1a" xref="S7.SS1.SSS0.Px3.p1.1.m1.1.1.cmml">+</mo><mrow id="S7.SS1.SSS0.Px3.p1.1.m1.1.1.2" xref="S7.SS1.SSS0.Px3.p1.1.m1.1.1.2.cmml"><mn id="S7.SS1.SSS0.Px3.p1.1.m1.1.1.2.2" xref="S7.SS1.SSS0.Px3.p1.1.m1.1.1.2.2.cmml">29.6</mn><mo id="S7.SS1.SSS0.Px3.p1.1.m1.1.1.2.1" xref="S7.SS1.SSS0.Px3.p1.1.m1.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.SSS0.Px3.p1.1.m1.1b"><apply id="S7.SS1.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S7.SS1.SSS0.Px3.p1.1.m1.1.1"><plus id="S7.SS1.SSS0.Px3.p1.1.m1.1.1.1.cmml" xref="S7.SS1.SSS0.Px3.p1.1.m1.1.1"></plus><apply id="S7.SS1.SSS0.Px3.p1.1.m1.1.1.2.cmml" xref="S7.SS1.SSS0.Px3.p1.1.m1.1.1.2"><csymbol cd="latexml" id="S7.SS1.SSS0.Px3.p1.1.m1.1.1.2.1.cmml" xref="S7.SS1.SSS0.Px3.p1.1.m1.1.1.2.1">percent</csymbol><cn type="float" id="S7.SS1.SSS0.Px3.p1.1.m1.1.1.2.2.cmml" xref="S7.SS1.SSS0.Px3.p1.1.m1.1.1.2.2">29.6</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.SSS0.Px3.p1.1.m1.1c">+29.6\%</annotation></semantics></math> on A <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib62" title="" class="ltx_ref">2021b</a>)</cite>, <math id="S7.SS1.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="+22.1\%" display="inline"><semantics id="S7.SS1.SSS0.Px3.p1.2.m2.1a"><mrow id="S7.SS1.SSS0.Px3.p1.2.m2.1.1" xref="S7.SS1.SSS0.Px3.p1.2.m2.1.1.cmml"><mo id="S7.SS1.SSS0.Px3.p1.2.m2.1.1a" xref="S7.SS1.SSS0.Px3.p1.2.m2.1.1.cmml">+</mo><mrow id="S7.SS1.SSS0.Px3.p1.2.m2.1.1.2" xref="S7.SS1.SSS0.Px3.p1.2.m2.1.1.2.cmml"><mn id="S7.SS1.SSS0.Px3.p1.2.m2.1.1.2.2" xref="S7.SS1.SSS0.Px3.p1.2.m2.1.1.2.2.cmml">22.1</mn><mo id="S7.SS1.SSS0.Px3.p1.2.m2.1.1.2.1" xref="S7.SS1.SSS0.Px3.p1.2.m2.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.SSS0.Px3.p1.2.m2.1b"><apply id="S7.SS1.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S7.SS1.SSS0.Px3.p1.2.m2.1.1"><plus id="S7.SS1.SSS0.Px3.p1.2.m2.1.1.1.cmml" xref="S7.SS1.SSS0.Px3.p1.2.m2.1.1"></plus><apply id="S7.SS1.SSS0.Px3.p1.2.m2.1.1.2.cmml" xref="S7.SS1.SSS0.Px3.p1.2.m2.1.1.2"><csymbol cd="latexml" id="S7.SS1.SSS0.Px3.p1.2.m2.1.1.2.1.cmml" xref="S7.SS1.SSS0.Px3.p1.2.m2.1.1.2.1">percent</csymbol><cn type="float" id="S7.SS1.SSS0.Px3.p1.2.m2.1.1.2.2.cmml" xref="S7.SS1.SSS0.Px3.p1.2.m2.1.1.2.2">22.1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.SSS0.Px3.p1.2.m2.1c">+22.1\%</annotation></semantics></math> on R <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib61" title="" class="ltx_ref">2021a</a>)</cite> and <math id="S7.SS1.SSS0.Px3.p1.3.m3.1" class="ltx_Math" alttext="+23.0\%" display="inline"><semantics id="S7.SS1.SSS0.Px3.p1.3.m3.1a"><mrow id="S7.SS1.SSS0.Px3.p1.3.m3.1.1" xref="S7.SS1.SSS0.Px3.p1.3.m3.1.1.cmml"><mo id="S7.SS1.SSS0.Px3.p1.3.m3.1.1a" xref="S7.SS1.SSS0.Px3.p1.3.m3.1.1.cmml">+</mo><mrow id="S7.SS1.SSS0.Px3.p1.3.m3.1.1.2" xref="S7.SS1.SSS0.Px3.p1.3.m3.1.1.2.cmml"><mn id="S7.SS1.SSS0.Px3.p1.3.m3.1.1.2.2" xref="S7.SS1.SSS0.Px3.p1.3.m3.1.1.2.2.cmml">23.0</mn><mo id="S7.SS1.SSS0.Px3.p1.3.m3.1.1.2.1" xref="S7.SS1.SSS0.Px3.p1.3.m3.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.SSS0.Px3.p1.3.m3.1b"><apply id="S7.SS1.SSS0.Px3.p1.3.m3.1.1.cmml" xref="S7.SS1.SSS0.Px3.p1.3.m3.1.1"><plus id="S7.SS1.SSS0.Px3.p1.3.m3.1.1.1.cmml" xref="S7.SS1.SSS0.Px3.p1.3.m3.1.1"></plus><apply id="S7.SS1.SSS0.Px3.p1.3.m3.1.1.2.cmml" xref="S7.SS1.SSS0.Px3.p1.3.m3.1.1.2"><csymbol cd="latexml" id="S7.SS1.SSS0.Px3.p1.3.m3.1.1.2.1.cmml" xref="S7.SS1.SSS0.Px3.p1.3.m3.1.1.2.1">percent</csymbol><cn type="float" id="S7.SS1.SSS0.Px3.p1.3.m3.1.1.2.2.cmml" xref="S7.SS1.SSS0.Px3.p1.3.m3.1.1.2.2">23.0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.SSS0.Px3.p1.3.m3.1c">+23.0\%</annotation></semantics></math> on Sketch <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib120" title="" class="ltx_ref">2019</a>)</cite> compared to iBOT).
Our model also improves upon the best weakly-supervised model on ImageNet-A while lagging behind on R and Sketch.</p>
</div>
<figure id="S7.T6" class="ltx_table">
<table id="S7.T6.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S7.T6.1.1" class="ltx_tr">
<td id="S7.T6.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">Method</td>
<td id="S7.T6.1.1.3" class="ltx_td ltx_align_left ltx_border_tt">Arch</td>
<td id="S7.T6.1.1.4" class="ltx_td ltx_align_left ltx_border_tt">Data</td>
<td id="S7.T6.1.1.5" class="ltx_td ltx_border_tt"></td>
<td id="S7.T6.1.1.6" class="ltx_td ltx_align_center ltx_border_tt">Im-A</td>
<td id="S7.T6.1.1.7" class="ltx_td ltx_align_center ltx_border_tt">Im-R</td>
<td id="S7.T6.1.1.1" class="ltx_td ltx_align_center ltx_border_tt">Im-C<math id="S7.T6.1.1.1.m1.1" class="ltx_Math" alttext="\downarrow" display="inline"><semantics id="S7.T6.1.1.1.m1.1a"><mo stretchy="false" id="S7.T6.1.1.1.m1.1.1" xref="S7.T6.1.1.1.m1.1.1.cmml">↓</mo><annotation-xml encoding="MathML-Content" id="S7.T6.1.1.1.m1.1b"><ci id="S7.T6.1.1.1.m1.1.1.cmml" xref="S7.T6.1.1.1.m1.1.1">↓</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.T6.1.1.1.m1.1c">\downarrow</annotation></semantics></math>
</td>
<td id="S7.T6.1.1.8" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_tt">Sketch</td>
</tr>
<tr id="S7.T6.1.2" class="ltx_tr">
<td id="S7.T6.1.2.1" class="ltx_td ltx_align_left ltx_border_t">OpenCLIP</td>
<td id="S7.T6.1.2.2" class="ltx_td ltx_align_left ltx_border_t">ViT-G/14</td>
<td id="S7.T6.1.2.3" class="ltx_td ltx_align_left ltx_border_t">LAION-2B</td>
<td id="S7.T6.1.2.4" class="ltx_td ltx_border_t"></td>
<td id="S7.T6.1.2.5" class="ltx_td ltx_align_center ltx_border_t">63.8</td>
<td id="S7.T6.1.2.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S7.T6.1.2.6.1" class="ltx_text ltx_font_bold">87.8</span></td>
<td id="S7.T6.1.2.7" class="ltx_td ltx_align_center ltx_border_t">45.3</td>
<td id="S7.T6.1.2.8" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t"><span id="S7.T6.1.2.8.1" class="ltx_text ltx_font_bold">66.4</span></td>
</tr>
<tr id="S7.T6.1.3" class="ltx_tr">
<td id="S7.T6.1.3.1" class="ltx_td ltx_align_left ltx_border_t">MAE</td>
<td id="S7.T6.1.3.2" class="ltx_td ltx_align_left ltx_border_t">ViT-H/14</td>
<td id="S7.T6.1.3.3" class="ltx_td ltx_align_left ltx_border_t">INet-1k</td>
<td id="S7.T6.1.3.4" class="ltx_td ltx_border_t"></td>
<td id="S7.T6.1.3.5" class="ltx_td ltx_align_center ltx_border_t">10.2</td>
<td id="S7.T6.1.3.6" class="ltx_td ltx_align_center ltx_border_t">34.4</td>
<td id="S7.T6.1.3.7" class="ltx_td ltx_align_center ltx_border_t">61.4</td>
<td id="S7.T6.1.3.8" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">21.9</td>
</tr>
<tr id="S7.T6.1.4" class="ltx_tr">
<td id="S7.T6.1.4.1" class="ltx_td ltx_align_left">DINO</td>
<td id="S7.T6.1.4.2" class="ltx_td ltx_align_left">ViT-B/8</td>
<td id="S7.T6.1.4.3" class="ltx_td ltx_align_left">INet-1k</td>
<td id="S7.T6.1.4.4" class="ltx_td"></td>
<td id="S7.T6.1.4.5" class="ltx_td ltx_align_center">23.9</td>
<td id="S7.T6.1.4.6" class="ltx_td ltx_align_center">37.0</td>
<td id="S7.T6.1.4.7" class="ltx_td ltx_align_center">56.6</td>
<td id="S7.T6.1.4.8" class="ltx_td ltx_nopad_r ltx_align_center">25.5</td>
</tr>
<tr id="S7.T6.1.5" class="ltx_tr">
<td id="S7.T6.1.5.1" class="ltx_td ltx_align_left">iBOT</td>
<td id="S7.T6.1.5.2" class="ltx_td ltx_align_left">ViT-L/16</td>
<td id="S7.T6.1.5.3" class="ltx_td ltx_align_left">INet-22k</td>
<td id="S7.T6.1.5.4" class="ltx_td"></td>
<td id="S7.T6.1.5.5" class="ltx_td ltx_align_center">41.5</td>
<td id="S7.T6.1.5.6" class="ltx_td ltx_align_center">51.0</td>
<td id="S7.T6.1.5.7" class="ltx_td ltx_align_center">43.9</td>
<td id="S7.T6.1.5.8" class="ltx_td ltx_nopad_r ltx_align_center">38.5</td>
</tr>
<tr id="S7.T6.1.6" class="ltx_tr">
<td id="S7.T6.1.6.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" rowspan="4"><span id="S7.T6.1.6.1.1" class="ltx_text">DINOv2</span></td>
<td id="S7.T6.1.6.2" class="ltx_td ltx_align_left ltx_border_t">ViT-S/14</td>
<td id="S7.T6.1.6.3" class="ltx_td ltx_align_left ltx_border_t">LVD-142M</td>
<td id="S7.T6.1.6.4" class="ltx_td ltx_border_t"></td>
<td id="S7.T6.1.6.5" class="ltx_td ltx_align_center ltx_border_t">33.5</td>
<td id="S7.T6.1.6.6" class="ltx_td ltx_align_center ltx_border_t">53.7</td>
<td id="S7.T6.1.6.7" class="ltx_td ltx_align_center ltx_border_t">54.4</td>
<td id="S7.T6.1.6.8" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">41.2</td>
</tr>
<tr id="S7.T6.1.7" class="ltx_tr">
<td id="S7.T6.1.7.1" class="ltx_td ltx_align_left">ViT-B/14</td>
<td id="S7.T6.1.7.2" class="ltx_td ltx_align_left">LVD-142M</td>
<td id="S7.T6.1.7.3" class="ltx_td"></td>
<td id="S7.T6.1.7.4" class="ltx_td ltx_align_center">55.1</td>
<td id="S7.T6.1.7.5" class="ltx_td ltx_align_center">63.3</td>
<td id="S7.T6.1.7.6" class="ltx_td ltx_align_center">42.7</td>
<td id="S7.T6.1.7.7" class="ltx_td ltx_nopad_r ltx_align_center">50.6</td>
</tr>
<tr id="S7.T6.1.8" class="ltx_tr">
<td id="S7.T6.1.8.1" class="ltx_td ltx_align_left">ViT-L/14</td>
<td id="S7.T6.1.8.2" class="ltx_td ltx_align_left">LVD-142M</td>
<td id="S7.T6.1.8.3" class="ltx_td"></td>
<td id="S7.T6.1.8.4" class="ltx_td ltx_align_center">71.3</td>
<td id="S7.T6.1.8.5" class="ltx_td ltx_align_center">74.4</td>
<td id="S7.T6.1.8.6" class="ltx_td ltx_align_center">31.5</td>
<td id="S7.T6.1.8.7" class="ltx_td ltx_nopad_r ltx_align_center">59.3</td>
</tr>
<tr id="S7.T6.1.9" class="ltx_tr">
<td id="S7.T6.1.9.1" class="ltx_td ltx_align_left ltx_border_bb">ViT-g/14</td>
<td id="S7.T6.1.9.2" class="ltx_td ltx_align_left ltx_border_bb">LVD-142M</td>
<td id="S7.T6.1.9.3" class="ltx_td ltx_border_bb"></td>
<td id="S7.T6.1.9.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T6.1.9.4.1" class="ltx_text ltx_font_bold">75.9</span></td>
<td id="S7.T6.1.9.5" class="ltx_td ltx_align_center ltx_border_bb">78.8</td>
<td id="S7.T6.1.9.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T6.1.9.6.1" class="ltx_text ltx_font_bold">28.2</span></td>
<td id="S7.T6.1.9.7" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb">62.5</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S7.T6.4.1.1" class="ltx_text" style="font-size:90%;">Table 6</span>: </span><span id="S7.T6.5.2" class="ltx_text" style="font-size:90%;">
<span id="S7.T6.5.2.1" class="ltx_text ltx_font_bold">Domain Generalization with a linear probe</span> on top of frozen features at a resolution of 224.
Higher numbers are better for all benchmarks except Im-C.
</span></figcaption>
</figure>
</section>
</section>
<section id="S7.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">7.2 </span>Additional Image and Video classification Benchmarks</h3>

<div id="S7.SS2.p1" class="ltx_para ltx_noindent">
<p id="S7.SS2.p1.3" class="ltx_p">In this section we study the generalization of our features on downstream classification benchmarks.
We consider two sets of evaluations in that context.
On one hand, we use large and finegrained datasets such as iNaturalist and Places205.
On the other, we use the 12 image classification tasks originally proposed in SimCLR <cite class="ltx_cite ltx_citemacro_citep">(Chen et al., <a href="#bib.bib21" title="" class="ltx_ref">2020</a>)</cite>.
For iNaturalist 2018, iNaturalist 2021, and Places205, we train a linear classifier with data augmentations as in Sec. <a href="#S7.SS1" title="7.1 ImageNet Classification ‣ 7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7.1</span></a>
We report top-1 accuracy for those three datasets in Table <a href="#S7.T7" title="Table 7 ‣ 7.2 Additional Image and Video classification Benchmarks ‣ 7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>.
Interestingly, our model significantly outperforms OpenCLIP ViT-G/14 on both variants of iNaturalist (<math id="S7.SS2.p1.1.m1.1" class="ltx_Math" alttext="+8.6\%" display="inline"><semantics id="S7.SS2.p1.1.m1.1a"><mrow id="S7.SS2.p1.1.m1.1.1" xref="S7.SS2.p1.1.m1.1.1.cmml"><mo id="S7.SS2.p1.1.m1.1.1a" xref="S7.SS2.p1.1.m1.1.1.cmml">+</mo><mrow id="S7.SS2.p1.1.m1.1.1.2" xref="S7.SS2.p1.1.m1.1.1.2.cmml"><mn id="S7.SS2.p1.1.m1.1.1.2.2" xref="S7.SS2.p1.1.m1.1.1.2.2.cmml">8.6</mn><mo id="S7.SS2.p1.1.m1.1.1.2.1" xref="S7.SS2.p1.1.m1.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS2.p1.1.m1.1b"><apply id="S7.SS2.p1.1.m1.1.1.cmml" xref="S7.SS2.p1.1.m1.1.1"><plus id="S7.SS2.p1.1.m1.1.1.1.cmml" xref="S7.SS2.p1.1.m1.1.1"></plus><apply id="S7.SS2.p1.1.m1.1.1.2.cmml" xref="S7.SS2.p1.1.m1.1.1.2"><csymbol cd="latexml" id="S7.SS2.p1.1.m1.1.1.2.1.cmml" xref="S7.SS2.p1.1.m1.1.1.2.1">percent</csymbol><cn type="float" id="S7.SS2.p1.1.m1.1.1.2.2.cmml" xref="S7.SS2.p1.1.m1.1.1.2.2">8.6</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p1.1.m1.1c">+8.6\%</annotation></semantics></math> and <math id="S7.SS2.p1.2.m2.1" class="ltx_Math" alttext="+9.7\%" display="inline"><semantics id="S7.SS2.p1.2.m2.1a"><mrow id="S7.SS2.p1.2.m2.1.1" xref="S7.SS2.p1.2.m2.1.1.cmml"><mo id="S7.SS2.p1.2.m2.1.1a" xref="S7.SS2.p1.2.m2.1.1.cmml">+</mo><mrow id="S7.SS2.p1.2.m2.1.1.2" xref="S7.SS2.p1.2.m2.1.1.2.cmml"><mn id="S7.SS2.p1.2.m2.1.1.2.2" xref="S7.SS2.p1.2.m2.1.1.2.2.cmml">9.7</mn><mo id="S7.SS2.p1.2.m2.1.1.2.1" xref="S7.SS2.p1.2.m2.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS2.p1.2.m2.1b"><apply id="S7.SS2.p1.2.m2.1.1.cmml" xref="S7.SS2.p1.2.m2.1.1"><plus id="S7.SS2.p1.2.m2.1.1.1.cmml" xref="S7.SS2.p1.2.m2.1.1"></plus><apply id="S7.SS2.p1.2.m2.1.1.2.cmml" xref="S7.SS2.p1.2.m2.1.1.2"><csymbol cd="latexml" id="S7.SS2.p1.2.m2.1.1.2.1.cmml" xref="S7.SS2.p1.2.m2.1.1.2.1">percent</csymbol><cn type="float" id="S7.SS2.p1.2.m2.1.1.2.2.cmml" xref="S7.SS2.p1.2.m2.1.1.2.2">9.7</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p1.2.m2.1c">+9.7\%</annotation></semantics></math> for 2018 and 2021 respectively), and lags slightly behind on Places 205 (<math id="S7.SS2.p1.3.m3.1" class="ltx_Math" alttext="-2.3\%" display="inline"><semantics id="S7.SS2.p1.3.m3.1a"><mrow id="S7.SS2.p1.3.m3.1.1" xref="S7.SS2.p1.3.m3.1.1.cmml"><mo id="S7.SS2.p1.3.m3.1.1a" xref="S7.SS2.p1.3.m3.1.1.cmml">−</mo><mrow id="S7.SS2.p1.3.m3.1.1.2" xref="S7.SS2.p1.3.m3.1.1.2.cmml"><mn id="S7.SS2.p1.3.m3.1.1.2.2" xref="S7.SS2.p1.3.m3.1.1.2.2.cmml">2.3</mn><mo id="S7.SS2.p1.3.m3.1.1.2.1" xref="S7.SS2.p1.3.m3.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS2.p1.3.m3.1b"><apply id="S7.SS2.p1.3.m3.1.1.cmml" xref="S7.SS2.p1.3.m3.1.1"><minus id="S7.SS2.p1.3.m3.1.1.1.cmml" xref="S7.SS2.p1.3.m3.1.1"></minus><apply id="S7.SS2.p1.3.m3.1.1.2.cmml" xref="S7.SS2.p1.3.m3.1.1.2"><csymbol cd="latexml" id="S7.SS2.p1.3.m3.1.1.2.1.cmml" xref="S7.SS2.p1.3.m3.1.1.2.1">percent</csymbol><cn type="float" id="S7.SS2.p1.3.m3.1.1.2.2.cmml" xref="S7.SS2.p1.3.m3.1.1.2.2">2.3</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p1.3.m3.1c">-2.3\%</annotation></semantics></math>).</p>
</div>
<figure id="S7.T7" class="ltx_table">
<table id="S7.T7.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S7.T7.2.1" class="ltx_tr">
<td id="S7.T7.2.1.1" class="ltx_td ltx_border_tt"></td>
<td id="S7.T7.2.1.2" class="ltx_td ltx_border_tt"></td>
<td id="S7.T7.2.1.3" class="ltx_td ltx_border_tt"></td>
<td id="S7.T7.2.1.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="3">Image classification</td>
<td id="S7.T7.2.1.5" class="ltx_td ltx_border_tt"></td>
<td id="S7.T7.2.1.6" class="ltx_td ltx_align_center ltx_border_tt" colspan="3">Video classification</td>
</tr>
<tr id="S7.T7.2.2" class="ltx_tr">
<td id="S7.T7.2.2.1" class="ltx_td ltx_align_left">Feature</td>
<td id="S7.T7.2.2.2" class="ltx_td ltx_align_left">Arch</td>
<td id="S7.T7.2.2.3" class="ltx_td"></td>
<td id="S7.T7.2.2.4" class="ltx_td ltx_align_center ltx_border_t">iNat2018</td>
<td id="S7.T7.2.2.5" class="ltx_td ltx_align_center ltx_border_t">iNat2021</td>
<td id="S7.T7.2.2.6" class="ltx_td ltx_align_center ltx_border_t">Places205</td>
<td id="S7.T7.2.2.7" class="ltx_td"></td>
<td id="S7.T7.2.2.8" class="ltx_td ltx_align_center ltx_border_t">K400</td>
<td id="S7.T7.2.2.9" class="ltx_td ltx_align_center ltx_border_t">UCF-101</td>
<td id="S7.T7.2.2.10" class="ltx_td ltx_align_center ltx_border_t">SSv2</td>
</tr>
<tr id="S7.T7.2.3" class="ltx_tr">
<td id="S7.T7.2.3.1" class="ltx_td ltx_align_left ltx_border_t">OpenCLIP</td>
<td id="S7.T7.2.3.2" class="ltx_td ltx_align_left ltx_border_t">ViT-G/14</td>
<td id="S7.T7.2.3.3" class="ltx_td ltx_border_t"></td>
<td id="S7.T7.2.3.4" class="ltx_td ltx_align_center ltx_border_t">73.0</td>
<td id="S7.T7.2.3.5" class="ltx_td ltx_align_center ltx_border_t">76.0</td>
<td id="S7.T7.2.3.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S7.T7.2.3.6.1" class="ltx_text ltx_font_bold">69.8</span></td>
<td id="S7.T7.2.3.7" class="ltx_td ltx_border_t"></td>
<td id="S7.T7.2.3.8" class="ltx_td ltx_align_center ltx_border_t">78.3</td>
<td id="S7.T7.2.3.9" class="ltx_td ltx_align_center ltx_border_t">90.7</td>
<td id="S7.T7.2.3.10" class="ltx_td ltx_align_center ltx_border_t">35.8</td>
</tr>
<tr id="S7.T7.2.4" class="ltx_tr">
<td id="S7.T7.2.4.1" class="ltx_td ltx_align_left ltx_border_t">MAE</td>
<td id="S7.T7.2.4.2" class="ltx_td ltx_align_left ltx_border_t">ViT-H/14</td>
<td id="S7.T7.2.4.3" class="ltx_td ltx_border_t"></td>
<td id="S7.T7.2.4.4" class="ltx_td ltx_align_center ltx_border_t">31.0</td>
<td id="S7.T7.2.4.5" class="ltx_td ltx_align_center ltx_border_t">32.3</td>
<td id="S7.T7.2.4.6" class="ltx_td ltx_align_center ltx_border_t">52.4</td>
<td id="S7.T7.2.4.7" class="ltx_td ltx_border_t"></td>
<td id="S7.T7.2.4.8" class="ltx_td ltx_align_center ltx_border_t">54.2</td>
<td id="S7.T7.2.4.9" class="ltx_td ltx_align_center ltx_border_t">70.6</td>
<td id="S7.T7.2.4.10" class="ltx_td ltx_align_center ltx_border_t">29.2</td>
</tr>
<tr id="S7.T7.2.5" class="ltx_tr">
<td id="S7.T7.2.5.1" class="ltx_td ltx_align_left">DINO</td>
<td id="S7.T7.2.5.2" class="ltx_td ltx_align_left">ViT-B/8</td>
<td id="S7.T7.2.5.3" class="ltx_td"></td>
<td id="S7.T7.2.5.4" class="ltx_td ltx_align_center">59.6</td>
<td id="S7.T7.2.5.5" class="ltx_td ltx_align_center">68.3</td>
<td id="S7.T7.2.5.6" class="ltx_td ltx_align_center">60.4</td>
<td id="S7.T7.2.5.7" class="ltx_td"></td>
<td id="S7.T7.2.5.8" class="ltx_td ltx_align_center">64.5</td>
<td id="S7.T7.2.5.9" class="ltx_td ltx_align_center">85.0</td>
<td id="S7.T7.2.5.10" class="ltx_td ltx_align_center">32.6</td>
</tr>
<tr id="S7.T7.2.6" class="ltx_tr">
<td id="S7.T7.2.6.1" class="ltx_td ltx_align_left">iBOT</td>
<td id="S7.T7.2.6.2" class="ltx_td ltx_align_left">ViT-L/16</td>
<td id="S7.T7.2.6.3" class="ltx_td"></td>
<td id="S7.T7.2.6.4" class="ltx_td ltx_align_center">66.3</td>
<td id="S7.T7.2.6.5" class="ltx_td ltx_align_center">74.6</td>
<td id="S7.T7.2.6.6" class="ltx_td ltx_align_center">64.4</td>
<td id="S7.T7.2.6.7" class="ltx_td"></td>
<td id="S7.T7.2.6.8" class="ltx_td ltx_align_center">72.6</td>
<td id="S7.T7.2.6.9" class="ltx_td ltx_align_center">88.6</td>
<td id="S7.T7.2.6.10" class="ltx_td ltx_align_center"><span id="S7.T7.2.6.10.1" class="ltx_text ltx_font_bold">38.7</span></td>
</tr>
<tr id="S7.T7.2.7" class="ltx_tr">
<td id="S7.T7.2.7.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" rowspan="4"><span id="S7.T7.2.7.1.1" class="ltx_text">DINOv2</span></td>
<td id="S7.T7.2.7.2" class="ltx_td ltx_align_left ltx_border_t">ViT-S/14</td>
<td id="S7.T7.2.7.3" class="ltx_td ltx_border_t"></td>
<td id="S7.T7.2.7.4" class="ltx_td ltx_align_center ltx_border_t">69.0</td>
<td id="S7.T7.2.7.5" class="ltx_td ltx_align_center ltx_border_t">74.2</td>
<td id="S7.T7.2.7.6" class="ltx_td ltx_align_center ltx_border_t">62.9</td>
<td id="S7.T7.2.7.7" class="ltx_td ltx_border_t"></td>
<td id="S7.T7.2.7.8" class="ltx_td ltx_align_center ltx_border_t">67.8</td>
<td id="S7.T7.2.7.9" class="ltx_td ltx_align_center ltx_border_t">87.0</td>
<td id="S7.T7.2.7.10" class="ltx_td ltx_align_center ltx_border_t">33.1</td>
</tr>
<tr id="S7.T7.2.8" class="ltx_tr">
<td id="S7.T7.2.8.1" class="ltx_td ltx_align_left">ViT-B/14</td>
<td id="S7.T7.2.8.2" class="ltx_td"></td>
<td id="S7.T7.2.8.3" class="ltx_td ltx_align_center">76.4</td>
<td id="S7.T7.2.8.4" class="ltx_td ltx_align_center">81.1</td>
<td id="S7.T7.2.8.5" class="ltx_td ltx_align_center">66.2</td>
<td id="S7.T7.2.8.6" class="ltx_td"></td>
<td id="S7.T7.2.8.7" class="ltx_td ltx_align_center">73.2</td>
<td id="S7.T7.2.8.8" class="ltx_td ltx_align_center">89.1</td>
<td id="S7.T7.2.8.9" class="ltx_td ltx_align_center">34.4</td>
</tr>
<tr id="S7.T7.2.9" class="ltx_tr">
<td id="S7.T7.2.9.1" class="ltx_td ltx_align_left">ViT-L/14</td>
<td id="S7.T7.2.9.2" class="ltx_td"></td>
<td id="S7.T7.2.9.3" class="ltx_td ltx_align_center">80.4</td>
<td id="S7.T7.2.9.4" class="ltx_td ltx_align_center">85.1</td>
<td id="S7.T7.2.9.5" class="ltx_td ltx_align_center">67.3</td>
<td id="S7.T7.2.9.6" class="ltx_td"></td>
<td id="S7.T7.2.9.7" class="ltx_td ltx_align_center">76.3</td>
<td id="S7.T7.2.9.8" class="ltx_td ltx_align_center">90.5</td>
<td id="S7.T7.2.9.9" class="ltx_td ltx_align_center">35.6</td>
</tr>
<tr id="S7.T7.2.10" class="ltx_tr">
<td id="S7.T7.2.10.1" class="ltx_td ltx_align_left ltx_border_bb">ViT-g/14</td>
<td id="S7.T7.2.10.2" class="ltx_td ltx_border_bb"></td>
<td id="S7.T7.2.10.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T7.2.10.3.1" class="ltx_text ltx_font_bold">81.6</span></td>
<td id="S7.T7.2.10.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T7.2.10.4.1" class="ltx_text ltx_font_bold">85.7</span></td>
<td id="S7.T7.2.10.5" class="ltx_td ltx_align_center ltx_border_bb">67.5</td>
<td id="S7.T7.2.10.6" class="ltx_td ltx_border_bb"></td>
<td id="S7.T7.2.10.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T7.2.10.7.1" class="ltx_text ltx_font_bold">78.4</span></td>
<td id="S7.T7.2.10.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T7.2.10.8.1" class="ltx_text ltx_font_bold">91.2</span></td>
<td id="S7.T7.2.10.9" class="ltx_td ltx_align_center ltx_border_bb">38.3</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S7.T7.4.1.1" class="ltx_text" style="font-size:90%;">Table 7</span>: </span><span id="S7.T7.5.2" class="ltx_text" style="font-size:90%;">
<span id="S7.T7.5.2.1" class="ltx_text ltx_font_bold">Linear evaluation on other image and video classification.</span>
The image benchmarks contain a large quantity of fine-grained examples about objects or scenes.
The video benchmarks cover action classification and human-object interaction.
All the features are frozen with a linear probe on top.
</span></figcaption>
</figure>
<div id="S7.SS2.p2" class="ltx_para ltx_noindent">
<p id="S7.SS2.p2.4" class="ltx_p">In a second set of evaluations, we measure the performance of our model on video action recognition even though our features were not trained on videos..
We evaluated features on three datasets, namely UCF-101 <cite class="ltx_cite ltx_citemacro_citep">(Soomro et al., <a href="#bib.bib107" title="" class="ltx_ref">2012</a>)</cite>, Kinetics-400 <cite class="ltx_cite ltx_citemacro_citep">(Kay et al., <a href="#bib.bib70" title="" class="ltx_ref">2017</a>)</cite> and Something-Something v2 <cite class="ltx_cite ltx_citemacro_citep">(Goyal et al., <a href="#bib.bib52" title="" class="ltx_ref">2017</a>)</cite>.
For this evaluation, we pick <math id="S7.SS2.p2.1.m1.1" class="ltx_Math" alttext="8" display="inline"><semantics id="S7.SS2.p2.1.m1.1a"><mn id="S7.SS2.p2.1.m1.1.1" xref="S7.SS2.p2.1.m1.1.1.cmml">8</mn><annotation-xml encoding="MathML-Content" id="S7.SS2.p2.1.m1.1b"><cn type="integer" id="S7.SS2.p2.1.m1.1.1.cmml" xref="S7.SS2.p2.1.m1.1.1">8</cn></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p2.1.m1.1c">8</annotation></semantics></math> evenly spaced frames in the video and train a linear classifier on the average of the features for UCF and K-400.
For SSv2, we opt for concatenation to retain more temporal information than with feature averaging.
For each dataset, we measure average accuracy and report the results in Table <a href="#S7.T7" title="Table 7 ‣ 7.2 Additional Image and Video classification Benchmarks ‣ 7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>.
We see that amongst self-supervised approaches, our model clearly sets a new state of the art.
Moreover, our model matches the accuracy of the OpenCLIP features on UCF and Kinetics (<math id="S7.SS2.p2.2.m2.1" class="ltx_Math" alttext="+0.1\%" display="inline"><semantics id="S7.SS2.p2.2.m2.1a"><mrow id="S7.SS2.p2.2.m2.1.1" xref="S7.SS2.p2.2.m2.1.1.cmml"><mo id="S7.SS2.p2.2.m2.1.1a" xref="S7.SS2.p2.2.m2.1.1.cmml">+</mo><mrow id="S7.SS2.p2.2.m2.1.1.2" xref="S7.SS2.p2.2.m2.1.1.2.cmml"><mn id="S7.SS2.p2.2.m2.1.1.2.2" xref="S7.SS2.p2.2.m2.1.1.2.2.cmml">0.1</mn><mo id="S7.SS2.p2.2.m2.1.1.2.1" xref="S7.SS2.p2.2.m2.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS2.p2.2.m2.1b"><apply id="S7.SS2.p2.2.m2.1.1.cmml" xref="S7.SS2.p2.2.m2.1.1"><plus id="S7.SS2.p2.2.m2.1.1.1.cmml" xref="S7.SS2.p2.2.m2.1.1"></plus><apply id="S7.SS2.p2.2.m2.1.1.2.cmml" xref="S7.SS2.p2.2.m2.1.1.2"><csymbol cd="latexml" id="S7.SS2.p2.2.m2.1.1.2.1.cmml" xref="S7.SS2.p2.2.m2.1.1.2.1">percent</csymbol><cn type="float" id="S7.SS2.p2.2.m2.1.1.2.2.cmml" xref="S7.SS2.p2.2.m2.1.1.2.2">0.1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p2.2.m2.1c">+0.1\%</annotation></semantics></math> and <math id="S7.SS2.p2.3.m3.1" class="ltx_Math" alttext="+0.5\%" display="inline"><semantics id="S7.SS2.p2.3.m3.1a"><mrow id="S7.SS2.p2.3.m3.1.1" xref="S7.SS2.p2.3.m3.1.1.cmml"><mo id="S7.SS2.p2.3.m3.1.1a" xref="S7.SS2.p2.3.m3.1.1.cmml">+</mo><mrow id="S7.SS2.p2.3.m3.1.1.2" xref="S7.SS2.p2.3.m3.1.1.2.cmml"><mn id="S7.SS2.p2.3.m3.1.1.2.2" xref="S7.SS2.p2.3.m3.1.1.2.2.cmml">0.5</mn><mo id="S7.SS2.p2.3.m3.1.1.2.1" xref="S7.SS2.p2.3.m3.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS2.p2.3.m3.1b"><apply id="S7.SS2.p2.3.m3.1.1.cmml" xref="S7.SS2.p2.3.m3.1.1"><plus id="S7.SS2.p2.3.m3.1.1.1.cmml" xref="S7.SS2.p2.3.m3.1.1"></plus><apply id="S7.SS2.p2.3.m3.1.1.2.cmml" xref="S7.SS2.p2.3.m3.1.1.2"><csymbol cd="latexml" id="S7.SS2.p2.3.m3.1.1.2.1.cmml" xref="S7.SS2.p2.3.m3.1.1.2.1">percent</csymbol><cn type="float" id="S7.SS2.p2.3.m3.1.1.2.2.cmml" xref="S7.SS2.p2.3.m3.1.1.2.2">0.5</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p2.3.m3.1c">+0.5\%</annotation></semantics></math> respectively) and clearly outperforms them on SSv2 (<math id="S7.SS2.p2.4.m4.1" class="ltx_Math" alttext="+2.5\%" display="inline"><semantics id="S7.SS2.p2.4.m4.1a"><mrow id="S7.SS2.p2.4.m4.1.1" xref="S7.SS2.p2.4.m4.1.1.cmml"><mo id="S7.SS2.p2.4.m4.1.1a" xref="S7.SS2.p2.4.m4.1.1.cmml">+</mo><mrow id="S7.SS2.p2.4.m4.1.1.2" xref="S7.SS2.p2.4.m4.1.1.2.cmml"><mn id="S7.SS2.p2.4.m4.1.1.2.2" xref="S7.SS2.p2.4.m4.1.1.2.2.cmml">2.5</mn><mo id="S7.SS2.p2.4.m4.1.1.2.1" xref="S7.SS2.p2.4.m4.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS2.p2.4.m4.1b"><apply id="S7.SS2.p2.4.m4.1.1.cmml" xref="S7.SS2.p2.4.m4.1.1"><plus id="S7.SS2.p2.4.m4.1.1.1.cmml" xref="S7.SS2.p2.4.m4.1.1"></plus><apply id="S7.SS2.p2.4.m4.1.1.2.cmml" xref="S7.SS2.p2.4.m4.1.1.2"><csymbol cd="latexml" id="S7.SS2.p2.4.m4.1.1.2.1.cmml" xref="S7.SS2.p2.4.m4.1.1.2.1">percent</csymbol><cn type="float" id="S7.SS2.p2.4.m4.1.1.2.2.cmml" xref="S7.SS2.p2.4.m4.1.1.2.2">2.5</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p2.4.m4.1c">+2.5\%</annotation></semantics></math>).
This is particularly interesting, as SSv2 requires a much richer understanding of the video frames.</p>
</div>
<div id="S7.SS2.p3" class="ltx_para ltx_noindent">
<p id="S7.SS2.p3.4" class="ltx_p">Finally, in Table <a href="#S7.T8" title="Table 8 ‣ 7.2 Additional Image and Video classification Benchmarks ‣ 7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>, we compare selected frozen features on 12 transfer classification benchmarks initially proposed by <cite class="ltx_cite ltx_citemacro_citet">Chen et al. (<a href="#bib.bib21" title="" class="ltx_ref">2020</a>)</cite>.
This benchmark covers scenes, objects (food, cars, planes), and textures.
We replace the Birdsnap dataset with CUB because the former was not publicly available in its entirety.
We follow the experimental protocol as outlined by <cite class="ltx_cite ltx_citemacro_citet">Chen et al. (<a href="#bib.bib21" title="" class="ltx_ref">2020</a>)</cite>, namely training a logistic regression on precomputed features.
Our model significantly outperforms state-of-the-art SSL models, with most notable differences on Stanford Cars (<math id="S7.SS2.p3.1.m1.1" class="ltx_Math" alttext="+14.8\%" display="inline"><semantics id="S7.SS2.p3.1.m1.1a"><mrow id="S7.SS2.p3.1.m1.1.1" xref="S7.SS2.p3.1.m1.1.1.cmml"><mo id="S7.SS2.p3.1.m1.1.1a" xref="S7.SS2.p3.1.m1.1.1.cmml">+</mo><mrow id="S7.SS2.p3.1.m1.1.1.2" xref="S7.SS2.p3.1.m1.1.1.2.cmml"><mn id="S7.SS2.p3.1.m1.1.1.2.2" xref="S7.SS2.p3.1.m1.1.1.2.2.cmml">14.8</mn><mo id="S7.SS2.p3.1.m1.1.1.2.1" xref="S7.SS2.p3.1.m1.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS2.p3.1.m1.1b"><apply id="S7.SS2.p3.1.m1.1.1.cmml" xref="S7.SS2.p3.1.m1.1.1"><plus id="S7.SS2.p3.1.m1.1.1.1.cmml" xref="S7.SS2.p3.1.m1.1.1"></plus><apply id="S7.SS2.p3.1.m1.1.1.2.cmml" xref="S7.SS2.p3.1.m1.1.1.2"><csymbol cd="latexml" id="S7.SS2.p3.1.m1.1.1.2.1.cmml" xref="S7.SS2.p3.1.m1.1.1.2.1">percent</csymbol><cn type="float" id="S7.SS2.p3.1.m1.1.1.2.2.cmml" xref="S7.SS2.p3.1.m1.1.1.2.2">14.8</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p3.1.m1.1c">+14.8\%</annotation></semantics></math> versus DINO ViT-B/8) and FGVC Aircraft (<math id="S7.SS2.p3.2.m2.1" class="ltx_Math" alttext="+14.8\%" display="inline"><semantics id="S7.SS2.p3.2.m2.1a"><mrow id="S7.SS2.p3.2.m2.1.1" xref="S7.SS2.p3.2.m2.1.1.cmml"><mo id="S7.SS2.p3.2.m2.1.1a" xref="S7.SS2.p3.2.m2.1.1.cmml">+</mo><mrow id="S7.SS2.p3.2.m2.1.1.2" xref="S7.SS2.p3.2.m2.1.1.2.cmml"><mn id="S7.SS2.p3.2.m2.1.1.2.2" xref="S7.SS2.p3.2.m2.1.1.2.2.cmml">14.8</mn><mo id="S7.SS2.p3.2.m2.1.1.2.1" xref="S7.SS2.p3.2.m2.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS2.p3.2.m2.1b"><apply id="S7.SS2.p3.2.m2.1.1.cmml" xref="S7.SS2.p3.2.m2.1.1"><plus id="S7.SS2.p3.2.m2.1.1.1.cmml" xref="S7.SS2.p3.2.m2.1.1"></plus><apply id="S7.SS2.p3.2.m2.1.1.2.cmml" xref="S7.SS2.p3.2.m2.1.1.2"><csymbol cd="latexml" id="S7.SS2.p3.2.m2.1.1.2.1.cmml" xref="S7.SS2.p3.2.m2.1.1.2.1">percent</csymbol><cn type="float" id="S7.SS2.p3.2.m2.1.1.2.2.cmml" xref="S7.SS2.p3.2.m2.1.1.2.2">14.8</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p3.2.m2.1c">+14.8\%</annotation></semantics></math> versus iBOT ViT-L/16).
Even though these benchmarks favor text-guided pretraining, our features are still competitive with OpenCLIP on most classification benchmarks, with the exception of a few datasets, especially SUN (<math id="S7.SS2.p3.3.m3.1" class="ltx_Math" alttext="-5.3\%" display="inline"><semantics id="S7.SS2.p3.3.m3.1a"><mrow id="S7.SS2.p3.3.m3.1.1" xref="S7.SS2.p3.3.m3.1.1.cmml"><mo id="S7.SS2.p3.3.m3.1.1a" xref="S7.SS2.p3.3.m3.1.1.cmml">−</mo><mrow id="S7.SS2.p3.3.m3.1.1.2" xref="S7.SS2.p3.3.m3.1.1.2.cmml"><mn id="S7.SS2.p3.3.m3.1.1.2.2" xref="S7.SS2.p3.3.m3.1.1.2.2.cmml">5.3</mn><mo id="S7.SS2.p3.3.m3.1.1.2.1" xref="S7.SS2.p3.3.m3.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS2.p3.3.m3.1b"><apply id="S7.SS2.p3.3.m3.1.1.cmml" xref="S7.SS2.p3.3.m3.1.1"><minus id="S7.SS2.p3.3.m3.1.1.1.cmml" xref="S7.SS2.p3.3.m3.1.1"></minus><apply id="S7.SS2.p3.3.m3.1.1.2.cmml" xref="S7.SS2.p3.3.m3.1.1.2"><csymbol cd="latexml" id="S7.SS2.p3.3.m3.1.1.2.1.cmml" xref="S7.SS2.p3.3.m3.1.1.2.1">percent</csymbol><cn type="float" id="S7.SS2.p3.3.m3.1.1.2.2.cmml" xref="S7.SS2.p3.3.m3.1.1.2.2">5.3</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p3.3.m3.1c">-5.3\%</annotation></semantics></math>) and Cars (<math id="S7.SS2.p3.4.m4.1" class="ltx_Math" alttext="-4.7\%" display="inline"><semantics id="S7.SS2.p3.4.m4.1a"><mrow id="S7.SS2.p3.4.m4.1.1" xref="S7.SS2.p3.4.m4.1.1.cmml"><mo id="S7.SS2.p3.4.m4.1.1a" xref="S7.SS2.p3.4.m4.1.1.cmml">−</mo><mrow id="S7.SS2.p3.4.m4.1.1.2" xref="S7.SS2.p3.4.m4.1.1.2.cmml"><mn id="S7.SS2.p3.4.m4.1.1.2.2" xref="S7.SS2.p3.4.m4.1.1.2.2.cmml">4.7</mn><mo id="S7.SS2.p3.4.m4.1.1.2.1" xref="S7.SS2.p3.4.m4.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS2.p3.4.m4.1b"><apply id="S7.SS2.p3.4.m4.1.1.cmml" xref="S7.SS2.p3.4.m4.1.1"><minus id="S7.SS2.p3.4.m4.1.1.1.cmml" xref="S7.SS2.p3.4.m4.1.1"></minus><apply id="S7.SS2.p3.4.m4.1.1.2.cmml" xref="S7.SS2.p3.4.m4.1.1.2"><csymbol cd="latexml" id="S7.SS2.p3.4.m4.1.1.2.1.cmml" xref="S7.SS2.p3.4.m4.1.1.2.1">percent</csymbol><cn type="float" id="S7.SS2.p3.4.m4.1.1.2.2.cmml" xref="S7.SS2.p3.4.m4.1.1.2.2">4.7</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS2.p3.4.m4.1c">-4.7\%</annotation></semantics></math>).</p>
</div>
<figure id="S7.T8" class="ltx_table">
<table id="S7.T8.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S7.T8.2.1" class="ltx_tr">
<td id="S7.T8.2.1.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;">Feature</td>
<td id="S7.T8.2.1.2" class="ltx_td ltx_nopad_l ltx_align_left ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;">Arch</td>
<td id="S7.T8.2.1.3" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.1.3.1" class="ltx_text" style="font-size:70%;">Food</span></td>
<td id="S7.T8.2.1.4" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.1.4.1" class="ltx_text" style="font-size:70%;">C10</span></td>
<td id="S7.T8.2.1.5" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.1.5.1" class="ltx_text" style="font-size:70%;">C100</span></td>
<td id="S7.T8.2.1.6" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.1.6.1" class="ltx_text" style="font-size:70%;">SUN</span></td>
<td id="S7.T8.2.1.7" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.1.7.1" class="ltx_text" style="font-size:70%;">Cars</span></td>
<td id="S7.T8.2.1.8" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.1.8.1" class="ltx_text" style="font-size:70%;">Aircr</span></td>
<td id="S7.T8.2.1.9" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.1.9.1" class="ltx_text" style="font-size:70%;">VOC</span></td>
<td id="S7.T8.2.1.10" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.1.10.1" class="ltx_text" style="font-size:70%;">DTD</span></td>
<td id="S7.T8.2.1.11" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.1.11.1" class="ltx_text" style="font-size:70%;">Pets</span></td>
<td id="S7.T8.2.1.12" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.1.12.1" class="ltx_text" style="font-size:70%;">Cal101</span></td>
<td id="S7.T8.2.1.13" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.1.13.1" class="ltx_text" style="font-size:70%;">Flowers</span></td>
<td id="S7.T8.2.1.14" class="ltx_td ltx_align_center ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.1.14.1" class="ltx_text" style="font-size:70%;">CUB</span></td>
<td id="S7.T8.2.1.15" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_tt" style="padding-left:4.0pt;padding-right:4.0pt;">Avg</td>
</tr>
<tr id="S7.T8.2.2" class="ltx_tr">
<td id="S7.T8.2.2.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">OpenCLIP</td>
<td id="S7.T8.2.2.2" class="ltx_td ltx_nopad_l ltx_align_left ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">ViT-G/14</td>
<td id="S7.T8.2.2.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">94.5</td>
<td id="S7.T8.2.2.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">98.7</td>
<td id="S7.T8.2.2.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">91.0</td>
<td id="S7.T8.2.2.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.2.6.1" class="ltx_text ltx_font_bold">84.0</span></td>
<td id="S7.T8.2.2.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.2.7.1" class="ltx_text ltx_font_bold">96.1</span></td>
<td id="S7.T8.2.2.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">80.2</td>
<td id="S7.T8.2.2.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.2.9.1" class="ltx_text ltx_font_bold">89.3</span></td>
<td id="S7.T8.2.2.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.2.10.1" class="ltx_text ltx_font_bold">86.0</span></td>
<td id="S7.T8.2.2.11" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">95.7</td>
<td id="S7.T8.2.2.12" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.2.12.1" class="ltx_text ltx_font_bold">98.1</span></td>
<td id="S7.T8.2.2.13" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">99.5</td>
<td id="S7.T8.2.2.14" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">89.9</td>
<td id="S7.T8.2.2.15" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">91.9</td>
</tr>
<tr id="S7.T8.2.3" class="ltx_tr">
<td id="S7.T8.2.3.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">MAE</td>
<td id="S7.T8.2.3.2" class="ltx_td ltx_nopad_l ltx_align_left ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">ViT-H/14</td>
<td id="S7.T8.2.3.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">78.4</td>
<td id="S7.T8.2.3.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">96.1</td>
<td id="S7.T8.2.3.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">83.9</td>
<td id="S7.T8.2.3.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">63.9</td>
<td id="S7.T8.2.3.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">56.1</td>
<td id="S7.T8.2.3.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">63.4</td>
<td id="S7.T8.2.3.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">84.3</td>
<td id="S7.T8.2.3.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">75.4</td>
<td id="S7.T8.2.3.11" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">89.4</td>
<td id="S7.T8.2.3.12" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">95.9</td>
<td id="S7.T8.2.3.13" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">92.3</td>
<td id="S7.T8.2.3.14" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">57.2</td>
<td id="S7.T8.2.3.15" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">78.0</td>
</tr>
<tr id="S7.T8.2.4" class="ltx_tr">
<td id="S7.T8.2.4.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:4.0pt;padding-right:4.0pt;">DINO</td>
<td id="S7.T8.2.4.2" class="ltx_td ltx_nopad_l ltx_align_left" style="padding-left:4.0pt;padding-right:4.0pt;">ViT-B/8</td>
<td id="S7.T8.2.4.3" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">85.1</td>
<td id="S7.T8.2.4.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">97.2</td>
<td id="S7.T8.2.4.5" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">86.9</td>
<td id="S7.T8.2.4.6" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">70.3</td>
<td id="S7.T8.2.4.7" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">76.6</td>
<td id="S7.T8.2.4.8" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">70.6</td>
<td id="S7.T8.2.4.9" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">86.7</td>
<td id="S7.T8.2.4.10" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">79.6</td>
<td id="S7.T8.2.4.11" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">93.2</td>
<td id="S7.T8.2.4.12" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">95.4</td>
<td id="S7.T8.2.4.13" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">97.6</td>
<td id="S7.T8.2.4.14" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">81.7</td>
<td id="S7.T8.2.4.15" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">85.1</td>
</tr>
<tr id="S7.T8.2.5" class="ltx_tr">
<td id="S7.T8.2.5.1" class="ltx_td ltx_nopad_r ltx_align_left" style="padding-left:4.0pt;padding-right:4.0pt;">iBOT</td>
<td id="S7.T8.2.5.2" class="ltx_td ltx_nopad_l ltx_align_left" style="padding-left:4.0pt;padding-right:4.0pt;">ViT-L/16</td>
<td id="S7.T8.2.5.3" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">91.0</td>
<td id="S7.T8.2.5.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">99.0</td>
<td id="S7.T8.2.5.5" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">92.8</td>
<td id="S7.T8.2.5.6" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">75.6</td>
<td id="S7.T8.2.5.7" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">71.8</td>
<td id="S7.T8.2.5.8" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">72.4</td>
<td id="S7.T8.2.5.9" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">89.0</td>
<td id="S7.T8.2.5.10" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">80.7</td>
<td id="S7.T8.2.5.11" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">87.7</td>
<td id="S7.T8.2.5.12" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">97.5</td>
<td id="S7.T8.2.5.13" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">99.6</td>
<td id="S7.T8.2.5.14" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">82.1</td>
<td id="S7.T8.2.5.15" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">86.6</td>
</tr>
<tr id="S7.T8.2.6" class="ltx_tr">
<td id="S7.T8.2.6.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;" rowspan="4"><span id="S7.T8.2.6.1.1" class="ltx_text">DINOv2</span></td>
<td id="S7.T8.2.6.2" class="ltx_td ltx_nopad_l ltx_align_left ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">ViT-S/14</td>
<td id="S7.T8.2.6.3" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">89.1</td>
<td id="S7.T8.2.6.4" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">97.7</td>
<td id="S7.T8.2.6.5" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">87.5</td>
<td id="S7.T8.2.6.6" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">74.4</td>
<td id="S7.T8.2.6.7" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">81.6</td>
<td id="S7.T8.2.6.8" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">74.0</td>
<td id="S7.T8.2.6.9" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">87.8</td>
<td id="S7.T8.2.6.10" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">80.6</td>
<td id="S7.T8.2.6.11" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">95.1</td>
<td id="S7.T8.2.6.12" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">97.0</td>
<td id="S7.T8.2.6.13" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">99.6</td>
<td id="S7.T8.2.6.14" class="ltx_td ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">88.1</td>
<td id="S7.T8.2.6.15" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t" style="padding-left:4.0pt;padding-right:4.0pt;">87.7</td>
</tr>
<tr id="S7.T8.2.7" class="ltx_tr">
<td id="S7.T8.2.7.1" class="ltx_td ltx_nopad_l ltx_align_left" style="padding-left:4.0pt;padding-right:4.0pt;">ViT-B/14</td>
<td id="S7.T8.2.7.2" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">92.8</td>
<td id="S7.T8.2.7.3" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">98.7</td>
<td id="S7.T8.2.7.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">91.3</td>
<td id="S7.T8.2.7.5" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">77.3</td>
<td id="S7.T8.2.7.6" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">88.2</td>
<td id="S7.T8.2.7.7" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">79.4</td>
<td id="S7.T8.2.7.8" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">88.2</td>
<td id="S7.T8.2.7.9" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">83.3</td>
<td id="S7.T8.2.7.10" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">96.2</td>
<td id="S7.T8.2.7.11" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">96.1</td>
<td id="S7.T8.2.7.12" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">99.6</td>
<td id="S7.T8.2.7.13" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">89.6</td>
<td id="S7.T8.2.7.14" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">90.1</td>
</tr>
<tr id="S7.T8.2.8" class="ltx_tr">
<td id="S7.T8.2.8.1" class="ltx_td ltx_nopad_l ltx_align_left" style="padding-left:4.0pt;padding-right:4.0pt;">ViT-L/14</td>
<td id="S7.T8.2.8.2" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">94.3</td>
<td id="S7.T8.2.8.3" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">99.3</td>
<td id="S7.T8.2.8.4" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">93.4</td>
<td id="S7.T8.2.8.5" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">78.7</td>
<td id="S7.T8.2.8.6" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">90.1</td>
<td id="S7.T8.2.8.7" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">81.5</td>
<td id="S7.T8.2.8.8" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">88.3</td>
<td id="S7.T8.2.8.9" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">84.0</td>
<td id="S7.T8.2.8.10" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">96.6</td>
<td id="S7.T8.2.8.11" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">97.5</td>
<td id="S7.T8.2.8.12" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">99.7</td>
<td id="S7.T8.2.8.13" class="ltx_td ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">90.5</td>
<td id="S7.T8.2.8.14" class="ltx_td ltx_nopad_r ltx_align_center" style="padding-left:4.0pt;padding-right:4.0pt;">91.2</td>
</tr>
<tr id="S7.T8.2.9" class="ltx_tr">
<td id="S7.T8.2.9.1" class="ltx_td ltx_nopad_l ltx_align_left ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;">ViT-g/14</td>
<td id="S7.T8.2.9.2" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.9.2.1" class="ltx_text ltx_font_bold">94.7</span></td>
<td id="S7.T8.2.9.3" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.9.3.1" class="ltx_text ltx_font_bold">99.5</span></td>
<td id="S7.T8.2.9.4" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.9.4.1" class="ltx_text ltx_font_bold">94.4</span></td>
<td id="S7.T8.2.9.5" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;">78.7</td>
<td id="S7.T8.2.9.6" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;">91.4</td>
<td id="S7.T8.2.9.7" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.9.7.1" class="ltx_text ltx_font_bold">87.2</span></td>
<td id="S7.T8.2.9.8" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;">89.0</td>
<td id="S7.T8.2.9.9" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;">84.5</td>
<td id="S7.T8.2.9.10" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.9.10.1" class="ltx_text ltx_font_bold">96.7</span></td>
<td id="S7.T8.2.9.11" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;">97.6</td>
<td id="S7.T8.2.9.12" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.9.12.1" class="ltx_text ltx_font_bold">99.7</span></td>
<td id="S7.T8.2.9.13" class="ltx_td ltx_align_center ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.9.13.1" class="ltx_text ltx_font_bold">91.6</span></td>
<td id="S7.T8.2.9.14" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb" style="padding-left:4.0pt;padding-right:4.0pt;"><span id="S7.T8.2.9.14.1" class="ltx_text ltx_font_bold">92.1</span></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S7.T8.4.1.1" class="ltx_text" style="font-size:90%;">Table 8</span>: </span><span id="S7.T8.5.2" class="ltx_text" style="font-size:90%;">
<span id="S7.T8.5.2.1" class="ltx_text ltx_font_bold">Linear evaluation of frozen features on fine-grained benchmarks.</span>
Accuracy on 12 benchmarks covering objects, scenes and textures following the evaluation protocol proposed in <cite class="ltx_cite ltx_citemacro_citet">Chen et al. (<a href="#bib.bib21" title="" class="ltx_ref">2020</a>)</cite>.
</span></figcaption>
</figure>
</section>
<section id="S7.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">7.3 </span>Instance Recognition</h3>

<div id="S7.SS3.p1" class="ltx_para ltx_noindent">
<p id="S7.SS3.p1.2" class="ltx_p">In this experiment, we probe our model on the task of instance-level recognition using a non-parametric approach.
Images from a database are ranked according to their cosine similarity with a query image.
We evaluated our model and compare to baselines on Paris and Oxford, that are landmark recognition benchmarks.
We also evaluated on Met, a dataset of artworks from the Metropolitan museum, and AmsterTime, containing street view images matched to archival images of Amsterdam.
We measure performance by computing the mean average precision and report our results in Table <a href="#S7.T9" title="Table 9 ‣ 7.3 Instance Recognition ‣ 7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>.
We see that our features significantly outperform both SSL (<math id="S7.SS3.p1.1.m1.1" class="ltx_Math" alttext="+41\%" display="inline"><semantics id="S7.SS3.p1.1.m1.1a"><mrow id="S7.SS3.p1.1.m1.1.1" xref="S7.SS3.p1.1.m1.1.1.cmml"><mo id="S7.SS3.p1.1.m1.1.1a" xref="S7.SS3.p1.1.m1.1.1.cmml">+</mo><mrow id="S7.SS3.p1.1.m1.1.1.2" xref="S7.SS3.p1.1.m1.1.1.2.cmml"><mn id="S7.SS3.p1.1.m1.1.1.2.2" xref="S7.SS3.p1.1.m1.1.1.2.2.cmml">41</mn><mo id="S7.SS3.p1.1.m1.1.1.2.1" xref="S7.SS3.p1.1.m1.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS3.p1.1.m1.1b"><apply id="S7.SS3.p1.1.m1.1.1.cmml" xref="S7.SS3.p1.1.m1.1.1"><plus id="S7.SS3.p1.1.m1.1.1.1.cmml" xref="S7.SS3.p1.1.m1.1.1"></plus><apply id="S7.SS3.p1.1.m1.1.1.2.cmml" xref="S7.SS3.p1.1.m1.1.1.2"><csymbol cd="latexml" id="S7.SS3.p1.1.m1.1.1.2.1.cmml" xref="S7.SS3.p1.1.m1.1.1.2.1">percent</csymbol><cn type="integer" id="S7.SS3.p1.1.m1.1.1.2.2.cmml" xref="S7.SS3.p1.1.m1.1.1.2.2">41</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS3.p1.1.m1.1c">+41\%</annotation></semantics></math> mAP on Oxford-Hard), and weakly-supervised (<math id="S7.SS3.p1.2.m2.1" class="ltx_Math" alttext="+34\%" display="inline"><semantics id="S7.SS3.p1.2.m2.1a"><mrow id="S7.SS3.p1.2.m2.1.1" xref="S7.SS3.p1.2.m2.1.1.cmml"><mo id="S7.SS3.p1.2.m2.1.1a" xref="S7.SS3.p1.2.m2.1.1.cmml">+</mo><mrow id="S7.SS3.p1.2.m2.1.1.2" xref="S7.SS3.p1.2.m2.1.1.2.cmml"><mn id="S7.SS3.p1.2.m2.1.1.2.2" xref="S7.SS3.p1.2.m2.1.1.2.2.cmml">34</mn><mo id="S7.SS3.p1.2.m2.1.1.2.1" xref="S7.SS3.p1.2.m2.1.1.2.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS3.p1.2.m2.1b"><apply id="S7.SS3.p1.2.m2.1.1.cmml" xref="S7.SS3.p1.2.m2.1.1"><plus id="S7.SS3.p1.2.m2.1.1.1.cmml" xref="S7.SS3.p1.2.m2.1.1"></plus><apply id="S7.SS3.p1.2.m2.1.1.2.cmml" xref="S7.SS3.p1.2.m2.1.1.2"><csymbol cd="latexml" id="S7.SS3.p1.2.m2.1.1.2.1.cmml" xref="S7.SS3.p1.2.m2.1.1.2.1">percent</csymbol><cn type="integer" id="S7.SS3.p1.2.m2.1.1.2.2.cmml" xref="S7.SS3.p1.2.m2.1.1.2.2">34</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS3.p1.2.m2.1c">+34\%</annotation></semantics></math> mAP on Oxford-Hard) ones.
It is interesting to see that our features perform well across task granularities, both at the category-level and instance-level.
This is a desirable property for strong off-the-shelf computer vision features.</p>
</div>
<figure id="S7.T9" class="ltx_table">
<table id="S7.T9.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S7.T9.2.1" class="ltx_tr">
<td id="S7.T9.2.1.1" class="ltx_td ltx_border_tt"></td>
<td id="S7.T9.2.1.2" class="ltx_td ltx_border_tt"></td>
<td id="S7.T9.2.1.3" class="ltx_td ltx_border_tt"></td>
<td id="S7.T9.2.1.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="2">Oxford</td>
<td id="S7.T9.2.1.5" class="ltx_td ltx_border_tt"></td>
<td id="S7.T9.2.1.6" class="ltx_td ltx_align_center ltx_border_tt" colspan="2">Paris</td>
<td id="S7.T9.2.1.7" class="ltx_td ltx_border_tt"></td>
<td id="S7.T9.2.1.8" class="ltx_td ltx_align_center ltx_border_tt" colspan="3">Met</td>
<td id="S7.T9.2.1.9" class="ltx_td ltx_border_tt"></td>
<td id="S7.T9.2.1.10" class="ltx_td ltx_align_center ltx_border_tt">AmsterTime</td>
</tr>
<tr id="S7.T9.2.2" class="ltx_tr">
<td id="S7.T9.2.2.1" class="ltx_td ltx_align_left">Feature</td>
<td id="S7.T9.2.2.2" class="ltx_td ltx_align_left">Arch</td>
<td id="S7.T9.2.2.3" class="ltx_td"></td>
<td id="S7.T9.2.2.4" class="ltx_td ltx_align_center ltx_border_t">M</td>
<td id="S7.T9.2.2.5" class="ltx_td ltx_align_center ltx_border_t">H</td>
<td id="S7.T9.2.2.6" class="ltx_td"></td>
<td id="S7.T9.2.2.7" class="ltx_td ltx_align_center ltx_border_t">M</td>
<td id="S7.T9.2.2.8" class="ltx_td ltx_align_center ltx_border_t">H</td>
<td id="S7.T9.2.2.9" class="ltx_td"></td>
<td id="S7.T9.2.2.10" class="ltx_td ltx_align_center ltx_border_t">GAP</td>
<td id="S7.T9.2.2.11" class="ltx_td ltx_align_center ltx_border_t">GAP-</td>
<td id="S7.T9.2.2.12" class="ltx_td ltx_align_center ltx_border_t">ACC</td>
<td id="S7.T9.2.2.13" class="ltx_td"></td>
<td id="S7.T9.2.2.14" class="ltx_td ltx_align_center ltx_border_t">mAP</td>
</tr>
<tr id="S7.T9.2.3" class="ltx_tr">
<td id="S7.T9.2.3.1" class="ltx_td ltx_align_left ltx_border_t">OpenCLIP</td>
<td id="S7.T9.2.3.2" class="ltx_td ltx_align_left ltx_border_t">ViT-G/14</td>
<td id="S7.T9.2.3.3" class="ltx_td ltx_border_t"></td>
<td id="S7.T9.2.3.4" class="ltx_td ltx_align_center ltx_border_t">50.7</td>
<td id="S7.T9.2.3.5" class="ltx_td ltx_align_center ltx_border_t">19.7</td>
<td id="S7.T9.2.3.6" class="ltx_td ltx_border_t"></td>
<td id="S7.T9.2.3.7" class="ltx_td ltx_align_center ltx_border_t">79.2</td>
<td id="S7.T9.2.3.8" class="ltx_td ltx_align_center ltx_border_t">60.2</td>
<td id="S7.T9.2.3.9" class="ltx_td ltx_border_t"></td>
<td id="S7.T9.2.3.10" class="ltx_td ltx_align_center ltx_border_t">6.5</td>
<td id="S7.T9.2.3.11" class="ltx_td ltx_align_center ltx_border_t">23.9</td>
<td id="S7.T9.2.3.12" class="ltx_td ltx_align_center ltx_border_t">34.4</td>
<td id="S7.T9.2.3.13" class="ltx_td ltx_border_t"></td>
<td id="S7.T9.2.3.14" class="ltx_td ltx_align_center ltx_border_t">24.6</td>
</tr>
<tr id="S7.T9.2.4" class="ltx_tr">
<td id="S7.T9.2.4.1" class="ltx_td ltx_align_left ltx_border_t">MAE</td>
<td id="S7.T9.2.4.2" class="ltx_td ltx_align_left ltx_border_t">ViT-H/14</td>
<td id="S7.T9.2.4.3" class="ltx_td ltx_border_t"></td>
<td id="S7.T9.2.4.4" class="ltx_td ltx_align_center ltx_border_t">11.7</td>
<td id="S7.T9.2.4.5" class="ltx_td ltx_align_center ltx_border_t">2.2</td>
<td id="S7.T9.2.4.6" class="ltx_td ltx_border_t"></td>
<td id="S7.T9.2.4.7" class="ltx_td ltx_align_center ltx_border_t">19.9</td>
<td id="S7.T9.2.4.8" class="ltx_td ltx_align_center ltx_border_t">4.7</td>
<td id="S7.T9.2.4.9" class="ltx_td ltx_border_t"></td>
<td id="S7.T9.2.4.10" class="ltx_td ltx_align_center ltx_border_t">7.5</td>
<td id="S7.T9.2.4.11" class="ltx_td ltx_align_center ltx_border_t">23.5</td>
<td id="S7.T9.2.4.12" class="ltx_td ltx_align_center ltx_border_t">30.5</td>
<td id="S7.T9.2.4.13" class="ltx_td ltx_border_t"></td>
<td id="S7.T9.2.4.14" class="ltx_td ltx_align_center ltx_border_t">4.2</td>
</tr>
<tr id="S7.T9.2.5" class="ltx_tr">
<td id="S7.T9.2.5.1" class="ltx_td ltx_align_left">DINO</td>
<td id="S7.T9.2.5.2" class="ltx_td ltx_align_left">ViT-B/8</td>
<td id="S7.T9.2.5.3" class="ltx_td"></td>
<td id="S7.T9.2.5.4" class="ltx_td ltx_align_center">40.1</td>
<td id="S7.T9.2.5.5" class="ltx_td ltx_align_center">13.7</td>
<td id="S7.T9.2.5.6" class="ltx_td"></td>
<td id="S7.T9.2.5.7" class="ltx_td ltx_align_center">65.3</td>
<td id="S7.T9.2.5.8" class="ltx_td ltx_align_center">35.3</td>
<td id="S7.T9.2.5.9" class="ltx_td"></td>
<td id="S7.T9.2.5.10" class="ltx_td ltx_align_center">17.1</td>
<td id="S7.T9.2.5.11" class="ltx_td ltx_align_center">37.7</td>
<td id="S7.T9.2.5.12" class="ltx_td ltx_align_center">43.9</td>
<td id="S7.T9.2.5.13" class="ltx_td"></td>
<td id="S7.T9.2.5.14" class="ltx_td ltx_align_center">24.6</td>
</tr>
<tr id="S7.T9.2.6" class="ltx_tr">
<td id="S7.T9.2.6.1" class="ltx_td ltx_align_left">iBOT</td>
<td id="S7.T9.2.6.2" class="ltx_td ltx_align_left">ViT-L/16</td>
<td id="S7.T9.2.6.3" class="ltx_td"></td>
<td id="S7.T9.2.6.4" class="ltx_td ltx_align_center">39.0</td>
<td id="S7.T9.2.6.5" class="ltx_td ltx_align_center">12.7</td>
<td id="S7.T9.2.6.6" class="ltx_td"></td>
<td id="S7.T9.2.6.7" class="ltx_td ltx_align_center">70.7</td>
<td id="S7.T9.2.6.8" class="ltx_td ltx_align_center">47.0</td>
<td id="S7.T9.2.6.9" class="ltx_td"></td>
<td id="S7.T9.2.6.10" class="ltx_td ltx_align_center">25.1</td>
<td id="S7.T9.2.6.11" class="ltx_td ltx_align_center">54.8</td>
<td id="S7.T9.2.6.12" class="ltx_td ltx_align_center">58.2</td>
<td id="S7.T9.2.6.13" class="ltx_td"></td>
<td id="S7.T9.2.6.14" class="ltx_td ltx_align_center">26.7</td>
</tr>
<tr id="S7.T9.2.7" class="ltx_tr">
<td id="S7.T9.2.7.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" rowspan="4"><span id="S7.T9.2.7.1.1" class="ltx_text">DINOv2</span></td>
<td id="S7.T9.2.7.2" class="ltx_td ltx_align_left ltx_border_t">ViT-S/14</td>
<td id="S7.T9.2.7.3" class="ltx_td ltx_border_t"></td>
<td id="S7.T9.2.7.4" class="ltx_td ltx_align_center ltx_border_t">68.8</td>
<td id="S7.T9.2.7.5" class="ltx_td ltx_align_center ltx_border_t">43.2</td>
<td id="S7.T9.2.7.6" class="ltx_td ltx_border_t"></td>
<td id="S7.T9.2.7.7" class="ltx_td ltx_align_center ltx_border_t">84.6</td>
<td id="S7.T9.2.7.8" class="ltx_td ltx_align_center ltx_border_t">68.5</td>
<td id="S7.T9.2.7.9" class="ltx_td ltx_border_t"></td>
<td id="S7.T9.2.7.10" class="ltx_td ltx_align_center ltx_border_t">29.4</td>
<td id="S7.T9.2.7.11" class="ltx_td ltx_align_center ltx_border_t">54.3</td>
<td id="S7.T9.2.7.12" class="ltx_td ltx_align_center ltx_border_t">57.7</td>
<td id="S7.T9.2.7.13" class="ltx_td ltx_border_t"></td>
<td id="S7.T9.2.7.14" class="ltx_td ltx_align_center ltx_border_t">43.5</td>
</tr>
<tr id="S7.T9.2.8" class="ltx_tr">
<td id="S7.T9.2.8.1" class="ltx_td ltx_align_left">ViT-B/14</td>
<td id="S7.T9.2.8.2" class="ltx_td"></td>
<td id="S7.T9.2.8.3" class="ltx_td ltx_align_center">72.9</td>
<td id="S7.T9.2.8.4" class="ltx_td ltx_align_center">49.5</td>
<td id="S7.T9.2.8.5" class="ltx_td"></td>
<td id="S7.T9.2.8.6" class="ltx_td ltx_align_center">90.3</td>
<td id="S7.T9.2.8.7" class="ltx_td ltx_align_center">78.6</td>
<td id="S7.T9.2.8.8" class="ltx_td"></td>
<td id="S7.T9.2.8.9" class="ltx_td ltx_align_center">36.7</td>
<td id="S7.T9.2.8.10" class="ltx_td ltx_align_center">63.5</td>
<td id="S7.T9.2.8.11" class="ltx_td ltx_align_center">66.1</td>
<td id="S7.T9.2.8.12" class="ltx_td"></td>
<td id="S7.T9.2.8.13" class="ltx_td ltx_align_center">45.6</td>
</tr>
<tr id="S7.T9.2.9" class="ltx_tr">
<td id="S7.T9.2.9.1" class="ltx_td ltx_align_left">ViT-L/14</td>
<td id="S7.T9.2.9.2" class="ltx_td"></td>
<td id="S7.T9.2.9.3" class="ltx_td ltx_align_center"><span id="S7.T9.2.9.3.1" class="ltx_text ltx_font_bold">75.1</span></td>
<td id="S7.T9.2.9.4" class="ltx_td ltx_align_center"><span id="S7.T9.2.9.4.1" class="ltx_text ltx_font_bold">54.0</span></td>
<td id="S7.T9.2.9.5" class="ltx_td"></td>
<td id="S7.T9.2.9.6" class="ltx_td ltx_align_center"><span id="S7.T9.2.9.6.1" class="ltx_text ltx_font_bold">92.7</span></td>
<td id="S7.T9.2.9.7" class="ltx_td ltx_align_center"><span id="S7.T9.2.9.7.1" class="ltx_text ltx_font_bold">83.5</span></td>
<td id="S7.T9.2.9.8" class="ltx_td"></td>
<td id="S7.T9.2.9.9" class="ltx_td ltx_align_center"><span id="S7.T9.2.9.9.1" class="ltx_text ltx_font_bold">40.0</span></td>
<td id="S7.T9.2.9.10" class="ltx_td ltx_align_center">68.9</td>
<td id="S7.T9.2.9.11" class="ltx_td ltx_align_center">71.6</td>
<td id="S7.T9.2.9.12" class="ltx_td"></td>
<td id="S7.T9.2.9.13" class="ltx_td ltx_align_center"><span id="S7.T9.2.9.13.1" class="ltx_text ltx_font_bold">50.0</span></td>
</tr>
<tr id="S7.T9.2.10" class="ltx_tr">
<td id="S7.T9.2.10.1" class="ltx_td ltx_align_left ltx_border_bb">ViT-g/14</td>
<td id="S7.T9.2.10.2" class="ltx_td ltx_border_bb"></td>
<td id="S7.T9.2.10.3" class="ltx_td ltx_align_center ltx_border_bb">73.6</td>
<td id="S7.T9.2.10.4" class="ltx_td ltx_align_center ltx_border_bb">52.3</td>
<td id="S7.T9.2.10.5" class="ltx_td ltx_border_bb"></td>
<td id="S7.T9.2.10.6" class="ltx_td ltx_align_center ltx_border_bb">92.1</td>
<td id="S7.T9.2.10.7" class="ltx_td ltx_align_center ltx_border_bb">82.6</td>
<td id="S7.T9.2.10.8" class="ltx_td ltx_border_bb"></td>
<td id="S7.T9.2.10.9" class="ltx_td ltx_align_center ltx_border_bb">36.8</td>
<td id="S7.T9.2.10.10" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T9.2.10.10.1" class="ltx_text ltx_font_bold">73.6</span></td>
<td id="S7.T9.2.10.11" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T9.2.10.11.1" class="ltx_text ltx_font_bold">76.5</span></td>
<td id="S7.T9.2.10.12" class="ltx_td ltx_border_bb"></td>
<td id="S7.T9.2.10.13" class="ltx_td ltx_align_center ltx_border_bb">46.7</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S7.T9.4.1.1" class="ltx_text" style="font-size:90%;">Table 9</span>: </span><span id="S7.T9.5.2" class="ltx_text" style="font-size:90%;">
<span id="S7.T9.5.2.1" class="ltx_text ltx_font_bold">Evaluation of frozen features on instance-level recognition.</span>
We consider 4 different benchmarks and report their main metrics.
</span></figcaption>
</figure>
</section>
<section id="S7.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">7.4 </span>Dense Recognition Tasks</h3>

<div id="S7.SS4.p1" class="ltx_para ltx_noindent">
<p id="S7.SS4.p1.1" class="ltx_p">We probe the quality of patch-level features extracted from our network on several dense downstream tasks.
We consider semantic image segmentation and monocular depth estimation in several settings and we conduct evaluations on multiple datasets for each.</p>
</div>
<section id="S7.SS4.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Semantic segmentation.</h5>

<div id="S7.SS4.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S7.SS4.SSS0.Px1.p1.1" class="ltx_p">For our semantic segmentation evaluation, we consider two different setups.
<span id="S7.SS4.SSS0.Px1.p1.1.1" class="ltx_text ltx_font_bold">Linear</span>: a linear layer is trained to predict class logits from a patch tokens.
It is used to produce a low-resolution logit map (eg 32x32 for a model with patch size 16), which is then upsampled to full resolution (512x512) to obtain a segmentation map.
This procedure is extremely simple but cannot easily produce high-resolution segmentations.
<span id="S7.SS4.SSS0.Px1.p1.1.2" class="ltx_text ltx_font_bold">+ms</span>: a boosted version of the linear setup.
We concatenate the patch tokens of the 4 last layers, use a larger image resolution of 640, and use multiscale test-time augmentations to improve the predictions.
We report the performance of our model variants as well as the baselines on three datasets under the two setups in Table <a href="#S7.T10" title="Table 10 ‣ Frozen backbone in a SOTA pipeline. ‣ 7.4 Dense Recognition Tasks ‣ 7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a>.</p>
</div>
<div id="S7.SS4.SSS0.Px1.p2" class="ltx_para ltx_noindent">
<p id="S7.SS4.SSS0.Px1.p2.4" class="ltx_p">Our models show very good performance on all datasets and for all setups.
Interestingly, our evaluation using <span id="S7.SS4.SSS0.Px1.p2.4.1" class="ltx_text ltx_font_bold">+ms</span> is on par with fully finetuning MAE with an Upernet decoder (<math id="S7.SS4.SSS0.Px1.p2.1.m1.1" class="ltx_Math" alttext="53.0" display="inline"><semantics id="S7.SS4.SSS0.Px1.p2.1.m1.1a"><mn id="S7.SS4.SSS0.Px1.p2.1.m1.1.1" xref="S7.SS4.SSS0.Px1.p2.1.m1.1.1.cmml">53.0</mn><annotation-xml encoding="MathML-Content" id="S7.SS4.SSS0.Px1.p2.1.m1.1b"><cn type="float" id="S7.SS4.SSS0.Px1.p2.1.m1.1.1.cmml" xref="S7.SS4.SSS0.Px1.p2.1.m1.1.1">53.0</cn></annotation-xml><annotation encoding="application/x-tex" id="S7.SS4.SSS0.Px1.p2.1.m1.1c">53.0</annotation></semantics></math> versus <math id="S7.SS4.SSS0.Px1.p2.2.m2.1" class="ltx_Math" alttext="53.6" display="inline"><semantics id="S7.SS4.SSS0.Px1.p2.2.m2.1a"><mn id="S7.SS4.SSS0.Px1.p2.2.m2.1.1" xref="S7.SS4.SSS0.Px1.p2.2.m2.1.1.cmml">53.6</mn><annotation-xml encoding="MathML-Content" id="S7.SS4.SSS0.Px1.p2.2.m2.1b"><cn type="float" id="S7.SS4.SSS0.Px1.p2.2.m2.1.1.cmml" xref="S7.SS4.SSS0.Px1.p2.2.m2.1.1">53.6</cn></annotation-xml><annotation encoding="application/x-tex" id="S7.SS4.SSS0.Px1.p2.2.m2.1c">53.6</annotation></semantics></math> mIoU).
This is surprising because we use a significantly simpler predictor.
Also, our best model, when evaluated using the boosted recipe, almost matches the state of the art on Pascal VOC (<math id="S7.SS4.SSS0.Px1.p2.3.m3.1" class="ltx_Math" alttext="86.2" display="inline"><semantics id="S7.SS4.SSS0.Px1.p2.3.m3.1a"><mn id="S7.SS4.SSS0.Px1.p2.3.m3.1.1" xref="S7.SS4.SSS0.Px1.p2.3.m3.1.1.cmml">86.2</mn><annotation-xml encoding="MathML-Content" id="S7.SS4.SSS0.Px1.p2.3.m3.1b"><cn type="float" id="S7.SS4.SSS0.Px1.p2.3.m3.1.1.cmml" xref="S7.SS4.SSS0.Px1.p2.3.m3.1.1">86.2</cn></annotation-xml><annotation encoding="application/x-tex" id="S7.SS4.SSS0.Px1.p2.3.m3.1c">86.2</annotation></semantics></math> versus <math id="S7.SS4.SSS0.Px1.p2.4.m4.1" class="ltx_Math" alttext="89.0" display="inline"><semantics id="S7.SS4.SSS0.Px1.p2.4.m4.1a"><mn id="S7.SS4.SSS0.Px1.p2.4.m4.1.1" xref="S7.SS4.SSS0.Px1.p2.4.m4.1.1.cmml">89.0</mn><annotation-xml encoding="MathML-Content" id="S7.SS4.SSS0.Px1.p2.4.m4.1b"><cn type="float" id="S7.SS4.SSS0.Px1.p2.4.m4.1.1.cmml" xref="S7.SS4.SSS0.Px1.p2.4.m4.1.1">89.0</cn></annotation-xml><annotation encoding="application/x-tex" id="S7.SS4.SSS0.Px1.p2.4.m4.1c">89.0</annotation></semantics></math> mIoU).</p>
</div>
</section>
<section id="S7.SS4.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Frozen backbone in a SOTA pipeline.</h5>

<div id="S7.SS4.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S7.SS4.SSS0.Px2.p1.1" class="ltx_p">In a final experiment, we freeze our backbone, and plug it into a ViT-Adapter <cite class="ltx_cite ltx_citemacro_cite">Chen et al. (<a href="#bib.bib25" title="" class="ltx_ref">2023b</a>)</cite> with a Mask2former head <cite class="ltx_cite ltx_citemacro_citep">(Cheng et al., <a href="#bib.bib26" title="" class="ltx_ref">2022</a>)</cite>.
We tune the weights of the adapter and head, but keep the backbone frozen, meaning 66% of the weights are frozen.
This allows for a lighter segmentation training than full end-to-end fine-tuning.
With this setup, we reach <math id="S7.SS4.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="60.2" display="inline"><semantics id="S7.SS4.SSS0.Px2.p1.1.m1.1a"><mn id="S7.SS4.SSS0.Px2.p1.1.m1.1.1" xref="S7.SS4.SSS0.Px2.p1.1.m1.1.1.cmml">60.2</mn><annotation-xml encoding="MathML-Content" id="S7.SS4.SSS0.Px2.p1.1.m1.1b"><cn type="float" id="S7.SS4.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S7.SS4.SSS0.Px2.p1.1.m1.1.1">60.2</cn></annotation-xml><annotation encoding="application/x-tex" id="S7.SS4.SSS0.Px2.p1.1.m1.1c">60.2</annotation></semantics></math> mIoU on ADE20k, close to the competitive state of the art, standing at 62.9 mIoU <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib119" title="" class="ltx_ref">2022</a>)</cite>.
Although our setup for this experiment doesn’t makes use of the optimisations described in Sec. <a href="#S5" title="5 Efficient implementation ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, the segmentation training in this experiment took 28 hours on 16 V100 GPUs.</p>
</div>
<figure id="S7.T10" class="ltx_table">
<table id="S7.T10.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S7.T10.2.1" class="ltx_tr">
<td id="S7.T10.2.1.1" class="ltx_td ltx_border_tt"></td>
<td id="S7.T10.2.1.2" class="ltx_td ltx_border_tt"></td>
<td id="S7.T10.2.1.3" class="ltx_td ltx_border_tt"></td>
<td id="S7.T10.2.1.4" class="ltx_td ltx_align_center ltx_border_tt" colspan="2">ADE20k</td>
<td id="S7.T10.2.1.5" class="ltx_td ltx_border_tt"></td>
<td id="S7.T10.2.1.6" class="ltx_td ltx_align_center ltx_border_tt" colspan="2">CityScapes</td>
<td id="S7.T10.2.1.7" class="ltx_td ltx_border_tt"></td>
<td id="S7.T10.2.1.8" class="ltx_td ltx_align_center ltx_border_tt" colspan="2">Pascal VOC</td>
</tr>
<tr id="S7.T10.2.2" class="ltx_tr">
<td id="S7.T10.2.2.1" class="ltx_td"></td>
<td id="S7.T10.2.2.2" class="ltx_td"></td>
<td id="S7.T10.2.2.3" class="ltx_td"></td>
<td id="S7.T10.2.2.4" class="ltx_td ltx_align_center" colspan="2">(62.9)</td>
<td id="S7.T10.2.2.5" class="ltx_td"></td>
<td id="S7.T10.2.2.6" class="ltx_td ltx_align_center" colspan="2">(86.9)</td>
<td id="S7.T10.2.2.7" class="ltx_td"></td>
<td id="S7.T10.2.2.8" class="ltx_td ltx_align_center" colspan="2">(89.0)</td>
</tr>
<tr id="S7.T10.2.3" class="ltx_tr">
<td id="S7.T10.2.3.1" class="ltx_td ltx_align_left">Method</td>
<td id="S7.T10.2.3.2" class="ltx_td ltx_align_left">Arch.</td>
<td id="S7.T10.2.3.3" class="ltx_td"></td>
<td id="S7.T10.2.3.4" class="ltx_td ltx_align_center ltx_border_t">lin.</td>
<td id="S7.T10.2.3.5" class="ltx_td ltx_align_center ltx_border_t">+ms</td>
<td id="S7.T10.2.3.6" class="ltx_td"></td>
<td id="S7.T10.2.3.7" class="ltx_td ltx_align_center ltx_border_t">lin.</td>
<td id="S7.T10.2.3.8" class="ltx_td ltx_align_center ltx_border_t">+ms</td>
<td id="S7.T10.2.3.9" class="ltx_td"></td>
<td id="S7.T10.2.3.10" class="ltx_td ltx_align_center ltx_border_t">lin.</td>
<td id="S7.T10.2.3.11" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">+ms</td>
</tr>
<tr id="S7.T10.2.4" class="ltx_tr">
<td id="S7.T10.2.4.1" class="ltx_td ltx_align_left ltx_border_t">OpenCLIP</td>
<td id="S7.T10.2.4.2" class="ltx_td ltx_align_left ltx_border_t">ViT-G/14</td>
<td id="S7.T10.2.4.3" class="ltx_td ltx_border_t"></td>
<td id="S7.T10.2.4.4" class="ltx_td ltx_align_center ltx_border_t">39.3</td>
<td id="S7.T10.2.4.5" class="ltx_td ltx_align_center ltx_border_t">46.0</td>
<td id="S7.T10.2.4.6" class="ltx_td ltx_border_t"></td>
<td id="S7.T10.2.4.7" class="ltx_td ltx_align_center ltx_border_t">60.3</td>
<td id="S7.T10.2.4.8" class="ltx_td ltx_align_center ltx_border_t">70.3</td>
<td id="S7.T10.2.4.9" class="ltx_td ltx_border_t"></td>
<td id="S7.T10.2.4.10" class="ltx_td ltx_align_center ltx_border_t">71.4</td>
<td id="S7.T10.2.4.11" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">79.2</td>
</tr>
<tr id="S7.T10.2.5" class="ltx_tr">
<td id="S7.T10.2.5.1" class="ltx_td ltx_align_left ltx_border_t">MAE</td>
<td id="S7.T10.2.5.2" class="ltx_td ltx_align_left ltx_border_t">ViT-H/14</td>
<td id="S7.T10.2.5.3" class="ltx_td ltx_border_t"></td>
<td id="S7.T10.2.5.4" class="ltx_td ltx_align_center ltx_border_t">33.3</td>
<td id="S7.T10.2.5.5" class="ltx_td ltx_align_center ltx_border_t">30.7</td>
<td id="S7.T10.2.5.6" class="ltx_td ltx_border_t"></td>
<td id="S7.T10.2.5.7" class="ltx_td ltx_align_center ltx_border_t">58.4</td>
<td id="S7.T10.2.5.8" class="ltx_td ltx_align_center ltx_border_t">61.0</td>
<td id="S7.T10.2.5.9" class="ltx_td ltx_border_t"></td>
<td id="S7.T10.2.5.10" class="ltx_td ltx_align_center ltx_border_t">67.6</td>
<td id="S7.T10.2.5.11" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">63.3</td>
</tr>
<tr id="S7.T10.2.6" class="ltx_tr">
<td id="S7.T10.2.6.1" class="ltx_td ltx_align_left">DINO</td>
<td id="S7.T10.2.6.2" class="ltx_td ltx_align_left">ViT-B/8</td>
<td id="S7.T10.2.6.3" class="ltx_td"></td>
<td id="S7.T10.2.6.4" class="ltx_td ltx_align_center">31.8</td>
<td id="S7.T10.2.6.5" class="ltx_td ltx_align_center">35.2</td>
<td id="S7.T10.2.6.6" class="ltx_td"></td>
<td id="S7.T10.2.6.7" class="ltx_td ltx_align_center">56.9</td>
<td id="S7.T10.2.6.8" class="ltx_td ltx_align_center">66.2</td>
<td id="S7.T10.2.6.9" class="ltx_td"></td>
<td id="S7.T10.2.6.10" class="ltx_td ltx_align_center">66.4</td>
<td id="S7.T10.2.6.11" class="ltx_td ltx_nopad_r ltx_align_center">75.6</td>
</tr>
<tr id="S7.T10.2.7" class="ltx_tr">
<td id="S7.T10.2.7.1" class="ltx_td ltx_align_left">iBOT</td>
<td id="S7.T10.2.7.2" class="ltx_td ltx_align_left">ViT-L/16</td>
<td id="S7.T10.2.7.3" class="ltx_td"></td>
<td id="S7.T10.2.7.4" class="ltx_td ltx_align_center">44.6</td>
<td id="S7.T10.2.7.5" class="ltx_td ltx_align_center">47.5</td>
<td id="S7.T10.2.7.6" class="ltx_td"></td>
<td id="S7.T10.2.7.7" class="ltx_td ltx_align_center">64.8</td>
<td id="S7.T10.2.7.8" class="ltx_td ltx_align_center">74.5</td>
<td id="S7.T10.2.7.9" class="ltx_td"></td>
<td id="S7.T10.2.7.10" class="ltx_td ltx_align_center">82.3</td>
<td id="S7.T10.2.7.11" class="ltx_td ltx_nopad_r ltx_align_center">84.3</td>
</tr>
<tr id="S7.T10.2.8" class="ltx_tr">
<td id="S7.T10.2.8.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" rowspan="4"><span id="S7.T10.2.8.1.1" class="ltx_text">DINOv2</span></td>
<td id="S7.T10.2.8.2" class="ltx_td ltx_align_left ltx_border_t">ViT-S/14</td>
<td id="S7.T10.2.8.3" class="ltx_td ltx_border_t"></td>
<td id="S7.T10.2.8.4" class="ltx_td ltx_align_center ltx_border_t">44.3</td>
<td id="S7.T10.2.8.5" class="ltx_td ltx_align_center ltx_border_t">47.2</td>
<td id="S7.T10.2.8.6" class="ltx_td ltx_border_t"></td>
<td id="S7.T10.2.8.7" class="ltx_td ltx_align_center ltx_border_t">66.6</td>
<td id="S7.T10.2.8.8" class="ltx_td ltx_align_center ltx_border_t">77.1</td>
<td id="S7.T10.2.8.9" class="ltx_td ltx_border_t"></td>
<td id="S7.T10.2.8.10" class="ltx_td ltx_align_center ltx_border_t">81.1</td>
<td id="S7.T10.2.8.11" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">82.6</td>
</tr>
<tr id="S7.T10.2.9" class="ltx_tr">
<td id="S7.T10.2.9.1" class="ltx_td ltx_align_left">ViT-B/14</td>
<td id="S7.T10.2.9.2" class="ltx_td"></td>
<td id="S7.T10.2.9.3" class="ltx_td ltx_align_center">47.3</td>
<td id="S7.T10.2.9.4" class="ltx_td ltx_align_center">51.3</td>
<td id="S7.T10.2.9.5" class="ltx_td"></td>
<td id="S7.T10.2.9.6" class="ltx_td ltx_align_center">69.4</td>
<td id="S7.T10.2.9.7" class="ltx_td ltx_align_center">80.0</td>
<td id="S7.T10.2.9.8" class="ltx_td"></td>
<td id="S7.T10.2.9.9" class="ltx_td ltx_align_center">82.5</td>
<td id="S7.T10.2.9.10" class="ltx_td ltx_nopad_r ltx_align_center">84.9</td>
</tr>
<tr id="S7.T10.2.10" class="ltx_tr">
<td id="S7.T10.2.10.1" class="ltx_td ltx_align_left">ViT-L/14</td>
<td id="S7.T10.2.10.2" class="ltx_td"></td>
<td id="S7.T10.2.10.3" class="ltx_td ltx_align_center">47.7</td>
<td id="S7.T10.2.10.4" class="ltx_td ltx_align_center"><span id="S7.T10.2.10.4.1" class="ltx_text ltx_font_bold">53.1</span></td>
<td id="S7.T10.2.10.5" class="ltx_td"></td>
<td id="S7.T10.2.10.6" class="ltx_td ltx_align_center">70.3</td>
<td id="S7.T10.2.10.7" class="ltx_td ltx_align_center">80.9</td>
<td id="S7.T10.2.10.8" class="ltx_td"></td>
<td id="S7.T10.2.10.9" class="ltx_td ltx_align_center">82.1</td>
<td id="S7.T10.2.10.10" class="ltx_td ltx_nopad_r ltx_align_center">86.0</td>
</tr>
<tr id="S7.T10.2.11" class="ltx_tr">
<td id="S7.T10.2.11.1" class="ltx_td ltx_align_left ltx_border_bb">ViT-g/14</td>
<td id="S7.T10.2.11.2" class="ltx_td ltx_border_bb"></td>
<td id="S7.T10.2.11.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T10.2.11.3.1" class="ltx_text ltx_font_bold">49.0</span></td>
<td id="S7.T10.2.11.4" class="ltx_td ltx_align_center ltx_border_bb">53.0</td>
<td id="S7.T10.2.11.5" class="ltx_td ltx_border_bb"></td>
<td id="S7.T10.2.11.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T10.2.11.6.1" class="ltx_text ltx_font_bold">71.3</span></td>
<td id="S7.T10.2.11.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T10.2.11.7.1" class="ltx_text ltx_font_bold">81.0</span></td>
<td id="S7.T10.2.11.8" class="ltx_td ltx_border_bb"></td>
<td id="S7.T10.2.11.9" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T10.2.11.9.1" class="ltx_text ltx_font_bold">83.0</span></td>
<td id="S7.T10.2.11.10" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S7.T10.2.11.10.1" class="ltx_text ltx_font_bold">86.2</span></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S7.T10.4.1.1" class="ltx_text" style="font-size:90%;">Table 10</span>: </span><span id="S7.T10.5.2" class="ltx_text" style="font-size:90%;">
<span id="S7.T10.5.2.1" class="ltx_text ltx_font_bold">Semantic segmentation on ADE20K, CityScapes and Pascal VOC with frozen features</span> and a linear classifier (lin.) and with multiscale (+ms).
The absolute state of the art – from <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib119" title="" class="ltx_ref">2022</a>)</cite>, <cite class="ltx_cite ltx_citemacro_cite">Liu et al. (<a href="#bib.bib78" title="" class="ltx_ref">2021</a>)</cite> and <cite class="ltx_cite ltx_citemacro_cite">Chen et al. (<a href="#bib.bib20" title="" class="ltx_ref">2018</a>)</cite> respectively – are mentioned at the top of the Table.
For reference, using the Mask2Former pipeline <cite class="ltx_cite ltx_citemacro_citep">(Steiner et al., <a href="#bib.bib108" title="" class="ltx_ref">2021</a>)</cite> with a ViT-Adapter <cite class="ltx_cite ltx_citemacro_citep">(Chen et al., <a href="#bib.bib25" title="" class="ltx_ref">2023b</a>)</cite> on top of our frozen ViT-g/14 backbone gives 60.2 mIoU on ADE-20k.
</span></figcaption>
</figure>
</section>
<section id="S7.SS4.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Depth estimation.</h5>

<div id="S7.SS4.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S7.SS4.SSS0.Px3.p1.3" class="ltx_p">In this experiment, we evaluate our patch-level features on three monocular depth estimation benchmarks: NYUd, KITTI and zero-shot transfer from NYUd to SUN3d.
We follow the evaluation protocol of <cite class="ltx_cite ltx_citemacro_citet">Li et al. (<a href="#bib.bib76" title="" class="ltx_ref">2022b</a>)</cite>.
We consider three different setups for this evaluation.
<span id="S7.SS4.SSS0.Px3.p1.3.1" class="ltx_text ltx_font_bold">lin. 1</span>: we extract the last layer of the frozen transformer and concatenate the <span id="S7.SS4.SSS0.Px3.p1.3.2" class="ltx_text ltx_font_typewriter">[CLS]</span> token to each patch token.
Then we bi-linearly upsample the tokens by a factor of 4 to increase the resolution.
Finally we train a simple linear layer using a classification loss by dividing the depth prediction range in 256 uniformly distributed bins and use a linear normalization following  <cite class="ltx_cite ltx_citemacro_cite">Bhat et al. (<a href="#bib.bib11" title="" class="ltx_ref">2021</a>)</cite>.
<span id="S7.SS4.SSS0.Px3.p1.3.3" class="ltx_text ltx_font_bold">lin. 4</span>: we use the same protocol that we use with one layer, but concatenate the tokens from layers <math id="S7.SS4.SSS0.Px3.p1.1.m1.4" class="ltx_Math" alttext="l=\{3,6,9,12\}" display="inline"><semantics id="S7.SS4.SSS0.Px3.p1.1.m1.4a"><mrow id="S7.SS4.SSS0.Px3.p1.1.m1.4.5" xref="S7.SS4.SSS0.Px3.p1.1.m1.4.5.cmml"><mi id="S7.SS4.SSS0.Px3.p1.1.m1.4.5.2" xref="S7.SS4.SSS0.Px3.p1.1.m1.4.5.2.cmml">l</mi><mo id="S7.SS4.SSS0.Px3.p1.1.m1.4.5.1" xref="S7.SS4.SSS0.Px3.p1.1.m1.4.5.1.cmml">=</mo><mrow id="S7.SS4.SSS0.Px3.p1.1.m1.4.5.3.2" xref="S7.SS4.SSS0.Px3.p1.1.m1.4.5.3.1.cmml"><mo stretchy="false" id="S7.SS4.SSS0.Px3.p1.1.m1.4.5.3.2.1" xref="S7.SS4.SSS0.Px3.p1.1.m1.4.5.3.1.cmml">{</mo><mn id="S7.SS4.SSS0.Px3.p1.1.m1.1.1" xref="S7.SS4.SSS0.Px3.p1.1.m1.1.1.cmml">3</mn><mo id="S7.SS4.SSS0.Px3.p1.1.m1.4.5.3.2.2" xref="S7.SS4.SSS0.Px3.p1.1.m1.4.5.3.1.cmml">,</mo><mn id="S7.SS4.SSS0.Px3.p1.1.m1.2.2" xref="S7.SS4.SSS0.Px3.p1.1.m1.2.2.cmml">6</mn><mo id="S7.SS4.SSS0.Px3.p1.1.m1.4.5.3.2.3" xref="S7.SS4.SSS0.Px3.p1.1.m1.4.5.3.1.cmml">,</mo><mn id="S7.SS4.SSS0.Px3.p1.1.m1.3.3" xref="S7.SS4.SSS0.Px3.p1.1.m1.3.3.cmml">9</mn><mo id="S7.SS4.SSS0.Px3.p1.1.m1.4.5.3.2.4" xref="S7.SS4.SSS0.Px3.p1.1.m1.4.5.3.1.cmml">,</mo><mn id="S7.SS4.SSS0.Px3.p1.1.m1.4.4" xref="S7.SS4.SSS0.Px3.p1.1.m1.4.4.cmml">12</mn><mo stretchy="false" id="S7.SS4.SSS0.Px3.p1.1.m1.4.5.3.2.5" xref="S7.SS4.SSS0.Px3.p1.1.m1.4.5.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS4.SSS0.Px3.p1.1.m1.4b"><apply id="S7.SS4.SSS0.Px3.p1.1.m1.4.5.cmml" xref="S7.SS4.SSS0.Px3.p1.1.m1.4.5"><eq id="S7.SS4.SSS0.Px3.p1.1.m1.4.5.1.cmml" xref="S7.SS4.SSS0.Px3.p1.1.m1.4.5.1"></eq><ci id="S7.SS4.SSS0.Px3.p1.1.m1.4.5.2.cmml" xref="S7.SS4.SSS0.Px3.p1.1.m1.4.5.2">𝑙</ci><set id="S7.SS4.SSS0.Px3.p1.1.m1.4.5.3.1.cmml" xref="S7.SS4.SSS0.Px3.p1.1.m1.4.5.3.2"><cn type="integer" id="S7.SS4.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S7.SS4.SSS0.Px3.p1.1.m1.1.1">3</cn><cn type="integer" id="S7.SS4.SSS0.Px3.p1.1.m1.2.2.cmml" xref="S7.SS4.SSS0.Px3.p1.1.m1.2.2">6</cn><cn type="integer" id="S7.SS4.SSS0.Px3.p1.1.m1.3.3.cmml" xref="S7.SS4.SSS0.Px3.p1.1.m1.3.3">9</cn><cn type="integer" id="S7.SS4.SSS0.Px3.p1.1.m1.4.4.cmml" xref="S7.SS4.SSS0.Px3.p1.1.m1.4.4">12</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS4.SSS0.Px3.p1.1.m1.4c">l=\{3,6,9,12\}</annotation></semantics></math> for ViT-S/B, <math id="S7.SS4.SSS0.Px3.p1.2.m2.4" class="ltx_Math" alttext="l=\{5,12,18,24\}" display="inline"><semantics id="S7.SS4.SSS0.Px3.p1.2.m2.4a"><mrow id="S7.SS4.SSS0.Px3.p1.2.m2.4.5" xref="S7.SS4.SSS0.Px3.p1.2.m2.4.5.cmml"><mi id="S7.SS4.SSS0.Px3.p1.2.m2.4.5.2" xref="S7.SS4.SSS0.Px3.p1.2.m2.4.5.2.cmml">l</mi><mo id="S7.SS4.SSS0.Px3.p1.2.m2.4.5.1" xref="S7.SS4.SSS0.Px3.p1.2.m2.4.5.1.cmml">=</mo><mrow id="S7.SS4.SSS0.Px3.p1.2.m2.4.5.3.2" xref="S7.SS4.SSS0.Px3.p1.2.m2.4.5.3.1.cmml"><mo stretchy="false" id="S7.SS4.SSS0.Px3.p1.2.m2.4.5.3.2.1" xref="S7.SS4.SSS0.Px3.p1.2.m2.4.5.3.1.cmml">{</mo><mn id="S7.SS4.SSS0.Px3.p1.2.m2.1.1" xref="S7.SS4.SSS0.Px3.p1.2.m2.1.1.cmml">5</mn><mo id="S7.SS4.SSS0.Px3.p1.2.m2.4.5.3.2.2" xref="S7.SS4.SSS0.Px3.p1.2.m2.4.5.3.1.cmml">,</mo><mn id="S7.SS4.SSS0.Px3.p1.2.m2.2.2" xref="S7.SS4.SSS0.Px3.p1.2.m2.2.2.cmml">12</mn><mo id="S7.SS4.SSS0.Px3.p1.2.m2.4.5.3.2.3" xref="S7.SS4.SSS0.Px3.p1.2.m2.4.5.3.1.cmml">,</mo><mn id="S7.SS4.SSS0.Px3.p1.2.m2.3.3" xref="S7.SS4.SSS0.Px3.p1.2.m2.3.3.cmml">18</mn><mo id="S7.SS4.SSS0.Px3.p1.2.m2.4.5.3.2.4" xref="S7.SS4.SSS0.Px3.p1.2.m2.4.5.3.1.cmml">,</mo><mn id="S7.SS4.SSS0.Px3.p1.2.m2.4.4" xref="S7.SS4.SSS0.Px3.p1.2.m2.4.4.cmml">24</mn><mo stretchy="false" id="S7.SS4.SSS0.Px3.p1.2.m2.4.5.3.2.5" xref="S7.SS4.SSS0.Px3.p1.2.m2.4.5.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS4.SSS0.Px3.p1.2.m2.4b"><apply id="S7.SS4.SSS0.Px3.p1.2.m2.4.5.cmml" xref="S7.SS4.SSS0.Px3.p1.2.m2.4.5"><eq id="S7.SS4.SSS0.Px3.p1.2.m2.4.5.1.cmml" xref="S7.SS4.SSS0.Px3.p1.2.m2.4.5.1"></eq><ci id="S7.SS4.SSS0.Px3.p1.2.m2.4.5.2.cmml" xref="S7.SS4.SSS0.Px3.p1.2.m2.4.5.2">𝑙</ci><set id="S7.SS4.SSS0.Px3.p1.2.m2.4.5.3.1.cmml" xref="S7.SS4.SSS0.Px3.p1.2.m2.4.5.3.2"><cn type="integer" id="S7.SS4.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S7.SS4.SSS0.Px3.p1.2.m2.1.1">5</cn><cn type="integer" id="S7.SS4.SSS0.Px3.p1.2.m2.2.2.cmml" xref="S7.SS4.SSS0.Px3.p1.2.m2.2.2">12</cn><cn type="integer" id="S7.SS4.SSS0.Px3.p1.2.m2.3.3.cmml" xref="S7.SS4.SSS0.Px3.p1.2.m2.3.3">18</cn><cn type="integer" id="S7.SS4.SSS0.Px3.p1.2.m2.4.4.cmml" xref="S7.SS4.SSS0.Px3.p1.2.m2.4.4">24</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS4.SSS0.Px3.p1.2.m2.4c">l=\{5,12,18,24\}</annotation></semantics></math> for ViT-L, and <math id="S7.SS4.SSS0.Px3.p1.3.m3.4" class="ltx_Math" alttext="l=\{10,20,30,40\}" display="inline"><semantics id="S7.SS4.SSS0.Px3.p1.3.m3.4a"><mrow id="S7.SS4.SSS0.Px3.p1.3.m3.4.5" xref="S7.SS4.SSS0.Px3.p1.3.m3.4.5.cmml"><mi id="S7.SS4.SSS0.Px3.p1.3.m3.4.5.2" xref="S7.SS4.SSS0.Px3.p1.3.m3.4.5.2.cmml">l</mi><mo id="S7.SS4.SSS0.Px3.p1.3.m3.4.5.1" xref="S7.SS4.SSS0.Px3.p1.3.m3.4.5.1.cmml">=</mo><mrow id="S7.SS4.SSS0.Px3.p1.3.m3.4.5.3.2" xref="S7.SS4.SSS0.Px3.p1.3.m3.4.5.3.1.cmml"><mo stretchy="false" id="S7.SS4.SSS0.Px3.p1.3.m3.4.5.3.2.1" xref="S7.SS4.SSS0.Px3.p1.3.m3.4.5.3.1.cmml">{</mo><mn id="S7.SS4.SSS0.Px3.p1.3.m3.1.1" xref="S7.SS4.SSS0.Px3.p1.3.m3.1.1.cmml">10</mn><mo id="S7.SS4.SSS0.Px3.p1.3.m3.4.5.3.2.2" xref="S7.SS4.SSS0.Px3.p1.3.m3.4.5.3.1.cmml">,</mo><mn id="S7.SS4.SSS0.Px3.p1.3.m3.2.2" xref="S7.SS4.SSS0.Px3.p1.3.m3.2.2.cmml">20</mn><mo id="S7.SS4.SSS0.Px3.p1.3.m3.4.5.3.2.3" xref="S7.SS4.SSS0.Px3.p1.3.m3.4.5.3.1.cmml">,</mo><mn id="S7.SS4.SSS0.Px3.p1.3.m3.3.3" xref="S7.SS4.SSS0.Px3.p1.3.m3.3.3.cmml">30</mn><mo id="S7.SS4.SSS0.Px3.p1.3.m3.4.5.3.2.4" xref="S7.SS4.SSS0.Px3.p1.3.m3.4.5.3.1.cmml">,</mo><mn id="S7.SS4.SSS0.Px3.p1.3.m3.4.4" xref="S7.SS4.SSS0.Px3.p1.3.m3.4.4.cmml">40</mn><mo stretchy="false" id="S7.SS4.SSS0.Px3.p1.3.m3.4.5.3.2.5" xref="S7.SS4.SSS0.Px3.p1.3.m3.4.5.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.SS4.SSS0.Px3.p1.3.m3.4b"><apply id="S7.SS4.SSS0.Px3.p1.3.m3.4.5.cmml" xref="S7.SS4.SSS0.Px3.p1.3.m3.4.5"><eq id="S7.SS4.SSS0.Px3.p1.3.m3.4.5.1.cmml" xref="S7.SS4.SSS0.Px3.p1.3.m3.4.5.1"></eq><ci id="S7.SS4.SSS0.Px3.p1.3.m3.4.5.2.cmml" xref="S7.SS4.SSS0.Px3.p1.3.m3.4.5.2">𝑙</ci><set id="S7.SS4.SSS0.Px3.p1.3.m3.4.5.3.1.cmml" xref="S7.SS4.SSS0.Px3.p1.3.m3.4.5.3.2"><cn type="integer" id="S7.SS4.SSS0.Px3.p1.3.m3.1.1.cmml" xref="S7.SS4.SSS0.Px3.p1.3.m3.1.1">10</cn><cn type="integer" id="S7.SS4.SSS0.Px3.p1.3.m3.2.2.cmml" xref="S7.SS4.SSS0.Px3.p1.3.m3.2.2">20</cn><cn type="integer" id="S7.SS4.SSS0.Px3.p1.3.m3.3.3.cmml" xref="S7.SS4.SSS0.Px3.p1.3.m3.3.3">30</cn><cn type="integer" id="S7.SS4.SSS0.Px3.p1.3.m3.4.4.cmml" xref="S7.SS4.SSS0.Px3.p1.3.m3.4.4">40</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS4.SSS0.Px3.p1.3.m3.4c">l=\{10,20,30,40\}</annotation></semantics></math> for ViT-g.
<span id="S7.SS4.SSS0.Px3.p1.3.4" class="ltx_text ltx_font_bold">DPT</span>: we use the DPT decoder <cite class="ltx_cite ltx_citemacro_citep">(Ranftl et al., <a href="#bib.bib95" title="" class="ltx_ref">2021</a>)</cite> on top of our frozen models and setup a regression task.
We scale the size of the head following the dimension of the features for each architecture.
We show results for all baselines, all datasets and all setups in Table <a href="#S7.T11" title="Table 11 ‣ Depth estimation. ‣ 7.4 Dense Recognition Tasks ‣ 7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">11</span></a>.</p>
</div>
<div id="S7.SS4.SSS0.Px3.p2" class="ltx_para ltx_noindent">
<p id="S7.SS4.SSS0.Px3.p2.1" class="ltx_p">From this table, we see that our features clearly surpass the best SSL and WSL features available.
It is interesting to see that iBOT features extracted from a ViT-L outperform the ones of OpenCLIP with a ViT-G.
This observation supports the intuition that caption-based feature learning fails to learn subtle patterns like this one.
Also, our model, with the DPT decoder and frozen backbone, matches or exceeds the performance of the recent work of <cite class="ltx_cite ltx_citemacro_citet">Li et al. (<a href="#bib.bib76" title="" class="ltx_ref">2022b</a>)</cite>.
Finally, the out-of-domain generalization result on SUN-RGBd shows that our features allow very good transfer between domains.
A depth prediction module trained on indoor scenes from NYUd generalizes pretty well to the outdoor examples of SUN-RGBd.</p>
</div>
<figure id="S7.T11" class="ltx_table">
<table id="S7.T11.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S7.T11.1.1" class="ltx_tr">
<td id="S7.T11.1.1.2" class="ltx_td ltx_border_tt"></td>
<td id="S7.T11.1.1.3" class="ltx_td ltx_nopad_r ltx_border_tt"></td>
<td id="S7.T11.1.1.4" class="ltx_td ltx_nopad_l ltx_border_tt"></td>
<td id="S7.T11.1.1.5" class="ltx_td ltx_align_center ltx_border_tt" colspan="3">NYUd</td>
<td id="S7.T11.1.1.6" class="ltx_td ltx_border_tt"></td>
<td id="S7.T11.1.1.7" class="ltx_td ltx_align_center ltx_border_tt" colspan="3">KITTI</td>
<td id="S7.T11.1.1.8" class="ltx_td ltx_border_tt"></td>
<td id="S7.T11.1.1.1" class="ltx_td ltx_align_center ltx_border_tt" colspan="3">NYUd <math id="S7.T11.1.1.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S7.T11.1.1.1.m1.1a"><mo stretchy="false" id="S7.T11.1.1.1.m1.1.1" xref="S7.T11.1.1.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S7.T11.1.1.1.m1.1b"><ci id="S7.T11.1.1.1.m1.1.1.cmml" xref="S7.T11.1.1.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.T11.1.1.1.m1.1c">\rightarrow</annotation></semantics></math> SUN RGB-D</td>
</tr>
<tr id="S7.T11.1.2" class="ltx_tr">
<td id="S7.T11.1.2.1" class="ltx_td"></td>
<td id="S7.T11.1.2.2" class="ltx_td ltx_nopad_r"></td>
<td id="S7.T11.1.2.3" class="ltx_td ltx_nopad_l"></td>
<td id="S7.T11.1.2.4" class="ltx_td ltx_align_center" colspan="3">(0.330)</td>
<td id="S7.T11.1.2.5" class="ltx_td"></td>
<td id="S7.T11.1.2.6" class="ltx_td ltx_align_center" colspan="3">(2.10)</td>
<td id="S7.T11.1.2.7" class="ltx_td"></td>
<td id="S7.T11.1.2.8" class="ltx_td ltx_align_center" colspan="3">(0.421)</td>
</tr>
<tr id="S7.T11.1.3" class="ltx_tr">
<td id="S7.T11.1.3.1" class="ltx_td ltx_align_left">Method</td>
<td id="S7.T11.1.3.2" class="ltx_td ltx_nopad_r ltx_align_left">Arch.</td>
<td id="S7.T11.1.3.3" class="ltx_td ltx_nopad_l"></td>
<td id="S7.T11.1.3.4" class="ltx_td ltx_align_center ltx_border_t">lin. 1</td>
<td id="S7.T11.1.3.5" class="ltx_td ltx_align_center ltx_border_t">lin. 4</td>
<td id="S7.T11.1.3.6" class="ltx_td ltx_align_center ltx_border_t">DPT</td>
<td id="S7.T11.1.3.7" class="ltx_td"></td>
<td id="S7.T11.1.3.8" class="ltx_td ltx_align_center ltx_border_t">lin. 1</td>
<td id="S7.T11.1.3.9" class="ltx_td ltx_align_center ltx_border_t">lin. 4</td>
<td id="S7.T11.1.3.10" class="ltx_td ltx_align_center ltx_border_t">DPT</td>
<td id="S7.T11.1.3.11" class="ltx_td"></td>
<td id="S7.T11.1.3.12" class="ltx_td ltx_align_center ltx_border_t">lin. 1</td>
<td id="S7.T11.1.3.13" class="ltx_td ltx_align_center ltx_border_t">lin. 4</td>
<td id="S7.T11.1.3.14" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">DPT</td>
</tr>
<tr id="S7.T11.1.4" class="ltx_tr">
<td id="S7.T11.1.4.1" class="ltx_td ltx_align_left ltx_border_t">OpenCLIP</td>
<td id="S7.T11.1.4.2" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">ViT-G/14</td>
<td id="S7.T11.1.4.3" class="ltx_td ltx_nopad_l ltx_border_t"></td>
<td id="S7.T11.1.4.4" class="ltx_td ltx_align_center ltx_border_t">0.541</td>
<td id="S7.T11.1.4.5" class="ltx_td ltx_align_center ltx_border_t">0.510</td>
<td id="S7.T11.1.4.6" class="ltx_td ltx_align_center ltx_border_t">0.414</td>
<td id="S7.T11.1.4.7" class="ltx_td ltx_border_t"></td>
<td id="S7.T11.1.4.8" class="ltx_td ltx_align_center ltx_border_t">3.57</td>
<td id="S7.T11.1.4.9" class="ltx_td ltx_align_center ltx_border_t">3.21</td>
<td id="S7.T11.1.4.10" class="ltx_td ltx_align_center ltx_border_t">2.56</td>
<td id="S7.T11.1.4.11" class="ltx_td ltx_border_t"></td>
<td id="S7.T11.1.4.12" class="ltx_td ltx_align_center ltx_border_t">0.537</td>
<td id="S7.T11.1.4.13" class="ltx_td ltx_align_center ltx_border_t">0.476</td>
<td id="S7.T11.1.4.14" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">0.408</td>
</tr>
<tr id="S7.T11.1.5" class="ltx_tr">
<td id="S7.T11.1.5.1" class="ltx_td ltx_align_left ltx_border_t">MAE</td>
<td id="S7.T11.1.5.2" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">ViT-H/14</td>
<td id="S7.T11.1.5.3" class="ltx_td ltx_nopad_l ltx_border_t"></td>
<td id="S7.T11.1.5.4" class="ltx_td ltx_align_center ltx_border_t">0.517</td>
<td id="S7.T11.1.5.5" class="ltx_td ltx_align_center ltx_border_t">0.483</td>
<td id="S7.T11.1.5.6" class="ltx_td ltx_align_center ltx_border_t">0.415</td>
<td id="S7.T11.1.5.7" class="ltx_td ltx_border_t"></td>
<td id="S7.T11.1.5.8" class="ltx_td ltx_align_center ltx_border_t">3.66</td>
<td id="S7.T11.1.5.9" class="ltx_td ltx_align_center ltx_border_t">3.26</td>
<td id="S7.T11.1.5.10" class="ltx_td ltx_align_center ltx_border_t">2.59</td>
<td id="S7.T11.1.5.11" class="ltx_td ltx_border_t"></td>
<td id="S7.T11.1.5.12" class="ltx_td ltx_align_center ltx_border_t">0.545</td>
<td id="S7.T11.1.5.13" class="ltx_td ltx_align_center ltx_border_t">0.523</td>
<td id="S7.T11.1.5.14" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">0.506</td>
</tr>
<tr id="S7.T11.1.6" class="ltx_tr">
<td id="S7.T11.1.6.1" class="ltx_td ltx_align_left">DINO</td>
<td id="S7.T11.1.6.2" class="ltx_td ltx_nopad_r ltx_align_left">ViT-B/8</td>
<td id="S7.T11.1.6.3" class="ltx_td ltx_nopad_l"></td>
<td id="S7.T11.1.6.4" class="ltx_td ltx_align_center">0.555</td>
<td id="S7.T11.1.6.5" class="ltx_td ltx_align_center">0.539</td>
<td id="S7.T11.1.6.6" class="ltx_td ltx_align_center">0.492</td>
<td id="S7.T11.1.6.7" class="ltx_td"></td>
<td id="S7.T11.1.6.8" class="ltx_td ltx_align_center">3.81</td>
<td id="S7.T11.1.6.9" class="ltx_td ltx_align_center">3.56</td>
<td id="S7.T11.1.6.10" class="ltx_td ltx_align_center">2.74</td>
<td id="S7.T11.1.6.11" class="ltx_td"></td>
<td id="S7.T11.1.6.12" class="ltx_td ltx_align_center">0.553</td>
<td id="S7.T11.1.6.13" class="ltx_td ltx_align_center">0.541</td>
<td id="S7.T11.1.6.14" class="ltx_td ltx_nopad_r ltx_align_center">0.520</td>
</tr>
<tr id="S7.T11.1.7" class="ltx_tr">
<td id="S7.T11.1.7.1" class="ltx_td ltx_align_left">iBOT</td>
<td id="S7.T11.1.7.2" class="ltx_td ltx_nopad_r ltx_align_left">ViT-L/16</td>
<td id="S7.T11.1.7.3" class="ltx_td ltx_nopad_l"></td>
<td id="S7.T11.1.7.4" class="ltx_td ltx_align_center">0.417</td>
<td id="S7.T11.1.7.5" class="ltx_td ltx_align_center">0.387</td>
<td id="S7.T11.1.7.6" class="ltx_td ltx_align_center">0.358</td>
<td id="S7.T11.1.7.7" class="ltx_td"></td>
<td id="S7.T11.1.7.8" class="ltx_td ltx_align_center">3.31</td>
<td id="S7.T11.1.7.9" class="ltx_td ltx_align_center">3.07</td>
<td id="S7.T11.1.7.10" class="ltx_td ltx_align_center">2.55</td>
<td id="S7.T11.1.7.11" class="ltx_td"></td>
<td id="S7.T11.1.7.12" class="ltx_td ltx_align_center">0.447</td>
<td id="S7.T11.1.7.13" class="ltx_td ltx_align_center">0.435</td>
<td id="S7.T11.1.7.14" class="ltx_td ltx_nopad_r ltx_align_center">0.426</td>
</tr>
<tr id="S7.T11.1.8" class="ltx_tr">
<td id="S7.T11.1.8.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t" rowspan="4"><span id="S7.T11.1.8.1.1" class="ltx_text">DINOv2</span></td>
<td id="S7.T11.1.8.2" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_t">ViT-S/14</td>
<td id="S7.T11.1.8.3" class="ltx_td ltx_nopad_l ltx_border_t"></td>
<td id="S7.T11.1.8.4" class="ltx_td ltx_align_center ltx_border_t">0.449</td>
<td id="S7.T11.1.8.5" class="ltx_td ltx_align_center ltx_border_t">0.417</td>
<td id="S7.T11.1.8.6" class="ltx_td ltx_align_center ltx_border_t">0.356</td>
<td id="S7.T11.1.8.7" class="ltx_td ltx_border_t"></td>
<td id="S7.T11.1.8.8" class="ltx_td ltx_align_center ltx_border_t">3.10</td>
<td id="S7.T11.1.8.9" class="ltx_td ltx_align_center ltx_border_t">2.86</td>
<td id="S7.T11.1.8.10" class="ltx_td ltx_align_center ltx_border_t">2.34</td>
<td id="S7.T11.1.8.11" class="ltx_td ltx_border_t"></td>
<td id="S7.T11.1.8.12" class="ltx_td ltx_align_center ltx_border_t">0.477</td>
<td id="S7.T11.1.8.13" class="ltx_td ltx_align_center ltx_border_t">0.431</td>
<td id="S7.T11.1.8.14" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">0.409</td>
</tr>
<tr id="S7.T11.1.9" class="ltx_tr">
<td id="S7.T11.1.9.1" class="ltx_td ltx_nopad_r ltx_align_left">ViT-B/14</td>
<td id="S7.T11.1.9.2" class="ltx_td ltx_nopad_l"></td>
<td id="S7.T11.1.9.3" class="ltx_td ltx_align_center">0.399</td>
<td id="S7.T11.1.9.4" class="ltx_td ltx_align_center">0.362</td>
<td id="S7.T11.1.9.5" class="ltx_td ltx_align_center">0.317</td>
<td id="S7.T11.1.9.6" class="ltx_td"></td>
<td id="S7.T11.1.9.7" class="ltx_td ltx_align_center">2.90</td>
<td id="S7.T11.1.9.8" class="ltx_td ltx_align_center">2.59</td>
<td id="S7.T11.1.9.9" class="ltx_td ltx_align_center">2.23</td>
<td id="S7.T11.1.9.10" class="ltx_td"></td>
<td id="S7.T11.1.9.11" class="ltx_td ltx_align_center">0.448</td>
<td id="S7.T11.1.9.12" class="ltx_td ltx_align_center">0.400</td>
<td id="S7.T11.1.9.13" class="ltx_td ltx_nopad_r ltx_align_center">0.377</td>
</tr>
<tr id="S7.T11.1.10" class="ltx_tr">
<td id="S7.T11.1.10.1" class="ltx_td ltx_nopad_r ltx_align_left">ViT-L/14</td>
<td id="S7.T11.1.10.2" class="ltx_td ltx_nopad_l"></td>
<td id="S7.T11.1.10.3" class="ltx_td ltx_align_center">0.384</td>
<td id="S7.T11.1.10.4" class="ltx_td ltx_align_center">0.333</td>
<td id="S7.T11.1.10.5" class="ltx_td ltx_align_center">0.293</td>
<td id="S7.T11.1.10.6" class="ltx_td"></td>
<td id="S7.T11.1.10.7" class="ltx_td ltx_align_center">2.78</td>
<td id="S7.T11.1.10.8" class="ltx_td ltx_align_center">2.50</td>
<td id="S7.T11.1.10.9" class="ltx_td ltx_align_center">2.14</td>
<td id="S7.T11.1.10.10" class="ltx_td"></td>
<td id="S7.T11.1.10.11" class="ltx_td ltx_align_center">0.429</td>
<td id="S7.T11.1.10.12" class="ltx_td ltx_align_center">0.396</td>
<td id="S7.T11.1.10.13" class="ltx_td ltx_nopad_r ltx_align_center">0.360</td>
</tr>
<tr id="S7.T11.1.11" class="ltx_tr">
<td id="S7.T11.1.11.1" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_bb">ViT-g/14</td>
<td id="S7.T11.1.11.2" class="ltx_td ltx_nopad_l ltx_border_bb"></td>
<td id="S7.T11.1.11.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T11.1.11.3.1" class="ltx_text ltx_font_bold">0.344</span></td>
<td id="S7.T11.1.11.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T11.1.11.4.1" class="ltx_text ltx_font_bold">0.298</span></td>
<td id="S7.T11.1.11.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T11.1.11.5.1" class="ltx_text ltx_font_bold">0.279</span></td>
<td id="S7.T11.1.11.6" class="ltx_td ltx_border_bb"></td>
<td id="S7.T11.1.11.7" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T11.1.11.7.1" class="ltx_text ltx_font_bold">2.62</span></td>
<td id="S7.T11.1.11.8" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T11.1.11.8.1" class="ltx_text ltx_font_bold">2.35</span></td>
<td id="S7.T11.1.11.9" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T11.1.11.9.1" class="ltx_text ltx_font_bold">2.11</span></td>
<td id="S7.T11.1.11.10" class="ltx_td ltx_border_bb"></td>
<td id="S7.T11.1.11.11" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T11.1.11.11.1" class="ltx_text ltx_font_bold">0.402</span></td>
<td id="S7.T11.1.11.12" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T11.1.11.12.1" class="ltx_text ltx_font_bold">0.362</span></td>
<td id="S7.T11.1.11.13" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb"><span id="S7.T11.1.11.13.1" class="ltx_text ltx_font_bold">0.338</span></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S7.T11.4.1.1" class="ltx_text" style="font-size:90%;">Table 11</span>: </span><span id="S7.T11.5.2" class="ltx_text" style="font-size:90%;">
<span id="S7.T11.5.2.1" class="ltx_text ltx_font_bold">Depth estimation with frozen features</span>.
We report performance when training a linear classifier on top of one (lin. 1) or four (lin. 4) transformer layers, as well, as the DPT decoder (DPT) of  <cite class="ltx_cite ltx_citemacro_cite">Ranftl et al. (<a href="#bib.bib95" title="" class="ltx_ref">2021</a>)</cite>.
We report the RMSE metric on the 3 datasets.
Lower is better.
For reference, we report state-of-the-art results taken from <cite class="ltx_cite ltx_citemacro_cite">Li et al. (<a href="#bib.bib76" title="" class="ltx_ref">2022b</a>)</cite> on each benchmark on top of the Table.
</span></figcaption>
</figure>
</section>
</section>
<section id="S7.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">7.5 </span>Qualitative Results</h3>

<div id="S7.SS5.p1" class="ltx_para ltx_noindent">
<p id="S7.SS5.p1.1" class="ltx_p">In this final section of the empirical evaluation of our features, we propose a few qualitative analyses.</p>
</div>
<section id="S7.SS5.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Semantic Segmentation and Depth Estimation.</h5>

<div id="S7.SS5.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S7.SS5.SSS0.Px1.p1.1" class="ltx_p">We show some qualitative results for our dense prediction evaluations: segmentation on ADE20K in Fig. <a href="#S7.F7" title="Figure 7 ‣ Semantic Segmentation and Depth Estimation. ‣ 7.5 Qualitative Results ‣ 7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> and depth estimation on NYUd, KITTI and SUN RGB-D in Fig. <a href="#S7.F7" title="Figure 7 ‣ Semantic Segmentation and Depth Estimation. ‣ 7.5 Qualitative Results ‣ 7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>.
We compare DINOv2 with OpenCLIP with a linear classifier on each dataset.
While not perfect, the linear segmentation model using our DINOv2 backbone produces good results and behaves much better than the OpenCLIP one under this evaluation setup.
Indeed, the segmentation mask produced by OpenCLIP-G shows many artifacts and disconnected components.
The qualitative results on depth estimation clearly illustrate the quantitative gap between OpenCLIP and DINOv2.
These results highlight that our features, as well as the features extracted from OpenCLIP, are able to linearly separate complex information such as depth, even though neither was trained with this type of information.
However, our features lead to a much smoother depth estimation, with less artifacts.
Some objects such as the chair on the SUN RGB-D image are completely ignored by OpenCLIP and correctly positioned using our features.</p>
</div>
<figure id="S7.F7" class="ltx_figure"><img src="/html/2304.07193/assets/new-figure-7.jpg" id="S7.F7.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="316" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S7.F7.3.1.1" class="ltx_text" style="font-size:90%;">Figure 7</span>: </span><span id="S7.F7.4.2" class="ltx_text" style="font-size:90%;">
<span id="S7.F7.4.2.1" class="ltx_text ltx_font_bold">Segmentation and depth estimation with linear classifiers.</span>
Examples from ADE20K, NYUd, SUN RGB-D and KITTI with a linear probe on frozen OpenCLIP-G and DINOv2-g features.
</span></figcaption>
</figure>
</section>
<section id="S7.SS5.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Out-of-distribution generalization.</h5>

<div id="S7.SS5.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="S7.SS5.SSS0.Px2.p1.1" class="ltx_p">We show a few examples of applying the depth prediction and segmentation linear classifiers to out-of-distribution examples in Fig. <a href="#S7.F8" title="Figure 8 ‣ Out-of-distribution generalization. ‣ 7.5 Qualitative Results ‣ 7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>.
The qualitative results support our claim that our features transfer between domains.
The quality of the depth and segmentation predicted for pictures of animals, or paintings is very good, even though the domains are very different.</p>
</div>
<figure id="S7.F8" class="ltx_figure"><img src="/html/2304.07193/assets/new-figure-8.jpg" id="S7.F8.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="230" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S7.F8.3.1.1" class="ltx_text" style="font-size:90%;">Figure 8</span>: </span><span id="S7.F8.4.2" class="ltx_text" style="font-size:90%;">
<span id="S7.F8.4.2.1" class="ltx_text ltx_font_bold">Examples of out-of-distribution examples</span> with frozen DINOv2-g features and a linear probe.
</span></figcaption>
</figure>
<figure id="S7.F9" class="ltx_figure"><img src="/html/2304.07193/assets/new-figure-9.jpg" id="S7.F9.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="324" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S7.F9.3.1.1" class="ltx_text" style="font-size:90%;">Figure 9</span>: </span><span id="S7.F9.4.2" class="ltx_text" style="font-size:90%;">
<span id="S7.F9.4.2.1" class="ltx_text ltx_font_bold">More visualization of the first PCA components.</span>
We compute the PCA between the patches from all of the images and show their first 3 components.
Each component corresponds to a specific color channel. Same parts are matched between related images depsite changes of pose, style or even objects.
Background is removed by removing patches with a negative score of the first PCA component.
</span></figcaption>
</figure>
</section>
<section id="S7.SS5.SSS0.Px3" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">PCA of patch features.</h5>

<div id="S7.SS5.SSS0.Px3.p1" class="ltx_para ltx_noindent">
<p id="S7.SS5.SSS0.Px3.p1.1" class="ltx_p">We show the results of the principal component analysis (PCA) performed on the patch features extracted by our model.
We keep only patches with a positive value after we threshold the first component.
This procedure turns out to separate the image’s main object from the background.
We compute a second PCA on the remaining patches across three images depicting the same category.
We color the three first components with three different colors and present the results in Fig. <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> and <a href="#S7.F9" title="Figure 9 ‣ Out-of-distribution generalization. ‣ 7.5 Qualitative Results ‣ 7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a>.
There are two interesting observations:
first, our unsupervised foreground / background detector, based on detecting the highest variance direction, performs very well and is capable of delineating the boundary of the main object in the picture.
Second, the other components correspond to "parts" of objects and match well for images of the same category.
This is an emerging property – our model was not trained to parse parts of objects.</p>
</div>
</section>
<section id="S7.SS5.SSS0.Px4" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Patch matching.</h5>

<div id="S7.SS5.SSS0.Px4.p1" class="ltx_para ltx_noindent">
<p id="S7.SS5.SSS0.Px4.p1.1" class="ltx_p">Finally, we explore what type of information our patch-level features contain by matching them across images.
We start by detecting the foreground object using the procedure described above.
Then, we compute the euclidean distance between patch features extracted from two images and map them by solving an assignment problem.
In order to reduce the number of matches, we then apply a non-maximum suppression to keep only the salient ones.
In Fig. <a href="#S7.F10" title="Figure 10 ‣ Patch matching. ‣ 7.5 Qualitative Results ‣ 7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a>, we show some examples of such matchings.</p>
</div>
<div id="S7.SS5.SSS0.Px4.p2" class="ltx_para ltx_noindent">
<p id="S7.SS5.SSS0.Px4.p2.1" class="ltx_p">We observe that the features seem to capture information about semantic regions that serve similar purpose in different objects or animals.
For instance, the wing of a plane matches the wing of a bird.
We also observe that the model is robust to style (image versus drawing), and to large variation of poses (see the elephant).</p>
</div>
<figure id="S7.F10" class="ltx_figure"><img src="/html/2304.07193/assets/new-figure-10.jpg" id="S7.F10.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="381" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S7.F10.3.1.1" class="ltx_text" style="font-size:90%;">Figure 10</span>: </span><span id="S7.F10.4.2" class="ltx_text" style="font-size:90%;">
<span id="S7.F10.4.2.1" class="ltx_text ltx_font_bold">Matching across images.</span> We match patch-level features between images from different domains, poses and even objects that share similar semantic information.
This exhibits the ability of our model to transfer across domains and understand relations between similar parts of different objects.
</span></figcaption>
</figure>
</section>
</section>
</section>
<section id="S8" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">8 </span>Fairness and Bias Analysis</h2>

<div id="S8.p1" class="ltx_para ltx_noindent">
<p id="S8.p1.1" class="ltx_p">We conduct two fairness evaluations of our models.
We probe for geographical fairness and potential harmful label associations.
For both evaluations, we experiment with our largest ViT-g model.</p>
</div>
<section id="S8.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">8.1 </span>Geographical Fairness</h3>

<div id="S8.SS1.p1" class="ltx_para ltx_noindent">
<p id="S8.SS1.p1.1" class="ltx_p">We evaluate geographical fairness on the Dollar Street dataset introduced in <cite class="ltx_cite ltx_citemacro_cite">De Vries et al. (<a href="#bib.bib32" title="" class="ltx_ref">2019</a>)</cite> using the evaluation protocol of <cite class="ltx_cite ltx_citemacro_citet">Goyal et al. (<a href="#bib.bib51" title="" class="ltx_ref">2022b</a>)</cite>.
This benchmark compares performance across countries and income levels. It contains 16,073 images from 289 households across 54 countries.
The task is to recognize 94 concepts that vary visually between households based on income or location.
In Table <a href="#S8.T12" title="Table 12 ‣ 8.1 Geographical Fairness ‣ 8 Fairness and Bias Analysis ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">12</span></a>, we compare our model with SEERv2 <cite class="ltx_cite ltx_citemacro_citep">(Goyal et al., <a href="#bib.bib50" title="" class="ltx_ref">2022a</a>)</cite>, a model trained on a geographically diverse set of images.
Our model is slightly fairer across regions and incomes than the SEERv2 model and significantly better than the supervised baseline reported by <cite class="ltx_cite ltx_citemacro_citet">Goyal et al. (<a href="#bib.bib50" title="" class="ltx_ref">2022a</a>)</cite>.
However, we still observe a significant difference between regions, particularly in Africa, where our model performance drops by 25.7% compared to Europe.
This shows that our model is still biased toward Western countries.
Similarly, our model performs significantly better on high-income households than low-income ones, with a difference of 31.7%.
Despite improvements, we observe significant biases in our models toward wealthy households from Western countries.</p>
</div>
<figure id="S8.T12" class="ltx_table">
<table id="S8.T12.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S8.T12.2.1" class="ltx_tr">
<td id="S8.T12.2.1.1" class="ltx_td ltx_border_tt"></td>
<td id="S8.T12.2.1.2" class="ltx_td ltx_border_tt"></td>
<td id="S8.T12.2.1.3" class="ltx_td ltx_border_tt"></td>
<td id="S8.T12.2.1.4" class="ltx_td ltx_border_tt"></td>
<td id="S8.T12.2.1.5" class="ltx_td ltx_align_center ltx_border_tt" colspan="3"><span id="S8.T12.2.1.5.1" class="ltx_text ltx_font_bold">Income buckets</span></td>
<td id="S8.T12.2.1.6" class="ltx_td ltx_border_tt"></td>
<td id="S8.T12.2.1.7" class="ltx_td ltx_align_center ltx_border_tt" colspan="4"><span id="S8.T12.2.1.7.1" class="ltx_text ltx_font_bold">Regions</span></td>
</tr>
<tr id="S8.T12.2.2" class="ltx_tr">
<td id="S8.T12.2.2.1" class="ltx_td ltx_align_left">Method</td>
<td id="S8.T12.2.2.2" class="ltx_td ltx_align_left">Arch.</td>
<td id="S8.T12.2.2.3" class="ltx_td ltx_align_left">Data</td>
<td id="S8.T12.2.2.4" class="ltx_td"></td>
<td id="S8.T12.2.2.5" class="ltx_td ltx_align_center ltx_border_t">low</td>
<td id="S8.T12.2.2.6" class="ltx_td ltx_align_center ltx_border_t">medium</td>
<td id="S8.T12.2.2.7" class="ltx_td ltx_align_center ltx_border_t">high</td>
<td id="S8.T12.2.2.8" class="ltx_td"></td>
<td id="S8.T12.2.2.9" class="ltx_td ltx_align_center ltx_border_t">Africa</td>
<td id="S8.T12.2.2.10" class="ltx_td ltx_align_center ltx_border_t">Asia</td>
<td id="S8.T12.2.2.11" class="ltx_td ltx_align_center ltx_border_t">Americas</td>
<td id="S8.T12.2.2.12" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">Europe</td>
</tr>
<tr id="S8.T12.2.3" class="ltx_tr">
<td id="S8.T12.2.3.1" class="ltx_td ltx_align_left ltx_border_t">SEERv2</td>
<td id="S8.T12.2.3.2" class="ltx_td ltx_align_left ltx_border_t">RG-10B</td>
<td id="S8.T12.2.3.3" class="ltx_td ltx_align_left ltx_border_t">IG-1B</td>
<td id="S8.T12.2.3.4" class="ltx_td ltx_border_t"></td>
<td id="S8.T12.2.3.5" class="ltx_td ltx_align_center ltx_border_t">59.7</td>
<td id="S8.T12.2.3.6" class="ltx_td ltx_align_center ltx_border_t">78.5</td>
<td id="S8.T12.2.3.7" class="ltx_td ltx_align_center ltx_border_t">86.6</td>
<td id="S8.T12.2.3.8" class="ltx_td ltx_border_t"></td>
<td id="S8.T12.2.3.9" class="ltx_td ltx_align_center ltx_border_t">65.9</td>
<td id="S8.T12.2.3.10" class="ltx_td ltx_align_center ltx_border_t">76.3</td>
<td id="S8.T12.2.3.11" class="ltx_td ltx_align_center ltx_border_t">81.1</td>
<td id="S8.T12.2.3.12" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">85.6</td>
</tr>
<tr id="S8.T12.2.4" class="ltx_tr">
<td id="S8.T12.2.4.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">DINOv2</td>
<td id="S8.T12.2.4.2" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">ViT-g/14</td>
<td id="S8.T12.2.4.3" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">LVD-142M</td>
<td id="S8.T12.2.4.4" class="ltx_td ltx_border_bb ltx_border_t"></td>
<td id="S8.T12.2.4.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">67.4</td>
<td id="S8.T12.2.4.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">83.3</td>
<td id="S8.T12.2.4.7" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">90.5</td>
<td id="S8.T12.2.4.8" class="ltx_td ltx_border_bb ltx_border_t"></td>
<td id="S8.T12.2.4.9" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">74.0</td>
<td id="S8.T12.2.4.10" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">81.6</td>
<td id="S8.T12.2.4.11" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">86.2</td>
<td id="S8.T12.2.4.12" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t">89.7</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S8.T12.4.1.1" class="ltx_text" style="font-size:90%;">Table 12</span>: </span><span id="S8.T12.5.2" class="ltx_text" style="font-size:90%;">
<span id="S8.T12.5.2.1" class="ltx_text ltx_font_bold">Geographical fairness and diversity analysis across income buckets and regions.</span>
</span></figcaption>
</figure>
</section>
<section id="S8.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">8.2 </span>Gender, Skintones and Age</h3>

<div id="S8.SS2.p1" class="ltx_para ltx_noindent">
<p id="S8.SS2.p1.1" class="ltx_p">In a second set of evaluations, we question how our model classifies images of people of different gender, skin tone, and age (all self-reported).
We follow the protocol of <cite class="ltx_cite ltx_citemacro_cite">Goyal et al. (<a href="#bib.bib51" title="" class="ltx_ref">2022b</a>)</cite>, where we train a multiclass classifier on a subset of 619 classes of ImageNet-22k.
We group the 619 classes into four broader categories: Human, Possibly Human, Non-Human, or Crime.
Non-Human and Crime are considered harmful.
Using this classifier, we run inference on 2955 images from the Casual Conversations dataset <cite class="ltx_cite ltx_citemacro_citep">(Hazirbas et al., <a href="#bib.bib56" title="" class="ltx_ref">2021</a>)</cite> and keep all labels in the top-5 that are assigned a probability of 0.1 or more.
Because of that, we can assign multiple classes to each image.
We make one modification to the original evaluation protocol: we do not backpropagate gradients to the backbone and keep it frozen.
We compare our model to SEERv2 in Table <a href="#S8.T13" title="Table 13 ‣ 8.2 Gender, Skintones and Age ‣ 8 Fairness and Bias Analysis ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">13</span></a>.</p>
</div>
<div id="S8.SS2.p2" class="ltx_para ltx_noindent">
<p id="S8.SS2.p2.1" class="ltx_p">Our model often classifies images of all groups as Human without large deviations across skin tones.
Neither SEERv2 nor DINOv2 predict harmful labels from the Non-Human or Crime meta-categories (except for two instances where the background contains bars visually similar to prison bars).
We see that our model triggers the Possibly-Human classes often.
This class is constructed from objects in ImageNet-22k that are often related to Humans, such as Scarf, Glasses, or Beard.
Our model often predicts the Possibly-Human class for men because of the prevalence of the Beard class.
No clear pattern indicates a bias against a particular group in this study.
While this is encouraging, we also acknowledge that a more thorough evaluation of biases may reveal flaws in our model.</p>
</div>
<figure id="S8.T13" class="ltx_table">
<table id="S8.T13.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S8.T13.2.1" class="ltx_tr">
<td id="S8.T13.2.1.1" class="ltx_td ltx_border_tt"></td>
<td id="S8.T13.2.1.2" class="ltx_td ltx_border_tt"></td>
<td id="S8.T13.2.1.3" class="ltx_td ltx_align_center ltx_border_tt" colspan="4"><span id="S8.T13.2.1.3.1" class="ltx_text ltx_font_bold">Gender Skintone</span></td>
<td id="S8.T13.2.1.4" class="ltx_td ltx_border_tt"></td>
<td id="S8.T13.2.1.5" class="ltx_td ltx_align_center ltx_border_tt" colspan="4"><span id="S8.T13.2.1.5.1" class="ltx_text ltx_font_bold">Age Groups</span></td>
</tr>
<tr id="S8.T13.2.2" class="ltx_tr">
<td id="S8.T13.2.2.1" class="ltx_td ltx_align_left"><span id="S8.T13.2.2.1.1" class="ltx_text ltx_font_bold">Model</span></td>
<td id="S8.T13.2.2.2" class="ltx_td ltx_align_left"><span id="S8.T13.2.2.2.1" class="ltx_text ltx_font_bold">Assoc.</span></td>
<td id="S8.T13.2.2.3" class="ltx_td ltx_align_center ltx_border_t">
<span id="S8.T13.2.2.3.1" class="ltx_text"></span> <span id="S8.T13.2.2.3.2" class="ltx_text">
<span id="S8.T13.2.2.3.2.1" class="ltx_tabular ltx_align_middle">
<span id="S8.T13.2.2.3.2.1.1" class="ltx_tr">
<span id="S8.T13.2.2.3.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">female</span></span>
<span id="S8.T13.2.2.3.2.1.2" class="ltx_tr">
<span id="S8.T13.2.2.3.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">darker</span></span>
</span></span><span id="S8.T13.2.2.3.3" class="ltx_text"></span>
</td>
<td id="S8.T13.2.2.4" class="ltx_td ltx_align_center ltx_border_t">
<span id="S8.T13.2.2.4.1" class="ltx_text"></span> <span id="S8.T13.2.2.4.2" class="ltx_text">
<span id="S8.T13.2.2.4.2.1" class="ltx_tabular ltx_align_middle">
<span id="S8.T13.2.2.4.2.1.1" class="ltx_tr">
<span id="S8.T13.2.2.4.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">female</span></span>
<span id="S8.T13.2.2.4.2.1.2" class="ltx_tr">
<span id="S8.T13.2.2.4.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">lighter</span></span>
</span></span><span id="S8.T13.2.2.4.3" class="ltx_text"></span>
</td>
<td id="S8.T13.2.2.5" class="ltx_td ltx_align_center ltx_border_t">
<span id="S8.T13.2.2.5.1" class="ltx_text"></span> <span id="S8.T13.2.2.5.2" class="ltx_text">
<span id="S8.T13.2.2.5.2.1" class="ltx_tabular ltx_align_middle">
<span id="S8.T13.2.2.5.2.1.1" class="ltx_tr">
<span id="S8.T13.2.2.5.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">male</span></span>
<span id="S8.T13.2.2.5.2.1.2" class="ltx_tr">
<span id="S8.T13.2.2.5.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">darker</span></span>
</span></span><span id="S8.T13.2.2.5.3" class="ltx_text"></span>
</td>
<td id="S8.T13.2.2.6" class="ltx_td ltx_align_center ltx_border_t">
<span id="S8.T13.2.2.6.1" class="ltx_text"></span> <span id="S8.T13.2.2.6.2" class="ltx_text">
<span id="S8.T13.2.2.6.2.1" class="ltx_tabular ltx_align_middle">
<span id="S8.T13.2.2.6.2.1.1" class="ltx_tr">
<span id="S8.T13.2.2.6.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">male</span></span>
<span id="S8.T13.2.2.6.2.1.2" class="ltx_tr">
<span id="S8.T13.2.2.6.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_center">lighter</span></span>
</span></span><span id="S8.T13.2.2.6.3" class="ltx_text"></span>
</td>
<td id="S8.T13.2.2.7" class="ltx_td"></td>
<td id="S8.T13.2.2.8" class="ltx_td ltx_align_center ltx_border_t">
<span id="S8.T13.2.2.8.1" class="ltx_text"></span> <span id="S8.T13.2.2.8.2" class="ltx_text">
<span id="S8.T13.2.2.8.2.1" class="ltx_tabular ltx_align_middle">
<span id="S8.T13.2.2.8.2.1.1" class="ltx_tr">
<span id="S8.T13.2.2.8.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">18-30</span></span>
</span></span><span id="S8.T13.2.2.8.3" class="ltx_text"></span>
</td>
<td id="S8.T13.2.2.9" class="ltx_td ltx_align_center ltx_border_t">
<span id="S8.T13.2.2.9.1" class="ltx_text"></span> <span id="S8.T13.2.2.9.2" class="ltx_text">
<span id="S8.T13.2.2.9.2.1" class="ltx_tabular ltx_align_middle">
<span id="S8.T13.2.2.9.2.1.1" class="ltx_tr">
<span id="S8.T13.2.2.9.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">30-45</span></span>
</span></span><span id="S8.T13.2.2.9.3" class="ltx_text"></span>
</td>
<td id="S8.T13.2.2.10" class="ltx_td ltx_align_center ltx_border_t">
<span id="S8.T13.2.2.10.1" class="ltx_text"></span> <span id="S8.T13.2.2.10.2" class="ltx_text">
<span id="S8.T13.2.2.10.2.1" class="ltx_tabular ltx_align_middle">
<span id="S8.T13.2.2.10.2.1.1" class="ltx_tr">
<span id="S8.T13.2.2.10.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">45-70</span></span>
</span></span><span id="S8.T13.2.2.10.3" class="ltx_text"></span>
</td>
<td id="S8.T13.2.2.11" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_t">
<span id="S8.T13.2.2.11.1" class="ltx_text"></span> <span id="S8.T13.2.2.11.2" class="ltx_text">
<span id="S8.T13.2.2.11.2.1" class="ltx_tabular ltx_align_middle">
<span id="S8.T13.2.2.11.2.1.1" class="ltx_tr">
<span id="S8.T13.2.2.11.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">70+</span></span>
</span></span><span id="S8.T13.2.2.11.3" class="ltx_text"></span>
</td>
</tr>
<tr id="S8.T13.2.3" class="ltx_tr">
<td id="S8.T13.2.3.1" class="ltx_td ltx_align_left ltx_border_t">SEER</td>
<td id="S8.T13.2.3.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S8.T13.2.3.2.1" class="ltx_text ltx_font_typewriter">Non-Human</span></td>
<td id="S8.T13.2.3.3" class="ltx_td ltx_align_right ltx_border_t">0.0</td>
<td id="S8.T13.2.3.4" class="ltx_td ltx_align_right ltx_border_t">0.0</td>
<td id="S8.T13.2.3.5" class="ltx_td ltx_align_right ltx_border_t">0.0</td>
<td id="S8.T13.2.3.6" class="ltx_td ltx_align_right ltx_border_t">0.0</td>
<td id="S8.T13.2.3.7" class="ltx_td ltx_border_t"></td>
<td id="S8.T13.2.3.8" class="ltx_td ltx_align_right ltx_border_t">0.0</td>
<td id="S8.T13.2.3.9" class="ltx_td ltx_align_right ltx_border_t">0.0</td>
<td id="S8.T13.2.3.10" class="ltx_td ltx_align_right ltx_border_t">0.0</td>
<td id="S8.T13.2.3.11" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t">0.0</td>
</tr>
<tr id="S8.T13.2.4" class="ltx_tr">
<td id="S8.T13.2.4.1" class="ltx_td ltx_align_left">RG-10B</td>
<td id="S8.T13.2.4.2" class="ltx_td ltx_align_left"><span id="S8.T13.2.4.2.1" class="ltx_text ltx_font_typewriter">Crime</span></td>
<td id="S8.T13.2.4.3" class="ltx_td ltx_align_right">0.0</td>
<td id="S8.T13.2.4.4" class="ltx_td ltx_align_right">0.0</td>
<td id="S8.T13.2.4.5" class="ltx_td ltx_align_right">0.0</td>
<td id="S8.T13.2.4.6" class="ltx_td ltx_align_right">0.0</td>
<td id="S8.T13.2.4.7" class="ltx_td"></td>
<td id="S8.T13.2.4.8" class="ltx_td ltx_align_right">0.0</td>
<td id="S8.T13.2.4.9" class="ltx_td ltx_align_right">0.0</td>
<td id="S8.T13.2.4.10" class="ltx_td ltx_align_right">0.0</td>
<td id="S8.T13.2.4.11" class="ltx_td ltx_nopad_r ltx_align_right">0.0</td>
</tr>
<tr id="S8.T13.2.5" class="ltx_tr">
<td id="S8.T13.2.5.1" class="ltx_td"></td>
<td id="S8.T13.2.5.2" class="ltx_td ltx_align_left"><span id="S8.T13.2.5.2.1" class="ltx_text ltx_font_typewriter">Human</span></td>
<td id="S8.T13.2.5.3" class="ltx_td ltx_align_right">94.9</td>
<td id="S8.T13.2.5.4" class="ltx_td ltx_align_right">95.8</td>
<td id="S8.T13.2.5.5" class="ltx_td ltx_align_right">86.6</td>
<td id="S8.T13.2.5.6" class="ltx_td ltx_align_right">79.0</td>
<td id="S8.T13.2.5.7" class="ltx_td"></td>
<td id="S8.T13.2.5.8" class="ltx_td ltx_align_right">90.5</td>
<td id="S8.T13.2.5.9" class="ltx_td ltx_align_right">88.3</td>
<td id="S8.T13.2.5.10" class="ltx_td ltx_align_right">91.9</td>
<td id="S8.T13.2.5.11" class="ltx_td ltx_nopad_r ltx_align_right">82.3</td>
</tr>
<tr id="S8.T13.2.6" class="ltx_tr">
<td id="S8.T13.2.6.1" class="ltx_td"></td>
<td id="S8.T13.2.6.2" class="ltx_td ltx_align_left"><span id="S8.T13.2.6.2.1" class="ltx_text ltx_font_typewriter">Possibly-Human</span></td>
<td id="S8.T13.2.6.3" class="ltx_td ltx_align_right">13.6</td>
<td id="S8.T13.2.6.4" class="ltx_td ltx_align_right">6.7</td>
<td id="S8.T13.2.6.5" class="ltx_td ltx_align_right">65.0</td>
<td id="S8.T13.2.6.6" class="ltx_td ltx_align_right">60.2</td>
<td id="S8.T13.2.6.7" class="ltx_td"></td>
<td id="S8.T13.2.6.8" class="ltx_td ltx_align_right">32.8</td>
<td id="S8.T13.2.6.9" class="ltx_td ltx_align_right">37.2</td>
<td id="S8.T13.2.6.10" class="ltx_td ltx_align_right">29.4</td>
<td id="S8.T13.2.6.11" class="ltx_td ltx_nopad_r ltx_align_right">6.5</td>
</tr>
<tr id="S8.T13.2.7" class="ltx_tr">
<td id="S8.T13.2.7.1" class="ltx_td ltx_align_left ltx_border_t">DINOv2</td>
<td id="S8.T13.2.7.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S8.T13.2.7.2.1" class="ltx_text ltx_font_typewriter">Non-Human</span></td>
<td id="S8.T13.2.7.3" class="ltx_td ltx_align_right ltx_border_t">0.0</td>
<td id="S8.T13.2.7.4" class="ltx_td ltx_align_right ltx_border_t">0.0</td>
<td id="S8.T13.2.7.5" class="ltx_td ltx_align_right ltx_border_t">0.0</td>
<td id="S8.T13.2.7.6" class="ltx_td ltx_align_right ltx_border_t">0.0</td>
<td id="S8.T13.2.7.7" class="ltx_td ltx_border_t"></td>
<td id="S8.T13.2.7.8" class="ltx_td ltx_align_right ltx_border_t">0.0</td>
<td id="S8.T13.2.7.9" class="ltx_td ltx_align_right ltx_border_t">0.0</td>
<td id="S8.T13.2.7.10" class="ltx_td ltx_align_right ltx_border_t">0.0</td>
<td id="S8.T13.2.7.11" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t">0.0</td>
</tr>
<tr id="S8.T13.2.8" class="ltx_tr">
<td id="S8.T13.2.8.1" class="ltx_td ltx_align_left">ViT-g/14</td>
<td id="S8.T13.2.8.2" class="ltx_td ltx_align_left"><span id="S8.T13.2.8.2.1" class="ltx_text ltx_font_typewriter">Crime</span></td>
<td id="S8.T13.2.8.3" class="ltx_td ltx_align_right">0.0</td>
<td id="S8.T13.2.8.4" class="ltx_td ltx_align_right">0.0</td>
<td id="S8.T13.2.8.5" class="ltx_td ltx_align_right">0.2</td>
<td id="S8.T13.2.8.6" class="ltx_td ltx_align_right">0.0</td>
<td id="S8.T13.2.8.7" class="ltx_td"></td>
<td id="S8.T13.2.8.8" class="ltx_td ltx_align_right">0.0</td>
<td id="S8.T13.2.8.9" class="ltx_td ltx_align_right">0.1</td>
<td id="S8.T13.2.8.10" class="ltx_td ltx_align_right">0.0</td>
<td id="S8.T13.2.8.11" class="ltx_td ltx_nopad_r ltx_align_right">0.0</td>
</tr>
<tr id="S8.T13.2.9" class="ltx_tr">
<td id="S8.T13.2.9.1" class="ltx_td"></td>
<td id="S8.T13.2.9.2" class="ltx_td ltx_align_left"><span id="S8.T13.2.9.2.1" class="ltx_text ltx_font_typewriter">Human</span></td>
<td id="S8.T13.2.9.3" class="ltx_td ltx_align_right">97.3</td>
<td id="S8.T13.2.9.4" class="ltx_td ltx_align_right">97.7</td>
<td id="S8.T13.2.9.5" class="ltx_td ltx_align_right">86.1</td>
<td id="S8.T13.2.9.6" class="ltx_td ltx_align_right">84.0</td>
<td id="S8.T13.2.9.7" class="ltx_td"></td>
<td id="S8.T13.2.9.8" class="ltx_td ltx_align_right">91.2</td>
<td id="S8.T13.2.9.9" class="ltx_td ltx_align_right">90.2</td>
<td id="S8.T13.2.9.10" class="ltx_td ltx_align_right">93.2</td>
<td id="S8.T13.2.9.11" class="ltx_td ltx_nopad_r ltx_align_right">88.7</td>
</tr>
<tr id="S8.T13.2.10" class="ltx_tr">
<td id="S8.T13.2.10.1" class="ltx_td ltx_border_bb"></td>
<td id="S8.T13.2.10.2" class="ltx_td ltx_align_left ltx_border_bb"><span id="S8.T13.2.10.2.1" class="ltx_text ltx_font_typewriter">Possibly-Human</span></td>
<td id="S8.T13.2.10.3" class="ltx_td ltx_align_right ltx_border_bb">15.8</td>
<td id="S8.T13.2.10.4" class="ltx_td ltx_align_right ltx_border_bb">17.2</td>
<td id="S8.T13.2.10.5" class="ltx_td ltx_align_right ltx_border_bb">52.2</td>
<td id="S8.T13.2.10.6" class="ltx_td ltx_align_right ltx_border_bb">48.1</td>
<td id="S8.T13.2.10.7" class="ltx_td ltx_border_bb"></td>
<td id="S8.T13.2.10.8" class="ltx_td ltx_align_right ltx_border_bb">35.3</td>
<td id="S8.T13.2.10.9" class="ltx_td ltx_align_right ltx_border_bb">37.3</td>
<td id="S8.T13.2.10.10" class="ltx_td ltx_align_right ltx_border_bb">23.0</td>
<td id="S8.T13.2.10.11" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_bb">9.7</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S8.T13.4.1.1" class="ltx_text" style="font-size:90%;">Table 13</span>: </span><span id="S8.T13.5.2" class="ltx_text" style="font-size:90%;">
<span id="S8.T13.5.2.1" class="ltx_text ltx_font_bold">Label association fairness evaluation across gender, skintones and age groups.</span>
We follow the protocol proposed by <cite class="ltx_cite ltx_citemacro_citet">Goyal et al. (<a href="#bib.bib51" title="" class="ltx_ref">2022b</a>)</cite> with a slight modification.
Instead of finetuning the backbone, we simply learn a linear classifier on the subset of 619 classes of ImageNet-22k.
</span></figcaption>
</figure>
</section>
</section>
<section id="S9" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">9 </span>Estimating the Environmental Impact of Training our Models</h2>

<figure id="S9.T14" class="ltx_table">
<table id="S9.T14.1" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S9.T14.1.2" class="ltx_tr">
<td id="S9.T14.1.2.1" class="ltx_td ltx_align_left ltx_border_tt">Model to</td>
<td id="S9.T14.1.2.2" class="ltx_td ltx_align_center ltx_border_tt" rowspan="2"><span id="S9.T14.1.2.2.1" class="ltx_text">GPU Type</span></td>
<td id="S9.T14.1.2.3" class="ltx_td ltx_align_center ltx_border_tt">GPU Power</td>
<td id="S9.T14.1.2.4" class="ltx_td ltx_align_center ltx_border_tt" rowspan="2"><span id="S9.T14.1.2.4.1" class="ltx_text">GPU-hours</span></td>
<td id="S9.T14.1.2.5" class="ltx_td ltx_align_center ltx_border_tt" rowspan="2"><span id="S9.T14.1.2.5.1" class="ltx_text">PUE</span></td>
<td id="S9.T14.1.2.6" class="ltx_td ltx_align_center ltx_border_tt">Total power</td>
<td id="S9.T14.1.2.7" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_tt">Carbon emitted</td>
</tr>
<tr id="S9.T14.1.1" class="ltx_tr">
<td id="S9.T14.1.1.2" class="ltx_td ltx_align_left">Reproduce</td>
<td id="S9.T14.1.1.3" class="ltx_td ltx_align_center">consumption</td>
<td id="S9.T14.1.1.4" class="ltx_td ltx_align_center">consumption</td>
<td id="S9.T14.1.1.1" class="ltx_td ltx_nopad_r ltx_align_center">(tCO<sub id="S9.T14.1.1.1.1" class="ltx_sub">2</sub>eq)</td>
</tr>
<tr id="S9.T14.1.3" class="ltx_tr">
<td id="S9.T14.1.3.1" class="ltx_td ltx_align_left ltx_border_bb ltx_border_t">DINOv2-g</td>
<td id="S9.T14.1.3.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">A100-40GB</td>
<td id="S9.T14.1.3.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">400W</td>
<td id="S9.T14.1.3.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">22,016</td>
<td id="S9.T14.1.3.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">1.1</td>
<td id="S9.T14.1.3.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">9.7 MWh</td>
<td id="S9.T14.1.3.7" class="ltx_td ltx_nopad_r ltx_align_center ltx_border_bb ltx_border_t">3.7</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S9.T14.7.2.1" class="ltx_text" style="font-size:90%;">Table 14</span>: </span><span id="S9.T14.3.1" class="ltx_text" style="font-size:90%;">
<span id="S9.T14.3.1.1" class="ltx_text ltx_font_bold">Carbon footprint of reproducing DINOv2.</span>
We report the potential carbon emission of reproducing DINOv2-g when assuming a power consumption for the A100-40GB of 400W, a PUE of 1.1 and carbon intensity factor of 0.385 kg CO<sub id="S9.T14.3.1.2" class="ltx_sub">2</sub>e per KWh.

</span></figcaption>
</figure>
<div id="S9.p1" class="ltx_para ltx_noindent">
<p id="S9.p1.2" class="ltx_p">Training foundation models consumes a significant amount of energy, resulting in carbon dioxide emissions.
<cite class="ltx_cite ltx_citemacro_citet">Patterson et al. (<a href="#bib.bib87" title="" class="ltx_ref">2021</a>)</cite> propose a methodology to report an estimation of the carbon emitted during the training of a model based on the specifics of the data center and its power grid.
This computation informs the design of the data center used for the training of models and the choice of location for data centers.
This methodology requires to know the specifics of the data center used for training, which can be complex when multiple data centers are involved over time.
Additionally, these specifics are most often not in the control of the AI practitioner, and hence, this methodology is less helpful when practioners make technical decisions about future trainings.
Instead, in this section, we follow an alternative that reports the potential carbon emission of retraining a similar model in an average data center located in the US.
This methodology was used in previous work in natural language processing <cite class="ltx_cite ltx_citemacro_citep">(Strubell et al., <a href="#bib.bib109" title="" class="ltx_ref">2019</a>; Touvron et al., <a href="#bib.bib115" title="" class="ltx_ref">2023</a>)</cite> to establish an apple-to-apple comparison between pretraining schemes.
More precisely, we fix the value of all exogenous variables, i.e., the Power Usage Effectiveness (PUE) and carbon intensity factor of a power grid to the same values as in <cite class="ltx_cite ltx_citemacro_citet">Touvron et al. (<a href="#bib.bib115" title="" class="ltx_ref">2023</a>)</cite>, that is, a PUE of 1.1 and the carbon intensity factor to the US average of 0.385 kg CO<sub id="S9.p1.2.1" class="ltx_sub">2</sub>eq/KWh.
We use the same formula as in <cite class="ltx_cite ltx_citemacro_cite">Patterson et al. (<a href="#bib.bib87" title="" class="ltx_ref">2021</a>)</cite> to estimate the potential energy consumption and the carbon emission.
For the power consumption of an A100-80GB, we take the thermal design power for NVLink systems, which is 400W.
We report the potential carbon emission of retraining a DINOv2 ViT-g in Table <a href="#S9.T14" title="Table 14 ‣ 9 Estimating the Environmental Impact of Training our Models ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">14</span></a>.
For comparison, retraining an OpenCLIP ViT-L or OpenCLIP ViT-G would require 22.4 MWh and 118.9 MWh, respectively, if run in the same data center.
This is 10<math id="S9.p1.2.m2.1" class="ltx_Math" alttext="\times" display="inline"><semantics id="S9.p1.2.m2.1a"><mo id="S9.p1.2.m2.1.1" xref="S9.p1.2.m2.1.1.cmml">×</mo><annotation-xml encoding="MathML-Content" id="S9.p1.2.m2.1b"><times id="S9.p1.2.m2.1.1.cmml" xref="S9.p1.2.m2.1.1"></times></annotation-xml><annotation encoding="application/x-tex" id="S9.p1.2.m2.1c">\times</annotation></semantics></math> more carbon emission.
Note that this comparison is not fair to them, since they also train a text encoder in parallel, and we thus do not report them in the table.
However, it gives a reasonable guideline for those who are interested in training only visual features: in this context, training a self-supervised model is preferable in terms of carbon emission.
Training a text-guided model still makes sense when planning to reuse the text encoder.</p>
</div>
<section id="S9.SS0.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Carbon footprint of the whole project.</h5>

<div id="S9.SS0.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="S9.SS0.SSS0.Px1.p1.5" class="ltx_p">Additionally, we estimate the footprint of the whole project to be between <math id="S9.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="0.5" display="inline"><semantics id="S9.SS0.SSS0.Px1.p1.1.m1.1a"><mn id="S9.SS0.SSS0.Px1.p1.1.m1.1.1" xref="S9.SS0.SSS0.Px1.p1.1.m1.1.1.cmml">0.5</mn><annotation-xml encoding="MathML-Content" id="S9.SS0.SSS0.Px1.p1.1.m1.1b"><cn type="float" id="S9.SS0.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S9.SS0.SSS0.Px1.p1.1.m1.1.1">0.5</cn></annotation-xml><annotation encoding="application/x-tex" id="S9.SS0.SSS0.Px1.p1.1.m1.1c">0.5</annotation></semantics></math>k and <math id="S9.SS0.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S9.SS0.SSS0.Px1.p1.2.m2.1a"><mn id="S9.SS0.SSS0.Px1.p1.2.m2.1.1" xref="S9.SS0.SSS0.Px1.p1.2.m2.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S9.SS0.SSS0.Px1.p1.2.m2.1b"><cn type="integer" id="S9.SS0.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S9.SS0.SSS0.Px1.p1.2.m2.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S9.SS0.SSS0.Px1.p1.2.m2.1c">1</annotation></semantics></math>k tCO<sub id="S9.SS0.SSS0.Px1.p1.5.1" class="ltx_sub">2</sub>eq using the same grid as presented above <span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>For context, a full Boeing 777 return flight between London and New York corresponds to approximately 560 tCO<sub id="footnote3.1" class="ltx_sub">2</sub>eq.</span></span></span>.
This carbon footprint represents in the order of <math id="S9.SS0.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="200" display="inline"><semantics id="S9.SS0.SSS0.Px1.p1.4.m4.1a"><mn id="S9.SS0.SSS0.Px1.p1.4.m4.1.1" xref="S9.SS0.SSS0.Px1.p1.4.m4.1.1.cmml">200</mn><annotation-xml encoding="MathML-Content" id="S9.SS0.SSS0.Px1.p1.4.m4.1b"><cn type="integer" id="S9.SS0.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S9.SS0.SSS0.Px1.p1.4.m4.1.1">200</cn></annotation-xml><annotation encoding="application/x-tex" id="S9.SS0.SSS0.Px1.p1.4.m4.1c">200</annotation></semantics></math>k GPU-days.
The primary sources of emissions are the self-supervised pre-trainings of the models.
For example, a single pre-training of a ViT-g model (22k GPU-hours) emits 3.7 tons of CO<sub id="S9.SS0.SSS0.Px1.p1.5.2" class="ltx_sub">2</sub>eq, while a finetuning on ImageNet-1k (1k GPU-hours) emits 0.2 tons.
This estimate only considers the GPUs’ electricity consumption and ignores other emissions, such as their manufacturing and disposal.</p>
</div>
</section>
</section>
<section id="S10" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">10 </span>Future work and Discussion</h2>

<div id="S10.p1" class="ltx_para ltx_noindent">
<p id="S10.p1.1" class="ltx_p">In this work, we present DINOv2, a new series of image encoders pretrained on large curated data with no supervision.
This is the first SSL work on image data that leads to visual features that close the performance gap with (weakly) supervised alternatives across a wide range of benchmarks and without the need for finetuning.

We can attribute the strong performance of the DINOv2 family of models to several factors:
<span id="S10.p1.1.1" class="ltx_text ltx_font_bold">i</span>) an improved training recipe with better hyperparameters and regularization (Table <a href="#S6.T1" title="Table 1 ‣ 6.1 Improved Training Recipe ‣ 6 Ablation Studies ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>),
<span id="S10.p1.1.2" class="ltx_text ltx_font_bold">ii</span>) a larger model scale with improved results regardless of the data used for training (Fig. <a href="#S6.F4" title="Figure 4 ‣ 6.3 Model Size and Data ‣ 6 Ablation Studies ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>),
<span id="S10.p1.1.3" class="ltx_text ltx_font_bold">iii</span>) a larger dataset (Fig. <a href="#S6.F4" title="Figure 4 ‣ 6.3 Model Size and Data ‣ 6 Ablation Studies ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>) and
<span id="S10.p1.1.4" class="ltx_text ltx_font_bold">iv</span>) the distillation process that makes smaller models benefit from the performance of the strongest ViT-g model (Fig. <a href="#S6.F5" title="Figure 5 ‣ 6.5 Impact of Knowledge Distillation ‣ 6 Ablation Studies ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>).

A few properties emerge from these models, such as an understanding of object parts and scene geometry regardless of the image domains.
We expect that more of these properties will emerge at larger scales of models and data, akin to instruction emergence in large language models, and plan to continue scaling along these axes.
This paper also demonstrates that these visual features are compatible with classifiers as simple as linear layers - meaning the underlying information is <span id="S10.p1.1.5" class="ltx_text ltx_font_italic">readily available</span>.
In future work, we plan to leverage this ability to train a a language-enabled AI system that can process visual features as if they were word tokens, and extract the required information to ground the system.</p>
</div>
<section id="S10.SS0.SSSx1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Acknowledgments.</h4>

<div id="S10.SS0.SSSx1.p1" class="ltx_para ltx_noindent">
<p id="S10.SS0.SSSx1.p1.1" class="ltx_p">We thank Mathilde Caron for initial discussions that led to this work.
Julien Mairal was supported by the ERC grant number 101087696 (APHELAIA project) and by ANR 3IA MIAI@Grenoble Alpes (ANR-19-P3IA-0003).
We thank Olivia Joulin for the horse drawing used in Fig. <a href="#S7.F10" title="Figure 10 ‣ Patch matching. ‣ 7.5 Qualitative Results ‣ 7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">10</span></a>.
We thank Madeleine and Léon for posing for Fig. <a href="#S7.F8" title="Figure 8 ‣ Out-of-distribution generalization. ‣ 7.5 Qualitative Results ‣ 7 Results ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>
We also thank the rest of FAIR and Meta AI for feedback on this work through the entire project.</p>
</div>
</section>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Amir et al. (2022)</span>
<span class="ltx_bibblock">
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel.

</span>
<span class="ltx_bibblock">Deep vit features as dense visual descriptors.

</span>
<span class="ltx_bibblock">In <em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">ECCV workshop on "What is Motion For?"</em>, 2022.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Asano et al. (2020)</span>
<span class="ltx_bibblock">
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi.

</span>
<span class="ltx_bibblock">Self-labelling via simultaneous clustering and representation
learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">ICLR</em>, 2020.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Assran et al. (2022)</span>
<span class="ltx_bibblock">
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes,
Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas.

</span>
<span class="ltx_bibblock">Masked siamese networks for label-efficient learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">ECCV</em>, 2022.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Assran et al. (2023)</span>
<span class="ltx_bibblock">
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent,
Michael Rabbat, Yann LeCun, and Nicolas Ballas.

</span>
<span class="ltx_bibblock">Self-supervised learning from images with a joint-embedding
predictive architecture.

</span>
<span class="ltx_bibblock">In <em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2023.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Baevski et al. (2022)</span>
<span class="ltx_bibblock">
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael
Auli.

</span>
<span class="ltx_bibblock">Data2vec: A general framework for self-supervised learning in speech,
vision and language.

</span>
<span class="ltx_bibblock">In <em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">ICML</em>, 2022.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bao et al. (2021)</span>
<span class="ltx_bibblock">
Hangbo Bao, Li Dong, and Furu Wei.

</span>
<span class="ltx_bibblock">Beit: Bert pre-training of image transformers.

</span>
<span class="ltx_bibblock">In <em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">ICLR</em>, 2021.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Beirlant et al. (1997)</span>
<span class="ltx_bibblock">
Jan Beirlant, Edward J Dudewicz, László Györfi, Edward C Van der
Meulen, et al.

</span>
<span class="ltx_bibblock">Nonparametric entropy estimation: An overview.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">International Journal of Mathematical and Statistical
Sciences</em>, 6(1):17–39, 1997.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Berman et al. (2019)</span>
<span class="ltx_bibblock">
Maxim Berman, Hervé Jégou, Vedaldi Andrea, Iasonas Kokkinos, and
Matthijs Douze.

</span>
<span class="ltx_bibblock">MultiGrain: a unified image embedding for classes and instances.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1902.05509</em>, 2019.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Beyer et al. (2020)</span>
<span class="ltx_bibblock">
Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and
Aäron van den Oord.

</span>
<span class="ltx_bibblock">Are we done with imagenet?

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2006.07159</em>, 2020.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Beyer et al. (2023)</span>
<span class="ltx_bibblock">
Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon
Kornblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim
Alabdulmohsin, and Filip Pavetic.

</span>
<span class="ltx_bibblock">Flexivit: One model for all patch sizes.

</span>
<span class="ltx_bibblock">In <em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2023.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bhat et al. (2021)</span>
<span class="ltx_bibblock">
Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka.

</span>
<span class="ltx_bibblock">AdaBins: Depth estimation using adaptive bins.

</span>
<span class="ltx_bibblock">In <em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2021.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bojanowski &amp; Joulin (2017)</span>
<span class="ltx_bibblock">
Piotr Bojanowski and Armand Joulin.

</span>
<span class="ltx_bibblock">Unsupervised learning by predicting noise.

</span>
<span class="ltx_bibblock">In <em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">ICML</em>, 2017.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bommasani et al. (2021)</span>
<span class="ltx_bibblock">
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney
von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma
Brunskill, et al.

</span>
<span class="ltx_bibblock">On the opportunities and risks of foundation models.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2108.07258</em>, 2021.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bossard et al. (2014)</span>
<span class="ltx_bibblock">
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.

</span>
<span class="ltx_bibblock">Food-101 – mining discriminative components with random forests.

</span>
<span class="ltx_bibblock">In <em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">ECCV</em>, 2014.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span>
<span class="ltx_bibblock">
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
et al.

</span>
<span class="ltx_bibblock">Language models are few-shot learners.

</span>
<span class="ltx_bibblock">In <em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>, 2020.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Caron et al. (2018)</span>
<span class="ltx_bibblock">
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.

</span>
<span class="ltx_bibblock">Deep clustering for unsupervised learning of visual features.

</span>
<span class="ltx_bibblock">In <em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">ECCV</em>, 2018.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Caron et al. (2019)</span>
<span class="ltx_bibblock">
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin.

</span>
<span class="ltx_bibblock">Unsupervised pre-training of image features on non-curated data.

</span>
<span class="ltx_bibblock">In <em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">ICCV</em>, 2019.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Caron et al. (2020)</span>
<span class="ltx_bibblock">
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and
Armand Joulin.

</span>
<span class="ltx_bibblock">Unsupervised learning of visual features by contrasting cluster
assignments.

</span>
<span class="ltx_bibblock">In <em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>, 2020.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Caron et al. (2021)</span>
<span class="ltx_bibblock">
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal,
Piotr Bojanowski, and Armand Joulin.

</span>
<span class="ltx_bibblock">Emerging properties in self-supervised vision transformers.

</span>
<span class="ltx_bibblock">In <em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">ICCV</em>, 2021.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2018)</span>
<span class="ltx_bibblock">
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig
Adam.

</span>
<span class="ltx_bibblock">Encoder-decoder with atrous separable convolution for semantic image
segmentation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">ECCV</em>, 2018.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2020)</span>
<span class="ltx_bibblock">
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.

</span>
<span class="ltx_bibblock">A simple framework for contrastive learning of visual
representations.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">ICML</em>, 2020.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2023a)</span>
<span class="ltx_bibblock">
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu
Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et al.

</span>
<span class="ltx_bibblock">Symbolic discovery of optimization algorithms.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2302.06675</em>, 2023a.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen &amp; He (2021)</span>
<span class="ltx_bibblock">
Xinlei Chen and Kaiming He.

</span>
<span class="ltx_bibblock">Exploring simple siamese representation learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2021.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2021)</span>
<span class="ltx_bibblock">
Xinlei Chen, Saining Xie, and Kaiming He.

</span>
<span class="ltx_bibblock">An empirical study of training self-supervised vision transformers.

</span>
<span class="ltx_bibblock">In <em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">ICCV</em>, 2021.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2023b)</span>
<span class="ltx_bibblock">
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and
Yu Qiao.

</span>
<span class="ltx_bibblock">Vision transformer adapter for dense predictions.

</span>
<span class="ltx_bibblock">In <em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">ICLR</em>, 2023b.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cheng et al. (2022)</span>
<span class="ltx_bibblock">
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit
Girdhar.

</span>
<span class="ltx_bibblock">Masked-attention mask transformer for universal image segmentation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2022.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cherti et al. (2023)</span>
<span class="ltx_bibblock">
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel
Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev.

</span>
<span class="ltx_bibblock">Reproducible scaling laws for contrastive language-image learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2023.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chowdhery et al. (2022)</span>
<span class="ltx_bibblock">
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra,
Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian
Gehrmann, et al.

</span>
<span class="ltx_bibblock">Palm: Scaling language modeling with pathways.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2204.02311</em>, 2022.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cimpoi et al. (2014)</span>
<span class="ltx_bibblock">
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi.

</span>
<span class="ltx_bibblock">Describing textures in the wild.

</span>
<span class="ltx_bibblock">In <em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2014.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cordts et al. (2016)</span>
<span class="ltx_bibblock">
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler,
Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.

</span>
<span class="ltx_bibblock">The cityscapes dataset for semantic urban scene understanding.

</span>
<span class="ltx_bibblock">In <em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2016.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dao et al. (2022)</span>
<span class="ltx_bibblock">
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.

</span>
<span class="ltx_bibblock">Flashattention: Fast and memory-efficient exact attention with
io-awareness.

</span>
<span class="ltx_bibblock">In <em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>, 2022.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">De Vries et al. (2019)</span>
<span class="ltx_bibblock">
Terrance De Vries, Ishan Misra, Changhan Wang, and Laurens Van der Maaten.

</span>
<span class="ltx_bibblock">Does object recognition work for everyone?

</span>
<span class="ltx_bibblock">In <em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">CVPR workshops</em>, 2019.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Delattre &amp; Fournier (2017)</span>
<span class="ltx_bibblock">
Sylvain Delattre and Nicolas Fournier.

</span>
<span class="ltx_bibblock">On the kozachenko–leonenko entropy estimator.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">Journal of Statistical Planning and Inference</em>, 185:69–93, 2017.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Deng et al. (2009)</span>
<span class="ltx_bibblock">
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.

</span>
<span class="ltx_bibblock">Imagenet: A large-scale hierarchical image database.

</span>
<span class="ltx_bibblock">In <em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2009.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Devlin et al. (2019)</span>
<span class="ltx_bibblock">
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.

</span>
<span class="ltx_bibblock">Bert: Pre-training of deep bidirectional transformers for language
understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">NAACL</em>, 2019.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Doersch et al. (2015)</span>
<span class="ltx_bibblock">
Carl Doersch, Abhinav Gupta, and Alexei A Efros.

</span>
<span class="ltx_bibblock">Unsupervised visual representation learning by context prediction.

</span>
<span class="ltx_bibblock">In <em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">ICCV</em>, 2015.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dosovitskiy et al. (2016)</span>
<span class="ltx_bibblock">
Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin
Riedmiller, and Thomas Brox.

</span>
<span class="ltx_bibblock">Discriminative unsupervised feature learning with exemplar
convolutional neural networks.

</span>
<span class="ltx_bibblock"><em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">TPAMI</em>, 2016.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dosovitskiy et al. (2021)</span>
<span class="ltx_bibblock">
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, et al.

</span>
<span class="ltx_bibblock">An image is worth 16x16 words: Transformers for image recognition at
scale.

</span>
<span class="ltx_bibblock">In <em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">ICLR</em>, 2021.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Douze et al. (2009)</span>
<span class="ltx_bibblock">
Matthijs Douze, Hervé Jégou, Harsimrat Sandhawalia, Laurent Amsaleg,
and Cordelia Schmid.

</span>
<span class="ltx_bibblock">Evaluation of gist descriptors for web-scale image search.

</span>
<span class="ltx_bibblock">In <em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">CIVR</em>, 2009.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Duval et al. (2023)</span>
<span class="ltx_bibblock">
Quentin Duval, Ishan Misra, and Nicolas Ballas.

</span>
<span class="ltx_bibblock">A simple recipe for competitive low-compute self supervised vision
models.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2301.09451</em>, 2023.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">El-Nouby et al. (2021)</span>
<span class="ltx_bibblock">
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé
Jegou, and Edouard Grave.

</span>
<span class="ltx_bibblock">Are large-scale datasets necessary for self-supervised pre-training?

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2112.10740</em>, 2021.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Everingham et al. (2010)</span>
<span class="ltx_bibblock">
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew
Zisserman.

</span>
<span class="ltx_bibblock">The pascal visual object classes (voc) challenge.

</span>
<span class="ltx_bibblock"><em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">IJCV</em>, 2010.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fang et al. (2023)</span>
<span class="ltx_bibblock">
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun
Huang, Xinlong Wang, and Yue Cao.

</span>
<span class="ltx_bibblock">Eva: Exploring the limits of masked visual representation learning at
scale.

</span>
<span class="ltx_bibblock">In <em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2023.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fei-Fei et al. (2004)</span>
<span class="ltx_bibblock">
Li Fei-Fei, Rob Fergus, and Pietro Perona.

</span>
<span class="ltx_bibblock">Learning generative visual models from few training examples: An
incremental bayesian approach tested on 101 object categories.

</span>
<span class="ltx_bibblock">In <em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2004.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Geiger et al. (2013)</span>
<span class="ltx_bibblock">
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun.

</span>
<span class="ltx_bibblock">Vision meets robotics: The kitti dataset.

</span>
<span class="ltx_bibblock"><em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">IJRR</em>, 2013.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Gidaris et al. (2018)</span>
<span class="ltx_bibblock">
Spyros Gidaris, Praveer Singh, and Nikos Komodakis.

</span>
<span class="ltx_bibblock">Unsupervised representation learning by predicting image rotations.

</span>
<span class="ltx_bibblock">In <em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">ICLR</em>, 2018.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Girdhar et al. (2023)</span>
<span class="ltx_bibblock">
Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand
Joulin, and Ishan Misra.

</span>
<span class="ltx_bibblock">Omnimae: Single model masked pretraining on images and videos.

</span>
<span class="ltx_bibblock">In <em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2023.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Goyal et al. (2019)</span>
<span class="ltx_bibblock">
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra.

</span>
<span class="ltx_bibblock">Scaling and benchmarking self-supervised visual representation
learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">ICCV</em>, 2019.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Goyal et al. (2021)</span>
<span class="ltx_bibblock">
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek
Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al.

</span>
<span class="ltx_bibblock">Self-supervised pretraining of visual features in the wild.

</span>
<span class="ltx_bibblock"><em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">preprint arXiv:2103.01988</em>, 2021.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Goyal et al. (2022a)</span>
<span class="ltx_bibblock">
Priya Goyal, Quentin Duval, Isaac Seessel, Mathilde Caron, Mannat Singh, Ishan
Misra, Levent Sagun, Armand Joulin, and Piotr Bojanowski.

</span>
<span class="ltx_bibblock">Vision models are more robust and fair when pretrained on uncurated
images without supervision.

</span>
<span class="ltx_bibblock"><em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2202.08360</em>, 2022a.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Goyal et al. (2022b)</span>
<span class="ltx_bibblock">
Priya Goyal, Adriana Romero Soriano, Caner Hazirbas, Levent Sagun, and Nicolas
Usunier.

</span>
<span class="ltx_bibblock">Fairness indicators for systematic assessments of visual feature
extractors.

</span>
<span class="ltx_bibblock">In <em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">FAcct</em>, 2022b.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Goyal et al. (2017)</span>
<span class="ltx_bibblock">
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska,
Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos,
Moritz Mueller-Freitag, et al.

</span>
<span class="ltx_bibblock">The "something something" video database for learning and evaluating
visual common sense.

</span>
<span class="ltx_bibblock">In <em id="bib.bib52.1.1" class="ltx_emph ltx_font_italic">ICCV</em>, 2017.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Grill et al. (2020)</span>
<span class="ltx_bibblock">
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec,
Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires,
Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu,
Rémi Munos, and Michal Valko.

</span>
<span class="ltx_bibblock">Bootstrap your own latent: A new approach to self-supervised
learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib53.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>, 2020.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hadsell et al. (2006)</span>
<span class="ltx_bibblock">
Raia Hadsell, Sumit Chopra, and Yann LeCun.

</span>
<span class="ltx_bibblock">Dimensionality reduction by learning an invariant mapping.

</span>
<span class="ltx_bibblock">In <em id="bib.bib54.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2006.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hamilton et al. (2022)</span>
<span class="ltx_bibblock">
Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T
Freeman.

</span>
<span class="ltx_bibblock">Unsupervised semantic segmentation by distilling feature
correspondences.

</span>
<span class="ltx_bibblock">In <em id="bib.bib55.1.1" class="ltx_emph ltx_font_italic">ICLR</em>, 2022.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hazirbas et al. (2021)</span>
<span class="ltx_bibblock">
Caner Hazirbas, Joanna Bitton, Brian Dolhansky, Jacqueline Pan, Albert Gordo,
and Cristian Canton Ferrer.

</span>
<span class="ltx_bibblock">Towards measuring fairness in ai: the casual conversations dataset.

</span>
<span class="ltx_bibblock"><em id="bib.bib56.1.1" class="ltx_emph ltx_font_italic">IEEE Transactions on Biometrics, Behavior, and Identity
Science</em>, 2021.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">He et al. (2020)</span>
<span class="ltx_bibblock">
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.

</span>
<span class="ltx_bibblock">Momentum contrast for unsupervised visual representation learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib57.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2020.

</span>
</li>
<li id="bib.bib58" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">He et al. (2022)</span>
<span class="ltx_bibblock">
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross
Girshick.

</span>
<span class="ltx_bibblock">Masked autoencoders are scalable vision learners.

</span>
<span class="ltx_bibblock">In <em id="bib.bib58.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2022.

</span>
</li>
<li id="bib.bib59" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hénaff et al. (2019)</span>
<span class="ltx_bibblock">
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl
Doersch, SM Eslami, and Aaron van den Oord.

</span>
<span class="ltx_bibblock">Data-efficient image recognition with contrastive predictive coding.

</span>
<span class="ltx_bibblock"><em id="bib.bib59.1.1" class="ltx_emph ltx_font_italic">PMLR</em>, 2019.

</span>
</li>
<li id="bib.bib60" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks &amp; Dietterich (2019)</span>
<span class="ltx_bibblock">
Dan Hendrycks and Thomas Dietterich.

</span>
<span class="ltx_bibblock">Benchmarking neural network robustness to common corruptions and
perturbations.

</span>
<span class="ltx_bibblock">In <em id="bib.bib60.1.1" class="ltx_emph ltx_font_italic">ICLR</em>, 2019.

</span>
</li>
<li id="bib.bib61" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. (2021a)</span>
<span class="ltx_bibblock">
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan
Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al.

</span>
<span class="ltx_bibblock">The many faces of robustness: A critical analysis of
out-of-distribution generalization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib61.1.1" class="ltx_emph ltx_font_italic">ICCV</em>, 2021a.

</span>
</li>
<li id="bib.bib62" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. (2021b)</span>
<span class="ltx_bibblock">
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.

</span>
<span class="ltx_bibblock">Natural adversarial examples.

</span>
<span class="ltx_bibblock">In <em id="bib.bib62.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2021b.

</span>
</li>
<li id="bib.bib63" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hinton et al. (2014)</span>
<span class="ltx_bibblock">
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.

</span>
<span class="ltx_bibblock">Distilling the knowledge in a neural network.

</span>
<span class="ltx_bibblock">In <em id="bib.bib63.1.1" class="ltx_emph ltx_font_italic">NeurIPS Deep Learning Workshop</em>, 2014.

</span>
</li>
<li id="bib.bib64" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hoffmann et al. (2022)</span>
<span class="ltx_bibblock">
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor
Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes
Welbl, Aidan Clark, et al.

</span>
<span class="ltx_bibblock">Training compute-optimal large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib64.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2203.15556</em>, 2022.

</span>
</li>
<li id="bib.bib65" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Huang et al. (2016)</span>
<span class="ltx_bibblock">
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger.

</span>
<span class="ltx_bibblock">Deep networks with stochastic depth.

</span>
<span class="ltx_bibblock">In <em id="bib.bib65.1.1" class="ltx_emph ltx_font_italic">ECCV</em>, 2016.

</span>
</li>
<li id="bib.bib66" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ilharco et al. (2021)</span>
<span class="ltx_bibblock">
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas
Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John
Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.

</span>
<span class="ltx_bibblock">Openclip.

</span>
<span class="ltx_bibblock">2021.

</span>
</li>
<li id="bib.bib67" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jegou et al. (2010)</span>
<span class="ltx_bibblock">
Herve Jegou, Matthijs Douze, and Cordelia Schmid.

</span>
<span class="ltx_bibblock">Product quantization for nearest neighbor search.

</span>
<span class="ltx_bibblock"><em id="bib.bib67.1.1" class="ltx_emph ltx_font_italic">TPAMI</em>, 2010.

</span>
</li>
<li id="bib.bib68" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Johnson et al. (2019)</span>
<span class="ltx_bibblock">
Jeff Johnson, Matthijs Douze, and Hervé Jégou.

</span>
<span class="ltx_bibblock">Billion-scale similarity search with GPUs.

</span>
<span class="ltx_bibblock"><em id="bib.bib68.1.1" class="ltx_emph ltx_font_italic">IEEE Transactions on Big Data</em>, 2019.

</span>
</li>
<li id="bib.bib69" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Joulin et al. (2016)</span>
<span class="ltx_bibblock">
Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache.

</span>
<span class="ltx_bibblock">Learning visual features from large weakly supervised data.

</span>
<span class="ltx_bibblock">In <em id="bib.bib69.1.1" class="ltx_emph ltx_font_italic">ECCV</em>, 2016.

</span>
</li>
<li id="bib.bib70" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kay et al. (2017)</span>
<span class="ltx_bibblock">
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra
Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.

</span>
<span class="ltx_bibblock">The kinetics human action video dataset.

</span>
<span class="ltx_bibblock"><em id="bib.bib70.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1705.06950</em>, 2017.

</span>
</li>
<li id="bib.bib71" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Krause et al. (2013)</span>
<span class="ltx_bibblock">
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.

</span>
<span class="ltx_bibblock">3d object representations for fine-grained categorization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib71.1.1" class="ltx_emph ltx_font_italic">3DRR</em>, 2013.

</span>
</li>
<li id="bib.bib72" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Krell et al. (2022)</span>
<span class="ltx_bibblock">
Mario Michael Krell, Matej Kosec, Sergio P. Perez, and Andrew Fitzgibbon.

</span>
<span class="ltx_bibblock">Efficient sequence packing without cross-contamination: Accelerating
large language models without impacting performance, 2022.

</span>
</li>
<li id="bib.bib73" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Krizhevsky et al. (2009)</span>
<span class="ltx_bibblock">
Alex Krizhevsky, Geoffrey Hinton, et al.

</span>
<span class="ltx_bibblock">Learning multiple layers of features from tiny images.

</span>
<span class="ltx_bibblock">2009.

</span>
</li>
<li id="bib.bib74" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lefaudeux et al. (2022)</span>
<span class="ltx_bibblock">
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio
Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick
Labatut, and Daniel Haziza.

</span>
<span class="ltx_bibblock">xformers: A modular and hackable transformer modelling library.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/facebookresearch/xformers" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/facebookresearch/xformers</a>, 2022.

</span>
</li>
<li id="bib.bib75" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2022a)</span>
<span class="ltx_bibblock">
Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai,
Lu Yuan, and Jianfeng Gao.

</span>
<span class="ltx_bibblock">Efficient self-supervised vision transformers for representation
learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib75.1.1" class="ltx_emph ltx_font_italic">ICLR</em>, 2022a.

</span>
</li>
<li id="bib.bib76" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2022b)</span>
<span class="ltx_bibblock">
Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang.

</span>
<span class="ltx_bibblock">Binsformer: Revisiting adaptive bins for monocular depth estimation.

</span>
<span class="ltx_bibblock"><em id="bib.bib76.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2204.00987</em>, 2022b.

</span>
</li>
<li id="bib.bib77" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Likhomanenko et al. (2021)</span>
<span class="ltx_bibblock">
Tatiana Likhomanenko, Qiantong Xu, Gabriel Synnaeve, Ronan Collobert, and Alex
Rogozhnikov.

</span>
<span class="ltx_bibblock">Cape: Encoding relative positions with continuous augmented
positional embeddings.

</span>
<span class="ltx_bibblock">In <em id="bib.bib77.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>, 2021.

</span>
</li>
<li id="bib.bib78" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2021)</span>
<span class="ltx_bibblock">
Huajun Liu, Fuqiang Liu, Xinyi Fan, and Dong Huang.

</span>
<span class="ltx_bibblock">Polarized self-attention: towards high-quality pixel-wise regression.

</span>
<span class="ltx_bibblock"><em id="bib.bib78.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2107.00782</em>, 2021.

</span>
</li>
<li id="bib.bib79" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mahajan et al. (2018)</span>
<span class="ltx_bibblock">
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri,
Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten.

</span>
<span class="ltx_bibblock">Exploring the limits of weakly supervised pretraining.

</span>
<span class="ltx_bibblock">In <em id="bib.bib79.1.1" class="ltx_emph ltx_font_italic">ECCV</em>, 2018.

</span>
</li>
<li id="bib.bib80" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Maji et al. (2013)</span>
<span class="ltx_bibblock">
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi.

</span>
<span class="ltx_bibblock">Fine-grained visual classification of aircraft.

</span>
<span class="ltx_bibblock">Technical report, 2013.

</span>
</li>
<li id="bib.bib81" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Misra &amp; Maaten (2020)</span>
<span class="ltx_bibblock">
Ishan Misra and Laurens van der Maaten.

</span>
<span class="ltx_bibblock">Self-supervised learning of pretext-invariant representations.

</span>
<span class="ltx_bibblock">In <em id="bib.bib81.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2020.

</span>
</li>
<li id="bib.bib82" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nilsback &amp; Zisserman (2008)</span>
<span class="ltx_bibblock">
Maria-Elena Nilsback and Andrew Zisserman.

</span>
<span class="ltx_bibblock">Automated flower classification over a large number of classes.

</span>
<span class="ltx_bibblock">In <em id="bib.bib82.1.1" class="ltx_emph ltx_font_italic">ICVGIP</em>, 2008.

</span>
</li>
<li id="bib.bib83" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Noroozi &amp; Favaro (2016)</span>
<span class="ltx_bibblock">
Mehdi Noroozi and Paolo Favaro.

</span>
<span class="ltx_bibblock">Unsupervised learning of visual representations by solving jigsaw
puzzles.

</span>
<span class="ltx_bibblock">In <em id="bib.bib83.1.1" class="ltx_emph ltx_font_italic">ECCV</em>, 2016.

</span>
</li>
<li id="bib.bib84" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ofri-Amar et al. (2023)</span>
<span class="ltx_bibblock">
Dolev Ofri-Amar, Michal Geyer, Yoni Kasten, and Tali Dekel.

</span>
<span class="ltx_bibblock">Neural congealing: Aligning images to a joint semantic atlas.

</span>
<span class="ltx_bibblock">In <em id="bib.bib84.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2023.

</span>
</li>
<li id="bib.bib85" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Parkhi et al. (2012)</span>
<span class="ltx_bibblock">
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar.

</span>
<span class="ltx_bibblock">Cats and dogs.

</span>
<span class="ltx_bibblock">In <em id="bib.bib85.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2012.

</span>
</li>
<li id="bib.bib86" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pathak et al. (2016)</span>
<span class="ltx_bibblock">
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei
Efros.

</span>
<span class="ltx_bibblock">Context encoders: Feature learning by inpainting.

</span>
<span class="ltx_bibblock">In <em id="bib.bib86.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2016.

</span>
</li>
<li id="bib.bib87" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Patterson et al. (2021)</span>
<span class="ltx_bibblock">
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia,
Daniel Rothchild, David So, Maud Texier, and Jeff Dean.

</span>
<span class="ltx_bibblock">Carbon emissions and large neural network training.

</span>
<span class="ltx_bibblock"><em id="bib.bib87.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2104.10350</em>, 2021.

</span>
</li>
<li id="bib.bib88" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pizzi et al. (2022)</span>
<span class="ltx_bibblock">
Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Matthijs
Douze.

</span>
<span class="ltx_bibblock">A self-supervised descriptor for image copy detection.

</span>
<span class="ltx_bibblock">In <em id="bib.bib88.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2022.

</span>
</li>
<li id="bib.bib89" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radenović et al. (2018a)</span>
<span class="ltx_bibblock">
Filip Radenović, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and
Ondřej Chum.

</span>
<span class="ltx_bibblock">Revisiting oxford and paris: Large-scale image retrieval
benchmarking.

</span>
<span class="ltx_bibblock">In <em id="bib.bib89.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2018a.

</span>
</li>
<li id="bib.bib90" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radenović et al. (2018b)</span>
<span class="ltx_bibblock">
Filip Radenović, Giorgos Tolias, and Ondřej Chum.

</span>
<span class="ltx_bibblock">Fine-tuning cnn image retrieval with no human annotation.

</span>
<span class="ltx_bibblock"><em id="bib.bib90.1.1" class="ltx_emph ltx_font_italic">TPAMI</em>, 2018b.

</span>
</li>
<li id="bib.bib91" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. (2017)</span>
<span class="ltx_bibblock">
Alec Radford, Rafal Jozefowicz, and Ilya Sutskever.

</span>
<span class="ltx_bibblock">Learning to generate reviews and discovering sentiment.

</span>
<span class="ltx_bibblock"><em id="bib.bib91.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1704.01444</em>, 2017.

</span>
</li>
<li id="bib.bib92" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. (2019)</span>
<span class="ltx_bibblock">
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya
Sutskever.

</span>
<span class="ltx_bibblock">Language models are unsupervised multitask learners.

</span>
<span class="ltx_bibblock">2019.

</span>
</li>
<li id="bib.bib93" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. (2021)</span>
<span class="ltx_bibblock">
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et al.

</span>
<span class="ltx_bibblock">Learning transferable visual models from natural language
supervision.

</span>
<span class="ltx_bibblock">In <em id="bib.bib93.1.1" class="ltx_emph ltx_font_italic">ICML</em>, 2021.

</span>
</li>
<li id="bib.bib94" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raffel et al. (2020)</span>
<span class="ltx_bibblock">
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael
Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al.

</span>
<span class="ltx_bibblock">Exploring the limits of transfer learning with a unified text-to-text
transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib94.1.1" class="ltx_emph ltx_font_italic">JMLR</em>, 2020.

</span>
</li>
<li id="bib.bib95" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ranftl et al. (2021)</span>
<span class="ltx_bibblock">
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.

</span>
<span class="ltx_bibblock">Vision transformers for dense prediction.

</span>
<span class="ltx_bibblock">In <em id="bib.bib95.1.1" class="ltx_emph ltx_font_italic">ICCV</em>, 2021.

</span>
</li>
<li id="bib.bib96" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Recht et al. (2019)</span>
<span class="ltx_bibblock">
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.

</span>
<span class="ltx_bibblock">Do imagenet classifiers generalize to imagenet?

</span>
<span class="ltx_bibblock">In <em id="bib.bib96.1.1" class="ltx_emph ltx_font_italic">ICML</em>, 2019.

</span>
</li>
<li id="bib.bib97" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Revaud et al. (2019)</span>
<span class="ltx_bibblock">
Jerome Revaud, Jon Almazán, Rafael S Rezende, and Cesar Roberto de Souza.

</span>
<span class="ltx_bibblock">Learning with average precision: Training image retrieval with a
listwise loss.

</span>
<span class="ltx_bibblock">In <em id="bib.bib97.1.1" class="ltx_emph ltx_font_italic">ICCV</em>, 2019.

</span>
</li>
<li id="bib.bib98" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ruan et al. (2023)</span>
<span class="ltx_bibblock">
Yangjun Ruan, Saurabh Singh, Warren Morningstar, Alexander A Alemi, Sergey
Ioffe, Ian Fischer, and Joshua V Dillon.

</span>
<span class="ltx_bibblock">Weighted ensemble self-supervised learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib98.1.1" class="ltx_emph ltx_font_italic">ICLR</em>, 2023.

</span>
</li>
<li id="bib.bib99" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Russakovsky et al. (2015)</span>
<span class="ltx_bibblock">
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma,
Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C
Berg, and Li Fei-Fei.

</span>
<span class="ltx_bibblock">Imagenet large scale visual recognition challenge.

</span>
<span class="ltx_bibblock"><em id="bib.bib99.1.1" class="ltx_emph ltx_font_italic">IJCV</em>, 2015.

</span>
</li>
<li id="bib.bib100" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sablayrolles et al. (2019)</span>
<span class="ltx_bibblock">
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé
Jégou.

</span>
<span class="ltx_bibblock">Spreading vectors for similarity search.

</span>
<span class="ltx_bibblock">In <em id="bib.bib100.1.1" class="ltx_emph ltx_font_italic">ICLR</em>, 2019.

</span>
</li>
<li id="bib.bib101" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schuhmann et al. (2021)</span>
<span class="ltx_bibblock">
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk,
Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran
Komatsuzaki.

</span>
<span class="ltx_bibblock">Laion-400m: Open dataset of clip-filtered 400 million image-text
pairs.

</span>
<span class="ltx_bibblock">In <em id="bib.bib101.1.1" class="ltx_emph ltx_font_italic">NeurIPS Data Centric AI Workshop</em>, 2021.

</span>
</li>
<li id="bib.bib102" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schuhmann et al. (2022)</span>
<span class="ltx_bibblock">
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross
Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell
Wortsman, et al.

</span>
<span class="ltx_bibblock">Laion-5b: An open large-scale dataset for training next generation
image-text models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib102.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>, 2022.

</span>
</li>
<li id="bib.bib103" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shazeer (2020)</span>
<span class="ltx_bibblock">
Noam Shazeer.

</span>
<span class="ltx_bibblock">Glu variants improve transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib103.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2002.05202</em>, 2020.

</span>
</li>
<li id="bib.bib104" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Silberman et al. (2012)</span>
<span class="ltx_bibblock">
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus.

</span>
<span class="ltx_bibblock">Indoor segmentation and support inference from rgbd images.

</span>
<span class="ltx_bibblock">In <em id="bib.bib104.1.1" class="ltx_emph ltx_font_italic">ECCV</em>, 2012.

</span>
</li>
<li id="bib.bib105" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Singh et al. (2022)</span>
<span class="ltx_bibblock">
Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra
Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár,
and Laurens van der Maaten.

</span>
<span class="ltx_bibblock">Revisiting Weakly Supervised Pre-Training of Visual Perception
Models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib105.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2022.

</span>
</li>
<li id="bib.bib106" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Song et al. (2015)</span>
<span class="ltx_bibblock">
Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao.

</span>
<span class="ltx_bibblock">Sun rgb-d: A rgb-d scene understanding benchmark suite.

</span>
<span class="ltx_bibblock">In <em id="bib.bib106.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2015.

</span>
</li>
<li id="bib.bib107" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Soomro et al. (2012)</span>
<span class="ltx_bibblock">
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.

</span>
<span class="ltx_bibblock">Ucf101: A dataset of 101 human actions classes from videos in the
wild.

</span>
<span class="ltx_bibblock"><em id="bib.bib107.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1212.0402</em>, 2012.

</span>
</li>
<li id="bib.bib108" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Steiner et al. (2021)</span>
<span class="ltx_bibblock">
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob
Uszkoreit, and Lucas Beyer.

</span>
<span class="ltx_bibblock">How to train your vit? data, augmentation, and regularization in
vision transformers.

</span>
<span class="ltx_bibblock"><em id="bib.bib108.1.1" class="ltx_emph ltx_font_italic">TMLR</em>, 2021.

</span>
</li>
<li id="bib.bib109" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Strubell et al. (2019)</span>
<span class="ltx_bibblock">
Emma Strubell, Ananya Ganesh, and Andrew McCallum.

</span>
<span class="ltx_bibblock">Energy and policy considerations for deep learning in nlp.

</span>
<span class="ltx_bibblock"><em id="bib.bib109.1.1" class="ltx_emph ltx_font_italic">ACL</em>, 2019.

</span>
</li>
<li id="bib.bib110" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tian et al. (2021)</span>
<span class="ltx_bibblock">
Yonglong Tian, Olivier J Henaff, and Aäron van den Oord.

</span>
<span class="ltx_bibblock">Divide and contrast: Self-supervised learning from uncurated data.

</span>
<span class="ltx_bibblock">In <em id="bib.bib110.1.1" class="ltx_emph ltx_font_italic">ICCV</em>, 2021.

</span>
</li>
<li id="bib.bib111" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tolias et al. (2016)</span>
<span class="ltx_bibblock">
Giorgos Tolias, Ronan Sicre, and Hervé Jégou.

</span>
<span class="ltx_bibblock">Particular object retrieval with integral max-pooling of cnn
activations.

</span>
<span class="ltx_bibblock">In <em id="bib.bib111.1.1" class="ltx_emph ltx_font_italic">ICLR</em>, 2016.

</span>
</li>
<li id="bib.bib112" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tong et al. (2022)</span>
<span class="ltx_bibblock">
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.

</span>
<span class="ltx_bibblock">Videomae: Masked autoencoders are data-efficient learners for
self-supervised video pre-training.

</span>
<span class="ltx_bibblock">In <em id="bib.bib112.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>, 2022.

</span>
</li>
<li id="bib.bib113" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2019)</span>
<span class="ltx_bibblock">
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou.

</span>
<span class="ltx_bibblock">Fixing the train-test resolution discrepancy.

</span>
<span class="ltx_bibblock">In <em id="bib.bib113.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>, 2019.

</span>
</li>
<li id="bib.bib114" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2022)</span>
<span class="ltx_bibblock">
Hugo Touvron, Matthieu Cord, and Hervé Jégou.

</span>
<span class="ltx_bibblock">Deit iii: Revenge of the vit.

</span>
<span class="ltx_bibblock">In <em id="bib.bib114.1.1" class="ltx_emph ltx_font_italic">ECCV</em>, 2022.

</span>
</li>
<li id="bib.bib115" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Touvron et al. (2023)</span>
<span class="ltx_bibblock">
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne
Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric
Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and
Guillaume Lample.

</span>
<span class="ltx_bibblock">Llama: Open and efficient foundation language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib115.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2302.13971</em>, 2023.

</span>
</li>
<li id="bib.bib116" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Tumanyan et al. (2022)</span>
<span class="ltx_bibblock">
Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel.

</span>
<span class="ltx_bibblock">Splicing vit features for semantic appearance transfer.

</span>
<span class="ltx_bibblock">In <em id="bib.bib116.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2022.

</span>
</li>
<li id="bib.bib117" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Van Horn et al. (2018)</span>
<span class="ltx_bibblock">
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard,
Hartwig Adam, Pietro Perona, and Serge Belongie.

</span>
<span class="ltx_bibblock">The inaturalist species classification and detection dataset.

</span>
<span class="ltx_bibblock">In <em id="bib.bib117.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2018.

</span>
</li>
<li id="bib.bib118" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Van Horn et al. (2021)</span>
<span class="ltx_bibblock">
Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and
Oisin Mac Aodha.

</span>
<span class="ltx_bibblock">Benchmarking representation learning for natural world image
collections.

</span>
<span class="ltx_bibblock">In <em id="bib.bib118.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2021.

</span>
</li>
<li id="bib.bib119" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2022)</span>
<span class="ltx_bibblock">
Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu,
Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al.

</span>
<span class="ltx_bibblock">Internimage: Exploring large-scale vision foundation models with
deformable convolutions.

</span>
<span class="ltx_bibblock">In <em id="bib.bib119.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2022.

</span>
</li>
<li id="bib.bib120" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2019)</span>
<span class="ltx_bibblock">
Xiaolong Wang, Allan Jabri, and Alexei A Efros.

</span>
<span class="ltx_bibblock">Learning correspondence from the cycle-consistency of time.

</span>
<span class="ltx_bibblock">In <em id="bib.bib120.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2019.

</span>
</li>
<li id="bib.bib121" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Warburg et al. (2020)</span>
<span class="ltx_bibblock">
Frederik Warburg, Soren Hauberg, Manuel Lopez-Antequera, Pau Gargallo, Yubin
Kuang, and Javier Civera.

</span>
<span class="ltx_bibblock">Mapillary street-level sequences: A dataset for lifelong place
recognition.

</span>
<span class="ltx_bibblock">In <em id="bib.bib121.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2020.

</span>
</li>
<li id="bib.bib122" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Weinzaepfel et al. (2021)</span>
<span class="ltx_bibblock">
Philippe Weinzaepfel, Thomas Lucas, Diane Larlus, and Yannis Kalantidis.

</span>
<span class="ltx_bibblock">Learning super-features for image retrieval.

</span>
<span class="ltx_bibblock">In <em id="bib.bib122.1.1" class="ltx_emph ltx_font_italic">ICLR</em>, 2021.

</span>
</li>
<li id="bib.bib123" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Welinder et al. (2010)</span>
<span class="ltx_bibblock">
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and
P. Perona.

</span>
<span class="ltx_bibblock">Caltech-UCSD Birds 200.

</span>
<span class="ltx_bibblock">Technical Report CNS-TR-2010-001, 2010.

</span>
</li>
<li id="bib.bib124" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wenzek et al. (2020)</span>
<span class="ltx_bibblock">
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary,
Francisco Guzmán, Armand Joulin, and Edouard Grave.

</span>
<span class="ltx_bibblock">Ccnet: Extracting high quality monolingual datasets from web crawl
data.

</span>
<span class="ltx_bibblock">In <em id="bib.bib124.1.1" class="ltx_emph ltx_font_italic">LREC</em>, 2020.

</span>
</li>
<li id="bib.bib125" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Weyand et al. (2020)</span>
<span class="ltx_bibblock">
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim.

</span>
<span class="ltx_bibblock">Google landmarks dataset v2 – a large-scale benchmark for
instance-level recognition and retrieval.

</span>
<span class="ltx_bibblock">In <em id="bib.bib125.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2020.

</span>
</li>
<li id="bib.bib126" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wu et al. (2018)</span>
<span class="ltx_bibblock">
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.

</span>
<span class="ltx_bibblock">Unsupervised feature learning via non-parametric instance
discrimination.

</span>
<span class="ltx_bibblock">In <em id="bib.bib126.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2018.

</span>
</li>
<li id="bib.bib127" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xiao et al. (2010)</span>
<span class="ltx_bibblock">
J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba.

</span>
<span class="ltx_bibblock">Sun database: Large-scale scene recognition from abbey to zoo.

</span>
<span class="ltx_bibblock">In <em id="bib.bib127.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2010.

</span>
</li>
<li id="bib.bib128" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xu et al. (2022)</span>
<span class="ltx_bibblock">
Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian
Metze, Christoph Feichtenhofer, et al.

</span>
<span class="ltx_bibblock">Masked autoencoders that listen.

</span>
<span class="ltx_bibblock"><em id="bib.bib128.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2207.06405</em>, 2022.

</span>
</li>
<li id="bib.bib129" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yalniz et al. (2019)</span>
<span class="ltx_bibblock">
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv
Mahajan.

</span>
<span class="ltx_bibblock">Billion-scale semi-supervised learning for image classification.

</span>
<span class="ltx_bibblock"><em id="bib.bib129.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1905.00546</em>, 2019.

</span>
</li>
<li id="bib.bib130" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yildiz et al. (2022)</span>
<span class="ltx_bibblock">
Burak Yildiz, Seyran Khademi, Ronald Maria Siebes, and Jan van Gemert.

</span>
<span class="ltx_bibblock">Amstertime: A visual place recognition benchmark dataset for severe
domain shift.

</span>
<span class="ltx_bibblock">In <em id="bib.bib130.1.1" class="ltx_emph ltx_font_italic">ICPR</em>, 2022.

</span>
</li>
<li id="bib.bib131" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ypsilantis et al. (2021)</span>
<span class="ltx_bibblock">
Nikolaos-Antonios Ypsilantis, Noa Garcia, Guangxing Han, Sarah Ibrahimi, Nanne
Van Noord, and Giorgos Tolias.

</span>
<span class="ltx_bibblock">The met dataset: Instance-level recognition for artworks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib131.1.1" class="ltx_emph ltx_font_italic">NeurIPS Datasets and Benchmarks Track</em>, 2021.

</span>
</li>
<li id="bib.bib132" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhai et al. (2022)</span>
<span class="ltx_bibblock">
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.

</span>
<span class="ltx_bibblock">Scaling vision transformers.

</span>
<span class="ltx_bibblock">In <em id="bib.bib132.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2022.

</span>
</li>
<li id="bib.bib133" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2016)</span>
<span class="ltx_bibblock">
Richard Zhang, Phillip Isola, and Alexei A Efros.

</span>
<span class="ltx_bibblock">Colorful image colorization.

</span>
<span class="ltx_bibblock">In <em id="bib.bib133.1.1" class="ltx_emph ltx_font_italic">ECCV</em>, 2016.

</span>
</li>
<li id="bib.bib134" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. (2014)</span>
<span class="ltx_bibblock">
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva.

</span>
<span class="ltx_bibblock">Learning deep features for scene recognition using places database.

</span>
<span class="ltx_bibblock">In <em id="bib.bib134.1.1" class="ltx_emph ltx_font_italic">NeurIPS</em>, 2014.

</span>
</li>
<li id="bib.bib135" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. (2017)</span>
<span class="ltx_bibblock">
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio
Torralba.

</span>
<span class="ltx_bibblock">Scene parsing through ade20k dataset.

</span>
<span class="ltx_bibblock">In <em id="bib.bib135.1.1" class="ltx_emph ltx_font_italic">CVPR</em>, 2017.

</span>
</li>
<li id="bib.bib136" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. (2022a)</span>
<span class="ltx_bibblock">
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao
Kong.

</span>
<span class="ltx_bibblock">ibot: Image bert pre-training with online tokenizer.

</span>
<span class="ltx_bibblock">In <em id="bib.bib136.1.1" class="ltx_emph ltx_font_italic">ICLR</em>, 2022a.

</span>
</li>
<li id="bib.bib137" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. (2022b)</span>
<span class="ltx_bibblock">
Pan Zhou, Yichen Zhou, Chenyang Si, Weihao Yu, Teck Khim Ng, and Shuicheng Yan.

</span>
<span class="ltx_bibblock">Mugs: A multi-granular self-supervised learning framework.

</span>
<span class="ltx_bibblock"><em id="bib.bib137.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2203.14415</em>, 2022b.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Data Processing</h2>

<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>Data selection</h3>

<div id="A1.SS1.p1" class="ltx_para ltx_noindent">
<p id="A1.SS1.p1.1" class="ltx_p">Our selection of datasets for building LVD-142M is detailed in Tab. <a href="#A1.T15" title="Table 15 ‣ A.4 Retrieval ‣ Appendix A Data Processing ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">15</span></a>.
This collection is intended to provide images covering well various downstream vision tasks both for image-level and dense recognition.</p>
</div>
</section>
<section id="A1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.2 </span>Image similarity</h3>

<div id="A1.SS2.p1" class="ltx_para ltx_noindent">
<p id="A1.SS2.p1.1" class="ltx_p">We employ cosine similarity to compare image features (whether ours or feature generated for deduplication) with the following similarity function <math id="A1.SS2.p1.1.m1.1" class="ltx_Math" alttext="m" display="inline"><semantics id="A1.SS2.p1.1.m1.1a"><mi id="A1.SS2.p1.1.m1.1.1" xref="A1.SS2.p1.1.m1.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.1.m1.1b"><ci id="A1.SS2.p1.1.m1.1.1.cmml" xref="A1.SS2.p1.1.m1.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.1.m1.1c">m</annotation></semantics></math>:</p>
<table id="A1.Ex4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A1.Ex4.m1.12" class="ltx_Math" alttext="m(s,r)=\text{cosine-similarity}\left(f\left(s\right),f\left(r\right)\right)=\frac{f(s)\cdot{}f(r)}{\lVert f(s)\rVert_{2}\lVert f(r)\rVert_{2}}" display="block"><semantics id="A1.Ex4.m1.12a"><mrow id="A1.Ex4.m1.12.12" xref="A1.Ex4.m1.12.12.cmml"><mrow id="A1.Ex4.m1.12.12.4" xref="A1.Ex4.m1.12.12.4.cmml"><mi id="A1.Ex4.m1.12.12.4.2" xref="A1.Ex4.m1.12.12.4.2.cmml">m</mi><mo lspace="0em" rspace="0em" id="A1.Ex4.m1.12.12.4.1" xref="A1.Ex4.m1.12.12.4.1.cmml">​</mo><mrow id="A1.Ex4.m1.12.12.4.3.2" xref="A1.Ex4.m1.12.12.4.3.1.cmml"><mo stretchy="false" id="A1.Ex4.m1.12.12.4.3.2.1" xref="A1.Ex4.m1.12.12.4.3.1.cmml">(</mo><mi id="A1.Ex4.m1.7.7" xref="A1.Ex4.m1.7.7.cmml">s</mi><mo id="A1.Ex4.m1.12.12.4.3.2.2" xref="A1.Ex4.m1.12.12.4.3.1.cmml">,</mo><mi id="A1.Ex4.m1.8.8" xref="A1.Ex4.m1.8.8.cmml">r</mi><mo stretchy="false" id="A1.Ex4.m1.12.12.4.3.2.3" xref="A1.Ex4.m1.12.12.4.3.1.cmml">)</mo></mrow></mrow><mo id="A1.Ex4.m1.12.12.5" xref="A1.Ex4.m1.12.12.5.cmml">=</mo><mrow id="A1.Ex4.m1.12.12.2" xref="A1.Ex4.m1.12.12.2.cmml"><mtext id="A1.Ex4.m1.12.12.2.4" xref="A1.Ex4.m1.12.12.2.4a.cmml">cosine-similarity</mtext><mo lspace="0em" rspace="0em" id="A1.Ex4.m1.12.12.2.3" xref="A1.Ex4.m1.12.12.2.3.cmml">​</mo><mrow id="A1.Ex4.m1.12.12.2.2.2" xref="A1.Ex4.m1.12.12.2.2.3.cmml"><mo id="A1.Ex4.m1.12.12.2.2.2.3" xref="A1.Ex4.m1.12.12.2.2.3.cmml">(</mo><mrow id="A1.Ex4.m1.11.11.1.1.1.1" xref="A1.Ex4.m1.11.11.1.1.1.1.cmml"><mi id="A1.Ex4.m1.11.11.1.1.1.1.2" xref="A1.Ex4.m1.11.11.1.1.1.1.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="A1.Ex4.m1.11.11.1.1.1.1.1" xref="A1.Ex4.m1.11.11.1.1.1.1.1.cmml">​</mo><mrow id="A1.Ex4.m1.11.11.1.1.1.1.3.2" xref="A1.Ex4.m1.11.11.1.1.1.1.cmml"><mo id="A1.Ex4.m1.11.11.1.1.1.1.3.2.1" xref="A1.Ex4.m1.11.11.1.1.1.1.cmml">(</mo><mi id="A1.Ex4.m1.9.9" xref="A1.Ex4.m1.9.9.cmml">s</mi><mo id="A1.Ex4.m1.11.11.1.1.1.1.3.2.2" xref="A1.Ex4.m1.11.11.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A1.Ex4.m1.12.12.2.2.2.4" xref="A1.Ex4.m1.12.12.2.2.3.cmml">,</mo><mrow id="A1.Ex4.m1.12.12.2.2.2.2" xref="A1.Ex4.m1.12.12.2.2.2.2.cmml"><mi id="A1.Ex4.m1.12.12.2.2.2.2.2" xref="A1.Ex4.m1.12.12.2.2.2.2.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="A1.Ex4.m1.12.12.2.2.2.2.1" xref="A1.Ex4.m1.12.12.2.2.2.2.1.cmml">​</mo><mrow id="A1.Ex4.m1.12.12.2.2.2.2.3.2" xref="A1.Ex4.m1.12.12.2.2.2.2.cmml"><mo id="A1.Ex4.m1.12.12.2.2.2.2.3.2.1" xref="A1.Ex4.m1.12.12.2.2.2.2.cmml">(</mo><mi id="A1.Ex4.m1.10.10" xref="A1.Ex4.m1.10.10.cmml">r</mi><mo id="A1.Ex4.m1.12.12.2.2.2.2.3.2.2" xref="A1.Ex4.m1.12.12.2.2.2.2.cmml">)</mo></mrow></mrow><mo id="A1.Ex4.m1.12.12.2.2.2.5" xref="A1.Ex4.m1.12.12.2.2.3.cmml">)</mo></mrow></mrow><mo id="A1.Ex4.m1.12.12.6" xref="A1.Ex4.m1.12.12.6.cmml">=</mo><mfrac id="A1.Ex4.m1.6.6" xref="A1.Ex4.m1.6.6.cmml"><mrow id="A1.Ex4.m1.2.2.2" xref="A1.Ex4.m1.2.2.2.cmml"><mrow id="A1.Ex4.m1.2.2.2.4" xref="A1.Ex4.m1.2.2.2.4.cmml"><mrow id="A1.Ex4.m1.2.2.2.4.2" xref="A1.Ex4.m1.2.2.2.4.2.cmml"><mi id="A1.Ex4.m1.2.2.2.4.2.2" xref="A1.Ex4.m1.2.2.2.4.2.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="A1.Ex4.m1.2.2.2.4.2.1" xref="A1.Ex4.m1.2.2.2.4.2.1.cmml">​</mo><mrow id="A1.Ex4.m1.2.2.2.4.2.3.2" xref="A1.Ex4.m1.2.2.2.4.2.cmml"><mo stretchy="false" id="A1.Ex4.m1.2.2.2.4.2.3.2.1" xref="A1.Ex4.m1.2.2.2.4.2.cmml">(</mo><mi id="A1.Ex4.m1.1.1.1.1" xref="A1.Ex4.m1.1.1.1.1.cmml">s</mi><mo rspace="0.055em" stretchy="false" id="A1.Ex4.m1.2.2.2.4.2.3.2.2" xref="A1.Ex4.m1.2.2.2.4.2.cmml">)</mo></mrow></mrow><mo rspace="0.222em" id="A1.Ex4.m1.2.2.2.4.1" xref="A1.Ex4.m1.2.2.2.4.1.cmml">⋅</mo><mi id="A1.Ex4.m1.2.2.2.4.3" xref="A1.Ex4.m1.2.2.2.4.3.cmml">f</mi></mrow><mo lspace="0em" rspace="0em" id="A1.Ex4.m1.2.2.2.3" xref="A1.Ex4.m1.2.2.2.3.cmml">​</mo><mrow id="A1.Ex4.m1.2.2.2.5.2" xref="A1.Ex4.m1.2.2.2.cmml"><mo stretchy="false" id="A1.Ex4.m1.2.2.2.5.2.1" xref="A1.Ex4.m1.2.2.2.cmml">(</mo><mi id="A1.Ex4.m1.2.2.2.2" xref="A1.Ex4.m1.2.2.2.2.cmml">r</mi><mo stretchy="false" id="A1.Ex4.m1.2.2.2.5.2.2" xref="A1.Ex4.m1.2.2.2.cmml">)</mo></mrow></mrow><mrow id="A1.Ex4.m1.6.6.6" xref="A1.Ex4.m1.6.6.6.cmml"><msub id="A1.Ex4.m1.5.5.5.3" xref="A1.Ex4.m1.5.5.5.3.cmml"><mrow id="A1.Ex4.m1.5.5.5.3.1.1" xref="A1.Ex4.m1.5.5.5.3.1.2.cmml"><mo fence="true" rspace="0em" id="A1.Ex4.m1.5.5.5.3.1.1.2" xref="A1.Ex4.m1.5.5.5.3.1.2.1.cmml">∥</mo><mrow id="A1.Ex4.m1.5.5.5.3.1.1.1" xref="A1.Ex4.m1.5.5.5.3.1.1.1.cmml"><mi id="A1.Ex4.m1.5.5.5.3.1.1.1.2" xref="A1.Ex4.m1.5.5.5.3.1.1.1.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="A1.Ex4.m1.5.5.5.3.1.1.1.1" xref="A1.Ex4.m1.5.5.5.3.1.1.1.1.cmml">​</mo><mrow id="A1.Ex4.m1.5.5.5.3.1.1.1.3.2" xref="A1.Ex4.m1.5.5.5.3.1.1.1.cmml"><mo stretchy="false" id="A1.Ex4.m1.5.5.5.3.1.1.1.3.2.1" xref="A1.Ex4.m1.5.5.5.3.1.1.1.cmml">(</mo><mi id="A1.Ex4.m1.3.3.3.1" xref="A1.Ex4.m1.3.3.3.1.cmml">s</mi><mo stretchy="false" id="A1.Ex4.m1.5.5.5.3.1.1.1.3.2.2" xref="A1.Ex4.m1.5.5.5.3.1.1.1.cmml">)</mo></mrow></mrow><mo fence="true" lspace="0em" id="A1.Ex4.m1.5.5.5.3.1.1.3" xref="A1.Ex4.m1.5.5.5.3.1.2.1.cmml">∥</mo></mrow><mn id="A1.Ex4.m1.5.5.5.3.3" xref="A1.Ex4.m1.5.5.5.3.3.cmml">2</mn></msub><mo lspace="0em" rspace="0em" id="A1.Ex4.m1.6.6.6.5" xref="A1.Ex4.m1.6.6.6.5.cmml">​</mo><msub id="A1.Ex4.m1.6.6.6.4" xref="A1.Ex4.m1.6.6.6.4.cmml"><mrow id="A1.Ex4.m1.6.6.6.4.1.1" xref="A1.Ex4.m1.6.6.6.4.1.2.cmml"><mo fence="true" rspace="0em" id="A1.Ex4.m1.6.6.6.4.1.1.2" xref="A1.Ex4.m1.6.6.6.4.1.2.1.cmml">∥</mo><mrow id="A1.Ex4.m1.6.6.6.4.1.1.1" xref="A1.Ex4.m1.6.6.6.4.1.1.1.cmml"><mi id="A1.Ex4.m1.6.6.6.4.1.1.1.2" xref="A1.Ex4.m1.6.6.6.4.1.1.1.2.cmml">f</mi><mo lspace="0em" rspace="0em" id="A1.Ex4.m1.6.6.6.4.1.1.1.1" xref="A1.Ex4.m1.6.6.6.4.1.1.1.1.cmml">​</mo><mrow id="A1.Ex4.m1.6.6.6.4.1.1.1.3.2" xref="A1.Ex4.m1.6.6.6.4.1.1.1.cmml"><mo stretchy="false" id="A1.Ex4.m1.6.6.6.4.1.1.1.3.2.1" xref="A1.Ex4.m1.6.6.6.4.1.1.1.cmml">(</mo><mi id="A1.Ex4.m1.4.4.4.2" xref="A1.Ex4.m1.4.4.4.2.cmml">r</mi><mo stretchy="false" id="A1.Ex4.m1.6.6.6.4.1.1.1.3.2.2" xref="A1.Ex4.m1.6.6.6.4.1.1.1.cmml">)</mo></mrow></mrow><mo fence="true" lspace="0em" id="A1.Ex4.m1.6.6.6.4.1.1.3" xref="A1.Ex4.m1.6.6.6.4.1.2.1.cmml">∥</mo></mrow><mn id="A1.Ex4.m1.6.6.6.4.3" xref="A1.Ex4.m1.6.6.6.4.3.cmml">2</mn></msub></mrow></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A1.Ex4.m1.12b"><apply id="A1.Ex4.m1.12.12.cmml" xref="A1.Ex4.m1.12.12"><and id="A1.Ex4.m1.12.12a.cmml" xref="A1.Ex4.m1.12.12"></and><apply id="A1.Ex4.m1.12.12b.cmml" xref="A1.Ex4.m1.12.12"><eq id="A1.Ex4.m1.12.12.5.cmml" xref="A1.Ex4.m1.12.12.5"></eq><apply id="A1.Ex4.m1.12.12.4.cmml" xref="A1.Ex4.m1.12.12.4"><times id="A1.Ex4.m1.12.12.4.1.cmml" xref="A1.Ex4.m1.12.12.4.1"></times><ci id="A1.Ex4.m1.12.12.4.2.cmml" xref="A1.Ex4.m1.12.12.4.2">𝑚</ci><interval closure="open" id="A1.Ex4.m1.12.12.4.3.1.cmml" xref="A1.Ex4.m1.12.12.4.3.2"><ci id="A1.Ex4.m1.7.7.cmml" xref="A1.Ex4.m1.7.7">𝑠</ci><ci id="A1.Ex4.m1.8.8.cmml" xref="A1.Ex4.m1.8.8">𝑟</ci></interval></apply><apply id="A1.Ex4.m1.12.12.2.cmml" xref="A1.Ex4.m1.12.12.2"><times id="A1.Ex4.m1.12.12.2.3.cmml" xref="A1.Ex4.m1.12.12.2.3"></times><ci id="A1.Ex4.m1.12.12.2.4a.cmml" xref="A1.Ex4.m1.12.12.2.4"><mtext id="A1.Ex4.m1.12.12.2.4.cmml" xref="A1.Ex4.m1.12.12.2.4">cosine-similarity</mtext></ci><interval closure="open" id="A1.Ex4.m1.12.12.2.2.3.cmml" xref="A1.Ex4.m1.12.12.2.2.2"><apply id="A1.Ex4.m1.11.11.1.1.1.1.cmml" xref="A1.Ex4.m1.11.11.1.1.1.1"><times id="A1.Ex4.m1.11.11.1.1.1.1.1.cmml" xref="A1.Ex4.m1.11.11.1.1.1.1.1"></times><ci id="A1.Ex4.m1.11.11.1.1.1.1.2.cmml" xref="A1.Ex4.m1.11.11.1.1.1.1.2">𝑓</ci><ci id="A1.Ex4.m1.9.9.cmml" xref="A1.Ex4.m1.9.9">𝑠</ci></apply><apply id="A1.Ex4.m1.12.12.2.2.2.2.cmml" xref="A1.Ex4.m1.12.12.2.2.2.2"><times id="A1.Ex4.m1.12.12.2.2.2.2.1.cmml" xref="A1.Ex4.m1.12.12.2.2.2.2.1"></times><ci id="A1.Ex4.m1.12.12.2.2.2.2.2.cmml" xref="A1.Ex4.m1.12.12.2.2.2.2.2">𝑓</ci><ci id="A1.Ex4.m1.10.10.cmml" xref="A1.Ex4.m1.10.10">𝑟</ci></apply></interval></apply></apply><apply id="A1.Ex4.m1.12.12c.cmml" xref="A1.Ex4.m1.12.12"><eq id="A1.Ex4.m1.12.12.6.cmml" xref="A1.Ex4.m1.12.12.6"></eq><share href="#A1.Ex4.m1.12.12.2.cmml" id="A1.Ex4.m1.12.12d.cmml" xref="A1.Ex4.m1.12.12"></share><apply id="A1.Ex4.m1.6.6.cmml" xref="A1.Ex4.m1.6.6"><divide id="A1.Ex4.m1.6.6.7.cmml" xref="A1.Ex4.m1.6.6"></divide><apply id="A1.Ex4.m1.2.2.2.cmml" xref="A1.Ex4.m1.2.2.2"><times id="A1.Ex4.m1.2.2.2.3.cmml" xref="A1.Ex4.m1.2.2.2.3"></times><apply id="A1.Ex4.m1.2.2.2.4.cmml" xref="A1.Ex4.m1.2.2.2.4"><ci id="A1.Ex4.m1.2.2.2.4.1.cmml" xref="A1.Ex4.m1.2.2.2.4.1">⋅</ci><apply id="A1.Ex4.m1.2.2.2.4.2.cmml" xref="A1.Ex4.m1.2.2.2.4.2"><times id="A1.Ex4.m1.2.2.2.4.2.1.cmml" xref="A1.Ex4.m1.2.2.2.4.2.1"></times><ci id="A1.Ex4.m1.2.2.2.4.2.2.cmml" xref="A1.Ex4.m1.2.2.2.4.2.2">𝑓</ci><ci id="A1.Ex4.m1.1.1.1.1.cmml" xref="A1.Ex4.m1.1.1.1.1">𝑠</ci></apply><ci id="A1.Ex4.m1.2.2.2.4.3.cmml" xref="A1.Ex4.m1.2.2.2.4.3">𝑓</ci></apply><ci id="A1.Ex4.m1.2.2.2.2.cmml" xref="A1.Ex4.m1.2.2.2.2">𝑟</ci></apply><apply id="A1.Ex4.m1.6.6.6.cmml" xref="A1.Ex4.m1.6.6.6"><times id="A1.Ex4.m1.6.6.6.5.cmml" xref="A1.Ex4.m1.6.6.6.5"></times><apply id="A1.Ex4.m1.5.5.5.3.cmml" xref="A1.Ex4.m1.5.5.5.3"><csymbol cd="ambiguous" id="A1.Ex4.m1.5.5.5.3.2.cmml" xref="A1.Ex4.m1.5.5.5.3">subscript</csymbol><apply id="A1.Ex4.m1.5.5.5.3.1.2.cmml" xref="A1.Ex4.m1.5.5.5.3.1.1"><csymbol cd="latexml" id="A1.Ex4.m1.5.5.5.3.1.2.1.cmml" xref="A1.Ex4.m1.5.5.5.3.1.1.2">delimited-∥∥</csymbol><apply id="A1.Ex4.m1.5.5.5.3.1.1.1.cmml" xref="A1.Ex4.m1.5.5.5.3.1.1.1"><times id="A1.Ex4.m1.5.5.5.3.1.1.1.1.cmml" xref="A1.Ex4.m1.5.5.5.3.1.1.1.1"></times><ci id="A1.Ex4.m1.5.5.5.3.1.1.1.2.cmml" xref="A1.Ex4.m1.5.5.5.3.1.1.1.2">𝑓</ci><ci id="A1.Ex4.m1.3.3.3.1.cmml" xref="A1.Ex4.m1.3.3.3.1">𝑠</ci></apply></apply><cn type="integer" id="A1.Ex4.m1.5.5.5.3.3.cmml" xref="A1.Ex4.m1.5.5.5.3.3">2</cn></apply><apply id="A1.Ex4.m1.6.6.6.4.cmml" xref="A1.Ex4.m1.6.6.6.4"><csymbol cd="ambiguous" id="A1.Ex4.m1.6.6.6.4.2.cmml" xref="A1.Ex4.m1.6.6.6.4">subscript</csymbol><apply id="A1.Ex4.m1.6.6.6.4.1.2.cmml" xref="A1.Ex4.m1.6.6.6.4.1.1"><csymbol cd="latexml" id="A1.Ex4.m1.6.6.6.4.1.2.1.cmml" xref="A1.Ex4.m1.6.6.6.4.1.1.2">delimited-∥∥</csymbol><apply id="A1.Ex4.m1.6.6.6.4.1.1.1.cmml" xref="A1.Ex4.m1.6.6.6.4.1.1.1"><times id="A1.Ex4.m1.6.6.6.4.1.1.1.1.cmml" xref="A1.Ex4.m1.6.6.6.4.1.1.1.1"></times><ci id="A1.Ex4.m1.6.6.6.4.1.1.1.2.cmml" xref="A1.Ex4.m1.6.6.6.4.1.1.1.2">𝑓</ci><ci id="A1.Ex4.m1.4.4.4.2.cmml" xref="A1.Ex4.m1.4.4.4.2">𝑟</ci></apply></apply><cn type="integer" id="A1.Ex4.m1.6.6.6.4.3.cmml" xref="A1.Ex4.m1.6.6.6.4.3">2</cn></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.Ex4.m1.12c">m(s,r)=\text{cosine-similarity}\left(f\left(s\right),f\left(r\right)\right)=\frac{f(s)\cdot{}f(r)}{\lVert f(s)\rVert_{2}\lVert f(r)\rVert_{2}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A1.SS2.p1.4" class="ltx_p">where <math id="A1.SS2.p1.2.m1.1" class="ltx_Math" alttext="s" display="inline"><semantics id="A1.SS2.p1.2.m1.1a"><mi id="A1.SS2.p1.2.m1.1.1" xref="A1.SS2.p1.2.m1.1.1.cmml">s</mi><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.2.m1.1b"><ci id="A1.SS2.p1.2.m1.1.1.cmml" xref="A1.SS2.p1.2.m1.1.1">𝑠</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.2.m1.1c">s</annotation></semantics></math> and <math id="A1.SS2.p1.3.m2.1" class="ltx_Math" alttext="r" display="inline"><semantics id="A1.SS2.p1.3.m2.1a"><mi id="A1.SS2.p1.3.m2.1.1" xref="A1.SS2.p1.3.m2.1.1.cmml">r</mi><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.3.m2.1b"><ci id="A1.SS2.p1.3.m2.1.1.cmml" xref="A1.SS2.p1.3.m2.1.1">𝑟</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.3.m2.1c">r</annotation></semantics></math> are a pair of images to compare and <math id="A1.SS2.p1.4.m3.1" class="ltx_Math" alttext="f" display="inline"><semantics id="A1.SS2.p1.4.m3.1a"><mi id="A1.SS2.p1.4.m3.1.1" xref="A1.SS2.p1.4.m3.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="A1.SS2.p1.4.m3.1b"><ci id="A1.SS2.p1.4.m3.1.1.cmml" xref="A1.SS2.p1.4.m3.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS2.p1.4.m3.1c">f</annotation></semantics></math> is the model generating features.</p>
</div>
</section>
<section id="A1.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.3 </span>Deduplication</h3>

<section id="A1.SS3.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Self-deduplication.</h5>

<div id="A1.SS3.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="A1.SS3.SSS0.Px1.p1.3" class="ltx_p">To deduplicate our uncurated data source of 1.3B images, we compute and use the embeddings generated by <cite class="ltx_cite ltx_citemacro_cite">Pizzi et al. (<a href="#bib.bib88" title="" class="ltx_ref">2022</a>)</cite> and retrieve the <math id="A1.SS3.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="k=64" display="inline"><semantics id="A1.SS3.SSS0.Px1.p1.1.m1.1a"><mrow id="A1.SS3.SSS0.Px1.p1.1.m1.1.1" xref="A1.SS3.SSS0.Px1.p1.1.m1.1.1.cmml"><mi id="A1.SS3.SSS0.Px1.p1.1.m1.1.1.2" xref="A1.SS3.SSS0.Px1.p1.1.m1.1.1.2.cmml">k</mi><mo id="A1.SS3.SSS0.Px1.p1.1.m1.1.1.1" xref="A1.SS3.SSS0.Px1.p1.1.m1.1.1.1.cmml">=</mo><mn id="A1.SS3.SSS0.Px1.p1.1.m1.1.1.3" xref="A1.SS3.SSS0.Px1.p1.1.m1.1.1.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.SSS0.Px1.p1.1.m1.1b"><apply id="A1.SS3.SSS0.Px1.p1.1.m1.1.1.cmml" xref="A1.SS3.SSS0.Px1.p1.1.m1.1.1"><eq id="A1.SS3.SSS0.Px1.p1.1.m1.1.1.1.cmml" xref="A1.SS3.SSS0.Px1.p1.1.m1.1.1.1"></eq><ci id="A1.SS3.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="A1.SS3.SSS0.Px1.p1.1.m1.1.1.2">𝑘</ci><cn type="integer" id="A1.SS3.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="A1.SS3.SSS0.Px1.p1.1.m1.1.1.3">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.SSS0.Px1.p1.1.m1.1c">k=64</annotation></semantics></math> nearest neighbors of each image (using cosine similarity).
Considering only neighbors with a similarity <math id="A1.SS3.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="{&gt;}0.6" display="inline"><semantics id="A1.SS3.SSS0.Px1.p1.2.m2.1a"><mrow id="A1.SS3.SSS0.Px1.p1.2.m2.1.1" xref="A1.SS3.SSS0.Px1.p1.2.m2.1.1.cmml"><mi id="A1.SS3.SSS0.Px1.p1.2.m2.1.1.2" xref="A1.SS3.SSS0.Px1.p1.2.m2.1.1.2.cmml"></mi><mo id="A1.SS3.SSS0.Px1.p1.2.m2.1.1.1" xref="A1.SS3.SSS0.Px1.p1.2.m2.1.1.1.cmml">&gt;</mo><mn id="A1.SS3.SSS0.Px1.p1.2.m2.1.1.3" xref="A1.SS3.SSS0.Px1.p1.2.m2.1.1.3.cmml">0.6</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.SSS0.Px1.p1.2.m2.1b"><apply id="A1.SS3.SSS0.Px1.p1.2.m2.1.1.cmml" xref="A1.SS3.SSS0.Px1.p1.2.m2.1.1"><gt id="A1.SS3.SSS0.Px1.p1.2.m2.1.1.1.cmml" xref="A1.SS3.SSS0.Px1.p1.2.m2.1.1.1"></gt><csymbol cd="latexml" id="A1.SS3.SSS0.Px1.p1.2.m2.1.1.2.cmml" xref="A1.SS3.SSS0.Px1.p1.2.m2.1.1.2">absent</csymbol><cn type="float" id="A1.SS3.SSS0.Px1.p1.2.m2.1.1.3.cmml" xref="A1.SS3.SSS0.Px1.p1.2.m2.1.1.3">0.6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.SSS0.Px1.p1.2.m2.1c">{&gt;}0.6</annotation></semantics></math>, we extract the connected components of the associated <math id="A1.SS3.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A1.SS3.SSS0.Px1.p1.3.m3.1a"><mi id="A1.SS3.SSS0.Px1.p1.3.m3.1.1" xref="A1.SS3.SSS0.Px1.p1.3.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A1.SS3.SSS0.Px1.p1.3.m3.1b"><ci id="A1.SS3.SSS0.Px1.p1.3.m3.1.1.cmml" xref="A1.SS3.SSS0.Px1.p1.3.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.SSS0.Px1.p1.3.m3.1c">k</annotation></semantics></math>-NN graph thanks to a scalable disjoint set data structure implementation.
We then only keep one representative for each component of duplicate images.
This results in a self-deduplicated data source of 1.1B images.</p>
</div>
</section>
<section id="A1.SS3.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">Relative deduplication</h5>

<div id="A1.SS3.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="A1.SS3.SSS0.Px2.p1.1" class="ltx_p">To reduce redundancy and also properly evaluate the performance of our features, we discard remaining images of our self-deduplicated data source that are too similar to train and test splits of our evaluation datasets.
To achieve this, we apply a similar procedure as for self-deduplication, with a stricter similarity <math id="A1.SS3.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="{&gt;}0.45" display="inline"><semantics id="A1.SS3.SSS0.Px2.p1.1.m1.1a"><mrow id="A1.SS3.SSS0.Px2.p1.1.m1.1.1" xref="A1.SS3.SSS0.Px2.p1.1.m1.1.1.cmml"><mi id="A1.SS3.SSS0.Px2.p1.1.m1.1.1.2" xref="A1.SS3.SSS0.Px2.p1.1.m1.1.1.2.cmml"></mi><mo id="A1.SS3.SSS0.Px2.p1.1.m1.1.1.1" xref="A1.SS3.SSS0.Px2.p1.1.m1.1.1.1.cmml">&gt;</mo><mn id="A1.SS3.SSS0.Px2.p1.1.m1.1.1.3" xref="A1.SS3.SSS0.Px2.p1.1.m1.1.1.3.cmml">0.45</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS3.SSS0.Px2.p1.1.m1.1b"><apply id="A1.SS3.SSS0.Px2.p1.1.m1.1.1.cmml" xref="A1.SS3.SSS0.Px2.p1.1.m1.1.1"><gt id="A1.SS3.SSS0.Px2.p1.1.m1.1.1.1.cmml" xref="A1.SS3.SSS0.Px2.p1.1.m1.1.1.1"></gt><csymbol cd="latexml" id="A1.SS3.SSS0.Px2.p1.1.m1.1.1.2.cmml" xref="A1.SS3.SSS0.Px2.p1.1.m1.1.1.2">absent</csymbol><cn type="float" id="A1.SS3.SSS0.Px2.p1.1.m1.1.1.3.cmml" xref="A1.SS3.SSS0.Px2.p1.1.m1.1.1.3">0.45</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS3.SSS0.Px2.p1.1.m1.1c">{&gt;}0.45</annotation></semantics></math>, this time identifying the duplicate components (if any) to which each reference image belong and discarding it entirely.
This results in a self- and relatively-deduplicated data source of 744M images.</p>
</div>
</section>
</section>
<section id="A1.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.4 </span>Retrieval</h3>

<div id="A1.SS4.p1" class="ltx_para ltx_noindent">
<p id="A1.SS4.p1.8" class="ltx_p">We employ two approaches to augment dataset via retrieval: sample-based and cluster-based.
The first one, sample-based, applies to datasets larger than 1M images and consists in collecting a fixed number <math id="A1.SS4.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A1.SS4.p1.1.m1.1a"><mi id="A1.SS4.p1.1.m1.1.1" xref="A1.SS4.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A1.SS4.p1.1.m1.1b"><ci id="A1.SS4.p1.1.m1.1.1.cmml" xref="A1.SS4.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p1.1.m1.1c">k</annotation></semantics></math> of nearest images for each sample image of the dataset to retrieve, effectively trying to multiply by <math id="A1.SS4.p1.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A1.SS4.p1.2.m2.1a"><mi id="A1.SS4.p1.2.m2.1.1" xref="A1.SS4.p1.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A1.SS4.p1.2.m2.1b"><ci id="A1.SS4.p1.2.m2.1.1.cmml" xref="A1.SS4.p1.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p1.2.m2.1c">k</annotation></semantics></math> the size of the dataset.
We use <math id="A1.SS4.p1.3.m3.1" class="ltx_Math" alttext="k=4" display="inline"><semantics id="A1.SS4.p1.3.m3.1a"><mrow id="A1.SS4.p1.3.m3.1.1" xref="A1.SS4.p1.3.m3.1.1.cmml"><mi id="A1.SS4.p1.3.m3.1.1.2" xref="A1.SS4.p1.3.m3.1.1.2.cmml">k</mi><mo id="A1.SS4.p1.3.m3.1.1.1" xref="A1.SS4.p1.3.m3.1.1.1.cmml">=</mo><mn id="A1.SS4.p1.3.m3.1.1.3" xref="A1.SS4.p1.3.m3.1.1.3.cmml">4</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p1.3.m3.1b"><apply id="A1.SS4.p1.3.m3.1.1.cmml" xref="A1.SS4.p1.3.m3.1.1"><eq id="A1.SS4.p1.3.m3.1.1.1.cmml" xref="A1.SS4.p1.3.m3.1.1.1"></eq><ci id="A1.SS4.p1.3.m3.1.1.2.cmml" xref="A1.SS4.p1.3.m3.1.1.2">𝑘</ci><cn type="integer" id="A1.SS4.p1.3.m3.1.1.3.cmml" xref="A1.SS4.p1.3.m3.1.1.3">4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p1.3.m3.1c">k=4</annotation></semantics></math> for Google Landmarks v2 and ImageNet-22k but a larger <math id="A1.SS4.p1.4.m4.1" class="ltx_Math" alttext="k=32" display="inline"><semantics id="A1.SS4.p1.4.m4.1a"><mrow id="A1.SS4.p1.4.m4.1.1" xref="A1.SS4.p1.4.m4.1.1.cmml"><mi id="A1.SS4.p1.4.m4.1.1.2" xref="A1.SS4.p1.4.m4.1.1.2.cmml">k</mi><mo id="A1.SS4.p1.4.m4.1.1.1" xref="A1.SS4.p1.4.m4.1.1.1.cmml">=</mo><mn id="A1.SS4.p1.4.m4.1.1.3" xref="A1.SS4.p1.4.m4.1.1.3.cmml">32</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p1.4.m4.1b"><apply id="A1.SS4.p1.4.m4.1.1.cmml" xref="A1.SS4.p1.4.m4.1.1"><eq id="A1.SS4.p1.4.m4.1.1.1.cmml" xref="A1.SS4.p1.4.m4.1.1.1"></eq><ci id="A1.SS4.p1.4.m4.1.1.2.cmml" xref="A1.SS4.p1.4.m4.1.1.2">𝑘</ci><cn type="integer" id="A1.SS4.p1.4.m4.1.1.3.cmml" xref="A1.SS4.p1.4.m4.1.1.3">32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p1.4.m4.1c">k=32</annotation></semantics></math> to make this specific retrieval a core part of our LVD-142M dataset.
For smaller datasets, the second approach, cluster-based, consists in first clustering our uncurated data source into <math id="A1.SS4.p1.5.m5.2" class="ltx_Math" alttext="100,000" display="inline"><semantics id="A1.SS4.p1.5.m5.2a"><mrow id="A1.SS4.p1.5.m5.2.3.2" xref="A1.SS4.p1.5.m5.2.3.1.cmml"><mn id="A1.SS4.p1.5.m5.1.1" xref="A1.SS4.p1.5.m5.1.1.cmml">100</mn><mo id="A1.SS4.p1.5.m5.2.3.2.1" xref="A1.SS4.p1.5.m5.2.3.1.cmml">,</mo><mn id="A1.SS4.p1.5.m5.2.2" xref="A1.SS4.p1.5.m5.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p1.5.m5.2b"><list id="A1.SS4.p1.5.m5.2.3.1.cmml" xref="A1.SS4.p1.5.m5.2.3.2"><cn type="integer" id="A1.SS4.p1.5.m5.1.1.cmml" xref="A1.SS4.p1.5.m5.1.1">100</cn><cn type="integer" id="A1.SS4.p1.5.m5.2.2.cmml" xref="A1.SS4.p1.5.m5.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p1.5.m5.2c">100,000</annotation></semantics></math> separate clusters thanks to a distributed <math id="A1.SS4.p1.6.m6.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A1.SS4.p1.6.m6.1a"><mi id="A1.SS4.p1.6.m6.1.1" xref="A1.SS4.p1.6.m6.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A1.SS4.p1.6.m6.1b"><ci id="A1.SS4.p1.6.m6.1.1.cmml" xref="A1.SS4.p1.6.m6.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p1.6.m6.1c">k</annotation></semantics></math>-means implementation.
Each cluster should capture different types of image concept and contents.
We then pick <math id="A1.SS4.p1.7.m7.2" class="ltx_Math" alttext="10,000" display="inline"><semantics id="A1.SS4.p1.7.m7.2a"><mrow id="A1.SS4.p1.7.m7.2.3.2" xref="A1.SS4.p1.7.m7.2.3.1.cmml"><mn id="A1.SS4.p1.7.m7.1.1" xref="A1.SS4.p1.7.m7.1.1.cmml">10</mn><mo id="A1.SS4.p1.7.m7.2.3.2.1" xref="A1.SS4.p1.7.m7.2.3.1.cmml">,</mo><mn id="A1.SS4.p1.7.m7.2.2" xref="A1.SS4.p1.7.m7.2.2.cmml">000</mn></mrow><annotation-xml encoding="MathML-Content" id="A1.SS4.p1.7.m7.2b"><list id="A1.SS4.p1.7.m7.2.3.1.cmml" xref="A1.SS4.p1.7.m7.2.3.2"><cn type="integer" id="A1.SS4.p1.7.m7.1.1.cmml" xref="A1.SS4.p1.7.m7.1.1">10</cn><cn type="integer" id="A1.SS4.p1.7.m7.2.2.cmml" xref="A1.SS4.p1.7.m7.2.2">000</cn></list></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p1.7.m7.2c">10,000</annotation></semantics></math> images from each cluster associated with more than <math id="A1.SS4.p1.8.m8.1" class="ltx_Math" alttext="3" display="inline"><semantics id="A1.SS4.p1.8.m8.1a"><mn id="A1.SS4.p1.8.m8.1.1" xref="A1.SS4.p1.8.m8.1.1.cmml">3</mn><annotation-xml encoding="MathML-Content" id="A1.SS4.p1.8.m8.1b"><cn type="integer" id="A1.SS4.p1.8.m8.1.1.cmml" xref="A1.SS4.p1.8.m8.1.1">3</cn></annotation-xml><annotation encoding="application/x-tex" id="A1.SS4.p1.8.m8.1c">3</annotation></semantics></math> images of the retrieved dataset.
As this can result in a very large number of retrieved images for some dataset, we restrict such retrievals to a maximum of 1M images to maintain the balance between the different datasets within LVD-142M.</p>
</div>
<figure id="A1.T15" class="ltx_table">
<div id="A1.T15.2" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:457.2pt;vertical-align:-0.9pt;"><span class="ltx_transformed_inner" style="transform:translate(-31.2pt,32.9pt) scale(0.874109706124546,0.874109706124546) ;">
<table id="A1.T15.2.1" class="ltx_tabular ltx_align_middle">
<tr id="A1.T15.2.1.1" class="ltx_tr">
<td id="A1.T15.2.1.1.1" class="ltx_td ltx_align_left ltx_border_tt">Task</td>
<td id="A1.T15.2.1.1.2" class="ltx_td ltx_align_left ltx_border_tt">Dataset / Split</td>
<td id="A1.T15.2.1.1.3" class="ltx_td ltx_align_right ltx_border_tt">Images</td>
<td id="A1.T15.2.1.1.4" class="ltx_td ltx_align_center ltx_border_tt">Retrieval</td>
<td id="A1.T15.2.1.1.5" class="ltx_td ltx_align_right ltx_border_tt">Retrieved</td>
<td id="A1.T15.2.1.1.6" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_tt">Final</td>
</tr>
<tr id="A1.T15.2.1.2" class="ltx_tr">
<td id="A1.T15.2.1.2.1" class="ltx_td ltx_align_left ltx_border_t">classification</td>
<td id="A1.T15.2.1.2.2" class="ltx_td ltx_align_left ltx_border_t">ImageNet-22k / –</td>
<td id="A1.T15.2.1.2.3" class="ltx_td ltx_align_right ltx_border_t">14,197,086</td>
<td id="A1.T15.2.1.2.4" class="ltx_td ltx_align_center ltx_border_t">as is</td>
<td id="A1.T15.2.1.2.5" class="ltx_td ltx_align_right ltx_border_t">–</td>
<td id="A1.T15.2.1.2.6" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t">14,197,086</td>
</tr>
<tr id="A1.T15.2.1.3" class="ltx_tr">
<td id="A1.T15.2.1.3.1" class="ltx_td ltx_align_left">classification</td>
<td id="A1.T15.2.1.3.2" class="ltx_td ltx_align_left">ImageNet-22k / –</td>
<td id="A1.T15.2.1.3.3" class="ltx_td ltx_align_right">14,197,086</td>
<td id="A1.T15.2.1.3.4" class="ltx_td ltx_align_center">sample</td>
<td id="A1.T15.2.1.3.5" class="ltx_td ltx_align_right">56,788,344</td>
<td id="A1.T15.2.1.3.6" class="ltx_td ltx_nopad_r ltx_align_right">56,788,344</td>
</tr>
<tr id="A1.T15.2.1.4" class="ltx_tr">
<td id="A1.T15.2.1.4.1" class="ltx_td ltx_align_left">classification</td>
<td id="A1.T15.2.1.4.2" class="ltx_td ltx_align_left">ImageNet-1k / train</td>
<td id="A1.T15.2.1.4.3" class="ltx_td ltx_align_right">1,281,167</td>
<td id="A1.T15.2.1.4.4" class="ltx_td ltx_align_center">sample</td>
<td id="A1.T15.2.1.4.5" class="ltx_td ltx_align_right">40,997,344</td>
<td id="A1.T15.2.1.4.6" class="ltx_td ltx_nopad_r ltx_align_right">40,997,344</td>
</tr>
<tr id="A1.T15.2.1.5" class="ltx_tr">
<td id="A1.T15.2.1.5.1" class="ltx_td ltx_align_left ltx_border_t">fine-grained classif.</td>
<td id="A1.T15.2.1.5.2" class="ltx_td ltx_align_left ltx_border_t">Caltech 101 / train</td>
<td id="A1.T15.2.1.5.3" class="ltx_td ltx_align_right ltx_border_t">3,030</td>
<td id="A1.T15.2.1.5.4" class="ltx_td ltx_align_center ltx_border_t">cluster</td>
<td id="A1.T15.2.1.5.5" class="ltx_td ltx_align_right ltx_border_t">2,630,000</td>
<td id="A1.T15.2.1.5.6" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.6" class="ltx_tr">
<td id="A1.T15.2.1.6.1" class="ltx_td ltx_align_left">fine-grained classif.</td>
<td id="A1.T15.2.1.6.2" class="ltx_td ltx_align_left">CUB-200-2011 / train</td>
<td id="A1.T15.2.1.6.3" class="ltx_td ltx_align_right">5,994</td>
<td id="A1.T15.2.1.6.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.6.5" class="ltx_td ltx_align_right">1,300,000</td>
<td id="A1.T15.2.1.6.6" class="ltx_td ltx_nopad_r ltx_align_right">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.7" class="ltx_tr">
<td id="A1.T15.2.1.7.1" class="ltx_td ltx_align_left">fine-grained classif.</td>
<td id="A1.T15.2.1.7.2" class="ltx_td ltx_align_left">DTD / train1</td>
<td id="A1.T15.2.1.7.3" class="ltx_td ltx_align_right">1,880</td>
<td id="A1.T15.2.1.7.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.7.5" class="ltx_td ltx_align_right">1,580,000</td>
<td id="A1.T15.2.1.7.6" class="ltx_td ltx_nopad_r ltx_align_right">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.8" class="ltx_tr">
<td id="A1.T15.2.1.8.1" class="ltx_td ltx_align_left">fine-grained classif.</td>
<td id="A1.T15.2.1.8.2" class="ltx_td ltx_align_left">FGVC-Aircraft / train</td>
<td id="A1.T15.2.1.8.3" class="ltx_td ltx_align_right">3,334</td>
<td id="A1.T15.2.1.8.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.8.5" class="ltx_td ltx_align_right">1,170,000</td>
<td id="A1.T15.2.1.8.6" class="ltx_td ltx_nopad_r ltx_align_right">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.9" class="ltx_tr">
<td id="A1.T15.2.1.9.1" class="ltx_td ltx_align_left">fine-grained classif.</td>
<td id="A1.T15.2.1.9.2" class="ltx_td ltx_align_left">Flowers-102 / train</td>
<td id="A1.T15.2.1.9.3" class="ltx_td ltx_align_right">1,020</td>
<td id="A1.T15.2.1.9.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.9.5" class="ltx_td ltx_align_right">1,060,000</td>
<td id="A1.T15.2.1.9.6" class="ltx_td ltx_nopad_r ltx_align_right">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.10" class="ltx_tr">
<td id="A1.T15.2.1.10.1" class="ltx_td ltx_align_left">fine-grained classif.</td>
<td id="A1.T15.2.1.10.2" class="ltx_td ltx_align_left">Food-101 / train</td>
<td id="A1.T15.2.1.10.3" class="ltx_td ltx_align_right">75,750</td>
<td id="A1.T15.2.1.10.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.10.5" class="ltx_td ltx_align_right">21,670,000</td>
<td id="A1.T15.2.1.10.6" class="ltx_td ltx_nopad_r ltx_align_right">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.11" class="ltx_tr">
<td id="A1.T15.2.1.11.1" class="ltx_td ltx_align_left">fine-grained classif.</td>
<td id="A1.T15.2.1.11.2" class="ltx_td ltx_align_left">Oxford-IIIT Pet / trainval</td>
<td id="A1.T15.2.1.11.3" class="ltx_td ltx_align_right">3,680</td>
<td id="A1.T15.2.1.11.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.11.5" class="ltx_td ltx_align_right">2,750,000</td>
<td id="A1.T15.2.1.11.6" class="ltx_td ltx_nopad_r ltx_align_right">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.12" class="ltx_tr">
<td id="A1.T15.2.1.12.1" class="ltx_td ltx_align_left">fine-grained classif.</td>
<td id="A1.T15.2.1.12.2" class="ltx_td ltx_align_left">Stanford Cars / train</td>
<td id="A1.T15.2.1.12.3" class="ltx_td ltx_align_right">8,144</td>
<td id="A1.T15.2.1.12.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.12.5" class="ltx_td ltx_align_right">7,220,000</td>
<td id="A1.T15.2.1.12.6" class="ltx_td ltx_nopad_r ltx_align_right">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.13" class="ltx_tr">
<td id="A1.T15.2.1.13.1" class="ltx_td ltx_align_left">fine-grained classif.</td>
<td id="A1.T15.2.1.13.2" class="ltx_td ltx_align_left">SUN397 / train1</td>
<td id="A1.T15.2.1.13.3" class="ltx_td ltx_align_right">19,850</td>
<td id="A1.T15.2.1.13.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.13.5" class="ltx_td ltx_align_right">18,950,000</td>
<td id="A1.T15.2.1.13.6" class="ltx_td ltx_nopad_r ltx_align_right">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.14" class="ltx_tr">
<td id="A1.T15.2.1.14.1" class="ltx_td ltx_align_left">fine-grained classif.</td>
<td id="A1.T15.2.1.14.2" class="ltx_td ltx_align_left">Pascal VOC 2007 / train</td>
<td id="A1.T15.2.1.14.3" class="ltx_td ltx_align_right">2,501</td>
<td id="A1.T15.2.1.14.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.14.5" class="ltx_td ltx_align_right">1,010,000</td>
<td id="A1.T15.2.1.14.6" class="ltx_td ltx_nopad_r ltx_align_right">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.15" class="ltx_tr">
<td id="A1.T15.2.1.15.1" class="ltx_td ltx_align_left ltx_border_t">segmentation</td>
<td id="A1.T15.2.1.15.2" class="ltx_td ltx_align_left ltx_border_t">ADE20K / train</td>
<td id="A1.T15.2.1.15.3" class="ltx_td ltx_align_right ltx_border_t">20,210</td>
<td id="A1.T15.2.1.15.4" class="ltx_td ltx_align_center ltx_border_t">cluster</td>
<td id="A1.T15.2.1.15.5" class="ltx_td ltx_align_right ltx_border_t">20,720,000</td>
<td id="A1.T15.2.1.15.6" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.16" class="ltx_tr">
<td id="A1.T15.2.1.16.1" class="ltx_td ltx_align_left">segmentation</td>
<td id="A1.T15.2.1.16.2" class="ltx_td ltx_align_left">Cityscapes / train</td>
<td id="A1.T15.2.1.16.3" class="ltx_td ltx_align_right">2,975</td>
<td id="A1.T15.2.1.16.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.16.5" class="ltx_td ltx_align_right">1,390,000</td>
<td id="A1.T15.2.1.16.6" class="ltx_td ltx_nopad_r ltx_align_right">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.17" class="ltx_tr">
<td id="A1.T15.2.1.17.1" class="ltx_td ltx_align_left">segmentation</td>
<td id="A1.T15.2.1.17.2" class="ltx_td ltx_align_left">Pascal VOC 2012 (seg.) / trainaug</td>
<td id="A1.T15.2.1.17.3" class="ltx_td ltx_align_right">1,464</td>
<td id="A1.T15.2.1.17.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.17.5" class="ltx_td ltx_align_right">10,140,000</td>
<td id="A1.T15.2.1.17.6" class="ltx_td ltx_nopad_r ltx_align_right">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.18" class="ltx_tr">
<td id="A1.T15.2.1.18.1" class="ltx_td ltx_align_left ltx_border_t">depth estimation</td>
<td id="A1.T15.2.1.18.2" class="ltx_td ltx_align_left ltx_border_t">Mapillary SLS / train</td>
<td id="A1.T15.2.1.18.3" class="ltx_td ltx_align_right ltx_border_t">1,434,262</td>
<td id="A1.T15.2.1.18.4" class="ltx_td ltx_align_center ltx_border_t">as is</td>
<td id="A1.T15.2.1.18.5" class="ltx_td ltx_align_right ltx_border_t">–</td>
<td id="A1.T15.2.1.18.6" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t">1,434,262</td>
</tr>
<tr id="A1.T15.2.1.19" class="ltx_tr">
<td id="A1.T15.2.1.19.1" class="ltx_td ltx_align_left">depth estimation</td>
<td id="A1.T15.2.1.19.2" class="ltx_td ltx_align_left">KITTI / train (Eigen)</td>
<td id="A1.T15.2.1.19.3" class="ltx_td ltx_align_right">23,158</td>
<td id="A1.T15.2.1.19.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.19.5" class="ltx_td ltx_align_right">3,700,000</td>
<td id="A1.T15.2.1.19.6" class="ltx_td ltx_nopad_r ltx_align_right">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.20" class="ltx_tr">
<td id="A1.T15.2.1.20.1" class="ltx_td ltx_align_left">depth estimation</td>
<td id="A1.T15.2.1.20.2" class="ltx_td ltx_align_left">NYU Depth V2 / train</td>
<td id="A1.T15.2.1.20.3" class="ltx_td ltx_align_right">24,231</td>
<td id="A1.T15.2.1.20.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.20.5" class="ltx_td ltx_align_right">10,850,000</td>
<td id="A1.T15.2.1.20.6" class="ltx_td ltx_nopad_r ltx_align_right">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.21" class="ltx_tr">
<td id="A1.T15.2.1.21.1" class="ltx_td ltx_align_left">depth estimation</td>
<td id="A1.T15.2.1.21.2" class="ltx_td ltx_align_left">SUN RGB-D / train</td>
<td id="A1.T15.2.1.21.3" class="ltx_td ltx_align_right">4,829</td>
<td id="A1.T15.2.1.21.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.21.5" class="ltx_td ltx_align_right">4,870,000</td>
<td id="A1.T15.2.1.21.6" class="ltx_td ltx_nopad_r ltx_align_right">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.22" class="ltx_tr">
<td id="A1.T15.2.1.22.1" class="ltx_td ltx_align_left ltx_border_t">retrieval</td>
<td id="A1.T15.2.1.22.2" class="ltx_td ltx_align_left ltx_border_t">Google Landmarks v2 / train (clean)</td>
<td id="A1.T15.2.1.22.3" class="ltx_td ltx_align_right ltx_border_t">1,580,470</td>
<td id="A1.T15.2.1.22.4" class="ltx_td ltx_align_center ltx_border_t">as is</td>
<td id="A1.T15.2.1.22.5" class="ltx_td ltx_align_right ltx_border_t">–</td>
<td id="A1.T15.2.1.22.6" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_t">1,580,470</td>
</tr>
<tr id="A1.T15.2.1.23" class="ltx_tr">
<td id="A1.T15.2.1.23.1" class="ltx_td ltx_align_left">retrieval</td>
<td id="A1.T15.2.1.23.2" class="ltx_td ltx_align_left">Google Landmarks v2 / train (clean)</td>
<td id="A1.T15.2.1.23.3" class="ltx_td ltx_align_right">1,580,470</td>
<td id="A1.T15.2.1.23.4" class="ltx_td ltx_align_center">sample</td>
<td id="A1.T15.2.1.23.5" class="ltx_td ltx_align_right">6,321,880</td>
<td id="A1.T15.2.1.23.6" class="ltx_td ltx_nopad_r ltx_align_right">6,321,880</td>
</tr>
<tr id="A1.T15.2.1.24" class="ltx_tr">
<td id="A1.T15.2.1.24.1" class="ltx_td ltx_align_left">retrieval</td>
<td id="A1.T15.2.1.24.2" class="ltx_td ltx_align_left">AmsterTime / new</td>
<td id="A1.T15.2.1.24.3" class="ltx_td ltx_align_right">1,231</td>
<td id="A1.T15.2.1.24.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.24.5" class="ltx_td ltx_align_right">960,000</td>
<td id="A1.T15.2.1.24.6" class="ltx_td ltx_nopad_r ltx_align_right">960,000</td>
</tr>
<tr id="A1.T15.2.1.25" class="ltx_tr">
<td id="A1.T15.2.1.25.1" class="ltx_td ltx_align_left">retrieval</td>
<td id="A1.T15.2.1.25.2" class="ltx_td ltx_align_left">AmsterTime / old</td>
<td id="A1.T15.2.1.25.3" class="ltx_td ltx_align_right">1,231</td>
<td id="A1.T15.2.1.25.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.25.5" class="ltx_td ltx_align_right">830,000</td>
<td id="A1.T15.2.1.25.6" class="ltx_td ltx_nopad_r ltx_align_right">830,000</td>
</tr>
<tr id="A1.T15.2.1.26" class="ltx_tr">
<td id="A1.T15.2.1.26.1" class="ltx_td ltx_align_left">retrieval</td>
<td id="A1.T15.2.1.26.2" class="ltx_td ltx_align_left">Met / train</td>
<td id="A1.T15.2.1.26.3" class="ltx_td ltx_align_right">397,121</td>
<td id="A1.T15.2.1.26.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.26.5" class="ltx_td ltx_align_right">62,860,000</td>
<td id="A1.T15.2.1.26.6" class="ltx_td ltx_nopad_r ltx_align_right">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.27" class="ltx_tr">
<td id="A1.T15.2.1.27.1" class="ltx_td ltx_align_left">retrieval</td>
<td id="A1.T15.2.1.27.2" class="ltx_td ltx_align_left">Revisiting Oxford / base</td>
<td id="A1.T15.2.1.27.3" class="ltx_td ltx_align_right">4,993</td>
<td id="A1.T15.2.1.27.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.27.5" class="ltx_td ltx_align_right">3,680,000</td>
<td id="A1.T15.2.1.27.6" class="ltx_td ltx_nopad_r ltx_align_right">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.28" class="ltx_tr">
<td id="A1.T15.2.1.28.1" class="ltx_td ltx_align_left">retrieval</td>
<td id="A1.T15.2.1.28.2" class="ltx_td ltx_align_left">Revisiting Paris / base</td>
<td id="A1.T15.2.1.28.3" class="ltx_td ltx_align_right">6,322</td>
<td id="A1.T15.2.1.28.4" class="ltx_td ltx_align_center">cluster</td>
<td id="A1.T15.2.1.28.5" class="ltx_td ltx_align_right">3,660,000</td>
<td id="A1.T15.2.1.28.6" class="ltx_td ltx_nopad_r ltx_align_right">1,000,000</td>
</tr>
<tr id="A1.T15.2.1.29" class="ltx_tr">
<td id="A1.T15.2.1.29.1" class="ltx_td ltx_border_tt"></td>
<td id="A1.T15.2.1.29.2" class="ltx_td ltx_border_tt"></td>
<td id="A1.T15.2.1.29.3" class="ltx_td ltx_border_tt"></td>
<td id="A1.T15.2.1.29.4" class="ltx_td ltx_border_tt"></td>
<td id="A1.T15.2.1.29.5" class="ltx_td ltx_border_tt"></td>
<td id="A1.T15.2.1.29.6" class="ltx_td ltx_nopad_r ltx_align_right ltx_border_tt">142,109,386</td>
</tr>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="A1.T15.4.1.1" class="ltx_text" style="font-size:90%;">Table 15</span>: </span><span id="A1.T15.5.2" class="ltx_text" style="font-size:90%;">
<span id="A1.T15.5.2.1" class="ltx_text ltx_font_bold">Composition of our LVD-142M dataset.</span>
We report the list of datasets and associated splits used to build the dataset, how they were included (as is without retrieval or via sample-based or cluster-based retrieval).
For retrievals, we indicate the actual number of retrieved images and the final number included in the dataset.

We chose to include as many datasets as possible in the pretraining data in order to cover as many domains as possible.
We kept a few datasets aside in order to evaluate performance outside of the pretraining domain.
More details about dataset usages can be found in Table <a href="#A3" title="Appendix C List of Datasets used ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>.

</span></figcaption>
</figure>
</section>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Implementation Details</h2>

<section id="A2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.1 </span>Unsupervised pre-training</h3>

<div id="A2.SS1.p1" class="ltx_para ltx_noindent">
<p id="A2.SS1.p1.1" class="ltx_p">For unsupervised pre-training we build on the DINO and iBOT codebases. We use hyperparameters shown in
Table <a href="#A2.T16" title="Table 16 ‣ B.1 Unsupervised pre-training ‣ Appendix B Implementation Details ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">16</span></a>, ViT architectures described in Table <a href="#A2.T17" title="Table 17 ‣ B.1 Unsupervised pre-training ‣ Appendix B Implementation Details ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">17</span></a>.</p>
</div>
<figure id="A2.T16" class="ltx_table">
<table id="A2.T16.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A2.T16.2.1" class="ltx_tr">
<td id="A2.T16.2.1.1" class="ltx_td ltx_border_tt"></td>
<td id="A2.T16.2.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="A2.T16.2.1.2.1" class="ltx_text" style="font-size:90%;">Arch.</span></td>
<td id="A2.T16.2.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="A2.T16.2.1.3.1" class="ltx_text" style="font-size:90%;">Drop-rate</span></td>
<td id="A2.T16.2.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="A2.T16.2.1.4.1" class="ltx_text" style="font-size:90%;">LR</span></td>
<td id="A2.T16.2.1.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="A2.T16.2.1.5.1" class="ltx_text" style="font-size:90%;">Batch size</span></td>
</tr>
<tr id="A2.T16.2.2" class="ltx_tr">
<td id="A2.T16.2.2.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A2.T16.2.2.1.1" class="ltx_text" style="font-size:90%;">DINOv2-S (distilled)</span></td>
<td id="A2.T16.2.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A2.T16.2.2.2.1" class="ltx_text" style="font-size:90%;">ViT-S/14</span></td>
<td id="A2.T16.2.2.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A2.T16.2.2.3.1" class="ltx_text" style="font-size:90%;">0</span></td>
<td id="A2.T16.2.2.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A2.T16.2.2.4.1" class="ltx_text" style="font-size:90%;">1e-3</span></td>
<td id="A2.T16.2.2.5" class="ltx_td ltx_align_center ltx_border_t"><span id="A2.T16.2.2.5.1" class="ltx_text" style="font-size:90%;">2048</span></td>
</tr>
<tr id="A2.T16.2.3" class="ltx_tr">
<td id="A2.T16.2.3.1" class="ltx_td ltx_align_left"><span id="A2.T16.2.3.1.1" class="ltx_text" style="font-size:90%;">DINOv2-B (distilled)</span></td>
<td id="A2.T16.2.3.2" class="ltx_td ltx_align_center"><span id="A2.T16.2.3.2.1" class="ltx_text" style="font-size:90%;">ViT-B/14</span></td>
<td id="A2.T16.2.3.3" class="ltx_td ltx_align_center"><span id="A2.T16.2.3.3.1" class="ltx_text" style="font-size:90%;">0</span></td>
<td id="A2.T16.2.3.4" class="ltx_td ltx_align_center"><span id="A2.T16.2.3.4.1" class="ltx_text" style="font-size:90%;">1e-3</span></td>
<td id="A2.T16.2.3.5" class="ltx_td ltx_align_center"><span id="A2.T16.2.3.5.1" class="ltx_text" style="font-size:90%;">2048</span></td>
</tr>
<tr id="A2.T16.2.4" class="ltx_tr">
<td id="A2.T16.2.4.1" class="ltx_td ltx_align_left"><span id="A2.T16.2.4.1.1" class="ltx_text" style="font-size:90%;">DINOv2-L (distilled)</span></td>
<td id="A2.T16.2.4.2" class="ltx_td ltx_align_center"><span id="A2.T16.2.4.2.1" class="ltx_text" style="font-size:90%;">ViT-L/14</span></td>
<td id="A2.T16.2.4.3" class="ltx_td ltx_align_center"><span id="A2.T16.2.4.3.1" class="ltx_text" style="font-size:90%;">0</span></td>
<td id="A2.T16.2.4.4" class="ltx_td ltx_align_center"><span id="A2.T16.2.4.4.1" class="ltx_text" style="font-size:90%;">1e-3</span></td>
<td id="A2.T16.2.4.5" class="ltx_td ltx_align_center"><span id="A2.T16.2.4.5.1" class="ltx_text" style="font-size:90%;">2048</span></td>
</tr>
<tr id="A2.T16.2.5" class="ltx_tr">
<td id="A2.T16.2.5.1" class="ltx_td ltx_align_left"><span id="A2.T16.2.5.1.1" class="ltx_text" style="font-size:90%;">DINOv2-L (from scratch)</span></td>
<td id="A2.T16.2.5.2" class="ltx_td ltx_align_center"><span id="A2.T16.2.5.2.1" class="ltx_text" style="font-size:90%;">ViT-L/14</span></td>
<td id="A2.T16.2.5.3" class="ltx_td ltx_align_center"><span id="A2.T16.2.5.3.1" class="ltx_text" style="font-size:90%;">0.4</span></td>
<td id="A2.T16.2.5.4" class="ltx_td ltx_align_center"><span id="A2.T16.2.5.4.1" class="ltx_text" style="font-size:90%;">3.5e-4</span></td>
<td id="A2.T16.2.5.5" class="ltx_td ltx_align_center"><span id="A2.T16.2.5.5.1" class="ltx_text" style="font-size:90%;">3072</span></td>
</tr>
<tr id="A2.T16.2.6" class="ltx_tr">
<td id="A2.T16.2.6.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="A2.T16.2.6.1.1" class="ltx_text" style="font-size:90%;">DINOv2-g (from scratch)</span></td>
<td id="A2.T16.2.6.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="A2.T16.2.6.2.1" class="ltx_text" style="font-size:90%;">ViT-g/14</span></td>
<td id="A2.T16.2.6.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="A2.T16.2.6.3.1" class="ltx_text" style="font-size:90%;">0.4</span></td>
<td id="A2.T16.2.6.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="A2.T16.2.6.4.1" class="ltx_text" style="font-size:90%;">3.5e-4</span></td>
<td id="A2.T16.2.6.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="A2.T16.2.6.5.1" class="ltx_text" style="font-size:90%;">3072</span></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering" style="font-size:90%;"><span class="ltx_tag ltx_tag_table">Table 16: </span>
<span id="A2.T16.7.1" class="ltx_text ltx_font_bold">Training hyperparameters for DINOv2-S, DINOv2-B, DINOv2-L and DINOv2-g.</span>
All models run for 625k iterations with optimizer AdamW, an initial LayerScale value of 1e-5, a weight decay cosine schedule from 0.04 to 0.2, a learning rate warmup of 100k iterations, a teacher momentum cosine schedule from 0.994 to 1, and we train in float16 precision in all cases (except for the DINO heads where we reduce the gradients in float32).
</figcaption>
</figure>
<figure id="A2.T17" class="ltx_table">
<table id="A2.T17.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A2.T17.2.1" class="ltx_tr">
<td id="A2.T17.2.1.1" class="ltx_td ltx_align_left ltx_border_tt"><span id="A2.T17.2.1.1.1" class="ltx_text" style="font-size:90%;">Arch.</span></td>
<td id="A2.T17.2.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="A2.T17.2.1.2.1" class="ltx_text" style="font-size:90%;">Embed dim</span></td>
<td id="A2.T17.2.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="A2.T17.2.1.3.1" class="ltx_text" style="font-size:90%;">Heads</span></td>
<td id="A2.T17.2.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="A2.T17.2.1.4.1" class="ltx_text" style="font-size:90%;">Blocks</span></td>
<td id="A2.T17.2.1.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="A2.T17.2.1.5.1" class="ltx_text" style="font-size:90%;">FFN layer</span></td>
</tr>
<tr id="A2.T17.2.2" class="ltx_tr">
<td id="A2.T17.2.2.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A2.T17.2.2.1.1" class="ltx_text" style="font-size:90%;">ViT-S/14 (distilled)</span></td>
<td id="A2.T17.2.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A2.T17.2.2.2.1" class="ltx_text" style="font-size:90%;">384</span></td>
<td id="A2.T17.2.2.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A2.T17.2.2.3.1" class="ltx_text" style="font-size:90%;">6</span></td>
<td id="A2.T17.2.2.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A2.T17.2.2.4.1" class="ltx_text" style="font-size:90%;">12</span></td>
<td id="A2.T17.2.2.5" class="ltx_td ltx_align_center ltx_border_t"><span id="A2.T17.2.2.5.1" class="ltx_text" style="font-size:90%;">MLP</span></td>
</tr>
<tr id="A2.T17.2.3" class="ltx_tr">
<td id="A2.T17.2.3.1" class="ltx_td ltx_align_left"><span id="A2.T17.2.3.1.1" class="ltx_text" style="font-size:90%;">ViT-B/14 (distilled)</span></td>
<td id="A2.T17.2.3.2" class="ltx_td ltx_align_center"><span id="A2.T17.2.3.2.1" class="ltx_text" style="font-size:90%;">768</span></td>
<td id="A2.T17.2.3.3" class="ltx_td ltx_align_center"><span id="A2.T17.2.3.3.1" class="ltx_text" style="font-size:90%;">12</span></td>
<td id="A2.T17.2.3.4" class="ltx_td ltx_align_center"><span id="A2.T17.2.3.4.1" class="ltx_text" style="font-size:90%;">18</span></td>
<td id="A2.T17.2.3.5" class="ltx_td ltx_align_center"><span id="A2.T17.2.3.5.1" class="ltx_text" style="font-size:90%;">MLP</span></td>
</tr>
<tr id="A2.T17.2.4" class="ltx_tr">
<td id="A2.T17.2.4.1" class="ltx_td ltx_align_left"><span id="A2.T17.2.4.1.1" class="ltx_text" style="font-size:90%;">ViT-L/14 (distilled)</span></td>
<td id="A2.T17.2.4.2" class="ltx_td ltx_align_center"><span id="A2.T17.2.4.2.1" class="ltx_text" style="font-size:90%;">1024</span></td>
<td id="A2.T17.2.4.3" class="ltx_td ltx_align_center"><span id="A2.T17.2.4.3.1" class="ltx_text" style="font-size:90%;">16</span></td>
<td id="A2.T17.2.4.4" class="ltx_td ltx_align_center"><span id="A2.T17.2.4.4.1" class="ltx_text" style="font-size:90%;">24</span></td>
<td id="A2.T17.2.4.5" class="ltx_td ltx_align_center"><span id="A2.T17.2.4.5.1" class="ltx_text" style="font-size:90%;">MLP</span></td>
</tr>
<tr id="A2.T17.2.5" class="ltx_tr">
<td id="A2.T17.2.5.1" class="ltx_td ltx_align_left"><span id="A2.T17.2.5.1.1" class="ltx_text" style="font-size:90%;">ViT-L/14 (from scratch)</span></td>
<td id="A2.T17.2.5.2" class="ltx_td ltx_align_center"><span id="A2.T17.2.5.2.1" class="ltx_text" style="font-size:90%;">1024</span></td>
<td id="A2.T17.2.5.3" class="ltx_td ltx_align_center"><span id="A2.T17.2.5.3.1" class="ltx_text" style="font-size:90%;">16</span></td>
<td id="A2.T17.2.5.4" class="ltx_td ltx_align_center"><span id="A2.T17.2.5.4.1" class="ltx_text" style="font-size:90%;">24</span></td>
<td id="A2.T17.2.5.5" class="ltx_td ltx_align_center"><span id="A2.T17.2.5.5.1" class="ltx_text" style="font-size:90%;">SwiGLU</span></td>
</tr>
<tr id="A2.T17.2.6" class="ltx_tr">
<td id="A2.T17.2.6.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="A2.T17.2.6.1.1" class="ltx_text" style="font-size:90%;">ViT-g/14 (from scratch)</span></td>
<td id="A2.T17.2.6.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="A2.T17.2.6.2.1" class="ltx_text" style="font-size:90%;">1536</span></td>
<td id="A2.T17.2.6.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="A2.T17.2.6.3.1" class="ltx_text" style="font-size:90%;">24</span></td>
<td id="A2.T17.2.6.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="A2.T17.2.6.4.1" class="ltx_text" style="font-size:90%;">40</span></td>
<td id="A2.T17.2.6.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="A2.T17.2.6.5.1" class="ltx_text" style="font-size:90%;">SwiGLU</span></td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering" style="font-size:90%;"><span class="ltx_tag ltx_tag_table">Table 17: </span>
<span id="A2.T17.11.1" class="ltx_text ltx_font_bold">Architecture details of the ViT-S/B/L/g networks used in this work.</span> We use MLP feed-forward networks for distilled models, and SwiGLU  <cite class="ltx_cite ltx_citemacro_citep">(Shazeer, <a href="#bib.bib103" title="" class="ltx_ref">2020</a>)</cite> when training from scratch.
</figcaption>
</figure>
<section id="A2.SS1.SSS0.Px1" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">KoLeo regularization.</h5>

<div id="A2.SS1.SSS0.Px1.p1" class="ltx_para ltx_noindent">
<p id="A2.SS1.SSS0.Px1.p1.1" class="ltx_p">We apply the KoLeo regularizer with a weight of 0.1 between the class tokens of the first global crop, for all samples within a GPU without cross-communication for this step.</p>
</div>
</section>
<section id="A2.SS1.SSS0.Px2" class="ltx_paragraph">
<h5 class="ltx_title ltx_title_paragraph">EMA update for the teacher.</h5>

<div id="A2.SS1.SSS0.Px2.p1" class="ltx_para ltx_noindent">
<p id="A2.SS1.SSS0.Px2.p1.1" class="ltx_p">The teacher is initialized with the same state as the student, and is an exponential moving average of the student network, with a momentum value in [0.994, 1.0] following a cosine schedule. It is updated at the end of every training step.</p>
</div>
</section>
</section>
<section id="A2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.2 </span>High-Resolution adaptation</h3>

<div id="A2.SS2.p1" class="ltx_para ltx_noindent">
<p id="A2.SS2.p1.1" class="ltx_p">We initialise the model with the pretrained weights then train it for 10k iterations with the same procedure as the original pretraining. All the schedules are kept the same as in the original training, but compressed to fit in 10k iterations. All the hyperparameters are kept the same as in the first pretraining, except the base learning rate which is reduced.</p>
</div>
</section>
<section id="A2.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.3 </span>Linear probing evaluation</h3>

<div id="A2.SS3.p1" class="ltx_para ltx_noindent">
<p id="A2.SS3.p1.1" class="ltx_p">For linear probing we define 3 evaluation parameters: the learning rate, how many output layers we use, whether we concatenate the average-pooled patch token features with the class token (or use only the class token).
We train our linear layer with SGD for 12500 iterations, using random-resized-crop data augmentation, and perform the following grid search:</p>
<ul id="A2.I1" class="ltx_itemize">
<li id="A2.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A2.I1.i1.p1" class="ltx_para">
<p id="A2.I1.i1.p1.1" class="ltx_p">learning rate in <math id="A2.I1.i1.p1.1.m1.13" class="ltx_Math" alttext="\{0.0001,0.0002,0.0005,0.001,0.002,0.005,0.01,0.02,0.05,0.1,0.2,0.3,0.5\}" display="inline"><semantics id="A2.I1.i1.p1.1.m1.13a"><mrow id="A2.I1.i1.p1.1.m1.13.14.2" xref="A2.I1.i1.p1.1.m1.13.14.1.cmml"><mo stretchy="false" id="A2.I1.i1.p1.1.m1.13.14.2.1" xref="A2.I1.i1.p1.1.m1.13.14.1.cmml">{</mo><mn id="A2.I1.i1.p1.1.m1.1.1" xref="A2.I1.i1.p1.1.m1.1.1.cmml">0.0001</mn><mo id="A2.I1.i1.p1.1.m1.13.14.2.2" xref="A2.I1.i1.p1.1.m1.13.14.1.cmml">,</mo><mn id="A2.I1.i1.p1.1.m1.2.2" xref="A2.I1.i1.p1.1.m1.2.2.cmml">0.0002</mn><mo id="A2.I1.i1.p1.1.m1.13.14.2.3" xref="A2.I1.i1.p1.1.m1.13.14.1.cmml">,</mo><mn id="A2.I1.i1.p1.1.m1.3.3" xref="A2.I1.i1.p1.1.m1.3.3.cmml">0.0005</mn><mo id="A2.I1.i1.p1.1.m1.13.14.2.4" xref="A2.I1.i1.p1.1.m1.13.14.1.cmml">,</mo><mn id="A2.I1.i1.p1.1.m1.4.4" xref="A2.I1.i1.p1.1.m1.4.4.cmml">0.001</mn><mo id="A2.I1.i1.p1.1.m1.13.14.2.5" xref="A2.I1.i1.p1.1.m1.13.14.1.cmml">,</mo><mn id="A2.I1.i1.p1.1.m1.5.5" xref="A2.I1.i1.p1.1.m1.5.5.cmml">0.002</mn><mo id="A2.I1.i1.p1.1.m1.13.14.2.6" xref="A2.I1.i1.p1.1.m1.13.14.1.cmml">,</mo><mn id="A2.I1.i1.p1.1.m1.6.6" xref="A2.I1.i1.p1.1.m1.6.6.cmml">0.005</mn><mo id="A2.I1.i1.p1.1.m1.13.14.2.7" xref="A2.I1.i1.p1.1.m1.13.14.1.cmml">,</mo><mn id="A2.I1.i1.p1.1.m1.7.7" xref="A2.I1.i1.p1.1.m1.7.7.cmml">0.01</mn><mo id="A2.I1.i1.p1.1.m1.13.14.2.8" xref="A2.I1.i1.p1.1.m1.13.14.1.cmml">,</mo><mn id="A2.I1.i1.p1.1.m1.8.8" xref="A2.I1.i1.p1.1.m1.8.8.cmml">0.02</mn><mo id="A2.I1.i1.p1.1.m1.13.14.2.9" xref="A2.I1.i1.p1.1.m1.13.14.1.cmml">,</mo><mn id="A2.I1.i1.p1.1.m1.9.9" xref="A2.I1.i1.p1.1.m1.9.9.cmml">0.05</mn><mo id="A2.I1.i1.p1.1.m1.13.14.2.10" xref="A2.I1.i1.p1.1.m1.13.14.1.cmml">,</mo><mn id="A2.I1.i1.p1.1.m1.10.10" xref="A2.I1.i1.p1.1.m1.10.10.cmml">0.1</mn><mo id="A2.I1.i1.p1.1.m1.13.14.2.11" xref="A2.I1.i1.p1.1.m1.13.14.1.cmml">,</mo><mn id="A2.I1.i1.p1.1.m1.11.11" xref="A2.I1.i1.p1.1.m1.11.11.cmml">0.2</mn><mo id="A2.I1.i1.p1.1.m1.13.14.2.12" xref="A2.I1.i1.p1.1.m1.13.14.1.cmml">,</mo><mn id="A2.I1.i1.p1.1.m1.12.12" xref="A2.I1.i1.p1.1.m1.12.12.cmml">0.3</mn><mo id="A2.I1.i1.p1.1.m1.13.14.2.13" xref="A2.I1.i1.p1.1.m1.13.14.1.cmml">,</mo><mn id="A2.I1.i1.p1.1.m1.13.13" xref="A2.I1.i1.p1.1.m1.13.13.cmml">0.5</mn><mo stretchy="false" id="A2.I1.i1.p1.1.m1.13.14.2.14" xref="A2.I1.i1.p1.1.m1.13.14.1.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.I1.i1.p1.1.m1.13b"><set id="A2.I1.i1.p1.1.m1.13.14.1.cmml" xref="A2.I1.i1.p1.1.m1.13.14.2"><cn type="float" id="A2.I1.i1.p1.1.m1.1.1.cmml" xref="A2.I1.i1.p1.1.m1.1.1">0.0001</cn><cn type="float" id="A2.I1.i1.p1.1.m1.2.2.cmml" xref="A2.I1.i1.p1.1.m1.2.2">0.0002</cn><cn type="float" id="A2.I1.i1.p1.1.m1.3.3.cmml" xref="A2.I1.i1.p1.1.m1.3.3">0.0005</cn><cn type="float" id="A2.I1.i1.p1.1.m1.4.4.cmml" xref="A2.I1.i1.p1.1.m1.4.4">0.001</cn><cn type="float" id="A2.I1.i1.p1.1.m1.5.5.cmml" xref="A2.I1.i1.p1.1.m1.5.5">0.002</cn><cn type="float" id="A2.I1.i1.p1.1.m1.6.6.cmml" xref="A2.I1.i1.p1.1.m1.6.6">0.005</cn><cn type="float" id="A2.I1.i1.p1.1.m1.7.7.cmml" xref="A2.I1.i1.p1.1.m1.7.7">0.01</cn><cn type="float" id="A2.I1.i1.p1.1.m1.8.8.cmml" xref="A2.I1.i1.p1.1.m1.8.8">0.02</cn><cn type="float" id="A2.I1.i1.p1.1.m1.9.9.cmml" xref="A2.I1.i1.p1.1.m1.9.9">0.05</cn><cn type="float" id="A2.I1.i1.p1.1.m1.10.10.cmml" xref="A2.I1.i1.p1.1.m1.10.10">0.1</cn><cn type="float" id="A2.I1.i1.p1.1.m1.11.11.cmml" xref="A2.I1.i1.p1.1.m1.11.11">0.2</cn><cn type="float" id="A2.I1.i1.p1.1.m1.12.12.cmml" xref="A2.I1.i1.p1.1.m1.12.12">0.3</cn><cn type="float" id="A2.I1.i1.p1.1.m1.13.13.cmml" xref="A2.I1.i1.p1.1.m1.13.13">0.5</cn></set></annotation-xml><annotation encoding="application/x-tex" id="A2.I1.i1.p1.1.m1.13c">\{0.0001,0.0002,0.0005,0.001,0.002,0.005,0.01,0.02,0.05,0.1,0.2,0.3,0.5\}</annotation></semantics></math></p>
</div>
</li>
<li id="A2.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A2.I1.i2.p1" class="ltx_para">
<p id="A2.I1.i2.p1.1" class="ltx_p">output layers in <math id="A2.I1.i2.p1.1.m1.2" class="ltx_Math" alttext="\{1,4\}" display="inline"><semantics id="A2.I1.i2.p1.1.m1.2a"><mrow id="A2.I1.i2.p1.1.m1.2.3.2" xref="A2.I1.i2.p1.1.m1.2.3.1.cmml"><mo stretchy="false" id="A2.I1.i2.p1.1.m1.2.3.2.1" xref="A2.I1.i2.p1.1.m1.2.3.1.cmml">{</mo><mn id="A2.I1.i2.p1.1.m1.1.1" xref="A2.I1.i2.p1.1.m1.1.1.cmml">1</mn><mo id="A2.I1.i2.p1.1.m1.2.3.2.2" xref="A2.I1.i2.p1.1.m1.2.3.1.cmml">,</mo><mn id="A2.I1.i2.p1.1.m1.2.2" xref="A2.I1.i2.p1.1.m1.2.2.cmml">4</mn><mo stretchy="false" id="A2.I1.i2.p1.1.m1.2.3.2.3" xref="A2.I1.i2.p1.1.m1.2.3.1.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.I1.i2.p1.1.m1.2b"><set id="A2.I1.i2.p1.1.m1.2.3.1.cmml" xref="A2.I1.i2.p1.1.m1.2.3.2"><cn type="integer" id="A2.I1.i2.p1.1.m1.1.1.cmml" xref="A2.I1.i2.p1.1.m1.1.1">1</cn><cn type="integer" id="A2.I1.i2.p1.1.m1.2.2.cmml" xref="A2.I1.i2.p1.1.m1.2.2">4</cn></set></annotation-xml><annotation encoding="application/x-tex" id="A2.I1.i2.p1.1.m1.2c">\{1,4\}</annotation></semantics></math></p>
</div>
</li>
<li id="A2.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="A2.I1.i3.p1" class="ltx_para ltx_noindent">
<p id="A2.I1.i3.p1.1" class="ltx_p">concatenate average-pooled tokens in <math id="A2.I1.i3.p1.1.m1.2" class="ltx_Math" alttext="\{yes,no\}" display="inline"><semantics id="A2.I1.i3.p1.1.m1.2a"><mrow id="A2.I1.i3.p1.1.m1.2.2.2" xref="A2.I1.i3.p1.1.m1.2.2.3.cmml"><mo stretchy="false" id="A2.I1.i3.p1.1.m1.2.2.2.3" xref="A2.I1.i3.p1.1.m1.2.2.3.cmml">{</mo><mrow id="A2.I1.i3.p1.1.m1.1.1.1.1" xref="A2.I1.i3.p1.1.m1.1.1.1.1.cmml"><mi id="A2.I1.i3.p1.1.m1.1.1.1.1.2" xref="A2.I1.i3.p1.1.m1.1.1.1.1.2.cmml">y</mi><mo lspace="0em" rspace="0em" id="A2.I1.i3.p1.1.m1.1.1.1.1.1" xref="A2.I1.i3.p1.1.m1.1.1.1.1.1.cmml">​</mo><mi id="A2.I1.i3.p1.1.m1.1.1.1.1.3" xref="A2.I1.i3.p1.1.m1.1.1.1.1.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="A2.I1.i3.p1.1.m1.1.1.1.1.1a" xref="A2.I1.i3.p1.1.m1.1.1.1.1.1.cmml">​</mo><mi id="A2.I1.i3.p1.1.m1.1.1.1.1.4" xref="A2.I1.i3.p1.1.m1.1.1.1.1.4.cmml">s</mi></mrow><mo id="A2.I1.i3.p1.1.m1.2.2.2.4" xref="A2.I1.i3.p1.1.m1.2.2.3.cmml">,</mo><mrow id="A2.I1.i3.p1.1.m1.2.2.2.2" xref="A2.I1.i3.p1.1.m1.2.2.2.2.cmml"><mi id="A2.I1.i3.p1.1.m1.2.2.2.2.2" xref="A2.I1.i3.p1.1.m1.2.2.2.2.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="A2.I1.i3.p1.1.m1.2.2.2.2.1" xref="A2.I1.i3.p1.1.m1.2.2.2.2.1.cmml">​</mo><mi id="A2.I1.i3.p1.1.m1.2.2.2.2.3" xref="A2.I1.i3.p1.1.m1.2.2.2.2.3.cmml">o</mi></mrow><mo stretchy="false" id="A2.I1.i3.p1.1.m1.2.2.2.5" xref="A2.I1.i3.p1.1.m1.2.2.3.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.I1.i3.p1.1.m1.2b"><set id="A2.I1.i3.p1.1.m1.2.2.3.cmml" xref="A2.I1.i3.p1.1.m1.2.2.2"><apply id="A2.I1.i3.p1.1.m1.1.1.1.1.cmml" xref="A2.I1.i3.p1.1.m1.1.1.1.1"><times id="A2.I1.i3.p1.1.m1.1.1.1.1.1.cmml" xref="A2.I1.i3.p1.1.m1.1.1.1.1.1"></times><ci id="A2.I1.i3.p1.1.m1.1.1.1.1.2.cmml" xref="A2.I1.i3.p1.1.m1.1.1.1.1.2">𝑦</ci><ci id="A2.I1.i3.p1.1.m1.1.1.1.1.3.cmml" xref="A2.I1.i3.p1.1.m1.1.1.1.1.3">𝑒</ci><ci id="A2.I1.i3.p1.1.m1.1.1.1.1.4.cmml" xref="A2.I1.i3.p1.1.m1.1.1.1.1.4">𝑠</ci></apply><apply id="A2.I1.i3.p1.1.m1.2.2.2.2.cmml" xref="A2.I1.i3.p1.1.m1.2.2.2.2"><times id="A2.I1.i3.p1.1.m1.2.2.2.2.1.cmml" xref="A2.I1.i3.p1.1.m1.2.2.2.2.1"></times><ci id="A2.I1.i3.p1.1.m1.2.2.2.2.2.cmml" xref="A2.I1.i3.p1.1.m1.2.2.2.2.2">𝑛</ci><ci id="A2.I1.i3.p1.1.m1.2.2.2.2.3.cmml" xref="A2.I1.i3.p1.1.m1.2.2.2.2.3">𝑜</ci></apply></set></annotation-xml><annotation encoding="application/x-tex" id="A2.I1.i3.p1.1.m1.2c">\{yes,no\}</annotation></semantics></math></p>
</div>
</li>
</ul>
<p id="A2.SS3.p1.2" class="ltx_p">We then report the highest accuracy value obtained on the validation set as is common practice.
Note that this grid search is not expensive, because at each iteration we perform inference on the backbone only once, then feed the output to all linear classifiers (each performing a single matrix multiplication).</p>
</div>
</section>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>List of Datasets used</h2>

<div id="A3.p1" class="ltx_para ltx_noindent">
<p id="A3.p1.1" class="ltx_p">We show in Table <a href="#A3" title="Appendix C List of Datasets used ‣ DINOv2: Learning Robust Visual Features without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a> the list of benchmarks and datasets used and their purposes.</p>
</div>
<figure id="A3.tab1" class="ltx_table">
<table id="A3.tab1.1" class="ltx_tabular ltx_align_middle">
<tr id="A3.tab1.1.1" class="ltx_tr">
<td id="A3.tab1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="A3.tab1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.1.1.1.1" class="ltx_p" style="width:42.7pt;">Dataset</span>
</span>
</td>
<td id="A3.tab1.1.1.2" class="ltx_td ltx_nopad_r ltx_align_left ltx_border_tt">
<span id="A3.tab1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.1.2.1.1" class="ltx_p" style="width:42.7pt;">Pretraining</span>
</span>
</td>
<td id="A3.tab1.1.1.3" class="ltx_td ltx_border_tt"></td>
<td id="A3.tab1.1.1.4" class="ltx_td ltx_border_tt"></td>
<td id="A3.tab1.1.1.5" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt"></td>
<td id="A3.tab1.1.1.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt"></td>
</tr>
<tr id="A3.tab1.1.2" class="ltx_tr">
<td id="A3.tab1.1.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.2.1.1.1" class="ltx_p" style="width:71.1pt;">(as is)</span>
</span>
</td>
<td id="A3.tab1.1.2.2" class="ltx_td ltx_nopad_r ltx_align_left">
<span id="A3.tab1.1.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.2.2.1.1" class="ltx_p" style="width:42.7pt;">Retrieving</span>
</span>
</td>
<td id="A3.tab1.1.2.3" class="ltx_td"></td>
<td id="A3.tab1.1.2.4" class="ltx_td"></td>
<td id="A3.tab1.1.2.5" class="ltx_td ltx_align_justify ltx_align_top"></td>
<td id="A3.tab1.1.2.6" class="ltx_td ltx_align_justify ltx_align_top"></td>
</tr>
<tr id="A3.tab1.1.3" class="ltx_tr">
<td id="A3.tab1.1.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.3.1.1.1" class="ltx_p" style="width:71.1pt;">pretraining</span>
</span>
</td>
<td id="A3.tab1.1.3.2" class="ltx_td"></td>
<td id="A3.tab1.1.3.3" class="ltx_td"></td>
<td id="A3.tab1.1.3.4" class="ltx_td"></td>
<td id="A3.tab1.1.3.5" class="ltx_td ltx_align_justify ltx_align_top"></td>
<td id="A3.tab1.1.3.6" class="ltx_td ltx_align_justify ltx_align_top"></td>
</tr>
<tr id="A3.tab1.1.4" class="ltx_tr">
<td id="A3.tab1.1.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.4.1.1.1" class="ltx_p" style="width:71.1pt;">data</span>
</span>
</td>
<td id="A3.tab1.1.4.2" class="ltx_td ltx_align_left">
<span id="A3.tab1.1.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.4.2.1.1" class="ltx_p" style="width:28.5pt;">Eval.</span>
</span>
</td>
<td id="A3.tab1.1.4.3" class="ltx_td ltx_align_center">Task</td>
<td id="A3.tab1.1.4.4" class="ltx_td ltx_align_center">Citation</td>
<td id="A3.tab1.1.4.5" class="ltx_td ltx_align_justify ltx_align_top"></td>
<td id="A3.tab1.1.4.6" class="ltx_td ltx_align_justify ltx_align_top"></td>
</tr>
<tr id="A3.tab1.1.5" class="ltx_tr">
<td id="A3.tab1.1.5.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.5.1.1.1" class="ltx_p" style="width:71.1pt;">ImageNet-1k</span>
</span>
</td>
<td id="A3.tab1.1.5.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.5.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.5.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.5.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.5.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.5.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.5.5" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.5.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.5.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.5.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.5.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.5.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Russakovsky et al., <a href="#bib.bib99" title="" class="ltx_ref">2015</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.6" class="ltx_tr">
<td id="A3.tab1.1.6.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.6.1.1.1" class="ltx_p" style="width:71.1pt;">ImageNet-22k</span>
</span>
</td>
<td id="A3.tab1.1.6.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.6.2.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.6.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.6.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.6.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.6.4.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.6.5" class="ltx_td ltx_align_top"></td>
<td id="A3.tab1.1.6.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.6.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.6.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Deng et al., <a href="#bib.bib34" title="" class="ltx_ref">2009</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.7" class="ltx_tr">
<td id="A3.tab1.1.7.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.7.1.1.1" class="ltx_p" style="width:71.1pt;">ImageNet-V2</span>
</span>
</td>
<td id="A3.tab1.1.7.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.7.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.7.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.7.3.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.7.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.7.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.7.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.7.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.7.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.7.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.7.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.7.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Recht et al., <a href="#bib.bib96" title="" class="ltx_ref">2019</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.8" class="ltx_tr">
<td id="A3.tab1.1.8.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.8.1.1.1" class="ltx_p" style="width:71.1pt;">ImageNet-ReaL</span>
</span>
</td>
<td id="A3.tab1.1.8.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.8.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.8.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.8.3.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.8.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.8.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.8.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.8.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.8.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.8.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.8.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.8.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Beyer et al., <a href="#bib.bib9" title="" class="ltx_ref">2020</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.9" class="ltx_tr">
<td id="A3.tab1.1.9.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.9.1.1.1" class="ltx_p" style="width:71.1pt;">ImageNet-A</span>
</span>
</td>
<td id="A3.tab1.1.9.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.9.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.9.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.9.3.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.9.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.9.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.9.5" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.9.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.9.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.9.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.9.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.9.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib62" title="" class="ltx_ref">2021b</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.10" class="ltx_tr">
<td id="A3.tab1.1.10.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.10.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.10.1.1.1" class="ltx_p" style="width:71.1pt;">ImageNet-C</span>
</span>
</td>
<td id="A3.tab1.1.10.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.10.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.10.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.10.3.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.10.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.10.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.10.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.10.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.10.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.10.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.10.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.10.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks &amp; Dietterich, <a href="#bib.bib60" title="" class="ltx_ref">2019</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.11" class="ltx_tr">
<td id="A3.tab1.1.11.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.11.1.1.1" class="ltx_p" style="width:71.1pt;">ImageNet-R</span>
</span>
</td>
<td id="A3.tab1.1.11.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.11.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.11.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.11.3.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.11.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.11.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.11.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.11.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.11.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.11.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.11.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.11.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Hendrycks et al., <a href="#bib.bib61" title="" class="ltx_ref">2021a</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.12" class="ltx_tr">
<td id="A3.tab1.1.12.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.12.1.1.1" class="ltx_p" style="width:71.1pt;">ImageNet-Sk.</span>
</span>
</td>
<td id="A3.tab1.1.12.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.12.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.12.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.12.3.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.12.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.12.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.12.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.12.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.12.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.12.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.12.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.12.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib120" title="" class="ltx_ref">2019</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.13" class="ltx_tr">
<td id="A3.tab1.1.13.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.13.1.1.1" class="ltx_p" style="width:71.1pt;">Food-101</span>
</span>
</td>
<td id="A3.tab1.1.13.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.13.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.13.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.13.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.13.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.13.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.13.5" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.13.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.13.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.13.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.13.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.13.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Bossard et al., <a href="#bib.bib14" title="" class="ltx_ref">2014</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.14" class="ltx_tr">
<td id="A3.tab1.1.14.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.14.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.14.1.1.1" class="ltx_p" style="width:71.1pt;">CIFAR-10</span>
</span>
</td>
<td id="A3.tab1.1.14.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.14.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.14.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.14.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.14.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.14.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.14.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.14.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.14.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.14.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.14.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.14.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Krizhevsky et al., <a href="#bib.bib73" title="" class="ltx_ref">2009</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.15" class="ltx_tr">
<td id="A3.tab1.1.15.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.15.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.15.1.1.1" class="ltx_p" style="width:71.1pt;">CIFAR-100</span>
</span>
</td>
<td id="A3.tab1.1.15.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.15.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.15.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.15.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.15.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.15.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.15.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.15.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.15.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.15.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.15.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.15.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Krizhevsky et al., <a href="#bib.bib73" title="" class="ltx_ref">2009</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.16" class="ltx_tr">
<td id="A3.tab1.1.16.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.16.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.16.1.1.1" class="ltx_p" style="width:71.1pt;">SUN397</span>
</span>
</td>
<td id="A3.tab1.1.16.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.16.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.16.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.16.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.16.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.16.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.16.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.16.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.16.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.16.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.16.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.16.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Xiao et al., <a href="#bib.bib127" title="" class="ltx_ref">2010</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.17" class="ltx_tr">
<td id="A3.tab1.1.17.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.17.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.17.1.1.1" class="ltx_p" style="width:71.1pt;">StanfordCars</span>
</span>
</td>
<td id="A3.tab1.1.17.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.17.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.17.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.17.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.17.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.17.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.17.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.17.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.17.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.17.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.17.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.17.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Krause et al., <a href="#bib.bib71" title="" class="ltx_ref">2013</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.18" class="ltx_tr">
<td id="A3.tab1.1.18.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.18.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.18.1.1.1" class="ltx_p" style="width:71.1pt;">FGVC-Aircraft</span>
</span>
</td>
<td id="A3.tab1.1.18.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.18.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.18.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.18.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.18.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.18.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.18.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.18.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.18.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.18.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.18.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.18.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Maji et al., <a href="#bib.bib80" title="" class="ltx_ref">2013</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.19" class="ltx_tr">
<td id="A3.tab1.1.19.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.19.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.19.1.1.1" class="ltx_p" style="width:71.1pt;">VOC 2007</span>
</span>
</td>
<td id="A3.tab1.1.19.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.19.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.19.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.19.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.19.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.19.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.19.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.19.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.19.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.19.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.19.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.19.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Everingham et al., <a href="#bib.bib42" title="" class="ltx_ref">2010</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.20" class="ltx_tr">
<td id="A3.tab1.1.20.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.20.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.20.1.1.1" class="ltx_p" style="width:71.1pt;">DTD</span>
</span>
</td>
<td id="A3.tab1.1.20.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.20.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.20.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.20.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.20.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.20.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.20.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.20.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.20.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.20.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.20.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.20.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Cimpoi et al., <a href="#bib.bib29" title="" class="ltx_ref">2014</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.21" class="ltx_tr">
<td id="A3.tab1.1.21.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.21.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.21.1.1.1" class="ltx_p" style="width:71.1pt;">Oxford Pets</span>
</span>
</td>
<td id="A3.tab1.1.21.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.21.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.21.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.21.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.21.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.21.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.21.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.21.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.21.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.21.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.21.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.21.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Parkhi et al., <a href="#bib.bib85" title="" class="ltx_ref">2012</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.22" class="ltx_tr">
<td id="A3.tab1.1.22.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.22.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.22.1.1.1" class="ltx_p" style="width:71.1pt;">Caltech101</span>
</span>
</td>
<td id="A3.tab1.1.22.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.22.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.22.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.22.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.22.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.22.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.22.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.22.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.22.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.22.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.22.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.22.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Fei-Fei et al., <a href="#bib.bib44" title="" class="ltx_ref">2004</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.23" class="ltx_tr">
<td id="A3.tab1.1.23.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.23.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.23.1.1.1" class="ltx_p" style="width:71.1pt;">Flowers</span>
</span>
</td>
<td id="A3.tab1.1.23.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.23.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.23.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.23.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.23.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.23.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.23.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.23.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.23.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.23.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.23.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.23.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Nilsback &amp; Zisserman, <a href="#bib.bib82" title="" class="ltx_ref">2008</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.24" class="ltx_tr">
<td id="A3.tab1.1.24.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.24.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.24.1.1.1" class="ltx_p" style="width:71.1pt;">CUB200</span>
</span>
</td>
<td id="A3.tab1.1.24.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.24.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.24.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.24.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.24.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.24.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.24.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.24.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.24.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.24.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.24.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.24.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Welinder et al., <a href="#bib.bib123" title="" class="ltx_ref">2010</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.25" class="ltx_tr">
<td id="A3.tab1.1.25.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.25.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.25.1.1.1" class="ltx_p" style="width:71.1pt;">iNaturalist 2018</span>
</span>
</td>
<td id="A3.tab1.1.25.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.25.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.25.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.25.3.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.25.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.25.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.25.5" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.25.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.25.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.25.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.25.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.25.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Van Horn et al., <a href="#bib.bib117" title="" class="ltx_ref">2018</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.26" class="ltx_tr">
<td id="A3.tab1.1.26.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.26.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.26.1.1.1" class="ltx_p" style="width:71.1pt;">iNaturalist 2021</span>
</span>
</td>
<td id="A3.tab1.1.26.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.26.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.26.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.26.3.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.26.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.26.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.26.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.26.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.26.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.26.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.26.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.26.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Van Horn et al., <a href="#bib.bib118" title="" class="ltx_ref">2021</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.27" class="ltx_tr">
<td id="A3.tab1.1.27.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.27.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.27.1.1.1" class="ltx_p" style="width:71.1pt;">Places-205</span>
</span>
</td>
<td id="A3.tab1.1.27.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.27.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.27.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.27.3.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.27.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.27.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.27.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.27.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.27.5.1.1" class="ltx_p" style="width:56.9pt;">Classif.</span>
</span>
</td>
<td id="A3.tab1.1.27.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.27.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.27.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Zhou et al., <a href="#bib.bib134" title="" class="ltx_ref">2014</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.28" class="ltx_tr">
<td id="A3.tab1.1.28.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.28.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.28.1.1.1" class="ltx_p" style="width:71.1pt;">UCF101</span>
</span>
</td>
<td id="A3.tab1.1.28.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.28.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.28.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.28.3.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.28.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.28.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.28.5" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.28.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.28.5.1.1" class="ltx_p" style="width:56.9pt;">Video</span>
</span>
</td>
<td id="A3.tab1.1.28.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.28.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.28.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Soomro et al., <a href="#bib.bib107" title="" class="ltx_ref">2012</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.29" class="ltx_tr">
<td id="A3.tab1.1.29.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.29.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.29.1.1.1" class="ltx_p" style="width:71.1pt;">Kinetics-400</span>
</span>
</td>
<td id="A3.tab1.1.29.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.29.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.29.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.29.3.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.29.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.29.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.29.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.29.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.29.5.1.1" class="ltx_p" style="width:56.9pt;">Video</span>
</span>
</td>
<td id="A3.tab1.1.29.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.29.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.29.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Kay et al., <a href="#bib.bib70" title="" class="ltx_ref">2017</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.30" class="ltx_tr">
<td id="A3.tab1.1.30.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.30.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.30.1.1.1" class="ltx_p" style="width:71.1pt;">SSv2</span>
</span>
</td>
<td id="A3.tab1.1.30.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.30.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.30.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.30.3.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.30.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.30.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.30.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.30.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.30.5.1.1" class="ltx_p" style="width:56.9pt;">Video</span>
</span>
</td>
<td id="A3.tab1.1.30.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.30.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.30.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Goyal et al., <a href="#bib.bib52" title="" class="ltx_ref">2017</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.31" class="ltx_tr">
<td id="A3.tab1.1.31.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.31.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.31.1.1.1" class="ltx_p" style="width:71.1pt;">GLD v2</span>
</span>
</td>
<td id="A3.tab1.1.31.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.31.2.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.31.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.31.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.31.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.31.4.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.31.5" class="ltx_td ltx_align_top ltx_border_t"></td>
<td id="A3.tab1.1.31.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.31.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.31.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Weyand et al., <a href="#bib.bib125" title="" class="ltx_ref">2020</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.32" class="ltx_tr">
<td id="A3.tab1.1.32.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.32.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.32.1.1.1" class="ltx_p" style="width:71.1pt;">R-Paris</span>
</span>
</td>
<td id="A3.tab1.1.32.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.32.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.32.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.32.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.32.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.32.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.32.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.32.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.32.5.1.1" class="ltx_p" style="width:56.9pt;">Retrieval</span>
</span>
</td>
<td id="A3.tab1.1.32.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.32.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.32.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Radenović et al., <a href="#bib.bib89" title="" class="ltx_ref">2018a</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.33" class="ltx_tr">
<td id="A3.tab1.1.33.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.33.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.33.1.1.1" class="ltx_p" style="width:71.1pt;">R-Oxford</span>
</span>
</td>
<td id="A3.tab1.1.33.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.33.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.33.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.33.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.33.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.33.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.33.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.33.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.33.5.1.1" class="ltx_p" style="width:56.9pt;">Retrieval</span>
</span>
</td>
<td id="A3.tab1.1.33.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.33.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.33.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Radenović et al., <a href="#bib.bib89" title="" class="ltx_ref">2018a</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.34" class="ltx_tr">
<td id="A3.tab1.1.34.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.34.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.34.1.1.1" class="ltx_p" style="width:71.1pt;">Met</span>
</span>
</td>
<td id="A3.tab1.1.34.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.34.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.34.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.34.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.34.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.34.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.34.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.34.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.34.5.1.1" class="ltx_p" style="width:56.9pt;">Retrieval</span>
</span>
</td>
<td id="A3.tab1.1.34.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.34.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.34.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Ypsilantis et al., <a href="#bib.bib131" title="" class="ltx_ref">2021</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.35" class="ltx_tr">
<td id="A3.tab1.1.35.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.35.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.35.1.1.1" class="ltx_p" style="width:71.1pt;">Amstertime</span>
</span>
</td>
<td id="A3.tab1.1.35.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.35.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.35.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.35.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.35.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.35.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.35.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.35.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.35.5.1.1" class="ltx_p" style="width:56.9pt;">Retrieval</span>
</span>
</td>
<td id="A3.tab1.1.35.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.35.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.35.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Yildiz et al., <a href="#bib.bib130" title="" class="ltx_ref">2022</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.36" class="ltx_tr">
<td id="A3.tab1.1.36.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.36.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.36.1.1.1" class="ltx_p" style="width:71.1pt;">ADE20k</span>
</span>
</td>
<td id="A3.tab1.1.36.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.36.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.36.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.36.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.36.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.36.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.36.5" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.36.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.36.5.1.1" class="ltx_p" style="width:56.9pt;">Seg.</span>
</span>
</td>
<td id="A3.tab1.1.36.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.36.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.36.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Zhou et al., <a href="#bib.bib135" title="" class="ltx_ref">2017</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.37" class="ltx_tr">
<td id="A3.tab1.1.37.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.37.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.37.1.1.1" class="ltx_p" style="width:71.1pt;">Cityscapes</span>
</span>
</td>
<td id="A3.tab1.1.37.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.37.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.37.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.37.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.37.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.37.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.37.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.37.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.37.5.1.1" class="ltx_p" style="width:56.9pt;">Seg.</span>
</span>
</td>
<td id="A3.tab1.1.37.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.37.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.37.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Cordts et al., <a href="#bib.bib30" title="" class="ltx_ref">2016</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.38" class="ltx_tr">
<td id="A3.tab1.1.38.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.38.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.38.1.1.1" class="ltx_p" style="width:71.1pt;">VOC 2012</span>
</span>
</td>
<td id="A3.tab1.1.38.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.38.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.38.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.38.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.38.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.38.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.38.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.38.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.38.5.1.1" class="ltx_p" style="width:56.9pt;">Seg.</span>
</span>
</td>
<td id="A3.tab1.1.38.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.38.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.38.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Everingham et al., <a href="#bib.bib42" title="" class="ltx_ref">2010</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.39" class="ltx_tr">
<td id="A3.tab1.1.39.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.39.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.39.1.1.1" class="ltx_p" style="width:71.1pt;">Mapillary SLS</span>
</span>
</td>
<td id="A3.tab1.1.39.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.39.2.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.39.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.39.3.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.39.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.39.4.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.39.5" class="ltx_td ltx_align_top ltx_border_t"></td>
<td id="A3.tab1.1.39.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.39.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.39.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Warburg et al., <a href="#bib.bib121" title="" class="ltx_ref">2020</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.40" class="ltx_tr">
<td id="A3.tab1.1.40.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.40.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.40.1.1.1" class="ltx_p" style="width:71.1pt;">NYU-Depth V2</span>
</span>
</td>
<td id="A3.tab1.1.40.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.40.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.40.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.40.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.40.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.40.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.40.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.40.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.40.5.1.1" class="ltx_p" style="width:56.9pt;">Depth</span>
</span>
</td>
<td id="A3.tab1.1.40.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.40.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.40.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Silberman et al., <a href="#bib.bib104" title="" class="ltx_ref">2012</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.41" class="ltx_tr">
<td id="A3.tab1.1.41.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.41.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.41.1.1.1" class="ltx_p" style="width:71.1pt;">KITTI</span>
</span>
</td>
<td id="A3.tab1.1.41.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.41.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.41.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.41.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.41.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.41.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.41.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.41.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.41.5.1.1" class="ltx_p" style="width:56.9pt;">Depth</span>
</span>
</td>
<td id="A3.tab1.1.41.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.41.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.41.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Geiger et al., <a href="#bib.bib45" title="" class="ltx_ref">2013</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.42" class="ltx_tr">
<td id="A3.tab1.1.42.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.42.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.42.1.1.1" class="ltx_p" style="width:71.1pt;">SUN-RGBD</span>
</span>
</td>
<td id="A3.tab1.1.42.2" class="ltx_td ltx_align_center"><span id="A3.tab1.1.42.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.42.3" class="ltx_td ltx_align_center"><span id="A3.tab1.1.42.3.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.42.4" class="ltx_td ltx_align_center"><span id="A3.tab1.1.42.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.42.5" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.42.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.42.5.1.1" class="ltx_p" style="width:56.9pt;">Depth</span>
</span>
</td>
<td id="A3.tab1.1.42.6" class="ltx_td ltx_align_justify ltx_align_top">
<span id="A3.tab1.1.42.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.42.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Song et al., <a href="#bib.bib106" title="" class="ltx_ref">2015</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.43" class="ltx_tr">
<td id="A3.tab1.1.43.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.43.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.43.1.1.1" class="ltx_p" style="width:71.1pt;">DollarStreet</span>
</span>
</td>
<td id="A3.tab1.1.43.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.43.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.43.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.43.3.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.43.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A3.tab1.1.43.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.43.5" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.43.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.43.5.1.1" class="ltx_p" style="width:56.9pt;">Fairness</span>
</span>
</td>
<td id="A3.tab1.1.43.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="A3.tab1.1.43.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.43.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(De Vries et al., <a href="#bib.bib32" title="" class="ltx_ref">2019</a>)</cite></span>
</span>
</td>
</tr>
<tr id="A3.tab1.1.44" class="ltx_tr">
<td id="A3.tab1.1.44.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A3.tab1.1.44.1.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.44.1.1.1" class="ltx_p" style="width:71.1pt;">Casual Conv.</span>
</span>
</td>
<td id="A3.tab1.1.44.2" class="ltx_td ltx_align_center ltx_border_bb"><span id="A3.tab1.1.44.2.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.44.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="A3.tab1.1.44.3.1" class="ltx_text" style="color:#8C4040;">✗</span></td>
<td id="A3.tab1.1.44.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="A3.tab1.1.44.4.1" class="ltx_text" style="color:#408C40;">✓</span></td>
<td id="A3.tab1.1.44.5" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A3.tab1.1.44.5.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.44.5.1.1" class="ltx_p" style="width:56.9pt;">Fairness</span>
</span>
</td>
<td id="A3.tab1.1.44.6" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="A3.tab1.1.44.6.1" class="ltx_inline-block ltx_align_top">
<span id="A3.tab1.1.44.6.1.1" class="ltx_p" style="width:142.3pt;"> <cite class="ltx_cite ltx_citemacro_citep">(Hazirbas et al., <a href="#bib.bib56" title="" class="ltx_ref">2021</a>)</cite></span>
</span>
</td>
</tr>
</table>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="A3.tab1.3.1.1" class="ltx_text" style="font-size:90%;">Table 18</span>: </span><span id="A3.tab1.4.2" class="ltx_text" style="font-size:90%;">
<span id="A3.tab1.4.2.1" class="ltx_text ltx_font_bold">List of datasets used.</span>
</span></figcaption>
</figure>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2304.07192" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2304.07193" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2304.07193">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2304.07193" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2304.07194" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Thu Feb 29 14:16:12 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2304.07193
The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision.
These models could greatly simplify the use of images in any system by producing general-purpose visual features, i.e., features that work across image distributions and tasks without finetuning.
This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources.
We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size.
Most of the technical contributions aim at accelerating and stabilizing the training at scale.
In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature.
In terms of models, we train a ViT modelwith 1B parameters and distill it into a series of smaller models that surpass the best available general-purpose features, OpenCLIPon most of the benchmarks at image and pixel levels.
abstract
DINOv2: Learning Robust Visual Featureswithout Supervision
Abstract
2304.07193
1IntroductionLearning task-agnostic pretrained representations have become the standard in Natural Language Processing (NLP) .
One can use these features “as they are”, i.e., without fine-tuning, and achieve performances on downstream tasks that are significantly better than those produced by task-specific models.
This success has been fueled by pretraining on large quantities of raw text using pretext objectives, such as language modelingor word vectors, that require no supervision.Following this paradigm shift in NLP, we expect similar “foundation” models to appear in computer vision.
These models should generate visual features that work out of the box on any task, both at the image level, e.g., image classification, and pixel level, e.g., segmentation.
Most promising efforts towards these foundation models focus on text-guided pretraining, i.e., using a form of textual supervision to guide the training of the features.
This form of text-guided pretraining limits the information that can be retained about the image since captions only approximate the rich information in images, and complex pixel-level information may not surface with this supervision.
Furthermore, these image encoders require aligned text-image corpora and hence, do not offer the flexibility of their text counterparts, that is, to learn from raw data alone.Figure 1:Visualization of the first PCA components.We compute a PCA between the patches of the images from the same column (a, b, c and d)  and show their first 3 components.
Each component is matched to a different color channel.
Same parts are matched between related images despite changes of pose, style or even objects.
Background is removed by thresholding the first PCA component.An alternative to text-guided pretraining is self-supervised learningwhere features are learned from images alone.
These approaches are conceptually closer to pretext tasks such as language modeling and can capture information at the image and pixel level.
Additionally, the features output by self-supervised models have been shown to exhibit various useful properties, and have enabled enabled a wide variety of applications.
However, despite their potential to learn general-purpose features, most of the advances in self-supervised learning were made in the context of pretraining on a small curated dataset, ImageNet-1k.
Some efforts on scaling these approaches beyond ImageNet-1k have been attempted, but they focused on uncurated datasets, which typically lead to a significant drop in the quality of the features.
This is explained by the lack of control over the data quality and diversity, which are essential to produce good features.In this work, we explore if self-supervised learning has the potential to learn general-purpose visual features if pretrained on a large quantity of curated data.
We revisit existing discriminative self-supervised approaches that learn features at both the image and patch level, such as iBOT, and we reconsider some of their design choices under the lens of a larger dataset.
Most of our technical contributions are tailored toward stabilizing and accelerating discriminative self-supervised learning when scaling in model and data sizes.
These improvements make our approach around 2faster and require 3less memory than similar discriminative self-supervised methods, allowing us to leverage longer training with larger batch sizes.Regarding pretraining data, we have built an automatic pipeline to filter and rebalance datasets from an extensive collection of uncurated images.
This pipeline is inspired by pipelines used in NLP, where data similarities are used instead of external metadata and do not require manual annotation.
A major difficulty when dealing with images in the wild is to rebalance concepts and avoid overfitting on a few dominant modes.
In this work, a naive clustering approach works reasonably well to resolve this issue.
We gathered a small but diverse corpus of 142M images to validate our approach.Finally, we provide a variety of pretrained visual models, called DINOv2, trained with different Vision Transformers (ViT) architectures on our data.
We release all the models and the code to retrain DINOv2 on any data.
We validate the quality of DINOv2 on various computer vision benchmarks at both image and pixel levels as we scale them, as summarized in Fig.2.
We conclude that self-supervised pretraining alone is a good candidate for learning transferable frozen features that are competitive with the best openly available weakly-supervised models.
introduction
DINOv2: Learning Robust Visual Featureswithout Supervision
1Introduction
2304.07193
2Related WorkIntra-image self-supervised training.A first family of self-supervised methods focuses on pretext tasks built from the image, i.e., extracting a signal from the image to be predicted from the rest of the image.
This idea has become prevalent with the work of, where they train by predicting the context of a given patch.
Many other pretext tasks were introduced based on, for example, re-colorizing images, predicting transformations, inpaintingor patch re-ordering.
Recently, the emergence of patch-based architectures, like ViTs, has led to a revisit of inpainting for pre-training, potentially in feature space.
Of particular interest,show that a masked auto-encoder (MAE)  learns features that provide substantial improvements when finetuned on downstream tasks.
This property of MAEs has been further validated on video, audio, and across other modalities.
However, their features require supervised finetuning, while our features perform well out of the box.Figure 2:Evolution of performance when scaling in parameters.We show performance on eight types of vision tasks, as presented in Sec.7, and average metrics with each type.
Features are extracted from our self-supervised encoders, DINOv2 (dark blue) , and we compare them with self-supervised methods (pale orange) , as well as weakly-supervised methods (dark pink) .
We report the best-performing weakly-supervised model’s performance as a dashed horizontal line.
Our family of models drastically improves over the previous state of the art in self-supervised learning and reaches performance comparable with weakly-supervised features.
See Sec.7for a detailed analysis.Discriminative self-supervised learning.The second line of work, closer to ours, is using discriminative signals between images or groups of images to learn features.
This family of methods has roots in early deep learning workbut became popular with the emergence of instance classification methods.
Several improvements were made based either on instance-level objectivesor clustering.
These methods provide performant frozen features on standard benchmarks like ImageNet, but they are hard to scale to larger model sizes.
In this work, we revisit the training of these approaches in the context of large pretraining datasets and models.
In particular, we build on top ofthat we find particularly suited for scaling.Scaling self-supervised pretraining.A growing body of work has focused on the scaling abilities of self-supervised learning in terms of data and model size.
Most of these works use large quantities of uncurated data to train models without supervision.
They show evidence that discriminative methods scale with data, but because of the poor quality of the pretraining data, most of the results are obtained by finetuning the features.
Of particular interest,have also shown that these methods benefit from scaling in model size given enough pretrained data.
This line of work questions the ability of self-supervised methods to work on any data while we focus on producing the best pretrained encoders.Automatic data curation.Our dataset construction borrows from the image retrieval community.
In particular, the use of retrieval to augment the training set has been studied in the context of semi-supervised learning.
Similarly, others have used hashtags or other metadataor pretrained vision encodersto filter uncurated datasets.
Unlike these works, we use no pretrained encoders, metadata nor supervision to filter images and leverage visual similarity between images.
Our approach is inspired by text curation pipelines, where a language model is trained on Wikipedia to score texts extracted from an uncurated source.
body
DINOv2: Learning Robust Visual Featureswithout Supervision
2Related Work
2304.07193
3Data ProcessingWe assemble our curated LVD-142M dataset by retrieving, from a large pool of uncurated data, images that are close to those in several curated datasets.
We describe below the main components in our data pipeline including the curated/uncurated data sources, the image deduplication step and the retrieval system.
Our pipeline does not require any metadata or text and directly works with images, as shown in Fig.3.
We refer the reader to appendixAfor more details on our approach.Figure 3:Overview of our data processing pipeline.Images from curated and uncurated data sources are first mapped to embeddings.
Uncurated images are then deduplicated before being matched to curated images.
The resulting combination augments the initial dataset through a self-supervised retrieval system.Data sources.Our selection of curated datasets is detailed in the appendix (Table15)  and contains ImageNet-22k, the train split of ImageNet-1k, Google Landmarks and several fine-grained datasets.
For the uncurated data source, we collect a raw unfiltered dataset of images from a publicly available repository of crawled web data.
From each web page in the repository, we extract URL links of images from<img>tags.
We discard URLs that are unsafe or restricted by domains, and post-process the downloaded images (PCA hash deduplication, NSFW filtering, and blurring identifiable faces) .
This results in 1.2B unique images.Deduplication.We apply the copy detection pipeline ofto the uncurated data and remove near-duplicate images.
This reduces redundancy and increases diversity among images.
We also remove near-duplicates of images contained in the test or validation set of any benchmark used in this work.Self-supervised image retrieval.We build our curated pretraining dataset by retrieving images from our uncurated data source that are close to images in our curated sources.
In order to do this, we first compute an image embedding using a self-supervised ViT-H/16 network pretrained on ImageNet-22k, and use cosine-similarity as a distance measure between images.
Then, we perform k-means clustering of the uncurated data.
Given a query dataset for retrieval, if it is large enough we retrieve(typically 4)  nearest neighbors for each query image.
If it is small, we sampleimages from the cluster corresponding to each query image.
Although visual inspection seemed to indicate good retrieval quality formuch larger than 4, this leads to more collisions (images that are nearest-neighbor retrievals of multiple queries) . We chooseas it provides a good tradeoff in that sense.Implementation Details.The deduplication and retrieval stages of our pipeline rely on the Faiss libraryto efficiently index and compute batch searches of nearest embeddings.
In particular, we heavily leverage its support for GPU-accelerated indices, using inverted file indices with product quantization codes.
The whole processing is distributed on a compute cluster of 20 nodes equipped with 8 V100-32GB GPUs and takes less than two days to produce the LVD-142M dataset.
body
DINOv2: Learning Robust Visual Featureswithout Supervision
3Data Processing
2304.07193
4Discriminative Self-supervised Pre-trainingWe learn our features with a discriminative self-supervised method that can be seen as a combination of DINO and iBOT losses with the centering of SwAV.
We also add a regularizer to spread features and a short high-resolution training phase.
We rapidly introduce each of these approaches, but more details can be found in the related papers, or in our open-sourced code.•Image-level objective.We consider the cross-entropy loss between the features extracted from a student and a teacher network.
Both features are coming from the class token of a ViT, obtained from different crops of the same image.

We pass the student class token through the student DINO head.
This head is an MLP model outputting a vector of scores, that we call "prototype scores".
We then apply a softmax to obtain.
Similarly, we apply the teacher DINO head to the teacher class token to obtain teacher prototype scores.
We then apply a softmax followed by a centering with moving average (or a Sinkhorn-Knopp centering as detailed thereafter)  to obtain.
The DINO loss term corresponds to:We learn the parameters of the student and build the teacher head with an exponential moving average of past iterates.•Patch-level objective.We randomly mask some of the input patches given to the student, but not to the teacher.

We then apply the student iBOT head to the student mask tokens.
Similarly, we apply the teacher iBOT head to the (visible)  teacher patch tokens corresponding to the ones masked in the student.
We then apply the softmax and centering steps as above, and obtain the iBOT loss term:,
whereare patch indices for masked tokens.
Similarly to above, we learn the parameters of the student, and build the teacher head through exponential moving average.•Untying head weights between both objectives.Both the DINO and the iBOT loss use a learnable MLP projection head.
It is applied to the output tokens and the loss is compute atop.
In, an ablation study shows that sharing parameters between the DINO and iBOT heads leads to better performance.
At scale, we observed that the opposite is true, and we therefore use two separate heads in all our experiments.•Sinkhorn-Knopp centering.recommend to replace the teacher softmax-centering step of DINO and iBot by the Sinkhorn-Knopp (SK)  batch normalization of SwAV.
We run the Sinkhorn-Knopp algorithm steps for 3 iterations.
For the student, we apply the softmax normalization.•KoLeo regularizer.The KoLeo regularizer derives from the Kozachenko-Leonenko differential entropy estimator (see)  and encourages a uniform span of the features within a batch.
Given a set ofvectors, it is defined aswhereis the minimum distance betweenand any other point within the batch.
We also-normalize the features before computing this regularizer.•Adapting the resolution.Increasing image resolution is key to pixel-level downstream tasks such as segmentation or detection, where small objects disappear at low resolutions.
However, training at high resolution is time and memory demanding, and instead, we increase the resolution of images toduring a short period at the end of pretraining. This is also similar to UniViT training fromand FlexiViT training from.
body
DINOv2: Learning Robust Visual Featureswithout Supervision
4Discriminative Self-supervised Pre-training
2304.07193
5Efficient implementationWe consider several improvements to train models at a larger scale.
We train models on A100 GPUs using PyTorch 2.0.
The code and pretrained models are made available under Apache 2.0 license111https://github.com/facebookresearch/dinov2.
The details of our models are in the appendix, Table17. With the same hardware, compared to the iBOT implementation, the DINOv2 code runs aroundfaster using onlyof the memory.Fast and memory-efficient attention.We implemented our own version of FlashAttentionto improve memory usage and speed on the self-attention layers.
Our version is on par with or better than the original on all cases considered, while covering more use-cases and hardware.
Due to the GPU hardware specifics, the efficiency is best when the embedding dimension per head is a multiple of 64, and the matrix operations are even better when the full embedding dimension is a multiple of 256.
As a consequence, our ViT-g architecture slightly differs from the architecture proposed byin order to maximize compute efficiency, and we use an embedding dimension of 1536 with 24 heads (64 dim/head) , rather than 1408 with 16 heads (88 dim/head) . Our experiments did not show significant differences in final accuracy, and our ViT-g backbone counts 1.1B parameters.Sequence packing.The DINO algorithm requires forwarding both large crops (at resolution 224)  and small crops (resolution 98) .
When split into patches, these two groups are represented by token sequences of different lengths and cannot be forwarded together.
In order to accelerate training, we use a trick called "sequence packing," which originates from NLP.
The idea is simple: we concatenate the sequences we must forward through the transformers into a single long sequence.
We pass this sequence through the transformer blocks as usual.
However, a block-diagonal mask is applied to the self-attention matrix in attention layers, preventing attention between different sequences.
This way, the forward is strictly equivalent to forwarding each sequence separately.
This trick gives us significant compute efficiency gains compared to using separate forward and backward passes, as in prior implementations.
The lower-level components of our setup are available in the xFormers library222https://github.com/facebookresearch/xformers() .Efficient stochastic depth.We implement an improved version of stochastic depththat skips the computation of the dropped residuals rather than masking the result.
This saves memory and compute in proportion approximately equal to the drop rate, thanks to specific fused kernels.
With high drop rates (in this work) , this allows a drastic improvement in compute efficiency and memory usage.
The implementation consists of randomly shuffling thesamples over the batch dimension, and slicing the firstsamples for the computations in the block.Fully-Sharded Data Parallel (FSDP) .Minimizing our objective with the AdamW optimizer requires 4 model replicas in float32 precision – student, teacher, optimizer first moments, optimizer second moments.
This sums toof memory for a billion-parameter model such as our ViT-g.
In order to reduce this memory footprint per GPU, we split the model replicas across GPUs, i.e., shardingacross GPUs using the PyTorch implementation of FSDP.
Consequently, the model size is not bounded by the memory of a single GPU but by the total sum of GPU memory across compute nodes.
The Pytorch implementation of FSDP brings a second advantage, which is to save on the cross-GPU communication costs: the weight shards are stored in float32 precision as required by the optimizer, but broadcasting weights and reducing gradients is done in float16 precision for the backbone (MLP heads gradients are reduced in float32 to avoid training instabilities) .
This leads to approximately 50% reduction in communication costs compared to the float32 gradient all-reduce operation used in DistributedDataParallel (DDP) , which is used in other self-supervised pretraining methods.
As a consequence, the training procedure scales more efficiently than DDP with float16 autocast when scaling the number of GPU nodes.
Overall, Pytorch-FSDP mixed-precision is superior to DDP with autocast in virtually all cases we encountered.Model distillation.Most of our technical improvements to the training loop aim at improving the training of large models over large quantities of data.
For smaller models, we distill them from our largest model, the ViT-g, instead of training them from scratch.
Knowledge distillationaims at reproducing the output of a large model with a smaller model by minimizing some distance between both outputs for a set of given inputs.
Since our objective function is a form of distillation from the teacher network to the student network, we leverage the same training loop with a few exceptions:
we use a larger model as a frozen teacher, keep a spare EMA of the student that we use as our final model, remove the masking and stochastic depth, and, apply the iBOT loss on the two global crops.
In our ablations, we observe that this approach achieves better performance than training from scratch, even for a ViT-L.
Our distillation method ends up close to the one described by, except we do not modify the loss terms for distillation and evaluate the EMA of the student.
body
DINOv2: Learning Robust Visual Featureswithout Supervision
5Efficient implementation
2304.07193
6Ablation StudiesWe present a set of ablations to empirically validate different components of our pipeline:
the technical modifications described in Sec.4, the pretraining data and the impact of model distillation.
We consider various downstream tasks that are described in Sec.7.6.1Improved Training RecipeOur approach improves over the iBOT method by combining it with several existing components described in Sec.4.
To evaluate their importance, we train multiple models where we successively add components to a baseline iBOT model.
We report the Top-1 accuracy on the validation set of ImageNet-1k with a k-NN and a linear probe in Table1.
Generally, we observe that each component improves the performance on either k-NN or linear probing and even both in most cases.
Only LayerScale and Stochastic Depth incur a performance drop in linear probing but significantly improve the training stability in our experience.INet-1k k-NNINet-1k lineariBOT72.982.3+(our reproduction) 74.583.2+LayerScale, Stochastic Depth75.482.0+128k prototypes76.681.9+KoLeo78.982.5+SwiGLU FFN78.783.1+Patch size 1478.983.5+Teacher momentum 0.99479.483.6+Tweak warmup schedules80.583.8+Batch size 3k81.784.7+Sinkhorn-Knopp81.784.7+Untying heads = DINOv282.084.5Table 1:Ablation study of the training differences between iBOT and DINOv2.We optimize for k-NN performance, as in our experience, the linear probe performance is lower-bounded by the k-NN performance.
Some modifications, like LayerScale and a high Stochastic Depth (rate=) , incur a decrease in linear probe performance, but have the benefits of increasing the stability of training by avoiding NaN loss values during training.
Overall, these modifications allowed for the next set of improvements to be added. Experiments are run using the ViT-Large architecture on ImageNet-22k.6.2Pretraining Data SourceThe quality of features is directly related to the quality of the pretraining data.
In this experiment, we probe the impact of LVD-142M compared to ImageNet-22k, a commonly used pretraining dataset, or using directly raw and uncurated data.
For the uncurated dataset, we randomly samplemillion images from the same data source as LVD-142M.
We train a ViT-g/14 on each dataset for the same number of iterations.
We also include a variant of ImageNet-22k obtained by removing the synsets of ImageNet-1k (INet-22kINet-1k)  for completeness.
We report the comparisons in Table2.Training DataINet-1kIm-AADE-20kOxford-MiNat2018iNat2021Places205INet-22k85.973.546.662.581.185.667.0INet-22kINet-1k85.370.346.258.780.185.166.5Uncurated data83.359.448.554.368.076.467.2LVD-142M85.873.947.764.682.386.467.6Table 2:Ablation of the source of pretraining data.We compare the INet-22k dataset that was used in iBOT to our dataset, LVD-142M.
Each model is trained for the same number of iterations, that is smaller than in our final run, without high-resolution adaptation.
Pretraining on LVD-142M maintains the performance over INet-1k while leading to models that perform better in other domains.The most salient observation is that training on a curated set of images works better on most benchmarks than training on uncurated data.
This confirms the benefit of curating data, even in the case of self-supervised pretraining.
When compared with models trained on ImageNet-22k, training on LVD-142M is also superior on all the benchmarks but ImageNet-1k.
This confirms that training on a more diverse set of images improves the quality of the features in domains that are not covered by ImageNet-22k.
We also see that training on our curated data increases the performances on domains that are not used for the curation process (INaturalist 2018, 2021 and Places205) , proving that scale and diversity can benefit unseen domains.
Overall, the conclusion of this ablation is that our dataset provides a good balance of different types of images that leads to the best performance overall.6.3Model Size and DataWe quantify the importance of scaling data with the model size in Fig.4.
As the size of models grow, training on LVD-142M becomes more beneficial than training on ImageNet-22k.
For instance, a ViT-g trained on LVD-142M matches the performance on ImageNet-1k of a model trained on ImageNet-22k while significantly outperforming it on the other benchmarks.Figure 4:Model scale versus data scale.Evolution of performance as a function of model size for two different pretraining datasets: ImageNet-22k (14M images)  and LVD-142M (142M images) .
The ViT-g trained on LVD-142M surpasses the ViT-g trained on ImageNet-22k on most benchmarks.6.4Loss ComponentsWe validated the proposed technical improvements in Sec.6.1by adding them incrementally.
This section analyzes the performance hit observed if we ablate specific loss terms, starting from our best-performing model.
We ablate the importance of the KoLeo loss and the impact of the masked image modeling term.
For both, we report performance on ImageNet-1k using a linear classifier, ADE-20k segmentation using a linear classifier, and nearest-neighbor image retrieval on Oxford-M.
Table3(a) shows the impact of using the KoLeo loss.
We see that the instance retrieval performance improves by more than, confirming that this term helps spread features in the output space.
At the same time, the other metrics do not suffer from this regularization.
In Table3(b) , we show the impact of using the masked image modeling term from iBOT.
This term is critical for dense prediction tasks, leading to almostperformance improvement.KoLeoINet-1kIm-AADE-20kOxford-M✕85.370.647.255.6✓85.872.847.163.9(a) Koleo lossMIMINet-1kIm-AADE-20kOxford-M✕85.372.044.264.3✓85.872.847.163.9(b) MIM objective in iBOTTable 3:(a) Effect of the KoLeo loss term.(b) Effect of the iBOT Masked Image Modeling (MIM)  loss term.
Evaluation performed on ImageNet-{1k,A} (classification with linear probe, accuracy %) , ADE-20k (segmentation with linear layer, mIoU)  and Oxford-M (image retrieval, mAP) .
Each model is trained on the same number of iterations, that is smaller than our final run.
The KoLeo loss term improves nearest-neighbor search tasks (e.g. retrieval) , and the MIM loss improves patch-level tasks (e.g. segmentation) .6.5Impact of Knowledge DistillationFor small architectures, we distill larger models instead of training them from scratch.
We use the distillation procedure described in Sec.5.
We evaluate the effectiveness of this approach by comparing a ViT-L/14 trained from scratch with one distilled from a ViT-g/14 over 12 benchmarks in Fig.5.
We also report the performance of the ViT-g/14 used for distillation as a topline.
The distilled model outperforms the one trained from scratch on all 12 benchmarks, validating our pretraining approach for small models.(a) Comparison on individual metricsArchMethodINet-1kSegm.DepthClassif.ViT-g/14Scratch86.573.41.0092.1ViT-L/14Scratch84.572.21.1090.2ViT-L/14Distill86.373.31.0891.2ArchMethodFinegr.Retriev.ARSketchVideoViT-g/14Scratch78.375.277.069.3ViT-L/14Scratch75.871.369.567.3ViT-L/14Distill77.676.374.567.5(b) Averaged metrics on 8 vision tasksFigure 5:Effectiveness of knowledge distillation.Comparison between a ViT-L trained from scratch or distilled from DINOv2 using ViT-g/14.
For reference, we also report the performance of the ViT-g/14 teacher. We show that a ViT-L model distilled from a frozen ViT-g outperforms a the same model trained from scratch on all benchmarks, sometimes even outperforming the distillation target.6.6Impact of ResolutionWe measure the impact of changing the resolution during the pretraining on the performance of image and patch-level features.
We consider models trained from scratch using a fixed resolution of eitheror, and a model trained from scratch at, then resumed for 10k more iterations at.
High-resolution training is compute-intensive, so we conduct this ablation on a small setup: a ViT-L/16 trained on ImageNet1k.
In Fig.6, we report the performance of a linear probe on ImageNet-1k and ADE-20k, evaluated at various resolutions.
The model trained on high-resolution images performs best across resolutions, but this comes at a high cost: training atis approximatelymore compute-intensive than training at.
On the other hand, training at high resolution for only 10k iterations at the end of the training is almost as good and only requiring a fraction of the compute.
As a consequence, we include this step at the end of the training rather than training at a high resolution from scratch.Figure 6:Role of resolution.Performance of ViT-L/16 trained on ImageNet-1k at fixed resolution (“224” and “416”)  or trained at 224 then 416 for a short duration (“224416”) .
We train linear classifiers on top of frozen features at different resolutions and report Top-1 accuracy on ImageNet and mIoU on ADE-20k. We observe that performing SSL training at high resolution for a short duration achieve behavior and results close to training at the same high resolution for the full training, at a fraction of the cost.
body
DINOv2: Learning Robust Visual Featureswithout Supervision
6Ablation Studies
2304.07193
7ResultsIn this section, we present the empirical evaluation of our models on many image understanding tasks.
We evaluate both global and local image representations, on category and instance-level recognition, semantic segmentation, monocular depth prediction, and action recognition.
We detail the list of benchmarks in AppendixC.
The goal of this evaluation is twofold.
First, we show that our self-supervised features outperform the current state of the art by a very large margin.
Second, we show that they match, or surpass the performance of weakly-supervised ones on a substantial number of tasks.Baselines.In our comparisons, we use two kinds of models as baselines.
We compare to the best performing self-supervised models that are openly available.
First, we run our evaluations for MAE, DINO, SEERv2, MSN, EsViT, Mugsand iBOT.
When several architectural variants were proposed for a given method, we report results for the one that leads to best top-1 accuracy on ImageNet-1k.
Second, we report performance of open-source weakly-supervised models such as CLIP, OpenCLIP, and SWAG.
When evaluating models on ImageNet-1k, we report the performance for each of the aforementioned methods.
For all other evaluations, we report the four best-performing models amongst SSL ones.
Also, for reference, we report the best performing OpenCLIP-G for weakly-supervised ones.7.1ImageNet ClassificationAs a first evaluation, we probe the quality of the holistic image representation produced by the model on the ImageNet-1k classification dataset.
We evaluate the quality of features by training a simple classifier over a frozen backbone, and do not perform finetuning of the backbone weights.
Following previous work, we use a linear model for simplicity, ensuring a reproducible evaluation, despite the fact that classes may not be linearly separable.
Because most SSL methods were developped using ImageNet-1k validation performance as a debugging signal, we also report the top-1 accuracy on ImageNet-ReaL and ImageNet-V2.
In order to report this additional validation performance, for all models, we run the evaluation with our code.
We compare our frozen features to the best publicly available SSL features in Table4, regardless of architecture or pretraining data.
We see the components proposed in this work lead to a very significant improvement ()  over the previous state of the art (iBOT ViT-L/16 trained on ImageNet-22k)  on linear evaluation.
At the same time, we also see that the performance increase on the alternative test sets is larger for our method, indicating stronger generalization. We describe details of our linear evaluation in AppendixB.3.kNNlinearMethodArch.DataText sup.valvalReaLV2Weakly supervisedCLIPViT-L/14WIT-400M✓79.884.388.175.3CLIPViT-L/14WIT-400M✓80.585.388.875.8SWAGViT-H/14IG3.6B✓82.685.788.777.6OpenCLIPViT-H/14LAION-2B✓81.784.488.475.5OpenCLIPViT-G/14LAION-2B✓83.286.289.477.2EVA-CLIPViT-g/14custom∗✓83.586.489.377.4Self-supervisedMAEViT-H/14INet-1k✕49.476.683.364.8DINOViT-S/8INet-1k✕78.679.285.568.2SEERv2RG10BIG2B✕–79.8––MSNViT-L/7INet-1k✕79.280.786.069.7EsViTSwin-B/W=14INet-1k✕79.481.387.070.4MugsViT-L/16INet-1k✕80.282.186.970.8iBOTViT-L/16INet-22k✕72.982.387.572.4DINOv2ViT-S/14LVD-142M✕79.081.186.670.9ViT-B/14LVD-142M✕82.184.588.375.1ViT-L/14LVD-142M✕83.586.389.578.0ViT-g/14LVD-142M✕83.586.589.678.4Table 4:Linear evaluation on ImageNet-1k of frozen pretrained features.We report Top-1 accuracy on the validation set for publicly available models trained on public or private data, and with or without text supervision (text sup.) . For reference, we also report the kNN performance on the validation set.
We compare across any possible architectures (Arch.) , at resolutionunless stated otherwise.
The dataset used for training EVA-CLIP is a custom mixture, see paper for details.How far are we from weakly-supervised models?We also want to validate that our features are competitive with state-of-the-art open-source weakly supervised models.
To this end, we compare on ImageNet-1k, using the linear evaluation, to three off-the-shelf methods with several architectural variants.
For all models, we run the linear evaluation using our code, after making sure that our numbers match those reported in technical reports and papers.
We show the result of this evaluation in Table4.
We see that our backbone, surpases the performance of OpenCLIP with a ViT-G/14 architecture ()  and EVA-CLIP with a ViT-g/14 () .
At the same time, we also observe that our performance on the ImageNet-V2 test set is significantly better (versus EVA-CLIP) , indicating better generalization.
For the remainder of this section, we report OpenCLIP-G as a reference for weakly-supervised models.Can we finetune the encoders?We question if the ability of our models to produce high quality frozen features impact their performance when finetuned with supervision on a specific dataset.
While this is not core to this paper, this experiment is indicative of whether we have involuntarily specialized our models to the setting of linear evaluations of frozen features.
To run this sanity check, we apply the finetuning pipeline from, without tweaking hyper-parameters.
In Table5, we show that the Top-1 accuracy on the validation set of ImageNet-1k improves by more thanwhen the backbone is finetuned.
This is true both when using models at resolutionand.
Further gains can be obtained by tuning the hyper-parameters of the finetuning, but this is beyond the goal of this sanity check.
Nonetheless, our best finetuned performance ()  is only a couple of percent below ()  the absolute state of the arts () , obtained by.
As DINOv2 leads to features that are strong in both the linear and finetuning settings, a strong property of our approach is thatfinetuning is optional.Arch.Res.LinearFinetunedViT-g/1422486.588.5+2.044886.788.9+2.2Table 5:Supervised finetuning on ImageNet-1k.We use the pipeline ofto finetune our encoders on ImageNet-1k at resolutionsor. We compare with the accuracy obtained with linear probing and observe only modest improvements with fine-tuning: this suggests that DINOv2 features already perform well out-of-the-box.Robustness analysis.To complement our study, and probe the generalization of our features, we evaluate our ImageNet-1k models trained with linear classification heads on domain generalization benchmarks.
We use the best performing linear classifier as described above and simply run inference on those benchmarks.
Please note that most results in the literature are obtained with models that are finetuned end-to-end on ImageNet-1k.
We show the result of this experiment in Table6.
When comparing with state-of-the-art SSL methods, our models shows drastically better robustness (on A,on Randon Sketchcompared to iBOT) .
Our model also improves upon the best weakly-supervised model on ImageNet-A while lagging behind on R and Sketch.MethodArchDataIm-AIm-RIm-CSketchOpenCLIPViT-G/14LAION-2B63.887.845.366.4MAEViT-H/14INet-1k10.234.461.421.9DINOViT-B/8INet-1k23.937.056.625.5iBOTViT-L/16INet-22k41.551.043.938.5DINOv2ViT-S/14LVD-142M33.553.754.441.2ViT-B/14LVD-142M55.163.342.750.6ViT-L/14LVD-142M71.374.431.559.3ViT-g/14LVD-142M75.978.828.262.5Table 6:Domain Generalization with a linear probeon top of frozen features at a resolution of 224.
Higher numbers are better for all benchmarks except Im-C.7.2Additional Image and Video classification BenchmarksIn this section we study the generalization of our features on downstream classification benchmarks.
We consider two sets of evaluations in that context.
On one hand, we use large and finegrained datasets such as iNaturalist and Places205.
On the other, we use the 12 image classification tasks originally proposed in SimCLR.
For iNaturalist 2018, iNaturalist 2021, and Places205, we train a linear classifier with data augmentations as in Sec.7.1We report top-1 accuracy for those three datasets in Table7.
Interestingly, our model significantly outperforms OpenCLIP ViT-G/14 on both variants of iNaturalist (andfor 2018 and 2021 respectively) , and lags slightly behind on Places 205 () .Image classificationVideo classificationFeatureArchiNat2018iNat2021Places205K400UCF-101SSv2OpenCLIPViT-G/1473.076.069.878.390.735.8MAEViT-H/1431.032.352.454.270.629.2DINOViT-B/859.668.360.464.585.032.6iBOTViT-L/1666.374.664.472.688.638.7DINOv2ViT-S/1469.074.262.967.887.033.1ViT-B/1476.481.166.273.289.134.4ViT-L/1480.485.167.376.390.535.6ViT-g/1481.685.767.578.491.238.3Table 7:Linear evaluation on other image and video classification.The image benchmarks contain a large quantity of fine-grained examples about objects or scenes.
The video benchmarks cover action classification and human-object interaction.
All the features are frozen with a linear probe on top.In a second set of evaluations, we measure the performance of our model on video action recognition even though our features were not trained on videos..
We evaluated features on three datasets, namely UCF-101, Kinetics-400and Something-Something v2.
For this evaluation, we pickevenly spaced frames in the video and train a linear classifier on the average of the features for UCF and K-400.
For SSv2, we opt for concatenation to retain more temporal information than with feature averaging.
For each dataset, we measure average accuracy and report the results in Table7.
We see that amongst self-supervised approaches, our model clearly sets a new state of the art.
Moreover, our model matches the accuracy of the OpenCLIP features on UCF and Kinetics (andrespectively)  and clearly outperforms them on SSv2 () .
This is particularly interesting, as SSv2 requires a much richer understanding of the video frames.Finally, in Table8, we compare selected frozen features on 12 transfer classification benchmarks initially proposed by.
This benchmark covers scenes, objects (food, cars, planes) , and textures.
We replace the Birdsnap dataset with CUB because the former was not publicly available in its entirety.
We follow the experimental protocol as outlined by, namely training a logistic regression on precomputed features.
Our model significantly outperforms state-of-the-art SSL models, with most notable differences on Stanford Cars (versus DINO ViT-B/8)  and FGVC Aircraft (versus iBOT ViT-L/16) .
Even though these benchmarks favor text-guided pretraining, our features are still competitive with OpenCLIP on most classification benchmarks, with the exception of a few datasets, especially SUN ()  and Cars () .FeatureArchFoodC10C100SUNCarsAircrVOCDTDPetsCal101FlowersCUBAvgOpenCLIPViT-G/1494.598.791.084.096.180.289.386.095.798.199.589.991.9MAEViT-H/1478.496.183.963.956.163.484.375.489.495.992.357.278.0DINOViT-B/885.197.286.970.376.670.686.779.693.295.497.681.785.1iBOTViT-L/1691.099.092.875.671.872.489.080.787.797.599.682.186.6DINOv2ViT-S/1489.197.787.574.481.674.087.880.695.197.099.688.187.7ViT-B/1492.898.791.377.388.279.488.283.396.296.199.689.690.1ViT-L/1494.399.393.478.790.181.588.384.096.697.599.790.591.2ViT-g/1494.799.594.478.791.487.289.084.596.797.699.791.692.1Table 8:Linear evaluation of frozen features on fine-grained benchmarks.Accuracy on 12 benchmarks covering objects, scenes and textures following the evaluation protocol proposed in.7.3Instance RecognitionIn this experiment, we probe our model on the task of instance-level recognition using a non-parametric approach.
Images from a database are ranked according to their cosine similarity with a query image.
We evaluated our model and compare to baselines on Paris and Oxford, that are landmark recognition benchmarks.
We also evaluated on Met, a dataset of artworks from the Metropolitan museum, and AmsterTime, containing street view images matched to archival images of Amsterdam.
We measure performance by computing the mean average precision and report our results in Table9.
We see that our features significantly outperform both SSL (mAP on Oxford-Hard) , and weakly-supervised (mAP on Oxford-Hard)  ones.
It is interesting to see that our features perform well across task granularities, both at the category-level and instance-level.
This is a desirable property for strong off-the-shelf computer vision features.OxfordParisMetAmsterTimeFeatureArchMHMHGAPGAP-ACCmAPOpenCLIPViT-G/1450.719.779.260.26.523.934.424.6MAEViT-H/1411.72.219.94.77.523.530.54.2DINOViT-B/840.113.765.335.317.137.743.924.6iBOTViT-L/1639.012.770.747.025.154.858.226.7DINOv2ViT-S/1468.843.284.668.529.454.357.743.5ViT-B/1472.949.590.378.636.763.566.145.6ViT-L/1475.154.092.783.540.068.971.650.0ViT-g/1473.652.392.182.636.873.676.546.7Table 9:Evaluation of frozen features on instance-level recognition.We consider 4 different benchmarks and report their main metrics.7.4Dense Recognition TasksWe probe the quality of patch-level features extracted from our network on several dense downstream tasks.
We consider semantic image segmentation and monocular depth estimation in several settings and we conduct evaluations on multiple datasets for each.Semantic segmentation.For our semantic segmentation evaluation, we consider two different setups.Linear: a linear layer is trained to predict class logits from a patch tokens.
It is used to produce a low-resolution logit map (eg 32x32 for a model with patch size 16) , which is then upsampled to full resolution (512x512)  to obtain a segmentation map.
This procedure is extremely simple but cannot easily produce high-resolution segmentations.+ms: a boosted version of the linear setup.
We concatenate the patch tokens of the 4 last layers, use a larger image resolution of 640, and use multiscale test-time augmentations to improve the predictions.
We report the performance of our model variants as well as the baselines on three datasets under the two setups in Table10.Our models show very good performance on all datasets and for all setups.
Interestingly, our evaluation using+msis on par with fully finetuning MAE with an Upernet decoder (versusmIoU) .
This is surprising because we use a significantly simpler predictor.
Also, our best model, when evaluated using the boosted recipe, almost matches the state of the art on Pascal VOC (versusmIoU) .Frozen backbone in a SOTA pipeline.In a final experiment, we freeze our backbone, and plug it into a ViT-Adapterwith a Mask2former head.
We tune the weights of the adapter and head, but keep the backbone frozen, meaning 66% of the weights are frozen.
This allows for a lighter segmentation training than full end-to-end fine-tuning.
With this setup, we reachmIoU on ADE20k, close to the competitive state of the art, standing at 62.9 mIoU.
Although our setup for this experiment doesn’t makes use of the optimisations described in Sec.5, the segmentation training in this experiment took 28 hours on 16 V100 GPUs.ADE20kCityScapesPascal VOC(62.9) (86.9) (89.0) MethodArch.lin.+mslin.+mslin.+msOpenCLIPViT-G/1439.346.060.370.371.479.2MAEViT-H/1433.330.758.461.067.663.3DINOViT-B/831.835.256.966.266.475.6iBOTViT-L/1644.647.564.874.582.384.3DINOv2ViT-S/1444.347.266.677.181.182.6ViT-B/1447.351.369.480.082.584.9ViT-L/1447.753.170.380.982.186.0ViT-g/1449.053.071.381.083.086.2Table 10:Semantic segmentation on ADE20K, CityScapes and Pascal VOC with frozen featuresand a linear classifier (lin.)  and with multiscale (+ms) .
The absolute state of the art – from,andrespectively – are mentioned at the top of the Table.
For reference, using the Mask2Former pipelinewith a ViT-Adapteron top of our frozen ViT-g/14 backbone gives 60.2 mIoU on ADE-20k.Depth estimation.In this experiment, we evaluate our patch-level features on three monocular depth estimation benchmarks: NYUd, KITTI and zero-shot transfer from NYUd to SUN3d.
We follow the evaluation protocol of.
We consider three different setups for this evaluation.lin. 1: we extract the last layer of the frozen transformer and concatenate the[CLS]token to each patch token.
Then we bi-linearly upsample the tokens by a factor of 4 to increase the resolution.
Finally we train a simple linear layer using a classification loss by dividing the depth prediction range in 256 uniformly distributed bins and use a linear normalization following.lin. 4: we use the same protocol that we use with one layer, but concatenate the tokens from layersfor ViT-S/B,for ViT-L, andfor ViT-g.DPT: we use the DPT decoderon top of our frozen models and setup a regression task.
We scale the size of the head following the dimension of the features for each architecture.
We show results for all baselines, all datasets and all setups in Table11.From this table, we see that our features clearly surpass the best SSL and WSL features available.
It is interesting to see that iBOT features extracted from a ViT-L outperform the ones of OpenCLIP with a ViT-G.
This observation supports the intuition that caption-based feature learning fails to learn subtle patterns like this one.
Also, our model, with the DPT decoder and frozen backbone, matches or exceeds the performance of the recent work of.
Finally, the out-of-domain generalization result on SUN-RGBd shows that our features allow very good transfer between domains.
A depth prediction module trained on indoor scenes from NYUd generalizes pretty well to the outdoor examples of SUN-RGBd.NYUdKITTINYUdSUN RGB-D(0.330) (2.10) (0.421) MethodArch.lin. 1lin. 4DPTlin. 1lin. 4DPTlin. 1lin. 4DPTOpenCLIPViT-G/140.5410.5100.4143.573.212.560.5370.4760.408MAEViT-H/140.5170.4830.4153.663.262.590.5450.5230.506DINOViT-B/80.5550.5390.4923.813.562.740.5530.5410.520iBOTViT-L/160.4170.3870.3583.313.072.550.4470.4350.426DINOv2ViT-S/140.4490.4170.3563.102.862.340.4770.4310.409ViT-B/140.3990.3620.3172.902.592.230.4480.4000.377ViT-L/140.3840.3330.2932.782.502.140.4290.3960.360ViT-g/140.3440.2980.2792.622.352.110.4020.3620.338Table 11:Depth estimation with frozen features.
We report performance when training a linear classifier on top of one (lin. 1)  or four (lin. 4)  transformer layers, as well, as the DPT decoder (DPT)  of.
We report the RMSE metric on the 3 datasets.
Lower is better.
For reference, we report state-of-the-art results taken fromon each benchmark on top of the Table.7.5Qualitative ResultsIn this final section of the empirical evaluation of our features, we propose a few qualitative analyses.Semantic Segmentation and Depth Estimation.We show some qualitative results for our dense prediction evaluations: segmentation on ADE20K in Fig.7and depth estimation on NYUd, KITTI and SUN RGB-D in Fig.7.
We compare DINOv2 with OpenCLIP with a linear classifier on each dataset.
While not perfect, the linear segmentation model using our DINOv2 backbone produces good results and behaves much better than the OpenCLIP one under this evaluation setup.
Indeed, the segmentation mask produced by OpenCLIP-G shows many artifacts and disconnected components.
The qualitative results on depth estimation clearly illustrate the quantitative gap between OpenCLIP and DINOv2.
These results highlight that our features, as well as the features extracted from OpenCLIP, are able to linearly separate complex information such as depth, even though neither was trained with this type of information.
However, our features lead to a much smoother depth estimation, with less artifacts.
Some objects such as the chair on the SUN RGB-D image are completely ignored by OpenCLIP and correctly positioned using our features.Figure 7:Segmentation and depth estimation with linear classifiers.Examples from ADE20K, NYUd, SUN RGB-D and KITTI with a linear probe on frozen OpenCLIP-G and DINOv2-g features.Out-of-distribution generalization.We show a few examples of applying the depth prediction and segmentation linear classifiers to out-of-distribution examples in Fig.8.
The qualitative results support our claim that our features transfer between domains.
The quality of the depth and segmentation predicted for pictures of animals, or paintings is very good, even though the domains are very different.Figure 8:Examples of out-of-distribution exampleswith frozen DINOv2-g features and a linear probe.Figure 9:More visualization of the first PCA components.We compute the PCA between the patches from all of the images and show their first 3 components.
Each component corresponds to a specific color channel. Same parts are matched between related images depsite changes of pose, style or even objects.
Background is removed by removing patches with a negative score of the first PCA component.PCA of patch features.We show the results of the principal component analysis (PCA)  performed on the patch features extracted by our model.
We keep only patches with a positive value after we threshold the first component.
This procedure turns out to separate the image’s main object from the background.
We compute a second PCA on the remaining patches across three images depicting the same category.
We color the three first components with three different colors and present the results in Fig.1and9.
There are two interesting observations:
first, our unsupervised foreground / background detector, based on detecting the highest variance direction, performs very well and is capable of delineating the boundary of the main object in the picture.
Second, the other components correspond to "parts" of objects and match well for images of the same category.
This is an emerging property – our model was not trained to parse parts of objects.Patch matching.Finally, we explore what type of information our patch-level features contain by matching them across images.
We start by detecting the foreground object using the procedure described above.
Then, we compute the euclidean distance between patch features extracted from two images and map them by solving an assignment problem.
In order to reduce the number of matches, we then apply a non-maximum suppression to keep only the salient ones.
In Fig.10, we show some examples of such matchings.We observe that the features seem to capture information about semantic regions that serve similar purpose in different objects or animals.
For instance, the wing of a plane matches the wing of a bird.
We also observe that the model is robust to style (image versus drawing) , and to large variation of poses (see the elephant) .Figure 10:Matching across images.We match patch-level features between images from different domains, poses and even objects that share similar semantic information.
This exhibits the ability of our model to transfer across domains and understand relations between similar parts of different objects.
body
DINOv2: Learning Robust Visual Featureswithout Supervision
7Results
2304.07193
8Fairness and Bias AnalysisWe conduct two fairness evaluations of our models.
We probe for geographical fairness and potential harmful label associations.
For both evaluations, we experiment with our largest ViT-g model.8.1Geographical FairnessWe evaluate geographical fairness on the Dollar Street dataset introduced inusing the evaluation protocol of.
This benchmark compares performance across countries and income levels. It contains 16,073 images from 289 households across 54 countries.
The task is to recognize 94 concepts that vary visually between households based on income or location.
In Table12, we compare our model with SEERv2, a model trained on a geographically diverse set of images.
Our model is slightly fairer across regions and incomes than the SEERv2 model and significantly better than the supervised baseline reported by.
However, we still observe a significant difference between regions, particularly in Africa, where our model performance drops by 25.7% compared to Europe.
This shows that our model is still biased toward Western countries.
Similarly, our model performs significantly better on high-income households than low-income ones, with a difference of 31.7%.
Despite improvements, we observe significant biases in our models toward wealthy households from Western countries.Income bucketsRegionsMethodArch.DatalowmediumhighAfricaAsiaAmericasEuropeSEERv2RG-10BIG-1B59.778.586.665.976.381.185.6DINOv2ViT-g/14LVD-142M67.483.390.574.081.686.289.7Table 12:Geographical fairness and diversity analysis across income buckets and regions.8.2Gender, Skintones and AgeIn a second set of evaluations, we question how our model classifies images of people of different gender, skin tone, and age (all self-reported) .
We follow the protocol of, where we train a multiclass classifier on a subset of 619 classes of ImageNet-22k.
We group the 619 classes into four broader categories: Human, Possibly Human, Non-Human, or Crime.
Non-Human and Crime are considered harmful.
Using this classifier, we run inference on 2955 images from the Casual Conversations datasetand keep all labels in the top-5 that are assigned a probability of 0.1 or more.
Because of that, we can assign multiple classes to each image.
We make one modification to the original evaluation protocol: we do not backpropagate gradients to the backbone and keep it frozen.
We compare our model to SEERv2 in Table13.Our model often classifies images of all groups as Human without large deviations across skin tones.
Neither SEERv2 nor DINOv2 predict harmful labels from the Non-Human or Crime meta-categories (except for two instances where the background contains bars visually similar to prison bars) .
We see that our model triggers the Possibly-Human classes often.
This class is constructed from objects in ImageNet-22k that are often related to Humans, such as Scarf, Glasses, or Beard.
Our model often predicts the Possibly-Human class for men because of the prevalence of the Beard class.
No clear pattern indicates a bias against a particular group in this study.
While this is encouraging, we also acknowledge that a more thorough evaluation of biases may reveal flaws in our model.Gender SkintoneAge GroupsModelAssoc.femaledarkerfemalelightermaledarkermalelighter18-3030-4545-7070+SEERNon-Human0.00.00.00.00.00.00.00.0RG-10BCrime0.00.00.00.00.00.00.00.0Human94.995.886.679.090.588.391.982.3Possibly-Human13.66.765.060.232.837.229.46.5DINOv2Non-Human0.00.00.00.00.00.00.00.0ViT-g/14Crime0.00.00.20.00.00.10.00.0Human97.397.786.184.091.290.293.288.7Possibly-Human15.817.252.248.135.337.323.09.7Table 13:Label association fairness evaluation across gender, skintones and age groups.We follow the protocol proposed bywith a slight modification.
Instead of finetuning the backbone, we simply learn a linear classifier on the subset of 619 classes of ImageNet-22k.
body
DINOv2: Learning Robust Visual Featureswithout Supervision
8Fairness and Bias Analysis
2304.07193
9Estimating the Environmental Impact of Training our ModelsModel toGPU TypeGPU PowerGPU-hoursPUETotal powerCarbon emittedReproduceconsumptionconsumption(tCO2eq) DINOv2-gA100-40GB400W22,0161.19.7 MWh3.7Table 14:Carbon footprint of reproducing DINOv2.We report the potential carbon emission of reproducing DINOv2-g when assuming a power consumption for the A100-40GB of 400W, a PUE of 1.1 and carbon intensity factor of 0.385 kg CO2e per KWh.Training foundation models consumes a significant amount of energy, resulting in carbon dioxide emissions.propose a methodology to report an estimation of the carbon emitted during the training of a model based on the specifics of the data center and its power grid.
This computation informs the design of the data center used for the training of models and the choice of location for data centers.
This methodology requires to know the specifics of the data center used for training, which can be complex when multiple data centers are involved over time.
Additionally, these specifics are most often not in the control of the AI practitioner, and hence, this methodology is less helpful when practioners make technical decisions about future trainings.
Instead, in this section, we follow an alternative that reports the potential carbon emission of retraining a similar model in an average data center located in the US.
This methodology was used in previous work in natural language processingto establish an apple-to-apple comparison between pretraining schemes.
More precisely, we fix the value of all exogenous variables, i.e., the Power Usage Effectiveness (PUE)  and carbon intensity factor of a power grid to the same values as in, that is, a PUE of 1.1 and the carbon intensity factor to the US average of 0.385 kg CO2eq/KWh.
We use the same formula as into estimate the potential energy consumption and the carbon emission.
For the power consumption of an A100-80GB, we take the thermal design power for NVLink systems, which is 400W.
We report the potential carbon emission of retraining a DINOv2 ViT-g in Table14.
For comparison, retraining an OpenCLIP ViT-L or OpenCLIP ViT-G would require 22.4 MWh and 118.9 MWh, respectively, if run in the same data center.
This is 10more carbon emission.
Note that this comparison is not fair to them, since they also train a text encoder in parallel, and we thus do not report them in the table.
However, it gives a reasonable guideline for those who are interested in training only visual features: in this context, training a self-supervised model is preferable in terms of carbon emission.
Training a text-guided model still makes sense when planning to reuse the text encoder.Carbon footprint of the whole project.Additionally, we estimate the footprint of the whole project to be betweenk andk tCO2eq using the same grid as presented above333For context, a full Boeing 777 return flight between London and New York corresponds to approximately 560 tCO2eq..
This carbon footprint represents in the order ofk GPU-days.
The primary sources of emissions are the self-supervised pre-trainings of the models.
For example, a single pre-training of a ViT-g model (22k GPU-hours)  emits 3.7 tons of CO2eq, while a finetuning on ImageNet-1k (1k GPU-hours)  emits 0.2 tons.
This estimate only considers the GPUs’ electricity consumption and ignores other emissions, such as their manufacturing and disposal.
body
DINOv2: Learning Robust Visual Featureswithout Supervision
9Estimating the Environmental Impact of Training our Models
2304.07193
10Future work and DiscussionIn this work, we present DINOv2, a new series of image encoders pretrained on large curated data with no supervision.
This is the first SSL work on image data that leads to visual features that close the performance gap with (weakly)  supervised alternatives across a wide range of benchmarks and without the need for finetuning.

We can attribute the strong performance of the DINOv2 family of models to several factors:i)  an improved training recipe with better hyperparameters and regularization (Table1) ,ii)  a larger model scale with improved results regardless of the data used for training (Fig.4) ,iii)  a larger dataset (Fig.4)  andiv)  the distillation process that makes smaller models benefit from the performance of the strongest ViT-g model (Fig.5) .

A few properties emerge from these models, such as an understanding of object parts and scene geometry regardless of the image domains.
We expect that more of these properties will emerge at larger scales of models and data, akin to instruction emergence in large language models, and plan to continue scaling along these axes.
This paper also demonstrates that these visual features are compatible with classifiers as simple as linear layers - meaning the underlying information isreadily available.
In future work, we plan to leverage this ability to train a a language-enabled AI system that can process visual features as if they were word tokens, and extract the required information to ground the system.Acknowledgments.We thank Mathilde Caron for initial discussions that led to this work.
Julien Mairal was supported by the ERC grant number 101087696 (APHELAIA project)  and by ANR 3IA MIAI@Grenoble Alpes (ANR-19-P3IA-0003) .
We thank Olivia Joulin for the horse drawing used in Fig.10.
We thank Madeleine and Léon for posing for Fig.8We also thank the rest of FAIR and Meta AI for feedback on this work through the entire project.
conclusion
DINOv2: Learning Robust Visual Featureswithout Supervision
10Future work and Discussion