<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2402.18153] Diffusion-based Neural Network Weights Generation</title><meta property="og:description" content="Transfer learning is a topic of significant interest in recent deep learning research because it enables faster convergence and improved performance on new tasks. While the performance of transfer learning depends on t…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Diffusion-based Neural Network Weights Generation">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Diffusion-based Neural Network Weights Generation">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2402.18153">

<!--Generated on Tue Mar  5 16:12:45 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta name="keywords" lang="en" content="Machine Learning,  ICML">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_pruned_first">
<h1 class="ltx_title ltx_title_document">Diffusion-based Neural Network Weights Generation</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Bedionita Soro
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Bruno Andreis
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Hayeon Lee
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Song Chong
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Frank Hutter
</span></span>
<span class="ltx_author_before">  </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Sung Ju Hwang
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id1.id1" class="ltx_p">Transfer learning is a topic of significant interest in recent deep learning research because it enables faster convergence and improved performance on new tasks. While the performance of transfer learning depends on the similarity of the source data to the target data, it is costly to train a model on a large number of datasets. Therefore, pretrained models are generally blindly selected with the hope that they will achieve good performance on the given task. To tackle such suboptimality of the pretrained models, we propose an efficient and adaptive transfer learning scheme through dataset-conditioned pretrained weights sampling. Specifically, we use a latent diffusion model with a variational autoencoder that can reconstruct the neural network weights, to learn the distribution of a set of pretrained weights conditioned on each dataset for transfer learning on unseen datasets. By learning the distribution of a neural network on a variety pretrained models, our approach enables adaptive sampling weights for unseen datasets achieving faster convergence and reaching competitive performance.</p>
</div>
<div class="ltx_keywords">Machine Learning, ICML
</div>
<div id="p2" class="ltx_para">
<br class="ltx_break">
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">The goal of Automated Machine Learning (AutoML) <cite class="ltx_cite ltx_citemacro_citep">(Hutter et al., <a href="#bib.bib12" title="" class="ltx_ref">2019</a>; Doke &amp; Gaikwad, <a href="#bib.bib7" title="" class="ltx_ref">2021</a>)</cite> is to automate the end-to-end process of optimizing machine learning models for real-world problems. It has become a popular means of making machine learning more accessible to individuals with limited expertise in AI. AutoML encompasses various stages of the machine learning pipeline, including 1) Neural architecture selection: Choosing the most suitable neural architecture based on the problem at hand, 2) Hyperparameter tuning: Optimizing the hyperparameters of the selected neural architecture to achieve better performance and 3) Weight optimization: training the neural architecture or generating optimized weights on the provided data. Especially, transferable AutoML, which aims to effectively reuse pre-trained models by distilling knowledge from them and transferring it to an unseen dataset for fast adaptation, has become an important problem with significant interest in the AutoML community <cite class="ltx_cite ltx_citemacro_citep">(Hutter et al., <a href="#bib.bib12" title="" class="ltx_ref">2019</a>)</cite>. For the former two stages, such as neural architecture selection and hyperparameter tuning, there exist extensive set of previous works that address these problems <cite class="ltx_cite ltx_citemacro_citep">(Mlodozeniec et al., <a href="#bib.bib18" title="" class="ltx_ref">2023</a>; Jeong et al., <a href="#bib.bib13" title="" class="ltx_ref">2021</a>; Lee et al., <a href="#bib.bib16" title="" class="ltx_ref">2021</a>)</cite>. However, only a very few existing works have explored automating the weight optimization par <cite class="ltx_cite ltx_citemacro_citep">(Ha et al., <a href="#bib.bib9" title="" class="ltx_ref">2017</a>)</cite>. In other words, we are still heavily dependent on naive training or fine-tuning in the model training section, which becomes a bottleneck in the entire training process even with automated neural architectures and hyperparameter optimization.</p>
</div>
<figure id="S1.F1" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S1.F1.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2402.18153/assets/x1.png" id="S1.F1.sf1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="403" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S1.F1.sf1.3.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="S1.F1.sf1.4.2" class="ltx_text ltx_font_bold" style="font-size:90%;">Dessert Dataset</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S1.F1.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2402.18153/assets/x2.png" id="S1.F1.sf2.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="403" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S1.F1.sf2.3.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="S1.F1.sf2.4.2" class="ltx_text ltx_font_bold" style="font-size:90%;">Gemstones Dataset</span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S1.F1.3.1.1" class="ltx_text" style="font-size:90%;">Figure 1</span>: </span><span id="S1.F1.4.2" class="ltx_text" style="font-size:90%;">We train D2NWG on a modelzoo generated from the training split of the Kaggle Dataset of  <cite class="ltx_cite ltx_citemacro_citep">(Jeong et al., <a href="#bib.bib13" title="" class="ltx_ref">2021</a>)</cite> and generate weights directly for the datasets in the test split. In Figure <a href="#S1.F1.sf1" title="In Figure 1 ‣ 1 Introduction ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1(a)</span></a> we sample the full set of weights using D2NWG (without any finetuning) conditioned on the dataset encoding of the Dessert dataset. For the baselines, we finetune them on the full Dessert dataset. In Figure <a href="#S1.F1.sf2" title="In Figure 1 ‣ 1 Introduction ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1(b)</span></a>, we again perform dataset-conditioned sampling with D2NWG and fintune it, together with a model pretrained on ImageNet and a randomly initialized model (RandInit) for 50 epochs. As can be seen, D2NWG can sample high performing weights for unseen datasets and achieves better convergence rate when the sampled weights are finetuned. The full set of results for Kaggle dataset is provided in Table <a href="#S4.T3" title="Table 3 ‣ 4.3 Performance Evaluation on KaggleZoo ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> in the experimental section.</span></figcaption>
</figure>
<figure id="S1.F2" class="ltx_figure"><img src="/html/2402.18153/assets/x3.png" id="S1.F2.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="369" height="207" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S1.F2.7.1.1" class="ltx_text" style="font-size:90%;">Figure 2</span>: </span><span id="S1.F2.8.2" class="ltx_text ltx_font_bold" style="font-size:90%;">Stage 1:<span id="S1.F2.8.2.1" class="ltx_text ltx_font_medium"> This stage consists of an encoder-decoder model trained to learn to reconstruct pretrained weights in a modelzoo. Latent embeddings from this stage are manipulated, conditioned on a dataset, in the next stages. </span>Stage 2<span id="S1.F2.8.2.2" class="ltx_text ltx_font_medium">: In the second stage, we learn a dataset encoder for compressing a given dataset into a vector representation. The resulting dataset encoder is frozen in the next stage. </span>Stage 3:<span id="S1.F2.8.2.3" class="ltx_text ltx_font_medium"> In the third stage, we learn to manipulate the latent space of Stage 1 conditioned on the dataset encodings from Stage 2 using a diffusion model. During inference, we condition on a dataset (seen or unseen) to obtain a latent representation using the diffusion model and then decode this representation back to weights using the decoder in Stage 1. Each stage is trained independently and the <span id="S1.F2.8.2.3.1" class="ltx_text" style="color:#0000FF;">blue</span> arrow signifies joint training of Stages 1 and 2, an ablation provided in Table <a href="#A8.T8" title="Table 8 ‣ H.1 Coupling with Accuracy Predictor ‣ Appendix H Detail of Experimental Results ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> of the Appendix.</span></span></figcaption>
</figure>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">On weight optimization automation, there exist only a handful of few related works. <cite class="ltx_cite ltx_citemacro_citet">Schürholt et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022a</a>)</cite> proposed to learn the distribution of diverse pretrained weights enabling sampling for better initialization. Similar idea is investigated by <cite class="ltx_cite ltx_citemacro_citet">Peebles et al. (<a href="#bib.bib21" title="" class="ltx_ref">2022</a>)</cite>, where they used prompt based fine-tuning. However, these methods do not explore the relationship of the sampled weights and target dataset. In this regard, <cite class="ltx_cite ltx_citemacro_citet">Nava et al. (<a href="#bib.bib19" title="" class="ltx_ref">2023</a>)</cite> proposed a meta-hypernetwork name HyperLDM class-conditioned weights generation for visual question answering(VQA) where the text descriptor of each image(classes) is used as condition.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">Most existing weights generation methods for image datasets that exploit knowledge of pretrained weights are limited to unconditional parameters generation and are designed for a single model and datasets or restricted to class-conditioning such as conditioning on target loss on the same dataset. In such cases they do not take into account the correlation between the pretrained weights and the corresponding pretrained dataset. Additionally, previous works have not investigated how to properly generate weights for unseen datasets, which is a problem of significant practical importance.
Therefore, we take the parameters generation problem one step further and extend it to dataset-conditioned sampling while taking advantage of existing diffusion models and pretrained weights knowledge. We propose a new method called Diffusion based Neural Network Weights Generation (D2NWG) to learn the distribution of a dataset of pretrained weights conditioned on the their corresponding dataset embeddings enabling target datasets conditioned weight generation during inference for better transfer learning as shown in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.
We demonstrate generalization to seen and unseen datasets by sampling weights for an architecture conditioned on the corresponding dataset of interest. Our contributions are as follows:</p>
<ul id="S1.I1" class="ltx_itemize">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p">We efficiently adapt latent diffusion <cite class="ltx_cite ltx_citemacro_citep">(Rombach et al., <a href="#bib.bib24" title="" class="ltx_ref">2021</a>)</cite> to network weights generation to improve the sampling performance and transfer learning.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p">We combine CLIP <cite class="ltx_cite ltx_citemacro_citep">(Radford et al., <a href="#bib.bib22" title="" class="ltx_ref">2021</a>)</cite> together with Set Transformer <cite class="ltx_cite ltx_citemacro_citep">(Lee et al., <a href="#bib.bib17" title="" class="ltx_ref">2019</a>)</cite> for better dataset conditioning.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p">Our demonstrate that our method, D2NWG, allows dataset-conditioned diverse weights sampling and can efficiently sample weights conditioned on unseen datasets.</p>
</div>
</li>
<li id="S1.I1.i4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i4.p1" class="ltx_para">
<p id="S1.I1.i4.p1.1" class="ltx_p">Through extensive experimental evaluations, we show that our method achieves promising results compared to the relevant baselines.</p>
</div>
</li>
</ul>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related work</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p">With the growing applications of neural networks, learning from pretrained weights is becoming increasingly important for speeding up the transfer learning process. Learning to generate network weights from existing modelzoos without going through the traditional training pipeline has potential to increase model efficiency. In this regard, several hypernetwork approaches have been proposed for weight prediction <cite class="ltx_cite ltx_citemacro_citep">(Chauhan et al., <a href="#bib.bib3" title="" class="ltx_ref">2023</a>; Ratzlaff &amp; Fuxin, <a href="#bib.bib23" title="" class="ltx_ref">2020</a>; Denil et al., <a href="#bib.bib5" title="" class="ltx_ref">2013</a>; Ha et al., <a href="#bib.bib8" title="" class="ltx_ref">2016</a>)</cite>.</p>
</div>
<div id="S2.p2" class="ltx_para">
<p id="S2.p2.1" class="ltx_p"><span id="S2.p2.1.1" class="ltx_text ltx_font_bold">Deterministic Parameters Generation</span>:
Recently <cite class="ltx_cite ltx_citemacro_citet">Zhang et al. (<a href="#bib.bib29" title="" class="ltx_ref">2019</a>)</cite> proposed Graph Hypernetwork (GHN) for generating weights for an architecture using its directed graph representation as input. This idea was subsequently improved by <cite class="ltx_cite ltx_citemacro_citet">Knyazev et al. (<a href="#bib.bib14" title="" class="ltx_ref">2021</a>)</cite> in GHN2 focusing on generation across architectures for the same datasets. Analogously, <cite class="ltx_cite ltx_citemacro_citet">Zhmoginov et al. (<a href="#bib.bib30" title="" class="ltx_ref">2022</a>)</cite> consider weight generation as an autoregressive process and used a transformer architecture to sequentially generate weights per layer. However, this method is less scalable since it requires a transformer module per layer. Similarly, <cite class="ltx_cite ltx_citemacro_citet">Knyazev et al. (<a href="#bib.bib15" title="" class="ltx_ref">2023</a>)</cite> combined ideas from transformer based approach and GHN2 to improve the generalization of graph hyper networks across datasets and architectures in a new method called GHN3. Although, GHN3 achieve promising results, the sampling is deterministic, and the generalization depends solely on the generalization of the training set of architectures and pretrained dataset.</p>
</div>
<div id="S2.p3" class="ltx_para">
<p id="S2.p3.1" class="ltx_p"><span id="S2.p3.1.1" class="ltx_text ltx_font_bold">Meta Pretrained Weight Generators</span>: <cite class="ltx_cite ltx_citemacro_citet">Nava et al. (<a href="#bib.bib19" title="" class="ltx_ref">2023</a>)</cite> proposed HyperLDM; a generative model for weights generation for visual question answering tasks from distribution of weights pretrained in a meta-learning setting and sampling with latent diffusion. Although generating pretrained weights through meta training is promising, the meta learning process can be computationally expensive, moreover, the meta pretrained weights are not optimal for in-distribution weights generation.</p>
</div>
<div id="S2.p4" class="ltx_para">
<p id="S2.p4.1" class="ltx_p"><span id="S2.p4.1.1" class="ltx_text ltx_font_bold">AutoEncoder-based Weight Generators</span>: <cite class="ltx_cite ltx_citemacro_citet">Schürholt et al. (<a href="#bib.bib27" title="" class="ltx_ref">2021</a>)</cite> proposed to learn the distribution of weights by learning to reconstruct them using auto-encoder style architectures. A follow-up work, <cite class="ltx_cite ltx_citemacro_citet">Schürholt et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022a</a>)</cite>, proposed to learn the distribution of pretrained weights enabling unconditional sampling of diverse weights using kernel density estimation. A similar idea is developed by <cite class="ltx_cite ltx_citemacro_citet">Peebles et al. (<a href="#bib.bib21" title="" class="ltx_ref">2022</a>)</cite> where the weight generation is conditioned by the target loss in a diffusion transformer framework.</p>
</div>
<div id="S2.p5" class="ltx_para">
<p id="S2.p5.1" class="ltx_p"><span id="S2.p5.1.1" class="ltx_text ltx_font_bold">Diffusion Models</span>: Denoising Diffusion Probabilistic Models (DDPM) <cite class="ltx_cite ltx_citemacro_citep">(Ho et al., <a href="#bib.bib11" title="" class="ltx_ref">2020</a>; Rombach et al., <a href="#bib.bib24" title="" class="ltx_ref">2021</a>; Blattmann et al., <a href="#bib.bib1" title="" class="ltx_ref">2022</a>; Croitoru et al., <a href="#bib.bib4" title="" class="ltx_ref">2022</a>)</cite> enable mapping data representations to a Gaussian distribution and vice versa achieving state-of-the art performance in generative modeling. Latent diffusion models <cite class="ltx_cite ltx_citemacro_citep">(Dhariwal &amp; Nichol, <a href="#bib.bib6" title="" class="ltx_ref">2021</a>; Nichol &amp; Dhariwal, <a href="#bib.bib20" title="" class="ltx_ref">2021</a>; Ho &amp; Salimans, <a href="#bib.bib10" title="" class="ltx_ref">2021</a>; Zhu et al., <a href="#bib.bib31" title="" class="ltx_ref">2023</a>; Rombach et al., <a href="#bib.bib24" title="" class="ltx_ref">2021</a>)</cite> allow for manipulation of the learned latent space. We utilize such models for our dataset-conditioned weight sampling.</p>
</div>
<div id="S2.p6" class="ltx_para">
<p id="S2.p6.1" class="ltx_p">Our work combines components from each of the methods discussed sofar allowing for dataset-conditioned neural network weight generation for both seen and unseen datasets.</p>
</div>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Approach</h2>

<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Preliminaries</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.6" class="ltx_p"><span id="S3.SS1.p1.6.1" class="ltx_text ltx_font_bold">Problem Definition:</span>
In this work, we are interested in the following problem: Given a set of neural network models of the same architecture
pretrained on a set of <math id="S3.SS1.p1.1.m1.1" class="ltx_Math" alttext="M" display="inline"><semantics id="S3.SS1.p1.1.m1.1a"><mi id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml">M</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.1b"><ci id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">𝑀</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.1c">M</annotation></semantics></math> datasets <math id="S3.SS1.p1.2.m2.4" class="ltx_Math" alttext="\{\mathfrak{D}_{1},\mathfrak{D}_{2},...,\mathfrak{D}_{M}\}" display="inline"><semantics id="S3.SS1.p1.2.m2.4a"><mrow id="S3.SS1.p1.2.m2.4.4.3" xref="S3.SS1.p1.2.m2.4.4.4.cmml"><mo stretchy="false" id="S3.SS1.p1.2.m2.4.4.3.4" xref="S3.SS1.p1.2.m2.4.4.4.cmml">{</mo><msub id="S3.SS1.p1.2.m2.2.2.1.1" xref="S3.SS1.p1.2.m2.2.2.1.1.cmml"><mi id="S3.SS1.p1.2.m2.2.2.1.1.2" xref="S3.SS1.p1.2.m2.2.2.1.1.2.cmml">𝔇</mi><mn id="S3.SS1.p1.2.m2.2.2.1.1.3" xref="S3.SS1.p1.2.m2.2.2.1.1.3.cmml">1</mn></msub><mo id="S3.SS1.p1.2.m2.4.4.3.5" xref="S3.SS1.p1.2.m2.4.4.4.cmml">,</mo><msub id="S3.SS1.p1.2.m2.3.3.2.2" xref="S3.SS1.p1.2.m2.3.3.2.2.cmml"><mi id="S3.SS1.p1.2.m2.3.3.2.2.2" xref="S3.SS1.p1.2.m2.3.3.2.2.2.cmml">𝔇</mi><mn id="S3.SS1.p1.2.m2.3.3.2.2.3" xref="S3.SS1.p1.2.m2.3.3.2.2.3.cmml">2</mn></msub><mo id="S3.SS1.p1.2.m2.4.4.3.6" xref="S3.SS1.p1.2.m2.4.4.4.cmml">,</mo><mi mathvariant="normal" id="S3.SS1.p1.2.m2.1.1" xref="S3.SS1.p1.2.m2.1.1.cmml">…</mi><mo id="S3.SS1.p1.2.m2.4.4.3.7" xref="S3.SS1.p1.2.m2.4.4.4.cmml">,</mo><msub id="S3.SS1.p1.2.m2.4.4.3.3" xref="S3.SS1.p1.2.m2.4.4.3.3.cmml"><mi id="S3.SS1.p1.2.m2.4.4.3.3.2" xref="S3.SS1.p1.2.m2.4.4.3.3.2.cmml">𝔇</mi><mi id="S3.SS1.p1.2.m2.4.4.3.3.3" xref="S3.SS1.p1.2.m2.4.4.3.3.3.cmml">M</mi></msub><mo stretchy="false" id="S3.SS1.p1.2.m2.4.4.3.8" xref="S3.SS1.p1.2.m2.4.4.4.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m2.4b"><set id="S3.SS1.p1.2.m2.4.4.4.cmml" xref="S3.SS1.p1.2.m2.4.4.3"><apply id="S3.SS1.p1.2.m2.2.2.1.1.cmml" xref="S3.SS1.p1.2.m2.2.2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.2.m2.2.2.1.1.1.cmml" xref="S3.SS1.p1.2.m2.2.2.1.1">subscript</csymbol><ci id="S3.SS1.p1.2.m2.2.2.1.1.2.cmml" xref="S3.SS1.p1.2.m2.2.2.1.1.2">𝔇</ci><cn type="integer" id="S3.SS1.p1.2.m2.2.2.1.1.3.cmml" xref="S3.SS1.p1.2.m2.2.2.1.1.3">1</cn></apply><apply id="S3.SS1.p1.2.m2.3.3.2.2.cmml" xref="S3.SS1.p1.2.m2.3.3.2.2"><csymbol cd="ambiguous" id="S3.SS1.p1.2.m2.3.3.2.2.1.cmml" xref="S3.SS1.p1.2.m2.3.3.2.2">subscript</csymbol><ci id="S3.SS1.p1.2.m2.3.3.2.2.2.cmml" xref="S3.SS1.p1.2.m2.3.3.2.2.2">𝔇</ci><cn type="integer" id="S3.SS1.p1.2.m2.3.3.2.2.3.cmml" xref="S3.SS1.p1.2.m2.3.3.2.2.3">2</cn></apply><ci id="S3.SS1.p1.2.m2.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1">…</ci><apply id="S3.SS1.p1.2.m2.4.4.3.3.cmml" xref="S3.SS1.p1.2.m2.4.4.3.3"><csymbol cd="ambiguous" id="S3.SS1.p1.2.m2.4.4.3.3.1.cmml" xref="S3.SS1.p1.2.m2.4.4.3.3">subscript</csymbol><ci id="S3.SS1.p1.2.m2.4.4.3.3.2.cmml" xref="S3.SS1.p1.2.m2.4.4.3.3.2">𝔇</ci><ci id="S3.SS1.p1.2.m2.4.4.3.3.3.cmml" xref="S3.SS1.p1.2.m2.4.4.3.3.3">𝑀</ci></apply></set></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m2.4c">\{\mathfrak{D}_{1},\mathfrak{D}_{2},...,\mathfrak{D}_{M}\}</annotation></semantics></math>, we want to learn the distribution <math id="S3.SS1.p1.3.m3.1" class="ltx_Math" alttext="p(W)" display="inline"><semantics id="S3.SS1.p1.3.m3.1a"><mrow id="S3.SS1.p1.3.m3.1.2" xref="S3.SS1.p1.3.m3.1.2.cmml"><mi id="S3.SS1.p1.3.m3.1.2.2" xref="S3.SS1.p1.3.m3.1.2.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.3.m3.1.2.1" xref="S3.SS1.p1.3.m3.1.2.1.cmml">​</mo><mrow id="S3.SS1.p1.3.m3.1.2.3.2" xref="S3.SS1.p1.3.m3.1.2.cmml"><mo stretchy="false" id="S3.SS1.p1.3.m3.1.2.3.2.1" xref="S3.SS1.p1.3.m3.1.2.cmml">(</mo><mi id="S3.SS1.p1.3.m3.1.1" xref="S3.SS1.p1.3.m3.1.1.cmml">W</mi><mo stretchy="false" id="S3.SS1.p1.3.m3.1.2.3.2.2" xref="S3.SS1.p1.3.m3.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.3.m3.1b"><apply id="S3.SS1.p1.3.m3.1.2.cmml" xref="S3.SS1.p1.3.m3.1.2"><times id="S3.SS1.p1.3.m3.1.2.1.cmml" xref="S3.SS1.p1.3.m3.1.2.1"></times><ci id="S3.SS1.p1.3.m3.1.2.2.cmml" xref="S3.SS1.p1.3.m3.1.2.2">𝑝</ci><ci id="S3.SS1.p1.3.m3.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1">𝑊</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.3.m3.1c">p(W)</annotation></semantics></math> of the pretrained weights (<math id="S3.SS1.p1.4.m4.1" class="ltx_Math" alttext="W" display="inline"><semantics id="S3.SS1.p1.4.m4.1a"><mi id="S3.SS1.p1.4.m4.1.1" xref="S3.SS1.p1.4.m4.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.4.m4.1b"><ci id="S3.SS1.p1.4.m4.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.4.m4.1c">W</annotation></semantics></math>) of those models such that for a new dataset <math id="S3.SS1.p1.5.m5.2" class="ltx_Math" alttext="\mathfrak{D}_{new}(x,y)" display="inline"><semantics id="S3.SS1.p1.5.m5.2a"><mrow id="S3.SS1.p1.5.m5.2.3" xref="S3.SS1.p1.5.m5.2.3.cmml"><msub id="S3.SS1.p1.5.m5.2.3.2" xref="S3.SS1.p1.5.m5.2.3.2.cmml"><mi id="S3.SS1.p1.5.m5.2.3.2.2" xref="S3.SS1.p1.5.m5.2.3.2.2.cmml">𝔇</mi><mrow id="S3.SS1.p1.5.m5.2.3.2.3" xref="S3.SS1.p1.5.m5.2.3.2.3.cmml"><mi id="S3.SS1.p1.5.m5.2.3.2.3.2" xref="S3.SS1.p1.5.m5.2.3.2.3.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.5.m5.2.3.2.3.1" xref="S3.SS1.p1.5.m5.2.3.2.3.1.cmml">​</mo><mi id="S3.SS1.p1.5.m5.2.3.2.3.3" xref="S3.SS1.p1.5.m5.2.3.2.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.5.m5.2.3.2.3.1a" xref="S3.SS1.p1.5.m5.2.3.2.3.1.cmml">​</mo><mi id="S3.SS1.p1.5.m5.2.3.2.3.4" xref="S3.SS1.p1.5.m5.2.3.2.3.4.cmml">w</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S3.SS1.p1.5.m5.2.3.1" xref="S3.SS1.p1.5.m5.2.3.1.cmml">​</mo><mrow id="S3.SS1.p1.5.m5.2.3.3.2" xref="S3.SS1.p1.5.m5.2.3.3.1.cmml"><mo stretchy="false" id="S3.SS1.p1.5.m5.2.3.3.2.1" xref="S3.SS1.p1.5.m5.2.3.3.1.cmml">(</mo><mi id="S3.SS1.p1.5.m5.1.1" xref="S3.SS1.p1.5.m5.1.1.cmml">x</mi><mo id="S3.SS1.p1.5.m5.2.3.3.2.2" xref="S3.SS1.p1.5.m5.2.3.3.1.cmml">,</mo><mi id="S3.SS1.p1.5.m5.2.2" xref="S3.SS1.p1.5.m5.2.2.cmml">y</mi><mo stretchy="false" id="S3.SS1.p1.5.m5.2.3.3.2.3" xref="S3.SS1.p1.5.m5.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.5.m5.2b"><apply id="S3.SS1.p1.5.m5.2.3.cmml" xref="S3.SS1.p1.5.m5.2.3"><times id="S3.SS1.p1.5.m5.2.3.1.cmml" xref="S3.SS1.p1.5.m5.2.3.1"></times><apply id="S3.SS1.p1.5.m5.2.3.2.cmml" xref="S3.SS1.p1.5.m5.2.3.2"><csymbol cd="ambiguous" id="S3.SS1.p1.5.m5.2.3.2.1.cmml" xref="S3.SS1.p1.5.m5.2.3.2">subscript</csymbol><ci id="S3.SS1.p1.5.m5.2.3.2.2.cmml" xref="S3.SS1.p1.5.m5.2.3.2.2">𝔇</ci><apply id="S3.SS1.p1.5.m5.2.3.2.3.cmml" xref="S3.SS1.p1.5.m5.2.3.2.3"><times id="S3.SS1.p1.5.m5.2.3.2.3.1.cmml" xref="S3.SS1.p1.5.m5.2.3.2.3.1"></times><ci id="S3.SS1.p1.5.m5.2.3.2.3.2.cmml" xref="S3.SS1.p1.5.m5.2.3.2.3.2">𝑛</ci><ci id="S3.SS1.p1.5.m5.2.3.2.3.3.cmml" xref="S3.SS1.p1.5.m5.2.3.2.3.3">𝑒</ci><ci id="S3.SS1.p1.5.m5.2.3.2.3.4.cmml" xref="S3.SS1.p1.5.m5.2.3.2.3.4">𝑤</ci></apply></apply><interval closure="open" id="S3.SS1.p1.5.m5.2.3.3.1.cmml" xref="S3.SS1.p1.5.m5.2.3.3.2"><ci id="S3.SS1.p1.5.m5.1.1.cmml" xref="S3.SS1.p1.5.m5.1.1">𝑥</ci><ci id="S3.SS1.p1.5.m5.2.2.cmml" xref="S3.SS1.p1.5.m5.2.2">𝑦</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.5.m5.2c">\mathfrak{D}_{new}(x,y)</annotation></semantics></math> we can conditionally sample a set of weights with <math id="S3.SS1.p1.6.m6.1" class="ltx_Math" alttext="p(W_{new}|\mathfrak{D}_{new})" display="inline"><semantics id="S3.SS1.p1.6.m6.1a"><mrow id="S3.SS1.p1.6.m6.1.1" xref="S3.SS1.p1.6.m6.1.1.cmml"><mi id="S3.SS1.p1.6.m6.1.1.3" xref="S3.SS1.p1.6.m6.1.1.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.6.m6.1.1.2" xref="S3.SS1.p1.6.m6.1.1.2.cmml">​</mo><mrow id="S3.SS1.p1.6.m6.1.1.1.1" xref="S3.SS1.p1.6.m6.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p1.6.m6.1.1.1.1.2" xref="S3.SS1.p1.6.m6.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS1.p1.6.m6.1.1.1.1.1" xref="S3.SS1.p1.6.m6.1.1.1.1.1.cmml"><msub id="S3.SS1.p1.6.m6.1.1.1.1.1.2" xref="S3.SS1.p1.6.m6.1.1.1.1.1.2.cmml"><mi id="S3.SS1.p1.6.m6.1.1.1.1.1.2.2" xref="S3.SS1.p1.6.m6.1.1.1.1.1.2.2.cmml">W</mi><mrow id="S3.SS1.p1.6.m6.1.1.1.1.1.2.3" xref="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.cmml"><mi id="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.2" xref="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.1" xref="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.1.cmml">​</mo><mi id="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.3" xref="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.1a" xref="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.1.cmml">​</mo><mi id="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.4" xref="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.4.cmml">w</mi></mrow></msub><mo fence="false" id="S3.SS1.p1.6.m6.1.1.1.1.1.1" xref="S3.SS1.p1.6.m6.1.1.1.1.1.1.cmml">|</mo><msub id="S3.SS1.p1.6.m6.1.1.1.1.1.3" xref="S3.SS1.p1.6.m6.1.1.1.1.1.3.cmml"><mi id="S3.SS1.p1.6.m6.1.1.1.1.1.3.2" xref="S3.SS1.p1.6.m6.1.1.1.1.1.3.2.cmml">𝔇</mi><mrow id="S3.SS1.p1.6.m6.1.1.1.1.1.3.3" xref="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.cmml"><mi id="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.2" xref="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.1" xref="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.3" xref="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.1a" xref="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.4" xref="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.4.cmml">w</mi></mrow></msub></mrow><mo stretchy="false" id="S3.SS1.p1.6.m6.1.1.1.1.3" xref="S3.SS1.p1.6.m6.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.6.m6.1b"><apply id="S3.SS1.p1.6.m6.1.1.cmml" xref="S3.SS1.p1.6.m6.1.1"><times id="S3.SS1.p1.6.m6.1.1.2.cmml" xref="S3.SS1.p1.6.m6.1.1.2"></times><ci id="S3.SS1.p1.6.m6.1.1.3.cmml" xref="S3.SS1.p1.6.m6.1.1.3">𝑝</ci><apply id="S3.SS1.p1.6.m6.1.1.1.1.1.cmml" xref="S3.SS1.p1.6.m6.1.1.1.1"><csymbol cd="latexml" id="S3.SS1.p1.6.m6.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.6.m6.1.1.1.1.1.1">conditional</csymbol><apply id="S3.SS1.p1.6.m6.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.6.m6.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p1.6.m6.1.1.1.1.1.2.1.cmml" xref="S3.SS1.p1.6.m6.1.1.1.1.1.2">subscript</csymbol><ci id="S3.SS1.p1.6.m6.1.1.1.1.1.2.2.cmml" xref="S3.SS1.p1.6.m6.1.1.1.1.1.2.2">𝑊</ci><apply id="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.cmml" xref="S3.SS1.p1.6.m6.1.1.1.1.1.2.3"><times id="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.1.cmml" xref="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.1"></times><ci id="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.2.cmml" xref="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.2">𝑛</ci><ci id="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.3.cmml" xref="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.3">𝑒</ci><ci id="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.4.cmml" xref="S3.SS1.p1.6.m6.1.1.1.1.1.2.3.4">𝑤</ci></apply></apply><apply id="S3.SS1.p1.6.m6.1.1.1.1.1.3.cmml" xref="S3.SS1.p1.6.m6.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.6.m6.1.1.1.1.1.3.1.cmml" xref="S3.SS1.p1.6.m6.1.1.1.1.1.3">subscript</csymbol><ci id="S3.SS1.p1.6.m6.1.1.1.1.1.3.2.cmml" xref="S3.SS1.p1.6.m6.1.1.1.1.1.3.2">𝔇</ci><apply id="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.cmml" xref="S3.SS1.p1.6.m6.1.1.1.1.1.3.3"><times id="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.1.cmml" xref="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.1"></times><ci id="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.2.cmml" xref="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.2">𝑛</ci><ci id="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.3.cmml" xref="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.3">𝑒</ci><ci id="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.4.cmml" xref="S3.SS1.p1.6.m6.1.1.1.1.1.3.3.4">𝑤</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.6.m6.1c">p(W_{new}|\mathfrak{D}_{new})</annotation></semantics></math> that will achieve good performance on the new dataset without training or with few optimization steps.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.1" class="ltx_p"><span id="S3.SS1.p2.1.1" class="ltx_text ltx_font_bold">High-Level Intuition:</span>
Our approach uses a diffusion model in order to generate a given neural architecture’s weights for previously unseen datasets.
In order to train our model, we use a modelzoo of trained weights from various datasets. The diffusion model then learns from that data to predict new weights conditioned on a given dataset.</p>
</div>
<div id="S3.SS1.p3" class="ltx_para">
<p id="S3.SS1.p3.1" class="ltx_p">Intuitively, based on the training data, the diffusion model knows what trained networks look like and can relate this information to the characteristics of the dataset the network was trained on.
At test time, for a previously unseen dataset we compute a dataset encoding and use this to guide the backward diffusion process to generate weights that mimic trained weights for this particular dataset.</p>
</div>
<div id="S3.SS1.p4" class="ltx_para">
<p id="S3.SS1.p4.1" class="ltx_p">This process exploits the facts that at test time we do have the dataset as input and that there is a clear relationship between a network’s trained weights and the dataset it was trained on (For a more formal argument of this relationship, please see Appendix <a href="#A3" title="Appendix C Relationship Between Datasets and Trained Weights ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>.) Understanding the distribution of the pretrained weights and their relation with the pretrained datasets will enable generating highly performing weights conditioned on the target dataset with or without a few optimization steps. Through the next sections we investigate the problem of learning this conditional distribution of pretrained weights.</p>
</div>
<div id="S3.SS1.p5" class="ltx_para">
<p id="S3.SS1.p5.1" class="ltx_p"><span id="S3.SS1.p5.1.1" class="ltx_text ltx_font_bold">Overview of Model:</span>
We start by describing the overall structure of our approach consisting of three main parts as shown in Figure <a href="#S1.F2" title="Figure 2 ‣ 1 Introduction ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>. After obtaining the set of pretrained models (ie. modelzoo), we extract the pretrained weights and vectorize them into a single vector per architecture checkpoint. These vectors are then paired with their corresponding pretrained datasets. We then train a VAE model to encode the vectors to learn latent representations of weights. This stage is denoted by Stage 1 in Figure <a href="#S1.F2" title="Figure 2 ‣ 1 Introduction ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>. In the intermediate stage we encode the dataset associated with each weight vector using a contrastive learning process denoted by the Stage 2 in Figure <a href="#S1.F2" title="Figure 2 ‣ 1 Introduction ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>. The final stage of the proposed method, denoted by Stage 3, consists of a UNet based diffusion model. Each stage is trained separately. During inference, we encode the target dataset and use its embeddings to conditionally generate new weights for the target datasets. In the subsequent sections, we will discuss these stages in turn in detail.</p>
</div>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Weight Encoding</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.6" class="ltx_p">Given a modelzoo of pretrained weights, we flatten the model parameters to obtain vectorized weights for each network. We then train a Variational
AutoEncoder (VAE) without any conditioning. The VAE is trained such that the decoded weights, when evaluated on the test set, achieve similar
performance compared to the input weights. Concretely, we minimize the following <a href="#S3.E1" title="In 3.2 Weight Encoding ‣ 3 Approach ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> objective:</p>
<table id="S3.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E1.m1.2" class="ltx_math_unparsed" alttext="\mathcal{L}=-\mathbb{E}_{q_{\phi}(z|x)}\left[\log p_{\theta}(x|z)\right]+\text{KL}\left[q_{\phi}(z|x)\,||\,p(z)\right]" display="block"><semantics id="S3.E1.m1.2a"><mrow id="S3.E1.m1.2b"><mi class="ltx_font_mathcaligraphic" id="S3.E1.m1.2.3">ℒ</mi><mo rspace="0em" id="S3.E1.m1.2.4">=</mo><mo lspace="0em" id="S3.E1.m1.2.5">−</mo><msub id="S3.E1.m1.2.6"><mi id="S3.E1.m1.2.6.2">𝔼</mi><mrow id="S3.E1.m1.1.1.1"><msub id="S3.E1.m1.1.1.1.3"><mi id="S3.E1.m1.1.1.1.3.2">q</mi><mi id="S3.E1.m1.1.1.1.3.3">ϕ</mi></msub><mo lspace="0em" rspace="0em" id="S3.E1.m1.1.1.1.2">​</mo><mrow id="S3.E1.m1.1.1.1.1.1"><mo stretchy="false" id="S3.E1.m1.1.1.1.1.1.2">(</mo><mrow id="S3.E1.m1.1.1.1.1.1.1"><mi id="S3.E1.m1.1.1.1.1.1.1.2">z</mi><mo fence="false" id="S3.E1.m1.1.1.1.1.1.1.1">|</mo><mi id="S3.E1.m1.1.1.1.1.1.1.3">x</mi></mrow><mo stretchy="false" id="S3.E1.m1.1.1.1.1.1.3">)</mo></mrow></mrow></msub><mrow id="S3.E1.m1.2.7"><mo id="S3.E1.m1.2.7.1">[</mo><mi id="S3.E1.m1.2.7.2">log</mi><msub id="S3.E1.m1.2.7.3"><mi id="S3.E1.m1.2.7.3.2">p</mi><mi id="S3.E1.m1.2.7.3.3">θ</mi></msub><mrow id="S3.E1.m1.2.7.4"><mo stretchy="false" id="S3.E1.m1.2.7.4.1">(</mo><mi id="S3.E1.m1.2.7.4.2">x</mi><mo fence="false" rspace="0.167em" stretchy="false" id="S3.E1.m1.2.7.4.3">|</mo><mi id="S3.E1.m1.2.7.4.4">z</mi><mo stretchy="false" id="S3.E1.m1.2.7.4.5">)</mo></mrow><mo id="S3.E1.m1.2.7.5">]</mo></mrow><mo id="S3.E1.m1.2.8">+</mo><mtext id="S3.E1.m1.2.9">KL</mtext><mrow id="S3.E1.m1.2.10"><mo id="S3.E1.m1.2.10.1">[</mo><msub id="S3.E1.m1.2.10.2"><mi id="S3.E1.m1.2.10.2.2">q</mi><mi id="S3.E1.m1.2.10.2.3">ϕ</mi></msub><mrow id="S3.E1.m1.2.10.3"><mo stretchy="false" id="S3.E1.m1.2.10.3.1">(</mo><mi id="S3.E1.m1.2.10.3.2">z</mi><mo fence="false" rspace="0.167em" stretchy="false" id="S3.E1.m1.2.10.3.3">|</mo><mi id="S3.E1.m1.2.10.3.4">x</mi><mo rspace="0.170em" stretchy="false" id="S3.E1.m1.2.10.3.5">)</mo></mrow><mo fence="false" rspace="0.167em" stretchy="false" id="S3.E1.m1.2.10.4">|</mo><mo fence="false" rspace="0.337em" stretchy="false" id="S3.E1.m1.2.10.5">|</mo><mi id="S3.E1.m1.2.10.6">p</mi><mrow id="S3.E1.m1.2.10.7"><mo stretchy="false" id="S3.E1.m1.2.10.7.1">(</mo><mi id="S3.E1.m1.2.2">z</mi><mo stretchy="false" id="S3.E1.m1.2.10.7.2">)</mo></mrow><mo id="S3.E1.m1.2.10.8">]</mo></mrow></mrow><annotation encoding="application/x-tex" id="S3.E1.m1.2c">\mathcal{L}=-\mathbb{E}_{q_{\phi}(z|x)}\left[\log p_{\theta}(x|z)\right]+\text{KL}\left[q_{\phi}(z|x)\,||\,p(z)\right]</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="S3.SS2.p1.5" class="ltx_p">where <math id="S3.SS2.p1.1.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S3.SS2.p1.1.m1.1a"><mi id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.1b"><ci id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.1c">x</annotation></semantics></math> is the vectorized weight, <math id="S3.SS2.p1.2.m2.1" class="ltx_Math" alttext="z" display="inline"><semantics id="S3.SS2.p1.2.m2.1a"><mi id="S3.SS2.p1.2.m2.1.1" xref="S3.SS2.p1.2.m2.1.1.cmml">z</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.2.m2.1b"><ci id="S3.SS2.p1.2.m2.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1">𝑧</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.2.m2.1c">z</annotation></semantics></math> it’s latent representation, <math id="S3.SS2.p1.3.m3.1" class="ltx_Math" alttext="p_{\theta}" display="inline"><semantics id="S3.SS2.p1.3.m3.1a"><msub id="S3.SS2.p1.3.m3.1.1" xref="S3.SS2.p1.3.m3.1.1.cmml"><mi id="S3.SS2.p1.3.m3.1.1.2" xref="S3.SS2.p1.3.m3.1.1.2.cmml">p</mi><mi id="S3.SS2.p1.3.m3.1.1.3" xref="S3.SS2.p1.3.m3.1.1.3.cmml">θ</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.3.m3.1b"><apply id="S3.SS2.p1.3.m3.1.1.cmml" xref="S3.SS2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.3.m3.1.1.1.cmml" xref="S3.SS2.p1.3.m3.1.1">subscript</csymbol><ci id="S3.SS2.p1.3.m3.1.1.2.cmml" xref="S3.SS2.p1.3.m3.1.1.2">𝑝</ci><ci id="S3.SS2.p1.3.m3.1.1.3.cmml" xref="S3.SS2.p1.3.m3.1.1.3">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.3.m3.1c">p_{\theta}</annotation></semantics></math> and <math id="S3.SS2.p1.4.m4.1" class="ltx_Math" alttext="q_{\phi}" display="inline"><semantics id="S3.SS2.p1.4.m4.1a"><msub id="S3.SS2.p1.4.m4.1.1" xref="S3.SS2.p1.4.m4.1.1.cmml"><mi id="S3.SS2.p1.4.m4.1.1.2" xref="S3.SS2.p1.4.m4.1.1.2.cmml">q</mi><mi id="S3.SS2.p1.4.m4.1.1.3" xref="S3.SS2.p1.4.m4.1.1.3.cmml">ϕ</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.4.m4.1b"><apply id="S3.SS2.p1.4.m4.1.1.cmml" xref="S3.SS2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.4.m4.1.1.1.cmml" xref="S3.SS2.p1.4.m4.1.1">subscript</csymbol><ci id="S3.SS2.p1.4.m4.1.1.2.cmml" xref="S3.SS2.p1.4.m4.1.1.2">𝑞</ci><ci id="S3.SS2.p1.4.m4.1.1.3.cmml" xref="S3.SS2.p1.4.m4.1.1.3">italic-ϕ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.4.m4.1c">q_{\phi}</annotation></semantics></math> the reconstruction and approximate posterior terms
respectively, and <math id="S3.SS2.p1.5.m5.1" class="ltx_Math" alttext="p(z)" display="inline"><semantics id="S3.SS2.p1.5.m5.1a"><mrow id="S3.SS2.p1.5.m5.1.2" xref="S3.SS2.p1.5.m5.1.2.cmml"><mi id="S3.SS2.p1.5.m5.1.2.2" xref="S3.SS2.p1.5.m5.1.2.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.5.m5.1.2.1" xref="S3.SS2.p1.5.m5.1.2.1.cmml">​</mo><mrow id="S3.SS2.p1.5.m5.1.2.3.2" xref="S3.SS2.p1.5.m5.1.2.cmml"><mo stretchy="false" id="S3.SS2.p1.5.m5.1.2.3.2.1" xref="S3.SS2.p1.5.m5.1.2.cmml">(</mo><mi id="S3.SS2.p1.5.m5.1.1" xref="S3.SS2.p1.5.m5.1.1.cmml">z</mi><mo stretchy="false" id="S3.SS2.p1.5.m5.1.2.3.2.2" xref="S3.SS2.p1.5.m5.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.5.m5.1b"><apply id="S3.SS2.p1.5.m5.1.2.cmml" xref="S3.SS2.p1.5.m5.1.2"><times id="S3.SS2.p1.5.m5.1.2.1.cmml" xref="S3.SS2.p1.5.m5.1.2.1"></times><ci id="S3.SS2.p1.5.m5.1.2.2.cmml" xref="S3.SS2.p1.5.m5.1.2.2">𝑝</ci><ci id="S3.SS2.p1.5.m5.1.1.cmml" xref="S3.SS2.p1.5.m5.1.1">𝑧</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.5.m5.1c">p(z)</annotation></semantics></math> the prior distribution. For the prior, we use the standard Gaussian with zero mean and unit variance. The trained VAE will later be used for training a latent diffusion model for dataset-conditioned weight sampling.</p>
</div>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Dataset Encoding</h3>

<div id="S3.SS3.p1" class="ltx_para">
<p id="S3.SS3.p1.1" class="ltx_p">To be able to sample weights conditioned on a dataset, we require an efficient mapping between the latent representations of the pretrained weights and their corresponding datasets (ie. the datasets the networks were trained on). However, efficiently encoding an entire dataset with multiple classes and hundreds, even millions, of samples per class is a challenging task. One way to encode such datasets is to utilize dataset condensation techniques which consists of training a network to synthesize one example per classes, combining the class-wise condensations and using it as a representation for the dataset. However this process requires performing the same process for every test dataset increasing the computation cost of the model. Hence we explore existing dataset-adaptive Neural Architecture Search approaches <cite class="ltx_cite ltx_citemacro_citet">Jeong et al. (<a href="#bib.bib13" title="" class="ltx_ref">2021</a>); Lee et al. (<a href="#bib.bib16" title="" class="ltx_ref">2021</a>)</cite> where Set Transformer <cite class="ltx_cite ltx_citemacro_citep">(Lee et al., <a href="#bib.bib17" title="" class="ltx_ref">2019</a>)</cite> has been demonstrated as an efficient model for dataset encoding.</p>
</div>
<div id="S3.SS3.p2" class="ltx_para">
<p id="S3.SS3.p2.24" class="ltx_p"><span id="S3.SS3.p2.24.1" class="ltx_text ltx_font_bold">Set-based Dataset Encoding:</span> Given a dataset <math id="S3.SS3.p2.1.m1.2" class="ltx_Math" alttext="\mathfrak{D}=\{x_{i},y_{i}\}_{i=1}^{N}" display="inline"><semantics id="S3.SS3.p2.1.m1.2a"><mrow id="S3.SS3.p2.1.m1.2.2" xref="S3.SS3.p2.1.m1.2.2.cmml"><mi id="S3.SS3.p2.1.m1.2.2.4" xref="S3.SS3.p2.1.m1.2.2.4.cmml">𝔇</mi><mo id="S3.SS3.p2.1.m1.2.2.3" xref="S3.SS3.p2.1.m1.2.2.3.cmml">=</mo><msubsup id="S3.SS3.p2.1.m1.2.2.2" xref="S3.SS3.p2.1.m1.2.2.2.cmml"><mrow id="S3.SS3.p2.1.m1.2.2.2.2.2.2" xref="S3.SS3.p2.1.m1.2.2.2.2.2.3.cmml"><mo stretchy="false" id="S3.SS3.p2.1.m1.2.2.2.2.2.2.3" xref="S3.SS3.p2.1.m1.2.2.2.2.2.3.cmml">{</mo><msub id="S3.SS3.p2.1.m1.1.1.1.1.1.1.1" xref="S3.SS3.p2.1.m1.1.1.1.1.1.1.1.cmml"><mi id="S3.SS3.p2.1.m1.1.1.1.1.1.1.1.2" xref="S3.SS3.p2.1.m1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S3.SS3.p2.1.m1.1.1.1.1.1.1.1.3" xref="S3.SS3.p2.1.m1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S3.SS3.p2.1.m1.2.2.2.2.2.2.4" xref="S3.SS3.p2.1.m1.2.2.2.2.2.3.cmml">,</mo><msub id="S3.SS3.p2.1.m1.2.2.2.2.2.2.2" xref="S3.SS3.p2.1.m1.2.2.2.2.2.2.2.cmml"><mi id="S3.SS3.p2.1.m1.2.2.2.2.2.2.2.2" xref="S3.SS3.p2.1.m1.2.2.2.2.2.2.2.2.cmml">y</mi><mi id="S3.SS3.p2.1.m1.2.2.2.2.2.2.2.3" xref="S3.SS3.p2.1.m1.2.2.2.2.2.2.2.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS3.p2.1.m1.2.2.2.2.2.2.5" xref="S3.SS3.p2.1.m1.2.2.2.2.2.3.cmml">}</mo></mrow><mrow id="S3.SS3.p2.1.m1.2.2.2.2.4" xref="S3.SS3.p2.1.m1.2.2.2.2.4.cmml"><mi id="S3.SS3.p2.1.m1.2.2.2.2.4.2" xref="S3.SS3.p2.1.m1.2.2.2.2.4.2.cmml">i</mi><mo id="S3.SS3.p2.1.m1.2.2.2.2.4.1" xref="S3.SS3.p2.1.m1.2.2.2.2.4.1.cmml">=</mo><mn id="S3.SS3.p2.1.m1.2.2.2.2.4.3" xref="S3.SS3.p2.1.m1.2.2.2.2.4.3.cmml">1</mn></mrow><mi id="S3.SS3.p2.1.m1.2.2.2.4" xref="S3.SS3.p2.1.m1.2.2.2.4.cmml">N</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.1.m1.2b"><apply id="S3.SS3.p2.1.m1.2.2.cmml" xref="S3.SS3.p2.1.m1.2.2"><eq id="S3.SS3.p2.1.m1.2.2.3.cmml" xref="S3.SS3.p2.1.m1.2.2.3"></eq><ci id="S3.SS3.p2.1.m1.2.2.4.cmml" xref="S3.SS3.p2.1.m1.2.2.4">𝔇</ci><apply id="S3.SS3.p2.1.m1.2.2.2.cmml" xref="S3.SS3.p2.1.m1.2.2.2"><csymbol cd="ambiguous" id="S3.SS3.p2.1.m1.2.2.2.3.cmml" xref="S3.SS3.p2.1.m1.2.2.2">superscript</csymbol><apply id="S3.SS3.p2.1.m1.2.2.2.2.cmml" xref="S3.SS3.p2.1.m1.2.2.2"><csymbol cd="ambiguous" id="S3.SS3.p2.1.m1.2.2.2.2.3.cmml" xref="S3.SS3.p2.1.m1.2.2.2">subscript</csymbol><set id="S3.SS3.p2.1.m1.2.2.2.2.2.3.cmml" xref="S3.SS3.p2.1.m1.2.2.2.2.2.2"><apply id="S3.SS3.p2.1.m1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.1.m1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.1.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.1.m1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS3.p2.1.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS3.p2.1.m1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S3.SS3.p2.1.m1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.p2.1.m1.1.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="S3.SS3.p2.1.m1.2.2.2.2.2.2.2.cmml" xref="S3.SS3.p2.1.m1.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS3.p2.1.m1.2.2.2.2.2.2.2.1.cmml" xref="S3.SS3.p2.1.m1.2.2.2.2.2.2.2">subscript</csymbol><ci id="S3.SS3.p2.1.m1.2.2.2.2.2.2.2.2.cmml" xref="S3.SS3.p2.1.m1.2.2.2.2.2.2.2.2">𝑦</ci><ci id="S3.SS3.p2.1.m1.2.2.2.2.2.2.2.3.cmml" xref="S3.SS3.p2.1.m1.2.2.2.2.2.2.2.3">𝑖</ci></apply></set><apply id="S3.SS3.p2.1.m1.2.2.2.2.4.cmml" xref="S3.SS3.p2.1.m1.2.2.2.2.4"><eq id="S3.SS3.p2.1.m1.2.2.2.2.4.1.cmml" xref="S3.SS3.p2.1.m1.2.2.2.2.4.1"></eq><ci id="S3.SS3.p2.1.m1.2.2.2.2.4.2.cmml" xref="S3.SS3.p2.1.m1.2.2.2.2.4.2">𝑖</ci><cn type="integer" id="S3.SS3.p2.1.m1.2.2.2.2.4.3.cmml" xref="S3.SS3.p2.1.m1.2.2.2.2.4.3">1</cn></apply></apply><ci id="S3.SS3.p2.1.m1.2.2.2.4.cmml" xref="S3.SS3.p2.1.m1.2.2.2.4">𝑁</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.1.m1.2c">\mathfrak{D}=\{x_{i},y_{i}\}_{i=1}^{N}</annotation></semantics></math>, where <math id="S3.SS3.p2.2.m2.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S3.SS3.p2.2.m2.1a"><msub id="S3.SS3.p2.2.m2.1.1" xref="S3.SS3.p2.2.m2.1.1.cmml"><mi id="S3.SS3.p2.2.m2.1.1.2" xref="S3.SS3.p2.2.m2.1.1.2.cmml">x</mi><mi id="S3.SS3.p2.2.m2.1.1.3" xref="S3.SS3.p2.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.2.m2.1b"><apply id="S3.SS3.p2.2.m2.1.1.cmml" xref="S3.SS3.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.2.m2.1.1.1.cmml" xref="S3.SS3.p2.2.m2.1.1">subscript</csymbol><ci id="S3.SS3.p2.2.m2.1.1.2.cmml" xref="S3.SS3.p2.2.m2.1.1.2">𝑥</ci><ci id="S3.SS3.p2.2.m2.1.1.3.cmml" xref="S3.SS3.p2.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.2.m2.1c">x_{i}</annotation></semantics></math> and <math id="S3.SS3.p2.3.m3.1" class="ltx_Math" alttext="y_{i}" display="inline"><semantics id="S3.SS3.p2.3.m3.1a"><msub id="S3.SS3.p2.3.m3.1.1" xref="S3.SS3.p2.3.m3.1.1.cmml"><mi id="S3.SS3.p2.3.m3.1.1.2" xref="S3.SS3.p2.3.m3.1.1.2.cmml">y</mi><mi id="S3.SS3.p2.3.m3.1.1.3" xref="S3.SS3.p2.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.3.m3.1b"><apply id="S3.SS3.p2.3.m3.1.1.cmml" xref="S3.SS3.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.3.m3.1.1.1.cmml" xref="S3.SS3.p2.3.m3.1.1">subscript</csymbol><ci id="S3.SS3.p2.3.m3.1.1.2.cmml" xref="S3.SS3.p2.3.m3.1.1.2">𝑦</ci><ci id="S3.SS3.p2.3.m3.1.1.3.cmml" xref="S3.SS3.p2.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.3.m3.1c">y_{i}</annotation></semantics></math> are input-output pairs, we form subsets <math id="S3.SS3.p2.4.m4.1" class="ltx_Math" alttext="s_{i}=(x_{p}^{y_{i}})_{p=1}^{K_{i}}" display="inline"><semantics id="S3.SS3.p2.4.m4.1a"><mrow id="S3.SS3.p2.4.m4.1.1" xref="S3.SS3.p2.4.m4.1.1.cmml"><msub id="S3.SS3.p2.4.m4.1.1.3" xref="S3.SS3.p2.4.m4.1.1.3.cmml"><mi id="S3.SS3.p2.4.m4.1.1.3.2" xref="S3.SS3.p2.4.m4.1.1.3.2.cmml">s</mi><mi id="S3.SS3.p2.4.m4.1.1.3.3" xref="S3.SS3.p2.4.m4.1.1.3.3.cmml">i</mi></msub><mo id="S3.SS3.p2.4.m4.1.1.2" xref="S3.SS3.p2.4.m4.1.1.2.cmml">=</mo><msubsup id="S3.SS3.p2.4.m4.1.1.1" xref="S3.SS3.p2.4.m4.1.1.1.cmml"><mrow id="S3.SS3.p2.4.m4.1.1.1.1.1.1" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS3.p2.4.m4.1.1.1.1.1.1.2" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="S3.SS3.p2.4.m4.1.1.1.1.1.1.1" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.cmml"><mi id="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.2.2" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.2.3" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.2.3.cmml">p</mi><msub id="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.3" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.3.cmml"><mi id="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.3.2" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.3.2.cmml">y</mi><mi id="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.3.3" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.3.3.cmml">i</mi></msub></msubsup><mo stretchy="false" id="S3.SS3.p2.4.m4.1.1.1.1.1.1.3" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.cmml">)</mo></mrow><mrow id="S3.SS3.p2.4.m4.1.1.1.1.3" xref="S3.SS3.p2.4.m4.1.1.1.1.3.cmml"><mi id="S3.SS3.p2.4.m4.1.1.1.1.3.2" xref="S3.SS3.p2.4.m4.1.1.1.1.3.2.cmml">p</mi><mo id="S3.SS3.p2.4.m4.1.1.1.1.3.1" xref="S3.SS3.p2.4.m4.1.1.1.1.3.1.cmml">=</mo><mn id="S3.SS3.p2.4.m4.1.1.1.1.3.3" xref="S3.SS3.p2.4.m4.1.1.1.1.3.3.cmml">1</mn></mrow><msub id="S3.SS3.p2.4.m4.1.1.1.3" xref="S3.SS3.p2.4.m4.1.1.1.3.cmml"><mi id="S3.SS3.p2.4.m4.1.1.1.3.2" xref="S3.SS3.p2.4.m4.1.1.1.3.2.cmml">K</mi><mi id="S3.SS3.p2.4.m4.1.1.1.3.3" xref="S3.SS3.p2.4.m4.1.1.1.3.3.cmml">i</mi></msub></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.4.m4.1b"><apply id="S3.SS3.p2.4.m4.1.1.cmml" xref="S3.SS3.p2.4.m4.1.1"><eq id="S3.SS3.p2.4.m4.1.1.2.cmml" xref="S3.SS3.p2.4.m4.1.1.2"></eq><apply id="S3.SS3.p2.4.m4.1.1.3.cmml" xref="S3.SS3.p2.4.m4.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.1.1.3.1.cmml" xref="S3.SS3.p2.4.m4.1.1.3">subscript</csymbol><ci id="S3.SS3.p2.4.m4.1.1.3.2.cmml" xref="S3.SS3.p2.4.m4.1.1.3.2">𝑠</ci><ci id="S3.SS3.p2.4.m4.1.1.3.3.cmml" xref="S3.SS3.p2.4.m4.1.1.3.3">𝑖</ci></apply><apply id="S3.SS3.p2.4.m4.1.1.1.cmml" xref="S3.SS3.p2.4.m4.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.1.1.1.2.cmml" xref="S3.SS3.p2.4.m4.1.1.1">superscript</csymbol><apply id="S3.SS3.p2.4.m4.1.1.1.1.cmml" xref="S3.SS3.p2.4.m4.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.1.1.1.1.2.cmml" xref="S3.SS3.p2.4.m4.1.1.1">subscript</csymbol><apply id="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1">superscript</csymbol><apply id="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.2.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.2.1.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.2.2.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.2.3.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.2.3">𝑝</ci></apply><apply id="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.3.1.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.3.2.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.3.2">𝑦</ci><ci id="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.3.3.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.1.1.1.3.3">𝑖</ci></apply></apply><apply id="S3.SS3.p2.4.m4.1.1.1.1.3.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.3"><eq id="S3.SS3.p2.4.m4.1.1.1.1.3.1.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.3.1"></eq><ci id="S3.SS3.p2.4.m4.1.1.1.1.3.2.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.3.2">𝑝</ci><cn type="integer" id="S3.SS3.p2.4.m4.1.1.1.1.3.3.cmml" xref="S3.SS3.p2.4.m4.1.1.1.1.3.3">1</cn></apply></apply><apply id="S3.SS3.p2.4.m4.1.1.1.3.cmml" xref="S3.SS3.p2.4.m4.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p2.4.m4.1.1.1.3.1.cmml" xref="S3.SS3.p2.4.m4.1.1.1.3">subscript</csymbol><ci id="S3.SS3.p2.4.m4.1.1.1.3.2.cmml" xref="S3.SS3.p2.4.m4.1.1.1.3.2">𝐾</ci><ci id="S3.SS3.p2.4.m4.1.1.1.3.3.cmml" xref="S3.SS3.p2.4.m4.1.1.1.3.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.4.m4.1c">s_{i}=(x_{p}^{y_{i}})_{p=1}^{K_{i}}</annotation></semantics></math> with <math id="S3.SS3.p2.5.m5.1" class="ltx_Math" alttext="K_{i}=|s_{i}|" display="inline"><semantics id="S3.SS3.p2.5.m5.1a"><mrow id="S3.SS3.p2.5.m5.1.1" xref="S3.SS3.p2.5.m5.1.1.cmml"><msub id="S3.SS3.p2.5.m5.1.1.3" xref="S3.SS3.p2.5.m5.1.1.3.cmml"><mi id="S3.SS3.p2.5.m5.1.1.3.2" xref="S3.SS3.p2.5.m5.1.1.3.2.cmml">K</mi><mi id="S3.SS3.p2.5.m5.1.1.3.3" xref="S3.SS3.p2.5.m5.1.1.3.3.cmml">i</mi></msub><mo id="S3.SS3.p2.5.m5.1.1.2" xref="S3.SS3.p2.5.m5.1.1.2.cmml">=</mo><mrow id="S3.SS3.p2.5.m5.1.1.1.1" xref="S3.SS3.p2.5.m5.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS3.p2.5.m5.1.1.1.1.2" xref="S3.SS3.p2.5.m5.1.1.1.2.1.cmml">|</mo><msub id="S3.SS3.p2.5.m5.1.1.1.1.1" xref="S3.SS3.p2.5.m5.1.1.1.1.1.cmml"><mi id="S3.SS3.p2.5.m5.1.1.1.1.1.2" xref="S3.SS3.p2.5.m5.1.1.1.1.1.2.cmml">s</mi><mi id="S3.SS3.p2.5.m5.1.1.1.1.1.3" xref="S3.SS3.p2.5.m5.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS3.p2.5.m5.1.1.1.1.3" xref="S3.SS3.p2.5.m5.1.1.1.2.1.cmml">|</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.5.m5.1b"><apply id="S3.SS3.p2.5.m5.1.1.cmml" xref="S3.SS3.p2.5.m5.1.1"><eq id="S3.SS3.p2.5.m5.1.1.2.cmml" xref="S3.SS3.p2.5.m5.1.1.2"></eq><apply id="S3.SS3.p2.5.m5.1.1.3.cmml" xref="S3.SS3.p2.5.m5.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p2.5.m5.1.1.3.1.cmml" xref="S3.SS3.p2.5.m5.1.1.3">subscript</csymbol><ci id="S3.SS3.p2.5.m5.1.1.3.2.cmml" xref="S3.SS3.p2.5.m5.1.1.3.2">𝐾</ci><ci id="S3.SS3.p2.5.m5.1.1.3.3.cmml" xref="S3.SS3.p2.5.m5.1.1.3.3">𝑖</ci></apply><apply id="S3.SS3.p2.5.m5.1.1.1.2.cmml" xref="S3.SS3.p2.5.m5.1.1.1.1"><abs id="S3.SS3.p2.5.m5.1.1.1.2.1.cmml" xref="S3.SS3.p2.5.m5.1.1.1.1.2"></abs><apply id="S3.SS3.p2.5.m5.1.1.1.1.1.cmml" xref="S3.SS3.p2.5.m5.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.5.m5.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.5.m5.1.1.1.1.1">subscript</csymbol><ci id="S3.SS3.p2.5.m5.1.1.1.1.1.2.cmml" xref="S3.SS3.p2.5.m5.1.1.1.1.1.2">𝑠</ci><ci id="S3.SS3.p2.5.m5.1.1.1.1.1.3.cmml" xref="S3.SS3.p2.5.m5.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.5.m5.1c">K_{i}=|s_{i}|</annotation></semantics></math>, and <math id="S3.SS3.p2.6.m6.1" class="ltx_Math" alttext="{s_{i}}\in\mathbb{R}^{K_{i}\times d}" display="inline"><semantics id="S3.SS3.p2.6.m6.1a"><mrow id="S3.SS3.p2.6.m6.1.1" xref="S3.SS3.p2.6.m6.1.1.cmml"><msub id="S3.SS3.p2.6.m6.1.1.2" xref="S3.SS3.p2.6.m6.1.1.2.cmml"><mi id="S3.SS3.p2.6.m6.1.1.2.2" xref="S3.SS3.p2.6.m6.1.1.2.2.cmml">s</mi><mi id="S3.SS3.p2.6.m6.1.1.2.3" xref="S3.SS3.p2.6.m6.1.1.2.3.cmml">i</mi></msub><mo id="S3.SS3.p2.6.m6.1.1.1" xref="S3.SS3.p2.6.m6.1.1.1.cmml">∈</mo><msup id="S3.SS3.p2.6.m6.1.1.3" xref="S3.SS3.p2.6.m6.1.1.3.cmml"><mi id="S3.SS3.p2.6.m6.1.1.3.2" xref="S3.SS3.p2.6.m6.1.1.3.2.cmml">ℝ</mi><mrow id="S3.SS3.p2.6.m6.1.1.3.3" xref="S3.SS3.p2.6.m6.1.1.3.3.cmml"><msub id="S3.SS3.p2.6.m6.1.1.3.3.2" xref="S3.SS3.p2.6.m6.1.1.3.3.2.cmml"><mi id="S3.SS3.p2.6.m6.1.1.3.3.2.2" xref="S3.SS3.p2.6.m6.1.1.3.3.2.2.cmml">K</mi><mi id="S3.SS3.p2.6.m6.1.1.3.3.2.3" xref="S3.SS3.p2.6.m6.1.1.3.3.2.3.cmml">i</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S3.SS3.p2.6.m6.1.1.3.3.1" xref="S3.SS3.p2.6.m6.1.1.3.3.1.cmml">×</mo><mi id="S3.SS3.p2.6.m6.1.1.3.3.3" xref="S3.SS3.p2.6.m6.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.6.m6.1b"><apply id="S3.SS3.p2.6.m6.1.1.cmml" xref="S3.SS3.p2.6.m6.1.1"><in id="S3.SS3.p2.6.m6.1.1.1.cmml" xref="S3.SS3.p2.6.m6.1.1.1"></in><apply id="S3.SS3.p2.6.m6.1.1.2.cmml" xref="S3.SS3.p2.6.m6.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p2.6.m6.1.1.2.1.cmml" xref="S3.SS3.p2.6.m6.1.1.2">subscript</csymbol><ci id="S3.SS3.p2.6.m6.1.1.2.2.cmml" xref="S3.SS3.p2.6.m6.1.1.2.2">𝑠</ci><ci id="S3.SS3.p2.6.m6.1.1.2.3.cmml" xref="S3.SS3.p2.6.m6.1.1.2.3">𝑖</ci></apply><apply id="S3.SS3.p2.6.m6.1.1.3.cmml" xref="S3.SS3.p2.6.m6.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p2.6.m6.1.1.3.1.cmml" xref="S3.SS3.p2.6.m6.1.1.3">superscript</csymbol><ci id="S3.SS3.p2.6.m6.1.1.3.2.cmml" xref="S3.SS3.p2.6.m6.1.1.3.2">ℝ</ci><apply id="S3.SS3.p2.6.m6.1.1.3.3.cmml" xref="S3.SS3.p2.6.m6.1.1.3.3"><times id="S3.SS3.p2.6.m6.1.1.3.3.1.cmml" xref="S3.SS3.p2.6.m6.1.1.3.3.1"></times><apply id="S3.SS3.p2.6.m6.1.1.3.3.2.cmml" xref="S3.SS3.p2.6.m6.1.1.3.3.2"><csymbol cd="ambiguous" id="S3.SS3.p2.6.m6.1.1.3.3.2.1.cmml" xref="S3.SS3.p2.6.m6.1.1.3.3.2">subscript</csymbol><ci id="S3.SS3.p2.6.m6.1.1.3.3.2.2.cmml" xref="S3.SS3.p2.6.m6.1.1.3.3.2.2">𝐾</ci><ci id="S3.SS3.p2.6.m6.1.1.3.3.2.3.cmml" xref="S3.SS3.p2.6.m6.1.1.3.3.2.3">𝑖</ci></apply><ci id="S3.SS3.p2.6.m6.1.1.3.3.3.cmml" xref="S3.SS3.p2.6.m6.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.6.m6.1c">{s_{i}}\in\mathbb{R}^{K_{i}\times d}</annotation></semantics></math>. If <math id="S3.SS3.p2.7.m7.1" class="ltx_Math" alttext="\mathfrak{D}" display="inline"><semantics id="S3.SS3.p2.7.m7.1a"><mi id="S3.SS3.p2.7.m7.1.1" xref="S3.SS3.p2.7.m7.1.1.cmml">𝔇</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.7.m7.1b"><ci id="S3.SS3.p2.7.m7.1.1.cmml" xref="S3.SS3.p2.7.m7.1.1">𝔇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.7.m7.1c">\mathfrak{D}</annotation></semantics></math> is an image dataset, <math id="S3.SS3.p2.8.m8.1" class="ltx_Math" alttext="d=c\times h\times w" display="inline"><semantics id="S3.SS3.p2.8.m8.1a"><mrow id="S3.SS3.p2.8.m8.1.1" xref="S3.SS3.p2.8.m8.1.1.cmml"><mi id="S3.SS3.p2.8.m8.1.1.2" xref="S3.SS3.p2.8.m8.1.1.2.cmml">d</mi><mo id="S3.SS3.p2.8.m8.1.1.1" xref="S3.SS3.p2.8.m8.1.1.1.cmml">=</mo><mrow id="S3.SS3.p2.8.m8.1.1.3" xref="S3.SS3.p2.8.m8.1.1.3.cmml"><mi id="S3.SS3.p2.8.m8.1.1.3.2" xref="S3.SS3.p2.8.m8.1.1.3.2.cmml">c</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS3.p2.8.m8.1.1.3.1" xref="S3.SS3.p2.8.m8.1.1.3.1.cmml">×</mo><mi id="S3.SS3.p2.8.m8.1.1.3.3" xref="S3.SS3.p2.8.m8.1.1.3.3.cmml">h</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS3.p2.8.m8.1.1.3.1a" xref="S3.SS3.p2.8.m8.1.1.3.1.cmml">×</mo><mi id="S3.SS3.p2.8.m8.1.1.3.4" xref="S3.SS3.p2.8.m8.1.1.3.4.cmml">w</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.8.m8.1b"><apply id="S3.SS3.p2.8.m8.1.1.cmml" xref="S3.SS3.p2.8.m8.1.1"><eq id="S3.SS3.p2.8.m8.1.1.1.cmml" xref="S3.SS3.p2.8.m8.1.1.1"></eq><ci id="S3.SS3.p2.8.m8.1.1.2.cmml" xref="S3.SS3.p2.8.m8.1.1.2">𝑑</ci><apply id="S3.SS3.p2.8.m8.1.1.3.cmml" xref="S3.SS3.p2.8.m8.1.1.3"><times id="S3.SS3.p2.8.m8.1.1.3.1.cmml" xref="S3.SS3.p2.8.m8.1.1.3.1"></times><ci id="S3.SS3.p2.8.m8.1.1.3.2.cmml" xref="S3.SS3.p2.8.m8.1.1.3.2">𝑐</ci><ci id="S3.SS3.p2.8.m8.1.1.3.3.cmml" xref="S3.SS3.p2.8.m8.1.1.3.3">ℎ</ci><ci id="S3.SS3.p2.8.m8.1.1.3.4.cmml" xref="S3.SS3.p2.8.m8.1.1.3.4">𝑤</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.8.m8.1c">d=c\times h\times w</annotation></semantics></math> where <math id="S3.SS3.p2.9.m9.1" class="ltx_Math" alttext="c" display="inline"><semantics id="S3.SS3.p2.9.m9.1a"><mi id="S3.SS3.p2.9.m9.1.1" xref="S3.SS3.p2.9.m9.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.9.m9.1b"><ci id="S3.SS3.p2.9.m9.1.1.cmml" xref="S3.SS3.p2.9.m9.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.9.m9.1c">c</annotation></semantics></math>, <math id="S3.SS3.p2.10.m10.1" class="ltx_Math" alttext="h" display="inline"><semantics id="S3.SS3.p2.10.m10.1a"><mi id="S3.SS3.p2.10.m10.1.1" xref="S3.SS3.p2.10.m10.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.10.m10.1b"><ci id="S3.SS3.p2.10.m10.1.1.cmml" xref="S3.SS3.p2.10.m10.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.10.m10.1c">h</annotation></semantics></math> and <math id="S3.SS3.p2.11.m11.1" class="ltx_Math" alttext="w" display="inline"><semantics id="S3.SS3.p2.11.m11.1a"><mi id="S3.SS3.p2.11.m11.1.1" xref="S3.SS3.p2.11.m11.1.1.cmml">w</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.11.m11.1b"><ci id="S3.SS3.p2.11.m11.1.1.cmml" xref="S3.SS3.p2.11.m11.1.1">𝑤</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.11.m11.1c">w</annotation></semantics></math> correspond to the image channel, height and width respectively. Therefore, the entire dataset can be viewed as a collection of subsets <math id="S3.SS3.p2.12.m12.1" class="ltx_Math" alttext="\mathcal{S}=\{s_{i}\}_{i=1}^{C}" display="inline"><semantics id="S3.SS3.p2.12.m12.1a"><mrow id="S3.SS3.p2.12.m12.1.1" xref="S3.SS3.p2.12.m12.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.12.m12.1.1.3" xref="S3.SS3.p2.12.m12.1.1.3.cmml">𝒮</mi><mo id="S3.SS3.p2.12.m12.1.1.2" xref="S3.SS3.p2.12.m12.1.1.2.cmml">=</mo><msubsup id="S3.SS3.p2.12.m12.1.1.1" xref="S3.SS3.p2.12.m12.1.1.1.cmml"><mrow id="S3.SS3.p2.12.m12.1.1.1.1.1.1" xref="S3.SS3.p2.12.m12.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS3.p2.12.m12.1.1.1.1.1.1.2" xref="S3.SS3.p2.12.m12.1.1.1.1.1.2.cmml">{</mo><msub id="S3.SS3.p2.12.m12.1.1.1.1.1.1.1" xref="S3.SS3.p2.12.m12.1.1.1.1.1.1.1.cmml"><mi id="S3.SS3.p2.12.m12.1.1.1.1.1.1.1.2" xref="S3.SS3.p2.12.m12.1.1.1.1.1.1.1.2.cmml">s</mi><mi id="S3.SS3.p2.12.m12.1.1.1.1.1.1.1.3" xref="S3.SS3.p2.12.m12.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS3.p2.12.m12.1.1.1.1.1.1.3" xref="S3.SS3.p2.12.m12.1.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S3.SS3.p2.12.m12.1.1.1.1.3" xref="S3.SS3.p2.12.m12.1.1.1.1.3.cmml"><mi id="S3.SS3.p2.12.m12.1.1.1.1.3.2" xref="S3.SS3.p2.12.m12.1.1.1.1.3.2.cmml">i</mi><mo id="S3.SS3.p2.12.m12.1.1.1.1.3.1" xref="S3.SS3.p2.12.m12.1.1.1.1.3.1.cmml">=</mo><mn id="S3.SS3.p2.12.m12.1.1.1.1.3.3" xref="S3.SS3.p2.12.m12.1.1.1.1.3.3.cmml">1</mn></mrow><mi id="S3.SS3.p2.12.m12.1.1.1.3" xref="S3.SS3.p2.12.m12.1.1.1.3.cmml">C</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.12.m12.1b"><apply id="S3.SS3.p2.12.m12.1.1.cmml" xref="S3.SS3.p2.12.m12.1.1"><eq id="S3.SS3.p2.12.m12.1.1.2.cmml" xref="S3.SS3.p2.12.m12.1.1.2"></eq><ci id="S3.SS3.p2.12.m12.1.1.3.cmml" xref="S3.SS3.p2.12.m12.1.1.3">𝒮</ci><apply id="S3.SS3.p2.12.m12.1.1.1.cmml" xref="S3.SS3.p2.12.m12.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.12.m12.1.1.1.2.cmml" xref="S3.SS3.p2.12.m12.1.1.1">superscript</csymbol><apply id="S3.SS3.p2.12.m12.1.1.1.1.cmml" xref="S3.SS3.p2.12.m12.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.12.m12.1.1.1.1.2.cmml" xref="S3.SS3.p2.12.m12.1.1.1">subscript</csymbol><set id="S3.SS3.p2.12.m12.1.1.1.1.1.2.cmml" xref="S3.SS3.p2.12.m12.1.1.1.1.1.1"><apply id="S3.SS3.p2.12.m12.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.12.m12.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.12.m12.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p2.12.m12.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS3.p2.12.m12.1.1.1.1.1.1.1.2.cmml" xref="S3.SS3.p2.12.m12.1.1.1.1.1.1.1.2">𝑠</ci><ci id="S3.SS3.p2.12.m12.1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.p2.12.m12.1.1.1.1.1.1.1.3">𝑖</ci></apply></set><apply id="S3.SS3.p2.12.m12.1.1.1.1.3.cmml" xref="S3.SS3.p2.12.m12.1.1.1.1.3"><eq id="S3.SS3.p2.12.m12.1.1.1.1.3.1.cmml" xref="S3.SS3.p2.12.m12.1.1.1.1.3.1"></eq><ci id="S3.SS3.p2.12.m12.1.1.1.1.3.2.cmml" xref="S3.SS3.p2.12.m12.1.1.1.1.3.2">𝑖</ci><cn type="integer" id="S3.SS3.p2.12.m12.1.1.1.1.3.3.cmml" xref="S3.SS3.p2.12.m12.1.1.1.1.3.3">1</cn></apply></apply><ci id="S3.SS3.p2.12.m12.1.1.1.3.cmml" xref="S3.SS3.p2.12.m12.1.1.1.3">𝐶</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.12.m12.1c">\mathcal{S}=\{s_{i}\}_{i=1}^{C}</annotation></semantics></math> with <math id="S3.SS3.p2.13.m13.1" class="ltx_Math" alttext="C" display="inline"><semantics id="S3.SS3.p2.13.m13.1a"><mi id="S3.SS3.p2.13.m13.1.1" xref="S3.SS3.p2.13.m13.1.1.cmml">C</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.13.m13.1b"><ci id="S3.SS3.p2.13.m13.1.1.cmml" xref="S3.SS3.p2.13.m13.1.1">𝐶</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.13.m13.1c">C</annotation></semantics></math> being the number of distinct classes. To encode such dataset, we define a transformation <math id="S3.SS3.p2.14.m14.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S3.SS3.p2.14.m14.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.14.m14.1.1" xref="S3.SS3.p2.14.m14.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.14.m14.1b"><ci id="S3.SS3.p2.14.m14.1.1.cmml" xref="S3.SS3.p2.14.m14.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.14.m14.1c">\mathcal{T}</annotation></semantics></math> over the subsets that takes each set <math id="S3.SS3.p2.15.m15.1" class="ltx_Math" alttext="s_{i}" display="inline"><semantics id="S3.SS3.p2.15.m15.1a"><msub id="S3.SS3.p2.15.m15.1.1" xref="S3.SS3.p2.15.m15.1.1.cmml"><mi id="S3.SS3.p2.15.m15.1.1.2" xref="S3.SS3.p2.15.m15.1.1.2.cmml">s</mi><mi id="S3.SS3.p2.15.m15.1.1.3" xref="S3.SS3.p2.15.m15.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.15.m15.1b"><apply id="S3.SS3.p2.15.m15.1.1.cmml" xref="S3.SS3.p2.15.m15.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.15.m15.1.1.1.cmml" xref="S3.SS3.p2.15.m15.1.1">subscript</csymbol><ci id="S3.SS3.p2.15.m15.1.1.2.cmml" xref="S3.SS3.p2.15.m15.1.1.2">𝑠</ci><ci id="S3.SS3.p2.15.m15.1.1.3.cmml" xref="S3.SS3.p2.15.m15.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.15.m15.1c">s_{i}</annotation></semantics></math> and outputs an embedding <math id="S3.SS3.p2.16.m16.1" class="ltx_Math" alttext="z_{s_{i}}\in\mathbb{R}^{1\times d}" display="inline"><semantics id="S3.SS3.p2.16.m16.1a"><mrow id="S3.SS3.p2.16.m16.1.1" xref="S3.SS3.p2.16.m16.1.1.cmml"><msub id="S3.SS3.p2.16.m16.1.1.2" xref="S3.SS3.p2.16.m16.1.1.2.cmml"><mi id="S3.SS3.p2.16.m16.1.1.2.2" xref="S3.SS3.p2.16.m16.1.1.2.2.cmml">z</mi><msub id="S3.SS3.p2.16.m16.1.1.2.3" xref="S3.SS3.p2.16.m16.1.1.2.3.cmml"><mi id="S3.SS3.p2.16.m16.1.1.2.3.2" xref="S3.SS3.p2.16.m16.1.1.2.3.2.cmml">s</mi><mi id="S3.SS3.p2.16.m16.1.1.2.3.3" xref="S3.SS3.p2.16.m16.1.1.2.3.3.cmml">i</mi></msub></msub><mo id="S3.SS3.p2.16.m16.1.1.1" xref="S3.SS3.p2.16.m16.1.1.1.cmml">∈</mo><msup id="S3.SS3.p2.16.m16.1.1.3" xref="S3.SS3.p2.16.m16.1.1.3.cmml"><mi id="S3.SS3.p2.16.m16.1.1.3.2" xref="S3.SS3.p2.16.m16.1.1.3.2.cmml">ℝ</mi><mrow id="S3.SS3.p2.16.m16.1.1.3.3" xref="S3.SS3.p2.16.m16.1.1.3.3.cmml"><mn id="S3.SS3.p2.16.m16.1.1.3.3.2" xref="S3.SS3.p2.16.m16.1.1.3.3.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS3.p2.16.m16.1.1.3.3.1" xref="S3.SS3.p2.16.m16.1.1.3.3.1.cmml">×</mo><mi id="S3.SS3.p2.16.m16.1.1.3.3.3" xref="S3.SS3.p2.16.m16.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.16.m16.1b"><apply id="S3.SS3.p2.16.m16.1.1.cmml" xref="S3.SS3.p2.16.m16.1.1"><in id="S3.SS3.p2.16.m16.1.1.1.cmml" xref="S3.SS3.p2.16.m16.1.1.1"></in><apply id="S3.SS3.p2.16.m16.1.1.2.cmml" xref="S3.SS3.p2.16.m16.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p2.16.m16.1.1.2.1.cmml" xref="S3.SS3.p2.16.m16.1.1.2">subscript</csymbol><ci id="S3.SS3.p2.16.m16.1.1.2.2.cmml" xref="S3.SS3.p2.16.m16.1.1.2.2">𝑧</ci><apply id="S3.SS3.p2.16.m16.1.1.2.3.cmml" xref="S3.SS3.p2.16.m16.1.1.2.3"><csymbol cd="ambiguous" id="S3.SS3.p2.16.m16.1.1.2.3.1.cmml" xref="S3.SS3.p2.16.m16.1.1.2.3">subscript</csymbol><ci id="S3.SS3.p2.16.m16.1.1.2.3.2.cmml" xref="S3.SS3.p2.16.m16.1.1.2.3.2">𝑠</ci><ci id="S3.SS3.p2.16.m16.1.1.2.3.3.cmml" xref="S3.SS3.p2.16.m16.1.1.2.3.3">𝑖</ci></apply></apply><apply id="S3.SS3.p2.16.m16.1.1.3.cmml" xref="S3.SS3.p2.16.m16.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p2.16.m16.1.1.3.1.cmml" xref="S3.SS3.p2.16.m16.1.1.3">superscript</csymbol><ci id="S3.SS3.p2.16.m16.1.1.3.2.cmml" xref="S3.SS3.p2.16.m16.1.1.3.2">ℝ</ci><apply id="S3.SS3.p2.16.m16.1.1.3.3.cmml" xref="S3.SS3.p2.16.m16.1.1.3.3"><times id="S3.SS3.p2.16.m16.1.1.3.3.1.cmml" xref="S3.SS3.p2.16.m16.1.1.3.3.1"></times><cn type="integer" id="S3.SS3.p2.16.m16.1.1.3.3.2.cmml" xref="S3.SS3.p2.16.m16.1.1.3.3.2">1</cn><ci id="S3.SS3.p2.16.m16.1.1.3.3.3.cmml" xref="S3.SS3.p2.16.m16.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.16.m16.1c">z_{s_{i}}\in\mathbb{R}^{1\times d}</annotation></semantics></math>. We note the abuse of notation here where <math id="S3.SS3.p2.17.m17.1" class="ltx_Math" alttext="z" display="inline"><semantics id="S3.SS3.p2.17.m17.1a"><mi id="S3.SS3.p2.17.m17.1.1" xref="S3.SS3.p2.17.m17.1.1.cmml">z</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.17.m17.1b"><ci id="S3.SS3.p2.17.m17.1.1.cmml" xref="S3.SS3.p2.17.m17.1.1">𝑧</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.17.m17.1c">z</annotation></semantics></math> has been used for the weight embedding in Section <a href="#S3.SS2" title="3.2 Weight Encoding ‣ 3 Approach ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>. All embeddings are aggregated to form a new embedding set <math id="S3.SS3.p2.18.m18.1" class="ltx_Math" alttext="\tilde{s}_{i}\in\mathbb{R}^{C\times d}" display="inline"><semantics id="S3.SS3.p2.18.m18.1a"><mrow id="S3.SS3.p2.18.m18.1.1" xref="S3.SS3.p2.18.m18.1.1.cmml"><msub id="S3.SS3.p2.18.m18.1.1.2" xref="S3.SS3.p2.18.m18.1.1.2.cmml"><mover accent="true" id="S3.SS3.p2.18.m18.1.1.2.2" xref="S3.SS3.p2.18.m18.1.1.2.2.cmml"><mi id="S3.SS3.p2.18.m18.1.1.2.2.2" xref="S3.SS3.p2.18.m18.1.1.2.2.2.cmml">s</mi><mo id="S3.SS3.p2.18.m18.1.1.2.2.1" xref="S3.SS3.p2.18.m18.1.1.2.2.1.cmml">~</mo></mover><mi id="S3.SS3.p2.18.m18.1.1.2.3" xref="S3.SS3.p2.18.m18.1.1.2.3.cmml">i</mi></msub><mo id="S3.SS3.p2.18.m18.1.1.1" xref="S3.SS3.p2.18.m18.1.1.1.cmml">∈</mo><msup id="S3.SS3.p2.18.m18.1.1.3" xref="S3.SS3.p2.18.m18.1.1.3.cmml"><mi id="S3.SS3.p2.18.m18.1.1.3.2" xref="S3.SS3.p2.18.m18.1.1.3.2.cmml">ℝ</mi><mrow id="S3.SS3.p2.18.m18.1.1.3.3" xref="S3.SS3.p2.18.m18.1.1.3.3.cmml"><mi id="S3.SS3.p2.18.m18.1.1.3.3.2" xref="S3.SS3.p2.18.m18.1.1.3.3.2.cmml">C</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS3.p2.18.m18.1.1.3.3.1" xref="S3.SS3.p2.18.m18.1.1.3.3.1.cmml">×</mo><mi id="S3.SS3.p2.18.m18.1.1.3.3.3" xref="S3.SS3.p2.18.m18.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.18.m18.1b"><apply id="S3.SS3.p2.18.m18.1.1.cmml" xref="S3.SS3.p2.18.m18.1.1"><in id="S3.SS3.p2.18.m18.1.1.1.cmml" xref="S3.SS3.p2.18.m18.1.1.1"></in><apply id="S3.SS3.p2.18.m18.1.1.2.cmml" xref="S3.SS3.p2.18.m18.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p2.18.m18.1.1.2.1.cmml" xref="S3.SS3.p2.18.m18.1.1.2">subscript</csymbol><apply id="S3.SS3.p2.18.m18.1.1.2.2.cmml" xref="S3.SS3.p2.18.m18.1.1.2.2"><ci id="S3.SS3.p2.18.m18.1.1.2.2.1.cmml" xref="S3.SS3.p2.18.m18.1.1.2.2.1">~</ci><ci id="S3.SS3.p2.18.m18.1.1.2.2.2.cmml" xref="S3.SS3.p2.18.m18.1.1.2.2.2">𝑠</ci></apply><ci id="S3.SS3.p2.18.m18.1.1.2.3.cmml" xref="S3.SS3.p2.18.m18.1.1.2.3">𝑖</ci></apply><apply id="S3.SS3.p2.18.m18.1.1.3.cmml" xref="S3.SS3.p2.18.m18.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p2.18.m18.1.1.3.1.cmml" xref="S3.SS3.p2.18.m18.1.1.3">superscript</csymbol><ci id="S3.SS3.p2.18.m18.1.1.3.2.cmml" xref="S3.SS3.p2.18.m18.1.1.3.2">ℝ</ci><apply id="S3.SS3.p2.18.m18.1.1.3.3.cmml" xref="S3.SS3.p2.18.m18.1.1.3.3"><times id="S3.SS3.p2.18.m18.1.1.3.3.1.cmml" xref="S3.SS3.p2.18.m18.1.1.3.3.1"></times><ci id="S3.SS3.p2.18.m18.1.1.3.3.2.cmml" xref="S3.SS3.p2.18.m18.1.1.3.3.2">𝐶</ci><ci id="S3.SS3.p2.18.m18.1.1.3.3.3.cmml" xref="S3.SS3.p2.18.m18.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.18.m18.1c">\tilde{s}_{i}\in\mathbb{R}^{C\times d}</annotation></semantics></math> to which we apply another transformation <math id="S3.SS3.p2.19.m19.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S3.SS3.p2.19.m19.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.19.m19.1.1" xref="S3.SS3.p2.19.m19.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.19.m19.1b"><ci id="S3.SS3.p2.19.m19.1.1.cmml" xref="S3.SS3.p2.19.m19.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.19.m19.1c">\mathcal{T}</annotation></semantics></math> to <math id="S3.SS3.p2.20.m20.1" class="ltx_Math" alttext="\tilde{s}_{i}" display="inline"><semantics id="S3.SS3.p2.20.m20.1a"><msub id="S3.SS3.p2.20.m20.1.1" xref="S3.SS3.p2.20.m20.1.1.cmml"><mover accent="true" id="S3.SS3.p2.20.m20.1.1.2" xref="S3.SS3.p2.20.m20.1.1.2.cmml"><mi id="S3.SS3.p2.20.m20.1.1.2.2" xref="S3.SS3.p2.20.m20.1.1.2.2.cmml">s</mi><mo id="S3.SS3.p2.20.m20.1.1.2.1" xref="S3.SS3.p2.20.m20.1.1.2.1.cmml">~</mo></mover><mi id="S3.SS3.p2.20.m20.1.1.3" xref="S3.SS3.p2.20.m20.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.20.m20.1b"><apply id="S3.SS3.p2.20.m20.1.1.cmml" xref="S3.SS3.p2.20.m20.1.1"><csymbol cd="ambiguous" id="S3.SS3.p2.20.m20.1.1.1.cmml" xref="S3.SS3.p2.20.m20.1.1">subscript</csymbol><apply id="S3.SS3.p2.20.m20.1.1.2.cmml" xref="S3.SS3.p2.20.m20.1.1.2"><ci id="S3.SS3.p2.20.m20.1.1.2.1.cmml" xref="S3.SS3.p2.20.m20.1.1.2.1">~</ci><ci id="S3.SS3.p2.20.m20.1.1.2.2.cmml" xref="S3.SS3.p2.20.m20.1.1.2.2">𝑠</ci></apply><ci id="S3.SS3.p2.20.m20.1.1.3.cmml" xref="S3.SS3.p2.20.m20.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.20.m20.1c">\tilde{s}_{i}</annotation></semantics></math> to the encoded dataset to <math id="S3.SS3.p2.21.m21.1" class="ltx_Math" alttext="z_{\mathfrak{D}}\in\mathbb{R}^{d}" display="inline"><semantics id="S3.SS3.p2.21.m21.1a"><mrow id="S3.SS3.p2.21.m21.1.1" xref="S3.SS3.p2.21.m21.1.1.cmml"><msub id="S3.SS3.p2.21.m21.1.1.2" xref="S3.SS3.p2.21.m21.1.1.2.cmml"><mi id="S3.SS3.p2.21.m21.1.1.2.2" xref="S3.SS3.p2.21.m21.1.1.2.2.cmml">z</mi><mi id="S3.SS3.p2.21.m21.1.1.2.3" xref="S3.SS3.p2.21.m21.1.1.2.3.cmml">𝔇</mi></msub><mo id="S3.SS3.p2.21.m21.1.1.1" xref="S3.SS3.p2.21.m21.1.1.1.cmml">∈</mo><msup id="S3.SS3.p2.21.m21.1.1.3" xref="S3.SS3.p2.21.m21.1.1.3.cmml"><mi id="S3.SS3.p2.21.m21.1.1.3.2" xref="S3.SS3.p2.21.m21.1.1.3.2.cmml">ℝ</mi><mi id="S3.SS3.p2.21.m21.1.1.3.3" xref="S3.SS3.p2.21.m21.1.1.3.3.cmml">d</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.21.m21.1b"><apply id="S3.SS3.p2.21.m21.1.1.cmml" xref="S3.SS3.p2.21.m21.1.1"><in id="S3.SS3.p2.21.m21.1.1.1.cmml" xref="S3.SS3.p2.21.m21.1.1.1"></in><apply id="S3.SS3.p2.21.m21.1.1.2.cmml" xref="S3.SS3.p2.21.m21.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p2.21.m21.1.1.2.1.cmml" xref="S3.SS3.p2.21.m21.1.1.2">subscript</csymbol><ci id="S3.SS3.p2.21.m21.1.1.2.2.cmml" xref="S3.SS3.p2.21.m21.1.1.2.2">𝑧</ci><ci id="S3.SS3.p2.21.m21.1.1.2.3.cmml" xref="S3.SS3.p2.21.m21.1.1.2.3">𝔇</ci></apply><apply id="S3.SS3.p2.21.m21.1.1.3.cmml" xref="S3.SS3.p2.21.m21.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p2.21.m21.1.1.3.1.cmml" xref="S3.SS3.p2.21.m21.1.1.3">superscript</csymbol><ci id="S3.SS3.p2.21.m21.1.1.3.2.cmml" xref="S3.SS3.p2.21.m21.1.1.3.2">ℝ</ci><ci id="S3.SS3.p2.21.m21.1.1.3.3.cmml" xref="S3.SS3.p2.21.m21.1.1.3.3">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.21.m21.1c">z_{\mathfrak{D}}\in\mathbb{R}^{d}</annotation></semantics></math>. The encoding mechanism described above can be summarized as a composition of Set Transformer blocks <math id="S3.SS3.p2.22.m22.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S3.SS3.p2.22.m22.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.22.m22.1.1" xref="S3.SS3.p2.22.m22.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.22.m22.1b"><ci id="S3.SS3.p2.22.m22.1.1.cmml" xref="S3.SS3.p2.22.m22.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.22.m22.1c">\mathcal{T}</annotation></semantics></math> as follows <math id="S3.SS3.p2.23.m23.1" class="ltx_Math" alttext="z_{\mathfrak{D}}=\mathcal{T}o\mathcal{T}(\mathcal{S})" display="inline"><semantics id="S3.SS3.p2.23.m23.1a"><mrow id="S3.SS3.p2.23.m23.1.2" xref="S3.SS3.p2.23.m23.1.2.cmml"><msub id="S3.SS3.p2.23.m23.1.2.2" xref="S3.SS3.p2.23.m23.1.2.2.cmml"><mi id="S3.SS3.p2.23.m23.1.2.2.2" xref="S3.SS3.p2.23.m23.1.2.2.2.cmml">z</mi><mi id="S3.SS3.p2.23.m23.1.2.2.3" xref="S3.SS3.p2.23.m23.1.2.2.3.cmml">𝔇</mi></msub><mo id="S3.SS3.p2.23.m23.1.2.1" xref="S3.SS3.p2.23.m23.1.2.1.cmml">=</mo><mrow id="S3.SS3.p2.23.m23.1.2.3" xref="S3.SS3.p2.23.m23.1.2.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.23.m23.1.2.3.2" xref="S3.SS3.p2.23.m23.1.2.3.2.cmml">𝒯</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.23.m23.1.2.3.1" xref="S3.SS3.p2.23.m23.1.2.3.1.cmml">​</mo><mi id="S3.SS3.p2.23.m23.1.2.3.3" xref="S3.SS3.p2.23.m23.1.2.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.23.m23.1.2.3.1a" xref="S3.SS3.p2.23.m23.1.2.3.1.cmml">​</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.23.m23.1.2.3.4" xref="S3.SS3.p2.23.m23.1.2.3.4.cmml">𝒯</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p2.23.m23.1.2.3.1b" xref="S3.SS3.p2.23.m23.1.2.3.1.cmml">​</mo><mrow id="S3.SS3.p2.23.m23.1.2.3.5.2" xref="S3.SS3.p2.23.m23.1.2.3.cmml"><mo stretchy="false" id="S3.SS3.p2.23.m23.1.2.3.5.2.1" xref="S3.SS3.p2.23.m23.1.2.3.cmml">(</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.23.m23.1.1" xref="S3.SS3.p2.23.m23.1.1.cmml">𝒮</mi><mo stretchy="false" id="S3.SS3.p2.23.m23.1.2.3.5.2.2" xref="S3.SS3.p2.23.m23.1.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.23.m23.1b"><apply id="S3.SS3.p2.23.m23.1.2.cmml" xref="S3.SS3.p2.23.m23.1.2"><eq id="S3.SS3.p2.23.m23.1.2.1.cmml" xref="S3.SS3.p2.23.m23.1.2.1"></eq><apply id="S3.SS3.p2.23.m23.1.2.2.cmml" xref="S3.SS3.p2.23.m23.1.2.2"><csymbol cd="ambiguous" id="S3.SS3.p2.23.m23.1.2.2.1.cmml" xref="S3.SS3.p2.23.m23.1.2.2">subscript</csymbol><ci id="S3.SS3.p2.23.m23.1.2.2.2.cmml" xref="S3.SS3.p2.23.m23.1.2.2.2">𝑧</ci><ci id="S3.SS3.p2.23.m23.1.2.2.3.cmml" xref="S3.SS3.p2.23.m23.1.2.2.3">𝔇</ci></apply><apply id="S3.SS3.p2.23.m23.1.2.3.cmml" xref="S3.SS3.p2.23.m23.1.2.3"><times id="S3.SS3.p2.23.m23.1.2.3.1.cmml" xref="S3.SS3.p2.23.m23.1.2.3.1"></times><ci id="S3.SS3.p2.23.m23.1.2.3.2.cmml" xref="S3.SS3.p2.23.m23.1.2.3.2">𝒯</ci><ci id="S3.SS3.p2.23.m23.1.2.3.3.cmml" xref="S3.SS3.p2.23.m23.1.2.3.3">𝑜</ci><ci id="S3.SS3.p2.23.m23.1.2.3.4.cmml" xref="S3.SS3.p2.23.m23.1.2.3.4">𝒯</ci><ci id="S3.SS3.p2.23.m23.1.1.cmml" xref="S3.SS3.p2.23.m23.1.1">𝒮</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.23.m23.1c">z_{\mathfrak{D}}=\mathcal{T}o\mathcal{T}(\mathcal{S})</annotation></semantics></math>. To recapitulate, we refer to this operation of encoding the dataset as <math id="S3.SS3.p2.24.m24.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S3.SS3.p2.24.m24.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p2.24.m24.1.1" xref="S3.SS3.p2.24.m24.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p2.24.m24.1b"><ci id="S3.SS3.p2.24.m24.1.1.cmml" xref="S3.SS3.p2.24.m24.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p2.24.m24.1c">\mathcal{T}</annotation></semantics></math>. Training the Set Transformer modules together with the different modules in the pipeline can be computationally demanding. To resolve this issue, we next outline a CLIP style objective for training the Set Transformer Modules.</p>
</div>
<div id="S3.SS3.p3" class="ltx_para">
<p id="S3.SS3.p3.5" class="ltx_p"><span id="S3.SS3.p3.5.1" class="ltx_text ltx_font_bold">Contrastive Dataset Encoding:</span> To improve the effectiveness of the dataset encoding mechanism on large collections of datasets, we train the Set Transformer using a CLIP-style contrastive loss.
An analogous idea is exploited in HyperCLIP <cite class="ltx_cite ltx_citemacro_citep">(Nava et al., <a href="#bib.bib19" title="" class="ltx_ref">2023</a>)</cite>. However in our case, we replace the image encoder with the VAE of Section <a href="#S3.SS2" title="3.2 Weight Encoding ‣ 3 Approach ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>, and replace the image-text contrastive pairs with the weight embedding and its corresponding dataset embedding. Concretely, we utilize the following contrastive pair, <math id="S3.SS3.p3.1.m1.3" class="ltx_Math" alttext="\text{Embed}((\mathfrak{D}_{i}(x_{j},y_{j})_{j}^{c_{i}},W_{i}))=(z_{\mathfrak{D}_{i}},z_{i})" display="inline"><semantics id="S3.SS3.p3.1.m1.3a"><mrow id="S3.SS3.p3.1.m1.3.3" xref="S3.SS3.p3.1.m1.3.3.cmml"><mrow id="S3.SS3.p3.1.m1.1.1.1" xref="S3.SS3.p3.1.m1.1.1.1.cmml"><mtext id="S3.SS3.p3.1.m1.1.1.1.3" xref="S3.SS3.p3.1.m1.1.1.1.3a.cmml">Embed</mtext><mo lspace="0em" rspace="0em" id="S3.SS3.p3.1.m1.1.1.1.2" xref="S3.SS3.p3.1.m1.1.1.1.2.cmml">​</mo><mrow id="S3.SS3.p3.1.m1.1.1.1.1.1" xref="S3.SS3.p3.1.m1.1.1.1.cmml"><mo stretchy="false" id="S3.SS3.p3.1.m1.1.1.1.1.1.2" xref="S3.SS3.p3.1.m1.1.1.1.cmml">(</mo><mrow id="S3.SS3.p3.1.m1.1.1.1.1.1.1.2" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.3.cmml"><mo stretchy="false" id="S3.SS3.p3.1.m1.1.1.1.1.1.1.2.3" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.3.cmml">(</mo><mrow id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.cmml"><msub id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.4" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.4.cmml"><mi id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.4.2" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.4.2.cmml">𝔇</mi><mi id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.4.3" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.4.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.3" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.3.cmml">​</mo><msubsup id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.cmml"><mrow id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.3.cmml"><mo stretchy="false" id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2.3" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.3.cmml">(</mo><msub id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">j</mi></msub><mo id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2.4" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.3.cmml">,</mo><msub id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2.2" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.cmml"><mi id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.cmml">y</mi><mi id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.3" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.3.cmml">j</mi></msub><mo stretchy="false" id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2.5" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.3.cmml">)</mo></mrow><mi id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.4" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.4.cmml">j</mi><msub id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.4" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.4.cmml"><mi id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.4.2" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.4.2.cmml">c</mi><mi id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.4.3" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.4.3.cmml">i</mi></msub></msubsup></mrow><mo id="S3.SS3.p3.1.m1.1.1.1.1.1.1.2.4" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.3.cmml">,</mo><msub id="S3.SS3.p3.1.m1.1.1.1.1.1.1.2.2" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.2.2.cmml"><mi id="S3.SS3.p3.1.m1.1.1.1.1.1.1.2.2.2" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.2.2.2.cmml">W</mi><mi id="S3.SS3.p3.1.m1.1.1.1.1.1.1.2.2.3" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.2.2.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS3.p3.1.m1.1.1.1.1.1.1.2.5" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.3.cmml">)</mo></mrow><mo stretchy="false" id="S3.SS3.p3.1.m1.1.1.1.1.1.3" xref="S3.SS3.p3.1.m1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS3.p3.1.m1.3.3.4" xref="S3.SS3.p3.1.m1.3.3.4.cmml">=</mo><mrow id="S3.SS3.p3.1.m1.3.3.3.2" xref="S3.SS3.p3.1.m1.3.3.3.3.cmml"><mo stretchy="false" id="S3.SS3.p3.1.m1.3.3.3.2.3" xref="S3.SS3.p3.1.m1.3.3.3.3.cmml">(</mo><msub id="S3.SS3.p3.1.m1.2.2.2.1.1" xref="S3.SS3.p3.1.m1.2.2.2.1.1.cmml"><mi id="S3.SS3.p3.1.m1.2.2.2.1.1.2" xref="S3.SS3.p3.1.m1.2.2.2.1.1.2.cmml">z</mi><msub id="S3.SS3.p3.1.m1.2.2.2.1.1.3" xref="S3.SS3.p3.1.m1.2.2.2.1.1.3.cmml"><mi id="S3.SS3.p3.1.m1.2.2.2.1.1.3.2" xref="S3.SS3.p3.1.m1.2.2.2.1.1.3.2.cmml">𝔇</mi><mi id="S3.SS3.p3.1.m1.2.2.2.1.1.3.3" xref="S3.SS3.p3.1.m1.2.2.2.1.1.3.3.cmml">i</mi></msub></msub><mo id="S3.SS3.p3.1.m1.3.3.3.2.4" xref="S3.SS3.p3.1.m1.3.3.3.3.cmml">,</mo><msub id="S3.SS3.p3.1.m1.3.3.3.2.2" xref="S3.SS3.p3.1.m1.3.3.3.2.2.cmml"><mi id="S3.SS3.p3.1.m1.3.3.3.2.2.2" xref="S3.SS3.p3.1.m1.3.3.3.2.2.2.cmml">z</mi><mi id="S3.SS3.p3.1.m1.3.3.3.2.2.3" xref="S3.SS3.p3.1.m1.3.3.3.2.2.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS3.p3.1.m1.3.3.3.2.5" xref="S3.SS3.p3.1.m1.3.3.3.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.1.m1.3b"><apply id="S3.SS3.p3.1.m1.3.3.cmml" xref="S3.SS3.p3.1.m1.3.3"><eq id="S3.SS3.p3.1.m1.3.3.4.cmml" xref="S3.SS3.p3.1.m1.3.3.4"></eq><apply id="S3.SS3.p3.1.m1.1.1.1.cmml" xref="S3.SS3.p3.1.m1.1.1.1"><times id="S3.SS3.p3.1.m1.1.1.1.2.cmml" xref="S3.SS3.p3.1.m1.1.1.1.2"></times><ci id="S3.SS3.p3.1.m1.1.1.1.3a.cmml" xref="S3.SS3.p3.1.m1.1.1.1.3"><mtext id="S3.SS3.p3.1.m1.1.1.1.3.cmml" xref="S3.SS3.p3.1.m1.1.1.1.3">Embed</mtext></ci><interval closure="open" id="S3.SS3.p3.1.m1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.2"><apply id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1"><times id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.3"></times><apply id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.4.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.4.1.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.4">subscript</csymbol><ci id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.4.2.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.4.2">𝔇</ci><ci id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.4.3.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.4.3">𝑖</ci></apply><apply id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2">superscript</csymbol><apply id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.3.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2">subscript</csymbol><interval closure="open" id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.3.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2"><apply id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3">𝑗</ci></apply><apply id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.1.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2.2">subscript</csymbol><ci id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.2">𝑦</ci><ci id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.3.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.2.2.2.3">𝑗</ci></apply></interval><ci id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.4.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.2.4">𝑗</ci></apply><apply id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.4.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.4.1.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.4">subscript</csymbol><ci id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.4.2.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.4.2">𝑐</ci><ci id="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.4.3.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.1.1.2.4.3">𝑖</ci></apply></apply></apply><apply id="S3.SS3.p3.1.m1.1.1.1.1.1.1.2.2.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.SS3.p3.1.m1.1.1.1.1.1.1.2.2.1.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.2.2">subscript</csymbol><ci id="S3.SS3.p3.1.m1.1.1.1.1.1.1.2.2.2.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.2.2.2">𝑊</ci><ci id="S3.SS3.p3.1.m1.1.1.1.1.1.1.2.2.3.cmml" xref="S3.SS3.p3.1.m1.1.1.1.1.1.1.2.2.3">𝑖</ci></apply></interval></apply><interval closure="open" id="S3.SS3.p3.1.m1.3.3.3.3.cmml" xref="S3.SS3.p3.1.m1.3.3.3.2"><apply id="S3.SS3.p3.1.m1.2.2.2.1.1.cmml" xref="S3.SS3.p3.1.m1.2.2.2.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.1.m1.2.2.2.1.1.1.cmml" xref="S3.SS3.p3.1.m1.2.2.2.1.1">subscript</csymbol><ci id="S3.SS3.p3.1.m1.2.2.2.1.1.2.cmml" xref="S3.SS3.p3.1.m1.2.2.2.1.1.2">𝑧</ci><apply id="S3.SS3.p3.1.m1.2.2.2.1.1.3.cmml" xref="S3.SS3.p3.1.m1.2.2.2.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p3.1.m1.2.2.2.1.1.3.1.cmml" xref="S3.SS3.p3.1.m1.2.2.2.1.1.3">subscript</csymbol><ci id="S3.SS3.p3.1.m1.2.2.2.1.1.3.2.cmml" xref="S3.SS3.p3.1.m1.2.2.2.1.1.3.2">𝔇</ci><ci id="S3.SS3.p3.1.m1.2.2.2.1.1.3.3.cmml" xref="S3.SS3.p3.1.m1.2.2.2.1.1.3.3">𝑖</ci></apply></apply><apply id="S3.SS3.p3.1.m1.3.3.3.2.2.cmml" xref="S3.SS3.p3.1.m1.3.3.3.2.2"><csymbol cd="ambiguous" id="S3.SS3.p3.1.m1.3.3.3.2.2.1.cmml" xref="S3.SS3.p3.1.m1.3.3.3.2.2">subscript</csymbol><ci id="S3.SS3.p3.1.m1.3.3.3.2.2.2.cmml" xref="S3.SS3.p3.1.m1.3.3.3.2.2.2">𝑧</ci><ci id="S3.SS3.p3.1.m1.3.3.3.2.2.3.cmml" xref="S3.SS3.p3.1.m1.3.3.3.2.2.3">𝑖</ci></apply></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.1.m1.3c">\text{Embed}((\mathfrak{D}_{i}(x_{j},y_{j})_{j}^{c_{i}},W_{i}))=(z_{\mathfrak{D}_{i}},z_{i})</annotation></semantics></math> consisting of the class-wise dataset encodings and the VAE embedding of a corresponding network weight. Using the pretrained VAE encoder of Section <a href="#S3.SS2" title="3.2 Weight Encoding ‣ 3 Approach ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.2</span></a>, we train the Set Transformer dataset encoding module <math id="S3.SS3.p3.2.m2.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S3.SS3.p3.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p3.2.m2.1.1" xref="S3.SS3.p3.2.m2.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.2.m2.1b"><ci id="S3.SS3.p3.2.m2.1.1.cmml" xref="S3.SS3.p3.2.m2.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.2.m2.1c">\mathcal{T}</annotation></semantics></math> to align its embedding <math id="S3.SS3.p3.3.m3.1" class="ltx_Math" alttext="z_{\mathfrak{D}_{i}}=\mathcal{T}(\mathfrak{D}_{i}(x_{j},y_{j})_{j}^{c_{i}})" display="inline"><semantics id="S3.SS3.p3.3.m3.1a"><mrow id="S3.SS3.p3.3.m3.1.1" xref="S3.SS3.p3.3.m3.1.1.cmml"><msub id="S3.SS3.p3.3.m3.1.1.3" xref="S3.SS3.p3.3.m3.1.1.3.cmml"><mi id="S3.SS3.p3.3.m3.1.1.3.2" xref="S3.SS3.p3.3.m3.1.1.3.2.cmml">z</mi><msub id="S3.SS3.p3.3.m3.1.1.3.3" xref="S3.SS3.p3.3.m3.1.1.3.3.cmml"><mi id="S3.SS3.p3.3.m3.1.1.3.3.2" xref="S3.SS3.p3.3.m3.1.1.3.3.2.cmml">𝔇</mi><mi id="S3.SS3.p3.3.m3.1.1.3.3.3" xref="S3.SS3.p3.3.m3.1.1.3.3.3.cmml">i</mi></msub></msub><mo id="S3.SS3.p3.3.m3.1.1.2" xref="S3.SS3.p3.3.m3.1.1.2.cmml">=</mo><mrow id="S3.SS3.p3.3.m3.1.1.1" xref="S3.SS3.p3.3.m3.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS3.p3.3.m3.1.1.1.3" xref="S3.SS3.p3.3.m3.1.1.1.3.cmml">𝒯</mi><mo lspace="0em" rspace="0em" id="S3.SS3.p3.3.m3.1.1.1.2" xref="S3.SS3.p3.3.m3.1.1.1.2.cmml">​</mo><mrow id="S3.SS3.p3.3.m3.1.1.1.1.1" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS3.p3.3.m3.1.1.1.1.1.2" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS3.p3.3.m3.1.1.1.1.1.1" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.cmml"><msub id="S3.SS3.p3.3.m3.1.1.1.1.1.1.4" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.4.cmml"><mi id="S3.SS3.p3.3.m3.1.1.1.1.1.1.4.2" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.4.2.cmml">𝔇</mi><mi id="S3.SS3.p3.3.m3.1.1.1.1.1.1.4.3" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.4.3.cmml">i</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS3.p3.3.m3.1.1.1.1.1.1.3" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.3.cmml">​</mo><msubsup id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.cmml"><mrow id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.3.cmml"><mo stretchy="false" id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2.3" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.3.cmml">(</mo><msub id="S3.SS3.p3.3.m3.1.1.1.1.1.1.1.1.1.1.1" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.SS3.p3.3.m3.1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S3.SS3.p3.3.m3.1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.1.1.1.1.1.3.cmml">j</mi></msub><mo id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2.4" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.3.cmml">,</mo><msub id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2.2" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2.2.cmml"><mi id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2.2.2" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2.2.2.cmml">y</mi><mi id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2.2.3" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2.2.3.cmml">j</mi></msub><mo stretchy="false" id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2.5" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.3.cmml">)</mo></mrow><mi id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.4" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.4.cmml">j</mi><msub id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.4" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.4.cmml"><mi id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.4.2" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.4.2.cmml">c</mi><mi id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.4.3" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.4.3.cmml">i</mi></msub></msubsup></mrow><mo stretchy="false" id="S3.SS3.p3.3.m3.1.1.1.1.1.3" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.3.m3.1b"><apply id="S3.SS3.p3.3.m3.1.1.cmml" xref="S3.SS3.p3.3.m3.1.1"><eq id="S3.SS3.p3.3.m3.1.1.2.cmml" xref="S3.SS3.p3.3.m3.1.1.2"></eq><apply id="S3.SS3.p3.3.m3.1.1.3.cmml" xref="S3.SS3.p3.3.m3.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p3.3.m3.1.1.3.1.cmml" xref="S3.SS3.p3.3.m3.1.1.3">subscript</csymbol><ci id="S3.SS3.p3.3.m3.1.1.3.2.cmml" xref="S3.SS3.p3.3.m3.1.1.3.2">𝑧</ci><apply id="S3.SS3.p3.3.m3.1.1.3.3.cmml" xref="S3.SS3.p3.3.m3.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS3.p3.3.m3.1.1.3.3.1.cmml" xref="S3.SS3.p3.3.m3.1.1.3.3">subscript</csymbol><ci id="S3.SS3.p3.3.m3.1.1.3.3.2.cmml" xref="S3.SS3.p3.3.m3.1.1.3.3.2">𝔇</ci><ci id="S3.SS3.p3.3.m3.1.1.3.3.3.cmml" xref="S3.SS3.p3.3.m3.1.1.3.3.3">𝑖</ci></apply></apply><apply id="S3.SS3.p3.3.m3.1.1.1.cmml" xref="S3.SS3.p3.3.m3.1.1.1"><times id="S3.SS3.p3.3.m3.1.1.1.2.cmml" xref="S3.SS3.p3.3.m3.1.1.1.2"></times><ci id="S3.SS3.p3.3.m3.1.1.1.3.cmml" xref="S3.SS3.p3.3.m3.1.1.1.3">𝒯</ci><apply id="S3.SS3.p3.3.m3.1.1.1.1.1.1.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1"><times id="S3.SS3.p3.3.m3.1.1.1.1.1.1.3.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.3"></times><apply id="S3.SS3.p3.3.m3.1.1.1.1.1.1.4.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.4"><csymbol cd="ambiguous" id="S3.SS3.p3.3.m3.1.1.1.1.1.1.4.1.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.4">subscript</csymbol><ci id="S3.SS3.p3.3.m3.1.1.1.1.1.1.4.2.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.4.2">𝔇</ci><ci id="S3.SS3.p3.3.m3.1.1.1.1.1.1.4.3.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.4.3">𝑖</ci></apply><apply id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.3.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2">superscript</csymbol><apply id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.3.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2">subscript</csymbol><interval closure="open" id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.3.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2"><apply id="S3.SS3.p3.3.m3.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.3.m3.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS3.p3.3.m3.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S3.SS3.p3.3.m3.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.1.1.1.1.1.3">𝑗</ci></apply><apply id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2.2.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2.2.1.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2.2">subscript</csymbol><ci id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2.2.2.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2.2.2">𝑦</ci><ci id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2.2.3.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.2.2.2.3">𝑗</ci></apply></interval><ci id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.4.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.2.4">𝑗</ci></apply><apply id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.4.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.4.1.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.4">subscript</csymbol><ci id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.4.2.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.4.2">𝑐</ci><ci id="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.4.3.cmml" xref="S3.SS3.p3.3.m3.1.1.1.1.1.1.2.4.3">𝑖</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.3.m3.1c">z_{\mathfrak{D}_{i}}=\mathcal{T}(\mathfrak{D}_{i}(x_{j},y_{j})_{j}^{c_{i}})</annotation></semantics></math> where <math id="S3.SS3.p3.4.m4.1" class="ltx_Math" alttext="z_{\mathfrak{D}_{i}}\in\mathbb{R}^{1\times d_{z}}" display="inline"><semantics id="S3.SS3.p3.4.m4.1a"><mrow id="S3.SS3.p3.4.m4.1.1" xref="S3.SS3.p3.4.m4.1.1.cmml"><msub id="S3.SS3.p3.4.m4.1.1.2" xref="S3.SS3.p3.4.m4.1.1.2.cmml"><mi id="S3.SS3.p3.4.m4.1.1.2.2" xref="S3.SS3.p3.4.m4.1.1.2.2.cmml">z</mi><msub id="S3.SS3.p3.4.m4.1.1.2.3" xref="S3.SS3.p3.4.m4.1.1.2.3.cmml"><mi id="S3.SS3.p3.4.m4.1.1.2.3.2" xref="S3.SS3.p3.4.m4.1.1.2.3.2.cmml">𝔇</mi><mi id="S3.SS3.p3.4.m4.1.1.2.3.3" xref="S3.SS3.p3.4.m4.1.1.2.3.3.cmml">i</mi></msub></msub><mo id="S3.SS3.p3.4.m4.1.1.1" xref="S3.SS3.p3.4.m4.1.1.1.cmml">∈</mo><msup id="S3.SS3.p3.4.m4.1.1.3" xref="S3.SS3.p3.4.m4.1.1.3.cmml"><mi id="S3.SS3.p3.4.m4.1.1.3.2" xref="S3.SS3.p3.4.m4.1.1.3.2.cmml">ℝ</mi><mrow id="S3.SS3.p3.4.m4.1.1.3.3" xref="S3.SS3.p3.4.m4.1.1.3.3.cmml"><mn id="S3.SS3.p3.4.m4.1.1.3.3.2" xref="S3.SS3.p3.4.m4.1.1.3.3.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="S3.SS3.p3.4.m4.1.1.3.3.1" xref="S3.SS3.p3.4.m4.1.1.3.3.1.cmml">×</mo><msub id="S3.SS3.p3.4.m4.1.1.3.3.3" xref="S3.SS3.p3.4.m4.1.1.3.3.3.cmml"><mi id="S3.SS3.p3.4.m4.1.1.3.3.3.2" xref="S3.SS3.p3.4.m4.1.1.3.3.3.2.cmml">d</mi><mi id="S3.SS3.p3.4.m4.1.1.3.3.3.3" xref="S3.SS3.p3.4.m4.1.1.3.3.3.3.cmml">z</mi></msub></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.4.m4.1b"><apply id="S3.SS3.p3.4.m4.1.1.cmml" xref="S3.SS3.p3.4.m4.1.1"><in id="S3.SS3.p3.4.m4.1.1.1.cmml" xref="S3.SS3.p3.4.m4.1.1.1"></in><apply id="S3.SS3.p3.4.m4.1.1.2.cmml" xref="S3.SS3.p3.4.m4.1.1.2"><csymbol cd="ambiguous" id="S3.SS3.p3.4.m4.1.1.2.1.cmml" xref="S3.SS3.p3.4.m4.1.1.2">subscript</csymbol><ci id="S3.SS3.p3.4.m4.1.1.2.2.cmml" xref="S3.SS3.p3.4.m4.1.1.2.2">𝑧</ci><apply id="S3.SS3.p3.4.m4.1.1.2.3.cmml" xref="S3.SS3.p3.4.m4.1.1.2.3"><csymbol cd="ambiguous" id="S3.SS3.p3.4.m4.1.1.2.3.1.cmml" xref="S3.SS3.p3.4.m4.1.1.2.3">subscript</csymbol><ci id="S3.SS3.p3.4.m4.1.1.2.3.2.cmml" xref="S3.SS3.p3.4.m4.1.1.2.3.2">𝔇</ci><ci id="S3.SS3.p3.4.m4.1.1.2.3.3.cmml" xref="S3.SS3.p3.4.m4.1.1.2.3.3">𝑖</ci></apply></apply><apply id="S3.SS3.p3.4.m4.1.1.3.cmml" xref="S3.SS3.p3.4.m4.1.1.3"><csymbol cd="ambiguous" id="S3.SS3.p3.4.m4.1.1.3.1.cmml" xref="S3.SS3.p3.4.m4.1.1.3">superscript</csymbol><ci id="S3.SS3.p3.4.m4.1.1.3.2.cmml" xref="S3.SS3.p3.4.m4.1.1.3.2">ℝ</ci><apply id="S3.SS3.p3.4.m4.1.1.3.3.cmml" xref="S3.SS3.p3.4.m4.1.1.3.3"><times id="S3.SS3.p3.4.m4.1.1.3.3.1.cmml" xref="S3.SS3.p3.4.m4.1.1.3.3.1"></times><cn type="integer" id="S3.SS3.p3.4.m4.1.1.3.3.2.cmml" xref="S3.SS3.p3.4.m4.1.1.3.3.2">1</cn><apply id="S3.SS3.p3.4.m4.1.1.3.3.3.cmml" xref="S3.SS3.p3.4.m4.1.1.3.3.3"><csymbol cd="ambiguous" id="S3.SS3.p3.4.m4.1.1.3.3.3.1.cmml" xref="S3.SS3.p3.4.m4.1.1.3.3.3">subscript</csymbol><ci id="S3.SS3.p3.4.m4.1.1.3.3.3.2.cmml" xref="S3.SS3.p3.4.m4.1.1.3.3.3.2">𝑑</ci><ci id="S3.SS3.p3.4.m4.1.1.3.3.3.3.cmml" xref="S3.SS3.p3.4.m4.1.1.3.3.3.3">𝑧</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.4.m4.1c">z_{\mathfrak{D}_{i}}\in\mathbb{R}^{1\times d_{z}}</annotation></semantics></math> with the network embeddings <math id="S3.SS3.p3.5.m5.1" class="ltx_Math" alttext="z_{i}" display="inline"><semantics id="S3.SS3.p3.5.m5.1a"><msub id="S3.SS3.p3.5.m5.1.1" xref="S3.SS3.p3.5.m5.1.1.cmml"><mi id="S3.SS3.p3.5.m5.1.1.2" xref="S3.SS3.p3.5.m5.1.1.2.cmml">z</mi><mi id="S3.SS3.p3.5.m5.1.1.3" xref="S3.SS3.p3.5.m5.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS3.p3.5.m5.1b"><apply id="S3.SS3.p3.5.m5.1.1.cmml" xref="S3.SS3.p3.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS3.p3.5.m5.1.1.1.cmml" xref="S3.SS3.p3.5.m5.1.1">subscript</csymbol><ci id="S3.SS3.p3.5.m5.1.1.2.cmml" xref="S3.SS3.p3.5.m5.1.1.2">𝑧</ci><ci id="S3.SS3.p3.5.m5.1.1.3.cmml" xref="S3.SS3.p3.5.m5.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.p3.5.m5.1c">z_{i}</annotation></semantics></math> of the pretrained VAE. The Set Transformer training objective is given as:</p>
<table id="S3.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E2.m1.1" class="ltx_math_unparsed" alttext="\mathcal{L}_{CLIP}=-log\frac{exp(z_{i}.z_{\mathfrak{D}_{i}/\tau)}}{\sum_{k=0}^{N}exp(z_{i}.z_{\mathfrak{D}_{k}}/\tau)}" display="block"><semantics id="S3.E2.m1.1a"><mrow id="S3.E2.m1.1.1"><msub id="S3.E2.m1.1.1.2"><mi class="ltx_font_mathcaligraphic" id="S3.E2.m1.1.1.2.2">ℒ</mi><mrow id="S3.E2.m1.1.1.2.3"><mi id="S3.E2.m1.1.1.2.3.2">C</mi><mo lspace="0em" rspace="0em" id="S3.E2.m1.1.1.2.3.1">​</mo><mi id="S3.E2.m1.1.1.2.3.3">L</mi><mo lspace="0em" rspace="0em" id="S3.E2.m1.1.1.2.3.1a">​</mo><mi id="S3.E2.m1.1.1.2.3.4">I</mi><mo lspace="0em" rspace="0em" id="S3.E2.m1.1.1.2.3.1b">​</mo><mi id="S3.E2.m1.1.1.2.3.5">P</mi></mrow></msub><mo id="S3.E2.m1.1.1.1">=</mo><mrow id="S3.E2.m1.1.1.3"><mo id="S3.E2.m1.1.1.3a">−</mo><mrow id="S3.E2.m1.1.1.3.2"><mi id="S3.E2.m1.1.1.3.2.2">l</mi><mo lspace="0em" rspace="0em" id="S3.E2.m1.1.1.3.2.1">​</mo><mi id="S3.E2.m1.1.1.3.2.3">o</mi><mo lspace="0em" rspace="0em" id="S3.E2.m1.1.1.3.2.1a">​</mo><mi id="S3.E2.m1.1.1.3.2.4">g</mi><mo lspace="0em" rspace="0em" id="S3.E2.m1.1.1.3.2.1b">​</mo><mfrac id="S3.E2.m1.1.1.3.2.5"><mrow id="S3.E2.m1.1.1.3.2.5.2"><mi id="S3.E2.m1.1.1.3.2.5.2.1">e</mi><mi id="S3.E2.m1.1.1.3.2.5.2.2">x</mi><mi id="S3.E2.m1.1.1.3.2.5.2.3">p</mi><mrow id="S3.E2.m1.1.1.3.2.5.2.4"><mo stretchy="false" id="S3.E2.m1.1.1.3.2.5.2.4.1">(</mo><msub id="S3.E2.m1.1.1.3.2.5.2.4.2"><mi id="S3.E2.m1.1.1.3.2.5.2.4.2.2">z</mi><mi id="S3.E2.m1.1.1.3.2.5.2.4.2.3">i</mi></msub><mo lspace="0em" rspace="0.167em" id="S3.E2.m1.1.1.3.2.5.2.4.3">.</mo><msub id="S3.E2.m1.1.1.3.2.5.2.4.4"><mi id="S3.E2.m1.1.1.3.2.5.2.4.4.2">z</mi><mrow id="S3.E2.m1.1.1.3.2.5.2.4.4.3"><msub id="S3.E2.m1.1.1.3.2.5.2.4.4.3.1"><mi id="S3.E2.m1.1.1.3.2.5.2.4.4.3.1.2">𝔇</mi><mi id="S3.E2.m1.1.1.3.2.5.2.4.4.3.1.3">i</mi></msub><mo id="S3.E2.m1.1.1.3.2.5.2.4.4.3.2">/</mo><mi id="S3.E2.m1.1.1.3.2.5.2.4.4.3.3">τ</mi><mo stretchy="false" id="S3.E2.m1.1.1.3.2.5.2.4.4.3.4">)</mo></mrow></msub></mrow></mrow><mrow id="S3.E2.m1.1.1.3.2.5.3"><msubsup id="S3.E2.m1.1.1.3.2.5.3.1"><mo id="S3.E2.m1.1.1.3.2.5.3.1.2.2">∑</mo><mrow id="S3.E2.m1.1.1.3.2.5.3.1.2.3"><mi id="S3.E2.m1.1.1.3.2.5.3.1.2.3.2">k</mi><mo id="S3.E2.m1.1.1.3.2.5.3.1.2.3.1">=</mo><mn id="S3.E2.m1.1.1.3.2.5.3.1.2.3.3">0</mn></mrow><mi id="S3.E2.m1.1.1.3.2.5.3.1.3">N</mi></msubsup><mi id="S3.E2.m1.1.1.3.2.5.3.2">e</mi><mi id="S3.E2.m1.1.1.3.2.5.3.3">x</mi><mi id="S3.E2.m1.1.1.3.2.5.3.4">p</mi><mrow id="S3.E2.m1.1.1.3.2.5.3.5"><mo stretchy="false" id="S3.E2.m1.1.1.3.2.5.3.5.1">(</mo><msub id="S3.E2.m1.1.1.3.2.5.3.5.2"><mi id="S3.E2.m1.1.1.3.2.5.3.5.2.2">z</mi><mi id="S3.E2.m1.1.1.3.2.5.3.5.2.3">i</mi></msub><mo lspace="0em" rspace="0.167em" id="S3.E2.m1.1.1.3.2.5.3.5.3">.</mo><msub id="S3.E2.m1.1.1.3.2.5.3.5.4"><mi id="S3.E2.m1.1.1.3.2.5.3.5.4.2">z</mi><msub id="S3.E2.m1.1.1.3.2.5.3.5.4.3"><mi id="S3.E2.m1.1.1.3.2.5.3.5.4.3.2">𝔇</mi><mi id="S3.E2.m1.1.1.3.2.5.3.5.4.3.3">k</mi></msub></msub><mo id="S3.E2.m1.1.1.3.2.5.3.5.5">/</mo><mi id="S3.E2.m1.1.1.3.2.5.3.5.6">τ</mi><mo stretchy="false" id="S3.E2.m1.1.1.3.2.5.3.5.7">)</mo></mrow></mrow></mfrac></mrow></mrow></mrow><annotation encoding="application/x-tex" id="S3.E2.m1.1b">\mathcal{L}_{CLIP}=-log\frac{exp(z_{i}.z_{\mathfrak{D}_{i}/\tau)}}{\sum_{k=0}^{N}exp(z_{i}.z_{\mathfrak{D}_{k}}/\tau)}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
</div>
</section>
<section id="S3.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.4 </span>Dataset-Conditioned Weight Generation</h3>

<div id="S3.SS4.p1" class="ltx_para">
<p id="S3.SS4.p1.1" class="ltx_p">At this stage, we have access to a pretrained VAE for encoding neural network weights and a pretrained Set Transformer module for encoding entire datasets. The next stage involves defining a model to generate latent representations of weights conditioned on dataset representations. We achieve this by using Diffusion Denoising Probabilistic Models (DDPM) <cite class="ltx_cite ltx_citemacro_citep">(Ho et al., <a href="#bib.bib11" title="" class="ltx_ref">2020</a>; Rombach et al., <a href="#bib.bib24" title="" class="ltx_ref">2021</a>)</cite>.</p>
</div>
<div id="S3.SS4.p2" class="ltx_para">
<p id="S3.SS4.p2.4" class="ltx_p"><span id="S3.SS4.p2.4.1" class="ltx_text ltx_font_bold">Forward Process:</span> Given a weight embedding <math id="S3.SS4.p2.1.m1.1" class="ltx_Math" alttext="z" display="inline"><semantics id="S3.SS4.p2.1.m1.1a"><mi id="S3.SS4.p2.1.m1.1.1" xref="S3.SS4.p2.1.m1.1.1.cmml">z</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.1.m1.1b"><ci id="S3.SS4.p2.1.m1.1.1.cmml" xref="S3.SS4.p2.1.m1.1.1">𝑧</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.1.m1.1c">z</annotation></semantics></math>, obtained from the encoder of the pretrained VAE, the forward diffusion process involves successive Gaussion noise perturbations of <math id="S3.SS4.p2.2.m2.1" class="ltx_Math" alttext="z" display="inline"><semantics id="S3.SS4.p2.2.m2.1a"><mi id="S3.SS4.p2.2.m2.1.1" xref="S3.SS4.p2.2.m2.1.1.cmml">z</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.2.m2.1b"><ci id="S3.SS4.p2.2.m2.1.1.cmml" xref="S3.SS4.p2.2.m2.1.1">𝑧</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.2.m2.1c">z</annotation></semantics></math> over <math id="S3.SS4.p2.3.m3.1" class="ltx_Math" alttext="T" display="inline"><semantics id="S3.SS4.p2.3.m3.1a"><mi id="S3.SS4.p2.3.m3.1.1" xref="S3.SS4.p2.3.m3.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.3.m3.1b"><ci id="S3.SS4.p2.3.m3.1.1.cmml" xref="S3.SS4.p2.3.m3.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.3.m3.1c">T</annotation></semantics></math> time steps. At time step <math id="S3.SS4.p2.4.m4.1" class="ltx_Math" alttext="t" display="inline"><semantics id="S3.SS4.p2.4.m4.1a"><mi id="S3.SS4.p2.4.m4.1.1" xref="S3.SS4.p2.4.m4.1.1.cmml">t</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.4.m4.1b"><ci id="S3.SS4.p2.4.m4.1.1.cmml" xref="S3.SS4.p2.4.m4.1.1">𝑡</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.4.m4.1c">t</annotation></semantics></math>,</p>
<table id="S3.E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E3.m1.1" class="ltx_math_unparsed" alttext="p(z_{t}|z_{t-1})=\mathcal{N}(z_{t};\mu_{t}=\sqrt{1-\beta_{t}}z_{t-1},\beta_{t}I)" display="block"><semantics id="S3.E3.m1.1a"><mrow id="S3.E3.m1.1b"><mi id="S3.E3.m1.1.1">p</mi><mrow id="S3.E3.m1.1.2"><mo stretchy="false" id="S3.E3.m1.1.2.1">(</mo><msub id="S3.E3.m1.1.2.2"><mi id="S3.E3.m1.1.2.2.2">z</mi><mi id="S3.E3.m1.1.2.2.3">t</mi></msub><mo fence="false" rspace="0.167em" stretchy="false" id="S3.E3.m1.1.2.3">|</mo><msub id="S3.E3.m1.1.2.4"><mi id="S3.E3.m1.1.2.4.2">z</mi><mrow id="S3.E3.m1.1.2.4.3"><mi id="S3.E3.m1.1.2.4.3.2">t</mi><mo id="S3.E3.m1.1.2.4.3.1">−</mo><mn id="S3.E3.m1.1.2.4.3.3">1</mn></mrow></msub><mo stretchy="false" id="S3.E3.m1.1.2.5">)</mo></mrow><mo id="S3.E3.m1.1.3">=</mo><mi class="ltx_font_mathcaligraphic" id="S3.E3.m1.1.4">𝒩</mi><mrow id="S3.E3.m1.1.5"><mo stretchy="false" id="S3.E3.m1.1.5.1">(</mo><msub id="S3.E3.m1.1.5.2"><mi id="S3.E3.m1.1.5.2.2">z</mi><mi id="S3.E3.m1.1.5.2.3">t</mi></msub><mo id="S3.E3.m1.1.5.3">;</mo><msub id="S3.E3.m1.1.5.4"><mi id="S3.E3.m1.1.5.4.2">μ</mi><mi id="S3.E3.m1.1.5.4.3">t</mi></msub><mo id="S3.E3.m1.1.5.5">=</mo><msqrt id="S3.E3.m1.1.5.6"><mrow id="S3.E3.m1.1.5.6.2"><mn id="S3.E3.m1.1.5.6.2.2">1</mn><mo id="S3.E3.m1.1.5.6.2.1">−</mo><msub id="S3.E3.m1.1.5.6.2.3"><mi id="S3.E3.m1.1.5.6.2.3.2">β</mi><mi id="S3.E3.m1.1.5.6.2.3.3">t</mi></msub></mrow></msqrt><msub id="S3.E3.m1.1.5.7"><mi id="S3.E3.m1.1.5.7.2">z</mi><mrow id="S3.E3.m1.1.5.7.3"><mi id="S3.E3.m1.1.5.7.3.2">t</mi><mo id="S3.E3.m1.1.5.7.3.1">−</mo><mn id="S3.E3.m1.1.5.7.3.3">1</mn></mrow></msub><mo id="S3.E3.m1.1.5.8">,</mo><msub id="S3.E3.m1.1.5.9"><mi id="S3.E3.m1.1.5.9.2">β</mi><mi id="S3.E3.m1.1.5.9.3">t</mi></msub><mi id="S3.E3.m1.1.5.10">I</mi><mo stretchy="false" id="S3.E3.m1.1.5.11">)</mo></mrow></mrow><annotation encoding="application/x-tex" id="S3.E3.m1.1c">p(z_{t}|z_{t-1})=\mathcal{N}(z_{t};\mu_{t}=\sqrt{1-\beta_{t}}z_{t-1},\beta_{t}I)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S3.SS4.p2.6" class="ltx_p">where <math id="S3.SS4.p2.5.m1.2" class="ltx_Math" alttext="\beta_{t}\in(0,1)" display="inline"><semantics id="S3.SS4.p2.5.m1.2a"><mrow id="S3.SS4.p2.5.m1.2.3" xref="S3.SS4.p2.5.m1.2.3.cmml"><msub id="S3.SS4.p2.5.m1.2.3.2" xref="S3.SS4.p2.5.m1.2.3.2.cmml"><mi id="S3.SS4.p2.5.m1.2.3.2.2" xref="S3.SS4.p2.5.m1.2.3.2.2.cmml">β</mi><mi id="S3.SS4.p2.5.m1.2.3.2.3" xref="S3.SS4.p2.5.m1.2.3.2.3.cmml">t</mi></msub><mo id="S3.SS4.p2.5.m1.2.3.1" xref="S3.SS4.p2.5.m1.2.3.1.cmml">∈</mo><mrow id="S3.SS4.p2.5.m1.2.3.3.2" xref="S3.SS4.p2.5.m1.2.3.3.1.cmml"><mo stretchy="false" id="S3.SS4.p2.5.m1.2.3.3.2.1" xref="S3.SS4.p2.5.m1.2.3.3.1.cmml">(</mo><mn id="S3.SS4.p2.5.m1.1.1" xref="S3.SS4.p2.5.m1.1.1.cmml">0</mn><mo id="S3.SS4.p2.5.m1.2.3.3.2.2" xref="S3.SS4.p2.5.m1.2.3.3.1.cmml">,</mo><mn id="S3.SS4.p2.5.m1.2.2" xref="S3.SS4.p2.5.m1.2.2.cmml">1</mn><mo stretchy="false" id="S3.SS4.p2.5.m1.2.3.3.2.3" xref="S3.SS4.p2.5.m1.2.3.3.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.5.m1.2b"><apply id="S3.SS4.p2.5.m1.2.3.cmml" xref="S3.SS4.p2.5.m1.2.3"><in id="S3.SS4.p2.5.m1.2.3.1.cmml" xref="S3.SS4.p2.5.m1.2.3.1"></in><apply id="S3.SS4.p2.5.m1.2.3.2.cmml" xref="S3.SS4.p2.5.m1.2.3.2"><csymbol cd="ambiguous" id="S3.SS4.p2.5.m1.2.3.2.1.cmml" xref="S3.SS4.p2.5.m1.2.3.2">subscript</csymbol><ci id="S3.SS4.p2.5.m1.2.3.2.2.cmml" xref="S3.SS4.p2.5.m1.2.3.2.2">𝛽</ci><ci id="S3.SS4.p2.5.m1.2.3.2.3.cmml" xref="S3.SS4.p2.5.m1.2.3.2.3">𝑡</ci></apply><interval closure="open" id="S3.SS4.p2.5.m1.2.3.3.1.cmml" xref="S3.SS4.p2.5.m1.2.3.3.2"><cn type="integer" id="S3.SS4.p2.5.m1.1.1.cmml" xref="S3.SS4.p2.5.m1.1.1">0</cn><cn type="integer" id="S3.SS4.p2.5.m1.2.2.cmml" xref="S3.SS4.p2.5.m1.2.2">1</cn></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.5.m1.2c">\beta_{t}\in(0,1)</annotation></semantics></math> is the noise variance and <math id="S3.SS4.p2.6.m2.2" class="ltx_Math" alttext="p(z_{1:T}|z_{0})=\prod_{i=1}^{T}p(z_{t}|z_{t-1})" display="inline"><semantics id="S3.SS4.p2.6.m2.2a"><mrow id="S3.SS4.p2.6.m2.2.2" xref="S3.SS4.p2.6.m2.2.2.cmml"><mrow id="S3.SS4.p2.6.m2.1.1.1" xref="S3.SS4.p2.6.m2.1.1.1.cmml"><mi id="S3.SS4.p2.6.m2.1.1.1.3" xref="S3.SS4.p2.6.m2.1.1.1.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p2.6.m2.1.1.1.2" xref="S3.SS4.p2.6.m2.1.1.1.2.cmml">​</mo><mrow id="S3.SS4.p2.6.m2.1.1.1.1.1" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS4.p2.6.m2.1.1.1.1.1.2" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS4.p2.6.m2.1.1.1.1.1.1" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.cmml"><msub id="S3.SS4.p2.6.m2.1.1.1.1.1.1.2" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.cmml"><mi id="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.2" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.2.cmml">z</mi><mrow id="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.3" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.3.cmml"><mn id="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.3.2" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.3.2.cmml">1</mn><mo lspace="0.278em" rspace="0.278em" id="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.3.1" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.3.1.cmml">:</mo><mi id="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.3.3" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.3.3.cmml">T</mi></mrow></msub><mo fence="false" id="S3.SS4.p2.6.m2.1.1.1.1.1.1.1" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.1.cmml">|</mo><msub id="S3.SS4.p2.6.m2.1.1.1.1.1.1.3" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.3.cmml"><mi id="S3.SS4.p2.6.m2.1.1.1.1.1.1.3.2" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.3.2.cmml">z</mi><mn id="S3.SS4.p2.6.m2.1.1.1.1.1.1.3.3" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.3.3.cmml">0</mn></msub></mrow><mo stretchy="false" id="S3.SS4.p2.6.m2.1.1.1.1.1.3" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.111em" id="S3.SS4.p2.6.m2.2.2.3" xref="S3.SS4.p2.6.m2.2.2.3.cmml">=</mo><mrow id="S3.SS4.p2.6.m2.2.2.2" xref="S3.SS4.p2.6.m2.2.2.2.cmml"><msubsup id="S3.SS4.p2.6.m2.2.2.2.2" xref="S3.SS4.p2.6.m2.2.2.2.2.cmml"><mo id="S3.SS4.p2.6.m2.2.2.2.2.2.2" xref="S3.SS4.p2.6.m2.2.2.2.2.2.2.cmml">∏</mo><mrow id="S3.SS4.p2.6.m2.2.2.2.2.2.3" xref="S3.SS4.p2.6.m2.2.2.2.2.2.3.cmml"><mi id="S3.SS4.p2.6.m2.2.2.2.2.2.3.2" xref="S3.SS4.p2.6.m2.2.2.2.2.2.3.2.cmml">i</mi><mo id="S3.SS4.p2.6.m2.2.2.2.2.2.3.1" xref="S3.SS4.p2.6.m2.2.2.2.2.2.3.1.cmml">=</mo><mn id="S3.SS4.p2.6.m2.2.2.2.2.2.3.3" xref="S3.SS4.p2.6.m2.2.2.2.2.2.3.3.cmml">1</mn></mrow><mi id="S3.SS4.p2.6.m2.2.2.2.2.3" xref="S3.SS4.p2.6.m2.2.2.2.2.3.cmml">T</mi></msubsup><mrow id="S3.SS4.p2.6.m2.2.2.2.1" xref="S3.SS4.p2.6.m2.2.2.2.1.cmml"><mi id="S3.SS4.p2.6.m2.2.2.2.1.3" xref="S3.SS4.p2.6.m2.2.2.2.1.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p2.6.m2.2.2.2.1.2" xref="S3.SS4.p2.6.m2.2.2.2.1.2.cmml">​</mo><mrow id="S3.SS4.p2.6.m2.2.2.2.1.1.1" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS4.p2.6.m2.2.2.2.1.1.1.2" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.cmml">(</mo><mrow id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.cmml"><msub id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.2" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.2.cmml"><mi id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.2.2" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.2.2.cmml">z</mi><mi id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.2.3" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.2.3.cmml">t</mi></msub><mo fence="false" id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.1" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.1.cmml">|</mo><msub id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.cmml"><mi id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.2" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.2.cmml">z</mi><mrow id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.3" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.3.cmml"><mi id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.3.2" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.3.2.cmml">t</mi><mo id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.3.1" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.3.1.cmml">−</mo><mn id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.3.3" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.3.3.cmml">1</mn></mrow></msub></mrow><mo stretchy="false" id="S3.SS4.p2.6.m2.2.2.2.1.1.1.3" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p2.6.m2.2b"><apply id="S3.SS4.p2.6.m2.2.2.cmml" xref="S3.SS4.p2.6.m2.2.2"><eq id="S3.SS4.p2.6.m2.2.2.3.cmml" xref="S3.SS4.p2.6.m2.2.2.3"></eq><apply id="S3.SS4.p2.6.m2.1.1.1.cmml" xref="S3.SS4.p2.6.m2.1.1.1"><times id="S3.SS4.p2.6.m2.1.1.1.2.cmml" xref="S3.SS4.p2.6.m2.1.1.1.2"></times><ci id="S3.SS4.p2.6.m2.1.1.1.3.cmml" xref="S3.SS4.p2.6.m2.1.1.1.3">𝑝</ci><apply id="S3.SS4.p2.6.m2.1.1.1.1.1.1.cmml" xref="S3.SS4.p2.6.m2.1.1.1.1.1"><csymbol cd="latexml" id="S3.SS4.p2.6.m2.1.1.1.1.1.1.1.cmml" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.1">conditional</csymbol><apply id="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.cmml" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.1.cmml" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.2.cmml" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.2">𝑧</ci><apply id="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.3.cmml" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.3"><ci id="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.3.1.cmml" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.3.1">:</ci><cn type="integer" id="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.3.2.cmml" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.3.2">1</cn><ci id="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.3.3.cmml" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.2.3.3">𝑇</ci></apply></apply><apply id="S3.SS4.p2.6.m2.1.1.1.1.1.1.3.cmml" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS4.p2.6.m2.1.1.1.1.1.1.3.1.cmml" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.SS4.p2.6.m2.1.1.1.1.1.1.3.2.cmml" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.3.2">𝑧</ci><cn type="integer" id="S3.SS4.p2.6.m2.1.1.1.1.1.1.3.3.cmml" xref="S3.SS4.p2.6.m2.1.1.1.1.1.1.3.3">0</cn></apply></apply></apply><apply id="S3.SS4.p2.6.m2.2.2.2.cmml" xref="S3.SS4.p2.6.m2.2.2.2"><apply id="S3.SS4.p2.6.m2.2.2.2.2.cmml" xref="S3.SS4.p2.6.m2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS4.p2.6.m2.2.2.2.2.1.cmml" xref="S3.SS4.p2.6.m2.2.2.2.2">superscript</csymbol><apply id="S3.SS4.p2.6.m2.2.2.2.2.2.cmml" xref="S3.SS4.p2.6.m2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS4.p2.6.m2.2.2.2.2.2.1.cmml" xref="S3.SS4.p2.6.m2.2.2.2.2">subscript</csymbol><csymbol cd="latexml" id="S3.SS4.p2.6.m2.2.2.2.2.2.2.cmml" xref="S3.SS4.p2.6.m2.2.2.2.2.2.2">product</csymbol><apply id="S3.SS4.p2.6.m2.2.2.2.2.2.3.cmml" xref="S3.SS4.p2.6.m2.2.2.2.2.2.3"><eq id="S3.SS4.p2.6.m2.2.2.2.2.2.3.1.cmml" xref="S3.SS4.p2.6.m2.2.2.2.2.2.3.1"></eq><ci id="S3.SS4.p2.6.m2.2.2.2.2.2.3.2.cmml" xref="S3.SS4.p2.6.m2.2.2.2.2.2.3.2">𝑖</ci><cn type="integer" id="S3.SS4.p2.6.m2.2.2.2.2.2.3.3.cmml" xref="S3.SS4.p2.6.m2.2.2.2.2.2.3.3">1</cn></apply></apply><ci id="S3.SS4.p2.6.m2.2.2.2.2.3.cmml" xref="S3.SS4.p2.6.m2.2.2.2.2.3">𝑇</ci></apply><apply id="S3.SS4.p2.6.m2.2.2.2.1.cmml" xref="S3.SS4.p2.6.m2.2.2.2.1"><times id="S3.SS4.p2.6.m2.2.2.2.1.2.cmml" xref="S3.SS4.p2.6.m2.2.2.2.1.2"></times><ci id="S3.SS4.p2.6.m2.2.2.2.1.3.cmml" xref="S3.SS4.p2.6.m2.2.2.2.1.3">𝑝</ci><apply id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.cmml" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1"><csymbol cd="latexml" id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.1.cmml" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.1">conditional</csymbol><apply id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.2.cmml" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.2.1.cmml" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.2">subscript</csymbol><ci id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.2.2.cmml" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.2.2">𝑧</ci><ci id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.2.3.cmml" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.2.3">𝑡</ci></apply><apply id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.cmml" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.1.cmml" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3">subscript</csymbol><ci id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.2.cmml" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.2">𝑧</ci><apply id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.3.cmml" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.3"><minus id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.3.1.cmml" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.3.1"></minus><ci id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.3.2.cmml" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.3.2">𝑡</ci><cn type="integer" id="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.3.3.cmml" xref="S3.SS4.p2.6.m2.2.2.2.1.1.1.1.3.3.3">1</cn></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p2.6.m2.2c">p(z_{1:T}|z_{0})=\prod_{i=1}^{T}p(z_{t}|z_{t-1})</annotation></semantics></math>.</p>
</div>
<div id="S3.SS4.p3" class="ltx_para">
<p id="S3.SS4.p3.3" class="ltx_p"><span id="S3.SS4.p3.3.1" class="ltx_text ltx_font_bold">Reverse Process:</span> As in most DDPM approaches the reverse process is approximated by a neural network such that:</p>
<table id="S3.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E4.m1.3" class="ltx_Math" alttext="p_{\theta}(z_{t-1}|z_{t})=\mathcal{N}(z_{t-1};\mu_{\theta}(z_{t},t),\Sigma_{\theta}(z_{t},t))," display="block"><semantics id="S3.E4.m1.3a"><mrow id="S3.E4.m1.3.3.1" xref="S3.E4.m1.3.3.1.1.cmml"><mrow id="S3.E4.m1.3.3.1.1" xref="S3.E4.m1.3.3.1.1.cmml"><mrow id="S3.E4.m1.3.3.1.1.1" xref="S3.E4.m1.3.3.1.1.1.cmml"><msub id="S3.E4.m1.3.3.1.1.1.3" xref="S3.E4.m1.3.3.1.1.1.3.cmml"><mi id="S3.E4.m1.3.3.1.1.1.3.2" xref="S3.E4.m1.3.3.1.1.1.3.2.cmml">p</mi><mi id="S3.E4.m1.3.3.1.1.1.3.3" xref="S3.E4.m1.3.3.1.1.1.3.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.E4.m1.3.3.1.1.1.2" xref="S3.E4.m1.3.3.1.1.1.2.cmml">​</mo><mrow id="S3.E4.m1.3.3.1.1.1.1.1" xref="S3.E4.m1.3.3.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E4.m1.3.3.1.1.1.1.1.2" xref="S3.E4.m1.3.3.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E4.m1.3.3.1.1.1.1.1.1" xref="S3.E4.m1.3.3.1.1.1.1.1.1.cmml"><msub id="S3.E4.m1.3.3.1.1.1.1.1.1.2" xref="S3.E4.m1.3.3.1.1.1.1.1.1.2.cmml"><mi id="S3.E4.m1.3.3.1.1.1.1.1.1.2.2" xref="S3.E4.m1.3.3.1.1.1.1.1.1.2.2.cmml">z</mi><mrow id="S3.E4.m1.3.3.1.1.1.1.1.1.2.3" xref="S3.E4.m1.3.3.1.1.1.1.1.1.2.3.cmml"><mi id="S3.E4.m1.3.3.1.1.1.1.1.1.2.3.2" xref="S3.E4.m1.3.3.1.1.1.1.1.1.2.3.2.cmml">t</mi><mo id="S3.E4.m1.3.3.1.1.1.1.1.1.2.3.1" xref="S3.E4.m1.3.3.1.1.1.1.1.1.2.3.1.cmml">−</mo><mn id="S3.E4.m1.3.3.1.1.1.1.1.1.2.3.3" xref="S3.E4.m1.3.3.1.1.1.1.1.1.2.3.3.cmml">1</mn></mrow></msub><mo fence="false" id="S3.E4.m1.3.3.1.1.1.1.1.1.1" xref="S3.E4.m1.3.3.1.1.1.1.1.1.1.cmml">|</mo><msub id="S3.E4.m1.3.3.1.1.1.1.1.1.3" xref="S3.E4.m1.3.3.1.1.1.1.1.1.3.cmml"><mi id="S3.E4.m1.3.3.1.1.1.1.1.1.3.2" xref="S3.E4.m1.3.3.1.1.1.1.1.1.3.2.cmml">z</mi><mi id="S3.E4.m1.3.3.1.1.1.1.1.1.3.3" xref="S3.E4.m1.3.3.1.1.1.1.1.1.3.3.cmml">t</mi></msub></mrow><mo stretchy="false" id="S3.E4.m1.3.3.1.1.1.1.1.3" xref="S3.E4.m1.3.3.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.E4.m1.3.3.1.1.5" xref="S3.E4.m1.3.3.1.1.5.cmml">=</mo><mrow id="S3.E4.m1.3.3.1.1.4" xref="S3.E4.m1.3.3.1.1.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E4.m1.3.3.1.1.4.5" xref="S3.E4.m1.3.3.1.1.4.5.cmml">𝒩</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.3.3.1.1.4.4" xref="S3.E4.m1.3.3.1.1.4.4.cmml">​</mo><mrow id="S3.E4.m1.3.3.1.1.4.3.3" xref="S3.E4.m1.3.3.1.1.4.3.4.cmml"><mo stretchy="false" id="S3.E4.m1.3.3.1.1.4.3.3.4" xref="S3.E4.m1.3.3.1.1.4.3.4.cmml">(</mo><msub id="S3.E4.m1.3.3.1.1.2.1.1.1" xref="S3.E4.m1.3.3.1.1.2.1.1.1.cmml"><mi id="S3.E4.m1.3.3.1.1.2.1.1.1.2" xref="S3.E4.m1.3.3.1.1.2.1.1.1.2.cmml">z</mi><mrow id="S3.E4.m1.3.3.1.1.2.1.1.1.3" xref="S3.E4.m1.3.3.1.1.2.1.1.1.3.cmml"><mi id="S3.E4.m1.3.3.1.1.2.1.1.1.3.2" xref="S3.E4.m1.3.3.1.1.2.1.1.1.3.2.cmml">t</mi><mo id="S3.E4.m1.3.3.1.1.2.1.1.1.3.1" xref="S3.E4.m1.3.3.1.1.2.1.1.1.3.1.cmml">−</mo><mn id="S3.E4.m1.3.3.1.1.2.1.1.1.3.3" xref="S3.E4.m1.3.3.1.1.2.1.1.1.3.3.cmml">1</mn></mrow></msub><mo id="S3.E4.m1.3.3.1.1.4.3.3.5" xref="S3.E4.m1.3.3.1.1.4.3.4.cmml">;</mo><mrow id="S3.E4.m1.3.3.1.1.3.2.2.2" xref="S3.E4.m1.3.3.1.1.3.2.2.2.cmml"><msub id="S3.E4.m1.3.3.1.1.3.2.2.2.3" xref="S3.E4.m1.3.3.1.1.3.2.2.2.3.cmml"><mi id="S3.E4.m1.3.3.1.1.3.2.2.2.3.2" xref="S3.E4.m1.3.3.1.1.3.2.2.2.3.2.cmml">μ</mi><mi id="S3.E4.m1.3.3.1.1.3.2.2.2.3.3" xref="S3.E4.m1.3.3.1.1.3.2.2.2.3.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.E4.m1.3.3.1.1.3.2.2.2.2" xref="S3.E4.m1.3.3.1.1.3.2.2.2.2.cmml">​</mo><mrow id="S3.E4.m1.3.3.1.1.3.2.2.2.1.1" xref="S3.E4.m1.3.3.1.1.3.2.2.2.1.2.cmml"><mo stretchy="false" id="S3.E4.m1.3.3.1.1.3.2.2.2.1.1.2" xref="S3.E4.m1.3.3.1.1.3.2.2.2.1.2.cmml">(</mo><msub id="S3.E4.m1.3.3.1.1.3.2.2.2.1.1.1" xref="S3.E4.m1.3.3.1.1.3.2.2.2.1.1.1.cmml"><mi id="S3.E4.m1.3.3.1.1.3.2.2.2.1.1.1.2" xref="S3.E4.m1.3.3.1.1.3.2.2.2.1.1.1.2.cmml">z</mi><mi id="S3.E4.m1.3.3.1.1.3.2.2.2.1.1.1.3" xref="S3.E4.m1.3.3.1.1.3.2.2.2.1.1.1.3.cmml">t</mi></msub><mo id="S3.E4.m1.3.3.1.1.3.2.2.2.1.1.3" xref="S3.E4.m1.3.3.1.1.3.2.2.2.1.2.cmml">,</mo><mi id="S3.E4.m1.1.1" xref="S3.E4.m1.1.1.cmml">t</mi><mo stretchy="false" id="S3.E4.m1.3.3.1.1.3.2.2.2.1.1.4" xref="S3.E4.m1.3.3.1.1.3.2.2.2.1.2.cmml">)</mo></mrow></mrow><mo id="S3.E4.m1.3.3.1.1.4.3.3.6" xref="S3.E4.m1.3.3.1.1.4.3.4.cmml">,</mo><mrow id="S3.E4.m1.3.3.1.1.4.3.3.3" xref="S3.E4.m1.3.3.1.1.4.3.3.3.cmml"><msub id="S3.E4.m1.3.3.1.1.4.3.3.3.3" xref="S3.E4.m1.3.3.1.1.4.3.3.3.3.cmml"><mi mathvariant="normal" id="S3.E4.m1.3.3.1.1.4.3.3.3.3.2" xref="S3.E4.m1.3.3.1.1.4.3.3.3.3.2.cmml">Σ</mi><mi id="S3.E4.m1.3.3.1.1.4.3.3.3.3.3" xref="S3.E4.m1.3.3.1.1.4.3.3.3.3.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="S3.E4.m1.3.3.1.1.4.3.3.3.2" xref="S3.E4.m1.3.3.1.1.4.3.3.3.2.cmml">​</mo><mrow id="S3.E4.m1.3.3.1.1.4.3.3.3.1.1" xref="S3.E4.m1.3.3.1.1.4.3.3.3.1.2.cmml"><mo stretchy="false" id="S3.E4.m1.3.3.1.1.4.3.3.3.1.1.2" xref="S3.E4.m1.3.3.1.1.4.3.3.3.1.2.cmml">(</mo><msub id="S3.E4.m1.3.3.1.1.4.3.3.3.1.1.1" xref="S3.E4.m1.3.3.1.1.4.3.3.3.1.1.1.cmml"><mi id="S3.E4.m1.3.3.1.1.4.3.3.3.1.1.1.2" xref="S3.E4.m1.3.3.1.1.4.3.3.3.1.1.1.2.cmml">z</mi><mi id="S3.E4.m1.3.3.1.1.4.3.3.3.1.1.1.3" xref="S3.E4.m1.3.3.1.1.4.3.3.3.1.1.1.3.cmml">t</mi></msub><mo id="S3.E4.m1.3.3.1.1.4.3.3.3.1.1.3" xref="S3.E4.m1.3.3.1.1.4.3.3.3.1.2.cmml">,</mo><mi id="S3.E4.m1.2.2" xref="S3.E4.m1.2.2.cmml">t</mi><mo stretchy="false" id="S3.E4.m1.3.3.1.1.4.3.3.3.1.1.4" xref="S3.E4.m1.3.3.1.1.4.3.3.3.1.2.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S3.E4.m1.3.3.1.1.4.3.3.7" xref="S3.E4.m1.3.3.1.1.4.3.4.cmml">)</mo></mrow></mrow></mrow><mo id="S3.E4.m1.3.3.1.2" xref="S3.E4.m1.3.3.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E4.m1.3b"><apply id="S3.E4.m1.3.3.1.1.cmml" xref="S3.E4.m1.3.3.1"><eq id="S3.E4.m1.3.3.1.1.5.cmml" xref="S3.E4.m1.3.3.1.1.5"></eq><apply id="S3.E4.m1.3.3.1.1.1.cmml" xref="S3.E4.m1.3.3.1.1.1"><times id="S3.E4.m1.3.3.1.1.1.2.cmml" xref="S3.E4.m1.3.3.1.1.1.2"></times><apply id="S3.E4.m1.3.3.1.1.1.3.cmml" xref="S3.E4.m1.3.3.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4.m1.3.3.1.1.1.3.1.cmml" xref="S3.E4.m1.3.3.1.1.1.3">subscript</csymbol><ci id="S3.E4.m1.3.3.1.1.1.3.2.cmml" xref="S3.E4.m1.3.3.1.1.1.3.2">𝑝</ci><ci id="S3.E4.m1.3.3.1.1.1.3.3.cmml" xref="S3.E4.m1.3.3.1.1.1.3.3">𝜃</ci></apply><apply id="S3.E4.m1.3.3.1.1.1.1.1.1.cmml" xref="S3.E4.m1.3.3.1.1.1.1.1"><csymbol cd="latexml" id="S3.E4.m1.3.3.1.1.1.1.1.1.1.cmml" xref="S3.E4.m1.3.3.1.1.1.1.1.1.1">conditional</csymbol><apply id="S3.E4.m1.3.3.1.1.1.1.1.1.2.cmml" xref="S3.E4.m1.3.3.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E4.m1.3.3.1.1.1.1.1.1.2.1.cmml" xref="S3.E4.m1.3.3.1.1.1.1.1.1.2">subscript</csymbol><ci id="S3.E4.m1.3.3.1.1.1.1.1.1.2.2.cmml" xref="S3.E4.m1.3.3.1.1.1.1.1.1.2.2">𝑧</ci><apply id="S3.E4.m1.3.3.1.1.1.1.1.1.2.3.cmml" xref="S3.E4.m1.3.3.1.1.1.1.1.1.2.3"><minus id="S3.E4.m1.3.3.1.1.1.1.1.1.2.3.1.cmml" xref="S3.E4.m1.3.3.1.1.1.1.1.1.2.3.1"></minus><ci id="S3.E4.m1.3.3.1.1.1.1.1.1.2.3.2.cmml" xref="S3.E4.m1.3.3.1.1.1.1.1.1.2.3.2">𝑡</ci><cn type="integer" id="S3.E4.m1.3.3.1.1.1.1.1.1.2.3.3.cmml" xref="S3.E4.m1.3.3.1.1.1.1.1.1.2.3.3">1</cn></apply></apply><apply id="S3.E4.m1.3.3.1.1.1.1.1.1.3.cmml" xref="S3.E4.m1.3.3.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E4.m1.3.3.1.1.1.1.1.1.3.1.cmml" xref="S3.E4.m1.3.3.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E4.m1.3.3.1.1.1.1.1.1.3.2.cmml" xref="S3.E4.m1.3.3.1.1.1.1.1.1.3.2">𝑧</ci><ci id="S3.E4.m1.3.3.1.1.1.1.1.1.3.3.cmml" xref="S3.E4.m1.3.3.1.1.1.1.1.1.3.3">𝑡</ci></apply></apply></apply><apply id="S3.E4.m1.3.3.1.1.4.cmml" xref="S3.E4.m1.3.3.1.1.4"><times id="S3.E4.m1.3.3.1.1.4.4.cmml" xref="S3.E4.m1.3.3.1.1.4.4"></times><ci id="S3.E4.m1.3.3.1.1.4.5.cmml" xref="S3.E4.m1.3.3.1.1.4.5">𝒩</ci><list id="S3.E4.m1.3.3.1.1.4.3.4.cmml" xref="S3.E4.m1.3.3.1.1.4.3.3"><apply id="S3.E4.m1.3.3.1.1.2.1.1.1.cmml" xref="S3.E4.m1.3.3.1.1.2.1.1.1"><csymbol cd="ambiguous" id="S3.E4.m1.3.3.1.1.2.1.1.1.1.cmml" xref="S3.E4.m1.3.3.1.1.2.1.1.1">subscript</csymbol><ci id="S3.E4.m1.3.3.1.1.2.1.1.1.2.cmml" xref="S3.E4.m1.3.3.1.1.2.1.1.1.2">𝑧</ci><apply id="S3.E4.m1.3.3.1.1.2.1.1.1.3.cmml" xref="S3.E4.m1.3.3.1.1.2.1.1.1.3"><minus id="S3.E4.m1.3.3.1.1.2.1.1.1.3.1.cmml" xref="S3.E4.m1.3.3.1.1.2.1.1.1.3.1"></minus><ci id="S3.E4.m1.3.3.1.1.2.1.1.1.3.2.cmml" xref="S3.E4.m1.3.3.1.1.2.1.1.1.3.2">𝑡</ci><cn type="integer" id="S3.E4.m1.3.3.1.1.2.1.1.1.3.3.cmml" xref="S3.E4.m1.3.3.1.1.2.1.1.1.3.3">1</cn></apply></apply><apply id="S3.E4.m1.3.3.1.1.3.2.2.2.cmml" xref="S3.E4.m1.3.3.1.1.3.2.2.2"><times id="S3.E4.m1.3.3.1.1.3.2.2.2.2.cmml" xref="S3.E4.m1.3.3.1.1.3.2.2.2.2"></times><apply id="S3.E4.m1.3.3.1.1.3.2.2.2.3.cmml" xref="S3.E4.m1.3.3.1.1.3.2.2.2.3"><csymbol cd="ambiguous" id="S3.E4.m1.3.3.1.1.3.2.2.2.3.1.cmml" xref="S3.E4.m1.3.3.1.1.3.2.2.2.3">subscript</csymbol><ci id="S3.E4.m1.3.3.1.1.3.2.2.2.3.2.cmml" xref="S3.E4.m1.3.3.1.1.3.2.2.2.3.2">𝜇</ci><ci id="S3.E4.m1.3.3.1.1.3.2.2.2.3.3.cmml" xref="S3.E4.m1.3.3.1.1.3.2.2.2.3.3">𝜃</ci></apply><interval closure="open" id="S3.E4.m1.3.3.1.1.3.2.2.2.1.2.cmml" xref="S3.E4.m1.3.3.1.1.3.2.2.2.1.1"><apply id="S3.E4.m1.3.3.1.1.3.2.2.2.1.1.1.cmml" xref="S3.E4.m1.3.3.1.1.3.2.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.E4.m1.3.3.1.1.3.2.2.2.1.1.1.1.cmml" xref="S3.E4.m1.3.3.1.1.3.2.2.2.1.1.1">subscript</csymbol><ci id="S3.E4.m1.3.3.1.1.3.2.2.2.1.1.1.2.cmml" xref="S3.E4.m1.3.3.1.1.3.2.2.2.1.1.1.2">𝑧</ci><ci id="S3.E4.m1.3.3.1.1.3.2.2.2.1.1.1.3.cmml" xref="S3.E4.m1.3.3.1.1.3.2.2.2.1.1.1.3">𝑡</ci></apply><ci id="S3.E4.m1.1.1.cmml" xref="S3.E4.m1.1.1">𝑡</ci></interval></apply><apply id="S3.E4.m1.3.3.1.1.4.3.3.3.cmml" xref="S3.E4.m1.3.3.1.1.4.3.3.3"><times id="S3.E4.m1.3.3.1.1.4.3.3.3.2.cmml" xref="S3.E4.m1.3.3.1.1.4.3.3.3.2"></times><apply id="S3.E4.m1.3.3.1.1.4.3.3.3.3.cmml" xref="S3.E4.m1.3.3.1.1.4.3.3.3.3"><csymbol cd="ambiguous" id="S3.E4.m1.3.3.1.1.4.3.3.3.3.1.cmml" xref="S3.E4.m1.3.3.1.1.4.3.3.3.3">subscript</csymbol><ci id="S3.E4.m1.3.3.1.1.4.3.3.3.3.2.cmml" xref="S3.E4.m1.3.3.1.1.4.3.3.3.3.2">Σ</ci><ci id="S3.E4.m1.3.3.1.1.4.3.3.3.3.3.cmml" xref="S3.E4.m1.3.3.1.1.4.3.3.3.3.3">𝜃</ci></apply><interval closure="open" id="S3.E4.m1.3.3.1.1.4.3.3.3.1.2.cmml" xref="S3.E4.m1.3.3.1.1.4.3.3.3.1.1"><apply id="S3.E4.m1.3.3.1.1.4.3.3.3.1.1.1.cmml" xref="S3.E4.m1.3.3.1.1.4.3.3.3.1.1.1"><csymbol cd="ambiguous" id="S3.E4.m1.3.3.1.1.4.3.3.3.1.1.1.1.cmml" xref="S3.E4.m1.3.3.1.1.4.3.3.3.1.1.1">subscript</csymbol><ci id="S3.E4.m1.3.3.1.1.4.3.3.3.1.1.1.2.cmml" xref="S3.E4.m1.3.3.1.1.4.3.3.3.1.1.1.2">𝑧</ci><ci id="S3.E4.m1.3.3.1.1.4.3.3.3.1.1.1.3.cmml" xref="S3.E4.m1.3.3.1.1.4.3.3.3.1.1.1.3">𝑡</ci></apply><ci id="S3.E4.m1.2.2.cmml" xref="S3.E4.m1.2.2">𝑡</ci></interval></apply></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E4.m1.3c">p_{\theta}(z_{t-1}|z_{t})=\mathcal{N}(z_{t-1};\mu_{\theta}(z_{t},t),\Sigma_{\theta}(z_{t},t)),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
<p id="S3.SS4.p3.2" class="ltx_p">where <math id="S3.SS4.p3.1.m1.1" class="ltx_Math" alttext="\mu_{\theta}" display="inline"><semantics id="S3.SS4.p3.1.m1.1a"><msub id="S3.SS4.p3.1.m1.1.1" xref="S3.SS4.p3.1.m1.1.1.cmml"><mi id="S3.SS4.p3.1.m1.1.1.2" xref="S3.SS4.p3.1.m1.1.1.2.cmml">μ</mi><mi id="S3.SS4.p3.1.m1.1.1.3" xref="S3.SS4.p3.1.m1.1.1.3.cmml">θ</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p3.1.m1.1b"><apply id="S3.SS4.p3.1.m1.1.1.cmml" xref="S3.SS4.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p3.1.m1.1.1.1.cmml" xref="S3.SS4.p3.1.m1.1.1">subscript</csymbol><ci id="S3.SS4.p3.1.m1.1.1.2.cmml" xref="S3.SS4.p3.1.m1.1.1.2">𝜇</ci><ci id="S3.SS4.p3.1.m1.1.1.3.cmml" xref="S3.SS4.p3.1.m1.1.1.3">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p3.1.m1.1c">\mu_{\theta}</annotation></semantics></math> and <math id="S3.SS4.p3.2.m2.1" class="ltx_Math" alttext="\Sigma_{\theta}" display="inline"><semantics id="S3.SS4.p3.2.m2.1a"><msub id="S3.SS4.p3.2.m2.1.1" xref="S3.SS4.p3.2.m2.1.1.cmml"><mi mathvariant="normal" id="S3.SS4.p3.2.m2.1.1.2" xref="S3.SS4.p3.2.m2.1.1.2.cmml">Σ</mi><mi id="S3.SS4.p3.2.m2.1.1.3" xref="S3.SS4.p3.2.m2.1.1.3.cmml">θ</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p3.2.m2.1b"><apply id="S3.SS4.p3.2.m2.1.1.cmml" xref="S3.SS4.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS4.p3.2.m2.1.1.1.cmml" xref="S3.SS4.p3.2.m2.1.1">subscript</csymbol><ci id="S3.SS4.p3.2.m2.1.1.2.cmml" xref="S3.SS4.p3.2.m2.1.1.2">Σ</ci><ci id="S3.SS4.p3.2.m2.1.1.3.cmml" xref="S3.SS4.p3.2.m2.1.1.3">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p3.2.m2.1c">\Sigma_{\theta}</annotation></semantics></math> are neural networks.</p>
</div>
<div id="S3.SS4.p4" class="ltx_para">
<p id="S3.SS4.p4.1" class="ltx_p"><span id="S3.SS4.p4.1.1" class="ltx_text ltx_font_bold">Dataset-Conditioned Training:</span> The diffusion model is trained on the VAE embeddings <math id="S3.SS4.p4.1.m1.1" class="ltx_Math" alttext="z" display="inline"><semantics id="S3.SS4.p4.1.m1.1a"><mi id="S3.SS4.p4.1.m1.1.1" xref="S3.SS4.p4.1.m1.1.1.cmml">z</mi><annotation-xml encoding="MathML-Content" id="S3.SS4.p4.1.m1.1b"><ci id="S3.SS4.p4.1.m1.1.1.cmml" xref="S3.SS4.p4.1.m1.1.1">𝑧</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p4.1.m1.1c">z</annotation></semantics></math>, conditioned on the dataset encodings through concatenation. To take advantage of existing architectures, we design the VAE model to produce latent representations compatible with existing latent diffusion models with minimal modifications and optimize the following latent diffusion model objective:</p>
<table id="S3.E5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E5.m1.9" class="ltx_Math" alttext="\mathcal{L}_{LDM}=\mathbb{E}_{z,\varepsilon\sim\mathcal{N}(0,1),Z_{\mathfrak{D}},t}\left[||\varepsilon-\varepsilon_{\psi}(z_{t},z_{\mathfrak{D}},t)||^{2}_{2}\right]," display="block"><semantics id="S3.E5.m1.9a"><mrow id="S3.E5.m1.9.9.1" xref="S3.E5.m1.9.9.1.1.cmml"><mrow id="S3.E5.m1.9.9.1.1" xref="S3.E5.m1.9.9.1.1.cmml"><msub id="S3.E5.m1.9.9.1.1.3" xref="S3.E5.m1.9.9.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E5.m1.9.9.1.1.3.2" xref="S3.E5.m1.9.9.1.1.3.2.cmml">ℒ</mi><mrow id="S3.E5.m1.9.9.1.1.3.3" xref="S3.E5.m1.9.9.1.1.3.3.cmml"><mi id="S3.E5.m1.9.9.1.1.3.3.2" xref="S3.E5.m1.9.9.1.1.3.3.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="S3.E5.m1.9.9.1.1.3.3.1" xref="S3.E5.m1.9.9.1.1.3.3.1.cmml">​</mo><mi id="S3.E5.m1.9.9.1.1.3.3.3" xref="S3.E5.m1.9.9.1.1.3.3.3.cmml">D</mi><mo lspace="0em" rspace="0em" id="S3.E5.m1.9.9.1.1.3.3.1a" xref="S3.E5.m1.9.9.1.1.3.3.1.cmml">​</mo><mi id="S3.E5.m1.9.9.1.1.3.3.4" xref="S3.E5.m1.9.9.1.1.3.3.4.cmml">M</mi></mrow></msub><mo id="S3.E5.m1.9.9.1.1.2" xref="S3.E5.m1.9.9.1.1.2.cmml">=</mo><mrow id="S3.E5.m1.9.9.1.1.1" xref="S3.E5.m1.9.9.1.1.1.cmml"><msub id="S3.E5.m1.9.9.1.1.1.3" xref="S3.E5.m1.9.9.1.1.1.3.cmml"><mi id="S3.E5.m1.9.9.1.1.1.3.2" xref="S3.E5.m1.9.9.1.1.1.3.2.cmml">𝔼</mi><mrow id="S3.E5.m1.7.7.7.7" xref="S3.E5.m1.7.7.7.8.cmml"><mrow id="S3.E5.m1.6.6.6.6.1" xref="S3.E5.m1.6.6.6.6.1.cmml"><mrow id="S3.E5.m1.6.6.6.6.1.2.2" xref="S3.E5.m1.6.6.6.6.1.2.1.cmml"><mi id="S3.E5.m1.3.3.3.3" xref="S3.E5.m1.3.3.3.3.cmml">z</mi><mo id="S3.E5.m1.6.6.6.6.1.2.2.1" xref="S3.E5.m1.6.6.6.6.1.2.1.cmml">,</mo><mi id="S3.E5.m1.4.4.4.4" xref="S3.E5.m1.4.4.4.4.cmml">ε</mi></mrow><mo id="S3.E5.m1.6.6.6.6.1.1" xref="S3.E5.m1.6.6.6.6.1.1.cmml">∼</mo><mrow id="S3.E5.m1.6.6.6.6.1.3" xref="S3.E5.m1.6.6.6.6.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E5.m1.6.6.6.6.1.3.2" xref="S3.E5.m1.6.6.6.6.1.3.2.cmml">𝒩</mi><mo lspace="0em" rspace="0em" id="S3.E5.m1.6.6.6.6.1.3.1" xref="S3.E5.m1.6.6.6.6.1.3.1.cmml">​</mo><mrow id="S3.E5.m1.6.6.6.6.1.3.3.2" xref="S3.E5.m1.6.6.6.6.1.3.3.1.cmml"><mo stretchy="false" id="S3.E5.m1.6.6.6.6.1.3.3.2.1" xref="S3.E5.m1.6.6.6.6.1.3.3.1.cmml">(</mo><mn id="S3.E5.m1.1.1.1.1" xref="S3.E5.m1.1.1.1.1.cmml">0</mn><mo id="S3.E5.m1.6.6.6.6.1.3.3.2.2" xref="S3.E5.m1.6.6.6.6.1.3.3.1.cmml">,</mo><mn id="S3.E5.m1.2.2.2.2" xref="S3.E5.m1.2.2.2.2.cmml">1</mn><mo stretchy="false" id="S3.E5.m1.6.6.6.6.1.3.3.2.3" xref="S3.E5.m1.6.6.6.6.1.3.3.1.cmml">)</mo></mrow></mrow></mrow><mo id="S3.E5.m1.7.7.7.7.3" xref="S3.E5.m1.7.7.7.8a.cmml">,</mo><mrow id="S3.E5.m1.7.7.7.7.2.1" xref="S3.E5.m1.7.7.7.7.2.2.cmml"><msub id="S3.E5.m1.7.7.7.7.2.1.1" xref="S3.E5.m1.7.7.7.7.2.1.1.cmml"><mi id="S3.E5.m1.7.7.7.7.2.1.1.2" xref="S3.E5.m1.7.7.7.7.2.1.1.2.cmml">Z</mi><mi id="S3.E5.m1.7.7.7.7.2.1.1.3" xref="S3.E5.m1.7.7.7.7.2.1.1.3.cmml">𝔇</mi></msub><mo id="S3.E5.m1.7.7.7.7.2.1.2" xref="S3.E5.m1.7.7.7.7.2.2.cmml">,</mo><mi id="S3.E5.m1.5.5.5.5" xref="S3.E5.m1.5.5.5.5.cmml">t</mi></mrow></mrow></msub><mo lspace="0em" rspace="0em" id="S3.E5.m1.9.9.1.1.1.2" xref="S3.E5.m1.9.9.1.1.1.2.cmml">​</mo><mrow id="S3.E5.m1.9.9.1.1.1.1.1" xref="S3.E5.m1.9.9.1.1.1.1.2.cmml"><mo id="S3.E5.m1.9.9.1.1.1.1.1.2" xref="S3.E5.m1.9.9.1.1.1.1.2.1.cmml">[</mo><msubsup id="S3.E5.m1.9.9.1.1.1.1.1.1" xref="S3.E5.m1.9.9.1.1.1.1.1.1.cmml"><mrow id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.2" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.2.1.cmml">‖</mo><mrow id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.4" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.4.cmml">ε</mi><mo id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.3.cmml">−</mo><mrow id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.cmml"><msub id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.4" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.4.cmml"><mi id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.4.2" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.4.2.cmml">ε</mi><mi id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.4.3" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.4.3.cmml">ψ</mi></msub><mo lspace="0em" rspace="0em" id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.3" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.3.cmml">​</mo><mrow id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml"><mo stretchy="false" id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2.3" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml">(</mo><msub id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">z</mi><mi id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">t</mi></msub><mo id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2.4" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml">,</mo><msub id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2.2" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2.2.cmml"><mi id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2.2.2" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2.2.2.cmml">z</mi><mi id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2.2.3" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2.2.3.cmml">𝔇</mi></msub><mo id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2.5" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml">,</mo><mi id="S3.E5.m1.8.8" xref="S3.E5.m1.8.8.cmml">t</mi><mo stretchy="false" id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2.6" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.3" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.2.1.cmml">‖</mo></mrow><mn id="S3.E5.m1.9.9.1.1.1.1.1.1.3" xref="S3.E5.m1.9.9.1.1.1.1.1.1.3.cmml">2</mn><mn id="S3.E5.m1.9.9.1.1.1.1.1.1.1.3" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.3.cmml">2</mn></msubsup><mo id="S3.E5.m1.9.9.1.1.1.1.1.3" xref="S3.E5.m1.9.9.1.1.1.1.2.1.cmml">]</mo></mrow></mrow></mrow><mo id="S3.E5.m1.9.9.1.2" xref="S3.E5.m1.9.9.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E5.m1.9b"><apply id="S3.E5.m1.9.9.1.1.cmml" xref="S3.E5.m1.9.9.1"><eq id="S3.E5.m1.9.9.1.1.2.cmml" xref="S3.E5.m1.9.9.1.1.2"></eq><apply id="S3.E5.m1.9.9.1.1.3.cmml" xref="S3.E5.m1.9.9.1.1.3"><csymbol cd="ambiguous" id="S3.E5.m1.9.9.1.1.3.1.cmml" xref="S3.E5.m1.9.9.1.1.3">subscript</csymbol><ci id="S3.E5.m1.9.9.1.1.3.2.cmml" xref="S3.E5.m1.9.9.1.1.3.2">ℒ</ci><apply id="S3.E5.m1.9.9.1.1.3.3.cmml" xref="S3.E5.m1.9.9.1.1.3.3"><times id="S3.E5.m1.9.9.1.1.3.3.1.cmml" xref="S3.E5.m1.9.9.1.1.3.3.1"></times><ci id="S3.E5.m1.9.9.1.1.3.3.2.cmml" xref="S3.E5.m1.9.9.1.1.3.3.2">𝐿</ci><ci id="S3.E5.m1.9.9.1.1.3.3.3.cmml" xref="S3.E5.m1.9.9.1.1.3.3.3">𝐷</ci><ci id="S3.E5.m1.9.9.1.1.3.3.4.cmml" xref="S3.E5.m1.9.9.1.1.3.3.4">𝑀</ci></apply></apply><apply id="S3.E5.m1.9.9.1.1.1.cmml" xref="S3.E5.m1.9.9.1.1.1"><times id="S3.E5.m1.9.9.1.1.1.2.cmml" xref="S3.E5.m1.9.9.1.1.1.2"></times><apply id="S3.E5.m1.9.9.1.1.1.3.cmml" xref="S3.E5.m1.9.9.1.1.1.3"><csymbol cd="ambiguous" id="S3.E5.m1.9.9.1.1.1.3.1.cmml" xref="S3.E5.m1.9.9.1.1.1.3">subscript</csymbol><ci id="S3.E5.m1.9.9.1.1.1.3.2.cmml" xref="S3.E5.m1.9.9.1.1.1.3.2">𝔼</ci><apply id="S3.E5.m1.7.7.7.8.cmml" xref="S3.E5.m1.7.7.7.7"><csymbol cd="ambiguous" id="S3.E5.m1.7.7.7.8a.cmml" xref="S3.E5.m1.7.7.7.7.3">formulae-sequence</csymbol><apply id="S3.E5.m1.6.6.6.6.1.cmml" xref="S3.E5.m1.6.6.6.6.1"><csymbol cd="latexml" id="S3.E5.m1.6.6.6.6.1.1.cmml" xref="S3.E5.m1.6.6.6.6.1.1">similar-to</csymbol><list id="S3.E5.m1.6.6.6.6.1.2.1.cmml" xref="S3.E5.m1.6.6.6.6.1.2.2"><ci id="S3.E5.m1.3.3.3.3.cmml" xref="S3.E5.m1.3.3.3.3">𝑧</ci><ci id="S3.E5.m1.4.4.4.4.cmml" xref="S3.E5.m1.4.4.4.4">𝜀</ci></list><apply id="S3.E5.m1.6.6.6.6.1.3.cmml" xref="S3.E5.m1.6.6.6.6.1.3"><times id="S3.E5.m1.6.6.6.6.1.3.1.cmml" xref="S3.E5.m1.6.6.6.6.1.3.1"></times><ci id="S3.E5.m1.6.6.6.6.1.3.2.cmml" xref="S3.E5.m1.6.6.6.6.1.3.2">𝒩</ci><interval closure="open" id="S3.E5.m1.6.6.6.6.1.3.3.1.cmml" xref="S3.E5.m1.6.6.6.6.1.3.3.2"><cn type="integer" id="S3.E5.m1.1.1.1.1.cmml" xref="S3.E5.m1.1.1.1.1">0</cn><cn type="integer" id="S3.E5.m1.2.2.2.2.cmml" xref="S3.E5.m1.2.2.2.2">1</cn></interval></apply></apply><list id="S3.E5.m1.7.7.7.7.2.2.cmml" xref="S3.E5.m1.7.7.7.7.2.1"><apply id="S3.E5.m1.7.7.7.7.2.1.1.cmml" xref="S3.E5.m1.7.7.7.7.2.1.1"><csymbol cd="ambiguous" id="S3.E5.m1.7.7.7.7.2.1.1.1.cmml" xref="S3.E5.m1.7.7.7.7.2.1.1">subscript</csymbol><ci id="S3.E5.m1.7.7.7.7.2.1.1.2.cmml" xref="S3.E5.m1.7.7.7.7.2.1.1.2">𝑍</ci><ci id="S3.E5.m1.7.7.7.7.2.1.1.3.cmml" xref="S3.E5.m1.7.7.7.7.2.1.1.3">𝔇</ci></apply><ci id="S3.E5.m1.5.5.5.5.cmml" xref="S3.E5.m1.5.5.5.5">𝑡</ci></list></apply></apply><apply id="S3.E5.m1.9.9.1.1.1.1.2.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1"><csymbol cd="latexml" id="S3.E5.m1.9.9.1.1.1.1.2.1.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.2">delimited-[]</csymbol><apply id="S3.E5.m1.9.9.1.1.1.1.1.1.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E5.m1.9.9.1.1.1.1.1.1.2.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1">subscript</csymbol><apply id="S3.E5.m1.9.9.1.1.1.1.1.1.1.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E5.m1.9.9.1.1.1.1.1.1.1.2.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1">superscript</csymbol><apply id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.2">norm</csymbol><apply id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1"><minus id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.3"></minus><ci id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.4.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.4">𝜀</ci><apply id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2"><times id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.3"></times><apply id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.4.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.4"><csymbol cd="ambiguous" id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.4.1.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.4">subscript</csymbol><ci id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.4.2.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.4.2">𝜀</ci><ci id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.4.3.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.4.3">𝜓</ci></apply><vector id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.3.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2"><apply id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2">𝑧</ci><ci id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3">𝑡</ci></apply><apply id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2.2.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2.2"><csymbol cd="ambiguous" id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2.2.1.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2.2">subscript</csymbol><ci id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2.2.2.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2.2.2">𝑧</ci><ci id="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2.2.3.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.1.1.1.2.2.2.2.3">𝔇</ci></apply><ci id="S3.E5.m1.8.8.cmml" xref="S3.E5.m1.8.8">𝑡</ci></vector></apply></apply></apply><cn type="integer" id="S3.E5.m1.9.9.1.1.1.1.1.1.1.3.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.1.3">2</cn></apply><cn type="integer" id="S3.E5.m1.9.9.1.1.1.1.1.1.3.cmml" xref="S3.E5.m1.9.9.1.1.1.1.1.1.3">2</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E5.m1.9c">\mathcal{L}_{LDM}=\mathbb{E}_{z,\varepsilon\sim\mathcal{N}(0,1),Z_{\mathfrak{D}},t}\left[||\varepsilon-\varepsilon_{\psi}(z_{t},z_{\mathfrak{D}},t)||^{2}_{2}\right],</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
</table>
<p id="S3.SS4.p4.2" class="ltx_p">where <math id="S3.SS4.p4.2.m1.3" class="ltx_Math" alttext="\varepsilon_{\psi}(z_{t},z_{\mathfrak{D}},t)" display="inline"><semantics id="S3.SS4.p4.2.m1.3a"><mrow id="S3.SS4.p4.2.m1.3.3" xref="S3.SS4.p4.2.m1.3.3.cmml"><msub id="S3.SS4.p4.2.m1.3.3.4" xref="S3.SS4.p4.2.m1.3.3.4.cmml"><mi id="S3.SS4.p4.2.m1.3.3.4.2" xref="S3.SS4.p4.2.m1.3.3.4.2.cmml">ε</mi><mi id="S3.SS4.p4.2.m1.3.3.4.3" xref="S3.SS4.p4.2.m1.3.3.4.3.cmml">ψ</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS4.p4.2.m1.3.3.3" xref="S3.SS4.p4.2.m1.3.3.3.cmml">​</mo><mrow id="S3.SS4.p4.2.m1.3.3.2.2" xref="S3.SS4.p4.2.m1.3.3.2.3.cmml"><mo stretchy="false" id="S3.SS4.p4.2.m1.3.3.2.2.3" xref="S3.SS4.p4.2.m1.3.3.2.3.cmml">(</mo><msub id="S3.SS4.p4.2.m1.2.2.1.1.1" xref="S3.SS4.p4.2.m1.2.2.1.1.1.cmml"><mi id="S3.SS4.p4.2.m1.2.2.1.1.1.2" xref="S3.SS4.p4.2.m1.2.2.1.1.1.2.cmml">z</mi><mi id="S3.SS4.p4.2.m1.2.2.1.1.1.3" xref="S3.SS4.p4.2.m1.2.2.1.1.1.3.cmml">t</mi></msub><mo id="S3.SS4.p4.2.m1.3.3.2.2.4" xref="S3.SS4.p4.2.m1.3.3.2.3.cmml">,</mo><msub id="S3.SS4.p4.2.m1.3.3.2.2.2" xref="S3.SS4.p4.2.m1.3.3.2.2.2.cmml"><mi id="S3.SS4.p4.2.m1.3.3.2.2.2.2" xref="S3.SS4.p4.2.m1.3.3.2.2.2.2.cmml">z</mi><mi id="S3.SS4.p4.2.m1.3.3.2.2.2.3" xref="S3.SS4.p4.2.m1.3.3.2.2.2.3.cmml">𝔇</mi></msub><mo id="S3.SS4.p4.2.m1.3.3.2.2.5" xref="S3.SS4.p4.2.m1.3.3.2.3.cmml">,</mo><mi id="S3.SS4.p4.2.m1.1.1" xref="S3.SS4.p4.2.m1.1.1.cmml">t</mi><mo stretchy="false" id="S3.SS4.p4.2.m1.3.3.2.2.6" xref="S3.SS4.p4.2.m1.3.3.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p4.2.m1.3b"><apply id="S3.SS4.p4.2.m1.3.3.cmml" xref="S3.SS4.p4.2.m1.3.3"><times id="S3.SS4.p4.2.m1.3.3.3.cmml" xref="S3.SS4.p4.2.m1.3.3.3"></times><apply id="S3.SS4.p4.2.m1.3.3.4.cmml" xref="S3.SS4.p4.2.m1.3.3.4"><csymbol cd="ambiguous" id="S3.SS4.p4.2.m1.3.3.4.1.cmml" xref="S3.SS4.p4.2.m1.3.3.4">subscript</csymbol><ci id="S3.SS4.p4.2.m1.3.3.4.2.cmml" xref="S3.SS4.p4.2.m1.3.3.4.2">𝜀</ci><ci id="S3.SS4.p4.2.m1.3.3.4.3.cmml" xref="S3.SS4.p4.2.m1.3.3.4.3">𝜓</ci></apply><vector id="S3.SS4.p4.2.m1.3.3.2.3.cmml" xref="S3.SS4.p4.2.m1.3.3.2.2"><apply id="S3.SS4.p4.2.m1.2.2.1.1.1.cmml" xref="S3.SS4.p4.2.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS4.p4.2.m1.2.2.1.1.1.1.cmml" xref="S3.SS4.p4.2.m1.2.2.1.1.1">subscript</csymbol><ci id="S3.SS4.p4.2.m1.2.2.1.1.1.2.cmml" xref="S3.SS4.p4.2.m1.2.2.1.1.1.2">𝑧</ci><ci id="S3.SS4.p4.2.m1.2.2.1.1.1.3.cmml" xref="S3.SS4.p4.2.m1.2.2.1.1.1.3">𝑡</ci></apply><apply id="S3.SS4.p4.2.m1.3.3.2.2.2.cmml" xref="S3.SS4.p4.2.m1.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.SS4.p4.2.m1.3.3.2.2.2.1.cmml" xref="S3.SS4.p4.2.m1.3.3.2.2.2">subscript</csymbol><ci id="S3.SS4.p4.2.m1.3.3.2.2.2.2.cmml" xref="S3.SS4.p4.2.m1.3.3.2.2.2.2">𝑧</ci><ci id="S3.SS4.p4.2.m1.3.3.2.2.2.3.cmml" xref="S3.SS4.p4.2.m1.3.3.2.2.2.3">𝔇</ci></apply><ci id="S3.SS4.p4.2.m1.1.1.cmml" xref="S3.SS4.p4.2.m1.1.1">𝑡</ci></vector></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p4.2.m1.3c">\varepsilon_{\psi}(z_{t},z_{\mathfrak{D}},t)</annotation></semantics></math> is implemented as a UNet.</p>
</div>
<div id="S3.SS4.p5" class="ltx_para">
<p id="S3.SS4.p5.6" class="ltx_p"><span id="S3.SS4.p5.6.1" class="ltx_text ltx_font_bold">Sampling:</span> New weights are sampled conditionally through the revers diffusion process as follows:</p>
<table id="S3.E6" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E6.m1.1" class="ltx_math_unparsed" alttext="z_{t}=\frac{1}{\sqrt{a_{t}}}(z_{t}-\frac{\beta_{t}}{\sqrt{1-\tilde{a}_{t}}}\varepsilon_{\psi}(z_{t},z_{\mathfrak{D}},t,))+\sigma\xi," display="block"><semantics id="S3.E6.m1.1a"><mrow id="S3.E6.m1.1b"><msub id="S3.E6.m1.1.2"><mi id="S3.E6.m1.1.2.2">z</mi><mi id="S3.E6.m1.1.2.3">t</mi></msub><mo id="S3.E6.m1.1.3">=</mo><mfrac id="S3.E6.m1.1.4"><mn id="S3.E6.m1.1.4.2">1</mn><msqrt id="S3.E6.m1.1.4.3"><msub id="S3.E6.m1.1.4.3.2"><mi id="S3.E6.m1.1.4.3.2.2">a</mi><mi id="S3.E6.m1.1.4.3.2.3">t</mi></msub></msqrt></mfrac><mrow id="S3.E6.m1.1.5"><mo stretchy="false" id="S3.E6.m1.1.5.1">(</mo><msub id="S3.E6.m1.1.5.2"><mi id="S3.E6.m1.1.5.2.2">z</mi><mi id="S3.E6.m1.1.5.2.3">t</mi></msub><mo id="S3.E6.m1.1.5.3">−</mo><mfrac id="S3.E6.m1.1.5.4"><msub id="S3.E6.m1.1.5.4.2"><mi id="S3.E6.m1.1.5.4.2.2">β</mi><mi id="S3.E6.m1.1.5.4.2.3">t</mi></msub><msqrt id="S3.E6.m1.1.5.4.3"><mrow id="S3.E6.m1.1.5.4.3.2"><mn id="S3.E6.m1.1.5.4.3.2.2">1</mn><mo id="S3.E6.m1.1.5.4.3.2.1">−</mo><msub id="S3.E6.m1.1.5.4.3.2.3"><mover accent="true" id="S3.E6.m1.1.5.4.3.2.3.2"><mi id="S3.E6.m1.1.5.4.3.2.3.2.2">a</mi><mo id="S3.E6.m1.1.5.4.3.2.3.2.1">~</mo></mover><mi id="S3.E6.m1.1.5.4.3.2.3.3">t</mi></msub></mrow></msqrt></mfrac><msub id="S3.E6.m1.1.5.5"><mi id="S3.E6.m1.1.5.5.2">ε</mi><mi id="S3.E6.m1.1.5.5.3">ψ</mi></msub><mrow id="S3.E6.m1.1.5.6"><mo stretchy="false" id="S3.E6.m1.1.5.6.1">(</mo><msub id="S3.E6.m1.1.5.6.2"><mi id="S3.E6.m1.1.5.6.2.2">z</mi><mi id="S3.E6.m1.1.5.6.2.3">t</mi></msub><mo id="S3.E6.m1.1.5.6.3">,</mo><msub id="S3.E6.m1.1.5.6.4"><mi id="S3.E6.m1.1.5.6.4.2">z</mi><mi id="S3.E6.m1.1.5.6.4.3">𝔇</mi></msub><mo id="S3.E6.m1.1.5.6.5">,</mo><mi id="S3.E6.m1.1.1">t</mi><mo id="S3.E6.m1.1.5.6.6">,</mo><mo stretchy="false" id="S3.E6.m1.1.5.6.7">)</mo></mrow><mo stretchy="false" id="S3.E6.m1.1.5.7">)</mo></mrow><mo id="S3.E6.m1.1.6">+</mo><mi id="S3.E6.m1.1.7">σ</mi><mi id="S3.E6.m1.1.8">ξ</mi><mo id="S3.E6.m1.1.9">,</mo></mrow><annotation encoding="application/x-tex" id="S3.E6.m1.1c">z_{t}=\frac{1}{\sqrt{a_{t}}}(z_{t}-\frac{\beta_{t}}{\sqrt{1-\tilde{a}_{t}}}\varepsilon_{\psi}(z_{t},z_{\mathfrak{D}},t,))+\sigma\xi,</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
</table>
<p id="S3.SS4.p5.5" class="ltx_p">where <math id="S3.SS4.p5.1.m1.2" class="ltx_Math" alttext="\xi\sim\mathcal{N}(0,I)" display="inline"><semantics id="S3.SS4.p5.1.m1.2a"><mrow id="S3.SS4.p5.1.m1.2.3" xref="S3.SS4.p5.1.m1.2.3.cmml"><mi id="S3.SS4.p5.1.m1.2.3.2" xref="S3.SS4.p5.1.m1.2.3.2.cmml">ξ</mi><mo id="S3.SS4.p5.1.m1.2.3.1" xref="S3.SS4.p5.1.m1.2.3.1.cmml">∼</mo><mrow id="S3.SS4.p5.1.m1.2.3.3" xref="S3.SS4.p5.1.m1.2.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS4.p5.1.m1.2.3.3.2" xref="S3.SS4.p5.1.m1.2.3.3.2.cmml">𝒩</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p5.1.m1.2.3.3.1" xref="S3.SS4.p5.1.m1.2.3.3.1.cmml">​</mo><mrow id="S3.SS4.p5.1.m1.2.3.3.3.2" xref="S3.SS4.p5.1.m1.2.3.3.3.1.cmml"><mo stretchy="false" id="S3.SS4.p5.1.m1.2.3.3.3.2.1" xref="S3.SS4.p5.1.m1.2.3.3.3.1.cmml">(</mo><mn id="S3.SS4.p5.1.m1.1.1" xref="S3.SS4.p5.1.m1.1.1.cmml">0</mn><mo id="S3.SS4.p5.1.m1.2.3.3.3.2.2" xref="S3.SS4.p5.1.m1.2.3.3.3.1.cmml">,</mo><mi id="S3.SS4.p5.1.m1.2.2" xref="S3.SS4.p5.1.m1.2.2.cmml">I</mi><mo stretchy="false" id="S3.SS4.p5.1.m1.2.3.3.3.2.3" xref="S3.SS4.p5.1.m1.2.3.3.3.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p5.1.m1.2b"><apply id="S3.SS4.p5.1.m1.2.3.cmml" xref="S3.SS4.p5.1.m1.2.3"><csymbol cd="latexml" id="S3.SS4.p5.1.m1.2.3.1.cmml" xref="S3.SS4.p5.1.m1.2.3.1">similar-to</csymbol><ci id="S3.SS4.p5.1.m1.2.3.2.cmml" xref="S3.SS4.p5.1.m1.2.3.2">𝜉</ci><apply id="S3.SS4.p5.1.m1.2.3.3.cmml" xref="S3.SS4.p5.1.m1.2.3.3"><times id="S3.SS4.p5.1.m1.2.3.3.1.cmml" xref="S3.SS4.p5.1.m1.2.3.3.1"></times><ci id="S3.SS4.p5.1.m1.2.3.3.2.cmml" xref="S3.SS4.p5.1.m1.2.3.3.2">𝒩</ci><interval closure="open" id="S3.SS4.p5.1.m1.2.3.3.3.1.cmml" xref="S3.SS4.p5.1.m1.2.3.3.3.2"><cn type="integer" id="S3.SS4.p5.1.m1.1.1.cmml" xref="S3.SS4.p5.1.m1.1.1">0</cn><ci id="S3.SS4.p5.1.m1.2.2.cmml" xref="S3.SS4.p5.1.m1.2.2">𝐼</ci></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p5.1.m1.2c">\xi\sim\mathcal{N}(0,I)</annotation></semantics></math> and, <math id="S3.SS4.p5.2.m2.1" class="ltx_Math" alttext="\sigma_{t}" display="inline"><semantics id="S3.SS4.p5.2.m2.1a"><msub id="S3.SS4.p5.2.m2.1.1" xref="S3.SS4.p5.2.m2.1.1.cmml"><mi id="S3.SS4.p5.2.m2.1.1.2" xref="S3.SS4.p5.2.m2.1.1.2.cmml">σ</mi><mi id="S3.SS4.p5.2.m2.1.1.3" xref="S3.SS4.p5.2.m2.1.1.3.cmml">t</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS4.p5.2.m2.1b"><apply id="S3.SS4.p5.2.m2.1.1.cmml" xref="S3.SS4.p5.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS4.p5.2.m2.1.1.1.cmml" xref="S3.SS4.p5.2.m2.1.1">subscript</csymbol><ci id="S3.SS4.p5.2.m2.1.1.2.cmml" xref="S3.SS4.p5.2.m2.1.1.2">𝜎</ci><ci id="S3.SS4.p5.2.m2.1.1.3.cmml" xref="S3.SS4.p5.2.m2.1.1.3">𝑡</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p5.2.m2.1c">\sigma_{t}</annotation></semantics></math> a chosen value. After sampling a latent representation <math id="S3.SS4.p5.3.m3.1" class="ltx_Math" alttext="\bar{(}z" display="inline"><semantics id="S3.SS4.p5.3.m3.1a"><mrow id="S3.SS4.p5.3.m3.1.1" xref="S3.SS4.p5.3.m3.1.1.cmml"><mover accent="true" id="S3.SS4.p5.3.m3.1.1.2" xref="S3.SS4.p5.3.m3.1.1.2.cmml"><mo stretchy="false" id="S3.SS4.p5.3.m3.1.1.2.2" xref="S3.SS4.p5.3.m3.1.1.2.2.cmml">(</mo><mo id="S3.SS4.p5.3.m3.1.1.2.1" xref="S3.SS4.p5.3.m3.1.1.2.1.cmml">¯</mo></mover><mo lspace="0em" rspace="0em" id="S3.SS4.p5.3.m3.1.1.1" xref="S3.SS4.p5.3.m3.1.1.1.cmml">​</mo><mi id="S3.SS4.p5.3.m3.1.1.3" xref="S3.SS4.p5.3.m3.1.1.3.cmml">z</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p5.3.m3.1b"><apply id="S3.SS4.p5.3.m3.1.1.cmml" xref="S3.SS4.p5.3.m3.1.1"><times id="S3.SS4.p5.3.m3.1.1.1.cmml" xref="S3.SS4.p5.3.m3.1.1.1"></times><apply id="S3.SS4.p5.3.m3.1.1.2.cmml" xref="S3.SS4.p5.3.m3.1.1.2"><ci id="S3.SS4.p5.3.m3.1.1.2.1.cmml" xref="S3.SS4.p5.3.m3.1.1.2.1">¯</ci><ci id="S3.SS4.p5.3.m3.1.1.2.2.cmml" xref="S3.SS4.p5.3.m3.1.1.2.2">(</ci></apply><ci id="S3.SS4.p5.3.m3.1.1.3.cmml" xref="S3.SS4.p5.3.m3.1.1.3">𝑧</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p5.3.m3.1c">\bar{(}z</annotation></semantics></math> for a given dataset <math id="S3.SS4.p5.4.m4.1" class="ltx_math_unparsed" alttext="\mathfrak{D}_{i})" display="inline"><semantics id="S3.SS4.p5.4.m4.1a"><mrow id="S3.SS4.p5.4.m4.1b"><msub id="S3.SS4.p5.4.m4.1.1"><mi id="S3.SS4.p5.4.m4.1.1.2">𝔇</mi><mi id="S3.SS4.p5.4.m4.1.1.3">i</mi></msub><mo stretchy="false" id="S3.SS4.p5.4.m4.1.2">)</mo></mrow><annotation encoding="application/x-tex" id="S3.SS4.p5.4.m4.1c">\mathfrak{D}_{i})</annotation></semantics></math>. The pretrained VAE decoder is used to transform that latents into a weight <math id="S3.SS4.p5.5.m5.1" class="ltx_Math" alttext="\bar{w}=\mathcal{D}(\bar{z})" display="inline"><semantics id="S3.SS4.p5.5.m5.1a"><mrow id="S3.SS4.p5.5.m5.1.2" xref="S3.SS4.p5.5.m5.1.2.cmml"><mover accent="true" id="S3.SS4.p5.5.m5.1.2.2" xref="S3.SS4.p5.5.m5.1.2.2.cmml"><mi id="S3.SS4.p5.5.m5.1.2.2.2" xref="S3.SS4.p5.5.m5.1.2.2.2.cmml">w</mi><mo id="S3.SS4.p5.5.m5.1.2.2.1" xref="S3.SS4.p5.5.m5.1.2.2.1.cmml">¯</mo></mover><mo id="S3.SS4.p5.5.m5.1.2.1" xref="S3.SS4.p5.5.m5.1.2.1.cmml">=</mo><mrow id="S3.SS4.p5.5.m5.1.2.3" xref="S3.SS4.p5.5.m5.1.2.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS4.p5.5.m5.1.2.3.2" xref="S3.SS4.p5.5.m5.1.2.3.2.cmml">𝒟</mi><mo lspace="0em" rspace="0em" id="S3.SS4.p5.5.m5.1.2.3.1" xref="S3.SS4.p5.5.m5.1.2.3.1.cmml">​</mo><mrow id="S3.SS4.p5.5.m5.1.2.3.3.2" xref="S3.SS4.p5.5.m5.1.1.cmml"><mo stretchy="false" id="S3.SS4.p5.5.m5.1.2.3.3.2.1" xref="S3.SS4.p5.5.m5.1.1.cmml">(</mo><mover accent="true" id="S3.SS4.p5.5.m5.1.1" xref="S3.SS4.p5.5.m5.1.1.cmml"><mi id="S3.SS4.p5.5.m5.1.1.2" xref="S3.SS4.p5.5.m5.1.1.2.cmml">z</mi><mo id="S3.SS4.p5.5.m5.1.1.1" xref="S3.SS4.p5.5.m5.1.1.1.cmml">¯</mo></mover><mo stretchy="false" id="S3.SS4.p5.5.m5.1.2.3.3.2.2" xref="S3.SS4.p5.5.m5.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS4.p5.5.m5.1b"><apply id="S3.SS4.p5.5.m5.1.2.cmml" xref="S3.SS4.p5.5.m5.1.2"><eq id="S3.SS4.p5.5.m5.1.2.1.cmml" xref="S3.SS4.p5.5.m5.1.2.1"></eq><apply id="S3.SS4.p5.5.m5.1.2.2.cmml" xref="S3.SS4.p5.5.m5.1.2.2"><ci id="S3.SS4.p5.5.m5.1.2.2.1.cmml" xref="S3.SS4.p5.5.m5.1.2.2.1">¯</ci><ci id="S3.SS4.p5.5.m5.1.2.2.2.cmml" xref="S3.SS4.p5.5.m5.1.2.2.2">𝑤</ci></apply><apply id="S3.SS4.p5.5.m5.1.2.3.cmml" xref="S3.SS4.p5.5.m5.1.2.3"><times id="S3.SS4.p5.5.m5.1.2.3.1.cmml" xref="S3.SS4.p5.5.m5.1.2.3.1"></times><ci id="S3.SS4.p5.5.m5.1.2.3.2.cmml" xref="S3.SS4.p5.5.m5.1.2.3.2">𝒟</ci><apply id="S3.SS4.p5.5.m5.1.1.cmml" xref="S3.SS4.p5.5.m5.1.2.3.3.2"><ci id="S3.SS4.p5.5.m5.1.1.1.cmml" xref="S3.SS4.p5.5.m5.1.1.1">¯</ci><ci id="S3.SS4.p5.5.m5.1.1.2.cmml" xref="S3.SS4.p5.5.m5.1.1.2">𝑧</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS4.p5.5.m5.1c">\bar{w}=\mathcal{D}(\bar{z})</annotation></semantics></math> which is then used to initialize the target network as depicted in Figure <a href="#S1.F2" title="Figure 2 ‣ 1 Introduction ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Experiments</h2>

<figure id="S4.F3" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S4.F3.1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2402.18153/assets/x4.png" id="S4.F3.1.g1" class="ltx_graphics ltx_img_landscape" width="461" height="33" alt="Refer to caption">
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F3.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2402.18153/assets/x5.png" id="S4.F3.sf1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="403" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F3.sf1.2.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="S4.F3.sf1.3.2" class="ltx_text" style="font-size:90%;">MNIST</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F3.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2402.18153/assets/x6.png" id="S4.F3.sf2.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="403" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F3.sf2.2.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="S4.F3.sf2.3.2" class="ltx_text" style="font-size:90%;">SVHN</span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F3.sf3" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2402.18153/assets/x7.png" id="S4.F3.sf3.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="403" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F3.sf3.2.1.1" class="ltx_text" style="font-size:90%;">(c)</span> </span><span id="S4.F3.sf3.3.2" class="ltx_text" style="font-size:90%;">CIFAR-10</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F3.sf4" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2402.18153/assets/x8.png" id="S4.F3.sf4.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="403" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F3.sf4.2.1.1" class="ltx_text" style="font-size:90%;">(d)</span> </span><span id="S4.F3.sf4.3.2" class="ltx_text" style="font-size:90%;">STL-10</span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F3.4.1.1" class="ltx_text" style="font-size:90%;">Figure 3</span>: </span><span id="S4.F3.5.2" class="ltx_text ltx_font_bold" style="font-size:90%;">Convergence Plots on Fintuning Generated Weigths:<span id="S4.F3.5.2.1" class="ltx_text ltx_font_medium"> Weights generated by the competing methods are finetuned for 25 epochs on the training set. We utilize the modelzoos of  <cite class="ltx_cite ltx_citemacro_citet">Schürholt et al. (<a href="#bib.bib26" title="" class="ltx_ref">2022b</a>)</cite>.</span></span></figcaption>
</figure>
<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">We generate and evaluate weights in the following setting:</p>
<ul id="S4.I1" class="ltx_itemize">
<li id="S4.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I1.i1.p1" class="ltx_para">
<p id="S4.I1.i1.p1.1" class="ltx_p"><span id="S4.I1.i1.p1.1.1" class="ltx_text ltx_font_italic">Weight Generation without Finetuning:</span> For this task, we condition on a dataset, generate weights and directly evaluate it on the test set.</p>
</div>
</li>
<li id="S4.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I1.i2.p1" class="ltx_para">
<p id="S4.I1.i2.p1.1" class="ltx_p"><span id="S4.I1.i2.p1.1.1" class="ltx_text ltx_font_italic">Weight Generation with Finetuning:</span> The generated weights are finetuned on the training set for a fixed number of epochs before evaluating on the test set.</p>
</div>
</li>
</ul>
</div>
<div id="S4.p2" class="ltx_para">
<p id="S4.p2.1" class="ltx_p"><span id="S4.p2.1.1" class="ltx_text ltx_font_bold">Baselines:</span> We benchmark against the following:</p>
<ul id="S4.I2" class="ltx_itemize">
<li id="S4.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I2.i1.p1" class="ltx_para">
<p id="S4.I2.i1.p1.1" class="ltx_p"><span id="S4.I2.i1.p1.1.1" class="ltx_text ltx_font_bold">RandInit:</span> This is a randomly initialized model trained on the training set of the target dataset from scratch for a specified number of epochs.</p>
</div>
</li>
<li id="S4.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I2.i2.p1" class="ltx_para">
<p id="S4.I2.i2.p1.1" class="ltx_p"><span id="S4.I2.i2.p1.1.1" class="ltx_text ltx_font_bold">Pretrained:</span> These are models pretrained either on the target dataset or ImageNet. We follow the setting of <cite class="ltx_cite ltx_citemacro_citet">Schürholt et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022a</a>)</cite> and train this baseline for 25 epochs on the target datasets in Tables <a href="#S4.T3" title="Table 3 ‣ 4.3 Performance Evaluation on KaggleZoo ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> &amp; <a href="#S4.T2" title="Table 2 ‣ 4.1 In-distribution Transfer Learning ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>. All other pretranied baselines are trained on ImageNet.</p>
</div>
</li>
<li id="S4.I2.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I2.i3.p1" class="ltx_para">
<p id="S4.I2.i3.p1.1" class="ltx_p"><span id="S4.I2.i3.p1.1.1" class="ltx_text ltx_font_bold">S<sub id="S4.I2.i3.p1.1.1.1" class="ltx_sub"><span id="S4.I2.i3.p1.1.1.1.1" class="ltx_text ltx_font_typewriter">KDE30</span></sub></span> <cite class="ltx_cite ltx_citemacro_citep">(Schürholt et al., <a href="#bib.bib25" title="" class="ltx_ref">2022a</a>)</cite>: This method learns an autoencoder on a modelzoo of networks trained on a single dataset and uses Kernel Density Estimation (KDE) to sample new weights from the embedding of networks in the modelzoo.</p>
</div>
</li>
</ul>
<p id="S4.p2.2" class="ltx_p">Detailed experimental setup is outlined in the Appendix.</p>
</div>
<div id="S4.p3" class="ltx_para">
<p id="S4.p3.1" class="ltx_p"><span id="S4.p3.1.1" class="ltx_text ltx_font_bold">Modelzoo Generation:</span> We utilize the pretrained weights datasets of <cite class="ltx_cite ltx_citemacro_citet">Schürholt et al. (<a href="#bib.bib26" title="" class="ltx_ref">2022b</a>)</cite> which consists of four modelzoos of networks pretranied on MNIST, SVHN, CIFAR-10 and STL-10. Next we generate a modelzoo for the set of datasets introduced in <cite class="ltx_cite ltx_citemacro_citet">Jeong et al. (<a href="#bib.bib13" title="" class="ltx_ref">2021</a>)</cite> and refer to it as the KaggleZoo. The test datasets are used for weight generation on unseen tasks. Finally, we generate a modelzoo by creating 2000 subsets of ImageNet with 100 and 50 images per class in the training and test splits respectively. Additionally, we do not train on the CIFAR-10 and STL-10 modelzoos of <cite class="ltx_cite ltx_citemacro_citet">Schürholt et al. (<a href="#bib.bib26" title="" class="ltx_ref">2022b</a>)</cite> and use them for unseen dataset weight generation. Details for generating the modelzoos is provided in Appendix <a href="#A2" title="Appendix B Details of Modelzoo Generation ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>.</p>
</div>
<figure id="alg1" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg1.2.1.1" class="ltx_text ltx_font_bold">Algorithm 1</span> </span> Datasets Encoder Training</figcaption>
<div id="alg1.3" class="ltx_listing ltx_listing">
<div id="alg1.l1" class="ltx_listingline">  <span id="alg1.l1.1" class="ltx_text ltx_font_bold">Input:</span> pretrained weights <math id="alg1.l1.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="alg1.l1.m1.1a"><mi id="alg1.l1.m1.1.1" xref="alg1.l1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="alg1.l1.m1.1b"><ci id="alg1.l1.m1.1.1.cmml" xref="alg1.l1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m1.1c">x</annotation></semantics></math>, image features <math id="alg1.l1.m2.1" class="ltx_Math" alttext="y" display="inline"><semantics id="alg1.l1.m2.1a"><mi id="alg1.l1.m2.1.1" xref="alg1.l1.m2.1.1.cmml">y</mi><annotation-xml encoding="MathML-Content" id="alg1.l1.m2.1b"><ci id="alg1.l1.m2.1.1.cmml" xref="alg1.l1.m2.1.1">𝑦</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m2.1c">y</annotation></semantics></math>, batch_num <math id="alg1.l1.m3.1" class="ltx_Math" alttext="m" display="inline"><semantics id="alg1.l1.m3.1a"><mi id="alg1.l1.m3.1.1" xref="alg1.l1.m3.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="alg1.l1.m3.1b"><ci id="alg1.l1.m3.1.1.cmml" xref="alg1.l1.m3.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m3.1c">m</annotation></semantics></math>

</div>
<div id="alg1.l2" class="ltx_listingline">  Instanciate <math id="alg1.l2.m1.1" class="ltx_Math" alttext="\mathcal{T}={\verb|Set Transformer|}" display="inline"><semantics id="alg1.l2.m1.1a"><mrow id="alg1.l2.m1.1.1" xref="alg1.l2.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l2.m1.1.1.2" xref="alg1.l2.m1.1.1.2.cmml">𝒯</mi><mo id="alg1.l2.m1.1.1.1" xref="alg1.l2.m1.1.1.1.cmml">=</mo><mi class="ltx_mathvariant_monospace" mathvariant="monospace" id="alg1.l2.m1.1.1.3" xref="alg1.l2.m1.1.1.3.cmml">Set Transformer</mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.l2.m1.1b"><apply id="alg1.l2.m1.1.1.cmml" xref="alg1.l2.m1.1.1"><eq id="alg1.l2.m1.1.1.1.cmml" xref="alg1.l2.m1.1.1.1"></eq><ci id="alg1.l2.m1.1.1.2.cmml" xref="alg1.l2.m1.1.1.2">𝒯</ci><ci id="alg1.l2.m1.1.1.3.cmml" xref="alg1.l2.m1.1.1.3">monospace-Set Transformer</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m1.1c">\mathcal{T}={\verb|Set Transformer|}</annotation></semantics></math>, Load pretrained Encoder (<math id="alg1.l2.m2.1" class="ltx_Math" alttext="\mathcal{E}" display="inline"><semantics id="alg1.l2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="alg1.l2.m2.1.1" xref="alg1.l2.m2.1.1.cmml">ℰ</mi><annotation-xml encoding="MathML-Content" id="alg1.l2.m2.1b"><ci id="alg1.l2.m2.1.1.cmml" xref="alg1.l2.m2.1.1">ℰ</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m2.1c">\mathcal{E}</annotation></semantics></math>).

</div>
<div id="alg1.l3" class="ltx_listingline">  <span id="alg1.l3.1" class="ltx_text ltx_font_bold">repeat</span>


</div>
<div id="alg1.l4" class="ltx_listingline">     Initialize <math id="alg1.l4.m1.1" class="ltx_Math" alttext="loss=0.0" display="inline"><semantics id="alg1.l4.m1.1a"><mrow id="alg1.l4.m1.1.1" xref="alg1.l4.m1.1.1.cmml"><mrow id="alg1.l4.m1.1.1.2" xref="alg1.l4.m1.1.1.2.cmml"><mi id="alg1.l4.m1.1.1.2.2" xref="alg1.l4.m1.1.1.2.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l4.m1.1.1.2.1" xref="alg1.l4.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l4.m1.1.1.2.3" xref="alg1.l4.m1.1.1.2.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg1.l4.m1.1.1.2.1a" xref="alg1.l4.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l4.m1.1.1.2.4" xref="alg1.l4.m1.1.1.2.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l4.m1.1.1.2.1b" xref="alg1.l4.m1.1.1.2.1.cmml">​</mo><mi id="alg1.l4.m1.1.1.2.5" xref="alg1.l4.m1.1.1.2.5.cmml">s</mi></mrow><mo id="alg1.l4.m1.1.1.1" xref="alg1.l4.m1.1.1.1.cmml">=</mo><mn id="alg1.l4.m1.1.1.3" xref="alg1.l4.m1.1.1.3.cmml">0.0</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.l4.m1.1b"><apply id="alg1.l4.m1.1.1.cmml" xref="alg1.l4.m1.1.1"><eq id="alg1.l4.m1.1.1.1.cmml" xref="alg1.l4.m1.1.1.1"></eq><apply id="alg1.l4.m1.1.1.2.cmml" xref="alg1.l4.m1.1.1.2"><times id="alg1.l4.m1.1.1.2.1.cmml" xref="alg1.l4.m1.1.1.2.1"></times><ci id="alg1.l4.m1.1.1.2.2.cmml" xref="alg1.l4.m1.1.1.2.2">𝑙</ci><ci id="alg1.l4.m1.1.1.2.3.cmml" xref="alg1.l4.m1.1.1.2.3">𝑜</ci><ci id="alg1.l4.m1.1.1.2.4.cmml" xref="alg1.l4.m1.1.1.2.4">𝑠</ci><ci id="alg1.l4.m1.1.1.2.5.cmml" xref="alg1.l4.m1.1.1.2.5">𝑠</ci></apply><cn type="float" id="alg1.l4.m1.1.1.3.cmml" xref="alg1.l4.m1.1.1.3">0.0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.m1.1c">loss=0.0</annotation></semantics></math>

</div>
<div id="alg1.l5" class="ltx_listingline">     <span id="alg1.l5.1" class="ltx_text ltx_font_bold">for</span> <math id="alg1.l5.m1.1" class="ltx_Math" alttext="i=1" display="inline"><semantics id="alg1.l5.m1.1a"><mrow id="alg1.l5.m1.1.1" xref="alg1.l5.m1.1.1.cmml"><mi id="alg1.l5.m1.1.1.2" xref="alg1.l5.m1.1.1.2.cmml">i</mi><mo id="alg1.l5.m1.1.1.1" xref="alg1.l5.m1.1.1.1.cmml">=</mo><mn id="alg1.l5.m1.1.1.3" xref="alg1.l5.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.l5.m1.1b"><apply id="alg1.l5.m1.1.1.cmml" xref="alg1.l5.m1.1.1"><eq id="alg1.l5.m1.1.1.1.cmml" xref="alg1.l5.m1.1.1.1"></eq><ci id="alg1.l5.m1.1.1.2.cmml" xref="alg1.l5.m1.1.1.2">𝑖</ci><cn type="integer" id="alg1.l5.m1.1.1.3.cmml" xref="alg1.l5.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m1.1c">i=1</annotation></semantics></math> <span id="alg1.l5.2" class="ltx_text ltx_font_bold">to</span> <math id="alg1.l5.m2.1" class="ltx_Math" alttext="m-1" display="inline"><semantics id="alg1.l5.m2.1a"><mrow id="alg1.l5.m2.1.1" xref="alg1.l5.m2.1.1.cmml"><mi id="alg1.l5.m2.1.1.2" xref="alg1.l5.m2.1.1.2.cmml">m</mi><mo id="alg1.l5.m2.1.1.1" xref="alg1.l5.m2.1.1.1.cmml">−</mo><mn id="alg1.l5.m2.1.1.3" xref="alg1.l5.m2.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg1.l5.m2.1b"><apply id="alg1.l5.m2.1.1.cmml" xref="alg1.l5.m2.1.1"><minus id="alg1.l5.m2.1.1.1.cmml" xref="alg1.l5.m2.1.1.1"></minus><ci id="alg1.l5.m2.1.1.2.cmml" xref="alg1.l5.m2.1.1.2">𝑚</ci><cn type="integer" id="alg1.l5.m2.1.1.3.cmml" xref="alg1.l5.m2.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m2.1c">m-1</annotation></semantics></math> <span id="alg1.l5.3" class="ltx_text ltx_font_bold">do</span>



</div>
<div id="alg1.l6" class="ltx_listingline">        <math id="alg1.l6.m1.1" class="ltx_Math" alttext="x_{i}\sim x" display="inline"><semantics id="alg1.l6.m1.1a"><mrow id="alg1.l6.m1.1.1" xref="alg1.l6.m1.1.1.cmml"><msub id="alg1.l6.m1.1.1.2" xref="alg1.l6.m1.1.1.2.cmml"><mi id="alg1.l6.m1.1.1.2.2" xref="alg1.l6.m1.1.1.2.2.cmml">x</mi><mi id="alg1.l6.m1.1.1.2.3" xref="alg1.l6.m1.1.1.2.3.cmml">i</mi></msub><mo id="alg1.l6.m1.1.1.1" xref="alg1.l6.m1.1.1.1.cmml">∼</mo><mi id="alg1.l6.m1.1.1.3" xref="alg1.l6.m1.1.1.3.cmml">x</mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.l6.m1.1b"><apply id="alg1.l6.m1.1.1.cmml" xref="alg1.l6.m1.1.1"><csymbol cd="latexml" id="alg1.l6.m1.1.1.1.cmml" xref="alg1.l6.m1.1.1.1">similar-to</csymbol><apply id="alg1.l6.m1.1.1.2.cmml" xref="alg1.l6.m1.1.1.2"><csymbol cd="ambiguous" id="alg1.l6.m1.1.1.2.1.cmml" xref="alg1.l6.m1.1.1.2">subscript</csymbol><ci id="alg1.l6.m1.1.1.2.2.cmml" xref="alg1.l6.m1.1.1.2.2">𝑥</ci><ci id="alg1.l6.m1.1.1.2.3.cmml" xref="alg1.l6.m1.1.1.2.3">𝑖</ci></apply><ci id="alg1.l6.m1.1.1.3.cmml" xref="alg1.l6.m1.1.1.3">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l6.m1.1c">x_{i}\sim x</annotation></semantics></math>, <math id="alg1.l6.m2.1" class="ltx_Math" alttext="\mathfrak{D_{i}}\sim\mathfrak{D}" display="inline"><semantics id="alg1.l6.m2.1a"><mrow id="alg1.l6.m2.1.1" xref="alg1.l6.m2.1.1.cmml"><msub id="alg1.l6.m2.1.1.2" xref="alg1.l6.m2.1.1.2.cmml"><mi id="alg1.l6.m2.1.1.2.2" xref="alg1.l6.m2.1.1.2.2.cmml">𝔇</mi><mi id="alg1.l6.m2.1.1.2.3" xref="alg1.l6.m2.1.1.2.3.cmml">𝔦</mi></msub><mo id="alg1.l6.m2.1.1.1" xref="alg1.l6.m2.1.1.1.cmml">∼</mo><mi id="alg1.l6.m2.1.1.3" xref="alg1.l6.m2.1.1.3.cmml">𝔇</mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.l6.m2.1b"><apply id="alg1.l6.m2.1.1.cmml" xref="alg1.l6.m2.1.1"><csymbol cd="latexml" id="alg1.l6.m2.1.1.1.cmml" xref="alg1.l6.m2.1.1.1">similar-to</csymbol><apply id="alg1.l6.m2.1.1.2.cmml" xref="alg1.l6.m2.1.1.2"><csymbol cd="ambiguous" id="alg1.l6.m2.1.1.2.1.cmml" xref="alg1.l6.m2.1.1.2">subscript</csymbol><ci id="alg1.l6.m2.1.1.2.2.cmml" xref="alg1.l6.m2.1.1.2.2">𝔇</ci><ci id="alg1.l6.m2.1.1.2.3.cmml" xref="alg1.l6.m2.1.1.2.3">𝔦</ci></apply><ci id="alg1.l6.m2.1.1.3.cmml" xref="alg1.l6.m2.1.1.3">𝔇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l6.m2.1c">\mathfrak{D_{i}}\sim\mathfrak{D}</annotation></semantics></math>

</div>
<div id="alg1.l7" class="ltx_listingline">        <math id="alg1.l7.m1.1" class="ltx_Math" alttext="z_{i}=\text{Encoder}_{\text{VAE}}(x_{i})" display="inline"><semantics id="alg1.l7.m1.1a"><mrow id="alg1.l7.m1.1.1" xref="alg1.l7.m1.1.1.cmml"><msub id="alg1.l7.m1.1.1.3" xref="alg1.l7.m1.1.1.3.cmml"><mi id="alg1.l7.m1.1.1.3.2" xref="alg1.l7.m1.1.1.3.2.cmml">z</mi><mi id="alg1.l7.m1.1.1.3.3" xref="alg1.l7.m1.1.1.3.3.cmml">i</mi></msub><mo id="alg1.l7.m1.1.1.2" xref="alg1.l7.m1.1.1.2.cmml">=</mo><mrow id="alg1.l7.m1.1.1.1" xref="alg1.l7.m1.1.1.1.cmml"><msub id="alg1.l7.m1.1.1.1.3" xref="alg1.l7.m1.1.1.1.3.cmml"><mtext id="alg1.l7.m1.1.1.1.3.2" xref="alg1.l7.m1.1.1.1.3.2a.cmml">Encoder</mtext><mtext id="alg1.l7.m1.1.1.1.3.3" xref="alg1.l7.m1.1.1.1.3.3a.cmml">VAE</mtext></msub><mo lspace="0em" rspace="0em" id="alg1.l7.m1.1.1.1.2" xref="alg1.l7.m1.1.1.1.2.cmml">​</mo><mrow id="alg1.l7.m1.1.1.1.1.1" xref="alg1.l7.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l7.m1.1.1.1.1.1.2" xref="alg1.l7.m1.1.1.1.1.1.1.cmml">(</mo><msub id="alg1.l7.m1.1.1.1.1.1.1" xref="alg1.l7.m1.1.1.1.1.1.1.cmml"><mi id="alg1.l7.m1.1.1.1.1.1.1.2" xref="alg1.l7.m1.1.1.1.1.1.1.2.cmml">x</mi><mi id="alg1.l7.m1.1.1.1.1.1.1.3" xref="alg1.l7.m1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="alg1.l7.m1.1.1.1.1.1.3" xref="alg1.l7.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l7.m1.1b"><apply id="alg1.l7.m1.1.1.cmml" xref="alg1.l7.m1.1.1"><eq id="alg1.l7.m1.1.1.2.cmml" xref="alg1.l7.m1.1.1.2"></eq><apply id="alg1.l7.m1.1.1.3.cmml" xref="alg1.l7.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.l7.m1.1.1.3.1.cmml" xref="alg1.l7.m1.1.1.3">subscript</csymbol><ci id="alg1.l7.m1.1.1.3.2.cmml" xref="alg1.l7.m1.1.1.3.2">𝑧</ci><ci id="alg1.l7.m1.1.1.3.3.cmml" xref="alg1.l7.m1.1.1.3.3">𝑖</ci></apply><apply id="alg1.l7.m1.1.1.1.cmml" xref="alg1.l7.m1.1.1.1"><times id="alg1.l7.m1.1.1.1.2.cmml" xref="alg1.l7.m1.1.1.1.2"></times><apply id="alg1.l7.m1.1.1.1.3.cmml" xref="alg1.l7.m1.1.1.1.3"><csymbol cd="ambiguous" id="alg1.l7.m1.1.1.1.3.1.cmml" xref="alg1.l7.m1.1.1.1.3">subscript</csymbol><ci id="alg1.l7.m1.1.1.1.3.2a.cmml" xref="alg1.l7.m1.1.1.1.3.2"><mtext id="alg1.l7.m1.1.1.1.3.2.cmml" xref="alg1.l7.m1.1.1.1.3.2">Encoder</mtext></ci><ci id="alg1.l7.m1.1.1.1.3.3a.cmml" xref="alg1.l7.m1.1.1.1.3.3"><mtext mathsize="70%" id="alg1.l7.m1.1.1.1.3.3.cmml" xref="alg1.l7.m1.1.1.1.3.3">VAE</mtext></ci></apply><apply id="alg1.l7.m1.1.1.1.1.1.1.cmml" xref="alg1.l7.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l7.m1.1.1.1.1.1.1.1.cmml" xref="alg1.l7.m1.1.1.1.1.1">subscript</csymbol><ci id="alg1.l7.m1.1.1.1.1.1.1.2.cmml" xref="alg1.l7.m1.1.1.1.1.1.1.2">𝑥</ci><ci id="alg1.l7.m1.1.1.1.1.1.1.3.cmml" xref="alg1.l7.m1.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l7.m1.1c">z_{i}=\text{Encoder}_{\text{VAE}}(x_{i})</annotation></semantics></math> 
</div>
<div id="alg1.l8" class="ltx_listingline">        <math id="alg1.l8.m1.1" class="ltx_Math" alttext="z_{\mathfrak{D}_{i}}=\mathcal{T}(\mathfrak{D_{i}})" display="inline"><semantics id="alg1.l8.m1.1a"><mrow id="alg1.l8.m1.1.1" xref="alg1.l8.m1.1.1.cmml"><msub id="alg1.l8.m1.1.1.3" xref="alg1.l8.m1.1.1.3.cmml"><mi id="alg1.l8.m1.1.1.3.2" xref="alg1.l8.m1.1.1.3.2.cmml">z</mi><msub id="alg1.l8.m1.1.1.3.3" xref="alg1.l8.m1.1.1.3.3.cmml"><mi id="alg1.l8.m1.1.1.3.3.2" xref="alg1.l8.m1.1.1.3.3.2.cmml">𝔇</mi><mi id="alg1.l8.m1.1.1.3.3.3" xref="alg1.l8.m1.1.1.3.3.3.cmml">i</mi></msub></msub><mo id="alg1.l8.m1.1.1.2" xref="alg1.l8.m1.1.1.2.cmml">=</mo><mrow id="alg1.l8.m1.1.1.1" xref="alg1.l8.m1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l8.m1.1.1.1.3" xref="alg1.l8.m1.1.1.1.3.cmml">𝒯</mi><mo lspace="0em" rspace="0em" id="alg1.l8.m1.1.1.1.2" xref="alg1.l8.m1.1.1.1.2.cmml">​</mo><mrow id="alg1.l8.m1.1.1.1.1.1" xref="alg1.l8.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l8.m1.1.1.1.1.1.2" xref="alg1.l8.m1.1.1.1.1.1.1.cmml">(</mo><msub id="alg1.l8.m1.1.1.1.1.1.1" xref="alg1.l8.m1.1.1.1.1.1.1.cmml"><mi id="alg1.l8.m1.1.1.1.1.1.1.2" xref="alg1.l8.m1.1.1.1.1.1.1.2.cmml">𝔇</mi><mi id="alg1.l8.m1.1.1.1.1.1.1.3" xref="alg1.l8.m1.1.1.1.1.1.1.3.cmml">𝔦</mi></msub><mo stretchy="false" id="alg1.l8.m1.1.1.1.1.1.3" xref="alg1.l8.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l8.m1.1b"><apply id="alg1.l8.m1.1.1.cmml" xref="alg1.l8.m1.1.1"><eq id="alg1.l8.m1.1.1.2.cmml" xref="alg1.l8.m1.1.1.2"></eq><apply id="alg1.l8.m1.1.1.3.cmml" xref="alg1.l8.m1.1.1.3"><csymbol cd="ambiguous" id="alg1.l8.m1.1.1.3.1.cmml" xref="alg1.l8.m1.1.1.3">subscript</csymbol><ci id="alg1.l8.m1.1.1.3.2.cmml" xref="alg1.l8.m1.1.1.3.2">𝑧</ci><apply id="alg1.l8.m1.1.1.3.3.cmml" xref="alg1.l8.m1.1.1.3.3"><csymbol cd="ambiguous" id="alg1.l8.m1.1.1.3.3.1.cmml" xref="alg1.l8.m1.1.1.3.3">subscript</csymbol><ci id="alg1.l8.m1.1.1.3.3.2.cmml" xref="alg1.l8.m1.1.1.3.3.2">𝔇</ci><ci id="alg1.l8.m1.1.1.3.3.3.cmml" xref="alg1.l8.m1.1.1.3.3.3">𝑖</ci></apply></apply><apply id="alg1.l8.m1.1.1.1.cmml" xref="alg1.l8.m1.1.1.1"><times id="alg1.l8.m1.1.1.1.2.cmml" xref="alg1.l8.m1.1.1.1.2"></times><ci id="alg1.l8.m1.1.1.1.3.cmml" xref="alg1.l8.m1.1.1.1.3">𝒯</ci><apply id="alg1.l8.m1.1.1.1.1.1.1.cmml" xref="alg1.l8.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l8.m1.1.1.1.1.1.1.1.cmml" xref="alg1.l8.m1.1.1.1.1.1">subscript</csymbol><ci id="alg1.l8.m1.1.1.1.1.1.1.2.cmml" xref="alg1.l8.m1.1.1.1.1.1.1.2">𝔇</ci><ci id="alg1.l8.m1.1.1.1.1.1.1.3.cmml" xref="alg1.l8.m1.1.1.1.1.1.1.3">𝔦</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l8.m1.1c">z_{\mathfrak{D}_{i}}=\mathcal{T}(\mathfrak{D_{i}})</annotation></semantics></math>

</div>
<div id="alg1.l9" class="ltx_listingline">        <math id="alg1.l9.m1.2" class="ltx_Math" alttext="loss=loss+\mathcal{L}_{CLIP}(z_{i},z_{\mathfrak{D_{i}}})" display="inline"><semantics id="alg1.l9.m1.2a"><mrow id="alg1.l9.m1.2.2" xref="alg1.l9.m1.2.2.cmml"><mrow id="alg1.l9.m1.2.2.4" xref="alg1.l9.m1.2.2.4.cmml"><mi id="alg1.l9.m1.2.2.4.2" xref="alg1.l9.m1.2.2.4.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m1.2.2.4.1" xref="alg1.l9.m1.2.2.4.1.cmml">​</mo><mi id="alg1.l9.m1.2.2.4.3" xref="alg1.l9.m1.2.2.4.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m1.2.2.4.1a" xref="alg1.l9.m1.2.2.4.1.cmml">​</mo><mi id="alg1.l9.m1.2.2.4.4" xref="alg1.l9.m1.2.2.4.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m1.2.2.4.1b" xref="alg1.l9.m1.2.2.4.1.cmml">​</mo><mi id="alg1.l9.m1.2.2.4.5" xref="alg1.l9.m1.2.2.4.5.cmml">s</mi></mrow><mo id="alg1.l9.m1.2.2.3" xref="alg1.l9.m1.2.2.3.cmml">=</mo><mrow id="alg1.l9.m1.2.2.2" xref="alg1.l9.m1.2.2.2.cmml"><mrow id="alg1.l9.m1.2.2.2.4" xref="alg1.l9.m1.2.2.2.4.cmml"><mi id="alg1.l9.m1.2.2.2.4.2" xref="alg1.l9.m1.2.2.2.4.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m1.2.2.2.4.1" xref="alg1.l9.m1.2.2.2.4.1.cmml">​</mo><mi id="alg1.l9.m1.2.2.2.4.3" xref="alg1.l9.m1.2.2.2.4.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m1.2.2.2.4.1a" xref="alg1.l9.m1.2.2.2.4.1.cmml">​</mo><mi id="alg1.l9.m1.2.2.2.4.4" xref="alg1.l9.m1.2.2.2.4.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m1.2.2.2.4.1b" xref="alg1.l9.m1.2.2.2.4.1.cmml">​</mo><mi id="alg1.l9.m1.2.2.2.4.5" xref="alg1.l9.m1.2.2.2.4.5.cmml">s</mi></mrow><mo id="alg1.l9.m1.2.2.2.3" xref="alg1.l9.m1.2.2.2.3.cmml">+</mo><mrow id="alg1.l9.m1.2.2.2.2" xref="alg1.l9.m1.2.2.2.2.cmml"><msub id="alg1.l9.m1.2.2.2.2.4" xref="alg1.l9.m1.2.2.2.2.4.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l9.m1.2.2.2.2.4.2" xref="alg1.l9.m1.2.2.2.2.4.2.cmml">ℒ</mi><mrow id="alg1.l9.m1.2.2.2.2.4.3" xref="alg1.l9.m1.2.2.2.2.4.3.cmml"><mi id="alg1.l9.m1.2.2.2.2.4.3.2" xref="alg1.l9.m1.2.2.2.2.4.3.2.cmml">C</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m1.2.2.2.2.4.3.1" xref="alg1.l9.m1.2.2.2.2.4.3.1.cmml">​</mo><mi id="alg1.l9.m1.2.2.2.2.4.3.3" xref="alg1.l9.m1.2.2.2.2.4.3.3.cmml">L</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m1.2.2.2.2.4.3.1a" xref="alg1.l9.m1.2.2.2.2.4.3.1.cmml">​</mo><mi id="alg1.l9.m1.2.2.2.2.4.3.4" xref="alg1.l9.m1.2.2.2.2.4.3.4.cmml">I</mi><mo lspace="0em" rspace="0em" id="alg1.l9.m1.2.2.2.2.4.3.1b" xref="alg1.l9.m1.2.2.2.2.4.3.1.cmml">​</mo><mi id="alg1.l9.m1.2.2.2.2.4.3.5" xref="alg1.l9.m1.2.2.2.2.4.3.5.cmml">P</mi></mrow></msub><mo lspace="0em" rspace="0em" id="alg1.l9.m1.2.2.2.2.3" xref="alg1.l9.m1.2.2.2.2.3.cmml">​</mo><mrow id="alg1.l9.m1.2.2.2.2.2.2" xref="alg1.l9.m1.2.2.2.2.2.3.cmml"><mo stretchy="false" id="alg1.l9.m1.2.2.2.2.2.2.3" xref="alg1.l9.m1.2.2.2.2.2.3.cmml">(</mo><msub id="alg1.l9.m1.1.1.1.1.1.1.1" xref="alg1.l9.m1.1.1.1.1.1.1.1.cmml"><mi id="alg1.l9.m1.1.1.1.1.1.1.1.2" xref="alg1.l9.m1.1.1.1.1.1.1.1.2.cmml">z</mi><mi id="alg1.l9.m1.1.1.1.1.1.1.1.3" xref="alg1.l9.m1.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="alg1.l9.m1.2.2.2.2.2.2.4" xref="alg1.l9.m1.2.2.2.2.2.3.cmml">,</mo><msub id="alg1.l9.m1.2.2.2.2.2.2.2" xref="alg1.l9.m1.2.2.2.2.2.2.2.cmml"><mi id="alg1.l9.m1.2.2.2.2.2.2.2.2" xref="alg1.l9.m1.2.2.2.2.2.2.2.2.cmml">z</mi><msub id="alg1.l9.m1.2.2.2.2.2.2.2.3" xref="alg1.l9.m1.2.2.2.2.2.2.2.3.cmml"><mi id="alg1.l9.m1.2.2.2.2.2.2.2.3.2" xref="alg1.l9.m1.2.2.2.2.2.2.2.3.2.cmml">𝔇</mi><mi id="alg1.l9.m1.2.2.2.2.2.2.2.3.3" xref="alg1.l9.m1.2.2.2.2.2.2.2.3.3.cmml">𝔦</mi></msub></msub><mo stretchy="false" id="alg1.l9.m1.2.2.2.2.2.2.5" xref="alg1.l9.m1.2.2.2.2.2.3.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l9.m1.2b"><apply id="alg1.l9.m1.2.2.cmml" xref="alg1.l9.m1.2.2"><eq id="alg1.l9.m1.2.2.3.cmml" xref="alg1.l9.m1.2.2.3"></eq><apply id="alg1.l9.m1.2.2.4.cmml" xref="alg1.l9.m1.2.2.4"><times id="alg1.l9.m1.2.2.4.1.cmml" xref="alg1.l9.m1.2.2.4.1"></times><ci id="alg1.l9.m1.2.2.4.2.cmml" xref="alg1.l9.m1.2.2.4.2">𝑙</ci><ci id="alg1.l9.m1.2.2.4.3.cmml" xref="alg1.l9.m1.2.2.4.3">𝑜</ci><ci id="alg1.l9.m1.2.2.4.4.cmml" xref="alg1.l9.m1.2.2.4.4">𝑠</ci><ci id="alg1.l9.m1.2.2.4.5.cmml" xref="alg1.l9.m1.2.2.4.5">𝑠</ci></apply><apply id="alg1.l9.m1.2.2.2.cmml" xref="alg1.l9.m1.2.2.2"><plus id="alg1.l9.m1.2.2.2.3.cmml" xref="alg1.l9.m1.2.2.2.3"></plus><apply id="alg1.l9.m1.2.2.2.4.cmml" xref="alg1.l9.m1.2.2.2.4"><times id="alg1.l9.m1.2.2.2.4.1.cmml" xref="alg1.l9.m1.2.2.2.4.1"></times><ci id="alg1.l9.m1.2.2.2.4.2.cmml" xref="alg1.l9.m1.2.2.2.4.2">𝑙</ci><ci id="alg1.l9.m1.2.2.2.4.3.cmml" xref="alg1.l9.m1.2.2.2.4.3">𝑜</ci><ci id="alg1.l9.m1.2.2.2.4.4.cmml" xref="alg1.l9.m1.2.2.2.4.4">𝑠</ci><ci id="alg1.l9.m1.2.2.2.4.5.cmml" xref="alg1.l9.m1.2.2.2.4.5">𝑠</ci></apply><apply id="alg1.l9.m1.2.2.2.2.cmml" xref="alg1.l9.m1.2.2.2.2"><times id="alg1.l9.m1.2.2.2.2.3.cmml" xref="alg1.l9.m1.2.2.2.2.3"></times><apply id="alg1.l9.m1.2.2.2.2.4.cmml" xref="alg1.l9.m1.2.2.2.2.4"><csymbol cd="ambiguous" id="alg1.l9.m1.2.2.2.2.4.1.cmml" xref="alg1.l9.m1.2.2.2.2.4">subscript</csymbol><ci id="alg1.l9.m1.2.2.2.2.4.2.cmml" xref="alg1.l9.m1.2.2.2.2.4.2">ℒ</ci><apply id="alg1.l9.m1.2.2.2.2.4.3.cmml" xref="alg1.l9.m1.2.2.2.2.4.3"><times id="alg1.l9.m1.2.2.2.2.4.3.1.cmml" xref="alg1.l9.m1.2.2.2.2.4.3.1"></times><ci id="alg1.l9.m1.2.2.2.2.4.3.2.cmml" xref="alg1.l9.m1.2.2.2.2.4.3.2">𝐶</ci><ci id="alg1.l9.m1.2.2.2.2.4.3.3.cmml" xref="alg1.l9.m1.2.2.2.2.4.3.3">𝐿</ci><ci id="alg1.l9.m1.2.2.2.2.4.3.4.cmml" xref="alg1.l9.m1.2.2.2.2.4.3.4">𝐼</ci><ci id="alg1.l9.m1.2.2.2.2.4.3.5.cmml" xref="alg1.l9.m1.2.2.2.2.4.3.5">𝑃</ci></apply></apply><interval closure="open" id="alg1.l9.m1.2.2.2.2.2.3.cmml" xref="alg1.l9.m1.2.2.2.2.2.2"><apply id="alg1.l9.m1.1.1.1.1.1.1.1.cmml" xref="alg1.l9.m1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l9.m1.1.1.1.1.1.1.1.1.cmml" xref="alg1.l9.m1.1.1.1.1.1.1.1">subscript</csymbol><ci id="alg1.l9.m1.1.1.1.1.1.1.1.2.cmml" xref="alg1.l9.m1.1.1.1.1.1.1.1.2">𝑧</ci><ci id="alg1.l9.m1.1.1.1.1.1.1.1.3.cmml" xref="alg1.l9.m1.1.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="alg1.l9.m1.2.2.2.2.2.2.2.cmml" xref="alg1.l9.m1.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="alg1.l9.m1.2.2.2.2.2.2.2.1.cmml" xref="alg1.l9.m1.2.2.2.2.2.2.2">subscript</csymbol><ci id="alg1.l9.m1.2.2.2.2.2.2.2.2.cmml" xref="alg1.l9.m1.2.2.2.2.2.2.2.2">𝑧</ci><apply id="alg1.l9.m1.2.2.2.2.2.2.2.3.cmml" xref="alg1.l9.m1.2.2.2.2.2.2.2.3"><csymbol cd="ambiguous" id="alg1.l9.m1.2.2.2.2.2.2.2.3.1.cmml" xref="alg1.l9.m1.2.2.2.2.2.2.2.3">subscript</csymbol><ci id="alg1.l9.m1.2.2.2.2.2.2.2.3.2.cmml" xref="alg1.l9.m1.2.2.2.2.2.2.2.3.2">𝔇</ci><ci id="alg1.l9.m1.2.2.2.2.2.2.2.3.3.cmml" xref="alg1.l9.m1.2.2.2.2.2.2.2.3.3">𝔦</ci></apply></apply></interval></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l9.m1.2c">loss=loss+\mathcal{L}_{CLIP}(z_{i},z_{\mathfrak{D_{i}}})</annotation></semantics></math> (Equation <a href="#S3.E2" title="In 3.3 Dataset Encoding ‣ 3 Approach ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>)

</div>
<div id="alg1.l10" class="ltx_listingline">     <span id="alg1.l10.1" class="ltx_text ltx_font_bold">end</span> <span id="alg1.l10.2" class="ltx_text ltx_font_bold">for</span>
</div>
<div id="alg1.l11" class="ltx_listingline">     Update weights of <math id="alg1.l11.m1.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="alg1.l11.m1.1a"><mi class="ltx_font_mathcaligraphic" id="alg1.l11.m1.1.1" xref="alg1.l11.m1.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="alg1.l11.m1.1b"><ci id="alg1.l11.m1.1.1.cmml" xref="alg1.l11.m1.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l11.m1.1c">\mathcal{T}</annotation></semantics></math>

</div>
<div id="alg1.l12" class="ltx_listingline">  <span id="alg1.l12.1" class="ltx_text ltx_font_bold">until</span> convergence

</div>
</div>
</figure>
<div id="S4.p4" class="ltx_para">
<p id="S4.p4.1" class="ltx_p"><span id="S4.p4.1.1" class="ltx_text ltx_font_bold">Set-based Dataset Encoding</span>: We use Set Transformer <cite class="ltx_cite ltx_citemacro_citep">(Lee et al., <a href="#bib.bib17" title="" class="ltx_ref">2019</a>)</cite> for dataset encoding using features extracted from CLIP <cite class="ltx_cite ltx_citemacro_citep">(Radford et al., <a href="#bib.bib22" title="" class="ltx_ref">2021</a>)</cite> similar to the approach in <cite class="ltx_cite ltx_citemacro_citet">Lee et al. (<a href="#bib.bib16" title="" class="ltx_ref">2021</a>)</cite> where features were extracted using a pretrained ResNet18. This process is done before training with the dataset encoder defined in Section <a href="#S3.SS3" title="3.3 Dataset Encoding ‣ 3 Approach ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3.3</span></a>. We observed that jointly optimizing the dataset encoder with the diffusion model is more challenging given a large number of datasets hence we pretrain the dataset encoder and freeze it during the training of the diffusion model.</p>
</div>
<div id="S4.p5" class="ltx_para">
<p id="S4.p5.1" class="ltx_p"><span id="S4.p5.1.1" class="ltx_text ltx_font_bold">Compute:</span> For the zoo generation we use a single Titan RTX GPU with 24GB memory. The proposed model was also trained on a single GPU with 24GB memory.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>In-distribution Transfer Learning</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p"><span id="S4.SS1.p1.1.1" class="ltx_text ltx_font_bold">Task:</span> The main objective of this experiment is to evaluate convergence speed using weights from each of the methods as initialization. In this experiment the target dataset is the same as those used to generate the modelzoos.</p>
</div>
<div id="S4.SS1.p2" class="ltx_para">
<p id="S4.SS1.p2.1" class="ltx_p"><span id="S4.SS1.p2.1.1" class="ltx_text ltx_font_bold">Setup</span>: The pretrained weights used here are from epochs 21 to 25 for each dataset where 70% of the resulting modelzoo is used for training and 15% for validation and testing respectively. The number of pretrained weights in the modelzoos are 3500 for MNIST, CIFAR-10, and STL-10, and 2864 for SVHN. The flattened network weights’ length is 2864 for CIFAR-10 and STL-10 and, 2464 for MNIST and SVHN. We pad all the weights with zero to 2864. After training the diffusion model, we conditionally sample five set of weights based on images samples from the corresponding datasets and initialize each corresponding architectures and then finetune them for 25 epochs. We compare our method to the best approach in <cite class="ltx_cite ltx_citemacro_citet">Schürholt et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022a</a>)</cite> and the pretrained weights already trained for 25 epochs (we evolve this model for 25 more epochs resulting in 50 epochs of optimization on the training set).</p>
</div>
<figure id="S4.T1" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S4.T1.4.1.1" class="ltx_text" style="font-size:90%;">Table 1</span>: </span><span id="S4.T1.5.2" class="ltx_text" style="font-size:90%;">
<span id="S4.T1.5.2.1" class="ltx_text ltx_font_bold">Convergence Speed Comparison</span> Sampled weights from S<sub id="S4.T1.5.2.2" class="ltx_sub"><span id="S4.T1.5.2.2.1" class="ltx_text ltx_font_typewriter">KDE30</span></sub> <cite class="ltx_cite ltx_citemacro_citep">(Schürholt et al., <a href="#bib.bib25" title="" class="ltx_ref">2022a</a>)</cite> and our method are evaluated on the test set after 0 (meaning no optimization steps), 1 and 25 epochs of training on the training set. The Pretrained baseline is trained on the training set for a total of 50 epochs.</span></figcaption>
<div id="S4.T1.6" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:198.5pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(59.6pt,-27.3pt) scale(1.37869704325808,1.37869704325808) ;">
<table id="S4.T1.6.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T1.6.1.1.1" class="ltx_tr">
<th id="S4.T1.6.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">Model</th>
<th id="S4.T1.6.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_tt">Epoch</th>
<td id="S4.T1.6.1.1.1.3" class="ltx_td ltx_align_center ltx_border_tt">MNIST</td>
<td id="S4.T1.6.1.1.1.4" class="ltx_td ltx_align_center ltx_border_tt">SVHN</td>
<td id="S4.T1.6.1.1.1.5" class="ltx_td ltx_align_center ltx_border_tt">CIFAR-10</td>
<td id="S4.T1.6.1.1.1.6" class="ltx_td ltx_align_center ltx_border_tt">STL-10</td>
</tr>
<tr id="S4.T1.6.1.2.2" class="ltx_tr">
<th id="S4.T1.6.1.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">Pretrained</th>
<th id="S4.T1.6.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_tt">50</th>
<td id="S4.T1.6.1.2.2.3" class="ltx_td ltx_align_center ltx_border_tt">95.70</td>
<td id="S4.T1.6.1.2.2.4" class="ltx_td ltx_align_center ltx_border_tt">74.49</td>
<td id="S4.T1.6.1.2.2.5" class="ltx_td ltx_align_center ltx_border_tt">49.25</td>
<td id="S4.T1.6.1.2.2.6" class="ltx_td ltx_align_center ltx_border_tt">39.39</td>
</tr>
<tr id="S4.T1.6.1.3.3" class="ltx_tr">
<th id="S4.T1.6.1.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">S<sub id="S4.T1.6.1.3.3.1.1" class="ltx_sub"><span id="S4.T1.6.1.3.3.1.1.1" class="ltx_text ltx_font_typewriter">KDE30</span></sub>
</th>
<th id="S4.T1.6.1.3.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">0</th>
<td id="S4.T1.6.1.3.3.3" class="ltx_td ltx_align_center">69.73</td>
<td id="S4.T1.6.1.3.3.4" class="ltx_td ltx_align_center">50.25</td>
<td id="S4.T1.6.1.3.3.5" class="ltx_td ltx_align_center">26.06</td>
<td id="S4.T1.6.1.3.3.6" class="ltx_td ltx_align_center">17.20</td>
</tr>
<tr id="S4.T1.6.1.4.4" class="ltx_tr">
<th id="S4.T1.6.1.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">D2NWG (Ours)</th>
<th id="S4.T1.6.1.4.4.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">0</th>
<td id="S4.T1.6.1.4.4.3" class="ltx_td ltx_align_center"><span id="S4.T1.6.1.4.4.3.1" class="ltx_text ltx_font_bold">78.10</span></td>
<td id="S4.T1.6.1.4.4.4" class="ltx_td ltx_align_center"><span id="S4.T1.6.1.4.4.4.1" class="ltx_text ltx_font_bold">58.92</span></td>
<td id="S4.T1.6.1.4.4.5" class="ltx_td ltx_align_center"><span id="S4.T1.6.1.4.4.5.1" class="ltx_text ltx_font_bold">38.23</span></td>
<td id="S4.T1.6.1.4.4.6" class="ltx_td ltx_align_center"><span id="S4.T1.6.1.4.4.6.1" class="ltx_text ltx_font_bold">22.74</span></td>
</tr>
<tr id="S4.T1.6.1.5.5" class="ltx_tr">
<th id="S4.T1.6.1.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">S<sub id="S4.T1.6.1.5.5.1.1" class="ltx_sub"><span id="S4.T1.6.1.5.5.1.1.1" class="ltx_text ltx_font_typewriter">KDE30</span></sub>
</th>
<th id="S4.T1.6.1.5.5.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t">1</th>
<td id="S4.T1.6.1.5.5.3" class="ltx_td ltx_align_center ltx_border_t">82.69</td>
<td id="S4.T1.6.1.5.5.4" class="ltx_td ltx_align_center ltx_border_t">70.45</td>
<td id="S4.T1.6.1.5.5.5" class="ltx_td ltx_align_center ltx_border_t">44.75</td>
<td id="S4.T1.6.1.5.5.6" class="ltx_td ltx_align_center ltx_border_t">28.42</td>
</tr>
<tr id="S4.T1.6.1.6.6" class="ltx_tr">
<th id="S4.T1.6.1.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">D2NWG (Ours)</th>
<th id="S4.T1.6.1.6.6.2" class="ltx_td ltx_align_center ltx_th ltx_th_row">1</th>
<td id="S4.T1.6.1.6.6.3" class="ltx_td ltx_align_center"><span id="S4.T1.6.1.6.6.3.1" class="ltx_text ltx_font_bold">83.25</span></td>
<td id="S4.T1.6.1.6.6.4" class="ltx_td ltx_align_center"><span id="S4.T1.6.1.6.6.4.1" class="ltx_text ltx_font_bold">73.11</span></td>
<td id="S4.T1.6.1.6.6.5" class="ltx_td ltx_align_center"><span id="S4.T1.6.1.6.6.5.1" class="ltx_text ltx_font_bold">46.88</span></td>
<td id="S4.T1.6.1.6.6.6" class="ltx_td ltx_align_center"><span id="S4.T1.6.1.6.6.6.1" class="ltx_text ltx_font_bold">34.39</span></td>
</tr>
<tr id="S4.T1.6.1.7.7" class="ltx_tr">
<th id="S4.T1.6.1.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">S<sub id="S4.T1.6.1.7.7.1.1" class="ltx_sub"><span id="S4.T1.6.1.7.7.1.1.1" class="ltx_text ltx_font_typewriter">KDE30</span></sub>
</th>
<th id="S4.T1.6.1.7.7.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t">25</th>
<td id="S4.T1.6.1.7.7.3" class="ltx_td ltx_align_center ltx_border_t">95.70</td>
<td id="S4.T1.6.1.7.7.4" class="ltx_td ltx_align_center ltx_border_t">74.69</td>
<td id="S4.T1.6.1.7.7.5" class="ltx_td ltx_align_center ltx_border_t">49.14</td>
<td id="S4.T1.6.1.7.7.6" class="ltx_td ltx_align_center ltx_border_t">39.14</td>
</tr>
<tr id="S4.T1.6.1.8.8" class="ltx_tr">
<th id="S4.T1.6.1.8.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">D2NWG (Ours)</th>
<th id="S4.T1.6.1.8.8.2" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb">25</th>
<td id="S4.T1.6.1.8.8.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T1.6.1.8.8.3.1" class="ltx_text ltx_font_bold">96.17</span></td>
<td id="S4.T1.6.1.8.8.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T1.6.1.8.8.4.1" class="ltx_text ltx_font_bold">74.75</span></td>
<td id="S4.T1.6.1.8.8.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T1.6.1.8.8.5.1" class="ltx_text ltx_font_bold">49.48</span></td>
<td id="S4.T1.6.1.8.8.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T1.6.1.8.8.6.1" class="ltx_text ltx_font_bold">39.43</span></td>
</tr>
</tbody>
</table>
</span></div>
</figure>
<div id="S4.SS1.p3" class="ltx_para">
<p id="S4.SS1.p3.1" class="ltx_p"><span id="S4.SS1.p3.1.1" class="ltx_text ltx_font_bold">Results:</span> As reported in Table <a href="#S4.T1" title="Table 1 ‣ 4.1 In-distribution Transfer Learning ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, D2NWG consistently speeds up the convergence rate on the related tasks and after 25 epochs, achieves better performance than the pretrained model, consistently outperforming the Kernel Density Estimation approach of  <cite class="ltx_cite ltx_citemacro_citet">Schürholt et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022a</a>)</cite>. In Figure <a href="#S4.F3" title="Figure 3 ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>, we show the convergence plots for the 4 datasets considered in this task.</p>
</div>
<figure id="S4.T2" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S4.T2.3.1.1" class="ltx_text" style="font-size:90%;">Table 2</span>: </span><span id="S4.T2.4.2" class="ltx_text ltx_font_bold" style="font-size:90%;">No Fine-tuning Initialization on Unseen Datasets<span id="S4.T2.4.2.1" class="ltx_text ltx_font_medium"> We transfer from one dataset, or combinations of datasets, to unseen datasets at test time.</span></span></figcaption>
<div id="S4.T2.5" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:297pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(85.4pt,-58.5pt) scale(1.64996269577461,1.64996269577461) ;">
<table id="S4.T2.5.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T2.5.1.1.1" class="ltx_tr">
<th id="S4.T2.5.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Source</th>
<th id="S4.T2.5.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Target</th>
<th id="S4.T2.5.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Accuracy</th>
<th id="S4.T2.5.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Methods</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T2.5.1.2.1" class="ltx_tr">
<th id="S4.T2.5.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">MNIST</th>
<td id="S4.T2.5.1.2.1.2" class="ltx_td ltx_align_center ltx_border_t">SVHN</td>
<td id="S4.T2.5.1.2.1.3" class="ltx_td ltx_align_center ltx_border_t">13.25</td>
<td id="S4.T2.5.1.2.1.4" class="ltx_td ltx_align_center ltx_border_t" rowspan="4"><span id="S4.T2.5.1.2.1.4.1" class="ltx_text">S<sub id="S4.T2.5.1.2.1.4.1.1" class="ltx_sub"><span id="S4.T2.5.1.2.1.4.1.1.1" class="ltx_text ltx_font_typewriter">KDE30</span></sub></span></td>
</tr>
<tr id="S4.T2.5.1.3.2" class="ltx_tr">
<th id="S4.T2.5.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">SVHN</th>
<td id="S4.T2.5.1.3.2.2" class="ltx_td ltx_align_center">MNIST</td>
<td id="S4.T2.5.1.3.2.3" class="ltx_td ltx_align_center">29.30</td>
</tr>
<tr id="S4.T2.5.1.4.3" class="ltx_tr">
<th id="S4.T2.5.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">CIFAR-10</th>
<td id="S4.T2.5.1.4.3.2" class="ltx_td ltx_align_center">STL-10</td>
<td id="S4.T2.5.1.4.3.3" class="ltx_td ltx_align_center">15.20</td>
</tr>
<tr id="S4.T2.5.1.5.4" class="ltx_tr">
<th id="S4.T2.5.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">STL-10</th>
<td id="S4.T2.5.1.5.4.2" class="ltx_td ltx_align_center">CIFAR-10</td>
<td id="S4.T2.5.1.5.4.3" class="ltx_td ltx_align_center">15.40</td>
</tr>
<tr id="S4.T2.5.1.6.5" class="ltx_tr">
<th id="S4.T2.5.1.6.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="4">Sampling from Combined Weights Distribution</th>
</tr>
<tr id="S4.T2.5.1.7.6" class="ltx_tr">
<th id="S4.T2.5.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">MNIST+CIFAR-10</th>
<td id="S4.T2.5.1.7.6.2" class="ltx_td ltx_align_center ltx_border_t">SVHN</td>
<td id="S4.T2.5.1.7.6.3" class="ltx_td ltx_align_center ltx_border_t">18.80</td>
<td id="S4.T2.5.1.7.6.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t" rowspan="4"><span id="S4.T2.5.1.7.6.4.1" class="ltx_text">Ours</span></td>
</tr>
<tr id="S4.T2.5.1.8.7" class="ltx_tr">
<th id="S4.T2.5.1.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">MNist+CIFAR-10</th>
<td id="S4.T2.5.1.8.7.2" class="ltx_td ltx_align_center">STL-10</td>
<td id="S4.T2.5.1.8.7.3" class="ltx_td ltx_align_center">16.21</td>
</tr>
<tr id="S4.T2.5.1.9.8" class="ltx_tr">
<th id="S4.T2.5.1.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">SVHN + STL-10</th>
<td id="S4.T2.5.1.9.8.2" class="ltx_td ltx_align_center">MNIST</td>
<td id="S4.T2.5.1.9.8.3" class="ltx_td ltx_align_center">36.64</td>
</tr>
<tr id="S4.T2.5.1.10.9" class="ltx_tr">
<th id="S4.T2.5.1.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">SVHN + STL-10</th>
<td id="S4.T2.5.1.10.9.2" class="ltx_td ltx_align_center ltx_border_bb">CIFAR-10</td>
<td id="S4.T2.5.1.10.9.3" class="ltx_td ltx_align_center ltx_border_bb">18.00</td>
</tr>
</tbody>
</table>
</span></div>
</figure>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Sampling Weights for Unseen Datasets</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.1" class="ltx_p"><span id="S4.SS2.p1.1.1" class="ltx_text ltx_font_bold">Task:</span> We evaluate the transferability of the models on unseen datasets. We create disjoint modelzoos by combining MNIST and CIFAR-10 into a single modelzoo and combining the SVHN and STL-10 modelzoos. When we train on the MNIST plus CIFAR-10 modelzoos, we test on the SVHN and STL-10 modelzoos and vice-versa.</p>
</div>
<div id="S4.SS2.p2" class="ltx_para">
<p id="S4.SS2.p2.1" class="ltx_p"><span id="S4.SS2.p2.1.1" class="ltx_text ltx_font_bold">Results:</span> As shown in Table <a href="#S4.T2" title="Table 2 ‣ 4.1 In-distribution Transfer Learning ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, the D2NWG is able to sample weights with higher accuracy on unseen datasets as well as for in distribution. Through these experiments our method does not only outperform the baseline it also demonstrates promising results for dataset-conditioned sampling for unseen datasets.</p>
</div>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Performance Evaluation on KaggleZoo</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">We extend the performance evaluation on unseen dataset to much diverse datasets.
Since the baseline <cite class="ltx_cite ltx_citemacro_citet">Schürholt et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022a</a>)</cite> relies on unconditional sampling this method is not appropriate for unseen dataset evaluation. Therefore the baseline of this experiments are model pretrained on ImageNet and random initialization.</p>
</div>
<div id="S4.SS3.p2" class="ltx_para">
<p id="S4.SS3.p2.1" class="ltx_p"><span id="S4.SS3.p2.1.1" class="ltx_text ltx_font_bold">Task</span>: In this experiment, we aim to demonstrate that D2NWG can generate efficient weights for initialization and fine-tuning compared to random initialization and ImageNet pretrained models on unseen datasets with different number of classes than those encountered during training.</p>
</div>
<figure id="S4.T3" class="ltx_table">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table"><span id="S4.T3.3.1.1" class="ltx_text" style="font-size:90%;">Table 3</span>: </span><span id="S4.T3.4.2" class="ltx_text ltx_font_bold" style="font-size:90%;">Dataset Conditioned Sampling<span id="S4.T3.4.2.1" class="ltx_text ltx_font_medium"> Weight sampling on 10 unseen real-world datasets. We report two sets of results: (a) weight evaluation without finetuning and (b) weight evaluation after 50 epochs of finetuning. The pretrained baseline is trained on ImageNet1K. We report the accuracy on the testset.</span></span></figcaption>
<div id="S4.T3.5" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:346.9pt;height:176.6pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-38.7pt,19.7pt) scale(0.817617155736399,0.817617155736399) ;">
<table id="S4.T3.5.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T3.5.1.1.1" class="ltx_tr">
<th id="S4.T3.5.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt" rowspan="2"><span id="S4.T3.5.1.1.1.1.1" class="ltx_text">Datasets</span></th>
<th id="S4.T3.5.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="3">Initialization</th>
<th id="S4.T3.5.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="3">Fine-tuning</th>
</tr>
<tr id="S4.T3.5.1.2.2" class="ltx_tr">
<td id="S4.T3.5.1.2.2.1" class="ltx_td ltx_align_center">RandInit</td>
<td id="S4.T3.5.1.2.2.2" class="ltx_td ltx_align_center">Pretrained</td>
<td id="S4.T3.5.1.2.2.3" class="ltx_td ltx_align_center ltx_border_r">D2NWG</td>
<td id="S4.T3.5.1.2.2.4" class="ltx_td ltx_align_center">RandInit</td>
<td id="S4.T3.5.1.2.2.5" class="ltx_td ltx_align_center">Pretrained</td>
<td id="S4.T3.5.1.2.2.6" class="ltx_td ltx_align_center">D2NWG</td>
</tr>
<tr id="S4.T3.5.1.3.3" class="ltx_tr">
<th id="S4.T3.5.1.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt">Gemstones</th>
<th id="S4.T3.5.1.3.3.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">1.13</th>
<th id="S4.T3.5.1.3.3.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">0.62</th>
<th id="S4.T3.5.1.3.3.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt"><span id="S4.T3.5.1.3.3.4.1" class="ltx_text ltx_font_bold">1.86</span></th>
<th id="S4.T3.5.1.3.3.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">70.59</th>
<th id="S4.T3.5.1.3.3.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">67.49</th>
<th id="S4.T3.5.1.3.3.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T3.5.1.3.3.7.1" class="ltx_text ltx_font_bold">76.06</span></th>
</tr>
<tr id="S4.T3.5.1.4.4" class="ltx_tr">
<th id="S4.T3.5.1.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Dog Breeds</th>
<td id="S4.T3.5.1.4.4.2" class="ltx_td ltx_align_center">0.55</td>
<td id="S4.T3.5.1.4.4.3" class="ltx_td ltx_align_center">0.69</td>
<td id="S4.T3.5.1.4.4.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T3.5.1.4.4.4.1" class="ltx_text ltx_font_bold">1.87</span></td>
<td id="S4.T3.5.1.4.4.5" class="ltx_td ltx_align_center">80.78</td>
<td id="S4.T3.5.1.4.4.6" class="ltx_td ltx_align_center">78.13</td>
<td id="S4.T3.5.1.4.4.7" class="ltx_td ltx_align_center"><span id="S4.T3.5.1.4.4.7.1" class="ltx_text ltx_font_bold">80.88</span></td>
</tr>
<tr id="S4.T3.5.1.5.5" class="ltx_tr">
<th id="S4.T3.5.1.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_r">Dessert</th>
<th id="S4.T3.5.1.5.5.2" class="ltx_td ltx_align_center ltx_th ltx_th_column">21.03</th>
<th id="S4.T3.5.1.5.5.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">12.50</th>
<th id="S4.T3.5.1.5.5.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r"><span id="S4.T3.5.1.5.5.4.1" class="ltx_text ltx_font_bold">99.40</span></th>
<th id="S4.T3.5.1.5.5.5" class="ltx_td ltx_align_center ltx_th ltx_th_column">95.83</th>
<th id="S4.T3.5.1.5.5.6" class="ltx_td ltx_align_center ltx_th ltx_th_column">94.64</th>
<th id="S4.T3.5.1.5.5.7" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S4.T3.5.1.5.5.7.1" class="ltx_text ltx_font_bold">99.40</span></th>
</tr>
<tr id="S4.T3.5.1.6.6" class="ltx_tr">
<th id="S4.T3.5.1.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Colorectal Histology</th>
<td id="S4.T3.5.1.6.6.2" class="ltx_td ltx_align_center">11.77</td>
<td id="S4.T3.5.1.6.6.3" class="ltx_td ltx_align_center">11.00</td>
<td id="S4.T3.5.1.6.6.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T3.5.1.6.6.4.1" class="ltx_text ltx_font_bold">18.12</span></td>
<td id="S4.T3.5.1.6.6.5" class="ltx_td ltx_align_center">90.34</td>
<td id="S4.T3.5.1.6.6.6" class="ltx_td ltx_align_center">89.75</td>
<td id="S4.T3.5.1.6.6.7" class="ltx_td ltx_align_center"><span id="S4.T3.5.1.6.6.7.1" class="ltx_text ltx_font_bold">93.65</span></td>
</tr>
<tr id="S4.T3.5.1.7.7" class="ltx_tr">
<th id="S4.T3.5.1.7.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_r">Drawing</th>
<th id="S4.T3.5.1.7.7.2" class="ltx_td ltx_align_center ltx_th ltx_th_column">10.86</th>
<th id="S4.T3.5.1.7.7.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">11.00</th>
<th id="S4.T3.5.1.7.7.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r"><span id="S4.T3.5.1.7.7.4.1" class="ltx_text ltx_font_bold">11.87</span></th>
<th id="S4.T3.5.1.7.7.5" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S4.T3.5.1.7.7.5.1" class="ltx_text ltx_font_bold">90.20</span></th>
<th id="S4.T3.5.1.7.7.6" class="ltx_td ltx_align_center ltx_th ltx_th_column">90.00</th>
<th id="S4.T3.5.1.7.7.7" class="ltx_td ltx_align_center ltx_th ltx_th_column">89.00</th>
</tr>
<tr id="S4.T3.5.1.8.8" class="ltx_tr">
<th id="S4.T3.5.1.8.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">Alien vs Predator</th>
<td id="S4.T3.5.1.8.8.2" class="ltx_td ltx_align_center">51.48</td>
<td id="S4.T3.5.1.8.8.3" class="ltx_td ltx_align_center">28.88</td>
<td id="S4.T3.5.1.8.8.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T3.5.1.8.8.4.1" class="ltx_text ltx_font_bold">78.15</span></td>
<td id="S4.T3.5.1.8.8.5" class="ltx_td ltx_align_center">98.52</td>
<td id="S4.T3.5.1.8.8.6" class="ltx_td ltx_align_center"><span id="S4.T3.5.1.8.8.6.1" class="ltx_text ltx_font_bold">98.89</span></td>
<td id="S4.T3.5.1.8.8.7" class="ltx_td ltx_align_center">97.77</td>
</tr>
<tr id="S4.T3.5.1.9.9" class="ltx_tr">
<th id="S4.T3.5.1.9.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_r">COVID-19</th>
<th id="S4.T3.5.1.9.9.2" class="ltx_td ltx_align_center ltx_th ltx_th_column">20.13</th>
<th id="S4.T3.5.1.9.9.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">46.53</th>
<th id="S4.T3.5.1.9.9.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r"><span id="S4.T3.5.1.9.9.4.1" class="ltx_text ltx_font_bold">47.22</span></th>
<th id="S4.T3.5.1.9.9.5" class="ltx_td ltx_align_center ltx_th ltx_th_column">93.86</th>
<th id="S4.T3.5.1.9.9.6" class="ltx_td ltx_align_center ltx_th ltx_th_column">93.40</th>
<th id="S4.T3.5.1.9.9.7" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S4.T3.5.1.9.9.7.1" class="ltx_text ltx_font_bold">94.56</span></th>
</tr>
<tr id="S4.T3.5.1.10.10" class="ltx_tr">
<th id="S4.T3.5.1.10.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_r">honey-bee-pollen</th>
<td id="S4.T3.5.1.10.10.2" class="ltx_td ltx_align_center">49.54</td>
<td id="S4.T3.5.1.10.10.3" class="ltx_td ltx_align_center">50.00</td>
<td id="S4.T3.5.1.10.10.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S4.T3.5.1.10.10.4.1" class="ltx_text ltx_font_bold">56.94</span></td>
<td id="S4.T3.5.1.10.10.5" class="ltx_td ltx_align_center">93.05</td>
<td id="S4.T3.5.1.10.10.6" class="ltx_td ltx_align_center">88.89</td>
<td id="S4.T3.5.1.10.10.7" class="ltx_td ltx_align_center"><span id="S4.T3.5.1.10.10.7.1" class="ltx_text ltx_font_bold">93.55</span></td>
</tr>
<tr id="S4.T3.5.1.11.11" class="ltx_tr">
<th id="S4.T3.5.1.11.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_r">Speed Limit Signs</th>
<th id="S4.T3.5.1.11.11.2" class="ltx_td ltx_align_center ltx_th ltx_th_column">30.55</th>
<th id="S4.T3.5.1.11.11.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">25.00</th>
<th id="S4.T3.5.1.11.11.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r"><span id="S4.T3.5.1.11.11.4.1" class="ltx_text ltx_font_bold">31.48</span></th>
<th id="S4.T3.5.1.11.11.5" class="ltx_td ltx_align_center ltx_th ltx_th_column">83.33</th>
<th id="S4.T3.5.1.11.11.6" class="ltx_td ltx_align_center ltx_th ltx_th_column">86.11</th>
<th id="S4.T3.5.1.11.11.7" class="ltx_td ltx_align_center ltx_th ltx_th_column"><span id="S4.T3.5.1.11.11.7.1" class="ltx_text ltx_font_bold">90.74</span></th>
</tr>
<tr id="S4.T3.5.1.12.12" class="ltx_tr">
<th id="S4.T3.5.1.12.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_r">Japanese Characters</th>
<td id="S4.T3.5.1.12.12.2" class="ltx_td ltx_align_center ltx_border_bb">0.03</td>
<td id="S4.T3.5.1.12.12.3" class="ltx_td ltx_align_center ltx_border_bb">0.08</td>
<td id="S4.T3.5.1.12.12.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><span id="S4.T3.5.1.12.12.4.1" class="ltx_text ltx_font_bold">0.50</span></td>
<td id="S4.T3.5.1.12.12.5" class="ltx_td ltx_align_center ltx_border_bb">53.17</td>
<td id="S4.T3.5.1.12.12.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T3.5.1.12.12.6.1" class="ltx_text ltx_font_bold">62.33</span></td>
<td id="S4.T3.5.1.12.12.7" class="ltx_td ltx_align_center ltx_border_bb">62.16</td>
</tr>
</tbody>
</table>
</span></div>
</figure>
<figure id="S4.F4" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_4">
<figure id="S4.F4.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2402.18153/assets/x9.png" id="S4.F4.sf1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="403" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F4.sf1.2.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span><span id="S4.F4.sf1.3.2" class="ltx_text" style="font-size:90%;">Dog Breeds</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_4">
<figure id="S4.F4.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2402.18153/assets/x10.png" id="S4.F4.sf2.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="403" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F4.sf2.2.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span><span id="S4.F4.sf2.3.2" class="ltx_text" style="font-size:90%;">Alien vs Predator</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_4">
<figure id="S4.F4.sf3" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2402.18153/assets/x11.png" id="S4.F4.sf3.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="403" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F4.sf3.2.1.1" class="ltx_text" style="font-size:90%;">(c)</span> </span><span id="S4.F4.sf3.3.2" class="ltx_text" style="font-size:90%;">Japanese Characters</span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_4">
<figure id="S4.F4.sf4" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2402.18153/assets/x12.png" id="S4.F4.sf4.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="403" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F4.sf4.2.1.1" class="ltx_text" style="font-size:90%;">(d)</span> </span><span id="S4.F4.sf4.3.2" class="ltx_text" style="font-size:90%;">Colorectal Histology</span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F4.3.1.1" class="ltx_text" style="font-size:90%;">Figure 4</span>: </span><span id="S4.F4.4.2" class="ltx_text" style="font-size:90%;">Average accuracy evolution of fine-tuning for 50 epochs with sampled weights for unseen datasets from real world datasets The full results are reported in the appendix</span></figcaption>
</figure>
<div id="S4.SS3.p3" class="ltx_para">
<p id="S4.SS3.p3.1" class="ltx_p"><span id="S4.SS3.p3.1.1" class="ltx_text ltx_font_bold">Results</span>: As show in Table <a href="#S4.T3" title="Table 3 ‣ 4.3 Performance Evaluation on KaggleZoo ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> and in the convergence plots in Figure <a href="#S4.F4" title="Figure 4 ‣ 4.3 Performance Evaluation on KaggleZoo ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, weight initialization with D2NWG without any fine-tuning achieves 99.04% on dessert dataset outperforming the random initialized model trained for 50 epochs. Additionally, our method consistently speeds-up the convergence on all tasks compared to the random initialized and pretrained models.
Except on Japanese Characters, Gemstones and Dog Breeds datasets where both methods achieved similar results, the proposed method consistently outperformed the two baselines and converges faster on the finetuning task. Although the test subsets and the training subsets of image dataset have no overlapping classes our method is able to transfer well to the test set.</p>
</div>
<div id="S4.SS3.p4" class="ltx_para">
<p id="S4.SS3.p4.1" class="ltx_p">We investigate the reason why the proposed method achieves superior performance on some datasets. Our findings suggest that D2NWG achieves similar none finetuning performance as the random initialization in the cases where the target dataset has more classes than the maximum number classes in the pretrained datasets. The sapmled weight performance on the dessert dataset is due to the fact that this dataset contains similar images as those in the foods dataset which are a subsets of the food classification task in the pretrained image datasets. These results showed that the dataset encoder efficiently encodes the datasets and if some target datasets overlap with those in the modelzoo, the proposed method is able to retrieve the weights that transfer well. Detailed results can be found in Appendix <a href="#A8" title="Appendix H Detail of Experimental Results ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">H</span></a>.</p>
</div>
<figure id="S4.T4" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S4.T4.11.5.1" class="ltx_text" style="font-size:90%;">Table 4</span>: </span><span id="S4.T4.8.4" class="ltx_text ltx_font_bold" style="font-size:90%;">Cross-Dataset Trasfer<span id="S4.T4.8.4.4" class="ltx_text ltx_font_medium"> MNIST<math id="S4.T4.5.1.1.m1.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S4.T4.5.1.1.m1.1b"><mo stretchy="false" id="S4.T4.5.1.1.m1.1.1" xref="S4.T4.5.1.1.m1.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S4.T4.5.1.1.m1.1c"><ci id="S4.T4.5.1.1.m1.1.1.cmml" xref="S4.T4.5.1.1.m1.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T4.5.1.1.m1.1d">\rightarrow</annotation></semantics></math> SVHN, SVHN<math id="S4.T4.6.2.2.m2.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S4.T4.6.2.2.m2.1b"><mo stretchy="false" id="S4.T4.6.2.2.m2.1.1" xref="S4.T4.6.2.2.m2.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S4.T4.6.2.2.m2.1c"><ci id="S4.T4.6.2.2.m2.1.1.cmml" xref="S4.T4.6.2.2.m2.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T4.6.2.2.m2.1d">\rightarrow</annotation></semantics></math> MNIST, CIFAR-10<math id="S4.T4.7.3.3.m3.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S4.T4.7.3.3.m3.1b"><mo stretchy="false" id="S4.T4.7.3.3.m3.1.1" xref="S4.T4.7.3.3.m3.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S4.T4.7.3.3.m3.1c"><ci id="S4.T4.7.3.3.m3.1.1.cmml" xref="S4.T4.7.3.3.m3.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T4.7.3.3.m3.1d">\rightarrow</annotation></semantics></math> STL-10, STL-10 <math id="S4.T4.8.4.4.m4.1" class="ltx_Math" alttext="\rightarrow" display="inline"><semantics id="S4.T4.8.4.4.m4.1b"><mo stretchy="false" id="S4.T4.8.4.4.m4.1.1" xref="S4.T4.8.4.4.m4.1.1.cmml">→</mo><annotation-xml encoding="MathML-Content" id="S4.T4.8.4.4.m4.1c"><ci id="S4.T4.8.4.4.m4.1.1.cmml" xref="S4.T4.8.4.4.m4.1.1">→</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.T4.8.4.4.m4.1d">\rightarrow</annotation></semantics></math> CIFAR-10. The OFA model is pretrained on ImageNet.</span></span></figcaption>
<div id="S4.T4.12" class="ltx_inline-block ltx_transformed_outer" style="width:433.6pt;height:122.6pt;vertical-align:-1.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-7.9pt,2.2pt) scale(0.964849039938924,0.964849039938924) ;">
<table id="S4.T4.12.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T4.12.1.1.1" class="ltx_tr">
<th id="S4.T4.12.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Model</th>
<th id="S4.T4.12.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">MNIST</th>
<th id="S4.T4.12.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">SVHN</th>
<th id="S4.T4.12.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">CIFAR10</th>
<th id="S4.T4.12.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">STL10</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T4.12.1.2.1" class="ltx_tr">
<th id="S4.T4.12.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">Pretrained (seen)</th>
<td id="S4.T4.12.1.2.1.2" class="ltx_td ltx_align_center ltx_border_tt">97.44</td>
<td id="S4.T4.12.1.2.1.3" class="ltx_td ltx_align_center ltx_border_tt">84.04</td>
<td id="S4.T4.12.1.2.1.4" class="ltx_td ltx_align_center ltx_border_tt">81.83</td>
<td id="S4.T4.12.1.2.1.5" class="ltx_td ltx_align_center ltx_border_tt">66.62</td>
</tr>
<tr id="S4.T4.12.1.3.2" class="ltx_tr">
<th id="S4.T4.12.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">D2NWG (seen)</th>
<td id="S4.T4.12.1.3.2.2" class="ltx_td ltx_align_center">97.49</td>
<td id="S4.T4.12.1.3.2.3" class="ltx_td ltx_align_center">83.12</td>
<td id="S4.T4.12.1.3.2.4" class="ltx_td ltx_align_center">82.19</td>
<td id="S4.T4.12.1.3.2.5" class="ltx_td ltx_align_center">66.13</td>
</tr>
<tr id="S4.T4.12.1.4.3" class="ltx_tr">
<th id="S4.T4.12.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">OFA (Pretrained)<cite class="ltx_cite ltx_citemacro_citep">(Cai et al., <a href="#bib.bib2" title="" class="ltx_ref">2020</a>)</cite>
</th>
<td id="S4.T4.12.1.4.3.2" class="ltx_td ltx_align_center ltx_border_t">13.34</td>
<td id="S4.T4.12.1.4.3.3" class="ltx_td ltx_align_center ltx_border_t">8.90</td>
<td id="S4.T4.12.1.4.3.4" class="ltx_td ltx_align_center ltx_border_t">13.34</td>
<td id="S4.T4.12.1.4.3.5" class="ltx_td ltx_align_center ltx_border_t">8.90</td>
</tr>
<tr id="S4.T4.12.1.5.4" class="ltx_tr">
<th id="S4.T4.12.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">cross_dataset (unseen)</th>
<td id="S4.T4.12.1.5.4.2" class="ltx_td ltx_align_center"><span id="S4.T4.12.1.5.4.2.1" class="ltx_text ltx_font_bold">50.70</span></td>
<td id="S4.T4.12.1.5.4.3" class="ltx_td ltx_align_center">28.68</td>
<td id="S4.T4.12.1.5.4.4" class="ltx_td ltx_align_center">34.16</td>
<td id="S4.T4.12.1.5.4.5" class="ltx_td ltx_align_center"><span id="S4.T4.12.1.5.4.5.1" class="ltx_text ltx_font_bold">39.95</span></td>
</tr>
<tr id="S4.T4.12.1.6.5" class="ltx_tr">
<th id="S4.T4.12.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">D2NWG (unseen)</th>
<td id="S4.T4.12.1.6.5.2" class="ltx_td ltx_align_center">49.32</td>
<td id="S4.T4.12.1.6.5.3" class="ltx_td ltx_align_center"><span id="S4.T4.12.1.6.5.3.1" class="ltx_text ltx_font_bold">29.13</span></td>
<td id="S4.T4.12.1.6.5.4" class="ltx_td ltx_align_center"><span id="S4.T4.12.1.6.5.4.1" class="ltx_text ltx_font_bold">35.02</span></td>
<td id="S4.T4.12.1.6.5.5" class="ltx_td ltx_align_center">39.90</td>
</tr>
<tr id="S4.T4.12.1.7.6" class="ltx_tr">
<th id="S4.T4.12.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">D2NWG+average (unseen)</th>
<td id="S4.T4.12.1.7.6.2" class="ltx_td ltx_align_center ltx_border_bb">50.24</td>
<td id="S4.T4.12.1.7.6.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T4.12.1.7.6.3.1" class="ltx_text ltx_font_bold">29.13</span></td>
<td id="S4.T4.12.1.7.6.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S4.T4.12.1.7.6.4.1" class="ltx_text ltx_font_bold">35.03</span></td>
<td id="S4.T4.12.1.7.6.5" class="ltx_td ltx_align_center ltx_border_bb">40.00</td>
</tr>
</tbody>
</table>
</span></div>
</figure>
<div id="S4.SS3.p5" class="ltx_para">
<p id="S4.SS3.p5.1" class="ltx_p"><span id="S4.SS3.p5.1.1" class="ltx_text ltx_font_bold">Evaluation on Larger Architectures:</span> So far we have restricted ourselves to modelzoos populated by realitively simple CNN architectures. In this section we evaluate our method on MobileNetV3, a subnetwork sample from OFA <cite class="ltx_cite ltx_citemacro_citep">(Cai et al., <a href="#bib.bib2" title="" class="ltx_ref">2020</a>)</cite> pretrained on CIFAR-10, STL-10, SVHN and MNIST for 25 epochs. The last 10 checkpoints per dataset are collected and we learn the pretrained weights distribution using our method. The performance of the sampled weighs are presented in Table <a href="#S4.T4" title="Table 4 ‣ 4.3 Performance Evaluation on KaggleZoo ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> and shows that the proposed method transfers well to unseen dataset as well as on seen datasets. In the last row of Table <a href="#S4.T4" title="Table 4 ‣ 4.3 Performance Evaluation on KaggleZoo ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, we provide ModelSoup-type <cite class="ltx_cite ltx_citemacro_citep">(Wortsman et al., <a href="#bib.bib28" title="" class="ltx_ref">2022</a>)</cite> experiments where we average multiple weights sampled from our model.</p>
</div>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4 </span>Ablation study</h3>

<div id="S4.SS4.p1" class="ltx_para">
<p id="S4.SS4.p1.1" class="ltx_p">In this section, we investigate the importance and limitation of the different modules used to build our method.</p>
</div>
<section id="S4.SS4.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.4.1 </span>Effect of Modelzoo Size Generalization</h4>

<div id="S4.SS4.SSS1.p1" class="ltx_para">
<p id="S4.SS4.SSS1.p1.1" class="ltx_p">In this part we investigate whether increasing the number of datasets will increase the performance on unseen datasets. To achieve that goal, we perform experiments with modelzoos of size 5000, 10000 and 20000 created from subsets of ImageNet. We use CIFAR-10 and STL-10 as unseen target datasets.
We sample 50 weights and report the average of the top 5 performing weights in Figure <a href="#S4.F5.sf1" title="In Figure 5 ‣ 4.4.1 Effect of Modelzoo Size Generalization ‣ 4.4 Ablation study ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5(a)</span></a> where it can be observed that increasing the modelzoo size improve generalization.</p>
</div>
<div id="S4.SS4.SSS1.p2" class="ltx_para">
<p id="S4.SS4.SSS1.p2.6" class="ltx_p"><span id="S4.SS4.SSS1.p2.6.1" class="ltx_text ltx_font_bold">Results:</span> On CIFAR-10 and STL-10, we obtain accuracies of 39.60<math id="S4.SS4.SSS1.p2.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.SS4.SSS1.p2.1.m1.1a"><mo id="S4.SS4.SSS1.p2.1.m1.1.1" xref="S4.SS4.SSS1.p2.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS1.p2.1.m1.1b"><csymbol cd="latexml" id="S4.SS4.SSS1.p2.1.m1.1.1.cmml" xref="S4.SS4.SSS1.p2.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.SSS1.p2.1.m1.1c">\pm</annotation></semantics></math>1.31% and 44.66 <math id="S4.SS4.SSS1.p2.2.m2.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.SS4.SSS1.p2.2.m2.1a"><mo id="S4.SS4.SSS1.p2.2.m2.1.1" xref="S4.SS4.SSS1.p2.2.m2.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS1.p2.2.m2.1b"><csymbol cd="latexml" id="S4.SS4.SSS1.p2.2.m2.1.1.cmml" xref="S4.SS4.SSS1.p2.2.m2.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.SSS1.p2.2.m2.1c">\pm</annotation></semantics></math> 0.55% for 5000 subsets, 42.15<math id="S4.SS4.SSS1.p2.3.m3.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.SS4.SSS1.p2.3.m3.1a"><mo id="S4.SS4.SSS1.p2.3.m3.1.1" xref="S4.SS4.SSS1.p2.3.m3.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS1.p2.3.m3.1b"><csymbol cd="latexml" id="S4.SS4.SSS1.p2.3.m3.1.1.cmml" xref="S4.SS4.SSS1.p2.3.m3.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.SSS1.p2.3.m3.1c">\pm</annotation></semantics></math>2.12 and 64.83 <math id="S4.SS4.SSS1.p2.4.m4.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.SS4.SSS1.p2.4.m4.1a"><mo id="S4.SS4.SSS1.p2.4.m4.1.1" xref="S4.SS4.SSS1.p2.4.m4.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS1.p2.4.m4.1b"><csymbol cd="latexml" id="S4.SS4.SSS1.p2.4.m4.1.1.cmml" xref="S4.SS4.SSS1.p2.4.m4.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.SSS1.p2.4.m4.1c">\pm</annotation></semantics></math> 2.83 for 10000 subsets, and 52.64<math id="S4.SS4.SSS1.p2.5.m5.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.SS4.SSS1.p2.5.m5.1a"><mo id="S4.SS4.SSS1.p2.5.m5.1.1" xref="S4.SS4.SSS1.p2.5.m5.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS1.p2.5.m5.1b"><csymbol cd="latexml" id="S4.SS4.SSS1.p2.5.m5.1.1.cmml" xref="S4.SS4.SSS1.p2.5.m5.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.SSS1.p2.5.m5.1c">\pm</annotation></semantics></math>3.12% and, 80.49 <math id="S4.SS4.SSS1.p2.6.m6.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="S4.SS4.SSS1.p2.6.m6.1a"><mo id="S4.SS4.SSS1.p2.6.m6.1.1" xref="S4.SS4.SSS1.p2.6.m6.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="S4.SS4.SSS1.p2.6.m6.1b"><csymbol cd="latexml" id="S4.SS4.SSS1.p2.6.m6.1.1.cmml" xref="S4.SS4.SSS1.p2.6.m6.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S4.SS4.SSS1.p2.6.m6.1c">\pm</annotation></semantics></math> 1.77% for 20000 subsets.
The maximum accuracy with random initialization are 12.11%, and 17.12% on CIFAR-10 and STL-10 with no fine-tuning.</p>
</div>
<figure id="S4.F5" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F5.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2402.18153/assets/x13.png" id="S4.F5.sf1.g1" class="ltx_graphics ltx_centering ltx_img_portrait" width="461" height="638" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F5.sf1.2.1.1" class="ltx_text" style="font-size:90%;">(a)</span> </span></figcaption>
</figure>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<figure id="S4.F5.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2402.18153/assets/x14.png" id="S4.F5.sf2.g1" class="ltx_graphics ltx_centering ltx_img_portrait" width="461" height="638" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F5.sf2.2.1.1" class="ltx_text" style="font-size:90%;">(b)</span> </span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F5.3.1.1" class="ltx_text" style="font-size:90%;">Figure 5</span>: </span><span id="S4.F5.4.2" class="ltx_text" style="font-size:90%;">(a) Effect of the number of pretrained datasets on sampling weights performance on unseen datasets. (b) Performance comparison on in-distribution sampling of methods with VAE+DDPM vs DDPM</span></figcaption>
</figure>
</section>
<section id="S4.SS4.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.4.2 </span>Sampling without Latent Representation</h4>

<div id="S4.SS4.SSS2.p1" class="ltx_para">
<p id="S4.SS4.SSS2.p1.1" class="ltx_p">We investigate a variant of our model that learns the diffusion model directly on the weights without the AutoEncoder stage. We compare these two setting, with and without the AutoEncoder. Both variants are trained on 1000 subsets of ImageNet. We report the results of in-distribution sampling on three datasets in Figure <a href="#S4.F5.sf2" title="In Figure 5 ‣ 4.4.1 Effect of Modelzoo Size Generalization ‣ 4.4 Ablation study ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5(b)</span></a>.</p>
</div>
<div id="S4.SS4.SSS2.p2" class="ltx_para">
<p id="S4.SS4.SSS2.p2.1" class="ltx_p"><span id="S4.SS4.SSS2.p2.1.1" class="ltx_text ltx_font_bold">Results:</span> As reported in Figure <a href="#S4.F5.sf2" title="In Figure 5 ‣ 4.4.1 Effect of Modelzoo Size Generalization ‣ 4.4 Ablation study ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5(b)</span></a>, for the same diffusion architecture, it is much easier to learn the pretrained weights distribution in latent space. One of the reason for which the DDPM process failed on the raw pretrained may be due to the fact that raw pretrained weights may require higher model capacity. Note also that learning the diffusion model directly on the weight space is computationally expensive. Hence, perhaps learning diffusion on subsets of the weights might be a more promising direction both in terms of computation and performance and we leave this as future work.</p>
</div>
</section>
<section id="S4.SS4.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.4.3 </span>CLIP-based Dataset Encoding</h4>

<div id="S4.SS4.SSS3.p1" class="ltx_para">
<p id="S4.SS4.SSS3.p1.1" class="ltx_p">We further investigate the different between CLIP-based dataset encoding scheme trained in an intermediate stage and the Set Transformer encoder jointly trained with the diffusion process. We conduct the experiments on the 140 Kaggle datasets and their corresponding modelzoos.</p>
</div>
<div id="S4.SS4.SSS3.p2" class="ltx_para">
<p id="S4.SS4.SSS3.p2.1" class="ltx_p"><span id="S4.SS4.SSS3.p2.1.1" class="ltx_text ltx_font_bold">Results</span>: We compare the in-distribution sampling, out-distribution initialization, and fine-tuning results. As shown in Figure <a href="#S4.F6" title="Figure 6 ‣ 4.4.3 CLIP-based Dataset Encoding ‣ 4.4 Ablation study ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>, for in-distribution sampling both methods achieve similar results for small number of datasets. When the number of datasets increases the set-transformer jointly trained with diffusion approach struggle to converge.</p>
</div>
<figure id="S4.F6" class="ltx_figure"><img src="/html/2402.18153/assets/x15.png" id="S4.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="424" height="305" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F6.3.1.1" class="ltx_text" style="font-size:90%;">Figure 6</span>: </span><span id="S4.F6.4.2" class="ltx_text" style="font-size:90%;">Performance comparison at initialization of method with jointly trained set-transformer (Without CLIP) and method clip-based dataset encoder.</span></figcaption>
</figure>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Conclusion</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">In this work we presented a three stage approach to neural network weight generation that involves fitting an AutoEncoder to pretrained weights, learning a dataset encoder using CLIP embeddings and learning a diffusion model to manipulate the latent space conditioned on the dataset embeddings. We demonstrated that the proposed method can sample high quality weights for both seen and unseen datasets and achieves better performance when finetuned for a fixed number of epochs when benchmarked against the relevant baselines.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Blattmann et al. (2022)</span>
<span class="ltx_bibblock">
Blattmann, A., Rombach, R., Oktay, K., Müller, J., and Ommer, B.

</span>
<span class="ltx_bibblock">Retrieval-augmented diffusion models.

</span>
<span class="ltx_bibblock">In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), <em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, volume 35, pp.  15309–15324. Curran Associates, Inc., 2022.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cai et al. (2020)</span>
<span class="ltx_bibblock">
Cai, H., Gan, C., Wang, T., Zhang, Z., and Han, S.

</span>
<span class="ltx_bibblock">Once for all: Train one network and specialize it for efficient deployment.

</span>
<span class="ltx_bibblock">In <em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2020.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chauhan et al. (2023)</span>
<span class="ltx_bibblock">
Chauhan, V. K., Zhou, J., Lu, P., Molaei, S., and Clifton, D. A.

</span>
<span class="ltx_bibblock">A brief review of hypernetworks in deep learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2306.06955, 2023.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://api.semanticscholar.org/CorpusID:259138728" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://api.semanticscholar.org/CorpusID:259138728</a>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Croitoru et al. (2022)</span>
<span class="ltx_bibblock">
Croitoru, F.-A., Hondru, V., Ionescu, R. T., and Shah, M.

</span>
<span class="ltx_bibblock">Diffusion models in vision: A survey.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">IEEE Transactions on Pattern Analysis and Machine Intelligence</em>, 45:10850–10869, 2022.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Denil et al. (2013)</span>
<span class="ltx_bibblock">
Denil, M., Shakibi, B., Dinh, L., Ranzato, M. A., and de Freitas, N.

</span>
<span class="ltx_bibblock">Predicting parameters in deep learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, volume 26. Curran Associates, Inc., 2013.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dhariwal &amp; Nichol (2021)</span>
<span class="ltx_bibblock">
Dhariwal, P. and Nichol, A. Q.

</span>
<span class="ltx_bibblock">Diffusion models beat GANs on image synthesis.

</span>
<span class="ltx_bibblock">In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), <em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 2021.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Doke &amp; Gaikwad (2021)</span>
<span class="ltx_bibblock">
Doke, A. and Gaikwad, M.

</span>
<span class="ltx_bibblock">Survey on automated machine learning (automl) and meta learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">2021 12th International Conference on Computing Communication and Networking Technologies (ICCCNT)</em>, pp.  1–5, 2021.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1109/ICCCNT51525.2021.9579526</span>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ha et al. (2016)</span>
<span class="ltx_bibblock">
Ha, D., Dai, A., and Le, Q. V.

</span>
<span class="ltx_bibblock">Hypernetworks, 2016.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ha et al. (2017)</span>
<span class="ltx_bibblock">
Ha, D., Dai, A. M., and Le, Q. V.

</span>
<span class="ltx_bibblock">Hypernetworks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2017.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ho &amp; Salimans (2021)</span>
<span class="ltx_bibblock">
Ho, J. and Salimans, T.

</span>
<span class="ltx_bibblock">Classifier-free diffusion guidance.

</span>
<span class="ltx_bibblock">In <em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications</em>, 2021.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ho et al. (2020)</span>
<span class="ltx_bibblock">
Ho, J., Jain, A., and Abbeel, P.

</span>
<span class="ltx_bibblock">Denoising diffusion probabilistic models.

</span>
<span class="ltx_bibblock">In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), <em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, volume 33, pp.  6840–6851. Curran Associates, Inc., 2020.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hutter et al. (2019)</span>
<span class="ltx_bibblock">
Hutter, F., Kotthoff, L., and Vanschoren, J. (eds.).

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Automated Machine Learning - Methods, Systems, Challenges</em>.

</span>
<span class="ltx_bibblock">Springer, 2019.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jeong et al. (2021)</span>
<span class="ltx_bibblock">
Jeong, W., Lee, H., Park, G. H., Hyung, E., Baek, J., and Hwang, S. J.

</span>
<span class="ltx_bibblock">Task-adaptive neural network search with meta-contrastive learning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Neural Information Processing Systems</em>, 2021.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Knyazev et al. (2021)</span>
<span class="ltx_bibblock">
Knyazev, B., Drozdzal, M., Taylor, G. W., and Romero-Soriano, A.

</span>
<span class="ltx_bibblock">Parameter prediction for unseen deep architectures.

</span>
<span class="ltx_bibblock">In <em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>, 2021.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Knyazev et al. (2023)</span>
<span class="ltx_bibblock">
Knyazev, B., Hwang, D., and Lacoste-Julien, S.

</span>
<span class="ltx_bibblock">Can we scale transformers to predict parameters of diverse imagenet models?

</span>
<span class="ltx_bibblock">In <em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, 2023.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee et al. (2021)</span>
<span class="ltx_bibblock">
Lee, H., Hyung, E., and Hwang, S. J.

</span>
<span class="ltx_bibblock">Rapid neural architecture search by learning to generate graphs from datasets.

</span>
<span class="ltx_bibblock">In <em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2021.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee et al. (2019)</span>
<span class="ltx_bibblock">
Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S., and Teh, Y. W.

</span>
<span class="ltx_bibblock">Set transformer: A framework for attention-based permutation-invariant neural networks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 36th International Conference on Machine Learning</em>, pp.  3744–3753, 2019.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mlodozeniec et al. (2023)</span>
<span class="ltx_bibblock">
Mlodozeniec, B. K., Reisser, M., and Louizos, C.

</span>
<span class="ltx_bibblock">Hyperparameter optimization through neural network partitioning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">The Eleventh International Conference on Learning Representations</em>, 2023.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nava et al. (2023)</span>
<span class="ltx_bibblock">
Nava, E., Kobayashi, S., Yin, Y., Katzschmann, R. K., and Grewe, B. F.

</span>
<span class="ltx_bibblock">Meta-learning via classifier(-free) diffusion guidance.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">Transactions on Machine Learning Research</em>, 2023.

</span>
<span class="ltx_bibblock">ISSN 2835-8856.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nichol &amp; Dhariwal (2021)</span>
<span class="ltx_bibblock">
Nichol, A. Q. and Dhariwal, P.

</span>
<span class="ltx_bibblock">Improved denoising diffusion probabilistic models.

</span>
<span class="ltx_bibblock">In Meila, M. and Zhang, T. (eds.), <em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 38th International Conference on Machine Learning</em>, volume 139 of <em id="bib.bib20.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>, pp.  8162–8171. PMLR, 18–24 Jul 2021.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Peebles et al. (2022)</span>
<span class="ltx_bibblock">
Peebles, W., Radosavovic, I., Brooks, T., Efros, A. A., and Malik, J.

</span>
<span class="ltx_bibblock">Learning to learn with generative models of neural network checkpoints, 2022.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. (2021)</span>
<span class="ltx_bibblock">
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I.

</span>
<span class="ltx_bibblock">Learning transferable visual models from natural language supervision.

</span>
<span class="ltx_bibblock">In <em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">International Conference on Machine Learning</em>, 2021.

</span>
<span class="ltx_bibblock">URL <a target="_blank" href="https://api.semanticscholar.org/CorpusID:231591445" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://api.semanticscholar.org/CorpusID:231591445</a>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ratzlaff &amp; Fuxin (2020)</span>
<span class="ltx_bibblock">
Ratzlaff, N. and Fuxin, L.

</span>
<span class="ltx_bibblock">Hypergan: A generative model for diverse, performant neural networks, 2020.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rombach et al. (2021)</span>
<span class="ltx_bibblock">
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B.

</span>
<span class="ltx_bibblock">High-resolution image synthesis with latent diffusion models.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)</em>, pp.  10674–10685, 2021.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schürholt et al. (2022a)</span>
<span class="ltx_bibblock">
Schürholt, K., Knyazev, B., Giró-i Nieto, X., and Borth, D.

</span>
<span class="ltx_bibblock">Hyper-representations as generative models: Sampling unseen neural network weights.

</span>
<span class="ltx_bibblock">In <em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">Thirty-Sixth Conference on Neural Information Processing Systems (NeurIPS)</em>, September 2022a.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schürholt et al. (2022b)</span>
<span class="ltx_bibblock">
Schürholt, K., Taskiran, D., Knyazev, B., Giró-i Nieto, X., and Borth, D.

</span>
<span class="ltx_bibblock">Model zoos: A dataset of diverse populations of neural network models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">Thirty-Sixth Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks</em>, September 2022b.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schürholt et al. (2021)</span>
<span class="ltx_bibblock">
Schürholt, K., Kostadinov, D., and Borth, D.

</span>
<span class="ltx_bibblock">Self-supervised representation learning on neural network weights for model characteristic prediction.

</span>
<span class="ltx_bibblock">In <em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems (NeurIPS 2021)</em>, Sydney, Australia, 2021.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wortsman et al. (2022)</span>
<span class="ltx_bibblock">
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L.

</span>
<span class="ltx_bibblock">Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.

</span>
<span class="ltx_bibblock">In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), <em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 39th International Conference on Machine Learning</em>, volume 162 of <em id="bib.bib28.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>, pp.  23965–23998. PMLR, 17–23 Jul 2022.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2019)</span>
<span class="ltx_bibblock">
Zhang, C., Ren, M., and Urtasun, R.

</span>
<span class="ltx_bibblock">Graph hypernetworks for neural architecture search.

</span>
<span class="ltx_bibblock">In <em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations</em>, 2019.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhmoginov et al. (2022)</span>
<span class="ltx_bibblock">
Zhmoginov, A., Sandler, M., and Vladymyrov, M.

</span>
<span class="ltx_bibblock">HyperTransformer: Model generation for supervised and semi-supervised few-shot learning.

</span>
<span class="ltx_bibblock">In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), <em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 39th International Conference on Machine Learning</em>, volume 162 of <em id="bib.bib30.2.2" class="ltx_emph ltx_font_italic">Proceedings of Machine Learning Research</em>, pp.  27075–27098. PMLR, 17–23 Jul 2022.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhu et al. (2023)</span>
<span class="ltx_bibblock">
Zhu, Y., Li, Z., Wang, T., He, M., and Yao, C.

</span>
<span class="ltx_bibblock">Conditional text image generation with diffusion models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)</em>, pp.  14235–14244, 2023.

</span>
<span class="ltx_bibblock">doi: <span class="ltx_ref ltx_nolink ltx_Url ltx_ref_self">10.1109/CVPR52729.2023.01368</span>.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Discussion</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.1" class="ltx_p"><span id="A1.p1.1.1" class="ltx_text ltx_font_bold">Broader Impact</span> D2NWG addresses the resource-intensive nature of deep learning by proposing a method for efficient transfer learning. This has the potential to reduce the computational resources required for training neural networks, making it more accessible to a wider range of researchers and organizations.</p>
</div>
<div id="A1.p2" class="ltx_para">
<p id="A1.p2.1" class="ltx_p"><span id="A1.p2.1.1" class="ltx_text ltx_font_bold">Limitation</span>
In this work we focus mainly on generalization across datasets. Therefore, sampling for unseen architecture may not guarantee performance improvement. Additionally, while diffusion model may achieve impressive performance on image generation, there are still some challenges to efficiently recast it for weights generation including memory constraint and convergence challenge.</p>
</div>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Details of Modelzoo Generation</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.1" class="ltx_p">The code source will be released on Github <a target="_blank" href="https://github.com/sorobedio/DNNWG" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/sorobedio/DNNWG</a></p>
</div>
<section id="A2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.1 </span>Modelzoo and Pretrained Datasets</h3>

<div id="A2.SS1.p1" class="ltx_para">
<p id="A2.SS1.p1.1" class="ltx_p">All MLP based modelzoos are trained with Adam optimizer with learning rate 1e-3 for 30 epochs. And we collect the checkpoints from epochs 20 to 30.</p>
</div>
<div id="A2.SS1.p2" class="ltx_para">
<p id="A2.SS1.p2.1" class="ltx_p"><span id="A2.SS1.p2.1.1" class="ltx_text ltx_font_bold">Model zoo</span> We use the pretrained datasets from <cite class="ltx_cite ltx_citemacro_citet">Schürholt et al. (<a href="#bib.bib26" title="" class="ltx_ref">2022b</a>)</cite> as structured in <cite class="ltx_cite ltx_citemacro_citep">(Schürholt et al., <a href="#bib.bib25" title="" class="ltx_ref">2022a</a>)</cite>. This dataset consists of 4 different datasets with 5000 pretrained weights per architectures and datasets. The details of the architecture used to generate the pretrained weights are available in <cite class="ltx_cite ltx_citemacro_citet">Schürholt et al. (<a href="#bib.bib26" title="" class="ltx_ref">2022b</a>)</cite>.</p>
</div>
<div id="A2.SS1.p3" class="ltx_para">
<p id="A2.SS1.p3.1" class="ltx_p"><span id="A2.SS1.p3.1.1" class="ltx_text ltx_font_bold">KaggleZoo</span>
The datasets used to generate this modelzoo generation is the same as those defined in <cite class="ltx_cite ltx_citemacro_citep">(Jeong et al., <a href="#bib.bib13" title="" class="ltx_ref">2021</a>)</cite>.
To efficiently generate the pretrained weights, we first compute the features of each image then use a MLP with two layers with input size 512 hidden size 256 with leaky_relu activation. We train the MLP on clip features because it allows us to quickly generate high performing weights. For each datasets we used the last 10 checkpoints which results in 1400 pretrained vector for training.</p>
</div>
<div id="A2.SS1.p4" class="ltx_para">
<p id="A2.SS1.p4.1" class="ltx_p"><span id="A2.SS1.p4.1.1" class="ltx_text ltx_font_bold">ImageNet zoo</span>
To generate the pretrained modelzoo on ImageNet, we sample respectively 1000, 5000, 10000 and 20000 subsets with 10 classes respectively and 100 images per class in the training set and 50 for per class in test set. For 1000 and 5000 subsets we used the same MLP architecture as the KaggleZoo. For the 10000 we reduce the hidden dimension to 128 and, for 20000 subsets we only use one linear probing layer. On the other datasets linear probing shows similar generalization performance similar to the two layer MLP architectures. Reducing the model size is motivated by the computation cost. We use Adam optimizer with learning rate set 1e-3. All models are trained for 30 epochs.</p>
</div>
</section>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Relationship Between Datasets and Trained Weights</h2>

<div id="A3.p1" class="ltx_para">
<p id="A3.p1.1" class="ltx_p">Nowadays, gradient descent based optimization is the commonly used technique to generate optimal neural network weights through training by minimizing a loss function, ie. cross-entropy for classifiation tasks. The weights optimized with gradient descent thus contains some information about the training data. Therefore, understanding the correlation between the training dataset and the optimal weights is important for the generation of weights.</p>
</div>
<div id="A3.p2" class="ltx_para">
<p id="A3.p2.3" class="ltx_p">During the optimization process with gradient descent the weights of each layer <math id="A3.p2.1.m1.1" class="ltx_Math" alttext="i" display="inline"><semantics id="A3.p2.1.m1.1a"><mi id="A3.p2.1.m1.1.1" xref="A3.p2.1.m1.1.1.cmml">i</mi><annotation-xml encoding="MathML-Content" id="A3.p2.1.m1.1b"><ci id="A3.p2.1.m1.1.1.cmml" xref="A3.p2.1.m1.1.1">𝑖</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.p2.1.m1.1c">i</annotation></semantics></math> are updated as <math id="A3.p2.2.m2.4" class="ltx_Math" alttext="w_{i}=w_{i-1}-\eta\nabla_{w_{i}}\mathcal{L}(w_{1},w_{2},\ldots,w_{n})" display="inline"><semantics id="A3.p2.2.m2.4a"><mrow id="A3.p2.2.m2.4.4" xref="A3.p2.2.m2.4.4.cmml"><msub id="A3.p2.2.m2.4.4.5" xref="A3.p2.2.m2.4.4.5.cmml"><mi id="A3.p2.2.m2.4.4.5.2" xref="A3.p2.2.m2.4.4.5.2.cmml">w</mi><mi id="A3.p2.2.m2.4.4.5.3" xref="A3.p2.2.m2.4.4.5.3.cmml">i</mi></msub><mo id="A3.p2.2.m2.4.4.4" xref="A3.p2.2.m2.4.4.4.cmml">=</mo><mrow id="A3.p2.2.m2.4.4.3" xref="A3.p2.2.m2.4.4.3.cmml"><msub id="A3.p2.2.m2.4.4.3.5" xref="A3.p2.2.m2.4.4.3.5.cmml"><mi id="A3.p2.2.m2.4.4.3.5.2" xref="A3.p2.2.m2.4.4.3.5.2.cmml">w</mi><mrow id="A3.p2.2.m2.4.4.3.5.3" xref="A3.p2.2.m2.4.4.3.5.3.cmml"><mi id="A3.p2.2.m2.4.4.3.5.3.2" xref="A3.p2.2.m2.4.4.3.5.3.2.cmml">i</mi><mo id="A3.p2.2.m2.4.4.3.5.3.1" xref="A3.p2.2.m2.4.4.3.5.3.1.cmml">−</mo><mn id="A3.p2.2.m2.4.4.3.5.3.3" xref="A3.p2.2.m2.4.4.3.5.3.3.cmml">1</mn></mrow></msub><mo id="A3.p2.2.m2.4.4.3.4" xref="A3.p2.2.m2.4.4.3.4.cmml">−</mo><mrow id="A3.p2.2.m2.4.4.3.3" xref="A3.p2.2.m2.4.4.3.3.cmml"><mi id="A3.p2.2.m2.4.4.3.3.5" xref="A3.p2.2.m2.4.4.3.3.5.cmml">η</mi><mo lspace="0.167em" rspace="0em" id="A3.p2.2.m2.4.4.3.3.4" xref="A3.p2.2.m2.4.4.3.3.4.cmml">​</mo><mrow id="A3.p2.2.m2.4.4.3.3.6" xref="A3.p2.2.m2.4.4.3.3.6.cmml"><msub id="A3.p2.2.m2.4.4.3.3.6.1" xref="A3.p2.2.m2.4.4.3.3.6.1.cmml"><mo rspace="0.167em" id="A3.p2.2.m2.4.4.3.3.6.1.2" xref="A3.p2.2.m2.4.4.3.3.6.1.2.cmml">∇</mo><msub id="A3.p2.2.m2.4.4.3.3.6.1.3" xref="A3.p2.2.m2.4.4.3.3.6.1.3.cmml"><mi id="A3.p2.2.m2.4.4.3.3.6.1.3.2" xref="A3.p2.2.m2.4.4.3.3.6.1.3.2.cmml">w</mi><mi id="A3.p2.2.m2.4.4.3.3.6.1.3.3" xref="A3.p2.2.m2.4.4.3.3.6.1.3.3.cmml">i</mi></msub></msub><mi class="ltx_font_mathcaligraphic" id="A3.p2.2.m2.4.4.3.3.6.2" xref="A3.p2.2.m2.4.4.3.3.6.2.cmml">ℒ</mi></mrow><mo lspace="0em" rspace="0em" id="A3.p2.2.m2.4.4.3.3.4a" xref="A3.p2.2.m2.4.4.3.3.4.cmml">​</mo><mrow id="A3.p2.2.m2.4.4.3.3.3.3" xref="A3.p2.2.m2.4.4.3.3.3.4.cmml"><mo stretchy="false" id="A3.p2.2.m2.4.4.3.3.3.3.4" xref="A3.p2.2.m2.4.4.3.3.3.4.cmml">(</mo><msub id="A3.p2.2.m2.2.2.1.1.1.1.1" xref="A3.p2.2.m2.2.2.1.1.1.1.1.cmml"><mi id="A3.p2.2.m2.2.2.1.1.1.1.1.2" xref="A3.p2.2.m2.2.2.1.1.1.1.1.2.cmml">w</mi><mn id="A3.p2.2.m2.2.2.1.1.1.1.1.3" xref="A3.p2.2.m2.2.2.1.1.1.1.1.3.cmml">1</mn></msub><mo id="A3.p2.2.m2.4.4.3.3.3.3.5" xref="A3.p2.2.m2.4.4.3.3.3.4.cmml">,</mo><msub id="A3.p2.2.m2.3.3.2.2.2.2.2" xref="A3.p2.2.m2.3.3.2.2.2.2.2.cmml"><mi id="A3.p2.2.m2.3.3.2.2.2.2.2.2" xref="A3.p2.2.m2.3.3.2.2.2.2.2.2.cmml">w</mi><mn id="A3.p2.2.m2.3.3.2.2.2.2.2.3" xref="A3.p2.2.m2.3.3.2.2.2.2.2.3.cmml">2</mn></msub><mo id="A3.p2.2.m2.4.4.3.3.3.3.6" xref="A3.p2.2.m2.4.4.3.3.3.4.cmml">,</mo><mi mathvariant="normal" id="A3.p2.2.m2.1.1" xref="A3.p2.2.m2.1.1.cmml">…</mi><mo id="A3.p2.2.m2.4.4.3.3.3.3.7" xref="A3.p2.2.m2.4.4.3.3.3.4.cmml">,</mo><msub id="A3.p2.2.m2.4.4.3.3.3.3.3" xref="A3.p2.2.m2.4.4.3.3.3.3.3.cmml"><mi id="A3.p2.2.m2.4.4.3.3.3.3.3.2" xref="A3.p2.2.m2.4.4.3.3.3.3.3.2.cmml">w</mi><mi id="A3.p2.2.m2.4.4.3.3.3.3.3.3" xref="A3.p2.2.m2.4.4.3.3.3.3.3.3.cmml">n</mi></msub><mo stretchy="false" id="A3.p2.2.m2.4.4.3.3.3.3.8" xref="A3.p2.2.m2.4.4.3.3.3.4.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.p2.2.m2.4b"><apply id="A3.p2.2.m2.4.4.cmml" xref="A3.p2.2.m2.4.4"><eq id="A3.p2.2.m2.4.4.4.cmml" xref="A3.p2.2.m2.4.4.4"></eq><apply id="A3.p2.2.m2.4.4.5.cmml" xref="A3.p2.2.m2.4.4.5"><csymbol cd="ambiguous" id="A3.p2.2.m2.4.4.5.1.cmml" xref="A3.p2.2.m2.4.4.5">subscript</csymbol><ci id="A3.p2.2.m2.4.4.5.2.cmml" xref="A3.p2.2.m2.4.4.5.2">𝑤</ci><ci id="A3.p2.2.m2.4.4.5.3.cmml" xref="A3.p2.2.m2.4.4.5.3">𝑖</ci></apply><apply id="A3.p2.2.m2.4.4.3.cmml" xref="A3.p2.2.m2.4.4.3"><minus id="A3.p2.2.m2.4.4.3.4.cmml" xref="A3.p2.2.m2.4.4.3.4"></minus><apply id="A3.p2.2.m2.4.4.3.5.cmml" xref="A3.p2.2.m2.4.4.3.5"><csymbol cd="ambiguous" id="A3.p2.2.m2.4.4.3.5.1.cmml" xref="A3.p2.2.m2.4.4.3.5">subscript</csymbol><ci id="A3.p2.2.m2.4.4.3.5.2.cmml" xref="A3.p2.2.m2.4.4.3.5.2">𝑤</ci><apply id="A3.p2.2.m2.4.4.3.5.3.cmml" xref="A3.p2.2.m2.4.4.3.5.3"><minus id="A3.p2.2.m2.4.4.3.5.3.1.cmml" xref="A3.p2.2.m2.4.4.3.5.3.1"></minus><ci id="A3.p2.2.m2.4.4.3.5.3.2.cmml" xref="A3.p2.2.m2.4.4.3.5.3.2">𝑖</ci><cn type="integer" id="A3.p2.2.m2.4.4.3.5.3.3.cmml" xref="A3.p2.2.m2.4.4.3.5.3.3">1</cn></apply></apply><apply id="A3.p2.2.m2.4.4.3.3.cmml" xref="A3.p2.2.m2.4.4.3.3"><times id="A3.p2.2.m2.4.4.3.3.4.cmml" xref="A3.p2.2.m2.4.4.3.3.4"></times><ci id="A3.p2.2.m2.4.4.3.3.5.cmml" xref="A3.p2.2.m2.4.4.3.3.5">𝜂</ci><apply id="A3.p2.2.m2.4.4.3.3.6.cmml" xref="A3.p2.2.m2.4.4.3.3.6"><apply id="A3.p2.2.m2.4.4.3.3.6.1.cmml" xref="A3.p2.2.m2.4.4.3.3.6.1"><csymbol cd="ambiguous" id="A3.p2.2.m2.4.4.3.3.6.1.1.cmml" xref="A3.p2.2.m2.4.4.3.3.6.1">subscript</csymbol><ci id="A3.p2.2.m2.4.4.3.3.6.1.2.cmml" xref="A3.p2.2.m2.4.4.3.3.6.1.2">∇</ci><apply id="A3.p2.2.m2.4.4.3.3.6.1.3.cmml" xref="A3.p2.2.m2.4.4.3.3.6.1.3"><csymbol cd="ambiguous" id="A3.p2.2.m2.4.4.3.3.6.1.3.1.cmml" xref="A3.p2.2.m2.4.4.3.3.6.1.3">subscript</csymbol><ci id="A3.p2.2.m2.4.4.3.3.6.1.3.2.cmml" xref="A3.p2.2.m2.4.4.3.3.6.1.3.2">𝑤</ci><ci id="A3.p2.2.m2.4.4.3.3.6.1.3.3.cmml" xref="A3.p2.2.m2.4.4.3.3.6.1.3.3">𝑖</ci></apply></apply><ci id="A3.p2.2.m2.4.4.3.3.6.2.cmml" xref="A3.p2.2.m2.4.4.3.3.6.2">ℒ</ci></apply><vector id="A3.p2.2.m2.4.4.3.3.3.4.cmml" xref="A3.p2.2.m2.4.4.3.3.3.3"><apply id="A3.p2.2.m2.2.2.1.1.1.1.1.cmml" xref="A3.p2.2.m2.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="A3.p2.2.m2.2.2.1.1.1.1.1.1.cmml" xref="A3.p2.2.m2.2.2.1.1.1.1.1">subscript</csymbol><ci id="A3.p2.2.m2.2.2.1.1.1.1.1.2.cmml" xref="A3.p2.2.m2.2.2.1.1.1.1.1.2">𝑤</ci><cn type="integer" id="A3.p2.2.m2.2.2.1.1.1.1.1.3.cmml" xref="A3.p2.2.m2.2.2.1.1.1.1.1.3">1</cn></apply><apply id="A3.p2.2.m2.3.3.2.2.2.2.2.cmml" xref="A3.p2.2.m2.3.3.2.2.2.2.2"><csymbol cd="ambiguous" id="A3.p2.2.m2.3.3.2.2.2.2.2.1.cmml" xref="A3.p2.2.m2.3.3.2.2.2.2.2">subscript</csymbol><ci id="A3.p2.2.m2.3.3.2.2.2.2.2.2.cmml" xref="A3.p2.2.m2.3.3.2.2.2.2.2.2">𝑤</ci><cn type="integer" id="A3.p2.2.m2.3.3.2.2.2.2.2.3.cmml" xref="A3.p2.2.m2.3.3.2.2.2.2.2.3">2</cn></apply><ci id="A3.p2.2.m2.1.1.cmml" xref="A3.p2.2.m2.1.1">…</ci><apply id="A3.p2.2.m2.4.4.3.3.3.3.3.cmml" xref="A3.p2.2.m2.4.4.3.3.3.3.3"><csymbol cd="ambiguous" id="A3.p2.2.m2.4.4.3.3.3.3.3.1.cmml" xref="A3.p2.2.m2.4.4.3.3.3.3.3">subscript</csymbol><ci id="A3.p2.2.m2.4.4.3.3.3.3.3.2.cmml" xref="A3.p2.2.m2.4.4.3.3.3.3.3.2">𝑤</ci><ci id="A3.p2.2.m2.4.4.3.3.3.3.3.3.cmml" xref="A3.p2.2.m2.4.4.3.3.3.3.3.3">𝑛</ci></apply></vector></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p2.2.m2.4c">w_{i}=w_{i-1}-\eta\nabla_{w_{i}}\mathcal{L}(w_{1},w_{2},\ldots,w_{n})</annotation></semantics></math>, where <math id="A3.p2.3.m3.4" class="ltx_Math" alttext="\nabla_{w_{i}}\mathcal{L}(w_{1},w_{2},\ldots,w_{n})" display="inline"><semantics id="A3.p2.3.m3.4a"><mrow id="A3.p2.3.m3.4.4" xref="A3.p2.3.m3.4.4.cmml"><mrow id="A3.p2.3.m3.4.4.5" xref="A3.p2.3.m3.4.4.5.cmml"><msub id="A3.p2.3.m3.4.4.5.1" xref="A3.p2.3.m3.4.4.5.1.cmml"><mo id="A3.p2.3.m3.4.4.5.1.2" xref="A3.p2.3.m3.4.4.5.1.2.cmml">∇</mo><msub id="A3.p2.3.m3.4.4.5.1.3" xref="A3.p2.3.m3.4.4.5.1.3.cmml"><mi id="A3.p2.3.m3.4.4.5.1.3.2" xref="A3.p2.3.m3.4.4.5.1.3.2.cmml">w</mi><mi id="A3.p2.3.m3.4.4.5.1.3.3" xref="A3.p2.3.m3.4.4.5.1.3.3.cmml">i</mi></msub></msub><mi class="ltx_font_mathcaligraphic" id="A3.p2.3.m3.4.4.5.2" xref="A3.p2.3.m3.4.4.5.2.cmml">ℒ</mi></mrow><mo lspace="0em" rspace="0em" id="A3.p2.3.m3.4.4.4" xref="A3.p2.3.m3.4.4.4.cmml">​</mo><mrow id="A3.p2.3.m3.4.4.3.3" xref="A3.p2.3.m3.4.4.3.4.cmml"><mo stretchy="false" id="A3.p2.3.m3.4.4.3.3.4" xref="A3.p2.3.m3.4.4.3.4.cmml">(</mo><msub id="A3.p2.3.m3.2.2.1.1.1" xref="A3.p2.3.m3.2.2.1.1.1.cmml"><mi id="A3.p2.3.m3.2.2.1.1.1.2" xref="A3.p2.3.m3.2.2.1.1.1.2.cmml">w</mi><mn id="A3.p2.3.m3.2.2.1.1.1.3" xref="A3.p2.3.m3.2.2.1.1.1.3.cmml">1</mn></msub><mo id="A3.p2.3.m3.4.4.3.3.5" xref="A3.p2.3.m3.4.4.3.4.cmml">,</mo><msub id="A3.p2.3.m3.3.3.2.2.2" xref="A3.p2.3.m3.3.3.2.2.2.cmml"><mi id="A3.p2.3.m3.3.3.2.2.2.2" xref="A3.p2.3.m3.3.3.2.2.2.2.cmml">w</mi><mn id="A3.p2.3.m3.3.3.2.2.2.3" xref="A3.p2.3.m3.3.3.2.2.2.3.cmml">2</mn></msub><mo id="A3.p2.3.m3.4.4.3.3.6" xref="A3.p2.3.m3.4.4.3.4.cmml">,</mo><mi mathvariant="normal" id="A3.p2.3.m3.1.1" xref="A3.p2.3.m3.1.1.cmml">…</mi><mo id="A3.p2.3.m3.4.4.3.3.7" xref="A3.p2.3.m3.4.4.3.4.cmml">,</mo><msub id="A3.p2.3.m3.4.4.3.3.3" xref="A3.p2.3.m3.4.4.3.3.3.cmml"><mi id="A3.p2.3.m3.4.4.3.3.3.2" xref="A3.p2.3.m3.4.4.3.3.3.2.cmml">w</mi><mi id="A3.p2.3.m3.4.4.3.3.3.3" xref="A3.p2.3.m3.4.4.3.3.3.3.cmml">n</mi></msub><mo stretchy="false" id="A3.p2.3.m3.4.4.3.3.8" xref="A3.p2.3.m3.4.4.3.4.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.p2.3.m3.4b"><apply id="A3.p2.3.m3.4.4.cmml" xref="A3.p2.3.m3.4.4"><times id="A3.p2.3.m3.4.4.4.cmml" xref="A3.p2.3.m3.4.4.4"></times><apply id="A3.p2.3.m3.4.4.5.cmml" xref="A3.p2.3.m3.4.4.5"><apply id="A3.p2.3.m3.4.4.5.1.cmml" xref="A3.p2.3.m3.4.4.5.1"><csymbol cd="ambiguous" id="A3.p2.3.m3.4.4.5.1.1.cmml" xref="A3.p2.3.m3.4.4.5.1">subscript</csymbol><ci id="A3.p2.3.m3.4.4.5.1.2.cmml" xref="A3.p2.3.m3.4.4.5.1.2">∇</ci><apply id="A3.p2.3.m3.4.4.5.1.3.cmml" xref="A3.p2.3.m3.4.4.5.1.3"><csymbol cd="ambiguous" id="A3.p2.3.m3.4.4.5.1.3.1.cmml" xref="A3.p2.3.m3.4.4.5.1.3">subscript</csymbol><ci id="A3.p2.3.m3.4.4.5.1.3.2.cmml" xref="A3.p2.3.m3.4.4.5.1.3.2">𝑤</ci><ci id="A3.p2.3.m3.4.4.5.1.3.3.cmml" xref="A3.p2.3.m3.4.4.5.1.3.3">𝑖</ci></apply></apply><ci id="A3.p2.3.m3.4.4.5.2.cmml" xref="A3.p2.3.m3.4.4.5.2">ℒ</ci></apply><vector id="A3.p2.3.m3.4.4.3.4.cmml" xref="A3.p2.3.m3.4.4.3.3"><apply id="A3.p2.3.m3.2.2.1.1.1.cmml" xref="A3.p2.3.m3.2.2.1.1.1"><csymbol cd="ambiguous" id="A3.p2.3.m3.2.2.1.1.1.1.cmml" xref="A3.p2.3.m3.2.2.1.1.1">subscript</csymbol><ci id="A3.p2.3.m3.2.2.1.1.1.2.cmml" xref="A3.p2.3.m3.2.2.1.1.1.2">𝑤</ci><cn type="integer" id="A3.p2.3.m3.2.2.1.1.1.3.cmml" xref="A3.p2.3.m3.2.2.1.1.1.3">1</cn></apply><apply id="A3.p2.3.m3.3.3.2.2.2.cmml" xref="A3.p2.3.m3.3.3.2.2.2"><csymbol cd="ambiguous" id="A3.p2.3.m3.3.3.2.2.2.1.cmml" xref="A3.p2.3.m3.3.3.2.2.2">subscript</csymbol><ci id="A3.p2.3.m3.3.3.2.2.2.2.cmml" xref="A3.p2.3.m3.3.3.2.2.2.2">𝑤</ci><cn type="integer" id="A3.p2.3.m3.3.3.2.2.2.3.cmml" xref="A3.p2.3.m3.3.3.2.2.2.3">2</cn></apply><ci id="A3.p2.3.m3.1.1.cmml" xref="A3.p2.3.m3.1.1">…</ci><apply id="A3.p2.3.m3.4.4.3.3.3.cmml" xref="A3.p2.3.m3.4.4.3.3.3"><csymbol cd="ambiguous" id="A3.p2.3.m3.4.4.3.3.3.1.cmml" xref="A3.p2.3.m3.4.4.3.3.3">subscript</csymbol><ci id="A3.p2.3.m3.4.4.3.3.3.2.cmml" xref="A3.p2.3.m3.4.4.3.3.3.2">𝑤</ci><ci id="A3.p2.3.m3.4.4.3.3.3.3.cmml" xref="A3.p2.3.m3.4.4.3.3.3.3">𝑛</ci></apply></vector></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p2.3.m3.4c">\nabla_{w_{i}}\mathcal{L}(w_{1},w_{2},\ldots,w_{n})</annotation></semantics></math> is input dependent. As an example, let’s consider a two-layer neural network with feedforward operations as follows;</p>
<table id="A8.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A3.Ex1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A3.Ex1.m1.1" class="ltx_Math" alttext="\displaystyle x" display="inline"><semantics id="A3.Ex1.m1.1a"><mi id="A3.Ex1.m1.1.1" xref="A3.Ex1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="A3.Ex1.m1.1b"><ci id="A3.Ex1.m1.1.1.cmml" xref="A3.Ex1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex1.m1.1c">\displaystyle x</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A3.Ex1.m2.1" class="ltx_Math" alttext="\displaystyle:inputs" display="inline"><semantics id="A3.Ex1.m2.1a"><mrow id="A3.Ex1.m2.1.1" xref="A3.Ex1.m2.1.1.cmml"><mi id="A3.Ex1.m2.1.1.2" xref="A3.Ex1.m2.1.1.2.cmml"></mi><mo lspace="0.278em" rspace="0.278em" id="A3.Ex1.m2.1.1.1" xref="A3.Ex1.m2.1.1.1.cmml">:</mo><mrow id="A3.Ex1.m2.1.1.3" xref="A3.Ex1.m2.1.1.3.cmml"><mi id="A3.Ex1.m2.1.1.3.2" xref="A3.Ex1.m2.1.1.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="A3.Ex1.m2.1.1.3.1" xref="A3.Ex1.m2.1.1.3.1.cmml">​</mo><mi id="A3.Ex1.m2.1.1.3.3" xref="A3.Ex1.m2.1.1.3.3.cmml">n</mi><mo lspace="0em" rspace="0em" id="A3.Ex1.m2.1.1.3.1a" xref="A3.Ex1.m2.1.1.3.1.cmml">​</mo><mi id="A3.Ex1.m2.1.1.3.4" xref="A3.Ex1.m2.1.1.3.4.cmml">p</mi><mo lspace="0em" rspace="0em" id="A3.Ex1.m2.1.1.3.1b" xref="A3.Ex1.m2.1.1.3.1.cmml">​</mo><mi id="A3.Ex1.m2.1.1.3.5" xref="A3.Ex1.m2.1.1.3.5.cmml">u</mi><mo lspace="0em" rspace="0em" id="A3.Ex1.m2.1.1.3.1c" xref="A3.Ex1.m2.1.1.3.1.cmml">​</mo><mi id="A3.Ex1.m2.1.1.3.6" xref="A3.Ex1.m2.1.1.3.6.cmml">t</mi><mo lspace="0em" rspace="0em" id="A3.Ex1.m2.1.1.3.1d" xref="A3.Ex1.m2.1.1.3.1.cmml">​</mo><mi id="A3.Ex1.m2.1.1.3.7" xref="A3.Ex1.m2.1.1.3.7.cmml">s</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.Ex1.m2.1b"><apply id="A3.Ex1.m2.1.1.cmml" xref="A3.Ex1.m2.1.1"><ci id="A3.Ex1.m2.1.1.1.cmml" xref="A3.Ex1.m2.1.1.1">:</ci><csymbol cd="latexml" id="A3.Ex1.m2.1.1.2.cmml" xref="A3.Ex1.m2.1.1.2">absent</csymbol><apply id="A3.Ex1.m2.1.1.3.cmml" xref="A3.Ex1.m2.1.1.3"><times id="A3.Ex1.m2.1.1.3.1.cmml" xref="A3.Ex1.m2.1.1.3.1"></times><ci id="A3.Ex1.m2.1.1.3.2.cmml" xref="A3.Ex1.m2.1.1.3.2">𝑖</ci><ci id="A3.Ex1.m2.1.1.3.3.cmml" xref="A3.Ex1.m2.1.1.3.3">𝑛</ci><ci id="A3.Ex1.m2.1.1.3.4.cmml" xref="A3.Ex1.m2.1.1.3.4">𝑝</ci><ci id="A3.Ex1.m2.1.1.3.5.cmml" xref="A3.Ex1.m2.1.1.3.5">𝑢</ci><ci id="A3.Ex1.m2.1.1.3.6.cmml" xref="A3.Ex1.m2.1.1.3.6">𝑡</ci><ci id="A3.Ex1.m2.1.1.3.7.cmml" xref="A3.Ex1.m2.1.1.3.7">𝑠</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex1.m2.1c">\displaystyle:inputs</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright" colspan="3"></td>
</tr></tbody>
<tbody id="A3.Ex2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A3.Ex2.m1.1" class="ltx_Math" alttext="\displaystyle l_{1}" display="inline"><semantics id="A3.Ex2.m1.1a"><msub id="A3.Ex2.m1.1.1" xref="A3.Ex2.m1.1.1.cmml"><mi id="A3.Ex2.m1.1.1.2" xref="A3.Ex2.m1.1.1.2.cmml">l</mi><mn id="A3.Ex2.m1.1.1.3" xref="A3.Ex2.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A3.Ex2.m1.1b"><apply id="A3.Ex2.m1.1.1.cmml" xref="A3.Ex2.m1.1.1"><csymbol cd="ambiguous" id="A3.Ex2.m1.1.1.1.cmml" xref="A3.Ex2.m1.1.1">subscript</csymbol><ci id="A3.Ex2.m1.1.1.2.cmml" xref="A3.Ex2.m1.1.1.2">𝑙</ci><cn type="integer" id="A3.Ex2.m1.1.1.3.cmml" xref="A3.Ex2.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex2.m1.1c">\displaystyle l_{1}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A3.Ex2.m2.1" class="ltx_Math" alttext="\displaystyle=W_{1}x+b_{1}" display="inline"><semantics id="A3.Ex2.m2.1a"><mrow id="A3.Ex2.m2.1.1" xref="A3.Ex2.m2.1.1.cmml"><mi id="A3.Ex2.m2.1.1.2" xref="A3.Ex2.m2.1.1.2.cmml"></mi><mo id="A3.Ex2.m2.1.1.1" xref="A3.Ex2.m2.1.1.1.cmml">=</mo><mrow id="A3.Ex2.m2.1.1.3" xref="A3.Ex2.m2.1.1.3.cmml"><mrow id="A3.Ex2.m2.1.1.3.2" xref="A3.Ex2.m2.1.1.3.2.cmml"><msub id="A3.Ex2.m2.1.1.3.2.2" xref="A3.Ex2.m2.1.1.3.2.2.cmml"><mi id="A3.Ex2.m2.1.1.3.2.2.2" xref="A3.Ex2.m2.1.1.3.2.2.2.cmml">W</mi><mn id="A3.Ex2.m2.1.1.3.2.2.3" xref="A3.Ex2.m2.1.1.3.2.2.3.cmml">1</mn></msub><mo lspace="0em" rspace="0em" id="A3.Ex2.m2.1.1.3.2.1" xref="A3.Ex2.m2.1.1.3.2.1.cmml">​</mo><mi id="A3.Ex2.m2.1.1.3.2.3" xref="A3.Ex2.m2.1.1.3.2.3.cmml">x</mi></mrow><mo id="A3.Ex2.m2.1.1.3.1" xref="A3.Ex2.m2.1.1.3.1.cmml">+</mo><msub id="A3.Ex2.m2.1.1.3.3" xref="A3.Ex2.m2.1.1.3.3.cmml"><mi id="A3.Ex2.m2.1.1.3.3.2" xref="A3.Ex2.m2.1.1.3.3.2.cmml">b</mi><mn id="A3.Ex2.m2.1.1.3.3.3" xref="A3.Ex2.m2.1.1.3.3.3.cmml">1</mn></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.Ex2.m2.1b"><apply id="A3.Ex2.m2.1.1.cmml" xref="A3.Ex2.m2.1.1"><eq id="A3.Ex2.m2.1.1.1.cmml" xref="A3.Ex2.m2.1.1.1"></eq><csymbol cd="latexml" id="A3.Ex2.m2.1.1.2.cmml" xref="A3.Ex2.m2.1.1.2">absent</csymbol><apply id="A3.Ex2.m2.1.1.3.cmml" xref="A3.Ex2.m2.1.1.3"><plus id="A3.Ex2.m2.1.1.3.1.cmml" xref="A3.Ex2.m2.1.1.3.1"></plus><apply id="A3.Ex2.m2.1.1.3.2.cmml" xref="A3.Ex2.m2.1.1.3.2"><times id="A3.Ex2.m2.1.1.3.2.1.cmml" xref="A3.Ex2.m2.1.1.3.2.1"></times><apply id="A3.Ex2.m2.1.1.3.2.2.cmml" xref="A3.Ex2.m2.1.1.3.2.2"><csymbol cd="ambiguous" id="A3.Ex2.m2.1.1.3.2.2.1.cmml" xref="A3.Ex2.m2.1.1.3.2.2">subscript</csymbol><ci id="A3.Ex2.m2.1.1.3.2.2.2.cmml" xref="A3.Ex2.m2.1.1.3.2.2.2">𝑊</ci><cn type="integer" id="A3.Ex2.m2.1.1.3.2.2.3.cmml" xref="A3.Ex2.m2.1.1.3.2.2.3">1</cn></apply><ci id="A3.Ex2.m2.1.1.3.2.3.cmml" xref="A3.Ex2.m2.1.1.3.2.3">𝑥</ci></apply><apply id="A3.Ex2.m2.1.1.3.3.cmml" xref="A3.Ex2.m2.1.1.3.3"><csymbol cd="ambiguous" id="A3.Ex2.m2.1.1.3.3.1.cmml" xref="A3.Ex2.m2.1.1.3.3">subscript</csymbol><ci id="A3.Ex2.m2.1.1.3.3.2.cmml" xref="A3.Ex2.m2.1.1.3.3.2">𝑏</ci><cn type="integer" id="A3.Ex2.m2.1.1.3.3.3.cmml" xref="A3.Ex2.m2.1.1.3.3.3">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex2.m2.1c">\displaystyle=W_{1}x+b_{1}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A3.Ex2.m3.1" class="ltx_Math" alttext="\displaystyle h" display="inline"><semantics id="A3.Ex2.m3.1a"><mi id="A3.Ex2.m3.1.1" xref="A3.Ex2.m3.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="A3.Ex2.m3.1b"><ci id="A3.Ex2.m3.1.1.cmml" xref="A3.Ex2.m3.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex2.m3.1c">\displaystyle h</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A3.Ex2.m4.1" class="ltx_Math" alttext="\displaystyle=ReLU(l_{1})" display="inline"><semantics id="A3.Ex2.m4.1a"><mrow id="A3.Ex2.m4.1.1" xref="A3.Ex2.m4.1.1.cmml"><mi id="A3.Ex2.m4.1.1.3" xref="A3.Ex2.m4.1.1.3.cmml"></mi><mo id="A3.Ex2.m4.1.1.2" xref="A3.Ex2.m4.1.1.2.cmml">=</mo><mrow id="A3.Ex2.m4.1.1.1" xref="A3.Ex2.m4.1.1.1.cmml"><mi id="A3.Ex2.m4.1.1.1.3" xref="A3.Ex2.m4.1.1.1.3.cmml">R</mi><mo lspace="0em" rspace="0em" id="A3.Ex2.m4.1.1.1.2" xref="A3.Ex2.m4.1.1.1.2.cmml">​</mo><mi id="A3.Ex2.m4.1.1.1.4" xref="A3.Ex2.m4.1.1.1.4.cmml">e</mi><mo lspace="0em" rspace="0em" id="A3.Ex2.m4.1.1.1.2a" xref="A3.Ex2.m4.1.1.1.2.cmml">​</mo><mi id="A3.Ex2.m4.1.1.1.5" xref="A3.Ex2.m4.1.1.1.5.cmml">L</mi><mo lspace="0em" rspace="0em" id="A3.Ex2.m4.1.1.1.2b" xref="A3.Ex2.m4.1.1.1.2.cmml">​</mo><mi id="A3.Ex2.m4.1.1.1.6" xref="A3.Ex2.m4.1.1.1.6.cmml">U</mi><mo lspace="0em" rspace="0em" id="A3.Ex2.m4.1.1.1.2c" xref="A3.Ex2.m4.1.1.1.2.cmml">​</mo><mrow id="A3.Ex2.m4.1.1.1.1.1" xref="A3.Ex2.m4.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A3.Ex2.m4.1.1.1.1.1.2" xref="A3.Ex2.m4.1.1.1.1.1.1.cmml">(</mo><msub id="A3.Ex2.m4.1.1.1.1.1.1" xref="A3.Ex2.m4.1.1.1.1.1.1.cmml"><mi id="A3.Ex2.m4.1.1.1.1.1.1.2" xref="A3.Ex2.m4.1.1.1.1.1.1.2.cmml">l</mi><mn id="A3.Ex2.m4.1.1.1.1.1.1.3" xref="A3.Ex2.m4.1.1.1.1.1.1.3.cmml">1</mn></msub><mo stretchy="false" id="A3.Ex2.m4.1.1.1.1.1.3" xref="A3.Ex2.m4.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.Ex2.m4.1b"><apply id="A3.Ex2.m4.1.1.cmml" xref="A3.Ex2.m4.1.1"><eq id="A3.Ex2.m4.1.1.2.cmml" xref="A3.Ex2.m4.1.1.2"></eq><csymbol cd="latexml" id="A3.Ex2.m4.1.1.3.cmml" xref="A3.Ex2.m4.1.1.3">absent</csymbol><apply id="A3.Ex2.m4.1.1.1.cmml" xref="A3.Ex2.m4.1.1.1"><times id="A3.Ex2.m4.1.1.1.2.cmml" xref="A3.Ex2.m4.1.1.1.2"></times><ci id="A3.Ex2.m4.1.1.1.3.cmml" xref="A3.Ex2.m4.1.1.1.3">𝑅</ci><ci id="A3.Ex2.m4.1.1.1.4.cmml" xref="A3.Ex2.m4.1.1.1.4">𝑒</ci><ci id="A3.Ex2.m4.1.1.1.5.cmml" xref="A3.Ex2.m4.1.1.1.5">𝐿</ci><ci id="A3.Ex2.m4.1.1.1.6.cmml" xref="A3.Ex2.m4.1.1.1.6">𝑈</ci><apply id="A3.Ex2.m4.1.1.1.1.1.1.cmml" xref="A3.Ex2.m4.1.1.1.1.1"><csymbol cd="ambiguous" id="A3.Ex2.m4.1.1.1.1.1.1.1.cmml" xref="A3.Ex2.m4.1.1.1.1.1">subscript</csymbol><ci id="A3.Ex2.m4.1.1.1.1.1.1.2.cmml" xref="A3.Ex2.m4.1.1.1.1.1.1.2">𝑙</ci><cn type="integer" id="A3.Ex2.m4.1.1.1.1.1.1.3.cmml" xref="A3.Ex2.m4.1.1.1.1.1.1.3">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex2.m4.1c">\displaystyle=ReLU(l_{1})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="A3.Ex3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A3.Ex3.m1.1" class="ltx_Math" alttext="\displaystyle h" display="inline"><semantics id="A3.Ex3.m1.1a"><mi id="A3.Ex3.m1.1.1" xref="A3.Ex3.m1.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="A3.Ex3.m1.1b"><ci id="A3.Ex3.m1.1.1.cmml" xref="A3.Ex3.m1.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex3.m1.1c">\displaystyle h</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A3.Ex3.m2.1" class="ltx_Math" alttext="\displaystyle=ReLU(l_{1})" display="inline"><semantics id="A3.Ex3.m2.1a"><mrow id="A3.Ex3.m2.1.1" xref="A3.Ex3.m2.1.1.cmml"><mi id="A3.Ex3.m2.1.1.3" xref="A3.Ex3.m2.1.1.3.cmml"></mi><mo id="A3.Ex3.m2.1.1.2" xref="A3.Ex3.m2.1.1.2.cmml">=</mo><mrow id="A3.Ex3.m2.1.1.1" xref="A3.Ex3.m2.1.1.1.cmml"><mi id="A3.Ex3.m2.1.1.1.3" xref="A3.Ex3.m2.1.1.1.3.cmml">R</mi><mo lspace="0em" rspace="0em" id="A3.Ex3.m2.1.1.1.2" xref="A3.Ex3.m2.1.1.1.2.cmml">​</mo><mi id="A3.Ex3.m2.1.1.1.4" xref="A3.Ex3.m2.1.1.1.4.cmml">e</mi><mo lspace="0em" rspace="0em" id="A3.Ex3.m2.1.1.1.2a" xref="A3.Ex3.m2.1.1.1.2.cmml">​</mo><mi id="A3.Ex3.m2.1.1.1.5" xref="A3.Ex3.m2.1.1.1.5.cmml">L</mi><mo lspace="0em" rspace="0em" id="A3.Ex3.m2.1.1.1.2b" xref="A3.Ex3.m2.1.1.1.2.cmml">​</mo><mi id="A3.Ex3.m2.1.1.1.6" xref="A3.Ex3.m2.1.1.1.6.cmml">U</mi><mo lspace="0em" rspace="0em" id="A3.Ex3.m2.1.1.1.2c" xref="A3.Ex3.m2.1.1.1.2.cmml">​</mo><mrow id="A3.Ex3.m2.1.1.1.1.1" xref="A3.Ex3.m2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A3.Ex3.m2.1.1.1.1.1.2" xref="A3.Ex3.m2.1.1.1.1.1.1.cmml">(</mo><msub id="A3.Ex3.m2.1.1.1.1.1.1" xref="A3.Ex3.m2.1.1.1.1.1.1.cmml"><mi id="A3.Ex3.m2.1.1.1.1.1.1.2" xref="A3.Ex3.m2.1.1.1.1.1.1.2.cmml">l</mi><mn id="A3.Ex3.m2.1.1.1.1.1.1.3" xref="A3.Ex3.m2.1.1.1.1.1.1.3.cmml">1</mn></msub><mo stretchy="false" id="A3.Ex3.m2.1.1.1.1.1.3" xref="A3.Ex3.m2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.Ex3.m2.1b"><apply id="A3.Ex3.m2.1.1.cmml" xref="A3.Ex3.m2.1.1"><eq id="A3.Ex3.m2.1.1.2.cmml" xref="A3.Ex3.m2.1.1.2"></eq><csymbol cd="latexml" id="A3.Ex3.m2.1.1.3.cmml" xref="A3.Ex3.m2.1.1.3">absent</csymbol><apply id="A3.Ex3.m2.1.1.1.cmml" xref="A3.Ex3.m2.1.1.1"><times id="A3.Ex3.m2.1.1.1.2.cmml" xref="A3.Ex3.m2.1.1.1.2"></times><ci id="A3.Ex3.m2.1.1.1.3.cmml" xref="A3.Ex3.m2.1.1.1.3">𝑅</ci><ci id="A3.Ex3.m2.1.1.1.4.cmml" xref="A3.Ex3.m2.1.1.1.4">𝑒</ci><ci id="A3.Ex3.m2.1.1.1.5.cmml" xref="A3.Ex3.m2.1.1.1.5">𝐿</ci><ci id="A3.Ex3.m2.1.1.1.6.cmml" xref="A3.Ex3.m2.1.1.1.6">𝑈</ci><apply id="A3.Ex3.m2.1.1.1.1.1.1.cmml" xref="A3.Ex3.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="A3.Ex3.m2.1.1.1.1.1.1.1.cmml" xref="A3.Ex3.m2.1.1.1.1.1">subscript</csymbol><ci id="A3.Ex3.m2.1.1.1.1.1.1.2.cmml" xref="A3.Ex3.m2.1.1.1.1.1.1.2">𝑙</ci><cn type="integer" id="A3.Ex3.m2.1.1.1.1.1.1.3.cmml" xref="A3.Ex3.m2.1.1.1.1.1.1.3">1</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex3.m2.1c">\displaystyle=ReLU(l_{1})</annotation></semantics></math></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A3.Ex3.m3.1" class="ltx_Math" alttext="\displaystyle l_{2}" display="inline"><semantics id="A3.Ex3.m3.1a"><msub id="A3.Ex3.m3.1.1" xref="A3.Ex3.m3.1.1.cmml"><mi id="A3.Ex3.m3.1.1.2" xref="A3.Ex3.m3.1.1.2.cmml">l</mi><mn id="A3.Ex3.m3.1.1.3" xref="A3.Ex3.m3.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A3.Ex3.m3.1b"><apply id="A3.Ex3.m3.1.1.cmml" xref="A3.Ex3.m3.1.1"><csymbol cd="ambiguous" id="A3.Ex3.m3.1.1.1.cmml" xref="A3.Ex3.m3.1.1">subscript</csymbol><ci id="A3.Ex3.m3.1.1.2.cmml" xref="A3.Ex3.m3.1.1.2">𝑙</ci><cn type="integer" id="A3.Ex3.m3.1.1.3.cmml" xref="A3.Ex3.m3.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex3.m3.1c">\displaystyle l_{2}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A3.Ex3.m4.1" class="ltx_Math" alttext="\displaystyle=W_{2}h+b_{2}" display="inline"><semantics id="A3.Ex3.m4.1a"><mrow id="A3.Ex3.m4.1.1" xref="A3.Ex3.m4.1.1.cmml"><mi id="A3.Ex3.m4.1.1.2" xref="A3.Ex3.m4.1.1.2.cmml"></mi><mo id="A3.Ex3.m4.1.1.1" xref="A3.Ex3.m4.1.1.1.cmml">=</mo><mrow id="A3.Ex3.m4.1.1.3" xref="A3.Ex3.m4.1.1.3.cmml"><mrow id="A3.Ex3.m4.1.1.3.2" xref="A3.Ex3.m4.1.1.3.2.cmml"><msub id="A3.Ex3.m4.1.1.3.2.2" xref="A3.Ex3.m4.1.1.3.2.2.cmml"><mi id="A3.Ex3.m4.1.1.3.2.2.2" xref="A3.Ex3.m4.1.1.3.2.2.2.cmml">W</mi><mn id="A3.Ex3.m4.1.1.3.2.2.3" xref="A3.Ex3.m4.1.1.3.2.2.3.cmml">2</mn></msub><mo lspace="0em" rspace="0em" id="A3.Ex3.m4.1.1.3.2.1" xref="A3.Ex3.m4.1.1.3.2.1.cmml">​</mo><mi id="A3.Ex3.m4.1.1.3.2.3" xref="A3.Ex3.m4.1.1.3.2.3.cmml">h</mi></mrow><mo id="A3.Ex3.m4.1.1.3.1" xref="A3.Ex3.m4.1.1.3.1.cmml">+</mo><msub id="A3.Ex3.m4.1.1.3.3" xref="A3.Ex3.m4.1.1.3.3.cmml"><mi id="A3.Ex3.m4.1.1.3.3.2" xref="A3.Ex3.m4.1.1.3.3.2.cmml">b</mi><mn id="A3.Ex3.m4.1.1.3.3.3" xref="A3.Ex3.m4.1.1.3.3.3.cmml">2</mn></msub></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.Ex3.m4.1b"><apply id="A3.Ex3.m4.1.1.cmml" xref="A3.Ex3.m4.1.1"><eq id="A3.Ex3.m4.1.1.1.cmml" xref="A3.Ex3.m4.1.1.1"></eq><csymbol cd="latexml" id="A3.Ex3.m4.1.1.2.cmml" xref="A3.Ex3.m4.1.1.2">absent</csymbol><apply id="A3.Ex3.m4.1.1.3.cmml" xref="A3.Ex3.m4.1.1.3"><plus id="A3.Ex3.m4.1.1.3.1.cmml" xref="A3.Ex3.m4.1.1.3.1"></plus><apply id="A3.Ex3.m4.1.1.3.2.cmml" xref="A3.Ex3.m4.1.1.3.2"><times id="A3.Ex3.m4.1.1.3.2.1.cmml" xref="A3.Ex3.m4.1.1.3.2.1"></times><apply id="A3.Ex3.m4.1.1.3.2.2.cmml" xref="A3.Ex3.m4.1.1.3.2.2"><csymbol cd="ambiguous" id="A3.Ex3.m4.1.1.3.2.2.1.cmml" xref="A3.Ex3.m4.1.1.3.2.2">subscript</csymbol><ci id="A3.Ex3.m4.1.1.3.2.2.2.cmml" xref="A3.Ex3.m4.1.1.3.2.2.2">𝑊</ci><cn type="integer" id="A3.Ex3.m4.1.1.3.2.2.3.cmml" xref="A3.Ex3.m4.1.1.3.2.2.3">2</cn></apply><ci id="A3.Ex3.m4.1.1.3.2.3.cmml" xref="A3.Ex3.m4.1.1.3.2.3">ℎ</ci></apply><apply id="A3.Ex3.m4.1.1.3.3.cmml" xref="A3.Ex3.m4.1.1.3.3"><csymbol cd="ambiguous" id="A3.Ex3.m4.1.1.3.3.1.cmml" xref="A3.Ex3.m4.1.1.3.3">subscript</csymbol><ci id="A3.Ex3.m4.1.1.3.3.2.cmml" xref="A3.Ex3.m4.1.1.3.3.2">𝑏</ci><cn type="integer" id="A3.Ex3.m4.1.1.3.3.3.cmml" xref="A3.Ex3.m4.1.1.3.3.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex3.m4.1c">\displaystyle=W_{2}h+b_{2}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="A3.Ex4"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A3.Ex4.m1.1" class="ltx_Math" alttext="\displaystyle\hat{y}" display="inline"><semantics id="A3.Ex4.m1.1a"><mover accent="true" id="A3.Ex4.m1.1.1" xref="A3.Ex4.m1.1.1.cmml"><mi id="A3.Ex4.m1.1.1.2" xref="A3.Ex4.m1.1.1.2.cmml">y</mi><mo id="A3.Ex4.m1.1.1.1" xref="A3.Ex4.m1.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="A3.Ex4.m1.1b"><apply id="A3.Ex4.m1.1.1.cmml" xref="A3.Ex4.m1.1.1"><ci id="A3.Ex4.m1.1.1.1.cmml" xref="A3.Ex4.m1.1.1.1">^</ci><ci id="A3.Ex4.m1.1.1.2.cmml" xref="A3.Ex4.m1.1.1.2">𝑦</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex4.m1.1c">\displaystyle\hat{y}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A3.Ex4.m2.1" class="ltx_Math" alttext="\displaystyle=softmax(l_{2})" display="inline"><semantics id="A3.Ex4.m2.1a"><mrow id="A3.Ex4.m2.1.1" xref="A3.Ex4.m2.1.1.cmml"><mi id="A3.Ex4.m2.1.1.3" xref="A3.Ex4.m2.1.1.3.cmml"></mi><mo id="A3.Ex4.m2.1.1.2" xref="A3.Ex4.m2.1.1.2.cmml">=</mo><mrow id="A3.Ex4.m2.1.1.1" xref="A3.Ex4.m2.1.1.1.cmml"><mi id="A3.Ex4.m2.1.1.1.3" xref="A3.Ex4.m2.1.1.1.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="A3.Ex4.m2.1.1.1.2" xref="A3.Ex4.m2.1.1.1.2.cmml">​</mo><mi id="A3.Ex4.m2.1.1.1.4" xref="A3.Ex4.m2.1.1.1.4.cmml">o</mi><mo lspace="0em" rspace="0em" id="A3.Ex4.m2.1.1.1.2a" xref="A3.Ex4.m2.1.1.1.2.cmml">​</mo><mi id="A3.Ex4.m2.1.1.1.5" xref="A3.Ex4.m2.1.1.1.5.cmml">f</mi><mo lspace="0em" rspace="0em" id="A3.Ex4.m2.1.1.1.2b" xref="A3.Ex4.m2.1.1.1.2.cmml">​</mo><mi id="A3.Ex4.m2.1.1.1.6" xref="A3.Ex4.m2.1.1.1.6.cmml">t</mi><mo lspace="0em" rspace="0em" id="A3.Ex4.m2.1.1.1.2c" xref="A3.Ex4.m2.1.1.1.2.cmml">​</mo><mi id="A3.Ex4.m2.1.1.1.7" xref="A3.Ex4.m2.1.1.1.7.cmml">m</mi><mo lspace="0em" rspace="0em" id="A3.Ex4.m2.1.1.1.2d" xref="A3.Ex4.m2.1.1.1.2.cmml">​</mo><mi id="A3.Ex4.m2.1.1.1.8" xref="A3.Ex4.m2.1.1.1.8.cmml">a</mi><mo lspace="0em" rspace="0em" id="A3.Ex4.m2.1.1.1.2e" xref="A3.Ex4.m2.1.1.1.2.cmml">​</mo><mi id="A3.Ex4.m2.1.1.1.9" xref="A3.Ex4.m2.1.1.1.9.cmml">x</mi><mo lspace="0em" rspace="0em" id="A3.Ex4.m2.1.1.1.2f" xref="A3.Ex4.m2.1.1.1.2.cmml">​</mo><mrow id="A3.Ex4.m2.1.1.1.1.1" xref="A3.Ex4.m2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A3.Ex4.m2.1.1.1.1.1.2" xref="A3.Ex4.m2.1.1.1.1.1.1.cmml">(</mo><msub id="A3.Ex4.m2.1.1.1.1.1.1" xref="A3.Ex4.m2.1.1.1.1.1.1.cmml"><mi id="A3.Ex4.m2.1.1.1.1.1.1.2" xref="A3.Ex4.m2.1.1.1.1.1.1.2.cmml">l</mi><mn id="A3.Ex4.m2.1.1.1.1.1.1.3" xref="A3.Ex4.m2.1.1.1.1.1.1.3.cmml">2</mn></msub><mo stretchy="false" id="A3.Ex4.m2.1.1.1.1.1.3" xref="A3.Ex4.m2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.Ex4.m2.1b"><apply id="A3.Ex4.m2.1.1.cmml" xref="A3.Ex4.m2.1.1"><eq id="A3.Ex4.m2.1.1.2.cmml" xref="A3.Ex4.m2.1.1.2"></eq><csymbol cd="latexml" id="A3.Ex4.m2.1.1.3.cmml" xref="A3.Ex4.m2.1.1.3">absent</csymbol><apply id="A3.Ex4.m2.1.1.1.cmml" xref="A3.Ex4.m2.1.1.1"><times id="A3.Ex4.m2.1.1.1.2.cmml" xref="A3.Ex4.m2.1.1.1.2"></times><ci id="A3.Ex4.m2.1.1.1.3.cmml" xref="A3.Ex4.m2.1.1.1.3">𝑠</ci><ci id="A3.Ex4.m2.1.1.1.4.cmml" xref="A3.Ex4.m2.1.1.1.4">𝑜</ci><ci id="A3.Ex4.m2.1.1.1.5.cmml" xref="A3.Ex4.m2.1.1.1.5">𝑓</ci><ci id="A3.Ex4.m2.1.1.1.6.cmml" xref="A3.Ex4.m2.1.1.1.6">𝑡</ci><ci id="A3.Ex4.m2.1.1.1.7.cmml" xref="A3.Ex4.m2.1.1.1.7">𝑚</ci><ci id="A3.Ex4.m2.1.1.1.8.cmml" xref="A3.Ex4.m2.1.1.1.8">𝑎</ci><ci id="A3.Ex4.m2.1.1.1.9.cmml" xref="A3.Ex4.m2.1.1.1.9">𝑥</ci><apply id="A3.Ex4.m2.1.1.1.1.1.1.cmml" xref="A3.Ex4.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="A3.Ex4.m2.1.1.1.1.1.1.1.cmml" xref="A3.Ex4.m2.1.1.1.1.1">subscript</csymbol><ci id="A3.Ex4.m2.1.1.1.1.1.1.2.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.2">𝑙</ci><cn type="integer" id="A3.Ex4.m2.1.1.1.1.1.1.3.cmml" xref="A3.Ex4.m2.1.1.1.1.1.1.3">2</cn></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex4.m2.1c">\displaystyle=softmax(l_{2})</annotation></semantics></math></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A3.Ex4.m3.1" class="ltx_Math" alttext="\displaystyle J" display="inline"><semantics id="A3.Ex4.m3.1a"><mi id="A3.Ex4.m3.1.1" xref="A3.Ex4.m3.1.1.cmml">J</mi><annotation-xml encoding="MathML-Content" id="A3.Ex4.m3.1b"><ci id="A3.Ex4.m3.1.1.cmml" xref="A3.Ex4.m3.1.1">𝐽</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex4.m3.1c">\displaystyle J</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A3.Ex4.m4.2" class="ltx_Math" alttext="\displaystyle=CE(y,\hat{y})" display="inline"><semantics id="A3.Ex4.m4.2a"><mrow id="A3.Ex4.m4.2.3" xref="A3.Ex4.m4.2.3.cmml"><mi id="A3.Ex4.m4.2.3.2" xref="A3.Ex4.m4.2.3.2.cmml"></mi><mo id="A3.Ex4.m4.2.3.1" xref="A3.Ex4.m4.2.3.1.cmml">=</mo><mrow id="A3.Ex4.m4.2.3.3" xref="A3.Ex4.m4.2.3.3.cmml"><mi id="A3.Ex4.m4.2.3.3.2" xref="A3.Ex4.m4.2.3.3.2.cmml">C</mi><mo lspace="0em" rspace="0em" id="A3.Ex4.m4.2.3.3.1" xref="A3.Ex4.m4.2.3.3.1.cmml">​</mo><mi id="A3.Ex4.m4.2.3.3.3" xref="A3.Ex4.m4.2.3.3.3.cmml">E</mi><mo lspace="0em" rspace="0em" id="A3.Ex4.m4.2.3.3.1a" xref="A3.Ex4.m4.2.3.3.1.cmml">​</mo><mrow id="A3.Ex4.m4.2.3.3.4.2" xref="A3.Ex4.m4.2.3.3.4.1.cmml"><mo stretchy="false" id="A3.Ex4.m4.2.3.3.4.2.1" xref="A3.Ex4.m4.2.3.3.4.1.cmml">(</mo><mi id="A3.Ex4.m4.1.1" xref="A3.Ex4.m4.1.1.cmml">y</mi><mo id="A3.Ex4.m4.2.3.3.4.2.2" xref="A3.Ex4.m4.2.3.3.4.1.cmml">,</mo><mover accent="true" id="A3.Ex4.m4.2.2" xref="A3.Ex4.m4.2.2.cmml"><mi id="A3.Ex4.m4.2.2.2" xref="A3.Ex4.m4.2.2.2.cmml">y</mi><mo id="A3.Ex4.m4.2.2.1" xref="A3.Ex4.m4.2.2.1.cmml">^</mo></mover><mo stretchy="false" id="A3.Ex4.m4.2.3.3.4.2.3" xref="A3.Ex4.m4.2.3.3.4.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.Ex4.m4.2b"><apply id="A3.Ex4.m4.2.3.cmml" xref="A3.Ex4.m4.2.3"><eq id="A3.Ex4.m4.2.3.1.cmml" xref="A3.Ex4.m4.2.3.1"></eq><csymbol cd="latexml" id="A3.Ex4.m4.2.3.2.cmml" xref="A3.Ex4.m4.2.3.2">absent</csymbol><apply id="A3.Ex4.m4.2.3.3.cmml" xref="A3.Ex4.m4.2.3.3"><times id="A3.Ex4.m4.2.3.3.1.cmml" xref="A3.Ex4.m4.2.3.3.1"></times><ci id="A3.Ex4.m4.2.3.3.2.cmml" xref="A3.Ex4.m4.2.3.3.2">𝐶</ci><ci id="A3.Ex4.m4.2.3.3.3.cmml" xref="A3.Ex4.m4.2.3.3.3">𝐸</ci><interval closure="open" id="A3.Ex4.m4.2.3.3.4.1.cmml" xref="A3.Ex4.m4.2.3.3.4.2"><ci id="A3.Ex4.m4.1.1.cmml" xref="A3.Ex4.m4.1.1">𝑦</ci><apply id="A3.Ex4.m4.2.2.cmml" xref="A3.Ex4.m4.2.2"><ci id="A3.Ex4.m4.2.2.1.cmml" xref="A3.Ex4.m4.2.2.1">^</ci><ci id="A3.Ex4.m4.2.2.2.cmml" xref="A3.Ex4.m4.2.2.2">𝑦</ci></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex4.m4.2c">\displaystyle=CE(y,\hat{y})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A3.p2.4" class="ltx_p">Analysing the weights’ update below, we can observe that the optimal weights are noisy perturbation of the inputs features map and all together they contain information about the training either related to the raw input or the feature map at a given stage.</p>
<table id="A8.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A3.Ex5"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A3.Ex5.m1.1" class="ltx_Math" alttext="\displaystyle\delta_{1}" display="inline"><semantics id="A3.Ex5.m1.1a"><msub id="A3.Ex5.m1.1.1" xref="A3.Ex5.m1.1.1.cmml"><mi id="A3.Ex5.m1.1.1.2" xref="A3.Ex5.m1.1.1.2.cmml">δ</mi><mn id="A3.Ex5.m1.1.1.3" xref="A3.Ex5.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A3.Ex5.m1.1b"><apply id="A3.Ex5.m1.1.1.cmml" xref="A3.Ex5.m1.1.1"><csymbol cd="ambiguous" id="A3.Ex5.m1.1.1.1.cmml" xref="A3.Ex5.m1.1.1">subscript</csymbol><ci id="A3.Ex5.m1.1.1.2.cmml" xref="A3.Ex5.m1.1.1.2">𝛿</ci><cn type="integer" id="A3.Ex5.m1.1.1.3.cmml" xref="A3.Ex5.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex5.m1.1c">\displaystyle\delta_{1}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A3.Ex5.m2.1" class="ltx_Math" alttext="\displaystyle=\frac{\partial J}{\partial l_{2}}=(y-\hat{y})^{T}" display="inline"><semantics id="A3.Ex5.m2.1a"><mrow id="A3.Ex5.m2.1.1" xref="A3.Ex5.m2.1.1.cmml"><mi id="A3.Ex5.m2.1.1.3" xref="A3.Ex5.m2.1.1.3.cmml"></mi><mo id="A3.Ex5.m2.1.1.4" xref="A3.Ex5.m2.1.1.4.cmml">=</mo><mstyle displaystyle="true" id="A3.Ex5.m2.1.1.5" xref="A3.Ex5.m2.1.1.5.cmml"><mfrac id="A3.Ex5.m2.1.1.5a" xref="A3.Ex5.m2.1.1.5.cmml"><mrow id="A3.Ex5.m2.1.1.5.2" xref="A3.Ex5.m2.1.1.5.2.cmml"><mo rspace="0em" id="A3.Ex5.m2.1.1.5.2.1" xref="A3.Ex5.m2.1.1.5.2.1.cmml">∂</mo><mi id="A3.Ex5.m2.1.1.5.2.2" xref="A3.Ex5.m2.1.1.5.2.2.cmml">J</mi></mrow><mrow id="A3.Ex5.m2.1.1.5.3" xref="A3.Ex5.m2.1.1.5.3.cmml"><mo rspace="0em" id="A3.Ex5.m2.1.1.5.3.1" xref="A3.Ex5.m2.1.1.5.3.1.cmml">∂</mo><msub id="A3.Ex5.m2.1.1.5.3.2" xref="A3.Ex5.m2.1.1.5.3.2.cmml"><mi id="A3.Ex5.m2.1.1.5.3.2.2" xref="A3.Ex5.m2.1.1.5.3.2.2.cmml">l</mi><mn id="A3.Ex5.m2.1.1.5.3.2.3" xref="A3.Ex5.m2.1.1.5.3.2.3.cmml">2</mn></msub></mrow></mfrac></mstyle><mo id="A3.Ex5.m2.1.1.6" xref="A3.Ex5.m2.1.1.6.cmml">=</mo><msup id="A3.Ex5.m2.1.1.1" xref="A3.Ex5.m2.1.1.1.cmml"><mrow id="A3.Ex5.m2.1.1.1.1.1" xref="A3.Ex5.m2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A3.Ex5.m2.1.1.1.1.1.2" xref="A3.Ex5.m2.1.1.1.1.1.1.cmml">(</mo><mrow id="A3.Ex5.m2.1.1.1.1.1.1" xref="A3.Ex5.m2.1.1.1.1.1.1.cmml"><mi id="A3.Ex5.m2.1.1.1.1.1.1.2" xref="A3.Ex5.m2.1.1.1.1.1.1.2.cmml">y</mi><mo id="A3.Ex5.m2.1.1.1.1.1.1.1" xref="A3.Ex5.m2.1.1.1.1.1.1.1.cmml">−</mo><mover accent="true" id="A3.Ex5.m2.1.1.1.1.1.1.3" xref="A3.Ex5.m2.1.1.1.1.1.1.3.cmml"><mi id="A3.Ex5.m2.1.1.1.1.1.1.3.2" xref="A3.Ex5.m2.1.1.1.1.1.1.3.2.cmml">y</mi><mo id="A3.Ex5.m2.1.1.1.1.1.1.3.1" xref="A3.Ex5.m2.1.1.1.1.1.1.3.1.cmml">^</mo></mover></mrow><mo stretchy="false" id="A3.Ex5.m2.1.1.1.1.1.3" xref="A3.Ex5.m2.1.1.1.1.1.1.cmml">)</mo></mrow><mi id="A3.Ex5.m2.1.1.1.3" xref="A3.Ex5.m2.1.1.1.3.cmml">T</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="A3.Ex5.m2.1b"><apply id="A3.Ex5.m2.1.1.cmml" xref="A3.Ex5.m2.1.1"><and id="A3.Ex5.m2.1.1a.cmml" xref="A3.Ex5.m2.1.1"></and><apply id="A3.Ex5.m2.1.1b.cmml" xref="A3.Ex5.m2.1.1"><eq id="A3.Ex5.m2.1.1.4.cmml" xref="A3.Ex5.m2.1.1.4"></eq><csymbol cd="latexml" id="A3.Ex5.m2.1.1.3.cmml" xref="A3.Ex5.m2.1.1.3">absent</csymbol><apply id="A3.Ex5.m2.1.1.5.cmml" xref="A3.Ex5.m2.1.1.5"><divide id="A3.Ex5.m2.1.1.5.1.cmml" xref="A3.Ex5.m2.1.1.5"></divide><apply id="A3.Ex5.m2.1.1.5.2.cmml" xref="A3.Ex5.m2.1.1.5.2"><partialdiff id="A3.Ex5.m2.1.1.5.2.1.cmml" xref="A3.Ex5.m2.1.1.5.2.1"></partialdiff><ci id="A3.Ex5.m2.1.1.5.2.2.cmml" xref="A3.Ex5.m2.1.1.5.2.2">𝐽</ci></apply><apply id="A3.Ex5.m2.1.1.5.3.cmml" xref="A3.Ex5.m2.1.1.5.3"><partialdiff id="A3.Ex5.m2.1.1.5.3.1.cmml" xref="A3.Ex5.m2.1.1.5.3.1"></partialdiff><apply id="A3.Ex5.m2.1.1.5.3.2.cmml" xref="A3.Ex5.m2.1.1.5.3.2"><csymbol cd="ambiguous" id="A3.Ex5.m2.1.1.5.3.2.1.cmml" xref="A3.Ex5.m2.1.1.5.3.2">subscript</csymbol><ci id="A3.Ex5.m2.1.1.5.3.2.2.cmml" xref="A3.Ex5.m2.1.1.5.3.2.2">𝑙</ci><cn type="integer" id="A3.Ex5.m2.1.1.5.3.2.3.cmml" xref="A3.Ex5.m2.1.1.5.3.2.3">2</cn></apply></apply></apply></apply><apply id="A3.Ex5.m2.1.1c.cmml" xref="A3.Ex5.m2.1.1"><eq id="A3.Ex5.m2.1.1.6.cmml" xref="A3.Ex5.m2.1.1.6"></eq><share href="#A3.Ex5.m2.1.1.5.cmml" id="A3.Ex5.m2.1.1d.cmml" xref="A3.Ex5.m2.1.1"></share><apply id="A3.Ex5.m2.1.1.1.cmml" xref="A3.Ex5.m2.1.1.1"><csymbol cd="ambiguous" id="A3.Ex5.m2.1.1.1.2.cmml" xref="A3.Ex5.m2.1.1.1">superscript</csymbol><apply id="A3.Ex5.m2.1.1.1.1.1.1.cmml" xref="A3.Ex5.m2.1.1.1.1.1"><minus id="A3.Ex5.m2.1.1.1.1.1.1.1.cmml" xref="A3.Ex5.m2.1.1.1.1.1.1.1"></minus><ci id="A3.Ex5.m2.1.1.1.1.1.1.2.cmml" xref="A3.Ex5.m2.1.1.1.1.1.1.2">𝑦</ci><apply id="A3.Ex5.m2.1.1.1.1.1.1.3.cmml" xref="A3.Ex5.m2.1.1.1.1.1.1.3"><ci id="A3.Ex5.m2.1.1.1.1.1.1.3.1.cmml" xref="A3.Ex5.m2.1.1.1.1.1.1.3.1">^</ci><ci id="A3.Ex5.m2.1.1.1.1.1.1.3.2.cmml" xref="A3.Ex5.m2.1.1.1.1.1.1.3.2">𝑦</ci></apply></apply><ci id="A3.Ex5.m2.1.1.1.3.cmml" xref="A3.Ex5.m2.1.1.1.3">𝑇</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex5.m2.1c">\displaystyle=\frac{\partial J}{\partial l_{2}}=(y-\hat{y})^{T}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="A3.Ex6"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A3.Ex6.m1.1" class="ltx_Math" alttext="\displaystyle\delta_{2}" display="inline"><semantics id="A3.Ex6.m1.1a"><msub id="A3.Ex6.m1.1.1" xref="A3.Ex6.m1.1.1.cmml"><mi id="A3.Ex6.m1.1.1.2" xref="A3.Ex6.m1.1.1.2.cmml">δ</mi><mn id="A3.Ex6.m1.1.1.3" xref="A3.Ex6.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A3.Ex6.m1.1b"><apply id="A3.Ex6.m1.1.1.cmml" xref="A3.Ex6.m1.1.1"><csymbol cd="ambiguous" id="A3.Ex6.m1.1.1.1.cmml" xref="A3.Ex6.m1.1.1">subscript</csymbol><ci id="A3.Ex6.m1.1.1.2.cmml" xref="A3.Ex6.m1.1.1.2">𝛿</ci><cn type="integer" id="A3.Ex6.m1.1.1.3.cmml" xref="A3.Ex6.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex6.m1.1c">\displaystyle\delta_{2}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A3.Ex6.m2.1" class="ltx_Math" alttext="\displaystyle=\frac{\partial J}{\partial l_{1}}=\delta_{1}W_{2}osgn(h)" display="inline"><semantics id="A3.Ex6.m2.1a"><mrow id="A3.Ex6.m2.1.2" xref="A3.Ex6.m2.1.2.cmml"><mi id="A3.Ex6.m2.1.2.2" xref="A3.Ex6.m2.1.2.2.cmml"></mi><mo id="A3.Ex6.m2.1.2.3" xref="A3.Ex6.m2.1.2.3.cmml">=</mo><mstyle displaystyle="true" id="A3.Ex6.m2.1.2.4" xref="A3.Ex6.m2.1.2.4.cmml"><mfrac id="A3.Ex6.m2.1.2.4a" xref="A3.Ex6.m2.1.2.4.cmml"><mrow id="A3.Ex6.m2.1.2.4.2" xref="A3.Ex6.m2.1.2.4.2.cmml"><mo rspace="0em" id="A3.Ex6.m2.1.2.4.2.1" xref="A3.Ex6.m2.1.2.4.2.1.cmml">∂</mo><mi id="A3.Ex6.m2.1.2.4.2.2" xref="A3.Ex6.m2.1.2.4.2.2.cmml">J</mi></mrow><mrow id="A3.Ex6.m2.1.2.4.3" xref="A3.Ex6.m2.1.2.4.3.cmml"><mo rspace="0em" id="A3.Ex6.m2.1.2.4.3.1" xref="A3.Ex6.m2.1.2.4.3.1.cmml">∂</mo><msub id="A3.Ex6.m2.1.2.4.3.2" xref="A3.Ex6.m2.1.2.4.3.2.cmml"><mi id="A3.Ex6.m2.1.2.4.3.2.2" xref="A3.Ex6.m2.1.2.4.3.2.2.cmml">l</mi><mn id="A3.Ex6.m2.1.2.4.3.2.3" xref="A3.Ex6.m2.1.2.4.3.2.3.cmml">1</mn></msub></mrow></mfrac></mstyle><mo id="A3.Ex6.m2.1.2.5" xref="A3.Ex6.m2.1.2.5.cmml">=</mo><mrow id="A3.Ex6.m2.1.2.6" xref="A3.Ex6.m2.1.2.6.cmml"><msub id="A3.Ex6.m2.1.2.6.2" xref="A3.Ex6.m2.1.2.6.2.cmml"><mi id="A3.Ex6.m2.1.2.6.2.2" xref="A3.Ex6.m2.1.2.6.2.2.cmml">δ</mi><mn id="A3.Ex6.m2.1.2.6.2.3" xref="A3.Ex6.m2.1.2.6.2.3.cmml">1</mn></msub><mo lspace="0em" rspace="0em" id="A3.Ex6.m2.1.2.6.1" xref="A3.Ex6.m2.1.2.6.1.cmml">​</mo><msub id="A3.Ex6.m2.1.2.6.3" xref="A3.Ex6.m2.1.2.6.3.cmml"><mi id="A3.Ex6.m2.1.2.6.3.2" xref="A3.Ex6.m2.1.2.6.3.2.cmml">W</mi><mn id="A3.Ex6.m2.1.2.6.3.3" xref="A3.Ex6.m2.1.2.6.3.3.cmml">2</mn></msub><mo lspace="0em" rspace="0em" id="A3.Ex6.m2.1.2.6.1a" xref="A3.Ex6.m2.1.2.6.1.cmml">​</mo><mi id="A3.Ex6.m2.1.2.6.4" xref="A3.Ex6.m2.1.2.6.4.cmml">o</mi><mo lspace="0em" rspace="0em" id="A3.Ex6.m2.1.2.6.1b" xref="A3.Ex6.m2.1.2.6.1.cmml">​</mo><mi id="A3.Ex6.m2.1.2.6.5" xref="A3.Ex6.m2.1.2.6.5.cmml">s</mi><mo lspace="0em" rspace="0em" id="A3.Ex6.m2.1.2.6.1c" xref="A3.Ex6.m2.1.2.6.1.cmml">​</mo><mi id="A3.Ex6.m2.1.2.6.6" xref="A3.Ex6.m2.1.2.6.6.cmml">g</mi><mo lspace="0em" rspace="0em" id="A3.Ex6.m2.1.2.6.1d" xref="A3.Ex6.m2.1.2.6.1.cmml">​</mo><mi id="A3.Ex6.m2.1.2.6.7" xref="A3.Ex6.m2.1.2.6.7.cmml">n</mi><mo lspace="0em" rspace="0em" id="A3.Ex6.m2.1.2.6.1e" xref="A3.Ex6.m2.1.2.6.1.cmml">​</mo><mrow id="A3.Ex6.m2.1.2.6.8.2" xref="A3.Ex6.m2.1.2.6.cmml"><mo stretchy="false" id="A3.Ex6.m2.1.2.6.8.2.1" xref="A3.Ex6.m2.1.2.6.cmml">(</mo><mi id="A3.Ex6.m2.1.1" xref="A3.Ex6.m2.1.1.cmml">h</mi><mo stretchy="false" id="A3.Ex6.m2.1.2.6.8.2.2" xref="A3.Ex6.m2.1.2.6.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.Ex6.m2.1b"><apply id="A3.Ex6.m2.1.2.cmml" xref="A3.Ex6.m2.1.2"><and id="A3.Ex6.m2.1.2a.cmml" xref="A3.Ex6.m2.1.2"></and><apply id="A3.Ex6.m2.1.2b.cmml" xref="A3.Ex6.m2.1.2"><eq id="A3.Ex6.m2.1.2.3.cmml" xref="A3.Ex6.m2.1.2.3"></eq><csymbol cd="latexml" id="A3.Ex6.m2.1.2.2.cmml" xref="A3.Ex6.m2.1.2.2">absent</csymbol><apply id="A3.Ex6.m2.1.2.4.cmml" xref="A3.Ex6.m2.1.2.4"><divide id="A3.Ex6.m2.1.2.4.1.cmml" xref="A3.Ex6.m2.1.2.4"></divide><apply id="A3.Ex6.m2.1.2.4.2.cmml" xref="A3.Ex6.m2.1.2.4.2"><partialdiff id="A3.Ex6.m2.1.2.4.2.1.cmml" xref="A3.Ex6.m2.1.2.4.2.1"></partialdiff><ci id="A3.Ex6.m2.1.2.4.2.2.cmml" xref="A3.Ex6.m2.1.2.4.2.2">𝐽</ci></apply><apply id="A3.Ex6.m2.1.2.4.3.cmml" xref="A3.Ex6.m2.1.2.4.3"><partialdiff id="A3.Ex6.m2.1.2.4.3.1.cmml" xref="A3.Ex6.m2.1.2.4.3.1"></partialdiff><apply id="A3.Ex6.m2.1.2.4.3.2.cmml" xref="A3.Ex6.m2.1.2.4.3.2"><csymbol cd="ambiguous" id="A3.Ex6.m2.1.2.4.3.2.1.cmml" xref="A3.Ex6.m2.1.2.4.3.2">subscript</csymbol><ci id="A3.Ex6.m2.1.2.4.3.2.2.cmml" xref="A3.Ex6.m2.1.2.4.3.2.2">𝑙</ci><cn type="integer" id="A3.Ex6.m2.1.2.4.3.2.3.cmml" xref="A3.Ex6.m2.1.2.4.3.2.3">1</cn></apply></apply></apply></apply><apply id="A3.Ex6.m2.1.2c.cmml" xref="A3.Ex6.m2.1.2"><eq id="A3.Ex6.m2.1.2.5.cmml" xref="A3.Ex6.m2.1.2.5"></eq><share href="#A3.Ex6.m2.1.2.4.cmml" id="A3.Ex6.m2.1.2d.cmml" xref="A3.Ex6.m2.1.2"></share><apply id="A3.Ex6.m2.1.2.6.cmml" xref="A3.Ex6.m2.1.2.6"><times id="A3.Ex6.m2.1.2.6.1.cmml" xref="A3.Ex6.m2.1.2.6.1"></times><apply id="A3.Ex6.m2.1.2.6.2.cmml" xref="A3.Ex6.m2.1.2.6.2"><csymbol cd="ambiguous" id="A3.Ex6.m2.1.2.6.2.1.cmml" xref="A3.Ex6.m2.1.2.6.2">subscript</csymbol><ci id="A3.Ex6.m2.1.2.6.2.2.cmml" xref="A3.Ex6.m2.1.2.6.2.2">𝛿</ci><cn type="integer" id="A3.Ex6.m2.1.2.6.2.3.cmml" xref="A3.Ex6.m2.1.2.6.2.3">1</cn></apply><apply id="A3.Ex6.m2.1.2.6.3.cmml" xref="A3.Ex6.m2.1.2.6.3"><csymbol cd="ambiguous" id="A3.Ex6.m2.1.2.6.3.1.cmml" xref="A3.Ex6.m2.1.2.6.3">subscript</csymbol><ci id="A3.Ex6.m2.1.2.6.3.2.cmml" xref="A3.Ex6.m2.1.2.6.3.2">𝑊</ci><cn type="integer" id="A3.Ex6.m2.1.2.6.3.3.cmml" xref="A3.Ex6.m2.1.2.6.3.3">2</cn></apply><ci id="A3.Ex6.m2.1.2.6.4.cmml" xref="A3.Ex6.m2.1.2.6.4">𝑜</ci><ci id="A3.Ex6.m2.1.2.6.5.cmml" xref="A3.Ex6.m2.1.2.6.5">𝑠</ci><ci id="A3.Ex6.m2.1.2.6.6.cmml" xref="A3.Ex6.m2.1.2.6.6">𝑔</ci><ci id="A3.Ex6.m2.1.2.6.7.cmml" xref="A3.Ex6.m2.1.2.6.7">𝑛</ci><ci id="A3.Ex6.m2.1.1.cmml" xref="A3.Ex6.m2.1.1">ℎ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex6.m2.1c">\displaystyle=\frac{\partial J}{\partial l_{1}}=\delta_{1}W_{2}osgn(h)</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="A3.Ex7"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A3.Ex7.m1.1" class="ltx_Math" alttext="\displaystyle W^{(i+1)}_{1}" display="inline"><semantics id="A3.Ex7.m1.1a"><msubsup id="A3.Ex7.m1.1.2" xref="A3.Ex7.m1.1.2.cmml"><mi id="A3.Ex7.m1.1.2.2.2" xref="A3.Ex7.m1.1.2.2.2.cmml">W</mi><mn id="A3.Ex7.m1.1.2.3" xref="A3.Ex7.m1.1.2.3.cmml">1</mn><mrow id="A3.Ex7.m1.1.1.1.1" xref="A3.Ex7.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="A3.Ex7.m1.1.1.1.1.2" xref="A3.Ex7.m1.1.1.1.1.1.cmml">(</mo><mrow id="A3.Ex7.m1.1.1.1.1.1" xref="A3.Ex7.m1.1.1.1.1.1.cmml"><mi id="A3.Ex7.m1.1.1.1.1.1.2" xref="A3.Ex7.m1.1.1.1.1.1.2.cmml">i</mi><mo id="A3.Ex7.m1.1.1.1.1.1.1" xref="A3.Ex7.m1.1.1.1.1.1.1.cmml">+</mo><mn id="A3.Ex7.m1.1.1.1.1.1.3" xref="A3.Ex7.m1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A3.Ex7.m1.1.1.1.1.3" xref="A3.Ex7.m1.1.1.1.1.1.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="A3.Ex7.m1.1b"><apply id="A3.Ex7.m1.1.2.cmml" xref="A3.Ex7.m1.1.2"><csymbol cd="ambiguous" id="A3.Ex7.m1.1.2.1.cmml" xref="A3.Ex7.m1.1.2">subscript</csymbol><apply id="A3.Ex7.m1.1.2.2.cmml" xref="A3.Ex7.m1.1.2"><csymbol cd="ambiguous" id="A3.Ex7.m1.1.2.2.1.cmml" xref="A3.Ex7.m1.1.2">superscript</csymbol><ci id="A3.Ex7.m1.1.2.2.2.cmml" xref="A3.Ex7.m1.1.2.2.2">𝑊</ci><apply id="A3.Ex7.m1.1.1.1.1.1.cmml" xref="A3.Ex7.m1.1.1.1.1"><plus id="A3.Ex7.m1.1.1.1.1.1.1.cmml" xref="A3.Ex7.m1.1.1.1.1.1.1"></plus><ci id="A3.Ex7.m1.1.1.1.1.1.2.cmml" xref="A3.Ex7.m1.1.1.1.1.1.2">𝑖</ci><cn type="integer" id="A3.Ex7.m1.1.1.1.1.1.3.cmml" xref="A3.Ex7.m1.1.1.1.1.1.3">1</cn></apply></apply><cn type="integer" id="A3.Ex7.m1.1.2.3.cmml" xref="A3.Ex7.m1.1.2.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex7.m1.1c">\displaystyle W^{(i+1)}_{1}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A3.Ex7.m2.5" class="ltx_Math" alttext="\displaystyle=W^{(i)}_{1}-\eta\nabla_{w_{1}}\mathcal{L}(w_{1},w_{2},b_{1},b_{2})" display="inline"><semantics id="A3.Ex7.m2.5a"><mrow id="A3.Ex7.m2.5.5" xref="A3.Ex7.m2.5.5.cmml"><mi id="A3.Ex7.m2.5.5.6" xref="A3.Ex7.m2.5.5.6.cmml"></mi><mo id="A3.Ex7.m2.5.5.5" xref="A3.Ex7.m2.5.5.5.cmml">=</mo><mrow id="A3.Ex7.m2.5.5.4" xref="A3.Ex7.m2.5.5.4.cmml"><msubsup id="A3.Ex7.m2.5.5.4.6" xref="A3.Ex7.m2.5.5.4.6.cmml"><mi id="A3.Ex7.m2.5.5.4.6.2.2" xref="A3.Ex7.m2.5.5.4.6.2.2.cmml">W</mi><mn id="A3.Ex7.m2.5.5.4.6.3" xref="A3.Ex7.m2.5.5.4.6.3.cmml">1</mn><mrow id="A3.Ex7.m2.1.1.1.3" xref="A3.Ex7.m2.5.5.4.6.cmml"><mo stretchy="false" id="A3.Ex7.m2.1.1.1.3.1" xref="A3.Ex7.m2.5.5.4.6.cmml">(</mo><mi id="A3.Ex7.m2.1.1.1.1" xref="A3.Ex7.m2.1.1.1.1.cmml">i</mi><mo stretchy="false" id="A3.Ex7.m2.1.1.1.3.2" xref="A3.Ex7.m2.5.5.4.6.cmml">)</mo></mrow></msubsup><mo id="A3.Ex7.m2.5.5.4.5" xref="A3.Ex7.m2.5.5.4.5.cmml">−</mo><mrow id="A3.Ex7.m2.5.5.4.4" xref="A3.Ex7.m2.5.5.4.4.cmml"><mi id="A3.Ex7.m2.5.5.4.4.6" xref="A3.Ex7.m2.5.5.4.4.6.cmml">η</mi><mo lspace="0.167em" rspace="0em" id="A3.Ex7.m2.5.5.4.4.5" xref="A3.Ex7.m2.5.5.4.4.5.cmml">​</mo><mrow id="A3.Ex7.m2.5.5.4.4.7" xref="A3.Ex7.m2.5.5.4.4.7.cmml"><msub id="A3.Ex7.m2.5.5.4.4.7.1" xref="A3.Ex7.m2.5.5.4.4.7.1.cmml"><mo rspace="0.167em" id="A3.Ex7.m2.5.5.4.4.7.1.2" xref="A3.Ex7.m2.5.5.4.4.7.1.2.cmml">∇</mo><msub id="A3.Ex7.m2.5.5.4.4.7.1.3" xref="A3.Ex7.m2.5.5.4.4.7.1.3.cmml"><mi id="A3.Ex7.m2.5.5.4.4.7.1.3.2" xref="A3.Ex7.m2.5.5.4.4.7.1.3.2.cmml">w</mi><mn id="A3.Ex7.m2.5.5.4.4.7.1.3.3" xref="A3.Ex7.m2.5.5.4.4.7.1.3.3.cmml">1</mn></msub></msub><mi class="ltx_font_mathcaligraphic" id="A3.Ex7.m2.5.5.4.4.7.2" xref="A3.Ex7.m2.5.5.4.4.7.2.cmml">ℒ</mi></mrow><mo lspace="0em" rspace="0em" id="A3.Ex7.m2.5.5.4.4.5a" xref="A3.Ex7.m2.5.5.4.4.5.cmml">​</mo><mrow id="A3.Ex7.m2.5.5.4.4.4.4" xref="A3.Ex7.m2.5.5.4.4.4.5.cmml"><mo stretchy="false" id="A3.Ex7.m2.5.5.4.4.4.4.5" xref="A3.Ex7.m2.5.5.4.4.4.5.cmml">(</mo><msub id="A3.Ex7.m2.2.2.1.1.1.1.1" xref="A3.Ex7.m2.2.2.1.1.1.1.1.cmml"><mi id="A3.Ex7.m2.2.2.1.1.1.1.1.2" xref="A3.Ex7.m2.2.2.1.1.1.1.1.2.cmml">w</mi><mn id="A3.Ex7.m2.2.2.1.1.1.1.1.3" xref="A3.Ex7.m2.2.2.1.1.1.1.1.3.cmml">1</mn></msub><mo id="A3.Ex7.m2.5.5.4.4.4.4.6" xref="A3.Ex7.m2.5.5.4.4.4.5.cmml">,</mo><msub id="A3.Ex7.m2.3.3.2.2.2.2.2" xref="A3.Ex7.m2.3.3.2.2.2.2.2.cmml"><mi id="A3.Ex7.m2.3.3.2.2.2.2.2.2" xref="A3.Ex7.m2.3.3.2.2.2.2.2.2.cmml">w</mi><mn id="A3.Ex7.m2.3.3.2.2.2.2.2.3" xref="A3.Ex7.m2.3.3.2.2.2.2.2.3.cmml">2</mn></msub><mo id="A3.Ex7.m2.5.5.4.4.4.4.7" xref="A3.Ex7.m2.5.5.4.4.4.5.cmml">,</mo><msub id="A3.Ex7.m2.4.4.3.3.3.3.3" xref="A3.Ex7.m2.4.4.3.3.3.3.3.cmml"><mi id="A3.Ex7.m2.4.4.3.3.3.3.3.2" xref="A3.Ex7.m2.4.4.3.3.3.3.3.2.cmml">b</mi><mn id="A3.Ex7.m2.4.4.3.3.3.3.3.3" xref="A3.Ex7.m2.4.4.3.3.3.3.3.3.cmml">1</mn></msub><mo id="A3.Ex7.m2.5.5.4.4.4.4.8" xref="A3.Ex7.m2.5.5.4.4.4.5.cmml">,</mo><msub id="A3.Ex7.m2.5.5.4.4.4.4.4" xref="A3.Ex7.m2.5.5.4.4.4.4.4.cmml"><mi id="A3.Ex7.m2.5.5.4.4.4.4.4.2" xref="A3.Ex7.m2.5.5.4.4.4.4.4.2.cmml">b</mi><mn id="A3.Ex7.m2.5.5.4.4.4.4.4.3" xref="A3.Ex7.m2.5.5.4.4.4.4.4.3.cmml">2</mn></msub><mo stretchy="false" id="A3.Ex7.m2.5.5.4.4.4.4.9" xref="A3.Ex7.m2.5.5.4.4.4.5.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.Ex7.m2.5b"><apply id="A3.Ex7.m2.5.5.cmml" xref="A3.Ex7.m2.5.5"><eq id="A3.Ex7.m2.5.5.5.cmml" xref="A3.Ex7.m2.5.5.5"></eq><csymbol cd="latexml" id="A3.Ex7.m2.5.5.6.cmml" xref="A3.Ex7.m2.5.5.6">absent</csymbol><apply id="A3.Ex7.m2.5.5.4.cmml" xref="A3.Ex7.m2.5.5.4"><minus id="A3.Ex7.m2.5.5.4.5.cmml" xref="A3.Ex7.m2.5.5.4.5"></minus><apply id="A3.Ex7.m2.5.5.4.6.cmml" xref="A3.Ex7.m2.5.5.4.6"><csymbol cd="ambiguous" id="A3.Ex7.m2.5.5.4.6.1.cmml" xref="A3.Ex7.m2.5.5.4.6">subscript</csymbol><apply id="A3.Ex7.m2.5.5.4.6.2.cmml" xref="A3.Ex7.m2.5.5.4.6"><csymbol cd="ambiguous" id="A3.Ex7.m2.5.5.4.6.2.1.cmml" xref="A3.Ex7.m2.5.5.4.6">superscript</csymbol><ci id="A3.Ex7.m2.5.5.4.6.2.2.cmml" xref="A3.Ex7.m2.5.5.4.6.2.2">𝑊</ci><ci id="A3.Ex7.m2.1.1.1.1.cmml" xref="A3.Ex7.m2.1.1.1.1">𝑖</ci></apply><cn type="integer" id="A3.Ex7.m2.5.5.4.6.3.cmml" xref="A3.Ex7.m2.5.5.4.6.3">1</cn></apply><apply id="A3.Ex7.m2.5.5.4.4.cmml" xref="A3.Ex7.m2.5.5.4.4"><times id="A3.Ex7.m2.5.5.4.4.5.cmml" xref="A3.Ex7.m2.5.5.4.4.5"></times><ci id="A3.Ex7.m2.5.5.4.4.6.cmml" xref="A3.Ex7.m2.5.5.4.4.6">𝜂</ci><apply id="A3.Ex7.m2.5.5.4.4.7.cmml" xref="A3.Ex7.m2.5.5.4.4.7"><apply id="A3.Ex7.m2.5.5.4.4.7.1.cmml" xref="A3.Ex7.m2.5.5.4.4.7.1"><csymbol cd="ambiguous" id="A3.Ex7.m2.5.5.4.4.7.1.1.cmml" xref="A3.Ex7.m2.5.5.4.4.7.1">subscript</csymbol><ci id="A3.Ex7.m2.5.5.4.4.7.1.2.cmml" xref="A3.Ex7.m2.5.5.4.4.7.1.2">∇</ci><apply id="A3.Ex7.m2.5.5.4.4.7.1.3.cmml" xref="A3.Ex7.m2.5.5.4.4.7.1.3"><csymbol cd="ambiguous" id="A3.Ex7.m2.5.5.4.4.7.1.3.1.cmml" xref="A3.Ex7.m2.5.5.4.4.7.1.3">subscript</csymbol><ci id="A3.Ex7.m2.5.5.4.4.7.1.3.2.cmml" xref="A3.Ex7.m2.5.5.4.4.7.1.3.2">𝑤</ci><cn type="integer" id="A3.Ex7.m2.5.5.4.4.7.1.3.3.cmml" xref="A3.Ex7.m2.5.5.4.4.7.1.3.3">1</cn></apply></apply><ci id="A3.Ex7.m2.5.5.4.4.7.2.cmml" xref="A3.Ex7.m2.5.5.4.4.7.2">ℒ</ci></apply><vector id="A3.Ex7.m2.5.5.4.4.4.5.cmml" xref="A3.Ex7.m2.5.5.4.4.4.4"><apply id="A3.Ex7.m2.2.2.1.1.1.1.1.cmml" xref="A3.Ex7.m2.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="A3.Ex7.m2.2.2.1.1.1.1.1.1.cmml" xref="A3.Ex7.m2.2.2.1.1.1.1.1">subscript</csymbol><ci id="A3.Ex7.m2.2.2.1.1.1.1.1.2.cmml" xref="A3.Ex7.m2.2.2.1.1.1.1.1.2">𝑤</ci><cn type="integer" id="A3.Ex7.m2.2.2.1.1.1.1.1.3.cmml" xref="A3.Ex7.m2.2.2.1.1.1.1.1.3">1</cn></apply><apply id="A3.Ex7.m2.3.3.2.2.2.2.2.cmml" xref="A3.Ex7.m2.3.3.2.2.2.2.2"><csymbol cd="ambiguous" id="A3.Ex7.m2.3.3.2.2.2.2.2.1.cmml" xref="A3.Ex7.m2.3.3.2.2.2.2.2">subscript</csymbol><ci id="A3.Ex7.m2.3.3.2.2.2.2.2.2.cmml" xref="A3.Ex7.m2.3.3.2.2.2.2.2.2">𝑤</ci><cn type="integer" id="A3.Ex7.m2.3.3.2.2.2.2.2.3.cmml" xref="A3.Ex7.m2.3.3.2.2.2.2.2.3">2</cn></apply><apply id="A3.Ex7.m2.4.4.3.3.3.3.3.cmml" xref="A3.Ex7.m2.4.4.3.3.3.3.3"><csymbol cd="ambiguous" id="A3.Ex7.m2.4.4.3.3.3.3.3.1.cmml" xref="A3.Ex7.m2.4.4.3.3.3.3.3">subscript</csymbol><ci id="A3.Ex7.m2.4.4.3.3.3.3.3.2.cmml" xref="A3.Ex7.m2.4.4.3.3.3.3.3.2">𝑏</ci><cn type="integer" id="A3.Ex7.m2.4.4.3.3.3.3.3.3.cmml" xref="A3.Ex7.m2.4.4.3.3.3.3.3.3">1</cn></apply><apply id="A3.Ex7.m2.5.5.4.4.4.4.4.cmml" xref="A3.Ex7.m2.5.5.4.4.4.4.4"><csymbol cd="ambiguous" id="A3.Ex7.m2.5.5.4.4.4.4.4.1.cmml" xref="A3.Ex7.m2.5.5.4.4.4.4.4">subscript</csymbol><ci id="A3.Ex7.m2.5.5.4.4.4.4.4.2.cmml" xref="A3.Ex7.m2.5.5.4.4.4.4.4.2">𝑏</ci><cn type="integer" id="A3.Ex7.m2.5.5.4.4.4.4.4.3.cmml" xref="A3.Ex7.m2.5.5.4.4.4.4.4.3">2</cn></apply></vector></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex7.m2.5c">\displaystyle=W^{(i)}_{1}-\eta\nabla_{w_{1}}\mathcal{L}(w_{1},w_{2},b_{1},b_{2})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="A3.Ex8"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A3.Ex8.m1.1" class="ltx_Math" alttext="\displaystyle=W^{(i)}_{1}-\eta\delta_{2}^{T}x" display="inline"><semantics id="A3.Ex8.m1.1a"><mrow id="A3.Ex8.m1.1.2" xref="A3.Ex8.m1.1.2.cmml"><mi id="A3.Ex8.m1.1.2.2" xref="A3.Ex8.m1.1.2.2.cmml"></mi><mo id="A3.Ex8.m1.1.2.1" xref="A3.Ex8.m1.1.2.1.cmml">=</mo><mrow id="A3.Ex8.m1.1.2.3" xref="A3.Ex8.m1.1.2.3.cmml"><msubsup id="A3.Ex8.m1.1.2.3.2" xref="A3.Ex8.m1.1.2.3.2.cmml"><mi id="A3.Ex8.m1.1.2.3.2.2.2" xref="A3.Ex8.m1.1.2.3.2.2.2.cmml">W</mi><mn id="A3.Ex8.m1.1.2.3.2.3" xref="A3.Ex8.m1.1.2.3.2.3.cmml">1</mn><mrow id="A3.Ex8.m1.1.1.1.3" xref="A3.Ex8.m1.1.2.3.2.cmml"><mo stretchy="false" id="A3.Ex8.m1.1.1.1.3.1" xref="A3.Ex8.m1.1.2.3.2.cmml">(</mo><mi id="A3.Ex8.m1.1.1.1.1" xref="A3.Ex8.m1.1.1.1.1.cmml">i</mi><mo stretchy="false" id="A3.Ex8.m1.1.1.1.3.2" xref="A3.Ex8.m1.1.2.3.2.cmml">)</mo></mrow></msubsup><mo id="A3.Ex8.m1.1.2.3.1" xref="A3.Ex8.m1.1.2.3.1.cmml">−</mo><mrow id="A3.Ex8.m1.1.2.3.3" xref="A3.Ex8.m1.1.2.3.3.cmml"><mi id="A3.Ex8.m1.1.2.3.3.2" xref="A3.Ex8.m1.1.2.3.3.2.cmml">η</mi><mo lspace="0em" rspace="0em" id="A3.Ex8.m1.1.2.3.3.1" xref="A3.Ex8.m1.1.2.3.3.1.cmml">​</mo><msubsup id="A3.Ex8.m1.1.2.3.3.3" xref="A3.Ex8.m1.1.2.3.3.3.cmml"><mi id="A3.Ex8.m1.1.2.3.3.3.2.2" xref="A3.Ex8.m1.1.2.3.3.3.2.2.cmml">δ</mi><mn id="A3.Ex8.m1.1.2.3.3.3.2.3" xref="A3.Ex8.m1.1.2.3.3.3.2.3.cmml">2</mn><mi id="A3.Ex8.m1.1.2.3.3.3.3" xref="A3.Ex8.m1.1.2.3.3.3.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="A3.Ex8.m1.1.2.3.3.1a" xref="A3.Ex8.m1.1.2.3.3.1.cmml">​</mo><mi id="A3.Ex8.m1.1.2.3.3.4" xref="A3.Ex8.m1.1.2.3.3.4.cmml">x</mi></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.Ex8.m1.1b"><apply id="A3.Ex8.m1.1.2.cmml" xref="A3.Ex8.m1.1.2"><eq id="A3.Ex8.m1.1.2.1.cmml" xref="A3.Ex8.m1.1.2.1"></eq><csymbol cd="latexml" id="A3.Ex8.m1.1.2.2.cmml" xref="A3.Ex8.m1.1.2.2">absent</csymbol><apply id="A3.Ex8.m1.1.2.3.cmml" xref="A3.Ex8.m1.1.2.3"><minus id="A3.Ex8.m1.1.2.3.1.cmml" xref="A3.Ex8.m1.1.2.3.1"></minus><apply id="A3.Ex8.m1.1.2.3.2.cmml" xref="A3.Ex8.m1.1.2.3.2"><csymbol cd="ambiguous" id="A3.Ex8.m1.1.2.3.2.1.cmml" xref="A3.Ex8.m1.1.2.3.2">subscript</csymbol><apply id="A3.Ex8.m1.1.2.3.2.2.cmml" xref="A3.Ex8.m1.1.2.3.2"><csymbol cd="ambiguous" id="A3.Ex8.m1.1.2.3.2.2.1.cmml" xref="A3.Ex8.m1.1.2.3.2">superscript</csymbol><ci id="A3.Ex8.m1.1.2.3.2.2.2.cmml" xref="A3.Ex8.m1.1.2.3.2.2.2">𝑊</ci><ci id="A3.Ex8.m1.1.1.1.1.cmml" xref="A3.Ex8.m1.1.1.1.1">𝑖</ci></apply><cn type="integer" id="A3.Ex8.m1.1.2.3.2.3.cmml" xref="A3.Ex8.m1.1.2.3.2.3">1</cn></apply><apply id="A3.Ex8.m1.1.2.3.3.cmml" xref="A3.Ex8.m1.1.2.3.3"><times id="A3.Ex8.m1.1.2.3.3.1.cmml" xref="A3.Ex8.m1.1.2.3.3.1"></times><ci id="A3.Ex8.m1.1.2.3.3.2.cmml" xref="A3.Ex8.m1.1.2.3.3.2">𝜂</ci><apply id="A3.Ex8.m1.1.2.3.3.3.cmml" xref="A3.Ex8.m1.1.2.3.3.3"><csymbol cd="ambiguous" id="A3.Ex8.m1.1.2.3.3.3.1.cmml" xref="A3.Ex8.m1.1.2.3.3.3">superscript</csymbol><apply id="A3.Ex8.m1.1.2.3.3.3.2.cmml" xref="A3.Ex8.m1.1.2.3.3.3"><csymbol cd="ambiguous" id="A3.Ex8.m1.1.2.3.3.3.2.1.cmml" xref="A3.Ex8.m1.1.2.3.3.3">subscript</csymbol><ci id="A3.Ex8.m1.1.2.3.3.3.2.2.cmml" xref="A3.Ex8.m1.1.2.3.3.3.2.2">𝛿</ci><cn type="integer" id="A3.Ex8.m1.1.2.3.3.3.2.3.cmml" xref="A3.Ex8.m1.1.2.3.3.3.2.3">2</cn></apply><ci id="A3.Ex8.m1.1.2.3.3.3.3.cmml" xref="A3.Ex8.m1.1.2.3.3.3.3">𝑇</ci></apply><ci id="A3.Ex8.m1.1.2.3.3.4.cmml" xref="A3.Ex8.m1.1.2.3.3.4">𝑥</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex8.m1.1c">\displaystyle=W^{(i)}_{1}-\eta\delta_{2}^{T}x</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="A3.Ex9"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A3.Ex9.m1.1" class="ltx_Math" alttext="\displaystyle W^{(i+1)}_{2}" display="inline"><semantics id="A3.Ex9.m1.1a"><msubsup id="A3.Ex9.m1.1.2" xref="A3.Ex9.m1.1.2.cmml"><mi id="A3.Ex9.m1.1.2.2.2" xref="A3.Ex9.m1.1.2.2.2.cmml">W</mi><mn id="A3.Ex9.m1.1.2.3" xref="A3.Ex9.m1.1.2.3.cmml">2</mn><mrow id="A3.Ex9.m1.1.1.1.1" xref="A3.Ex9.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="A3.Ex9.m1.1.1.1.1.2" xref="A3.Ex9.m1.1.1.1.1.1.cmml">(</mo><mrow id="A3.Ex9.m1.1.1.1.1.1" xref="A3.Ex9.m1.1.1.1.1.1.cmml"><mi id="A3.Ex9.m1.1.1.1.1.1.2" xref="A3.Ex9.m1.1.1.1.1.1.2.cmml">i</mi><mo id="A3.Ex9.m1.1.1.1.1.1.1" xref="A3.Ex9.m1.1.1.1.1.1.1.cmml">+</mo><mn id="A3.Ex9.m1.1.1.1.1.1.3" xref="A3.Ex9.m1.1.1.1.1.1.3.cmml">1</mn></mrow><mo stretchy="false" id="A3.Ex9.m1.1.1.1.1.3" xref="A3.Ex9.m1.1.1.1.1.1.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="A3.Ex9.m1.1b"><apply id="A3.Ex9.m1.1.2.cmml" xref="A3.Ex9.m1.1.2"><csymbol cd="ambiguous" id="A3.Ex9.m1.1.2.1.cmml" xref="A3.Ex9.m1.1.2">subscript</csymbol><apply id="A3.Ex9.m1.1.2.2.cmml" xref="A3.Ex9.m1.1.2"><csymbol cd="ambiguous" id="A3.Ex9.m1.1.2.2.1.cmml" xref="A3.Ex9.m1.1.2">superscript</csymbol><ci id="A3.Ex9.m1.1.2.2.2.cmml" xref="A3.Ex9.m1.1.2.2.2">𝑊</ci><apply id="A3.Ex9.m1.1.1.1.1.1.cmml" xref="A3.Ex9.m1.1.1.1.1"><plus id="A3.Ex9.m1.1.1.1.1.1.1.cmml" xref="A3.Ex9.m1.1.1.1.1.1.1"></plus><ci id="A3.Ex9.m1.1.1.1.1.1.2.cmml" xref="A3.Ex9.m1.1.1.1.1.1.2">𝑖</ci><cn type="integer" id="A3.Ex9.m1.1.1.1.1.1.3.cmml" xref="A3.Ex9.m1.1.1.1.1.1.3">1</cn></apply></apply><cn type="integer" id="A3.Ex9.m1.1.2.3.cmml" xref="A3.Ex9.m1.1.2.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex9.m1.1c">\displaystyle W^{(i+1)}_{2}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A3.Ex9.m2.5" class="ltx_Math" alttext="\displaystyle=W^{(i)}_{2}-\eta\nabla_{w_{2}}\mathcal{L}(w_{1},w_{2},b_{1},b_{2})" display="inline"><semantics id="A3.Ex9.m2.5a"><mrow id="A3.Ex9.m2.5.5" xref="A3.Ex9.m2.5.5.cmml"><mi id="A3.Ex9.m2.5.5.6" xref="A3.Ex9.m2.5.5.6.cmml"></mi><mo id="A3.Ex9.m2.5.5.5" xref="A3.Ex9.m2.5.5.5.cmml">=</mo><mrow id="A3.Ex9.m2.5.5.4" xref="A3.Ex9.m2.5.5.4.cmml"><msubsup id="A3.Ex9.m2.5.5.4.6" xref="A3.Ex9.m2.5.5.4.6.cmml"><mi id="A3.Ex9.m2.5.5.4.6.2.2" xref="A3.Ex9.m2.5.5.4.6.2.2.cmml">W</mi><mn id="A3.Ex9.m2.5.5.4.6.3" xref="A3.Ex9.m2.5.5.4.6.3.cmml">2</mn><mrow id="A3.Ex9.m2.1.1.1.3" xref="A3.Ex9.m2.5.5.4.6.cmml"><mo stretchy="false" id="A3.Ex9.m2.1.1.1.3.1" xref="A3.Ex9.m2.5.5.4.6.cmml">(</mo><mi id="A3.Ex9.m2.1.1.1.1" xref="A3.Ex9.m2.1.1.1.1.cmml">i</mi><mo stretchy="false" id="A3.Ex9.m2.1.1.1.3.2" xref="A3.Ex9.m2.5.5.4.6.cmml">)</mo></mrow></msubsup><mo id="A3.Ex9.m2.5.5.4.5" xref="A3.Ex9.m2.5.5.4.5.cmml">−</mo><mrow id="A3.Ex9.m2.5.5.4.4" xref="A3.Ex9.m2.5.5.4.4.cmml"><mi id="A3.Ex9.m2.5.5.4.4.6" xref="A3.Ex9.m2.5.5.4.4.6.cmml">η</mi><mo lspace="0.167em" rspace="0em" id="A3.Ex9.m2.5.5.4.4.5" xref="A3.Ex9.m2.5.5.4.4.5.cmml">​</mo><mrow id="A3.Ex9.m2.5.5.4.4.7" xref="A3.Ex9.m2.5.5.4.4.7.cmml"><msub id="A3.Ex9.m2.5.5.4.4.7.1" xref="A3.Ex9.m2.5.5.4.4.7.1.cmml"><mo rspace="0.167em" id="A3.Ex9.m2.5.5.4.4.7.1.2" xref="A3.Ex9.m2.5.5.4.4.7.1.2.cmml">∇</mo><msub id="A3.Ex9.m2.5.5.4.4.7.1.3" xref="A3.Ex9.m2.5.5.4.4.7.1.3.cmml"><mi id="A3.Ex9.m2.5.5.4.4.7.1.3.2" xref="A3.Ex9.m2.5.5.4.4.7.1.3.2.cmml">w</mi><mn id="A3.Ex9.m2.5.5.4.4.7.1.3.3" xref="A3.Ex9.m2.5.5.4.4.7.1.3.3.cmml">2</mn></msub></msub><mi class="ltx_font_mathcaligraphic" id="A3.Ex9.m2.5.5.4.4.7.2" xref="A3.Ex9.m2.5.5.4.4.7.2.cmml">ℒ</mi></mrow><mo lspace="0em" rspace="0em" id="A3.Ex9.m2.5.5.4.4.5a" xref="A3.Ex9.m2.5.5.4.4.5.cmml">​</mo><mrow id="A3.Ex9.m2.5.5.4.4.4.4" xref="A3.Ex9.m2.5.5.4.4.4.5.cmml"><mo stretchy="false" id="A3.Ex9.m2.5.5.4.4.4.4.5" xref="A3.Ex9.m2.5.5.4.4.4.5.cmml">(</mo><msub id="A3.Ex9.m2.2.2.1.1.1.1.1" xref="A3.Ex9.m2.2.2.1.1.1.1.1.cmml"><mi id="A3.Ex9.m2.2.2.1.1.1.1.1.2" xref="A3.Ex9.m2.2.2.1.1.1.1.1.2.cmml">w</mi><mn id="A3.Ex9.m2.2.2.1.1.1.1.1.3" xref="A3.Ex9.m2.2.2.1.1.1.1.1.3.cmml">1</mn></msub><mo id="A3.Ex9.m2.5.5.4.4.4.4.6" xref="A3.Ex9.m2.5.5.4.4.4.5.cmml">,</mo><msub id="A3.Ex9.m2.3.3.2.2.2.2.2" xref="A3.Ex9.m2.3.3.2.2.2.2.2.cmml"><mi id="A3.Ex9.m2.3.3.2.2.2.2.2.2" xref="A3.Ex9.m2.3.3.2.2.2.2.2.2.cmml">w</mi><mn id="A3.Ex9.m2.3.3.2.2.2.2.2.3" xref="A3.Ex9.m2.3.3.2.2.2.2.2.3.cmml">2</mn></msub><mo id="A3.Ex9.m2.5.5.4.4.4.4.7" xref="A3.Ex9.m2.5.5.4.4.4.5.cmml">,</mo><msub id="A3.Ex9.m2.4.4.3.3.3.3.3" xref="A3.Ex9.m2.4.4.3.3.3.3.3.cmml"><mi id="A3.Ex9.m2.4.4.3.3.3.3.3.2" xref="A3.Ex9.m2.4.4.3.3.3.3.3.2.cmml">b</mi><mn id="A3.Ex9.m2.4.4.3.3.3.3.3.3" xref="A3.Ex9.m2.4.4.3.3.3.3.3.3.cmml">1</mn></msub><mo id="A3.Ex9.m2.5.5.4.4.4.4.8" xref="A3.Ex9.m2.5.5.4.4.4.5.cmml">,</mo><msub id="A3.Ex9.m2.5.5.4.4.4.4.4" xref="A3.Ex9.m2.5.5.4.4.4.4.4.cmml"><mi id="A3.Ex9.m2.5.5.4.4.4.4.4.2" xref="A3.Ex9.m2.5.5.4.4.4.4.4.2.cmml">b</mi><mn id="A3.Ex9.m2.5.5.4.4.4.4.4.3" xref="A3.Ex9.m2.5.5.4.4.4.4.4.3.cmml">2</mn></msub><mo stretchy="false" id="A3.Ex9.m2.5.5.4.4.4.4.9" xref="A3.Ex9.m2.5.5.4.4.4.5.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.Ex9.m2.5b"><apply id="A3.Ex9.m2.5.5.cmml" xref="A3.Ex9.m2.5.5"><eq id="A3.Ex9.m2.5.5.5.cmml" xref="A3.Ex9.m2.5.5.5"></eq><csymbol cd="latexml" id="A3.Ex9.m2.5.5.6.cmml" xref="A3.Ex9.m2.5.5.6">absent</csymbol><apply id="A3.Ex9.m2.5.5.4.cmml" xref="A3.Ex9.m2.5.5.4"><minus id="A3.Ex9.m2.5.5.4.5.cmml" xref="A3.Ex9.m2.5.5.4.5"></minus><apply id="A3.Ex9.m2.5.5.4.6.cmml" xref="A3.Ex9.m2.5.5.4.6"><csymbol cd="ambiguous" id="A3.Ex9.m2.5.5.4.6.1.cmml" xref="A3.Ex9.m2.5.5.4.6">subscript</csymbol><apply id="A3.Ex9.m2.5.5.4.6.2.cmml" xref="A3.Ex9.m2.5.5.4.6"><csymbol cd="ambiguous" id="A3.Ex9.m2.5.5.4.6.2.1.cmml" xref="A3.Ex9.m2.5.5.4.6">superscript</csymbol><ci id="A3.Ex9.m2.5.5.4.6.2.2.cmml" xref="A3.Ex9.m2.5.5.4.6.2.2">𝑊</ci><ci id="A3.Ex9.m2.1.1.1.1.cmml" xref="A3.Ex9.m2.1.1.1.1">𝑖</ci></apply><cn type="integer" id="A3.Ex9.m2.5.5.4.6.3.cmml" xref="A3.Ex9.m2.5.5.4.6.3">2</cn></apply><apply id="A3.Ex9.m2.5.5.4.4.cmml" xref="A3.Ex9.m2.5.5.4.4"><times id="A3.Ex9.m2.5.5.4.4.5.cmml" xref="A3.Ex9.m2.5.5.4.4.5"></times><ci id="A3.Ex9.m2.5.5.4.4.6.cmml" xref="A3.Ex9.m2.5.5.4.4.6">𝜂</ci><apply id="A3.Ex9.m2.5.5.4.4.7.cmml" xref="A3.Ex9.m2.5.5.4.4.7"><apply id="A3.Ex9.m2.5.5.4.4.7.1.cmml" xref="A3.Ex9.m2.5.5.4.4.7.1"><csymbol cd="ambiguous" id="A3.Ex9.m2.5.5.4.4.7.1.1.cmml" xref="A3.Ex9.m2.5.5.4.4.7.1">subscript</csymbol><ci id="A3.Ex9.m2.5.5.4.4.7.1.2.cmml" xref="A3.Ex9.m2.5.5.4.4.7.1.2">∇</ci><apply id="A3.Ex9.m2.5.5.4.4.7.1.3.cmml" xref="A3.Ex9.m2.5.5.4.4.7.1.3"><csymbol cd="ambiguous" id="A3.Ex9.m2.5.5.4.4.7.1.3.1.cmml" xref="A3.Ex9.m2.5.5.4.4.7.1.3">subscript</csymbol><ci id="A3.Ex9.m2.5.5.4.4.7.1.3.2.cmml" xref="A3.Ex9.m2.5.5.4.4.7.1.3.2">𝑤</ci><cn type="integer" id="A3.Ex9.m2.5.5.4.4.7.1.3.3.cmml" xref="A3.Ex9.m2.5.5.4.4.7.1.3.3">2</cn></apply></apply><ci id="A3.Ex9.m2.5.5.4.4.7.2.cmml" xref="A3.Ex9.m2.5.5.4.4.7.2">ℒ</ci></apply><vector id="A3.Ex9.m2.5.5.4.4.4.5.cmml" xref="A3.Ex9.m2.5.5.4.4.4.4"><apply id="A3.Ex9.m2.2.2.1.1.1.1.1.cmml" xref="A3.Ex9.m2.2.2.1.1.1.1.1"><csymbol cd="ambiguous" id="A3.Ex9.m2.2.2.1.1.1.1.1.1.cmml" xref="A3.Ex9.m2.2.2.1.1.1.1.1">subscript</csymbol><ci id="A3.Ex9.m2.2.2.1.1.1.1.1.2.cmml" xref="A3.Ex9.m2.2.2.1.1.1.1.1.2">𝑤</ci><cn type="integer" id="A3.Ex9.m2.2.2.1.1.1.1.1.3.cmml" xref="A3.Ex9.m2.2.2.1.1.1.1.1.3">1</cn></apply><apply id="A3.Ex9.m2.3.3.2.2.2.2.2.cmml" xref="A3.Ex9.m2.3.3.2.2.2.2.2"><csymbol cd="ambiguous" id="A3.Ex9.m2.3.3.2.2.2.2.2.1.cmml" xref="A3.Ex9.m2.3.3.2.2.2.2.2">subscript</csymbol><ci id="A3.Ex9.m2.3.3.2.2.2.2.2.2.cmml" xref="A3.Ex9.m2.3.3.2.2.2.2.2.2">𝑤</ci><cn type="integer" id="A3.Ex9.m2.3.3.2.2.2.2.2.3.cmml" xref="A3.Ex9.m2.3.3.2.2.2.2.2.3">2</cn></apply><apply id="A3.Ex9.m2.4.4.3.3.3.3.3.cmml" xref="A3.Ex9.m2.4.4.3.3.3.3.3"><csymbol cd="ambiguous" id="A3.Ex9.m2.4.4.3.3.3.3.3.1.cmml" xref="A3.Ex9.m2.4.4.3.3.3.3.3">subscript</csymbol><ci id="A3.Ex9.m2.4.4.3.3.3.3.3.2.cmml" xref="A3.Ex9.m2.4.4.3.3.3.3.3.2">𝑏</ci><cn type="integer" id="A3.Ex9.m2.4.4.3.3.3.3.3.3.cmml" xref="A3.Ex9.m2.4.4.3.3.3.3.3.3">1</cn></apply><apply id="A3.Ex9.m2.5.5.4.4.4.4.4.cmml" xref="A3.Ex9.m2.5.5.4.4.4.4.4"><csymbol cd="ambiguous" id="A3.Ex9.m2.5.5.4.4.4.4.4.1.cmml" xref="A3.Ex9.m2.5.5.4.4.4.4.4">subscript</csymbol><ci id="A3.Ex9.m2.5.5.4.4.4.4.4.2.cmml" xref="A3.Ex9.m2.5.5.4.4.4.4.4.2">𝑏</ci><cn type="integer" id="A3.Ex9.m2.5.5.4.4.4.4.4.3.cmml" xref="A3.Ex9.m2.5.5.4.4.4.4.4.3">2</cn></apply></vector></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex9.m2.5c">\displaystyle=W^{(i)}_{2}-\eta\nabla_{w_{2}}\mathcal{L}(w_{1},w_{2},b_{1},b_{2})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="A3.Ex10"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A3.Ex10.m1.1" class="ltx_Math" alttext="\displaystyle=W^{(i)}_{2}-\eta\delta_{1}^{T}h^{T}" display="inline"><semantics id="A3.Ex10.m1.1a"><mrow id="A3.Ex10.m1.1.2" xref="A3.Ex10.m1.1.2.cmml"><mi id="A3.Ex10.m1.1.2.2" xref="A3.Ex10.m1.1.2.2.cmml"></mi><mo id="A3.Ex10.m1.1.2.1" xref="A3.Ex10.m1.1.2.1.cmml">=</mo><mrow id="A3.Ex10.m1.1.2.3" xref="A3.Ex10.m1.1.2.3.cmml"><msubsup id="A3.Ex10.m1.1.2.3.2" xref="A3.Ex10.m1.1.2.3.2.cmml"><mi id="A3.Ex10.m1.1.2.3.2.2.2" xref="A3.Ex10.m1.1.2.3.2.2.2.cmml">W</mi><mn id="A3.Ex10.m1.1.2.3.2.3" xref="A3.Ex10.m1.1.2.3.2.3.cmml">2</mn><mrow id="A3.Ex10.m1.1.1.1.3" xref="A3.Ex10.m1.1.2.3.2.cmml"><mo stretchy="false" id="A3.Ex10.m1.1.1.1.3.1" xref="A3.Ex10.m1.1.2.3.2.cmml">(</mo><mi id="A3.Ex10.m1.1.1.1.1" xref="A3.Ex10.m1.1.1.1.1.cmml">i</mi><mo stretchy="false" id="A3.Ex10.m1.1.1.1.3.2" xref="A3.Ex10.m1.1.2.3.2.cmml">)</mo></mrow></msubsup><mo id="A3.Ex10.m1.1.2.3.1" xref="A3.Ex10.m1.1.2.3.1.cmml">−</mo><mrow id="A3.Ex10.m1.1.2.3.3" xref="A3.Ex10.m1.1.2.3.3.cmml"><mi id="A3.Ex10.m1.1.2.3.3.2" xref="A3.Ex10.m1.1.2.3.3.2.cmml">η</mi><mo lspace="0em" rspace="0em" id="A3.Ex10.m1.1.2.3.3.1" xref="A3.Ex10.m1.1.2.3.3.1.cmml">​</mo><msubsup id="A3.Ex10.m1.1.2.3.3.3" xref="A3.Ex10.m1.1.2.3.3.3.cmml"><mi id="A3.Ex10.m1.1.2.3.3.3.2.2" xref="A3.Ex10.m1.1.2.3.3.3.2.2.cmml">δ</mi><mn id="A3.Ex10.m1.1.2.3.3.3.2.3" xref="A3.Ex10.m1.1.2.3.3.3.2.3.cmml">1</mn><mi id="A3.Ex10.m1.1.2.3.3.3.3" xref="A3.Ex10.m1.1.2.3.3.3.3.cmml">T</mi></msubsup><mo lspace="0em" rspace="0em" id="A3.Ex10.m1.1.2.3.3.1a" xref="A3.Ex10.m1.1.2.3.3.1.cmml">​</mo><msup id="A3.Ex10.m1.1.2.3.3.4" xref="A3.Ex10.m1.1.2.3.3.4.cmml"><mi id="A3.Ex10.m1.1.2.3.3.4.2" xref="A3.Ex10.m1.1.2.3.3.4.2.cmml">h</mi><mi id="A3.Ex10.m1.1.2.3.3.4.3" xref="A3.Ex10.m1.1.2.3.3.4.3.cmml">T</mi></msup></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.Ex10.m1.1b"><apply id="A3.Ex10.m1.1.2.cmml" xref="A3.Ex10.m1.1.2"><eq id="A3.Ex10.m1.1.2.1.cmml" xref="A3.Ex10.m1.1.2.1"></eq><csymbol cd="latexml" id="A3.Ex10.m1.1.2.2.cmml" xref="A3.Ex10.m1.1.2.2">absent</csymbol><apply id="A3.Ex10.m1.1.2.3.cmml" xref="A3.Ex10.m1.1.2.3"><minus id="A3.Ex10.m1.1.2.3.1.cmml" xref="A3.Ex10.m1.1.2.3.1"></minus><apply id="A3.Ex10.m1.1.2.3.2.cmml" xref="A3.Ex10.m1.1.2.3.2"><csymbol cd="ambiguous" id="A3.Ex10.m1.1.2.3.2.1.cmml" xref="A3.Ex10.m1.1.2.3.2">subscript</csymbol><apply id="A3.Ex10.m1.1.2.3.2.2.cmml" xref="A3.Ex10.m1.1.2.3.2"><csymbol cd="ambiguous" id="A3.Ex10.m1.1.2.3.2.2.1.cmml" xref="A3.Ex10.m1.1.2.3.2">superscript</csymbol><ci id="A3.Ex10.m1.1.2.3.2.2.2.cmml" xref="A3.Ex10.m1.1.2.3.2.2.2">𝑊</ci><ci id="A3.Ex10.m1.1.1.1.1.cmml" xref="A3.Ex10.m1.1.1.1.1">𝑖</ci></apply><cn type="integer" id="A3.Ex10.m1.1.2.3.2.3.cmml" xref="A3.Ex10.m1.1.2.3.2.3">2</cn></apply><apply id="A3.Ex10.m1.1.2.3.3.cmml" xref="A3.Ex10.m1.1.2.3.3"><times id="A3.Ex10.m1.1.2.3.3.1.cmml" xref="A3.Ex10.m1.1.2.3.3.1"></times><ci id="A3.Ex10.m1.1.2.3.3.2.cmml" xref="A3.Ex10.m1.1.2.3.3.2">𝜂</ci><apply id="A3.Ex10.m1.1.2.3.3.3.cmml" xref="A3.Ex10.m1.1.2.3.3.3"><csymbol cd="ambiguous" id="A3.Ex10.m1.1.2.3.3.3.1.cmml" xref="A3.Ex10.m1.1.2.3.3.3">superscript</csymbol><apply id="A3.Ex10.m1.1.2.3.3.3.2.cmml" xref="A3.Ex10.m1.1.2.3.3.3"><csymbol cd="ambiguous" id="A3.Ex10.m1.1.2.3.3.3.2.1.cmml" xref="A3.Ex10.m1.1.2.3.3.3">subscript</csymbol><ci id="A3.Ex10.m1.1.2.3.3.3.2.2.cmml" xref="A3.Ex10.m1.1.2.3.3.3.2.2">𝛿</ci><cn type="integer" id="A3.Ex10.m1.1.2.3.3.3.2.3.cmml" xref="A3.Ex10.m1.1.2.3.3.3.2.3">1</cn></apply><ci id="A3.Ex10.m1.1.2.3.3.3.3.cmml" xref="A3.Ex10.m1.1.2.3.3.3.3">𝑇</ci></apply><apply id="A3.Ex10.m1.1.2.3.3.4.cmml" xref="A3.Ex10.m1.1.2.3.3.4"><csymbol cd="ambiguous" id="A3.Ex10.m1.1.2.3.3.4.1.cmml" xref="A3.Ex10.m1.1.2.3.3.4">superscript</csymbol><ci id="A3.Ex10.m1.1.2.3.3.4.2.cmml" xref="A3.Ex10.m1.1.2.3.3.4.2">ℎ</ci><ci id="A3.Ex10.m1.1.2.3.3.4.3.cmml" xref="A3.Ex10.m1.1.2.3.3.4.3">𝑇</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.Ex10.m1.1c">\displaystyle=W^{(i)}_{2}-\eta\delta_{1}^{T}h^{T}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Predictor Training</h2>

<div id="A4.p1" class="ltx_para">
<p id="A4.p1.2" class="ltx_p">To improve the reconstruction and sampling efficiency, we trained an accuracy predictor <math id="A4.p1.1.m1.1" class="ltx_Math" alttext="g" display="inline"><semantics id="A4.p1.1.m1.1a"><mi id="A4.p1.1.m1.1.1" xref="A4.p1.1.m1.1.1.cmml">g</mi><annotation-xml encoding="MathML-Content" id="A4.p1.1.m1.1b"><ci id="A4.p1.1.m1.1.1.cmml" xref="A4.p1.1.m1.1.1">𝑔</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.1.m1.1c">g</annotation></semantics></math> from pretrained weights <math id="A4.p1.2.m2.1" class="ltx_Math" alttext="w" display="inline"><semantics id="A4.p1.2.m2.1a"><mi id="A4.p1.2.m2.1.1" xref="A4.p1.2.m2.1.1.cmml">w</mi><annotation-xml encoding="MathML-Content" id="A4.p1.2.m2.1b"><ci id="A4.p1.2.m2.1.1.cmml" xref="A4.p1.2.m2.1.1">𝑤</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.2.m2.1c">w</annotation></semantics></math> then use the frozen predictor during the the training of the VAE as L<sub id="A4.p1.2.1" class="ltx_sub">2</sub>-regularizer as defined in Equation <a href="#A4.E7" title="In Appendix D Predictor Training ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>.</p>
<table id="A4.E7" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="A4.E7.m1.6" class="ltx_Math" alttext="\underset{\theta,\sigma}{\min}\frac{w-f_{\theta}(w)}{\sigma^{2}}+\log\sigma^{2}+||g(w)-g(f_{\theta}(w))||^{2}," display="block"><semantics id="A4.E7.m1.6a"><mrow id="A4.E7.m1.6.6.1" xref="A4.E7.m1.6.6.1.1.cmml"><mrow id="A4.E7.m1.6.6.1.1" xref="A4.E7.m1.6.6.1.1.cmml"><mrow id="A4.E7.m1.6.6.1.1.3" xref="A4.E7.m1.6.6.1.1.3.cmml"><munder accentunder="true" id="A4.E7.m1.2.2" xref="A4.E7.m1.2.2.cmml"><mi id="A4.E7.m1.2.2.3" xref="A4.E7.m1.2.2.3.cmml">min</mi><mrow id="A4.E7.m1.2.2.2.4" xref="A4.E7.m1.2.2.2.3.cmml"><mi id="A4.E7.m1.1.1.1.1" xref="A4.E7.m1.1.1.1.1.cmml">θ</mi><mo id="A4.E7.m1.2.2.2.4.1" xref="A4.E7.m1.2.2.2.3.cmml">,</mo><mi id="A4.E7.m1.2.2.2.2" xref="A4.E7.m1.2.2.2.2.cmml">σ</mi></mrow></munder><mo lspace="0.167em" rspace="0em" id="A4.E7.m1.6.6.1.1.3.1" xref="A4.E7.m1.6.6.1.1.3.1.cmml">​</mo><mfrac id="A4.E7.m1.3.3" xref="A4.E7.m1.3.3.cmml"><mrow id="A4.E7.m1.3.3.1" xref="A4.E7.m1.3.3.1.cmml"><mi id="A4.E7.m1.3.3.1.3" xref="A4.E7.m1.3.3.1.3.cmml">w</mi><mo id="A4.E7.m1.3.3.1.2" xref="A4.E7.m1.3.3.1.2.cmml">−</mo><mrow id="A4.E7.m1.3.3.1.4" xref="A4.E7.m1.3.3.1.4.cmml"><msub id="A4.E7.m1.3.3.1.4.2" xref="A4.E7.m1.3.3.1.4.2.cmml"><mi id="A4.E7.m1.3.3.1.4.2.2" xref="A4.E7.m1.3.3.1.4.2.2.cmml">f</mi><mi id="A4.E7.m1.3.3.1.4.2.3" xref="A4.E7.m1.3.3.1.4.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="A4.E7.m1.3.3.1.4.1" xref="A4.E7.m1.3.3.1.4.1.cmml">​</mo><mrow id="A4.E7.m1.3.3.1.4.3.2" xref="A4.E7.m1.3.3.1.4.cmml"><mo stretchy="false" id="A4.E7.m1.3.3.1.4.3.2.1" xref="A4.E7.m1.3.3.1.4.cmml">(</mo><mi id="A4.E7.m1.3.3.1.1" xref="A4.E7.m1.3.3.1.1.cmml">w</mi><mo stretchy="false" id="A4.E7.m1.3.3.1.4.3.2.2" xref="A4.E7.m1.3.3.1.4.cmml">)</mo></mrow></mrow></mrow><msup id="A4.E7.m1.3.3.3" xref="A4.E7.m1.3.3.3.cmml"><mi id="A4.E7.m1.3.3.3.2" xref="A4.E7.m1.3.3.3.2.cmml">σ</mi><mn id="A4.E7.m1.3.3.3.3" xref="A4.E7.m1.3.3.3.3.cmml">2</mn></msup></mfrac></mrow><mo id="A4.E7.m1.6.6.1.1.2" xref="A4.E7.m1.6.6.1.1.2.cmml">+</mo><mrow id="A4.E7.m1.6.6.1.1.4" xref="A4.E7.m1.6.6.1.1.4.cmml"><mi id="A4.E7.m1.6.6.1.1.4.1" xref="A4.E7.m1.6.6.1.1.4.1.cmml">log</mi><mo lspace="0.167em" id="A4.E7.m1.6.6.1.1.4a" xref="A4.E7.m1.6.6.1.1.4.cmml">⁡</mo><msup id="A4.E7.m1.6.6.1.1.4.2" xref="A4.E7.m1.6.6.1.1.4.2.cmml"><mi id="A4.E7.m1.6.6.1.1.4.2.2" xref="A4.E7.m1.6.6.1.1.4.2.2.cmml">σ</mi><mn id="A4.E7.m1.6.6.1.1.4.2.3" xref="A4.E7.m1.6.6.1.1.4.2.3.cmml">2</mn></msup></mrow><mo id="A4.E7.m1.6.6.1.1.2a" xref="A4.E7.m1.6.6.1.1.2.cmml">+</mo><msup id="A4.E7.m1.6.6.1.1.1" xref="A4.E7.m1.6.6.1.1.1.cmml"><mrow id="A4.E7.m1.6.6.1.1.1.1.1" xref="A4.E7.m1.6.6.1.1.1.1.2.cmml"><mo stretchy="false" id="A4.E7.m1.6.6.1.1.1.1.1.2" xref="A4.E7.m1.6.6.1.1.1.1.2.1.cmml">‖</mo><mrow id="A4.E7.m1.6.6.1.1.1.1.1.1" xref="A4.E7.m1.6.6.1.1.1.1.1.1.cmml"><mrow id="A4.E7.m1.6.6.1.1.1.1.1.1.3" xref="A4.E7.m1.6.6.1.1.1.1.1.1.3.cmml"><mi id="A4.E7.m1.6.6.1.1.1.1.1.1.3.2" xref="A4.E7.m1.6.6.1.1.1.1.1.1.3.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="A4.E7.m1.6.6.1.1.1.1.1.1.3.1" xref="A4.E7.m1.6.6.1.1.1.1.1.1.3.1.cmml">​</mo><mrow id="A4.E7.m1.6.6.1.1.1.1.1.1.3.3.2" xref="A4.E7.m1.6.6.1.1.1.1.1.1.3.cmml"><mo stretchy="false" id="A4.E7.m1.6.6.1.1.1.1.1.1.3.3.2.1" xref="A4.E7.m1.6.6.1.1.1.1.1.1.3.cmml">(</mo><mi id="A4.E7.m1.4.4" xref="A4.E7.m1.4.4.cmml">w</mi><mo stretchy="false" id="A4.E7.m1.6.6.1.1.1.1.1.1.3.3.2.2" xref="A4.E7.m1.6.6.1.1.1.1.1.1.3.cmml">)</mo></mrow></mrow><mo id="A4.E7.m1.6.6.1.1.1.1.1.1.2" xref="A4.E7.m1.6.6.1.1.1.1.1.1.2.cmml">−</mo><mrow id="A4.E7.m1.6.6.1.1.1.1.1.1.1" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.cmml"><mi id="A4.E7.m1.6.6.1.1.1.1.1.1.1.3" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.3.cmml">g</mi><mo lspace="0em" rspace="0em" id="A4.E7.m1.6.6.1.1.1.1.1.1.1.2" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.2" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.cmml"><msub id="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.2" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.2.2" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.2.2.cmml">f</mi><mi id="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.2.3" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.1" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.1.cmml">​</mo><mrow id="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.3.2" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.3.2.1" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mi id="A4.E7.m1.5.5" xref="A4.E7.m1.5.5.cmml">w</mi><mo stretchy="false" id="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.3.2.2" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.3" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="A4.E7.m1.6.6.1.1.1.1.1.3" xref="A4.E7.m1.6.6.1.1.1.1.2.1.cmml">‖</mo></mrow><mn id="A4.E7.m1.6.6.1.1.1.3" xref="A4.E7.m1.6.6.1.1.1.3.cmml">2</mn></msup></mrow><mo id="A4.E7.m1.6.6.1.2" xref="A4.E7.m1.6.6.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="A4.E7.m1.6b"><apply id="A4.E7.m1.6.6.1.1.cmml" xref="A4.E7.m1.6.6.1"><plus id="A4.E7.m1.6.6.1.1.2.cmml" xref="A4.E7.m1.6.6.1.1.2"></plus><apply id="A4.E7.m1.6.6.1.1.3.cmml" xref="A4.E7.m1.6.6.1.1.3"><times id="A4.E7.m1.6.6.1.1.3.1.cmml" xref="A4.E7.m1.6.6.1.1.3.1"></times><apply id="A4.E7.m1.2.2.cmml" xref="A4.E7.m1.2.2"><list id="A4.E7.m1.2.2.2.3.cmml" xref="A4.E7.m1.2.2.2.4"><ci id="A4.E7.m1.1.1.1.1.cmml" xref="A4.E7.m1.1.1.1.1">𝜃</ci><ci id="A4.E7.m1.2.2.2.2.cmml" xref="A4.E7.m1.2.2.2.2">𝜎</ci></list><min id="A4.E7.m1.2.2.3.cmml" xref="A4.E7.m1.2.2.3"></min></apply><apply id="A4.E7.m1.3.3.cmml" xref="A4.E7.m1.3.3"><divide id="A4.E7.m1.3.3.2.cmml" xref="A4.E7.m1.3.3"></divide><apply id="A4.E7.m1.3.3.1.cmml" xref="A4.E7.m1.3.3.1"><minus id="A4.E7.m1.3.3.1.2.cmml" xref="A4.E7.m1.3.3.1.2"></minus><ci id="A4.E7.m1.3.3.1.3.cmml" xref="A4.E7.m1.3.3.1.3">𝑤</ci><apply id="A4.E7.m1.3.3.1.4.cmml" xref="A4.E7.m1.3.3.1.4"><times id="A4.E7.m1.3.3.1.4.1.cmml" xref="A4.E7.m1.3.3.1.4.1"></times><apply id="A4.E7.m1.3.3.1.4.2.cmml" xref="A4.E7.m1.3.3.1.4.2"><csymbol cd="ambiguous" id="A4.E7.m1.3.3.1.4.2.1.cmml" xref="A4.E7.m1.3.3.1.4.2">subscript</csymbol><ci id="A4.E7.m1.3.3.1.4.2.2.cmml" xref="A4.E7.m1.3.3.1.4.2.2">𝑓</ci><ci id="A4.E7.m1.3.3.1.4.2.3.cmml" xref="A4.E7.m1.3.3.1.4.2.3">𝜃</ci></apply><ci id="A4.E7.m1.3.3.1.1.cmml" xref="A4.E7.m1.3.3.1.1">𝑤</ci></apply></apply><apply id="A4.E7.m1.3.3.3.cmml" xref="A4.E7.m1.3.3.3"><csymbol cd="ambiguous" id="A4.E7.m1.3.3.3.1.cmml" xref="A4.E7.m1.3.3.3">superscript</csymbol><ci id="A4.E7.m1.3.3.3.2.cmml" xref="A4.E7.m1.3.3.3.2">𝜎</ci><cn type="integer" id="A4.E7.m1.3.3.3.3.cmml" xref="A4.E7.m1.3.3.3.3">2</cn></apply></apply></apply><apply id="A4.E7.m1.6.6.1.1.4.cmml" xref="A4.E7.m1.6.6.1.1.4"><log id="A4.E7.m1.6.6.1.1.4.1.cmml" xref="A4.E7.m1.6.6.1.1.4.1"></log><apply id="A4.E7.m1.6.6.1.1.4.2.cmml" xref="A4.E7.m1.6.6.1.1.4.2"><csymbol cd="ambiguous" id="A4.E7.m1.6.6.1.1.4.2.1.cmml" xref="A4.E7.m1.6.6.1.1.4.2">superscript</csymbol><ci id="A4.E7.m1.6.6.1.1.4.2.2.cmml" xref="A4.E7.m1.6.6.1.1.4.2.2">𝜎</ci><cn type="integer" id="A4.E7.m1.6.6.1.1.4.2.3.cmml" xref="A4.E7.m1.6.6.1.1.4.2.3">2</cn></apply></apply><apply id="A4.E7.m1.6.6.1.1.1.cmml" xref="A4.E7.m1.6.6.1.1.1"><csymbol cd="ambiguous" id="A4.E7.m1.6.6.1.1.1.2.cmml" xref="A4.E7.m1.6.6.1.1.1">superscript</csymbol><apply id="A4.E7.m1.6.6.1.1.1.1.2.cmml" xref="A4.E7.m1.6.6.1.1.1.1.1"><csymbol cd="latexml" id="A4.E7.m1.6.6.1.1.1.1.2.1.cmml" xref="A4.E7.m1.6.6.1.1.1.1.1.2">norm</csymbol><apply id="A4.E7.m1.6.6.1.1.1.1.1.1.cmml" xref="A4.E7.m1.6.6.1.1.1.1.1.1"><minus id="A4.E7.m1.6.6.1.1.1.1.1.1.2.cmml" xref="A4.E7.m1.6.6.1.1.1.1.1.1.2"></minus><apply id="A4.E7.m1.6.6.1.1.1.1.1.1.3.cmml" xref="A4.E7.m1.6.6.1.1.1.1.1.1.3"><times id="A4.E7.m1.6.6.1.1.1.1.1.1.3.1.cmml" xref="A4.E7.m1.6.6.1.1.1.1.1.1.3.1"></times><ci id="A4.E7.m1.6.6.1.1.1.1.1.1.3.2.cmml" xref="A4.E7.m1.6.6.1.1.1.1.1.1.3.2">𝑔</ci><ci id="A4.E7.m1.4.4.cmml" xref="A4.E7.m1.4.4">𝑤</ci></apply><apply id="A4.E7.m1.6.6.1.1.1.1.1.1.1.cmml" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1"><times id="A4.E7.m1.6.6.1.1.1.1.1.1.1.2.cmml" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.2"></times><ci id="A4.E7.m1.6.6.1.1.1.1.1.1.1.3.cmml" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.3">𝑔</ci><apply id="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.cmml" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1"><times id="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.1"></times><apply id="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.2.2">𝑓</ci><ci id="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="A4.E7.m1.6.6.1.1.1.1.1.1.1.1.1.1.2.3">𝜃</ci></apply><ci id="A4.E7.m1.5.5.cmml" xref="A4.E7.m1.5.5">𝑤</ci></apply></apply></apply></apply><cn type="integer" id="A4.E7.m1.6.6.1.1.1.3.cmml" xref="A4.E7.m1.6.6.1.1.1.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.E7.m1.6c">\underset{\theta,\sigma}{\min}\frac{w-f_{\theta}(w)}{\sigma^{2}}+\log\sigma^{2}+||g(w)-g(f_{\theta}(w))||^{2},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(7)</span></td>
</tr></tbody>
</table>
<p id="A4.p1.4" class="ltx_p">where <math id="A4.p1.3.m1.1" class="ltx_Math" alttext="g(w)" display="inline"><semantics id="A4.p1.3.m1.1a"><mrow id="A4.p1.3.m1.1.2" xref="A4.p1.3.m1.1.2.cmml"><mi id="A4.p1.3.m1.1.2.2" xref="A4.p1.3.m1.1.2.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="A4.p1.3.m1.1.2.1" xref="A4.p1.3.m1.1.2.1.cmml">​</mo><mrow id="A4.p1.3.m1.1.2.3.2" xref="A4.p1.3.m1.1.2.cmml"><mo stretchy="false" id="A4.p1.3.m1.1.2.3.2.1" xref="A4.p1.3.m1.1.2.cmml">(</mo><mi id="A4.p1.3.m1.1.1" xref="A4.p1.3.m1.1.1.cmml">w</mi><mo stretchy="false" id="A4.p1.3.m1.1.2.3.2.2" xref="A4.p1.3.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.p1.3.m1.1b"><apply id="A4.p1.3.m1.1.2.cmml" xref="A4.p1.3.m1.1.2"><times id="A4.p1.3.m1.1.2.1.cmml" xref="A4.p1.3.m1.1.2.1"></times><ci id="A4.p1.3.m1.1.2.2.cmml" xref="A4.p1.3.m1.1.2.2">𝑔</ci><ci id="A4.p1.3.m1.1.1.cmml" xref="A4.p1.3.m1.1.1">𝑤</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.3.m1.1c">g(w)</annotation></semantics></math> is the embedding of the original input and <math id="A4.p1.4.m2.2" class="ltx_Math" alttext="g(f_{\theta}(w))" display="inline"><semantics id="A4.p1.4.m2.2a"><mrow id="A4.p1.4.m2.2.2" xref="A4.p1.4.m2.2.2.cmml"><mi id="A4.p1.4.m2.2.2.3" xref="A4.p1.4.m2.2.2.3.cmml">g</mi><mo lspace="0em" rspace="0em" id="A4.p1.4.m2.2.2.2" xref="A4.p1.4.m2.2.2.2.cmml">​</mo><mrow id="A4.p1.4.m2.2.2.1.1" xref="A4.p1.4.m2.2.2.1.1.1.cmml"><mo stretchy="false" id="A4.p1.4.m2.2.2.1.1.2" xref="A4.p1.4.m2.2.2.1.1.1.cmml">(</mo><mrow id="A4.p1.4.m2.2.2.1.1.1" xref="A4.p1.4.m2.2.2.1.1.1.cmml"><msub id="A4.p1.4.m2.2.2.1.1.1.2" xref="A4.p1.4.m2.2.2.1.1.1.2.cmml"><mi id="A4.p1.4.m2.2.2.1.1.1.2.2" xref="A4.p1.4.m2.2.2.1.1.1.2.2.cmml">f</mi><mi id="A4.p1.4.m2.2.2.1.1.1.2.3" xref="A4.p1.4.m2.2.2.1.1.1.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="A4.p1.4.m2.2.2.1.1.1.1" xref="A4.p1.4.m2.2.2.1.1.1.1.cmml">​</mo><mrow id="A4.p1.4.m2.2.2.1.1.1.3.2" xref="A4.p1.4.m2.2.2.1.1.1.cmml"><mo stretchy="false" id="A4.p1.4.m2.2.2.1.1.1.3.2.1" xref="A4.p1.4.m2.2.2.1.1.1.cmml">(</mo><mi id="A4.p1.4.m2.1.1" xref="A4.p1.4.m2.1.1.cmml">w</mi><mo stretchy="false" id="A4.p1.4.m2.2.2.1.1.1.3.2.2" xref="A4.p1.4.m2.2.2.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="A4.p1.4.m2.2.2.1.1.3" xref="A4.p1.4.m2.2.2.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.p1.4.m2.2b"><apply id="A4.p1.4.m2.2.2.cmml" xref="A4.p1.4.m2.2.2"><times id="A4.p1.4.m2.2.2.2.cmml" xref="A4.p1.4.m2.2.2.2"></times><ci id="A4.p1.4.m2.2.2.3.cmml" xref="A4.p1.4.m2.2.2.3">𝑔</ci><apply id="A4.p1.4.m2.2.2.1.1.1.cmml" xref="A4.p1.4.m2.2.2.1.1"><times id="A4.p1.4.m2.2.2.1.1.1.1.cmml" xref="A4.p1.4.m2.2.2.1.1.1.1"></times><apply id="A4.p1.4.m2.2.2.1.1.1.2.cmml" xref="A4.p1.4.m2.2.2.1.1.1.2"><csymbol cd="ambiguous" id="A4.p1.4.m2.2.2.1.1.1.2.1.cmml" xref="A4.p1.4.m2.2.2.1.1.1.2">subscript</csymbol><ci id="A4.p1.4.m2.2.2.1.1.1.2.2.cmml" xref="A4.p1.4.m2.2.2.1.1.1.2.2">𝑓</ci><ci id="A4.p1.4.m2.2.2.1.1.1.2.3.cmml" xref="A4.p1.4.m2.2.2.1.1.1.2.3">𝜃</ci></apply><ci id="A4.p1.4.m2.1.1.cmml" xref="A4.p1.4.m2.1.1">𝑤</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.4.m2.2c">g(f_{\theta}(w))</annotation></semantics></math> is the predictor embedding of the reconstructed weights. The predictor can be either dataset-conditioned or unconditioned. In general we found that dataset-conditioned predictor works only well for large number of samples per dataset. After the AutoEncoder is trained, we train the dataset-conditioned module which requires a dataset encoder.</p>
</div>
<figure id="alg2" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg2.2.1.1" class="ltx_text ltx_font_bold">Algorithm 2</span> </span> predictor-guided VAE</figcaption>
<div id="alg2.3" class="ltx_listing ltx_listing">
<div id="alg2.l1" class="ltx_listingline">  <span id="alg2.l1.1" class="ltx_text ltx_font_bold">Input:</span> pretrained weights <math id="alg2.l1.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="alg2.l1.m1.1a"><mi id="alg2.l1.m1.1.1" xref="alg2.l1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="alg2.l1.m1.1b"><ci id="alg2.l1.m1.1.1.cmml" xref="alg2.l1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l1.m1.1c">x</annotation></semantics></math>, accuracy <math id="alg2.l1.m2.1" class="ltx_Math" alttext="y" display="inline"><semantics id="alg2.l1.m2.1a"><mi id="alg2.l1.m2.1.1" xref="alg2.l1.m2.1.1.cmml">y</mi><annotation-xml encoding="MathML-Content" id="alg2.l1.m2.1b"><ci id="alg2.l1.m2.1.1.cmml" xref="alg2.l1.m2.1.1">𝑦</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l1.m2.1c">y</annotation></semantics></math>, batch_num <math id="alg2.l1.m3.1" class="ltx_Math" alttext="m" display="inline"><semantics id="alg2.l1.m3.1a"><mi id="alg2.l1.m3.1.1" xref="alg2.l1.m3.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="alg2.l1.m3.1b"><ci id="alg2.l1.m3.1.1.cmml" xref="alg2.l1.m3.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l1.m3.1c">m</annotation></semantics></math>

</div>
<div id="alg2.l2" class="ltx_listingline">  Instanciate <math id="alg2.l2.m1.1" class="ltx_Math" alttext="\mathcal{f}=set\_transformer" display="inline"><semantics id="alg2.l2.m1.1a"><mrow id="alg2.l2.m1.1.1" xref="alg2.l2.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg2.l2.m1.1.1.2" xref="alg2.l2.m1.1.1.2.cmml">𝒻</mi><mo id="alg2.l2.m1.1.1.1" xref="alg2.l2.m1.1.1.1.cmml">=</mo><mrow id="alg2.l2.m1.1.1.3" xref="alg2.l2.m1.1.1.3.cmml"><mi id="alg2.l2.m1.1.1.3.2" xref="alg2.l2.m1.1.1.3.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.1" xref="alg2.l2.m1.1.1.3.1.cmml">​</mo><mi id="alg2.l2.m1.1.1.3.3" xref="alg2.l2.m1.1.1.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.1a" xref="alg2.l2.m1.1.1.3.1.cmml">​</mo><mi id="alg2.l2.m1.1.1.3.4" xref="alg2.l2.m1.1.1.3.4.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.1b" xref="alg2.l2.m1.1.1.3.1.cmml">​</mo><mi mathvariant="normal" id="alg2.l2.m1.1.1.3.5" xref="alg2.l2.m1.1.1.3.5.cmml">_</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.1c" xref="alg2.l2.m1.1.1.3.1.cmml">​</mo><mi id="alg2.l2.m1.1.1.3.6" xref="alg2.l2.m1.1.1.3.6.cmml">t</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.1d" xref="alg2.l2.m1.1.1.3.1.cmml">​</mo><mi id="alg2.l2.m1.1.1.3.7" xref="alg2.l2.m1.1.1.3.7.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.1e" xref="alg2.l2.m1.1.1.3.1.cmml">​</mo><mi id="alg2.l2.m1.1.1.3.8" xref="alg2.l2.m1.1.1.3.8.cmml">a</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.1f" xref="alg2.l2.m1.1.1.3.1.cmml">​</mo><mi id="alg2.l2.m1.1.1.3.9" xref="alg2.l2.m1.1.1.3.9.cmml">n</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.1g" xref="alg2.l2.m1.1.1.3.1.cmml">​</mo><mi id="alg2.l2.m1.1.1.3.10" xref="alg2.l2.m1.1.1.3.10.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.1h" xref="alg2.l2.m1.1.1.3.1.cmml">​</mo><mi id="alg2.l2.m1.1.1.3.11" xref="alg2.l2.m1.1.1.3.11.cmml">f</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.1i" xref="alg2.l2.m1.1.1.3.1.cmml">​</mo><mi id="alg2.l2.m1.1.1.3.12" xref="alg2.l2.m1.1.1.3.12.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.1j" xref="alg2.l2.m1.1.1.3.1.cmml">​</mo><mi id="alg2.l2.m1.1.1.3.13" xref="alg2.l2.m1.1.1.3.13.cmml">r</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.1k" xref="alg2.l2.m1.1.1.3.1.cmml">​</mo><mi id="alg2.l2.m1.1.1.3.14" xref="alg2.l2.m1.1.1.3.14.cmml">m</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.1l" xref="alg2.l2.m1.1.1.3.1.cmml">​</mo><mi id="alg2.l2.m1.1.1.3.15" xref="alg2.l2.m1.1.1.3.15.cmml">e</mi><mo lspace="0em" rspace="0em" id="alg2.l2.m1.1.1.3.1m" xref="alg2.l2.m1.1.1.3.1.cmml">​</mo><mi id="alg2.l2.m1.1.1.3.16" xref="alg2.l2.m1.1.1.3.16.cmml">r</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l2.m1.1b"><apply id="alg2.l2.m1.1.1.cmml" xref="alg2.l2.m1.1.1"><eq id="alg2.l2.m1.1.1.1.cmml" xref="alg2.l2.m1.1.1.1"></eq><ci id="alg2.l2.m1.1.1.2.cmml" xref="alg2.l2.m1.1.1.2">𝒻</ci><apply id="alg2.l2.m1.1.1.3.cmml" xref="alg2.l2.m1.1.1.3"><times id="alg2.l2.m1.1.1.3.1.cmml" xref="alg2.l2.m1.1.1.3.1"></times><ci id="alg2.l2.m1.1.1.3.2.cmml" xref="alg2.l2.m1.1.1.3.2">𝑠</ci><ci id="alg2.l2.m1.1.1.3.3.cmml" xref="alg2.l2.m1.1.1.3.3">𝑒</ci><ci id="alg2.l2.m1.1.1.3.4.cmml" xref="alg2.l2.m1.1.1.3.4">𝑡</ci><ci id="alg2.l2.m1.1.1.3.5.cmml" xref="alg2.l2.m1.1.1.3.5">_</ci><ci id="alg2.l2.m1.1.1.3.6.cmml" xref="alg2.l2.m1.1.1.3.6">𝑡</ci><ci id="alg2.l2.m1.1.1.3.7.cmml" xref="alg2.l2.m1.1.1.3.7">𝑟</ci><ci id="alg2.l2.m1.1.1.3.8.cmml" xref="alg2.l2.m1.1.1.3.8">𝑎</ci><ci id="alg2.l2.m1.1.1.3.9.cmml" xref="alg2.l2.m1.1.1.3.9">𝑛</ci><ci id="alg2.l2.m1.1.1.3.10.cmml" xref="alg2.l2.m1.1.1.3.10">𝑠</ci><ci id="alg2.l2.m1.1.1.3.11.cmml" xref="alg2.l2.m1.1.1.3.11">𝑓</ci><ci id="alg2.l2.m1.1.1.3.12.cmml" xref="alg2.l2.m1.1.1.3.12">𝑜</ci><ci id="alg2.l2.m1.1.1.3.13.cmml" xref="alg2.l2.m1.1.1.3.13">𝑟</ci><ci id="alg2.l2.m1.1.1.3.14.cmml" xref="alg2.l2.m1.1.1.3.14">𝑚</ci><ci id="alg2.l2.m1.1.1.3.15.cmml" xref="alg2.l2.m1.1.1.3.15">𝑒</ci><ci id="alg2.l2.m1.1.1.3.16.cmml" xref="alg2.l2.m1.1.1.3.16">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l2.m1.1c">\mathcal{f}=set\_transformer</annotation></semantics></math>, Load pretrained predictor (<math id="alg2.l2.m2.1" class="ltx_Math" alttext="\mathcal{g}" display="inline"><semantics id="alg2.l2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="alg2.l2.m2.1.1" xref="alg2.l2.m2.1.1.cmml">ℊ</mi><annotation-xml encoding="MathML-Content" id="alg2.l2.m2.1b"><ci id="alg2.l2.m2.1.1.cmml" xref="alg2.l2.m2.1.1">ℊ</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l2.m2.1c">\mathcal{g}</annotation></semantics></math>).

</div>
<div id="alg2.l3" class="ltx_listingline">  <span id="alg2.l3.1" class="ltx_text ltx_font_bold">repeat</span>


</div>
<div id="alg2.l4" class="ltx_listingline">     Initialize <math id="alg2.l4.m1.1" class="ltx_Math" alttext="loss=0.0" display="inline"><semantics id="alg2.l4.m1.1a"><mrow id="alg2.l4.m1.1.1" xref="alg2.l4.m1.1.1.cmml"><mrow id="alg2.l4.m1.1.1.2" xref="alg2.l4.m1.1.1.2.cmml"><mi id="alg2.l4.m1.1.1.2.2" xref="alg2.l4.m1.1.1.2.2.cmml">l</mi><mo lspace="0em" rspace="0em" id="alg2.l4.m1.1.1.2.1" xref="alg2.l4.m1.1.1.2.1.cmml">​</mo><mi id="alg2.l4.m1.1.1.2.3" xref="alg2.l4.m1.1.1.2.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="alg2.l4.m1.1.1.2.1a" xref="alg2.l4.m1.1.1.2.1.cmml">​</mo><mi id="alg2.l4.m1.1.1.2.4" xref="alg2.l4.m1.1.1.2.4.cmml">s</mi><mo lspace="0em" rspace="0em" id="alg2.l4.m1.1.1.2.1b" xref="alg2.l4.m1.1.1.2.1.cmml">​</mo><mi id="alg2.l4.m1.1.1.2.5" xref="alg2.l4.m1.1.1.2.5.cmml">s</mi></mrow><mo id="alg2.l4.m1.1.1.1" xref="alg2.l4.m1.1.1.1.cmml">=</mo><mn id="alg2.l4.m1.1.1.3" xref="alg2.l4.m1.1.1.3.cmml">0.0</mn></mrow><annotation-xml encoding="MathML-Content" id="alg2.l4.m1.1b"><apply id="alg2.l4.m1.1.1.cmml" xref="alg2.l4.m1.1.1"><eq id="alg2.l4.m1.1.1.1.cmml" xref="alg2.l4.m1.1.1.1"></eq><apply id="alg2.l4.m1.1.1.2.cmml" xref="alg2.l4.m1.1.1.2"><times id="alg2.l4.m1.1.1.2.1.cmml" xref="alg2.l4.m1.1.1.2.1"></times><ci id="alg2.l4.m1.1.1.2.2.cmml" xref="alg2.l4.m1.1.1.2.2">𝑙</ci><ci id="alg2.l4.m1.1.1.2.3.cmml" xref="alg2.l4.m1.1.1.2.3">𝑜</ci><ci id="alg2.l4.m1.1.1.2.4.cmml" xref="alg2.l4.m1.1.1.2.4">𝑠</ci><ci id="alg2.l4.m1.1.1.2.5.cmml" xref="alg2.l4.m1.1.1.2.5">𝑠</ci></apply><cn type="float" id="alg2.l4.m1.1.1.3.cmml" xref="alg2.l4.m1.1.1.3">0.0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l4.m1.1c">loss=0.0</annotation></semantics></math>

</div>
<div id="alg2.l5" class="ltx_listingline">     <span id="alg2.l5.1" class="ltx_text ltx_font_bold">for</span> <math id="alg2.l5.m1.1" class="ltx_Math" alttext="i=1" display="inline"><semantics id="alg2.l5.m1.1a"><mrow id="alg2.l5.m1.1.1" xref="alg2.l5.m1.1.1.cmml"><mi id="alg2.l5.m1.1.1.2" xref="alg2.l5.m1.1.1.2.cmml">i</mi><mo id="alg2.l5.m1.1.1.1" xref="alg2.l5.m1.1.1.1.cmml">=</mo><mn id="alg2.l5.m1.1.1.3" xref="alg2.l5.m1.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg2.l5.m1.1b"><apply id="alg2.l5.m1.1.1.cmml" xref="alg2.l5.m1.1.1"><eq id="alg2.l5.m1.1.1.1.cmml" xref="alg2.l5.m1.1.1.1"></eq><ci id="alg2.l5.m1.1.1.2.cmml" xref="alg2.l5.m1.1.1.2">𝑖</ci><cn type="integer" id="alg2.l5.m1.1.1.3.cmml" xref="alg2.l5.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l5.m1.1c">i=1</annotation></semantics></math> <span id="alg2.l5.2" class="ltx_text ltx_font_bold">to</span> <math id="alg2.l5.m2.1" class="ltx_Math" alttext="m-1" display="inline"><semantics id="alg2.l5.m2.1a"><mrow id="alg2.l5.m2.1.1" xref="alg2.l5.m2.1.1.cmml"><mi id="alg2.l5.m2.1.1.2" xref="alg2.l5.m2.1.1.2.cmml">m</mi><mo id="alg2.l5.m2.1.1.1" xref="alg2.l5.m2.1.1.1.cmml">−</mo><mn id="alg2.l5.m2.1.1.3" xref="alg2.l5.m2.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="alg2.l5.m2.1b"><apply id="alg2.l5.m2.1.1.cmml" xref="alg2.l5.m2.1.1"><minus id="alg2.l5.m2.1.1.1.cmml" xref="alg2.l5.m2.1.1.1"></minus><ci id="alg2.l5.m2.1.1.2.cmml" xref="alg2.l5.m2.1.1.2">𝑚</ci><cn type="integer" id="alg2.l5.m2.1.1.3.cmml" xref="alg2.l5.m2.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l5.m2.1c">m-1</annotation></semantics></math> <span id="alg2.l5.3" class="ltx_text ltx_font_bold">do</span>



</div>
<div id="alg2.l6" class="ltx_listingline">        <math id="alg2.l6.m1.1" class="ltx_Math" alttext="\bar{x}=f_{\theta}(x)" display="inline"><semantics id="alg2.l6.m1.1a"><mrow id="alg2.l6.m1.1.2" xref="alg2.l6.m1.1.2.cmml"><mover accent="true" id="alg2.l6.m1.1.2.2" xref="alg2.l6.m1.1.2.2.cmml"><mi id="alg2.l6.m1.1.2.2.2" xref="alg2.l6.m1.1.2.2.2.cmml">x</mi><mo id="alg2.l6.m1.1.2.2.1" xref="alg2.l6.m1.1.2.2.1.cmml">¯</mo></mover><mo id="alg2.l6.m1.1.2.1" xref="alg2.l6.m1.1.2.1.cmml">=</mo><mrow id="alg2.l6.m1.1.2.3" xref="alg2.l6.m1.1.2.3.cmml"><msub id="alg2.l6.m1.1.2.3.2" xref="alg2.l6.m1.1.2.3.2.cmml"><mi id="alg2.l6.m1.1.2.3.2.2" xref="alg2.l6.m1.1.2.3.2.2.cmml">f</mi><mi id="alg2.l6.m1.1.2.3.2.3" xref="alg2.l6.m1.1.2.3.2.3.cmml">θ</mi></msub><mo lspace="0em" rspace="0em" id="alg2.l6.m1.1.2.3.1" xref="alg2.l6.m1.1.2.3.1.cmml">​</mo><mrow id="alg2.l6.m1.1.2.3.3.2" xref="alg2.l6.m1.1.2.3.cmml"><mo stretchy="false" id="alg2.l6.m1.1.2.3.3.2.1" xref="alg2.l6.m1.1.2.3.cmml">(</mo><mi id="alg2.l6.m1.1.1" xref="alg2.l6.m1.1.1.cmml">x</mi><mo stretchy="false" id="alg2.l6.m1.1.2.3.3.2.2" xref="alg2.l6.m1.1.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l6.m1.1b"><apply id="alg2.l6.m1.1.2.cmml" xref="alg2.l6.m1.1.2"><eq id="alg2.l6.m1.1.2.1.cmml" xref="alg2.l6.m1.1.2.1"></eq><apply id="alg2.l6.m1.1.2.2.cmml" xref="alg2.l6.m1.1.2.2"><ci id="alg2.l6.m1.1.2.2.1.cmml" xref="alg2.l6.m1.1.2.2.1">¯</ci><ci id="alg2.l6.m1.1.2.2.2.cmml" xref="alg2.l6.m1.1.2.2.2">𝑥</ci></apply><apply id="alg2.l6.m1.1.2.3.cmml" xref="alg2.l6.m1.1.2.3"><times id="alg2.l6.m1.1.2.3.1.cmml" xref="alg2.l6.m1.1.2.3.1"></times><apply id="alg2.l6.m1.1.2.3.2.cmml" xref="alg2.l6.m1.1.2.3.2"><csymbol cd="ambiguous" id="alg2.l6.m1.1.2.3.2.1.cmml" xref="alg2.l6.m1.1.2.3.2">subscript</csymbol><ci id="alg2.l6.m1.1.2.3.2.2.cmml" xref="alg2.l6.m1.1.2.3.2.2">𝑓</ci><ci id="alg2.l6.m1.1.2.3.2.3.cmml" xref="alg2.l6.m1.1.2.3.2.3">𝜃</ci></apply><ci id="alg2.l6.m1.1.1.cmml" xref="alg2.l6.m1.1.1">𝑥</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l6.m1.1c">\bar{x}=f_{\theta}(x)</annotation></semantics></math>, <math id="alg2.l6.m2.1" class="ltx_Math" alttext="\bar{y}=g(\bar{x})" display="inline"><semantics id="alg2.l6.m2.1a"><mrow id="alg2.l6.m2.1.2" xref="alg2.l6.m2.1.2.cmml"><mover accent="true" id="alg2.l6.m2.1.2.2" xref="alg2.l6.m2.1.2.2.cmml"><mi id="alg2.l6.m2.1.2.2.2" xref="alg2.l6.m2.1.2.2.2.cmml">y</mi><mo id="alg2.l6.m2.1.2.2.1" xref="alg2.l6.m2.1.2.2.1.cmml">¯</mo></mover><mo id="alg2.l6.m2.1.2.1" xref="alg2.l6.m2.1.2.1.cmml">=</mo><mrow id="alg2.l6.m2.1.2.3" xref="alg2.l6.m2.1.2.3.cmml"><mi id="alg2.l6.m2.1.2.3.2" xref="alg2.l6.m2.1.2.3.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="alg2.l6.m2.1.2.3.1" xref="alg2.l6.m2.1.2.3.1.cmml">​</mo><mrow id="alg2.l6.m2.1.2.3.3.2" xref="alg2.l6.m2.1.1.cmml"><mo stretchy="false" id="alg2.l6.m2.1.2.3.3.2.1" xref="alg2.l6.m2.1.1.cmml">(</mo><mover accent="true" id="alg2.l6.m2.1.1" xref="alg2.l6.m2.1.1.cmml"><mi id="alg2.l6.m2.1.1.2" xref="alg2.l6.m2.1.1.2.cmml">x</mi><mo id="alg2.l6.m2.1.1.1" xref="alg2.l6.m2.1.1.1.cmml">¯</mo></mover><mo stretchy="false" id="alg2.l6.m2.1.2.3.3.2.2" xref="alg2.l6.m2.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l6.m2.1b"><apply id="alg2.l6.m2.1.2.cmml" xref="alg2.l6.m2.1.2"><eq id="alg2.l6.m2.1.2.1.cmml" xref="alg2.l6.m2.1.2.1"></eq><apply id="alg2.l6.m2.1.2.2.cmml" xref="alg2.l6.m2.1.2.2"><ci id="alg2.l6.m2.1.2.2.1.cmml" xref="alg2.l6.m2.1.2.2.1">¯</ci><ci id="alg2.l6.m2.1.2.2.2.cmml" xref="alg2.l6.m2.1.2.2.2">𝑦</ci></apply><apply id="alg2.l6.m2.1.2.3.cmml" xref="alg2.l6.m2.1.2.3"><times id="alg2.l6.m2.1.2.3.1.cmml" xref="alg2.l6.m2.1.2.3.1"></times><ci id="alg2.l6.m2.1.2.3.2.cmml" xref="alg2.l6.m2.1.2.3.2">𝑔</ci><apply id="alg2.l6.m2.1.1.cmml" xref="alg2.l6.m2.1.2.3.3.2"><ci id="alg2.l6.m2.1.1.1.cmml" xref="alg2.l6.m2.1.1.1">¯</ci><ci id="alg2.l6.m2.1.1.2.cmml" xref="alg2.l6.m2.1.1.2">𝑥</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l6.m2.1c">\bar{y}=g(\bar{x})</annotation></semantics></math> <math id="alg2.l6.m3.1" class="ltx_Math" alttext="\hat{y}=g(x)" display="inline"><semantics id="alg2.l6.m3.1a"><mrow id="alg2.l6.m3.1.2" xref="alg2.l6.m3.1.2.cmml"><mover accent="true" id="alg2.l6.m3.1.2.2" xref="alg2.l6.m3.1.2.2.cmml"><mi id="alg2.l6.m3.1.2.2.2" xref="alg2.l6.m3.1.2.2.2.cmml">y</mi><mo id="alg2.l6.m3.1.2.2.1" xref="alg2.l6.m3.1.2.2.1.cmml">^</mo></mover><mo id="alg2.l6.m3.1.2.1" xref="alg2.l6.m3.1.2.1.cmml">=</mo><mrow id="alg2.l6.m3.1.2.3" xref="alg2.l6.m3.1.2.3.cmml"><mi id="alg2.l6.m3.1.2.3.2" xref="alg2.l6.m3.1.2.3.2.cmml">g</mi><mo lspace="0em" rspace="0em" id="alg2.l6.m3.1.2.3.1" xref="alg2.l6.m3.1.2.3.1.cmml">​</mo><mrow id="alg2.l6.m3.1.2.3.3.2" xref="alg2.l6.m3.1.2.3.cmml"><mo stretchy="false" id="alg2.l6.m3.1.2.3.3.2.1" xref="alg2.l6.m3.1.2.3.cmml">(</mo><mi id="alg2.l6.m3.1.1" xref="alg2.l6.m3.1.1.cmml">x</mi><mo stretchy="false" id="alg2.l6.m3.1.2.3.3.2.2" xref="alg2.l6.m3.1.2.3.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg2.l6.m3.1b"><apply id="alg2.l6.m3.1.2.cmml" xref="alg2.l6.m3.1.2"><eq id="alg2.l6.m3.1.2.1.cmml" xref="alg2.l6.m3.1.2.1"></eq><apply id="alg2.l6.m3.1.2.2.cmml" xref="alg2.l6.m3.1.2.2"><ci id="alg2.l6.m3.1.2.2.1.cmml" xref="alg2.l6.m3.1.2.2.1">^</ci><ci id="alg2.l6.m3.1.2.2.2.cmml" xref="alg2.l6.m3.1.2.2.2">𝑦</ci></apply><apply id="alg2.l6.m3.1.2.3.cmml" xref="alg2.l6.m3.1.2.3"><times id="alg2.l6.m3.1.2.3.1.cmml" xref="alg2.l6.m3.1.2.3.1"></times><ci id="alg2.l6.m3.1.2.3.2.cmml" xref="alg2.l6.m3.1.2.3.2">𝑔</ci><ci id="alg2.l6.m3.1.1.cmml" xref="alg2.l6.m3.1.1">𝑥</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l6.m3.1c">\hat{y}=g(x)</annotation></semantics></math>

</div>
<div id="alg2.l7" class="ltx_listingline">        <math id="alg2.l7.m1.1" class="ltx_Math" alttext="\underset{\theta}{L}\frac{x-\bar{x}}{\sigma^{2}}+\log\sigma^{2}+||\hat{y}-\bar{y}||^{2}" display="inline"><semantics id="alg2.l7.m1.1a"><mrow id="alg2.l7.m1.1.1" xref="alg2.l7.m1.1.1.cmml"><mrow id="alg2.l7.m1.1.1.3" xref="alg2.l7.m1.1.1.3.cmml"><munder accentunder="true" id="alg2.l7.m1.1.1.3.2" xref="alg2.l7.m1.1.1.3.2.cmml"><mi id="alg2.l7.m1.1.1.3.2.2" xref="alg2.l7.m1.1.1.3.2.2.cmml">L</mi><mo id="alg2.l7.m1.1.1.3.2.1" xref="alg2.l7.m1.1.1.3.2.1.cmml">𝜃</mo></munder><mo lspace="0em" rspace="0em" id="alg2.l7.m1.1.1.3.1" xref="alg2.l7.m1.1.1.3.1.cmml">​</mo><mfrac id="alg2.l7.m1.1.1.3.3" xref="alg2.l7.m1.1.1.3.3.cmml"><mrow id="alg2.l7.m1.1.1.3.3.2" xref="alg2.l7.m1.1.1.3.3.2.cmml"><mi id="alg2.l7.m1.1.1.3.3.2.2" xref="alg2.l7.m1.1.1.3.3.2.2.cmml">x</mi><mo id="alg2.l7.m1.1.1.3.3.2.1" xref="alg2.l7.m1.1.1.3.3.2.1.cmml">−</mo><mover accent="true" id="alg2.l7.m1.1.1.3.3.2.3" xref="alg2.l7.m1.1.1.3.3.2.3.cmml"><mi id="alg2.l7.m1.1.1.3.3.2.3.2" xref="alg2.l7.m1.1.1.3.3.2.3.2.cmml">x</mi><mo id="alg2.l7.m1.1.1.3.3.2.3.1" xref="alg2.l7.m1.1.1.3.3.2.3.1.cmml">¯</mo></mover></mrow><msup id="alg2.l7.m1.1.1.3.3.3" xref="alg2.l7.m1.1.1.3.3.3.cmml"><mi id="alg2.l7.m1.1.1.3.3.3.2" xref="alg2.l7.m1.1.1.3.3.3.2.cmml">σ</mi><mn id="alg2.l7.m1.1.1.3.3.3.3" xref="alg2.l7.m1.1.1.3.3.3.3.cmml">2</mn></msup></mfrac></mrow><mo id="alg2.l7.m1.1.1.2" xref="alg2.l7.m1.1.1.2.cmml">+</mo><mrow id="alg2.l7.m1.1.1.4" xref="alg2.l7.m1.1.1.4.cmml"><mi id="alg2.l7.m1.1.1.4.1" xref="alg2.l7.m1.1.1.4.1.cmml">log</mi><mo lspace="0.167em" id="alg2.l7.m1.1.1.4a" xref="alg2.l7.m1.1.1.4.cmml">⁡</mo><msup id="alg2.l7.m1.1.1.4.2" xref="alg2.l7.m1.1.1.4.2.cmml"><mi id="alg2.l7.m1.1.1.4.2.2" xref="alg2.l7.m1.1.1.4.2.2.cmml">σ</mi><mn id="alg2.l7.m1.1.1.4.2.3" xref="alg2.l7.m1.1.1.4.2.3.cmml">2</mn></msup></mrow><mo id="alg2.l7.m1.1.1.2a" xref="alg2.l7.m1.1.1.2.cmml">+</mo><msup id="alg2.l7.m1.1.1.1" xref="alg2.l7.m1.1.1.1.cmml"><mrow id="alg2.l7.m1.1.1.1.1.1" xref="alg2.l7.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="alg2.l7.m1.1.1.1.1.1.2" xref="alg2.l7.m1.1.1.1.1.2.1.cmml">‖</mo><mrow id="alg2.l7.m1.1.1.1.1.1.1" xref="alg2.l7.m1.1.1.1.1.1.1.cmml"><mover accent="true" id="alg2.l7.m1.1.1.1.1.1.1.2" xref="alg2.l7.m1.1.1.1.1.1.1.2.cmml"><mi id="alg2.l7.m1.1.1.1.1.1.1.2.2" xref="alg2.l7.m1.1.1.1.1.1.1.2.2.cmml">y</mi><mo id="alg2.l7.m1.1.1.1.1.1.1.2.1" xref="alg2.l7.m1.1.1.1.1.1.1.2.1.cmml">^</mo></mover><mo id="alg2.l7.m1.1.1.1.1.1.1.1" xref="alg2.l7.m1.1.1.1.1.1.1.1.cmml">−</mo><mover accent="true" id="alg2.l7.m1.1.1.1.1.1.1.3" xref="alg2.l7.m1.1.1.1.1.1.1.3.cmml"><mi id="alg2.l7.m1.1.1.1.1.1.1.3.2" xref="alg2.l7.m1.1.1.1.1.1.1.3.2.cmml">y</mi><mo id="alg2.l7.m1.1.1.1.1.1.1.3.1" xref="alg2.l7.m1.1.1.1.1.1.1.3.1.cmml">¯</mo></mover></mrow><mo stretchy="false" id="alg2.l7.m1.1.1.1.1.1.3" xref="alg2.l7.m1.1.1.1.1.2.1.cmml">‖</mo></mrow><mn id="alg2.l7.m1.1.1.1.3" xref="alg2.l7.m1.1.1.1.3.cmml">2</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="alg2.l7.m1.1b"><apply id="alg2.l7.m1.1.1.cmml" xref="alg2.l7.m1.1.1"><plus id="alg2.l7.m1.1.1.2.cmml" xref="alg2.l7.m1.1.1.2"></plus><apply id="alg2.l7.m1.1.1.3.cmml" xref="alg2.l7.m1.1.1.3"><times id="alg2.l7.m1.1.1.3.1.cmml" xref="alg2.l7.m1.1.1.3.1"></times><apply id="alg2.l7.m1.1.1.3.2.cmml" xref="alg2.l7.m1.1.1.3.2"><ci id="alg2.l7.m1.1.1.3.2.1.cmml" xref="alg2.l7.m1.1.1.3.2.1">𝜃</ci><ci id="alg2.l7.m1.1.1.3.2.2.cmml" xref="alg2.l7.m1.1.1.3.2.2">𝐿</ci></apply><apply id="alg2.l7.m1.1.1.3.3.cmml" xref="alg2.l7.m1.1.1.3.3"><divide id="alg2.l7.m1.1.1.3.3.1.cmml" xref="alg2.l7.m1.1.1.3.3"></divide><apply id="alg2.l7.m1.1.1.3.3.2.cmml" xref="alg2.l7.m1.1.1.3.3.2"><minus id="alg2.l7.m1.1.1.3.3.2.1.cmml" xref="alg2.l7.m1.1.1.3.3.2.1"></minus><ci id="alg2.l7.m1.1.1.3.3.2.2.cmml" xref="alg2.l7.m1.1.1.3.3.2.2">𝑥</ci><apply id="alg2.l7.m1.1.1.3.3.2.3.cmml" xref="alg2.l7.m1.1.1.3.3.2.3"><ci id="alg2.l7.m1.1.1.3.3.2.3.1.cmml" xref="alg2.l7.m1.1.1.3.3.2.3.1">¯</ci><ci id="alg2.l7.m1.1.1.3.3.2.3.2.cmml" xref="alg2.l7.m1.1.1.3.3.2.3.2">𝑥</ci></apply></apply><apply id="alg2.l7.m1.1.1.3.3.3.cmml" xref="alg2.l7.m1.1.1.3.3.3"><csymbol cd="ambiguous" id="alg2.l7.m1.1.1.3.3.3.1.cmml" xref="alg2.l7.m1.1.1.3.3.3">superscript</csymbol><ci id="alg2.l7.m1.1.1.3.3.3.2.cmml" xref="alg2.l7.m1.1.1.3.3.3.2">𝜎</ci><cn type="integer" id="alg2.l7.m1.1.1.3.3.3.3.cmml" xref="alg2.l7.m1.1.1.3.3.3.3">2</cn></apply></apply></apply><apply id="alg2.l7.m1.1.1.4.cmml" xref="alg2.l7.m1.1.1.4"><log id="alg2.l7.m1.1.1.4.1.cmml" xref="alg2.l7.m1.1.1.4.1"></log><apply id="alg2.l7.m1.1.1.4.2.cmml" xref="alg2.l7.m1.1.1.4.2"><csymbol cd="ambiguous" id="alg2.l7.m1.1.1.4.2.1.cmml" xref="alg2.l7.m1.1.1.4.2">superscript</csymbol><ci id="alg2.l7.m1.1.1.4.2.2.cmml" xref="alg2.l7.m1.1.1.4.2.2">𝜎</ci><cn type="integer" id="alg2.l7.m1.1.1.4.2.3.cmml" xref="alg2.l7.m1.1.1.4.2.3">2</cn></apply></apply><apply id="alg2.l7.m1.1.1.1.cmml" xref="alg2.l7.m1.1.1.1"><csymbol cd="ambiguous" id="alg2.l7.m1.1.1.1.2.cmml" xref="alg2.l7.m1.1.1.1">superscript</csymbol><apply id="alg2.l7.m1.1.1.1.1.2.cmml" xref="alg2.l7.m1.1.1.1.1.1"><csymbol cd="latexml" id="alg2.l7.m1.1.1.1.1.2.1.cmml" xref="alg2.l7.m1.1.1.1.1.1.2">norm</csymbol><apply id="alg2.l7.m1.1.1.1.1.1.1.cmml" xref="alg2.l7.m1.1.1.1.1.1.1"><minus id="alg2.l7.m1.1.1.1.1.1.1.1.cmml" xref="alg2.l7.m1.1.1.1.1.1.1.1"></minus><apply id="alg2.l7.m1.1.1.1.1.1.1.2.cmml" xref="alg2.l7.m1.1.1.1.1.1.1.2"><ci id="alg2.l7.m1.1.1.1.1.1.1.2.1.cmml" xref="alg2.l7.m1.1.1.1.1.1.1.2.1">^</ci><ci id="alg2.l7.m1.1.1.1.1.1.1.2.2.cmml" xref="alg2.l7.m1.1.1.1.1.1.1.2.2">𝑦</ci></apply><apply id="alg2.l7.m1.1.1.1.1.1.1.3.cmml" xref="alg2.l7.m1.1.1.1.1.1.1.3"><ci id="alg2.l7.m1.1.1.1.1.1.1.3.1.cmml" xref="alg2.l7.m1.1.1.1.1.1.1.3.1">¯</ci><ci id="alg2.l7.m1.1.1.1.1.1.1.3.2.cmml" xref="alg2.l7.m1.1.1.1.1.1.1.3.2">𝑦</ci></apply></apply></apply><cn type="integer" id="alg2.l7.m1.1.1.1.3.cmml" xref="alg2.l7.m1.1.1.1.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg2.l7.m1.1c">\underset{\theta}{L}\frac{x-\bar{x}}{\sigma^{2}}+\log\sigma^{2}+||\hat{y}-\bar{y}||^{2}</annotation></semantics></math>

</div>
<div id="alg2.l8" class="ltx_listingline">     <span id="alg2.l8.1" class="ltx_text ltx_font_bold">end</span> <span id="alg2.l8.2" class="ltx_text ltx_font_bold">for</span>
</div>
<div id="alg2.l9" class="ltx_listingline">     Update weights of <math id="alg2.l9.m1.1" class="ltx_Math" alttext="\mathcal{f}" display="inline"><semantics id="alg2.l9.m1.1a"><mi class="ltx_font_mathcaligraphic" id="alg2.l9.m1.1.1" xref="alg2.l9.m1.1.1.cmml">𝒻</mi><annotation-xml encoding="MathML-Content" id="alg2.l9.m1.1b"><ci id="alg2.l9.m1.1.1.cmml" xref="alg2.l9.m1.1.1">𝒻</ci></annotation-xml><annotation encoding="application/x-tex" id="alg2.l9.m1.1c">\mathcal{f}</annotation></semantics></math>

</div>
<div id="alg2.l10" class="ltx_listingline">  <span id="alg2.l10.1" class="ltx_text ltx_font_bold">until</span> convergence

</div>
</div>
</figure>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Details of the Proposed Model</h2>

<div id="A5.p1" class="ltx_para">
<p id="A5.p1.1" class="ltx_p">We build our dataset conditioned weight generation model using latent diffusion <cite class="ltx_cite ltx_citemacro_citep">(Rombach et al., <a href="#bib.bib24" title="" class="ltx_ref">2021</a>)</cite>.</p>
</div>
<div id="A5.p2" class="ltx_para">
<p id="A5.p2.1" class="ltx_p"><span id="A5.p2.1.1" class="ltx_text ltx_font_bold">AutoEncoder</span>: We use the same VAE modules of latent diffusion with minor modifications and use the same structure for all experiments except adaptation of the inputs and output dimensions. We insert a linear layer before the first layer of the encoder such that we can reshape its output to a representation for the convolution layers. Analogously, another linear layer is place as the last layer of the decoder adapting the output to the vectorized weights representations. For the VAE loss function we removed the discriminator in the original latent diffusion VAE.</p>
</div>
<div id="A5.p3" class="ltx_para">
<p id="A5.p3.1" class="ltx_p"><span id="A5.p3.1.1" class="ltx_text ltx_font_bold">Diffusion Model</span>: Regarding the diffusion process, we use an architecture similar to the UNet model as in latent diffusion with minor changes. The training process in this module is the same as in latent diffusion.</p>
</div>
<div id="A5.p4" class="ltx_para">
<p id="A5.p4.1" class="ltx_p"><span id="A5.p4.1.1" class="ltx_text ltx_font_bold">Dataset Encoder</span>: We investigated three different mechanisms of dataset encoding. Firstly, we use Set Transformer which can be difficult to train when optimized together with the diffusion using the weights encoder from the VAE and the Set Transformer. Finally, we also observe that two linear layers can also be used to efficiently mimic the Set Transfomer. The main limitation of the contrastive based dataset encoding with Set Transformer is that this method can easily overfit the training datasets while jointly optimizing Set Transformer with the diffusion process often takes longer to converge.</p>
</div>
<div id="A5.p5" class="ltx_para">
<p id="A5.p5.16" class="ltx_p"><span id="A5.p5.16.1" class="ltx_text ltx_font_bold">Alternate Dataset Encoding Mechanisms</span>
We also use a two-layers MLP model as dataset encoder. The first layer is a dynamic layer with a maximum input feature size set to <math id="A5.p5.1.m1.2" class="ltx_Math" alttext="n\_max.c\_max" display="inline"><semantics id="A5.p5.1.m1.2a"><mrow id="A5.p5.1.m1.2.2.2" xref="A5.p5.1.m1.2.2.3.cmml"><mrow id="A5.p5.1.m1.1.1.1.1" xref="A5.p5.1.m1.1.1.1.1.cmml"><mi id="A5.p5.1.m1.1.1.1.1.2" xref="A5.p5.1.m1.1.1.1.1.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="A5.p5.1.m1.1.1.1.1.1" xref="A5.p5.1.m1.1.1.1.1.1.cmml">​</mo><mi mathvariant="normal" id="A5.p5.1.m1.1.1.1.1.3" xref="A5.p5.1.m1.1.1.1.1.3.cmml">_</mi><mo lspace="0em" rspace="0em" id="A5.p5.1.m1.1.1.1.1.1a" xref="A5.p5.1.m1.1.1.1.1.1.cmml">​</mo><mi id="A5.p5.1.m1.1.1.1.1.4" xref="A5.p5.1.m1.1.1.1.1.4.cmml">m</mi><mo lspace="0em" rspace="0em" id="A5.p5.1.m1.1.1.1.1.1b" xref="A5.p5.1.m1.1.1.1.1.1.cmml">​</mo><mi id="A5.p5.1.m1.1.1.1.1.5" xref="A5.p5.1.m1.1.1.1.1.5.cmml">a</mi><mo lspace="0em" rspace="0em" id="A5.p5.1.m1.1.1.1.1.1c" xref="A5.p5.1.m1.1.1.1.1.1.cmml">​</mo><mi id="A5.p5.1.m1.1.1.1.1.6" xref="A5.p5.1.m1.1.1.1.1.6.cmml">x</mi></mrow><mo lspace="0em" rspace="0.167em" id="A5.p5.1.m1.2.2.2.3" xref="A5.p5.1.m1.2.2.3a.cmml">.</mo><mrow id="A5.p5.1.m1.2.2.2.2" xref="A5.p5.1.m1.2.2.2.2.cmml"><mi id="A5.p5.1.m1.2.2.2.2.2" xref="A5.p5.1.m1.2.2.2.2.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="A5.p5.1.m1.2.2.2.2.1" xref="A5.p5.1.m1.2.2.2.2.1.cmml">​</mo><mi mathvariant="normal" id="A5.p5.1.m1.2.2.2.2.3" xref="A5.p5.1.m1.2.2.2.2.3.cmml">_</mi><mo lspace="0em" rspace="0em" id="A5.p5.1.m1.2.2.2.2.1a" xref="A5.p5.1.m1.2.2.2.2.1.cmml">​</mo><mi id="A5.p5.1.m1.2.2.2.2.4" xref="A5.p5.1.m1.2.2.2.2.4.cmml">m</mi><mo lspace="0em" rspace="0em" id="A5.p5.1.m1.2.2.2.2.1b" xref="A5.p5.1.m1.2.2.2.2.1.cmml">​</mo><mi id="A5.p5.1.m1.2.2.2.2.5" xref="A5.p5.1.m1.2.2.2.2.5.cmml">a</mi><mo lspace="0em" rspace="0em" id="A5.p5.1.m1.2.2.2.2.1c" xref="A5.p5.1.m1.2.2.2.2.1.cmml">​</mo><mi id="A5.p5.1.m1.2.2.2.2.6" xref="A5.p5.1.m1.2.2.2.2.6.cmml">x</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="A5.p5.1.m1.2b"><apply id="A5.p5.1.m1.2.2.3.cmml" xref="A5.p5.1.m1.2.2.2"><csymbol cd="ambiguous" id="A5.p5.1.m1.2.2.3a.cmml" xref="A5.p5.1.m1.2.2.2.3">formulae-sequence</csymbol><apply id="A5.p5.1.m1.1.1.1.1.cmml" xref="A5.p5.1.m1.1.1.1.1"><times id="A5.p5.1.m1.1.1.1.1.1.cmml" xref="A5.p5.1.m1.1.1.1.1.1"></times><ci id="A5.p5.1.m1.1.1.1.1.2.cmml" xref="A5.p5.1.m1.1.1.1.1.2">𝑛</ci><ci id="A5.p5.1.m1.1.1.1.1.3.cmml" xref="A5.p5.1.m1.1.1.1.1.3">_</ci><ci id="A5.p5.1.m1.1.1.1.1.4.cmml" xref="A5.p5.1.m1.1.1.1.1.4">𝑚</ci><ci id="A5.p5.1.m1.1.1.1.1.5.cmml" xref="A5.p5.1.m1.1.1.1.1.5">𝑎</ci><ci id="A5.p5.1.m1.1.1.1.1.6.cmml" xref="A5.p5.1.m1.1.1.1.1.6">𝑥</ci></apply><apply id="A5.p5.1.m1.2.2.2.2.cmml" xref="A5.p5.1.m1.2.2.2.2"><times id="A5.p5.1.m1.2.2.2.2.1.cmml" xref="A5.p5.1.m1.2.2.2.2.1"></times><ci id="A5.p5.1.m1.2.2.2.2.2.cmml" xref="A5.p5.1.m1.2.2.2.2.2">𝑐</ci><ci id="A5.p5.1.m1.2.2.2.2.3.cmml" xref="A5.p5.1.m1.2.2.2.2.3">_</ci><ci id="A5.p5.1.m1.2.2.2.2.4.cmml" xref="A5.p5.1.m1.2.2.2.2.4">𝑚</ci><ci id="A5.p5.1.m1.2.2.2.2.5.cmml" xref="A5.p5.1.m1.2.2.2.2.5">𝑎</ci><ci id="A5.p5.1.m1.2.2.2.2.6.cmml" xref="A5.p5.1.m1.2.2.2.2.6">𝑥</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p5.1.m1.2c">n\_max.c\_max</annotation></semantics></math> with <math id="A5.p5.2.m2.1" class="ltx_Math" alttext="n\_max" display="inline"><semantics id="A5.p5.2.m2.1a"><mrow id="A5.p5.2.m2.1.1" xref="A5.p5.2.m2.1.1.cmml"><mi id="A5.p5.2.m2.1.1.2" xref="A5.p5.2.m2.1.1.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="A5.p5.2.m2.1.1.1" xref="A5.p5.2.m2.1.1.1.cmml">​</mo><mi mathvariant="normal" id="A5.p5.2.m2.1.1.3" xref="A5.p5.2.m2.1.1.3.cmml">_</mi><mo lspace="0em" rspace="0em" id="A5.p5.2.m2.1.1.1a" xref="A5.p5.2.m2.1.1.1.cmml">​</mo><mi id="A5.p5.2.m2.1.1.4" xref="A5.p5.2.m2.1.1.4.cmml">m</mi><mo lspace="0em" rspace="0em" id="A5.p5.2.m2.1.1.1b" xref="A5.p5.2.m2.1.1.1.cmml">​</mo><mi id="A5.p5.2.m2.1.1.5" xref="A5.p5.2.m2.1.1.5.cmml">a</mi><mo lspace="0em" rspace="0em" id="A5.p5.2.m2.1.1.1c" xref="A5.p5.2.m2.1.1.1.cmml">​</mo><mi id="A5.p5.2.m2.1.1.6" xref="A5.p5.2.m2.1.1.6.cmml">x</mi></mrow><annotation-xml encoding="MathML-Content" id="A5.p5.2.m2.1b"><apply id="A5.p5.2.m2.1.1.cmml" xref="A5.p5.2.m2.1.1"><times id="A5.p5.2.m2.1.1.1.cmml" xref="A5.p5.2.m2.1.1.1"></times><ci id="A5.p5.2.m2.1.1.2.cmml" xref="A5.p5.2.m2.1.1.2">𝑛</ci><ci id="A5.p5.2.m2.1.1.3.cmml" xref="A5.p5.2.m2.1.1.3">_</ci><ci id="A5.p5.2.m2.1.1.4.cmml" xref="A5.p5.2.m2.1.1.4">𝑚</ci><ci id="A5.p5.2.m2.1.1.5.cmml" xref="A5.p5.2.m2.1.1.5">𝑎</ci><ci id="A5.p5.2.m2.1.1.6.cmml" xref="A5.p5.2.m2.1.1.6">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p5.2.m2.1c">n\_max</annotation></semantics></math> max number of images per class and <math id="A5.p5.3.m3.1" class="ltx_Math" alttext="c\_max" display="inline"><semantics id="A5.p5.3.m3.1a"><mrow id="A5.p5.3.m3.1.1" xref="A5.p5.3.m3.1.1.cmml"><mi id="A5.p5.3.m3.1.1.2" xref="A5.p5.3.m3.1.1.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="A5.p5.3.m3.1.1.1" xref="A5.p5.3.m3.1.1.1.cmml">​</mo><mi mathvariant="normal" id="A5.p5.3.m3.1.1.3" xref="A5.p5.3.m3.1.1.3.cmml">_</mi><mo lspace="0em" rspace="0em" id="A5.p5.3.m3.1.1.1a" xref="A5.p5.3.m3.1.1.1.cmml">​</mo><mi id="A5.p5.3.m3.1.1.4" xref="A5.p5.3.m3.1.1.4.cmml">m</mi><mo lspace="0em" rspace="0em" id="A5.p5.3.m3.1.1.1b" xref="A5.p5.3.m3.1.1.1.cmml">​</mo><mi id="A5.p5.3.m3.1.1.5" xref="A5.p5.3.m3.1.1.5.cmml">a</mi><mo lspace="0em" rspace="0em" id="A5.p5.3.m3.1.1.1c" xref="A5.p5.3.m3.1.1.1.cmml">​</mo><mi id="A5.p5.3.m3.1.1.6" xref="A5.p5.3.m3.1.1.6.cmml">x</mi></mrow><annotation-xml encoding="MathML-Content" id="A5.p5.3.m3.1b"><apply id="A5.p5.3.m3.1.1.cmml" xref="A5.p5.3.m3.1.1"><times id="A5.p5.3.m3.1.1.1.cmml" xref="A5.p5.3.m3.1.1.1"></times><ci id="A5.p5.3.m3.1.1.2.cmml" xref="A5.p5.3.m3.1.1.2">𝑐</ci><ci id="A5.p5.3.m3.1.1.3.cmml" xref="A5.p5.3.m3.1.1.3">_</ci><ci id="A5.p5.3.m3.1.1.4.cmml" xref="A5.p5.3.m3.1.1.4">𝑚</ci><ci id="A5.p5.3.m3.1.1.5.cmml" xref="A5.p5.3.m3.1.1.5">𝑎</ci><ci id="A5.p5.3.m3.1.1.6.cmml" xref="A5.p5.3.m3.1.1.6">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p5.3.m3.1c">c\_max</annotation></semantics></math> maximum number of classes among all subsets of pretrained datasets.
The shape of the each images datasets features obtained with clip image encoder is <math id="A5.p5.4.m4.1" class="ltx_Math" alttext="x\in\mathbf{R}^{c\times n\times d}" display="inline"><semantics id="A5.p5.4.m4.1a"><mrow id="A5.p5.4.m4.1.1" xref="A5.p5.4.m4.1.1.cmml"><mi id="A5.p5.4.m4.1.1.2" xref="A5.p5.4.m4.1.1.2.cmml">x</mi><mo id="A5.p5.4.m4.1.1.1" xref="A5.p5.4.m4.1.1.1.cmml">∈</mo><msup id="A5.p5.4.m4.1.1.3" xref="A5.p5.4.m4.1.1.3.cmml"><mi id="A5.p5.4.m4.1.1.3.2" xref="A5.p5.4.m4.1.1.3.2.cmml">𝐑</mi><mrow id="A5.p5.4.m4.1.1.3.3" xref="A5.p5.4.m4.1.1.3.3.cmml"><mi id="A5.p5.4.m4.1.1.3.3.2" xref="A5.p5.4.m4.1.1.3.3.2.cmml">c</mi><mo lspace="0.222em" rspace="0.222em" id="A5.p5.4.m4.1.1.3.3.1" xref="A5.p5.4.m4.1.1.3.3.1.cmml">×</mo><mi id="A5.p5.4.m4.1.1.3.3.3" xref="A5.p5.4.m4.1.1.3.3.3.cmml">n</mi><mo lspace="0.222em" rspace="0.222em" id="A5.p5.4.m4.1.1.3.3.1a" xref="A5.p5.4.m4.1.1.3.3.1.cmml">×</mo><mi id="A5.p5.4.m4.1.1.3.3.4" xref="A5.p5.4.m4.1.1.3.3.4.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A5.p5.4.m4.1b"><apply id="A5.p5.4.m4.1.1.cmml" xref="A5.p5.4.m4.1.1"><in id="A5.p5.4.m4.1.1.1.cmml" xref="A5.p5.4.m4.1.1.1"></in><ci id="A5.p5.4.m4.1.1.2.cmml" xref="A5.p5.4.m4.1.1.2">𝑥</ci><apply id="A5.p5.4.m4.1.1.3.cmml" xref="A5.p5.4.m4.1.1.3"><csymbol cd="ambiguous" id="A5.p5.4.m4.1.1.3.1.cmml" xref="A5.p5.4.m4.1.1.3">superscript</csymbol><ci id="A5.p5.4.m4.1.1.3.2.cmml" xref="A5.p5.4.m4.1.1.3.2">𝐑</ci><apply id="A5.p5.4.m4.1.1.3.3.cmml" xref="A5.p5.4.m4.1.1.3.3"><times id="A5.p5.4.m4.1.1.3.3.1.cmml" xref="A5.p5.4.m4.1.1.3.3.1"></times><ci id="A5.p5.4.m4.1.1.3.3.2.cmml" xref="A5.p5.4.m4.1.1.3.3.2">𝑐</ci><ci id="A5.p5.4.m4.1.1.3.3.3.cmml" xref="A5.p5.4.m4.1.1.3.3.3">𝑛</ci><ci id="A5.p5.4.m4.1.1.3.3.4.cmml" xref="A5.p5.4.m4.1.1.3.3.4">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p5.4.m4.1c">x\in\mathbf{R}^{c\times n\times d}</annotation></semantics></math> with <math id="A5.p5.5.m5.1" class="ltx_Math" alttext="d" display="inline"><semantics id="A5.p5.5.m5.1a"><mi id="A5.p5.5.m5.1.1" xref="A5.p5.5.m5.1.1.cmml">d</mi><annotation-xml encoding="MathML-Content" id="A5.p5.5.m5.1b"><ci id="A5.p5.5.m5.1.1.cmml" xref="A5.p5.5.m5.1.1">𝑑</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.p5.5.m5.1c">d</annotation></semantics></math> the features dimension for each corresponding pretrained weight vector. While the Set Transformer based encoder uses these inputs directly, with MLP encoder we reshape each input <math id="A5.p5.6.m6.1" class="ltx_Math" alttext="x\in\mathbf{R}^{c\times n\times d}\rightarrow x\in\mathbf{R}^{d\times nd}" display="inline"><semantics id="A5.p5.6.m6.1a"><mrow id="A5.p5.6.m6.1.1" xref="A5.p5.6.m6.1.1.cmml"><mi id="A5.p5.6.m6.1.1.2" xref="A5.p5.6.m6.1.1.2.cmml">x</mi><mo id="A5.p5.6.m6.1.1.3" xref="A5.p5.6.m6.1.1.3.cmml">∈</mo><msup id="A5.p5.6.m6.1.1.4" xref="A5.p5.6.m6.1.1.4.cmml"><mi id="A5.p5.6.m6.1.1.4.2" xref="A5.p5.6.m6.1.1.4.2.cmml">𝐑</mi><mrow id="A5.p5.6.m6.1.1.4.3" xref="A5.p5.6.m6.1.1.4.3.cmml"><mi id="A5.p5.6.m6.1.1.4.3.2" xref="A5.p5.6.m6.1.1.4.3.2.cmml">c</mi><mo lspace="0.222em" rspace="0.222em" id="A5.p5.6.m6.1.1.4.3.1" xref="A5.p5.6.m6.1.1.4.3.1.cmml">×</mo><mi id="A5.p5.6.m6.1.1.4.3.3" xref="A5.p5.6.m6.1.1.4.3.3.cmml">n</mi><mo lspace="0.222em" rspace="0.222em" id="A5.p5.6.m6.1.1.4.3.1a" xref="A5.p5.6.m6.1.1.4.3.1.cmml">×</mo><mi id="A5.p5.6.m6.1.1.4.3.4" xref="A5.p5.6.m6.1.1.4.3.4.cmml">d</mi></mrow></msup><mo stretchy="false" id="A5.p5.6.m6.1.1.5" xref="A5.p5.6.m6.1.1.5.cmml">→</mo><mi id="A5.p5.6.m6.1.1.6" xref="A5.p5.6.m6.1.1.6.cmml">x</mi><mo id="A5.p5.6.m6.1.1.7" xref="A5.p5.6.m6.1.1.7.cmml">∈</mo><msup id="A5.p5.6.m6.1.1.8" xref="A5.p5.6.m6.1.1.8.cmml"><mi id="A5.p5.6.m6.1.1.8.2" xref="A5.p5.6.m6.1.1.8.2.cmml">𝐑</mi><mrow id="A5.p5.6.m6.1.1.8.3" xref="A5.p5.6.m6.1.1.8.3.cmml"><mrow id="A5.p5.6.m6.1.1.8.3.2" xref="A5.p5.6.m6.1.1.8.3.2.cmml"><mi id="A5.p5.6.m6.1.1.8.3.2.2" xref="A5.p5.6.m6.1.1.8.3.2.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="A5.p5.6.m6.1.1.8.3.2.1" xref="A5.p5.6.m6.1.1.8.3.2.1.cmml">×</mo><mi id="A5.p5.6.m6.1.1.8.3.2.3" xref="A5.p5.6.m6.1.1.8.3.2.3.cmml">n</mi></mrow><mo lspace="0em" rspace="0em" id="A5.p5.6.m6.1.1.8.3.1" xref="A5.p5.6.m6.1.1.8.3.1.cmml">​</mo><mi id="A5.p5.6.m6.1.1.8.3.3" xref="A5.p5.6.m6.1.1.8.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A5.p5.6.m6.1b"><apply id="A5.p5.6.m6.1.1.cmml" xref="A5.p5.6.m6.1.1"><and id="A5.p5.6.m6.1.1a.cmml" xref="A5.p5.6.m6.1.1"></and><apply id="A5.p5.6.m6.1.1b.cmml" xref="A5.p5.6.m6.1.1"><in id="A5.p5.6.m6.1.1.3.cmml" xref="A5.p5.6.m6.1.1.3"></in><ci id="A5.p5.6.m6.1.1.2.cmml" xref="A5.p5.6.m6.1.1.2">𝑥</ci><apply id="A5.p5.6.m6.1.1.4.cmml" xref="A5.p5.6.m6.1.1.4"><csymbol cd="ambiguous" id="A5.p5.6.m6.1.1.4.1.cmml" xref="A5.p5.6.m6.1.1.4">superscript</csymbol><ci id="A5.p5.6.m6.1.1.4.2.cmml" xref="A5.p5.6.m6.1.1.4.2">𝐑</ci><apply id="A5.p5.6.m6.1.1.4.3.cmml" xref="A5.p5.6.m6.1.1.4.3"><times id="A5.p5.6.m6.1.1.4.3.1.cmml" xref="A5.p5.6.m6.1.1.4.3.1"></times><ci id="A5.p5.6.m6.1.1.4.3.2.cmml" xref="A5.p5.6.m6.1.1.4.3.2">𝑐</ci><ci id="A5.p5.6.m6.1.1.4.3.3.cmml" xref="A5.p5.6.m6.1.1.4.3.3">𝑛</ci><ci id="A5.p5.6.m6.1.1.4.3.4.cmml" xref="A5.p5.6.m6.1.1.4.3.4">𝑑</ci></apply></apply></apply><apply id="A5.p5.6.m6.1.1c.cmml" xref="A5.p5.6.m6.1.1"><ci id="A5.p5.6.m6.1.1.5.cmml" xref="A5.p5.6.m6.1.1.5">→</ci><share href="#A5.p5.6.m6.1.1.4.cmml" id="A5.p5.6.m6.1.1d.cmml" xref="A5.p5.6.m6.1.1"></share><ci id="A5.p5.6.m6.1.1.6.cmml" xref="A5.p5.6.m6.1.1.6">𝑥</ci></apply><apply id="A5.p5.6.m6.1.1e.cmml" xref="A5.p5.6.m6.1.1"><in id="A5.p5.6.m6.1.1.7.cmml" xref="A5.p5.6.m6.1.1.7"></in><share href="#A5.p5.6.m6.1.1.6.cmml" id="A5.p5.6.m6.1.1f.cmml" xref="A5.p5.6.m6.1.1"></share><apply id="A5.p5.6.m6.1.1.8.cmml" xref="A5.p5.6.m6.1.1.8"><csymbol cd="ambiguous" id="A5.p5.6.m6.1.1.8.1.cmml" xref="A5.p5.6.m6.1.1.8">superscript</csymbol><ci id="A5.p5.6.m6.1.1.8.2.cmml" xref="A5.p5.6.m6.1.1.8.2">𝐑</ci><apply id="A5.p5.6.m6.1.1.8.3.cmml" xref="A5.p5.6.m6.1.1.8.3"><times id="A5.p5.6.m6.1.1.8.3.1.cmml" xref="A5.p5.6.m6.1.1.8.3.1"></times><apply id="A5.p5.6.m6.1.1.8.3.2.cmml" xref="A5.p5.6.m6.1.1.8.3.2"><times id="A5.p5.6.m6.1.1.8.3.2.1.cmml" xref="A5.p5.6.m6.1.1.8.3.2.1"></times><ci id="A5.p5.6.m6.1.1.8.3.2.2.cmml" xref="A5.p5.6.m6.1.1.8.3.2.2">𝑑</ci><ci id="A5.p5.6.m6.1.1.8.3.2.3.cmml" xref="A5.p5.6.m6.1.1.8.3.2.3">𝑛</ci></apply><ci id="A5.p5.6.m6.1.1.8.3.3.cmml" xref="A5.p5.6.m6.1.1.8.3.3">𝑑</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p5.6.m6.1c">x\in\mathbf{R}^{c\times n\times d}\rightarrow x\in\mathbf{R}^{d\times nd}</annotation></semantics></math> then applied the dynamic linear layer.If a dataset has more classes respectively more samples than <math id="A5.p5.7.m7.1" class="ltx_Math" alttext="c\_max" display="inline"><semantics id="A5.p5.7.m7.1a"><mrow id="A5.p5.7.m7.1.1" xref="A5.p5.7.m7.1.1.cmml"><mi id="A5.p5.7.m7.1.1.2" xref="A5.p5.7.m7.1.1.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="A5.p5.7.m7.1.1.1" xref="A5.p5.7.m7.1.1.1.cmml">​</mo><mi mathvariant="normal" id="A5.p5.7.m7.1.1.3" xref="A5.p5.7.m7.1.1.3.cmml">_</mi><mo lspace="0em" rspace="0em" id="A5.p5.7.m7.1.1.1a" xref="A5.p5.7.m7.1.1.1.cmml">​</mo><mi id="A5.p5.7.m7.1.1.4" xref="A5.p5.7.m7.1.1.4.cmml">m</mi><mo lspace="0em" rspace="0em" id="A5.p5.7.m7.1.1.1b" xref="A5.p5.7.m7.1.1.1.cmml">​</mo><mi id="A5.p5.7.m7.1.1.5" xref="A5.p5.7.m7.1.1.5.cmml">a</mi><mo lspace="0em" rspace="0em" id="A5.p5.7.m7.1.1.1c" xref="A5.p5.7.m7.1.1.1.cmml">​</mo><mi id="A5.p5.7.m7.1.1.6" xref="A5.p5.7.m7.1.1.6.cmml">x</mi></mrow><annotation-xml encoding="MathML-Content" id="A5.p5.7.m7.1b"><apply id="A5.p5.7.m7.1.1.cmml" xref="A5.p5.7.m7.1.1"><times id="A5.p5.7.m7.1.1.1.cmml" xref="A5.p5.7.m7.1.1.1"></times><ci id="A5.p5.7.m7.1.1.2.cmml" xref="A5.p5.7.m7.1.1.2">𝑐</ci><ci id="A5.p5.7.m7.1.1.3.cmml" xref="A5.p5.7.m7.1.1.3">_</ci><ci id="A5.p5.7.m7.1.1.4.cmml" xref="A5.p5.7.m7.1.1.4">𝑚</ci><ci id="A5.p5.7.m7.1.1.5.cmml" xref="A5.p5.7.m7.1.1.5">𝑎</ci><ci id="A5.p5.7.m7.1.1.6.cmml" xref="A5.p5.7.m7.1.1.6">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p5.7.m7.1c">c\_max</annotation></semantics></math> respectively <math id="A5.p5.8.m8.1" class="ltx_Math" alttext="n\_max" display="inline"><semantics id="A5.p5.8.m8.1a"><mrow id="A5.p5.8.m8.1.1" xref="A5.p5.8.m8.1.1.cmml"><mi id="A5.p5.8.m8.1.1.2" xref="A5.p5.8.m8.1.1.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="A5.p5.8.m8.1.1.1" xref="A5.p5.8.m8.1.1.1.cmml">​</mo><mi mathvariant="normal" id="A5.p5.8.m8.1.1.3" xref="A5.p5.8.m8.1.1.3.cmml">_</mi><mo lspace="0em" rspace="0em" id="A5.p5.8.m8.1.1.1a" xref="A5.p5.8.m8.1.1.1.cmml">​</mo><mi id="A5.p5.8.m8.1.1.4" xref="A5.p5.8.m8.1.1.4.cmml">m</mi><mo lspace="0em" rspace="0em" id="A5.p5.8.m8.1.1.1b" xref="A5.p5.8.m8.1.1.1.cmml">​</mo><mi id="A5.p5.8.m8.1.1.5" xref="A5.p5.8.m8.1.1.5.cmml">a</mi><mo lspace="0em" rspace="0em" id="A5.p5.8.m8.1.1.1c" xref="A5.p5.8.m8.1.1.1.cmml">​</mo><mi id="A5.p5.8.m8.1.1.6" xref="A5.p5.8.m8.1.1.6.cmml">x</mi></mrow><annotation-xml encoding="MathML-Content" id="A5.p5.8.m8.1b"><apply id="A5.p5.8.m8.1.1.cmml" xref="A5.p5.8.m8.1.1"><times id="A5.p5.8.m8.1.1.1.cmml" xref="A5.p5.8.m8.1.1.1"></times><ci id="A5.p5.8.m8.1.1.2.cmml" xref="A5.p5.8.m8.1.1.2">𝑛</ci><ci id="A5.p5.8.m8.1.1.3.cmml" xref="A5.p5.8.m8.1.1.3">_</ci><ci id="A5.p5.8.m8.1.1.4.cmml" xref="A5.p5.8.m8.1.1.4">𝑚</ci><ci id="A5.p5.8.m8.1.1.5.cmml" xref="A5.p5.8.m8.1.1.5">𝑎</ci><ci id="A5.p5.8.m8.1.1.6.cmml" xref="A5.p5.8.m8.1.1.6">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p5.8.m8.1c">n\_max</annotation></semantics></math>, we only consider the first <math id="A5.p5.9.m9.1" class="ltx_Math" alttext="c\_max" display="inline"><semantics id="A5.p5.9.m9.1a"><mrow id="A5.p5.9.m9.1.1" xref="A5.p5.9.m9.1.1.cmml"><mi id="A5.p5.9.m9.1.1.2" xref="A5.p5.9.m9.1.1.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="A5.p5.9.m9.1.1.1" xref="A5.p5.9.m9.1.1.1.cmml">​</mo><mi mathvariant="normal" id="A5.p5.9.m9.1.1.3" xref="A5.p5.9.m9.1.1.3.cmml">_</mi><mo lspace="0em" rspace="0em" id="A5.p5.9.m9.1.1.1a" xref="A5.p5.9.m9.1.1.1.cmml">​</mo><mi id="A5.p5.9.m9.1.1.4" xref="A5.p5.9.m9.1.1.4.cmml">m</mi><mo lspace="0em" rspace="0em" id="A5.p5.9.m9.1.1.1b" xref="A5.p5.9.m9.1.1.1.cmml">​</mo><mi id="A5.p5.9.m9.1.1.5" xref="A5.p5.9.m9.1.1.5.cmml">a</mi><mo lspace="0em" rspace="0em" id="A5.p5.9.m9.1.1.1c" xref="A5.p5.9.m9.1.1.1.cmml">​</mo><mi id="A5.p5.9.m9.1.1.6" xref="A5.p5.9.m9.1.1.6.cmml">x</mi></mrow><annotation-xml encoding="MathML-Content" id="A5.p5.9.m9.1b"><apply id="A5.p5.9.m9.1.1.cmml" xref="A5.p5.9.m9.1.1"><times id="A5.p5.9.m9.1.1.1.cmml" xref="A5.p5.9.m9.1.1.1"></times><ci id="A5.p5.9.m9.1.1.2.cmml" xref="A5.p5.9.m9.1.1.2">𝑐</ci><ci id="A5.p5.9.m9.1.1.3.cmml" xref="A5.p5.9.m9.1.1.3">_</ci><ci id="A5.p5.9.m9.1.1.4.cmml" xref="A5.p5.9.m9.1.1.4">𝑚</ci><ci id="A5.p5.9.m9.1.1.5.cmml" xref="A5.p5.9.m9.1.1.5">𝑎</ci><ci id="A5.p5.9.m9.1.1.6.cmml" xref="A5.p5.9.m9.1.1.6">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p5.9.m9.1c">c\_max</annotation></semantics></math> classes and <math id="A5.p5.10.m10.1" class="ltx_Math" alttext="n\_max" display="inline"><semantics id="A5.p5.10.m10.1a"><mrow id="A5.p5.10.m10.1.1" xref="A5.p5.10.m10.1.1.cmml"><mi id="A5.p5.10.m10.1.1.2" xref="A5.p5.10.m10.1.1.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="A5.p5.10.m10.1.1.1" xref="A5.p5.10.m10.1.1.1.cmml">​</mo><mi mathvariant="normal" id="A5.p5.10.m10.1.1.3" xref="A5.p5.10.m10.1.1.3.cmml">_</mi><mo lspace="0em" rspace="0em" id="A5.p5.10.m10.1.1.1a" xref="A5.p5.10.m10.1.1.1.cmml">​</mo><mi id="A5.p5.10.m10.1.1.4" xref="A5.p5.10.m10.1.1.4.cmml">m</mi><mo lspace="0em" rspace="0em" id="A5.p5.10.m10.1.1.1b" xref="A5.p5.10.m10.1.1.1.cmml">​</mo><mi id="A5.p5.10.m10.1.1.5" xref="A5.p5.10.m10.1.1.5.cmml">a</mi><mo lspace="0em" rspace="0em" id="A5.p5.10.m10.1.1.1c" xref="A5.p5.10.m10.1.1.1.cmml">​</mo><mi id="A5.p5.10.m10.1.1.6" xref="A5.p5.10.m10.1.1.6.cmml">x</mi></mrow><annotation-xml encoding="MathML-Content" id="A5.p5.10.m10.1b"><apply id="A5.p5.10.m10.1.1.cmml" xref="A5.p5.10.m10.1.1"><times id="A5.p5.10.m10.1.1.1.cmml" xref="A5.p5.10.m10.1.1.1"></times><ci id="A5.p5.10.m10.1.1.2.cmml" xref="A5.p5.10.m10.1.1.2">𝑛</ci><ci id="A5.p5.10.m10.1.1.3.cmml" xref="A5.p5.10.m10.1.1.3">_</ci><ci id="A5.p5.10.m10.1.1.4.cmml" xref="A5.p5.10.m10.1.1.4">𝑚</ci><ci id="A5.p5.10.m10.1.1.5.cmml" xref="A5.p5.10.m10.1.1.5">𝑎</ci><ci id="A5.p5.10.m10.1.1.6.cmml" xref="A5.p5.10.m10.1.1.6">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p5.10.m10.1c">n\_max</annotation></semantics></math> samples per classes. In the case that the dataset has less classes or samples we reset the dynamic linear layer dimension accordingly. The output of the dynamic linear layer is <math id="A5.p5.11.m11.1" class="ltx_Math" alttext="z\in\mathbf{R}^{d\times h}" display="inline"><semantics id="A5.p5.11.m11.1a"><mrow id="A5.p5.11.m11.1.1" xref="A5.p5.11.m11.1.1.cmml"><mi id="A5.p5.11.m11.1.1.2" xref="A5.p5.11.m11.1.1.2.cmml">z</mi><mo id="A5.p5.11.m11.1.1.1" xref="A5.p5.11.m11.1.1.1.cmml">∈</mo><msup id="A5.p5.11.m11.1.1.3" xref="A5.p5.11.m11.1.1.3.cmml"><mi id="A5.p5.11.m11.1.1.3.2" xref="A5.p5.11.m11.1.1.3.2.cmml">𝐑</mi><mrow id="A5.p5.11.m11.1.1.3.3" xref="A5.p5.11.m11.1.1.3.3.cmml"><mi id="A5.p5.11.m11.1.1.3.3.2" xref="A5.p5.11.m11.1.1.3.3.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="A5.p5.11.m11.1.1.3.3.1" xref="A5.p5.11.m11.1.1.3.3.1.cmml">×</mo><mi id="A5.p5.11.m11.1.1.3.3.3" xref="A5.p5.11.m11.1.1.3.3.3.cmml">h</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A5.p5.11.m11.1b"><apply id="A5.p5.11.m11.1.1.cmml" xref="A5.p5.11.m11.1.1"><in id="A5.p5.11.m11.1.1.1.cmml" xref="A5.p5.11.m11.1.1.1"></in><ci id="A5.p5.11.m11.1.1.2.cmml" xref="A5.p5.11.m11.1.1.2">𝑧</ci><apply id="A5.p5.11.m11.1.1.3.cmml" xref="A5.p5.11.m11.1.1.3"><csymbol cd="ambiguous" id="A5.p5.11.m11.1.1.3.1.cmml" xref="A5.p5.11.m11.1.1.3">superscript</csymbol><ci id="A5.p5.11.m11.1.1.3.2.cmml" xref="A5.p5.11.m11.1.1.3.2">𝐑</ci><apply id="A5.p5.11.m11.1.1.3.3.cmml" xref="A5.p5.11.m11.1.1.3.3"><times id="A5.p5.11.m11.1.1.3.3.1.cmml" xref="A5.p5.11.m11.1.1.3.3.1"></times><ci id="A5.p5.11.m11.1.1.3.3.2.cmml" xref="A5.p5.11.m11.1.1.3.3.2">𝑑</ci><ci id="A5.p5.11.m11.1.1.3.3.3.cmml" xref="A5.p5.11.m11.1.1.3.3.3">ℎ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p5.11.m11.1c">z\in\mathbf{R}^{d\times h}</annotation></semantics></math> where <math id="A5.p5.12.m12.1" class="ltx_Math" alttext="h" display="inline"><semantics id="A5.p5.12.m12.1a"><mi id="A5.p5.12.m12.1.1" xref="A5.p5.12.m12.1.1.cmml">h</mi><annotation-xml encoding="MathML-Content" id="A5.p5.12.m12.1b"><ci id="A5.p5.12.m12.1.1.cmml" xref="A5.p5.12.m12.1.1">ℎ</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.p5.12.m12.1c">h</annotation></semantics></math> s an arbitrary chosen number <math id="A5.p5.13.m13.1" class="ltx_Math" alttext="h&gt;0" display="inline"><semantics id="A5.p5.13.m13.1a"><mrow id="A5.p5.13.m13.1.1" xref="A5.p5.13.m13.1.1.cmml"><mi id="A5.p5.13.m13.1.1.2" xref="A5.p5.13.m13.1.1.2.cmml">h</mi><mo id="A5.p5.13.m13.1.1.1" xref="A5.p5.13.m13.1.1.1.cmml">&gt;</mo><mn id="A5.p5.13.m13.1.1.3" xref="A5.p5.13.m13.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.p5.13.m13.1b"><apply id="A5.p5.13.m13.1.1.cmml" xref="A5.p5.13.m13.1.1"><gt id="A5.p5.13.m13.1.1.1.cmml" xref="A5.p5.13.m13.1.1.1"></gt><ci id="A5.p5.13.m13.1.1.2.cmml" xref="A5.p5.13.m13.1.1.2">ℎ</ci><cn type="integer" id="A5.p5.13.m13.1.1.3.cmml" xref="A5.p5.13.m13.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p5.13.m13.1c">h&gt;0</annotation></semantics></math>. We then reshape <math id="A5.p5.14.m14.1" class="ltx_Math" alttext="z\in\mathbf{R}^{d\times h}" display="inline"><semantics id="A5.p5.14.m14.1a"><mrow id="A5.p5.14.m14.1.1" xref="A5.p5.14.m14.1.1.cmml"><mi id="A5.p5.14.m14.1.1.2" xref="A5.p5.14.m14.1.1.2.cmml">z</mi><mo id="A5.p5.14.m14.1.1.1" xref="A5.p5.14.m14.1.1.1.cmml">∈</mo><msup id="A5.p5.14.m14.1.1.3" xref="A5.p5.14.m14.1.1.3.cmml"><mi id="A5.p5.14.m14.1.1.3.2" xref="A5.p5.14.m14.1.1.3.2.cmml">𝐑</mi><mrow id="A5.p5.14.m14.1.1.3.3" xref="A5.p5.14.m14.1.1.3.3.cmml"><mi id="A5.p5.14.m14.1.1.3.3.2" xref="A5.p5.14.m14.1.1.3.3.2.cmml">d</mi><mo lspace="0.222em" rspace="0.222em" id="A5.p5.14.m14.1.1.3.3.1" xref="A5.p5.14.m14.1.1.3.3.1.cmml">×</mo><mi id="A5.p5.14.m14.1.1.3.3.3" xref="A5.p5.14.m14.1.1.3.3.3.cmml">h</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A5.p5.14.m14.1b"><apply id="A5.p5.14.m14.1.1.cmml" xref="A5.p5.14.m14.1.1"><in id="A5.p5.14.m14.1.1.1.cmml" xref="A5.p5.14.m14.1.1.1"></in><ci id="A5.p5.14.m14.1.1.2.cmml" xref="A5.p5.14.m14.1.1.2">𝑧</ci><apply id="A5.p5.14.m14.1.1.3.cmml" xref="A5.p5.14.m14.1.1.3"><csymbol cd="ambiguous" id="A5.p5.14.m14.1.1.3.1.cmml" xref="A5.p5.14.m14.1.1.3">superscript</csymbol><ci id="A5.p5.14.m14.1.1.3.2.cmml" xref="A5.p5.14.m14.1.1.3.2">𝐑</ci><apply id="A5.p5.14.m14.1.1.3.3.cmml" xref="A5.p5.14.m14.1.1.3.3"><times id="A5.p5.14.m14.1.1.3.3.1.cmml" xref="A5.p5.14.m14.1.1.3.3.1"></times><ci id="A5.p5.14.m14.1.1.3.3.2.cmml" xref="A5.p5.14.m14.1.1.3.3.2">𝑑</ci><ci id="A5.p5.14.m14.1.1.3.3.3.cmml" xref="A5.p5.14.m14.1.1.3.3.3">ℎ</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p5.14.m14.1c">z\in\mathbf{R}^{d\times h}</annotation></semantics></math> to <math id="A5.p5.15.m15.1" class="ltx_Math" alttext="z\in\mathbf{R}^{1\times hd}" display="inline"><semantics id="A5.p5.15.m15.1a"><mrow id="A5.p5.15.m15.1.1" xref="A5.p5.15.m15.1.1.cmml"><mi id="A5.p5.15.m15.1.1.2" xref="A5.p5.15.m15.1.1.2.cmml">z</mi><mo id="A5.p5.15.m15.1.1.1" xref="A5.p5.15.m15.1.1.1.cmml">∈</mo><msup id="A5.p5.15.m15.1.1.3" xref="A5.p5.15.m15.1.1.3.cmml"><mi id="A5.p5.15.m15.1.1.3.2" xref="A5.p5.15.m15.1.1.3.2.cmml">𝐑</mi><mrow id="A5.p5.15.m15.1.1.3.3" xref="A5.p5.15.m15.1.1.3.3.cmml"><mrow id="A5.p5.15.m15.1.1.3.3.2" xref="A5.p5.15.m15.1.1.3.3.2.cmml"><mn id="A5.p5.15.m15.1.1.3.3.2.2" xref="A5.p5.15.m15.1.1.3.3.2.2.cmml">1</mn><mo lspace="0.222em" rspace="0.222em" id="A5.p5.15.m15.1.1.3.3.2.1" xref="A5.p5.15.m15.1.1.3.3.2.1.cmml">×</mo><mi id="A5.p5.15.m15.1.1.3.3.2.3" xref="A5.p5.15.m15.1.1.3.3.2.3.cmml">h</mi></mrow><mo lspace="0em" rspace="0em" id="A5.p5.15.m15.1.1.3.3.1" xref="A5.p5.15.m15.1.1.3.3.1.cmml">​</mo><mi id="A5.p5.15.m15.1.1.3.3.3" xref="A5.p5.15.m15.1.1.3.3.3.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="A5.p5.15.m15.1b"><apply id="A5.p5.15.m15.1.1.cmml" xref="A5.p5.15.m15.1.1"><in id="A5.p5.15.m15.1.1.1.cmml" xref="A5.p5.15.m15.1.1.1"></in><ci id="A5.p5.15.m15.1.1.2.cmml" xref="A5.p5.15.m15.1.1.2">𝑧</ci><apply id="A5.p5.15.m15.1.1.3.cmml" xref="A5.p5.15.m15.1.1.3"><csymbol cd="ambiguous" id="A5.p5.15.m15.1.1.3.1.cmml" xref="A5.p5.15.m15.1.1.3">superscript</csymbol><ci id="A5.p5.15.m15.1.1.3.2.cmml" xref="A5.p5.15.m15.1.1.3.2">𝐑</ci><apply id="A5.p5.15.m15.1.1.3.3.cmml" xref="A5.p5.15.m15.1.1.3.3"><times id="A5.p5.15.m15.1.1.3.3.1.cmml" xref="A5.p5.15.m15.1.1.3.3.1"></times><apply id="A5.p5.15.m15.1.1.3.3.2.cmml" xref="A5.p5.15.m15.1.1.3.3.2"><times id="A5.p5.15.m15.1.1.3.3.2.1.cmml" xref="A5.p5.15.m15.1.1.3.3.2.1"></times><cn type="integer" id="A5.p5.15.m15.1.1.3.3.2.2.cmml" xref="A5.p5.15.m15.1.1.3.3.2.2">1</cn><ci id="A5.p5.15.m15.1.1.3.3.2.3.cmml" xref="A5.p5.15.m15.1.1.3.3.2.3">ℎ</ci></apply><ci id="A5.p5.15.m15.1.1.3.3.3.cmml" xref="A5.p5.15.m15.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p5.15.m15.1c">z\in\mathbf{R}^{1\times hd}</annotation></semantics></math> (<math id="A5.p5.16.m16.1" class="ltx_Math" alttext="hd" display="inline"><semantics id="A5.p5.16.m16.1a"><mrow id="A5.p5.16.m16.1.1" xref="A5.p5.16.m16.1.1.cmml"><mi id="A5.p5.16.m16.1.1.2" xref="A5.p5.16.m16.1.1.2.cmml">h</mi><mo lspace="0em" rspace="0em" id="A5.p5.16.m16.1.1.1" xref="A5.p5.16.m16.1.1.1.cmml">​</mo><mi id="A5.p5.16.m16.1.1.3" xref="A5.p5.16.m16.1.1.3.cmml">d</mi></mrow><annotation-xml encoding="MathML-Content" id="A5.p5.16.m16.1b"><apply id="A5.p5.16.m16.1.1.cmml" xref="A5.p5.16.m16.1.1"><times id="A5.p5.16.m16.1.1.1.cmml" xref="A5.p5.16.m16.1.1.1"></times><ci id="A5.p5.16.m16.1.1.2.cmml" xref="A5.p5.16.m16.1.1.2">ℎ</ci><ci id="A5.p5.16.m16.1.1.3.cmml" xref="A5.p5.16.m16.1.1.3">𝑑</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.p5.16.m16.1c">hd</annotation></semantics></math> fixed) and apply the final linear layer to obtain the desired output. This model can be jointly optimized with diffusion model while achieving good performance.</p>
</div>
<div id="A5.p6" class="ltx_para">
<p id="A5.p6.1" class="ltx_p"><span id="A5.p6.1.1" class="ltx_text ltx_font_bold">Pretrained Set Transformer</span> We make use of Set Transformer for dataset encoding pretrained in <cite class="ltx_cite ltx_citemacro_citep">(Lee et al., <a href="#bib.bib16" title="" class="ltx_ref">2021</a>)</cite>. The idea is to use the frozen Set Transformer then add a single linear layer to adapt the output to our problem and use it as dataset encoder. This approach reduces the computation cost of training the Set Transformer enabling jointly optimization of the dataset encoder and the diffusion model. We provide the results of these datasets encoding schemes in Table<a href="#A8.T8" title="Table 8 ‣ H.1 Coupling with Accuracy Predictor ‣ Appendix H Detail of Experimental Results ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a> on the Hyperzoo dataset.</p>
</div>
<figure id="A5.T5" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="A5.T5.8.3.1" class="ltx_text" style="font-size:90%;">Table 5</span>: </span><span id="A5.T5.4.2" class="ltx_text" style="font-size:90%;">Models seting, <math id="A5.T5.3.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A5.T5.3.1.m1.1b"><mi id="A5.T5.3.1.m1.1.1" xref="A5.T5.3.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A5.T5.3.1.m1.1c"><ci id="A5.T5.3.1.m1.1.1.cmml" xref="A5.T5.3.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.T5.3.1.m1.1d">n</annotation></semantics></math> and <math id="A5.T5.4.2.m2.1" class="ltx_Math" alttext="c" display="inline"><semantics id="A5.T5.4.2.m2.1b"><mi id="A5.T5.4.2.m2.1.1" xref="A5.T5.4.2.m2.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="A5.T5.4.2.m2.1c"><ci id="A5.T5.4.2.m2.1.1.cmml" xref="A5.T5.4.2.m2.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="A5.T5.4.2.m2.1d">c</annotation></semantics></math> in the dataset configuration represent respectively the number of samples per class n=5 for training and c the total number of classes per dataset. The VAE and the diffusion models share similar configuration and architectures as <cite class="ltx_cite ltx_citemacro_citep">(Rombach et al., <a href="#bib.bib24" title="" class="ltx_ref">2021</a>)</cite></span></figcaption>
<div id="A5.T5.6" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:664.4pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(105.2pt,-161.2pt) scale(1.94257700027145,1.94257700027145) ;">
<table id="A5.T5.6.2" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A5.T5.6.2.3.1" class="ltx_tr">
<th id="A5.T5.6.2.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">Parameters</th>
<td id="A5.T5.6.2.3.1.2" class="ltx_td ltx_align_center ltx_border_tt">Values</td>
</tr>
<tr id="A5.T5.6.2.4.2" class="ltx_tr">
<th id="A5.T5.6.2.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">epochs</th>
<td id="A5.T5.6.2.4.2.2" class="ltx_td ltx_align_center ltx_border_t">[50, 2000]</td>
</tr>
<tr id="A5.T5.6.2.5.3" class="ltx_tr">
<th id="A5.T5.6.2.5.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="2">VAE model</th>
</tr>
<tr id="A5.T5.6.2.6.4" class="ltx_tr">
<th id="A5.T5.6.2.6.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">optimizer</th>
<td id="A5.T5.6.2.6.4.2" class="ltx_td ltx_align_center ltx_border_t">Adam</td>
</tr>
<tr id="A5.T5.6.2.7.5" class="ltx_tr">
<th id="A5.T5.6.2.7.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Learning rate</th>
<td id="A5.T5.6.2.7.5.2" class="ltx_td ltx_align_center">1e-3</td>
</tr>
<tr id="A5.T5.6.2.8.6" class="ltx_tr">
<th id="A5.T5.6.2.8.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">latent_size</th>
<td id="A5.T5.6.2.8.6.2" class="ltx_td ltx_align_center">1024</td>
</tr>
<tr id="A5.T5.6.2.9.7" class="ltx_tr">
<th id="A5.T5.6.2.9.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">kl_weights</th>
<td id="A5.T5.6.2.9.7.2" class="ltx_td ltx_align_center">1e-6</td>
</tr>
<tr id="A5.T5.6.2.10.8" class="ltx_tr">
<th id="A5.T5.6.2.10.8.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="2">Dataset encoder</th>
</tr>
<tr id="A5.T5.6.2.11.9" class="ltx_tr">
<th id="A5.T5.6.2.11.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">architecture</th>
<td id="A5.T5.6.2.11.9.2" class="ltx_td ltx_align_center ltx_border_t">Set transformer</td>
</tr>
<tr id="A5.T5.5.1.1" class="ltx_tr">
<th id="A5.T5.5.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">input dim</th>
<td id="A5.T5.5.1.1.1" class="ltx_td ltx_align_center">
<math id="A5.T5.5.1.1.1.m1.1" class="ltx_Math" alttext="c\times n\times 512" display="inline"><semantics id="A5.T5.5.1.1.1.m1.1a"><mrow id="A5.T5.5.1.1.1.m1.1.1" xref="A5.T5.5.1.1.1.m1.1.1.cmml"><mi id="A5.T5.5.1.1.1.m1.1.1.2" xref="A5.T5.5.1.1.1.m1.1.1.2.cmml">c</mi><mo lspace="0.222em" rspace="0.222em" id="A5.T5.5.1.1.1.m1.1.1.1" xref="A5.T5.5.1.1.1.m1.1.1.1.cmml">×</mo><mi id="A5.T5.5.1.1.1.m1.1.1.3" xref="A5.T5.5.1.1.1.m1.1.1.3.cmml">n</mi><mo lspace="0.222em" rspace="0.222em" id="A5.T5.5.1.1.1.m1.1.1.1a" xref="A5.T5.5.1.1.1.m1.1.1.1.cmml">×</mo><mn id="A5.T5.5.1.1.1.m1.1.1.4" xref="A5.T5.5.1.1.1.m1.1.1.4.cmml">512</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.T5.5.1.1.1.m1.1b"><apply id="A5.T5.5.1.1.1.m1.1.1.cmml" xref="A5.T5.5.1.1.1.m1.1.1"><times id="A5.T5.5.1.1.1.m1.1.1.1.cmml" xref="A5.T5.5.1.1.1.m1.1.1.1"></times><ci id="A5.T5.5.1.1.1.m1.1.1.2.cmml" xref="A5.T5.5.1.1.1.m1.1.1.2">𝑐</ci><ci id="A5.T5.5.1.1.1.m1.1.1.3.cmml" xref="A5.T5.5.1.1.1.m1.1.1.3">𝑛</ci><cn type="integer" id="A5.T5.5.1.1.1.m1.1.1.4.cmml" xref="A5.T5.5.1.1.1.m1.1.1.4">512</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T5.5.1.1.1.m1.1c">c\times n\times 512</annotation></semantics></math>(min)</td>
</tr>
<tr id="A5.T5.6.2.12.10" class="ltx_tr">
<th id="A5.T5.6.2.12.10.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">output dim</th>
<td id="A5.T5.6.2.12.10.2" class="ltx_td ltx_align_center">1024 (min)</td>
</tr>
<tr id="A5.T5.6.2.13.11" class="ltx_tr">
<th id="A5.T5.6.2.13.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">number of set transformers</th>
<td id="A5.T5.6.2.13.11.2" class="ltx_td ltx_align_center">2</td>
</tr>
<tr id="A5.T5.6.2.14.12" class="ltx_tr">
<th id="A5.T5.6.2.14.12.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t" colspan="2">Diffusion</th>
</tr>
<tr id="A5.T5.6.2.15.13" class="ltx_tr">
<th id="A5.T5.6.2.15.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">optimizer</th>
<td id="A5.T5.6.2.15.13.2" class="ltx_td ltx_align_center ltx_border_t">AdamW</td>
</tr>
<tr id="A5.T5.6.2.16.14" class="ltx_tr">
<th id="A5.T5.6.2.16.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">base learning rate</th>
<td id="A5.T5.6.2.16.14.2" class="ltx_td ltx_align_center">1e-4</td>
</tr>
<tr id="A5.T5.6.2.17.15" class="ltx_tr">
<th id="A5.T5.6.2.17.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">scheduler</th>
<td id="A5.T5.6.2.17.15.2" class="ltx_td ltx_align_center">linear</td>
</tr>
<tr id="A5.T5.6.2.18.16" class="ltx_tr">
<th id="A5.T5.6.2.18.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Time step</th>
<td id="A5.T5.6.2.18.16.2" class="ltx_td ltx_align_center">1024</td>
</tr>
<tr id="A5.T5.6.2.19.17" class="ltx_tr">
<th id="A5.T5.6.2.19.17.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">backward network</th>
<td id="A5.T5.6.2.19.17.2" class="ltx_td ltx_align_center">Unet</td>
</tr>
<tr id="A5.T5.6.2.2" class="ltx_tr">
<th id="A5.T5.6.2.2.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">input to Unet</th>
<td id="A5.T5.6.2.2.1" class="ltx_td ltx_align_center ltx_border_bb"><math id="A5.T5.6.2.2.1.m1.1" class="ltx_Math" alttext="(c\times 32\times 32)" display="inline"><semantics id="A5.T5.6.2.2.1.m1.1a"><mrow id="A5.T5.6.2.2.1.m1.1.1.1" xref="A5.T5.6.2.2.1.m1.1.1.1.1.cmml"><mo stretchy="false" id="A5.T5.6.2.2.1.m1.1.1.1.2" xref="A5.T5.6.2.2.1.m1.1.1.1.1.cmml">(</mo><mrow id="A5.T5.6.2.2.1.m1.1.1.1.1" xref="A5.T5.6.2.2.1.m1.1.1.1.1.cmml"><mi id="A5.T5.6.2.2.1.m1.1.1.1.1.2" xref="A5.T5.6.2.2.1.m1.1.1.1.1.2.cmml">c</mi><mo lspace="0.222em" rspace="0.222em" id="A5.T5.6.2.2.1.m1.1.1.1.1.1" xref="A5.T5.6.2.2.1.m1.1.1.1.1.1.cmml">×</mo><mn id="A5.T5.6.2.2.1.m1.1.1.1.1.3" xref="A5.T5.6.2.2.1.m1.1.1.1.1.3.cmml">32</mn><mo lspace="0.222em" rspace="0.222em" id="A5.T5.6.2.2.1.m1.1.1.1.1.1a" xref="A5.T5.6.2.2.1.m1.1.1.1.1.1.cmml">×</mo><mn id="A5.T5.6.2.2.1.m1.1.1.1.1.4" xref="A5.T5.6.2.2.1.m1.1.1.1.1.4.cmml">32</mn></mrow><mo stretchy="false" id="A5.T5.6.2.2.1.m1.1.1.1.3" xref="A5.T5.6.2.2.1.m1.1.1.1.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="A5.T5.6.2.2.1.m1.1b"><apply id="A5.T5.6.2.2.1.m1.1.1.1.1.cmml" xref="A5.T5.6.2.2.1.m1.1.1.1"><times id="A5.T5.6.2.2.1.m1.1.1.1.1.1.cmml" xref="A5.T5.6.2.2.1.m1.1.1.1.1.1"></times><ci id="A5.T5.6.2.2.1.m1.1.1.1.1.2.cmml" xref="A5.T5.6.2.2.1.m1.1.1.1.1.2">𝑐</ci><cn type="integer" id="A5.T5.6.2.2.1.m1.1.1.1.1.3.cmml" xref="A5.T5.6.2.2.1.m1.1.1.1.1.3">32</cn><cn type="integer" id="A5.T5.6.2.2.1.m1.1.1.1.1.4.cmml" xref="A5.T5.6.2.2.1.m1.1.1.1.1.4">32</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T5.6.2.2.1.m1.1c">(c\times 32\times 32)</annotation></semantics></math></td>
</tr>
</tbody>
</table>
</span></div>
</figure>
</section>
<section id="A6" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix F </span>Training Details</h2>

<div id="A6.p1" class="ltx_para">
<p id="A6.p1.1" class="ltx_p"><span id="A6.p1.1.1" class="ltx_text ltx_font_bold">Training D2NWG</span>. The training process consists of two main stages: In stage 1 we train the predictor then unconditionally train the VAE accoding to equation <a href="#A4.E7" title="In Appendix D Predictor Training ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>. However, we also train the VAE following equation <a href="#S3.E1" title="In 3.2 Weight Encoding ‣ 3 Approach ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> without a predictor in the main experiments. In stage 2 we train the LDM module with the Set Transforter based dataset encoding module(not that depending on the choice of dataset encoder, another pretraining stage may be required). Although, the dataset encoder can be optimized together with the diffusion processes, we prefer to train them separately to speed-up the training process while reducing the memory requirement. The VAE and the dataset encoder are trained with Adam optimizer and learning rate set to 1e-4. The diffusion model in each experiment is trained with a linear scheduler, base learning rate 1e-4 and, AdamW optimizer as defined in <cite class="ltx_cite ltx_citemacro_citep">(Rombach et al., <a href="#bib.bib24" title="" class="ltx_ref">2021</a>)</cite>. During the training process of the diffusion model the output of the dataset encoder is concatenated to the latent representation of the input weights forming the input to the UNET module. Additionally, we investigate joint training of the diffusion process in the ablation study and appendix section <a href="#S4.SS4.SSS3" title="4.4.3 CLIP-based Dataset Encoding ‣ 4.4 Ablation study ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.4.3</span></a> <a href="#A5" title="Appendix E Details of the Proposed Model ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E</span></a>. Additional details can be found in Table <a href="#A5.T5" title="Table 5 ‣ Appendix E Details of the Proposed Model ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a></p>
</div>
</section>
<section id="A7" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix G </span>Additional results</h2>

<section id="A7.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.1 </span>Fast Convergence Performance Evaluation</h3>

<div id="A7.SS1.p1" class="ltx_para">
<p id="A7.SS1.p1.1" class="ltx_p">In this section we report supplementary results for experiment in section <a href="#S4.SS1" title="4.1 In-distribution Transfer Learning ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1</span></a> in Table <a href="#A7.T6" title="Table 6 ‣ G.1 Fast Convergence Performance Evaluation ‣ Appendix G Additional results ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> which is a full version of Table <a href="#S4.T1" title="Table 1 ‣ 4.1 In-distribution Transfer Learning ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.</p>
</div>
<figure id="A7.T6" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="A7.T6.39.1.1" class="ltx_text" style="font-size:90%;">Table 6</span>: </span><span id="A7.T6.40.2" class="ltx_text" style="font-size:90%;">Convergence speed comparison with S<sub id="A7.T6.40.2.1" class="ltx_sub"><span id="A7.T6.40.2.1.1" class="ltx_text ltx_font_typewriter">KDE30</span></sub> <cite class="ltx_cite ltx_citemacro_citep">(Schürholt et al., <a href="#bib.bib25" title="" class="ltx_ref">2022a</a>)</cite>.</span></figcaption>
<div id="A7.T6.36.36" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:303.5pt;height:148.5pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-32.2pt,15.7pt) scale(0.825151752786948,0.825151752786948) ;">
<table id="A7.T6.36.36.36" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A7.T6.36.36.36.37.1" class="ltx_tr">
<th id="A7.T6.36.36.36.37.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="A7.T6.36.36.36.37.1.1.1" class="ltx_text" style="font-size:90%;">Model</span></th>
<th id="A7.T6.36.36.36.37.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A7.T6.36.36.36.37.1.2.1" class="ltx_text" style="font-size:90%;">Epoch</span></th>
<th id="A7.T6.36.36.36.37.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A7.T6.36.36.36.37.1.3.1" class="ltx_text" style="font-size:90%;">MNIST</span></th>
<th id="A7.T6.36.36.36.37.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A7.T6.36.36.36.37.1.4.1" class="ltx_text" style="font-size:90%;">SVHN</span></th>
<th id="A7.T6.36.36.36.37.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A7.T6.36.36.36.37.1.5.1" class="ltx_text" style="font-size:90%;">CIFAR-10</span></th>
<th id="A7.T6.36.36.36.37.1.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A7.T6.36.36.36.37.1.6.1" class="ltx_text" style="font-size:90%;">STL-10</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A7.T6.4.4.4.4" class="ltx_tr">
<th id="A7.T6.4.4.4.4.5" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t"><span id="A7.T6.4.4.4.4.5.1" class="ltx_text" style="font-size:90%;">Pretrained models</span></th>
<td id="A7.T6.4.4.4.4.6" class="ltx_td ltx_align_center ltx_border_t"><span id="A7.T6.4.4.4.4.6.1" class="ltx_text" style="font-size:90%;">25</span></td>
<td id="A7.T6.1.1.1.1.1" class="ltx_td ltx_align_center ltx_border_t"><span id="A7.T6.1.1.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">82.82<math id="A7.T6.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.1.1.1.1.1.1.m1.1a"><mo id="A7.T6.1.1.1.1.1.1.m1.1.1" xref="A7.T6.1.1.1.1.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.1.1.1.1.1.1.m1.1b"><csymbol cd="latexml" id="A7.T6.1.1.1.1.1.1.m1.1.1.cmml" xref="A7.T6.1.1.1.1.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.1.1.1.1.1.1.m1.1c">\pm</annotation></semantics></math> 1.38</span></td>
<td id="A7.T6.2.2.2.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A7.T6.2.2.2.2.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">67.57<math id="A7.T6.2.2.2.2.2.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.2.2.2.2.2.1.m1.1a"><mo id="A7.T6.2.2.2.2.2.1.m1.1.1" xref="A7.T6.2.2.2.2.2.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.2.2.2.2.2.1.m1.1b"><csymbol cd="latexml" id="A7.T6.2.2.2.2.2.1.m1.1.1.cmml" xref="A7.T6.2.2.2.2.2.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.2.2.2.2.2.1.m1.1c">\pm</annotation></semantics></math> 0.59</span></td>
<td id="A7.T6.3.3.3.3.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A7.T6.3.3.3.3.3.1" class="ltx_text ltx_font_bold" style="font-size:90%;">44.68<math id="A7.T6.3.3.3.3.3.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.3.3.3.3.3.1.m1.1a"><mo id="A7.T6.3.3.3.3.3.1.m1.1.1" xref="A7.T6.3.3.3.3.3.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.3.3.3.3.3.1.m1.1b"><csymbol cd="latexml" id="A7.T6.3.3.3.3.3.1.m1.1.1.cmml" xref="A7.T6.3.3.3.3.3.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.3.3.3.3.3.1.m1.1c">\pm</annotation></semantics></math> 3.15</span></td>
<td id="A7.T6.4.4.4.4.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A7.T6.4.4.4.4.4.1" class="ltx_text ltx_font_bold" style="font-size:90%;">35.99<math id="A7.T6.4.4.4.4.4.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.4.4.4.4.4.1.m1.1a"><mo id="A7.T6.4.4.4.4.4.1.m1.1.1" xref="A7.T6.4.4.4.4.4.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.4.4.4.4.4.1.m1.1b"><csymbol cd="latexml" id="A7.T6.4.4.4.4.4.1.m1.1.1.cmml" xref="A7.T6.4.4.4.4.4.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.4.4.4.4.4.1.m1.1c">\pm</annotation></semantics></math> 1.15</span></td>
</tr>
<tr id="A7.T6.8.8.8.8" class="ltx_tr">
<th id="A7.T6.8.8.8.8.5" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T6.8.8.8.8.5.1" class="ltx_text" style="font-size:90%;">S</span><sub id="A7.T6.8.8.8.8.5.2" class="ltx_sub"><span id="A7.T6.8.8.8.8.5.2.1" class="ltx_text ltx_font_typewriter" style="font-size:90%;">KDE30</span></sub>
</th>
<td id="A7.T6.8.8.8.8.6" class="ltx_td ltx_align_center"><span id="A7.T6.8.8.8.8.6.1" class="ltx_text" style="font-size:90%;">0</span></td>
<td id="A7.T6.5.5.5.5.1" class="ltx_td ltx_align_center">
<span id="A7.T6.5.5.5.5.1.1" class="ltx_text" style="font-size:90%;">69.73</span><math id="A7.T6.5.5.5.5.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.5.5.5.5.1.m1.1a"><mo mathsize="90%" id="A7.T6.5.5.5.5.1.m1.1.1" xref="A7.T6.5.5.5.5.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.5.5.5.5.1.m1.1b"><csymbol cd="latexml" id="A7.T6.5.5.5.5.1.m1.1.1.cmml" xref="A7.T6.5.5.5.5.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.5.5.5.5.1.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.5.5.5.5.1.2" class="ltx_text" style="font-size:90%;"> 5.12</span>
</td>
<td id="A7.T6.6.6.6.6.2" class="ltx_td ltx_align_center">
<span id="A7.T6.6.6.6.6.2.1" class="ltx_text" style="font-size:90%;">50.25</span><math id="A7.T6.6.6.6.6.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.6.6.6.6.2.m1.1a"><mo mathsize="90%" id="A7.T6.6.6.6.6.2.m1.1.1" xref="A7.T6.6.6.6.6.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.6.6.6.6.2.m1.1b"><csymbol cd="latexml" id="A7.T6.6.6.6.6.2.m1.1.1.cmml" xref="A7.T6.6.6.6.6.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.6.6.6.6.2.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.6.6.6.6.2.2" class="ltx_text" style="font-size:90%;"> 6.12</span>
</td>
<td id="A7.T6.7.7.7.7.3" class="ltx_td ltx_align_center">
<span id="A7.T6.7.7.7.7.3.1" class="ltx_text" style="font-size:90%;">26.06</span><math id="A7.T6.7.7.7.7.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.7.7.7.7.3.m1.1a"><mo mathsize="90%" id="A7.T6.7.7.7.7.3.m1.1.1" xref="A7.T6.7.7.7.7.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.7.7.7.7.3.m1.1b"><csymbol cd="latexml" id="A7.T6.7.7.7.7.3.m1.1.1.cmml" xref="A7.T6.7.7.7.7.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.7.7.7.7.3.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.7.7.7.7.3.2" class="ltx_text" style="font-size:90%;"> 3.01</span>
</td>
<td id="A7.T6.8.8.8.8.4" class="ltx_td ltx_align_center">
<span id="A7.T6.8.8.8.8.4.1" class="ltx_text" style="font-size:90%;">17.20</span><math id="A7.T6.8.8.8.8.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.8.8.8.8.4.m1.1a"><mo mathsize="90%" id="A7.T6.8.8.8.8.4.m1.1.1" xref="A7.T6.8.8.8.8.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.8.8.8.8.4.m1.1b"><csymbol cd="latexml" id="A7.T6.8.8.8.8.4.m1.1.1.cmml" xref="A7.T6.8.8.8.8.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.8.8.8.8.4.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.8.8.8.8.4.2" class="ltx_text" style="font-size:90%;"> 3.43</span>
</td>
</tr>
<tr id="A7.T6.12.12.12.12" class="ltx_tr">
<th id="A7.T6.12.12.12.12.5" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A7.T6.12.12.12.12.5.1" class="ltx_text" style="font-size:90%;">D2NWG (Ours)</span></th>
<td id="A7.T6.12.12.12.12.6" class="ltx_td ltx_align_center"><span id="A7.T6.12.12.12.12.6.1" class="ltx_text" style="font-size:90%;">0</span></td>
<td id="A7.T6.9.9.9.9.1" class="ltx_td ltx_align_center">
<span id="A7.T6.9.9.9.9.1.1" class="ltx_text" style="font-size:90%;">78.10</span><math id="A7.T6.9.9.9.9.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.9.9.9.9.1.m1.1a"><mo mathsize="90%" id="A7.T6.9.9.9.9.1.m1.1.1" xref="A7.T6.9.9.9.9.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.9.9.9.9.1.m1.1b"><csymbol cd="latexml" id="A7.T6.9.9.9.9.1.m1.1.1.cmml" xref="A7.T6.9.9.9.9.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.9.9.9.9.1.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.9.9.9.9.1.2" class="ltx_text" style="font-size:90%;">0.29</span>
</td>
<td id="A7.T6.10.10.10.10.2" class="ltx_td ltx_align_center">
<span id="A7.T6.10.10.10.10.2.1" class="ltx_text" style="font-size:90%;">58.92</span><math id="A7.T6.10.10.10.10.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.10.10.10.10.2.m1.1a"><mo mathsize="90%" id="A7.T6.10.10.10.10.2.m1.1.1" xref="A7.T6.10.10.10.10.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.10.10.10.10.2.m1.1b"><csymbol cd="latexml" id="A7.T6.10.10.10.10.2.m1.1.1.cmml" xref="A7.T6.10.10.10.10.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.10.10.10.10.2.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.10.10.10.10.2.2" class="ltx_text" style="font-size:90%;"> 0.76</span>
</td>
<td id="A7.T6.11.11.11.11.3" class="ltx_td ltx_align_center">
<span id="A7.T6.11.11.11.11.3.1" class="ltx_text" style="font-size:90%;">38.23</span><math id="A7.T6.11.11.11.11.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.11.11.11.11.3.m1.1a"><mo mathsize="90%" id="A7.T6.11.11.11.11.3.m1.1.1" xref="A7.T6.11.11.11.11.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.11.11.11.11.3.m1.1b"><csymbol cd="latexml" id="A7.T6.11.11.11.11.3.m1.1.1.cmml" xref="A7.T6.11.11.11.11.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.11.11.11.11.3.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.11.11.11.11.3.2" class="ltx_text" style="font-size:90%;"> 5.45</span>
</td>
<td id="A7.T6.12.12.12.12.4" class="ltx_td ltx_align_center">
<span id="A7.T6.12.12.12.12.4.1" class="ltx_text" style="font-size:90%;">22.74</span><math id="A7.T6.12.12.12.12.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.12.12.12.12.4.m1.1a"><mo mathsize="90%" id="A7.T6.12.12.12.12.4.m1.1.1" xref="A7.T6.12.12.12.12.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.12.12.12.12.4.m1.1b"><csymbol cd="latexml" id="A7.T6.12.12.12.12.4.m1.1.1.cmml" xref="A7.T6.12.12.12.12.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.12.12.12.12.4.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.12.12.12.12.4.2" class="ltx_text" style="font-size:90%;"> 2.08</span>
</td>
</tr>
<tr id="A7.T6.16.16.16.16" class="ltx_tr">
<th id="A7.T6.16.16.16.16.5" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t"><span id="A7.T6.16.16.16.16.5.1" class="ltx_text" style="font-size:90%;">Pretrained models</span></th>
<td id="A7.T6.16.16.16.16.6" class="ltx_td ltx_align_center ltx_border_t"><span id="A7.T6.16.16.16.16.6.1" class="ltx_text" style="font-size:90%;">26</span></td>
<td id="A7.T6.13.13.13.13.1" class="ltx_td ltx_align_center ltx_border_t"><span id="A7.T6.13.13.13.13.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">85.50<math id="A7.T6.13.13.13.13.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.13.13.13.13.1.1.m1.1a"><mo id="A7.T6.13.13.13.13.1.1.m1.1.1" xref="A7.T6.13.13.13.13.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.13.13.13.13.1.1.m1.1b"><csymbol cd="latexml" id="A7.T6.13.13.13.13.1.1.m1.1.1.cmml" xref="A7.T6.13.13.13.13.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.13.13.13.13.1.1.m1.1c">\pm</annotation></semantics></math> 0.99</span></td>
<td id="A7.T6.14.14.14.14.2" class="ltx_td ltx_align_center ltx_border_t">
<span id="A7.T6.14.14.14.14.2.1" class="ltx_text" style="font-size:90%;">67.79</span><math id="A7.T6.14.14.14.14.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.14.14.14.14.2.m1.1a"><mo mathsize="90%" id="A7.T6.14.14.14.14.2.m1.1.1" xref="A7.T6.14.14.14.14.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.14.14.14.14.2.m1.1b"><csymbol cd="latexml" id="A7.T6.14.14.14.14.2.m1.1.1.cmml" xref="A7.T6.14.14.14.14.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.14.14.14.14.2.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.14.14.14.14.2.2" class="ltx_text" style="font-size:90%;"> 1.05</span>
</td>
<td id="A7.T6.15.15.15.15.3" class="ltx_td ltx_align_center ltx_border_t">
<span id="A7.T6.15.15.15.15.3.1" class="ltx_text" style="font-size:90%;">44.81</span><math id="A7.T6.15.15.15.15.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.15.15.15.15.3.m1.1a"><mo mathsize="90%" id="A7.T6.15.15.15.15.3.m1.1.1" xref="A7.T6.15.15.15.15.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.15.15.15.15.3.m1.1b"><csymbol cd="latexml" id="A7.T6.15.15.15.15.3.m1.1.1.cmml" xref="A7.T6.15.15.15.15.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.15.15.15.15.3.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.15.15.15.15.3.2" class="ltx_text" style="font-size:90%;"> 3.20</span>
</td>
<td id="A7.T6.16.16.16.16.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A7.T6.16.16.16.16.4.1" class="ltx_text ltx_font_bold" style="font-size:90%;">36.28<math id="A7.T6.16.16.16.16.4.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.16.16.16.16.4.1.m1.1a"><mo id="A7.T6.16.16.16.16.4.1.m1.1.1" xref="A7.T6.16.16.16.16.4.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.16.16.16.16.4.1.m1.1b"><csymbol cd="latexml" id="A7.T6.16.16.16.16.4.1.m1.1.1.cmml" xref="A7.T6.16.16.16.16.4.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.16.16.16.16.4.1.m1.1c">\pm</annotation></semantics></math> 1.18</span></td>
</tr>
<tr id="A7.T6.20.20.20.20" class="ltx_tr">
<th id="A7.T6.20.20.20.20.5" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T6.20.20.20.20.5.1" class="ltx_text" style="font-size:90%;">S</span><sub id="A7.T6.20.20.20.20.5.2" class="ltx_sub"><span id="A7.T6.20.20.20.20.5.2.1" class="ltx_text ltx_font_typewriter" style="font-size:90%;">KDE30</span></sub>
</th>
<td id="A7.T6.20.20.20.20.6" class="ltx_td ltx_align_center"><span id="A7.T6.20.20.20.20.6.1" class="ltx_text" style="font-size:90%;">1</span></td>
<td id="A7.T6.17.17.17.17.1" class="ltx_td ltx_align_center">
<span id="A7.T6.17.17.17.17.1.1" class="ltx_text" style="font-size:90%;">82.69</span><math id="A7.T6.17.17.17.17.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.17.17.17.17.1.m1.1a"><mo mathsize="90%" id="A7.T6.17.17.17.17.1.m1.1.1" xref="A7.T6.17.17.17.17.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.17.17.17.17.1.m1.1b"><csymbol cd="latexml" id="A7.T6.17.17.17.17.1.m1.1.1.cmml" xref="A7.T6.17.17.17.17.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.17.17.17.17.1.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.17.17.17.17.1.2" class="ltx_text" style="font-size:90%;"> 0.81</span>
</td>
<td id="A7.T6.18.18.18.18.2" class="ltx_td ltx_align_center">
<span id="A7.T6.18.18.18.18.2.1" class="ltx_text" style="font-size:90%;">70.45</span><math id="A7.T6.18.18.18.18.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.18.18.18.18.2.m1.1a"><mo mathsize="90%" id="A7.T6.18.18.18.18.2.m1.1.1" xref="A7.T6.18.18.18.18.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.18.18.18.18.2.m1.1b"><csymbol cd="latexml" id="A7.T6.18.18.18.18.2.m1.1.1.cmml" xref="A7.T6.18.18.18.18.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.18.18.18.18.2.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.18.18.18.18.2.2" class="ltx_text" style="font-size:90%;"> 0.92</span>
</td>
<td id="A7.T6.19.19.19.19.3" class="ltx_td ltx_align_center">
<span id="A7.T6.19.19.19.19.3.1" class="ltx_text" style="font-size:90%;">44.75</span><math id="A7.T6.19.19.19.19.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.19.19.19.19.3.m1.1a"><mo mathsize="90%" id="A7.T6.19.19.19.19.3.m1.1.1" xref="A7.T6.19.19.19.19.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.19.19.19.19.3.m1.1b"><csymbol cd="latexml" id="A7.T6.19.19.19.19.3.m1.1.1.cmml" xref="A7.T6.19.19.19.19.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.19.19.19.19.3.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.19.19.19.19.3.2" class="ltx_text" style="font-size:90%;"> 0.81</span>
</td>
<td id="A7.T6.20.20.20.20.4" class="ltx_td ltx_align_center">
<span id="A7.T6.20.20.20.20.4.1" class="ltx_text" style="font-size:90%;">28.42</span><math id="A7.T6.20.20.20.20.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.20.20.20.20.4.m1.1a"><mo mathsize="90%" id="A7.T6.20.20.20.20.4.m1.1.1" xref="A7.T6.20.20.20.20.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.20.20.20.20.4.m1.1b"><csymbol cd="latexml" id="A7.T6.20.20.20.20.4.m1.1.1.cmml" xref="A7.T6.20.20.20.20.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.20.20.20.20.4.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.20.20.20.20.4.2" class="ltx_text" style="font-size:90%;"> 1.53</span>
</td>
</tr>
<tr id="A7.T6.24.24.24.24" class="ltx_tr">
<th id="A7.T6.24.24.24.24.5" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="A7.T6.24.24.24.24.5.1" class="ltx_text" style="font-size:90%;">D2NWG (Ours)</span></th>
<td id="A7.T6.24.24.24.24.6" class="ltx_td ltx_align_center"><span id="A7.T6.24.24.24.24.6.1" class="ltx_text" style="font-size:90%;">1</span></td>
<td id="A7.T6.21.21.21.21.1" class="ltx_td ltx_align_center">
<span id="A7.T6.21.21.21.21.1.1" class="ltx_text" style="font-size:90%;">83.25</span><math id="A7.T6.21.21.21.21.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.21.21.21.21.1.m1.1a"><mo mathsize="90%" id="A7.T6.21.21.21.21.1.m1.1.1" xref="A7.T6.21.21.21.21.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.21.21.21.21.1.m1.1b"><csymbol cd="latexml" id="A7.T6.21.21.21.21.1.m1.1.1.cmml" xref="A7.T6.21.21.21.21.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.21.21.21.21.1.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.21.21.21.21.1.2" class="ltx_text" style="font-size:90%;"> 0.25</span>
</td>
<td id="A7.T6.22.22.22.22.2" class="ltx_td ltx_align_center"><span id="A7.T6.22.22.22.22.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">73.11<math id="A7.T6.22.22.22.22.2.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.22.22.22.22.2.1.m1.1a"><mo id="A7.T6.22.22.22.22.2.1.m1.1.1" xref="A7.T6.22.22.22.22.2.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.22.22.22.22.2.1.m1.1b"><csymbol cd="latexml" id="A7.T6.22.22.22.22.2.1.m1.1.1.cmml" xref="A7.T6.22.22.22.22.2.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.22.22.22.22.2.1.m1.1c">\pm</annotation></semantics></math>0.24</span></td>
<td id="A7.T6.23.23.23.23.3" class="ltx_td ltx_align_center"><span id="A7.T6.23.23.23.23.3.1" class="ltx_text ltx_font_bold" style="font-size:90%;">46.88<math id="A7.T6.23.23.23.23.3.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.23.23.23.23.3.1.m1.1a"><mo id="A7.T6.23.23.23.23.3.1.m1.1.1" xref="A7.T6.23.23.23.23.3.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.23.23.23.23.3.1.m1.1b"><csymbol cd="latexml" id="A7.T6.23.23.23.23.3.1.m1.1.1.cmml" xref="A7.T6.23.23.23.23.3.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.23.23.23.23.3.1.m1.1c">\pm</annotation></semantics></math> 0.85</span></td>
<td id="A7.T6.24.24.24.24.4" class="ltx_td ltx_align_center">
<span id="A7.T6.24.24.24.24.4.1" class="ltx_text" style="font-size:90%;">34.39</span><math id="A7.T6.24.24.24.24.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.24.24.24.24.4.m1.1a"><mo mathsize="90%" id="A7.T6.24.24.24.24.4.m1.1.1" xref="A7.T6.24.24.24.24.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.24.24.24.24.4.m1.1b"><csymbol cd="latexml" id="A7.T6.24.24.24.24.4.m1.1.1.cmml" xref="A7.T6.24.24.24.24.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.24.24.24.24.4.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.24.24.24.24.4.2" class="ltx_text" style="font-size:90%;"> 0.51</span>
</td>
</tr>
<tr id="A7.T6.28.28.28.28" class="ltx_tr">
<th id="A7.T6.28.28.28.28.5" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t"><span id="A7.T6.28.28.28.28.5.1" class="ltx_text" style="font-size:90%;">Pretrained models</span></th>
<td id="A7.T6.28.28.28.28.6" class="ltx_td ltx_align_center ltx_border_t"><span id="A7.T6.28.28.28.28.6.1" class="ltx_text" style="font-size:90%;">50</span></td>
<td id="A7.T6.25.25.25.25.1" class="ltx_td ltx_align_center ltx_border_t">
<span id="A7.T6.25.25.25.25.1.1" class="ltx_text" style="font-size:90%;">95.70</span><math id="A7.T6.25.25.25.25.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.25.25.25.25.1.m1.1a"><mo mathsize="90%" id="A7.T6.25.25.25.25.1.m1.1.1" xref="A7.T6.25.25.25.25.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.25.25.25.25.1.m1.1b"><csymbol cd="latexml" id="A7.T6.25.25.25.25.1.m1.1.1.cmml" xref="A7.T6.25.25.25.25.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.25.25.25.25.1.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.25.25.25.25.1.2" class="ltx_text" style="font-size:90%;"> 0.36</span>
</td>
<td id="A7.T6.26.26.26.26.2" class="ltx_td ltx_align_center ltx_border_t">
<span id="A7.T6.26.26.26.26.2.1" class="ltx_text" style="font-size:90%;">74.49</span><math id="A7.T6.26.26.26.26.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.26.26.26.26.2.m1.1a"><mo mathsize="90%" id="A7.T6.26.26.26.26.2.m1.1.1" xref="A7.T6.26.26.26.26.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.26.26.26.26.2.m1.1b"><csymbol cd="latexml" id="A7.T6.26.26.26.26.2.m1.1.1.cmml" xref="A7.T6.26.26.26.26.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.26.26.26.26.2.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.26.26.26.26.2.2" class="ltx_text" style="font-size:90%;"> 1.53</span>
</td>
<td id="A7.T6.27.27.27.27.3" class="ltx_td ltx_align_center ltx_border_t">
<span id="A7.T6.27.27.27.27.3.1" class="ltx_text" style="font-size:90%;">49.25</span><math id="A7.T6.27.27.27.27.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.27.27.27.27.3.m1.1a"><mo mathsize="90%" id="A7.T6.27.27.27.27.3.m1.1.1" xref="A7.T6.27.27.27.27.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.27.27.27.27.3.m1.1b"><csymbol cd="latexml" id="A7.T6.27.27.27.27.3.m1.1.1.cmml" xref="A7.T6.27.27.27.27.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.27.27.27.27.3.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.27.27.27.27.3.2" class="ltx_text" style="font-size:90%;"> 3.00</span>
</td>
<td id="A7.T6.28.28.28.28.4" class="ltx_td ltx_align_center ltx_border_t">
<span id="A7.T6.28.28.28.28.4.1" class="ltx_text" style="font-size:90%;">39.39</span><math id="A7.T6.28.28.28.28.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.28.28.28.28.4.m1.1a"><mo mathsize="90%" id="A7.T6.28.28.28.28.4.m1.1.1" xref="A7.T6.28.28.28.28.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.28.28.28.28.4.m1.1b"><csymbol cd="latexml" id="A7.T6.28.28.28.28.4.m1.1.1.cmml" xref="A7.T6.28.28.28.28.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.28.28.28.28.4.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.28.28.28.28.4.2" class="ltx_text" style="font-size:90%;">0.89</span>
</td>
</tr>
<tr id="A7.T6.32.32.32.32" class="ltx_tr">
<th id="A7.T6.32.32.32.32.5" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="A7.T6.32.32.32.32.5.1" class="ltx_text" style="font-size:90%;">S</span><sub id="A7.T6.32.32.32.32.5.2" class="ltx_sub"><span id="A7.T6.32.32.32.32.5.2.1" class="ltx_text ltx_font_typewriter" style="font-size:90%;">KDE30</span></sub>
</th>
<td id="A7.T6.32.32.32.32.6" class="ltx_td ltx_align_center"><span id="A7.T6.32.32.32.32.6.1" class="ltx_text" style="font-size:90%;">25</span></td>
<td id="A7.T6.29.29.29.29.1" class="ltx_td ltx_align_center">
<span id="A7.T6.29.29.29.29.1.1" class="ltx_text" style="font-size:90%;">95.70</span><math id="A7.T6.29.29.29.29.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.29.29.29.29.1.m1.1a"><mo mathsize="90%" id="A7.T6.29.29.29.29.1.m1.1.1" xref="A7.T6.29.29.29.29.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.29.29.29.29.1.m1.1b"><csymbol cd="latexml" id="A7.T6.29.29.29.29.1.m1.1.1.cmml" xref="A7.T6.29.29.29.29.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.29.29.29.29.1.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.29.29.29.29.1.2" class="ltx_text" style="font-size:90%;"> 0.30</span>
</td>
<td id="A7.T6.30.30.30.30.2" class="ltx_td ltx_align_center">
<span id="A7.T6.30.30.30.30.2.1" class="ltx_text" style="font-size:90%;">74.69</span><math id="A7.T6.30.30.30.30.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.30.30.30.30.2.m1.1a"><mo mathsize="90%" id="A7.T6.30.30.30.30.2.m1.1.1" xref="A7.T6.30.30.30.30.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.30.30.30.30.2.m1.1b"><csymbol cd="latexml" id="A7.T6.30.30.30.30.2.m1.1.1.cmml" xref="A7.T6.30.30.30.30.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.30.30.30.30.2.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.30.30.30.30.2.2" class="ltx_text" style="font-size:90%;"> 0.70</span>
</td>
<td id="A7.T6.31.31.31.31.3" class="ltx_td ltx_align_center">
<span id="A7.T6.31.31.31.31.3.1" class="ltx_text" style="font-size:90%;">49.14</span><math id="A7.T6.31.31.31.31.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.31.31.31.31.3.m1.1a"><mo mathsize="90%" id="A7.T6.31.31.31.31.3.m1.1.1" xref="A7.T6.31.31.31.31.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.31.31.31.31.3.m1.1b"><csymbol cd="latexml" id="A7.T6.31.31.31.31.3.m1.1.1.cmml" xref="A7.T6.31.31.31.31.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.31.31.31.31.3.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.31.31.31.31.3.2" class="ltx_text" style="font-size:90%;"> 0.65</span>
</td>
<td id="A7.T6.32.32.32.32.4" class="ltx_td ltx_align_center">
<span id="A7.T6.32.32.32.32.4.1" class="ltx_text" style="font-size:90%;">39.14</span><math id="A7.T6.32.32.32.32.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.32.32.32.32.4.m1.1a"><mo mathsize="90%" id="A7.T6.32.32.32.32.4.m1.1.1" xref="A7.T6.32.32.32.32.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.32.32.32.32.4.m1.1b"><csymbol cd="latexml" id="A7.T6.32.32.32.32.4.m1.1.1.cmml" xref="A7.T6.32.32.32.32.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.32.32.32.32.4.m1.1c">\pm</annotation></semantics></math><span id="A7.T6.32.32.32.32.4.2" class="ltx_text" style="font-size:90%;"> 0.76</span>
</td>
</tr>
<tr id="A7.T6.36.36.36.36" class="ltx_tr">
<th id="A7.T6.36.36.36.36.5" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="A7.T6.36.36.36.36.5.1" class="ltx_text" style="font-size:90%;">D2NWG (Ours)</span></th>
<td id="A7.T6.36.36.36.36.6" class="ltx_td ltx_align_center ltx_border_bb"><span id="A7.T6.36.36.36.36.6.1" class="ltx_text" style="font-size:90%;">25</span></td>
<td id="A7.T6.33.33.33.33.1" class="ltx_td ltx_align_center ltx_border_bb"><span id="A7.T6.33.33.33.33.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">96.17<math id="A7.T6.33.33.33.33.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.33.33.33.33.1.1.m1.1a"><mo id="A7.T6.33.33.33.33.1.1.m1.1.1" xref="A7.T6.33.33.33.33.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.33.33.33.33.1.1.m1.1b"><csymbol cd="latexml" id="A7.T6.33.33.33.33.1.1.m1.1.1.cmml" xref="A7.T6.33.33.33.33.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.33.33.33.33.1.1.m1.1c">\pm</annotation></semantics></math>0.0.33</span></td>
<td id="A7.T6.34.34.34.34.2" class="ltx_td ltx_align_center ltx_border_bb">
<span id="A7.T6.34.34.34.34.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">74.75<math id="A7.T6.34.34.34.34.2.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.34.34.34.34.2.1.m1.1a"><mo id="A7.T6.34.34.34.34.2.1.m1.1.1" xref="A7.T6.34.34.34.34.2.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.34.34.34.34.2.1.m1.1b"><csymbol cd="latexml" id="A7.T6.34.34.34.34.2.1.m1.1.1.cmml" xref="A7.T6.34.34.34.34.2.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.34.34.34.34.2.1.m1.1c">\pm</annotation></semantics></math></span><span id="A7.T6.34.34.34.34.2.2" class="ltx_text" style="font-size:90%;"> 0.33</span>
</td>
<td id="A7.T6.35.35.35.35.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="A7.T6.35.35.35.35.3.1" class="ltx_text ltx_font_bold" style="font-size:90%;">49.48<math id="A7.T6.35.35.35.35.3.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.35.35.35.35.3.1.m1.1a"><mo id="A7.T6.35.35.35.35.3.1.m1.1.1" xref="A7.T6.35.35.35.35.3.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.35.35.35.35.3.1.m1.1b"><csymbol cd="latexml" id="A7.T6.35.35.35.35.3.1.m1.1.1.cmml" xref="A7.T6.35.35.35.35.3.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.35.35.35.35.3.1.m1.1c">\pm</annotation></semantics></math> 0.97</span></td>
<td id="A7.T6.36.36.36.36.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="A7.T6.36.36.36.36.4.1" class="ltx_text ltx_font_bold" style="font-size:90%;">39.43<math id="A7.T6.36.36.36.36.4.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A7.T6.36.36.36.36.4.1.m1.1a"><mo id="A7.T6.36.36.36.36.4.1.m1.1.1" xref="A7.T6.36.36.36.36.4.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A7.T6.36.36.36.36.4.1.m1.1b"><csymbol cd="latexml" id="A7.T6.36.36.36.36.4.1.m1.1.1.cmml" xref="A7.T6.36.36.36.36.4.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A7.T6.36.36.36.36.4.1.m1.1c">\pm</annotation></semantics></math> 0.13</span></td>
</tr>
</tbody>
</table>
</span></div>
</figure>
</section>
</section>
<section id="A8" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix H </span>Detail of Experimental Results</h2>

<section id="A8.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">H.1 </span>Coupling with Accuracy Predictor</h3>

<div id="A8.SS1.p1" class="ltx_para">
<p id="A8.SS1.p1.1" class="ltx_p">This section reports the extended results of Table <a href="#S4.T2" title="Table 2 ‣ 4.1 In-distribution Transfer Learning ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> in which we compared our method in-distribution and out-of distribution with and without accuracy predictor.</p>
</div>
<div id="A8.SS1.p2" class="ltx_para">
<p id="A8.SS1.p2.1" class="ltx_p"><span id="A8.SS1.p2.1.1" class="ltx_text ltx_font_bold">Results.</span>: The full results of Table <a href="#S4.T2" title="Table 2 ‣ 4.1 In-distribution Transfer Learning ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> are reported in Table <a href="#A8.T7" title="Table 7 ‣ H.1 Coupling with Accuracy Predictor ‣ Appendix H Detail of Experimental Results ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>. Using an accuracy predictor enable easily selecting highly performing when sampling in-distribution. However, in our case the accuracy predictor struggles to generalize well for unseen dataset as shown in Table <a href="#A8.T7" title="Table 7 ‣ H.1 Coupling with Accuracy Predictor ‣ Appendix H Detail of Experimental Results ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a></p>
</div>
<figure id="A8.T7" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="A8.T7.34.1.1" class="ltx_text" style="font-size:90%;">Table 7</span>: </span><span id="A8.T7.35.2" class="ltx_text" style="font-size:90%;">Performance evaluation at initialization without fine-tuning. For the baseline we use weights of SVHN for MNIST and vice versa similarly for CIFAR-10 and STL-10</span></figcaption>
<div id="A8.T7.32.32" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:346.9pt;height:83.9pt;vertical-align:-0.5pt;"><span class="ltx_transformed_inner" style="transform:translate(-182.4pt,43.9pt) scale(0.48743304921364,0.48743304921364) ;">
<table id="A8.T7.32.32.32" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A8.T7.32.32.32.33.1" class="ltx_tr">
<th id="A8.T7.32.32.32.33.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_tt">Datasets</th>
<th id="A8.T7.32.32.32.33.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">MNIST</th>
<th id="A8.T7.32.32.32.33.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">SVHN</th>
<th id="A8.T7.32.32.32.33.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">CIFAR10</th>
<th id="A8.T7.32.32.32.33.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">STL10</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A8.T7.4.4.4.4" class="ltx_tr">
<th id="A8.T7.4.4.4.4.5" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t">Random</th>
<td id="A8.T7.1.1.1.1.1" class="ltx_td ltx_align_center ltx_border_t">10.23<math id="A8.T7.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.1.1.1.1.1.m1.1a"><mo id="A8.T7.1.1.1.1.1.m1.1.1" xref="A8.T7.1.1.1.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.1.1.1.1.1.m1.1b"><csymbol cd="latexml" id="A8.T7.1.1.1.1.1.m1.1.1.cmml" xref="A8.T7.1.1.1.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.1.1.1.1.1.m1.1c">\pm</annotation></semantics></math>0.56</td>
<td id="A8.T7.2.2.2.2.2" class="ltx_td ltx_align_center ltx_border_t">12.21<math id="A8.T7.2.2.2.2.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.2.2.2.2.2.m1.1a"><mo id="A8.T7.2.2.2.2.2.m1.1.1" xref="A8.T7.2.2.2.2.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.2.2.2.2.2.m1.1b"><csymbol cd="latexml" id="A8.T7.2.2.2.2.2.m1.1.1.cmml" xref="A8.T7.2.2.2.2.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.2.2.2.2.2.m1.1c">\pm</annotation></semantics></math>3.76</td>
<td id="A8.T7.3.3.3.3.3" class="ltx_td ltx_align_center ltx_border_t">9.98<math id="A8.T7.3.3.3.3.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.3.3.3.3.3.m1.1a"><mo id="A8.T7.3.3.3.3.3.m1.1.1" xref="A8.T7.3.3.3.3.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.3.3.3.3.3.m1.1b"><csymbol cd="latexml" id="A8.T7.3.3.3.3.3.m1.1.1.cmml" xref="A8.T7.3.3.3.3.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.3.3.3.3.3.m1.1c">\pm</annotation></semantics></math>1.47</td>
<td id="A8.T7.4.4.4.4.4" class="ltx_td ltx_align_center ltx_border_t">9.56<math id="A8.T7.4.4.4.4.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.4.4.4.4.4.m1.1a"><mo id="A8.T7.4.4.4.4.4.m1.1.1" xref="A8.T7.4.4.4.4.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.4.4.4.4.4.m1.1b"><csymbol cd="latexml" id="A8.T7.4.4.4.4.4.m1.1.1.cmml" xref="A8.T7.4.4.4.4.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.4.4.4.4.4.m1.1c">\pm</annotation></semantics></math>1.02</td>
</tr>
<tr id="A8.T7.8.8.8.8" class="ltx_tr">
<th id="A8.T7.8.8.8.8.5" class="ltx_td ltx_align_center ltx_th ltx_th_row">Pretrained models</th>
<td id="A8.T7.5.5.5.5.1" class="ltx_td ltx_align_center">82.82<math id="A8.T7.5.5.5.5.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.5.5.5.5.1.m1.1a"><mo id="A8.T7.5.5.5.5.1.m1.1.1" xref="A8.T7.5.5.5.5.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.5.5.5.5.1.m1.1b"><csymbol cd="latexml" id="A8.T7.5.5.5.5.1.m1.1.1.cmml" xref="A8.T7.5.5.5.5.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.5.5.5.5.1.m1.1c">\pm</annotation></semantics></math> 1.38</td>
<td id="A8.T7.6.6.6.6.2" class="ltx_td ltx_align_center"><span id="A8.T7.6.6.6.6.2.1" class="ltx_text ltx_font_bold">67.57<math id="A8.T7.6.6.6.6.2.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.6.6.6.6.2.1.m1.1a"><mo id="A8.T7.6.6.6.6.2.1.m1.1.1" xref="A8.T7.6.6.6.6.2.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.6.6.6.6.2.1.m1.1b"><csymbol cd="latexml" id="A8.T7.6.6.6.6.2.1.m1.1.1.cmml" xref="A8.T7.6.6.6.6.2.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.6.6.6.6.2.1.m1.1c">\pm</annotation></semantics></math> 0.59</span></td>
<td id="A8.T7.7.7.7.7.3" class="ltx_td ltx_align_center"><span id="A8.T7.7.7.7.7.3.1" class="ltx_text ltx_font_bold">44.68<math id="A8.T7.7.7.7.7.3.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.7.7.7.7.3.1.m1.1a"><mo id="A8.T7.7.7.7.7.3.1.m1.1.1" xref="A8.T7.7.7.7.7.3.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.7.7.7.7.3.1.m1.1b"><csymbol cd="latexml" id="A8.T7.7.7.7.7.3.1.m1.1.1.cmml" xref="A8.T7.7.7.7.7.3.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.7.7.7.7.3.1.m1.1c">\pm</annotation></semantics></math> 3.15</span></td>
<td id="A8.T7.8.8.8.8.4" class="ltx_td ltx_align_center"><span id="A8.T7.8.8.8.8.4.1" class="ltx_text ltx_font_bold">35.99<math id="A8.T7.8.8.8.8.4.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.8.8.8.8.4.1.m1.1a"><mo id="A8.T7.8.8.8.8.4.1.m1.1.1" xref="A8.T7.8.8.8.8.4.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.8.8.8.8.4.1.m1.1b"><csymbol cd="latexml" id="A8.T7.8.8.8.8.4.1.m1.1.1.cmml" xref="A8.T7.8.8.8.8.4.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.8.8.8.8.4.1.m1.1c">\pm</annotation></semantics></math> 1.15</span></td>
</tr>
<tr id="A8.T7.12.12.12.12" class="ltx_tr">
<th id="A8.T7.12.12.12.12.5" class="ltx_td ltx_align_center ltx_th ltx_th_row">S<sub id="A8.T7.12.12.12.12.5.1" class="ltx_sub">kde30</sub><cite class="ltx_cite ltx_citemacro_citet">Schürholt et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022a</a>)</cite>
</th>
<td id="A8.T7.9.9.9.9.1" class="ltx_td ltx_align_center">69.73<math id="A8.T7.9.9.9.9.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.9.9.9.9.1.m1.1a"><mo id="A8.T7.9.9.9.9.1.m1.1.1" xref="A8.T7.9.9.9.9.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.9.9.9.9.1.m1.1b"><csymbol cd="latexml" id="A8.T7.9.9.9.9.1.m1.1.1.cmml" xref="A8.T7.9.9.9.9.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.9.9.9.9.1.m1.1c">\pm</annotation></semantics></math> 5.12</td>
<td id="A8.T7.10.10.10.10.2" class="ltx_td ltx_align_center">50.25<math id="A8.T7.10.10.10.10.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.10.10.10.10.2.m1.1a"><mo id="A8.T7.10.10.10.10.2.m1.1.1" xref="A8.T7.10.10.10.10.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.10.10.10.10.2.m1.1b"><csymbol cd="latexml" id="A8.T7.10.10.10.10.2.m1.1.1.cmml" xref="A8.T7.10.10.10.10.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.10.10.10.10.2.m1.1c">\pm</annotation></semantics></math> 6.12</td>
<td id="A8.T7.11.11.11.11.3" class="ltx_td ltx_align_center">26.06<math id="A8.T7.11.11.11.11.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.11.11.11.11.3.m1.1a"><mo id="A8.T7.11.11.11.11.3.m1.1.1" xref="A8.T7.11.11.11.11.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.11.11.11.11.3.m1.1b"><csymbol cd="latexml" id="A8.T7.11.11.11.11.3.m1.1.1.cmml" xref="A8.T7.11.11.11.11.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.11.11.11.11.3.m1.1c">\pm</annotation></semantics></math> 3.01</td>
<td id="A8.T7.12.12.12.12.4" class="ltx_td ltx_align_center">17.20<math id="A8.T7.12.12.12.12.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.12.12.12.12.4.m1.1a"><mo id="A8.T7.12.12.12.12.4.m1.1.1" xref="A8.T7.12.12.12.12.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.12.12.12.12.4.m1.1b"><csymbol cd="latexml" id="A8.T7.12.12.12.12.4.m1.1.1.cmml" xref="A8.T7.12.12.12.12.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.12.12.12.12.4.m1.1c">\pm</annotation></semantics></math> 3.43</td>
</tr>
<tr id="A8.T7.16.16.16.16" class="ltx_tr">
<th id="A8.T7.16.16.16.16.5" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t">seen (D2NWG)</th>
<td id="A8.T7.13.13.13.13.1" class="ltx_td ltx_align_center ltx_border_t">83.92<math id="A8.T7.13.13.13.13.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.13.13.13.13.1.m1.1a"><mo id="A8.T7.13.13.13.13.1.m1.1.1" xref="A8.T7.13.13.13.13.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.13.13.13.13.1.m1.1b"><csymbol cd="latexml" id="A8.T7.13.13.13.13.1.m1.1.1.cmml" xref="A8.T7.13.13.13.13.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.13.13.13.13.1.m1.1c">\pm</annotation></semantics></math>1.92</td>
<td id="A8.T7.14.14.14.14.2" class="ltx_td ltx_align_center ltx_border_t">61.81 <math id="A8.T7.14.14.14.14.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.14.14.14.14.2.m1.1a"><mo id="A8.T7.14.14.14.14.2.m1.1.1" xref="A8.T7.14.14.14.14.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.14.14.14.14.2.m1.1b"><csymbol cd="latexml" id="A8.T7.14.14.14.14.2.m1.1.1.cmml" xref="A8.T7.14.14.14.14.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.14.14.14.14.2.m1.1c">\pm</annotation></semantics></math> 3.13</td>
<td id="A8.T7.15.15.15.15.3" class="ltx_td ltx_align_center ltx_border_t">43.08<math id="A8.T7.15.15.15.15.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.15.15.15.15.3.m1.1a"><mo id="A8.T7.15.15.15.15.3.m1.1.1" xref="A8.T7.15.15.15.15.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.15.15.15.15.3.m1.1b"><csymbol cd="latexml" id="A8.T7.15.15.15.15.3.m1.1.1.cmml" xref="A8.T7.15.15.15.15.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.15.15.15.15.3.m1.1c">\pm</annotation></semantics></math>0.55</td>
<td id="A8.T7.16.16.16.16.4" class="ltx_td ltx_align_center ltx_border_t">31.45<math id="A8.T7.16.16.16.16.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.16.16.16.16.4.m1.1a"><mo id="A8.T7.16.16.16.16.4.m1.1.1" xref="A8.T7.16.16.16.16.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.16.16.16.16.4.m1.1b"><csymbol cd="latexml" id="A8.T7.16.16.16.16.4.m1.1.1.cmml" xref="A8.T7.16.16.16.16.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.16.16.16.16.4.m1.1c">\pm</annotation></semantics></math>0.35</td>
</tr>
<tr id="A8.T7.20.20.20.20" class="ltx_tr">
<th id="A8.T7.20.20.20.20.5" class="ltx_td ltx_align_center ltx_th ltx_th_row">seen(D2NWG)(with Pred)</th>
<td id="A8.T7.17.17.17.17.1" class="ltx_td ltx_align_center">84.85<math id="A8.T7.17.17.17.17.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.17.17.17.17.1.m1.1a"><mo id="A8.T7.17.17.17.17.1.m1.1.1" xref="A8.T7.17.17.17.17.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.17.17.17.17.1.m1.1b"><csymbol cd="latexml" id="A8.T7.17.17.17.17.1.m1.1.1.cmml" xref="A8.T7.17.17.17.17.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.17.17.17.17.1.m1.1c">\pm</annotation></semantics></math>0.83</td>
<td id="A8.T7.18.18.18.18.2" class="ltx_td ltx_align_center">66.03 <math id="A8.T7.18.18.18.18.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.18.18.18.18.2.m1.1a"><mo id="A8.T7.18.18.18.18.2.m1.1.1" xref="A8.T7.18.18.18.18.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.18.18.18.18.2.m1.1b"><csymbol cd="latexml" id="A8.T7.18.18.18.18.2.m1.1.1.cmml" xref="A8.T7.18.18.18.18.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.18.18.18.18.2.m1.1c">\pm</annotation></semantics></math> 1.36</td>
<td id="A8.T7.19.19.19.19.3" class="ltx_td ltx_align_center">43.89<math id="A8.T7.19.19.19.19.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.19.19.19.19.3.m1.1a"><mo id="A8.T7.19.19.19.19.3.m1.1.1" xref="A8.T7.19.19.19.19.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.19.19.19.19.3.m1.1b"><csymbol cd="latexml" id="A8.T7.19.19.19.19.3.m1.1.1.cmml" xref="A8.T7.19.19.19.19.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.19.19.19.19.3.m1.1c">\pm</annotation></semantics></math>0.15</td>
<td id="A8.T7.20.20.20.20.4" class="ltx_td ltx_align_center">34.29<math id="A8.T7.20.20.20.20.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.20.20.20.20.4.m1.1a"><mo id="A8.T7.20.20.20.20.4.m1.1.1" xref="A8.T7.20.20.20.20.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.20.20.20.20.4.m1.1b"><csymbol cd="latexml" id="A8.T7.20.20.20.20.4.m1.1.1.cmml" xref="A8.T7.20.20.20.20.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.20.20.20.20.4.m1.1c">\pm</annotation></semantics></math>0.13</td>
</tr>
<tr id="A8.T7.24.24.24.24" class="ltx_tr">
<th id="A8.T7.24.24.24.24.5" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t">S<sub id="A8.T7.24.24.24.24.5.1" class="ltx_sub">kde30</sub><cite class="ltx_cite ltx_citemacro_citet">Schürholt et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022a</a>)</cite>(cross)</th>
<td id="A8.T7.21.21.21.21.1" class="ltx_td ltx_align_center ltx_border_t">29.30<math id="A8.T7.21.21.21.21.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.21.21.21.21.1.m1.1a"><mo id="A8.T7.21.21.21.21.1.m1.1.1" xref="A8.T7.21.21.21.21.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.21.21.21.21.1.m1.1b"><csymbol cd="latexml" id="A8.T7.21.21.21.21.1.m1.1.1.cmml" xref="A8.T7.21.21.21.21.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.21.21.21.21.1.m1.1c">\pm</annotation></semantics></math> 3.46</td>
<td id="A8.T7.22.22.22.22.2" class="ltx_td ltx_align_center ltx_border_t">13.25<math id="A8.T7.22.22.22.22.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.22.22.22.22.2.m1.1a"><mo id="A8.T7.22.22.22.22.2.m1.1.1" xref="A8.T7.22.22.22.22.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.22.22.22.22.2.m1.1b"><csymbol cd="latexml" id="A8.T7.22.22.22.22.2.m1.1.1.cmml" xref="A8.T7.22.22.22.22.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.22.22.22.22.2.m1.1c">\pm</annotation></semantics></math> 1.12</td>
<td id="A8.T7.23.23.23.23.3" class="ltx_td ltx_align_center ltx_border_t">15.40<math id="A8.T7.23.23.23.23.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.23.23.23.23.3.m1.1a"><mo id="A8.T7.23.23.23.23.3.m1.1.1" xref="A8.T7.23.23.23.23.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.23.23.23.23.3.m1.1b"><csymbol cd="latexml" id="A8.T7.23.23.23.23.3.m1.1.1.cmml" xref="A8.T7.23.23.23.23.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.23.23.23.23.3.m1.1c">\pm</annotation></semantics></math> 0.51</td>
<td id="A8.T7.24.24.24.24.4" class="ltx_td ltx_align_center ltx_border_t">15.20<math id="A8.T7.24.24.24.24.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.24.24.24.24.4.m1.1a"><mo id="A8.T7.24.24.24.24.4.m1.1.1" xref="A8.T7.24.24.24.24.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.24.24.24.24.4.m1.1b"><csymbol cd="latexml" id="A8.T7.24.24.24.24.4.m1.1.1.cmml" xref="A8.T7.24.24.24.24.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.24.24.24.24.4.m1.1c">\pm</annotation></semantics></math>1.24</td>
</tr>
<tr id="A8.T7.28.28.28.28" class="ltx_tr">
<th id="A8.T7.28.28.28.28.5" class="ltx_td ltx_align_center ltx_th ltx_th_row">not seen(D2NWG)</th>
<td id="A8.T7.25.25.25.25.1" class="ltx_td ltx_align_center">36.64<math id="A8.T7.25.25.25.25.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.25.25.25.25.1.m1.1a"><mo id="A8.T7.25.25.25.25.1.m1.1.1" xref="A8.T7.25.25.25.25.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.25.25.25.25.1.m1.1b"><csymbol cd="latexml" id="A8.T7.25.25.25.25.1.m1.1.1.cmml" xref="A8.T7.25.25.25.25.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.25.25.25.25.1.m1.1c">\pm</annotation></semantics></math>4.69</td>
<td id="A8.T7.26.26.26.26.2" class="ltx_td ltx_align_center">18.80<math id="A8.T7.26.26.26.26.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.26.26.26.26.2.m1.1a"><mo id="A8.T7.26.26.26.26.2.m1.1.1" xref="A8.T7.26.26.26.26.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.26.26.26.26.2.m1.1b"><csymbol cd="latexml" id="A8.T7.26.26.26.26.2.m1.1.1.cmml" xref="A8.T7.26.26.26.26.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.26.26.26.26.2.m1.1c">\pm</annotation></semantics></math>0.58</td>
<td id="A8.T7.27.27.27.27.3" class="ltx_td ltx_align_center">18.00<math id="A8.T7.27.27.27.27.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.27.27.27.27.3.m1.1a"><mo id="A8.T7.27.27.27.27.3.m1.1.1" xref="A8.T7.27.27.27.27.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.27.27.27.27.3.m1.1b"><csymbol cd="latexml" id="A8.T7.27.27.27.27.3.m1.1.1.cmml" xref="A8.T7.27.27.27.27.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.27.27.27.27.3.m1.1c">\pm</annotation></semantics></math>0.22</td>
<td id="A8.T7.28.28.28.28.4" class="ltx_td ltx_align_center">16.21<math id="A8.T7.28.28.28.28.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.28.28.28.28.4.m1.1a"><mo id="A8.T7.28.28.28.28.4.m1.1.1" xref="A8.T7.28.28.28.28.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.28.28.28.28.4.m1.1b"><csymbol cd="latexml" id="A8.T7.28.28.28.28.4.m1.1.1.cmml" xref="A8.T7.28.28.28.28.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.28.28.28.28.4.m1.1c">\pm</annotation></semantics></math>0.52</td>
</tr>
<tr id="A8.T7.32.32.32.32" class="ltx_tr">
<th id="A8.T7.32.32.32.32.5" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb">not seen(D2NWG)(with Pred)</th>
<td id="A8.T7.29.29.29.29.1" class="ltx_td ltx_align_center ltx_border_bb">30.15<math id="A8.T7.29.29.29.29.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.29.29.29.29.1.m1.1a"><mo id="A8.T7.29.29.29.29.1.m1.1.1" xref="A8.T7.29.29.29.29.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.29.29.29.29.1.m1.1b"><csymbol cd="latexml" id="A8.T7.29.29.29.29.1.m1.1.1.cmml" xref="A8.T7.29.29.29.29.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.29.29.29.29.1.m1.1c">\pm</annotation></semantics></math>5.09</td>
<td id="A8.T7.30.30.30.30.2" class="ltx_td ltx_align_center ltx_border_bb">15.76<math id="A8.T7.30.30.30.30.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.30.30.30.30.2.m1.1a"><mo id="A8.T7.30.30.30.30.2.m1.1.1" xref="A8.T7.30.30.30.30.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.30.30.30.30.2.m1.1b"><csymbol cd="latexml" id="A8.T7.30.30.30.30.2.m1.1.1.cmml" xref="A8.T7.30.30.30.30.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.30.30.30.30.2.m1.1c">\pm</annotation></semantics></math>1.43</td>
<td id="A8.T7.31.31.31.31.3" class="ltx_td ltx_align_center ltx_border_bb">17.10<math id="A8.T7.31.31.31.31.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.31.31.31.31.3.m1.1a"><mo id="A8.T7.31.31.31.31.3.m1.1.1" xref="A8.T7.31.31.31.31.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.31.31.31.31.3.m1.1b"><csymbol cd="latexml" id="A8.T7.31.31.31.31.3.m1.1.1.cmml" xref="A8.T7.31.31.31.31.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.31.31.31.31.3.m1.1c">\pm</annotation></semantics></math>1.12</td>
<td id="A8.T7.32.32.32.32.4" class="ltx_td ltx_align_center ltx_border_bb">15.37<math id="A8.T7.32.32.32.32.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T7.32.32.32.32.4.m1.1a"><mo id="A8.T7.32.32.32.32.4.m1.1.1" xref="A8.T7.32.32.32.32.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T7.32.32.32.32.4.m1.1b"><csymbol cd="latexml" id="A8.T7.32.32.32.32.4.m1.1.1.cmml" xref="A8.T7.32.32.32.32.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T7.32.32.32.32.4.m1.1c">\pm</annotation></semantics></math>0.52</td>
</tr>
</tbody>
</table>
</span></div>
</figure>
<figure id="A8.T8" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="A8.T8.26.1.1" class="ltx_text" style="font-size:90%;">Table 8</span>: </span><span id="A8.T8.27.2" class="ltx_text" style="font-size:90%;">In-distribution performance comparison of different image dataset encoding schemes on model zoo dataset</span></figcaption>
<div id="A8.T8.24" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:346.9pt;height:62.7pt;vertical-align:-0.5pt;"><span class="ltx_transformed_inner" style="transform:translate(-177.9pt,31.9pt) scale(0.49361756843149,0.49361756843149) ;">
<table id="A8.T8.24.24" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A8.T8.24.24.25.1" class="ltx_tr">
<th id="A8.T8.24.24.25.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_tt">Datasets</th>
<td id="A8.T8.24.24.25.1.2" class="ltx_td ltx_align_center ltx_border_tt">MNIST</td>
<td id="A8.T8.24.24.25.1.3" class="ltx_td ltx_align_center ltx_border_tt">SVHN</td>
<td id="A8.T8.24.24.25.1.4" class="ltx_td ltx_align_center ltx_border_tt">CIFAR10</td>
<td id="A8.T8.24.24.25.1.5" class="ltx_td ltx_align_center ltx_border_tt">STL10</td>
</tr>
<tr id="A8.T8.4.4.4" class="ltx_tr">
<th id="A8.T8.4.4.4.5" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t">Pretrained models</th>
<td id="A8.T8.1.1.1.1" class="ltx_td ltx_align_center ltx_border_t">82.82<math id="A8.T8.1.1.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.1.1.1.1.m1.1a"><mo id="A8.T8.1.1.1.1.m1.1.1" xref="A8.T8.1.1.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.1.1.1.1.m1.1b"><csymbol cd="latexml" id="A8.T8.1.1.1.1.m1.1.1.cmml" xref="A8.T8.1.1.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.1.1.1.1.m1.1c">\pm</annotation></semantics></math> 1.38</td>
<td id="A8.T8.2.2.2.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A8.T8.2.2.2.2.1" class="ltx_text ltx_font_bold">67.57<math id="A8.T8.2.2.2.2.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.2.2.2.2.1.m1.1a"><mo id="A8.T8.2.2.2.2.1.m1.1.1" xref="A8.T8.2.2.2.2.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.2.2.2.2.1.m1.1b"><csymbol cd="latexml" id="A8.T8.2.2.2.2.1.m1.1.1.cmml" xref="A8.T8.2.2.2.2.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.2.2.2.2.1.m1.1c">\pm</annotation></semantics></math> 0.59</span></td>
<td id="A8.T8.3.3.3.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A8.T8.3.3.3.3.1" class="ltx_text ltx_font_bold">44.68<math id="A8.T8.3.3.3.3.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.3.3.3.3.1.m1.1a"><mo id="A8.T8.3.3.3.3.1.m1.1.1" xref="A8.T8.3.3.3.3.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.3.3.3.3.1.m1.1b"><csymbol cd="latexml" id="A8.T8.3.3.3.3.1.m1.1.1.cmml" xref="A8.T8.3.3.3.3.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.3.3.3.3.1.m1.1c">\pm</annotation></semantics></math> 3.15</span></td>
<td id="A8.T8.4.4.4.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A8.T8.4.4.4.4.1" class="ltx_text ltx_font_bold">35.99<math id="A8.T8.4.4.4.4.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.4.4.4.4.1.m1.1a"><mo id="A8.T8.4.4.4.4.1.m1.1.1" xref="A8.T8.4.4.4.4.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.4.4.4.4.1.m1.1b"><csymbol cd="latexml" id="A8.T8.4.4.4.4.1.m1.1.1.cmml" xref="A8.T8.4.4.4.4.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.4.4.4.4.1.m1.1c">\pm</annotation></semantics></math> 1.15</span></td>
</tr>
<tr id="A8.T8.8.8.8" class="ltx_tr">
<th id="A8.T8.8.8.8.5" class="ltx_td ltx_align_center ltx_th ltx_th_row">S<sub id="A8.T8.8.8.8.5.1" class="ltx_sub">kde30</sub><cite class="ltx_cite ltx_citemacro_citet">Schürholt et al. (<a href="#bib.bib25" title="" class="ltx_ref">2022a</a>)</cite>
</th>
<td id="A8.T8.5.5.5.1" class="ltx_td ltx_align_center">69.73<math id="A8.T8.5.5.5.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.5.5.5.1.m1.1a"><mo id="A8.T8.5.5.5.1.m1.1.1" xref="A8.T8.5.5.5.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.5.5.5.1.m1.1b"><csymbol cd="latexml" id="A8.T8.5.5.5.1.m1.1.1.cmml" xref="A8.T8.5.5.5.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.5.5.5.1.m1.1c">\pm</annotation></semantics></math> 5.12</td>
<td id="A8.T8.6.6.6.2" class="ltx_td ltx_align_center">50.25<math id="A8.T8.6.6.6.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.6.6.6.2.m1.1a"><mo id="A8.T8.6.6.6.2.m1.1.1" xref="A8.T8.6.6.6.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.6.6.6.2.m1.1b"><csymbol cd="latexml" id="A8.T8.6.6.6.2.m1.1.1.cmml" xref="A8.T8.6.6.6.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.6.6.6.2.m1.1c">\pm</annotation></semantics></math> 6.12</td>
<td id="A8.T8.7.7.7.3" class="ltx_td ltx_align_center">26.06<math id="A8.T8.7.7.7.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.7.7.7.3.m1.1a"><mo id="A8.T8.7.7.7.3.m1.1.1" xref="A8.T8.7.7.7.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.7.7.7.3.m1.1b"><csymbol cd="latexml" id="A8.T8.7.7.7.3.m1.1.1.cmml" xref="A8.T8.7.7.7.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.7.7.7.3.m1.1c">\pm</annotation></semantics></math> 3.01</td>
<td id="A8.T8.8.8.8.4" class="ltx_td ltx_align_center">17.20<math id="A8.T8.8.8.8.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.8.8.8.4.m1.1a"><mo id="A8.T8.8.8.8.4.m1.1.1" xref="A8.T8.8.8.8.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.8.8.8.4.m1.1b"><csymbol cd="latexml" id="A8.T8.8.8.8.4.m1.1.1.cmml" xref="A8.T8.8.8.8.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.8.8.8.4.m1.1c">\pm</annotation></semantics></math> 3.43</td>
</tr>
<tr id="A8.T8.12.12.12" class="ltx_tr">
<th id="A8.T8.12.12.12.5" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_t">MLP_Encoder</th>
<td id="A8.T8.9.9.9.1" class="ltx_td ltx_align_center ltx_border_t">67.04<math id="A8.T8.9.9.9.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.9.9.9.1.m1.1a"><mo id="A8.T8.9.9.9.1.m1.1.1" xref="A8.T8.9.9.9.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.9.9.9.1.m1.1b"><csymbol cd="latexml" id="A8.T8.9.9.9.1.m1.1.1.cmml" xref="A8.T8.9.9.9.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.9.9.9.1.m1.1c">\pm</annotation></semantics></math>17.73</td>
<td id="A8.T8.10.10.10.2" class="ltx_td ltx_align_center ltx_border_t">35.65 <math id="A8.T8.10.10.10.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.10.10.10.2.m1.1a"><mo id="A8.T8.10.10.10.2.m1.1.1" xref="A8.T8.10.10.10.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.10.10.10.2.m1.1b"><csymbol cd="latexml" id="A8.T8.10.10.10.2.m1.1.1.cmml" xref="A8.T8.10.10.10.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.10.10.10.2.m1.1c">\pm</annotation></semantics></math> 13.03</td>
<td id="A8.T8.11.11.11.3" class="ltx_td ltx_align_center ltx_border_t">17.41<math id="A8.T8.11.11.11.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.11.11.11.3.m1.1a"><mo id="A8.T8.11.11.11.3.m1.1.1" xref="A8.T8.11.11.11.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.11.11.11.3.m1.1b"><csymbol cd="latexml" id="A8.T8.11.11.11.3.m1.1.1.cmml" xref="A8.T8.11.11.11.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.11.11.11.3.m1.1c">\pm</annotation></semantics></math>3.02</td>
<td id="A8.T8.12.12.12.4" class="ltx_td ltx_align_center ltx_border_t">20.36<math id="A8.T8.12.12.12.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.12.12.12.4.m1.1a"><mo id="A8.T8.12.12.12.4.m1.1.1" xref="A8.T8.12.12.12.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.12.12.12.4.m1.1b"><csymbol cd="latexml" id="A8.T8.12.12.12.4.m1.1.1.cmml" xref="A8.T8.12.12.12.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.12.12.12.4.m1.1c">\pm</annotation></semantics></math>7.38</td>
</tr>
<tr id="A8.T8.16.16.16" class="ltx_tr">
<th id="A8.T8.16.16.16.5" class="ltx_td ltx_align_center ltx_th ltx_th_row">Set_transf(pret)</th>
<td id="A8.T8.13.13.13.1" class="ltx_td ltx_align_center">78.21<math id="A8.T8.13.13.13.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.13.13.13.1.m1.1a"><mo id="A8.T8.13.13.13.1.m1.1.1" xref="A8.T8.13.13.13.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.13.13.13.1.m1.1b"><csymbol cd="latexml" id="A8.T8.13.13.13.1.m1.1.1.cmml" xref="A8.T8.13.13.13.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.13.13.13.1.m1.1c">\pm</annotation></semantics></math>1.76</td>
<td id="A8.T8.14.14.14.2" class="ltx_td ltx_align_center">60.90 <math id="A8.T8.14.14.14.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.14.14.14.2.m1.1a"><mo id="A8.T8.14.14.14.2.m1.1.1" xref="A8.T8.14.14.14.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.14.14.14.2.m1.1b"><csymbol cd="latexml" id="A8.T8.14.14.14.2.m1.1.1.cmml" xref="A8.T8.14.14.14.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.14.14.14.2.m1.1c">\pm</annotation></semantics></math> 1.08</td>
<td id="A8.T8.15.15.15.3" class="ltx_td ltx_align_center">28.68<math id="A8.T8.15.15.15.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.15.15.15.3.m1.1a"><mo id="A8.T8.15.15.15.3.m1.1.1" xref="A8.T8.15.15.15.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.15.15.15.3.m1.1b"><csymbol cd="latexml" id="A8.T8.15.15.15.3.m1.1.1.cmml" xref="A8.T8.15.15.15.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.15.15.15.3.m1.1c">\pm</annotation></semantics></math>1.84</td>
<td id="A8.T8.16.16.16.4" class="ltx_td ltx_align_center">34.75<math id="A8.T8.16.16.16.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.16.16.16.4.m1.1a"><mo id="A8.T8.16.16.16.4.m1.1.1" xref="A8.T8.16.16.16.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.16.16.16.4.m1.1b"><csymbol cd="latexml" id="A8.T8.16.16.16.4.m1.1.1.cmml" xref="A8.T8.16.16.16.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.16.16.16.4.m1.1c">\pm</annotation></semantics></math>00.38</td>
</tr>
<tr id="A8.T8.20.20.20" class="ltx_tr">
<th id="A8.T8.20.20.20.5" class="ltx_td ltx_align_center ltx_th ltx_th_row">seen (D2NWG)</th>
<td id="A8.T8.17.17.17.1" class="ltx_td ltx_align_center">83.92<math id="A8.T8.17.17.17.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.17.17.17.1.m1.1a"><mo id="A8.T8.17.17.17.1.m1.1.1" xref="A8.T8.17.17.17.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.17.17.17.1.m1.1b"><csymbol cd="latexml" id="A8.T8.17.17.17.1.m1.1.1.cmml" xref="A8.T8.17.17.17.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.17.17.17.1.m1.1c">\pm</annotation></semantics></math>1.92</td>
<td id="A8.T8.18.18.18.2" class="ltx_td ltx_align_center">61.81 <math id="A8.T8.18.18.18.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.18.18.18.2.m1.1a"><mo id="A8.T8.18.18.18.2.m1.1.1" xref="A8.T8.18.18.18.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.18.18.18.2.m1.1b"><csymbol cd="latexml" id="A8.T8.18.18.18.2.m1.1.1.cmml" xref="A8.T8.18.18.18.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.18.18.18.2.m1.1c">\pm</annotation></semantics></math> 3.13</td>
<td id="A8.T8.19.19.19.3" class="ltx_td ltx_align_center">43.08<math id="A8.T8.19.19.19.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.19.19.19.3.m1.1a"><mo id="A8.T8.19.19.19.3.m1.1.1" xref="A8.T8.19.19.19.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.19.19.19.3.m1.1b"><csymbol cd="latexml" id="A8.T8.19.19.19.3.m1.1.1.cmml" xref="A8.T8.19.19.19.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.19.19.19.3.m1.1c">\pm</annotation></semantics></math>0.55</td>
<td id="A8.T8.20.20.20.4" class="ltx_td ltx_align_center">31.45<math id="A8.T8.20.20.20.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.20.20.20.4.m1.1a"><mo id="A8.T8.20.20.20.4.m1.1.1" xref="A8.T8.20.20.20.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.20.20.20.4.m1.1b"><csymbol cd="latexml" id="A8.T8.20.20.20.4.m1.1.1.cmml" xref="A8.T8.20.20.20.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.20.20.20.4.m1.1c">\pm</annotation></semantics></math>0.35</td>
</tr>
<tr id="A8.T8.24.24.24" class="ltx_tr">
<th id="A8.T8.24.24.24.5" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb">seen(D2NWG)(with Pred)</th>
<td id="A8.T8.21.21.21.1" class="ltx_td ltx_align_center ltx_border_bb">84.85<math id="A8.T8.21.21.21.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.21.21.21.1.m1.1a"><mo id="A8.T8.21.21.21.1.m1.1.1" xref="A8.T8.21.21.21.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.21.21.21.1.m1.1b"><csymbol cd="latexml" id="A8.T8.21.21.21.1.m1.1.1.cmml" xref="A8.T8.21.21.21.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.21.21.21.1.m1.1c">\pm</annotation></semantics></math>0.83</td>
<td id="A8.T8.22.22.22.2" class="ltx_td ltx_align_center ltx_border_bb">66.03 <math id="A8.T8.22.22.22.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.22.22.22.2.m1.1a"><mo id="A8.T8.22.22.22.2.m1.1.1" xref="A8.T8.22.22.22.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.22.22.22.2.m1.1b"><csymbol cd="latexml" id="A8.T8.22.22.22.2.m1.1.1.cmml" xref="A8.T8.22.22.22.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.22.22.22.2.m1.1c">\pm</annotation></semantics></math> 1.36</td>
<td id="A8.T8.23.23.23.3" class="ltx_td ltx_align_center ltx_border_bb">43.89<math id="A8.T8.23.23.23.3.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.23.23.23.3.m1.1a"><mo id="A8.T8.23.23.23.3.m1.1.1" xref="A8.T8.23.23.23.3.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.23.23.23.3.m1.1b"><csymbol cd="latexml" id="A8.T8.23.23.23.3.m1.1.1.cmml" xref="A8.T8.23.23.23.3.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.23.23.23.3.m1.1c">\pm</annotation></semantics></math>0.15</td>
<td id="A8.T8.24.24.24.4" class="ltx_td ltx_align_center ltx_border_bb">34.29<math id="A8.T8.24.24.24.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T8.24.24.24.4.m1.1a"><mo id="A8.T8.24.24.24.4.m1.1.1" xref="A8.T8.24.24.24.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T8.24.24.24.4.m1.1b"><csymbol cd="latexml" id="A8.T8.24.24.24.4.m1.1.1.cmml" xref="A8.T8.24.24.24.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T8.24.24.24.4.m1.1c">\pm</annotation></semantics></math>0.13</td>
</tr>
</tbody>
</table>
</span></div>
</figure>
</section>
<section id="A8.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">H.2 </span>Performance Evaluation on Unseen Datasets</h3>

<div id="A8.SS2.p1" class="ltx_para">
<p id="A8.SS2.p1.1" class="ltx_p">In this section we report supplementary results of experiment in Section <a href="#S4.SS2" title="4.2 Sampling Weights for Unseen Datasets ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2</span></a>. Table <a href="#A8.T9" title="Table 9 ‣ H.2 Performance Evaluation on Unseen Datasets ‣ Appendix H Detail of Experimental Results ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">9</span></a> reports the detail results presented in Table <a href="#S4.T3" title="Table 3 ‣ 4.3 Performance Evaluation on KaggleZoo ‣ 4 Experiments ‣ Diffusion-based Neural Network Weights Generation" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.</p>
</div>
<figure id="A8.T9" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="A8.T9.62.1.1" class="ltx_text" style="font-size:90%;">Table 9</span>: </span><span id="A8.T9.63.2" class="ltx_text" style="font-size:90%;">Performance of the datasets conditional sampling on 10 unseen real-world datasets. We report the averaged accuracy on ten unseen test datasets over 3 different runs fine-tuned for 50 epochs. pret(imnet): pretrained on imagenet1k</span></figcaption>
<div id="A8.T9.60.60" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:156.8pt;vertical-align:-0.7pt;"><span class="ltx_transformed_inner" style="transform:translate(-83.2pt,30.0pt) scale(0.722581612377245,0.722581612377245) ;">
<table id="A8.T9.60.60.60" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A8.T9.60.60.60.61.1" class="ltx_tr">
<th id="A8.T9.60.60.60.61.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_r ltx_border_tt" rowspan="2"><span id="A8.T9.60.60.60.61.1.1.1" class="ltx_text">Datasets</span></th>
<th id="A8.T9.60.60.60.61.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt" colspan="3">No-fine-tuning</th>
<th id="A8.T9.60.60.60.61.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="3">50 epochs Fine-Tuning</th>
<th id="A8.T9.60.60.60.61.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" rowspan="2"><span id="A8.T9.60.60.60.61.1.4.1" class="ltx_text"># of classes</span></th>
</tr>
<tr id="A8.T9.60.60.60.62.2" class="ltx_tr">
<th id="A8.T9.60.60.60.62.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column">Random init.</th>
<th id="A8.T9.60.60.60.62.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column">pret(imnet)</th>
<th id="A8.T9.60.60.60.62.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">D2NWG(ours)</th>
<th id="A8.T9.60.60.60.62.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column">Random init.</th>
<th id="A8.T9.60.60.60.62.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column">pret(imnet)</th>
<th id="A8.T9.60.60.60.62.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">D2NWG(ours)</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A8.T9.6.6.6.6" class="ltx_tr">
<td id="A8.T9.6.6.6.6.7" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Gemstones</td>
<td id="A8.T9.1.1.1.1.1" class="ltx_td ltx_align_center ltx_border_t">1.13 <math id="A8.T9.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.1.1.1.1.1.m1.1a"><mo id="A8.T9.1.1.1.1.1.m1.1.1" xref="A8.T9.1.1.1.1.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.1.1.1.1.1.m1.1b"><csymbol cd="latexml" id="A8.T9.1.1.1.1.1.m1.1.1.cmml" xref="A8.T9.1.1.1.1.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.1.1.1.1.1.m1.1c">\pm</annotation></semantics></math> 0.52</td>
<td id="A8.T9.2.2.2.2.2" class="ltx_td ltx_align_center ltx_border_t">0.62 <math id="A8.T9.2.2.2.2.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.2.2.2.2.2.m1.1a"><mo id="A8.T9.2.2.2.2.2.m1.1.1" xref="A8.T9.2.2.2.2.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.2.2.2.2.2.m1.1b"><csymbol cd="latexml" id="A8.T9.2.2.2.2.2.m1.1.1.cmml" xref="A8.T9.2.2.2.2.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.2.2.2.2.2.m1.1c">\pm</annotation></semantics></math> 0.00</td>
<td id="A8.T9.3.3.3.3.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="A8.T9.3.3.3.3.3.1" class="ltx_text ltx_font_bold">1.86 <math id="A8.T9.3.3.3.3.3.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.3.3.3.3.3.1.m1.1a"><mo id="A8.T9.3.3.3.3.3.1.m1.1.1" xref="A8.T9.3.3.3.3.3.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.3.3.3.3.3.1.m1.1b"><csymbol cd="latexml" id="A8.T9.3.3.3.3.3.1.m1.1.1.cmml" xref="A8.T9.3.3.3.3.3.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.3.3.3.3.3.1.m1.1c">\pm</annotation></semantics></math> 0.25</span></td>
<td id="A8.T9.4.4.4.4.4" class="ltx_td ltx_align_center ltx_border_t">70.59<math id="A8.T9.4.4.4.4.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.4.4.4.4.4.m1.1a"><mo id="A8.T9.4.4.4.4.4.m1.1.1" xref="A8.T9.4.4.4.4.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.4.4.4.4.4.m1.1b"><csymbol cd="latexml" id="A8.T9.4.4.4.4.4.m1.1.1.cmml" xref="A8.T9.4.4.4.4.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.4.4.4.4.4.m1.1c">\pm</annotation></semantics></math> 0.91</td>
<td id="A8.T9.5.5.5.5.5" class="ltx_td ltx_align_center ltx_border_t">67.49<math id="A8.T9.5.5.5.5.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.5.5.5.5.5.m1.1a"><mo id="A8.T9.5.5.5.5.5.m1.1.1" xref="A8.T9.5.5.5.5.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.5.5.5.5.5.m1.1b"><csymbol cd="latexml" id="A8.T9.5.5.5.5.5.m1.1.1.cmml" xref="A8.T9.5.5.5.5.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.5.5.5.5.5.m1.1c">\pm</annotation></semantics></math> 0.43</td>
<td id="A8.T9.6.6.6.6.6" class="ltx_td ltx_align_center ltx_border_r ltx_border_t"><span id="A8.T9.6.6.6.6.6.1" class="ltx_text ltx_font_bold">76.06 <math id="A8.T9.6.6.6.6.6.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.6.6.6.6.6.1.m1.1a"><mo id="A8.T9.6.6.6.6.6.1.m1.1.1" xref="A8.T9.6.6.6.6.6.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.6.6.6.6.6.1.m1.1b"><csymbol cd="latexml" id="A8.T9.6.6.6.6.6.1.m1.1.1.cmml" xref="A8.T9.6.6.6.6.6.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.6.6.6.6.6.1.m1.1c">\pm</annotation></semantics></math> 0.88</span></td>
<td id="A8.T9.6.6.6.6.8" class="ltx_td ltx_align_center ltx_border_t">87</td>
</tr>
<tr id="A8.T9.12.12.12.12" class="ltx_tr">
<td id="A8.T9.12.12.12.12.7" class="ltx_td ltx_align_left ltx_border_r">Dog Breeds</td>
<td id="A8.T9.7.7.7.7.1" class="ltx_td ltx_align_center">0.55 <math id="A8.T9.7.7.7.7.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.7.7.7.7.1.m1.1a"><mo id="A8.T9.7.7.7.7.1.m1.1.1" xref="A8.T9.7.7.7.7.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.7.7.7.7.1.m1.1b"><csymbol cd="latexml" id="A8.T9.7.7.7.7.1.m1.1.1.cmml" xref="A8.T9.7.7.7.7.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.7.7.7.7.1.m1.1c">\pm</annotation></semantics></math> 0.22</td>
<td id="A8.T9.8.8.8.8.2" class="ltx_td ltx_align_center">0.69 <math id="A8.T9.8.8.8.8.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.8.8.8.8.2.m1.1a"><mo id="A8.T9.8.8.8.8.2.m1.1.1" xref="A8.T9.8.8.8.8.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.8.8.8.8.2.m1.1b"><csymbol cd="latexml" id="A8.T9.8.8.8.8.2.m1.1.1.cmml" xref="A8.T9.8.8.8.8.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.8.8.8.8.2.m1.1c">\pm</annotation></semantics></math> 0.00</td>
<td id="A8.T9.9.9.9.9.3" class="ltx_td ltx_align_center ltx_border_r"><span id="A8.T9.9.9.9.9.3.1" class="ltx_text ltx_font_bold">1.87 <math id="A8.T9.9.9.9.9.3.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.9.9.9.9.3.1.m1.1a"><mo id="A8.T9.9.9.9.9.3.1.m1.1.1" xref="A8.T9.9.9.9.9.3.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.9.9.9.9.3.1.m1.1b"><csymbol cd="latexml" id="A8.T9.9.9.9.9.3.1.m1.1.1.cmml" xref="A8.T9.9.9.9.9.3.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.9.9.9.9.3.1.m1.1c">\pm</annotation></semantics></math> 0.39</span></td>
<td id="A8.T9.10.10.10.10.4" class="ltx_td ltx_align_center">80.78<math id="A8.T9.10.10.10.10.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.10.10.10.10.4.m1.1a"><mo id="A8.T9.10.10.10.10.4.m1.1.1" xref="A8.T9.10.10.10.10.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.10.10.10.10.4.m1.1b"><csymbol cd="latexml" id="A8.T9.10.10.10.10.4.m1.1.1.cmml" xref="A8.T9.10.10.10.10.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.10.10.10.10.4.m1.1c">\pm</annotation></semantics></math> 0.28</td>
<td id="A8.T9.11.11.11.11.5" class="ltx_td ltx_align_center">78.13<math id="A8.T9.11.11.11.11.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.11.11.11.11.5.m1.1a"><mo id="A8.T9.11.11.11.11.5.m1.1.1" xref="A8.T9.11.11.11.11.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.11.11.11.11.5.m1.1b"><csymbol cd="latexml" id="A8.T9.11.11.11.11.5.m1.1.1.cmml" xref="A8.T9.11.11.11.11.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.11.11.11.11.5.m1.1c">\pm</annotation></semantics></math> 0.49</td>
<td id="A8.T9.12.12.12.12.6" class="ltx_td ltx_align_center ltx_border_r"><span id="A8.T9.12.12.12.12.6.1" class="ltx_text ltx_font_bold">80.88 <math id="A8.T9.12.12.12.12.6.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.12.12.12.12.6.1.m1.1a"><mo id="A8.T9.12.12.12.12.6.1.m1.1.1" xref="A8.T9.12.12.12.12.6.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.12.12.12.12.6.1.m1.1b"><csymbol cd="latexml" id="A8.T9.12.12.12.12.6.1.m1.1.1.cmml" xref="A8.T9.12.12.12.12.6.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.12.12.12.12.6.1.m1.1c">\pm</annotation></semantics></math> 0.88</span></td>
<td id="A8.T9.12.12.12.12.8" class="ltx_td ltx_align_center">133</td>
</tr>
<tr id="A8.T9.18.18.18.18" class="ltx_tr">
<td id="A8.T9.18.18.18.18.7" class="ltx_td ltx_align_left ltx_border_r">Dessert</td>
<td id="A8.T9.13.13.13.13.1" class="ltx_td ltx_align_center">21.03 <math id="A8.T9.13.13.13.13.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.13.13.13.13.1.m1.1a"><mo id="A8.T9.13.13.13.13.1.m1.1.1" xref="A8.T9.13.13.13.13.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.13.13.13.13.1.m1.1b"><csymbol cd="latexml" id="A8.T9.13.13.13.13.1.m1.1.1.cmml" xref="A8.T9.13.13.13.13.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.13.13.13.13.1.m1.1c">\pm</annotation></semantics></math> 2.44</td>
<td id="A8.T9.14.14.14.14.2" class="ltx_td ltx_align_center">12.50 <math id="A8.T9.14.14.14.14.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.14.14.14.14.2.m1.1a"><mo id="A8.T9.14.14.14.14.2.m1.1.1" xref="A8.T9.14.14.14.14.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.14.14.14.14.2.m1.1b"><csymbol cd="latexml" id="A8.T9.14.14.14.14.2.m1.1.1.cmml" xref="A8.T9.14.14.14.14.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.14.14.14.14.2.m1.1c">\pm</annotation></semantics></math> 0.00</td>
<td id="A8.T9.15.15.15.15.3" class="ltx_td ltx_align_center ltx_border_r"><span id="A8.T9.15.15.15.15.3.1" class="ltx_text ltx_font_bold">99.40 <math id="A8.T9.15.15.15.15.3.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.15.15.15.15.3.1.m1.1a"><mo id="A8.T9.15.15.15.15.3.1.m1.1.1" xref="A8.T9.15.15.15.15.3.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.15.15.15.15.3.1.m1.1b"><csymbol cd="latexml" id="A8.T9.15.15.15.15.3.1.m1.1.1.cmml" xref="A8.T9.15.15.15.15.3.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.15.15.15.15.3.1.m1.1c">\pm</annotation></semantics></math>0.02</span></td>
<td id="A8.T9.16.16.16.16.4" class="ltx_td ltx_align_center">95.83<math id="A8.T9.16.16.16.16.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.16.16.16.16.4.m1.1a"><mo id="A8.T9.16.16.16.16.4.m1.1.1" xref="A8.T9.16.16.16.16.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.16.16.16.16.4.m1.1b"><csymbol cd="latexml" id="A8.T9.16.16.16.16.4.m1.1.1.cmml" xref="A8.T9.16.16.16.16.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.16.16.16.16.4.m1.1c">\pm</annotation></semantics></math>0.34</td>
<td id="A8.T9.17.17.17.17.5" class="ltx_td ltx_align_center">94.64<math id="A8.T9.17.17.17.17.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.17.17.17.17.5.m1.1a"><mo id="A8.T9.17.17.17.17.5.m1.1.1" xref="A8.T9.17.17.17.17.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.17.17.17.17.5.m1.1b"><csymbol cd="latexml" id="A8.T9.17.17.17.17.5.m1.1.1.cmml" xref="A8.T9.17.17.17.17.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.17.17.17.17.5.m1.1c">\pm</annotation></semantics></math> 0.00</td>
<td id="A8.T9.18.18.18.18.6" class="ltx_td ltx_align_center ltx_border_r"><span id="A8.T9.18.18.18.18.6.1" class="ltx_text ltx_font_bold">99.40 <math id="A8.T9.18.18.18.18.6.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.18.18.18.18.6.1.m1.1a"><mo id="A8.T9.18.18.18.18.6.1.m1.1.1" xref="A8.T9.18.18.18.18.6.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.18.18.18.18.6.1.m1.1b"><csymbol cd="latexml" id="A8.T9.18.18.18.18.6.1.m1.1.1.cmml" xref="A8.T9.18.18.18.18.6.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.18.18.18.18.6.1.m1.1c">\pm</annotation></semantics></math> 0.02</span></td>
<td id="A8.T9.18.18.18.18.8" class="ltx_td ltx_align_center">5</td>
</tr>
<tr id="A8.T9.24.24.24.24" class="ltx_tr">
<td id="A8.T9.24.24.24.24.7" class="ltx_td ltx_align_left ltx_border_r">Colorectal Histology</td>
<td id="A8.T9.19.19.19.19.1" class="ltx_td ltx_align_center">11.77 <math id="A8.T9.19.19.19.19.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.19.19.19.19.1.m1.1a"><mo id="A8.T9.19.19.19.19.1.m1.1.1" xref="A8.T9.19.19.19.19.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.19.19.19.19.1.m1.1b"><csymbol cd="latexml" id="A8.T9.19.19.19.19.1.m1.1.1.cmml" xref="A8.T9.19.19.19.19.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.19.19.19.19.1.m1.1c">\pm</annotation></semantics></math>2.88</td>
<td id="A8.T9.20.20.20.20.2" class="ltx_td ltx_align_center">11.00 <math id="A8.T9.20.20.20.20.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.20.20.20.20.2.m1.1a"><mo id="A8.T9.20.20.20.20.2.m1.1.1" xref="A8.T9.20.20.20.20.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.20.20.20.20.2.m1.1b"><csymbol cd="latexml" id="A8.T9.20.20.20.20.2.m1.1.1.cmml" xref="A8.T9.20.20.20.20.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.20.20.20.20.2.m1.1c">\pm</annotation></semantics></math> 0.00</td>
<td id="A8.T9.21.21.21.21.3" class="ltx_td ltx_align_center ltx_border_r"><span id="A8.T9.21.21.21.21.3.1" class="ltx_text ltx_font_bold">18.12 <math id="A8.T9.21.21.21.21.3.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.21.21.21.21.3.1.m1.1a"><mo id="A8.T9.21.21.21.21.3.1.m1.1.1" xref="A8.T9.21.21.21.21.3.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.21.21.21.21.3.1.m1.1b"><csymbol cd="latexml" id="A8.T9.21.21.21.21.3.1.m1.1.1.cmml" xref="A8.T9.21.21.21.21.3.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.21.21.21.21.3.1.m1.1c">\pm</annotation></semantics></math> 0.25</span></td>
<td id="A8.T9.22.22.22.22.4" class="ltx_td ltx_align_center">90.34 <math id="A8.T9.22.22.22.22.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.22.22.22.22.4.m1.1a"><mo id="A8.T9.22.22.22.22.4.m1.1.1" xref="A8.T9.22.22.22.22.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.22.22.22.22.4.m1.1b"><csymbol cd="latexml" id="A8.T9.22.22.22.22.4.m1.1.1.cmml" xref="A8.T9.22.22.22.22.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.22.22.22.22.4.m1.1c">\pm</annotation></semantics></math> 0.33</td>
<td id="A8.T9.23.23.23.23.5" class="ltx_td ltx_align_center">89.75<math id="A8.T9.23.23.23.23.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.23.23.23.23.5.m1.1a"><mo id="A8.T9.23.23.23.23.5.m1.1.1" xref="A8.T9.23.23.23.23.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.23.23.23.23.5.m1.1b"><csymbol cd="latexml" id="A8.T9.23.23.23.23.5.m1.1.1.cmml" xref="A8.T9.23.23.23.23.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.23.23.23.23.5.m1.1c">\pm</annotation></semantics></math> 0.19</td>
<td id="A8.T9.24.24.24.24.6" class="ltx_td ltx_align_center ltx_border_r"><span id="A8.T9.24.24.24.24.6.1" class="ltx_text ltx_font_bold">93.65 <math id="A8.T9.24.24.24.24.6.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.24.24.24.24.6.1.m1.1a"><mo id="A8.T9.24.24.24.24.6.1.m1.1.1" xref="A8.T9.24.24.24.24.6.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.24.24.24.24.6.1.m1.1b"><csymbol cd="latexml" id="A8.T9.24.24.24.24.6.1.m1.1.1.cmml" xref="A8.T9.24.24.24.24.6.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.24.24.24.24.6.1.m1.1c">\pm</annotation></semantics></math> 0.10</span></td>
<td id="A8.T9.24.24.24.24.8" class="ltx_td ltx_align_center">8</td>
</tr>
<tr id="A8.T9.30.30.30.30" class="ltx_tr">
<td id="A8.T9.30.30.30.30.7" class="ltx_td ltx_align_left ltx_border_r">Drawing</td>
<td id="A8.T9.25.25.25.25.1" class="ltx_td ltx_align_center">10.86 <math id="A8.T9.25.25.25.25.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.25.25.25.25.1.m1.1a"><mo id="A8.T9.25.25.25.25.1.m1.1.1" xref="A8.T9.25.25.25.25.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.25.25.25.25.1.m1.1b"><csymbol cd="latexml" id="A8.T9.25.25.25.25.1.m1.1.1.cmml" xref="A8.T9.25.25.25.25.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.25.25.25.25.1.m1.1c">\pm</annotation></semantics></math> 1.22</td>
<td id="A8.T9.26.26.26.26.2" class="ltx_td ltx_align_center">11.00 <math id="A8.T9.26.26.26.26.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.26.26.26.26.2.m1.1a"><mo id="A8.T9.26.26.26.26.2.m1.1.1" xref="A8.T9.26.26.26.26.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.26.26.26.26.2.m1.1b"><csymbol cd="latexml" id="A8.T9.26.26.26.26.2.m1.1.1.cmml" xref="A8.T9.26.26.26.26.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.26.26.26.26.2.m1.1c">\pm</annotation></semantics></math> 0.00</td>
<td id="A8.T9.27.27.27.27.3" class="ltx_td ltx_align_center ltx_border_r"><span id="A8.T9.27.27.27.27.3.1" class="ltx_text ltx_font_bold">11.87 <math id="A8.T9.27.27.27.27.3.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.27.27.27.27.3.1.m1.1a"><mo id="A8.T9.27.27.27.27.3.1.m1.1.1" xref="A8.T9.27.27.27.27.3.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.27.27.27.27.3.1.m1.1b"><csymbol cd="latexml" id="A8.T9.27.27.27.27.3.1.m1.1.1.cmml" xref="A8.T9.27.27.27.27.3.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.27.27.27.27.3.1.m1.1c">\pm</annotation></semantics></math>0.93</span></td>
<td id="A8.T9.28.28.28.28.4" class="ltx_td ltx_align_center"><span id="A8.T9.28.28.28.28.4.1" class="ltx_text ltx_font_bold">90.20 <math id="A8.T9.28.28.28.28.4.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.28.28.28.28.4.1.m1.1a"><mo id="A8.T9.28.28.28.28.4.1.m1.1.1" xref="A8.T9.28.28.28.28.4.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.28.28.28.28.4.1.m1.1b"><csymbol cd="latexml" id="A8.T9.28.28.28.28.4.1.m1.1.1.cmml" xref="A8.T9.28.28.28.28.4.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.28.28.28.28.4.1.m1.1c">\pm</annotation></semantics></math> 0.16</span></td>
<td id="A8.T9.29.29.29.29.5" class="ltx_td ltx_align_center">90.00<math id="A8.T9.29.29.29.29.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.29.29.29.29.5.m1.1a"><mo id="A8.T9.29.29.29.29.5.m1.1.1" xref="A8.T9.29.29.29.29.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.29.29.29.29.5.m1.1b"><csymbol cd="latexml" id="A8.T9.29.29.29.29.5.m1.1.1.cmml" xref="A8.T9.29.29.29.29.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.29.29.29.29.5.m1.1c">\pm</annotation></semantics></math> 0.16</td>
<td id="A8.T9.30.30.30.30.6" class="ltx_td ltx_align_center ltx_border_r">89.00 <math id="A8.T9.30.30.30.30.6.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.30.30.30.30.6.m1.1a"><mo id="A8.T9.30.30.30.30.6.m1.1.1" xref="A8.T9.30.30.30.30.6.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.30.30.30.30.6.m1.1b"><csymbol cd="latexml" id="A8.T9.30.30.30.30.6.m1.1.1.cmml" xref="A8.T9.30.30.30.30.6.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.30.30.30.30.6.m1.1c">\pm</annotation></semantics></math> 0.16</td>
<td id="A8.T9.30.30.30.30.8" class="ltx_td ltx_align_center">10</td>
</tr>
<tr id="A8.T9.36.36.36.36" class="ltx_tr">
<td id="A8.T9.36.36.36.36.7" class="ltx_td ltx_align_left ltx_border_r">Alien vs Predator</td>
<td id="A8.T9.31.31.31.31.1" class="ltx_td ltx_align_center">51.48 <math id="A8.T9.31.31.31.31.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.31.31.31.31.1.m1.1a"><mo id="A8.T9.31.31.31.31.1.m1.1.1" xref="A8.T9.31.31.31.31.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.31.31.31.31.1.m1.1b"><csymbol cd="latexml" id="A8.T9.31.31.31.31.1.m1.1.1.cmml" xref="A8.T9.31.31.31.31.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.31.31.31.31.1.m1.1c">\pm</annotation></semantics></math>2.09</td>
<td id="A8.T9.32.32.32.32.2" class="ltx_td ltx_align_center">28.88 <math id="A8.T9.32.32.32.32.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.32.32.32.32.2.m1.1a"><mo id="A8.T9.32.32.32.32.2.m1.1.1" xref="A8.T9.32.32.32.32.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.32.32.32.32.2.m1.1b"><csymbol cd="latexml" id="A8.T9.32.32.32.32.2.m1.1.1.cmml" xref="A8.T9.32.32.32.32.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.32.32.32.32.2.m1.1c">\pm</annotation></semantics></math> 0.00</td>
<td id="A8.T9.33.33.33.33.3" class="ltx_td ltx_align_center ltx_border_r"><span id="A8.T9.33.33.33.33.3.1" class="ltx_text ltx_font_bold">78.15 <math id="A8.T9.33.33.33.33.3.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.33.33.33.33.3.1.m1.1a"><mo id="A8.T9.33.33.33.33.3.1.m1.1.1" xref="A8.T9.33.33.33.33.3.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.33.33.33.33.3.1.m1.1b"><csymbol cd="latexml" id="A8.T9.33.33.33.33.3.1.m1.1.1.cmml" xref="A8.T9.33.33.33.33.3.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.33.33.33.33.3.1.m1.1c">\pm</annotation></semantics></math>0.52</span></td>
<td id="A8.T9.34.34.34.34.4" class="ltx_td ltx_align_center">98.52<math id="A8.T9.34.34.34.34.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.34.34.34.34.4.m1.1a"><mo id="A8.T9.34.34.34.34.4.m1.1.1" xref="A8.T9.34.34.34.34.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.34.34.34.34.4.m1.1b"><csymbol cd="latexml" id="A8.T9.34.34.34.34.4.m1.1.1.cmml" xref="A8.T9.34.34.34.34.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.34.34.34.34.4.m1.1c">\pm</annotation></semantics></math> 0.52</td>
<td id="A8.T9.35.35.35.35.5" class="ltx_td ltx_align_center"><span id="A8.T9.35.35.35.35.5.1" class="ltx_text ltx_font_bold">98.89<math id="A8.T9.35.35.35.35.5.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.35.35.35.35.5.1.m1.1a"><mo id="A8.T9.35.35.35.35.5.1.m1.1.1" xref="A8.T9.35.35.35.35.5.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.35.35.35.35.5.1.m1.1b"><csymbol cd="latexml" id="A8.T9.35.35.35.35.5.1.m1.1.1.cmml" xref="A8.T9.35.35.35.35.5.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.35.35.35.35.5.1.m1.1c">\pm</annotation></semantics></math> 1.42</span></td>
<td id="A8.T9.36.36.36.36.6" class="ltx_td ltx_align_center ltx_border_r">97.77 <math id="A8.T9.36.36.36.36.6.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.36.36.36.36.6.m1.1a"><mo id="A8.T9.36.36.36.36.6.m1.1.1" xref="A8.T9.36.36.36.36.6.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.36.36.36.36.6.m1.1b"><csymbol cd="latexml" id="A8.T9.36.36.36.36.6.m1.1.1.cmml" xref="A8.T9.36.36.36.36.6.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.36.36.36.36.6.m1.1c">\pm</annotation></semantics></math> 0.00</td>
<td id="A8.T9.36.36.36.36.8" class="ltx_td ltx_align_center">2</td>
</tr>
<tr id="A8.T9.42.42.42.42" class="ltx_tr">
<td id="A8.T9.42.42.42.42.7" class="ltx_td ltx_align_left ltx_border_r">COVID-19</td>
<td id="A8.T9.37.37.37.37.1" class="ltx_td ltx_align_center">20.13 <math id="A8.T9.37.37.37.37.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.37.37.37.37.1.m1.1a"><mo id="A8.T9.37.37.37.37.1.m1.1.1" xref="A8.T9.37.37.37.37.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.37.37.37.37.1.m1.1b"><csymbol cd="latexml" id="A8.T9.37.37.37.37.1.m1.1.1.cmml" xref="A8.T9.37.37.37.37.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.37.37.37.37.1.m1.1c">\pm</annotation></semantics></math>18.66</td>
<td id="A8.T9.38.38.38.38.2" class="ltx_td ltx_align_center">46.53 <math id="A8.T9.38.38.38.38.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.38.38.38.38.2.m1.1a"><mo id="A8.T9.38.38.38.38.2.m1.1.1" xref="A8.T9.38.38.38.38.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.38.38.38.38.2.m1.1b"><csymbol cd="latexml" id="A8.T9.38.38.38.38.2.m1.1.1.cmml" xref="A8.T9.38.38.38.38.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.38.38.38.38.2.m1.1c">\pm</annotation></semantics></math> 0.00</td>
<td id="A8.T9.39.39.39.39.3" class="ltx_td ltx_align_center ltx_border_r"><span id="A8.T9.39.39.39.39.3.1" class="ltx_text ltx_font_bold">47.22 <math id="A8.T9.39.39.39.39.3.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.39.39.39.39.3.1.m1.1a"><mo id="A8.T9.39.39.39.39.3.1.m1.1.1" xref="A8.T9.39.39.39.39.3.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.39.39.39.39.3.1.m1.1b"><csymbol cd="latexml" id="A8.T9.39.39.39.39.3.1.m1.1.1.cmml" xref="A8.T9.39.39.39.39.3.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.39.39.39.39.3.1.m1.1c">\pm</annotation></semantics></math>0.00</span></td>
<td id="A8.T9.40.40.40.40.4" class="ltx_td ltx_align_center">93.86<math id="A8.T9.40.40.40.40.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.40.40.40.40.4.m1.1a"><mo id="A8.T9.40.40.40.40.4.m1.1.1" xref="A8.T9.40.40.40.40.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.40.40.40.40.4.m1.1b"><csymbol cd="latexml" id="A8.T9.40.40.40.40.4.m1.1.1.cmml" xref="A8.T9.40.40.40.40.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.40.40.40.40.4.m1.1c">\pm</annotation></semantics></math>0.16</td>
<td id="A8.T9.41.41.41.41.5" class="ltx_td ltx_align_center">93.40<math id="A8.T9.41.41.41.41.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.41.41.41.41.5.m1.1a"><mo id="A8.T9.41.41.41.41.5.m1.1.1" xref="A8.T9.41.41.41.41.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.41.41.41.41.5.m1.1b"><csymbol cd="latexml" id="A8.T9.41.41.41.41.5.m1.1.1.cmml" xref="A8.T9.41.41.41.41.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.41.41.41.41.5.m1.1c">\pm</annotation></semantics></math> 0.49</td>
<td id="A8.T9.42.42.42.42.6" class="ltx_td ltx_align_center ltx_border_r"><span id="A8.T9.42.42.42.42.6.1" class="ltx_text ltx_font_bold">94.56 <math id="A8.T9.42.42.42.42.6.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.42.42.42.42.6.1.m1.1a"><mo id="A8.T9.42.42.42.42.6.1.m1.1.1" xref="A8.T9.42.42.42.42.6.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.42.42.42.42.6.1.m1.1b"><csymbol cd="latexml" id="A8.T9.42.42.42.42.6.1.m1.1.1.cmml" xref="A8.T9.42.42.42.42.6.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.42.42.42.42.6.1.m1.1c">\pm</annotation></semantics></math> 0.71</span></td>
<td id="A8.T9.42.42.42.42.8" class="ltx_td ltx_align_center">3</td>
</tr>
<tr id="A8.T9.48.48.48.48" class="ltx_tr">
<td id="A8.T9.48.48.48.48.7" class="ltx_td ltx_align_left ltx_border_r">honey-bee-pollen</td>
<td id="A8.T9.43.43.43.43.1" class="ltx_td ltx_align_center">49.54 <math id="A8.T9.43.43.43.43.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.43.43.43.43.1.m1.1a"><mo id="A8.T9.43.43.43.43.1.m1.1.1" xref="A8.T9.43.43.43.43.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.43.43.43.43.1.m1.1b"><csymbol cd="latexml" id="A8.T9.43.43.43.43.1.m1.1.1.cmml" xref="A8.T9.43.43.43.43.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.43.43.43.43.1.m1.1c">\pm</annotation></semantics></math>1.30</td>
<td id="A8.T9.44.44.44.44.2" class="ltx_td ltx_align_center">50.00 <math id="A8.T9.44.44.44.44.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.44.44.44.44.2.m1.1a"><mo id="A8.T9.44.44.44.44.2.m1.1.1" xref="A8.T9.44.44.44.44.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.44.44.44.44.2.m1.1b"><csymbol cd="latexml" id="A8.T9.44.44.44.44.2.m1.1.1.cmml" xref="A8.T9.44.44.44.44.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.44.44.44.44.2.m1.1c">\pm</annotation></semantics></math> 0.00</td>
<td id="A8.T9.45.45.45.45.3" class="ltx_td ltx_align_center ltx_border_r"><span id="A8.T9.45.45.45.45.3.1" class="ltx_text ltx_font_bold">56.94 <math id="A8.T9.45.45.45.45.3.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.45.45.45.45.3.1.m1.1a"><mo id="A8.T9.45.45.45.45.3.1.m1.1.1" xref="A8.T9.45.45.45.45.3.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.45.45.45.45.3.1.m1.1b"><csymbol cd="latexml" id="A8.T9.45.45.45.45.3.1.m1.1.1.cmml" xref="A8.T9.45.45.45.45.3.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.45.45.45.45.3.1.m1.1c">\pm</annotation></semantics></math>4.53</span></td>
<td id="A8.T9.46.46.46.46.4" class="ltx_td ltx_align_center">93.05 <math id="A8.T9.46.46.46.46.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.46.46.46.46.4.m1.1a"><mo id="A8.T9.46.46.46.46.4.m1.1.1" xref="A8.T9.46.46.46.46.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.46.46.46.46.4.m1.1b"><csymbol cd="latexml" id="A8.T9.46.46.46.46.4.m1.1.1.cmml" xref="A8.T9.46.46.46.46.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.46.46.46.46.4.m1.1c">\pm</annotation></semantics></math> 0.00</td>
<td id="A8.T9.47.47.47.47.5" class="ltx_td ltx_align_center">88.89<math id="A8.T9.47.47.47.47.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.47.47.47.47.5.m1.1a"><mo id="A8.T9.47.47.47.47.5.m1.1.1" xref="A8.T9.47.47.47.47.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.47.47.47.47.5.m1.1b"><csymbol cd="latexml" id="A8.T9.47.47.47.47.5.m1.1.1.cmml" xref="A8.T9.47.47.47.47.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.47.47.47.47.5.m1.1c">\pm</annotation></semantics></math> 0.00</td>
<td id="A8.T9.48.48.48.48.6" class="ltx_td ltx_align_center ltx_border_r"><span id="A8.T9.48.48.48.48.6.1" class="ltx_text ltx_font_bold">93.55 <math id="A8.T9.48.48.48.48.6.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.48.48.48.48.6.1.m1.1a"><mo id="A8.T9.48.48.48.48.6.1.m1.1.1" xref="A8.T9.48.48.48.48.6.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.48.48.48.48.6.1.m1.1b"><csymbol cd="latexml" id="A8.T9.48.48.48.48.6.1.m1.1.1.cmml" xref="A8.T9.48.48.48.48.6.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.48.48.48.48.6.1.m1.1c">\pm</annotation></semantics></math> 4.53</span></td>
<td id="A8.T9.48.48.48.48.8" class="ltx_td ltx_align_center">2</td>
</tr>
<tr id="A8.T9.54.54.54.54" class="ltx_tr">
<td id="A8.T9.54.54.54.54.7" class="ltx_td ltx_align_left ltx_border_r">Speed Limit Signs</td>
<td id="A8.T9.49.49.49.49.1" class="ltx_td ltx_align_center">30.55 <math id="A8.T9.49.49.49.49.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.49.49.49.49.1.m1.1a"><mo id="A8.T9.49.49.49.49.1.m1.1.1" xref="A8.T9.49.49.49.49.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.49.49.49.49.1.m1.1b"><csymbol cd="latexml" id="A8.T9.49.49.49.49.1.m1.1.1.cmml" xref="A8.T9.49.49.49.49.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.49.49.49.49.1.m1.1c">\pm</annotation></semantics></math>2.27</td>
<td id="A8.T9.50.50.50.50.2" class="ltx_td ltx_align_center">25.00 <math id="A8.T9.50.50.50.50.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.50.50.50.50.2.m1.1a"><mo id="A8.T9.50.50.50.50.2.m1.1.1" xref="A8.T9.50.50.50.50.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.50.50.50.50.2.m1.1b"><csymbol cd="latexml" id="A8.T9.50.50.50.50.2.m1.1.1.cmml" xref="A8.T9.50.50.50.50.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.50.50.50.50.2.m1.1c">\pm</annotation></semantics></math> 0.00</td>
<td id="A8.T9.51.51.51.51.3" class="ltx_td ltx_align_center ltx_border_r"><span id="A8.T9.51.51.51.51.3.1" class="ltx_text ltx_font_bold">31.48 <math id="A8.T9.51.51.51.51.3.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.51.51.51.51.3.1.m1.1a"><mo id="A8.T9.51.51.51.51.3.1.m1.1.1" xref="A8.T9.51.51.51.51.3.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.51.51.51.51.3.1.m1.1b"><csymbol cd="latexml" id="A8.T9.51.51.51.51.3.1.m1.1.1.cmml" xref="A8.T9.51.51.51.51.3.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.51.51.51.51.3.1.m1.1c">\pm</annotation></semantics></math>10.23</span></td>
<td id="A8.T9.52.52.52.52.4" class="ltx_td ltx_align_center">83.33<math id="A8.T9.52.52.52.52.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.52.52.52.52.4.m1.1a"><mo id="A8.T9.52.52.52.52.4.m1.1.1" xref="A8.T9.52.52.52.52.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.52.52.52.52.4.m1.1b"><csymbol cd="latexml" id="A8.T9.52.52.52.52.4.m1.1.1.cmml" xref="A8.T9.52.52.52.52.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.52.52.52.52.4.m1.1c">\pm</annotation></semantics></math> 0.00</td>
<td id="A8.T9.53.53.53.53.5" class="ltx_td ltx_align_center">86.11<math id="A8.T9.53.53.53.53.5.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.53.53.53.53.5.m1.1a"><mo id="A8.T9.53.53.53.53.5.m1.1.1" xref="A8.T9.53.53.53.53.5.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.53.53.53.53.5.m1.1b"><csymbol cd="latexml" id="A8.T9.53.53.53.53.5.m1.1.1.cmml" xref="A8.T9.53.53.53.53.5.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.53.53.53.53.5.m1.1c">\pm</annotation></semantics></math> 0.00</td>
<td id="A8.T9.54.54.54.54.6" class="ltx_td ltx_align_center ltx_border_r"><span id="A8.T9.54.54.54.54.6.1" class="ltx_text ltx_font_bold">90.74 <math id="A8.T9.54.54.54.54.6.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.54.54.54.54.6.1.m1.1a"><mo id="A8.T9.54.54.54.54.6.1.m1.1.1" xref="A8.T9.54.54.54.54.6.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.54.54.54.54.6.1.m1.1b"><csymbol cd="latexml" id="A8.T9.54.54.54.54.6.1.m1.1.1.cmml" xref="A8.T9.54.54.54.54.6.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.54.54.54.54.6.1.m1.1c">\pm</annotation></semantics></math> 1.31</span></td>
<td id="A8.T9.54.54.54.54.8" class="ltx_td ltx_align_center">4</td>
</tr>
<tr id="A8.T9.60.60.60.60" class="ltx_tr">
<td id="A8.T9.60.60.60.60.7" class="ltx_td ltx_align_left ltx_border_bb ltx_border_r">Japanese Characters</td>
<td id="A8.T9.55.55.55.55.1" class="ltx_td ltx_align_center ltx_border_bb">0.03<math id="A8.T9.55.55.55.55.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.55.55.55.55.1.m1.1a"><mo id="A8.T9.55.55.55.55.1.m1.1.1" xref="A8.T9.55.55.55.55.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.55.55.55.55.1.m1.1b"><csymbol cd="latexml" id="A8.T9.55.55.55.55.1.m1.1.1.cmml" xref="A8.T9.55.55.55.55.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.55.55.55.55.1.m1.1c">\pm</annotation></semantics></math>0.00</td>
<td id="A8.T9.56.56.56.56.2" class="ltx_td ltx_align_center ltx_border_bb">0.08 <math id="A8.T9.56.56.56.56.2.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.56.56.56.56.2.m1.1a"><mo id="A8.T9.56.56.56.56.2.m1.1.1" xref="A8.T9.56.56.56.56.2.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.56.56.56.56.2.m1.1b"><csymbol cd="latexml" id="A8.T9.56.56.56.56.2.m1.1.1.cmml" xref="A8.T9.56.56.56.56.2.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.56.56.56.56.2.m1.1c">\pm</annotation></semantics></math> 0.00</td>
<td id="A8.T9.57.57.57.57.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r"><span id="A8.T9.57.57.57.57.3.1" class="ltx_text ltx_font_bold">0.50<math id="A8.T9.57.57.57.57.3.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.57.57.57.57.3.1.m1.1a"><mo id="A8.T9.57.57.57.57.3.1.m1.1.1" xref="A8.T9.57.57.57.57.3.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.57.57.57.57.3.1.m1.1b"><csymbol cd="latexml" id="A8.T9.57.57.57.57.3.1.m1.1.1.cmml" xref="A8.T9.57.57.57.57.3.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.57.57.57.57.3.1.m1.1c">\pm</annotation></semantics></math>0.22</span></td>
<td id="A8.T9.58.58.58.58.4" class="ltx_td ltx_align_center ltx_border_bb">53.17 <math id="A8.T9.58.58.58.58.4.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.58.58.58.58.4.m1.1a"><mo id="A8.T9.58.58.58.58.4.m1.1.1" xref="A8.T9.58.58.58.58.4.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.58.58.58.58.4.m1.1b"><csymbol cd="latexml" id="A8.T9.58.58.58.58.4.m1.1.1.cmml" xref="A8.T9.58.58.58.58.4.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.58.58.58.58.4.m1.1c">\pm</annotation></semantics></math> 0.15</td>
<td id="A8.T9.59.59.59.59.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="A8.T9.59.59.59.59.5.1" class="ltx_text ltx_font_bold">62.33 <math id="A8.T9.59.59.59.59.5.1.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.59.59.59.59.5.1.m1.1a"><mo id="A8.T9.59.59.59.59.5.1.m1.1.1" xref="A8.T9.59.59.59.59.5.1.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.59.59.59.59.5.1.m1.1b"><csymbol cd="latexml" id="A8.T9.59.59.59.59.5.1.m1.1.1.cmml" xref="A8.T9.59.59.59.59.5.1.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.59.59.59.59.5.1.m1.1c">\pm</annotation></semantics></math> 0.16</span></td>
<td id="A8.T9.60.60.60.60.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">62.16 <math id="A8.T9.60.60.60.60.6.m1.1" class="ltx_Math" alttext="\pm" display="inline"><semantics id="A8.T9.60.60.60.60.6.m1.1a"><mo id="A8.T9.60.60.60.60.6.m1.1.1" xref="A8.T9.60.60.60.60.6.m1.1.1.cmml">±</mo><annotation-xml encoding="MathML-Content" id="A8.T9.60.60.60.60.6.m1.1b"><csymbol cd="latexml" id="A8.T9.60.60.60.60.6.m1.1.1.cmml" xref="A8.T9.60.60.60.60.6.m1.1.1">plus-or-minus</csymbol></annotation-xml><annotation encoding="application/x-tex" id="A8.T9.60.60.60.60.6.m1.1c">\pm</annotation></semantics></math>0.47 0.45</td>
<td id="A8.T9.60.60.60.60.8" class="ltx_td ltx_align_center ltx_border_bb">1566</td>
</tr>
</tbody>
</table>
</span></div>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2402.18152" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2402.18153" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2402.18153">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2402.18153" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2402.18154" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Tue Mar  5 16:12:45 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2402.18153
Transfer learning is a topic of significant interest in recent deep learning research because it enables faster convergence and improved performance on new tasks. While the performance of transfer learning depends on the similarity of the source data to the target data, it is costly to train a model on a large number of datasets. Therefore, pretrained models are generally blindly selected with the hope that they will achieve good performance on the given task. To tackle such suboptimality of the pretrained models, we propose an efficient and adaptive transfer learning scheme through dataset-conditioned pretrained weights sampling. Specifically, we use a latent diffusion model with a variational autoencoder that can reconstruct the neural network weights, to learn the distribution of a set of pretrained weights conditioned on each dataset for transfer learning on unseen datasets. By learning the distribution of a neural network on a variety pretrained models, our approach enables adaptive sampling weights for unseen datasets achieving faster convergence and reaching competitive performance.
abstract
Diffusion-based Neural Network Weights Generation
Abstract
2402.18153
1IntroductionThe goal of Automated Machine Learning (AutoML) is to automate the end-to-end process of optimizing machine learning models for real-world problems. It has become a popular means of making machine learning more accessible to individuals with limited expertise in AI. AutoML encompasses various stages of the machine learning pipeline, including 1)  Neural architecture selection: Choosing the most suitable neural architecture based on the problem at hand, 2)  Hyperparameter tuning: Optimizing the hyperparameters of the selected neural architecture to achieve better performance and 3)  Weight optimization: training the neural architecture or generating optimized weights on the provided data. Especially, transferable AutoML, which aims to effectively reuse pre-trained models by distilling knowledge from them and transferring it to an unseen dataset for fast adaptation, has become an important problem with significant interest in the AutoML community. For the former two stages, such as neural architecture selection and hyperparameter tuning, there exist extensive set of previous works that address these problems. However, only a very few existing works have explored automating the weight optimization par. In other words, we are still heavily dependent on naive training or fine-tuning in the model training section, which becomes a bottleneck in the entire training process even with automated neural architectures and hyperparameter optimization.(a) Dessert Dataset(b) Gemstones DatasetFigure 1:We train D2NWG on a modelzoo generated from the training split of the Kaggle Dataset ofand generate weights directly for the datasets in the test split. In Figure1(a) we sample the full set of weights using D2NWG (without any finetuning)  conditioned on the dataset encoding of the Dessert dataset. For the baselines, we finetune them on the full Dessert dataset. In Figure1(b) , we again perform dataset-conditioned sampling with D2NWG and fintune it, together with a model pretrained on ImageNet and a randomly initialized model (RandInit)  for 50 epochs. As can be seen, D2NWG can sample high performing weights for unseen datasets and achieves better convergence rate when the sampled weights are finetuned. The full set of results for Kaggle dataset is provided in Table3in the experimental section.Figure 2:Stage 1:This stage consists of an encoder-decoder model trained to learn to reconstruct pretrained weights in a modelzoo. Latent embeddings from this stage are manipulated, conditioned on a dataset, in the next stages.Stage 2: In the second stage, we learn a dataset encoder for compressing a given dataset into a vector representation. The resulting dataset encoder is frozen in the next stage.Stage 3:In the third stage, we learn to manipulate the latent space of Stage 1 conditioned on the dataset encodings from Stage 2 using a diffusion model. During inference, we condition on a dataset (seen or unseen)  to obtain a latent representation using the diffusion model and then decode this representation back to weights using the decoder in Stage 1. Each stage is trained independently and thebluearrow signifies joint training of Stages 1 and 2, an ablation provided in Table8of the Appendix.On weight optimization automation, there exist only a handful of few related works.proposed to learn the distribution of diverse pretrained weights enabling sampling for better initialization. Similar idea is investigated by, where they used prompt based fine-tuning. However, these methods do not explore the relationship of the sampled weights and target dataset. In this regard,proposed a meta-hypernetwork name HyperLDM class-conditioned weights generation for visual question answering(VQA)  where the text descriptor of each image(classes)  is used as condition.Most existing weights generation methods for image datasets that exploit knowledge of pretrained weights are limited to unconditional parameters generation and are designed for a single model and datasets or restricted to class-conditioning such as conditioning on target loss on the same dataset. In such cases they do not take into account the correlation between the pretrained weights and the corresponding pretrained dataset. Additionally, previous works have not investigated how to properly generate weights for unseen datasets, which is a problem of significant practical importance.
Therefore, we take the parameters generation problem one step further and extend it to dataset-conditioned sampling while taking advantage of existing diffusion models and pretrained weights knowledge. We propose a new method called Diffusion based Neural Network Weights Generation (D2NWG)  to learn the distribution of a dataset of pretrained weights conditioned on the their corresponding dataset embeddings enabling target datasets conditioned weight generation during inference for better transfer learning as shown in Figure1.
We demonstrate generalization to seen and unseen datasets by sampling weights for an architecture conditioned on the corresponding dataset of interest. Our contributions are as follows:•We efficiently adapt latent diffusionto network weights generation to improve the sampling performance and transfer learning.•We combine CLIPtogether with Set Transformerfor better dataset conditioning.•Our demonstrate that our method, D2NWG, allows dataset-conditioned diverse weights sampling and can efficiently sample weights conditioned on unseen datasets.•Through extensive experimental evaluations, we show that our method achieves promising results compared to the relevant baselines.
introduction
Diffusion-based Neural Network Weights Generation
1Introduction
2402.18153
2Related workWith the growing applications of neural networks, learning from pretrained weights is becoming increasingly important for speeding up the transfer learning process. Learning to generate network weights from existing modelzoos without going through the traditional training pipeline has potential to increase model efficiency. In this regard, several hypernetwork approaches have been proposed for weight prediction.Deterministic Parameters Generation:
Recentlyproposed Graph Hypernetwork (GHN)  for generating weights for an architecture using its directed graph representation as input. This idea was subsequently improved byin GHN2 focusing on generation across architectures for the same datasets. Analogously,consider weight generation as an autoregressive process and used a transformer architecture to sequentially generate weights per layer. However, this method is less scalable since it requires a transformer module per layer. Similarly,combined ideas from transformer based approach and GHN2 to improve the generalization of graph hyper networks across datasets and architectures in a new method called GHN3. Although, GHN3 achieve promising results, the sampling is deterministic, and the generalization depends solely on the generalization of the training set of architectures and pretrained dataset.Meta Pretrained Weight Generators:proposed HyperLDM; a generative model for weights generation for visual question answering tasks from distribution of weights pretrained in a meta-learning setting and sampling with latent diffusion. Although generating pretrained weights through meta training is promising, the meta learning process can be computationally expensive, moreover, the meta pretrained weights are not optimal for in-distribution weights generation.AutoEncoder-based Weight Generators:proposed to learn the distribution of weights by learning to reconstruct them using auto-encoder style architectures. A follow-up work,, proposed to learn the distribution of pretrained weights enabling unconditional sampling of diverse weights using kernel density estimation. A similar idea is developed bywhere the weight generation is conditioned by the target loss in a diffusion transformer framework.Diffusion Models: Denoising Diffusion Probabilistic Models (DDPM) enable mapping data representations to a Gaussian distribution and vice versa achieving state-of-the art performance in generative modeling. Latent diffusion modelsallow for manipulation of the learned latent space. We utilize such models for our dataset-conditioned weight sampling.Our work combines components from each of the methods discussed sofar allowing for dataset-conditioned neural network weight generation for both seen and unseen datasets.
body
Diffusion-based Neural Network Weights Generation
2Related work
2402.18153
3Approach3.1PreliminariesProblem Definition:In this work, we are interested in the following problem: Given a set of neural network models of the same architecture
pretrained on a set ofdatasets, we want to learn the distributionof the pretrained weights ()  of those models such that for a new datasetwe can conditionally sample a set of weights withthat will achieve good performance on the new dataset without training or with few optimization steps.High-Level Intuition:Our approach uses a diffusion model in order to generate a given neural architecture’s weights for previously unseen datasets.
In order to train our model, we use a modelzoo of trained weights from various datasets. The diffusion model then learns from that data to predict new weights conditioned on a given dataset.Intuitively, based on the training data, the diffusion model knows what trained networks look like and can relate this information to the characteristics of the dataset the network was trained on.
At test time, for a previously unseen dataset we compute a dataset encoding and use this to guide the backward diffusion process to generate weights that mimic trained weights for this particular dataset.This process exploits the facts that at test time we do have the dataset as input and that there is a clear relationship between a network’s trained weights and the dataset it was trained on (For a more formal argument of this relationship, please see AppendixC.)  Understanding the distribution of the pretrained weights and their relation with the pretrained datasets will enable generating highly performing weights conditioned on the target dataset with or without a few optimization steps. Through the next sections we investigate the problem of learning this conditional distribution of pretrained weights.Overview of Model:We start by describing the overall structure of our approach consisting of three main parts as shown in Figure2. After obtaining the set of pretrained models (ie. modelzoo) , we extract the pretrained weights and vectorize them into a single vector per architecture checkpoint. These vectors are then paired with their corresponding pretrained datasets. We then train a VAE model to encode the vectors to learn latent representations of weights. This stage is denoted by Stage 1 in Figure2. In the intermediate stage we encode the dataset associated with each weight vector using a contrastive learning process denoted by the Stage 2 in Figure2. The final stage of the proposed method, denoted by Stage 3, consists of a UNet based diffusion model. Each stage is trained separately. During inference, we encode the target dataset and use its embeddings to conditionally generate new weights for the target datasets. In the subsequent sections, we will discuss these stages in turn in detail.3.2Weight EncodingGiven a modelzoo of pretrained weights, we flatten the model parameters to obtain vectorized weights for each network. We then train a Variational
AutoEncoder (VAE)  without any conditioning. The VAE is trained such that the decoded weights, when evaluated on the test set, achieve similar
performance compared to the input weights. Concretely, we minimize the following1objective:(1) whereis the vectorized weight,it’s latent representation,andthe reconstruction and approximate posterior terms
respectively, andthe prior distribution. For the prior, we use the standard Gaussian with zero mean and unit variance. The trained VAE will later be used for training a latent diffusion model for dataset-conditioned weight sampling.3.3Dataset EncodingTo be able to sample weights conditioned on a dataset, we require an efficient mapping between the latent representations of the pretrained weights and their corresponding datasets (ie. the datasets the networks were trained on) . However, efficiently encoding an entire dataset with multiple classes and hundreds, even millions, of samples per class is a challenging task. One way to encode such datasets is to utilize dataset condensation techniques which consists of training a network to synthesize one example per classes, combining the class-wise condensations and using it as a representation for the dataset. However this process requires performing the same process for every test dataset increasing the computation cost of the model. Hence we explore existing dataset-adaptive Neural Architecture Search approacheswhere Set Transformerhas been demonstrated as an efficient model for dataset encoding.Set-based Dataset Encoding:Given a dataset, whereandare input-output pairs, we form subsetswith, and. Ifis an image dataset,where,andcorrespond to the image channel, height and width respectively. Therefore, the entire dataset can be viewed as a collection of subsetswithbeing the number of distinct classes. To encode such dataset, we define a transformationover the subsets that takes each setand outputs an embedding. We note the abuse of notation here wherehas been used for the weight embedding in Section3.2. All embeddings are aggregated to form a new embedding setto which we apply another transformationtoto the encoded dataset to. The encoding mechanism described above can be summarized as a composition of Set Transformer blocksas follows. To recapitulate, we refer to this operation of encoding the dataset as. Training the Set Transformer modules together with the different modules in the pipeline can be computationally demanding. To resolve this issue, we next outline a CLIP style objective for training the Set Transformer Modules.Contrastive Dataset Encoding:To improve the effectiveness of the dataset encoding mechanism on large collections of datasets, we train the Set Transformer using a CLIP-style contrastive loss.
An analogous idea is exploited in HyperCLIP. However in our case, we replace the image encoder with the VAE of Section3.2, and replace the image-text contrastive pairs with the weight embedding and its corresponding dataset embedding. Concretely, we utilize the following contrastive pair,consisting of the class-wise dataset encodings and the VAE embedding of a corresponding network weight. Using the pretrained VAE encoder of Section3.2, we train the Set Transformer dataset encoding moduleto align its embeddingwherewith the network embeddingsof the pretrained VAE. The Set Transformer training objective is given as:(2) 3.4Dataset-Conditioned Weight GenerationAt this stage, we have access to a pretrained VAE for encoding neural network weights and a pretrained Set Transformer module for encoding entire datasets. The next stage involves defining a model to generate latent representations of weights conditioned on dataset representations. We achieve this by using Diffusion Denoising Probabilistic Models (DDPM) .Forward Process:Given a weight embedding, obtained from the encoder of the pretrained VAE, the forward diffusion process involves successive Gaussion noise perturbations ofovertime steps. At time step,(3) whereis the noise variance and.Reverse Process:As in most DDPM approaches the reverse process is approximated by a neural network such that:(4) whereandare neural networks.Dataset-Conditioned Training:The diffusion model is trained on the VAE embeddings, conditioned on the dataset encodings through concatenation. To take advantage of existing architectures, we design the VAE model to produce latent representations compatible with existing latent diffusion models with minimal modifications and optimize the following latent diffusion model objective:(5) whereis implemented as a UNet.Sampling:New weights are sampled conditionally through the revers diffusion process as follows:(6) whereand,a chosen value. After sampling a latent representationfor a given dataset. The pretrained VAE decoder is used to transform that latents into a weightwhich is then used to initialize the target network as depicted in Figure2.
body
Diffusion-based Neural Network Weights Generation
3Approach
2402.18153
4Experiments(a) MNIST(b) SVHN(c) CIFAR-10(d) STL-10Figure 3:Convergence Plots on Fintuning Generated Weigths:Weights generated by the competing methods are finetuned for 25 epochs on the training set. We utilize the modelzoos of.We generate and evaluate weights in the following setting:•Weight Generation without Finetuning:For this task, we condition on a dataset, generate weights and directly evaluate it on the test set.•Weight Generation with Finetuning:The generated weights are finetuned on the training set for a fixed number of epochs before evaluating on the test set.Baselines:We benchmark against the following:•RandInit:This is a randomly initialized model trained on the training set of the target dataset from scratch for a specified number of epochs.•Pretrained:These are models pretrained either on the target dataset or ImageNet. We follow the setting ofand train this baseline for 25 epochs on the target datasets in Tables3&2. All other pretranied baselines are trained on ImageNet.•SKDE30: This method learns an autoencoder on a modelzoo of networks trained on a single dataset and uses Kernel Density Estimation (KDE)  to sample new weights from the embedding of networks in the modelzoo.Detailed experimental setup is outlined in the Appendix.Modelzoo Generation:We utilize the pretrained weights datasets ofwhich consists of four modelzoos of networks pretranied on MNIST, SVHN, CIFAR-10 and STL-10. Next we generate a modelzoo for the set of datasets introduced inand refer to it as the KaggleZoo. The test datasets are used for weight generation on unseen tasks. Finally, we generate a modelzoo by creating 2000 subsets of ImageNet with 100 and 50 images per class in the training and test splits respectively. Additionally, we do not train on the CIFAR-10 and STL-10 modelzoos ofand use them for unseen dataset weight generation. Details for generating the modelzoos is provided in AppendixB.Algorithm 1Datasets Encoder TrainingInput:pretrained weights, image features, batch_numInstanciate, Load pretrained Encoder () .repeatInitializefortodo,(Equation2) endforUpdate weights ofuntilconvergenceSet-based Dataset Encoding: We use Set Transformerfor dataset encoding using features extracted from CLIPsimilar to the approach inwhere features were extracted using a pretrained ResNet18. This process is done before training with the dataset encoder defined in Section3.3. We observed that jointly optimizing the dataset encoder with the diffusion model is more challenging given a large number of datasets hence we pretrain the dataset encoder and freeze it during the training of the diffusion model.Compute:For the zoo generation we use a single Titan RTX GPU with 24GB memory. The proposed model was also trained on a single GPU with 24GB memory.4.1In-distribution Transfer LearningTask:The main objective of this experiment is to evaluate convergence speed using weights from each of the methods as initialization. In this experiment the target dataset is the same as those used to generate the modelzoos.Setup: The pretrained weights used here are from epochs 21 to 25 for each dataset where 70% of the resulting modelzoo is used for training and 15% for validation and testing respectively. The number of pretrained weights in the modelzoos are 3500 for MNIST, CIFAR-10, and STL-10, and 2864 for SVHN. The flattened network weights’ length is 2864 for CIFAR-10 and STL-10 and, 2464 for MNIST and SVHN. We pad all the weights with zero to 2864. After training the diffusion model, we conditionally sample five set of weights based on images samples from the corresponding datasets and initialize each corresponding architectures and then finetune them for 25 epochs. We compare our method to the best approach inand the pretrained weights already trained for 25 epochs (we evolve this model for 25 more epochs resulting in 50 epochs of optimization on the training set) .Table 1:Convergence Speed ComparisonSampled weights from SKDE30and our method are evaluated on the test set after 0 (meaning no optimization steps) , 1 and 25 epochs of training on the training set. The Pretrained baseline is trained on the training set for a total of 50 epochs.ModelEpochMNISTSVHNCIFAR-10STL-10Pretrained5095.7074.4949.2539.39SKDE30069.7350.2526.0617.20D2NWG (Ours) 078.1058.9238.2322.74SKDE30182.6970.4544.7528.42D2NWG (Ours) 183.2573.1146.8834.39SKDE302595.7074.6949.1439.14D2NWG (Ours) 2596.1774.7549.4839.43Results:As reported in Table1, D2NWG consistently speeds up the convergence rate on the related tasks and after 25 epochs, achieves better performance than the pretrained model, consistently outperforming the Kernel Density Estimation approach of. In Figure3, we show the convergence plots for the 4 datasets considered in this task.Table 2:No Fine-tuning Initialization on Unseen DatasetsWe transfer from one dataset, or combinations of datasets, to unseen datasets at test time.SourceTargetAccuracyMethodsMNISTSVHN13.25SKDE30SVHNMNIST29.30CIFAR-10STL-1015.20STL-10CIFAR-1015.40Sampling from Combined Weights DistributionMNIST+CIFAR-10SVHN18.80OursMNist+CIFAR-10STL-1016.21SVHN + STL-10MNIST36.64SVHN + STL-10CIFAR-1018.004.2Sampling Weights for Unseen DatasetsTask:We evaluate the transferability of the models on unseen datasets. We create disjoint modelzoos by combining MNIST and CIFAR-10 into a single modelzoo and combining the SVHN and STL-10 modelzoos. When we train on the MNIST plus CIFAR-10 modelzoos, we test on the SVHN and STL-10 modelzoos and vice-versa.Results:As shown in Table2, the D2NWG is able to sample weights with higher accuracy on unseen datasets as well as for in distribution. Through these experiments our method does not only outperform the baseline it also demonstrates promising results for dataset-conditioned sampling for unseen datasets.4.3Performance Evaluation on KaggleZooWe extend the performance evaluation on unseen dataset to much diverse datasets.
Since the baselinerelies on unconditional sampling this method is not appropriate for unseen dataset evaluation. Therefore the baseline of this experiments are model pretrained on ImageNet and random initialization.Task: In this experiment, we aim to demonstrate that D2NWG can generate efficient weights for initialization and fine-tuning compared to random initialization and ImageNet pretrained models on unseen datasets with different number of classes than those encountered during training.Table 3:Dataset Conditioned SamplingWeight sampling on 10 unseen real-world datasets. We report two sets of results: (a)  weight evaluation without finetuning and (b)  weight evaluation after 50 epochs of finetuning. The pretrained baseline is trained on ImageNet1K. We report the accuracy on the testset.DatasetsInitializationFine-tuningRandInitPretrainedD2NWGRandInitPretrainedD2NWGGemstones1.130.621.8670.5967.4976.06Dog Breeds0.550.691.8780.7878.1380.88Dessert21.0312.5099.4095.8394.6499.40Colorectal Histology11.7711.0018.1290.3489.7593.65Drawing10.8611.0011.8790.2090.0089.00Alien vs Predator51.4828.8878.1598.5298.8997.77COVID-1920.1346.5347.2293.8693.4094.56honey-bee-pollen49.5450.0056.9493.0588.8993.55Speed Limit Signs30.5525.0031.4883.3386.1190.74Japanese Characters0.030.080.5053.1762.3362.16(a) Dog Breeds(b) Alien vs Predator(c) Japanese Characters(d) Colorectal HistologyFigure 4:Average accuracy evolution of fine-tuning for 50 epochs with sampled weights for unseen datasets from real world datasets The full results are reported in the appendixResults: As show in Table3and in the convergence plots in Figure4, weight initialization with D2NWG without any fine-tuning achieves 99.04% on dessert dataset outperforming the random initialized model trained for 50 epochs. Additionally, our method consistently speeds-up the convergence on all tasks compared to the random initialized and pretrained models.
Except on Japanese Characters, Gemstones and Dog Breeds datasets where both methods achieved similar results, the proposed method consistently outperformed the two baselines and converges faster on the finetuning task. Although the test subsets and the training subsets of image dataset have no overlapping classes our method is able to transfer well to the test set.We investigate the reason why the proposed method achieves superior performance on some datasets. Our findings suggest that D2NWG achieves similar none finetuning performance as the random initialization in the cases where the target dataset has more classes than the maximum number classes in the pretrained datasets. The sapmled weight performance on the dessert dataset is due to the fact that this dataset contains similar images as those in the foods dataset which are a subsets of the food classification task in the pretrained image datasets. These results showed that the dataset encoder efficiently encodes the datasets and if some target datasets overlap with those in the modelzoo, the proposed method is able to retrieve the weights that transfer well. Detailed results can be found in AppendixH.Table 4:Cross-Dataset TrasferMNISTSVHN, SVHNMNIST, CIFAR-10STL-10, STL-10CIFAR-10. The OFA model is pretrained on ImageNet.ModelMNISTSVHNCIFAR10STL10Pretrained (seen) 97.4484.0481.8366.62D2NWG (seen) 97.4983.1282.1966.13OFA (Pretrained) 13.348.9013.348.90cross_dataset (unseen) 50.7028.6834.1639.95D2NWG (unseen) 49.3229.1335.0239.90D2NWG+average (unseen) 50.2429.1335.0340.00Evaluation on Larger Architectures:So far we have restricted ourselves to modelzoos populated by realitively simple CNN architectures. In this section we evaluate our method on MobileNetV3, a subnetwork sample from OFApretrained on CIFAR-10, STL-10, SVHN and MNIST for 25 epochs. The last 10 checkpoints per dataset are collected and we learn the pretrained weights distribution using our method. The performance of the sampled weighs are presented in Table4and shows that the proposed method transfers well to unseen dataset as well as on seen datasets. In the last row of Table4, we provide ModelSoup-typeexperiments where we average multiple weights sampled from our model.4.4Ablation studyIn this section, we investigate the importance and limitation of the different modules used to build our method.4.4.1Effect of Modelzoo Size GeneralizationIn this part we investigate whether increasing the number of datasets will increase the performance on unseen datasets. To achieve that goal, we perform experiments with modelzoos of size 5000, 10000 and 20000 created from subsets of ImageNet. We use CIFAR-10 and STL-10 as unseen target datasets.
We sample 50 weights and report the average of the top 5 performing weights in Figure5(a) where it can be observed that increasing the modelzoo size improve generalization.Results:On CIFAR-10 and STL-10, we obtain accuracies of 39.601.31% and 44.660.55% for 5000 subsets, 42.152.12 and 64.832.83 for 10000 subsets, and 52.643.12% and, 80.491.77% for 20000 subsets.
The maximum accuracy with random initialization are 12.11%, and 17.12% on CIFAR-10 and STL-10 with no fine-tuning.(a) (b) Figure 5:(a)  Effect of the number of pretrained datasets on sampling weights performance on unseen datasets. (b)  Performance comparison on in-distribution sampling of methods with VAE+DDPM vs DDPM4.4.2Sampling without Latent RepresentationWe investigate a variant of our model that learns the diffusion model directly on the weights without the AutoEncoder stage. We compare these two setting, with and without the AutoEncoder. Both variants are trained on 1000 subsets of ImageNet. We report the results of in-distribution sampling on three datasets in Figure5(b) .Results:As reported in Figure5(b) , for the same diffusion architecture, it is much easier to learn the pretrained weights distribution in latent space. One of the reason for which the DDPM process failed on the raw pretrained may be due to the fact that raw pretrained weights may require higher model capacity. Note also that learning the diffusion model directly on the weight space is computationally expensive. Hence, perhaps learning diffusion on subsets of the weights might be a more promising direction both in terms of computation and performance and we leave this as future work.4.4.3CLIP-based Dataset EncodingWe further investigate the different between CLIP-based dataset encoding scheme trained in an intermediate stage and the Set Transformer encoder jointly trained with the diffusion process. We conduct the experiments on the 140 Kaggle datasets and their corresponding modelzoos.Results: We compare the in-distribution sampling, out-distribution initialization, and fine-tuning results. As shown in Figure6, for in-distribution sampling both methods achieve similar results for small number of datasets. When the number of datasets increases the set-transformer jointly trained with diffusion approach struggle to converge.Figure 6:Performance comparison at initialization of method with jointly trained set-transformer (Without CLIP)  and method clip-based dataset encoder.
body
Diffusion-based Neural Network Weights Generation
4Experiments
2402.18153
5ConclusionIn this work we presented a three stage approach to neural network weight generation that involves fitting an AutoEncoder to pretrained weights, learning a dataset encoder using CLIP embeddings and learning a diffusion model to manipulate the latent space conditioned on the dataset embeddings. We demonstrated that the proposed method can sample high quality weights for both seen and unseen datasets and achieves better performance when finetuned for a fixed number of epochs when benchmarked against the relevant baselines.
conclusion
Diffusion-based Neural Network Weights Generation
5Conclusion