<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2407.20243] Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions</title><meta property="og:description" content="Embeddings from Large Language Models (LLMs) have emerged as critical components in various applications, particularly for information retrieval.
While high-dimensional embeddings generally demonstrate superior perform…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2407.20243">

<!--Generated on Mon Aug  5 16:43:15 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document">
<h1 class="ltx_title ltx_title_document">Matryoshka-Adaptor: Unsupervised and Supervised Tuning 
<br class="ltx_break">for Smaller Embedding Dimensions</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Jinsung Yoon, Raj Sinha, Sercan Ö. Arık, Tomas Pfister 
<br class="ltx_break">Google Cloud AI 
<br class="ltx_break"><span id="id1.1.id1" class="ltx_text ltx_font_typewriter">{jinsungyoon, sinharaj, soarik, tpfister}@google.com</span>
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id2.id1" class="ltx_p">Embeddings from Large Language Models (LLMs) have emerged as critical components in various applications, particularly for information retrieval.
While high-dimensional embeddings generally demonstrate superior performance as they contain more salient information, their practical application is frequently hindered by elevated computational latency and the associated higher cost.
To address these challenges, we propose Matryoshka-Adaptor, a novel tuning framework designed for the customization of LLM embeddings.
Matryoshka-Adaptor facilitates substantial dimensionality reduction while maintaining comparable performance levels, thereby achieving a significant enhancement in computational efficiency and cost-effectiveness.
Our framework directly modifies the embeddings from pre-trained LLMs which is designed to be seamlessly integrated with any LLM architecture, encompassing those accessible exclusively through black-box APIs.
Also, it exhibits efficacy in both unsupervised and supervised learning settings.
A rigorous evaluation conducted across a diverse corpus of English, multilingual, and multimodal datasets consistently reveals substantial gains with Matryoshka-Adaptor.
Notably, with Google and OpenAI Embedding APIs, Matryoshka-Adaptor achieves a reduction in dimensionality ranging from two- to twelve-fold without compromising performance across multiple BEIR datasets.</p>
</div>
<div id="p1" class="ltx_para ltx_noindent">
<div id="p1.1" class="ltx_block ltx_align_bottom">
<p id="p1.1.1" class="ltx_p"><span id="p1.1.1.1" class="ltx_text ltx_font_bold">Matryoshka-Adaptor: Unsupervised and Supervised Tuning 
<br class="ltx_break">for Smaller Embedding Dimensions</span></p>
<br class="ltx_break ltx_centering">
<p id="p1.1.2" class="ltx_p ltx_align_center" style="width:433.6pt;"><span id="p1.1.2.1" class="ltx_text ltx_inline-block" style="width:0.0pt;">
<span id="p1.1.2.1.1" class="ltx_tabular ltx_align_top">
<span class="ltx_tbody">
<span id="p1.1.2.1.1.1.1" class="ltx_tr">
<span id="p1.1.2.1.1.1.1.1" class="ltx_td ltx_align_center"><span id="p1.1.2.1.1.1.1.1.1" class="ltx_text ltx_font_bold">Jinsung Yoon, Raj Sinha, Sercan Ö. Arık, Tomas Pfister</span></span></span>
<span id="p1.1.2.1.1.2.2" class="ltx_tr">
<span id="p1.1.2.1.1.2.2.1" class="ltx_td ltx_align_center">Google Cloud AI</span></span>
<span id="p1.1.2.1.1.3.3" class="ltx_tr">
<span id="p1.1.2.1.1.3.3.1" class="ltx_td ltx_align_center"><span id="p1.1.2.1.1.3.3.1.1" class="ltx_text ltx_font_typewriter">{jinsungyoon, sinharaj, soarik, tpfister}@google.com</span></span></span>
</span>
</span></span></p>
<br class="ltx_break ltx_centering">
</div>
</div>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<figure id="S1.F1" class="ltx_figure"><img src="/html/2407.20243/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="388" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>The effectiveness of the Matryoshka Adaptor in dimensionality reduction.
In both unsupervised (red line) and supervised (black line) settings, the Matryoshka Adaptor showcases the capability to considerably decrease embedding dimensions while maintaining a negligible impact on nDCG@10 retrieval performance with BEIR SciFact dataset.
Notably, at the same embedding dimensionality, the utilization of our approach results in significantly improved performance.</figcaption>
</figure>
<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Large language models (LLMs) have showcased remarkable proficiency in handling various text processing tasks, encompassing question answering, summarization, and mathematical reasoning <cite class="ltx_cite ltx_citemacro_citep">(Brown et al., <a href="#bib.bib6" title="" class="ltx_ref">2020</a>; Chowdhery et al., <a href="#bib.bib8" title="" class="ltx_ref">2022</a>; Zhang et al., <a href="#bib.bib26" title="" class="ltx_ref">2022a</a>)</cite>.
This success can be partially attributed to their ability to transform raw text into semantically enriched representations, with the quality of these text-to-embedding mappings being of paramount importance <cite class="ltx_cite ltx_citemacro_citep">(Ouyang et al., <a href="#bib.bib21" title="" class="ltx_ref">2022</a>)</cite>.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">Embeddings find extensive utilization in a wide array of downstream tasks, with information retrieval (IR) serving as a prominent one <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib24" title="" class="ltx_ref">2023</a>; Muennighoff et al., <a href="#bib.bib19" title="" class="ltx_ref">2024</a>; Li et al., <a href="#bib.bib17" title="" class="ltx_ref">2023</a>)</cite>.
IR involves the process of searching for relevant information within a corpus database using queries, and language modeling plays a pivotal role as both queries and corpus data are frequently textual in nature.
In IR systems, text embeddings are commonly employed to rank relevant corpora based on their similarity to queries.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">Numerous LLMs have been developed specifically for extracting embeddings from raw text, including Sentence T5 <cite class="ltx_cite ltx_citemacro_citep">(Ni et al., <a href="#bib.bib20" title="" class="ltx_ref">2021</a>)</cite>, OpenAI Embedding APIs <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib4" title="" class="ltx_ref">ope, </a>)</cite>, and Google Embedding APIs <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib2" title="" class="ltx_ref">gcp, </a>)</cite>.
However, generation of high-dimensional embeddings often entail high latency and computational costs, thereby limiting their practical application in latency-sensitive scenarios such as large-scale recommendation systems.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">Matryoshka representation learning (MRL) <cite class="ltx_cite ltx_citemacro_citep">(Kusupati et al., <a href="#bib.bib14" title="" class="ltx_ref">2022</a>; Cai et al., <a href="#bib.bib7" title="" class="ltx_ref">2024</a>)</cite> addresses this limitation at pretraining stage, by generating embeddings that retain similar characteristics even when utilizing only a subset of their dimensions (i.e., Matryoshka properties).
This enables efficient similarity comparisons with lower-dimensional embeddings and is leveraged by many state-of-the-art models, including those from Google and OpenAI <cite class="ltx_cite ltx_citemacro_cite">Lee et al. (<a href="#bib.bib16" title="" class="ltx_ref">2024</a>); <a href="#bib.bib4" title="" class="ltx_ref">ope </a></cite>.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">In this study, we present Matryoshka-Adaptor, a novel framework designed to transform arbitrary embeddings into embeddings with Matryoshka properties in both unsupervised and supervised learning setups.
In an unsupervised learning setting, the adaptor learns to transform input embeddings into Matryoshka embeddings using only corpus embeddings.
We introduced pairwise and top-k similarity loss functions to facilitate this process.
In supervised learning setup, the adaptor can be further refined by incorporating relevant (query, corpus) pairs as labeled tuning data.
Note that we customize embeddings to better suit specific datasets, using both unsupervised and supervised approaches.
This refinement results in improved Matryoshka properties, surpassing even those of embeddings derived from MRL-trained models.</p>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p">Extensive experiments conducted across 13 BEIR datasets <cite class="ltx_cite ltx_citemacro_citep">(Thakur et al., <a href="#bib.bib23" title="" class="ltx_ref">2021</a>)</cite>, 17 MIRACL datasets <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib27" title="" class="ltx_ref">2022b</a>)</cite>, and 5 multimodal Fashion-200K datasets <cite class="ltx_cite ltx_citemacro_citep">(Han et al., <a href="#bib.bib10" title="" class="ltx_ref">2017</a>)</cite> validate the effectiveness of Matryoshka-Adaptor.
We demonstrate significant performance improvements over the latest Google (for English only, multilingual, and multimodal data) <cite class="ltx_cite ltx_citemacro_citep">(Lee et al., <a href="#bib.bib16" title="" class="ltx_ref">2024</a>)</cite> and OpenAI text embedding models <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib4" title="" class="ltx_ref">ope, </a>)</cite>.
Matryoshka-Adaptor exhibits broad applicability to diverse embedding types, encompassing text, multimodal, multilingual, and use case-specific embeddings. The contributions of this paper are threefold:</p>
<ul id="S1.I1" class="ltx_itemize">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p">We propose a novel tuning framework, Matryoshka-Adaptor that achieves substantial dimensionality reduction in embeddings without sacrificing performance achieved by tailoring embeddings to better fit specific datasets.</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p">Matryoshka-Adaptor is applicable in both unsupervised and supervised settings, consistently improving retrieval performance across various datasets. Also, Matryoshka-Adaptor demonstrates a roughly two-fold (unsupervised) and six-fold (supervised) reduction in dimensionality, with no loss in performance.</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p">The benefits of Matryoshka-Adaptor extend to multimodal learning and multilingual scenarios.</p>
</div>
</li>
</ul>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<div id="S2.p1" class="ltx_para">
<p id="S2.p1.1" class="ltx_p"><span id="S2.p1.1.1" class="ltx_text ltx_font_bold">Matryoshka embeddings. </span>
<cite class="ltx_cite ltx_citemacro_citep">(Kusupati et al., <a href="#bib.bib14" title="" class="ltx_ref">2022</a>)</cite> pioneered the development of embedding models whose representations could be substantially reduced in dimensionality without incurring a significant degradation in performance.
These Matryoshka embedding models are specifically trained to ensure the utility of such truncated embeddings.
The Matryoshka properties inherent in these models enable fine-grained control over the trade-off between latency and accuracy in downstream tasks utilizing the embeddings.
Given this advantage, recent embedding models, including those from OpenAI and Google <cite class="ltx_cite ltx_citemacro_cite">Lee et al. (<a href="#bib.bib16" title="" class="ltx_ref">2024</a>)</cite>, have integrated Matryoshka properties by employing Matryoshka Representation Learning (MRL) during pre-training.
Our proposed work further enhances the Matryoshka properties through tuning.
We introduce modifications to embeddings that are tailored to target datasets in both an unsupervised and supervised tuning setups.
The proposed tuning yields enhanced Matryoshka properties compared to the original embeddings, even those derived from MRL-trained models.</p>
</div>
<div id="S2.p2" class="ltx_para">
<p id="S2.p2.1" class="ltx_p"><span id="S2.p2.1.1" class="ltx_text ltx_font_bold">Dimensionality reduction. </span>
Dimensionality reduction is a well-established framework for reducing the dimensionality of vectors while preserving their inherent properties.
Principal Component Analysis (PCA) <cite class="ltx_cite ltx_citemacro_citep">(Jolliffe and Cadima, <a href="#bib.bib13" title="" class="ltx_ref">2016</a>)</cite>, Independent Component Analysis (ICA) <cite class="ltx_cite ltx_citemacro_citep">(Hyvärinen and Oja, <a href="#bib.bib11" title="" class="ltx_ref">2000</a>)</cite>, and Non-negative Matrix Factorization (NMF) <cite class="ltx_cite ltx_citemacro_citep">(Lee and Seung, <a href="#bib.bib15" title="" class="ltx_ref">2000</a>)</cite> are widely used unsupervised dimensionality reduction techniques.
Linear Discriminant Analysis (LDA) <cite class="ltx_cite ltx_citemacro_citep">(McLachlan, <a href="#bib.bib18" title="" class="ltx_ref">2005</a>)</cite> is a supervised method that leverages labeled data for dimensionality reduction.
While ICA is possible, its limitations in this context arise due to the equal importance of all components, requiring separate model training for each reduced dimension.
NMF, on the other hand, is not suitable for this application as it is only applicable to non-negative embeddings.
Also, PCA’s orthogonality properties may limit its reliability for dimensionality reduction for vectors with highly non-linear relationships, especially at high dimensionality.</p>
</div>
<div id="S2.p3" class="ltx_para">
<p id="S2.p3.1" class="ltx_p"><span id="S2.p3.1.1" class="ltx_text ltx_font_bold">Embedding customization. </span>
In lieu of employing a single unified model for zero-shot retrieval, embedding customization tailored to individual datasets presents a viable alternative.
TART <cite class="ltx_cite ltx_citemacro_citep">(Asai et al., <a href="#bib.bib5" title="" class="ltx_ref">2022</a>)</cite> constructs a retrieval system that adapts retrieval based on instructions, incorporating different tasks (e.g. for code, question, or answer) to enhance dense embedding retrieval.
InstructOR <cite class="ltx_cite ltx_citemacro_cite">Su et al. (<a href="#bib.bib22" title="" class="ltx_ref">2022</a>)</cite> integrates task and domain descriptions for retrieval while tuning embedding models.
Promptagator <cite class="ltx_cite ltx_citemacro_cite">Dai et al. (<a href="#bib.bib9" title="" class="ltx_ref">2022</a>)</cite> leverages in-context learning to generate synthetic query-corpus pairs using a limited number of original pairs, subsequently fine-tuning pre-trained LLMs with these synthetic pairs.
Search-Adaptor <cite class="ltx_cite ltx_citemacro_citep">(Yoon et al., <a href="#bib.bib25" title="" class="ltx_ref">2023</a>)</cite> customizes embeddings for information retrieval using a small number of query-corpus pairs in supervised learning setup.
Unlike these approaches, Matryoshka-Adaptor is applicable to both supervised and unsupervised settings, eliminating the need for labeled query-corpus pairs.
Additionally, Matryoshka-Adaptor aims not only to enhance full-dimensional embedding performance but also to improve performance across all reduced-dimensionality embeddings.</p>
</div>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Unsupervised Matryoshka-Adaptor</h2>

<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Problem formulation</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.6" class="ltx_p">To facilitate a clear understanding, the proposed framework is formulated within the context of information retrieval.
However, it is crucial to emphasize that it can be readily generalized to any application involving embeddings.
In unsupervised settings, we assume the availability of a corpus set, denoted as <math id="S3.SS1.p1.1.m1.4" class="ltx_Math" alttext="\mathcal{C}=\{c_{1},c_{2},...,c_{N}\}" display="inline"><semantics id="S3.SS1.p1.1.m1.4a"><mrow id="S3.SS1.p1.1.m1.4.4" xref="S3.SS1.p1.1.m1.4.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.1.m1.4.4.5" xref="S3.SS1.p1.1.m1.4.4.5.cmml">𝒞</mi><mo id="S3.SS1.p1.1.m1.4.4.4" xref="S3.SS1.p1.1.m1.4.4.4.cmml">=</mo><mrow id="S3.SS1.p1.1.m1.4.4.3.3" xref="S3.SS1.p1.1.m1.4.4.3.4.cmml"><mo stretchy="false" id="S3.SS1.p1.1.m1.4.4.3.3.4" xref="S3.SS1.p1.1.m1.4.4.3.4.cmml">{</mo><msub id="S3.SS1.p1.1.m1.2.2.1.1.1" xref="S3.SS1.p1.1.m1.2.2.1.1.1.cmml"><mi id="S3.SS1.p1.1.m1.2.2.1.1.1.2" xref="S3.SS1.p1.1.m1.2.2.1.1.1.2.cmml">c</mi><mn id="S3.SS1.p1.1.m1.2.2.1.1.1.3" xref="S3.SS1.p1.1.m1.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S3.SS1.p1.1.m1.4.4.3.3.5" xref="S3.SS1.p1.1.m1.4.4.3.4.cmml">,</mo><msub id="S3.SS1.p1.1.m1.3.3.2.2.2" xref="S3.SS1.p1.1.m1.3.3.2.2.2.cmml"><mi id="S3.SS1.p1.1.m1.3.3.2.2.2.2" xref="S3.SS1.p1.1.m1.3.3.2.2.2.2.cmml">c</mi><mn id="S3.SS1.p1.1.m1.3.3.2.2.2.3" xref="S3.SS1.p1.1.m1.3.3.2.2.2.3.cmml">2</mn></msub><mo id="S3.SS1.p1.1.m1.4.4.3.3.6" xref="S3.SS1.p1.1.m1.4.4.3.4.cmml">,</mo><mi mathvariant="normal" id="S3.SS1.p1.1.m1.1.1" xref="S3.SS1.p1.1.m1.1.1.cmml">…</mi><mo id="S3.SS1.p1.1.m1.4.4.3.3.7" xref="S3.SS1.p1.1.m1.4.4.3.4.cmml">,</mo><msub id="S3.SS1.p1.1.m1.4.4.3.3.3" xref="S3.SS1.p1.1.m1.4.4.3.3.3.cmml"><mi id="S3.SS1.p1.1.m1.4.4.3.3.3.2" xref="S3.SS1.p1.1.m1.4.4.3.3.3.2.cmml">c</mi><mi id="S3.SS1.p1.1.m1.4.4.3.3.3.3" xref="S3.SS1.p1.1.m1.4.4.3.3.3.3.cmml">N</mi></msub><mo stretchy="false" id="S3.SS1.p1.1.m1.4.4.3.3.8" xref="S3.SS1.p1.1.m1.4.4.3.4.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.1.m1.4b"><apply id="S3.SS1.p1.1.m1.4.4.cmml" xref="S3.SS1.p1.1.m1.4.4"><eq id="S3.SS1.p1.1.m1.4.4.4.cmml" xref="S3.SS1.p1.1.m1.4.4.4"></eq><ci id="S3.SS1.p1.1.m1.4.4.5.cmml" xref="S3.SS1.p1.1.m1.4.4.5">𝒞</ci><set id="S3.SS1.p1.1.m1.4.4.3.4.cmml" xref="S3.SS1.p1.1.m1.4.4.3.3"><apply id="S3.SS1.p1.1.m1.2.2.1.1.1.cmml" xref="S3.SS1.p1.1.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.1.m1.2.2.1.1.1.1.cmml" xref="S3.SS1.p1.1.m1.2.2.1.1.1">subscript</csymbol><ci id="S3.SS1.p1.1.m1.2.2.1.1.1.2.cmml" xref="S3.SS1.p1.1.m1.2.2.1.1.1.2">𝑐</ci><cn type="integer" id="S3.SS1.p1.1.m1.2.2.1.1.1.3.cmml" xref="S3.SS1.p1.1.m1.2.2.1.1.1.3">1</cn></apply><apply id="S3.SS1.p1.1.m1.3.3.2.2.2.cmml" xref="S3.SS1.p1.1.m1.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.SS1.p1.1.m1.3.3.2.2.2.1.cmml" xref="S3.SS1.p1.1.m1.3.3.2.2.2">subscript</csymbol><ci id="S3.SS1.p1.1.m1.3.3.2.2.2.2.cmml" xref="S3.SS1.p1.1.m1.3.3.2.2.2.2">𝑐</ci><cn type="integer" id="S3.SS1.p1.1.m1.3.3.2.2.2.3.cmml" xref="S3.SS1.p1.1.m1.3.3.2.2.2.3">2</cn></apply><ci id="S3.SS1.p1.1.m1.1.1.cmml" xref="S3.SS1.p1.1.m1.1.1">…</ci><apply id="S3.SS1.p1.1.m1.4.4.3.3.3.cmml" xref="S3.SS1.p1.1.m1.4.4.3.3.3"><csymbol cd="ambiguous" id="S3.SS1.p1.1.m1.4.4.3.3.3.1.cmml" xref="S3.SS1.p1.1.m1.4.4.3.3.3">subscript</csymbol><ci id="S3.SS1.p1.1.m1.4.4.3.3.3.2.cmml" xref="S3.SS1.p1.1.m1.4.4.3.3.3.2">𝑐</ci><ci id="S3.SS1.p1.1.m1.4.4.3.3.3.3.cmml" xref="S3.SS1.p1.1.m1.4.4.3.3.3.3">𝑁</ci></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.1.m1.4c">\mathcal{C}=\{c_{1},c_{2},...,c_{N}\}</annotation></semantics></math> and a pre-trained embedding model (<math id="S3.SS1.p1.2.m2.1" class="ltx_Math" alttext="E" display="inline"><semantics id="S3.SS1.p1.2.m2.1a"><mi id="S3.SS1.p1.2.m2.1.1" xref="S3.SS1.p1.2.m2.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.2.m2.1b"><ci id="S3.SS1.p1.2.m2.1.1.cmml" xref="S3.SS1.p1.2.m2.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.2.m2.1c">E</annotation></semantics></math>).
The embeddings extracted from the corpus are represented as <math id="S3.SS1.p1.3.m3.4" class="ltx_Math" alttext="\mathcal{C}_{E}=\{ce_{1},ce_{2},...,ce_{N}\}" display="inline"><semantics id="S3.SS1.p1.3.m3.4a"><mrow id="S3.SS1.p1.3.m3.4.4" xref="S3.SS1.p1.3.m3.4.4.cmml"><msub id="S3.SS1.p1.3.m3.4.4.5" xref="S3.SS1.p1.3.m3.4.4.5.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.3.m3.4.4.5.2" xref="S3.SS1.p1.3.m3.4.4.5.2.cmml">𝒞</mi><mi id="S3.SS1.p1.3.m3.4.4.5.3" xref="S3.SS1.p1.3.m3.4.4.5.3.cmml">E</mi></msub><mo id="S3.SS1.p1.3.m3.4.4.4" xref="S3.SS1.p1.3.m3.4.4.4.cmml">=</mo><mrow id="S3.SS1.p1.3.m3.4.4.3.3" xref="S3.SS1.p1.3.m3.4.4.3.4.cmml"><mo stretchy="false" id="S3.SS1.p1.3.m3.4.4.3.3.4" xref="S3.SS1.p1.3.m3.4.4.3.4.cmml">{</mo><mrow id="S3.SS1.p1.3.m3.2.2.1.1.1" xref="S3.SS1.p1.3.m3.2.2.1.1.1.cmml"><mi id="S3.SS1.p1.3.m3.2.2.1.1.1.2" xref="S3.SS1.p1.3.m3.2.2.1.1.1.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.3.m3.2.2.1.1.1.1" xref="S3.SS1.p1.3.m3.2.2.1.1.1.1.cmml">​</mo><msub id="S3.SS1.p1.3.m3.2.2.1.1.1.3" xref="S3.SS1.p1.3.m3.2.2.1.1.1.3.cmml"><mi id="S3.SS1.p1.3.m3.2.2.1.1.1.3.2" xref="S3.SS1.p1.3.m3.2.2.1.1.1.3.2.cmml">e</mi><mn id="S3.SS1.p1.3.m3.2.2.1.1.1.3.3" xref="S3.SS1.p1.3.m3.2.2.1.1.1.3.3.cmml">1</mn></msub></mrow><mo id="S3.SS1.p1.3.m3.4.4.3.3.5" xref="S3.SS1.p1.3.m3.4.4.3.4.cmml">,</mo><mrow id="S3.SS1.p1.3.m3.3.3.2.2.2" xref="S3.SS1.p1.3.m3.3.3.2.2.2.cmml"><mi id="S3.SS1.p1.3.m3.3.3.2.2.2.2" xref="S3.SS1.p1.3.m3.3.3.2.2.2.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.3.m3.3.3.2.2.2.1" xref="S3.SS1.p1.3.m3.3.3.2.2.2.1.cmml">​</mo><msub id="S3.SS1.p1.3.m3.3.3.2.2.2.3" xref="S3.SS1.p1.3.m3.3.3.2.2.2.3.cmml"><mi id="S3.SS1.p1.3.m3.3.3.2.2.2.3.2" xref="S3.SS1.p1.3.m3.3.3.2.2.2.3.2.cmml">e</mi><mn id="S3.SS1.p1.3.m3.3.3.2.2.2.3.3" xref="S3.SS1.p1.3.m3.3.3.2.2.2.3.3.cmml">2</mn></msub></mrow><mo id="S3.SS1.p1.3.m3.4.4.3.3.6" xref="S3.SS1.p1.3.m3.4.4.3.4.cmml">,</mo><mi mathvariant="normal" id="S3.SS1.p1.3.m3.1.1" xref="S3.SS1.p1.3.m3.1.1.cmml">…</mi><mo id="S3.SS1.p1.3.m3.4.4.3.3.7" xref="S3.SS1.p1.3.m3.4.4.3.4.cmml">,</mo><mrow id="S3.SS1.p1.3.m3.4.4.3.3.3" xref="S3.SS1.p1.3.m3.4.4.3.3.3.cmml"><mi id="S3.SS1.p1.3.m3.4.4.3.3.3.2" xref="S3.SS1.p1.3.m3.4.4.3.3.3.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.3.m3.4.4.3.3.3.1" xref="S3.SS1.p1.3.m3.4.4.3.3.3.1.cmml">​</mo><msub id="S3.SS1.p1.3.m3.4.4.3.3.3.3" xref="S3.SS1.p1.3.m3.4.4.3.3.3.3.cmml"><mi id="S3.SS1.p1.3.m3.4.4.3.3.3.3.2" xref="S3.SS1.p1.3.m3.4.4.3.3.3.3.2.cmml">e</mi><mi id="S3.SS1.p1.3.m3.4.4.3.3.3.3.3" xref="S3.SS1.p1.3.m3.4.4.3.3.3.3.3.cmml">N</mi></msub></mrow><mo stretchy="false" id="S3.SS1.p1.3.m3.4.4.3.3.8" xref="S3.SS1.p1.3.m3.4.4.3.4.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.3.m3.4b"><apply id="S3.SS1.p1.3.m3.4.4.cmml" xref="S3.SS1.p1.3.m3.4.4"><eq id="S3.SS1.p1.3.m3.4.4.4.cmml" xref="S3.SS1.p1.3.m3.4.4.4"></eq><apply id="S3.SS1.p1.3.m3.4.4.5.cmml" xref="S3.SS1.p1.3.m3.4.4.5"><csymbol cd="ambiguous" id="S3.SS1.p1.3.m3.4.4.5.1.cmml" xref="S3.SS1.p1.3.m3.4.4.5">subscript</csymbol><ci id="S3.SS1.p1.3.m3.4.4.5.2.cmml" xref="S3.SS1.p1.3.m3.4.4.5.2">𝒞</ci><ci id="S3.SS1.p1.3.m3.4.4.5.3.cmml" xref="S3.SS1.p1.3.m3.4.4.5.3">𝐸</ci></apply><set id="S3.SS1.p1.3.m3.4.4.3.4.cmml" xref="S3.SS1.p1.3.m3.4.4.3.3"><apply id="S3.SS1.p1.3.m3.2.2.1.1.1.cmml" xref="S3.SS1.p1.3.m3.2.2.1.1.1"><times id="S3.SS1.p1.3.m3.2.2.1.1.1.1.cmml" xref="S3.SS1.p1.3.m3.2.2.1.1.1.1"></times><ci id="S3.SS1.p1.3.m3.2.2.1.1.1.2.cmml" xref="S3.SS1.p1.3.m3.2.2.1.1.1.2">𝑐</ci><apply id="S3.SS1.p1.3.m3.2.2.1.1.1.3.cmml" xref="S3.SS1.p1.3.m3.2.2.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p1.3.m3.2.2.1.1.1.3.1.cmml" xref="S3.SS1.p1.3.m3.2.2.1.1.1.3">subscript</csymbol><ci id="S3.SS1.p1.3.m3.2.2.1.1.1.3.2.cmml" xref="S3.SS1.p1.3.m3.2.2.1.1.1.3.2">𝑒</ci><cn type="integer" id="S3.SS1.p1.3.m3.2.2.1.1.1.3.3.cmml" xref="S3.SS1.p1.3.m3.2.2.1.1.1.3.3">1</cn></apply></apply><apply id="S3.SS1.p1.3.m3.3.3.2.2.2.cmml" xref="S3.SS1.p1.3.m3.3.3.2.2.2"><times id="S3.SS1.p1.3.m3.3.3.2.2.2.1.cmml" xref="S3.SS1.p1.3.m3.3.3.2.2.2.1"></times><ci id="S3.SS1.p1.3.m3.3.3.2.2.2.2.cmml" xref="S3.SS1.p1.3.m3.3.3.2.2.2.2">𝑐</ci><apply id="S3.SS1.p1.3.m3.3.3.2.2.2.3.cmml" xref="S3.SS1.p1.3.m3.3.3.2.2.2.3"><csymbol cd="ambiguous" id="S3.SS1.p1.3.m3.3.3.2.2.2.3.1.cmml" xref="S3.SS1.p1.3.m3.3.3.2.2.2.3">subscript</csymbol><ci id="S3.SS1.p1.3.m3.3.3.2.2.2.3.2.cmml" xref="S3.SS1.p1.3.m3.3.3.2.2.2.3.2">𝑒</ci><cn type="integer" id="S3.SS1.p1.3.m3.3.3.2.2.2.3.3.cmml" xref="S3.SS1.p1.3.m3.3.3.2.2.2.3.3">2</cn></apply></apply><ci id="S3.SS1.p1.3.m3.1.1.cmml" xref="S3.SS1.p1.3.m3.1.1">…</ci><apply id="S3.SS1.p1.3.m3.4.4.3.3.3.cmml" xref="S3.SS1.p1.3.m3.4.4.3.3.3"><times id="S3.SS1.p1.3.m3.4.4.3.3.3.1.cmml" xref="S3.SS1.p1.3.m3.4.4.3.3.3.1"></times><ci id="S3.SS1.p1.3.m3.4.4.3.3.3.2.cmml" xref="S3.SS1.p1.3.m3.4.4.3.3.3.2">𝑐</ci><apply id="S3.SS1.p1.3.m3.4.4.3.3.3.3.cmml" xref="S3.SS1.p1.3.m3.4.4.3.3.3.3"><csymbol cd="ambiguous" id="S3.SS1.p1.3.m3.4.4.3.3.3.3.1.cmml" xref="S3.SS1.p1.3.m3.4.4.3.3.3.3">subscript</csymbol><ci id="S3.SS1.p1.3.m3.4.4.3.3.3.3.2.cmml" xref="S3.SS1.p1.3.m3.4.4.3.3.3.3.2">𝑒</ci><ci id="S3.SS1.p1.3.m3.4.4.3.3.3.3.3.cmml" xref="S3.SS1.p1.3.m3.4.4.3.3.3.3.3">𝑁</ci></apply></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.3.m3.4c">\mathcal{C}_{E}=\{ce_{1},ce_{2},...,ce_{N}\}</annotation></semantics></math>, where each embedding vector <math id="S3.SS1.p1.4.m4.1" class="ltx_Math" alttext="ce_{i}=E(c_{i})\in\mathbb{R}^{d}" display="inline"><semantics id="S3.SS1.p1.4.m4.1a"><mrow id="S3.SS1.p1.4.m4.1.1" xref="S3.SS1.p1.4.m4.1.1.cmml"><mrow id="S3.SS1.p1.4.m4.1.1.3" xref="S3.SS1.p1.4.m4.1.1.3.cmml"><mi id="S3.SS1.p1.4.m4.1.1.3.2" xref="S3.SS1.p1.4.m4.1.1.3.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.4.m4.1.1.3.1" xref="S3.SS1.p1.4.m4.1.1.3.1.cmml">​</mo><msub id="S3.SS1.p1.4.m4.1.1.3.3" xref="S3.SS1.p1.4.m4.1.1.3.3.cmml"><mi id="S3.SS1.p1.4.m4.1.1.3.3.2" xref="S3.SS1.p1.4.m4.1.1.3.3.2.cmml">e</mi><mi id="S3.SS1.p1.4.m4.1.1.3.3.3" xref="S3.SS1.p1.4.m4.1.1.3.3.3.cmml">i</mi></msub></mrow><mo id="S3.SS1.p1.4.m4.1.1.4" xref="S3.SS1.p1.4.m4.1.1.4.cmml">=</mo><mrow id="S3.SS1.p1.4.m4.1.1.1" xref="S3.SS1.p1.4.m4.1.1.1.cmml"><mi id="S3.SS1.p1.4.m4.1.1.1.3" xref="S3.SS1.p1.4.m4.1.1.1.3.cmml">E</mi><mo lspace="0em" rspace="0em" id="S3.SS1.p1.4.m4.1.1.1.2" xref="S3.SS1.p1.4.m4.1.1.1.2.cmml">​</mo><mrow id="S3.SS1.p1.4.m4.1.1.1.1.1" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p1.4.m4.1.1.1.1.1.2" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.cmml">(</mo><msub id="S3.SS1.p1.4.m4.1.1.1.1.1.1" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.cmml"><mi id="S3.SS1.p1.4.m4.1.1.1.1.1.1.2" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.2.cmml">c</mi><mi id="S3.SS1.p1.4.m4.1.1.1.1.1.1.3" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S3.SS1.p1.4.m4.1.1.1.1.1.3" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS1.p1.4.m4.1.1.5" xref="S3.SS1.p1.4.m4.1.1.5.cmml">∈</mo><msup id="S3.SS1.p1.4.m4.1.1.6" xref="S3.SS1.p1.4.m4.1.1.6.cmml"><mi id="S3.SS1.p1.4.m4.1.1.6.2" xref="S3.SS1.p1.4.m4.1.1.6.2.cmml">ℝ</mi><mi id="S3.SS1.p1.4.m4.1.1.6.3" xref="S3.SS1.p1.4.m4.1.1.6.3.cmml">d</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.4.m4.1b"><apply id="S3.SS1.p1.4.m4.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1"><and id="S3.SS1.p1.4.m4.1.1a.cmml" xref="S3.SS1.p1.4.m4.1.1"></and><apply id="S3.SS1.p1.4.m4.1.1b.cmml" xref="S3.SS1.p1.4.m4.1.1"><eq id="S3.SS1.p1.4.m4.1.1.4.cmml" xref="S3.SS1.p1.4.m4.1.1.4"></eq><apply id="S3.SS1.p1.4.m4.1.1.3.cmml" xref="S3.SS1.p1.4.m4.1.1.3"><times id="S3.SS1.p1.4.m4.1.1.3.1.cmml" xref="S3.SS1.p1.4.m4.1.1.3.1"></times><ci id="S3.SS1.p1.4.m4.1.1.3.2.cmml" xref="S3.SS1.p1.4.m4.1.1.3.2">𝑐</ci><apply id="S3.SS1.p1.4.m4.1.1.3.3.cmml" xref="S3.SS1.p1.4.m4.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS1.p1.4.m4.1.1.3.3.1.cmml" xref="S3.SS1.p1.4.m4.1.1.3.3">subscript</csymbol><ci id="S3.SS1.p1.4.m4.1.1.3.3.2.cmml" xref="S3.SS1.p1.4.m4.1.1.3.3.2">𝑒</ci><ci id="S3.SS1.p1.4.m4.1.1.3.3.3.cmml" xref="S3.SS1.p1.4.m4.1.1.3.3.3">𝑖</ci></apply></apply><apply id="S3.SS1.p1.4.m4.1.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1"><times id="S3.SS1.p1.4.m4.1.1.1.2.cmml" xref="S3.SS1.p1.4.m4.1.1.1.2"></times><ci id="S3.SS1.p1.4.m4.1.1.1.3.cmml" xref="S3.SS1.p1.4.m4.1.1.1.3">𝐸</ci><apply id="S3.SS1.p1.4.m4.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.4.m4.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1">subscript</csymbol><ci id="S3.SS1.p1.4.m4.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.2">𝑐</ci><ci id="S3.SS1.p1.4.m4.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p1.4.m4.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply><apply id="S3.SS1.p1.4.m4.1.1c.cmml" xref="S3.SS1.p1.4.m4.1.1"><in id="S3.SS1.p1.4.m4.1.1.5.cmml" xref="S3.SS1.p1.4.m4.1.1.5"></in><share href="#S3.SS1.p1.4.m4.1.1.1.cmml" id="S3.SS1.p1.4.m4.1.1d.cmml" xref="S3.SS1.p1.4.m4.1.1"></share><apply id="S3.SS1.p1.4.m4.1.1.6.cmml" xref="S3.SS1.p1.4.m4.1.1.6"><csymbol cd="ambiguous" id="S3.SS1.p1.4.m4.1.1.6.1.cmml" xref="S3.SS1.p1.4.m4.1.1.6">superscript</csymbol><ci id="S3.SS1.p1.4.m4.1.1.6.2.cmml" xref="S3.SS1.p1.4.m4.1.1.6.2">ℝ</ci><ci id="S3.SS1.p1.4.m4.1.1.6.3.cmml" xref="S3.SS1.p1.4.m4.1.1.6.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.4.m4.1c">ce_{i}=E(c_{i})\in\mathbb{R}^{d}</annotation></semantics></math>.
Notably, our framework permits the embedding model, <math id="S3.SS1.p1.5.m5.1" class="ltx_Math" alttext="E" display="inline"><semantics id="S3.SS1.p1.5.m5.1a"><mi id="S3.SS1.p1.5.m5.1.1" xref="S3.SS1.p1.5.m5.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.5.m5.1b"><ci id="S3.SS1.p1.5.m5.1.1.cmml" xref="S3.SS1.p1.5.m5.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.5.m5.1c">E</annotation></semantics></math>, to be treated as a black-box model, and Matryoshka-Adaptor can be directly applied to the extracted embeddings, <math id="S3.SS1.p1.6.m6.1" class="ltx_Math" alttext="\mathcal{C}_{E}" display="inline"><semantics id="S3.SS1.p1.6.m6.1a"><msub id="S3.SS1.p1.6.m6.1.1" xref="S3.SS1.p1.6.m6.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p1.6.m6.1.1.2" xref="S3.SS1.p1.6.m6.1.1.2.cmml">𝒞</mi><mi id="S3.SS1.p1.6.m6.1.1.3" xref="S3.SS1.p1.6.m6.1.1.3.cmml">E</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p1.6.m6.1b"><apply id="S3.SS1.p1.6.m6.1.1.cmml" xref="S3.SS1.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS1.p1.6.m6.1.1.1.cmml" xref="S3.SS1.p1.6.m6.1.1">subscript</csymbol><ci id="S3.SS1.p1.6.m6.1.1.2.cmml" xref="S3.SS1.p1.6.m6.1.1.2">𝒞</ci><ci id="S3.SS1.p1.6.m6.1.1.3.cmml" xref="S3.SS1.p1.6.m6.1.1.3">𝐸</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p1.6.m6.1c">\mathcal{C}_{E}</annotation></semantics></math>.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para">
<p id="S3.SS1.p2.6" class="ltx_p">A Matryoshka embedding, characterized by <math id="S3.SS1.p2.1.m1.1" class="ltx_Math" alttext="m" display="inline"><semantics id="S3.SS1.p2.1.m1.1a"><mi id="S3.SS1.p2.1.m1.1.1" xref="S3.SS1.p2.1.m1.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.1.m1.1b"><ci id="S3.SS1.p2.1.m1.1.1.cmml" xref="S3.SS1.p2.1.m1.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.1.m1.1c">m</annotation></semantics></math> dimensions, is defined as the initial <math id="S3.SS1.p2.2.m2.1" class="ltx_Math" alttext="m" display="inline"><semantics id="S3.SS1.p2.2.m2.1a"><mi id="S3.SS1.p2.2.m2.1.1" xref="S3.SS1.p2.2.m2.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.2.m2.1b"><ci id="S3.SS1.p2.2.m2.1.1.cmml" xref="S3.SS1.p2.2.m2.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.2.m2.1c">m</annotation></semantics></math> dimensions of the original <math id="S3.SS1.p2.3.m3.1" class="ltx_Math" alttext="d" display="inline"><semantics id="S3.SS1.p2.3.m3.1a"><mi id="S3.SS1.p2.3.m3.1.1" xref="S3.SS1.p2.3.m3.1.1.cmml">d</mi><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.3.m3.1b"><ci id="S3.SS1.p2.3.m3.1.1.cmml" xref="S3.SS1.p2.3.m3.1.1">𝑑</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.3.m3.1c">d</annotation></semantics></math>-dimensional embedding, where <math id="S3.SS1.p2.4.m4.1" class="ltx_Math" alttext="m&lt;d" display="inline"><semantics id="S3.SS1.p2.4.m4.1a"><mrow id="S3.SS1.p2.4.m4.1.1" xref="S3.SS1.p2.4.m4.1.1.cmml"><mi id="S3.SS1.p2.4.m4.1.1.2" xref="S3.SS1.p2.4.m4.1.1.2.cmml">m</mi><mo id="S3.SS1.p2.4.m4.1.1.1" xref="S3.SS1.p2.4.m4.1.1.1.cmml">&lt;</mo><mi id="S3.SS1.p2.4.m4.1.1.3" xref="S3.SS1.p2.4.m4.1.1.3.cmml">d</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p2.4.m4.1b"><apply id="S3.SS1.p2.4.m4.1.1.cmml" xref="S3.SS1.p2.4.m4.1.1"><lt id="S3.SS1.p2.4.m4.1.1.1.cmml" xref="S3.SS1.p2.4.m4.1.1.1"></lt><ci id="S3.SS1.p2.4.m4.1.1.2.cmml" xref="S3.SS1.p2.4.m4.1.1.2">𝑚</ci><ci id="S3.SS1.p2.4.m4.1.1.3.cmml" xref="S3.SS1.p2.4.m4.1.1.3">𝑑</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p2.4.m4.1c">m&lt;d</annotation></semantics></math>.
This can be expressed as <math id="S3.SS1.p2.5.m5.1" class="ltx_math_unparsed" alttext="\mathcal{C}_{E}[:m]=\{ce_{1}[:m],ce_{2}[:m],...,ce_{N}[:m]\}" display="inline"><semantics id="S3.SS1.p2.5.m5.1a"><mrow id="S3.SS1.p2.5.m5.1b"><msub id="S3.SS1.p2.5.m5.1.1"><mi class="ltx_font_mathcaligraphic" id="S3.SS1.p2.5.m5.1.1.2">𝒞</mi><mi id="S3.SS1.p2.5.m5.1.1.3">E</mi></msub><mrow id="S3.SS1.p2.5.m5.1.2"><mo stretchy="false" id="S3.SS1.p2.5.m5.1.2.1">[</mo><mo rspace="0.278em" id="S3.SS1.p2.5.m5.1.2.2">:</mo><mi id="S3.SS1.p2.5.m5.1.2.3">m</mi><mo stretchy="false" id="S3.SS1.p2.5.m5.1.2.4">]</mo></mrow><mo id="S3.SS1.p2.5.m5.1.3">=</mo><mrow id="S3.SS1.p2.5.m5.1.4"><mo stretchy="false" id="S3.SS1.p2.5.m5.1.4.1">{</mo><mi id="S3.SS1.p2.5.m5.1.4.2">c</mi><msub id="S3.SS1.p2.5.m5.1.4.3"><mi id="S3.SS1.p2.5.m5.1.4.3.2">e</mi><mn id="S3.SS1.p2.5.m5.1.4.3.3">1</mn></msub><mrow id="S3.SS1.p2.5.m5.1.4.4"><mo stretchy="false" id="S3.SS1.p2.5.m5.1.4.4.1">[</mo><mo rspace="0.278em" id="S3.SS1.p2.5.m5.1.4.4.2">:</mo><mi id="S3.SS1.p2.5.m5.1.4.4.3">m</mi><mo stretchy="false" id="S3.SS1.p2.5.m5.1.4.4.4">]</mo></mrow><mo id="S3.SS1.p2.5.m5.1.4.5">,</mo><mi id="S3.SS1.p2.5.m5.1.4.6">c</mi><msub id="S3.SS1.p2.5.m5.1.4.7"><mi id="S3.SS1.p2.5.m5.1.4.7.2">e</mi><mn id="S3.SS1.p2.5.m5.1.4.7.3">2</mn></msub><mrow id="S3.SS1.p2.5.m5.1.4.8"><mo stretchy="false" id="S3.SS1.p2.5.m5.1.4.8.1">[</mo><mo rspace="0.278em" id="S3.SS1.p2.5.m5.1.4.8.2">:</mo><mi id="S3.SS1.p2.5.m5.1.4.8.3">m</mi><mo stretchy="false" id="S3.SS1.p2.5.m5.1.4.8.4">]</mo></mrow><mo id="S3.SS1.p2.5.m5.1.4.9">,</mo><mi mathvariant="normal" id="S3.SS1.p2.5.m5.1.4.10">…</mi><mo id="S3.SS1.p2.5.m5.1.4.11">,</mo><mi id="S3.SS1.p2.5.m5.1.4.12">c</mi><msub id="S3.SS1.p2.5.m5.1.4.13"><mi id="S3.SS1.p2.5.m5.1.4.13.2">e</mi><mi id="S3.SS1.p2.5.m5.1.4.13.3">N</mi></msub><mrow id="S3.SS1.p2.5.m5.1.4.14"><mo stretchy="false" id="S3.SS1.p2.5.m5.1.4.14.1">[</mo><mo rspace="0.278em" id="S3.SS1.p2.5.m5.1.4.14.2">:</mo><mi id="S3.SS1.p2.5.m5.1.4.14.3">m</mi><mo stretchy="false" id="S3.SS1.p2.5.m5.1.4.14.4">]</mo></mrow><mo stretchy="false" id="S3.SS1.p2.5.m5.1.4.15">}</mo></mrow></mrow><annotation encoding="application/x-tex" id="S3.SS1.p2.5.m5.1c">\mathcal{C}_{E}[:m]=\{ce_{1}[:m],ce_{2}[:m],...,ce_{N}[:m]\}</annotation></semantics></math> where each reduced embedding vector <math id="S3.SS1.p2.6.m6.1" class="ltx_math_unparsed" alttext="ce_{i}[:m]\in\mathbb{R}^{m}" display="inline"><semantics id="S3.SS1.p2.6.m6.1a"><mrow id="S3.SS1.p2.6.m6.1b"><mi id="S3.SS1.p2.6.m6.1.1">c</mi><msub id="S3.SS1.p2.6.m6.1.2"><mi id="S3.SS1.p2.6.m6.1.2.2">e</mi><mi id="S3.SS1.p2.6.m6.1.2.3">i</mi></msub><mrow id="S3.SS1.p2.6.m6.1.3"><mo stretchy="false" id="S3.SS1.p2.6.m6.1.3.1">[</mo><mo rspace="0.278em" id="S3.SS1.p2.6.m6.1.3.2">:</mo><mi id="S3.SS1.p2.6.m6.1.3.3">m</mi><mo stretchy="false" id="S3.SS1.p2.6.m6.1.3.4">]</mo></mrow><mo id="S3.SS1.p2.6.m6.1.4">∈</mo><msup id="S3.SS1.p2.6.m6.1.5"><mi id="S3.SS1.p2.6.m6.1.5.2">ℝ</mi><mi id="S3.SS1.p2.6.m6.1.5.3">m</mi></msup></mrow><annotation encoding="application/x-tex" id="S3.SS1.p2.6.m6.1c">ce_{i}[:m]\in\mathbb{R}^{m}</annotation></semantics></math>.
A fundamental characteristic of Matryoshka embeddings is their capacity to preserve the essential properties of the original embeddings, even within a reduced-dimensional space.</p>
</div>
<figure id="S3.F2" class="ltx_figure"><img src="/html/2407.20243/assets/x2.png" id="S3.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="282" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>
Similarity loss is a measure of the discrepancy between the similarity of two embeddings in their original dimensional space and their similarity in a reduced dimensional space.
If the orange and blue embeddings are chosen randomly, this loss is referred to as pairwise similarity loss.
If the orange and blue embeddings are selected based on similarity in their original dimensional space (top-k nearest embeddings), this loss is referred to as top-k similarity loss.
Note that top-k similarity loss focuses on preserving local similarity relationships among neighboring embeddings.</figcaption>
</figure>
<figure id="S3.F3" class="ltx_figure"><img src="/html/2407.20243/assets/x3.png" id="S3.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="456" height="167" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Block diagrams illustrating both the unsupervised and supervised Matryoshka-Adaptor frameworks.
<span id="S3.F3.3.1" class="ltx_text ltx_font_italic">Unsupervised Matryoshka-Adaptor</span>: This variant exclusively utilizes corpus embeddings as input.
The training of the adaptor is achieved through a combination of top-k similarity loss and pairwise loss, which are calculated across multiple Matryoshka embeddings with various reduced dimensions.
<span id="S3.F3.4.2" class="ltx_text ltx_font_italic">Supervised Matryoshka-Adaptor</span>: In this variant, query embeddings and query-corpus pairs are provided as supplementary inputs.
A ranking loss is incorporated alongside the top-k and pairwise losses to facilitate the training of the adaptor.
Similar to the unsupervised setting, all losses are computed across Matryoshka embeddings with various reduced dimensions.</figcaption>
</figure>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Tuning objective functions</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.5" class="ltx_p">The proposed Matryoshka-Adaptor, represented by the function <math id="S3.SS2.p1.1.m1.1" class="ltx_Math" alttext="f:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}" display="inline"><semantics id="S3.SS2.p1.1.m1.1a"><mrow id="S3.SS2.p1.1.m1.1.1" xref="S3.SS2.p1.1.m1.1.1.cmml"><mi id="S3.SS2.p1.1.m1.1.1.2" xref="S3.SS2.p1.1.m1.1.1.2.cmml">f</mi><mo lspace="0.278em" rspace="0.278em" id="S3.SS2.p1.1.m1.1.1.1" xref="S3.SS2.p1.1.m1.1.1.1.cmml">:</mo><mrow id="S3.SS2.p1.1.m1.1.1.3" xref="S3.SS2.p1.1.m1.1.1.3.cmml"><msup id="S3.SS2.p1.1.m1.1.1.3.2" xref="S3.SS2.p1.1.m1.1.1.3.2.cmml"><mi id="S3.SS2.p1.1.m1.1.1.3.2.2" xref="S3.SS2.p1.1.m1.1.1.3.2.2.cmml">ℝ</mi><mi id="S3.SS2.p1.1.m1.1.1.3.2.3" xref="S3.SS2.p1.1.m1.1.1.3.2.3.cmml">d</mi></msup><mo stretchy="false" id="S3.SS2.p1.1.m1.1.1.3.1" xref="S3.SS2.p1.1.m1.1.1.3.1.cmml">→</mo><msup id="S3.SS2.p1.1.m1.1.1.3.3" xref="S3.SS2.p1.1.m1.1.1.3.3.cmml"><mi id="S3.SS2.p1.1.m1.1.1.3.3.2" xref="S3.SS2.p1.1.m1.1.1.3.3.2.cmml">ℝ</mi><mi id="S3.SS2.p1.1.m1.1.1.3.3.3" xref="S3.SS2.p1.1.m1.1.1.3.3.3.cmml">d</mi></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.1.m1.1b"><apply id="S3.SS2.p1.1.m1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1"><ci id="S3.SS2.p1.1.m1.1.1.1.cmml" xref="S3.SS2.p1.1.m1.1.1.1">:</ci><ci id="S3.SS2.p1.1.m1.1.1.2.cmml" xref="S3.SS2.p1.1.m1.1.1.2">𝑓</ci><apply id="S3.SS2.p1.1.m1.1.1.3.cmml" xref="S3.SS2.p1.1.m1.1.1.3"><ci id="S3.SS2.p1.1.m1.1.1.3.1.cmml" xref="S3.SS2.p1.1.m1.1.1.3.1">→</ci><apply id="S3.SS2.p1.1.m1.1.1.3.2.cmml" xref="S3.SS2.p1.1.m1.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS2.p1.1.m1.1.1.3.2.1.cmml" xref="S3.SS2.p1.1.m1.1.1.3.2">superscript</csymbol><ci id="S3.SS2.p1.1.m1.1.1.3.2.2.cmml" xref="S3.SS2.p1.1.m1.1.1.3.2.2">ℝ</ci><ci id="S3.SS2.p1.1.m1.1.1.3.2.3.cmml" xref="S3.SS2.p1.1.m1.1.1.3.2.3">𝑑</ci></apply><apply id="S3.SS2.p1.1.m1.1.1.3.3.cmml" xref="S3.SS2.p1.1.m1.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS2.p1.1.m1.1.1.3.3.1.cmml" xref="S3.SS2.p1.1.m1.1.1.3.3">superscript</csymbol><ci id="S3.SS2.p1.1.m1.1.1.3.3.2.cmml" xref="S3.SS2.p1.1.m1.1.1.3.3.2">ℝ</ci><ci id="S3.SS2.p1.1.m1.1.1.3.3.3.cmml" xref="S3.SS2.p1.1.m1.1.1.3.3.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.1.m1.1c">f:\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}</annotation></semantics></math>, is designed to modify the original embeddings in order to enhance their inherent Matryoshka properties.
We define the set of customized corpus embeddings as <math id="S3.SS2.p1.2.m2.4" class="ltx_Math" alttext="\hat{\mathcal{C}}_{E}=\{\hat{ce}_{1},\hat{ce}_{2},...,\hat{ce}_{N}\}" display="inline"><semantics id="S3.SS2.p1.2.m2.4a"><mrow id="S3.SS2.p1.2.m2.4.4" xref="S3.SS2.p1.2.m2.4.4.cmml"><msub id="S3.SS2.p1.2.m2.4.4.5" xref="S3.SS2.p1.2.m2.4.4.5.cmml"><mover accent="true" id="S3.SS2.p1.2.m2.4.4.5.2" xref="S3.SS2.p1.2.m2.4.4.5.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.2.m2.4.4.5.2.2" xref="S3.SS2.p1.2.m2.4.4.5.2.2.cmml">𝒞</mi><mo id="S3.SS2.p1.2.m2.4.4.5.2.1" xref="S3.SS2.p1.2.m2.4.4.5.2.1.cmml">^</mo></mover><mi id="S3.SS2.p1.2.m2.4.4.5.3" xref="S3.SS2.p1.2.m2.4.4.5.3.cmml">E</mi></msub><mo id="S3.SS2.p1.2.m2.4.4.4" xref="S3.SS2.p1.2.m2.4.4.4.cmml">=</mo><mrow id="S3.SS2.p1.2.m2.4.4.3.3" xref="S3.SS2.p1.2.m2.4.4.3.4.cmml"><mo stretchy="false" id="S3.SS2.p1.2.m2.4.4.3.3.4" xref="S3.SS2.p1.2.m2.4.4.3.4.cmml">{</mo><msub id="S3.SS2.p1.2.m2.2.2.1.1.1" xref="S3.SS2.p1.2.m2.2.2.1.1.1.cmml"><mover accent="true" id="S3.SS2.p1.2.m2.2.2.1.1.1.2" xref="S3.SS2.p1.2.m2.2.2.1.1.1.2.cmml"><mrow id="S3.SS2.p1.2.m2.2.2.1.1.1.2.2" xref="S3.SS2.p1.2.m2.2.2.1.1.1.2.2.cmml"><mi id="S3.SS2.p1.2.m2.2.2.1.1.1.2.2.2" xref="S3.SS2.p1.2.m2.2.2.1.1.1.2.2.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.2.2.1.1.1.2.2.1" xref="S3.SS2.p1.2.m2.2.2.1.1.1.2.2.1.cmml">​</mo><mi id="S3.SS2.p1.2.m2.2.2.1.1.1.2.2.3" xref="S3.SS2.p1.2.m2.2.2.1.1.1.2.2.3.cmml">e</mi></mrow><mo id="S3.SS2.p1.2.m2.2.2.1.1.1.2.1" xref="S3.SS2.p1.2.m2.2.2.1.1.1.2.1.cmml">^</mo></mover><mn id="S3.SS2.p1.2.m2.2.2.1.1.1.3" xref="S3.SS2.p1.2.m2.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S3.SS2.p1.2.m2.4.4.3.3.5" xref="S3.SS2.p1.2.m2.4.4.3.4.cmml">,</mo><msub id="S3.SS2.p1.2.m2.3.3.2.2.2" xref="S3.SS2.p1.2.m2.3.3.2.2.2.cmml"><mover accent="true" id="S3.SS2.p1.2.m2.3.3.2.2.2.2" xref="S3.SS2.p1.2.m2.3.3.2.2.2.2.cmml"><mrow id="S3.SS2.p1.2.m2.3.3.2.2.2.2.2" xref="S3.SS2.p1.2.m2.3.3.2.2.2.2.2.cmml"><mi id="S3.SS2.p1.2.m2.3.3.2.2.2.2.2.2" xref="S3.SS2.p1.2.m2.3.3.2.2.2.2.2.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.3.3.2.2.2.2.2.1" xref="S3.SS2.p1.2.m2.3.3.2.2.2.2.2.1.cmml">​</mo><mi id="S3.SS2.p1.2.m2.3.3.2.2.2.2.2.3" xref="S3.SS2.p1.2.m2.3.3.2.2.2.2.2.3.cmml">e</mi></mrow><mo id="S3.SS2.p1.2.m2.3.3.2.2.2.2.1" xref="S3.SS2.p1.2.m2.3.3.2.2.2.2.1.cmml">^</mo></mover><mn id="S3.SS2.p1.2.m2.3.3.2.2.2.3" xref="S3.SS2.p1.2.m2.3.3.2.2.2.3.cmml">2</mn></msub><mo id="S3.SS2.p1.2.m2.4.4.3.3.6" xref="S3.SS2.p1.2.m2.4.4.3.4.cmml">,</mo><mi mathvariant="normal" id="S3.SS2.p1.2.m2.1.1" xref="S3.SS2.p1.2.m2.1.1.cmml">…</mi><mo id="S3.SS2.p1.2.m2.4.4.3.3.7" xref="S3.SS2.p1.2.m2.4.4.3.4.cmml">,</mo><msub id="S3.SS2.p1.2.m2.4.4.3.3.3" xref="S3.SS2.p1.2.m2.4.4.3.3.3.cmml"><mover accent="true" id="S3.SS2.p1.2.m2.4.4.3.3.3.2" xref="S3.SS2.p1.2.m2.4.4.3.3.3.2.cmml"><mrow id="S3.SS2.p1.2.m2.4.4.3.3.3.2.2" xref="S3.SS2.p1.2.m2.4.4.3.3.3.2.2.cmml"><mi id="S3.SS2.p1.2.m2.4.4.3.3.3.2.2.2" xref="S3.SS2.p1.2.m2.4.4.3.3.3.2.2.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.2.m2.4.4.3.3.3.2.2.1" xref="S3.SS2.p1.2.m2.4.4.3.3.3.2.2.1.cmml">​</mo><mi id="S3.SS2.p1.2.m2.4.4.3.3.3.2.2.3" xref="S3.SS2.p1.2.m2.4.4.3.3.3.2.2.3.cmml">e</mi></mrow><mo id="S3.SS2.p1.2.m2.4.4.3.3.3.2.1" xref="S3.SS2.p1.2.m2.4.4.3.3.3.2.1.cmml">^</mo></mover><mi id="S3.SS2.p1.2.m2.4.4.3.3.3.3" xref="S3.SS2.p1.2.m2.4.4.3.3.3.3.cmml">N</mi></msub><mo stretchy="false" id="S3.SS2.p1.2.m2.4.4.3.3.8" xref="S3.SS2.p1.2.m2.4.4.3.4.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.2.m2.4b"><apply id="S3.SS2.p1.2.m2.4.4.cmml" xref="S3.SS2.p1.2.m2.4.4"><eq id="S3.SS2.p1.2.m2.4.4.4.cmml" xref="S3.SS2.p1.2.m2.4.4.4"></eq><apply id="S3.SS2.p1.2.m2.4.4.5.cmml" xref="S3.SS2.p1.2.m2.4.4.5"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m2.4.4.5.1.cmml" xref="S3.SS2.p1.2.m2.4.4.5">subscript</csymbol><apply id="S3.SS2.p1.2.m2.4.4.5.2.cmml" xref="S3.SS2.p1.2.m2.4.4.5.2"><ci id="S3.SS2.p1.2.m2.4.4.5.2.1.cmml" xref="S3.SS2.p1.2.m2.4.4.5.2.1">^</ci><ci id="S3.SS2.p1.2.m2.4.4.5.2.2.cmml" xref="S3.SS2.p1.2.m2.4.4.5.2.2">𝒞</ci></apply><ci id="S3.SS2.p1.2.m2.4.4.5.3.cmml" xref="S3.SS2.p1.2.m2.4.4.5.3">𝐸</ci></apply><set id="S3.SS2.p1.2.m2.4.4.3.4.cmml" xref="S3.SS2.p1.2.m2.4.4.3.3"><apply id="S3.SS2.p1.2.m2.2.2.1.1.1.cmml" xref="S3.SS2.p1.2.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m2.2.2.1.1.1.1.cmml" xref="S3.SS2.p1.2.m2.2.2.1.1.1">subscript</csymbol><apply id="S3.SS2.p1.2.m2.2.2.1.1.1.2.cmml" xref="S3.SS2.p1.2.m2.2.2.1.1.1.2"><ci id="S3.SS2.p1.2.m2.2.2.1.1.1.2.1.cmml" xref="S3.SS2.p1.2.m2.2.2.1.1.1.2.1">^</ci><apply id="S3.SS2.p1.2.m2.2.2.1.1.1.2.2.cmml" xref="S3.SS2.p1.2.m2.2.2.1.1.1.2.2"><times id="S3.SS2.p1.2.m2.2.2.1.1.1.2.2.1.cmml" xref="S3.SS2.p1.2.m2.2.2.1.1.1.2.2.1"></times><ci id="S3.SS2.p1.2.m2.2.2.1.1.1.2.2.2.cmml" xref="S3.SS2.p1.2.m2.2.2.1.1.1.2.2.2">𝑐</ci><ci id="S3.SS2.p1.2.m2.2.2.1.1.1.2.2.3.cmml" xref="S3.SS2.p1.2.m2.2.2.1.1.1.2.2.3">𝑒</ci></apply></apply><cn type="integer" id="S3.SS2.p1.2.m2.2.2.1.1.1.3.cmml" xref="S3.SS2.p1.2.m2.2.2.1.1.1.3">1</cn></apply><apply id="S3.SS2.p1.2.m2.3.3.2.2.2.cmml" xref="S3.SS2.p1.2.m2.3.3.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m2.3.3.2.2.2.1.cmml" xref="S3.SS2.p1.2.m2.3.3.2.2.2">subscript</csymbol><apply id="S3.SS2.p1.2.m2.3.3.2.2.2.2.cmml" xref="S3.SS2.p1.2.m2.3.3.2.2.2.2"><ci id="S3.SS2.p1.2.m2.3.3.2.2.2.2.1.cmml" xref="S3.SS2.p1.2.m2.3.3.2.2.2.2.1">^</ci><apply id="S3.SS2.p1.2.m2.3.3.2.2.2.2.2.cmml" xref="S3.SS2.p1.2.m2.3.3.2.2.2.2.2"><times id="S3.SS2.p1.2.m2.3.3.2.2.2.2.2.1.cmml" xref="S3.SS2.p1.2.m2.3.3.2.2.2.2.2.1"></times><ci id="S3.SS2.p1.2.m2.3.3.2.2.2.2.2.2.cmml" xref="S3.SS2.p1.2.m2.3.3.2.2.2.2.2.2">𝑐</ci><ci id="S3.SS2.p1.2.m2.3.3.2.2.2.2.2.3.cmml" xref="S3.SS2.p1.2.m2.3.3.2.2.2.2.2.3">𝑒</ci></apply></apply><cn type="integer" id="S3.SS2.p1.2.m2.3.3.2.2.2.3.cmml" xref="S3.SS2.p1.2.m2.3.3.2.2.2.3">2</cn></apply><ci id="S3.SS2.p1.2.m2.1.1.cmml" xref="S3.SS2.p1.2.m2.1.1">…</ci><apply id="S3.SS2.p1.2.m2.4.4.3.3.3.cmml" xref="S3.SS2.p1.2.m2.4.4.3.3.3"><csymbol cd="ambiguous" id="S3.SS2.p1.2.m2.4.4.3.3.3.1.cmml" xref="S3.SS2.p1.2.m2.4.4.3.3.3">subscript</csymbol><apply id="S3.SS2.p1.2.m2.4.4.3.3.3.2.cmml" xref="S3.SS2.p1.2.m2.4.4.3.3.3.2"><ci id="S3.SS2.p1.2.m2.4.4.3.3.3.2.1.cmml" xref="S3.SS2.p1.2.m2.4.4.3.3.3.2.1">^</ci><apply id="S3.SS2.p1.2.m2.4.4.3.3.3.2.2.cmml" xref="S3.SS2.p1.2.m2.4.4.3.3.3.2.2"><times id="S3.SS2.p1.2.m2.4.4.3.3.3.2.2.1.cmml" xref="S3.SS2.p1.2.m2.4.4.3.3.3.2.2.1"></times><ci id="S3.SS2.p1.2.m2.4.4.3.3.3.2.2.2.cmml" xref="S3.SS2.p1.2.m2.4.4.3.3.3.2.2.2">𝑐</ci><ci id="S3.SS2.p1.2.m2.4.4.3.3.3.2.2.3.cmml" xref="S3.SS2.p1.2.m2.4.4.3.3.3.2.2.3">𝑒</ci></apply></apply><ci id="S3.SS2.p1.2.m2.4.4.3.3.3.3.cmml" xref="S3.SS2.p1.2.m2.4.4.3.3.3.3">𝑁</ci></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.2.m2.4c">\hat{\mathcal{C}}_{E}=\{\hat{ce}_{1},\hat{ce}_{2},...,\hat{ce}_{N}\}</annotation></semantics></math> and their corresponding Matryoshka embeddings as <math id="S3.SS2.p1.3.m3.1" class="ltx_math_unparsed" alttext="\hat{\mathcal{C}}_{E}[:m]=\{\hat{ce}_{1}[:m],\hat{ce}_{2}[:m],...,\hat{ce}_{N}[:m]\}" display="inline"><semantics id="S3.SS2.p1.3.m3.1a"><mrow id="S3.SS2.p1.3.m3.1b"><msub id="S3.SS2.p1.3.m3.1.1"><mover accent="true" id="S3.SS2.p1.3.m3.1.1.2"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p1.3.m3.1.1.2.2">𝒞</mi><mo id="S3.SS2.p1.3.m3.1.1.2.1">^</mo></mover><mi id="S3.SS2.p1.3.m3.1.1.3">E</mi></msub><mrow id="S3.SS2.p1.3.m3.1.2"><mo stretchy="false" id="S3.SS2.p1.3.m3.1.2.1">[</mo><mo rspace="0.278em" id="S3.SS2.p1.3.m3.1.2.2">:</mo><mi id="S3.SS2.p1.3.m3.1.2.3">m</mi><mo stretchy="false" id="S3.SS2.p1.3.m3.1.2.4">]</mo></mrow><mo id="S3.SS2.p1.3.m3.1.3">=</mo><mrow id="S3.SS2.p1.3.m3.1.4"><mo stretchy="false" id="S3.SS2.p1.3.m3.1.4.1">{</mo><msub id="S3.SS2.p1.3.m3.1.4.2"><mover accent="true" id="S3.SS2.p1.3.m3.1.4.2.2"><mrow id="S3.SS2.p1.3.m3.1.4.2.2.2"><mi id="S3.SS2.p1.3.m3.1.4.2.2.2.2">c</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.3.m3.1.4.2.2.2.1">​</mo><mi id="S3.SS2.p1.3.m3.1.4.2.2.2.3">e</mi></mrow><mo id="S3.SS2.p1.3.m3.1.4.2.2.1">^</mo></mover><mn id="S3.SS2.p1.3.m3.1.4.2.3">1</mn></msub><mrow id="S3.SS2.p1.3.m3.1.4.3"><mo stretchy="false" id="S3.SS2.p1.3.m3.1.4.3.1">[</mo><mo rspace="0.278em" id="S3.SS2.p1.3.m3.1.4.3.2">:</mo><mi id="S3.SS2.p1.3.m3.1.4.3.3">m</mi><mo stretchy="false" id="S3.SS2.p1.3.m3.1.4.3.4">]</mo></mrow><mo id="S3.SS2.p1.3.m3.1.4.4">,</mo><msub id="S3.SS2.p1.3.m3.1.4.5"><mover accent="true" id="S3.SS2.p1.3.m3.1.4.5.2"><mrow id="S3.SS2.p1.3.m3.1.4.5.2.2"><mi id="S3.SS2.p1.3.m3.1.4.5.2.2.2">c</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.3.m3.1.4.5.2.2.1">​</mo><mi id="S3.SS2.p1.3.m3.1.4.5.2.2.3">e</mi></mrow><mo id="S3.SS2.p1.3.m3.1.4.5.2.1">^</mo></mover><mn id="S3.SS2.p1.3.m3.1.4.5.3">2</mn></msub><mrow id="S3.SS2.p1.3.m3.1.4.6"><mo stretchy="false" id="S3.SS2.p1.3.m3.1.4.6.1">[</mo><mo rspace="0.278em" id="S3.SS2.p1.3.m3.1.4.6.2">:</mo><mi id="S3.SS2.p1.3.m3.1.4.6.3">m</mi><mo stretchy="false" id="S3.SS2.p1.3.m3.1.4.6.4">]</mo></mrow><mo id="S3.SS2.p1.3.m3.1.4.7">,</mo><mi mathvariant="normal" id="S3.SS2.p1.3.m3.1.4.8">…</mi><mo id="S3.SS2.p1.3.m3.1.4.9">,</mo><msub id="S3.SS2.p1.3.m3.1.4.10"><mover accent="true" id="S3.SS2.p1.3.m3.1.4.10.2"><mrow id="S3.SS2.p1.3.m3.1.4.10.2.2"><mi id="S3.SS2.p1.3.m3.1.4.10.2.2.2">c</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.3.m3.1.4.10.2.2.1">​</mo><mi id="S3.SS2.p1.3.m3.1.4.10.2.2.3">e</mi></mrow><mo id="S3.SS2.p1.3.m3.1.4.10.2.1">^</mo></mover><mi id="S3.SS2.p1.3.m3.1.4.10.3">N</mi></msub><mrow id="S3.SS2.p1.3.m3.1.4.11"><mo stretchy="false" id="S3.SS2.p1.3.m3.1.4.11.1">[</mo><mo rspace="0.278em" id="S3.SS2.p1.3.m3.1.4.11.2">:</mo><mi id="S3.SS2.p1.3.m3.1.4.11.3">m</mi><mo stretchy="false" id="S3.SS2.p1.3.m3.1.4.11.4">]</mo></mrow><mo stretchy="false" id="S3.SS2.p1.3.m3.1.4.12">}</mo></mrow></mrow><annotation encoding="application/x-tex" id="S3.SS2.p1.3.m3.1c">\hat{\mathcal{C}}_{E}[:m]=\{\hat{ce}_{1}[:m],\hat{ce}_{2}[:m],...,\hat{ce}_{N}[:m]\}</annotation></semantics></math> where <math id="S3.SS2.p1.4.m4.1" class="ltx_Math" alttext="\hat{ce}_{i}=f(ce_{i})" display="inline"><semantics id="S3.SS2.p1.4.m4.1a"><mrow id="S3.SS2.p1.4.m4.1.1" xref="S3.SS2.p1.4.m4.1.1.cmml"><msub id="S3.SS2.p1.4.m4.1.1.3" xref="S3.SS2.p1.4.m4.1.1.3.cmml"><mover accent="true" id="S3.SS2.p1.4.m4.1.1.3.2" xref="S3.SS2.p1.4.m4.1.1.3.2.cmml"><mrow id="S3.SS2.p1.4.m4.1.1.3.2.2" xref="S3.SS2.p1.4.m4.1.1.3.2.2.cmml"><mi id="S3.SS2.p1.4.m4.1.1.3.2.2.2" xref="S3.SS2.p1.4.m4.1.1.3.2.2.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.4.m4.1.1.3.2.2.1" xref="S3.SS2.p1.4.m4.1.1.3.2.2.1.cmml">​</mo><mi id="S3.SS2.p1.4.m4.1.1.3.2.2.3" xref="S3.SS2.p1.4.m4.1.1.3.2.2.3.cmml">e</mi></mrow><mo id="S3.SS2.p1.4.m4.1.1.3.2.1" xref="S3.SS2.p1.4.m4.1.1.3.2.1.cmml">^</mo></mover><mi id="S3.SS2.p1.4.m4.1.1.3.3" xref="S3.SS2.p1.4.m4.1.1.3.3.cmml">i</mi></msub><mo id="S3.SS2.p1.4.m4.1.1.2" xref="S3.SS2.p1.4.m4.1.1.2.cmml">=</mo><mrow id="S3.SS2.p1.4.m4.1.1.1" xref="S3.SS2.p1.4.m4.1.1.1.cmml"><mi id="S3.SS2.p1.4.m4.1.1.1.3" xref="S3.SS2.p1.4.m4.1.1.1.3.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.4.m4.1.1.1.2" xref="S3.SS2.p1.4.m4.1.1.1.2.cmml">​</mo><mrow id="S3.SS2.p1.4.m4.1.1.1.1.1" xref="S3.SS2.p1.4.m4.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p1.4.m4.1.1.1.1.1.2" xref="S3.SS2.p1.4.m4.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p1.4.m4.1.1.1.1.1.1" xref="S3.SS2.p1.4.m4.1.1.1.1.1.1.cmml"><mi id="S3.SS2.p1.4.m4.1.1.1.1.1.1.2" xref="S3.SS2.p1.4.m4.1.1.1.1.1.1.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p1.4.m4.1.1.1.1.1.1.1" xref="S3.SS2.p1.4.m4.1.1.1.1.1.1.1.cmml">​</mo><msub id="S3.SS2.p1.4.m4.1.1.1.1.1.1.3" xref="S3.SS2.p1.4.m4.1.1.1.1.1.1.3.cmml"><mi id="S3.SS2.p1.4.m4.1.1.1.1.1.1.3.2" xref="S3.SS2.p1.4.m4.1.1.1.1.1.1.3.2.cmml">e</mi><mi id="S3.SS2.p1.4.m4.1.1.1.1.1.1.3.3" xref="S3.SS2.p1.4.m4.1.1.1.1.1.1.3.3.cmml">i</mi></msub></mrow><mo stretchy="false" id="S3.SS2.p1.4.m4.1.1.1.1.1.3" xref="S3.SS2.p1.4.m4.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.4.m4.1b"><apply id="S3.SS2.p1.4.m4.1.1.cmml" xref="S3.SS2.p1.4.m4.1.1"><eq id="S3.SS2.p1.4.m4.1.1.2.cmml" xref="S3.SS2.p1.4.m4.1.1.2"></eq><apply id="S3.SS2.p1.4.m4.1.1.3.cmml" xref="S3.SS2.p1.4.m4.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p1.4.m4.1.1.3.1.cmml" xref="S3.SS2.p1.4.m4.1.1.3">subscript</csymbol><apply id="S3.SS2.p1.4.m4.1.1.3.2.cmml" xref="S3.SS2.p1.4.m4.1.1.3.2"><ci id="S3.SS2.p1.4.m4.1.1.3.2.1.cmml" xref="S3.SS2.p1.4.m4.1.1.3.2.1">^</ci><apply id="S3.SS2.p1.4.m4.1.1.3.2.2.cmml" xref="S3.SS2.p1.4.m4.1.1.3.2.2"><times id="S3.SS2.p1.4.m4.1.1.3.2.2.1.cmml" xref="S3.SS2.p1.4.m4.1.1.3.2.2.1"></times><ci id="S3.SS2.p1.4.m4.1.1.3.2.2.2.cmml" xref="S3.SS2.p1.4.m4.1.1.3.2.2.2">𝑐</ci><ci id="S3.SS2.p1.4.m4.1.1.3.2.2.3.cmml" xref="S3.SS2.p1.4.m4.1.1.3.2.2.3">𝑒</ci></apply></apply><ci id="S3.SS2.p1.4.m4.1.1.3.3.cmml" xref="S3.SS2.p1.4.m4.1.1.3.3">𝑖</ci></apply><apply id="S3.SS2.p1.4.m4.1.1.1.cmml" xref="S3.SS2.p1.4.m4.1.1.1"><times id="S3.SS2.p1.4.m4.1.1.1.2.cmml" xref="S3.SS2.p1.4.m4.1.1.1.2"></times><ci id="S3.SS2.p1.4.m4.1.1.1.3.cmml" xref="S3.SS2.p1.4.m4.1.1.1.3">𝑓</ci><apply id="S3.SS2.p1.4.m4.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.4.m4.1.1.1.1.1"><times id="S3.SS2.p1.4.m4.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p1.4.m4.1.1.1.1.1.1.1"></times><ci id="S3.SS2.p1.4.m4.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p1.4.m4.1.1.1.1.1.1.2">𝑐</ci><apply id="S3.SS2.p1.4.m4.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p1.4.m4.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p1.4.m4.1.1.1.1.1.1.3.1.cmml" xref="S3.SS2.p1.4.m4.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.SS2.p1.4.m4.1.1.1.1.1.1.3.2.cmml" xref="S3.SS2.p1.4.m4.1.1.1.1.1.1.3.2">𝑒</ci><ci id="S3.SS2.p1.4.m4.1.1.1.1.1.1.3.3.cmml" xref="S3.SS2.p1.4.m4.1.1.1.1.1.1.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.4.m4.1c">\hat{ce}_{i}=f(ce_{i})</annotation></semantics></math>.
The primary objective of the function <math id="S3.SS2.p1.5.m5.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S3.SS2.p1.5.m5.1a"><mi id="S3.SS2.p1.5.m5.1.1" xref="S3.SS2.p1.5.m5.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p1.5.m5.1b"><ci id="S3.SS2.p1.5.m5.1.1.cmml" xref="S3.SS2.p1.5.m5.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p1.5.m5.1c">f</annotation></semantics></math> is to maximize the Matryoshka properties through this customization process.
This means ensuring the similarity between any two embeddings remains as consistent as possible, whether they are represented in the original high-dimensional space or the reduced low-dimensional space.</p>
</div>
<div id="S3.SS2.p2" class="ltx_para">
<p id="S3.SS2.p2.1" class="ltx_p">To achieve this objective, we introduce two loss functions. The first loss function, denoted as <math id="S3.SS2.p2.1.m1.1" class="ltx_Math" alttext="\mathcal{L}_{pair}" display="inline"><semantics id="S3.SS2.p2.1.m1.1a"><msub id="S3.SS2.p2.1.m1.1.1" xref="S3.SS2.p2.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p2.1.m1.1.1.2" xref="S3.SS2.p2.1.m1.1.1.2.cmml">ℒ</mi><mrow id="S3.SS2.p2.1.m1.1.1.3" xref="S3.SS2.p2.1.m1.1.1.3.cmml"><mi id="S3.SS2.p2.1.m1.1.1.3.2" xref="S3.SS2.p2.1.m1.1.1.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.1.m1.1.1.3.1" xref="S3.SS2.p2.1.m1.1.1.3.1.cmml">​</mo><mi id="S3.SS2.p2.1.m1.1.1.3.3" xref="S3.SS2.p2.1.m1.1.1.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.1.m1.1.1.3.1a" xref="S3.SS2.p2.1.m1.1.1.3.1.cmml">​</mo><mi id="S3.SS2.p2.1.m1.1.1.3.4" xref="S3.SS2.p2.1.m1.1.1.3.4.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.1.m1.1.1.3.1b" xref="S3.SS2.p2.1.m1.1.1.3.1.cmml">​</mo><mi id="S3.SS2.p2.1.m1.1.1.3.5" xref="S3.SS2.p2.1.m1.1.1.3.5.cmml">r</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.1.m1.1b"><apply id="S3.SS2.p2.1.m1.1.1.cmml" xref="S3.SS2.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p2.1.m1.1.1.1.cmml" xref="S3.SS2.p2.1.m1.1.1">subscript</csymbol><ci id="S3.SS2.p2.1.m1.1.1.2.cmml" xref="S3.SS2.p2.1.m1.1.1.2">ℒ</ci><apply id="S3.SS2.p2.1.m1.1.1.3.cmml" xref="S3.SS2.p2.1.m1.1.1.3"><times id="S3.SS2.p2.1.m1.1.1.3.1.cmml" xref="S3.SS2.p2.1.m1.1.1.3.1"></times><ci id="S3.SS2.p2.1.m1.1.1.3.2.cmml" xref="S3.SS2.p2.1.m1.1.1.3.2">𝑝</ci><ci id="S3.SS2.p2.1.m1.1.1.3.3.cmml" xref="S3.SS2.p2.1.m1.1.1.3.3">𝑎</ci><ci id="S3.SS2.p2.1.m1.1.1.3.4.cmml" xref="S3.SS2.p2.1.m1.1.1.3.4">𝑖</ci><ci id="S3.SS2.p2.1.m1.1.1.3.5.cmml" xref="S3.SS2.p2.1.m1.1.1.3.5">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.1.m1.1c">\mathcal{L}_{pair}</annotation></semantics></math>, is designed to preserve the pairwise similarity between the original embeddings in their reduced-dimension Matryoshka form, expressed as:</p>
<table id="A6.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E1.m1.1" class="ltx_Math" alttext="\displaystyle\mathcal{L}_{pair}=\sum_{i}" display="inline"><semantics id="S3.E1.m1.1a"><mrow id="S3.E1.m1.1.1" xref="S3.E1.m1.1.1.cmml"><msub id="S3.E1.m1.1.1.2" xref="S3.E1.m1.1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E1.m1.1.1.2.2" xref="S3.E1.m1.1.1.2.2.cmml">ℒ</mi><mrow id="S3.E1.m1.1.1.2.3" xref="S3.E1.m1.1.1.2.3.cmml"><mi id="S3.E1.m1.1.1.2.3.2" xref="S3.E1.m1.1.1.2.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.E1.m1.1.1.2.3.1" xref="S3.E1.m1.1.1.2.3.1.cmml">​</mo><mi id="S3.E1.m1.1.1.2.3.3" xref="S3.E1.m1.1.1.2.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S3.E1.m1.1.1.2.3.1a" xref="S3.E1.m1.1.1.2.3.1.cmml">​</mo><mi id="S3.E1.m1.1.1.2.3.4" xref="S3.E1.m1.1.1.2.3.4.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.E1.m1.1.1.2.3.1b" xref="S3.E1.m1.1.1.2.3.1.cmml">​</mo><mi id="S3.E1.m1.1.1.2.3.5" xref="S3.E1.m1.1.1.2.3.5.cmml">r</mi></mrow></msub><mo id="S3.E1.m1.1.1.1" xref="S3.E1.m1.1.1.1.cmml">=</mo><mstyle displaystyle="true" id="S3.E1.m1.1.1.3" xref="S3.E1.m1.1.1.3.cmml"><munder id="S3.E1.m1.1.1.3a" xref="S3.E1.m1.1.1.3.cmml"><mo movablelimits="false" id="S3.E1.m1.1.1.3.2" xref="S3.E1.m1.1.1.3.2.cmml">∑</mo><mi id="S3.E1.m1.1.1.3.3" xref="S3.E1.m1.1.1.3.3.cmml">i</mi></munder></mstyle></mrow><annotation-xml encoding="MathML-Content" id="S3.E1.m1.1b"><apply id="S3.E1.m1.1.1.cmml" xref="S3.E1.m1.1.1"><eq id="S3.E1.m1.1.1.1.cmml" xref="S3.E1.m1.1.1.1"></eq><apply id="S3.E1.m1.1.1.2.cmml" xref="S3.E1.m1.1.1.2"><csymbol cd="ambiguous" id="S3.E1.m1.1.1.2.1.cmml" xref="S3.E1.m1.1.1.2">subscript</csymbol><ci id="S3.E1.m1.1.1.2.2.cmml" xref="S3.E1.m1.1.1.2.2">ℒ</ci><apply id="S3.E1.m1.1.1.2.3.cmml" xref="S3.E1.m1.1.1.2.3"><times id="S3.E1.m1.1.1.2.3.1.cmml" xref="S3.E1.m1.1.1.2.3.1"></times><ci id="S3.E1.m1.1.1.2.3.2.cmml" xref="S3.E1.m1.1.1.2.3.2">𝑝</ci><ci id="S3.E1.m1.1.1.2.3.3.cmml" xref="S3.E1.m1.1.1.2.3.3">𝑎</ci><ci id="S3.E1.m1.1.1.2.3.4.cmml" xref="S3.E1.m1.1.1.2.3.4">𝑖</ci><ci id="S3.E1.m1.1.1.2.3.5.cmml" xref="S3.E1.m1.1.1.2.3.5">𝑟</ci></apply></apply><apply id="S3.E1.m1.1.1.3.cmml" xref="S3.E1.m1.1.1.3"><csymbol cd="ambiguous" id="S3.E1.m1.1.1.3.1.cmml" xref="S3.E1.m1.1.1.3">subscript</csymbol><sum id="S3.E1.m1.1.1.3.2.cmml" xref="S3.E1.m1.1.1.3.2"></sum><ci id="S3.E1.m1.1.1.3.3.cmml" xref="S3.E1.m1.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E1.m1.1c">\displaystyle\mathcal{L}_{pair}=\sum_{i}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E1.m2.2" class="ltx_Math" alttext="\displaystyle\sum_{j}\sum_{m}|Sim(ce_{i},ce_{j})-" display="inline"><semantics id="S3.E1.m2.2a"><mrow id="S3.E1.m2.2.2" xref="S3.E1.m2.2.2.cmml"><mrow id="S3.E1.m2.2.2.4" xref="S3.E1.m2.2.2.4.cmml"><mstyle displaystyle="true" id="S3.E1.m2.2.2.4.1" xref="S3.E1.m2.2.2.4.1.cmml"><munder id="S3.E1.m2.2.2.4.1a" xref="S3.E1.m2.2.2.4.1.cmml"><mo movablelimits="false" id="S3.E1.m2.2.2.4.1.2" xref="S3.E1.m2.2.2.4.1.2.cmml">∑</mo><mi id="S3.E1.m2.2.2.4.1.3" xref="S3.E1.m2.2.2.4.1.3.cmml">j</mi></munder></mstyle><mstyle displaystyle="true" id="S3.E1.m2.2.2.4.2" xref="S3.E1.m2.2.2.4.2.cmml"><munder id="S3.E1.m2.2.2.4.2a" xref="S3.E1.m2.2.2.4.2.cmml"><mo movablelimits="false" id="S3.E1.m2.2.2.4.2.2" xref="S3.E1.m2.2.2.4.2.2.cmml">∑</mo><mi id="S3.E1.m2.2.2.4.2.3" xref="S3.E1.m2.2.2.4.2.3.cmml">m</mi></munder></mstyle></mrow><mo fence="false" id="S3.E1.m2.2.2.3" xref="S3.E1.m2.2.2.3.cmml">|</mo><mrow id="S3.E1.m2.2.2.2" xref="S3.E1.m2.2.2.2.cmml"><mrow id="S3.E1.m2.2.2.2.2" xref="S3.E1.m2.2.2.2.2.cmml"><mi id="S3.E1.m2.2.2.2.2.4" xref="S3.E1.m2.2.2.2.2.4.cmml">S</mi><mo lspace="0em" rspace="0em" id="S3.E1.m2.2.2.2.2.3" xref="S3.E1.m2.2.2.2.2.3.cmml">​</mo><mi id="S3.E1.m2.2.2.2.2.5" xref="S3.E1.m2.2.2.2.2.5.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.E1.m2.2.2.2.2.3a" xref="S3.E1.m2.2.2.2.2.3.cmml">​</mo><mi id="S3.E1.m2.2.2.2.2.6" xref="S3.E1.m2.2.2.2.2.6.cmml">m</mi><mo lspace="0em" rspace="0em" id="S3.E1.m2.2.2.2.2.3b" xref="S3.E1.m2.2.2.2.2.3.cmml">​</mo><mrow id="S3.E1.m2.2.2.2.2.2.2" xref="S3.E1.m2.2.2.2.2.2.3.cmml"><mo stretchy="false" id="S3.E1.m2.2.2.2.2.2.2.3" xref="S3.E1.m2.2.2.2.2.2.3.cmml">(</mo><mrow id="S3.E1.m2.1.1.1.1.1.1.1" xref="S3.E1.m2.1.1.1.1.1.1.1.cmml"><mi id="S3.E1.m2.1.1.1.1.1.1.1.2" xref="S3.E1.m2.1.1.1.1.1.1.1.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.E1.m2.1.1.1.1.1.1.1.1" xref="S3.E1.m2.1.1.1.1.1.1.1.1.cmml">​</mo><msub id="S3.E1.m2.1.1.1.1.1.1.1.3" xref="S3.E1.m2.1.1.1.1.1.1.1.3.cmml"><mi id="S3.E1.m2.1.1.1.1.1.1.1.3.2" xref="S3.E1.m2.1.1.1.1.1.1.1.3.2.cmml">e</mi><mi id="S3.E1.m2.1.1.1.1.1.1.1.3.3" xref="S3.E1.m2.1.1.1.1.1.1.1.3.3.cmml">i</mi></msub></mrow><mo id="S3.E1.m2.2.2.2.2.2.2.4" xref="S3.E1.m2.2.2.2.2.2.3.cmml">,</mo><mrow id="S3.E1.m2.2.2.2.2.2.2.2" xref="S3.E1.m2.2.2.2.2.2.2.2.cmml"><mi id="S3.E1.m2.2.2.2.2.2.2.2.2" xref="S3.E1.m2.2.2.2.2.2.2.2.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.E1.m2.2.2.2.2.2.2.2.1" xref="S3.E1.m2.2.2.2.2.2.2.2.1.cmml">​</mo><msub id="S3.E1.m2.2.2.2.2.2.2.2.3" xref="S3.E1.m2.2.2.2.2.2.2.2.3.cmml"><mi id="S3.E1.m2.2.2.2.2.2.2.2.3.2" xref="S3.E1.m2.2.2.2.2.2.2.2.3.2.cmml">e</mi><mi id="S3.E1.m2.2.2.2.2.2.2.2.3.3" xref="S3.E1.m2.2.2.2.2.2.2.2.3.3.cmml">j</mi></msub></mrow><mo stretchy="false" id="S3.E1.m2.2.2.2.2.2.2.5" xref="S3.E1.m2.2.2.2.2.2.3.cmml">)</mo></mrow></mrow><mo id="S3.E1.m2.2.2.2.4" xref="S3.E1.m2.2.2.2.4.cmml">−</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E1.m2.2b"><apply id="S3.E1.m2.2.2.cmml" xref="S3.E1.m2.2.2"><csymbol cd="latexml" id="S3.E1.m2.2.2.3.cmml" xref="S3.E1.m2.2.2.3">conditional</csymbol><apply id="S3.E1.m2.2.2.4.cmml" xref="S3.E1.m2.2.2.4"><apply id="S3.E1.m2.2.2.4.1.cmml" xref="S3.E1.m2.2.2.4.1"><csymbol cd="ambiguous" id="S3.E1.m2.2.2.4.1.1.cmml" xref="S3.E1.m2.2.2.4.1">subscript</csymbol><sum id="S3.E1.m2.2.2.4.1.2.cmml" xref="S3.E1.m2.2.2.4.1.2"></sum><ci id="S3.E1.m2.2.2.4.1.3.cmml" xref="S3.E1.m2.2.2.4.1.3">𝑗</ci></apply><apply id="S3.E1.m2.2.2.4.2.cmml" xref="S3.E1.m2.2.2.4.2"><csymbol cd="ambiguous" id="S3.E1.m2.2.2.4.2.1.cmml" xref="S3.E1.m2.2.2.4.2">subscript</csymbol><sum id="S3.E1.m2.2.2.4.2.2.cmml" xref="S3.E1.m2.2.2.4.2.2"></sum><ci id="S3.E1.m2.2.2.4.2.3.cmml" xref="S3.E1.m2.2.2.4.2.3">𝑚</ci></apply></apply><apply id="S3.E1.m2.2.2.2.cmml" xref="S3.E1.m2.2.2.2"><csymbol cd="latexml" id="S3.E1.m2.2.2.2.3.cmml" xref="S3.E1.m2.2.2.2">limit-from</csymbol><apply id="S3.E1.m2.2.2.2.2.cmml" xref="S3.E1.m2.2.2.2.2"><times id="S3.E1.m2.2.2.2.2.3.cmml" xref="S3.E1.m2.2.2.2.2.3"></times><ci id="S3.E1.m2.2.2.2.2.4.cmml" xref="S3.E1.m2.2.2.2.2.4">𝑆</ci><ci id="S3.E1.m2.2.2.2.2.5.cmml" xref="S3.E1.m2.2.2.2.2.5">𝑖</ci><ci id="S3.E1.m2.2.2.2.2.6.cmml" xref="S3.E1.m2.2.2.2.2.6">𝑚</ci><interval closure="open" id="S3.E1.m2.2.2.2.2.2.3.cmml" xref="S3.E1.m2.2.2.2.2.2.2"><apply id="S3.E1.m2.1.1.1.1.1.1.1.cmml" xref="S3.E1.m2.1.1.1.1.1.1.1"><times id="S3.E1.m2.1.1.1.1.1.1.1.1.cmml" xref="S3.E1.m2.1.1.1.1.1.1.1.1"></times><ci id="S3.E1.m2.1.1.1.1.1.1.1.2.cmml" xref="S3.E1.m2.1.1.1.1.1.1.1.2">𝑐</ci><apply id="S3.E1.m2.1.1.1.1.1.1.1.3.cmml" xref="S3.E1.m2.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E1.m2.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E1.m2.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E1.m2.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E1.m2.1.1.1.1.1.1.1.3.2">𝑒</ci><ci id="S3.E1.m2.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E1.m2.1.1.1.1.1.1.1.3.3">𝑖</ci></apply></apply><apply id="S3.E1.m2.2.2.2.2.2.2.2.cmml" xref="S3.E1.m2.2.2.2.2.2.2.2"><times id="S3.E1.m2.2.2.2.2.2.2.2.1.cmml" xref="S3.E1.m2.2.2.2.2.2.2.2.1"></times><ci id="S3.E1.m2.2.2.2.2.2.2.2.2.cmml" xref="S3.E1.m2.2.2.2.2.2.2.2.2">𝑐</ci><apply id="S3.E1.m2.2.2.2.2.2.2.2.3.cmml" xref="S3.E1.m2.2.2.2.2.2.2.2.3"><csymbol cd="ambiguous" id="S3.E1.m2.2.2.2.2.2.2.2.3.1.cmml" xref="S3.E1.m2.2.2.2.2.2.2.2.3">subscript</csymbol><ci id="S3.E1.m2.2.2.2.2.2.2.2.3.2.cmml" xref="S3.E1.m2.2.2.2.2.2.2.2.3.2">𝑒</ci><ci id="S3.E1.m2.2.2.2.2.2.2.2.3.3.cmml" xref="S3.E1.m2.2.2.2.2.2.2.2.3.3">𝑗</ci></apply></apply></interval></apply><minus id="S3.E1.m2.2.2.2.4.cmml" xref="S3.E1.m2.2.2.2.4"></minus></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E1.m2.2c">\displaystyle\sum_{j}\sum_{m}|Sim(ce_{i},ce_{j})-</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
<tbody id="S3.Ex1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.Ex1.m1.1" class="ltx_math_unparsed" alttext="\displaystyle Sim(f(ce_{i})[:m],f(ce_{j})[:m])|," display="inline"><semantics id="S3.Ex1.m1.1a"><mrow id="S3.Ex1.m1.1b"><mi id="S3.Ex1.m1.1.1">S</mi><mi id="S3.Ex1.m1.1.2">i</mi><mi id="S3.Ex1.m1.1.3">m</mi><mrow id="S3.Ex1.m1.1.4"><mo stretchy="false" id="S3.Ex1.m1.1.4.1">(</mo><mi id="S3.Ex1.m1.1.4.2">f</mi><mrow id="S3.Ex1.m1.1.4.3"><mo stretchy="false" id="S3.Ex1.m1.1.4.3.1">(</mo><mi id="S3.Ex1.m1.1.4.3.2">c</mi><msub id="S3.Ex1.m1.1.4.3.3"><mi id="S3.Ex1.m1.1.4.3.3.2">e</mi><mi id="S3.Ex1.m1.1.4.3.3.3">i</mi></msub><mo stretchy="false" id="S3.Ex1.m1.1.4.3.4">)</mo></mrow><mrow id="S3.Ex1.m1.1.4.4"><mo stretchy="false" id="S3.Ex1.m1.1.4.4.1">[</mo><mo rspace="0.278em" id="S3.Ex1.m1.1.4.4.2">:</mo><mi id="S3.Ex1.m1.1.4.4.3">m</mi><mo stretchy="false" id="S3.Ex1.m1.1.4.4.4">]</mo></mrow><mo id="S3.Ex1.m1.1.4.5">,</mo><mi id="S3.Ex1.m1.1.4.6">f</mi><mrow id="S3.Ex1.m1.1.4.7"><mo stretchy="false" id="S3.Ex1.m1.1.4.7.1">(</mo><mi id="S3.Ex1.m1.1.4.7.2">c</mi><msub id="S3.Ex1.m1.1.4.7.3"><mi id="S3.Ex1.m1.1.4.7.3.2">e</mi><mi id="S3.Ex1.m1.1.4.7.3.3">j</mi></msub><mo stretchy="false" id="S3.Ex1.m1.1.4.7.4">)</mo></mrow><mrow id="S3.Ex1.m1.1.4.8"><mo stretchy="false" id="S3.Ex1.m1.1.4.8.1">[</mo><mo rspace="0.278em" id="S3.Ex1.m1.1.4.8.2">:</mo><mi id="S3.Ex1.m1.1.4.8.3">m</mi><mo stretchy="false" id="S3.Ex1.m1.1.4.8.4">]</mo></mrow><mo stretchy="false" id="S3.Ex1.m1.1.4.9">)</mo></mrow><mo fence="false" rspace="0.167em" stretchy="false" id="S3.Ex1.m1.1.5">|</mo><mo id="S3.Ex1.m1.1.6">,</mo></mrow><annotation encoding="application/x-tex" id="S3.Ex1.m1.1c">\displaystyle Sim(f(ce_{i})[:m],f(ce_{j})[:m])|,</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S3.SS2.p2.2" class="ltx_p">where <math id="S3.SS2.p2.2.m1.1" class="ltx_Math" alttext="Sim" display="inline"><semantics id="S3.SS2.p2.2.m1.1a"><mrow id="S3.SS2.p2.2.m1.1.1" xref="S3.SS2.p2.2.m1.1.1.cmml"><mi id="S3.SS2.p2.2.m1.1.1.2" xref="S3.SS2.p2.2.m1.1.1.2.cmml">S</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.2.m1.1.1.1" xref="S3.SS2.p2.2.m1.1.1.1.cmml">​</mo><mi id="S3.SS2.p2.2.m1.1.1.3" xref="S3.SS2.p2.2.m1.1.1.3.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p2.2.m1.1.1.1a" xref="S3.SS2.p2.2.m1.1.1.1.cmml">​</mo><mi id="S3.SS2.p2.2.m1.1.1.4" xref="S3.SS2.p2.2.m1.1.1.4.cmml">m</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p2.2.m1.1b"><apply id="S3.SS2.p2.2.m1.1.1.cmml" xref="S3.SS2.p2.2.m1.1.1"><times id="S3.SS2.p2.2.m1.1.1.1.cmml" xref="S3.SS2.p2.2.m1.1.1.1"></times><ci id="S3.SS2.p2.2.m1.1.1.2.cmml" xref="S3.SS2.p2.2.m1.1.1.2">𝑆</ci><ci id="S3.SS2.p2.2.m1.1.1.3.cmml" xref="S3.SS2.p2.2.m1.1.1.3">𝑖</ci><ci id="S3.SS2.p2.2.m1.1.1.4.cmml" xref="S3.SS2.p2.2.m1.1.1.4">𝑚</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p2.2.m1.1c">Sim</annotation></semantics></math> represents an arbitrary similarity function, which is chosen to be the cosine similarity.</p>
</div>
<div id="S3.SS2.p3" class="ltx_para">
<p id="S3.SS2.p3.1" class="ltx_p">The second loss function, denoted as <math id="S3.SS2.p3.1.m1.1" class="ltx_Math" alttext="\mathcal{L}_{topk}" display="inline"><semantics id="S3.SS2.p3.1.m1.1a"><msub id="S3.SS2.p3.1.m1.1.1" xref="S3.SS2.p3.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p3.1.m1.1.1.2" xref="S3.SS2.p3.1.m1.1.1.2.cmml">ℒ</mi><mrow id="S3.SS2.p3.1.m1.1.1.3" xref="S3.SS2.p3.1.m1.1.1.3.cmml"><mi id="S3.SS2.p3.1.m1.1.1.3.2" xref="S3.SS2.p3.1.m1.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p3.1.m1.1.1.3.1" xref="S3.SS2.p3.1.m1.1.1.3.1.cmml">​</mo><mi id="S3.SS2.p3.1.m1.1.1.3.3" xref="S3.SS2.p3.1.m1.1.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p3.1.m1.1.1.3.1a" xref="S3.SS2.p3.1.m1.1.1.3.1.cmml">​</mo><mi id="S3.SS2.p3.1.m1.1.1.3.4" xref="S3.SS2.p3.1.m1.1.1.3.4.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p3.1.m1.1.1.3.1b" xref="S3.SS2.p3.1.m1.1.1.3.1.cmml">​</mo><mi id="S3.SS2.p3.1.m1.1.1.3.5" xref="S3.SS2.p3.1.m1.1.1.3.5.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.1.m1.1b"><apply id="S3.SS2.p3.1.m1.1.1.cmml" xref="S3.SS2.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS2.p3.1.m1.1.1.1.cmml" xref="S3.SS2.p3.1.m1.1.1">subscript</csymbol><ci id="S3.SS2.p3.1.m1.1.1.2.cmml" xref="S3.SS2.p3.1.m1.1.1.2">ℒ</ci><apply id="S3.SS2.p3.1.m1.1.1.3.cmml" xref="S3.SS2.p3.1.m1.1.1.3"><times id="S3.SS2.p3.1.m1.1.1.3.1.cmml" xref="S3.SS2.p3.1.m1.1.1.3.1"></times><ci id="S3.SS2.p3.1.m1.1.1.3.2.cmml" xref="S3.SS2.p3.1.m1.1.1.3.2">𝑡</ci><ci id="S3.SS2.p3.1.m1.1.1.3.3.cmml" xref="S3.SS2.p3.1.m1.1.1.3.3">𝑜</ci><ci id="S3.SS2.p3.1.m1.1.1.3.4.cmml" xref="S3.SS2.p3.1.m1.1.1.3.4">𝑝</ci><ci id="S3.SS2.p3.1.m1.1.1.3.5.cmml" xref="S3.SS2.p3.1.m1.1.1.3.5">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.1.m1.1c">\mathcal{L}_{topk}</annotation></semantics></math>, focuses on preserving local similarity relationships among neighboring embeddings:</p>
<table id="A6.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E2.m1.1" class="ltx_Math" alttext="\displaystyle\mathcal{L}_{topk}=\sum_{i}" display="inline"><semantics id="S3.E2.m1.1a"><mrow id="S3.E2.m1.1.1" xref="S3.E2.m1.1.1.cmml"><msub id="S3.E2.m1.1.1.2" xref="S3.E2.m1.1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E2.m1.1.1.2.2" xref="S3.E2.m1.1.1.2.2.cmml">ℒ</mi><mrow id="S3.E2.m1.1.1.2.3" xref="S3.E2.m1.1.1.2.3.cmml"><mi id="S3.E2.m1.1.1.2.3.2" xref="S3.E2.m1.1.1.2.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S3.E2.m1.1.1.2.3.1" xref="S3.E2.m1.1.1.2.3.1.cmml">​</mo><mi id="S3.E2.m1.1.1.2.3.3" xref="S3.E2.m1.1.1.2.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S3.E2.m1.1.1.2.3.1a" xref="S3.E2.m1.1.1.2.3.1.cmml">​</mo><mi id="S3.E2.m1.1.1.2.3.4" xref="S3.E2.m1.1.1.2.3.4.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.E2.m1.1.1.2.3.1b" xref="S3.E2.m1.1.1.2.3.1.cmml">​</mo><mi id="S3.E2.m1.1.1.2.3.5" xref="S3.E2.m1.1.1.2.3.5.cmml">k</mi></mrow></msub><mo id="S3.E2.m1.1.1.1" xref="S3.E2.m1.1.1.1.cmml">=</mo><mstyle displaystyle="true" id="S3.E2.m1.1.1.3" xref="S3.E2.m1.1.1.3.cmml"><munder id="S3.E2.m1.1.1.3a" xref="S3.E2.m1.1.1.3.cmml"><mo movablelimits="false" id="S3.E2.m1.1.1.3.2" xref="S3.E2.m1.1.1.3.2.cmml">∑</mo><mi id="S3.E2.m1.1.1.3.3" xref="S3.E2.m1.1.1.3.3.cmml">i</mi></munder></mstyle></mrow><annotation-xml encoding="MathML-Content" id="S3.E2.m1.1b"><apply id="S3.E2.m1.1.1.cmml" xref="S3.E2.m1.1.1"><eq id="S3.E2.m1.1.1.1.cmml" xref="S3.E2.m1.1.1.1"></eq><apply id="S3.E2.m1.1.1.2.cmml" xref="S3.E2.m1.1.1.2"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.2.1.cmml" xref="S3.E2.m1.1.1.2">subscript</csymbol><ci id="S3.E2.m1.1.1.2.2.cmml" xref="S3.E2.m1.1.1.2.2">ℒ</ci><apply id="S3.E2.m1.1.1.2.3.cmml" xref="S3.E2.m1.1.1.2.3"><times id="S3.E2.m1.1.1.2.3.1.cmml" xref="S3.E2.m1.1.1.2.3.1"></times><ci id="S3.E2.m1.1.1.2.3.2.cmml" xref="S3.E2.m1.1.1.2.3.2">𝑡</ci><ci id="S3.E2.m1.1.1.2.3.3.cmml" xref="S3.E2.m1.1.1.2.3.3">𝑜</ci><ci id="S3.E2.m1.1.1.2.3.4.cmml" xref="S3.E2.m1.1.1.2.3.4">𝑝</ci><ci id="S3.E2.m1.1.1.2.3.5.cmml" xref="S3.E2.m1.1.1.2.3.5">𝑘</ci></apply></apply><apply id="S3.E2.m1.1.1.3.cmml" xref="S3.E2.m1.1.1.3"><csymbol cd="ambiguous" id="S3.E2.m1.1.1.3.1.cmml" xref="S3.E2.m1.1.1.3">subscript</csymbol><sum id="S3.E2.m1.1.1.3.2.cmml" xref="S3.E2.m1.1.1.3.2"></sum><ci id="S3.E2.m1.1.1.3.3.cmml" xref="S3.E2.m1.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E2.m1.1c">\displaystyle\mathcal{L}_{topk}=\sum_{i}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.E2.m2.3" class="ltx_Math" alttext="\displaystyle\sum_{j\in NN_{k}(i)}\sum_{m}|Sim(ce_{i},ce_{j})-" display="inline"><semantics id="S3.E2.m2.3a"><mrow id="S3.E2.m2.3.3" xref="S3.E2.m2.3.3.cmml"><mrow id="S3.E2.m2.3.3.4" xref="S3.E2.m2.3.3.4.cmml"><mstyle displaystyle="true" id="S3.E2.m2.3.3.4.1" xref="S3.E2.m2.3.3.4.1.cmml"><munder id="S3.E2.m2.3.3.4.1a" xref="S3.E2.m2.3.3.4.1.cmml"><mo movablelimits="false" id="S3.E2.m2.3.3.4.1.2" xref="S3.E2.m2.3.3.4.1.2.cmml">∑</mo><mrow id="S3.E2.m2.1.1.1" xref="S3.E2.m2.1.1.1.cmml"><mi id="S3.E2.m2.1.1.1.3" xref="S3.E2.m2.1.1.1.3.cmml">j</mi><mo id="S3.E2.m2.1.1.1.2" xref="S3.E2.m2.1.1.1.2.cmml">∈</mo><mrow id="S3.E2.m2.1.1.1.4" xref="S3.E2.m2.1.1.1.4.cmml"><mi id="S3.E2.m2.1.1.1.4.2" xref="S3.E2.m2.1.1.1.4.2.cmml">N</mi><mo lspace="0em" rspace="0em" id="S3.E2.m2.1.1.1.4.1" xref="S3.E2.m2.1.1.1.4.1.cmml">​</mo><msub id="S3.E2.m2.1.1.1.4.3" xref="S3.E2.m2.1.1.1.4.3.cmml"><mi id="S3.E2.m2.1.1.1.4.3.2" xref="S3.E2.m2.1.1.1.4.3.2.cmml">N</mi><mi id="S3.E2.m2.1.1.1.4.3.3" xref="S3.E2.m2.1.1.1.4.3.3.cmml">k</mi></msub><mo lspace="0em" rspace="0em" id="S3.E2.m2.1.1.1.4.1a" xref="S3.E2.m2.1.1.1.4.1.cmml">​</mo><mrow id="S3.E2.m2.1.1.1.4.4.2" xref="S3.E2.m2.1.1.1.4.cmml"><mo stretchy="false" id="S3.E2.m2.1.1.1.4.4.2.1" xref="S3.E2.m2.1.1.1.4.cmml">(</mo><mi id="S3.E2.m2.1.1.1.1" xref="S3.E2.m2.1.1.1.1.cmml">i</mi><mo stretchy="false" id="S3.E2.m2.1.1.1.4.4.2.2" xref="S3.E2.m2.1.1.1.4.cmml">)</mo></mrow></mrow></mrow></munder></mstyle><mstyle displaystyle="true" id="S3.E2.m2.3.3.4.2" xref="S3.E2.m2.3.3.4.2.cmml"><munder id="S3.E2.m2.3.3.4.2a" xref="S3.E2.m2.3.3.4.2.cmml"><mo movablelimits="false" id="S3.E2.m2.3.3.4.2.2" xref="S3.E2.m2.3.3.4.2.2.cmml">∑</mo><mi id="S3.E2.m2.3.3.4.2.3" xref="S3.E2.m2.3.3.4.2.3.cmml">m</mi></munder></mstyle></mrow><mo fence="false" id="S3.E2.m2.3.3.3" xref="S3.E2.m2.3.3.3.cmml">|</mo><mrow id="S3.E2.m2.3.3.2" xref="S3.E2.m2.3.3.2.cmml"><mrow id="S3.E2.m2.3.3.2.2" xref="S3.E2.m2.3.3.2.2.cmml"><mi id="S3.E2.m2.3.3.2.2.4" xref="S3.E2.m2.3.3.2.2.4.cmml">S</mi><mo lspace="0em" rspace="0em" id="S3.E2.m2.3.3.2.2.3" xref="S3.E2.m2.3.3.2.2.3.cmml">​</mo><mi id="S3.E2.m2.3.3.2.2.5" xref="S3.E2.m2.3.3.2.2.5.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.E2.m2.3.3.2.2.3a" xref="S3.E2.m2.3.3.2.2.3.cmml">​</mo><mi id="S3.E2.m2.3.3.2.2.6" xref="S3.E2.m2.3.3.2.2.6.cmml">m</mi><mo lspace="0em" rspace="0em" id="S3.E2.m2.3.3.2.2.3b" xref="S3.E2.m2.3.3.2.2.3.cmml">​</mo><mrow id="S3.E2.m2.3.3.2.2.2.2" xref="S3.E2.m2.3.3.2.2.2.3.cmml"><mo stretchy="false" id="S3.E2.m2.3.3.2.2.2.2.3" xref="S3.E2.m2.3.3.2.2.2.3.cmml">(</mo><mrow id="S3.E2.m2.2.2.1.1.1.1.1" xref="S3.E2.m2.2.2.1.1.1.1.1.cmml"><mi id="S3.E2.m2.2.2.1.1.1.1.1.2" xref="S3.E2.m2.2.2.1.1.1.1.1.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.E2.m2.2.2.1.1.1.1.1.1" xref="S3.E2.m2.2.2.1.1.1.1.1.1.cmml">​</mo><msub id="S3.E2.m2.2.2.1.1.1.1.1.3" xref="S3.E2.m2.2.2.1.1.1.1.1.3.cmml"><mi id="S3.E2.m2.2.2.1.1.1.1.1.3.2" xref="S3.E2.m2.2.2.1.1.1.1.1.3.2.cmml">e</mi><mi id="S3.E2.m2.2.2.1.1.1.1.1.3.3" xref="S3.E2.m2.2.2.1.1.1.1.1.3.3.cmml">i</mi></msub></mrow><mo id="S3.E2.m2.3.3.2.2.2.2.4" xref="S3.E2.m2.3.3.2.2.2.3.cmml">,</mo><mrow id="S3.E2.m2.3.3.2.2.2.2.2" xref="S3.E2.m2.3.3.2.2.2.2.2.cmml"><mi id="S3.E2.m2.3.3.2.2.2.2.2.2" xref="S3.E2.m2.3.3.2.2.2.2.2.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.E2.m2.3.3.2.2.2.2.2.1" xref="S3.E2.m2.3.3.2.2.2.2.2.1.cmml">​</mo><msub id="S3.E2.m2.3.3.2.2.2.2.2.3" xref="S3.E2.m2.3.3.2.2.2.2.2.3.cmml"><mi id="S3.E2.m2.3.3.2.2.2.2.2.3.2" xref="S3.E2.m2.3.3.2.2.2.2.2.3.2.cmml">e</mi><mi id="S3.E2.m2.3.3.2.2.2.2.2.3.3" xref="S3.E2.m2.3.3.2.2.2.2.2.3.3.cmml">j</mi></msub></mrow><mo stretchy="false" id="S3.E2.m2.3.3.2.2.2.2.5" xref="S3.E2.m2.3.3.2.2.2.3.cmml">)</mo></mrow></mrow><mo id="S3.E2.m2.3.3.2.4" xref="S3.E2.m2.3.3.2.4.cmml">−</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.E2.m2.3b"><apply id="S3.E2.m2.3.3.cmml" xref="S3.E2.m2.3.3"><csymbol cd="latexml" id="S3.E2.m2.3.3.3.cmml" xref="S3.E2.m2.3.3.3">conditional</csymbol><apply id="S3.E2.m2.3.3.4.cmml" xref="S3.E2.m2.3.3.4"><apply id="S3.E2.m2.3.3.4.1.cmml" xref="S3.E2.m2.3.3.4.1"><csymbol cd="ambiguous" id="S3.E2.m2.3.3.4.1.1.cmml" xref="S3.E2.m2.3.3.4.1">subscript</csymbol><sum id="S3.E2.m2.3.3.4.1.2.cmml" xref="S3.E2.m2.3.3.4.1.2"></sum><apply id="S3.E2.m2.1.1.1.cmml" xref="S3.E2.m2.1.1.1"><in id="S3.E2.m2.1.1.1.2.cmml" xref="S3.E2.m2.1.1.1.2"></in><ci id="S3.E2.m2.1.1.1.3.cmml" xref="S3.E2.m2.1.1.1.3">𝑗</ci><apply id="S3.E2.m2.1.1.1.4.cmml" xref="S3.E2.m2.1.1.1.4"><times id="S3.E2.m2.1.1.1.4.1.cmml" xref="S3.E2.m2.1.1.1.4.1"></times><ci id="S3.E2.m2.1.1.1.4.2.cmml" xref="S3.E2.m2.1.1.1.4.2">𝑁</ci><apply id="S3.E2.m2.1.1.1.4.3.cmml" xref="S3.E2.m2.1.1.1.4.3"><csymbol cd="ambiguous" id="S3.E2.m2.1.1.1.4.3.1.cmml" xref="S3.E2.m2.1.1.1.4.3">subscript</csymbol><ci id="S3.E2.m2.1.1.1.4.3.2.cmml" xref="S3.E2.m2.1.1.1.4.3.2">𝑁</ci><ci id="S3.E2.m2.1.1.1.4.3.3.cmml" xref="S3.E2.m2.1.1.1.4.3.3">𝑘</ci></apply><ci id="S3.E2.m2.1.1.1.1.cmml" xref="S3.E2.m2.1.1.1.1">𝑖</ci></apply></apply></apply><apply id="S3.E2.m2.3.3.4.2.cmml" xref="S3.E2.m2.3.3.4.2"><csymbol cd="ambiguous" id="S3.E2.m2.3.3.4.2.1.cmml" xref="S3.E2.m2.3.3.4.2">subscript</csymbol><sum id="S3.E2.m2.3.3.4.2.2.cmml" xref="S3.E2.m2.3.3.4.2.2"></sum><ci id="S3.E2.m2.3.3.4.2.3.cmml" xref="S3.E2.m2.3.3.4.2.3">𝑚</ci></apply></apply><apply id="S3.E2.m2.3.3.2.cmml" xref="S3.E2.m2.3.3.2"><csymbol cd="latexml" id="S3.E2.m2.3.3.2.3.cmml" xref="S3.E2.m2.3.3.2">limit-from</csymbol><apply id="S3.E2.m2.3.3.2.2.cmml" xref="S3.E2.m2.3.3.2.2"><times id="S3.E2.m2.3.3.2.2.3.cmml" xref="S3.E2.m2.3.3.2.2.3"></times><ci id="S3.E2.m2.3.3.2.2.4.cmml" xref="S3.E2.m2.3.3.2.2.4">𝑆</ci><ci id="S3.E2.m2.3.3.2.2.5.cmml" xref="S3.E2.m2.3.3.2.2.5">𝑖</ci><ci id="S3.E2.m2.3.3.2.2.6.cmml" xref="S3.E2.m2.3.3.2.2.6">𝑚</ci><interval closure="open" id="S3.E2.m2.3.3.2.2.2.3.cmml" xref="S3.E2.m2.3.3.2.2.2.2"><apply id="S3.E2.m2.2.2.1.1.1.1.1.cmml" xref="S3.E2.m2.2.2.1.1.1.1.1"><times id="S3.E2.m2.2.2.1.1.1.1.1.1.cmml" xref="S3.E2.m2.2.2.1.1.1.1.1.1"></times><ci id="S3.E2.m2.2.2.1.1.1.1.1.2.cmml" xref="S3.E2.m2.2.2.1.1.1.1.1.2">𝑐</ci><apply id="S3.E2.m2.2.2.1.1.1.1.1.3.cmml" xref="S3.E2.m2.2.2.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E2.m2.2.2.1.1.1.1.1.3.1.cmml" xref="S3.E2.m2.2.2.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E2.m2.2.2.1.1.1.1.1.3.2.cmml" xref="S3.E2.m2.2.2.1.1.1.1.1.3.2">𝑒</ci><ci id="S3.E2.m2.2.2.1.1.1.1.1.3.3.cmml" xref="S3.E2.m2.2.2.1.1.1.1.1.3.3">𝑖</ci></apply></apply><apply id="S3.E2.m2.3.3.2.2.2.2.2.cmml" xref="S3.E2.m2.3.3.2.2.2.2.2"><times id="S3.E2.m2.3.3.2.2.2.2.2.1.cmml" xref="S3.E2.m2.3.3.2.2.2.2.2.1"></times><ci id="S3.E2.m2.3.3.2.2.2.2.2.2.cmml" xref="S3.E2.m2.3.3.2.2.2.2.2.2">𝑐</ci><apply id="S3.E2.m2.3.3.2.2.2.2.2.3.cmml" xref="S3.E2.m2.3.3.2.2.2.2.2.3"><csymbol cd="ambiguous" id="S3.E2.m2.3.3.2.2.2.2.2.3.1.cmml" xref="S3.E2.m2.3.3.2.2.2.2.2.3">subscript</csymbol><ci id="S3.E2.m2.3.3.2.2.2.2.2.3.2.cmml" xref="S3.E2.m2.3.3.2.2.2.2.2.3.2">𝑒</ci><ci id="S3.E2.m2.3.3.2.2.2.2.2.3.3.cmml" xref="S3.E2.m2.3.3.2.2.2.2.2.3.3">𝑗</ci></apply></apply></interval></apply><minus id="S3.E2.m2.3.3.2.4.cmml" xref="S3.E2.m2.3.3.2.4"></minus></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E2.m2.3c">\displaystyle\sum_{j\in NN_{k}(i)}\sum_{m}|Sim(ce_{i},ce_{j})-</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
<tbody id="S3.Ex2"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S3.Ex2.m1.1" class="ltx_math_unparsed" alttext="\displaystyle Sim(f(ce_{i})[:m],f(ce_{j})[:m])|," display="inline"><semantics id="S3.Ex2.m1.1a"><mrow id="S3.Ex2.m1.1b"><mi id="S3.Ex2.m1.1.1">S</mi><mi id="S3.Ex2.m1.1.2">i</mi><mi id="S3.Ex2.m1.1.3">m</mi><mrow id="S3.Ex2.m1.1.4"><mo stretchy="false" id="S3.Ex2.m1.1.4.1">(</mo><mi id="S3.Ex2.m1.1.4.2">f</mi><mrow id="S3.Ex2.m1.1.4.3"><mo stretchy="false" id="S3.Ex2.m1.1.4.3.1">(</mo><mi id="S3.Ex2.m1.1.4.3.2">c</mi><msub id="S3.Ex2.m1.1.4.3.3"><mi id="S3.Ex2.m1.1.4.3.3.2">e</mi><mi id="S3.Ex2.m1.1.4.3.3.3">i</mi></msub><mo stretchy="false" id="S3.Ex2.m1.1.4.3.4">)</mo></mrow><mrow id="S3.Ex2.m1.1.4.4"><mo stretchy="false" id="S3.Ex2.m1.1.4.4.1">[</mo><mo rspace="0.278em" id="S3.Ex2.m1.1.4.4.2">:</mo><mi id="S3.Ex2.m1.1.4.4.3">m</mi><mo stretchy="false" id="S3.Ex2.m1.1.4.4.4">]</mo></mrow><mo id="S3.Ex2.m1.1.4.5">,</mo><mi id="S3.Ex2.m1.1.4.6">f</mi><mrow id="S3.Ex2.m1.1.4.7"><mo stretchy="false" id="S3.Ex2.m1.1.4.7.1">(</mo><mi id="S3.Ex2.m1.1.4.7.2">c</mi><msub id="S3.Ex2.m1.1.4.7.3"><mi id="S3.Ex2.m1.1.4.7.3.2">e</mi><mi id="S3.Ex2.m1.1.4.7.3.3">j</mi></msub><mo stretchy="false" id="S3.Ex2.m1.1.4.7.4">)</mo></mrow><mrow id="S3.Ex2.m1.1.4.8"><mo stretchy="false" id="S3.Ex2.m1.1.4.8.1">[</mo><mo rspace="0.278em" id="S3.Ex2.m1.1.4.8.2">:</mo><mi id="S3.Ex2.m1.1.4.8.3">m</mi><mo stretchy="false" id="S3.Ex2.m1.1.4.8.4">]</mo></mrow><mo stretchy="false" id="S3.Ex2.m1.1.4.9">)</mo></mrow><mo fence="false" rspace="0.167em" stretchy="false" id="S3.Ex2.m1.1.5">|</mo><mo id="S3.Ex2.m1.1.6">,</mo></mrow><annotation encoding="application/x-tex" id="S3.Ex2.m1.1c">\displaystyle Sim(f(ce_{i})[:m],f(ce_{j})[:m])|,</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S3.SS2.p3.3" class="ltx_p">where <math id="S3.SS2.p3.2.m1.1" class="ltx_Math" alttext="NN_{k}(i)" display="inline"><semantics id="S3.SS2.p3.2.m1.1a"><mrow id="S3.SS2.p3.2.m1.1.2" xref="S3.SS2.p3.2.m1.1.2.cmml"><mi id="S3.SS2.p3.2.m1.1.2.2" xref="S3.SS2.p3.2.m1.1.2.2.cmml">N</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p3.2.m1.1.2.1" xref="S3.SS2.p3.2.m1.1.2.1.cmml">​</mo><msub id="S3.SS2.p3.2.m1.1.2.3" xref="S3.SS2.p3.2.m1.1.2.3.cmml"><mi id="S3.SS2.p3.2.m1.1.2.3.2" xref="S3.SS2.p3.2.m1.1.2.3.2.cmml">N</mi><mi id="S3.SS2.p3.2.m1.1.2.3.3" xref="S3.SS2.p3.2.m1.1.2.3.3.cmml">k</mi></msub><mo lspace="0em" rspace="0em" id="S3.SS2.p3.2.m1.1.2.1a" xref="S3.SS2.p3.2.m1.1.2.1.cmml">​</mo><mrow id="S3.SS2.p3.2.m1.1.2.4.2" xref="S3.SS2.p3.2.m1.1.2.cmml"><mo stretchy="false" id="S3.SS2.p3.2.m1.1.2.4.2.1" xref="S3.SS2.p3.2.m1.1.2.cmml">(</mo><mi id="S3.SS2.p3.2.m1.1.1" xref="S3.SS2.p3.2.m1.1.1.cmml">i</mi><mo stretchy="false" id="S3.SS2.p3.2.m1.1.2.4.2.2" xref="S3.SS2.p3.2.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.2.m1.1b"><apply id="S3.SS2.p3.2.m1.1.2.cmml" xref="S3.SS2.p3.2.m1.1.2"><times id="S3.SS2.p3.2.m1.1.2.1.cmml" xref="S3.SS2.p3.2.m1.1.2.1"></times><ci id="S3.SS2.p3.2.m1.1.2.2.cmml" xref="S3.SS2.p3.2.m1.1.2.2">𝑁</ci><apply id="S3.SS2.p3.2.m1.1.2.3.cmml" xref="S3.SS2.p3.2.m1.1.2.3"><csymbol cd="ambiguous" id="S3.SS2.p3.2.m1.1.2.3.1.cmml" xref="S3.SS2.p3.2.m1.1.2.3">subscript</csymbol><ci id="S3.SS2.p3.2.m1.1.2.3.2.cmml" xref="S3.SS2.p3.2.m1.1.2.3.2">𝑁</ci><ci id="S3.SS2.p3.2.m1.1.2.3.3.cmml" xref="S3.SS2.p3.2.m1.1.2.3.3">𝑘</ci></apply><ci id="S3.SS2.p3.2.m1.1.1.cmml" xref="S3.SS2.p3.2.m1.1.1">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.2.m1.1c">NN_{k}(i)</annotation></semantics></math> denotes the set of the top k most similar embeddings to <math id="S3.SS2.p3.3.m2.1" class="ltx_Math" alttext="ce_{i}" display="inline"><semantics id="S3.SS2.p3.3.m2.1a"><mrow id="S3.SS2.p3.3.m2.1.1" xref="S3.SS2.p3.3.m2.1.1.cmml"><mi id="S3.SS2.p3.3.m2.1.1.2" xref="S3.SS2.p3.3.m2.1.1.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p3.3.m2.1.1.1" xref="S3.SS2.p3.3.m2.1.1.1.cmml">​</mo><msub id="S3.SS2.p3.3.m2.1.1.3" xref="S3.SS2.p3.3.m2.1.1.3.cmml"><mi id="S3.SS2.p3.3.m2.1.1.3.2" xref="S3.SS2.p3.3.m2.1.1.3.2.cmml">e</mi><mi id="S3.SS2.p3.3.m2.1.1.3.3" xref="S3.SS2.p3.3.m2.1.1.3.3.cmml">i</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p3.3.m2.1b"><apply id="S3.SS2.p3.3.m2.1.1.cmml" xref="S3.SS2.p3.3.m2.1.1"><times id="S3.SS2.p3.3.m2.1.1.1.cmml" xref="S3.SS2.p3.3.m2.1.1.1"></times><ci id="S3.SS2.p3.3.m2.1.1.2.cmml" xref="S3.SS2.p3.3.m2.1.1.2">𝑐</ci><apply id="S3.SS2.p3.3.m2.1.1.3.cmml" xref="S3.SS2.p3.3.m2.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p3.3.m2.1.1.3.1.cmml" xref="S3.SS2.p3.3.m2.1.1.3">subscript</csymbol><ci id="S3.SS2.p3.3.m2.1.1.3.2.cmml" xref="S3.SS2.p3.3.m2.1.1.3.2">𝑒</ci><ci id="S3.SS2.p3.3.m2.1.1.3.3.cmml" xref="S3.SS2.p3.3.m2.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p3.3.m2.1c">ce_{i}</annotation></semantics></math>.
A visual representation of these two loss functions, illustrating their application across multiple Matryoshka embeddings of varying dimensions, is provided in Fig. <a href="#S3.F2" title="Figure 2 ‣ 3.1 Problem formulation ‣ 3 Unsupervised Matryoshka-Adaptor ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
<div id="S3.SS2.p4" class="ltx_para">
<p id="S3.SS2.p4.3" class="ltx_p">In order to mitigate any substantial deviation from the original embeddings, we have integrated regularizations into our methodology.
Primarily, a skip connection is implemented within the architecture of the learnable function, <math id="S3.SS2.p4.1.m1.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S3.SS2.p4.1.m1.1a"><mi id="S3.SS2.p4.1.m1.1.1" xref="S3.SS2.p4.1.m1.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.1.m1.1b"><ci id="S3.SS2.p4.1.m1.1.1.cmml" xref="S3.SS2.p4.1.m1.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.1.m1.1c">f</annotation></semantics></math>, ensuring that this function learns solely the difference from the original embedding, represented as <math id="S3.SS2.p4.2.m2.1" class="ltx_Math" alttext="\hat{ce}_{i}=ce_{i}+f(ce_{i})" display="inline"><semantics id="S3.SS2.p4.2.m2.1a"><mrow id="S3.SS2.p4.2.m2.1.1" xref="S3.SS2.p4.2.m2.1.1.cmml"><msub id="S3.SS2.p4.2.m2.1.1.3" xref="S3.SS2.p4.2.m2.1.1.3.cmml"><mover accent="true" id="S3.SS2.p4.2.m2.1.1.3.2" xref="S3.SS2.p4.2.m2.1.1.3.2.cmml"><mrow id="S3.SS2.p4.2.m2.1.1.3.2.2" xref="S3.SS2.p4.2.m2.1.1.3.2.2.cmml"><mi id="S3.SS2.p4.2.m2.1.1.3.2.2.2" xref="S3.SS2.p4.2.m2.1.1.3.2.2.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p4.2.m2.1.1.3.2.2.1" xref="S3.SS2.p4.2.m2.1.1.3.2.2.1.cmml">​</mo><mi id="S3.SS2.p4.2.m2.1.1.3.2.2.3" xref="S3.SS2.p4.2.m2.1.1.3.2.2.3.cmml">e</mi></mrow><mo id="S3.SS2.p4.2.m2.1.1.3.2.1" xref="S3.SS2.p4.2.m2.1.1.3.2.1.cmml">^</mo></mover><mi id="S3.SS2.p4.2.m2.1.1.3.3" xref="S3.SS2.p4.2.m2.1.1.3.3.cmml">i</mi></msub><mo id="S3.SS2.p4.2.m2.1.1.2" xref="S3.SS2.p4.2.m2.1.1.2.cmml">=</mo><mrow id="S3.SS2.p4.2.m2.1.1.1" xref="S3.SS2.p4.2.m2.1.1.1.cmml"><mrow id="S3.SS2.p4.2.m2.1.1.1.3" xref="S3.SS2.p4.2.m2.1.1.1.3.cmml"><mi id="S3.SS2.p4.2.m2.1.1.1.3.2" xref="S3.SS2.p4.2.m2.1.1.1.3.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p4.2.m2.1.1.1.3.1" xref="S3.SS2.p4.2.m2.1.1.1.3.1.cmml">​</mo><msub id="S3.SS2.p4.2.m2.1.1.1.3.3" xref="S3.SS2.p4.2.m2.1.1.1.3.3.cmml"><mi id="S3.SS2.p4.2.m2.1.1.1.3.3.2" xref="S3.SS2.p4.2.m2.1.1.1.3.3.2.cmml">e</mi><mi id="S3.SS2.p4.2.m2.1.1.1.3.3.3" xref="S3.SS2.p4.2.m2.1.1.1.3.3.3.cmml">i</mi></msub></mrow><mo id="S3.SS2.p4.2.m2.1.1.1.2" xref="S3.SS2.p4.2.m2.1.1.1.2.cmml">+</mo><mrow id="S3.SS2.p4.2.m2.1.1.1.1" xref="S3.SS2.p4.2.m2.1.1.1.1.cmml"><mi id="S3.SS2.p4.2.m2.1.1.1.1.3" xref="S3.SS2.p4.2.m2.1.1.1.1.3.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p4.2.m2.1.1.1.1.2" xref="S3.SS2.p4.2.m2.1.1.1.1.2.cmml">​</mo><mrow id="S3.SS2.p4.2.m2.1.1.1.1.1.1" xref="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS2.p4.2.m2.1.1.1.1.1.1.2" xref="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.SS2.p4.2.m2.1.1.1.1.1.1.1" xref="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.cmml"><mi id="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.2" xref="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.1" xref="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.1.cmml">​</mo><msub id="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.3" xref="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.3.cmml"><mi id="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.3.2" xref="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.3.2.cmml">e</mi><mi id="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.3.3" xref="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.3.3.cmml">i</mi></msub></mrow><mo stretchy="false" id="S3.SS2.p4.2.m2.1.1.1.1.1.1.3" xref="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.2.m2.1b"><apply id="S3.SS2.p4.2.m2.1.1.cmml" xref="S3.SS2.p4.2.m2.1.1"><eq id="S3.SS2.p4.2.m2.1.1.2.cmml" xref="S3.SS2.p4.2.m2.1.1.2"></eq><apply id="S3.SS2.p4.2.m2.1.1.3.cmml" xref="S3.SS2.p4.2.m2.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p4.2.m2.1.1.3.1.cmml" xref="S3.SS2.p4.2.m2.1.1.3">subscript</csymbol><apply id="S3.SS2.p4.2.m2.1.1.3.2.cmml" xref="S3.SS2.p4.2.m2.1.1.3.2"><ci id="S3.SS2.p4.2.m2.1.1.3.2.1.cmml" xref="S3.SS2.p4.2.m2.1.1.3.2.1">^</ci><apply id="S3.SS2.p4.2.m2.1.1.3.2.2.cmml" xref="S3.SS2.p4.2.m2.1.1.3.2.2"><times id="S3.SS2.p4.2.m2.1.1.3.2.2.1.cmml" xref="S3.SS2.p4.2.m2.1.1.3.2.2.1"></times><ci id="S3.SS2.p4.2.m2.1.1.3.2.2.2.cmml" xref="S3.SS2.p4.2.m2.1.1.3.2.2.2">𝑐</ci><ci id="S3.SS2.p4.2.m2.1.1.3.2.2.3.cmml" xref="S3.SS2.p4.2.m2.1.1.3.2.2.3">𝑒</ci></apply></apply><ci id="S3.SS2.p4.2.m2.1.1.3.3.cmml" xref="S3.SS2.p4.2.m2.1.1.3.3">𝑖</ci></apply><apply id="S3.SS2.p4.2.m2.1.1.1.cmml" xref="S3.SS2.p4.2.m2.1.1.1"><plus id="S3.SS2.p4.2.m2.1.1.1.2.cmml" xref="S3.SS2.p4.2.m2.1.1.1.2"></plus><apply id="S3.SS2.p4.2.m2.1.1.1.3.cmml" xref="S3.SS2.p4.2.m2.1.1.1.3"><times id="S3.SS2.p4.2.m2.1.1.1.3.1.cmml" xref="S3.SS2.p4.2.m2.1.1.1.3.1"></times><ci id="S3.SS2.p4.2.m2.1.1.1.3.2.cmml" xref="S3.SS2.p4.2.m2.1.1.1.3.2">𝑐</ci><apply id="S3.SS2.p4.2.m2.1.1.1.3.3.cmml" xref="S3.SS2.p4.2.m2.1.1.1.3.3"><csymbol cd="ambiguous" id="S3.SS2.p4.2.m2.1.1.1.3.3.1.cmml" xref="S3.SS2.p4.2.m2.1.1.1.3.3">subscript</csymbol><ci id="S3.SS2.p4.2.m2.1.1.1.3.3.2.cmml" xref="S3.SS2.p4.2.m2.1.1.1.3.3.2">𝑒</ci><ci id="S3.SS2.p4.2.m2.1.1.1.3.3.3.cmml" xref="S3.SS2.p4.2.m2.1.1.1.3.3.3">𝑖</ci></apply></apply><apply id="S3.SS2.p4.2.m2.1.1.1.1.cmml" xref="S3.SS2.p4.2.m2.1.1.1.1"><times id="S3.SS2.p4.2.m2.1.1.1.1.2.cmml" xref="S3.SS2.p4.2.m2.1.1.1.1.2"></times><ci id="S3.SS2.p4.2.m2.1.1.1.1.3.cmml" xref="S3.SS2.p4.2.m2.1.1.1.1.3">𝑓</ci><apply id="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p4.2.m2.1.1.1.1.1.1"><times id="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.1.cmml" xref="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.1"></times><ci id="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.2.cmml" xref="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.2">𝑐</ci><apply id="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.3.cmml" xref="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.3.1.cmml" xref="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.3.2.cmml" xref="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.3.2">𝑒</ci><ci id="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.3.3.cmml" xref="S3.SS2.p4.2.m2.1.1.1.1.1.1.1.3.3">𝑖</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.2.m2.1c">\hat{ce}_{i}=ce_{i}+f(ce_{i})</annotation></semantics></math>.
Furthermore, a reconstruction loss, denoted as <math id="S3.SS2.p4.3.m3.1" class="ltx_Math" alttext="\mathcal{L}_{rec}" display="inline"><semantics id="S3.SS2.p4.3.m3.1a"><msub id="S3.SS2.p4.3.m3.1.1" xref="S3.SS2.p4.3.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS2.p4.3.m3.1.1.2" xref="S3.SS2.p4.3.m3.1.1.2.cmml">ℒ</mi><mrow id="S3.SS2.p4.3.m3.1.1.3" xref="S3.SS2.p4.3.m3.1.1.3.cmml"><mi id="S3.SS2.p4.3.m3.1.1.3.2" xref="S3.SS2.p4.3.m3.1.1.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p4.3.m3.1.1.3.1" xref="S3.SS2.p4.3.m3.1.1.3.1.cmml">​</mo><mi id="S3.SS2.p4.3.m3.1.1.3.3" xref="S3.SS2.p4.3.m3.1.1.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.SS2.p4.3.m3.1.1.3.1a" xref="S3.SS2.p4.3.m3.1.1.3.1.cmml">​</mo><mi id="S3.SS2.p4.3.m3.1.1.3.4" xref="S3.SS2.p4.3.m3.1.1.3.4.cmml">c</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.3.m3.1b"><apply id="S3.SS2.p4.3.m3.1.1.cmml" xref="S3.SS2.p4.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS2.p4.3.m3.1.1.1.cmml" xref="S3.SS2.p4.3.m3.1.1">subscript</csymbol><ci id="S3.SS2.p4.3.m3.1.1.2.cmml" xref="S3.SS2.p4.3.m3.1.1.2">ℒ</ci><apply id="S3.SS2.p4.3.m3.1.1.3.cmml" xref="S3.SS2.p4.3.m3.1.1.3"><times id="S3.SS2.p4.3.m3.1.1.3.1.cmml" xref="S3.SS2.p4.3.m3.1.1.3.1"></times><ci id="S3.SS2.p4.3.m3.1.1.3.2.cmml" xref="S3.SS2.p4.3.m3.1.1.3.2">𝑟</ci><ci id="S3.SS2.p4.3.m3.1.1.3.3.cmml" xref="S3.SS2.p4.3.m3.1.1.3.3">𝑒</ci><ci id="S3.SS2.p4.3.m3.1.1.3.4.cmml" xref="S3.SS2.p4.3.m3.1.1.3.4">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.3.m3.1c">\mathcal{L}_{rec}</annotation></semantics></math>, is introduced as an additional regularizer:</p>
<table id="A6.EGx3" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S3.E3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S3.E3.m1.1" class="ltx_Math" alttext="\displaystyle\mathcal{L}_{rec}=\sum_{i}|ce_{i}-f(ce_{i})|." display="inline"><semantics id="S3.E3.m1.1a"><mrow id="S3.E3.m1.1.1.1" xref="S3.E3.m1.1.1.1.1.cmml"><mrow id="S3.E3.m1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.cmml"><msub id="S3.E3.m1.1.1.1.1.3" xref="S3.E3.m1.1.1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E3.m1.1.1.1.1.3.2" xref="S3.E3.m1.1.1.1.1.3.2.cmml">ℒ</mi><mrow id="S3.E3.m1.1.1.1.1.3.3" xref="S3.E3.m1.1.1.1.1.3.3.cmml"><mi id="S3.E3.m1.1.1.1.1.3.3.2" xref="S3.E3.m1.1.1.1.1.3.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.1.1.1.1.3.3.1" xref="S3.E3.m1.1.1.1.1.3.3.1.cmml">​</mo><mi id="S3.E3.m1.1.1.1.1.3.3.3" xref="S3.E3.m1.1.1.1.1.3.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.1.1.1.1.3.3.1a" xref="S3.E3.m1.1.1.1.1.3.3.1.cmml">​</mo><mi id="S3.E3.m1.1.1.1.1.3.3.4" xref="S3.E3.m1.1.1.1.1.3.3.4.cmml">c</mi></mrow></msub><mo id="S3.E3.m1.1.1.1.1.2" xref="S3.E3.m1.1.1.1.1.2.cmml">=</mo><mrow id="S3.E3.m1.1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.1.cmml"><mstyle displaystyle="true" id="S3.E3.m1.1.1.1.1.1.2" xref="S3.E3.m1.1.1.1.1.1.2.cmml"><munder id="S3.E3.m1.1.1.1.1.1.2a" xref="S3.E3.m1.1.1.1.1.1.2.cmml"><mo movablelimits="false" id="S3.E3.m1.1.1.1.1.1.2.2" xref="S3.E3.m1.1.1.1.1.1.2.2.cmml">∑</mo><mi id="S3.E3.m1.1.1.1.1.1.2.3" xref="S3.E3.m1.1.1.1.1.1.2.3.cmml">i</mi></munder></mstyle><mrow id="S3.E3.m1.1.1.1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.E3.m1.1.1.1.1.1.1.1.2" xref="S3.E3.m1.1.1.1.1.1.1.2.1.cmml">|</mo><mrow id="S3.E3.m1.1.1.1.1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.1.1.1.1.cmml"><mrow id="S3.E3.m1.1.1.1.1.1.1.1.1.3" xref="S3.E3.m1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.E3.m1.1.1.1.1.1.1.1.1.3.2" xref="S3.E3.m1.1.1.1.1.1.1.1.1.3.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.1.1.1.1.1.1.1.1.3.1" xref="S3.E3.m1.1.1.1.1.1.1.1.1.3.1.cmml">​</mo><msub id="S3.E3.m1.1.1.1.1.1.1.1.1.3.3" xref="S3.E3.m1.1.1.1.1.1.1.1.1.3.3.cmml"><mi id="S3.E3.m1.1.1.1.1.1.1.1.1.3.3.2" xref="S3.E3.m1.1.1.1.1.1.1.1.1.3.3.2.cmml">e</mi><mi id="S3.E3.m1.1.1.1.1.1.1.1.1.3.3.3" xref="S3.E3.m1.1.1.1.1.1.1.1.1.3.3.3.cmml">i</mi></msub></mrow><mo id="S3.E3.m1.1.1.1.1.1.1.1.1.2" xref="S3.E3.m1.1.1.1.1.1.1.1.1.2.cmml">−</mo><mrow id="S3.E3.m1.1.1.1.1.1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E3.m1.1.1.1.1.1.1.1.1.1.3" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.3.cmml">f</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.1.1.1.1.1.1.1.1.1.2" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">c</mi><mo lspace="0em" rspace="0em" id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">​</mo><msub id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.2" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml">e</mi><mi id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.3" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml">i</mi></msub></mrow><mo stretchy="false" id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.E3.m1.1.1.1.1.1.1.1.3" xref="S3.E3.m1.1.1.1.1.1.1.2.1.cmml">|</mo></mrow></mrow></mrow><mo lspace="0em" id="S3.E3.m1.1.1.1.2" xref="S3.E3.m1.1.1.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E3.m1.1b"><apply id="S3.E3.m1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1"><eq id="S3.E3.m1.1.1.1.1.2.cmml" xref="S3.E3.m1.1.1.1.1.2"></eq><apply id="S3.E3.m1.1.1.1.1.3.cmml" xref="S3.E3.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.1.3.1.cmml" xref="S3.E3.m1.1.1.1.1.3">subscript</csymbol><ci id="S3.E3.m1.1.1.1.1.3.2.cmml" xref="S3.E3.m1.1.1.1.1.3.2">ℒ</ci><apply id="S3.E3.m1.1.1.1.1.3.3.cmml" xref="S3.E3.m1.1.1.1.1.3.3"><times id="S3.E3.m1.1.1.1.1.3.3.1.cmml" xref="S3.E3.m1.1.1.1.1.3.3.1"></times><ci id="S3.E3.m1.1.1.1.1.3.3.2.cmml" xref="S3.E3.m1.1.1.1.1.3.3.2">𝑟</ci><ci id="S3.E3.m1.1.1.1.1.3.3.3.cmml" xref="S3.E3.m1.1.1.1.1.3.3.3">𝑒</ci><ci id="S3.E3.m1.1.1.1.1.3.3.4.cmml" xref="S3.E3.m1.1.1.1.1.3.3.4">𝑐</ci></apply></apply><apply id="S3.E3.m1.1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1.1"><apply id="S3.E3.m1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.1.1.2.1.cmml" xref="S3.E3.m1.1.1.1.1.1.2">subscript</csymbol><sum id="S3.E3.m1.1.1.1.1.1.2.2.cmml" xref="S3.E3.m1.1.1.1.1.1.2.2"></sum><ci id="S3.E3.m1.1.1.1.1.1.2.3.cmml" xref="S3.E3.m1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S3.E3.m1.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1"><abs id="S3.E3.m1.1.1.1.1.1.1.2.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.2"></abs><apply id="S3.E3.m1.1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1"><minus id="S3.E3.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.2"></minus><apply id="S3.E3.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.3"><times id="S3.E3.m1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.3.1"></times><ci id="S3.E3.m1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.3.2">𝑐</ci><apply id="S3.E3.m1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.1.1.1.1.1.3.3.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.3.3">subscript</csymbol><ci id="S3.E3.m1.1.1.1.1.1.1.1.1.3.3.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.3.3.2">𝑒</ci><ci id="S3.E3.m1.1.1.1.1.1.1.1.1.3.3.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.3.3.3">𝑖</ci></apply></apply><apply id="S3.E3.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1"><times id="S3.E3.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.2"></times><ci id="S3.E3.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.3">𝑓</ci><apply id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1"><times id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.1"></times><ci id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.2">𝑐</ci><apply id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.2">𝑒</ci><ci id="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S3.E3.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.3">𝑖</ci></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E3.m1.1c">\displaystyle\mathcal{L}_{rec}=\sum_{i}|ce_{i}-f(ce_{i})|.</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S3.SS2.p4.6" class="ltx_p">The overall objective function, designed to minimize the aggregate loss, is given as:</p>
<table id="S3.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S3.E4.m1.4" class="ltx_Math" alttext="\min_{f}\mathcal{L}_{topk}(f)+\alpha\mathcal{L}_{pair}(f)+\beta\mathcal{L}_{rec}(f)," display="block"><semantics id="S3.E4.m1.4a"><mrow id="S3.E4.m1.4.4.1" xref="S3.E4.m1.4.4.1.1.cmml"><mrow id="S3.E4.m1.4.4.1.1" xref="S3.E4.m1.4.4.1.1.cmml"><mrow id="S3.E4.m1.4.4.1.1.2" xref="S3.E4.m1.4.4.1.1.2.cmml"><mrow id="S3.E4.m1.4.4.1.1.2.2" xref="S3.E4.m1.4.4.1.1.2.2.cmml"><munder id="S3.E4.m1.4.4.1.1.2.2.1" xref="S3.E4.m1.4.4.1.1.2.2.1.cmml"><mi id="S3.E4.m1.4.4.1.1.2.2.1.2" xref="S3.E4.m1.4.4.1.1.2.2.1.2.cmml">min</mi><mi id="S3.E4.m1.4.4.1.1.2.2.1.3" xref="S3.E4.m1.4.4.1.1.2.2.1.3.cmml">f</mi></munder><mo lspace="0.167em" id="S3.E4.m1.4.4.1.1.2.2a" xref="S3.E4.m1.4.4.1.1.2.2.cmml">⁡</mo><msub id="S3.E4.m1.4.4.1.1.2.2.2" xref="S3.E4.m1.4.4.1.1.2.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E4.m1.4.4.1.1.2.2.2.2" xref="S3.E4.m1.4.4.1.1.2.2.2.2.cmml">ℒ</mi><mrow id="S3.E4.m1.4.4.1.1.2.2.2.3" xref="S3.E4.m1.4.4.1.1.2.2.2.3.cmml"><mi id="S3.E4.m1.4.4.1.1.2.2.2.3.2" xref="S3.E4.m1.4.4.1.1.2.2.2.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.4.4.1.1.2.2.2.3.1" xref="S3.E4.m1.4.4.1.1.2.2.2.3.1.cmml">​</mo><mi id="S3.E4.m1.4.4.1.1.2.2.2.3.3" xref="S3.E4.m1.4.4.1.1.2.2.2.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.4.4.1.1.2.2.2.3.1a" xref="S3.E4.m1.4.4.1.1.2.2.2.3.1.cmml">​</mo><mi id="S3.E4.m1.4.4.1.1.2.2.2.3.4" xref="S3.E4.m1.4.4.1.1.2.2.2.3.4.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.4.4.1.1.2.2.2.3.1b" xref="S3.E4.m1.4.4.1.1.2.2.2.3.1.cmml">​</mo><mi id="S3.E4.m1.4.4.1.1.2.2.2.3.5" xref="S3.E4.m1.4.4.1.1.2.2.2.3.5.cmml">k</mi></mrow></msub></mrow><mo lspace="0em" rspace="0em" id="S3.E4.m1.4.4.1.1.2.1" xref="S3.E4.m1.4.4.1.1.2.1.cmml">​</mo><mrow id="S3.E4.m1.4.4.1.1.2.3.2" xref="S3.E4.m1.4.4.1.1.2.cmml"><mo stretchy="false" id="S3.E4.m1.4.4.1.1.2.3.2.1" xref="S3.E4.m1.4.4.1.1.2.cmml">(</mo><mi id="S3.E4.m1.1.1" xref="S3.E4.m1.1.1.cmml">f</mi><mo stretchy="false" id="S3.E4.m1.4.4.1.1.2.3.2.2" xref="S3.E4.m1.4.4.1.1.2.cmml">)</mo></mrow></mrow><mo id="S3.E4.m1.4.4.1.1.1" xref="S3.E4.m1.4.4.1.1.1.cmml">+</mo><mrow id="S3.E4.m1.4.4.1.1.3" xref="S3.E4.m1.4.4.1.1.3.cmml"><mi id="S3.E4.m1.4.4.1.1.3.2" xref="S3.E4.m1.4.4.1.1.3.2.cmml">α</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.4.4.1.1.3.1" xref="S3.E4.m1.4.4.1.1.3.1.cmml">​</mo><msub id="S3.E4.m1.4.4.1.1.3.3" xref="S3.E4.m1.4.4.1.1.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E4.m1.4.4.1.1.3.3.2" xref="S3.E4.m1.4.4.1.1.3.3.2.cmml">ℒ</mi><mrow id="S3.E4.m1.4.4.1.1.3.3.3" xref="S3.E4.m1.4.4.1.1.3.3.3.cmml"><mi id="S3.E4.m1.4.4.1.1.3.3.3.2" xref="S3.E4.m1.4.4.1.1.3.3.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.4.4.1.1.3.3.3.1" xref="S3.E4.m1.4.4.1.1.3.3.3.1.cmml">​</mo><mi id="S3.E4.m1.4.4.1.1.3.3.3.3" xref="S3.E4.m1.4.4.1.1.3.3.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.4.4.1.1.3.3.3.1a" xref="S3.E4.m1.4.4.1.1.3.3.3.1.cmml">​</mo><mi id="S3.E4.m1.4.4.1.1.3.3.3.4" xref="S3.E4.m1.4.4.1.1.3.3.3.4.cmml">i</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.4.4.1.1.3.3.3.1b" xref="S3.E4.m1.4.4.1.1.3.3.3.1.cmml">​</mo><mi id="S3.E4.m1.4.4.1.1.3.3.3.5" xref="S3.E4.m1.4.4.1.1.3.3.3.5.cmml">r</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S3.E4.m1.4.4.1.1.3.1a" xref="S3.E4.m1.4.4.1.1.3.1.cmml">​</mo><mrow id="S3.E4.m1.4.4.1.1.3.4.2" xref="S3.E4.m1.4.4.1.1.3.cmml"><mo stretchy="false" id="S3.E4.m1.4.4.1.1.3.4.2.1" xref="S3.E4.m1.4.4.1.1.3.cmml">(</mo><mi id="S3.E4.m1.2.2" xref="S3.E4.m1.2.2.cmml">f</mi><mo stretchy="false" id="S3.E4.m1.4.4.1.1.3.4.2.2" xref="S3.E4.m1.4.4.1.1.3.cmml">)</mo></mrow></mrow><mo id="S3.E4.m1.4.4.1.1.1a" xref="S3.E4.m1.4.4.1.1.1.cmml">+</mo><mrow id="S3.E4.m1.4.4.1.1.4" xref="S3.E4.m1.4.4.1.1.4.cmml"><mi id="S3.E4.m1.4.4.1.1.4.2" xref="S3.E4.m1.4.4.1.1.4.2.cmml">β</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.4.4.1.1.4.1" xref="S3.E4.m1.4.4.1.1.4.1.cmml">​</mo><msub id="S3.E4.m1.4.4.1.1.4.3" xref="S3.E4.m1.4.4.1.1.4.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.E4.m1.4.4.1.1.4.3.2" xref="S3.E4.m1.4.4.1.1.4.3.2.cmml">ℒ</mi><mrow id="S3.E4.m1.4.4.1.1.4.3.3" xref="S3.E4.m1.4.4.1.1.4.3.3.cmml"><mi id="S3.E4.m1.4.4.1.1.4.3.3.2" xref="S3.E4.m1.4.4.1.1.4.3.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.4.4.1.1.4.3.3.1" xref="S3.E4.m1.4.4.1.1.4.3.3.1.cmml">​</mo><mi id="S3.E4.m1.4.4.1.1.4.3.3.3" xref="S3.E4.m1.4.4.1.1.4.3.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S3.E4.m1.4.4.1.1.4.3.3.1a" xref="S3.E4.m1.4.4.1.1.4.3.3.1.cmml">​</mo><mi id="S3.E4.m1.4.4.1.1.4.3.3.4" xref="S3.E4.m1.4.4.1.1.4.3.3.4.cmml">c</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S3.E4.m1.4.4.1.1.4.1a" xref="S3.E4.m1.4.4.1.1.4.1.cmml">​</mo><mrow id="S3.E4.m1.4.4.1.1.4.4.2" xref="S3.E4.m1.4.4.1.1.4.cmml"><mo stretchy="false" id="S3.E4.m1.4.4.1.1.4.4.2.1" xref="S3.E4.m1.4.4.1.1.4.cmml">(</mo><mi id="S3.E4.m1.3.3" xref="S3.E4.m1.3.3.cmml">f</mi><mo stretchy="false" id="S3.E4.m1.4.4.1.1.4.4.2.2" xref="S3.E4.m1.4.4.1.1.4.cmml">)</mo></mrow></mrow></mrow><mo id="S3.E4.m1.4.4.1.2" xref="S3.E4.m1.4.4.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.E4.m1.4b"><apply id="S3.E4.m1.4.4.1.1.cmml" xref="S3.E4.m1.4.4.1"><plus id="S3.E4.m1.4.4.1.1.1.cmml" xref="S3.E4.m1.4.4.1.1.1"></plus><apply id="S3.E4.m1.4.4.1.1.2.cmml" xref="S3.E4.m1.4.4.1.1.2"><times id="S3.E4.m1.4.4.1.1.2.1.cmml" xref="S3.E4.m1.4.4.1.1.2.1"></times><apply id="S3.E4.m1.4.4.1.1.2.2.cmml" xref="S3.E4.m1.4.4.1.1.2.2"><apply id="S3.E4.m1.4.4.1.1.2.2.1.cmml" xref="S3.E4.m1.4.4.1.1.2.2.1"><csymbol cd="ambiguous" id="S3.E4.m1.4.4.1.1.2.2.1.1.cmml" xref="S3.E4.m1.4.4.1.1.2.2.1">subscript</csymbol><min id="S3.E4.m1.4.4.1.1.2.2.1.2.cmml" xref="S3.E4.m1.4.4.1.1.2.2.1.2"></min><ci id="S3.E4.m1.4.4.1.1.2.2.1.3.cmml" xref="S3.E4.m1.4.4.1.1.2.2.1.3">𝑓</ci></apply><apply id="S3.E4.m1.4.4.1.1.2.2.2.cmml" xref="S3.E4.m1.4.4.1.1.2.2.2"><csymbol cd="ambiguous" id="S3.E4.m1.4.4.1.1.2.2.2.1.cmml" xref="S3.E4.m1.4.4.1.1.2.2.2">subscript</csymbol><ci id="S3.E4.m1.4.4.1.1.2.2.2.2.cmml" xref="S3.E4.m1.4.4.1.1.2.2.2.2">ℒ</ci><apply id="S3.E4.m1.4.4.1.1.2.2.2.3.cmml" xref="S3.E4.m1.4.4.1.1.2.2.2.3"><times id="S3.E4.m1.4.4.1.1.2.2.2.3.1.cmml" xref="S3.E4.m1.4.4.1.1.2.2.2.3.1"></times><ci id="S3.E4.m1.4.4.1.1.2.2.2.3.2.cmml" xref="S3.E4.m1.4.4.1.1.2.2.2.3.2">𝑡</ci><ci id="S3.E4.m1.4.4.1.1.2.2.2.3.3.cmml" xref="S3.E4.m1.4.4.1.1.2.2.2.3.3">𝑜</ci><ci id="S3.E4.m1.4.4.1.1.2.2.2.3.4.cmml" xref="S3.E4.m1.4.4.1.1.2.2.2.3.4">𝑝</ci><ci id="S3.E4.m1.4.4.1.1.2.2.2.3.5.cmml" xref="S3.E4.m1.4.4.1.1.2.2.2.3.5">𝑘</ci></apply></apply></apply><ci id="S3.E4.m1.1.1.cmml" xref="S3.E4.m1.1.1">𝑓</ci></apply><apply id="S3.E4.m1.4.4.1.1.3.cmml" xref="S3.E4.m1.4.4.1.1.3"><times id="S3.E4.m1.4.4.1.1.3.1.cmml" xref="S3.E4.m1.4.4.1.1.3.1"></times><ci id="S3.E4.m1.4.4.1.1.3.2.cmml" xref="S3.E4.m1.4.4.1.1.3.2">𝛼</ci><apply id="S3.E4.m1.4.4.1.1.3.3.cmml" xref="S3.E4.m1.4.4.1.1.3.3"><csymbol cd="ambiguous" id="S3.E4.m1.4.4.1.1.3.3.1.cmml" xref="S3.E4.m1.4.4.1.1.3.3">subscript</csymbol><ci id="S3.E4.m1.4.4.1.1.3.3.2.cmml" xref="S3.E4.m1.4.4.1.1.3.3.2">ℒ</ci><apply id="S3.E4.m1.4.4.1.1.3.3.3.cmml" xref="S3.E4.m1.4.4.1.1.3.3.3"><times id="S3.E4.m1.4.4.1.1.3.3.3.1.cmml" xref="S3.E4.m1.4.4.1.1.3.3.3.1"></times><ci id="S3.E4.m1.4.4.1.1.3.3.3.2.cmml" xref="S3.E4.m1.4.4.1.1.3.3.3.2">𝑝</ci><ci id="S3.E4.m1.4.4.1.1.3.3.3.3.cmml" xref="S3.E4.m1.4.4.1.1.3.3.3.3">𝑎</ci><ci id="S3.E4.m1.4.4.1.1.3.3.3.4.cmml" xref="S3.E4.m1.4.4.1.1.3.3.3.4">𝑖</ci><ci id="S3.E4.m1.4.4.1.1.3.3.3.5.cmml" xref="S3.E4.m1.4.4.1.1.3.3.3.5">𝑟</ci></apply></apply><ci id="S3.E4.m1.2.2.cmml" xref="S3.E4.m1.2.2">𝑓</ci></apply><apply id="S3.E4.m1.4.4.1.1.4.cmml" xref="S3.E4.m1.4.4.1.1.4"><times id="S3.E4.m1.4.4.1.1.4.1.cmml" xref="S3.E4.m1.4.4.1.1.4.1"></times><ci id="S3.E4.m1.4.4.1.1.4.2.cmml" xref="S3.E4.m1.4.4.1.1.4.2">𝛽</ci><apply id="S3.E4.m1.4.4.1.1.4.3.cmml" xref="S3.E4.m1.4.4.1.1.4.3"><csymbol cd="ambiguous" id="S3.E4.m1.4.4.1.1.4.3.1.cmml" xref="S3.E4.m1.4.4.1.1.4.3">subscript</csymbol><ci id="S3.E4.m1.4.4.1.1.4.3.2.cmml" xref="S3.E4.m1.4.4.1.1.4.3.2">ℒ</ci><apply id="S3.E4.m1.4.4.1.1.4.3.3.cmml" xref="S3.E4.m1.4.4.1.1.4.3.3"><times id="S3.E4.m1.4.4.1.1.4.3.3.1.cmml" xref="S3.E4.m1.4.4.1.1.4.3.3.1"></times><ci id="S3.E4.m1.4.4.1.1.4.3.3.2.cmml" xref="S3.E4.m1.4.4.1.1.4.3.3.2">𝑟</ci><ci id="S3.E4.m1.4.4.1.1.4.3.3.3.cmml" xref="S3.E4.m1.4.4.1.1.4.3.3.3">𝑒</ci><ci id="S3.E4.m1.4.4.1.1.4.3.3.4.cmml" xref="S3.E4.m1.4.4.1.1.4.3.3.4">𝑐</ci></apply></apply><ci id="S3.E4.m1.3.3.cmml" xref="S3.E4.m1.3.3">𝑓</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.E4.m1.4c">\min_{f}\mathcal{L}_{topk}(f)+\alpha\mathcal{L}_{pair}(f)+\beta\mathcal{L}_{rec}(f),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
<p id="S3.SS2.p4.5" class="ltx_p">with <math id="S3.SS2.p4.4.m1.2" class="ltx_Math" alttext="\alpha,\beta&gt;0" display="inline"><semantics id="S3.SS2.p4.4.m1.2a"><mrow id="S3.SS2.p4.4.m1.2.3" xref="S3.SS2.p4.4.m1.2.3.cmml"><mrow id="S3.SS2.p4.4.m1.2.3.2.2" xref="S3.SS2.p4.4.m1.2.3.2.1.cmml"><mi id="S3.SS2.p4.4.m1.1.1" xref="S3.SS2.p4.4.m1.1.1.cmml">α</mi><mo id="S3.SS2.p4.4.m1.2.3.2.2.1" xref="S3.SS2.p4.4.m1.2.3.2.1.cmml">,</mo><mi id="S3.SS2.p4.4.m1.2.2" xref="S3.SS2.p4.4.m1.2.2.cmml">β</mi></mrow><mo id="S3.SS2.p4.4.m1.2.3.1" xref="S3.SS2.p4.4.m1.2.3.1.cmml">&gt;</mo><mn id="S3.SS2.p4.4.m1.2.3.3" xref="S3.SS2.p4.4.m1.2.3.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.4.m1.2b"><apply id="S3.SS2.p4.4.m1.2.3.cmml" xref="S3.SS2.p4.4.m1.2.3"><gt id="S3.SS2.p4.4.m1.2.3.1.cmml" xref="S3.SS2.p4.4.m1.2.3.1"></gt><list id="S3.SS2.p4.4.m1.2.3.2.1.cmml" xref="S3.SS2.p4.4.m1.2.3.2.2"><ci id="S3.SS2.p4.4.m1.1.1.cmml" xref="S3.SS2.p4.4.m1.1.1">𝛼</ci><ci id="S3.SS2.p4.4.m1.2.2.cmml" xref="S3.SS2.p4.4.m1.2.2">𝛽</ci></list><cn type="integer" id="S3.SS2.p4.4.m1.2.3.3.cmml" xref="S3.SS2.p4.4.m1.2.3.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.4.m1.2c">\alpha,\beta&gt;0</annotation></semantics></math> being hyperparameters.
Within the context of the unsupervised tuning setting, we fix their values as <math id="S3.SS2.p4.5.m2.2" class="ltx_Math" alttext="\alpha=1.0,\beta=1.0" display="inline"><semantics id="S3.SS2.p4.5.m2.2a"><mrow id="S3.SS2.p4.5.m2.2.2.2" xref="S3.SS2.p4.5.m2.2.2.3.cmml"><mrow id="S3.SS2.p4.5.m2.1.1.1.1" xref="S3.SS2.p4.5.m2.1.1.1.1.cmml"><mi id="S3.SS2.p4.5.m2.1.1.1.1.2" xref="S3.SS2.p4.5.m2.1.1.1.1.2.cmml">α</mi><mo id="S3.SS2.p4.5.m2.1.1.1.1.1" xref="S3.SS2.p4.5.m2.1.1.1.1.1.cmml">=</mo><mn id="S3.SS2.p4.5.m2.1.1.1.1.3" xref="S3.SS2.p4.5.m2.1.1.1.1.3.cmml">1.0</mn></mrow><mo id="S3.SS2.p4.5.m2.2.2.2.3" xref="S3.SS2.p4.5.m2.2.2.3a.cmml">,</mo><mrow id="S3.SS2.p4.5.m2.2.2.2.2" xref="S3.SS2.p4.5.m2.2.2.2.2.cmml"><mi id="S3.SS2.p4.5.m2.2.2.2.2.2" xref="S3.SS2.p4.5.m2.2.2.2.2.2.cmml">β</mi><mo id="S3.SS2.p4.5.m2.2.2.2.2.1" xref="S3.SS2.p4.5.m2.2.2.2.2.1.cmml">=</mo><mn id="S3.SS2.p4.5.m2.2.2.2.2.3" xref="S3.SS2.p4.5.m2.2.2.2.2.3.cmml">1.0</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.p4.5.m2.2b"><apply id="S3.SS2.p4.5.m2.2.2.3.cmml" xref="S3.SS2.p4.5.m2.2.2.2"><csymbol cd="ambiguous" id="S3.SS2.p4.5.m2.2.2.3a.cmml" xref="S3.SS2.p4.5.m2.2.2.2.3">formulae-sequence</csymbol><apply id="S3.SS2.p4.5.m2.1.1.1.1.cmml" xref="S3.SS2.p4.5.m2.1.1.1.1"><eq id="S3.SS2.p4.5.m2.1.1.1.1.1.cmml" xref="S3.SS2.p4.5.m2.1.1.1.1.1"></eq><ci id="S3.SS2.p4.5.m2.1.1.1.1.2.cmml" xref="S3.SS2.p4.5.m2.1.1.1.1.2">𝛼</ci><cn type="float" id="S3.SS2.p4.5.m2.1.1.1.1.3.cmml" xref="S3.SS2.p4.5.m2.1.1.1.1.3">1.0</cn></apply><apply id="S3.SS2.p4.5.m2.2.2.2.2.cmml" xref="S3.SS2.p4.5.m2.2.2.2.2"><eq id="S3.SS2.p4.5.m2.2.2.2.2.1.cmml" xref="S3.SS2.p4.5.m2.2.2.2.2.1"></eq><ci id="S3.SS2.p4.5.m2.2.2.2.2.2.cmml" xref="S3.SS2.p4.5.m2.2.2.2.2.2">𝛽</ci><cn type="float" id="S3.SS2.p4.5.m2.2.2.2.2.3.cmml" xref="S3.SS2.p4.5.m2.2.2.2.2.3">1.0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.p4.5.m2.2c">\alpha=1.0,\beta=1.0</annotation></semantics></math>.</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Supervised Matryoshka-Adaptor</h2>

<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Problem formulation</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.1" class="ltx_p">For many information retrieval applications, the availability of pairwise data, which indicates the relevance between specific queries and corpora, can considerably improve retrieval performance <cite class="ltx_cite ltx_citemacro_citep">(Yoon et al., <a href="#bib.bib25" title="" class="ltx_ref">2023</a>)</cite>.
In this section, we introduce a method to further refine the Matryoshka-Adaptor by utilizing (a limited number of) such paired samples.</p>
</div>
<div id="S4.SS1.p2" class="ltx_para">
<p id="S4.SS1.p2.11" class="ltx_p">Let <math id="S4.SS1.p2.1.m1.4" class="ltx_Math" alttext="\mathcal{Q}=\{q_{1},q_{2},...,q_{M}\}" display="inline"><semantics id="S4.SS1.p2.1.m1.4a"><mrow id="S4.SS1.p2.1.m1.4.4" xref="S4.SS1.p2.1.m1.4.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS1.p2.1.m1.4.4.5" xref="S4.SS1.p2.1.m1.4.4.5.cmml">𝒬</mi><mo id="S4.SS1.p2.1.m1.4.4.4" xref="S4.SS1.p2.1.m1.4.4.4.cmml">=</mo><mrow id="S4.SS1.p2.1.m1.4.4.3.3" xref="S4.SS1.p2.1.m1.4.4.3.4.cmml"><mo stretchy="false" id="S4.SS1.p2.1.m1.4.4.3.3.4" xref="S4.SS1.p2.1.m1.4.4.3.4.cmml">{</mo><msub id="S4.SS1.p2.1.m1.2.2.1.1.1" xref="S4.SS1.p2.1.m1.2.2.1.1.1.cmml"><mi id="S4.SS1.p2.1.m1.2.2.1.1.1.2" xref="S4.SS1.p2.1.m1.2.2.1.1.1.2.cmml">q</mi><mn id="S4.SS1.p2.1.m1.2.2.1.1.1.3" xref="S4.SS1.p2.1.m1.2.2.1.1.1.3.cmml">1</mn></msub><mo id="S4.SS1.p2.1.m1.4.4.3.3.5" xref="S4.SS1.p2.1.m1.4.4.3.4.cmml">,</mo><msub id="S4.SS1.p2.1.m1.3.3.2.2.2" xref="S4.SS1.p2.1.m1.3.3.2.2.2.cmml"><mi id="S4.SS1.p2.1.m1.3.3.2.2.2.2" xref="S4.SS1.p2.1.m1.3.3.2.2.2.2.cmml">q</mi><mn id="S4.SS1.p2.1.m1.3.3.2.2.2.3" xref="S4.SS1.p2.1.m1.3.3.2.2.2.3.cmml">2</mn></msub><mo id="S4.SS1.p2.1.m1.4.4.3.3.6" xref="S4.SS1.p2.1.m1.4.4.3.4.cmml">,</mo><mi mathvariant="normal" id="S4.SS1.p2.1.m1.1.1" xref="S4.SS1.p2.1.m1.1.1.cmml">…</mi><mo id="S4.SS1.p2.1.m1.4.4.3.3.7" xref="S4.SS1.p2.1.m1.4.4.3.4.cmml">,</mo><msub id="S4.SS1.p2.1.m1.4.4.3.3.3" xref="S4.SS1.p2.1.m1.4.4.3.3.3.cmml"><mi id="S4.SS1.p2.1.m1.4.4.3.3.3.2" xref="S4.SS1.p2.1.m1.4.4.3.3.3.2.cmml">q</mi><mi id="S4.SS1.p2.1.m1.4.4.3.3.3.3" xref="S4.SS1.p2.1.m1.4.4.3.3.3.3.cmml">M</mi></msub><mo stretchy="false" id="S4.SS1.p2.1.m1.4.4.3.3.8" xref="S4.SS1.p2.1.m1.4.4.3.4.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.1.m1.4b"><apply id="S4.SS1.p2.1.m1.4.4.cmml" xref="S4.SS1.p2.1.m1.4.4"><eq id="S4.SS1.p2.1.m1.4.4.4.cmml" xref="S4.SS1.p2.1.m1.4.4.4"></eq><ci id="S4.SS1.p2.1.m1.4.4.5.cmml" xref="S4.SS1.p2.1.m1.4.4.5">𝒬</ci><set id="S4.SS1.p2.1.m1.4.4.3.4.cmml" xref="S4.SS1.p2.1.m1.4.4.3.3"><apply id="S4.SS1.p2.1.m1.2.2.1.1.1.cmml" xref="S4.SS1.p2.1.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="S4.SS1.p2.1.m1.2.2.1.1.1.1.cmml" xref="S4.SS1.p2.1.m1.2.2.1.1.1">subscript</csymbol><ci id="S4.SS1.p2.1.m1.2.2.1.1.1.2.cmml" xref="S4.SS1.p2.1.m1.2.2.1.1.1.2">𝑞</ci><cn type="integer" id="S4.SS1.p2.1.m1.2.2.1.1.1.3.cmml" xref="S4.SS1.p2.1.m1.2.2.1.1.1.3">1</cn></apply><apply id="S4.SS1.p2.1.m1.3.3.2.2.2.cmml" xref="S4.SS1.p2.1.m1.3.3.2.2.2"><csymbol cd="ambiguous" id="S4.SS1.p2.1.m1.3.3.2.2.2.1.cmml" xref="S4.SS1.p2.1.m1.3.3.2.2.2">subscript</csymbol><ci id="S4.SS1.p2.1.m1.3.3.2.2.2.2.cmml" xref="S4.SS1.p2.1.m1.3.3.2.2.2.2">𝑞</ci><cn type="integer" id="S4.SS1.p2.1.m1.3.3.2.2.2.3.cmml" xref="S4.SS1.p2.1.m1.3.3.2.2.2.3">2</cn></apply><ci id="S4.SS1.p2.1.m1.1.1.cmml" xref="S4.SS1.p2.1.m1.1.1">…</ci><apply id="S4.SS1.p2.1.m1.4.4.3.3.3.cmml" xref="S4.SS1.p2.1.m1.4.4.3.3.3"><csymbol cd="ambiguous" id="S4.SS1.p2.1.m1.4.4.3.3.3.1.cmml" xref="S4.SS1.p2.1.m1.4.4.3.3.3">subscript</csymbol><ci id="S4.SS1.p2.1.m1.4.4.3.3.3.2.cmml" xref="S4.SS1.p2.1.m1.4.4.3.3.3.2">𝑞</ci><ci id="S4.SS1.p2.1.m1.4.4.3.3.3.3.cmml" xref="S4.SS1.p2.1.m1.4.4.3.3.3.3">𝑀</ci></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.1.m1.4c">\mathcal{Q}=\{q_{1},q_{2},...,q_{M}\}</annotation></semantics></math> denote the set of queries, and let <math id="S4.SS1.p2.2.m2.3" class="ltx_Math" alttext="R=\{(q_{i},c_{j},y_{ij})\}_{i=1:M,j=1:N}" display="inline"><semantics id="S4.SS1.p2.2.m2.3a"><mrow id="S4.SS1.p2.2.m2.3.3" xref="S4.SS1.p2.2.m2.3.3.cmml"><mi id="S4.SS1.p2.2.m2.3.3.3" xref="S4.SS1.p2.2.m2.3.3.3.cmml">R</mi><mo id="S4.SS1.p2.2.m2.3.3.2" xref="S4.SS1.p2.2.m2.3.3.2.cmml">=</mo><msub id="S4.SS1.p2.2.m2.3.3.1" xref="S4.SS1.p2.2.m2.3.3.1.cmml"><mrow id="S4.SS1.p2.2.m2.3.3.1.1.1" xref="S4.SS1.p2.2.m2.3.3.1.1.2.cmml"><mo stretchy="false" id="S4.SS1.p2.2.m2.3.3.1.1.1.2" xref="S4.SS1.p2.2.m2.3.3.1.1.2.cmml">{</mo><mrow id="S4.SS1.p2.2.m2.3.3.1.1.1.1.3" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.4.cmml"><mo stretchy="false" id="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.4" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.4.cmml">(</mo><msub id="S4.SS1.p2.2.m2.3.3.1.1.1.1.1.1" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.1.1.cmml"><mi id="S4.SS1.p2.2.m2.3.3.1.1.1.1.1.1.2" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.1.1.2.cmml">q</mi><mi id="S4.SS1.p2.2.m2.3.3.1.1.1.1.1.1.3" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.1.1.3.cmml">i</mi></msub><mo id="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.5" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.4.cmml">,</mo><msub id="S4.SS1.p2.2.m2.3.3.1.1.1.1.2.2" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.2.2.cmml"><mi id="S4.SS1.p2.2.m2.3.3.1.1.1.1.2.2.2" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.2.2.2.cmml">c</mi><mi id="S4.SS1.p2.2.m2.3.3.1.1.1.1.2.2.3" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.2.2.3.cmml">j</mi></msub><mo id="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.6" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.4.cmml">,</mo><msub id="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.cmml"><mi id="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.2" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.2.cmml">y</mi><mrow id="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.3" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.3.cmml"><mi id="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.3.2" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.3.1" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.3.1.cmml">​</mo><mi id="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.3.3" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.3.3.cmml">j</mi></mrow></msub><mo stretchy="false" id="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.7" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.4.cmml">)</mo></mrow><mo stretchy="false" id="S4.SS1.p2.2.m2.3.3.1.1.1.3" xref="S4.SS1.p2.2.m2.3.3.1.1.2.cmml">}</mo></mrow><mrow id="S4.SS1.p2.2.m2.2.2.2" xref="S4.SS1.p2.2.m2.2.2.2.cmml"><mrow id="S4.SS1.p2.2.m2.2.2.2.4" xref="S4.SS1.p2.2.m2.2.2.2.4.cmml"><mi id="S4.SS1.p2.2.m2.2.2.2.4.2" xref="S4.SS1.p2.2.m2.2.2.2.4.2.cmml">i</mi><mo id="S4.SS1.p2.2.m2.2.2.2.4.1" xref="S4.SS1.p2.2.m2.2.2.2.4.1.cmml">=</mo><mn id="S4.SS1.p2.2.m2.2.2.2.4.3" xref="S4.SS1.p2.2.m2.2.2.2.4.3.cmml">1</mn></mrow><mo lspace="0.278em" rspace="0.278em" id="S4.SS1.p2.2.m2.2.2.2.5" xref="S4.SS1.p2.2.m2.2.2.2.5.cmml">:</mo><mrow id="S4.SS1.p2.2.m2.2.2.2.6" xref="S4.SS1.p2.2.m2.2.2.2.6.cmml"><mrow id="S4.SS1.p2.2.m2.2.2.2.6.2.2" xref="S4.SS1.p2.2.m2.2.2.2.6.2.1.cmml"><mi id="S4.SS1.p2.2.m2.1.1.1.1" xref="S4.SS1.p2.2.m2.1.1.1.1.cmml">M</mi><mo id="S4.SS1.p2.2.m2.2.2.2.6.2.2.1" xref="S4.SS1.p2.2.m2.2.2.2.6.2.1.cmml">,</mo><mi id="S4.SS1.p2.2.m2.2.2.2.2" xref="S4.SS1.p2.2.m2.2.2.2.2.cmml">j</mi></mrow><mo id="S4.SS1.p2.2.m2.2.2.2.6.1" xref="S4.SS1.p2.2.m2.2.2.2.6.1.cmml">=</mo><mn id="S4.SS1.p2.2.m2.2.2.2.6.3" xref="S4.SS1.p2.2.m2.2.2.2.6.3.cmml">1</mn></mrow><mo lspace="0.278em" rspace="0.278em" id="S4.SS1.p2.2.m2.2.2.2.7" xref="S4.SS1.p2.2.m2.2.2.2.7.cmml">:</mo><mi id="S4.SS1.p2.2.m2.2.2.2.8" xref="S4.SS1.p2.2.m2.2.2.2.8.cmml">N</mi></mrow></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.2.m2.3b"><apply id="S4.SS1.p2.2.m2.3.3.cmml" xref="S4.SS1.p2.2.m2.3.3"><eq id="S4.SS1.p2.2.m2.3.3.2.cmml" xref="S4.SS1.p2.2.m2.3.3.2"></eq><ci id="S4.SS1.p2.2.m2.3.3.3.cmml" xref="S4.SS1.p2.2.m2.3.3.3">𝑅</ci><apply id="S4.SS1.p2.2.m2.3.3.1.cmml" xref="S4.SS1.p2.2.m2.3.3.1"><csymbol cd="ambiguous" id="S4.SS1.p2.2.m2.3.3.1.2.cmml" xref="S4.SS1.p2.2.m2.3.3.1">subscript</csymbol><set id="S4.SS1.p2.2.m2.3.3.1.1.2.cmml" xref="S4.SS1.p2.2.m2.3.3.1.1.1"><vector id="S4.SS1.p2.2.m2.3.3.1.1.1.1.4.cmml" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.3"><apply id="S4.SS1.p2.2.m2.3.3.1.1.1.1.1.1.cmml" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.SS1.p2.2.m2.3.3.1.1.1.1.1.1.1.cmml" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.1.1">subscript</csymbol><ci id="S4.SS1.p2.2.m2.3.3.1.1.1.1.1.1.2.cmml" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.1.1.2">𝑞</ci><ci id="S4.SS1.p2.2.m2.3.3.1.1.1.1.1.1.3.cmml" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="S4.SS1.p2.2.m2.3.3.1.1.1.1.2.2.cmml" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S4.SS1.p2.2.m2.3.3.1.1.1.1.2.2.1.cmml" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.2.2">subscript</csymbol><ci id="S4.SS1.p2.2.m2.3.3.1.1.1.1.2.2.2.cmml" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.2.2.2">𝑐</ci><ci id="S4.SS1.p2.2.m2.3.3.1.1.1.1.2.2.3.cmml" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.2.2.3">𝑗</ci></apply><apply id="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.cmml" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3"><csymbol cd="ambiguous" id="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.1.cmml" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3">subscript</csymbol><ci id="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.2.cmml" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.2">𝑦</ci><apply id="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.3.cmml" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.3"><times id="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.3.1.cmml" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.3.1"></times><ci id="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.3.2.cmml" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.3.2">𝑖</ci><ci id="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.3.3.cmml" xref="S4.SS1.p2.2.m2.3.3.1.1.1.1.3.3.3.3">𝑗</ci></apply></apply></vector></set><apply id="S4.SS1.p2.2.m2.2.2.2.cmml" xref="S4.SS1.p2.2.m2.2.2.2"><and id="S4.SS1.p2.2.m2.2.2.2a.cmml" xref="S4.SS1.p2.2.m2.2.2.2"></and><apply id="S4.SS1.p2.2.m2.2.2.2b.cmml" xref="S4.SS1.p2.2.m2.2.2.2"><ci id="S4.SS1.p2.2.m2.2.2.2.5.cmml" xref="S4.SS1.p2.2.m2.2.2.2.5">:</ci><apply id="S4.SS1.p2.2.m2.2.2.2.4.cmml" xref="S4.SS1.p2.2.m2.2.2.2.4"><eq id="S4.SS1.p2.2.m2.2.2.2.4.1.cmml" xref="S4.SS1.p2.2.m2.2.2.2.4.1"></eq><ci id="S4.SS1.p2.2.m2.2.2.2.4.2.cmml" xref="S4.SS1.p2.2.m2.2.2.2.4.2">𝑖</ci><cn type="integer" id="S4.SS1.p2.2.m2.2.2.2.4.3.cmml" xref="S4.SS1.p2.2.m2.2.2.2.4.3">1</cn></apply><apply id="S4.SS1.p2.2.m2.2.2.2.6.cmml" xref="S4.SS1.p2.2.m2.2.2.2.6"><eq id="S4.SS1.p2.2.m2.2.2.2.6.1.cmml" xref="S4.SS1.p2.2.m2.2.2.2.6.1"></eq><list id="S4.SS1.p2.2.m2.2.2.2.6.2.1.cmml" xref="S4.SS1.p2.2.m2.2.2.2.6.2.2"><ci id="S4.SS1.p2.2.m2.1.1.1.1.cmml" xref="S4.SS1.p2.2.m2.1.1.1.1">𝑀</ci><ci id="S4.SS1.p2.2.m2.2.2.2.2.cmml" xref="S4.SS1.p2.2.m2.2.2.2.2">𝑗</ci></list><cn type="integer" id="S4.SS1.p2.2.m2.2.2.2.6.3.cmml" xref="S4.SS1.p2.2.m2.2.2.2.6.3">1</cn></apply></apply><apply id="S4.SS1.p2.2.m2.2.2.2c.cmml" xref="S4.SS1.p2.2.m2.2.2.2"><ci id="S4.SS1.p2.2.m2.2.2.2.7.cmml" xref="S4.SS1.p2.2.m2.2.2.2.7">:</ci><share href="#S4.SS1.p2.2.m2.2.2.2.6.cmml" id="S4.SS1.p2.2.m2.2.2.2d.cmml" xref="S4.SS1.p2.2.m2.2.2.2"></share><ci id="S4.SS1.p2.2.m2.2.2.2.8.cmml" xref="S4.SS1.p2.2.m2.2.2.2.8">𝑁</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.2.m2.3c">R=\{(q_{i},c_{j},y_{ij})\}_{i=1:M,j=1:N}</annotation></semantics></math> represent the set of query-corpus relevance triplets, where <math id="S4.SS1.p2.3.m3.1" class="ltx_Math" alttext="y_{ij}&gt;0" display="inline"><semantics id="S4.SS1.p2.3.m3.1a"><mrow id="S4.SS1.p2.3.m3.1.1" xref="S4.SS1.p2.3.m3.1.1.cmml"><msub id="S4.SS1.p2.3.m3.1.1.2" xref="S4.SS1.p2.3.m3.1.1.2.cmml"><mi id="S4.SS1.p2.3.m3.1.1.2.2" xref="S4.SS1.p2.3.m3.1.1.2.2.cmml">y</mi><mrow id="S4.SS1.p2.3.m3.1.1.2.3" xref="S4.SS1.p2.3.m3.1.1.2.3.cmml"><mi id="S4.SS1.p2.3.m3.1.1.2.3.2" xref="S4.SS1.p2.3.m3.1.1.2.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S4.SS1.p2.3.m3.1.1.2.3.1" xref="S4.SS1.p2.3.m3.1.1.2.3.1.cmml">​</mo><mi id="S4.SS1.p2.3.m3.1.1.2.3.3" xref="S4.SS1.p2.3.m3.1.1.2.3.3.cmml">j</mi></mrow></msub><mo id="S4.SS1.p2.3.m3.1.1.1" xref="S4.SS1.p2.3.m3.1.1.1.cmml">&gt;</mo><mn id="S4.SS1.p2.3.m3.1.1.3" xref="S4.SS1.p2.3.m3.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.3.m3.1b"><apply id="S4.SS1.p2.3.m3.1.1.cmml" xref="S4.SS1.p2.3.m3.1.1"><gt id="S4.SS1.p2.3.m3.1.1.1.cmml" xref="S4.SS1.p2.3.m3.1.1.1"></gt><apply id="S4.SS1.p2.3.m3.1.1.2.cmml" xref="S4.SS1.p2.3.m3.1.1.2"><csymbol cd="ambiguous" id="S4.SS1.p2.3.m3.1.1.2.1.cmml" xref="S4.SS1.p2.3.m3.1.1.2">subscript</csymbol><ci id="S4.SS1.p2.3.m3.1.1.2.2.cmml" xref="S4.SS1.p2.3.m3.1.1.2.2">𝑦</ci><apply id="S4.SS1.p2.3.m3.1.1.2.3.cmml" xref="S4.SS1.p2.3.m3.1.1.2.3"><times id="S4.SS1.p2.3.m3.1.1.2.3.1.cmml" xref="S4.SS1.p2.3.m3.1.1.2.3.1"></times><ci id="S4.SS1.p2.3.m3.1.1.2.3.2.cmml" xref="S4.SS1.p2.3.m3.1.1.2.3.2">𝑖</ci><ci id="S4.SS1.p2.3.m3.1.1.2.3.3.cmml" xref="S4.SS1.p2.3.m3.1.1.2.3.3">𝑗</ci></apply></apply><cn type="integer" id="S4.SS1.p2.3.m3.1.1.3.cmml" xref="S4.SS1.p2.3.m3.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.3.m3.1c">y_{ij}&gt;0</annotation></semantics></math> signifies the relevance score between query <math id="S4.SS1.p2.4.m4.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S4.SS1.p2.4.m4.1a"><msub id="S4.SS1.p2.4.m4.1.1" xref="S4.SS1.p2.4.m4.1.1.cmml"><mi id="S4.SS1.p2.4.m4.1.1.2" xref="S4.SS1.p2.4.m4.1.1.2.cmml">q</mi><mi id="S4.SS1.p2.4.m4.1.1.3" xref="S4.SS1.p2.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.4.m4.1b"><apply id="S4.SS1.p2.4.m4.1.1.cmml" xref="S4.SS1.p2.4.m4.1.1"><csymbol cd="ambiguous" id="S4.SS1.p2.4.m4.1.1.1.cmml" xref="S4.SS1.p2.4.m4.1.1">subscript</csymbol><ci id="S4.SS1.p2.4.m4.1.1.2.cmml" xref="S4.SS1.p2.4.m4.1.1.2">𝑞</ci><ci id="S4.SS1.p2.4.m4.1.1.3.cmml" xref="S4.SS1.p2.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.4.m4.1c">q_{i}</annotation></semantics></math> and corpus <math id="S4.SS1.p2.5.m5.1" class="ltx_Math" alttext="c_{j}" display="inline"><semantics id="S4.SS1.p2.5.m5.1a"><msub id="S4.SS1.p2.5.m5.1.1" xref="S4.SS1.p2.5.m5.1.1.cmml"><mi id="S4.SS1.p2.5.m5.1.1.2" xref="S4.SS1.p2.5.m5.1.1.2.cmml">c</mi><mi id="S4.SS1.p2.5.m5.1.1.3" xref="S4.SS1.p2.5.m5.1.1.3.cmml">j</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.5.m5.1b"><apply id="S4.SS1.p2.5.m5.1.1.cmml" xref="S4.SS1.p2.5.m5.1.1"><csymbol cd="ambiguous" id="S4.SS1.p2.5.m5.1.1.1.cmml" xref="S4.SS1.p2.5.m5.1.1">subscript</csymbol><ci id="S4.SS1.p2.5.m5.1.1.2.cmml" xref="S4.SS1.p2.5.m5.1.1.2">𝑐</ci><ci id="S4.SS1.p2.5.m5.1.1.3.cmml" xref="S4.SS1.p2.5.m5.1.1.3">𝑗</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.5.m5.1c">c_{j}</annotation></semantics></math>.
The query embeddings extracted from the model (<math id="S4.SS1.p2.6.m6.1" class="ltx_Math" alttext="E" display="inline"><semantics id="S4.SS1.p2.6.m6.1a"><mi id="S4.SS1.p2.6.m6.1.1" xref="S4.SS1.p2.6.m6.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.6.m6.1b"><ci id="S4.SS1.p2.6.m6.1.1.cmml" xref="S4.SS1.p2.6.m6.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.6.m6.1c">E</annotation></semantics></math>) are denoted as <math id="S4.SS1.p2.7.m7.4" class="ltx_Math" alttext="\mathcal{Q}_{E}=\{qe_{1},qe_{2},...,qe_{N}\}" display="inline"><semantics id="S4.SS1.p2.7.m7.4a"><mrow id="S4.SS1.p2.7.m7.4.4" xref="S4.SS1.p2.7.m7.4.4.cmml"><msub id="S4.SS1.p2.7.m7.4.4.5" xref="S4.SS1.p2.7.m7.4.4.5.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS1.p2.7.m7.4.4.5.2" xref="S4.SS1.p2.7.m7.4.4.5.2.cmml">𝒬</mi><mi id="S4.SS1.p2.7.m7.4.4.5.3" xref="S4.SS1.p2.7.m7.4.4.5.3.cmml">E</mi></msub><mo id="S4.SS1.p2.7.m7.4.4.4" xref="S4.SS1.p2.7.m7.4.4.4.cmml">=</mo><mrow id="S4.SS1.p2.7.m7.4.4.3.3" xref="S4.SS1.p2.7.m7.4.4.3.4.cmml"><mo stretchy="false" id="S4.SS1.p2.7.m7.4.4.3.3.4" xref="S4.SS1.p2.7.m7.4.4.3.4.cmml">{</mo><mrow id="S4.SS1.p2.7.m7.2.2.1.1.1" xref="S4.SS1.p2.7.m7.2.2.1.1.1.cmml"><mi id="S4.SS1.p2.7.m7.2.2.1.1.1.2" xref="S4.SS1.p2.7.m7.2.2.1.1.1.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="S4.SS1.p2.7.m7.2.2.1.1.1.1" xref="S4.SS1.p2.7.m7.2.2.1.1.1.1.cmml">​</mo><msub id="S4.SS1.p2.7.m7.2.2.1.1.1.3" xref="S4.SS1.p2.7.m7.2.2.1.1.1.3.cmml"><mi id="S4.SS1.p2.7.m7.2.2.1.1.1.3.2" xref="S4.SS1.p2.7.m7.2.2.1.1.1.3.2.cmml">e</mi><mn id="S4.SS1.p2.7.m7.2.2.1.1.1.3.3" xref="S4.SS1.p2.7.m7.2.2.1.1.1.3.3.cmml">1</mn></msub></mrow><mo id="S4.SS1.p2.7.m7.4.4.3.3.5" xref="S4.SS1.p2.7.m7.4.4.3.4.cmml">,</mo><mrow id="S4.SS1.p2.7.m7.3.3.2.2.2" xref="S4.SS1.p2.7.m7.3.3.2.2.2.cmml"><mi id="S4.SS1.p2.7.m7.3.3.2.2.2.2" xref="S4.SS1.p2.7.m7.3.3.2.2.2.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="S4.SS1.p2.7.m7.3.3.2.2.2.1" xref="S4.SS1.p2.7.m7.3.3.2.2.2.1.cmml">​</mo><msub id="S4.SS1.p2.7.m7.3.3.2.2.2.3" xref="S4.SS1.p2.7.m7.3.3.2.2.2.3.cmml"><mi id="S4.SS1.p2.7.m7.3.3.2.2.2.3.2" xref="S4.SS1.p2.7.m7.3.3.2.2.2.3.2.cmml">e</mi><mn id="S4.SS1.p2.7.m7.3.3.2.2.2.3.3" xref="S4.SS1.p2.7.m7.3.3.2.2.2.3.3.cmml">2</mn></msub></mrow><mo id="S4.SS1.p2.7.m7.4.4.3.3.6" xref="S4.SS1.p2.7.m7.4.4.3.4.cmml">,</mo><mi mathvariant="normal" id="S4.SS1.p2.7.m7.1.1" xref="S4.SS1.p2.7.m7.1.1.cmml">…</mi><mo id="S4.SS1.p2.7.m7.4.4.3.3.7" xref="S4.SS1.p2.7.m7.4.4.3.4.cmml">,</mo><mrow id="S4.SS1.p2.7.m7.4.4.3.3.3" xref="S4.SS1.p2.7.m7.4.4.3.3.3.cmml"><mi id="S4.SS1.p2.7.m7.4.4.3.3.3.2" xref="S4.SS1.p2.7.m7.4.4.3.3.3.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="S4.SS1.p2.7.m7.4.4.3.3.3.1" xref="S4.SS1.p2.7.m7.4.4.3.3.3.1.cmml">​</mo><msub id="S4.SS1.p2.7.m7.4.4.3.3.3.3" xref="S4.SS1.p2.7.m7.4.4.3.3.3.3.cmml"><mi id="S4.SS1.p2.7.m7.4.4.3.3.3.3.2" xref="S4.SS1.p2.7.m7.4.4.3.3.3.3.2.cmml">e</mi><mi id="S4.SS1.p2.7.m7.4.4.3.3.3.3.3" xref="S4.SS1.p2.7.m7.4.4.3.3.3.3.3.cmml">N</mi></msub></mrow><mo stretchy="false" id="S4.SS1.p2.7.m7.4.4.3.3.8" xref="S4.SS1.p2.7.m7.4.4.3.4.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.7.m7.4b"><apply id="S4.SS1.p2.7.m7.4.4.cmml" xref="S4.SS1.p2.7.m7.4.4"><eq id="S4.SS1.p2.7.m7.4.4.4.cmml" xref="S4.SS1.p2.7.m7.4.4.4"></eq><apply id="S4.SS1.p2.7.m7.4.4.5.cmml" xref="S4.SS1.p2.7.m7.4.4.5"><csymbol cd="ambiguous" id="S4.SS1.p2.7.m7.4.4.5.1.cmml" xref="S4.SS1.p2.7.m7.4.4.5">subscript</csymbol><ci id="S4.SS1.p2.7.m7.4.4.5.2.cmml" xref="S4.SS1.p2.7.m7.4.4.5.2">𝒬</ci><ci id="S4.SS1.p2.7.m7.4.4.5.3.cmml" xref="S4.SS1.p2.7.m7.4.4.5.3">𝐸</ci></apply><set id="S4.SS1.p2.7.m7.4.4.3.4.cmml" xref="S4.SS1.p2.7.m7.4.4.3.3"><apply id="S4.SS1.p2.7.m7.2.2.1.1.1.cmml" xref="S4.SS1.p2.7.m7.2.2.1.1.1"><times id="S4.SS1.p2.7.m7.2.2.1.1.1.1.cmml" xref="S4.SS1.p2.7.m7.2.2.1.1.1.1"></times><ci id="S4.SS1.p2.7.m7.2.2.1.1.1.2.cmml" xref="S4.SS1.p2.7.m7.2.2.1.1.1.2">𝑞</ci><apply id="S4.SS1.p2.7.m7.2.2.1.1.1.3.cmml" xref="S4.SS1.p2.7.m7.2.2.1.1.1.3"><csymbol cd="ambiguous" id="S4.SS1.p2.7.m7.2.2.1.1.1.3.1.cmml" xref="S4.SS1.p2.7.m7.2.2.1.1.1.3">subscript</csymbol><ci id="S4.SS1.p2.7.m7.2.2.1.1.1.3.2.cmml" xref="S4.SS1.p2.7.m7.2.2.1.1.1.3.2">𝑒</ci><cn type="integer" id="S4.SS1.p2.7.m7.2.2.1.1.1.3.3.cmml" xref="S4.SS1.p2.7.m7.2.2.1.1.1.3.3">1</cn></apply></apply><apply id="S4.SS1.p2.7.m7.3.3.2.2.2.cmml" xref="S4.SS1.p2.7.m7.3.3.2.2.2"><times id="S4.SS1.p2.7.m7.3.3.2.2.2.1.cmml" xref="S4.SS1.p2.7.m7.3.3.2.2.2.1"></times><ci id="S4.SS1.p2.7.m7.3.3.2.2.2.2.cmml" xref="S4.SS1.p2.7.m7.3.3.2.2.2.2">𝑞</ci><apply id="S4.SS1.p2.7.m7.3.3.2.2.2.3.cmml" xref="S4.SS1.p2.7.m7.3.3.2.2.2.3"><csymbol cd="ambiguous" id="S4.SS1.p2.7.m7.3.3.2.2.2.3.1.cmml" xref="S4.SS1.p2.7.m7.3.3.2.2.2.3">subscript</csymbol><ci id="S4.SS1.p2.7.m7.3.3.2.2.2.3.2.cmml" xref="S4.SS1.p2.7.m7.3.3.2.2.2.3.2">𝑒</ci><cn type="integer" id="S4.SS1.p2.7.m7.3.3.2.2.2.3.3.cmml" xref="S4.SS1.p2.7.m7.3.3.2.2.2.3.3">2</cn></apply></apply><ci id="S4.SS1.p2.7.m7.1.1.cmml" xref="S4.SS1.p2.7.m7.1.1">…</ci><apply id="S4.SS1.p2.7.m7.4.4.3.3.3.cmml" xref="S4.SS1.p2.7.m7.4.4.3.3.3"><times id="S4.SS1.p2.7.m7.4.4.3.3.3.1.cmml" xref="S4.SS1.p2.7.m7.4.4.3.3.3.1"></times><ci id="S4.SS1.p2.7.m7.4.4.3.3.3.2.cmml" xref="S4.SS1.p2.7.m7.4.4.3.3.3.2">𝑞</ci><apply id="S4.SS1.p2.7.m7.4.4.3.3.3.3.cmml" xref="S4.SS1.p2.7.m7.4.4.3.3.3.3"><csymbol cd="ambiguous" id="S4.SS1.p2.7.m7.4.4.3.3.3.3.1.cmml" xref="S4.SS1.p2.7.m7.4.4.3.3.3.3">subscript</csymbol><ci id="S4.SS1.p2.7.m7.4.4.3.3.3.3.2.cmml" xref="S4.SS1.p2.7.m7.4.4.3.3.3.3.2">𝑒</ci><ci id="S4.SS1.p2.7.m7.4.4.3.3.3.3.3.cmml" xref="S4.SS1.p2.7.m7.4.4.3.3.3.3.3">𝑁</ci></apply></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.7.m7.4c">\mathcal{Q}_{E}=\{qe_{1},qe_{2},...,qe_{N}\}</annotation></semantics></math> with each query embedding vector <math id="S4.SS1.p2.8.m8.1" class="ltx_Math" alttext="qe_{i}=E(q_{i})\in\mathbb{R}^{d}" display="inline"><semantics id="S4.SS1.p2.8.m8.1a"><mrow id="S4.SS1.p2.8.m8.1.1" xref="S4.SS1.p2.8.m8.1.1.cmml"><mrow id="S4.SS1.p2.8.m8.1.1.3" xref="S4.SS1.p2.8.m8.1.1.3.cmml"><mi id="S4.SS1.p2.8.m8.1.1.3.2" xref="S4.SS1.p2.8.m8.1.1.3.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="S4.SS1.p2.8.m8.1.1.3.1" xref="S4.SS1.p2.8.m8.1.1.3.1.cmml">​</mo><msub id="S4.SS1.p2.8.m8.1.1.3.3" xref="S4.SS1.p2.8.m8.1.1.3.3.cmml"><mi id="S4.SS1.p2.8.m8.1.1.3.3.2" xref="S4.SS1.p2.8.m8.1.1.3.3.2.cmml">e</mi><mi id="S4.SS1.p2.8.m8.1.1.3.3.3" xref="S4.SS1.p2.8.m8.1.1.3.3.3.cmml">i</mi></msub></mrow><mo id="S4.SS1.p2.8.m8.1.1.4" xref="S4.SS1.p2.8.m8.1.1.4.cmml">=</mo><mrow id="S4.SS1.p2.8.m8.1.1.1" xref="S4.SS1.p2.8.m8.1.1.1.cmml"><mi id="S4.SS1.p2.8.m8.1.1.1.3" xref="S4.SS1.p2.8.m8.1.1.1.3.cmml">E</mi><mo lspace="0em" rspace="0em" id="S4.SS1.p2.8.m8.1.1.1.2" xref="S4.SS1.p2.8.m8.1.1.1.2.cmml">​</mo><mrow id="S4.SS1.p2.8.m8.1.1.1.1.1" xref="S4.SS1.p2.8.m8.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS1.p2.8.m8.1.1.1.1.1.2" xref="S4.SS1.p2.8.m8.1.1.1.1.1.1.cmml">(</mo><msub id="S4.SS1.p2.8.m8.1.1.1.1.1.1" xref="S4.SS1.p2.8.m8.1.1.1.1.1.1.cmml"><mi id="S4.SS1.p2.8.m8.1.1.1.1.1.1.2" xref="S4.SS1.p2.8.m8.1.1.1.1.1.1.2.cmml">q</mi><mi id="S4.SS1.p2.8.m8.1.1.1.1.1.1.3" xref="S4.SS1.p2.8.m8.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S4.SS1.p2.8.m8.1.1.1.1.1.3" xref="S4.SS1.p2.8.m8.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S4.SS1.p2.8.m8.1.1.5" xref="S4.SS1.p2.8.m8.1.1.5.cmml">∈</mo><msup id="S4.SS1.p2.8.m8.1.1.6" xref="S4.SS1.p2.8.m8.1.1.6.cmml"><mi id="S4.SS1.p2.8.m8.1.1.6.2" xref="S4.SS1.p2.8.m8.1.1.6.2.cmml">ℝ</mi><mi id="S4.SS1.p2.8.m8.1.1.6.3" xref="S4.SS1.p2.8.m8.1.1.6.3.cmml">d</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.8.m8.1b"><apply id="S4.SS1.p2.8.m8.1.1.cmml" xref="S4.SS1.p2.8.m8.1.1"><and id="S4.SS1.p2.8.m8.1.1a.cmml" xref="S4.SS1.p2.8.m8.1.1"></and><apply id="S4.SS1.p2.8.m8.1.1b.cmml" xref="S4.SS1.p2.8.m8.1.1"><eq id="S4.SS1.p2.8.m8.1.1.4.cmml" xref="S4.SS1.p2.8.m8.1.1.4"></eq><apply id="S4.SS1.p2.8.m8.1.1.3.cmml" xref="S4.SS1.p2.8.m8.1.1.3"><times id="S4.SS1.p2.8.m8.1.1.3.1.cmml" xref="S4.SS1.p2.8.m8.1.1.3.1"></times><ci id="S4.SS1.p2.8.m8.1.1.3.2.cmml" xref="S4.SS1.p2.8.m8.1.1.3.2">𝑞</ci><apply id="S4.SS1.p2.8.m8.1.1.3.3.cmml" xref="S4.SS1.p2.8.m8.1.1.3.3"><csymbol cd="ambiguous" id="S4.SS1.p2.8.m8.1.1.3.3.1.cmml" xref="S4.SS1.p2.8.m8.1.1.3.3">subscript</csymbol><ci id="S4.SS1.p2.8.m8.1.1.3.3.2.cmml" xref="S4.SS1.p2.8.m8.1.1.3.3.2">𝑒</ci><ci id="S4.SS1.p2.8.m8.1.1.3.3.3.cmml" xref="S4.SS1.p2.8.m8.1.1.3.3.3">𝑖</ci></apply></apply><apply id="S4.SS1.p2.8.m8.1.1.1.cmml" xref="S4.SS1.p2.8.m8.1.1.1"><times id="S4.SS1.p2.8.m8.1.1.1.2.cmml" xref="S4.SS1.p2.8.m8.1.1.1.2"></times><ci id="S4.SS1.p2.8.m8.1.1.1.3.cmml" xref="S4.SS1.p2.8.m8.1.1.1.3">𝐸</ci><apply id="S4.SS1.p2.8.m8.1.1.1.1.1.1.cmml" xref="S4.SS1.p2.8.m8.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.SS1.p2.8.m8.1.1.1.1.1.1.1.cmml" xref="S4.SS1.p2.8.m8.1.1.1.1.1">subscript</csymbol><ci id="S4.SS1.p2.8.m8.1.1.1.1.1.1.2.cmml" xref="S4.SS1.p2.8.m8.1.1.1.1.1.1.2">𝑞</ci><ci id="S4.SS1.p2.8.m8.1.1.1.1.1.1.3.cmml" xref="S4.SS1.p2.8.m8.1.1.1.1.1.1.3">𝑖</ci></apply></apply></apply><apply id="S4.SS1.p2.8.m8.1.1c.cmml" xref="S4.SS1.p2.8.m8.1.1"><in id="S4.SS1.p2.8.m8.1.1.5.cmml" xref="S4.SS1.p2.8.m8.1.1.5"></in><share href="#S4.SS1.p2.8.m8.1.1.1.cmml" id="S4.SS1.p2.8.m8.1.1d.cmml" xref="S4.SS1.p2.8.m8.1.1"></share><apply id="S4.SS1.p2.8.m8.1.1.6.cmml" xref="S4.SS1.p2.8.m8.1.1.6"><csymbol cd="ambiguous" id="S4.SS1.p2.8.m8.1.1.6.1.cmml" xref="S4.SS1.p2.8.m8.1.1.6">superscript</csymbol><ci id="S4.SS1.p2.8.m8.1.1.6.2.cmml" xref="S4.SS1.p2.8.m8.1.1.6.2">ℝ</ci><ci id="S4.SS1.p2.8.m8.1.1.6.3.cmml" xref="S4.SS1.p2.8.m8.1.1.6.3">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.8.m8.1c">qe_{i}=E(q_{i})\in\mathbb{R}^{d}</annotation></semantics></math>.
Furthermore, we define Matryoshka query embeddings as <math id="S4.SS1.p2.9.m9.1" class="ltx_math_unparsed" alttext="\mathcal{Q}_{E}[:m]=\{qe_{1}[:m],qe_{2}[:m],...,qe_{N}[:m]\}" display="inline"><semantics id="S4.SS1.p2.9.m9.1a"><mrow id="S4.SS1.p2.9.m9.1b"><msub id="S4.SS1.p2.9.m9.1.1"><mi class="ltx_font_mathcaligraphic" id="S4.SS1.p2.9.m9.1.1.2">𝒬</mi><mi id="S4.SS1.p2.9.m9.1.1.3">E</mi></msub><mrow id="S4.SS1.p2.9.m9.1.2"><mo stretchy="false" id="S4.SS1.p2.9.m9.1.2.1">[</mo><mo rspace="0.278em" id="S4.SS1.p2.9.m9.1.2.2">:</mo><mi id="S4.SS1.p2.9.m9.1.2.3">m</mi><mo stretchy="false" id="S4.SS1.p2.9.m9.1.2.4">]</mo></mrow><mo id="S4.SS1.p2.9.m9.1.3">=</mo><mrow id="S4.SS1.p2.9.m9.1.4"><mo stretchy="false" id="S4.SS1.p2.9.m9.1.4.1">{</mo><mi id="S4.SS1.p2.9.m9.1.4.2">q</mi><msub id="S4.SS1.p2.9.m9.1.4.3"><mi id="S4.SS1.p2.9.m9.1.4.3.2">e</mi><mn id="S4.SS1.p2.9.m9.1.4.3.3">1</mn></msub><mrow id="S4.SS1.p2.9.m9.1.4.4"><mo stretchy="false" id="S4.SS1.p2.9.m9.1.4.4.1">[</mo><mo rspace="0.278em" id="S4.SS1.p2.9.m9.1.4.4.2">:</mo><mi id="S4.SS1.p2.9.m9.1.4.4.3">m</mi><mo stretchy="false" id="S4.SS1.p2.9.m9.1.4.4.4">]</mo></mrow><mo id="S4.SS1.p2.9.m9.1.4.5">,</mo><mi id="S4.SS1.p2.9.m9.1.4.6">q</mi><msub id="S4.SS1.p2.9.m9.1.4.7"><mi id="S4.SS1.p2.9.m9.1.4.7.2">e</mi><mn id="S4.SS1.p2.9.m9.1.4.7.3">2</mn></msub><mrow id="S4.SS1.p2.9.m9.1.4.8"><mo stretchy="false" id="S4.SS1.p2.9.m9.1.4.8.1">[</mo><mo rspace="0.278em" id="S4.SS1.p2.9.m9.1.4.8.2">:</mo><mi id="S4.SS1.p2.9.m9.1.4.8.3">m</mi><mo stretchy="false" id="S4.SS1.p2.9.m9.1.4.8.4">]</mo></mrow><mo id="S4.SS1.p2.9.m9.1.4.9">,</mo><mi mathvariant="normal" id="S4.SS1.p2.9.m9.1.4.10">…</mi><mo id="S4.SS1.p2.9.m9.1.4.11">,</mo><mi id="S4.SS1.p2.9.m9.1.4.12">q</mi><msub id="S4.SS1.p2.9.m9.1.4.13"><mi id="S4.SS1.p2.9.m9.1.4.13.2">e</mi><mi id="S4.SS1.p2.9.m9.1.4.13.3">N</mi></msub><mrow id="S4.SS1.p2.9.m9.1.4.14"><mo stretchy="false" id="S4.SS1.p2.9.m9.1.4.14.1">[</mo><mo rspace="0.278em" id="S4.SS1.p2.9.m9.1.4.14.2">:</mo><mi id="S4.SS1.p2.9.m9.1.4.14.3">m</mi><mo stretchy="false" id="S4.SS1.p2.9.m9.1.4.14.4">]</mo></mrow><mo stretchy="false" id="S4.SS1.p2.9.m9.1.4.15">}</mo></mrow></mrow><annotation encoding="application/x-tex" id="S4.SS1.p2.9.m9.1c">\mathcal{Q}_{E}[:m]=\{qe_{1}[:m],qe_{2}[:m],...,qe_{N}[:m]\}</annotation></semantics></math>, where <math id="S4.SS1.p2.10.m10.1" class="ltx_math_unparsed" alttext="qe_{i}[:m]\in\mathbb{R}^{m}" display="inline"><semantics id="S4.SS1.p2.10.m10.1a"><mrow id="S4.SS1.p2.10.m10.1b"><mi id="S4.SS1.p2.10.m10.1.1">q</mi><msub id="S4.SS1.p2.10.m10.1.2"><mi id="S4.SS1.p2.10.m10.1.2.2">e</mi><mi id="S4.SS1.p2.10.m10.1.2.3">i</mi></msub><mrow id="S4.SS1.p2.10.m10.1.3"><mo stretchy="false" id="S4.SS1.p2.10.m10.1.3.1">[</mo><mo rspace="0.278em" id="S4.SS1.p2.10.m10.1.3.2">:</mo><mi id="S4.SS1.p2.10.m10.1.3.3">m</mi><mo stretchy="false" id="S4.SS1.p2.10.m10.1.3.4">]</mo></mrow><mo id="S4.SS1.p2.10.m10.1.4">∈</mo><msup id="S4.SS1.p2.10.m10.1.5"><mi id="S4.SS1.p2.10.m10.1.5.2">ℝ</mi><mi id="S4.SS1.p2.10.m10.1.5.3">m</mi></msup></mrow><annotation encoding="application/x-tex" id="S4.SS1.p2.10.m10.1c">qe_{i}[:m]\in\mathbb{R}^{m}</annotation></semantics></math>.
Note that <math id="S4.SS1.p2.11.m11.1" class="ltx_Math" alttext="m" display="inline"><semantics id="S4.SS1.p2.11.m11.1a"><mi id="S4.SS1.p2.11.m11.1.1" xref="S4.SS1.p2.11.m11.1.1.cmml">m</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p2.11.m11.1b"><ci id="S4.SS1.p2.11.m11.1.1.cmml" xref="S4.SS1.p2.11.m11.1.1">𝑚</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p2.11.m11.1c">m</annotation></semantics></math> can be any integer less than the original embedding dimension.</p>
</div>
<figure id="S4.F4" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S4.F4.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.20243/assets/x4.png" id="S4.F4.sf1.g1" class="ltx_graphics ltx_img_square" width="147" height="134" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S4.F4.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.20243/assets/x5.png" id="S4.F4.sf2.g1" class="ltx_graphics ltx_img_square" width="147" height="134" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S4.F4.sf3" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.20243/assets/x6.png" id="S4.F4.sf3.g1" class="ltx_graphics ltx_img_square" width="147" height="134" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(c) </span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>Experimental results of the unsupervised Matryoshka-Adaptor applied to three different embedding models: OpenAI text-embedding-3-large (with 3072 dimensions), OpenAI text-embedding-3-small (with 1536 dimensions), and Google multimodal (with 1408 dimensions).
Text embedding results were obtained using 8 BEIR datasets, while multimodal embedding results were obtained using 5 Fashion-200K datasets.</figcaption>
</figure>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Tuning objective functions</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.2" class="ltx_p">In the supervised setting, the Matryoshka-Adaptor (<math id="S4.SS2.p1.1.m1.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S4.SS2.p1.1.m1.1a"><mi id="S4.SS2.p1.1.m1.1.1" xref="S4.SS2.p1.1.m1.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.1.m1.1b"><ci id="S4.SS2.p1.1.m1.1.1.cmml" xref="S4.SS2.p1.1.m1.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.1.m1.1c">f</annotation></semantics></math>) undergoes optimization to enhance both its Matryoshka properties and the retrieval performance of the Matryoshka embeddings.
This is accomplished through the utilization of paired query-corpus samples, in conjunction with the original query and corpus embeddings.
Matryoshka ranking loss, denoted as <math id="S4.SS2.p1.2.m2.1" class="ltx_Math" alttext="\mathcal{L}_{rank}" display="inline"><semantics id="S4.SS2.p1.2.m2.1a"><msub id="S4.SS2.p1.2.m2.1.1" xref="S4.SS2.p1.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS2.p1.2.m2.1.1.2" xref="S4.SS2.p1.2.m2.1.1.2.cmml">ℒ</mi><mrow id="S4.SS2.p1.2.m2.1.1.3" xref="S4.SS2.p1.2.m2.1.1.3.cmml"><mi id="S4.SS2.p1.2.m2.1.1.3.2" xref="S4.SS2.p1.2.m2.1.1.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.2.m2.1.1.3.1" xref="S4.SS2.p1.2.m2.1.1.3.1.cmml">​</mo><mi id="S4.SS2.p1.2.m2.1.1.3.3" xref="S4.SS2.p1.2.m2.1.1.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.2.m2.1.1.3.1a" xref="S4.SS2.p1.2.m2.1.1.3.1.cmml">​</mo><mi id="S4.SS2.p1.2.m2.1.1.3.4" xref="S4.SS2.p1.2.m2.1.1.3.4.cmml">n</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.2.m2.1.1.3.1b" xref="S4.SS2.p1.2.m2.1.1.3.1.cmml">​</mo><mi id="S4.SS2.p1.2.m2.1.1.3.5" xref="S4.SS2.p1.2.m2.1.1.3.5.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.2.m2.1b"><apply id="S4.SS2.p1.2.m2.1.1.cmml" xref="S4.SS2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.2.m2.1.1.1.cmml" xref="S4.SS2.p1.2.m2.1.1">subscript</csymbol><ci id="S4.SS2.p1.2.m2.1.1.2.cmml" xref="S4.SS2.p1.2.m2.1.1.2">ℒ</ci><apply id="S4.SS2.p1.2.m2.1.1.3.cmml" xref="S4.SS2.p1.2.m2.1.1.3"><times id="S4.SS2.p1.2.m2.1.1.3.1.cmml" xref="S4.SS2.p1.2.m2.1.1.3.1"></times><ci id="S4.SS2.p1.2.m2.1.1.3.2.cmml" xref="S4.SS2.p1.2.m2.1.1.3.2">𝑟</ci><ci id="S4.SS2.p1.2.m2.1.1.3.3.cmml" xref="S4.SS2.p1.2.m2.1.1.3.3">𝑎</ci><ci id="S4.SS2.p1.2.m2.1.1.3.4.cmml" xref="S4.SS2.p1.2.m2.1.1.3.4">𝑛</ci><ci id="S4.SS2.p1.2.m2.1.1.3.5.cmml" xref="S4.SS2.p1.2.m2.1.1.3.5">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.2.m2.1c">\mathcal{L}_{rank}</annotation></semantics></math>, is introduced to align the ranking between query and corpus considering different Matryoshka embedding dimensions:</p>
<table id="A6.EGx4" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S4.E5"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S4.E5.m1.1" class="ltx_Math" alttext="\displaystyle\mathcal{L}_{rank}" display="inline"><semantics id="S4.E5.m1.1a"><msub id="S4.E5.m1.1.1" xref="S4.E5.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.E5.m1.1.1.2" xref="S4.E5.m1.1.1.2.cmml">ℒ</mi><mrow id="S4.E5.m1.1.1.3" xref="S4.E5.m1.1.1.3.cmml"><mi id="S4.E5.m1.1.1.3.2" xref="S4.E5.m1.1.1.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="S4.E5.m1.1.1.3.1" xref="S4.E5.m1.1.1.3.1.cmml">​</mo><mi id="S4.E5.m1.1.1.3.3" xref="S4.E5.m1.1.1.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S4.E5.m1.1.1.3.1a" xref="S4.E5.m1.1.1.3.1.cmml">​</mo><mi id="S4.E5.m1.1.1.3.4" xref="S4.E5.m1.1.1.3.4.cmml">n</mi><mo lspace="0em" rspace="0em" id="S4.E5.m1.1.1.3.1b" xref="S4.E5.m1.1.1.3.1.cmml">​</mo><mi id="S4.E5.m1.1.1.3.5" xref="S4.E5.m1.1.1.3.5.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S4.E5.m1.1b"><apply id="S4.E5.m1.1.1.cmml" xref="S4.E5.m1.1.1"><csymbol cd="ambiguous" id="S4.E5.m1.1.1.1.cmml" xref="S4.E5.m1.1.1">subscript</csymbol><ci id="S4.E5.m1.1.1.2.cmml" xref="S4.E5.m1.1.1.2">ℒ</ci><apply id="S4.E5.m1.1.1.3.cmml" xref="S4.E5.m1.1.1.3"><times id="S4.E5.m1.1.1.3.1.cmml" xref="S4.E5.m1.1.1.3.1"></times><ci id="S4.E5.m1.1.1.3.2.cmml" xref="S4.E5.m1.1.1.3.2">𝑟</ci><ci id="S4.E5.m1.1.1.3.3.cmml" xref="S4.E5.m1.1.1.3.3">𝑎</ci><ci id="S4.E5.m1.1.1.3.4.cmml" xref="S4.E5.m1.1.1.3.4">𝑛</ci><ci id="S4.E5.m1.1.1.3.5.cmml" xref="S4.E5.m1.1.1.3.5">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E5.m1.1c">\displaystyle\mathcal{L}_{rank}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S4.E5.m2.1" class="ltx_Math" alttext="\displaystyle=\sum_{i}\sum_{j}\sum_{k}\sum_{m}I(y_{ij}&gt;y_{ik})" display="inline"><semantics id="S4.E5.m2.1a"><mrow id="S4.E5.m2.1.1" xref="S4.E5.m2.1.1.cmml"><mi id="S4.E5.m2.1.1.3" xref="S4.E5.m2.1.1.3.cmml"></mi><mo id="S4.E5.m2.1.1.2" xref="S4.E5.m2.1.1.2.cmml">=</mo><mrow id="S4.E5.m2.1.1.1" xref="S4.E5.m2.1.1.1.cmml"><mstyle displaystyle="true" id="S4.E5.m2.1.1.1.2" xref="S4.E5.m2.1.1.1.2.cmml"><munder id="S4.E5.m2.1.1.1.2a" xref="S4.E5.m2.1.1.1.2.cmml"><mo movablelimits="false" id="S4.E5.m2.1.1.1.2.2" xref="S4.E5.m2.1.1.1.2.2.cmml">∑</mo><mi id="S4.E5.m2.1.1.1.2.3" xref="S4.E5.m2.1.1.1.2.3.cmml">i</mi></munder></mstyle><mrow id="S4.E5.m2.1.1.1.1" xref="S4.E5.m2.1.1.1.1.cmml"><mstyle displaystyle="true" id="S4.E5.m2.1.1.1.1.2" xref="S4.E5.m2.1.1.1.1.2.cmml"><munder id="S4.E5.m2.1.1.1.1.2a" xref="S4.E5.m2.1.1.1.1.2.cmml"><mo movablelimits="false" id="S4.E5.m2.1.1.1.1.2.2" xref="S4.E5.m2.1.1.1.1.2.2.cmml">∑</mo><mi id="S4.E5.m2.1.1.1.1.2.3" xref="S4.E5.m2.1.1.1.1.2.3.cmml">j</mi></munder></mstyle><mrow id="S4.E5.m2.1.1.1.1.1" xref="S4.E5.m2.1.1.1.1.1.cmml"><mstyle displaystyle="true" id="S4.E5.m2.1.1.1.1.1.2" xref="S4.E5.m2.1.1.1.1.1.2.cmml"><munder id="S4.E5.m2.1.1.1.1.1.2a" xref="S4.E5.m2.1.1.1.1.1.2.cmml"><mo movablelimits="false" id="S4.E5.m2.1.1.1.1.1.2.2" xref="S4.E5.m2.1.1.1.1.1.2.2.cmml">∑</mo><mi id="S4.E5.m2.1.1.1.1.1.2.3" xref="S4.E5.m2.1.1.1.1.1.2.3.cmml">k</mi></munder></mstyle><mrow id="S4.E5.m2.1.1.1.1.1.1" xref="S4.E5.m2.1.1.1.1.1.1.cmml"><mstyle displaystyle="true" id="S4.E5.m2.1.1.1.1.1.1.2" xref="S4.E5.m2.1.1.1.1.1.1.2.cmml"><munder id="S4.E5.m2.1.1.1.1.1.1.2a" xref="S4.E5.m2.1.1.1.1.1.1.2.cmml"><mo movablelimits="false" id="S4.E5.m2.1.1.1.1.1.1.2.2" xref="S4.E5.m2.1.1.1.1.1.1.2.2.cmml">∑</mo><mi id="S4.E5.m2.1.1.1.1.1.1.2.3" xref="S4.E5.m2.1.1.1.1.1.1.2.3.cmml">m</mi></munder></mstyle><mrow id="S4.E5.m2.1.1.1.1.1.1.1" xref="S4.E5.m2.1.1.1.1.1.1.1.cmml"><mi id="S4.E5.m2.1.1.1.1.1.1.1.3" xref="S4.E5.m2.1.1.1.1.1.1.1.3.cmml">I</mi><mo lspace="0em" rspace="0em" id="S4.E5.m2.1.1.1.1.1.1.1.2" xref="S4.E5.m2.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S4.E5.m2.1.1.1.1.1.1.1.1.1" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E5.m2.1.1.1.1.1.1.1.1.1.2" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.cmml"><msub id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.2" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.2.cmml">y</mi><mrow id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.3" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.3.cmml"><mi id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.3.2" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.3.1" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.3.1.cmml">​</mo><mi id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.3.3" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.3.3.cmml">j</mi></mrow></msub><mo id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.1" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.1.cmml">&gt;</mo><msub id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.2" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.2.cmml">y</mi><mrow id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.3" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.3.cmml"><mi id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.3.2" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.cmml">i</mi><mo lspace="0em" rspace="0em" id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.3.1" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.3.1.cmml">​</mo><mi id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.3.3" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.3.3.cmml">k</mi></mrow></msub></mrow><mo stretchy="false" id="S4.E5.m2.1.1.1.1.1.1.1.1.1.3" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E5.m2.1b"><apply id="S4.E5.m2.1.1.cmml" xref="S4.E5.m2.1.1"><eq id="S4.E5.m2.1.1.2.cmml" xref="S4.E5.m2.1.1.2"></eq><csymbol cd="latexml" id="S4.E5.m2.1.1.3.cmml" xref="S4.E5.m2.1.1.3">absent</csymbol><apply id="S4.E5.m2.1.1.1.cmml" xref="S4.E5.m2.1.1.1"><apply id="S4.E5.m2.1.1.1.2.cmml" xref="S4.E5.m2.1.1.1.2"><csymbol cd="ambiguous" id="S4.E5.m2.1.1.1.2.1.cmml" xref="S4.E5.m2.1.1.1.2">subscript</csymbol><sum id="S4.E5.m2.1.1.1.2.2.cmml" xref="S4.E5.m2.1.1.1.2.2"></sum><ci id="S4.E5.m2.1.1.1.2.3.cmml" xref="S4.E5.m2.1.1.1.2.3">𝑖</ci></apply><apply id="S4.E5.m2.1.1.1.1.cmml" xref="S4.E5.m2.1.1.1.1"><apply id="S4.E5.m2.1.1.1.1.2.cmml" xref="S4.E5.m2.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.E5.m2.1.1.1.1.2.1.cmml" xref="S4.E5.m2.1.1.1.1.2">subscript</csymbol><sum id="S4.E5.m2.1.1.1.1.2.2.cmml" xref="S4.E5.m2.1.1.1.1.2.2"></sum><ci id="S4.E5.m2.1.1.1.1.2.3.cmml" xref="S4.E5.m2.1.1.1.1.2.3">𝑗</ci></apply><apply id="S4.E5.m2.1.1.1.1.1.cmml" xref="S4.E5.m2.1.1.1.1.1"><apply id="S4.E5.m2.1.1.1.1.1.2.cmml" xref="S4.E5.m2.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.E5.m2.1.1.1.1.1.2.1.cmml" xref="S4.E5.m2.1.1.1.1.1.2">subscript</csymbol><sum id="S4.E5.m2.1.1.1.1.1.2.2.cmml" xref="S4.E5.m2.1.1.1.1.1.2.2"></sum><ci id="S4.E5.m2.1.1.1.1.1.2.3.cmml" xref="S4.E5.m2.1.1.1.1.1.2.3">𝑘</ci></apply><apply id="S4.E5.m2.1.1.1.1.1.1.cmml" xref="S4.E5.m2.1.1.1.1.1.1"><apply id="S4.E5.m2.1.1.1.1.1.1.2.cmml" xref="S4.E5.m2.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.E5.m2.1.1.1.1.1.1.2.1.cmml" xref="S4.E5.m2.1.1.1.1.1.1.2">subscript</csymbol><sum id="S4.E5.m2.1.1.1.1.1.1.2.2.cmml" xref="S4.E5.m2.1.1.1.1.1.1.2.2"></sum><ci id="S4.E5.m2.1.1.1.1.1.1.2.3.cmml" xref="S4.E5.m2.1.1.1.1.1.1.2.3">𝑚</ci></apply><apply id="S4.E5.m2.1.1.1.1.1.1.1.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1"><times id="S4.E5.m2.1.1.1.1.1.1.1.2.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1.2"></times><ci id="S4.E5.m2.1.1.1.1.1.1.1.3.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1.3">𝐼</ci><apply id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1"><gt id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.1"></gt><apply id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.2">𝑦</ci><apply id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.3"><times id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.3.1.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.3.1"></times><ci id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.3.2.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.3.2">𝑖</ci><ci id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.3.3.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.2.3.3">𝑗</ci></apply></apply><apply id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.2">𝑦</ci><apply id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.3"><times id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.3.1.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.3.1"></times><ci id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.3.2.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.3.2">𝑖</ci><ci id="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.3.3.cmml" xref="S4.E5.m2.1.1.1.1.1.1.1.1.1.1.3.3.3">𝑘</ci></apply></apply></apply></apply></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E5.m2.1c">\displaystyle=\sum_{i}\sum_{j}\sum_{k}\sum_{m}I(y_{ij}&gt;y_{ik})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(5)</span></td>
</tr></tbody>
<tbody id="S4.Ex3"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S4.Ex3.m1.1" class="ltx_math_unparsed" alttext="\displaystyle(y_{ij}-y_{ik})\log(1+\exp(s_{ik}[:m]-s_{ij}[:m]))," display="inline"><semantics id="S4.Ex3.m1.1a"><mrow id="S4.Ex3.m1.1b"><mrow id="S4.Ex3.m1.1.1"><mo stretchy="false" id="S4.Ex3.m1.1.1.1">(</mo><msub id="S4.Ex3.m1.1.1.2"><mi id="S4.Ex3.m1.1.1.2.2">y</mi><mrow id="S4.Ex3.m1.1.1.2.3"><mi id="S4.Ex3.m1.1.1.2.3.2">i</mi><mo lspace="0em" rspace="0em" id="S4.Ex3.m1.1.1.2.3.1">​</mo><mi id="S4.Ex3.m1.1.1.2.3.3">j</mi></mrow></msub><mo id="S4.Ex3.m1.1.1.3">−</mo><msub id="S4.Ex3.m1.1.1.4"><mi id="S4.Ex3.m1.1.1.4.2">y</mi><mrow id="S4.Ex3.m1.1.1.4.3"><mi id="S4.Ex3.m1.1.1.4.3.2">i</mi><mo lspace="0em" rspace="0em" id="S4.Ex3.m1.1.1.4.3.1">​</mo><mi id="S4.Ex3.m1.1.1.4.3.3">k</mi></mrow></msub><mo rspace="0.167em" stretchy="false" id="S4.Ex3.m1.1.1.5">)</mo></mrow><mi id="S4.Ex3.m1.1.2">log</mi><mrow id="S4.Ex3.m1.1.3"><mo stretchy="false" id="S4.Ex3.m1.1.3.1">(</mo><mn id="S4.Ex3.m1.1.3.2">1</mn><mo id="S4.Ex3.m1.1.3.3">+</mo><mi id="S4.Ex3.m1.1.3.4">exp</mi><mrow id="S4.Ex3.m1.1.3.5"><mo stretchy="false" id="S4.Ex3.m1.1.3.5.1">(</mo><msub id="S4.Ex3.m1.1.3.5.2"><mi id="S4.Ex3.m1.1.3.5.2.2">s</mi><mrow id="S4.Ex3.m1.1.3.5.2.3"><mi id="S4.Ex3.m1.1.3.5.2.3.2">i</mi><mo lspace="0em" rspace="0em" id="S4.Ex3.m1.1.3.5.2.3.1">​</mo><mi id="S4.Ex3.m1.1.3.5.2.3.3">k</mi></mrow></msub><mrow id="S4.Ex3.m1.1.3.5.3"><mo stretchy="false" id="S4.Ex3.m1.1.3.5.3.1">[</mo><mo rspace="0.278em" id="S4.Ex3.m1.1.3.5.3.2">:</mo><mi id="S4.Ex3.m1.1.3.5.3.3">m</mi><mo stretchy="false" id="S4.Ex3.m1.1.3.5.3.4">]</mo></mrow><mo id="S4.Ex3.m1.1.3.5.4">−</mo><msub id="S4.Ex3.m1.1.3.5.5"><mi id="S4.Ex3.m1.1.3.5.5.2">s</mi><mrow id="S4.Ex3.m1.1.3.5.5.3"><mi id="S4.Ex3.m1.1.3.5.5.3.2">i</mi><mo lspace="0em" rspace="0em" id="S4.Ex3.m1.1.3.5.5.3.1">​</mo><mi id="S4.Ex3.m1.1.3.5.5.3.3">j</mi></mrow></msub><mrow id="S4.Ex3.m1.1.3.5.6"><mo stretchy="false" id="S4.Ex3.m1.1.3.5.6.1">[</mo><mo rspace="0.278em" id="S4.Ex3.m1.1.3.5.6.2">:</mo><mi id="S4.Ex3.m1.1.3.5.6.3">m</mi><mo stretchy="false" id="S4.Ex3.m1.1.3.5.6.4">]</mo></mrow><mo stretchy="false" id="S4.Ex3.m1.1.3.5.7">)</mo></mrow><mo stretchy="false" id="S4.Ex3.m1.1.3.6">)</mo></mrow><mo id="S4.Ex3.m1.1.4">,</mo></mrow><annotation encoding="application/x-tex" id="S4.Ex3.m1.1c">\displaystyle(y_{ij}-y_{ik})\log(1+\exp(s_{ik}[:m]-s_{ij}[:m])),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S4.SS2.p1.7" class="ltx_p">where <math id="S4.SS2.p1.3.m1.1" class="ltx_math_unparsed" alttext="s_{ij}[:m]" display="inline"><semantics id="S4.SS2.p1.3.m1.1a"><mrow id="S4.SS2.p1.3.m1.1b"><msub id="S4.SS2.p1.3.m1.1.1"><mi id="S4.SS2.p1.3.m1.1.1.2">s</mi><mrow id="S4.SS2.p1.3.m1.1.1.3"><mi id="S4.SS2.p1.3.m1.1.1.3.2">i</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.3.m1.1.1.3.1">​</mo><mi id="S4.SS2.p1.3.m1.1.1.3.3">j</mi></mrow></msub><mrow id="S4.SS2.p1.3.m1.1.2"><mo stretchy="false" id="S4.SS2.p1.3.m1.1.2.1">[</mo><mo rspace="0.278em" id="S4.SS2.p1.3.m1.1.2.2">:</mo><mi id="S4.SS2.p1.3.m1.1.2.3">m</mi><mo stretchy="false" id="S4.SS2.p1.3.m1.1.2.4">]</mo></mrow></mrow><annotation encoding="application/x-tex" id="S4.SS2.p1.3.m1.1c">s_{ij}[:m]</annotation></semantics></math> represents the cosine similarity between the adapted query embedding <math id="S4.SS2.p1.4.m2.1" class="ltx_math_unparsed" alttext="\hat{qe}_{i}[:m]" display="inline"><semantics id="S4.SS2.p1.4.m2.1a"><mrow id="S4.SS2.p1.4.m2.1b"><msub id="S4.SS2.p1.4.m2.1.1"><mover accent="true" id="S4.SS2.p1.4.m2.1.1.2"><mrow id="S4.SS2.p1.4.m2.1.1.2.2"><mi id="S4.SS2.p1.4.m2.1.1.2.2.2">q</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.4.m2.1.1.2.2.1">​</mo><mi id="S4.SS2.p1.4.m2.1.1.2.2.3">e</mi></mrow><mo id="S4.SS2.p1.4.m2.1.1.2.1">^</mo></mover><mi id="S4.SS2.p1.4.m2.1.1.3">i</mi></msub><mrow id="S4.SS2.p1.4.m2.1.2"><mo stretchy="false" id="S4.SS2.p1.4.m2.1.2.1">[</mo><mo rspace="0.278em" id="S4.SS2.p1.4.m2.1.2.2">:</mo><mi id="S4.SS2.p1.4.m2.1.2.3">m</mi><mo stretchy="false" id="S4.SS2.p1.4.m2.1.2.4">]</mo></mrow></mrow><annotation encoding="application/x-tex" id="S4.SS2.p1.4.m2.1c">\hat{qe}_{i}[:m]</annotation></semantics></math> (where <math id="S4.SS2.p1.5.m3.1" class="ltx_Math" alttext="\hat{qe}_{i}=qe_{i}+f(qe_{i})" display="inline"><semantics id="S4.SS2.p1.5.m3.1a"><mrow id="S4.SS2.p1.5.m3.1.1" xref="S4.SS2.p1.5.m3.1.1.cmml"><msub id="S4.SS2.p1.5.m3.1.1.3" xref="S4.SS2.p1.5.m3.1.1.3.cmml"><mover accent="true" id="S4.SS2.p1.5.m3.1.1.3.2" xref="S4.SS2.p1.5.m3.1.1.3.2.cmml"><mrow id="S4.SS2.p1.5.m3.1.1.3.2.2" xref="S4.SS2.p1.5.m3.1.1.3.2.2.cmml"><mi id="S4.SS2.p1.5.m3.1.1.3.2.2.2" xref="S4.SS2.p1.5.m3.1.1.3.2.2.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.5.m3.1.1.3.2.2.1" xref="S4.SS2.p1.5.m3.1.1.3.2.2.1.cmml">​</mo><mi id="S4.SS2.p1.5.m3.1.1.3.2.2.3" xref="S4.SS2.p1.5.m3.1.1.3.2.2.3.cmml">e</mi></mrow><mo id="S4.SS2.p1.5.m3.1.1.3.2.1" xref="S4.SS2.p1.5.m3.1.1.3.2.1.cmml">^</mo></mover><mi id="S4.SS2.p1.5.m3.1.1.3.3" xref="S4.SS2.p1.5.m3.1.1.3.3.cmml">i</mi></msub><mo id="S4.SS2.p1.5.m3.1.1.2" xref="S4.SS2.p1.5.m3.1.1.2.cmml">=</mo><mrow id="S4.SS2.p1.5.m3.1.1.1" xref="S4.SS2.p1.5.m3.1.1.1.cmml"><mrow id="S4.SS2.p1.5.m3.1.1.1.3" xref="S4.SS2.p1.5.m3.1.1.1.3.cmml"><mi id="S4.SS2.p1.5.m3.1.1.1.3.2" xref="S4.SS2.p1.5.m3.1.1.1.3.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.5.m3.1.1.1.3.1" xref="S4.SS2.p1.5.m3.1.1.1.3.1.cmml">​</mo><msub id="S4.SS2.p1.5.m3.1.1.1.3.3" xref="S4.SS2.p1.5.m3.1.1.1.3.3.cmml"><mi id="S4.SS2.p1.5.m3.1.1.1.3.3.2" xref="S4.SS2.p1.5.m3.1.1.1.3.3.2.cmml">e</mi><mi id="S4.SS2.p1.5.m3.1.1.1.3.3.3" xref="S4.SS2.p1.5.m3.1.1.1.3.3.3.cmml">i</mi></msub></mrow><mo id="S4.SS2.p1.5.m3.1.1.1.2" xref="S4.SS2.p1.5.m3.1.1.1.2.cmml">+</mo><mrow id="S4.SS2.p1.5.m3.1.1.1.1" xref="S4.SS2.p1.5.m3.1.1.1.1.cmml"><mi id="S4.SS2.p1.5.m3.1.1.1.1.3" xref="S4.SS2.p1.5.m3.1.1.1.1.3.cmml">f</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.5.m3.1.1.1.1.2" xref="S4.SS2.p1.5.m3.1.1.1.1.2.cmml">​</mo><mrow id="S4.SS2.p1.5.m3.1.1.1.1.1.1" xref="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS2.p1.5.m3.1.1.1.1.1.1.2" xref="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.SS2.p1.5.m3.1.1.1.1.1.1.1" xref="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.cmml"><mi id="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.2" xref="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.2.cmml">q</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.1" xref="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.1.cmml">​</mo><msub id="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.3" xref="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.3.cmml"><mi id="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.3.2" xref="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.3.2.cmml">e</mi><mi id="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.3.3" xref="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.3.3.cmml">i</mi></msub></mrow><mo stretchy="false" id="S4.SS2.p1.5.m3.1.1.1.1.1.1.3" xref="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.5.m3.1b"><apply id="S4.SS2.p1.5.m3.1.1.cmml" xref="S4.SS2.p1.5.m3.1.1"><eq id="S4.SS2.p1.5.m3.1.1.2.cmml" xref="S4.SS2.p1.5.m3.1.1.2"></eq><apply id="S4.SS2.p1.5.m3.1.1.3.cmml" xref="S4.SS2.p1.5.m3.1.1.3"><csymbol cd="ambiguous" id="S4.SS2.p1.5.m3.1.1.3.1.cmml" xref="S4.SS2.p1.5.m3.1.1.3">subscript</csymbol><apply id="S4.SS2.p1.5.m3.1.1.3.2.cmml" xref="S4.SS2.p1.5.m3.1.1.3.2"><ci id="S4.SS2.p1.5.m3.1.1.3.2.1.cmml" xref="S4.SS2.p1.5.m3.1.1.3.2.1">^</ci><apply id="S4.SS2.p1.5.m3.1.1.3.2.2.cmml" xref="S4.SS2.p1.5.m3.1.1.3.2.2"><times id="S4.SS2.p1.5.m3.1.1.3.2.2.1.cmml" xref="S4.SS2.p1.5.m3.1.1.3.2.2.1"></times><ci id="S4.SS2.p1.5.m3.1.1.3.2.2.2.cmml" xref="S4.SS2.p1.5.m3.1.1.3.2.2.2">𝑞</ci><ci id="S4.SS2.p1.5.m3.1.1.3.2.2.3.cmml" xref="S4.SS2.p1.5.m3.1.1.3.2.2.3">𝑒</ci></apply></apply><ci id="S4.SS2.p1.5.m3.1.1.3.3.cmml" xref="S4.SS2.p1.5.m3.1.1.3.3">𝑖</ci></apply><apply id="S4.SS2.p1.5.m3.1.1.1.cmml" xref="S4.SS2.p1.5.m3.1.1.1"><plus id="S4.SS2.p1.5.m3.1.1.1.2.cmml" xref="S4.SS2.p1.5.m3.1.1.1.2"></plus><apply id="S4.SS2.p1.5.m3.1.1.1.3.cmml" xref="S4.SS2.p1.5.m3.1.1.1.3"><times id="S4.SS2.p1.5.m3.1.1.1.3.1.cmml" xref="S4.SS2.p1.5.m3.1.1.1.3.1"></times><ci id="S4.SS2.p1.5.m3.1.1.1.3.2.cmml" xref="S4.SS2.p1.5.m3.1.1.1.3.2">𝑞</ci><apply id="S4.SS2.p1.5.m3.1.1.1.3.3.cmml" xref="S4.SS2.p1.5.m3.1.1.1.3.3"><csymbol cd="ambiguous" id="S4.SS2.p1.5.m3.1.1.1.3.3.1.cmml" xref="S4.SS2.p1.5.m3.1.1.1.3.3">subscript</csymbol><ci id="S4.SS2.p1.5.m3.1.1.1.3.3.2.cmml" xref="S4.SS2.p1.5.m3.1.1.1.3.3.2">𝑒</ci><ci id="S4.SS2.p1.5.m3.1.1.1.3.3.3.cmml" xref="S4.SS2.p1.5.m3.1.1.1.3.3.3">𝑖</ci></apply></apply><apply id="S4.SS2.p1.5.m3.1.1.1.1.cmml" xref="S4.SS2.p1.5.m3.1.1.1.1"><times id="S4.SS2.p1.5.m3.1.1.1.1.2.cmml" xref="S4.SS2.p1.5.m3.1.1.1.1.2"></times><ci id="S4.SS2.p1.5.m3.1.1.1.1.3.cmml" xref="S4.SS2.p1.5.m3.1.1.1.1.3">𝑓</ci><apply id="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.cmml" xref="S4.SS2.p1.5.m3.1.1.1.1.1.1"><times id="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.1.cmml" xref="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.1"></times><ci id="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.2.cmml" xref="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.2">𝑞</ci><apply id="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.3.cmml" xref="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.3.1.cmml" xref="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.3.2.cmml" xref="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.3.2">𝑒</ci><ci id="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.3.3.cmml" xref="S4.SS2.p1.5.m3.1.1.1.1.1.1.1.3.3">𝑖</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.5.m3.1c">\hat{qe}_{i}=qe_{i}+f(qe_{i})</annotation></semantics></math>) and the adapted corpus embedding <math id="S4.SS2.p1.6.m4.1" class="ltx_math_unparsed" alttext="\hat{ce}_{j}[:m]" display="inline"><semantics id="S4.SS2.p1.6.m4.1a"><mrow id="S4.SS2.p1.6.m4.1b"><msub id="S4.SS2.p1.6.m4.1.1"><mover accent="true" id="S4.SS2.p1.6.m4.1.1.2"><mrow id="S4.SS2.p1.6.m4.1.1.2.2"><mi id="S4.SS2.p1.6.m4.1.1.2.2.2">c</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.6.m4.1.1.2.2.1">​</mo><mi id="S4.SS2.p1.6.m4.1.1.2.2.3">e</mi></mrow><mo id="S4.SS2.p1.6.m4.1.1.2.1">^</mo></mover><mi id="S4.SS2.p1.6.m4.1.1.3">j</mi></msub><mrow id="S4.SS2.p1.6.m4.1.2"><mo stretchy="false" id="S4.SS2.p1.6.m4.1.2.1">[</mo><mo rspace="0.278em" id="S4.SS2.p1.6.m4.1.2.2">:</mo><mi id="S4.SS2.p1.6.m4.1.2.3">m</mi><mo stretchy="false" id="S4.SS2.p1.6.m4.1.2.4">]</mo></mrow></mrow><annotation encoding="application/x-tex" id="S4.SS2.p1.6.m4.1c">\hat{ce}_{j}[:m]</annotation></semantics></math>.
We use the same adaptor (<math id="S4.SS2.p1.7.m5.1" class="ltx_Math" alttext="f" display="inline"><semantics id="S4.SS2.p1.7.m5.1a"><mi id="S4.SS2.p1.7.m5.1.1" xref="S4.SS2.p1.7.m5.1.1.cmml">f</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.7.m5.1b"><ci id="S4.SS2.p1.7.m5.1.1.cmml" xref="S4.SS2.p1.7.m5.1.1">𝑓</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.7.m5.1c">f</annotation></semantics></math>) for both query and corpus embeddings.
This ranking loss is crucial for effective learning of lower dimensional representations with their information content for the ranking objective being considered.</p>
</div>
<div id="S4.SS2.p2" class="ltx_para">
<p id="S4.SS2.p2.3" class="ltx_p">The supervised Matryoshka-Adaptor is trained using a joint objective function that encompasses the ranking loss as well as the unsupervised Matryoshka losses (<math id="S4.SS2.p2.1.m1.1" class="ltx_Math" alttext="\mathcal{L}_{topk}" display="inline"><semantics id="S4.SS2.p2.1.m1.1a"><msub id="S4.SS2.p2.1.m1.1.1" xref="S4.SS2.p2.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS2.p2.1.m1.1.1.2" xref="S4.SS2.p2.1.m1.1.1.2.cmml">ℒ</mi><mrow id="S4.SS2.p2.1.m1.1.1.3" xref="S4.SS2.p2.1.m1.1.1.3.cmml"><mi id="S4.SS2.p2.1.m1.1.1.3.2" xref="S4.SS2.p2.1.m1.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p2.1.m1.1.1.3.1" xref="S4.SS2.p2.1.m1.1.1.3.1.cmml">​</mo><mi id="S4.SS2.p2.1.m1.1.1.3.3" xref="S4.SS2.p2.1.m1.1.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p2.1.m1.1.1.3.1a" xref="S4.SS2.p2.1.m1.1.1.3.1.cmml">​</mo><mi id="S4.SS2.p2.1.m1.1.1.3.4" xref="S4.SS2.p2.1.m1.1.1.3.4.cmml">p</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p2.1.m1.1.1.3.1b" xref="S4.SS2.p2.1.m1.1.1.3.1.cmml">​</mo><mi id="S4.SS2.p2.1.m1.1.1.3.5" xref="S4.SS2.p2.1.m1.1.1.3.5.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.1.m1.1b"><apply id="S4.SS2.p2.1.m1.1.1.cmml" xref="S4.SS2.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S4.SS2.p2.1.m1.1.1.1.cmml" xref="S4.SS2.p2.1.m1.1.1">subscript</csymbol><ci id="S4.SS2.p2.1.m1.1.1.2.cmml" xref="S4.SS2.p2.1.m1.1.1.2">ℒ</ci><apply id="S4.SS2.p2.1.m1.1.1.3.cmml" xref="S4.SS2.p2.1.m1.1.1.3"><times id="S4.SS2.p2.1.m1.1.1.3.1.cmml" xref="S4.SS2.p2.1.m1.1.1.3.1"></times><ci id="S4.SS2.p2.1.m1.1.1.3.2.cmml" xref="S4.SS2.p2.1.m1.1.1.3.2">𝑡</ci><ci id="S4.SS2.p2.1.m1.1.1.3.3.cmml" xref="S4.SS2.p2.1.m1.1.1.3.3">𝑜</ci><ci id="S4.SS2.p2.1.m1.1.1.3.4.cmml" xref="S4.SS2.p2.1.m1.1.1.3.4">𝑝</ci><ci id="S4.SS2.p2.1.m1.1.1.3.5.cmml" xref="S4.SS2.p2.1.m1.1.1.3.5">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.1.m1.1c">\mathcal{L}_{topk}</annotation></semantics></math> , <math id="S4.SS2.p2.2.m2.1" class="ltx_Math" alttext="\mathcal{L}_{pair}" display="inline"><semantics id="S4.SS2.p2.2.m2.1a"><msub id="S4.SS2.p2.2.m2.1.1" xref="S4.SS2.p2.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS2.p2.2.m2.1.1.2" xref="S4.SS2.p2.2.m2.1.1.2.cmml">ℒ</mi><mrow id="S4.SS2.p2.2.m2.1.1.3" xref="S4.SS2.p2.2.m2.1.1.3.cmml"><mi id="S4.SS2.p2.2.m2.1.1.3.2" xref="S4.SS2.p2.2.m2.1.1.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p2.2.m2.1.1.3.1" xref="S4.SS2.p2.2.m2.1.1.3.1.cmml">​</mo><mi id="S4.SS2.p2.2.m2.1.1.3.3" xref="S4.SS2.p2.2.m2.1.1.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p2.2.m2.1.1.3.1a" xref="S4.SS2.p2.2.m2.1.1.3.1.cmml">​</mo><mi id="S4.SS2.p2.2.m2.1.1.3.4" xref="S4.SS2.p2.2.m2.1.1.3.4.cmml">i</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p2.2.m2.1.1.3.1b" xref="S4.SS2.p2.2.m2.1.1.3.1.cmml">​</mo><mi id="S4.SS2.p2.2.m2.1.1.3.5" xref="S4.SS2.p2.2.m2.1.1.3.5.cmml">r</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.2.m2.1b"><apply id="S4.SS2.p2.2.m2.1.1.cmml" xref="S4.SS2.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S4.SS2.p2.2.m2.1.1.1.cmml" xref="S4.SS2.p2.2.m2.1.1">subscript</csymbol><ci id="S4.SS2.p2.2.m2.1.1.2.cmml" xref="S4.SS2.p2.2.m2.1.1.2">ℒ</ci><apply id="S4.SS2.p2.2.m2.1.1.3.cmml" xref="S4.SS2.p2.2.m2.1.1.3"><times id="S4.SS2.p2.2.m2.1.1.3.1.cmml" xref="S4.SS2.p2.2.m2.1.1.3.1"></times><ci id="S4.SS2.p2.2.m2.1.1.3.2.cmml" xref="S4.SS2.p2.2.m2.1.1.3.2">𝑝</ci><ci id="S4.SS2.p2.2.m2.1.1.3.3.cmml" xref="S4.SS2.p2.2.m2.1.1.3.3">𝑎</ci><ci id="S4.SS2.p2.2.m2.1.1.3.4.cmml" xref="S4.SS2.p2.2.m2.1.1.3.4">𝑖</ci><ci id="S4.SS2.p2.2.m2.1.1.3.5.cmml" xref="S4.SS2.p2.2.m2.1.1.3.5">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.2.m2.1c">\mathcal{L}_{pair}</annotation></semantics></math>, and <math id="S4.SS2.p2.3.m3.1" class="ltx_Math" alttext="\mathcal{L}_{rec}" display="inline"><semantics id="S4.SS2.p2.3.m3.1a"><msub id="S4.SS2.p2.3.m3.1.1" xref="S4.SS2.p2.3.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS2.p2.3.m3.1.1.2" xref="S4.SS2.p2.3.m3.1.1.2.cmml">ℒ</mi><mrow id="S4.SS2.p2.3.m3.1.1.3" xref="S4.SS2.p2.3.m3.1.1.3.cmml"><mi id="S4.SS2.p2.3.m3.1.1.3.2" xref="S4.SS2.p2.3.m3.1.1.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p2.3.m3.1.1.3.1" xref="S4.SS2.p2.3.m3.1.1.3.1.cmml">​</mo><mi id="S4.SS2.p2.3.m3.1.1.3.3" xref="S4.SS2.p2.3.m3.1.1.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p2.3.m3.1.1.3.1a" xref="S4.SS2.p2.3.m3.1.1.3.1.cmml">​</mo><mi id="S4.SS2.p2.3.m3.1.1.3.4" xref="S4.SS2.p2.3.m3.1.1.3.4.cmml">c</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.3.m3.1b"><apply id="S4.SS2.p2.3.m3.1.1.cmml" xref="S4.SS2.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S4.SS2.p2.3.m3.1.1.1.cmml" xref="S4.SS2.p2.3.m3.1.1">subscript</csymbol><ci id="S4.SS2.p2.3.m3.1.1.2.cmml" xref="S4.SS2.p2.3.m3.1.1.2">ℒ</ci><apply id="S4.SS2.p2.3.m3.1.1.3.cmml" xref="S4.SS2.p2.3.m3.1.1.3"><times id="S4.SS2.p2.3.m3.1.1.3.1.cmml" xref="S4.SS2.p2.3.m3.1.1.3.1"></times><ci id="S4.SS2.p2.3.m3.1.1.3.2.cmml" xref="S4.SS2.p2.3.m3.1.1.3.2">𝑟</ci><ci id="S4.SS2.p2.3.m3.1.1.3.3.cmml" xref="S4.SS2.p2.3.m3.1.1.3.3">𝑒</ci><ci id="S4.SS2.p2.3.m3.1.1.3.4.cmml" xref="S4.SS2.p2.3.m3.1.1.3.4">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.3.m3.1c">\mathcal{L}_{rec}</annotation></semantics></math>).
This joint training approach aims to improve the quality of the embeddings while preserving their Matryoshka representations.
Query-corpus pairs are employed for the ranking loss, while query and corpus embeddings are utilized for the Matryoshka representation learning. The overall objective function is:</p>
<table id="A6.EGx5" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="S4.E6"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="S4.E6.m1.1" class="ltx_Math" alttext="\displaystyle\min_{f}" display="inline"><semantics id="S4.E6.m1.1a"><munder id="S4.E6.m1.1.1" xref="S4.E6.m1.1.1.cmml"><mi id="S4.E6.m1.1.1.2" xref="S4.E6.m1.1.1.2.cmml">min</mi><mi id="S4.E6.m1.1.1.3" xref="S4.E6.m1.1.1.3.cmml">f</mi></munder><annotation-xml encoding="MathML-Content" id="S4.E6.m1.1b"><apply id="S4.E6.m1.1.1.cmml" xref="S4.E6.m1.1.1"><csymbol cd="ambiguous" id="S4.E6.m1.1.1.1.cmml" xref="S4.E6.m1.1.1">subscript</csymbol><min id="S4.E6.m1.1.1.2.cmml" xref="S4.E6.m1.1.1.2"></min><ci id="S4.E6.m1.1.1.3.cmml" xref="S4.E6.m1.1.1.3">𝑓</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E6.m1.1c">\displaystyle\min_{f}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S4.E6.m2.2" class="ltx_Math" alttext="\displaystyle\mathcal{L}_{topk}(f)+\alpha\mathcal{L}_{pair}(f)+" display="inline"><semantics id="S4.E6.m2.2a"><mrow id="S4.E6.m2.2.3" xref="S4.E6.m2.2.3.cmml"><mrow id="S4.E6.m2.2.3.2" xref="S4.E6.m2.2.3.2.cmml"><msub id="S4.E6.m2.2.3.2.2" xref="S4.E6.m2.2.3.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.E6.m2.2.3.2.2.2" xref="S4.E6.m2.2.3.2.2.2.cmml">ℒ</mi><mrow id="S4.E6.m2.2.3.2.2.3" xref="S4.E6.m2.2.3.2.2.3.cmml"><mi id="S4.E6.m2.2.3.2.2.3.2" xref="S4.E6.m2.2.3.2.2.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S4.E6.m2.2.3.2.2.3.1" xref="S4.E6.m2.2.3.2.2.3.1.cmml">​</mo><mi id="S4.E6.m2.2.3.2.2.3.3" xref="S4.E6.m2.2.3.2.2.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S4.E6.m2.2.3.2.2.3.1a" xref="S4.E6.m2.2.3.2.2.3.1.cmml">​</mo><mi id="S4.E6.m2.2.3.2.2.3.4" xref="S4.E6.m2.2.3.2.2.3.4.cmml">p</mi><mo lspace="0em" rspace="0em" id="S4.E6.m2.2.3.2.2.3.1b" xref="S4.E6.m2.2.3.2.2.3.1.cmml">​</mo><mi id="S4.E6.m2.2.3.2.2.3.5" xref="S4.E6.m2.2.3.2.2.3.5.cmml">k</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S4.E6.m2.2.3.2.1" xref="S4.E6.m2.2.3.2.1.cmml">​</mo><mrow id="S4.E6.m2.2.3.2.3.2" xref="S4.E6.m2.2.3.2.cmml"><mo stretchy="false" id="S4.E6.m2.2.3.2.3.2.1" xref="S4.E6.m2.2.3.2.cmml">(</mo><mi id="S4.E6.m2.1.1" xref="S4.E6.m2.1.1.cmml">f</mi><mo stretchy="false" id="S4.E6.m2.2.3.2.3.2.2" xref="S4.E6.m2.2.3.2.cmml">)</mo></mrow></mrow><mo id="S4.E6.m2.2.3.1" xref="S4.E6.m2.2.3.1.cmml">+</mo><mrow id="S4.E6.m2.2.3.3" xref="S4.E6.m2.2.3.3.cmml"><mrow id="S4.E6.m2.2.3.3.2" xref="S4.E6.m2.2.3.3.2.cmml"><mi id="S4.E6.m2.2.3.3.2.2" xref="S4.E6.m2.2.3.3.2.2.cmml">α</mi><mo lspace="0em" rspace="0em" id="S4.E6.m2.2.3.3.2.1" xref="S4.E6.m2.2.3.3.2.1.cmml">​</mo><msub id="S4.E6.m2.2.3.3.2.3" xref="S4.E6.m2.2.3.3.2.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.E6.m2.2.3.3.2.3.2" xref="S4.E6.m2.2.3.3.2.3.2.cmml">ℒ</mi><mrow id="S4.E6.m2.2.3.3.2.3.3" xref="S4.E6.m2.2.3.3.2.3.3.cmml"><mi id="S4.E6.m2.2.3.3.2.3.3.2" xref="S4.E6.m2.2.3.3.2.3.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="S4.E6.m2.2.3.3.2.3.3.1" xref="S4.E6.m2.2.3.3.2.3.3.1.cmml">​</mo><mi id="S4.E6.m2.2.3.3.2.3.3.3" xref="S4.E6.m2.2.3.3.2.3.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S4.E6.m2.2.3.3.2.3.3.1a" xref="S4.E6.m2.2.3.3.2.3.3.1.cmml">​</mo><mi id="S4.E6.m2.2.3.3.2.3.3.4" xref="S4.E6.m2.2.3.3.2.3.3.4.cmml">i</mi><mo lspace="0em" rspace="0em" id="S4.E6.m2.2.3.3.2.3.3.1b" xref="S4.E6.m2.2.3.3.2.3.3.1.cmml">​</mo><mi id="S4.E6.m2.2.3.3.2.3.3.5" xref="S4.E6.m2.2.3.3.2.3.3.5.cmml">r</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S4.E6.m2.2.3.3.2.1a" xref="S4.E6.m2.2.3.3.2.1.cmml">​</mo><mrow id="S4.E6.m2.2.3.3.2.4.2" xref="S4.E6.m2.2.3.3.2.cmml"><mo stretchy="false" id="S4.E6.m2.2.3.3.2.4.2.1" xref="S4.E6.m2.2.3.3.2.cmml">(</mo><mi id="S4.E6.m2.2.2" xref="S4.E6.m2.2.2.cmml">f</mi><mo stretchy="false" id="S4.E6.m2.2.3.3.2.4.2.2" xref="S4.E6.m2.2.3.3.2.cmml">)</mo></mrow></mrow><mo id="S4.E6.m2.2.3.3.3" xref="S4.E6.m2.2.3.3.3.cmml">+</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.E6.m2.2b"><apply id="S4.E6.m2.2.3.cmml" xref="S4.E6.m2.2.3"><plus id="S4.E6.m2.2.3.1.cmml" xref="S4.E6.m2.2.3.1"></plus><apply id="S4.E6.m2.2.3.2.cmml" xref="S4.E6.m2.2.3.2"><times id="S4.E6.m2.2.3.2.1.cmml" xref="S4.E6.m2.2.3.2.1"></times><apply id="S4.E6.m2.2.3.2.2.cmml" xref="S4.E6.m2.2.3.2.2"><csymbol cd="ambiguous" id="S4.E6.m2.2.3.2.2.1.cmml" xref="S4.E6.m2.2.3.2.2">subscript</csymbol><ci id="S4.E6.m2.2.3.2.2.2.cmml" xref="S4.E6.m2.2.3.2.2.2">ℒ</ci><apply id="S4.E6.m2.2.3.2.2.3.cmml" xref="S4.E6.m2.2.3.2.2.3"><times id="S4.E6.m2.2.3.2.2.3.1.cmml" xref="S4.E6.m2.2.3.2.2.3.1"></times><ci id="S4.E6.m2.2.3.2.2.3.2.cmml" xref="S4.E6.m2.2.3.2.2.3.2">𝑡</ci><ci id="S4.E6.m2.2.3.2.2.3.3.cmml" xref="S4.E6.m2.2.3.2.2.3.3">𝑜</ci><ci id="S4.E6.m2.2.3.2.2.3.4.cmml" xref="S4.E6.m2.2.3.2.2.3.4">𝑝</ci><ci id="S4.E6.m2.2.3.2.2.3.5.cmml" xref="S4.E6.m2.2.3.2.2.3.5">𝑘</ci></apply></apply><ci id="S4.E6.m2.1.1.cmml" xref="S4.E6.m2.1.1">𝑓</ci></apply><apply id="S4.E6.m2.2.3.3.cmml" xref="S4.E6.m2.2.3.3"><csymbol cd="latexml" id="S4.E6.m2.2.3.3.1.cmml" xref="S4.E6.m2.2.3.3">limit-from</csymbol><apply id="S4.E6.m2.2.3.3.2.cmml" xref="S4.E6.m2.2.3.3.2"><times id="S4.E6.m2.2.3.3.2.1.cmml" xref="S4.E6.m2.2.3.3.2.1"></times><ci id="S4.E6.m2.2.3.3.2.2.cmml" xref="S4.E6.m2.2.3.3.2.2">𝛼</ci><apply id="S4.E6.m2.2.3.3.2.3.cmml" xref="S4.E6.m2.2.3.3.2.3"><csymbol cd="ambiguous" id="S4.E6.m2.2.3.3.2.3.1.cmml" xref="S4.E6.m2.2.3.3.2.3">subscript</csymbol><ci id="S4.E6.m2.2.3.3.2.3.2.cmml" xref="S4.E6.m2.2.3.3.2.3.2">ℒ</ci><apply id="S4.E6.m2.2.3.3.2.3.3.cmml" xref="S4.E6.m2.2.3.3.2.3.3"><times id="S4.E6.m2.2.3.3.2.3.3.1.cmml" xref="S4.E6.m2.2.3.3.2.3.3.1"></times><ci id="S4.E6.m2.2.3.3.2.3.3.2.cmml" xref="S4.E6.m2.2.3.3.2.3.3.2">𝑝</ci><ci id="S4.E6.m2.2.3.3.2.3.3.3.cmml" xref="S4.E6.m2.2.3.3.2.3.3.3">𝑎</ci><ci id="S4.E6.m2.2.3.3.2.3.3.4.cmml" xref="S4.E6.m2.2.3.3.2.3.3.4">𝑖</ci><ci id="S4.E6.m2.2.3.3.2.3.3.5.cmml" xref="S4.E6.m2.2.3.3.2.3.3.5">𝑟</ci></apply></apply><ci id="S4.E6.m2.2.2.cmml" xref="S4.E6.m2.2.2">𝑓</ci></apply><plus id="S4.E6.m2.2.3.3.3.cmml" xref="S4.E6.m2.2.3.3.3"></plus></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E6.m2.2c">\displaystyle\mathcal{L}_{topk}(f)+\alpha\mathcal{L}_{pair}(f)+</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(6)</span></td>
</tr></tbody>
<tbody id="S4.Ex4"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_eqn_cell"></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="S4.Ex4.m1.3" class="ltx_Math" alttext="\displaystyle\beta\mathcal{L}_{rec}(f)+\gamma\mathcal{L}_{rank}(f)," display="inline"><semantics id="S4.Ex4.m1.3a"><mrow id="S4.Ex4.m1.3.3.1" xref="S4.Ex4.m1.3.3.1.1.cmml"><mrow id="S4.Ex4.m1.3.3.1.1" xref="S4.Ex4.m1.3.3.1.1.cmml"><mrow id="S4.Ex4.m1.3.3.1.1.2" xref="S4.Ex4.m1.3.3.1.1.2.cmml"><mi id="S4.Ex4.m1.3.3.1.1.2.2" xref="S4.Ex4.m1.3.3.1.1.2.2.cmml">β</mi><mo lspace="0em" rspace="0em" id="S4.Ex4.m1.3.3.1.1.2.1" xref="S4.Ex4.m1.3.3.1.1.2.1.cmml">​</mo><msub id="S4.Ex4.m1.3.3.1.1.2.3" xref="S4.Ex4.m1.3.3.1.1.2.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.Ex4.m1.3.3.1.1.2.3.2" xref="S4.Ex4.m1.3.3.1.1.2.3.2.cmml">ℒ</mi><mrow id="S4.Ex4.m1.3.3.1.1.2.3.3" xref="S4.Ex4.m1.3.3.1.1.2.3.3.cmml"><mi id="S4.Ex4.m1.3.3.1.1.2.3.3.2" xref="S4.Ex4.m1.3.3.1.1.2.3.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="S4.Ex4.m1.3.3.1.1.2.3.3.1" xref="S4.Ex4.m1.3.3.1.1.2.3.3.1.cmml">​</mo><mi id="S4.Ex4.m1.3.3.1.1.2.3.3.3" xref="S4.Ex4.m1.3.3.1.1.2.3.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S4.Ex4.m1.3.3.1.1.2.3.3.1a" xref="S4.Ex4.m1.3.3.1.1.2.3.3.1.cmml">​</mo><mi id="S4.Ex4.m1.3.3.1.1.2.3.3.4" xref="S4.Ex4.m1.3.3.1.1.2.3.3.4.cmml">c</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S4.Ex4.m1.3.3.1.1.2.1a" xref="S4.Ex4.m1.3.3.1.1.2.1.cmml">​</mo><mrow id="S4.Ex4.m1.3.3.1.1.2.4.2" xref="S4.Ex4.m1.3.3.1.1.2.cmml"><mo stretchy="false" id="S4.Ex4.m1.3.3.1.1.2.4.2.1" xref="S4.Ex4.m1.3.3.1.1.2.cmml">(</mo><mi id="S4.Ex4.m1.1.1" xref="S4.Ex4.m1.1.1.cmml">f</mi><mo stretchy="false" id="S4.Ex4.m1.3.3.1.1.2.4.2.2" xref="S4.Ex4.m1.3.3.1.1.2.cmml">)</mo></mrow></mrow><mo id="S4.Ex4.m1.3.3.1.1.1" xref="S4.Ex4.m1.3.3.1.1.1.cmml">+</mo><mrow id="S4.Ex4.m1.3.3.1.1.3" xref="S4.Ex4.m1.3.3.1.1.3.cmml"><mi id="S4.Ex4.m1.3.3.1.1.3.2" xref="S4.Ex4.m1.3.3.1.1.3.2.cmml">γ</mi><mo lspace="0em" rspace="0em" id="S4.Ex4.m1.3.3.1.1.3.1" xref="S4.Ex4.m1.3.3.1.1.3.1.cmml">​</mo><msub id="S4.Ex4.m1.3.3.1.1.3.3" xref="S4.Ex4.m1.3.3.1.1.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.Ex4.m1.3.3.1.1.3.3.2" xref="S4.Ex4.m1.3.3.1.1.3.3.2.cmml">ℒ</mi><mrow id="S4.Ex4.m1.3.3.1.1.3.3.3" xref="S4.Ex4.m1.3.3.1.1.3.3.3.cmml"><mi id="S4.Ex4.m1.3.3.1.1.3.3.3.2" xref="S4.Ex4.m1.3.3.1.1.3.3.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="S4.Ex4.m1.3.3.1.1.3.3.3.1" xref="S4.Ex4.m1.3.3.1.1.3.3.3.1.cmml">​</mo><mi id="S4.Ex4.m1.3.3.1.1.3.3.3.3" xref="S4.Ex4.m1.3.3.1.1.3.3.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S4.Ex4.m1.3.3.1.1.3.3.3.1a" xref="S4.Ex4.m1.3.3.1.1.3.3.3.1.cmml">​</mo><mi id="S4.Ex4.m1.3.3.1.1.3.3.3.4" xref="S4.Ex4.m1.3.3.1.1.3.3.3.4.cmml">n</mi><mo lspace="0em" rspace="0em" id="S4.Ex4.m1.3.3.1.1.3.3.3.1b" xref="S4.Ex4.m1.3.3.1.1.3.3.3.1.cmml">​</mo><mi id="S4.Ex4.m1.3.3.1.1.3.3.3.5" xref="S4.Ex4.m1.3.3.1.1.3.3.3.5.cmml">k</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S4.Ex4.m1.3.3.1.1.3.1a" xref="S4.Ex4.m1.3.3.1.1.3.1.cmml">​</mo><mrow id="S4.Ex4.m1.3.3.1.1.3.4.2" xref="S4.Ex4.m1.3.3.1.1.3.cmml"><mo stretchy="false" id="S4.Ex4.m1.3.3.1.1.3.4.2.1" xref="S4.Ex4.m1.3.3.1.1.3.cmml">(</mo><mi id="S4.Ex4.m1.2.2" xref="S4.Ex4.m1.2.2.cmml">f</mi><mo stretchy="false" id="S4.Ex4.m1.3.3.1.1.3.4.2.2" xref="S4.Ex4.m1.3.3.1.1.3.cmml">)</mo></mrow></mrow></mrow><mo id="S4.Ex4.m1.3.3.1.2" xref="S4.Ex4.m1.3.3.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.Ex4.m1.3b"><apply id="S4.Ex4.m1.3.3.1.1.cmml" xref="S4.Ex4.m1.3.3.1"><plus id="S4.Ex4.m1.3.3.1.1.1.cmml" xref="S4.Ex4.m1.3.3.1.1.1"></plus><apply id="S4.Ex4.m1.3.3.1.1.2.cmml" xref="S4.Ex4.m1.3.3.1.1.2"><times id="S4.Ex4.m1.3.3.1.1.2.1.cmml" xref="S4.Ex4.m1.3.3.1.1.2.1"></times><ci id="S4.Ex4.m1.3.3.1.1.2.2.cmml" xref="S4.Ex4.m1.3.3.1.1.2.2">𝛽</ci><apply id="S4.Ex4.m1.3.3.1.1.2.3.cmml" xref="S4.Ex4.m1.3.3.1.1.2.3"><csymbol cd="ambiguous" id="S4.Ex4.m1.3.3.1.1.2.3.1.cmml" xref="S4.Ex4.m1.3.3.1.1.2.3">subscript</csymbol><ci id="S4.Ex4.m1.3.3.1.1.2.3.2.cmml" xref="S4.Ex4.m1.3.3.1.1.2.3.2">ℒ</ci><apply id="S4.Ex4.m1.3.3.1.1.2.3.3.cmml" xref="S4.Ex4.m1.3.3.1.1.2.3.3"><times id="S4.Ex4.m1.3.3.1.1.2.3.3.1.cmml" xref="S4.Ex4.m1.3.3.1.1.2.3.3.1"></times><ci id="S4.Ex4.m1.3.3.1.1.2.3.3.2.cmml" xref="S4.Ex4.m1.3.3.1.1.2.3.3.2">𝑟</ci><ci id="S4.Ex4.m1.3.3.1.1.2.3.3.3.cmml" xref="S4.Ex4.m1.3.3.1.1.2.3.3.3">𝑒</ci><ci id="S4.Ex4.m1.3.3.1.1.2.3.3.4.cmml" xref="S4.Ex4.m1.3.3.1.1.2.3.3.4">𝑐</ci></apply></apply><ci id="S4.Ex4.m1.1.1.cmml" xref="S4.Ex4.m1.1.1">𝑓</ci></apply><apply id="S4.Ex4.m1.3.3.1.1.3.cmml" xref="S4.Ex4.m1.3.3.1.1.3"><times id="S4.Ex4.m1.3.3.1.1.3.1.cmml" xref="S4.Ex4.m1.3.3.1.1.3.1"></times><ci id="S4.Ex4.m1.3.3.1.1.3.2.cmml" xref="S4.Ex4.m1.3.3.1.1.3.2">𝛾</ci><apply id="S4.Ex4.m1.3.3.1.1.3.3.cmml" xref="S4.Ex4.m1.3.3.1.1.3.3"><csymbol cd="ambiguous" id="S4.Ex4.m1.3.3.1.1.3.3.1.cmml" xref="S4.Ex4.m1.3.3.1.1.3.3">subscript</csymbol><ci id="S4.Ex4.m1.3.3.1.1.3.3.2.cmml" xref="S4.Ex4.m1.3.3.1.1.3.3.2">ℒ</ci><apply id="S4.Ex4.m1.3.3.1.1.3.3.3.cmml" xref="S4.Ex4.m1.3.3.1.1.3.3.3"><times id="S4.Ex4.m1.3.3.1.1.3.3.3.1.cmml" xref="S4.Ex4.m1.3.3.1.1.3.3.3.1"></times><ci id="S4.Ex4.m1.3.3.1.1.3.3.3.2.cmml" xref="S4.Ex4.m1.3.3.1.1.3.3.3.2">𝑟</ci><ci id="S4.Ex4.m1.3.3.1.1.3.3.3.3.cmml" xref="S4.Ex4.m1.3.3.1.1.3.3.3.3">𝑎</ci><ci id="S4.Ex4.m1.3.3.1.1.3.3.3.4.cmml" xref="S4.Ex4.m1.3.3.1.1.3.3.3.4">𝑛</ci><ci id="S4.Ex4.m1.3.3.1.1.3.3.3.5.cmml" xref="S4.Ex4.m1.3.3.1.1.3.3.3.5">𝑘</ci></apply></apply><ci id="S4.Ex4.m1.2.2.cmml" xref="S4.Ex4.m1.2.2">𝑓</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Ex4.m1.3c">\displaystyle\beta\mathcal{L}_{rec}(f)+\gamma\mathcal{L}_{rank}(f),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S4.SS2.p2.6" class="ltx_p">with <math id="S4.SS2.p2.4.m1.3" class="ltx_Math" alttext="\alpha,\beta,\gamma\geq 0" display="inline"><semantics id="S4.SS2.p2.4.m1.3a"><mrow id="S4.SS2.p2.4.m1.3.4" xref="S4.SS2.p2.4.m1.3.4.cmml"><mrow id="S4.SS2.p2.4.m1.3.4.2.2" xref="S4.SS2.p2.4.m1.3.4.2.1.cmml"><mi id="S4.SS2.p2.4.m1.1.1" xref="S4.SS2.p2.4.m1.1.1.cmml">α</mi><mo id="S4.SS2.p2.4.m1.3.4.2.2.1" xref="S4.SS2.p2.4.m1.3.4.2.1.cmml">,</mo><mi id="S4.SS2.p2.4.m1.2.2" xref="S4.SS2.p2.4.m1.2.2.cmml">β</mi><mo id="S4.SS2.p2.4.m1.3.4.2.2.2" xref="S4.SS2.p2.4.m1.3.4.2.1.cmml">,</mo><mi id="S4.SS2.p2.4.m1.3.3" xref="S4.SS2.p2.4.m1.3.3.cmml">γ</mi></mrow><mo id="S4.SS2.p2.4.m1.3.4.1" xref="S4.SS2.p2.4.m1.3.4.1.cmml">≥</mo><mn id="S4.SS2.p2.4.m1.3.4.3" xref="S4.SS2.p2.4.m1.3.4.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.4.m1.3b"><apply id="S4.SS2.p2.4.m1.3.4.cmml" xref="S4.SS2.p2.4.m1.3.4"><geq id="S4.SS2.p2.4.m1.3.4.1.cmml" xref="S4.SS2.p2.4.m1.3.4.1"></geq><list id="S4.SS2.p2.4.m1.3.4.2.1.cmml" xref="S4.SS2.p2.4.m1.3.4.2.2"><ci id="S4.SS2.p2.4.m1.1.1.cmml" xref="S4.SS2.p2.4.m1.1.1">𝛼</ci><ci id="S4.SS2.p2.4.m1.2.2.cmml" xref="S4.SS2.p2.4.m1.2.2">𝛽</ci><ci id="S4.SS2.p2.4.m1.3.3.cmml" xref="S4.SS2.p2.4.m1.3.3">𝛾</ci></list><cn type="integer" id="S4.SS2.p2.4.m1.3.4.3.cmml" xref="S4.SS2.p2.4.m1.3.4.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.4.m1.3c">\alpha,\beta,\gamma\geq 0</annotation></semantics></math> being hyper-parameters with fixed values as <math id="S4.SS2.p2.5.m2.2" class="ltx_Math" alttext="\alpha=1.0,\beta=1.0" display="inline"><semantics id="S4.SS2.p2.5.m2.2a"><mrow id="S4.SS2.p2.5.m2.2.2.2" xref="S4.SS2.p2.5.m2.2.2.3.cmml"><mrow id="S4.SS2.p2.5.m2.1.1.1.1" xref="S4.SS2.p2.5.m2.1.1.1.1.cmml"><mi id="S4.SS2.p2.5.m2.1.1.1.1.2" xref="S4.SS2.p2.5.m2.1.1.1.1.2.cmml">α</mi><mo id="S4.SS2.p2.5.m2.1.1.1.1.1" xref="S4.SS2.p2.5.m2.1.1.1.1.1.cmml">=</mo><mn id="S4.SS2.p2.5.m2.1.1.1.1.3" xref="S4.SS2.p2.5.m2.1.1.1.1.3.cmml">1.0</mn></mrow><mo id="S4.SS2.p2.5.m2.2.2.2.3" xref="S4.SS2.p2.5.m2.2.2.3a.cmml">,</mo><mrow id="S4.SS2.p2.5.m2.2.2.2.2" xref="S4.SS2.p2.5.m2.2.2.2.2.cmml"><mi id="S4.SS2.p2.5.m2.2.2.2.2.2" xref="S4.SS2.p2.5.m2.2.2.2.2.2.cmml">β</mi><mo id="S4.SS2.p2.5.m2.2.2.2.2.1" xref="S4.SS2.p2.5.m2.2.2.2.2.1.cmml">=</mo><mn id="S4.SS2.p2.5.m2.2.2.2.2.3" xref="S4.SS2.p2.5.m2.2.2.2.2.3.cmml">1.0</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.5.m2.2b"><apply id="S4.SS2.p2.5.m2.2.2.3.cmml" xref="S4.SS2.p2.5.m2.2.2.2"><csymbol cd="ambiguous" id="S4.SS2.p2.5.m2.2.2.3a.cmml" xref="S4.SS2.p2.5.m2.2.2.2.3">formulae-sequence</csymbol><apply id="S4.SS2.p2.5.m2.1.1.1.1.cmml" xref="S4.SS2.p2.5.m2.1.1.1.1"><eq id="S4.SS2.p2.5.m2.1.1.1.1.1.cmml" xref="S4.SS2.p2.5.m2.1.1.1.1.1"></eq><ci id="S4.SS2.p2.5.m2.1.1.1.1.2.cmml" xref="S4.SS2.p2.5.m2.1.1.1.1.2">𝛼</ci><cn type="float" id="S4.SS2.p2.5.m2.1.1.1.1.3.cmml" xref="S4.SS2.p2.5.m2.1.1.1.1.3">1.0</cn></apply><apply id="S4.SS2.p2.5.m2.2.2.2.2.cmml" xref="S4.SS2.p2.5.m2.2.2.2.2"><eq id="S4.SS2.p2.5.m2.2.2.2.2.1.cmml" xref="S4.SS2.p2.5.m2.2.2.2.2.1"></eq><ci id="S4.SS2.p2.5.m2.2.2.2.2.2.cmml" xref="S4.SS2.p2.5.m2.2.2.2.2.2">𝛽</ci><cn type="float" id="S4.SS2.p2.5.m2.2.2.2.2.3.cmml" xref="S4.SS2.p2.5.m2.2.2.2.2.3">1.0</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.5.m2.2c">\alpha=1.0,\beta=1.0</annotation></semantics></math> and <math id="S4.SS2.p2.6.m3.1" class="ltx_Math" alttext="\gamma=1.0" display="inline"><semantics id="S4.SS2.p2.6.m3.1a"><mrow id="S4.SS2.p2.6.m3.1.1" xref="S4.SS2.p2.6.m3.1.1.cmml"><mi id="S4.SS2.p2.6.m3.1.1.2" xref="S4.SS2.p2.6.m3.1.1.2.cmml">γ</mi><mo id="S4.SS2.p2.6.m3.1.1.1" xref="S4.SS2.p2.6.m3.1.1.1.cmml">=</mo><mn id="S4.SS2.p2.6.m3.1.1.3" xref="S4.SS2.p2.6.m3.1.1.3.cmml">1.0</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p2.6.m3.1b"><apply id="S4.SS2.p2.6.m3.1.1.cmml" xref="S4.SS2.p2.6.m3.1.1"><eq id="S4.SS2.p2.6.m3.1.1.1.cmml" xref="S4.SS2.p2.6.m3.1.1.1"></eq><ci id="S4.SS2.p2.6.m3.1.1.2.cmml" xref="S4.SS2.p2.6.m3.1.1.2">𝛾</ci><cn type="float" id="S4.SS2.p2.6.m3.1.1.3.cmml" xref="S4.SS2.p2.6.m3.1.1.3">1.0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p2.6.m3.1c">\gamma=1.0</annotation></semantics></math>.</p>
</div>
<div id="S4.SS2.p3" class="ltx_para">
<p id="S4.SS2.p3.1" class="ltx_p">To improve convergence, two-stage training strategy is employed. Initially, the Matryoshka-Adaptor is trained in an unsupervised way using Eq. <a href="#S3.E4" title="In 3.2 Tuning objective functions ‣ 3 Unsupervised Matryoshka-Adaptor ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.
Subsequently, further tuning is conducted in a supervised way, utilizing Eq. <a href="#S4.E6" title="In 4.2 Tuning objective functions ‣ 4 Supervised Matryoshka-Adaptor ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.</p>
</div>
<figure id="S4.F5" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S4.F5.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.20243/assets/x7.png" id="S4.F5.sf1.g1" class="ltx_graphics ltx_img_square" width="147" height="134" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S4.F5.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.20243/assets/x8.png" id="S4.F5.sf2.g1" class="ltx_graphics ltx_img_square" width="147" height="134" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S4.F5.sf3" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.20243/assets/x9.png" id="S4.F5.sf3.g1" class="ltx_graphics ltx_img_square" width="147" height="134" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(c) </span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>Experimental results of the supervised Matryoshka-Adaptor on retrieval tasks, utilizing three different embedding models: OpenAI text-embedding-3-large (on 8 BEIR datasets), Google Gecko multilingual (on 17 MIRACL datasets), and Google multimodal (on 5 Fashion-200K datasets). Additional results are in Appendix. <a href="#A4" title="Appendix D Additional results on supervised Matryoshka-Adaptor ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D</span></a>.</figcaption>
</figure>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Experiments</h2>

<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Experimental settings</h3>

<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.1" class="ltx_p">The Matryoshka-Adaptor is extensively evaluated across a diverse set of 13 BEIR datasets <cite class="ltx_cite ltx_citemacro_cite"><a href="#bib.bib1" title="" class="ltx_ref">bei </a></cite>, 17 MIRACL datasets <cite class="ltx_cite ltx_citemacro_cite"><a href="#bib.bib3" title="" class="ltx_ref">mir </a></cite>, and 5 Fashion-200K datasets <cite class="ltx_cite ltx_citemacro_cite">Han et al. (<a href="#bib.bib10" title="" class="ltx_ref">2017</a>)</cite>.
Such a comprehensive evaluation highlights the data-agnostic nature of the Matryoshka-Adaptor.
Query and corpus embeddings are generated using state-of-the-art models, including Google Gecko text embeddings (English and multilingual) <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib2" title="" class="ltx_ref">gcp, </a>)</cite>, Google multimodal embeddings, and OpenAI text embeddings <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib4" title="" class="ltx_ref">ope, </a>)</cite>.
This further highlights the model-agnostic nature of the proposed Matryoshka-Adaptor.</p>
</div>
<div id="S5.SS1.p2" class="ltx_para">
<p id="S5.SS1.p2.1" class="ltx_p">During the evaluation phase, both query and corpus embeddings are transformed using the trained Matryoshka-Adaptor.
Cosine similarity is then computed between the transformed query and corpus embeddings across a spectrum of reduced dimensions (e.g., <math id="S5.SS1.p2.1.m1.8" class="ltx_Math" alttext="d=\{8,16,32,64,128,256,512,768\}" display="inline"><semantics id="S5.SS1.p2.1.m1.8a"><mrow id="S5.SS1.p2.1.m1.8.9" xref="S5.SS1.p2.1.m1.8.9.cmml"><mi id="S5.SS1.p2.1.m1.8.9.2" xref="S5.SS1.p2.1.m1.8.9.2.cmml">d</mi><mo id="S5.SS1.p2.1.m1.8.9.1" xref="S5.SS1.p2.1.m1.8.9.1.cmml">=</mo><mrow id="S5.SS1.p2.1.m1.8.9.3.2" xref="S5.SS1.p2.1.m1.8.9.3.1.cmml"><mo stretchy="false" id="S5.SS1.p2.1.m1.8.9.3.2.1" xref="S5.SS1.p2.1.m1.8.9.3.1.cmml">{</mo><mn id="S5.SS1.p2.1.m1.1.1" xref="S5.SS1.p2.1.m1.1.1.cmml">8</mn><mo id="S5.SS1.p2.1.m1.8.9.3.2.2" xref="S5.SS1.p2.1.m1.8.9.3.1.cmml">,</mo><mn id="S5.SS1.p2.1.m1.2.2" xref="S5.SS1.p2.1.m1.2.2.cmml">16</mn><mo id="S5.SS1.p2.1.m1.8.9.3.2.3" xref="S5.SS1.p2.1.m1.8.9.3.1.cmml">,</mo><mn id="S5.SS1.p2.1.m1.3.3" xref="S5.SS1.p2.1.m1.3.3.cmml">32</mn><mo id="S5.SS1.p2.1.m1.8.9.3.2.4" xref="S5.SS1.p2.1.m1.8.9.3.1.cmml">,</mo><mn id="S5.SS1.p2.1.m1.4.4" xref="S5.SS1.p2.1.m1.4.4.cmml">64</mn><mo id="S5.SS1.p2.1.m1.8.9.3.2.5" xref="S5.SS1.p2.1.m1.8.9.3.1.cmml">,</mo><mn id="S5.SS1.p2.1.m1.5.5" xref="S5.SS1.p2.1.m1.5.5.cmml">128</mn><mo id="S5.SS1.p2.1.m1.8.9.3.2.6" xref="S5.SS1.p2.1.m1.8.9.3.1.cmml">,</mo><mn id="S5.SS1.p2.1.m1.6.6" xref="S5.SS1.p2.1.m1.6.6.cmml">256</mn><mo id="S5.SS1.p2.1.m1.8.9.3.2.7" xref="S5.SS1.p2.1.m1.8.9.3.1.cmml">,</mo><mn id="S5.SS1.p2.1.m1.7.7" xref="S5.SS1.p2.1.m1.7.7.cmml">512</mn><mo id="S5.SS1.p2.1.m1.8.9.3.2.8" xref="S5.SS1.p2.1.m1.8.9.3.1.cmml">,</mo><mn id="S5.SS1.p2.1.m1.8.8" xref="S5.SS1.p2.1.m1.8.8.cmml">768</mn><mo stretchy="false" id="S5.SS1.p2.1.m1.8.9.3.2.9" xref="S5.SS1.p2.1.m1.8.9.3.1.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p2.1.m1.8b"><apply id="S5.SS1.p2.1.m1.8.9.cmml" xref="S5.SS1.p2.1.m1.8.9"><eq id="S5.SS1.p2.1.m1.8.9.1.cmml" xref="S5.SS1.p2.1.m1.8.9.1"></eq><ci id="S5.SS1.p2.1.m1.8.9.2.cmml" xref="S5.SS1.p2.1.m1.8.9.2">𝑑</ci><set id="S5.SS1.p2.1.m1.8.9.3.1.cmml" xref="S5.SS1.p2.1.m1.8.9.3.2"><cn type="integer" id="S5.SS1.p2.1.m1.1.1.cmml" xref="S5.SS1.p2.1.m1.1.1">8</cn><cn type="integer" id="S5.SS1.p2.1.m1.2.2.cmml" xref="S5.SS1.p2.1.m1.2.2">16</cn><cn type="integer" id="S5.SS1.p2.1.m1.3.3.cmml" xref="S5.SS1.p2.1.m1.3.3">32</cn><cn type="integer" id="S5.SS1.p2.1.m1.4.4.cmml" xref="S5.SS1.p2.1.m1.4.4">64</cn><cn type="integer" id="S5.SS1.p2.1.m1.5.5.cmml" xref="S5.SS1.p2.1.m1.5.5">128</cn><cn type="integer" id="S5.SS1.p2.1.m1.6.6.cmml" xref="S5.SS1.p2.1.m1.6.6">256</cn><cn type="integer" id="S5.SS1.p2.1.m1.7.7.cmml" xref="S5.SS1.p2.1.m1.7.7">512</cn><cn type="integer" id="S5.SS1.p2.1.m1.8.8.cmml" xref="S5.SS1.p2.1.m1.8.8">768</cn></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p2.1.m1.8c">d=\{8,16,32,64,128,256,512,768\}</annotation></semantics></math> for Gecko text embeddings).
Retrieval performance is evaluated using the normalized discounted cumulative gain at rank 10 (nDCG@10) metric <cite class="ltx_cite ltx_citemacro_cite">Järvelin and Kekäläinen (<a href="#bib.bib12" title="" class="ltx_ref">2002</a>)</cite>, facilitating a comprehensive assessment of performance across various Matryoshka embedding dimensions.
All reported results represent the average value across the datasets.
Data statistics and hyper-parameters used in the experiments can be found in Appendix. <a href="#A1" title="Appendix A Data Statistics ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a> and <a href="#A2" title="Appendix B Hyper-parameters ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>.
Detailed results can be found in Appendix. <a href="#A5" title="Appendix E Detailed Experimental Results on Unsupervised Settings ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E</span></a> and <a href="#A6" title="Appendix F Detailed Experimental Results on Supervised Settings ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">F</span></a>.</p>
</div>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Unsupervised tuning</h3>

<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.1" class="ltx_p">The Matryoshka-Adaptor can be applied exclusively with corpus embeddings, referred to as the unsupervised setting, enabling customization of embeddings solely on the corpus side.
In this subsection, we present the impact of the unsupervised Matryoshka-Adaptor on two OpenAI text embedding models and compare its performance with the commonly-used unsupervised dimensionality reduction method, Principal Component Analysis (PCA) <cite class="ltx_cite ltx_citemacro_cite">Jolliffe and Cadima (<a href="#bib.bib13" title="" class="ltx_ref">2016</a>)</cite>.</p>
</div>
<div id="S5.SS2.p2" class="ltx_para">
<p id="S5.SS2.p2.1" class="ltx_p">Fig. <a href="#S4.F4.sf1" title="In Figure 4 ‣ 4.1 Problem formulation ‣ 4 Supervised Matryoshka-Adaptor ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4(a)</span></a> and <a href="#S4.F4.sf2" title="In Figure 4 ‣ 4.1 Problem formulation ‣ 4 Supervised Matryoshka-Adaptor ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4(b)</span></a> illustrate that the Matryoshka-Adaptor yields significant performance improvements, particularly for lower dimensions, compared to the embeddings of the same dimensionality without it.
Furthermore, lower dimensional embeddings processed with Matryoshka-Adaptor achieve comparable performance to original embeddings of high dimensionality.
The adaptor achieves a faster saturation in performance with embedding dimensionality, towards the retrieval performance of original embeddings, underscoring its substantial impact in significantly reducing latency and memory requirements for retrieval applications.</p>
</div>
<div id="S5.SS2.p3" class="ltx_para">
<p id="S5.SS2.p3.1" class="ltx_p">Notably, the latest OpenAI embeddings are already trained with Matryoshka Representation Learning (MRL) <cite class="ltx_cite ltx_citemacro_citep">(<a href="#bib.bib4" title="" class="ltx_ref">ope, </a>)</cite>.
The additional performance gains achieved by Matryoshka-Adaptor are attributed to the tuning process.
With PCA, some improvements are observed for lower dimensions, however, at higher dimensions, noticeable performance degradation occurs, resulting in performance inferior even to the original embedding.
This highlights the superiority of the Matryoshka-Adaptor, designed with the goal of salient information preservation for similarity within lower dimensions compared to PCA, a generic unsupervised dimensionality reduction approach.</p>
</div>
</section>
<section id="S5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.3 </span>Supervised tuning</h3>

<div id="S5.SS3.p1" class="ltx_para">
<p id="S5.SS3.p1.1" class="ltx_p">The Matryoshka-Adaptor can be applied in supervised learning setup where a limited number of query-corpus pairs are available.
In this context, the Supervised Matryoshka-Adaptor is trained utilizing paired query-corpus data.
The effectiveness of the Supervised Matryoshka-Adaptor is evaluated on 13 BEIR, 17 MIRACL, and 5 Fashion-200K datasets, with query and corpus embeddings being generated using the latest Google Gecko models, including its multilingual and multimodal versions.</p>
</div>
<div id="S5.SS3.p2" class="ltx_para">
<p id="S5.SS3.p2.1" class="ltx_p">As illustrated in Fig. <a href="#S4.F5" title="Figure 5 ‣ 4.2 Tuning objective functions ‣ 4 Supervised Matryoshka-Adaptor ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, the Supervised Matryoshka-Adaptor consistently outperforms the alternatives, such as Search-Adaptor <cite class="ltx_cite ltx_citemacro_cite">Yoon et al. (<a href="#bib.bib25" title="" class="ltx_ref">2023</a>)</cite>, particularly for lower embedding dimensions.
Additionally, lower dimensional embeddings processed with the Supervised Matryoshka-Adaptor perform comparably to high dimensional embeddings, showcasing its potential to significantly reduce latency and memory requirements for applications like retrieval.</p>
</div>
<figure id="S5.F6" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S5.F6.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.20243/assets/x10.png" id="S5.F6.sf1.g1" class="ltx_graphics ltx_img_square" width="147" height="134" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S5.F6.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.20243/assets/x11.png" id="S5.F6.sf2.g1" class="ltx_graphics ltx_img_square" width="147" height="134" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S5.F6.sf3" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.20243/assets/x12.png" id="S5.F6.sf3.g1" class="ltx_graphics ltx_img_square" width="147" height="134" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(c) </span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>Experimental results of the unsupervised Matryoshka-Adaptor with three different text embedding models: Google Gecko, Google Gecko multilingual, and Google Gecko-003 (which is not trained with the Matryoshka Representation Learning technique).</figcaption>
</figure>
</section>
<section id="S5.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.4 </span>Tuning for Multimodal Embeddings</h3>

<div id="S5.SS4.p1" class="ltx_para">
<p id="S5.SS4.p1.1" class="ltx_p">As previously established, the Matryoshka-Adaptor framework is not confined to text embeddings but can be generalized to multimodal embeddings as well.
To illustrate this capability, we applied the Matryoshka-Adaptor to the latest gecko-multimodal embeddings, utilizing the Fashion-200K dataset, which comprises 5 sub-datasets designed for text-to-image retrieval tasks.</p>
</div>
<div id="S5.SS4.p2" class="ltx_para">
<p id="S5.SS4.p2.1" class="ltx_p">Fig. <a href="#S4.F4.sf3" title="In Figure 4 ‣ 4.1 Problem formulation ‣ 4 Supervised Matryoshka-Adaptor ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4(c)</span></a> and <a href="#S4.F5.sf3" title="In Figure 5 ‣ 4.2 Tuning objective functions ‣ 4 Supervised Matryoshka-Adaptor ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5(c)</span></a> demonstrate the effectiveness of the Matryoshka-Adaptor in consistently improving the performance of multimodal base embedding models for text-to-image retrieval tasks.
These highlight that the Matryoshka-Adaptor significantly outperforms alternative methods such as PCA, in unsupervised learning setups and Search-Adaptor in supervised learning setups, particularly when lower embedding dimensions are considered.</p>
</div>
</section>
<section id="S5.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.5 </span>Tuning for Multilingual Embeddings</h3>

<div id="S5.SS5.p1" class="ltx_para">
<p id="S5.SS5.p1.1" class="ltx_p">Matryoshka-Adaptor is not only model-agnostic but also data-agnostic. Its applicability even extends beyond a single language.
To validate this, we evaluate the performance of Matryoshka-Adaptor on MIRACL datasets, which comprise 17 non-English languages.</p>
</div>
<div id="S5.SS5.p2" class="ltx_para">
<p id="S5.SS5.p2.1" class="ltx_p">Fig. <a href="#S5.F6.sf2" title="In Figure 6 ‣ 5.3 Supervised tuning ‣ 5 Experiments ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6(b)</span></a> and <a href="#S4.F5.sf2" title="In Figure 5 ‣ 4.2 Tuning objective functions ‣ 4 Supervised Matryoshka-Adaptor ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5(b)</span></a> present the results of applying Matryoshka-Adaptor to multilingual retrieval tasks.
The findings demonstrate that the performance gains achieved through the proposed tuning method are not limited to English but also extend to non-English language datasets.
Furthermore, the improvements are observed to be model-agnostic, as evident from the successful application of Matryoshka-Adaptor to the latest Gecko multilingual embedding models.</p>
</div>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Discussions</h2>

<section id="S6.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.1 </span>Models that are not pretrained with MRL</h3>

<div id="S6.SS1.p1" class="ltx_para">
<p id="S6.SS1.p1.1" class="ltx_p">A significant advantage of the Matryoshka-Adaptor framework lies in its broad applicability to a wide array of embedding models.
We demonstrate the efficacy of Matryoshka-Adaptor when applied to embedding models that have not been trained using MRL.
Specifically, we utilize earlier versions of the Gecko embedding models (Google Gecko-003), which do not utilize MRL in their pretraining, in conjunction with BEIR datasets to illustrate the impact of the Unsupervised Matryoshka-Adaptor.</p>
</div>
<div id="S6.SS1.p2" class="ltx_para">
<p id="S6.SS1.p2.1" class="ltx_p">Fig. <a href="#S5.F6.sf3" title="In Figure 6 ‣ 5.3 Supervised tuning ‣ 5 Experiments ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6(c)</span></a> presents evidence of the consistent performance improvements achieved by the Matryoshka-Adaptor when applied to non-MRL trained embedding models.
This observation underscores that the performance gains of Matryoshka-Adaptor originate from customizing the embedding to the specific corpus, and this beneficial impact can be extended to any embedding model, even when they had been trained with MRL.</p>
</div>
<figure id="S6.F7" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S6.F7.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.20243/assets/x13.png" id="S6.F7.sf1.g1" class="ltx_graphics ltx_img_square" width="147" height="123" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S6.F7.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.20243/assets/x14.png" id="S6.F7.sf2.g1" class="ltx_graphics ltx_img_square" width="147" height="123" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="S6.F7.sf3" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.20243/assets/x15.png" id="S6.F7.sf3.g1" class="ltx_graphics ltx_img_square" width="147" height="123" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(c) </span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>Analysis of distance metrics in unsupervised settings.
(a) Average pairwise and top-k distances across varying embedding dimensions, compared to base embeddings.
(b) Correlation between nDCG@10 and average pairwise distance.
(c) Correlation between nDCG@10 and average top-k distance.</figcaption>
</figure>
</section>
<section id="S6.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.2 </span>Ablation studies</h3>

<div id="S6.SS2.p1" class="ltx_para">
<p id="S6.SS2.p1.6" class="ltx_p">To examine the individual contribution of each loss component to the overall performance of Matryoshka-Adaptor, we conduct ablation studies.
In the unsupervised setting, each of the top-k loss (<math id="S6.SS2.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{L}_{topk}" display="inline"><semantics id="S6.SS2.p1.1.m1.1a"><msub id="S6.SS2.p1.1.m1.1.1" xref="S6.SS2.p1.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.SS2.p1.1.m1.1.1.2" xref="S6.SS2.p1.1.m1.1.1.2.cmml">ℒ</mi><mrow id="S6.SS2.p1.1.m1.1.1.3" xref="S6.SS2.p1.1.m1.1.1.3.cmml"><mi id="S6.SS2.p1.1.m1.1.1.3.2" xref="S6.SS2.p1.1.m1.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p1.1.m1.1.1.3.1" xref="S6.SS2.p1.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p1.1.m1.1.1.3.3" xref="S6.SS2.p1.1.m1.1.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p1.1.m1.1.1.3.1a" xref="S6.SS2.p1.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p1.1.m1.1.1.3.4" xref="S6.SS2.p1.1.m1.1.1.3.4.cmml">p</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p1.1.m1.1.1.3.1b" xref="S6.SS2.p1.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p1.1.m1.1.1.3.5" xref="S6.SS2.p1.1.m1.1.1.3.5.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.1.m1.1b"><apply id="S6.SS2.p1.1.m1.1.1.cmml" xref="S6.SS2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S6.SS2.p1.1.m1.1.1.1.cmml" xref="S6.SS2.p1.1.m1.1.1">subscript</csymbol><ci id="S6.SS2.p1.1.m1.1.1.2.cmml" xref="S6.SS2.p1.1.m1.1.1.2">ℒ</ci><apply id="S6.SS2.p1.1.m1.1.1.3.cmml" xref="S6.SS2.p1.1.m1.1.1.3"><times id="S6.SS2.p1.1.m1.1.1.3.1.cmml" xref="S6.SS2.p1.1.m1.1.1.3.1"></times><ci id="S6.SS2.p1.1.m1.1.1.3.2.cmml" xref="S6.SS2.p1.1.m1.1.1.3.2">𝑡</ci><ci id="S6.SS2.p1.1.m1.1.1.3.3.cmml" xref="S6.SS2.p1.1.m1.1.1.3.3">𝑜</ci><ci id="S6.SS2.p1.1.m1.1.1.3.4.cmml" xref="S6.SS2.p1.1.m1.1.1.3.4">𝑝</ci><ci id="S6.SS2.p1.1.m1.1.1.3.5.cmml" xref="S6.SS2.p1.1.m1.1.1.3.5">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.1.m1.1c">\mathcal{L}_{topk}</annotation></semantics></math>), pairwise loss (<math id="S6.SS2.p1.2.m2.1" class="ltx_Math" alttext="\mathcal{L}_{pair}" display="inline"><semantics id="S6.SS2.p1.2.m2.1a"><msub id="S6.SS2.p1.2.m2.1.1" xref="S6.SS2.p1.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.SS2.p1.2.m2.1.1.2" xref="S6.SS2.p1.2.m2.1.1.2.cmml">ℒ</mi><mrow id="S6.SS2.p1.2.m2.1.1.3" xref="S6.SS2.p1.2.m2.1.1.3.cmml"><mi id="S6.SS2.p1.2.m2.1.1.3.2" xref="S6.SS2.p1.2.m2.1.1.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p1.2.m2.1.1.3.1" xref="S6.SS2.p1.2.m2.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p1.2.m2.1.1.3.3" xref="S6.SS2.p1.2.m2.1.1.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p1.2.m2.1.1.3.1a" xref="S6.SS2.p1.2.m2.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p1.2.m2.1.1.3.4" xref="S6.SS2.p1.2.m2.1.1.3.4.cmml">i</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p1.2.m2.1.1.3.1b" xref="S6.SS2.p1.2.m2.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p1.2.m2.1.1.3.5" xref="S6.SS2.p1.2.m2.1.1.3.5.cmml">r</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.2.m2.1b"><apply id="S6.SS2.p1.2.m2.1.1.cmml" xref="S6.SS2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S6.SS2.p1.2.m2.1.1.1.cmml" xref="S6.SS2.p1.2.m2.1.1">subscript</csymbol><ci id="S6.SS2.p1.2.m2.1.1.2.cmml" xref="S6.SS2.p1.2.m2.1.1.2">ℒ</ci><apply id="S6.SS2.p1.2.m2.1.1.3.cmml" xref="S6.SS2.p1.2.m2.1.1.3"><times id="S6.SS2.p1.2.m2.1.1.3.1.cmml" xref="S6.SS2.p1.2.m2.1.1.3.1"></times><ci id="S6.SS2.p1.2.m2.1.1.3.2.cmml" xref="S6.SS2.p1.2.m2.1.1.3.2">𝑝</ci><ci id="S6.SS2.p1.2.m2.1.1.3.3.cmml" xref="S6.SS2.p1.2.m2.1.1.3.3">𝑎</ci><ci id="S6.SS2.p1.2.m2.1.1.3.4.cmml" xref="S6.SS2.p1.2.m2.1.1.3.4">𝑖</ci><ci id="S6.SS2.p1.2.m2.1.1.3.5.cmml" xref="S6.SS2.p1.2.m2.1.1.3.5">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.2.m2.1c">\mathcal{L}_{pair}</annotation></semantics></math>), and reconstruction loss (<math id="S6.SS2.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{L}_{rec}" display="inline"><semantics id="S6.SS2.p1.3.m3.1a"><msub id="S6.SS2.p1.3.m3.1.1" xref="S6.SS2.p1.3.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.SS2.p1.3.m3.1.1.2" xref="S6.SS2.p1.3.m3.1.1.2.cmml">ℒ</mi><mrow id="S6.SS2.p1.3.m3.1.1.3" xref="S6.SS2.p1.3.m3.1.1.3.cmml"><mi id="S6.SS2.p1.3.m3.1.1.3.2" xref="S6.SS2.p1.3.m3.1.1.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p1.3.m3.1.1.3.1" xref="S6.SS2.p1.3.m3.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p1.3.m3.1.1.3.3" xref="S6.SS2.p1.3.m3.1.1.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p1.3.m3.1.1.3.1a" xref="S6.SS2.p1.3.m3.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p1.3.m3.1.1.3.4" xref="S6.SS2.p1.3.m3.1.1.3.4.cmml">c</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.3.m3.1b"><apply id="S6.SS2.p1.3.m3.1.1.cmml" xref="S6.SS2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S6.SS2.p1.3.m3.1.1.1.cmml" xref="S6.SS2.p1.3.m3.1.1">subscript</csymbol><ci id="S6.SS2.p1.3.m3.1.1.2.cmml" xref="S6.SS2.p1.3.m3.1.1.2">ℒ</ci><apply id="S6.SS2.p1.3.m3.1.1.3.cmml" xref="S6.SS2.p1.3.m3.1.1.3"><times id="S6.SS2.p1.3.m3.1.1.3.1.cmml" xref="S6.SS2.p1.3.m3.1.1.3.1"></times><ci id="S6.SS2.p1.3.m3.1.1.3.2.cmml" xref="S6.SS2.p1.3.m3.1.1.3.2">𝑟</ci><ci id="S6.SS2.p1.3.m3.1.1.3.3.cmml" xref="S6.SS2.p1.3.m3.1.1.3.3">𝑒</ci><ci id="S6.SS2.p1.3.m3.1.1.3.4.cmml" xref="S6.SS2.p1.3.m3.1.1.3.4">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.3.m3.1c">\mathcal{L}_{rec}</annotation></semantics></math>) is individually excluded from the original loss summation in Eq. <a href="#S3.E4" title="In 3.2 Tuning objective functions ‣ 3 Unsupervised Matryoshka-Adaptor ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, and the corresponding performance degradation is monitored.
Specifically, the pairwise loss was excluded by setting <math id="S6.SS2.p1.4.m4.1" class="ltx_Math" alttext="\alpha=0" display="inline"><semantics id="S6.SS2.p1.4.m4.1a"><mrow id="S6.SS2.p1.4.m4.1.1" xref="S6.SS2.p1.4.m4.1.1.cmml"><mi id="S6.SS2.p1.4.m4.1.1.2" xref="S6.SS2.p1.4.m4.1.1.2.cmml">α</mi><mo id="S6.SS2.p1.4.m4.1.1.1" xref="S6.SS2.p1.4.m4.1.1.1.cmml">=</mo><mn id="S6.SS2.p1.4.m4.1.1.3" xref="S6.SS2.p1.4.m4.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.4.m4.1b"><apply id="S6.SS2.p1.4.m4.1.1.cmml" xref="S6.SS2.p1.4.m4.1.1"><eq id="S6.SS2.p1.4.m4.1.1.1.cmml" xref="S6.SS2.p1.4.m4.1.1.1"></eq><ci id="S6.SS2.p1.4.m4.1.1.2.cmml" xref="S6.SS2.p1.4.m4.1.1.2">𝛼</ci><cn type="integer" id="S6.SS2.p1.4.m4.1.1.3.cmml" xref="S6.SS2.p1.4.m4.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.4.m4.1c">\alpha=0</annotation></semantics></math>, and the reconstruction loss is excluded by setting <math id="S6.SS2.p1.5.m5.1" class="ltx_Math" alttext="\beta=0" display="inline"><semantics id="S6.SS2.p1.5.m5.1a"><mrow id="S6.SS2.p1.5.m5.1.1" xref="S6.SS2.p1.5.m5.1.1.cmml"><mi id="S6.SS2.p1.5.m5.1.1.2" xref="S6.SS2.p1.5.m5.1.1.2.cmml">β</mi><mo id="S6.SS2.p1.5.m5.1.1.1" xref="S6.SS2.p1.5.m5.1.1.1.cmml">=</mo><mn id="S6.SS2.p1.5.m5.1.1.3" xref="S6.SS2.p1.5.m5.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.5.m5.1b"><apply id="S6.SS2.p1.5.m5.1.1.cmml" xref="S6.SS2.p1.5.m5.1.1"><eq id="S6.SS2.p1.5.m5.1.1.1.cmml" xref="S6.SS2.p1.5.m5.1.1.1"></eq><ci id="S6.SS2.p1.5.m5.1.1.2.cmml" xref="S6.SS2.p1.5.m5.1.1.2">𝛽</ci><cn type="integer" id="S6.SS2.p1.5.m5.1.1.3.cmml" xref="S6.SS2.p1.5.m5.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.5.m5.1c">\beta=0</annotation></semantics></math>.
In the supervised setting, we utilize only the ranking loss (<math id="S6.SS2.p1.6.m6.1" class="ltx_Math" alttext="\mathcal{L}_{rank}" display="inline"><semantics id="S6.SS2.p1.6.m6.1a"><msub id="S6.SS2.p1.6.m6.1.1" xref="S6.SS2.p1.6.m6.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.SS2.p1.6.m6.1.1.2" xref="S6.SS2.p1.6.m6.1.1.2.cmml">ℒ</mi><mrow id="S6.SS2.p1.6.m6.1.1.3" xref="S6.SS2.p1.6.m6.1.1.3.cmml"><mi id="S6.SS2.p1.6.m6.1.1.3.2" xref="S6.SS2.p1.6.m6.1.1.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p1.6.m6.1.1.3.1" xref="S6.SS2.p1.6.m6.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p1.6.m6.1.1.3.3" xref="S6.SS2.p1.6.m6.1.1.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p1.6.m6.1.1.3.1a" xref="S6.SS2.p1.6.m6.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p1.6.m6.1.1.3.4" xref="S6.SS2.p1.6.m6.1.1.3.4.cmml">n</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p1.6.m6.1.1.3.1b" xref="S6.SS2.p1.6.m6.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p1.6.m6.1.1.3.5" xref="S6.SS2.p1.6.m6.1.1.3.5.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.6.m6.1b"><apply id="S6.SS2.p1.6.m6.1.1.cmml" xref="S6.SS2.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S6.SS2.p1.6.m6.1.1.1.cmml" xref="S6.SS2.p1.6.m6.1.1">subscript</csymbol><ci id="S6.SS2.p1.6.m6.1.1.2.cmml" xref="S6.SS2.p1.6.m6.1.1.2">ℒ</ci><apply id="S6.SS2.p1.6.m6.1.1.3.cmml" xref="S6.SS2.p1.6.m6.1.1.3"><times id="S6.SS2.p1.6.m6.1.1.3.1.cmml" xref="S6.SS2.p1.6.m6.1.1.3.1"></times><ci id="S6.SS2.p1.6.m6.1.1.3.2.cmml" xref="S6.SS2.p1.6.m6.1.1.3.2">𝑟</ci><ci id="S6.SS2.p1.6.m6.1.1.3.3.cmml" xref="S6.SS2.p1.6.m6.1.1.3.3">𝑎</ci><ci id="S6.SS2.p1.6.m6.1.1.3.4.cmml" xref="S6.SS2.p1.6.m6.1.1.3.4">𝑛</ci><ci id="S6.SS2.p1.6.m6.1.1.3.5.cmml" xref="S6.SS2.p1.6.m6.1.1.3.5">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.6.m6.1c">\mathcal{L}_{rank}</annotation></semantics></math>) to assess the impact of three unsupervised losses in Eq. <a href="#S4.E6" title="In 4.2 Tuning objective functions ‣ 4 Supervised Matryoshka-Adaptor ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>.</p>
</div>
<figure id="S6.T1" class="ltx_table">
<table id="S6.T1.4" class="ltx_tabular ltx_centering ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S6.T1.4.5.1" class="ltx_tr">
<td id="S6.T1.4.5.1.1" class="ltx_td ltx_align_center ltx_border_tt" colspan="5">
<span id="S6.T1.4.5.1.1.1" class="ltx_text ltx_font_bold">Unsuperivsed Matryoshka-Adaptor</span> (nDCG@10)</td>
</tr>
<tr id="S6.T1.4.6.2" class="ltx_tr">
<td id="S6.T1.4.6.2.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">Reduced dims</td>
<td id="S6.T1.4.6.2.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">64</td>
<td id="S6.T1.4.6.2.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">128</td>
<td id="S6.T1.4.6.2.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">256</td>
<td id="S6.T1.4.6.2.5" class="ltx_td ltx_align_center ltx_border_t">512</td>
</tr>
<tr id="S6.T1.4.7.3" class="ltx_tr">
<td id="S6.T1.4.7.3.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">Baseline</td>
<td id="S6.T1.4.7.3.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.4332</td>
<td id="S6.T1.4.7.3.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.5044</td>
<td id="S6.T1.4.7.3.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.5461</td>
<td id="S6.T1.4.7.3.5" class="ltx_td ltx_align_center ltx_border_t">0.5590</td>
</tr>
<tr id="S6.T1.4.8.4" class="ltx_tr">
<td id="S6.T1.4.8.4.1" class="ltx_td ltx_align_center ltx_border_r">All three losses</td>
<td id="S6.T1.4.8.4.2" class="ltx_td ltx_align_center ltx_border_r"><span id="S6.T1.4.8.4.2.1" class="ltx_text ltx_font_bold">0.4845</span></td>
<td id="S6.T1.4.8.4.3" class="ltx_td ltx_align_center ltx_border_r"><span id="S6.T1.4.8.4.3.1" class="ltx_text ltx_font_bold">0.5380</span></td>
<td id="S6.T1.4.8.4.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S6.T1.4.8.4.4.1" class="ltx_text ltx_font_bold">0.5580</span></td>
<td id="S6.T1.4.8.4.5" class="ltx_td ltx_align_center"><span id="S6.T1.4.8.4.5.1" class="ltx_text ltx_font_bold">0.5652</span></td>
</tr>
<tr id="S6.T1.1.1" class="ltx_tr">
<td id="S6.T1.1.1.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">w/o <math id="S6.T1.1.1.1.m1.1" class="ltx_Math" alttext="\mathcal{L}_{topk}" display="inline"><semantics id="S6.T1.1.1.1.m1.1a"><msub id="S6.T1.1.1.1.m1.1.1" xref="S6.T1.1.1.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.T1.1.1.1.m1.1.1.2" xref="S6.T1.1.1.1.m1.1.1.2.cmml">ℒ</mi><mrow id="S6.T1.1.1.1.m1.1.1.3" xref="S6.T1.1.1.1.m1.1.1.3.cmml"><mi id="S6.T1.1.1.1.m1.1.1.3.2" xref="S6.T1.1.1.1.m1.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S6.T1.1.1.1.m1.1.1.3.1" xref="S6.T1.1.1.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.T1.1.1.1.m1.1.1.3.3" xref="S6.T1.1.1.1.m1.1.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S6.T1.1.1.1.m1.1.1.3.1a" xref="S6.T1.1.1.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.T1.1.1.1.m1.1.1.3.4" xref="S6.T1.1.1.1.m1.1.1.3.4.cmml">p</mi><mo lspace="0em" rspace="0em" id="S6.T1.1.1.1.m1.1.1.3.1b" xref="S6.T1.1.1.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.T1.1.1.1.m1.1.1.3.5" xref="S6.T1.1.1.1.m1.1.1.3.5.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S6.T1.1.1.1.m1.1b"><apply id="S6.T1.1.1.1.m1.1.1.cmml" xref="S6.T1.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="S6.T1.1.1.1.m1.1.1.1.cmml" xref="S6.T1.1.1.1.m1.1.1">subscript</csymbol><ci id="S6.T1.1.1.1.m1.1.1.2.cmml" xref="S6.T1.1.1.1.m1.1.1.2">ℒ</ci><apply id="S6.T1.1.1.1.m1.1.1.3.cmml" xref="S6.T1.1.1.1.m1.1.1.3"><times id="S6.T1.1.1.1.m1.1.1.3.1.cmml" xref="S6.T1.1.1.1.m1.1.1.3.1"></times><ci id="S6.T1.1.1.1.m1.1.1.3.2.cmml" xref="S6.T1.1.1.1.m1.1.1.3.2">𝑡</ci><ci id="S6.T1.1.1.1.m1.1.1.3.3.cmml" xref="S6.T1.1.1.1.m1.1.1.3.3">𝑜</ci><ci id="S6.T1.1.1.1.m1.1.1.3.4.cmml" xref="S6.T1.1.1.1.m1.1.1.3.4">𝑝</ci><ci id="S6.T1.1.1.1.m1.1.1.3.5.cmml" xref="S6.T1.1.1.1.m1.1.1.3.5">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.1.1.1.m1.1c">\mathcal{L}_{topk}</annotation></semantics></math>
</td>
<td id="S6.T1.1.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.4798</td>
<td id="S6.T1.1.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.5236</td>
<td id="S6.T1.1.1.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.5463</td>
<td id="S6.T1.1.1.5" class="ltx_td ltx_align_center ltx_border_t">0.5630</td>
</tr>
<tr id="S6.T1.2.2" class="ltx_tr">
<td id="S6.T1.2.2.1" class="ltx_td ltx_align_center ltx_border_r">w/o <math id="S6.T1.2.2.1.m1.1" class="ltx_Math" alttext="\mathcal{L}_{pair}" display="inline"><semantics id="S6.T1.2.2.1.m1.1a"><msub id="S6.T1.2.2.1.m1.1.1" xref="S6.T1.2.2.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.T1.2.2.1.m1.1.1.2" xref="S6.T1.2.2.1.m1.1.1.2.cmml">ℒ</mi><mrow id="S6.T1.2.2.1.m1.1.1.3" xref="S6.T1.2.2.1.m1.1.1.3.cmml"><mi id="S6.T1.2.2.1.m1.1.1.3.2" xref="S6.T1.2.2.1.m1.1.1.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="S6.T1.2.2.1.m1.1.1.3.1" xref="S6.T1.2.2.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.T1.2.2.1.m1.1.1.3.3" xref="S6.T1.2.2.1.m1.1.1.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S6.T1.2.2.1.m1.1.1.3.1a" xref="S6.T1.2.2.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.T1.2.2.1.m1.1.1.3.4" xref="S6.T1.2.2.1.m1.1.1.3.4.cmml">i</mi><mo lspace="0em" rspace="0em" id="S6.T1.2.2.1.m1.1.1.3.1b" xref="S6.T1.2.2.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.T1.2.2.1.m1.1.1.3.5" xref="S6.T1.2.2.1.m1.1.1.3.5.cmml">r</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S6.T1.2.2.1.m1.1b"><apply id="S6.T1.2.2.1.m1.1.1.cmml" xref="S6.T1.2.2.1.m1.1.1"><csymbol cd="ambiguous" id="S6.T1.2.2.1.m1.1.1.1.cmml" xref="S6.T1.2.2.1.m1.1.1">subscript</csymbol><ci id="S6.T1.2.2.1.m1.1.1.2.cmml" xref="S6.T1.2.2.1.m1.1.1.2">ℒ</ci><apply id="S6.T1.2.2.1.m1.1.1.3.cmml" xref="S6.T1.2.2.1.m1.1.1.3"><times id="S6.T1.2.2.1.m1.1.1.3.1.cmml" xref="S6.T1.2.2.1.m1.1.1.3.1"></times><ci id="S6.T1.2.2.1.m1.1.1.3.2.cmml" xref="S6.T1.2.2.1.m1.1.1.3.2">𝑝</ci><ci id="S6.T1.2.2.1.m1.1.1.3.3.cmml" xref="S6.T1.2.2.1.m1.1.1.3.3">𝑎</ci><ci id="S6.T1.2.2.1.m1.1.1.3.4.cmml" xref="S6.T1.2.2.1.m1.1.1.3.4">𝑖</ci><ci id="S6.T1.2.2.1.m1.1.1.3.5.cmml" xref="S6.T1.2.2.1.m1.1.1.3.5">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.2.2.1.m1.1c">\mathcal{L}_{pair}</annotation></semantics></math>
</td>
<td id="S6.T1.2.2.2" class="ltx_td ltx_align_center ltx_border_r">0.4745</td>
<td id="S6.T1.2.2.3" class="ltx_td ltx_align_center ltx_border_r">0.5230</td>
<td id="S6.T1.2.2.4" class="ltx_td ltx_align_center ltx_border_r">0.5423</td>
<td id="S6.T1.2.2.5" class="ltx_td ltx_align_center">0.5598</td>
</tr>
<tr id="S6.T1.3.3" class="ltx_tr">
<td id="S6.T1.3.3.1" class="ltx_td ltx_align_center ltx_border_r">w/o <math id="S6.T1.3.3.1.m1.1" class="ltx_Math" alttext="\mathcal{L}_{rec}" display="inline"><semantics id="S6.T1.3.3.1.m1.1a"><msub id="S6.T1.3.3.1.m1.1.1" xref="S6.T1.3.3.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.T1.3.3.1.m1.1.1.2" xref="S6.T1.3.3.1.m1.1.1.2.cmml">ℒ</mi><mrow id="S6.T1.3.3.1.m1.1.1.3" xref="S6.T1.3.3.1.m1.1.1.3.cmml"><mi id="S6.T1.3.3.1.m1.1.1.3.2" xref="S6.T1.3.3.1.m1.1.1.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="S6.T1.3.3.1.m1.1.1.3.1" xref="S6.T1.3.3.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.T1.3.3.1.m1.1.1.3.3" xref="S6.T1.3.3.1.m1.1.1.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="S6.T1.3.3.1.m1.1.1.3.1a" xref="S6.T1.3.3.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.T1.3.3.1.m1.1.1.3.4" xref="S6.T1.3.3.1.m1.1.1.3.4.cmml">c</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S6.T1.3.3.1.m1.1b"><apply id="S6.T1.3.3.1.m1.1.1.cmml" xref="S6.T1.3.3.1.m1.1.1"><csymbol cd="ambiguous" id="S6.T1.3.3.1.m1.1.1.1.cmml" xref="S6.T1.3.3.1.m1.1.1">subscript</csymbol><ci id="S6.T1.3.3.1.m1.1.1.2.cmml" xref="S6.T1.3.3.1.m1.1.1.2">ℒ</ci><apply id="S6.T1.3.3.1.m1.1.1.3.cmml" xref="S6.T1.3.3.1.m1.1.1.3"><times id="S6.T1.3.3.1.m1.1.1.3.1.cmml" xref="S6.T1.3.3.1.m1.1.1.3.1"></times><ci id="S6.T1.3.3.1.m1.1.1.3.2.cmml" xref="S6.T1.3.3.1.m1.1.1.3.2">𝑟</ci><ci id="S6.T1.3.3.1.m1.1.1.3.3.cmml" xref="S6.T1.3.3.1.m1.1.1.3.3">𝑒</ci><ci id="S6.T1.3.3.1.m1.1.1.3.4.cmml" xref="S6.T1.3.3.1.m1.1.1.3.4">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.3.3.1.m1.1c">\mathcal{L}_{rec}</annotation></semantics></math>
</td>
<td id="S6.T1.3.3.2" class="ltx_td ltx_align_center ltx_border_r">0.4824</td>
<td id="S6.T1.3.3.3" class="ltx_td ltx_align_center ltx_border_r">0.5342</td>
<td id="S6.T1.3.3.4" class="ltx_td ltx_align_center ltx_border_r">0.5477</td>
<td id="S6.T1.3.3.5" class="ltx_td ltx_align_center">0.5621</td>
</tr>
<tr id="S6.T1.4.9.5" class="ltx_tr">
<td id="S6.T1.4.9.5.1" class="ltx_td ltx_align_center ltx_border_t" colspan="5">
<span id="S6.T1.4.9.5.1.1" class="ltx_text ltx_font_bold">Supervised Matryoshka-Adaptor</span> (nDCG@10)</td>
</tr>
<tr id="S6.T1.4.10.6" class="ltx_tr">
<td id="S6.T1.4.10.6.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">Reduced dims</td>
<td id="S6.T1.4.10.6.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">64</td>
<td id="S6.T1.4.10.6.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">128</td>
<td id="S6.T1.4.10.6.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">256</td>
<td id="S6.T1.4.10.6.5" class="ltx_td ltx_align_center ltx_border_t">512</td>
</tr>
<tr id="S6.T1.4.11.7" class="ltx_tr">
<td id="S6.T1.4.11.7.1" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">Baseline</td>
<td id="S6.T1.4.11.7.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.4332</td>
<td id="S6.T1.4.11.7.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.5044</td>
<td id="S6.T1.4.11.7.4" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">0.5461</td>
<td id="S6.T1.4.11.7.5" class="ltx_td ltx_align_center ltx_border_t">0.5590</td>
</tr>
<tr id="S6.T1.4.12.8" class="ltx_tr">
<td id="S6.T1.4.12.8.1" class="ltx_td ltx_align_center ltx_border_r">All four losses</td>
<td id="S6.T1.4.12.8.2" class="ltx_td ltx_align_center ltx_border_r"><span id="S6.T1.4.12.8.2.1" class="ltx_text ltx_font_bold">0.5047</span></td>
<td id="S6.T1.4.12.8.3" class="ltx_td ltx_align_center ltx_border_r"><span id="S6.T1.4.12.8.3.1" class="ltx_text ltx_font_bold">0.5473</span></td>
<td id="S6.T1.4.12.8.4" class="ltx_td ltx_align_center ltx_border_r"><span id="S6.T1.4.12.8.4.1" class="ltx_text ltx_font_bold">0.5714</span></td>
<td id="S6.T1.4.12.8.5" class="ltx_td ltx_align_center"><span id="S6.T1.4.12.8.5.1" class="ltx_text ltx_font_bold">0.5902</span></td>
</tr>
<tr id="S6.T1.4.4" class="ltx_tr">
<td id="S6.T1.4.4.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t">Only with <math id="S6.T1.4.4.1.m1.1" class="ltx_Math" alttext="\mathcal{L}_{rank}" display="inline"><semantics id="S6.T1.4.4.1.m1.1a"><msub id="S6.T1.4.4.1.m1.1.1" xref="S6.T1.4.4.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.T1.4.4.1.m1.1.1.2" xref="S6.T1.4.4.1.m1.1.1.2.cmml">ℒ</mi><mrow id="S6.T1.4.4.1.m1.1.1.3" xref="S6.T1.4.4.1.m1.1.1.3.cmml"><mi id="S6.T1.4.4.1.m1.1.1.3.2" xref="S6.T1.4.4.1.m1.1.1.3.2.cmml">r</mi><mo lspace="0em" rspace="0em" id="S6.T1.4.4.1.m1.1.1.3.1" xref="S6.T1.4.4.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.T1.4.4.1.m1.1.1.3.3" xref="S6.T1.4.4.1.m1.1.1.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S6.T1.4.4.1.m1.1.1.3.1a" xref="S6.T1.4.4.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.T1.4.4.1.m1.1.1.3.4" xref="S6.T1.4.4.1.m1.1.1.3.4.cmml">n</mi><mo lspace="0em" rspace="0em" id="S6.T1.4.4.1.m1.1.1.3.1b" xref="S6.T1.4.4.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.T1.4.4.1.m1.1.1.3.5" xref="S6.T1.4.4.1.m1.1.1.3.5.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S6.T1.4.4.1.m1.1b"><apply id="S6.T1.4.4.1.m1.1.1.cmml" xref="S6.T1.4.4.1.m1.1.1"><csymbol cd="ambiguous" id="S6.T1.4.4.1.m1.1.1.1.cmml" xref="S6.T1.4.4.1.m1.1.1">subscript</csymbol><ci id="S6.T1.4.4.1.m1.1.1.2.cmml" xref="S6.T1.4.4.1.m1.1.1.2">ℒ</ci><apply id="S6.T1.4.4.1.m1.1.1.3.cmml" xref="S6.T1.4.4.1.m1.1.1.3"><times id="S6.T1.4.4.1.m1.1.1.3.1.cmml" xref="S6.T1.4.4.1.m1.1.1.3.1"></times><ci id="S6.T1.4.4.1.m1.1.1.3.2.cmml" xref="S6.T1.4.4.1.m1.1.1.3.2">𝑟</ci><ci id="S6.T1.4.4.1.m1.1.1.3.3.cmml" xref="S6.T1.4.4.1.m1.1.1.3.3">𝑎</ci><ci id="S6.T1.4.4.1.m1.1.1.3.4.cmml" xref="S6.T1.4.4.1.m1.1.1.3.4">𝑛</ci><ci id="S6.T1.4.4.1.m1.1.1.3.5.cmml" xref="S6.T1.4.4.1.m1.1.1.3.5">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T1.4.4.1.m1.1c">\mathcal{L}_{rank}</annotation></semantics></math>
</td>
<td id="S6.T1.4.4.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t">0.4767</td>
<td id="S6.T1.4.4.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t">0.5209</td>
<td id="S6.T1.4.4.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r ltx_border_t">0.5607</td>
<td id="S6.T1.4.4.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">0.5787</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Ablation studies on both supervised and unsupervised Matryoshka-Adaptor across 8 BEIR datasets.
The best performance in each scenario is in bold.</figcaption>
</figure>
<div id="S6.SS2.p2" class="ltx_para">
<p id="S6.SS2.p2.3" class="ltx_p">As evident in Table <a href="#S6.T1" title="Table 1 ‣ 6.2 Ablation studies ‣ 6 Discussions ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>, each loss function contributes uniquely to the overall performance of Matryoshka-Adaptor in the unsupervised setting.
Notably, <math id="S6.SS2.p2.1.m1.1" class="ltx_Math" alttext="\mathcal{L}_{topk}" display="inline"><semantics id="S6.SS2.p2.1.m1.1a"><msub id="S6.SS2.p2.1.m1.1.1" xref="S6.SS2.p2.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.SS2.p2.1.m1.1.1.2" xref="S6.SS2.p2.1.m1.1.1.2.cmml">ℒ</mi><mrow id="S6.SS2.p2.1.m1.1.1.3" xref="S6.SS2.p2.1.m1.1.1.3.cmml"><mi id="S6.SS2.p2.1.m1.1.1.3.2" xref="S6.SS2.p2.1.m1.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p2.1.m1.1.1.3.1" xref="S6.SS2.p2.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p2.1.m1.1.1.3.3" xref="S6.SS2.p2.1.m1.1.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p2.1.m1.1.1.3.1a" xref="S6.SS2.p2.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p2.1.m1.1.1.3.4" xref="S6.SS2.p2.1.m1.1.1.3.4.cmml">p</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p2.1.m1.1.1.3.1b" xref="S6.SS2.p2.1.m1.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p2.1.m1.1.1.3.5" xref="S6.SS2.p2.1.m1.1.1.3.5.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S6.SS2.p2.1.m1.1b"><apply id="S6.SS2.p2.1.m1.1.1.cmml" xref="S6.SS2.p2.1.m1.1.1"><csymbol cd="ambiguous" id="S6.SS2.p2.1.m1.1.1.1.cmml" xref="S6.SS2.p2.1.m1.1.1">subscript</csymbol><ci id="S6.SS2.p2.1.m1.1.1.2.cmml" xref="S6.SS2.p2.1.m1.1.1.2">ℒ</ci><apply id="S6.SS2.p2.1.m1.1.1.3.cmml" xref="S6.SS2.p2.1.m1.1.1.3"><times id="S6.SS2.p2.1.m1.1.1.3.1.cmml" xref="S6.SS2.p2.1.m1.1.1.3.1"></times><ci id="S6.SS2.p2.1.m1.1.1.3.2.cmml" xref="S6.SS2.p2.1.m1.1.1.3.2">𝑡</ci><ci id="S6.SS2.p2.1.m1.1.1.3.3.cmml" xref="S6.SS2.p2.1.m1.1.1.3.3">𝑜</ci><ci id="S6.SS2.p2.1.m1.1.1.3.4.cmml" xref="S6.SS2.p2.1.m1.1.1.3.4">𝑝</ci><ci id="S6.SS2.p2.1.m1.1.1.3.5.cmml" xref="S6.SS2.p2.1.m1.1.1.3.5">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p2.1.m1.1c">\mathcal{L}_{topk}</annotation></semantics></math> is particularly beneficial for higher dimensions.
In the supervised setting, the incorporation of the additional unsupervised losses, including <math id="S6.SS2.p2.2.m2.1" class="ltx_Math" alttext="\mathcal{L}_{topk}" display="inline"><semantics id="S6.SS2.p2.2.m2.1a"><msub id="S6.SS2.p2.2.m2.1.1" xref="S6.SS2.p2.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.SS2.p2.2.m2.1.1.2" xref="S6.SS2.p2.2.m2.1.1.2.cmml">ℒ</mi><mrow id="S6.SS2.p2.2.m2.1.1.3" xref="S6.SS2.p2.2.m2.1.1.3.cmml"><mi id="S6.SS2.p2.2.m2.1.1.3.2" xref="S6.SS2.p2.2.m2.1.1.3.2.cmml">t</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p2.2.m2.1.1.3.1" xref="S6.SS2.p2.2.m2.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p2.2.m2.1.1.3.3" xref="S6.SS2.p2.2.m2.1.1.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p2.2.m2.1.1.3.1a" xref="S6.SS2.p2.2.m2.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p2.2.m2.1.1.3.4" xref="S6.SS2.p2.2.m2.1.1.3.4.cmml">p</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p2.2.m2.1.1.3.1b" xref="S6.SS2.p2.2.m2.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p2.2.m2.1.1.3.5" xref="S6.SS2.p2.2.m2.1.1.3.5.cmml">k</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S6.SS2.p2.2.m2.1b"><apply id="S6.SS2.p2.2.m2.1.1.cmml" xref="S6.SS2.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S6.SS2.p2.2.m2.1.1.1.cmml" xref="S6.SS2.p2.2.m2.1.1">subscript</csymbol><ci id="S6.SS2.p2.2.m2.1.1.2.cmml" xref="S6.SS2.p2.2.m2.1.1.2">ℒ</ci><apply id="S6.SS2.p2.2.m2.1.1.3.cmml" xref="S6.SS2.p2.2.m2.1.1.3"><times id="S6.SS2.p2.2.m2.1.1.3.1.cmml" xref="S6.SS2.p2.2.m2.1.1.3.1"></times><ci id="S6.SS2.p2.2.m2.1.1.3.2.cmml" xref="S6.SS2.p2.2.m2.1.1.3.2">𝑡</ci><ci id="S6.SS2.p2.2.m2.1.1.3.3.cmml" xref="S6.SS2.p2.2.m2.1.1.3.3">𝑜</ci><ci id="S6.SS2.p2.2.m2.1.1.3.4.cmml" xref="S6.SS2.p2.2.m2.1.1.3.4">𝑝</ci><ci id="S6.SS2.p2.2.m2.1.1.3.5.cmml" xref="S6.SS2.p2.2.m2.1.1.3.5">𝑘</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p2.2.m2.1c">\mathcal{L}_{topk}</annotation></semantics></math> and <math id="S6.SS2.p2.3.m3.1" class="ltx_Math" alttext="\mathcal{L}_{pair}" display="inline"><semantics id="S6.SS2.p2.3.m3.1a"><msub id="S6.SS2.p2.3.m3.1.1" xref="S6.SS2.p2.3.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.SS2.p2.3.m3.1.1.2" xref="S6.SS2.p2.3.m3.1.1.2.cmml">ℒ</mi><mrow id="S6.SS2.p2.3.m3.1.1.3" xref="S6.SS2.p2.3.m3.1.1.3.cmml"><mi id="S6.SS2.p2.3.m3.1.1.3.2" xref="S6.SS2.p2.3.m3.1.1.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p2.3.m3.1.1.3.1" xref="S6.SS2.p2.3.m3.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p2.3.m3.1.1.3.3" xref="S6.SS2.p2.3.m3.1.1.3.3.cmml">a</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p2.3.m3.1.1.3.1a" xref="S6.SS2.p2.3.m3.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p2.3.m3.1.1.3.4" xref="S6.SS2.p2.3.m3.1.1.3.4.cmml">i</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p2.3.m3.1.1.3.1b" xref="S6.SS2.p2.3.m3.1.1.3.1.cmml">​</mo><mi id="S6.SS2.p2.3.m3.1.1.3.5" xref="S6.SS2.p2.3.m3.1.1.3.5.cmml">r</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S6.SS2.p2.3.m3.1b"><apply id="S6.SS2.p2.3.m3.1.1.cmml" xref="S6.SS2.p2.3.m3.1.1"><csymbol cd="ambiguous" id="S6.SS2.p2.3.m3.1.1.1.cmml" xref="S6.SS2.p2.3.m3.1.1">subscript</csymbol><ci id="S6.SS2.p2.3.m3.1.1.2.cmml" xref="S6.SS2.p2.3.m3.1.1.2">ℒ</ci><apply id="S6.SS2.p2.3.m3.1.1.3.cmml" xref="S6.SS2.p2.3.m3.1.1.3"><times id="S6.SS2.p2.3.m3.1.1.3.1.cmml" xref="S6.SS2.p2.3.m3.1.1.3.1"></times><ci id="S6.SS2.p2.3.m3.1.1.3.2.cmml" xref="S6.SS2.p2.3.m3.1.1.3.2">𝑝</ci><ci id="S6.SS2.p2.3.m3.1.1.3.3.cmml" xref="S6.SS2.p2.3.m3.1.1.3.3">𝑎</ci><ci id="S6.SS2.p2.3.m3.1.1.3.4.cmml" xref="S6.SS2.p2.3.m3.1.1.3.4">𝑖</ci><ci id="S6.SS2.p2.3.m3.1.1.3.5.cmml" xref="S6.SS2.p2.3.m3.1.1.3.5">𝑟</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p2.3.m3.1c">\mathcal{L}_{pair}</annotation></semantics></math>, is critical for achieving performance improvements.
Without these unsupervised losses, the performance gain diminishes by approximately 50%.</p>
</div>
</section>
<section id="S6.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.3 </span>Relationship with distance metrics</h3>

<div id="S6.SS3.p1" class="ltx_para">
<p id="S6.SS3.p1.1" class="ltx_p">In unsupervised learning setup, the absence of labeled data presents a challenge in evaluating the impact of Matryoshka-Adaptor.
Towards circumventing this issue and shedding light on the impact of dimensionality reduction, we analyze the average pairwise distances and top-k distances within the corpus embeddings on the disjoint validation corpus.
Subsequently, we examine the relationship between these unsupervised metrics and the supervised retrieval metric nDCG@10, utilizing 8 BEIR datasets with Google Gecko embedding models.</p>
</div>
<div id="S6.SS3.p2" class="ltx_para">
<p id="S6.SS3.p2.1" class="ltx_p">Fig. <a href="#S6.F7.sf1" title="In Figure 7 ‣ 6.1 Models that are not pretrained with MRL ‣ 6 Discussions ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7(a)</span></a> reveals that the unsupervised Matryoshka-Adaptor results in significantly lower distance metrics compared to the original embeddings.
This suggests that the proposed method effectively preserves both pairwise and top-k distances among corpus embeddings when employing a reduced subset of embedding dimensions.
These distance metrics can serve as viable unsupervised proxies for assessing the efficacy of the unsupervised Matryoshka-Adaptor.
To further elucidate this relationship, we also show the correlation between these distance metrics and nDCG@10 (Fig. <a href="#S6.F7.sf2" title="In Figure 7 ‣ 6.1 Models that are not pretrained with MRL ‣ 6 Discussions ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7(b)</span></a> and <a href="#S6.F7.sf3" title="In Figure 7 ‣ 6.1 Models that are not pretrained with MRL ‣ 6 Discussions ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7(c)</span></a>).
A strong correlation is observed, indicating that lower distance metrics are associated with smaller performance drop in retrieval tasks, even when using reduced embedding dimensions.
This suggests that the unsupervised distance metrics can be indicators of retrieval performance, even in the absence of labeled data.</p>
</div>
</section>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Conclusions</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.1" class="ltx_p">While high-dimensional embeddings often provide a richer representation of the original data, their practical deployment in real-world applications is frequently hampered by computational costs and latency issues.
As a result, many applications opt for lower-dimensional embeddings, accepting a compromise in performance.
The presented Matryoshka-Adaptor framework offers a solution to this dilemma by enabling substantial dimensionality reduction while minimizing performance degradation, in both supervised and unsupervised scenarios.
Furthermore, due to its versatile nature and applicability to any embedding model, regardless of access to model parameters, the Matryoshka-Adaptor constitutes a valuable tool for improving the efficiency and practicality of embedding-based applications.</p>
</div>
</section>
<section id="S8" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">8 </span>Limitations and Future Works</h2>

<div id="S8.p1" class="ltx_para">
<p id="S8.p1.1" class="ltx_p">In the unsupervised setting, picking the optimal hyperparameters for Matryoshka-Adaptor poses a challenge due to the absence of validation data.
While alternative metrics, such as the proposed distance metrics illustrated in Fig. <a href="#S6.F7" title="Figure 7 ‣ 6.1 Models that are not pretrained with MRL ‣ 6 Discussions ‣ Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a>, can be employed, these may exhibit higher levels of noise compared to supervised validation metrics.
One major risk associated with Matryoshka-Adaptor lies in the potential for overfitting to the tuning data, during the tuning process.
Future research endeavors might include extending the proposed approach to encompass multiple modalities in the tuning objective;
exploring semi-supervised variations; and
investigating simultaneous utilization of multiple datasets during tuning.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(1)</span>
<span class="ltx_bibblock">

</span>
<span class="ltx_bibblock">Beir data repository.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/beir-cellar/beir" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/beir-cellar/beir</a>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(2)</span>
<span class="ltx_bibblock">

</span>
<span class="ltx_bibblock">Google cloud text embedding.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://cloud.google.com/vertex-ai/docs/generative-ai/embeddings/get-text-embeddings" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://cloud.google.com/vertex-ai/docs/generative-ai/embeddings/get-text-embeddings</a>.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(3)</span>
<span class="ltx_bibblock">

</span>
<span class="ltx_bibblock">Miracl data repository.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://project-miracl.github.io/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://project-miracl.github.io/</a>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(4)</span>
<span class="ltx_bibblock">

</span>
<span class="ltx_bibblock">Openai text embedding.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://https://openai.com/index/new-embedding-models-and-api-updates" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://https://openai.com/index/new-embedding-models-and-api-updates</a>.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Asai et al. (2022)</span>
<span class="ltx_bibblock">
Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian
Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. 2022.

</span>
<span class="ltx_bibblock">Task-aware retrieval with instructions.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2211.09260</em>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span>
<span class="ltx_bibblock">
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
et al. 2020.

</span>
<span class="ltx_bibblock">Language models are few-shot learners.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>,
33:1877–1901.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cai et al. (2024)</span>
<span class="ltx_bibblock">
Mu Cai, Jianwei Yang, Jianfeng Gao, and Yong Jae Lee. 2024.

</span>
<span class="ltx_bibblock">Matryoshka multimodal models.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2405.17430</em>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chowdhery et al. (2022)</span>
<span class="ltx_bibblock">
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra,
Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian
Gehrmann, et al. 2022.

</span>
<span class="ltx_bibblock">Palm: Scaling language modeling with pathways.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2204.02311</em>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dai et al. (2022)</span>
<span class="ltx_bibblock">
Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov,
Kelvin Guu, Keith B Hall, and Ming-Wei Chang. 2022.

</span>
<span class="ltx_bibblock">Promptagator: Few-shot dense retrieval from 8 examples.

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2209.11755</em>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Han et al. (2017)</span>
<span class="ltx_bibblock">
Xintong Han, Zuxuan Wu, Phoenix X Huang, Xiao Zhang, Menglong Zhu, Yuan Li,
Yang Zhao, and Larry S Davis. 2017.

</span>
<span class="ltx_bibblock">Automatic spatially-aware fashion concept discovery.

</span>
<span class="ltx_bibblock">In <em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">Proceedings of the IEEE international conference on computer
vision</em>, pages 1463–1471.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hyvärinen and Oja (2000)</span>
<span class="ltx_bibblock">
Aapo Hyvärinen and Erkki Oja. 2000.

</span>
<span class="ltx_bibblock">Independent component analysis: algorithms and applications.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">Neural networks</em>, 13(4-5):411–430.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Järvelin and Kekäläinen (2002)</span>
<span class="ltx_bibblock">
Kalervo Järvelin and Jaana Kekäläinen. 2002.

</span>
<span class="ltx_bibblock">Cumulated gain-based evaluation of ir techniques.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">ACM Transactions on Information Systems (TOIS)</em>,
20(4):422–446.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jolliffe and Cadima (2016)</span>
<span class="ltx_bibblock">
Ian T Jolliffe and Jorge Cadima. 2016.

</span>
<span class="ltx_bibblock">Principal component analysis: a review and recent developments.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Philosophical transactions of the royal society A:
Mathematical, Physical and Engineering Sciences</em>, 374(2065):20150202.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kusupati et al. (2022)</span>
<span class="ltx_bibblock">
Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya
Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade,
Prateek Jain, et al. 2022.

</span>
<span class="ltx_bibblock">Matryoshka representation learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
35:30233–30249.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee and Seung (2000)</span>
<span class="ltx_bibblock">
Daniel Lee and H Sebastian Seung. 2000.

</span>
<span class="ltx_bibblock">Algorithms for non-negative matrix factorization.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>, 13.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee et al. (2024)</span>
<span class="ltx_bibblock">
Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R Cole, Kai
Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, et al. 2024.

</span>
<span class="ltx_bibblock">Gecko: Versatile text embeddings distilled from large language
models.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2403.20327</em>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Li et al. (2023)</span>
<span class="ltx_bibblock">
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan
Zhang. 2023.

</span>
<span class="ltx_bibblock">Towards general text embeddings with multi-stage contrastive
learning.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2308.03281</em>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">McLachlan (2005)</span>
<span class="ltx_bibblock">
Geoffrey J McLachlan. 2005.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">Discriminant analysis and statistical pattern recognition</em>.

</span>
<span class="ltx_bibblock">John Wiley &amp; Sons.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Muennighoff et al. (2024)</span>
<span class="ltx_bibblock">
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu,
Amanpreet Singh, and Douwe Kiela. 2024.

</span>
<span class="ltx_bibblock">Generative representational instruction tuning.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2402.09906</em>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ni et al. (2021)</span>
<span class="ltx_bibblock">
Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B
Hall, Daniel Cer, and Yinfei Yang. 2021.

</span>
<span class="ltx_bibblock">Sentence-t5: Scalable sentence encoders from pre-trained text-to-text
models.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2108.08877</em>.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ouyang et al. (2022)</span>
<span class="ltx_bibblock">
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela
Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.
2022.

</span>
<span class="ltx_bibblock">Training language models to follow instructions with human feedback.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems</em>,
35:27730–27744.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Su et al. (2022)</span>
<span class="ltx_bibblock">
Hongjin Su, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih,
Noah A Smith, Luke Zettlemoyer, Tao Yu, et al. 2022.

</span>
<span class="ltx_bibblock">One embedder, any task: Instruction-finetuned text embeddings.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2212.09741</em>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Thakur et al. (2021)</span>
<span class="ltx_bibblock">
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and
Iryna Gurevych. 2021.

</span>
<span class="ltx_bibblock">Beir: A heterogenous benchmark for zero-shot evaluation of
information retrieval models.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2104.08663</em>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2023)</span>
<span class="ltx_bibblock">
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu
Wei. 2023.

</span>
<span class="ltx_bibblock">Improving text embeddings with large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2401.00368</em>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yoon et al. (2023)</span>
<span class="ltx_bibblock">
Jinsung Yoon, Sercan O Arik, Yanfei Chen, and Tomas Pfister. 2023.

</span>
<span class="ltx_bibblock">Search-adaptor: Text embedding customization for information
retrieval.

</span>
<span class="ltx_bibblock"><em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.08750</em>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2022a)</span>
<span class="ltx_bibblock">
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui
Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.
2022a.

</span>
<span class="ltx_bibblock">Opt: Open pre-trained transformer language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2205.01068</em>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2022b)</span>
<span class="ltx_bibblock">
Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David
Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin.
2022b.

</span>
<span class="ltx_bibblock">Making a miracl: Multilingual information retrieval across a
continuum of languages.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2210.09984</em>.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Data Statistics</h2>

<section id="A1.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.1 </span>BEIR datasets</h3>

<figure id="A1.T2" class="ltx_table">
<table id="A1.T2.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T2.1.1.1" class="ltx_tr">
<th id="A1.T2.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt" rowspan="2"><span id="A1.T2.1.1.1.1.1" class="ltx_text">Datasets</span></th>
<th id="A1.T2.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt">Number of</th>
<th id="A1.T2.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt">Number of</th>
<th id="A1.T2.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Number of</th>
</tr>
<tr id="A1.T2.1.2.2" class="ltx_tr">
<th id="A1.T2.1.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">train pairs</th>
<th id="A1.T2.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">test pairs</th>
<th id="A1.T2.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">corpus</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T2.1.3.1" class="ltx_tr">
<th id="A1.T2.1.3.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">NFCorpus</th>
<td id="A1.T2.1.3.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">110575</td>
<td id="A1.T2.1.3.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">12334</td>
<td id="A1.T2.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t">3633</td>
</tr>
<tr id="A1.T2.1.4.2" class="ltx_tr">
<th id="A1.T2.1.4.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">SciFact</th>
<td id="A1.T2.1.4.2.2" class="ltx_td ltx_align_center ltx_border_r">919</td>
<td id="A1.T2.1.4.2.3" class="ltx_td ltx_align_center ltx_border_r">339</td>
<td id="A1.T2.1.4.2.4" class="ltx_td ltx_align_center">5183</td>
</tr>
<tr id="A1.T2.1.5.3" class="ltx_tr">
<th id="A1.T2.1.5.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Arguana</th>
<td id="A1.T2.1.5.3.2" class="ltx_td ltx_align_center ltx_border_r">703</td>
<td id="A1.T2.1.5.3.3" class="ltx_td ltx_align_center ltx_border_r">703</td>
<td id="A1.T2.1.5.3.4" class="ltx_td ltx_align_center">8674</td>
</tr>
<tr id="A1.T2.1.6.4" class="ltx_tr">
<th id="A1.T2.1.6.4.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">SciDocs</th>
<td id="A1.T2.1.6.4.2" class="ltx_td ltx_align_center ltx_border_r">14972</td>
<td id="A1.T2.1.6.4.3" class="ltx_td ltx_align_center ltx_border_r">14956</td>
<td id="A1.T2.1.6.4.4" class="ltx_td ltx_align_center">25657</td>
</tr>
<tr id="A1.T2.1.7.5" class="ltx_tr">
<th id="A1.T2.1.7.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">FiQA</th>
<td id="A1.T2.1.7.5.2" class="ltx_td ltx_align_center ltx_border_r">14166</td>
<td id="A1.T2.1.7.5.3" class="ltx_td ltx_align_center ltx_border_r">1706</td>
<td id="A1.T2.1.7.5.4" class="ltx_td ltx_align_center">57638</td>
</tr>
<tr id="A1.T2.1.8.6" class="ltx_tr">
<th id="A1.T2.1.8.6.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Trec-Covid</th>
<td id="A1.T2.1.8.6.2" class="ltx_td ltx_align_center ltx_border_r">35460</td>
<td id="A1.T2.1.8.6.3" class="ltx_td ltx_align_center ltx_border_r">30876</td>
<td id="A1.T2.1.8.6.4" class="ltx_td ltx_align_center">171332</td>
</tr>
<tr id="A1.T2.1.9.7" class="ltx_tr">
<th id="A1.T2.1.9.7.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Touche</th>
<td id="A1.T2.1.9.7.2" class="ltx_td ltx_align_center ltx_border_r">1077</td>
<td id="A1.T2.1.9.7.3" class="ltx_td ltx_align_center ltx_border_r">1137</td>
<td id="A1.T2.1.9.7.4" class="ltx_td ltx_align_center">382545</td>
</tr>
<tr id="A1.T2.1.10.8" class="ltx_tr">
<th id="A1.T2.1.10.8.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Quora</th>
<td id="A1.T2.1.10.8.2" class="ltx_td ltx_align_center ltx_border_r">7626</td>
<td id="A1.T2.1.10.8.3" class="ltx_td ltx_align_center ltx_border_r">15675</td>
<td id="A1.T2.1.10.8.4" class="ltx_td ltx_align_center">522931</td>
</tr>
<tr id="A1.T2.1.11.9" class="ltx_tr">
<th id="A1.T2.1.11.9.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">NQ</th>
<td id="A1.T2.1.11.9.2" class="ltx_td ltx_align_center ltx_border_r">2097</td>
<td id="A1.T2.1.11.9.3" class="ltx_td ltx_align_center ltx_border_r">2104</td>
<td id="A1.T2.1.11.9.4" class="ltx_td ltx_align_center">2681468</td>
</tr>
<tr id="A1.T2.1.12.10" class="ltx_tr">
<th id="A1.T2.1.12.10.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">DBPedia</th>
<td id="A1.T2.1.12.10.2" class="ltx_td ltx_align_center ltx_border_r">5673</td>
<td id="A1.T2.1.12.10.3" class="ltx_td ltx_align_center ltx_border_r">43515</td>
<td id="A1.T2.1.12.10.4" class="ltx_td ltx_align_center">4635922</td>
</tr>
<tr id="A1.T2.1.13.11" class="ltx_tr">
<th id="A1.T2.1.13.11.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">HotPotQA</th>
<td id="A1.T2.1.13.11.2" class="ltx_td ltx_align_center ltx_border_r">170000</td>
<td id="A1.T2.1.13.11.3" class="ltx_td ltx_align_center ltx_border_r">14810</td>
<td id="A1.T2.1.13.11.4" class="ltx_td ltx_align_center">5233329</td>
</tr>
<tr id="A1.T2.1.14.12" class="ltx_tr">
<th id="A1.T2.1.14.12.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Fever</th>
<td id="A1.T2.1.14.12.2" class="ltx_td ltx_align_center ltx_border_r">140085</td>
<td id="A1.T2.1.14.12.3" class="ltx_td ltx_align_center ltx_border_r">7937</td>
<td id="A1.T2.1.14.12.4" class="ltx_td ltx_align_center">5416568</td>
</tr>
<tr id="A1.T2.1.15.13" class="ltx_tr">
<th id="A1.T2.1.15.13.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb ltx_border_r">Climate-fever</th>
<td id="A1.T2.1.15.13.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">2299</td>
<td id="A1.T2.1.15.13.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">2382</td>
<td id="A1.T2.1.15.13.4" class="ltx_td ltx_align_center ltx_border_bb">5416593</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>The statistics of 13 BEIR datasets (sorted by the number of corpus).</figcaption>
</figure>
</section>
<section id="A1.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.2 </span>MIRACL datasets</h3>

<figure id="A1.T3" class="ltx_table">
<table id="A1.T3.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T3.1.1.1" class="ltx_tr">
<th id="A1.T3.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt" rowspan="2"><span id="A1.T3.1.1.1.1.1" class="ltx_text">Datasets</span></th>
<th id="A1.T3.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt">Number of</th>
<th id="A1.T3.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt">Number of</th>
<th id="A1.T3.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Number of</th>
</tr>
<tr id="A1.T3.1.2.2" class="ltx_tr">
<th id="A1.T3.1.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">train pairs</th>
<th id="A1.T3.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">test pairs</th>
<th id="A1.T3.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">corpus</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T3.1.3.1" class="ltx_tr">
<th id="A1.T3.1.3.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">Yoruba (yo)</th>
<td id="A1.T3.1.3.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">959</td>
<td id="A1.T3.1.3.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">229</td>
<td id="A1.T3.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t">49043</td>
</tr>
<tr id="A1.T3.1.4.2" class="ltx_tr">
<th id="A1.T3.1.4.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Swahilli (sw)</th>
<td id="A1.T3.1.4.2.2" class="ltx_td ltx_align_center ltx_border_r">9359</td>
<td id="A1.T3.1.4.2.3" class="ltx_td ltx_align_center ltx_border_r">5092</td>
<td id="A1.T3.1.4.2.4" class="ltx_td ltx_align_center">131924</td>
</tr>
<tr id="A1.T3.1.5.3" class="ltx_tr">
<th id="A1.T3.1.5.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Bengali (bn)</th>
<td id="A1.T3.1.5.3.2" class="ltx_td ltx_align_center ltx_border_r">16754</td>
<td id="A1.T3.1.5.3.3" class="ltx_td ltx_align_center ltx_border_r">4206</td>
<td id="A1.T3.1.5.3.4" class="ltx_td ltx_align_center">297265</td>
</tr>
<tr id="A1.T3.1.6.4" class="ltx_tr">
<th id="A1.T3.1.6.4.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Hindi (hi)</th>
<td id="A1.T3.1.6.4.2" class="ltx_td ltx_align_center ltx_border_r">11668</td>
<td id="A1.T3.1.6.4.3" class="ltx_td ltx_align_center ltx_border_r">3494</td>
<td id="A1.T3.1.6.4.4" class="ltx_td ltx_align_center">506264</td>
</tr>
<tr id="A1.T3.1.7.5" class="ltx_tr">
<th id="A1.T3.1.7.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Telugu (te)</th>
<td id="A1.T3.1.7.5.2" class="ltx_td ltx_align_center ltx_border_r">18608</td>
<td id="A1.T3.1.7.5.3" class="ltx_td ltx_align_center ltx_border_r">1606</td>
<td id="A1.T3.1.7.5.4" class="ltx_td ltx_align_center">518079</td>
</tr>
<tr id="A1.T3.1.8.6" class="ltx_tr">
<th id="A1.T3.1.8.6.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Thai (th)</th>
<td id="A1.T3.1.8.6.2" class="ltx_td ltx_align_center ltx_border_r">21293</td>
<td id="A1.T3.1.8.6.3" class="ltx_td ltx_align_center ltx_border_r">7573</td>
<td id="A1.T3.1.8.6.4" class="ltx_td ltx_align_center">542166</td>
</tr>
<tr id="A1.T3.1.9.7" class="ltx_tr">
<th id="A1.T3.1.9.7.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Indonesian (id)</th>
<td id="A1.T3.1.9.7.2" class="ltx_td ltx_align_center ltx_border_r">41358</td>
<td id="A1.T3.1.9.7.3" class="ltx_td ltx_align_center ltx_border_r">9668</td>
<td id="A1.T3.1.9.7.4" class="ltx_td ltx_align_center">1446315</td>
</tr>
<tr id="A1.T3.1.10.8" class="ltx_tr">
<th id="A1.T3.1.10.8.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Korean (ko)</th>
<td id="A1.T3.1.10.8.2" class="ltx_td ltx_align_center ltx_border_r">12767</td>
<td id="A1.T3.1.10.8.3" class="ltx_td ltx_align_center ltx_border_r">3057</td>
<td id="A1.T3.1.10.8.4" class="ltx_td ltx_align_center">1486752</td>
</tr>
<tr id="A1.T3.1.11.9" class="ltx_tr">
<th id="A1.T3.1.11.9.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Finnish (fi)</th>
<td id="A1.T3.1.11.9.2" class="ltx_td ltx_align_center ltx_border_r">20350</td>
<td id="A1.T3.1.11.9.3" class="ltx_td ltx_align_center ltx_border_r">12008</td>
<td id="A1.T3.1.11.9.4" class="ltx_td ltx_align_center">1883509</td>
</tr>
<tr id="A1.T3.1.12.10" class="ltx_tr">
<th id="A1.T3.1.12.10.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Arabic (ar)</th>
<td id="A1.T3.1.12.10.2" class="ltx_td ltx_align_center ltx_border_r">25382</td>
<td id="A1.T3.1.12.10.3" class="ltx_td ltx_align_center ltx_border_r">29197</td>
<td id="A1.T3.1.12.10.4" class="ltx_td ltx_align_center">2061414</td>
</tr>
<tr id="A1.T3.1.13.11" class="ltx_tr">
<th id="A1.T3.1.13.11.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Persian (fa)</th>
<td id="A1.T3.1.13.11.2" class="ltx_td ltx_align_center ltx_border_r">21844</td>
<td id="A1.T3.1.13.11.3" class="ltx_td ltx_align_center ltx_border_r">6571</td>
<td id="A1.T3.1.13.11.4" class="ltx_td ltx_align_center">2207172</td>
</tr>
<tr id="A1.T3.1.14.12" class="ltx_tr">
<th id="A1.T3.1.14.12.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Chinese (zh)</th>
<td id="A1.T3.1.14.12.2" class="ltx_td ltx_align_center ltx_border_r">13113</td>
<td id="A1.T3.1.14.12.3" class="ltx_td ltx_align_center ltx_border_r">3928</td>
<td id="A1.T3.1.14.12.4" class="ltx_td ltx_align_center">4934368</td>
</tr>
<tr id="A1.T3.1.15.13" class="ltx_tr">
<th id="A1.T3.1.15.13.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Japanese (ja)</th>
<td id="A1.T3.1.15.13.2" class="ltx_td ltx_align_center ltx_border_r">34387</td>
<td id="A1.T3.1.15.13.3" class="ltx_td ltx_align_center ltx_border_r">8354</td>
<td id="A1.T3.1.15.13.4" class="ltx_td ltx_align_center">6953614</td>
</tr>
<tr id="A1.T3.1.16.14" class="ltx_tr">
<th id="A1.T3.1.16.14.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Russian (ru)</th>
<td id="A1.T3.1.16.14.2" class="ltx_td ltx_align_center ltx_border_r">33921</td>
<td id="A1.T3.1.16.14.3" class="ltx_td ltx_align_center ltx_border_r">13100</td>
<td id="A1.T3.1.16.14.4" class="ltx_td ltx_align_center">9543918</td>
</tr>
<tr id="A1.T3.1.17.15" class="ltx_tr">
<th id="A1.T3.1.17.15.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Spanish (es)</th>
<td id="A1.T3.1.17.15.2" class="ltx_td ltx_align_center ltx_border_r">21531</td>
<td id="A1.T3.1.17.15.3" class="ltx_td ltx_align_center ltx_border_r">6443</td>
<td id="A1.T3.1.17.15.4" class="ltx_td ltx_align_center">10373953</td>
</tr>
<tr id="A1.T3.1.18.16" class="ltx_tr">
<th id="A1.T3.1.18.16.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">French (fr)</th>
<td id="A1.T3.1.18.16.2" class="ltx_td ltx_align_center ltx_border_r">11426</td>
<td id="A1.T3.1.18.16.3" class="ltx_td ltx_align_center ltx_border_r">3429</td>
<td id="A1.T3.1.18.16.4" class="ltx_td ltx_align_center">14636953</td>
</tr>
<tr id="A1.T3.1.19.17" class="ltx_tr">
<th id="A1.T3.1.19.17.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb ltx_border_r">Germany (de)</th>
<td id="A1.T3.1.19.17.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">2526</td>
<td id="A1.T3.1.19.17.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">628</td>
<td id="A1.T3.1.19.17.4" class="ltx_td ltx_align_center ltx_border_bb">15866222</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>The statistics of 17 MIRACL datasets (sorted by the number of corpus).</figcaption>
</figure>
</section>
<section id="A1.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">A.3 </span>Fashion-200K datasets</h3>

<figure id="A1.T4" class="ltx_table">
<table id="A1.T4.1" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T4.1.1.1" class="ltx_tr">
<th id="A1.T4.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_th_row ltx_border_r ltx_border_tt" rowspan="2"><span id="A1.T4.1.1.1.1.1" class="ltx_text">Datasets</span></th>
<th id="A1.T4.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt">Number of</th>
<th id="A1.T4.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r ltx_border_tt">Number of</th>
<th id="A1.T4.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt">Number of</th>
</tr>
<tr id="A1.T4.1.2.2" class="ltx_tr">
<th id="A1.T4.1.2.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">train pairs</th>
<th id="A1.T4.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_r">test pairs</th>
<th id="A1.T4.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">corpus</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T4.1.3.1" class="ltx_tr">
<th id="A1.T4.1.3.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r ltx_border_t">Dresses</th>
<td id="A1.T4.1.3.1.2" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">15127</td>
<td id="A1.T4.1.3.1.3" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">1567</td>
<td id="A1.T4.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t">72376</td>
</tr>
<tr id="A1.T4.1.4.2" class="ltx_tr">
<th id="A1.T4.1.4.2.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Jackets</th>
<td id="A1.T4.1.4.2.2" class="ltx_td ltx_align_center ltx_border_r">8105</td>
<td id="A1.T4.1.4.2.3" class="ltx_td ltx_align_center ltx_border_r">1511</td>
<td id="A1.T4.1.4.2.4" class="ltx_td ltx_align_center">71118</td>
</tr>
<tr id="A1.T4.1.5.3" class="ltx_tr">
<th id="A1.T4.1.5.3.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Pants</th>
<td id="A1.T4.1.5.3.2" class="ltx_td ltx_align_center ltx_border_r">9264</td>
<td id="A1.T4.1.5.3.3" class="ltx_td ltx_align_center ltx_border_r">1758</td>
<td id="A1.T4.1.5.3.4" class="ltx_td ltx_align_center">74470</td>
</tr>
<tr id="A1.T4.1.6.4" class="ltx_tr">
<th id="A1.T4.1.6.4.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_r">Skirts</th>
<td id="A1.T4.1.6.4.2" class="ltx_td ltx_align_center ltx_border_r">6822</td>
<td id="A1.T4.1.6.4.3" class="ltx_td ltx_align_center ltx_border_r">1247</td>
<td id="A1.T4.1.6.4.4" class="ltx_td ltx_align_center">47931</td>
</tr>
<tr id="A1.T4.1.7.5" class="ltx_tr">
<th id="A1.T4.1.7.5.1" class="ltx_td ltx_align_center ltx_th ltx_th_row ltx_border_bb ltx_border_r">Tops</th>
<td id="A1.T4.1.7.5.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">13809</td>
<td id="A1.T4.1.7.5.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_r">2536</td>
<td id="A1.T4.1.7.5.4" class="ltx_td ltx_align_center ltx_border_bb">72444</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 4: </span>The statistics of 5 Fashion-200K datasets.</figcaption>
</figure>
</section>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Hyper-parameters</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.1" class="ltx_p">We summarize the hyper-parameters used to train Matryoshka-Adaptor.
In all experiments (in both unsupervised and supervised settings), we utilize the fixed hyper-parameters that enable applying Matryoshka-Adaptor without extensive hyper-parameter tuning.</p>
</div>
<figure id="A2.T5" class="ltx_table">
<table id="A2.T5.6" class="ltx_tabular ltx_centering ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A2.T5.6.7.1" class="ltx_tr">
<th id="A2.T5.6.7.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_th ltx_th_column ltx_border_r ltx_border_tt">
<span id="A2.T5.6.7.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T5.6.7.1.1.1.1" class="ltx_p" style="width:142.3pt;">Hyper-parameters</span>
</span>
</th>
<th id="A2.T5.6.7.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt">Fixed values</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A2.T5.2.2" class="ltx_tr">
<td id="A2.T5.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_r ltx_border_t">
<span id="A2.T5.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T5.1.1.1.1.1" class="ltx_p" style="width:142.3pt;">Pairwise loss coefficient (<math id="A2.T5.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\alpha" display="inline"><semantics id="A2.T5.1.1.1.1.1.m1.1a"><mi id="A2.T5.1.1.1.1.1.m1.1.1" xref="A2.T5.1.1.1.1.1.m1.1.1.cmml">α</mi><annotation-xml encoding="MathML-Content" id="A2.T5.1.1.1.1.1.m1.1b"><ci id="A2.T5.1.1.1.1.1.m1.1.1.cmml" xref="A2.T5.1.1.1.1.1.m1.1.1">𝛼</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.1.1.1.1.1.m1.1c">\alpha</annotation></semantics></math>)</span>
</span>
</td>
<td id="A2.T5.2.2.2" class="ltx_td ltx_align_left ltx_border_t"><math id="A2.T5.2.2.2.m1.1" class="ltx_Math" alttext="1.0" display="inline"><semantics id="A2.T5.2.2.2.m1.1a"><mn id="A2.T5.2.2.2.m1.1.1" xref="A2.T5.2.2.2.m1.1.1.cmml">1.0</mn><annotation-xml encoding="MathML-Content" id="A2.T5.2.2.2.m1.1b"><cn type="float" id="A2.T5.2.2.2.m1.1.1.cmml" xref="A2.T5.2.2.2.m1.1.1">1.0</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.2.2.2.m1.1c">1.0</annotation></semantics></math></td>
</tr>
<tr id="A2.T5.4.4" class="ltx_tr">
<td id="A2.T5.3.3.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_r">
<span id="A2.T5.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T5.3.3.1.1.1" class="ltx_p" style="width:142.3pt;">Recovery loss coefficient (<math id="A2.T5.3.3.1.1.1.m1.1" class="ltx_Math" alttext="\beta" display="inline"><semantics id="A2.T5.3.3.1.1.1.m1.1a"><mi id="A2.T5.3.3.1.1.1.m1.1.1" xref="A2.T5.3.3.1.1.1.m1.1.1.cmml">β</mi><annotation-xml encoding="MathML-Content" id="A2.T5.3.3.1.1.1.m1.1b"><ci id="A2.T5.3.3.1.1.1.m1.1.1.cmml" xref="A2.T5.3.3.1.1.1.m1.1.1">𝛽</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.3.3.1.1.1.m1.1c">\beta</annotation></semantics></math>)</span>
</span>
</td>
<td id="A2.T5.4.4.2" class="ltx_td ltx_align_left"><math id="A2.T5.4.4.2.m1.1" class="ltx_Math" alttext="1.0" display="inline"><semantics id="A2.T5.4.4.2.m1.1a"><mn id="A2.T5.4.4.2.m1.1.1" xref="A2.T5.4.4.2.m1.1.1.cmml">1.0</mn><annotation-xml encoding="MathML-Content" id="A2.T5.4.4.2.m1.1b"><cn type="float" id="A2.T5.4.4.2.m1.1.1.cmml" xref="A2.T5.4.4.2.m1.1.1">1.0</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.4.4.2.m1.1c">1.0</annotation></semantics></math></td>
</tr>
<tr id="A2.T5.6.6" class="ltx_tr">
<td id="A2.T5.5.5.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_r">
<span id="A2.T5.5.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T5.5.5.1.1.1" class="ltx_p" style="width:142.3pt;">Ranking loss coefficient (<math id="A2.T5.5.5.1.1.1.m1.1" class="ltx_Math" alttext="\gamma" display="inline"><semantics id="A2.T5.5.5.1.1.1.m1.1a"><mi id="A2.T5.5.5.1.1.1.m1.1.1" xref="A2.T5.5.5.1.1.1.m1.1.1.cmml">γ</mi><annotation-xml encoding="MathML-Content" id="A2.T5.5.5.1.1.1.m1.1b"><ci id="A2.T5.5.5.1.1.1.m1.1.1.cmml" xref="A2.T5.5.5.1.1.1.m1.1.1">𝛾</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.5.5.1.1.1.m1.1c">\gamma</annotation></semantics></math>)</span>
</span>
</td>
<td id="A2.T5.6.6.2" class="ltx_td ltx_align_left"><math id="A2.T5.6.6.2.m1.1" class="ltx_Math" alttext="1.0" display="inline"><semantics id="A2.T5.6.6.2.m1.1a"><mn id="A2.T5.6.6.2.m1.1.1" xref="A2.T5.6.6.2.m1.1.1.cmml">1.0</mn><annotation-xml encoding="MathML-Content" id="A2.T5.6.6.2.m1.1b"><cn type="float" id="A2.T5.6.6.2.m1.1.1.cmml" xref="A2.T5.6.6.2.m1.1.1">1.0</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.T5.6.6.2.m1.1c">1.0</annotation></semantics></math></td>
</tr>
<tr id="A2.T5.6.8.1" class="ltx_tr">
<td id="A2.T5.6.8.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_r">
<span id="A2.T5.6.8.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T5.6.8.1.1.1.1" class="ltx_p" style="width:142.3pt;">Batch size for training</span>
</span>
</td>
<td id="A2.T5.6.8.1.2" class="ltx_td ltx_align_left">128</td>
</tr>
<tr id="A2.T5.6.9.2" class="ltx_tr">
<td id="A2.T5.6.9.2.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_r">
<span id="A2.T5.6.9.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T5.6.9.2.1.1.1" class="ltx_p" style="width:142.3pt;">Batch size for corpus during training</span>
</span>
</td>
<td id="A2.T5.6.9.2.2" class="ltx_td ltx_align_left">50000</td>
</tr>
<tr id="A2.T5.6.10.3" class="ltx_tr">
<td id="A2.T5.6.10.3.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_r">
<span id="A2.T5.6.10.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T5.6.10.3.1.1.1" class="ltx_p" style="width:142.3pt;">Maximum number of training iterations</span>
</span>
</td>
<td id="A2.T5.6.10.3.2" class="ltx_td ltx_align_left">5000</td>
</tr>
<tr id="A2.T5.6.11.4" class="ltx_tr">
<td id="A2.T5.6.11.4.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_r">
<span id="A2.T5.6.11.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T5.6.11.4.1.1.1" class="ltx_p" style="width:142.3pt;">Patience for early stopping</span>
</span>
</td>
<td id="A2.T5.6.11.4.2" class="ltx_td ltx_align_left">500</td>
</tr>
<tr id="A2.T5.6.12.5" class="ltx_tr">
<td id="A2.T5.6.12.5.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_r">
<span id="A2.T5.6.12.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T5.6.12.5.1.1.1" class="ltx_p" style="width:142.3pt;">Learning rates</span>
</span>
</td>
<td id="A2.T5.6.12.5.2" class="ltx_td ltx_align_left">0.001</td>
</tr>
<tr id="A2.T5.6.13.6" class="ltx_tr">
<td id="A2.T5.6.13.6.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb ltx_border_r">
<span id="A2.T5.6.13.6.1.1" class="ltx_inline-block ltx_align_top">
<span id="A2.T5.6.13.6.1.1.1" class="ltx_p" style="width:142.3pt;">Optimizer</span>
</span>
</td>
<td id="A2.T5.6.13.6.2" class="ltx_td ltx_align_left ltx_border_bb">Adam</td>
</tr>
</tbody>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 5: </span>Hyper-parameters used to train Matryoshka-Adaptor in all experiments.</figcaption>
</figure>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Computational complexity</h2>

<div id="A3.p1" class="ltx_para">
<p id="A3.p1.1" class="ltx_p">All experimental procedures were conducted utilizing a single NVIDIA V100 GPU with 16GB of memory.
In the unsupervised Matryoshka-Adaptor training regime, processing time did not exceed 10 minutes for datasets ranging from 3,000 to 10 million corpus samples.
For supervised Matryoshka-Adaptor training, datasets with fewer than 1 million corpus samples were processed in under 30 minutes, while datasets with up to 10 million corpus samples were processed in less than one hour.
It is important to note that the adaptor architecture consists of a shallow multi-layer perceptron, rendering the computational complexity of inference negligible.</p>
</div>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Additional results on supervised Matryoshka-Adaptor</h2>

<figure id="A4.F8" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A4.F8.sf1" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.20243/assets/x16.png" id="A4.F8.sf1.g1" class="ltx_graphics ltx_img_square" width="184" height="167" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A4.F8.sf2" class="ltx_figure ltx_figure_panel ltx_align_center"><img src="/html/2407.20243/assets/x17.png" id="A4.F8.sf2.g1" class="ltx_graphics ltx_img_square" width="184" height="167" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 8: </span>Experimental results of the supervised Matryoshka-Adaptor on retrieval tasks, utilizing two different embedding models: Google Gecko (on 13 BEIR datasets), and OpenAI text-embedding-3-small (on 8 BEIR datasets).</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Detailed Experimental Results on Unsupervised Settings</h2>

<div id="A5.p1" class="ltx_para">
<p id="A5.p1.1" class="ltx_p">In this section, we present the experimental results per each dataset in unsupervised settings.
In the main manuscript, we report the average values among the entire datasets.</p>
</div>
<section id="A5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.1 </span>Unsupervised Matryoshka-Adaptor with OpenAI embedding models</h3>

<figure id="A5.F9" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F9.sf1" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x18.png" id="A5.F9.sf1.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F9.sf2" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x19.png" id="A5.F9.sf2.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F9.sf3" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x20.png" id="A5.F9.sf3.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(c) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F9.sf4" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x21.png" id="A5.F9.sf4.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(d) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F9.sf5" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x22.png" id="A5.F9.sf5.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(e) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F9.sf6" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x23.png" id="A5.F9.sf6.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(f) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F9.sf7" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x24.png" id="A5.F9.sf7.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(g) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F9.sf8" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x25.png" id="A5.F9.sf8.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(h) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 9: </span>Experimental results of unsupervised Matryoshka-Adaptor with OpenAI-textembedding-3-large on 8 BEIR datasets.</figcaption>
</figure>
<figure id="A5.F10" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F10.sf1" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x26.png" id="A5.F10.sf1.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F10.sf2" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x27.png" id="A5.F10.sf2.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F10.sf3" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x28.png" id="A5.F10.sf3.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(c) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F10.sf4" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x29.png" id="A5.F10.sf4.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(d) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F10.sf5" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x30.png" id="A5.F10.sf5.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(e) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F10.sf6" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x31.png" id="A5.F10.sf6.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(f) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F10.sf7" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x32.png" id="A5.F10.sf7.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(g) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F10.sf8" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x33.png" id="A5.F10.sf8.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(h) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 10: </span>Experimental results of unsupervised Matryoshka-Adaptor with OpenAI-textembedding-3-small on 8 BEIR datasets.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.2 </span>Unsupervised Matryoshka-Adaptor with Google multimodal embedding models</h3>

<figure id="A5.F11" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F11.sf1" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x34.png" id="A5.F11.sf1.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F11.sf2" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x35.png" id="A5.F11.sf2.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F11.sf3" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x36.png" id="A5.F11.sf3.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(c) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F11.sf4" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x37.png" id="A5.F11.sf4.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(d) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F11.sf5" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x38.png" id="A5.F11.sf5.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(e) </span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 11: </span>Experimental results of unsupervised Matryoshka-Adaptor with Google multimodal embedding models on 5 Fashion-200K datasets.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A5.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">E.3 </span>Unsupervised Matryoshka-Adaptor with Google Gecko multilingual embedding models</h3>

<figure id="A5.F12" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F12.sf1" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x39.png" id="A5.F12.sf1.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F12.sf2" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x40.png" id="A5.F12.sf2.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F12.sf3" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x41.png" id="A5.F12.sf3.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(c) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F12.sf4" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x42.png" id="A5.F12.sf4.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(d) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F12.sf5" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x43.png" id="A5.F12.sf5.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(e) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F12.sf6" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x44.png" id="A5.F12.sf6.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(f) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F12.sf7" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x45.png" id="A5.F12.sf7.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(g) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F12.sf8" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x46.png" id="A5.F12.sf8.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(h) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F12.sf9" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x47.png" id="A5.F12.sf9.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(i) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F12.sf10" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x48.png" id="A5.F12.sf10.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(j) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F12.sf11" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x49.png" id="A5.F12.sf11.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(k) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F12.sf12" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x50.png" id="A5.F12.sf12.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(l) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F12.sf13" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x51.png" id="A5.F12.sf13.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(m) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F12.sf14" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x52.png" id="A5.F12.sf14.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(n) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F12.sf15" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x53.png" id="A5.F12.sf15.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(o) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F12.sf16" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x54.png" id="A5.F12.sf16.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(p) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A5.F12.sf17" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x55.png" id="A5.F12.sf17.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(q) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 12: </span>Experimental results of unsupervised Matryoshka-Adaptor with Google Gecko multilingual embedding models on 17 MIRACL datasets.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
<section id="A6" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix F </span>Detailed Experimental Results on Supervised Settings</h2>

<div id="A6.p1" class="ltx_para">
<p id="A6.p1.1" class="ltx_p">In this section, we present the experimental results per each dataset in supervised settings.
In the main manuscript, we report the average values among the entire datasets.</p>
</div>
<section id="A6.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">F.1 </span>Supervised Matryoshka-Adaptor with Google Gecko embedding models</h3>

<figure id="A6.F13" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F13.sf1" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x56.png" id="A6.F13.sf1.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F13.sf2" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x57.png" id="A6.F13.sf2.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F13.sf3" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x58.png" id="A6.F13.sf3.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(c) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F13.sf4" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x59.png" id="A6.F13.sf4.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(d) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F13.sf5" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x60.png" id="A6.F13.sf5.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(e) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F13.sf6" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x61.png" id="A6.F13.sf6.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(f) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F13.sf7" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x62.png" id="A6.F13.sf7.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(g) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F13.sf8" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x63.png" id="A6.F13.sf8.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(h) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F13.sf9" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x64.png" id="A6.F13.sf9.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(i) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F13.sf10" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x65.png" id="A6.F13.sf10.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(j) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F13.sf11" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x66.png" id="A6.F13.sf11.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(k) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F13.sf12" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x67.png" id="A6.F13.sf12.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(l) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F13.sf13" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x68.png" id="A6.F13.sf13.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(m) </span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 13: </span>Experimental results of supervised Matryoshka-Adaptor with Google latest Gecko embedding models on 13 BEIR datasets.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A6.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">F.2 </span>Supervised Matryoshka-Adaptor with OpenAI text-embedding-3-large models</h3>

<figure id="A6.F14" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F14.sf1" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x69.png" id="A6.F14.sf1.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F14.sf2" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x70.png" id="A6.F14.sf2.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F14.sf3" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x71.png" id="A6.F14.sf3.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(c) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F14.sf4" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x72.png" id="A6.F14.sf4.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(d) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F14.sf5" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x73.png" id="A6.F14.sf5.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(e) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F14.sf6" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x74.png" id="A6.F14.sf6.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(f) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F14.sf7" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x75.png" id="A6.F14.sf7.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(g) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F14.sf8" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x76.png" id="A6.F14.sf8.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(h) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 14: </span>Experimental results of supervised Matryoshka-Adaptor with OpenAI text-embedding-3-large models on 8 BEIR datasets.</figcaption>
</figure>
</section>
<section id="A6.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">F.3 </span>Supervised Matryoshka-Adaptor with OpenAI text-embedding-3-small models</h3>

<figure id="A6.F15" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F15.sf1" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x77.png" id="A6.F15.sf1.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F15.sf2" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x78.png" id="A6.F15.sf2.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F15.sf3" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x79.png" id="A6.F15.sf3.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(c) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F15.sf4" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x80.png" id="A6.F15.sf4.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(d) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F15.sf5" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x81.png" id="A6.F15.sf5.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(e) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F15.sf6" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x82.png" id="A6.F15.sf6.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(f) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F15.sf7" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x83.png" id="A6.F15.sf7.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(g) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F15.sf8" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x84.png" id="A6.F15.sf8.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(h) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 15: </span>Experimental results of supervised Matryoshka-Adaptor with OpenAI text-embedding-3-small models on 8 BEIR datasets.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A6.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">F.4 </span>Supervised Matryoshka-Adaptor with Google Gecko multilingual embedding models</h3>

<figure id="A6.F16" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F16.sf1" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x85.png" id="A6.F16.sf1.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F16.sf2" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x86.png" id="A6.F16.sf2.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F16.sf3" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x87.png" id="A6.F16.sf3.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(c) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F16.sf4" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x88.png" id="A6.F16.sf4.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(d) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F16.sf5" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x89.png" id="A6.F16.sf5.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(e) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F16.sf6" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x90.png" id="A6.F16.sf6.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(f) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F16.sf7" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x91.png" id="A6.F16.sf7.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(g) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F16.sf8" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x92.png" id="A6.F16.sf8.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(h) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F16.sf9" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x93.png" id="A6.F16.sf9.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(i) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F16.sf10" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x94.png" id="A6.F16.sf10.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(j) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F16.sf11" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x95.png" id="A6.F16.sf11.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(k) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F16.sf12" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x96.png" id="A6.F16.sf12.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(l) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F16.sf13" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x97.png" id="A6.F16.sf13.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(m) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F16.sf14" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x98.png" id="A6.F16.sf14.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(n) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F16.sf15" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x99.png" id="A6.F16.sf15.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(o) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F16.sf16" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x100.png" id="A6.F16.sf16.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(p) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F16.sf17" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x101.png" id="A6.F16.sf17.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(q) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 16: </span>Experimental results of supervised Matryoshka-Adaptor with Google Gecko multilingual embedding models on 17 MIRACL datasets.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A6.SS5" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">F.5 </span>Supervised Matryoshka-Adaptor with Google multimodal embedding models</h3>

<figure id="A6.F17" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F17.sf1" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x102.png" id="A6.F17.sf1.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(a) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F17.sf2" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x103.png" id="A6.F17.sf2.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(b) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F17.sf3" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x104.png" id="A6.F17.sf3.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(c) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F17.sf4" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x105.png" id="A6.F17.sf4.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(d) </span></figcaption>
</figure>
</div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<figure id="A6.F17.sf5" class="ltx_figure ltx_figure_panel"><img src="/html/2407.20243/assets/x106.png" id="A6.F17.sf5.g1" class="ltx_graphics ltx_img_square" width="111" height="100" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">(e) </span></figcaption>
</figure>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 17: </span>Experimental results of supervised Matryoshka-Adaptor with Google multimodal embedding models on 5 Fashion-200K datasets.</figcaption>
</figure>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2407.20242" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2407.20243" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2407.20243">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2407.20243" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2407.20244" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Mon Aug  5 16:43:15 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2407.20243
Embeddings from Large Language Models (LLMs)  have emerged as critical components in various applications, particularly for information retrieval.
While high-dimensional embeddings generally demonstrate superior performance as they contain more salient information, their practical application is frequently hindered by elevated computational latency and the associated higher cost.
To address these challenges, we propose Matryoshka-Adaptor, a novel tuning framework designed for the customization of LLM embeddings.
Matryoshka-Adaptor facilitates substantial dimensionality reduction while maintaining comparable performance levels, thereby achieving a significant enhancement in computational efficiency and cost-effectiveness.
Our framework directly modifies the embeddings from pre-trained LLMs which is designed to be seamlessly integrated with any LLM architecture, encompassing those accessible exclusively through black-box APIs.
Also, it exhibits efficacy in both unsupervised and supervised learning settings.
A rigorous evaluation conducted across a diverse corpus of English, multilingual, and multimodal datasets consistently reveals substantial gains with Matryoshka-Adaptor.
Notably, with Google and OpenAI Embedding APIs, Matryoshka-Adaptor achieves a reduction in dimensionality ranging from two- to twelve-fold without compromising performance across multiple BEIR datasets.
abstract
Matryoshka-Adaptor: Unsupervised and Supervised Tuningfor Smaller Embedding Dimensions
Abstract
2407.20243
1IntroductionFigure 1:The effectiveness of the Matryoshka Adaptor in dimensionality reduction.
In both unsupervised (red line)  and supervised (black line)  settings, the Matryoshka Adaptor showcases the capability to considerably decrease embedding dimensions while maintaining a negligible impact on nDCG@10 retrieval performance with BEIR SciFact dataset.
Notably, at the same embedding dimensionality, the utilization of our approach results in significantly improved performance.Large language models (LLMs)  have showcased remarkable proficiency in handling various text processing tasks, encompassing question answering, summarization, and mathematical reasoning.
This success can be partially attributed to their ability to transform raw text into semantically enriched representations, with the quality of these text-to-embedding mappings being of paramount importance.Embeddings find extensive utilization in a wide array of downstream tasks, with information retrieval (IR)  serving as a prominent one.
IR involves the process of searching for relevant information within a corpus database using queries, and language modeling plays a pivotal role as both queries and corpus data are frequently textual in nature.
In IR systems, text embeddings are commonly employed to rank relevant corpora based on their similarity to queries.Numerous LLMs have been developed specifically for extracting embeddings from raw text, including Sentence T5, OpenAI Embedding APIs, and Google Embedding APIs.
However, generation of high-dimensional embeddings often entail high latency and computational costs, thereby limiting their practical application in latency-sensitive scenarios such as large-scale recommendation systems.Matryoshka representation learning (MRL) addresses this limitation at pretraining stage, by generating embeddings that retain similar characteristics even when utilizing only a subset of their dimensions (i.e., Matryoshka properties) .
This enables efficient similarity comparisons with lower-dimensional embeddings and is leveraged by many state-of-the-art models, including those from Google and OpenAI.In this study, we present Matryoshka-Adaptor, a novel framework designed to transform arbitrary embeddings into embeddings with Matryoshka properties in both unsupervised and supervised learning setups.
In an unsupervised learning setting, the adaptor learns to transform input embeddings into Matryoshka embeddings using only corpus embeddings.
We introduced pairwise and top-k similarity loss functions to facilitate this process.
In supervised learning setup, the adaptor can be further refined by incorporating relevant (query, corpus)  pairs as labeled tuning data.
Note that we customize embeddings to better suit specific datasets, using both unsupervised and supervised approaches.
This refinement results in improved Matryoshka properties, surpassing even those of embeddings derived from MRL-trained models.Extensive experiments conducted across 13 BEIR datasets, 17 MIRACL datasets, and 5 multimodal Fashion-200K datasetsvalidate the effectiveness of Matryoshka-Adaptor.
We demonstrate significant performance improvements over the latest Google (for English only, multilingual, and multimodal data) and OpenAI text embedding models.
Matryoshka-Adaptor exhibits broad applicability to diverse embedding types, encompassing text, multimodal, multilingual, and use case-specific embeddings. The contributions of this paper are threefold:•We propose a novel tuning framework, Matryoshka-Adaptor that achieves substantial dimensionality reduction in embeddings without sacrificing performance achieved by tailoring embeddings to better fit specific datasets.•Matryoshka-Adaptor is applicable in both unsupervised and supervised settings, consistently improving retrieval performance across various datasets. Also, Matryoshka-Adaptor demonstrates a roughly two-fold (unsupervised)  and six-fold (supervised)  reduction in dimensionality, with no loss in performance.•The benefits of Matryoshka-Adaptor extend to multimodal learning and multilingual scenarios.
introduction
Matryoshka-Adaptor: Unsupervised and Supervised Tuningfor Smaller Embedding Dimensions
1Introduction
2407.20243
2Related WorkMatryoshka embeddings.pioneered the development of embedding models whose representations could be substantially reduced in dimensionality without incurring a significant degradation in performance.
These Matryoshka embedding models are specifically trained to ensure the utility of such truncated embeddings.
The Matryoshka properties inherent in these models enable fine-grained control over the trade-off between latency and accuracy in downstream tasks utilizing the embeddings.
Given this advantage, recent embedding models, including those from OpenAI and Google, have integrated Matryoshka properties by employing Matryoshka Representation Learning (MRL)  during pre-training.
Our proposed work further enhances the Matryoshka properties through tuning.
We introduce modifications to embeddings that are tailored to target datasets in both an unsupervised and supervised tuning setups.
The proposed tuning yields enhanced Matryoshka properties compared to the original embeddings, even those derived from MRL-trained models.Dimensionality reduction.Dimensionality reduction is a well-established framework for reducing the dimensionality of vectors while preserving their inherent properties.
Principal Component Analysis (PCA) , Independent Component Analysis (ICA) , and Non-negative Matrix Factorization (NMF) are widely used unsupervised dimensionality reduction techniques.
Linear Discriminant Analysis (LDA) is a supervised method that leverages labeled data for dimensionality reduction.
While ICA is possible, its limitations in this context arise due to the equal importance of all components, requiring separate model training for each reduced dimension.
NMF, on the other hand, is not suitable for this application as it is only applicable to non-negative embeddings.
Also, PCA’s orthogonality properties may limit its reliability for dimensionality reduction for vectors with highly non-linear relationships, especially at high dimensionality.Embedding customization.In lieu of employing a single unified model for zero-shot retrieval, embedding customization tailored to individual datasets presents a viable alternative.
TARTconstructs a retrieval system that adapts retrieval based on instructions, incorporating different tasks (e.g. for code, question, or answer)  to enhance dense embedding retrieval.
InstructORintegrates task and domain descriptions for retrieval while tuning embedding models.
Promptagatorleverages in-context learning to generate synthetic query-corpus pairs using a limited number of original pairs, subsequently fine-tuning pre-trained LLMs with these synthetic pairs.
Search-Adaptorcustomizes embeddings for information retrieval using a small number of query-corpus pairs in supervised learning setup.
Unlike these approaches, Matryoshka-Adaptor is applicable to both supervised and unsupervised settings, eliminating the need for labeled query-corpus pairs.
Additionally, Matryoshka-Adaptor aims not only to enhance full-dimensional embedding performance but also to improve performance across all reduced-dimensionality embeddings.
body
Matryoshka-Adaptor: Unsupervised and Supervised Tuningfor Smaller Embedding Dimensions
2Related Work
2407.20243
3Unsupervised Matryoshka-Adaptor3.1Problem formulationTo facilitate a clear understanding, the proposed framework is formulated within the context of information retrieval.
However, it is crucial to emphasize that it can be readily generalized to any application involving embeddings.
In unsupervised settings, we assume the availability of a corpus set, denoted asand a pre-trained embedding model () .
The embeddings extracted from the corpus are represented as, where each embedding vector.
Notably, our framework permits the embedding model,, to be treated as a black-box model, and Matryoshka-Adaptor can be directly applied to the extracted embeddings,.A Matryoshka embedding, characterized bydimensions, is defined as the initialdimensions of the original-dimensional embedding, where.
This can be expressed aswhere each reduced embedding vector.
A fundamental characteristic of Matryoshka embeddings is their capacity to preserve the essential properties of the original embeddings, even within a reduced-dimensional space.Figure 2:Similarity loss is a measure of the discrepancy between the similarity of two embeddings in their original dimensional space and their similarity in a reduced dimensional space.
If the orange and blue embeddings are chosen randomly, this loss is referred to as pairwise similarity loss.
If the orange and blue embeddings are selected based on similarity in their original dimensional space (top-k nearest embeddings) , this loss is referred to as top-k similarity loss.
Note that top-k similarity loss focuses on preserving local similarity relationships among neighboring embeddings.Figure 3:Block diagrams illustrating both the unsupervised and supervised Matryoshka-Adaptor frameworks.Unsupervised Matryoshka-Adaptor: This variant exclusively utilizes corpus embeddings as input.
The training of the adaptor is achieved through a combination of top-k similarity loss and pairwise loss, which are calculated across multiple Matryoshka embeddings with various reduced dimensions.Supervised Matryoshka-Adaptor: In this variant, query embeddings and query-corpus pairs are provided as supplementary inputs.
A ranking loss is incorporated alongside the top-k and pairwise losses to facilitate the training of the adaptor.
Similar to the unsupervised setting, all losses are computed across Matryoshka embeddings with various reduced dimensions.3.2Tuning objective functionsThe proposed Matryoshka-Adaptor, represented by the function, is designed to modify the original embeddings in order to enhance their inherent Matryoshka properties.
We define the set of customized corpus embeddings asand their corresponding Matryoshka embeddings aswhere.
The primary objective of the functionis to maximize the Matryoshka properties through this customization process.
This means ensuring the similarity between any two embeddings remains as consistent as possible, whether they are represented in the original high-dimensional space or the reduced low-dimensional space.To achieve this objective, we introduce two loss functions. The first loss function, denoted as, is designed to preserve the pairwise similarity between the original embeddings in their reduced-dimension Matryoshka form, expressed as:(1) whererepresents an arbitrary similarity function, which is chosen to be the cosine similarity.The second loss function, denoted as, focuses on preserving local similarity relationships among neighboring embeddings:(2) wheredenotes the set of the top k most similar embeddings to.
A visual representation of these two loss functions, illustrating their application across multiple Matryoshka embeddings of varying dimensions, is provided in Fig.2.In order to mitigate any substantial deviation from the original embeddings, we have integrated regularizations into our methodology.
Primarily, a skip connection is implemented within the architecture of the learnable function,, ensuring that this function learns solely the difference from the original embedding, represented as.
Furthermore, a reconstruction loss, denoted as, is introduced as an additional regularizer:(3) The overall objective function, designed to minimize the aggregate loss, is given as:(4) withbeing hyperparameters.
Within the context of the unsupervised tuning setting, we fix their values as.
body
Matryoshka-Adaptor: Unsupervised and Supervised Tuningfor Smaller Embedding Dimensions
3Unsupervised Matryoshka-Adaptor
2407.20243
4Supervised Matryoshka-Adaptor4.1Problem formulationFor many information retrieval applications, the availability of pairwise data, which indicates the relevance between specific queries and corpora, can considerably improve retrieval performance.
In this section, we introduce a method to further refine the Matryoshka-Adaptor by utilizing (a limited number of)  such paired samples.Letdenote the set of queries, and letrepresent the set of query-corpus relevance triplets, wheresignifies the relevance score between queryand corpus.
The query embeddings extracted from the model ()  are denoted aswith each query embedding vector.
Furthermore, we define Matryoshka query embeddings as, where.
Note thatcan be any integer less than the original embedding dimension.(a) (b) (c) Figure 4:Experimental results of the unsupervised Matryoshka-Adaptor applied to three different embedding models: OpenAI text-embedding-3-large (with 3072 dimensions) , OpenAI text-embedding-3-small (with 1536 dimensions) , and Google multimodal (with 1408 dimensions) .
Text embedding results were obtained using 8 BEIR datasets, while multimodal embedding results were obtained using 5 Fashion-200K datasets.4.2Tuning objective functionsIn the supervised setting, the Matryoshka-Adaptor ()  undergoes optimization to enhance both its Matryoshka properties and the retrieval performance of the Matryoshka embeddings.
This is accomplished through the utilization of paired query-corpus samples, in conjunction with the original query and corpus embeddings.
Matryoshka ranking loss, denoted as, is introduced to align the ranking between query and corpus considering different Matryoshka embedding dimensions:(5) whererepresents the cosine similarity between the adapted query embedding(where)  and the adapted corpus embedding.
We use the same adaptor ()  for both query and corpus embeddings.
This ranking loss is crucial for effective learning of lower dimensional representations with their information content for the ranking objective being considered.The supervised Matryoshka-Adaptor is trained using a joint objective function that encompasses the ranking loss as well as the unsupervised Matryoshka losses (,, and) .
This joint training approach aims to improve the quality of the embeddings while preserving their Matryoshka representations.
Query-corpus pairs are employed for the ranking loss, while query and corpus embeddings are utilized for the Matryoshka representation learning. The overall objective function is:(6) withbeing hyper-parameters with fixed values asand.To improve convergence, two-stage training strategy is employed. Initially, the Matryoshka-Adaptor is trained in an unsupervised way using Eq.4.
Subsequently, further tuning is conducted in a supervised way, utilizing Eq.6.(a) (b) (c) Figure 5:Experimental results of the supervised Matryoshka-Adaptor on retrieval tasks, utilizing three different embedding models: OpenAI text-embedding-3-large (on 8 BEIR datasets) , Google Gecko multilingual (on 17 MIRACL datasets) , and Google multimodal (on 5 Fashion-200K datasets) . Additional results are in Appendix.D.
body
Matryoshka-Adaptor: Unsupervised and Supervised Tuningfor Smaller Embedding Dimensions
4Supervised Matryoshka-Adaptor
2407.20243
5Experiments5.1Experimental settingsThe Matryoshka-Adaptor is extensively evaluated across a diverse set of 13 BEIR datasets, 17 MIRACL datasets, and 5 Fashion-200K datasets.
Such a comprehensive evaluation highlights the data-agnostic nature of the Matryoshka-Adaptor.
Query and corpus embeddings are generated using state-of-the-art models, including Google Gecko text embeddings (English and multilingual) , Google multimodal embeddings, and OpenAI text embeddings.
This further highlights the model-agnostic nature of the proposed Matryoshka-Adaptor.During the evaluation phase, both query and corpus embeddings are transformed using the trained Matryoshka-Adaptor.
Cosine similarity is then computed between the transformed query and corpus embeddings across a spectrum of reduced dimensions (e.g.,for Gecko text embeddings) .
Retrieval performance is evaluated using the normalized discounted cumulative gain at rank 10 (nDCG@10)  metric, facilitating a comprehensive assessment of performance across various Matryoshka embedding dimensions.
All reported results represent the average value across the datasets.
Data statistics and hyper-parameters used in the experiments can be found in Appendix.AandB.
Detailed results can be found in Appendix.EandF.5.2Unsupervised tuningThe Matryoshka-Adaptor can be applied exclusively with corpus embeddings, referred to as the unsupervised setting, enabling customization of embeddings solely on the corpus side.
In this subsection, we present the impact of the unsupervised Matryoshka-Adaptor on two OpenAI text embedding models and compare its performance with the commonly-used unsupervised dimensionality reduction method, Principal Component Analysis (PCA) .Fig.4(a) and4(b) illustrate that the Matryoshka-Adaptor yields significant performance improvements, particularly for lower dimensions, compared to the embeddings of the same dimensionality without it.
Furthermore, lower dimensional embeddings processed with Matryoshka-Adaptor achieve comparable performance to original embeddings of high dimensionality.
The adaptor achieves a faster saturation in performance with embedding dimensionality, towards the retrieval performance of original embeddings, underscoring its substantial impact in significantly reducing latency and memory requirements for retrieval applications.Notably, the latest OpenAI embeddings are already trained with Matryoshka Representation Learning (MRL) .
The additional performance gains achieved by Matryoshka-Adaptor are attributed to the tuning process.
With PCA, some improvements are observed for lower dimensions, however, at higher dimensions, noticeable performance degradation occurs, resulting in performance inferior even to the original embedding.
This highlights the superiority of the Matryoshka-Adaptor, designed with the goal of salient information preservation for similarity within lower dimensions compared to PCA, a generic unsupervised dimensionality reduction approach.5.3Supervised tuningThe Matryoshka-Adaptor can be applied in supervised learning setup where a limited number of query-corpus pairs are available.
In this context, the Supervised Matryoshka-Adaptor is trained utilizing paired query-corpus data.
The effectiveness of the Supervised Matryoshka-Adaptor is evaluated on 13 BEIR, 17 MIRACL, and 5 Fashion-200K datasets, with query and corpus embeddings being generated using the latest Google Gecko models, including its multilingual and multimodal versions.As illustrated in Fig.5, the Supervised Matryoshka-Adaptor consistently outperforms the alternatives, such as Search-Adaptor, particularly for lower embedding dimensions.
Additionally, lower dimensional embeddings processed with the Supervised Matryoshka-Adaptor perform comparably to high dimensional embeddings, showcasing its potential to significantly reduce latency and memory requirements for applications like retrieval.(a) (b) (c) Figure 6:Experimental results of the unsupervised Matryoshka-Adaptor with three different text embedding models: Google Gecko, Google Gecko multilingual, and Google Gecko-003 (which is not trained with the Matryoshka Representation Learning technique) .5.4Tuning for Multimodal EmbeddingsAs previously established, the Matryoshka-Adaptor framework is not confined to text embeddings but can be generalized to multimodal embeddings as well.
To illustrate this capability, we applied the Matryoshka-Adaptor to the latest gecko-multimodal embeddings, utilizing the Fashion-200K dataset, which comprises 5 sub-datasets designed for text-to-image retrieval tasks.Fig.4(c) and5(c) demonstrate the effectiveness of the Matryoshka-Adaptor in consistently improving the performance of multimodal base embedding models for text-to-image retrieval tasks.
These highlight that the Matryoshka-Adaptor significantly outperforms alternative methods such as PCA, in unsupervised learning setups and Search-Adaptor in supervised learning setups, particularly when lower embedding dimensions are considered.5.5Tuning for Multilingual EmbeddingsMatryoshka-Adaptor is not only model-agnostic but also data-agnostic. Its applicability even extends beyond a single language.
To validate this, we evaluate the performance of Matryoshka-Adaptor on MIRACL datasets, which comprise 17 non-English languages.Fig.6(b) and5(b) present the results of applying Matryoshka-Adaptor to multilingual retrieval tasks.
The findings demonstrate that the performance gains achieved through the proposed tuning method are not limited to English but also extend to non-English language datasets.
Furthermore, the improvements are observed to be model-agnostic, as evident from the successful application of Matryoshka-Adaptor to the latest Gecko multilingual embedding models.
body
Matryoshka-Adaptor: Unsupervised and Supervised Tuningfor Smaller Embedding Dimensions
5Experiments
2407.20243
6Discussions6.1Models that are not pretrained with MRLA significant advantage of the Matryoshka-Adaptor framework lies in its broad applicability to a wide array of embedding models.
We demonstrate the efficacy of Matryoshka-Adaptor when applied to embedding models that have not been trained using MRL.
Specifically, we utilize earlier versions of the Gecko embedding models (Google Gecko-003) , which do not utilize MRL in their pretraining, in conjunction with BEIR datasets to illustrate the impact of the Unsupervised Matryoshka-Adaptor.Fig.6(c) presents evidence of the consistent performance improvements achieved by the Matryoshka-Adaptor when applied to non-MRL trained embedding models.
This observation underscores that the performance gains of Matryoshka-Adaptor originate from customizing the embedding to the specific corpus, and this beneficial impact can be extended to any embedding model, even when they had been trained with MRL.(a) (b) (c) Figure 7:Analysis of distance metrics in unsupervised settings.
(a)  Average pairwise and top-k distances across varying embedding dimensions, compared to base embeddings.
(b)  Correlation between nDCG@10 and average pairwise distance.
(c)  Correlation between nDCG@10 and average top-k distance.6.2Ablation studiesTo examine the individual contribution of each loss component to the overall performance of Matryoshka-Adaptor, we conduct ablation studies.
In the unsupervised setting, each of the top-k loss () , pairwise loss () , and reconstruction loss ()  is individually excluded from the original loss summation in Eq.4, and the corresponding performance degradation is monitored.
Specifically, the pairwise loss was excluded by setting, and the reconstruction loss is excluded by setting.
In the supervised setting, we utilize only the ranking loss ()  to assess the impact of three unsupervised losses in Eq.6.Unsuperivsed Matryoshka-Adaptor(nDCG@10) Reduced dims64128256512Baseline0.43320.50440.54610.5590All three losses0.48450.53800.55800.5652w/o0.47980.52360.54630.5630w/o0.47450.52300.54230.5598w/o0.48240.53420.54770.5621Supervised Matryoshka-Adaptor(nDCG@10) Reduced dims64128256512Baseline0.43320.50440.54610.5590All four losses0.50470.54730.57140.5902Only with0.47670.52090.56070.5787Table 1:Ablation studies on both supervised and unsupervised Matryoshka-Adaptor across 8 BEIR datasets.
The best performance in each scenario is in bold.As evident in Table1, each loss function contributes uniquely to the overall performance of Matryoshka-Adaptor in the unsupervised setting.
Notably,is particularly beneficial for higher dimensions.
In the supervised setting, the incorporation of the additional unsupervised losses, includingand, is critical for achieving performance improvements.
Without these unsupervised losses, the performance gain diminishes by approximately 50%.6.3Relationship with distance metricsIn unsupervised learning setup, the absence of labeled data presents a challenge in evaluating the impact of Matryoshka-Adaptor.
Towards circumventing this issue and shedding light on the impact of dimensionality reduction, we analyze the average pairwise distances and top-k distances within the corpus embeddings on the disjoint validation corpus.
Subsequently, we examine the relationship between these unsupervised metrics and the supervised retrieval metric nDCG@10, utilizing 8 BEIR datasets with Google Gecko embedding models.Fig.7(a) reveals that the unsupervised Matryoshka-Adaptor results in significantly lower distance metrics compared to the original embeddings.
This suggests that the proposed method effectively preserves both pairwise and top-k distances among corpus embeddings when employing a reduced subset of embedding dimensions.
These distance metrics can serve as viable unsupervised proxies for assessing the efficacy of the unsupervised Matryoshka-Adaptor.
To further elucidate this relationship, we also show the correlation between these distance metrics and nDCG@10 (Fig.7(b) and7(c) ) .
A strong correlation is observed, indicating that lower distance metrics are associated with smaller performance drop in retrieval tasks, even when using reduced embedding dimensions.
This suggests that the unsupervised distance metrics can be indicators of retrieval performance, even in the absence of labeled data.
body
Matryoshka-Adaptor: Unsupervised and Supervised Tuningfor Smaller Embedding Dimensions
6Discussions
2407.20243
7ConclusionsWhile high-dimensional embeddings often provide a richer representation of the original data, their practical deployment in real-world applications is frequently hampered by computational costs and latency issues.
As a result, many applications opt for lower-dimensional embeddings, accepting a compromise in performance.
The presented Matryoshka-Adaptor framework offers a solution to this dilemma by enabling substantial dimensionality reduction while minimizing performance degradation, in both supervised and unsupervised scenarios.
Furthermore, due to its versatile nature and applicability to any embedding model, regardless of access to model parameters, the Matryoshka-Adaptor constitutes a valuable tool for improving the efficiency and practicality of embedding-based applications.
body
Matryoshka-Adaptor: Unsupervised and Supervised Tuningfor Smaller Embedding Dimensions
7Conclusions
2407.20243
8Limitations and Future WorksIn the unsupervised setting, picking the optimal hyperparameters for Matryoshka-Adaptor poses a challenge due to the absence of validation data.
While alternative metrics, such as the proposed distance metrics illustrated in Fig.7, can be employed, these may exhibit higher levels of noise compared to supervised validation metrics.
One major risk associated with Matryoshka-Adaptor lies in the potential for overfitting to the tuning data, during the tuning process.
Future research endeavors might include extending the proposed approach to encompass multiple modalities in the tuning objective;
exploring semi-supervised variations; and
investigating simultaneous utilization of multiple datasets during tuning.
conclusion
Matryoshka-Adaptor: Unsupervised and Supervised Tuningfor Smaller Embedding Dimensions
8Limitations and Future Works