<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2012.15723] Making Pre-trained Language Models Better Few-shot Learners</title><meta property="og:description" content="The recent GPT-3 model Brown et al. (2020) achieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context.
Inspired by their findings, we study fe…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Making Pre-trained Language Models Better Few-shot Learners">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Making Pre-trained Language Models Better Few-shot Learners">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2012.15723">

<!--Generated on Fri Mar  8 07:36:36 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Making Pre-trained Language Models Better Few-shot Learners</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Tianyu Gao<sup id="id6.6.id1" class="ltx_sup"><span id="id6.6.id1.1" class="ltx_text ltx_font_italic">†∗</span></sup>  Adam Fisch<sup id="id7.7.id2" class="ltx_sup"><span id="id7.7.id2.1" class="ltx_text ltx_font_italic">‡∗</span></sup>  Danqi Chen<sup id="id8.8.id3" class="ltx_sup"><span id="id8.8.id3.1" class="ltx_text ltx_font_italic">†</span></sup> 
<br class="ltx_break"><sup id="id9.9.id4" class="ltx_sup"><span id="id9.9.id4.1" class="ltx_text ltx_font_italic">†</span></sup>Princeton University <sup id="id10.10.id5" class="ltx_sup"><span id="id10.10.id5.1" class="ltx_text ltx_font_italic">‡</span></sup>Massachusetts Institute of Technology
<br class="ltx_break"><span id="id11.11.id6" class="ltx_text ltx_font_typewriter">{tianyug,danqic}@cs.princeton.edu
<br class="ltx_break">fisch@csail.mit.edu</span>
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id12.id1" class="ltx_p">The recent GPT-3 model <cite class="ltx_cite ltx_citemacro_cite">Brown et al. (<a href="#bib.bib8" title="" class="ltx_ref">2020</a>)</cite> achieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context.
Inspired by their findings, we study few-shot learning in a more practical scenario, where we use smaller language models for which fine-tuning is computationally efficient.
We present LM-BFF—<span id="id12.id1.1" class="ltx_text ltx_framed ltx_framed_underline">b</span>etter <span id="id12.id1.2" class="ltx_text ltx_framed ltx_framed_underline">f</span>ew-shot <span id="id12.id1.3" class="ltx_text ltx_framed ltx_framed_underline">f</span>ine-tuning of <span id="id12.id1.4" class="ltx_text ltx_framed ltx_framed_underline">l</span>anguage <span id="id12.id1.5" class="ltx_text ltx_framed ltx_framed_underline">m</span>odels<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>Alternatively, language models’ <span id="footnote1.1" class="ltx_text ltx_framed ltx_framed_underline">b</span>est <span id="footnote1.2" class="ltx_text ltx_framed ltx_framed_underline">f</span>riends <span id="footnote1.3" class="ltx_text ltx_framed ltx_framed_underline">f</span>orever.</span></span></span>—a suite of simple and complementary techniques for fine-tuning language models on a small number of annotated examples. Our approach includes (1) prompt-based fine-tuning together with a novel pipeline for automating prompt generation; and (2) a refined strategy for dynamically and selectively incorporating demonstrations into each context.
Finally, we present a systematic evaluation for analyzing few-shot performance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dramatically outperform standard fine-tuning procedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning.<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>Our implementation is publicly available at <a target="_blank" href="https://github.com/princeton-nlp/LM-BFF" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/princeton-nlp/LM-BFF</a>.</span></span></span></p>
</div>
<span id="footnotex1" class="ltx_note ltx_role_footnotetext"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_note_type">footnotetext: </span>The first two authors contributed equally.</span></span></span>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<figure id="S1.F1" class="ltx_figure"><img src="/html/2012.15723/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="460" height="172" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>An illustration of (a) masked language model (MLM) pre-training, (b) standard fine-tuning, and (c) our proposed LM-BFF using prompt-based fine-tuning with demonstrations. The underlined text is the task-specific <em id="S1.F1.3.1" class="ltx_emph ltx_font_italic">template</em>, and colored words are <em id="S1.F1.4.2" class="ltx_emph ltx_font_italic">label words</em>.</figcaption>
</figure>
<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">The GPT-3 model <cite class="ltx_cite ltx_citemacro_cite">Brown et al. (<a href="#bib.bib8" title="" class="ltx_ref">2020</a>)</cite> has made waves in the NLP community by demonstrating astounding few-shot capabilities on myriad language understanding tasks.
Given only a <span id="S1.p1.1.1" class="ltx_text ltx_font_italic">natural language prompt</span> and a few <em id="S1.p1.1.2" class="ltx_emph ltx_font_italic">demonstrations</em> of the task, GPT-3 is able to make accurate predictions without updating any of the weights of its underlying language model.
However, while remarkable, GPT-3 consists of 175B parameters, which makes it challenging to use in most real-wold applications.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">In this work, we study a more practical scenario in which we only assume access to a moderately-sized language model such as BERT <cite class="ltx_cite ltx_citemacro_cite">Devlin et al. (<a href="#bib.bib13" title="" class="ltx_ref">2019</a>)</cite> or RoBERTa <cite class="ltx_cite ltx_citemacro_cite">Liu et al. (<a href="#bib.bib22" title="" class="ltx_ref">2019</a>)</cite>, and a small number of examples (i.e., a <em id="S1.p2.1.1" class="ltx_emph ltx_font_italic">few-shot</em> setting), which we can use to fine-tune the weights of
the language model.
This setting is appealing as
(1) such models can be trained on typical research hardware;
(2) few-shot settings are realistic, as it is generally both easy to acquire a few annotations (e.g., 32 examples) and efficient to train on them; and
(3) updating parameters typically leads to better performance.
Inspired by GPT-3’s findings, we propose several novel strategies for expanding its few-shot learning abilities to our setting, considering both classification and—for the first time—regression.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">First, we follow the route of <span id="S1.p3.1.1" class="ltx_text ltx_font_italic">prompt-based</span> prediction, first developed by the GPT series <cite class="ltx_cite ltx_citemacro_cite">Radford et al. (<a href="#bib.bib29" title="" class="ltx_ref">2018</a>, <a href="#bib.bib30" title="" class="ltx_ref">2019</a>); Brown et al. (<a href="#bib.bib8" title="" class="ltx_ref">2020</a>)</cite> for zero-shot prediction and recently studied by PET <cite class="ltx_cite ltx_citemacro_cite">Schick and Schütze (<a href="#bib.bib35" title="" class="ltx_ref">2021a</a>, <a href="#bib.bib36" title="" class="ltx_ref">b</a>)</cite> for fine-tuning.
Prompt-based prediction treats the downstream task as a (masked) language modeling problem, where the model directly generates a textual response (referred to as a <em id="S1.p3.1.2" class="ltx_emph ltx_font_italic">label word</em>) to a given prompt defined by a task-specific <em id="S1.p3.1.3" class="ltx_emph ltx_font_italic">template</em> (see Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>(c)).
Finding the right prompts, however, is an art—requiring both domain expertise and an understanding of the language model’s inner workings.
Even if significant effort is invested, manual prompts are likely to be suboptimal.
We address this issue by introducing automatic prompt generation, including a pruned brute-force search to identify the best working label words, and a novel decoding objective to automatically generate templates using the generative T5 model <cite class="ltx_cite ltx_citemacro_cite">Raffel et al. (<a href="#bib.bib31" title="" class="ltx_ref">2020</a>)</cite>—all of which only require the few-shot training data. This allows us to cheaply obtain effective prompts that match or outperform our manually chosen ones.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">Second, we adopt the idea of incorporating demonstrations as
additional context.
GPT-3’s naive “in-context learning” paradigm picks up to 32 randomly sampled examples, and concatenates them with the input. This method is not guaranteed to prioritize the most informative demonstrations, and mixing random examples from different classes together creates long contexts which can be hard to learn from. Additionally, the number of usable demonstrations is bounded by the model’s maximum input length.
We develop a more refined strategy, where, for each input, we randomly sample a <em id="S1.p4.1.1" class="ltx_emph ltx_font_italic">single</em> example at a time from <em id="S1.p4.1.2" class="ltx_emph ltx_font_italic">each</em> class to create multiple, minimal demonstration <em id="S1.p4.1.3" class="ltx_emph ltx_font_italic">sets</em>.
We also devise a novel sampling strategy that pairs inputs with similar examples, thereby providing the model with more discriminative comparisons.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">We present a systematic evaluation for analyzing few-shot performance on 8 single-sentence and 7 sentence-pair NLP tasks. We observe that given a small number of training examples, (1) prompt-based fine-tuning largely outperforms standard fine-tuning; (2) our automatic prompt search method matches or outperforms manual prompts; and (3) incorporating demonstrations is effective for fine-tuning, and boosts few-shot performance.
Together, these simple-yet-effective methods contribute towards a dramatic improvement across the tasks we evaluate on,
and we obtain gains up to 30% absolute improvement (11% on average) compared to standard fine-tuning.
For instance, we find that a RoBERTa-large model achieves around 90% accuracy on most binary sentence classification tasks, while only relying on 32 training examples. We refer to our approach as LM-BFF, <span id="S1.p5.1.1" class="ltx_text ltx_framed ltx_framed_underline">b</span>etter <span id="S1.p5.1.2" class="ltx_text ltx_framed ltx_framed_underline">f</span>ew-shot <span id="S1.p5.1.3" class="ltx_text ltx_framed ltx_framed_underline">f</span>ine-tuning of <span id="S1.p5.1.4" class="ltx_text ltx_framed ltx_framed_underline">l</span>anguage <span id="S1.p5.1.5" class="ltx_text ltx_framed ltx_framed_underline">m</span>odels: a strong, task-agnostic method for few-shot learning.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Related Work</h2>

<div id="S2.p1" class="ltx_para ltx_noindent">
<p id="S2.p1.1" class="ltx_p"><span id="S2.p1.1.1" class="ltx_text ltx_font_bold">Language model prompting.</span> The GPT series <cite class="ltx_cite ltx_citemacro_citep">(Radford et al., <a href="#bib.bib29" title="" class="ltx_ref">2018</a>, <a href="#bib.bib30" title="" class="ltx_ref">2019</a>; Brown et al., <a href="#bib.bib8" title="" class="ltx_ref">2020</a>)</cite> fueled the development of prompt-based learning,
and we follow many of its core concepts.
We are also greatly inspired by the recent PET work <cite class="ltx_cite ltx_citemacro_citep">(Schick and Schütze, <a href="#bib.bib35" title="" class="ltx_ref">2021a</a>, <a href="#bib.bib36" title="" class="ltx_ref">b</a>)</cite>, although they mainly focus on a semi-supervised setting where a large set of unlabeled examples are provided.
We only use a few annotated examples as supervision, and also explore automatically generated prompts and fine-tuning with demonstrations.
Furthermore, we deviate from their evaluation by providing a more rigorous framework, as we will discuss in §<a href="#S3" title="3 Problem Setup ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>.
Finally, there is a large body of work on prompting for mining knowledge from pre-trained models <cite class="ltx_cite ltx_citemacro_cite">(Trinh and Le, <a href="#bib.bib40" title="" class="ltx_ref">2018</a>; Petroni et al., <a href="#bib.bib27" title="" class="ltx_ref">2019</a>; Davison et al., <a href="#bib.bib12" title="" class="ltx_ref">2019</a>; Talmor et al., <a href="#bib.bib39" title="" class="ltx_ref">2020</a>, <em id="S2.p1.1.2.1" class="ltx_emph ltx_font_italic">inter alia</em>)</cite>. Different from these works, we focus on leveraging prompting for fine-tuning on downstream tasks.</p>
</div>
<section id="S2.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Automatic prompt search.</h4>

<div id="S2.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px1.p1.1" class="ltx_p"><cite class="ltx_cite ltx_citemacro_citet">Schick and Schütze (<a href="#bib.bib35" title="" class="ltx_ref">2021a</a>)</cite> and <cite class="ltx_cite ltx_citemacro_citet">Schick et al. (<a href="#bib.bib34" title="" class="ltx_ref">2020</a>)</cite> explore ways of identifying label words automatically,
however, none of these results lead to better performance compared to hand-picked ones.
In contrast, our method searches over both templates and label words, and is able to match or outperform our manual prompts.
Several other attempts have been made in addition—yet these approaches either operate in limited domains,
such as finding patterns to express specific relations <cite class="ltx_cite ltx_citemacro_cite">Jiang et al. (<a href="#bib.bib20" title="" class="ltx_ref">2020</a>)</cite>,
or require a large number of examples for gradient-guided search <cite class="ltx_cite ltx_citemacro_cite">Shin et al. (<a href="#bib.bib37" title="" class="ltx_ref">2020</a>); Zhong et al. (<a href="#bib.bib50" title="" class="ltx_ref">2021</a>)</cite>. Our approach aims to develop general-purpose search methods that rely only on a few annotations.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Fine-tuning of language models.</h4>

<div id="S2.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px2.p1.1" class="ltx_p">A number of recent studies have focused on better methods for fine-tuning language models <cite class="ltx_cite ltx_citemacro_cite">Howard and Ruder (<a href="#bib.bib18" title="" class="ltx_ref">2018</a>); Dodge et al. (<a href="#bib.bib14" title="" class="ltx_ref">2020</a>); Lee et al. (<a href="#bib.bib21" title="" class="ltx_ref">2020</a>); Zhang et al. (<a href="#bib.bib49" title="" class="ltx_ref">2021</a>)</cite>. These works mainly focus on optimization and regularization techniques to stabilize fine-tuning. Here we use standard optimization techniques, and instead mainly focus our efforts on better prompt-based fine-tuning in a more extreme few-shot setting. We anticipate that results of these studies are largely complementary to ours.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Few-shot learning.</h4>

<div id="S2.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px3.p1.1" class="ltx_p">Broadly speaking, our setting is also connected to other few-shot learning paradigms in NLP, including
(1) semi-supervised learning <cite class="ltx_cite ltx_citemacro_cite">Miyato et al. (<a href="#bib.bib24" title="" class="ltx_ref">2017</a>); Xie et al. (<a href="#bib.bib46" title="" class="ltx_ref">2020</a>); Chen et al. (<a href="#bib.bib10" title="" class="ltx_ref">2020</a>)</cite>, where a set of unlabeled examples are given;
(2) meta-learning <cite class="ltx_cite ltx_citemacro_cite">Yu et al. (<a href="#bib.bib48" title="" class="ltx_ref">2018</a>); Han et al. (<a href="#bib.bib17" title="" class="ltx_ref">2018</a>); Bansal et al. (<a href="#bib.bib1" title="" class="ltx_ref">2020a</a>, <a href="#bib.bib2" title="" class="ltx_ref">b</a>); Bao et al. (<a href="#bib.bib3" title="" class="ltx_ref">2020</a>)</cite>, where a set of auxiliary tasks are given; and
(3) intermediate training <cite class="ltx_cite ltx_citemacro_cite">Phang et al. (<a href="#bib.bib28" title="" class="ltx_ref">2018</a>); Yin et al. (<a href="#bib.bib47" title="" class="ltx_ref">2020</a>)</cite>, where a related, intermediate task is given. We deviate from these settings by making minimal assumptions about available resources: we only assume a few annotated examples and a pre-trained language model. Our focus is on understanding how far we can push without any other advantages.</p>
</div>
<figure id="S2.T1" class="ltx_table">
<div id="S2.T1.23.23" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:858.6pt;height:472.6pt;vertical-align:-1.6pt;"><span class="ltx_transformed_inner" style="transform:translate(166.8pt,-91.5pt) scale(1.63548137950343,1.63548137950343) ;">
<table id="S2.T1.23.23.23" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S2.T1.23.23.23.24.1" class="ltx_tr">
<td id="S2.T1.23.23.23.24.1.1" class="ltx_td ltx_align_left ltx_border_tt"><span id="S2.T1.23.23.23.24.1.1.1" class="ltx_text ltx_font_bold">Task</span></td>
<td id="S2.T1.23.23.23.24.1.2" class="ltx_td ltx_align_left ltx_border_tt"><span id="S2.T1.23.23.23.24.1.2.1" class="ltx_text ltx_font_bold">Template</span></td>
<td id="S2.T1.23.23.23.24.1.3" class="ltx_td ltx_align_left ltx_border_tt"><span id="S2.T1.23.23.23.24.1.3.1" class="ltx_text ltx_font_bold">Label words</span></td>
</tr>
<tr id="S2.T1.1.1.1.1" class="ltx_tr">
<td id="S2.T1.1.1.1.1.2" class="ltx_td ltx_align_left ltx_border_t">SST-2</td>
<td id="S2.T1.1.1.1.1.1" class="ltx_td ltx_align_left ltx_border_t">
<span id="S2.T1.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S2.T1.1.1.1.1.1.1.m1.1a"><msub id="S2.T1.1.1.1.1.1.1.m1.1.1" xref="S2.T1.1.1.1.1.1.1.m1.1.1.cmml"><mi id="S2.T1.1.1.1.1.1.1.m1.1.1.2" xref="S2.T1.1.1.1.1.1.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.1.1.1.1.1.1.m1.1.1.3" xref="S2.T1.1.1.1.1.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.1.1.1.1.1.1.m1.1b"><apply id="S2.T1.1.1.1.1.1.1.m1.1.1.cmml" xref="S2.T1.1.1.1.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.1.1.1.1.1.1.m1.1.1.1.cmml" xref="S2.T1.1.1.1.1.1.1.m1.1.1">subscript</csymbol><ci id="S2.T1.1.1.1.1.1.1.m1.1.1.2.cmml" xref="S2.T1.1.1.1.1.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.1.1.1.1.1.1.m1.1.1.3.cmml" xref="S2.T1.1.1.1.1.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.1.1.1.1.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> It was <span id="S2.T1.1.1.1.1.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="S2.T1.1.1.1.1.3" class="ltx_td ltx_align_left ltx_border_t">positive: great, negative: terrible</td>
</tr>
<tr id="S2.T1.2.2.2.2" class="ltx_tr">
<td id="S2.T1.2.2.2.2.2" class="ltx_td ltx_align_left">SST-5</td>
<td id="S2.T1.2.2.2.2.1" class="ltx_td ltx_align_left">
<span id="S2.T1.2.2.2.2.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.2.2.2.2.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S2.T1.2.2.2.2.1.1.m1.1a"><msub id="S2.T1.2.2.2.2.1.1.m1.1.1" xref="S2.T1.2.2.2.2.1.1.m1.1.1.cmml"><mi id="S2.T1.2.2.2.2.1.1.m1.1.1.2" xref="S2.T1.2.2.2.2.1.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.2.2.2.2.1.1.m1.1.1.3" xref="S2.T1.2.2.2.2.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.2.2.2.2.1.1.m1.1b"><apply id="S2.T1.2.2.2.2.1.1.m1.1.1.cmml" xref="S2.T1.2.2.2.2.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.2.2.2.2.1.1.m1.1.1.1.cmml" xref="S2.T1.2.2.2.2.1.1.m1.1.1">subscript</csymbol><ci id="S2.T1.2.2.2.2.1.1.m1.1.1.2.cmml" xref="S2.T1.2.2.2.2.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.2.2.2.2.1.1.m1.1.1.3.cmml" xref="S2.T1.2.2.2.2.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.2.2.2.2.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> It was <span id="S2.T1.2.2.2.2.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="S2.T1.2.2.2.2.3" class="ltx_td ltx_align_left">v.positive: great, positive: good, neutral: okay, negative: bad, v.negative: terrible</td>
</tr>
<tr id="S2.T1.3.3.3.3" class="ltx_tr">
<td id="S2.T1.3.3.3.3.2" class="ltx_td ltx_align_left">MR</td>
<td id="S2.T1.3.3.3.3.1" class="ltx_td ltx_align_left">
<span id="S2.T1.3.3.3.3.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.3.3.3.3.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S2.T1.3.3.3.3.1.1.m1.1a"><msub id="S2.T1.3.3.3.3.1.1.m1.1.1" xref="S2.T1.3.3.3.3.1.1.m1.1.1.cmml"><mi id="S2.T1.3.3.3.3.1.1.m1.1.1.2" xref="S2.T1.3.3.3.3.1.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.3.3.3.3.1.1.m1.1.1.3" xref="S2.T1.3.3.3.3.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.3.3.3.3.1.1.m1.1b"><apply id="S2.T1.3.3.3.3.1.1.m1.1.1.cmml" xref="S2.T1.3.3.3.3.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.3.3.3.3.1.1.m1.1.1.1.cmml" xref="S2.T1.3.3.3.3.1.1.m1.1.1">subscript</csymbol><ci id="S2.T1.3.3.3.3.1.1.m1.1.1.2.cmml" xref="S2.T1.3.3.3.3.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.3.3.3.3.1.1.m1.1.1.3.cmml" xref="S2.T1.3.3.3.3.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.3.3.3.3.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> It was <span id="S2.T1.3.3.3.3.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="S2.T1.3.3.3.3.3" class="ltx_td ltx_align_left">positive: great, negative: terrible</td>
</tr>
<tr id="S2.T1.4.4.4.4" class="ltx_tr">
<td id="S2.T1.4.4.4.4.2" class="ltx_td ltx_align_left">CR</td>
<td id="S2.T1.4.4.4.4.1" class="ltx_td ltx_align_left">
<span id="S2.T1.4.4.4.4.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.4.4.4.4.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S2.T1.4.4.4.4.1.1.m1.1a"><msub id="S2.T1.4.4.4.4.1.1.m1.1.1" xref="S2.T1.4.4.4.4.1.1.m1.1.1.cmml"><mi id="S2.T1.4.4.4.4.1.1.m1.1.1.2" xref="S2.T1.4.4.4.4.1.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.4.4.4.4.1.1.m1.1.1.3" xref="S2.T1.4.4.4.4.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.4.4.4.4.1.1.m1.1b"><apply id="S2.T1.4.4.4.4.1.1.m1.1.1.cmml" xref="S2.T1.4.4.4.4.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.4.4.4.4.1.1.m1.1.1.1.cmml" xref="S2.T1.4.4.4.4.1.1.m1.1.1">subscript</csymbol><ci id="S2.T1.4.4.4.4.1.1.m1.1.1.2.cmml" xref="S2.T1.4.4.4.4.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.4.4.4.4.1.1.m1.1.1.3.cmml" xref="S2.T1.4.4.4.4.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.4.4.4.4.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> It was <span id="S2.T1.4.4.4.4.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="S2.T1.4.4.4.4.3" class="ltx_td ltx_align_left">positive: great, negative: terrible</td>
</tr>
<tr id="S2.T1.5.5.5.5" class="ltx_tr">
<td id="S2.T1.5.5.5.5.2" class="ltx_td ltx_align_left">Subj</td>
<td id="S2.T1.5.5.5.5.1" class="ltx_td ltx_align_left">
<span id="S2.T1.5.5.5.5.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.5.5.5.5.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S2.T1.5.5.5.5.1.1.m1.1a"><msub id="S2.T1.5.5.5.5.1.1.m1.1.1" xref="S2.T1.5.5.5.5.1.1.m1.1.1.cmml"><mi id="S2.T1.5.5.5.5.1.1.m1.1.1.2" xref="S2.T1.5.5.5.5.1.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.5.5.5.5.1.1.m1.1.1.3" xref="S2.T1.5.5.5.5.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.5.5.5.5.1.1.m1.1b"><apply id="S2.T1.5.5.5.5.1.1.m1.1.1.cmml" xref="S2.T1.5.5.5.5.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.5.5.5.5.1.1.m1.1.1.1.cmml" xref="S2.T1.5.5.5.5.1.1.m1.1.1">subscript</csymbol><ci id="S2.T1.5.5.5.5.1.1.m1.1.1.2.cmml" xref="S2.T1.5.5.5.5.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.5.5.5.5.1.1.m1.1.1.3.cmml" xref="S2.T1.5.5.5.5.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.5.5.5.5.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> This is <span id="S2.T1.5.5.5.5.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="S2.T1.5.5.5.5.3" class="ltx_td ltx_align_left">subjective: subjective, objective: objective</td>
</tr>
<tr id="S2.T1.6.6.6.6" class="ltx_tr">
<td id="S2.T1.6.6.6.6.2" class="ltx_td ltx_align_left">TREC</td>
<td id="S2.T1.6.6.6.6.1" class="ltx_td ltx_align_left">
<span id="S2.T1.6.6.6.6.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> : <span id="S2.T1.6.6.6.6.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.6.6.6.6.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S2.T1.6.6.6.6.1.1.m1.1a"><msub id="S2.T1.6.6.6.6.1.1.m1.1.1" xref="S2.T1.6.6.6.6.1.1.m1.1.1.cmml"><mi id="S2.T1.6.6.6.6.1.1.m1.1.1.2" xref="S2.T1.6.6.6.6.1.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.6.6.6.6.1.1.m1.1.1.3" xref="S2.T1.6.6.6.6.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.6.6.6.6.1.1.m1.1b"><apply id="S2.T1.6.6.6.6.1.1.m1.1.1.cmml" xref="S2.T1.6.6.6.6.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.6.6.6.6.1.1.m1.1.1.1.cmml" xref="S2.T1.6.6.6.6.1.1.m1.1.1">subscript</csymbol><ci id="S2.T1.6.6.6.6.1.1.m1.1.1.2.cmml" xref="S2.T1.6.6.6.6.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.6.6.6.6.1.1.m1.1.1.3.cmml" xref="S2.T1.6.6.6.6.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.6.6.6.6.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span>
</td>
<td id="S2.T1.6.6.6.6.3" class="ltx_td ltx_align_left">abbreviation: Expression, entity: Entity, description: Description</td>
</tr>
<tr id="S2.T1.23.23.23.25.2" class="ltx_tr">
<td id="S2.T1.23.23.23.25.2.1" class="ltx_td"></td>
<td id="S2.T1.23.23.23.25.2.2" class="ltx_td"></td>
<td id="S2.T1.23.23.23.25.2.3" class="ltx_td ltx_align_left">human: Human, location: Location, numeric: Number</td>
</tr>
<tr id="S2.T1.7.7.7.7" class="ltx_tr">
<td id="S2.T1.7.7.7.7.2" class="ltx_td ltx_align_left">COLA</td>
<td id="S2.T1.7.7.7.7.1" class="ltx_td ltx_align_left">
<span id="S2.T1.7.7.7.7.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.7.7.7.7.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S2.T1.7.7.7.7.1.1.m1.1a"><msub id="S2.T1.7.7.7.7.1.1.m1.1.1" xref="S2.T1.7.7.7.7.1.1.m1.1.1.cmml"><mi id="S2.T1.7.7.7.7.1.1.m1.1.1.2" xref="S2.T1.7.7.7.7.1.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.7.7.7.7.1.1.m1.1.1.3" xref="S2.T1.7.7.7.7.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.7.7.7.7.1.1.m1.1b"><apply id="S2.T1.7.7.7.7.1.1.m1.1.1.cmml" xref="S2.T1.7.7.7.7.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.7.7.7.7.1.1.m1.1.1.1.cmml" xref="S2.T1.7.7.7.7.1.1.m1.1.1">subscript</csymbol><ci id="S2.T1.7.7.7.7.1.1.m1.1.1.2.cmml" xref="S2.T1.7.7.7.7.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.7.7.7.7.1.1.m1.1.1.3.cmml" xref="S2.T1.7.7.7.7.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.7.7.7.7.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> This is <span id="S2.T1.7.7.7.7.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="S2.T1.7.7.7.7.3" class="ltx_td ltx_align_left">grammatical: correct, not_grammatical: incorrect</td>
</tr>
<tr id="S2.T1.9.9.9.9" class="ltx_tr">
<td id="S2.T1.9.9.9.9.3" class="ltx_td ltx_align_left ltx_border_t">MNLI</td>
<td id="S2.T1.9.9.9.9.2" class="ltx_td ltx_align_left ltx_border_t">
<span id="S2.T1.8.8.8.8.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.8.8.8.8.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S2.T1.8.8.8.8.1.1.m1.1a"><msub id="S2.T1.8.8.8.8.1.1.m1.1.1" xref="S2.T1.8.8.8.8.1.1.m1.1.1.cmml"><mi id="S2.T1.8.8.8.8.1.1.m1.1.1.2" xref="S2.T1.8.8.8.8.1.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.8.8.8.8.1.1.m1.1.1.3" xref="S2.T1.8.8.8.8.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.8.8.8.8.1.1.m1.1b"><apply id="S2.T1.8.8.8.8.1.1.m1.1.1.cmml" xref="S2.T1.8.8.8.8.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.8.8.8.8.1.1.m1.1.1.1.cmml" xref="S2.T1.8.8.8.8.1.1.m1.1.1">subscript</csymbol><ci id="S2.T1.8.8.8.8.1.1.m1.1.1.2.cmml" xref="S2.T1.8.8.8.8.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.8.8.8.8.1.1.m1.1.1.3.cmml" xref="S2.T1.8.8.8.8.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.8.8.8.8.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> ? <span id="S2.T1.9.9.9.9.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , <span id="S2.T1.9.9.9.9.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.9.9.9.9.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="S2.T1.9.9.9.9.2.2.m1.1a"><msub id="S2.T1.9.9.9.9.2.2.m1.1.1" xref="S2.T1.9.9.9.9.2.2.m1.1.1.cmml"><mi id="S2.T1.9.9.9.9.2.2.m1.1.1.2" xref="S2.T1.9.9.9.9.2.2.m1.1.1.2.cmml">S</mi><mn id="S2.T1.9.9.9.9.2.2.m1.1.1.3" xref="S2.T1.9.9.9.9.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.9.9.9.9.2.2.m1.1b"><apply id="S2.T1.9.9.9.9.2.2.m1.1.1.cmml" xref="S2.T1.9.9.9.9.2.2.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.9.9.9.9.2.2.m1.1.1.1.cmml" xref="S2.T1.9.9.9.9.2.2.m1.1.1">subscript</csymbol><ci id="S2.T1.9.9.9.9.2.2.m1.1.1.2.cmml" xref="S2.T1.9.9.9.9.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.9.9.9.9.2.2.m1.1.1.3.cmml" xref="S2.T1.9.9.9.9.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.9.9.9.9.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="S2.T1.9.9.9.9.4" class="ltx_td ltx_align_left ltx_border_t">entailment: Yes, netural: Maybe, contradiction: No</td>
</tr>
<tr id="S2.T1.11.11.11.11" class="ltx_tr">
<td id="S2.T1.11.11.11.11.3" class="ltx_td ltx_align_left">SNLI</td>
<td id="S2.T1.11.11.11.11.2" class="ltx_td ltx_align_left">
<span id="S2.T1.10.10.10.10.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.10.10.10.10.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S2.T1.10.10.10.10.1.1.m1.1a"><msub id="S2.T1.10.10.10.10.1.1.m1.1.1" xref="S2.T1.10.10.10.10.1.1.m1.1.1.cmml"><mi id="S2.T1.10.10.10.10.1.1.m1.1.1.2" xref="S2.T1.10.10.10.10.1.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.10.10.10.10.1.1.m1.1.1.3" xref="S2.T1.10.10.10.10.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.10.10.10.10.1.1.m1.1b"><apply id="S2.T1.10.10.10.10.1.1.m1.1.1.cmml" xref="S2.T1.10.10.10.10.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.10.10.10.10.1.1.m1.1.1.1.cmml" xref="S2.T1.10.10.10.10.1.1.m1.1.1">subscript</csymbol><ci id="S2.T1.10.10.10.10.1.1.m1.1.1.2.cmml" xref="S2.T1.10.10.10.10.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.10.10.10.10.1.1.m1.1.1.3.cmml" xref="S2.T1.10.10.10.10.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.10.10.10.10.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> ? <span id="S2.T1.11.11.11.11.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , <span id="S2.T1.11.11.11.11.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.11.11.11.11.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="S2.T1.11.11.11.11.2.2.m1.1a"><msub id="S2.T1.11.11.11.11.2.2.m1.1.1" xref="S2.T1.11.11.11.11.2.2.m1.1.1.cmml"><mi id="S2.T1.11.11.11.11.2.2.m1.1.1.2" xref="S2.T1.11.11.11.11.2.2.m1.1.1.2.cmml">S</mi><mn id="S2.T1.11.11.11.11.2.2.m1.1.1.3" xref="S2.T1.11.11.11.11.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.11.11.11.11.2.2.m1.1b"><apply id="S2.T1.11.11.11.11.2.2.m1.1.1.cmml" xref="S2.T1.11.11.11.11.2.2.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.11.11.11.11.2.2.m1.1.1.1.cmml" xref="S2.T1.11.11.11.11.2.2.m1.1.1">subscript</csymbol><ci id="S2.T1.11.11.11.11.2.2.m1.1.1.2.cmml" xref="S2.T1.11.11.11.11.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.11.11.11.11.2.2.m1.1.1.3.cmml" xref="S2.T1.11.11.11.11.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.11.11.11.11.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="S2.T1.11.11.11.11.4" class="ltx_td ltx_align_left">entailment: Yes, netural: Maybe, contradiction: No</td>
</tr>
<tr id="S2.T1.13.13.13.13" class="ltx_tr">
<td id="S2.T1.13.13.13.13.3" class="ltx_td ltx_align_left">QNLI</td>
<td id="S2.T1.13.13.13.13.2" class="ltx_td ltx_align_left">
<span id="S2.T1.12.12.12.12.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.12.12.12.12.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S2.T1.12.12.12.12.1.1.m1.1a"><msub id="S2.T1.12.12.12.12.1.1.m1.1.1" xref="S2.T1.12.12.12.12.1.1.m1.1.1.cmml"><mi id="S2.T1.12.12.12.12.1.1.m1.1.1.2" xref="S2.T1.12.12.12.12.1.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.12.12.12.12.1.1.m1.1.1.3" xref="S2.T1.12.12.12.12.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.12.12.12.12.1.1.m1.1b"><apply id="S2.T1.12.12.12.12.1.1.m1.1.1.cmml" xref="S2.T1.12.12.12.12.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.12.12.12.12.1.1.m1.1.1.1.cmml" xref="S2.T1.12.12.12.12.1.1.m1.1.1">subscript</csymbol><ci id="S2.T1.12.12.12.12.1.1.m1.1.1.2.cmml" xref="S2.T1.12.12.12.12.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.12.12.12.12.1.1.m1.1.1.3.cmml" xref="S2.T1.12.12.12.12.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.12.12.12.12.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> ? <span id="S2.T1.13.13.13.13.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , <span id="S2.T1.13.13.13.13.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.13.13.13.13.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="S2.T1.13.13.13.13.2.2.m1.1a"><msub id="S2.T1.13.13.13.13.2.2.m1.1.1" xref="S2.T1.13.13.13.13.2.2.m1.1.1.cmml"><mi id="S2.T1.13.13.13.13.2.2.m1.1.1.2" xref="S2.T1.13.13.13.13.2.2.m1.1.1.2.cmml">S</mi><mn id="S2.T1.13.13.13.13.2.2.m1.1.1.3" xref="S2.T1.13.13.13.13.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.13.13.13.13.2.2.m1.1b"><apply id="S2.T1.13.13.13.13.2.2.m1.1.1.cmml" xref="S2.T1.13.13.13.13.2.2.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.13.13.13.13.2.2.m1.1.1.1.cmml" xref="S2.T1.13.13.13.13.2.2.m1.1.1">subscript</csymbol><ci id="S2.T1.13.13.13.13.2.2.m1.1.1.2.cmml" xref="S2.T1.13.13.13.13.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.13.13.13.13.2.2.m1.1.1.3.cmml" xref="S2.T1.13.13.13.13.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.13.13.13.13.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="S2.T1.13.13.13.13.4" class="ltx_td ltx_align_left">entailment: Yes, not_entailment: No</td>
</tr>
<tr id="S2.T1.15.15.15.15" class="ltx_tr">
<td id="S2.T1.15.15.15.15.3" class="ltx_td ltx_align_left">RTE</td>
<td id="S2.T1.15.15.15.15.2" class="ltx_td ltx_align_left">
<span id="S2.T1.14.14.14.14.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.14.14.14.14.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S2.T1.14.14.14.14.1.1.m1.1a"><msub id="S2.T1.14.14.14.14.1.1.m1.1.1" xref="S2.T1.14.14.14.14.1.1.m1.1.1.cmml"><mi id="S2.T1.14.14.14.14.1.1.m1.1.1.2" xref="S2.T1.14.14.14.14.1.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.14.14.14.14.1.1.m1.1.1.3" xref="S2.T1.14.14.14.14.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.14.14.14.14.1.1.m1.1b"><apply id="S2.T1.14.14.14.14.1.1.m1.1.1.cmml" xref="S2.T1.14.14.14.14.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.14.14.14.14.1.1.m1.1.1.1.cmml" xref="S2.T1.14.14.14.14.1.1.m1.1.1">subscript</csymbol><ci id="S2.T1.14.14.14.14.1.1.m1.1.1.2.cmml" xref="S2.T1.14.14.14.14.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.14.14.14.14.1.1.m1.1.1.3.cmml" xref="S2.T1.14.14.14.14.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.14.14.14.14.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> ? <span id="S2.T1.15.15.15.15.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , <span id="S2.T1.15.15.15.15.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.15.15.15.15.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="S2.T1.15.15.15.15.2.2.m1.1a"><msub id="S2.T1.15.15.15.15.2.2.m1.1.1" xref="S2.T1.15.15.15.15.2.2.m1.1.1.cmml"><mi id="S2.T1.15.15.15.15.2.2.m1.1.1.2" xref="S2.T1.15.15.15.15.2.2.m1.1.1.2.cmml">S</mi><mn id="S2.T1.15.15.15.15.2.2.m1.1.1.3" xref="S2.T1.15.15.15.15.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.15.15.15.15.2.2.m1.1b"><apply id="S2.T1.15.15.15.15.2.2.m1.1.1.cmml" xref="S2.T1.15.15.15.15.2.2.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.15.15.15.15.2.2.m1.1.1.1.cmml" xref="S2.T1.15.15.15.15.2.2.m1.1.1">subscript</csymbol><ci id="S2.T1.15.15.15.15.2.2.m1.1.1.2.cmml" xref="S2.T1.15.15.15.15.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.15.15.15.15.2.2.m1.1.1.3.cmml" xref="S2.T1.15.15.15.15.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.15.15.15.15.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="S2.T1.15.15.15.15.4" class="ltx_td ltx_align_left">entailment: Yes, not_entailment: No</td>
</tr>
<tr id="S2.T1.17.17.17.17" class="ltx_tr">
<td id="S2.T1.17.17.17.17.3" class="ltx_td ltx_align_left">MRPC</td>
<td id="S2.T1.17.17.17.17.2" class="ltx_td ltx_align_left">
<span id="S2.T1.16.16.16.16.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.16.16.16.16.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S2.T1.16.16.16.16.1.1.m1.1a"><msub id="S2.T1.16.16.16.16.1.1.m1.1.1" xref="S2.T1.16.16.16.16.1.1.m1.1.1.cmml"><mi id="S2.T1.16.16.16.16.1.1.m1.1.1.2" xref="S2.T1.16.16.16.16.1.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.16.16.16.16.1.1.m1.1.1.3" xref="S2.T1.16.16.16.16.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.16.16.16.16.1.1.m1.1b"><apply id="S2.T1.16.16.16.16.1.1.m1.1.1.cmml" xref="S2.T1.16.16.16.16.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.16.16.16.16.1.1.m1.1.1.1.cmml" xref="S2.T1.16.16.16.16.1.1.m1.1.1">subscript</csymbol><ci id="S2.T1.16.16.16.16.1.1.m1.1.1.2.cmml" xref="S2.T1.16.16.16.16.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.16.16.16.16.1.1.m1.1.1.3.cmml" xref="S2.T1.16.16.16.16.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.16.16.16.16.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> <span id="S2.T1.17.17.17.17.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , <span id="S2.T1.17.17.17.17.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.17.17.17.17.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="S2.T1.17.17.17.17.2.2.m1.1a"><msub id="S2.T1.17.17.17.17.2.2.m1.1.1" xref="S2.T1.17.17.17.17.2.2.m1.1.1.cmml"><mi id="S2.T1.17.17.17.17.2.2.m1.1.1.2" xref="S2.T1.17.17.17.17.2.2.m1.1.1.2.cmml">S</mi><mn id="S2.T1.17.17.17.17.2.2.m1.1.1.3" xref="S2.T1.17.17.17.17.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.17.17.17.17.2.2.m1.1b"><apply id="S2.T1.17.17.17.17.2.2.m1.1.1.cmml" xref="S2.T1.17.17.17.17.2.2.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.17.17.17.17.2.2.m1.1.1.1.cmml" xref="S2.T1.17.17.17.17.2.2.m1.1.1">subscript</csymbol><ci id="S2.T1.17.17.17.17.2.2.m1.1.1.2.cmml" xref="S2.T1.17.17.17.17.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.17.17.17.17.2.2.m1.1.1.3.cmml" xref="S2.T1.17.17.17.17.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.17.17.17.17.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="S2.T1.17.17.17.17.4" class="ltx_td ltx_align_left">equivalent: Yes, not_equivalent: No</td>
</tr>
<tr id="S2.T1.19.19.19.19" class="ltx_tr">
<td id="S2.T1.19.19.19.19.3" class="ltx_td ltx_align_left">QQP</td>
<td id="S2.T1.19.19.19.19.2" class="ltx_td ltx_align_left">
<span id="S2.T1.18.18.18.18.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.18.18.18.18.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S2.T1.18.18.18.18.1.1.m1.1a"><msub id="S2.T1.18.18.18.18.1.1.m1.1.1" xref="S2.T1.18.18.18.18.1.1.m1.1.1.cmml"><mi id="S2.T1.18.18.18.18.1.1.m1.1.1.2" xref="S2.T1.18.18.18.18.1.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.18.18.18.18.1.1.m1.1.1.3" xref="S2.T1.18.18.18.18.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.18.18.18.18.1.1.m1.1b"><apply id="S2.T1.18.18.18.18.1.1.m1.1.1.cmml" xref="S2.T1.18.18.18.18.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.18.18.18.18.1.1.m1.1.1.1.cmml" xref="S2.T1.18.18.18.18.1.1.m1.1.1">subscript</csymbol><ci id="S2.T1.18.18.18.18.1.1.m1.1.1.2.cmml" xref="S2.T1.18.18.18.18.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.18.18.18.18.1.1.m1.1.1.3.cmml" xref="S2.T1.18.18.18.18.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.18.18.18.18.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> <span id="S2.T1.19.19.19.19.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , <span id="S2.T1.19.19.19.19.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.19.19.19.19.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="S2.T1.19.19.19.19.2.2.m1.1a"><msub id="S2.T1.19.19.19.19.2.2.m1.1.1" xref="S2.T1.19.19.19.19.2.2.m1.1.1.cmml"><mi id="S2.T1.19.19.19.19.2.2.m1.1.1.2" xref="S2.T1.19.19.19.19.2.2.m1.1.1.2.cmml">S</mi><mn id="S2.T1.19.19.19.19.2.2.m1.1.1.3" xref="S2.T1.19.19.19.19.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.19.19.19.19.2.2.m1.1b"><apply id="S2.T1.19.19.19.19.2.2.m1.1.1.cmml" xref="S2.T1.19.19.19.19.2.2.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.19.19.19.19.2.2.m1.1.1.1.cmml" xref="S2.T1.19.19.19.19.2.2.m1.1.1">subscript</csymbol><ci id="S2.T1.19.19.19.19.2.2.m1.1.1.2.cmml" xref="S2.T1.19.19.19.19.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.19.19.19.19.2.2.m1.1.1.3.cmml" xref="S2.T1.19.19.19.19.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.19.19.19.19.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="S2.T1.19.19.19.19.4" class="ltx_td ltx_align_left">equivalent: Yes, not_equivalent: No</td>
</tr>
<tr id="S2.T1.23.23.23.23" class="ltx_tr">
<td id="S2.T1.23.23.23.23.5" class="ltx_td ltx_align_left ltx_border_bb">STS-B</td>
<td id="S2.T1.21.21.21.21.2" class="ltx_td ltx_align_left ltx_border_bb">
<span id="S2.T1.20.20.20.20.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.20.20.20.20.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S2.T1.20.20.20.20.1.1.m1.1a"><msub id="S2.T1.20.20.20.20.1.1.m1.1.1" xref="S2.T1.20.20.20.20.1.1.m1.1.1.cmml"><mi id="S2.T1.20.20.20.20.1.1.m1.1.1.2" xref="S2.T1.20.20.20.20.1.1.m1.1.1.2.cmml">S</mi><mn id="S2.T1.20.20.20.20.1.1.m1.1.1.3" xref="S2.T1.20.20.20.20.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.20.20.20.20.1.1.m1.1b"><apply id="S2.T1.20.20.20.20.1.1.m1.1.1.cmml" xref="S2.T1.20.20.20.20.1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.20.20.20.20.1.1.m1.1.1.1.cmml" xref="S2.T1.20.20.20.20.1.1.m1.1.1">subscript</csymbol><ci id="S2.T1.20.20.20.20.1.1.m1.1.1.2.cmml" xref="S2.T1.20.20.20.20.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.20.20.20.20.1.1.m1.1.1.3.cmml" xref="S2.T1.20.20.20.20.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.20.20.20.20.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> <span id="S2.T1.21.21.21.21.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , <span id="S2.T1.21.21.21.21.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="S2.T1.21.21.21.21.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="S2.T1.21.21.21.21.2.2.m1.1a"><msub id="S2.T1.21.21.21.21.2.2.m1.1.1" xref="S2.T1.21.21.21.21.2.2.m1.1.1.cmml"><mi id="S2.T1.21.21.21.21.2.2.m1.1.1.2" xref="S2.T1.21.21.21.21.2.2.m1.1.1.2.cmml">S</mi><mn id="S2.T1.21.21.21.21.2.2.m1.1.1.3" xref="S2.T1.21.21.21.21.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S2.T1.21.21.21.21.2.2.m1.1b"><apply id="S2.T1.21.21.21.21.2.2.m1.1.1.cmml" xref="S2.T1.21.21.21.21.2.2.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.21.21.21.21.2.2.m1.1.1.1.cmml" xref="S2.T1.21.21.21.21.2.2.m1.1.1">subscript</csymbol><ci id="S2.T1.21.21.21.21.2.2.m1.1.1.2.cmml" xref="S2.T1.21.21.21.21.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="S2.T1.21.21.21.21.2.2.m1.1.1.3.cmml" xref="S2.T1.21.21.21.21.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.21.21.21.21.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="S2.T1.23.23.23.23.4" class="ltx_td ltx_align_left ltx_border_bb">
<math id="S2.T1.22.22.22.22.3.m1.1" class="ltx_Math" alttext="y_{u}" display="inline"><semantics id="S2.T1.22.22.22.22.3.m1.1a"><msub id="S2.T1.22.22.22.22.3.m1.1.1" xref="S2.T1.22.22.22.22.3.m1.1.1.cmml"><mi id="S2.T1.22.22.22.22.3.m1.1.1.2" xref="S2.T1.22.22.22.22.3.m1.1.1.2.cmml">y</mi><mi id="S2.T1.22.22.22.22.3.m1.1.1.3" xref="S2.T1.22.22.22.22.3.m1.1.1.3.cmml">u</mi></msub><annotation-xml encoding="MathML-Content" id="S2.T1.22.22.22.22.3.m1.1b"><apply id="S2.T1.22.22.22.22.3.m1.1.1.cmml" xref="S2.T1.22.22.22.22.3.m1.1.1"><csymbol cd="ambiguous" id="S2.T1.22.22.22.22.3.m1.1.1.1.cmml" xref="S2.T1.22.22.22.22.3.m1.1.1">subscript</csymbol><ci id="S2.T1.22.22.22.22.3.m1.1.1.2.cmml" xref="S2.T1.22.22.22.22.3.m1.1.1.2">𝑦</ci><ci id="S2.T1.22.22.22.22.3.m1.1.1.3.cmml" xref="S2.T1.22.22.22.22.3.m1.1.1.3">𝑢</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.22.22.22.22.3.m1.1c">y_{u}</annotation></semantics></math>: Yes, <math id="S2.T1.23.23.23.23.4.m2.1" class="ltx_Math" alttext="y_{l}" display="inline"><semantics id="S2.T1.23.23.23.23.4.m2.1a"><msub id="S2.T1.23.23.23.23.4.m2.1.1" xref="S2.T1.23.23.23.23.4.m2.1.1.cmml"><mi id="S2.T1.23.23.23.23.4.m2.1.1.2" xref="S2.T1.23.23.23.23.4.m2.1.1.2.cmml">y</mi><mi id="S2.T1.23.23.23.23.4.m2.1.1.3" xref="S2.T1.23.23.23.23.4.m2.1.1.3.cmml">l</mi></msub><annotation-xml encoding="MathML-Content" id="S2.T1.23.23.23.23.4.m2.1b"><apply id="S2.T1.23.23.23.23.4.m2.1.1.cmml" xref="S2.T1.23.23.23.23.4.m2.1.1"><csymbol cd="ambiguous" id="S2.T1.23.23.23.23.4.m2.1.1.1.cmml" xref="S2.T1.23.23.23.23.4.m2.1.1">subscript</csymbol><ci id="S2.T1.23.23.23.23.4.m2.1.1.2.cmml" xref="S2.T1.23.23.23.23.4.m2.1.1.2">𝑦</ci><ci id="S2.T1.23.23.23.23.4.m2.1.1.3.cmml" xref="S2.T1.23.23.23.23.4.m2.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.T1.23.23.23.23.4.m2.1c">y_{l}</annotation></semantics></math>: No</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 1: </span>Manual templates and label words that we used in our experiments.
STS-B is a regression task (§<a href="#S4.SS2" title="4.2 Regression ‣ 4 Prompt-based Fine-tuning ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2</span></a>).
</figcaption>
</figure>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Problem Setup</h2>

<section id="S3.SS0.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Task formulation.</h4>

<div id="S3.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S3.SS0.SSS0.Px1.p1.12" class="ltx_p">In this work, we assume access to a pre-trained language model <math id="S3.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{L}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS0.SSS0.Px1.p1.1.m1.1.1" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1.cmml">ℒ</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.1.m1.1b"><ci id="S3.SS0.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.1.m1.1.1">ℒ</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.1.m1.1c">\mathcal{L}</annotation></semantics></math> that we wish to fine-tune on a task <math id="S3.SS0.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="\mathcal{D}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS0.SSS0.Px1.p1.2.m2.1.1" xref="S3.SS0.SSS0.Px1.p1.2.m2.1.1.cmml">𝒟</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.2.m2.1b"><ci id="S3.SS0.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.2.m2.1.1">𝒟</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.2.m2.1c">\mathcal{D}</annotation></semantics></math> with a label space <math id="S3.SS0.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{Y}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="S3.SS0.SSS0.Px1.p1.3.m3.1.1" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1.cmml">𝒴</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.3.m3.1b"><ci id="S3.SS0.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.3.m3.1.1">𝒴</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.3.m3.1c">\mathcal{Y}</annotation></semantics></math>. For the task, we only assume <math id="S3.SS0.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.4.m4.1a"><mi id="S3.SS0.SSS0.Px1.p1.4.m4.1.1" xref="S3.SS0.SSS0.Px1.p1.4.m4.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.4.m4.1b"><ci id="S3.SS0.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.4.m4.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.4.m4.1c">K</annotation></semantics></math> training examples <em id="S3.SS0.SSS0.Px1.p1.12.1" class="ltx_emph ltx_font_italic">per class</em><span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>For regression, we partition the data into two “classes” according to being above or below the median value.</span></span></span> for the task’s training set <math id="S3.SS0.SSS0.Px1.p1.5.m5.1" class="ltx_Math" alttext="\mathcal{D}_{\text{train}}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.5.m5.1a"><msub id="S3.SS0.SSS0.Px1.p1.5.m5.1.1" xref="S3.SS0.SSS0.Px1.p1.5.m5.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS0.SSS0.Px1.p1.5.m5.1.1.2" xref="S3.SS0.SSS0.Px1.p1.5.m5.1.1.2.cmml">𝒟</mi><mtext id="S3.SS0.SSS0.Px1.p1.5.m5.1.1.3" xref="S3.SS0.SSS0.Px1.p1.5.m5.1.1.3a.cmml">train</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.5.m5.1b"><apply id="S3.SS0.SSS0.Px1.p1.5.m5.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.5.m5.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.5.m5.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.5.m5.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.5.m5.1.1.2">𝒟</ci><ci id="S3.SS0.SSS0.Px1.p1.5.m5.1.1.3a.cmml" xref="S3.SS0.SSS0.Px1.p1.5.m5.1.1.3"><mtext mathsize="70%" id="S3.SS0.SSS0.Px1.p1.5.m5.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.5.m5.1.1.3">train</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.5.m5.1c">\mathcal{D}_{\text{train}}</annotation></semantics></math>, such that the total number of examples is <math id="S3.SS0.SSS0.Px1.p1.6.m6.1" class="ltx_Math" alttext="K_{\text{tot}}=K\times|\mathcal{Y}|" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.6.m6.1a"><mrow id="S3.SS0.SSS0.Px1.p1.6.m6.1.2" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.cmml"><msub id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.2" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.2.cmml"><mi id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.2.2" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.2.2.cmml">K</mi><mtext id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.2.3" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.2.3a.cmml">tot</mtext></msub><mo id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.1" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.1.cmml">=</mo><mrow id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.cmml"><mi id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.2" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.2.cmml">K</mi><mo lspace="0.222em" rspace="0.222em" id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.1" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.1.cmml">×</mo><mrow id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.3.2" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.3.1.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.3.2.1" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.3.1.1.cmml">|</mo><mi class="ltx_font_mathcaligraphic" id="S3.SS0.SSS0.Px1.p1.6.m6.1.1" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.1.cmml">𝒴</mi><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.3.2.2" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.3.1.1.cmml">|</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.6.m6.1b"><apply id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2"><eq id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.1"></eq><apply id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.2">subscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.2.2">𝐾</ci><ci id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.2.3a.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.2.3"><mtext mathsize="70%" id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.2.3">tot</mtext></ci></apply><apply id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3"><times id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.1"></times><ci id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.2">𝐾</ci><apply id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.3.2"><abs id="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.3.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.2.3.3.2.1"></abs><ci id="S3.SS0.SSS0.Px1.p1.6.m6.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.6.m6.1.1">𝒴</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.6.m6.1c">K_{\text{tot}}=K\times|\mathcal{Y}|</annotation></semantics></math>, and <math id="S3.SS0.SSS0.Px1.p1.7.m7.1" class="ltx_Math" alttext="\mathcal{D}_{\text{train}}=\{({x}_{\mathrm{in}}^{i},y^{i})\}_{i=1}^{K_{\text{tot}}}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.7.m7.1a"><mrow id="S3.SS0.SSS0.Px1.p1.7.m7.1.1" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.cmml"><msub id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.2" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.2.cmml">𝒟</mi><mtext id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.3" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.3a.cmml">train</mtext></msub><mo id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.cmml">=</mo><msubsup id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.cmml"><mrow id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.2" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.2.cmml">{</mo><mrow id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.3.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.3" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.3.cmml">(</mo><msubsup id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1.2.2" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1.2.3" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1.2.3.cmml">in</mi><mi id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1.3.cmml">i</mi></msubsup><mo id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.4" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.3.cmml">,</mo><msup id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.2" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.2.cmml"><mi id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.2.2" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.2.2.cmml">y</mi><mi id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.2.3" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.2.3.cmml">i</mi></msup><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.5" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.3.cmml">)</mo></mrow><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.cmml"><mi id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.2" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.2.cmml">i</mi><mo id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.1" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.1.cmml">=</mo><mn id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.3" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.3.cmml">1</mn></mrow><msub id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3.cmml"><mi id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3.2" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3.2.cmml">K</mi><mtext id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3.3" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3.3a.cmml">tot</mtext></msub></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.7.m7.1b"><apply id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1"><eq id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.2"></eq><apply id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3">subscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.2">𝒟</ci><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.3a.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.3"><mtext mathsize="70%" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.3.3">train</mtext></ci></apply><apply id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1">superscript</csymbol><apply id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1">subscript</csymbol><set id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1"><interval closure="open" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2"><apply id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1">superscript</csymbol><apply id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1.2.3">in</ci></apply><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.1.1.3">𝑖</ci></apply><apply id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.2">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.2.2">𝑦</ci><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.1.1.1.2.2.3">𝑖</ci></apply></interval></set><apply id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3"><eq id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.1"></eq><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.2">𝑖</ci><cn type="integer" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.1.3.3">1</cn></apply></apply><apply id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3.1.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3">subscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3.2.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3.2">𝐾</ci><ci id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3.3a.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3.3"><mtext mathsize="50%" id="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3.3.cmml" xref="S3.SS0.SSS0.Px1.p1.7.m7.1.1.1.3.3">tot</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.7.m7.1c">\mathcal{D}_{\text{train}}=\{({x}_{\mathrm{in}}^{i},y^{i})\}_{i=1}^{K_{\text{tot}}}</annotation></semantics></math>.
Our goal is then to develop task-agnostic learning strategies that generalize well to an unseen test set <math id="S3.SS0.SSS0.Px1.p1.8.m8.2" class="ltx_Math" alttext="({x}_{\mathrm{in}}^{\text{test}},y^{\text{test}})\sim\mathcal{D}_{\text{test}}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.8.m8.2a"><mrow id="S3.SS0.SSS0.Px1.p1.8.m8.2.2" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.cmml"><mrow id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.3.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.3" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.3.cmml">(</mo><msubsup id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.cmml"><mi id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.2.2" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.2.2.cmml">x</mi><mi id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.2.3" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.2.3.cmml">in</mi><mtext id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.3a.cmml">test</mtext></msubsup><mo id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.4" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.3.cmml">,</mo><msup id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.2" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.2.cmml"><mi id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.2.2" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.2.2.cmml">y</mi><mtext id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.2.3" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.2.3a.cmml">test</mtext></msup><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.5" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.3.cmml">)</mo></mrow><mo id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.3" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.3.cmml">∼</mo><msub id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.4" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.4.2" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.4.2.cmml">𝒟</mi><mtext id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.4.3" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.4.3a.cmml">test</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.8.m8.2b"><apply id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2"><csymbol cd="latexml" id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.3">similar-to</csymbol><interval closure="open" id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2"><apply id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1">superscript</csymbol><apply id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.2.2">𝑥</ci><ci id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.2.3">in</ci></apply><ci id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.3a.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.3"><mtext mathsize="70%" id="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.1.1.1.1.1.3">test</mtext></ci></apply><apply id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.2">superscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.2.2">𝑦</ci><ci id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.2.3a.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.2.3"><mtext mathsize="70%" id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.2.2.2.3">test</mtext></ci></apply></interval><apply id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.4.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.4"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.4.1.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.4">subscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.4.2.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.4.2">𝒟</ci><ci id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.4.3a.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.4.3"><mtext mathsize="70%" id="S3.SS0.SSS0.Px1.p1.8.m8.2.2.4.3.cmml" xref="S3.SS0.SSS0.Px1.p1.8.m8.2.2.4.3">test</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.8.m8.2c">({x}_{\mathrm{in}}^{\text{test}},y^{\text{test}})\sim\mathcal{D}_{\text{test}}</annotation></semantics></math>. For model selection and hyper-parameter tuning, we assume a development set <math id="S3.SS0.SSS0.Px1.p1.9.m9.1" class="ltx_Math" alttext="\mathcal{D}_{\text{dev}}" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.9.m9.1a"><msub id="S3.SS0.SSS0.Px1.p1.9.m9.1.1" xref="S3.SS0.SSS0.Px1.p1.9.m9.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS0.SSS0.Px1.p1.9.m9.1.1.2" xref="S3.SS0.SSS0.Px1.p1.9.m9.1.1.2.cmml">𝒟</mi><mtext id="S3.SS0.SSS0.Px1.p1.9.m9.1.1.3" xref="S3.SS0.SSS0.Px1.p1.9.m9.1.1.3a.cmml">dev</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.9.m9.1b"><apply id="S3.SS0.SSS0.Px1.p1.9.m9.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.9.m9.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.9.m9.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.1.1.2">𝒟</ci><ci id="S3.SS0.SSS0.Px1.p1.9.m9.1.1.3a.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.1.1.3"><mtext mathsize="70%" id="S3.SS0.SSS0.Px1.p1.9.m9.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.9.m9.1.1.3">dev</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.9.m9.1c">\mathcal{D}_{\text{dev}}</annotation></semantics></math>, of the same size as the few-shot training set, i.e., <math id="S3.SS0.SSS0.Px1.p1.10.m10.2" class="ltx_Math" alttext="|\mathcal{D}_{\text{dev}}|=|\mathcal{D}_{\text{train}}|" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.10.m10.2a"><mrow id="S3.SS0.SSS0.Px1.p1.10.m10.2.2" xref="S3.SS0.SSS0.Px1.p1.10.m10.2.2.cmml"><mrow id="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1" xref="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.2" xref="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.2.1.cmml">|</mo><msub id="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.1" xref="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.1.2" xref="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.1.2.cmml">𝒟</mi><mtext id="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.1.3a.cmml">dev</mtext></msub><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.3" xref="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.2.1.cmml">|</mo></mrow><mo id="S3.SS0.SSS0.Px1.p1.10.m10.2.2.3" xref="S3.SS0.SSS0.Px1.p1.10.m10.2.2.3.cmml">=</mo><mrow id="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1" xref="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.2.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.2" xref="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.2.1.cmml">|</mo><msub id="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.1" xref="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.1.2" xref="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.1.2.cmml">𝒟</mi><mtext id="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.1.3" xref="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.1.3a.cmml">train</mtext></msub><mo stretchy="false" id="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.3" xref="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.2.1.cmml">|</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.10.m10.2b"><apply id="S3.SS0.SSS0.Px1.p1.10.m10.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m10.2.2"><eq id="S3.SS0.SSS0.Px1.p1.10.m10.2.2.3.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m10.2.2.3"></eq><apply id="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1"><abs id="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.2"></abs><apply id="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.1.2">𝒟</ci><ci id="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.1.3a.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.1.3"><mtext mathsize="70%" id="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m10.1.1.1.1.1.3">dev</mtext></ci></apply></apply><apply id="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.2.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1"><abs id="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.2.1.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.2"></abs><apply id="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.1.2">𝒟</ci><ci id="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.1.3a.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.1.3"><mtext mathsize="70%" id="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.10.m10.2.2.2.1.1.3">train</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.10.m10.2c">|\mathcal{D}_{\text{dev}}|=|\mathcal{D}_{\text{train}}|</annotation></semantics></math>. This distinction is important: using a larger development set confers a significant advantage
(see our experiments in Appendix <a href="#A1" title="Appendix A Impact of Development Sets ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A</span></a>),
and subverts our initial goal of learning from limited data.<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>In contrast, <cite class="ltx_cite ltx_citemacro_citet">Schick and Schütze (<a href="#bib.bib35" title="" class="ltx_ref">2021a</a>, <a href="#bib.bib36" title="" class="ltx_ref">b</a>)</cite> do not use a development set, and adopt a set of hyper-parameters based on practical considerations.
This is akin to “shooting in the dark” on a setting that we show can have unintuitive outcomes.</span></span></span>
For all of the following experiments (unless specified otherwise), we take <math id="S3.SS0.SSS0.Px1.p1.11.m11.1" class="ltx_Math" alttext="\mathcal{L}=" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.11.m11.1a"><mrow id="S3.SS0.SSS0.Px1.p1.11.m11.1.1" xref="S3.SS0.SSS0.Px1.p1.11.m11.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS0.SSS0.Px1.p1.11.m11.1.1.2" xref="S3.SS0.SSS0.Px1.p1.11.m11.1.1.2.cmml">ℒ</mi><mo id="S3.SS0.SSS0.Px1.p1.11.m11.1.1.1" xref="S3.SS0.SSS0.Px1.p1.11.m11.1.1.1.cmml">=</mo><mi id="S3.SS0.SSS0.Px1.p1.11.m11.1.1.3" xref="S3.SS0.SSS0.Px1.p1.11.m11.1.1.3.cmml"></mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.11.m11.1b"><apply id="S3.SS0.SSS0.Px1.p1.11.m11.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.11.m11.1.1"><eq id="S3.SS0.SSS0.Px1.p1.11.m11.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.11.m11.1.1.1"></eq><ci id="S3.SS0.SSS0.Px1.p1.11.m11.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.11.m11.1.1.2">ℒ</ci><csymbol cd="latexml" id="S3.SS0.SSS0.Px1.p1.11.m11.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.11.m11.1.1.3">absent</csymbol></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.11.m11.1c">\mathcal{L}=</annotation></semantics></math> RoBERTa-large and <math id="S3.SS0.SSS0.Px1.p1.12.m12.1" class="ltx_Math" alttext="K=16" display="inline"><semantics id="S3.SS0.SSS0.Px1.p1.12.m12.1a"><mrow id="S3.SS0.SSS0.Px1.p1.12.m12.1.1" xref="S3.SS0.SSS0.Px1.p1.12.m12.1.1.cmml"><mi id="S3.SS0.SSS0.Px1.p1.12.m12.1.1.2" xref="S3.SS0.SSS0.Px1.p1.12.m12.1.1.2.cmml">K</mi><mo id="S3.SS0.SSS0.Px1.p1.12.m12.1.1.1" xref="S3.SS0.SSS0.Px1.p1.12.m12.1.1.1.cmml">=</mo><mn id="S3.SS0.SSS0.Px1.p1.12.m12.1.1.3" xref="S3.SS0.SSS0.Px1.p1.12.m12.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px1.p1.12.m12.1b"><apply id="S3.SS0.SSS0.Px1.p1.12.m12.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.12.m12.1.1"><eq id="S3.SS0.SSS0.Px1.p1.12.m12.1.1.1.cmml" xref="S3.SS0.SSS0.Px1.p1.12.m12.1.1.1"></eq><ci id="S3.SS0.SSS0.Px1.p1.12.m12.1.1.2.cmml" xref="S3.SS0.SSS0.Px1.p1.12.m12.1.1.2">𝐾</ci><cn type="integer" id="S3.SS0.SSS0.Px1.p1.12.m12.1.1.3.cmml" xref="S3.SS0.SSS0.Px1.p1.12.m12.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px1.p1.12.m12.1c">K=16</annotation></semantics></math>.</p>
</div>
</section>
<section id="S3.SS0.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Evaluation datasets.</h4>

<div id="S3.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S3.SS0.SSS0.Px2.p1.7" class="ltx_p">We conduct a systematic study across <math id="S3.SS0.SSS0.Px2.p1.1.m1.1" class="ltx_Math" alttext="8" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.1.m1.1a"><mn id="S3.SS0.SSS0.Px2.p1.1.m1.1.1" xref="S3.SS0.SSS0.Px2.p1.1.m1.1.1.cmml">8</mn><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.1.m1.1b"><cn type="integer" id="S3.SS0.SSS0.Px2.p1.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.1.m1.1.1">8</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.1.m1.1c">8</annotation></semantics></math> single-sentence and <math id="S3.SS0.SSS0.Px2.p1.2.m2.1" class="ltx_Math" alttext="7" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.2.m2.1a"><mn id="S3.SS0.SSS0.Px2.p1.2.m2.1.1" xref="S3.SS0.SSS0.Px2.p1.2.m2.1.1.cmml">7</mn><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.2.m2.1b"><cn type="integer" id="S3.SS0.SSS0.Px2.p1.2.m2.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.2.m2.1.1">7</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.2.m2.1c">7</annotation></semantics></math> sentence-pair English tasks, including
8 tasks from the GLUE benchmark <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib42" title="" class="ltx_ref">2019</a>)</cite>,
SNLI <cite class="ltx_cite ltx_citemacro_cite">Bowman et al. (<a href="#bib.bib7" title="" class="ltx_ref">2015</a>)</cite>, and 6 other popular sentence classification tasks (SST-5, MR, CR, MPQA, Subj, TREC). All of the dataset details are provided in Appendix <a href="#A2" title="Appendix B Datasets ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B</span></a>. For <em id="S3.SS0.SSS0.Px2.p1.7.4" class="ltx_emph ltx_font_italic">single-sentence</em> tasks, the goal is to make a prediction based on an input sentence <math id="S3.SS0.SSS0.Px2.p1.3.m3.1" class="ltx_Math" alttext="{x}_{\mathrm{in}}=x_{1}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.3.m3.1a"><mrow id="S3.SS0.SSS0.Px2.p1.3.m3.1.1" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.cmml"><msub id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.2" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.2.cmml"><mi id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.2.2" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.2.2.cmml">x</mi><mi id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.2.3" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.2.3.cmml">in</mi></msub><mo id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.1" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.1.cmml">=</mo><msub id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.3" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.3.cmml"><mi id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.3.2" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.3.2.cmml">x</mi><mn id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.3.3" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.cmml">1</mn></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.3.m3.1b"><apply id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1"><eq id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.1"></eq><apply id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.2.1.cmml" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.2">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.2.2.cmml" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.2.2">𝑥</ci><ci id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.2.3.cmml" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.2.3">in</ci></apply><apply id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.3.1.cmml" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.3">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.3.2.cmml" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.3.2">𝑥</ci><cn type="integer" id="S3.SS0.SSS0.Px2.p1.3.m3.1.1.3.3.cmml" xref="S3.SS0.SSS0.Px2.p1.3.m3.1.1.3.3">1</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.3.m3.1c">{x}_{\mathrm{in}}=x_{1}</annotation></semantics></math>, such as whether a movie review is positive or not. For <em id="S3.SS0.SSS0.Px2.p1.7.5" class="ltx_emph ltx_font_italic">sentence-pair</em> tasks, the goal is to take a pair of input sentences <math id="S3.SS0.SSS0.Px2.p1.4.m4.2" class="ltx_Math" alttext="{x}_{\mathrm{in}}=(x_{1},x_{2})" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.4.m4.2a"><mrow id="S3.SS0.SSS0.Px2.p1.4.m4.2.2" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.cmml"><msub id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.4" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.4.cmml"><mi id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.4.2" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.4.2.cmml">x</mi><mi id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.4.3" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.4.3.cmml">in</mi></msub><mo id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.3" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.3.cmml">=</mo><mrow id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.3.cmml"><mo stretchy="false" id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2.3" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.3.cmml">(</mo><msub id="S3.SS0.SSS0.Px2.p1.4.m4.1.1.1.1.1" xref="S3.SS0.SSS0.Px2.p1.4.m4.1.1.1.1.1.cmml"><mi id="S3.SS0.SSS0.Px2.p1.4.m4.1.1.1.1.1.2" xref="S3.SS0.SSS0.Px2.p1.4.m4.1.1.1.1.1.2.cmml">x</mi><mn id="S3.SS0.SSS0.Px2.p1.4.m4.1.1.1.1.1.3" xref="S3.SS0.SSS0.Px2.p1.4.m4.1.1.1.1.1.3.cmml">1</mn></msub><mo id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2.4" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.3.cmml">,</mo><msub id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2.2" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2.2.cmml"><mi id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2.2.2" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2.2.2.cmml">x</mi><mn id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2.2.3" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2.2.3.cmml">2</mn></msub><mo stretchy="false" id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2.5" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.4.m4.2b"><apply id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.cmml" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2"><eq id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.3.cmml" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.3"></eq><apply id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.4.cmml" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.4"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.4.1.cmml" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.4">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.4.2.cmml" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.4.2">𝑥</ci><ci id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.4.3.cmml" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.4.3">in</ci></apply><interval closure="open" id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.3.cmml" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2"><apply id="S3.SS0.SSS0.Px2.p1.4.m4.1.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.4.m4.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p1.4.m4.1.1.1.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.4.m4.1.1.1.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p1.4.m4.1.1.1.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p1.4.m4.1.1.1.1.1.2">𝑥</ci><cn type="integer" id="S3.SS0.SSS0.Px2.p1.4.m4.1.1.1.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p1.4.m4.1.1.1.1.1.3">1</cn></apply><apply id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2.2.cmml" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2.2"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2.2.1.cmml" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2.2">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2.2.2.cmml" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2.2.2">𝑥</ci><cn type="integer" id="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2.2.3.cmml" xref="S3.SS0.SSS0.Px2.p1.4.m4.2.2.2.2.2.3">2</cn></apply></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.4.m4.2c">{x}_{\mathrm{in}}=(x_{1},x_{2})</annotation></semantics></math> and predict the relationship between them. We also interchangeably refer to the inputs as <span id="S3.SS0.SSS0.Px2.p1.5.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S3.SS0.SSS0.Px2.p1.5.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.5.1.m1.1a"><msub id="S3.SS0.SSS0.Px2.p1.5.1.m1.1.1" xref="S3.SS0.SSS0.Px2.p1.5.1.m1.1.1.cmml"><mi id="S3.SS0.SSS0.Px2.p1.5.1.m1.1.1.2" xref="S3.SS0.SSS0.Px2.p1.5.1.m1.1.1.2.cmml">S</mi><mn id="S3.SS0.SSS0.Px2.p1.5.1.m1.1.1.3" xref="S3.SS0.SSS0.Px2.p1.5.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.5.1.m1.1b"><apply id="S3.SS0.SSS0.Px2.p1.5.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.5.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p1.5.1.m1.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.5.1.m1.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p1.5.1.m1.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p1.5.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S3.SS0.SSS0.Px2.p1.5.1.m1.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p1.5.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.5.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> or (<span id="S3.SS0.SSS0.Px2.p1.6.2" class="ltx_text ltx_font_typewriter">&lt;<math id="S3.SS0.SSS0.Px2.p1.6.2.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.6.2.m1.1a"><msub id="S3.SS0.SSS0.Px2.p1.6.2.m1.1.1" xref="S3.SS0.SSS0.Px2.p1.6.2.m1.1.1.cmml"><mi id="S3.SS0.SSS0.Px2.p1.6.2.m1.1.1.2" xref="S3.SS0.SSS0.Px2.p1.6.2.m1.1.1.2.cmml">S</mi><mn id="S3.SS0.SSS0.Px2.p1.6.2.m1.1.1.3" xref="S3.SS0.SSS0.Px2.p1.6.2.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.6.2.m1.1b"><apply id="S3.SS0.SSS0.Px2.p1.6.2.m1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.6.2.m1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p1.6.2.m1.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.6.2.m1.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p1.6.2.m1.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p1.6.2.m1.1.1.2">𝑆</ci><cn type="integer" id="S3.SS0.SSS0.Px2.p1.6.2.m1.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p1.6.2.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.6.2.m1.1c">S_{1}</annotation></semantics></math>&gt;</span>, <span id="S3.SS0.SSS0.Px2.p1.7.3" class="ltx_text ltx_font_typewriter">&lt;<math id="S3.SS0.SSS0.Px2.p1.7.3.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="S3.SS0.SSS0.Px2.p1.7.3.m1.1a"><msub id="S3.SS0.SSS0.Px2.p1.7.3.m1.1.1" xref="S3.SS0.SSS0.Px2.p1.7.3.m1.1.1.cmml"><mi id="S3.SS0.SSS0.Px2.p1.7.3.m1.1.1.2" xref="S3.SS0.SSS0.Px2.p1.7.3.m1.1.1.2.cmml">S</mi><mn id="S3.SS0.SSS0.Px2.p1.7.3.m1.1.1.3" xref="S3.SS0.SSS0.Px2.p1.7.3.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px2.p1.7.3.m1.1b"><apply id="S3.SS0.SSS0.Px2.p1.7.3.m1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.7.3.m1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px2.p1.7.3.m1.1.1.1.cmml" xref="S3.SS0.SSS0.Px2.p1.7.3.m1.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px2.p1.7.3.m1.1.1.2.cmml" xref="S3.SS0.SSS0.Px2.p1.7.3.m1.1.1.2">𝑆</ci><cn type="integer" id="S3.SS0.SSS0.Px2.p1.7.3.m1.1.1.3.cmml" xref="S3.SS0.SSS0.Px2.p1.7.3.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px2.p1.7.3.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>).
Note that we mainly use SST-2 and SNLI for pilot experiments and model development, making it close to a true few-shot setting, at least for all the other datasets we evaluate on.
</p>
</div>
</section>
<section id="S3.SS0.SSS0.Px3" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Evaluation protocol.</h4>

<div id="S3.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="S3.SS0.SSS0.Px3.p1.3" class="ltx_p">Systematically evaluating few-shot performance can be tricky.
It is well-known that
fine-tuning on small datasets can suffer from instability <cite class="ltx_cite ltx_citemacro_cite">Dodge et al. (<a href="#bib.bib14" title="" class="ltx_ref">2020</a>); Zhang et al. (<a href="#bib.bib49" title="" class="ltx_ref">2021</a>)</cite>, and results may change dramatically given a new split of data.
To account for this, we measure average performance across 5 different randomly sampled <math id="S3.SS0.SSS0.Px3.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{D}_{\text{train}}" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.1.m1.1a"><msub id="S3.SS0.SSS0.Px3.p1.1.m1.1.1" xref="S3.SS0.SSS0.Px3.p1.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS0.SSS0.Px3.p1.1.m1.1.1.2" xref="S3.SS0.SSS0.Px3.p1.1.m1.1.1.2.cmml">𝒟</mi><mtext id="S3.SS0.SSS0.Px3.p1.1.m1.1.1.3" xref="S3.SS0.SSS0.Px3.p1.1.m1.1.1.3a.cmml">train</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.1.m1.1b"><apply id="S3.SS0.SSS0.Px3.p1.1.m1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px3.p1.1.m1.1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px3.p1.1.m1.1.1.2.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.1.1.2">𝒟</ci><ci id="S3.SS0.SSS0.Px3.p1.1.m1.1.1.3a.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.1.1.3"><mtext mathsize="70%" id="S3.SS0.SSS0.Px3.p1.1.m1.1.1.3.cmml" xref="S3.SS0.SSS0.Px3.p1.1.m1.1.1.3">train</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.1.m1.1c">\mathcal{D}_{\text{train}}</annotation></semantics></math> and <math id="S3.SS0.SSS0.Px3.p1.2.m2.1" class="ltx_Math" alttext="\mathcal{D}_{\text{dev}}" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.2.m2.1a"><msub id="S3.SS0.SSS0.Px3.p1.2.m2.1.1" xref="S3.SS0.SSS0.Px3.p1.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS0.SSS0.Px3.p1.2.m2.1.1.2" xref="S3.SS0.SSS0.Px3.p1.2.m2.1.1.2.cmml">𝒟</mi><mtext id="S3.SS0.SSS0.Px3.p1.2.m2.1.1.3" xref="S3.SS0.SSS0.Px3.p1.2.m2.1.1.3a.cmml">dev</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.2.m2.1b"><apply id="S3.SS0.SSS0.Px3.p1.2.m2.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px3.p1.2.m2.1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.2.m2.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px3.p1.2.m2.1.1.2.cmml" xref="S3.SS0.SSS0.Px3.p1.2.m2.1.1.2">𝒟</ci><ci id="S3.SS0.SSS0.Px3.p1.2.m2.1.1.3a.cmml" xref="S3.SS0.SSS0.Px3.p1.2.m2.1.1.3"><mtext mathsize="70%" id="S3.SS0.SSS0.Px3.p1.2.m2.1.1.3.cmml" xref="S3.SS0.SSS0.Px3.p1.2.m2.1.1.3">dev</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.2.m2.1c">\mathcal{D}_{\text{dev}}</annotation></semantics></math> splits.
This issue has also been discussed in
<cite class="ltx_cite ltx_citemacro_citet">Schick and Schütze (<a href="#bib.bib36" title="" class="ltx_ref">2021b</a>)</cite>—they suggest using a fixed set of training examples. We argue that sampling
multiple splits gives a
more robust measure of performance,
and a better estimate of the variance.
We also observe that hyper-parameters
can make a significant difference,
thus we sweep multiple hyper-parameters for each data sample, and take the best setting as measured on the <math id="S3.SS0.SSS0.Px3.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{D}_{\text{dev}}" display="inline"><semantics id="S3.SS0.SSS0.Px3.p1.3.m3.1a"><msub id="S3.SS0.SSS0.Px3.p1.3.m3.1.1" xref="S3.SS0.SSS0.Px3.p1.3.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S3.SS0.SSS0.Px3.p1.3.m3.1.1.2" xref="S3.SS0.SSS0.Px3.p1.3.m3.1.1.2.cmml">𝒟</mi><mtext id="S3.SS0.SSS0.Px3.p1.3.m3.1.1.3" xref="S3.SS0.SSS0.Px3.p1.3.m3.1.1.3a.cmml">dev</mtext></msub><annotation-xml encoding="MathML-Content" id="S3.SS0.SSS0.Px3.p1.3.m3.1b"><apply id="S3.SS0.SSS0.Px3.p1.3.m3.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S3.SS0.SSS0.Px3.p1.3.m3.1.1.1.cmml" xref="S3.SS0.SSS0.Px3.p1.3.m3.1.1">subscript</csymbol><ci id="S3.SS0.SSS0.Px3.p1.3.m3.1.1.2.cmml" xref="S3.SS0.SSS0.Px3.p1.3.m3.1.1.2">𝒟</ci><ci id="S3.SS0.SSS0.Px3.p1.3.m3.1.1.3a.cmml" xref="S3.SS0.SSS0.Px3.p1.3.m3.1.1.3"><mtext mathsize="70%" id="S3.SS0.SSS0.Px3.p1.3.m3.1.1.3.cmml" xref="S3.SS0.SSS0.Px3.p1.3.m3.1.1.3">dev</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS0.SSS0.Px3.p1.3.m3.1c">\mathcal{D}_{\text{dev}}</annotation></semantics></math> of that sample (see Appendix <a href="#A3.SS1" title="C.1 Hyper-parameter selection ‣ Appendix C Experimental Details ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C.1</span></a>).</p>
</div>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Prompt-based Fine-tuning</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.14" class="ltx_p">Given a masked language model
<math id="S4.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{L}" display="inline"><semantics id="S4.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S4.p1.1.m1.1.1" xref="S4.p1.1.m1.1.1.cmml">ℒ</mi><annotation-xml encoding="MathML-Content" id="S4.p1.1.m1.1b"><ci id="S4.p1.1.m1.1.1.cmml" xref="S4.p1.1.m1.1.1">ℒ</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.1.m1.1c">\mathcal{L}</annotation></semantics></math>,
we first convert input <math id="S4.p1.2.m2.1" class="ltx_Math" alttext="{x}_{\mathrm{in}}" display="inline"><semantics id="S4.p1.2.m2.1a"><msub id="S4.p1.2.m2.1.1" xref="S4.p1.2.m2.1.1.cmml"><mi id="S4.p1.2.m2.1.1.2" xref="S4.p1.2.m2.1.1.2.cmml">x</mi><mi id="S4.p1.2.m2.1.1.3" xref="S4.p1.2.m2.1.1.3.cmml">in</mi></msub><annotation-xml encoding="MathML-Content" id="S4.p1.2.m2.1b"><apply id="S4.p1.2.m2.1.1.cmml" xref="S4.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S4.p1.2.m2.1.1.1.cmml" xref="S4.p1.2.m2.1.1">subscript</csymbol><ci id="S4.p1.2.m2.1.1.2.cmml" xref="S4.p1.2.m2.1.1.2">𝑥</ci><ci id="S4.p1.2.m2.1.1.3.cmml" xref="S4.p1.2.m2.1.1.3">in</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.2.m2.1c">{x}_{\mathrm{in}}</annotation></semantics></math> to a token sequence <math id="S4.p1.3.m3.1" class="ltx_Math" alttext="\tilde{x}" display="inline"><semantics id="S4.p1.3.m3.1a"><mover accent="true" id="S4.p1.3.m3.1.1" xref="S4.p1.3.m3.1.1.cmml"><mi id="S4.p1.3.m3.1.1.2" xref="S4.p1.3.m3.1.1.2.cmml">x</mi><mo id="S4.p1.3.m3.1.1.1" xref="S4.p1.3.m3.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="S4.p1.3.m3.1b"><apply id="S4.p1.3.m3.1.1.cmml" xref="S4.p1.3.m3.1.1"><ci id="S4.p1.3.m3.1.1.1.cmml" xref="S4.p1.3.m3.1.1.1">~</ci><ci id="S4.p1.3.m3.1.1.2.cmml" xref="S4.p1.3.m3.1.1.2">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.3.m3.1c">\tilde{x}</annotation></semantics></math>, and the language model <math id="S4.p1.4.m4.1" class="ltx_Math" alttext="\mathcal{L}" display="inline"><semantics id="S4.p1.4.m4.1a"><mi class="ltx_font_mathcaligraphic" id="S4.p1.4.m4.1.1" xref="S4.p1.4.m4.1.1.cmml">ℒ</mi><annotation-xml encoding="MathML-Content" id="S4.p1.4.m4.1b"><ci id="S4.p1.4.m4.1.1.cmml" xref="S4.p1.4.m4.1.1">ℒ</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.4.m4.1c">\mathcal{L}</annotation></semantics></math> then
maps <math id="S4.p1.5.m5.1" class="ltx_Math" alttext="\tilde{x}" display="inline"><semantics id="S4.p1.5.m5.1a"><mover accent="true" id="S4.p1.5.m5.1.1" xref="S4.p1.5.m5.1.1.cmml"><mi id="S4.p1.5.m5.1.1.2" xref="S4.p1.5.m5.1.1.2.cmml">x</mi><mo id="S4.p1.5.m5.1.1.1" xref="S4.p1.5.m5.1.1.1.cmml">~</mo></mover><annotation-xml encoding="MathML-Content" id="S4.p1.5.m5.1b"><apply id="S4.p1.5.m5.1.1.cmml" xref="S4.p1.5.m5.1.1"><ci id="S4.p1.5.m5.1.1.1.cmml" xref="S4.p1.5.m5.1.1.1">~</ci><ci id="S4.p1.5.m5.1.1.2.cmml" xref="S4.p1.5.m5.1.1.2">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.5.m5.1c">\tilde{x}</annotation></semantics></math> to a sequence of hidden vectors <math id="S4.p1.6.m6.1" class="ltx_Math" alttext="\{\mathbf{h}_{k}\in\mathbb{R}^{d}\}" display="inline"><semantics id="S4.p1.6.m6.1a"><mrow id="S4.p1.6.m6.1.1.1" xref="S4.p1.6.m6.1.1.2.cmml"><mo stretchy="false" id="S4.p1.6.m6.1.1.1.2" xref="S4.p1.6.m6.1.1.2.cmml">{</mo><mrow id="S4.p1.6.m6.1.1.1.1" xref="S4.p1.6.m6.1.1.1.1.cmml"><msub id="S4.p1.6.m6.1.1.1.1.2" xref="S4.p1.6.m6.1.1.1.1.2.cmml"><mi id="S4.p1.6.m6.1.1.1.1.2.2" xref="S4.p1.6.m6.1.1.1.1.2.2.cmml">𝐡</mi><mi id="S4.p1.6.m6.1.1.1.1.2.3" xref="S4.p1.6.m6.1.1.1.1.2.3.cmml">k</mi></msub><mo id="S4.p1.6.m6.1.1.1.1.1" xref="S4.p1.6.m6.1.1.1.1.1.cmml">∈</mo><msup id="S4.p1.6.m6.1.1.1.1.3" xref="S4.p1.6.m6.1.1.1.1.3.cmml"><mi id="S4.p1.6.m6.1.1.1.1.3.2" xref="S4.p1.6.m6.1.1.1.1.3.2.cmml">ℝ</mi><mi id="S4.p1.6.m6.1.1.1.1.3.3" xref="S4.p1.6.m6.1.1.1.1.3.3.cmml">d</mi></msup></mrow><mo stretchy="false" id="S4.p1.6.m6.1.1.1.3" xref="S4.p1.6.m6.1.1.2.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.6.m6.1b"><set id="S4.p1.6.m6.1.1.2.cmml" xref="S4.p1.6.m6.1.1.1"><apply id="S4.p1.6.m6.1.1.1.1.cmml" xref="S4.p1.6.m6.1.1.1.1"><in id="S4.p1.6.m6.1.1.1.1.1.cmml" xref="S4.p1.6.m6.1.1.1.1.1"></in><apply id="S4.p1.6.m6.1.1.1.1.2.cmml" xref="S4.p1.6.m6.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.p1.6.m6.1.1.1.1.2.1.cmml" xref="S4.p1.6.m6.1.1.1.1.2">subscript</csymbol><ci id="S4.p1.6.m6.1.1.1.1.2.2.cmml" xref="S4.p1.6.m6.1.1.1.1.2.2">𝐡</ci><ci id="S4.p1.6.m6.1.1.1.1.2.3.cmml" xref="S4.p1.6.m6.1.1.1.1.2.3">𝑘</ci></apply><apply id="S4.p1.6.m6.1.1.1.1.3.cmml" xref="S4.p1.6.m6.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.p1.6.m6.1.1.1.1.3.1.cmml" xref="S4.p1.6.m6.1.1.1.1.3">superscript</csymbol><ci id="S4.p1.6.m6.1.1.1.1.3.2.cmml" xref="S4.p1.6.m6.1.1.1.1.3.2">ℝ</ci><ci id="S4.p1.6.m6.1.1.1.1.3.3.cmml" xref="S4.p1.6.m6.1.1.1.1.3.3">𝑑</ci></apply></apply></set></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.6.m6.1c">\{\mathbf{h}_{k}\in\mathbb{R}^{d}\}</annotation></semantics></math>.
During standard fine-tuning, we usually take <math id="S4.p1.7.m7.1" class="ltx_Math" alttext="\tilde{x}_{\text{single}}=\texttt{[CLS]}x_{1}\texttt{[SEP]}" display="inline"><semantics id="S4.p1.7.m7.1a"><mrow id="S4.p1.7.m7.1.1" xref="S4.p1.7.m7.1.1.cmml"><msub id="S4.p1.7.m7.1.1.2" xref="S4.p1.7.m7.1.1.2.cmml"><mover accent="true" id="S4.p1.7.m7.1.1.2.2" xref="S4.p1.7.m7.1.1.2.2.cmml"><mi id="S4.p1.7.m7.1.1.2.2.2" xref="S4.p1.7.m7.1.1.2.2.2.cmml">x</mi><mo id="S4.p1.7.m7.1.1.2.2.1" xref="S4.p1.7.m7.1.1.2.2.1.cmml">~</mo></mover><mtext id="S4.p1.7.m7.1.1.2.3" xref="S4.p1.7.m7.1.1.2.3a.cmml">single</mtext></msub><mo id="S4.p1.7.m7.1.1.1" xref="S4.p1.7.m7.1.1.1.cmml">=</mo><mrow id="S4.p1.7.m7.1.1.3" xref="S4.p1.7.m7.1.1.3.cmml"><mtext class="ltx_mathvariant_monospace" id="S4.p1.7.m7.1.1.3.2" xref="S4.p1.7.m7.1.1.3.2a.cmml">[CLS]</mtext><mo lspace="0em" rspace="0em" id="S4.p1.7.m7.1.1.3.1" xref="S4.p1.7.m7.1.1.3.1.cmml">​</mo><msub id="S4.p1.7.m7.1.1.3.3" xref="S4.p1.7.m7.1.1.3.3.cmml"><mi id="S4.p1.7.m7.1.1.3.3.2" xref="S4.p1.7.m7.1.1.3.3.2.cmml">x</mi><mn id="S4.p1.7.m7.1.1.3.3.3" xref="S4.p1.7.m7.1.1.3.3.3.cmml">1</mn></msub><mo lspace="0em" rspace="0em" id="S4.p1.7.m7.1.1.3.1a" xref="S4.p1.7.m7.1.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S4.p1.7.m7.1.1.3.4" xref="S4.p1.7.m7.1.1.3.4a.cmml">[SEP]</mtext></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.7.m7.1b"><apply id="S4.p1.7.m7.1.1.cmml" xref="S4.p1.7.m7.1.1"><eq id="S4.p1.7.m7.1.1.1.cmml" xref="S4.p1.7.m7.1.1.1"></eq><apply id="S4.p1.7.m7.1.1.2.cmml" xref="S4.p1.7.m7.1.1.2"><csymbol cd="ambiguous" id="S4.p1.7.m7.1.1.2.1.cmml" xref="S4.p1.7.m7.1.1.2">subscript</csymbol><apply id="S4.p1.7.m7.1.1.2.2.cmml" xref="S4.p1.7.m7.1.1.2.2"><ci id="S4.p1.7.m7.1.1.2.2.1.cmml" xref="S4.p1.7.m7.1.1.2.2.1">~</ci><ci id="S4.p1.7.m7.1.1.2.2.2.cmml" xref="S4.p1.7.m7.1.1.2.2.2">𝑥</ci></apply><ci id="S4.p1.7.m7.1.1.2.3a.cmml" xref="S4.p1.7.m7.1.1.2.3"><mtext mathsize="70%" id="S4.p1.7.m7.1.1.2.3.cmml" xref="S4.p1.7.m7.1.1.2.3">single</mtext></ci></apply><apply id="S4.p1.7.m7.1.1.3.cmml" xref="S4.p1.7.m7.1.1.3"><times id="S4.p1.7.m7.1.1.3.1.cmml" xref="S4.p1.7.m7.1.1.3.1"></times><ci id="S4.p1.7.m7.1.1.3.2a.cmml" xref="S4.p1.7.m7.1.1.3.2"><mtext class="ltx_mathvariant_monospace" id="S4.p1.7.m7.1.1.3.2.cmml" xref="S4.p1.7.m7.1.1.3.2">[CLS]</mtext></ci><apply id="S4.p1.7.m7.1.1.3.3.cmml" xref="S4.p1.7.m7.1.1.3.3"><csymbol cd="ambiguous" id="S4.p1.7.m7.1.1.3.3.1.cmml" xref="S4.p1.7.m7.1.1.3.3">subscript</csymbol><ci id="S4.p1.7.m7.1.1.3.3.2.cmml" xref="S4.p1.7.m7.1.1.3.3.2">𝑥</ci><cn type="integer" id="S4.p1.7.m7.1.1.3.3.3.cmml" xref="S4.p1.7.m7.1.1.3.3.3">1</cn></apply><ci id="S4.p1.7.m7.1.1.3.4a.cmml" xref="S4.p1.7.m7.1.1.3.4"><mtext class="ltx_mathvariant_monospace" id="S4.p1.7.m7.1.1.3.4.cmml" xref="S4.p1.7.m7.1.1.3.4">[SEP]</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.7.m7.1c">\tilde{x}_{\text{single}}=\texttt{[CLS]}x_{1}\texttt{[SEP]}</annotation></semantics></math> or <math id="S4.p1.8.m8.1" class="ltx_Math" alttext="\tilde{x}_{\text{pair}}=\texttt{[CLS]}x_{1}\texttt{[SEP]}x_{2}\texttt{[SEP]}" display="inline"><semantics id="S4.p1.8.m8.1a"><mrow id="S4.p1.8.m8.1.1" xref="S4.p1.8.m8.1.1.cmml"><msub id="S4.p1.8.m8.1.1.2" xref="S4.p1.8.m8.1.1.2.cmml"><mover accent="true" id="S4.p1.8.m8.1.1.2.2" xref="S4.p1.8.m8.1.1.2.2.cmml"><mi id="S4.p1.8.m8.1.1.2.2.2" xref="S4.p1.8.m8.1.1.2.2.2.cmml">x</mi><mo id="S4.p1.8.m8.1.1.2.2.1" xref="S4.p1.8.m8.1.1.2.2.1.cmml">~</mo></mover><mtext id="S4.p1.8.m8.1.1.2.3" xref="S4.p1.8.m8.1.1.2.3a.cmml">pair</mtext></msub><mo id="S4.p1.8.m8.1.1.1" xref="S4.p1.8.m8.1.1.1.cmml">=</mo><mrow id="S4.p1.8.m8.1.1.3" xref="S4.p1.8.m8.1.1.3.cmml"><mtext class="ltx_mathvariant_monospace" id="S4.p1.8.m8.1.1.3.2" xref="S4.p1.8.m8.1.1.3.2a.cmml">[CLS]</mtext><mo lspace="0em" rspace="0em" id="S4.p1.8.m8.1.1.3.1" xref="S4.p1.8.m8.1.1.3.1.cmml">​</mo><msub id="S4.p1.8.m8.1.1.3.3" xref="S4.p1.8.m8.1.1.3.3.cmml"><mi id="S4.p1.8.m8.1.1.3.3.2" xref="S4.p1.8.m8.1.1.3.3.2.cmml">x</mi><mn id="S4.p1.8.m8.1.1.3.3.3" xref="S4.p1.8.m8.1.1.3.3.3.cmml">1</mn></msub><mo lspace="0em" rspace="0em" id="S4.p1.8.m8.1.1.3.1a" xref="S4.p1.8.m8.1.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S4.p1.8.m8.1.1.3.4" xref="S4.p1.8.m8.1.1.3.4a.cmml">[SEP]</mtext><mo lspace="0em" rspace="0em" id="S4.p1.8.m8.1.1.3.1b" xref="S4.p1.8.m8.1.1.3.1.cmml">​</mo><msub id="S4.p1.8.m8.1.1.3.5" xref="S4.p1.8.m8.1.1.3.5.cmml"><mi id="S4.p1.8.m8.1.1.3.5.2" xref="S4.p1.8.m8.1.1.3.5.2.cmml">x</mi><mn id="S4.p1.8.m8.1.1.3.5.3" xref="S4.p1.8.m8.1.1.3.5.3.cmml">2</mn></msub><mo lspace="0em" rspace="0em" id="S4.p1.8.m8.1.1.3.1c" xref="S4.p1.8.m8.1.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S4.p1.8.m8.1.1.3.6" xref="S4.p1.8.m8.1.1.3.6a.cmml">[SEP]</mtext></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.8.m8.1b"><apply id="S4.p1.8.m8.1.1.cmml" xref="S4.p1.8.m8.1.1"><eq id="S4.p1.8.m8.1.1.1.cmml" xref="S4.p1.8.m8.1.1.1"></eq><apply id="S4.p1.8.m8.1.1.2.cmml" xref="S4.p1.8.m8.1.1.2"><csymbol cd="ambiguous" id="S4.p1.8.m8.1.1.2.1.cmml" xref="S4.p1.8.m8.1.1.2">subscript</csymbol><apply id="S4.p1.8.m8.1.1.2.2.cmml" xref="S4.p1.8.m8.1.1.2.2"><ci id="S4.p1.8.m8.1.1.2.2.1.cmml" xref="S4.p1.8.m8.1.1.2.2.1">~</ci><ci id="S4.p1.8.m8.1.1.2.2.2.cmml" xref="S4.p1.8.m8.1.1.2.2.2">𝑥</ci></apply><ci id="S4.p1.8.m8.1.1.2.3a.cmml" xref="S4.p1.8.m8.1.1.2.3"><mtext mathsize="70%" id="S4.p1.8.m8.1.1.2.3.cmml" xref="S4.p1.8.m8.1.1.2.3">pair</mtext></ci></apply><apply id="S4.p1.8.m8.1.1.3.cmml" xref="S4.p1.8.m8.1.1.3"><times id="S4.p1.8.m8.1.1.3.1.cmml" xref="S4.p1.8.m8.1.1.3.1"></times><ci id="S4.p1.8.m8.1.1.3.2a.cmml" xref="S4.p1.8.m8.1.1.3.2"><mtext class="ltx_mathvariant_monospace" id="S4.p1.8.m8.1.1.3.2.cmml" xref="S4.p1.8.m8.1.1.3.2">[CLS]</mtext></ci><apply id="S4.p1.8.m8.1.1.3.3.cmml" xref="S4.p1.8.m8.1.1.3.3"><csymbol cd="ambiguous" id="S4.p1.8.m8.1.1.3.3.1.cmml" xref="S4.p1.8.m8.1.1.3.3">subscript</csymbol><ci id="S4.p1.8.m8.1.1.3.3.2.cmml" xref="S4.p1.8.m8.1.1.3.3.2">𝑥</ci><cn type="integer" id="S4.p1.8.m8.1.1.3.3.3.cmml" xref="S4.p1.8.m8.1.1.3.3.3">1</cn></apply><ci id="S4.p1.8.m8.1.1.3.4a.cmml" xref="S4.p1.8.m8.1.1.3.4"><mtext class="ltx_mathvariant_monospace" id="S4.p1.8.m8.1.1.3.4.cmml" xref="S4.p1.8.m8.1.1.3.4">[SEP]</mtext></ci><apply id="S4.p1.8.m8.1.1.3.5.cmml" xref="S4.p1.8.m8.1.1.3.5"><csymbol cd="ambiguous" id="S4.p1.8.m8.1.1.3.5.1.cmml" xref="S4.p1.8.m8.1.1.3.5">subscript</csymbol><ci id="S4.p1.8.m8.1.1.3.5.2.cmml" xref="S4.p1.8.m8.1.1.3.5.2">𝑥</ci><cn type="integer" id="S4.p1.8.m8.1.1.3.5.3.cmml" xref="S4.p1.8.m8.1.1.3.5.3">2</cn></apply><ci id="S4.p1.8.m8.1.1.3.6a.cmml" xref="S4.p1.8.m8.1.1.3.6"><mtext class="ltx_mathvariant_monospace" id="S4.p1.8.m8.1.1.3.6.cmml" xref="S4.p1.8.m8.1.1.3.6">[SEP]</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.8.m8.1c">\tilde{x}_{\text{pair}}=\texttt{[CLS]}x_{1}\texttt{[SEP]}x_{2}\texttt{[SEP]}</annotation></semantics></math>.
For downstream classification tasks with a label space <math id="S4.p1.9.m9.1" class="ltx_Math" alttext="\mathcal{Y}" display="inline"><semantics id="S4.p1.9.m9.1a"><mi class="ltx_font_mathcaligraphic" id="S4.p1.9.m9.1.1" xref="S4.p1.9.m9.1.1.cmml">𝒴</mi><annotation-xml encoding="MathML-Content" id="S4.p1.9.m9.1b"><ci id="S4.p1.9.m9.1.1.cmml" xref="S4.p1.9.m9.1.1">𝒴</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.9.m9.1c">\mathcal{Y}</annotation></semantics></math>, we train a task-specific head, <math id="S4.p1.10.m10.1" class="ltx_Math" alttext="\mathrm{softmax}(\mathbf{W}_{o}\mathbf{h}_{\texttt{[CLS]}})" display="inline"><semantics id="S4.p1.10.m10.1a"><mrow id="S4.p1.10.m10.1.1" xref="S4.p1.10.m10.1.1.cmml"><mi id="S4.p1.10.m10.1.1.3" xref="S4.p1.10.m10.1.1.3.cmml">softmax</mi><mo lspace="0em" rspace="0em" id="S4.p1.10.m10.1.1.2" xref="S4.p1.10.m10.1.1.2.cmml">​</mo><mrow id="S4.p1.10.m10.1.1.1.1" xref="S4.p1.10.m10.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.p1.10.m10.1.1.1.1.2" xref="S4.p1.10.m10.1.1.1.1.1.cmml">(</mo><mrow id="S4.p1.10.m10.1.1.1.1.1" xref="S4.p1.10.m10.1.1.1.1.1.cmml"><msub id="S4.p1.10.m10.1.1.1.1.1.2" xref="S4.p1.10.m10.1.1.1.1.1.2.cmml"><mi id="S4.p1.10.m10.1.1.1.1.1.2.2" xref="S4.p1.10.m10.1.1.1.1.1.2.2.cmml">𝐖</mi><mi id="S4.p1.10.m10.1.1.1.1.1.2.3" xref="S4.p1.10.m10.1.1.1.1.1.2.3.cmml">o</mi></msub><mo lspace="0em" rspace="0em" id="S4.p1.10.m10.1.1.1.1.1.1" xref="S4.p1.10.m10.1.1.1.1.1.1.cmml">​</mo><msub id="S4.p1.10.m10.1.1.1.1.1.3" xref="S4.p1.10.m10.1.1.1.1.1.3.cmml"><mi id="S4.p1.10.m10.1.1.1.1.1.3.2" xref="S4.p1.10.m10.1.1.1.1.1.3.2.cmml">𝐡</mi><mtext class="ltx_mathvariant_monospace" id="S4.p1.10.m10.1.1.1.1.1.3.3" xref="S4.p1.10.m10.1.1.1.1.1.3.3a.cmml">[CLS]</mtext></msub></mrow><mo stretchy="false" id="S4.p1.10.m10.1.1.1.1.3" xref="S4.p1.10.m10.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.10.m10.1b"><apply id="S4.p1.10.m10.1.1.cmml" xref="S4.p1.10.m10.1.1"><times id="S4.p1.10.m10.1.1.2.cmml" xref="S4.p1.10.m10.1.1.2"></times><ci id="S4.p1.10.m10.1.1.3.cmml" xref="S4.p1.10.m10.1.1.3">softmax</ci><apply id="S4.p1.10.m10.1.1.1.1.1.cmml" xref="S4.p1.10.m10.1.1.1.1"><times id="S4.p1.10.m10.1.1.1.1.1.1.cmml" xref="S4.p1.10.m10.1.1.1.1.1.1"></times><apply id="S4.p1.10.m10.1.1.1.1.1.2.cmml" xref="S4.p1.10.m10.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.p1.10.m10.1.1.1.1.1.2.1.cmml" xref="S4.p1.10.m10.1.1.1.1.1.2">subscript</csymbol><ci id="S4.p1.10.m10.1.1.1.1.1.2.2.cmml" xref="S4.p1.10.m10.1.1.1.1.1.2.2">𝐖</ci><ci id="S4.p1.10.m10.1.1.1.1.1.2.3.cmml" xref="S4.p1.10.m10.1.1.1.1.1.2.3">𝑜</ci></apply><apply id="S4.p1.10.m10.1.1.1.1.1.3.cmml" xref="S4.p1.10.m10.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.p1.10.m10.1.1.1.1.1.3.1.cmml" xref="S4.p1.10.m10.1.1.1.1.1.3">subscript</csymbol><ci id="S4.p1.10.m10.1.1.1.1.1.3.2.cmml" xref="S4.p1.10.m10.1.1.1.1.1.3.2">𝐡</ci><ci id="S4.p1.10.m10.1.1.1.1.1.3.3a.cmml" xref="S4.p1.10.m10.1.1.1.1.1.3.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.p1.10.m10.1.1.1.1.1.3.3.cmml" xref="S4.p1.10.m10.1.1.1.1.1.3.3">[CLS]</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.10.m10.1c">\mathrm{softmax}(\mathbf{W}_{o}\mathbf{h}_{\texttt{[CLS]}})</annotation></semantics></math>, by maximizing the log-probability of the correct label, where <math id="S4.p1.11.m11.1" class="ltx_Math" alttext="\mathbf{h}_{\texttt{[CLS]}}" display="inline"><semantics id="S4.p1.11.m11.1a"><msub id="S4.p1.11.m11.1.1" xref="S4.p1.11.m11.1.1.cmml"><mi id="S4.p1.11.m11.1.1.2" xref="S4.p1.11.m11.1.1.2.cmml">𝐡</mi><mtext class="ltx_mathvariant_monospace" id="S4.p1.11.m11.1.1.3" xref="S4.p1.11.m11.1.1.3a.cmml">[CLS]</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.p1.11.m11.1b"><apply id="S4.p1.11.m11.1.1.cmml" xref="S4.p1.11.m11.1.1"><csymbol cd="ambiguous" id="S4.p1.11.m11.1.1.1.cmml" xref="S4.p1.11.m11.1.1">subscript</csymbol><ci id="S4.p1.11.m11.1.1.2.cmml" xref="S4.p1.11.m11.1.1.2">𝐡</ci><ci id="S4.p1.11.m11.1.1.3a.cmml" xref="S4.p1.11.m11.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.p1.11.m11.1.1.3.cmml" xref="S4.p1.11.m11.1.1.3">[CLS]</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.11.m11.1c">\mathbf{h}_{\texttt{[CLS]}}</annotation></semantics></math> is the hidden vector of <span id="S4.p1.14.1" class="ltx_text ltx_font_typewriter">[CLS]</span>, and <math id="S4.p1.12.m12.1" class="ltx_Math" alttext="\mathbf{W}_{o}\in\mathbb{R}^{\mathcal{|\mathcal{Y}|}\times d}" display="inline"><semantics id="S4.p1.12.m12.1a"><mrow id="S4.p1.12.m12.1.2" xref="S4.p1.12.m12.1.2.cmml"><msub id="S4.p1.12.m12.1.2.2" xref="S4.p1.12.m12.1.2.2.cmml"><mi id="S4.p1.12.m12.1.2.2.2" xref="S4.p1.12.m12.1.2.2.2.cmml">𝐖</mi><mi id="S4.p1.12.m12.1.2.2.3" xref="S4.p1.12.m12.1.2.2.3.cmml">o</mi></msub><mo id="S4.p1.12.m12.1.2.1" xref="S4.p1.12.m12.1.2.1.cmml">∈</mo><msup id="S4.p1.12.m12.1.2.3" xref="S4.p1.12.m12.1.2.3.cmml"><mi id="S4.p1.12.m12.1.2.3.2" xref="S4.p1.12.m12.1.2.3.2.cmml">ℝ</mi><mrow id="S4.p1.12.m12.1.1.1" xref="S4.p1.12.m12.1.1.1.cmml"><mrow id="S4.p1.12.m12.1.1.1.3.2" xref="S4.p1.12.m12.1.1.1.3.1.cmml"><mo stretchy="false" id="S4.p1.12.m12.1.1.1.3.2.1" xref="S4.p1.12.m12.1.1.1.3.1.1.cmml">|</mo><mi class="ltx_font_mathcaligraphic" id="S4.p1.12.m12.1.1.1.1" xref="S4.p1.12.m12.1.1.1.1.cmml">𝒴</mi><mo rspace="0.055em" stretchy="false" id="S4.p1.12.m12.1.1.1.3.2.2" xref="S4.p1.12.m12.1.1.1.3.1.1.cmml">|</mo></mrow><mo rspace="0.222em" id="S4.p1.12.m12.1.1.1.2" xref="S4.p1.12.m12.1.1.1.2.cmml">×</mo><mi id="S4.p1.12.m12.1.1.1.4" xref="S4.p1.12.m12.1.1.1.4.cmml">d</mi></mrow></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.12.m12.1b"><apply id="S4.p1.12.m12.1.2.cmml" xref="S4.p1.12.m12.1.2"><in id="S4.p1.12.m12.1.2.1.cmml" xref="S4.p1.12.m12.1.2.1"></in><apply id="S4.p1.12.m12.1.2.2.cmml" xref="S4.p1.12.m12.1.2.2"><csymbol cd="ambiguous" id="S4.p1.12.m12.1.2.2.1.cmml" xref="S4.p1.12.m12.1.2.2">subscript</csymbol><ci id="S4.p1.12.m12.1.2.2.2.cmml" xref="S4.p1.12.m12.1.2.2.2">𝐖</ci><ci id="S4.p1.12.m12.1.2.2.3.cmml" xref="S4.p1.12.m12.1.2.2.3">𝑜</ci></apply><apply id="S4.p1.12.m12.1.2.3.cmml" xref="S4.p1.12.m12.1.2.3"><csymbol cd="ambiguous" id="S4.p1.12.m12.1.2.3.1.cmml" xref="S4.p1.12.m12.1.2.3">superscript</csymbol><ci id="S4.p1.12.m12.1.2.3.2.cmml" xref="S4.p1.12.m12.1.2.3.2">ℝ</ci><apply id="S4.p1.12.m12.1.1.1.cmml" xref="S4.p1.12.m12.1.1.1"><times id="S4.p1.12.m12.1.1.1.2.cmml" xref="S4.p1.12.m12.1.1.1.2"></times><apply id="S4.p1.12.m12.1.1.1.3.1.cmml" xref="S4.p1.12.m12.1.1.1.3.2"><abs id="S4.p1.12.m12.1.1.1.3.1.1.cmml" xref="S4.p1.12.m12.1.1.1.3.2.1"></abs><ci id="S4.p1.12.m12.1.1.1.1.cmml" xref="S4.p1.12.m12.1.1.1.1">𝒴</ci></apply><ci id="S4.p1.12.m12.1.1.1.4.cmml" xref="S4.p1.12.m12.1.1.1.4">𝑑</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.12.m12.1c">\mathbf{W}_{o}\in\mathbb{R}^{\mathcal{|\mathcal{Y}|}\times d}</annotation></semantics></math> is a set of randomly initialized parameters introduced at the start of fine-tuning.
Similarly, for a regression task, we can introduce <math id="S4.p1.13.m13.1" class="ltx_Math" alttext="\mathbf{w}_{o}\in\mathbb{R}^{d}" display="inline"><semantics id="S4.p1.13.m13.1a"><mrow id="S4.p1.13.m13.1.1" xref="S4.p1.13.m13.1.1.cmml"><msub id="S4.p1.13.m13.1.1.2" xref="S4.p1.13.m13.1.1.2.cmml"><mi id="S4.p1.13.m13.1.1.2.2" xref="S4.p1.13.m13.1.1.2.2.cmml">𝐰</mi><mi id="S4.p1.13.m13.1.1.2.3" xref="S4.p1.13.m13.1.1.2.3.cmml">o</mi></msub><mo id="S4.p1.13.m13.1.1.1" xref="S4.p1.13.m13.1.1.1.cmml">∈</mo><msup id="S4.p1.13.m13.1.1.3" xref="S4.p1.13.m13.1.1.3.cmml"><mi id="S4.p1.13.m13.1.1.3.2" xref="S4.p1.13.m13.1.1.3.2.cmml">ℝ</mi><mi id="S4.p1.13.m13.1.1.3.3" xref="S4.p1.13.m13.1.1.3.3.cmml">d</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.13.m13.1b"><apply id="S4.p1.13.m13.1.1.cmml" xref="S4.p1.13.m13.1.1"><in id="S4.p1.13.m13.1.1.1.cmml" xref="S4.p1.13.m13.1.1.1"></in><apply id="S4.p1.13.m13.1.1.2.cmml" xref="S4.p1.13.m13.1.1.2"><csymbol cd="ambiguous" id="S4.p1.13.m13.1.1.2.1.cmml" xref="S4.p1.13.m13.1.1.2">subscript</csymbol><ci id="S4.p1.13.m13.1.1.2.2.cmml" xref="S4.p1.13.m13.1.1.2.2">𝐰</ci><ci id="S4.p1.13.m13.1.1.2.3.cmml" xref="S4.p1.13.m13.1.1.2.3">𝑜</ci></apply><apply id="S4.p1.13.m13.1.1.3.cmml" xref="S4.p1.13.m13.1.1.3"><csymbol cd="ambiguous" id="S4.p1.13.m13.1.1.3.1.cmml" xref="S4.p1.13.m13.1.1.3">superscript</csymbol><ci id="S4.p1.13.m13.1.1.3.2.cmml" xref="S4.p1.13.m13.1.1.3.2">ℝ</ci><ci id="S4.p1.13.m13.1.1.3.3.cmml" xref="S4.p1.13.m13.1.1.3.3">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.13.m13.1c">\mathbf{w}_{o}\in\mathbb{R}^{d}</annotation></semantics></math> and optimize the mean squared error between <math id="S4.p1.14.m14.1" class="ltx_Math" alttext="\mathbf{w}_{o}\cdot\mathbf{h}_{\texttt{[CLS]}}" display="inline"><semantics id="S4.p1.14.m14.1a"><mrow id="S4.p1.14.m14.1.1" xref="S4.p1.14.m14.1.1.cmml"><msub id="S4.p1.14.m14.1.1.2" xref="S4.p1.14.m14.1.1.2.cmml"><mi id="S4.p1.14.m14.1.1.2.2" xref="S4.p1.14.m14.1.1.2.2.cmml">𝐰</mi><mi id="S4.p1.14.m14.1.1.2.3" xref="S4.p1.14.m14.1.1.2.3.cmml">o</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S4.p1.14.m14.1.1.1" xref="S4.p1.14.m14.1.1.1.cmml">⋅</mo><msub id="S4.p1.14.m14.1.1.3" xref="S4.p1.14.m14.1.1.3.cmml"><mi id="S4.p1.14.m14.1.1.3.2" xref="S4.p1.14.m14.1.1.3.2.cmml">𝐡</mi><mtext class="ltx_mathvariant_monospace" id="S4.p1.14.m14.1.1.3.3" xref="S4.p1.14.m14.1.1.3.3a.cmml">[CLS]</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S4.p1.14.m14.1b"><apply id="S4.p1.14.m14.1.1.cmml" xref="S4.p1.14.m14.1.1"><ci id="S4.p1.14.m14.1.1.1.cmml" xref="S4.p1.14.m14.1.1.1">⋅</ci><apply id="S4.p1.14.m14.1.1.2.cmml" xref="S4.p1.14.m14.1.1.2"><csymbol cd="ambiguous" id="S4.p1.14.m14.1.1.2.1.cmml" xref="S4.p1.14.m14.1.1.2">subscript</csymbol><ci id="S4.p1.14.m14.1.1.2.2.cmml" xref="S4.p1.14.m14.1.1.2.2">𝐰</ci><ci id="S4.p1.14.m14.1.1.2.3.cmml" xref="S4.p1.14.m14.1.1.2.3">𝑜</ci></apply><apply id="S4.p1.14.m14.1.1.3.cmml" xref="S4.p1.14.m14.1.1.3"><csymbol cd="ambiguous" id="S4.p1.14.m14.1.1.3.1.cmml" xref="S4.p1.14.m14.1.1.3">subscript</csymbol><ci id="S4.p1.14.m14.1.1.3.2.cmml" xref="S4.p1.14.m14.1.1.3.2">𝐡</ci><ci id="S4.p1.14.m14.1.1.3.3a.cmml" xref="S4.p1.14.m14.1.1.3.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.p1.14.m14.1.1.3.3.cmml" xref="S4.p1.14.m14.1.1.3.3">[CLS]</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p1.14.m14.1c">\mathbf{w}_{o}\cdot\mathbf{h}_{\texttt{[CLS]}}</annotation></semantics></math> and the gold label.
In either case, the number of new parameters can be substantial—for example, a simple binary classification task will introduce 2,048 new parameters for a RoBERTa-large model—making it challenging to learn from a small amount of annotated data (e.g., 32 examples).</p>
</div>
<div id="S4.p2" class="ltx_para">
<p id="S4.p2.2" class="ltx_p">An alternative approach to solving this problem is <span id="S4.p2.2.1" class="ltx_text ltx_font_italic">prompt-based fine-tuning</span>, in which <math id="S4.p2.1.m1.1" class="ltx_Math" alttext="\mathcal{L}" display="inline"><semantics id="S4.p2.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S4.p2.1.m1.1.1" xref="S4.p2.1.m1.1.1.cmml">ℒ</mi><annotation-xml encoding="MathML-Content" id="S4.p2.1.m1.1b"><ci id="S4.p2.1.m1.1.1.cmml" xref="S4.p2.1.m1.1.1">ℒ</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.1.m1.1c">\mathcal{L}</annotation></semantics></math> is directly tasked with “auto-completing” natural language prompts. For instance, we can formulate a binary sentiment classification task using a prompt with
input <math id="S4.p2.2.m2.1" class="ltx_Math" alttext="x_{1}" display="inline"><semantics id="S4.p2.2.m2.1a"><msub id="S4.p2.2.m2.1.1" xref="S4.p2.2.m2.1.1.cmml"><mi id="S4.p2.2.m2.1.1.2" xref="S4.p2.2.m2.1.1.2.cmml">x</mi><mn id="S4.p2.2.m2.1.1.3" xref="S4.p2.2.m2.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.p2.2.m2.1b"><apply id="S4.p2.2.m2.1.1.cmml" xref="S4.p2.2.m2.1.1"><csymbol cd="ambiguous" id="S4.p2.2.m2.1.1.1.cmml" xref="S4.p2.2.m2.1.1">subscript</csymbol><ci id="S4.p2.2.m2.1.1.2.cmml" xref="S4.p2.2.m2.1.1.2">𝑥</ci><cn type="integer" id="S4.p2.2.m2.1.1.3.cmml" xref="S4.p2.2.m2.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.2.m2.1c">x_{1}</annotation></semantics></math>
(e.g., “<span id="S4.p2.2.2" class="ltx_text ltx_font_italic">No reason to watch it .</span>”) as:</p>
<table id="S4.Ex1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center">
<div id="S4.Ex1.m1.1.1.1" class="ltx_inline-block ltx_markedasmath ltx_transformed_outer" style="width:399.3pt;height:23.6pt;vertical-align:-6.4pt;"><span class="ltx_transformed_inner" style="transform:translate(112.4pt,-4.8pt) scale(2.28932202899025,2.28932202899025) ;">
<p id="S4.Ex1.m1.1.1.1.1" class="ltx_p"><math id="S4.Ex1.m1.1.1.1.1.m1.1" class="ltx_Math" alttext="x_{\mathrm{prompt}}=\text{{[CLS]}~{}$x_{1}$~{}{It was}~{}{[MASK]}~{}. {[SEP]}}" display="inline"><semantics id="S4.Ex1.m1.1.1.1.1.m1.1a"><mrow id="S4.Ex1.m1.1.1.1.1.m1.1.2" xref="S4.Ex1.m1.1.1.1.1.m1.1.2.cmml"><msub id="S4.Ex1.m1.1.1.1.1.m1.1.2.2" xref="S4.Ex1.m1.1.1.1.1.m1.1.2.2.cmml"><mi id="S4.Ex1.m1.1.1.1.1.m1.1.2.2.2" xref="S4.Ex1.m1.1.1.1.1.m1.1.2.2.2.cmml">x</mi><mi id="S4.Ex1.m1.1.1.1.1.m1.1.2.2.3" xref="S4.Ex1.m1.1.1.1.1.m1.1.2.2.3.cmml">prompt</mi></msub><mo id="S4.Ex1.m1.1.1.1.1.m1.1.2.1" xref="S4.Ex1.m1.1.1.1.1.m1.1.2.1.cmml">=</mo><mrow id="S4.Ex1.m1.1.1.1.1.m1.1.1.1" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1g.cmml"><mtext class="ltx_mathvariant_monospace" id="S4.Ex1.m1.1.1.1.1.m1.1.1.1a" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1g.cmml">[CLS]</mtext><mtext id="S4.Ex1.m1.1.1.1.1.m1.1.1.1b" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1g.cmml"> </mtext><msub id="S4.Ex1.m1.1.1.1.1.m1.1.1.1.m1.1.1" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1.m1.1.1.cmml"><mi id="S4.Ex1.m1.1.1.1.1.m1.1.1.1.m1.1.1.2" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1.m1.1.1.2.cmml">x</mi><mn id="S4.Ex1.m1.1.1.1.1.m1.1.1.1.m1.1.1.3" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1.m1.1.1.3.cmml">1</mn></msub><mtext id="S4.Ex1.m1.1.1.1.1.m1.1.1.1c" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1g.cmml"> It was </mtext><mtext class="ltx_mathvariant_monospace" id="S4.Ex1.m1.1.1.1.1.m1.1.1.1d" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1g.cmml">[MASK]</mtext><mtext id="S4.Ex1.m1.1.1.1.1.m1.1.1.1e" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1g.cmml"> . </mtext><mtext class="ltx_mathvariant_monospace" id="S4.Ex1.m1.1.1.1.1.m1.1.1.1f" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1g.cmml">[SEP]</mtext></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.Ex1.m1.1.1.1.1.m1.1b"><apply id="S4.Ex1.m1.1.1.1.1.m1.1.2.cmml" xref="S4.Ex1.m1.1.1.1.1.m1.1.2"><eq id="S4.Ex1.m1.1.1.1.1.m1.1.2.1.cmml" xref="S4.Ex1.m1.1.1.1.1.m1.1.2.1"></eq><apply id="S4.Ex1.m1.1.1.1.1.m1.1.2.2.cmml" xref="S4.Ex1.m1.1.1.1.1.m1.1.2.2"><csymbol cd="ambiguous" id="S4.Ex1.m1.1.1.1.1.m1.1.2.2.1.cmml" xref="S4.Ex1.m1.1.1.1.1.m1.1.2.2">subscript</csymbol><ci id="S4.Ex1.m1.1.1.1.1.m1.1.2.2.2.cmml" xref="S4.Ex1.m1.1.1.1.1.m1.1.2.2.2">𝑥</ci><ci id="S4.Ex1.m1.1.1.1.1.m1.1.2.2.3.cmml" xref="S4.Ex1.m1.1.1.1.1.m1.1.2.2.3">prompt</ci></apply><ci id="S4.Ex1.m1.1.1.1.1.m1.1.1.1g.cmml" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1"><mrow id="S4.Ex1.m1.1.1.1.1.m1.1.1.1.cmml" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1"><mtext class="ltx_mathvariant_monospace" id="S4.Ex1.m1.1.1.1.1.m1.1.1.1a.cmml" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1">[CLS]</mtext><mtext id="S4.Ex1.m1.1.1.1.1.m1.1.1.1b.cmml" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1"> </mtext><msub id="S4.Ex1.m1.1.1.1.1.m1.1.1.1.m1.1.1.cmml" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1.m1.1.1"><mi id="S4.Ex1.m1.1.1.1.1.m1.1.1.1.m1.1.1.2.cmml" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1.m1.1.1.2">x</mi><mn id="S4.Ex1.m1.1.1.1.1.m1.1.1.1.m1.1.1.3.cmml" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1.m1.1.1.3">1</mn></msub><mtext id="S4.Ex1.m1.1.1.1.1.m1.1.1.1c.cmml" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1"> It was </mtext><mtext class="ltx_mathvariant_monospace" id="S4.Ex1.m1.1.1.1.1.m1.1.1.1d.cmml" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1">[MASK]</mtext><mtext id="S4.Ex1.m1.1.1.1.1.m1.1.1.1e.cmml" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1"> . </mtext><mtext class="ltx_mathvariant_monospace" id="S4.Ex1.m1.1.1.1.1.m1.1.1.1f.cmml" xref="S4.Ex1.m1.1.1.1.1.m1.1.1.1">[SEP]</mtext></mrow></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.Ex1.m1.1.1.1.1.m1.1c">x_{\mathrm{prompt}}=\text{{[CLS]}~{}$x_{1}$~{}{It was}~{}{[MASK]}~{}. {[SEP]}}</annotation></semantics></math></p>
</span></div>
</td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S4.p2.3" class="ltx_p">and let <math id="S4.p2.3.m1.1" class="ltx_Math" alttext="\mathcal{L}" display="inline"><semantics id="S4.p2.3.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S4.p2.3.m1.1.1" xref="S4.p2.3.m1.1.1.cmml">ℒ</mi><annotation-xml encoding="MathML-Content" id="S4.p2.3.m1.1b"><ci id="S4.p2.3.m1.1.1.cmml" xref="S4.p2.3.m1.1.1">ℒ</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.p2.3.m1.1c">\mathcal{L}</annotation></semantics></math> decide whether it is more appropriate to fill in “<em id="S4.p2.3.1" class="ltx_emph ltx_font_italic">great</em>” (positive) or “<em id="S4.p2.3.2" class="ltx_emph ltx_font_italic">terrible</em>” (negative) for <span id="S4.p2.3.3" class="ltx_text ltx_font_typewriter">[MASK]</span>.
We now formalize this approach for classification and regression (§<a href="#S4.SS1" title="4.1 Classification ‣ 4 Prompt-based Fine-tuning ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1</span></a> and §<a href="#S4.SS2" title="4.2 Regression ‣ 4 Prompt-based Fine-tuning ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2</span></a>), and discuss the importance of prompt selection (§<a href="#S4.SS3" title="4.3 Manual prompts: the good and the bad ‣ 4 Prompt-based Fine-tuning ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.3</span></a>).</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1 </span>Classification</h3>

<div id="S4.SS1.p1" class="ltx_para">
<p id="S4.SS1.p1.6" class="ltx_p">Let <math id="S4.SS1.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{M}\colon\mathcal{Y}\rightarrow\mathcal{V}" display="inline"><semantics id="S4.SS1.p1.1.m1.1a"><mrow id="S4.SS1.p1.1.m1.1.1" xref="S4.SS1.p1.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS1.p1.1.m1.1.1.2" xref="S4.SS1.p1.1.m1.1.1.2.cmml">ℳ</mi><mo lspace="0.278em" rspace="0.278em" id="S4.SS1.p1.1.m1.1.1.1" xref="S4.SS1.p1.1.m1.1.1.1.cmml">:</mo><mrow id="S4.SS1.p1.1.m1.1.1.3" xref="S4.SS1.p1.1.m1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS1.p1.1.m1.1.1.3.2" xref="S4.SS1.p1.1.m1.1.1.3.2.cmml">𝒴</mi><mo stretchy="false" id="S4.SS1.p1.1.m1.1.1.3.1" xref="S4.SS1.p1.1.m1.1.1.3.1.cmml">→</mo><mi class="ltx_font_mathcaligraphic" id="S4.SS1.p1.1.m1.1.1.3.3" xref="S4.SS1.p1.1.m1.1.1.3.3.cmml">𝒱</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.1.m1.1b"><apply id="S4.SS1.p1.1.m1.1.1.cmml" xref="S4.SS1.p1.1.m1.1.1"><ci id="S4.SS1.p1.1.m1.1.1.1.cmml" xref="S4.SS1.p1.1.m1.1.1.1">:</ci><ci id="S4.SS1.p1.1.m1.1.1.2.cmml" xref="S4.SS1.p1.1.m1.1.1.2">ℳ</ci><apply id="S4.SS1.p1.1.m1.1.1.3.cmml" xref="S4.SS1.p1.1.m1.1.1.3"><ci id="S4.SS1.p1.1.m1.1.1.3.1.cmml" xref="S4.SS1.p1.1.m1.1.1.3.1">→</ci><ci id="S4.SS1.p1.1.m1.1.1.3.2.cmml" xref="S4.SS1.p1.1.m1.1.1.3.2">𝒴</ci><ci id="S4.SS1.p1.1.m1.1.1.3.3.cmml" xref="S4.SS1.p1.1.m1.1.1.3.3">𝒱</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.1.m1.1c">\mathcal{M}\colon\mathcal{Y}\rightarrow\mathcal{V}</annotation></semantics></math> be a mapping from the task label space to individual words<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span>More generally, we can consider a one-to-many mapping <math id="footnote5.m1.1" class="ltx_Math" alttext="\mathcal{M}\colon\mathcal{Y}\rightarrow 2^{|\mathcal{Y}|}" display="inline"><semantics id="footnote5.m1.1b"><mrow id="footnote5.m1.1.2" xref="footnote5.m1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="footnote5.m1.1.2.2" xref="footnote5.m1.1.2.2.cmml">ℳ</mi><mo lspace="0.278em" rspace="0.278em" id="footnote5.m1.1.2.1" xref="footnote5.m1.1.2.1.cmml">:</mo><mrow id="footnote5.m1.1.2.3" xref="footnote5.m1.1.2.3.cmml"><mi class="ltx_font_mathcaligraphic" id="footnote5.m1.1.2.3.2" xref="footnote5.m1.1.2.3.2.cmml">𝒴</mi><mo stretchy="false" id="footnote5.m1.1.2.3.1" xref="footnote5.m1.1.2.3.1.cmml">→</mo><msup id="footnote5.m1.1.2.3.3" xref="footnote5.m1.1.2.3.3.cmml"><mn id="footnote5.m1.1.2.3.3.2" xref="footnote5.m1.1.2.3.3.2.cmml">2</mn><mrow id="footnote5.m1.1.1.1.3" xref="footnote5.m1.1.1.1.2.cmml"><mo stretchy="false" id="footnote5.m1.1.1.1.3.1" xref="footnote5.m1.1.1.1.2.1.cmml">|</mo><mi class="ltx_font_mathcaligraphic" id="footnote5.m1.1.1.1.1" xref="footnote5.m1.1.1.1.1.cmml">𝒴</mi><mo stretchy="false" id="footnote5.m1.1.1.1.3.2" xref="footnote5.m1.1.1.1.2.1.cmml">|</mo></mrow></msup></mrow></mrow><annotation-xml encoding="MathML-Content" id="footnote5.m1.1c"><apply id="footnote5.m1.1.2.cmml" xref="footnote5.m1.1.2"><ci id="footnote5.m1.1.2.1.cmml" xref="footnote5.m1.1.2.1">:</ci><ci id="footnote5.m1.1.2.2.cmml" xref="footnote5.m1.1.2.2">ℳ</ci><apply id="footnote5.m1.1.2.3.cmml" xref="footnote5.m1.1.2.3"><ci id="footnote5.m1.1.2.3.1.cmml" xref="footnote5.m1.1.2.3.1">→</ci><ci id="footnote5.m1.1.2.3.2.cmml" xref="footnote5.m1.1.2.3.2">𝒴</ci><apply id="footnote5.m1.1.2.3.3.cmml" xref="footnote5.m1.1.2.3.3"><csymbol cd="ambiguous" id="footnote5.m1.1.2.3.3.1.cmml" xref="footnote5.m1.1.2.3.3">superscript</csymbol><cn type="integer" id="footnote5.m1.1.2.3.3.2.cmml" xref="footnote5.m1.1.2.3.3.2">2</cn><apply id="footnote5.m1.1.1.1.2.cmml" xref="footnote5.m1.1.1.1.3"><abs id="footnote5.m1.1.1.1.2.1.cmml" xref="footnote5.m1.1.1.1.3.1"></abs><ci id="footnote5.m1.1.1.1.1.cmml" xref="footnote5.m1.1.1.1.1">𝒴</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="footnote5.m1.1d">\mathcal{M}\colon\mathcal{Y}\rightarrow 2^{|\mathcal{Y}|}</annotation></semantics></math> in which we map labels to sets of words. However, we did not find significant gains in our experiments.</span></span></span>
in the vocabulary <math id="S4.SS1.p1.2.m2.1" class="ltx_Math" alttext="\mathcal{V}" display="inline"><semantics id="S4.SS1.p1.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S4.SS1.p1.2.m2.1.1" xref="S4.SS1.p1.2.m2.1.1.cmml">𝒱</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.2.m2.1b"><ci id="S4.SS1.p1.2.m2.1.1.cmml" xref="S4.SS1.p1.2.m2.1.1">𝒱</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.2.m2.1c">\mathcal{V}</annotation></semantics></math> of <math id="S4.SS1.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{L}" display="inline"><semantics id="S4.SS1.p1.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="S4.SS1.p1.3.m3.1.1" xref="S4.SS1.p1.3.m3.1.1.cmml">ℒ</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.3.m3.1b"><ci id="S4.SS1.p1.3.m3.1.1.cmml" xref="S4.SS1.p1.3.m3.1.1">ℒ</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.3.m3.1c">\mathcal{L}</annotation></semantics></math>.
Then for each <math id="S4.SS1.p1.4.m4.1" class="ltx_Math" alttext="{x}_{\mathrm{in}}" display="inline"><semantics id="S4.SS1.p1.4.m4.1a"><msub id="S4.SS1.p1.4.m4.1.1" xref="S4.SS1.p1.4.m4.1.1.cmml"><mi id="S4.SS1.p1.4.m4.1.1.2" xref="S4.SS1.p1.4.m4.1.1.2.cmml">x</mi><mi id="S4.SS1.p1.4.m4.1.1.3" xref="S4.SS1.p1.4.m4.1.1.3.cmml">in</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.4.m4.1b"><apply id="S4.SS1.p1.4.m4.1.1.cmml" xref="S4.SS1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S4.SS1.p1.4.m4.1.1.1.cmml" xref="S4.SS1.p1.4.m4.1.1">subscript</csymbol><ci id="S4.SS1.p1.4.m4.1.1.2.cmml" xref="S4.SS1.p1.4.m4.1.1.2">𝑥</ci><ci id="S4.SS1.p1.4.m4.1.1.3.cmml" xref="S4.SS1.p1.4.m4.1.1.3">in</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.4.m4.1c">{x}_{\mathrm{in}}</annotation></semantics></math>, let the manipulation <math id="S4.SS1.p1.5.m5.1" class="ltx_Math" alttext="{x}_{\mathrm{prompt}}=\mathcal{T}({x}_{\mathrm{in}})" display="inline"><semantics id="S4.SS1.p1.5.m5.1a"><mrow id="S4.SS1.p1.5.m5.1.1" xref="S4.SS1.p1.5.m5.1.1.cmml"><msub id="S4.SS1.p1.5.m5.1.1.3" xref="S4.SS1.p1.5.m5.1.1.3.cmml"><mi id="S4.SS1.p1.5.m5.1.1.3.2" xref="S4.SS1.p1.5.m5.1.1.3.2.cmml">x</mi><mi id="S4.SS1.p1.5.m5.1.1.3.3" xref="S4.SS1.p1.5.m5.1.1.3.3.cmml">prompt</mi></msub><mo id="S4.SS1.p1.5.m5.1.1.2" xref="S4.SS1.p1.5.m5.1.1.2.cmml">=</mo><mrow id="S4.SS1.p1.5.m5.1.1.1" xref="S4.SS1.p1.5.m5.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS1.p1.5.m5.1.1.1.3" xref="S4.SS1.p1.5.m5.1.1.1.3.cmml">𝒯</mi><mo lspace="0em" rspace="0em" id="S4.SS1.p1.5.m5.1.1.1.2" xref="S4.SS1.p1.5.m5.1.1.1.2.cmml">​</mo><mrow id="S4.SS1.p1.5.m5.1.1.1.1.1" xref="S4.SS1.p1.5.m5.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS1.p1.5.m5.1.1.1.1.1.2" xref="S4.SS1.p1.5.m5.1.1.1.1.1.1.cmml">(</mo><msub id="S4.SS1.p1.5.m5.1.1.1.1.1.1" xref="S4.SS1.p1.5.m5.1.1.1.1.1.1.cmml"><mi id="S4.SS1.p1.5.m5.1.1.1.1.1.1.2" xref="S4.SS1.p1.5.m5.1.1.1.1.1.1.2.cmml">x</mi><mi id="S4.SS1.p1.5.m5.1.1.1.1.1.1.3" xref="S4.SS1.p1.5.m5.1.1.1.1.1.1.3.cmml">in</mi></msub><mo stretchy="false" id="S4.SS1.p1.5.m5.1.1.1.1.1.3" xref="S4.SS1.p1.5.m5.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.5.m5.1b"><apply id="S4.SS1.p1.5.m5.1.1.cmml" xref="S4.SS1.p1.5.m5.1.1"><eq id="S4.SS1.p1.5.m5.1.1.2.cmml" xref="S4.SS1.p1.5.m5.1.1.2"></eq><apply id="S4.SS1.p1.5.m5.1.1.3.cmml" xref="S4.SS1.p1.5.m5.1.1.3"><csymbol cd="ambiguous" id="S4.SS1.p1.5.m5.1.1.3.1.cmml" xref="S4.SS1.p1.5.m5.1.1.3">subscript</csymbol><ci id="S4.SS1.p1.5.m5.1.1.3.2.cmml" xref="S4.SS1.p1.5.m5.1.1.3.2">𝑥</ci><ci id="S4.SS1.p1.5.m5.1.1.3.3.cmml" xref="S4.SS1.p1.5.m5.1.1.3.3">prompt</ci></apply><apply id="S4.SS1.p1.5.m5.1.1.1.cmml" xref="S4.SS1.p1.5.m5.1.1.1"><times id="S4.SS1.p1.5.m5.1.1.1.2.cmml" xref="S4.SS1.p1.5.m5.1.1.1.2"></times><ci id="S4.SS1.p1.5.m5.1.1.1.3.cmml" xref="S4.SS1.p1.5.m5.1.1.1.3">𝒯</ci><apply id="S4.SS1.p1.5.m5.1.1.1.1.1.1.cmml" xref="S4.SS1.p1.5.m5.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.SS1.p1.5.m5.1.1.1.1.1.1.1.cmml" xref="S4.SS1.p1.5.m5.1.1.1.1.1">subscript</csymbol><ci id="S4.SS1.p1.5.m5.1.1.1.1.1.1.2.cmml" xref="S4.SS1.p1.5.m5.1.1.1.1.1.1.2">𝑥</ci><ci id="S4.SS1.p1.5.m5.1.1.1.1.1.1.3.cmml" xref="S4.SS1.p1.5.m5.1.1.1.1.1.1.3">in</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.5.m5.1c">{x}_{\mathrm{prompt}}=\mathcal{T}({x}_{\mathrm{in}})</annotation></semantics></math>
be a <em id="S4.SS1.p1.6.1" class="ltx_emph ltx_font_italic">masked language modeling</em> (MLM) input which contains one <span id="S4.SS1.p1.6.2" class="ltx_text ltx_font_typewriter">[MASK]</span> token.
In this way, we can treat our task as an
MLM, and model the probability of predicting class <math id="S4.SS1.p1.6.m6.1" class="ltx_Math" alttext="y\in\mathcal{Y}" display="inline"><semantics id="S4.SS1.p1.6.m6.1a"><mrow id="S4.SS1.p1.6.m6.1.1" xref="S4.SS1.p1.6.m6.1.1.cmml"><mi id="S4.SS1.p1.6.m6.1.1.2" xref="S4.SS1.p1.6.m6.1.1.2.cmml">y</mi><mo id="S4.SS1.p1.6.m6.1.1.1" xref="S4.SS1.p1.6.m6.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S4.SS1.p1.6.m6.1.1.3" xref="S4.SS1.p1.6.m6.1.1.3.cmml">𝒴</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.6.m6.1b"><apply id="S4.SS1.p1.6.m6.1.1.cmml" xref="S4.SS1.p1.6.m6.1.1"><in id="S4.SS1.p1.6.m6.1.1.1.cmml" xref="S4.SS1.p1.6.m6.1.1.1"></in><ci id="S4.SS1.p1.6.m6.1.1.2.cmml" xref="S4.SS1.p1.6.m6.1.1.2">𝑦</ci><ci id="S4.SS1.p1.6.m6.1.1.3.cmml" xref="S4.SS1.p1.6.m6.1.1.3">𝒴</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.6.m6.1c">y\in\mathcal{Y}</annotation></semantics></math> as:</p>
<table id="S4.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center">
<div id="S4.E1.m1.1.1.1" class="ltx_inline-block ltx_markedasmath ltx_transformed_outer" style="width:399.3pt;height:105.2pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(126.0pt,-33.2pt) scale(2.71180568725192,2.71180568725192) ;">
<p id="S4.E1.m1.1.1.1.1" class="ltx_p"><math id="S4.E1.m1.1.1.1.1.m1.10" class="ltx_Math" alttext="\begin{aligned} p(y\mid{x}_{\mathrm{in}})&amp;=p\left(\texttt{[MASK]}=\mathcal{M}(y)\mid x_{\mathrm{prompt}}\right)\\
&amp;=\frac{\exp\left(\mathbf{w}_{\mathcal{M}(y)}\cdot\mathbf{h}_{\texttt{[MASK]}}\right)}{\sum_{y^{\prime}\in\mathcal{Y}}{\exp\left(\mathbf{w}_{\mathcal{M}(y^{\prime})}\cdot\mathbf{h}_{\texttt{[MASK]}}\right)}},\end{aligned}" display="inline"><semantics id="S4.E1.m1.1.1.1.1.m1.10a"><mtable columnspacing="0pt" rowspacing="0pt" id="S4.E1.m1.1.1.1.1.m1.10.10" xref="S4.E1.m1.1.1.1.1.m1.10.10.cmml"><mtr id="S4.E1.m1.1.1.1.1.m1.10.10a" xref="S4.E1.m1.1.1.1.1.m1.10.10.cmml"><mtd class="ltx_align_right" columnalign="right" id="S4.E1.m1.1.1.1.1.m1.10.10b" xref="S4.E1.m1.1.1.1.1.m1.10.10.cmml"><mrow id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.cmml"><mi id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.3" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.2" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.2.cmml">​</mo><mrow id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.cmml"><mi id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.2" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.2.cmml">y</mi><mo id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.cmml">∣</mo><msub id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.2" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.2.cmml">x</mi><mi id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.3" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.3.cmml">in</mi></msub></mrow><mo stretchy="false" id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.3" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E1.m1.1.1.1.1.m1.10.10c" xref="S4.E1.m1.1.1.1.1.m1.10.10.cmml"><mrow id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.cmml"><mi id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.4" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.4.cmml"></mi><mo id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.3" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.3.cmml">=</mo><mrow id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.cmml"><mi id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.3" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.2" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.2.cmml">​</mo><mrow id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.cmml"><mo id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.2" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.cmml">(</mo><mrow id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.2" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.2a.cmml">[MASK]</mtext><mo id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.1" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.1.cmml">=</mo><mrow id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.cmml"><mrow id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.2" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.2.2" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.2.2.cmml">ℳ</mi><mo lspace="0em" rspace="0em" id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.2.1" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.2.1.cmml">​</mo><mrow id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.2.3.2" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.2.cmml"><mo stretchy="false" id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.2.3.2.1" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.2.cmml">(</mo><mi id="S4.E1.m1.1.1.1.1.m1.2.2.2.2.1.1" xref="S4.E1.m1.1.1.1.1.m1.2.2.2.2.1.1.cmml">y</mi><mo stretchy="false" id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.2.3.2.2" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.2.cmml">)</mo></mrow></mrow><mo id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.1" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.1.cmml">∣</mo><msub id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.3" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.3.cmml"><mi id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.3.2" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.3.2.cmml">x</mi><mi id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.3.3" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.3.3.cmml">prompt</mi></msub></mrow></mrow><mo id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.3" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.cmml">)</mo></mrow></mrow></mrow></mtd></mtr><mtr id="S4.E1.m1.1.1.1.1.m1.10.10d" xref="S4.E1.m1.1.1.1.1.m1.10.10.cmml"><mtd id="S4.E1.m1.1.1.1.1.m1.10.10e" xref="S4.E1.m1.1.1.1.1.m1.10.10.cmml"></mtd><mtd class="ltx_align_left" columnalign="left" id="S4.E1.m1.1.1.1.1.m1.10.10f" xref="S4.E1.m1.1.1.1.1.m1.10.10.cmml"><mrow id="S4.E1.m1.1.1.1.1.m1.10.10.10.7.7.7" xref="S4.E1.m1.1.1.1.1.m1.10.10.10.7.7.7.1.cmml"><mrow id="S4.E1.m1.1.1.1.1.m1.10.10.10.7.7.7.1" xref="S4.E1.m1.1.1.1.1.m1.10.10.10.7.7.7.1.cmml"><mi id="S4.E1.m1.1.1.1.1.m1.10.10.10.7.7.7.1.2" xref="S4.E1.m1.1.1.1.1.m1.10.10.10.7.7.7.1.2.cmml"></mi><mo id="S4.E1.m1.1.1.1.1.m1.10.10.10.7.7.7.1.1" xref="S4.E1.m1.1.1.1.1.m1.10.10.10.7.7.7.1.1.cmml">=</mo><mstyle displaystyle="true" id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.cmml"><mfrac id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6a" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.cmml"><mrow id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.4.cmml"><mi id="S4.E1.m1.1.1.1.1.m1.5.5.5.2.2.2.2.2" xref="S4.E1.m1.1.1.1.1.m1.5.5.5.2.2.2.2.2.cmml">exp</mi><mo id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3a" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.4.cmml">⁡</mo><mrow id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.4.cmml"><mo id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.2" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.4.cmml">(</mo><mrow id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.cmml"><msub id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.2" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.2.cmml"><mi id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.2.2" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.2.2.cmml">𝐰</mi><mrow id="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1" xref="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.3" xref="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.3.cmml">ℳ</mi><mo lspace="0em" rspace="0em" id="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.2" xref="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.4.2" xref="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.4.2.1" xref="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.cmml">(</mo><mi id="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.1" xref="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.1.cmml">y</mi><mo stretchy="false" id="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.4.2.2" xref="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></msub><mo lspace="0.222em" rspace="0.222em" id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.1" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.1.cmml">⋅</mo><msub id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.3" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.3.cmml"><mi id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.3.2" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.3.2.cmml">𝐡</mi><mtext class="ltx_mathvariant_monospace" id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.3.3" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.3.3a.cmml">[MASK]</mtext></msub></mrow><mo id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.3" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.4.cmml">)</mo></mrow></mrow><mrow id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.cmml"><msub id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.cmml"><mo id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.2" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.2.cmml">∑</mo><mrow id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.cmml"><msup id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.2" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.2.cmml"><mi id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.2.2" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.2.2.cmml">y</mi><mo id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.2.3" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.2.3.cmml">′</mo></msup><mo id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.1" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.3" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.3.cmml">𝒴</mi></mrow></msub><mrow id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.2.cmml"><mi id="S4.E1.m1.1.1.1.1.m1.8.8.8.5.5.5.5.2" xref="S4.E1.m1.1.1.1.1.m1.8.8.8.5.5.5.5.2.cmml">exp</mi><mo id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1a" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.2.cmml">⁡</mo><mrow id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.2.cmml"><mo id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.2" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.2.cmml">(</mo><mrow id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.cmml"><msub id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.2" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.2.cmml"><mi id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.2.2" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.2.2.cmml">𝐰</mi><mrow id="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1" xref="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.3" xref="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.3.cmml">ℳ</mi><mo lspace="0em" rspace="0em" id="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.2" xref="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.2.cmml">​</mo><mrow id="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1" xref="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1.2" xref="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1.1.cmml">(</mo><msup id="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1.1" xref="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1.1.cmml"><mi id="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1.1.2" xref="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1.1.2.cmml">y</mi><mo id="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1.1.3" xref="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1.1.3.cmml">′</mo></msup><mo stretchy="false" id="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1.3" xref="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1.1.cmml">)</mo></mrow></mrow></msub><mo lspace="0.222em" rspace="0.222em" id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.1" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.1.cmml">⋅</mo><msub id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.3" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.3.cmml"><mi id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.3.2" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.3.2.cmml">𝐡</mi><mtext class="ltx_mathvariant_monospace" id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.3.3" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.3.3a.cmml">[MASK]</mtext></msub></mrow><mo id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.3" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.2.cmml">)</mo></mrow></mrow></mrow></mfrac></mstyle></mrow><mo id="S4.E1.m1.1.1.1.1.m1.10.10.10.7.7.7.2" xref="S4.E1.m1.1.1.1.1.m1.10.10.10.7.7.7.1.cmml">,</mo></mrow></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="S4.E1.m1.1.1.1.1.m1.10b"><matrix id="S4.E1.m1.1.1.1.1.m1.10.10.cmml" xref="S4.E1.m1.1.1.1.1.m1.10.10"><matrixrow id="S4.E1.m1.1.1.1.1.m1.10.10a.cmml" xref="S4.E1.m1.1.1.1.1.m1.10.10"><apply id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1"><times id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.2"></times><ci id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.3.cmml" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.3">𝑝</ci><apply id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1">conditional</csymbol><ci id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.2">𝑦</ci><apply id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.2">𝑥</ci><ci id="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S4.E1.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.3">in</ci></apply></apply></apply><apply id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2"><eq id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.3.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.3"></eq><csymbol cd="latexml" id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.4.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.4">absent</csymbol><apply id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2"><times id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.2"></times><ci id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.3.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.3">𝑝</ci><apply id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1"><eq id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.1"></eq><ci id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.2a.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.2"><mtext class="ltx_mathvariant_monospace" id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.2">[MASK]</mtext></ci><apply id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3"><csymbol cd="latexml" id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.1">conditional</csymbol><apply id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.2"><times id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.2.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.2.1"></times><ci id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.2.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.2.2">ℳ</ci><ci id="S4.E1.m1.1.1.1.1.m1.2.2.2.2.1.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.2.2.2.2.1.1">𝑦</ci></apply><apply id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.3.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.3"><csymbol cd="ambiguous" id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.3.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.3">subscript</csymbol><ci id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.3.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.3.2">𝑥</ci><ci id="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.3.3.cmml" xref="S4.E1.m1.1.1.1.1.m1.3.3.3.3.2.2.1.1.1.3.3.3">prompt</ci></apply></apply></apply></apply></apply></matrixrow><matrixrow id="S4.E1.m1.1.1.1.1.m1.10.10b.cmml" xref="S4.E1.m1.1.1.1.1.m1.10.10"><cerror id="S4.E1.m1.1.1.1.1.m1.10.10c.cmml" xref="S4.E1.m1.1.1.1.1.m1.10.10"><csymbol cd="ambiguous" id="S4.E1.m1.1.1.1.1.m1.10.10d.cmml" xref="S4.E1.m1.1.1.1.1.m1.10.10">missing-subexpression</csymbol></cerror><apply id="S4.E1.m1.1.1.1.1.m1.10.10.10.7.7.7.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.10.10.10.7.7.7"><eq id="S4.E1.m1.1.1.1.1.m1.10.10.10.7.7.7.1.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.10.10.10.7.7.7.1.1"></eq><csymbol cd="latexml" id="S4.E1.m1.1.1.1.1.m1.10.10.10.7.7.7.1.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.10.10.10.7.7.7.1.2">absent</csymbol><apply id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6"><divide id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.7.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6"></divide><apply id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.4.cmml" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3"><exp id="S4.E1.m1.1.1.1.1.m1.5.5.5.2.2.2.2.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.5.5.5.2.2.2.2.2"></exp><apply id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1"><ci id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.1">⋅</ci><apply id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.2"><csymbol cd="ambiguous" id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.2.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.2">subscript</csymbol><ci id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.2.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.2.2">𝐰</ci><apply id="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1"><times id="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.2"></times><ci id="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.3.cmml" xref="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.3">ℳ</ci><ci id="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.4.4.4.1.1.1.1.1.1.1">𝑦</ci></apply></apply><apply id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.3.cmml" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.3"><csymbol cd="ambiguous" id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.3.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.3">subscript</csymbol><ci id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.3.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.3.2">𝐡</ci><ci id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.3.3a.cmml" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.3.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.3.3.cmml" xref="S4.E1.m1.1.1.1.1.m1.6.6.6.3.3.3.3.3.1.1.3.3">[MASK]</mtext></ci></apply></apply></apply><apply id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6"><apply id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4"><csymbol cd="ambiguous" id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4">subscript</csymbol><sum id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.2"></sum><apply id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3"><in id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.1"></in><apply id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.2"><csymbol cd="ambiguous" id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.2.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.2">superscript</csymbol><ci id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.2.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.2.2">𝑦</ci><ci id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.2.3.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.2.3">′</ci></apply><ci id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.3.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.4.3.3">𝒴</ci></apply></apply><apply id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1"><exp id="S4.E1.m1.1.1.1.1.m1.8.8.8.5.5.5.5.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.8.8.8.5.5.5.5.2"></exp><apply id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1"><ci id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.1">⋅</ci><apply id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.2"><csymbol cd="ambiguous" id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.2.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.2">subscript</csymbol><ci id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.2.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.2.2">𝐰</ci><apply id="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1"><times id="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.2"></times><ci id="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.3.cmml" xref="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.3">ℳ</ci><apply id="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1"><csymbol cd="ambiguous" id="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1.1.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1">superscript</csymbol><ci id="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1.1.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1.1.2">𝑦</ci><ci id="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1.1.3.cmml" xref="S4.E1.m1.1.1.1.1.m1.7.7.7.4.4.4.4.1.1.1.1.1.3">′</ci></apply></apply></apply><apply id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.3.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.3"><csymbol cd="ambiguous" id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.3.1.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.3">subscript</csymbol><ci id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.3.2.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.3.2">𝐡</ci><ci id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.3.3a.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.3.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.3.3.cmml" xref="S4.E1.m1.1.1.1.1.m1.9.9.9.6.6.6.6.3.1.1.1.3.3">[MASK]</mtext></ci></apply></apply></apply></apply></apply></apply></matrixrow></matrix></annotation-xml><annotation encoding="application/x-tex" id="S4.E1.m1.1.1.1.1.m1.10c">\begin{aligned} p(y\mid{x}_{\mathrm{in}})&amp;=p\left(\texttt{[MASK]}=\mathcal{M}(y)\mid x_{\mathrm{prompt}}\right)\\
&amp;=\frac{\exp\left(\mathbf{w}_{\mathcal{M}(y)}\cdot\mathbf{h}_{\texttt{[MASK]}}\right)}{\sum_{y^{\prime}\in\mathcal{Y}}{\exp\left(\mathbf{w}_{\mathcal{M}(y^{\prime})}\cdot\mathbf{h}_{\texttt{[MASK]}}\right)}},\end{aligned}</annotation></semantics></math></p>
</span></div>
</td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="0" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="S4.SS1.p1.12" class="ltx_p">where <math id="S4.SS1.p1.7.m1.1" class="ltx_Math" alttext="\mathbf{h}_{\texttt{[MASK]}}" display="inline"><semantics id="S4.SS1.p1.7.m1.1a"><msub id="S4.SS1.p1.7.m1.1.1" xref="S4.SS1.p1.7.m1.1.1.cmml"><mi id="S4.SS1.p1.7.m1.1.1.2" xref="S4.SS1.p1.7.m1.1.1.2.cmml">𝐡</mi><mtext class="ltx_mathvariant_monospace" id="S4.SS1.p1.7.m1.1.1.3" xref="S4.SS1.p1.7.m1.1.1.3a.cmml">[MASK]</mtext></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.7.m1.1b"><apply id="S4.SS1.p1.7.m1.1.1.cmml" xref="S4.SS1.p1.7.m1.1.1"><csymbol cd="ambiguous" id="S4.SS1.p1.7.m1.1.1.1.cmml" xref="S4.SS1.p1.7.m1.1.1">subscript</csymbol><ci id="S4.SS1.p1.7.m1.1.1.2.cmml" xref="S4.SS1.p1.7.m1.1.1.2">𝐡</ci><ci id="S4.SS1.p1.7.m1.1.1.3a.cmml" xref="S4.SS1.p1.7.m1.1.1.3"><mtext class="ltx_mathvariant_monospace" mathsize="70%" id="S4.SS1.p1.7.m1.1.1.3.cmml" xref="S4.SS1.p1.7.m1.1.1.3">[MASK]</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.7.m1.1c">\mathbf{h}_{\texttt{[MASK]}}</annotation></semantics></math> is the hidden vector of <span id="S4.SS1.p1.12.1" class="ltx_text ltx_font_typewriter">[MASK]</span> and <math id="S4.SS1.p1.8.m2.1" class="ltx_Math" alttext="\mathbf{w}_{v}" display="inline"><semantics id="S4.SS1.p1.8.m2.1a"><msub id="S4.SS1.p1.8.m2.1.1" xref="S4.SS1.p1.8.m2.1.1.cmml"><mi id="S4.SS1.p1.8.m2.1.1.2" xref="S4.SS1.p1.8.m2.1.1.2.cmml">𝐰</mi><mi id="S4.SS1.p1.8.m2.1.1.3" xref="S4.SS1.p1.8.m2.1.1.3.cmml">v</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.8.m2.1b"><apply id="S4.SS1.p1.8.m2.1.1.cmml" xref="S4.SS1.p1.8.m2.1.1"><csymbol cd="ambiguous" id="S4.SS1.p1.8.m2.1.1.1.cmml" xref="S4.SS1.p1.8.m2.1.1">subscript</csymbol><ci id="S4.SS1.p1.8.m2.1.1.2.cmml" xref="S4.SS1.p1.8.m2.1.1.2">𝐰</ci><ci id="S4.SS1.p1.8.m2.1.1.3.cmml" xref="S4.SS1.p1.8.m2.1.1.3">𝑣</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.8.m2.1c">\mathbf{w}_{v}</annotation></semantics></math> denotes the pre-softmax vector corresponding to <math id="S4.SS1.p1.9.m3.1" class="ltx_Math" alttext="v\in\mathcal{V}" display="inline"><semantics id="S4.SS1.p1.9.m3.1a"><mrow id="S4.SS1.p1.9.m3.1.1" xref="S4.SS1.p1.9.m3.1.1.cmml"><mi id="S4.SS1.p1.9.m3.1.1.2" xref="S4.SS1.p1.9.m3.1.1.2.cmml">v</mi><mo id="S4.SS1.p1.9.m3.1.1.1" xref="S4.SS1.p1.9.m3.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S4.SS1.p1.9.m3.1.1.3" xref="S4.SS1.p1.9.m3.1.1.3.cmml">𝒱</mi></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.9.m3.1b"><apply id="S4.SS1.p1.9.m3.1.1.cmml" xref="S4.SS1.p1.9.m3.1.1"><in id="S4.SS1.p1.9.m3.1.1.1.cmml" xref="S4.SS1.p1.9.m3.1.1.1"></in><ci id="S4.SS1.p1.9.m3.1.1.2.cmml" xref="S4.SS1.p1.9.m3.1.1.2">𝑣</ci><ci id="S4.SS1.p1.9.m3.1.1.3.cmml" xref="S4.SS1.p1.9.m3.1.1.3">𝒱</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.9.m3.1c">v\in\mathcal{V}</annotation></semantics></math>.
When supervised examples <math id="S4.SS1.p1.10.m4.2" class="ltx_Math" alttext="\{({x}_{\mathrm{in}},y)\}" display="inline"><semantics id="S4.SS1.p1.10.m4.2a"><mrow id="S4.SS1.p1.10.m4.2.2.1" xref="S4.SS1.p1.10.m4.2.2.2.cmml"><mo stretchy="false" id="S4.SS1.p1.10.m4.2.2.1.2" xref="S4.SS1.p1.10.m4.2.2.2.cmml">{</mo><mrow id="S4.SS1.p1.10.m4.2.2.1.1.1" xref="S4.SS1.p1.10.m4.2.2.1.1.2.cmml"><mo stretchy="false" id="S4.SS1.p1.10.m4.2.2.1.1.1.2" xref="S4.SS1.p1.10.m4.2.2.1.1.2.cmml">(</mo><msub id="S4.SS1.p1.10.m4.2.2.1.1.1.1" xref="S4.SS1.p1.10.m4.2.2.1.1.1.1.cmml"><mi id="S4.SS1.p1.10.m4.2.2.1.1.1.1.2" xref="S4.SS1.p1.10.m4.2.2.1.1.1.1.2.cmml">x</mi><mi id="S4.SS1.p1.10.m4.2.2.1.1.1.1.3" xref="S4.SS1.p1.10.m4.2.2.1.1.1.1.3.cmml">in</mi></msub><mo id="S4.SS1.p1.10.m4.2.2.1.1.1.3" xref="S4.SS1.p1.10.m4.2.2.1.1.2.cmml">,</mo><mi id="S4.SS1.p1.10.m4.1.1" xref="S4.SS1.p1.10.m4.1.1.cmml">y</mi><mo stretchy="false" id="S4.SS1.p1.10.m4.2.2.1.1.1.4" xref="S4.SS1.p1.10.m4.2.2.1.1.2.cmml">)</mo></mrow><mo stretchy="false" id="S4.SS1.p1.10.m4.2.2.1.3" xref="S4.SS1.p1.10.m4.2.2.2.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.10.m4.2b"><set id="S4.SS1.p1.10.m4.2.2.2.cmml" xref="S4.SS1.p1.10.m4.2.2.1"><interval closure="open" id="S4.SS1.p1.10.m4.2.2.1.1.2.cmml" xref="S4.SS1.p1.10.m4.2.2.1.1.1"><apply id="S4.SS1.p1.10.m4.2.2.1.1.1.1.cmml" xref="S4.SS1.p1.10.m4.2.2.1.1.1.1"><csymbol cd="ambiguous" id="S4.SS1.p1.10.m4.2.2.1.1.1.1.1.cmml" xref="S4.SS1.p1.10.m4.2.2.1.1.1.1">subscript</csymbol><ci id="S4.SS1.p1.10.m4.2.2.1.1.1.1.2.cmml" xref="S4.SS1.p1.10.m4.2.2.1.1.1.1.2">𝑥</ci><ci id="S4.SS1.p1.10.m4.2.2.1.1.1.1.3.cmml" xref="S4.SS1.p1.10.m4.2.2.1.1.1.1.3">in</ci></apply><ci id="S4.SS1.p1.10.m4.1.1.cmml" xref="S4.SS1.p1.10.m4.1.1">𝑦</ci></interval></set></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.10.m4.2c">\{({x}_{\mathrm{in}},y)\}</annotation></semantics></math> are available, <math id="S4.SS1.p1.11.m5.1" class="ltx_Math" alttext="\mathcal{L}" display="inline"><semantics id="S4.SS1.p1.11.m5.1a"><mi class="ltx_font_mathcaligraphic" id="S4.SS1.p1.11.m5.1.1" xref="S4.SS1.p1.11.m5.1.1.cmml">ℒ</mi><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.11.m5.1b"><ci id="S4.SS1.p1.11.m5.1.1.cmml" xref="S4.SS1.p1.11.m5.1.1">ℒ</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.11.m5.1c">\mathcal{L}</annotation></semantics></math> can be fine-tuned to minimize the cross-entropy loss.
It is important to note that this approach re-uses the pre-trained weights <math id="S4.SS1.p1.12.m6.1" class="ltx_Math" alttext="\mathbf{w}_{v}" display="inline"><semantics id="S4.SS1.p1.12.m6.1a"><msub id="S4.SS1.p1.12.m6.1.1" xref="S4.SS1.p1.12.m6.1.1.cmml"><mi id="S4.SS1.p1.12.m6.1.1.2" xref="S4.SS1.p1.12.m6.1.1.2.cmml">𝐰</mi><mi id="S4.SS1.p1.12.m6.1.1.3" xref="S4.SS1.p1.12.m6.1.1.3.cmml">v</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS1.p1.12.m6.1b"><apply id="S4.SS1.p1.12.m6.1.1.cmml" xref="S4.SS1.p1.12.m6.1.1"><csymbol cd="ambiguous" id="S4.SS1.p1.12.m6.1.1.1.cmml" xref="S4.SS1.p1.12.m6.1.1">subscript</csymbol><ci id="S4.SS1.p1.12.m6.1.1.2.cmml" xref="S4.SS1.p1.12.m6.1.1.2">𝐰</ci><ci id="S4.SS1.p1.12.m6.1.1.3.cmml" xref="S4.SS1.p1.12.m6.1.1.3">𝑣</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.p1.12.m6.1c">\mathbf{w}_{v}</annotation></semantics></math> and does not introduce any new parameters. It also reduces the gap between pre-training and fine-tuning, making it more effective in few-shot scenarios.</p>
</div>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2 </span>Regression</h3>

<div id="S4.SS2.p1" class="ltx_para">
<p id="S4.SS2.p1.9" class="ltx_p">We assume the same basic setup as in classification, but treat the label space <math id="S4.SS2.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{Y}" display="inline"><semantics id="S4.SS2.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S4.SS2.p1.1.m1.1.1" xref="S4.SS2.p1.1.m1.1.1.cmml">𝒴</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.1.m1.1b"><ci id="S4.SS2.p1.1.m1.1.1.cmml" xref="S4.SS2.p1.1.m1.1.1">𝒴</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.1.m1.1c">\mathcal{Y}</annotation></semantics></math> as a bounded interval <math id="S4.SS2.p1.2.m2.2" class="ltx_Math" alttext="[v_{l},v_{u}]" display="inline"><semantics id="S4.SS2.p1.2.m2.2a"><mrow id="S4.SS2.p1.2.m2.2.2.2" xref="S4.SS2.p1.2.m2.2.2.3.cmml"><mo stretchy="false" id="S4.SS2.p1.2.m2.2.2.2.3" xref="S4.SS2.p1.2.m2.2.2.3.cmml">[</mo><msub id="S4.SS2.p1.2.m2.1.1.1.1" xref="S4.SS2.p1.2.m2.1.1.1.1.cmml"><mi id="S4.SS2.p1.2.m2.1.1.1.1.2" xref="S4.SS2.p1.2.m2.1.1.1.1.2.cmml">v</mi><mi id="S4.SS2.p1.2.m2.1.1.1.1.3" xref="S4.SS2.p1.2.m2.1.1.1.1.3.cmml">l</mi></msub><mo id="S4.SS2.p1.2.m2.2.2.2.4" xref="S4.SS2.p1.2.m2.2.2.3.cmml">,</mo><msub id="S4.SS2.p1.2.m2.2.2.2.2" xref="S4.SS2.p1.2.m2.2.2.2.2.cmml"><mi id="S4.SS2.p1.2.m2.2.2.2.2.2" xref="S4.SS2.p1.2.m2.2.2.2.2.2.cmml">v</mi><mi id="S4.SS2.p1.2.m2.2.2.2.2.3" xref="S4.SS2.p1.2.m2.2.2.2.2.3.cmml">u</mi></msub><mo stretchy="false" id="S4.SS2.p1.2.m2.2.2.2.5" xref="S4.SS2.p1.2.m2.2.2.3.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.2.m2.2b"><interval closure="closed" id="S4.SS2.p1.2.m2.2.2.3.cmml" xref="S4.SS2.p1.2.m2.2.2.2"><apply id="S4.SS2.p1.2.m2.1.1.1.1.cmml" xref="S4.SS2.p1.2.m2.1.1.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.2.m2.1.1.1.1.1.cmml" xref="S4.SS2.p1.2.m2.1.1.1.1">subscript</csymbol><ci id="S4.SS2.p1.2.m2.1.1.1.1.2.cmml" xref="S4.SS2.p1.2.m2.1.1.1.1.2">𝑣</ci><ci id="S4.SS2.p1.2.m2.1.1.1.1.3.cmml" xref="S4.SS2.p1.2.m2.1.1.1.1.3">𝑙</ci></apply><apply id="S4.SS2.p1.2.m2.2.2.2.2.cmml" xref="S4.SS2.p1.2.m2.2.2.2.2"><csymbol cd="ambiguous" id="S4.SS2.p1.2.m2.2.2.2.2.1.cmml" xref="S4.SS2.p1.2.m2.2.2.2.2">subscript</csymbol><ci id="S4.SS2.p1.2.m2.2.2.2.2.2.cmml" xref="S4.SS2.p1.2.m2.2.2.2.2.2">𝑣</ci><ci id="S4.SS2.p1.2.m2.2.2.2.2.3.cmml" xref="S4.SS2.p1.2.m2.2.2.2.2.3">𝑢</ci></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.2.m2.2c">[v_{l},v_{u}]</annotation></semantics></math>.
Inspired by  <cite class="ltx_cite ltx_citemacro_citet">Mettes et al. (<a href="#bib.bib23" title="" class="ltx_ref">2019</a>)</cite>,
we model the problem as an interpolation between two opposing poles, <math id="S4.SS2.p1.3.m3.2" class="ltx_Math" alttext="\{y_{l},y_{u}\}" display="inline"><semantics id="S4.SS2.p1.3.m3.2a"><mrow id="S4.SS2.p1.3.m3.2.2.2" xref="S4.SS2.p1.3.m3.2.2.3.cmml"><mo stretchy="false" id="S4.SS2.p1.3.m3.2.2.2.3" xref="S4.SS2.p1.3.m3.2.2.3.cmml">{</mo><msub id="S4.SS2.p1.3.m3.1.1.1.1" xref="S4.SS2.p1.3.m3.1.1.1.1.cmml"><mi id="S4.SS2.p1.3.m3.1.1.1.1.2" xref="S4.SS2.p1.3.m3.1.1.1.1.2.cmml">y</mi><mi id="S4.SS2.p1.3.m3.1.1.1.1.3" xref="S4.SS2.p1.3.m3.1.1.1.1.3.cmml">l</mi></msub><mo id="S4.SS2.p1.3.m3.2.2.2.4" xref="S4.SS2.p1.3.m3.2.2.3.cmml">,</mo><msub id="S4.SS2.p1.3.m3.2.2.2.2" xref="S4.SS2.p1.3.m3.2.2.2.2.cmml"><mi id="S4.SS2.p1.3.m3.2.2.2.2.2" xref="S4.SS2.p1.3.m3.2.2.2.2.2.cmml">y</mi><mi id="S4.SS2.p1.3.m3.2.2.2.2.3" xref="S4.SS2.p1.3.m3.2.2.2.2.3.cmml">u</mi></msub><mo stretchy="false" id="S4.SS2.p1.3.m3.2.2.2.5" xref="S4.SS2.p1.3.m3.2.2.3.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.3.m3.2b"><set id="S4.SS2.p1.3.m3.2.2.3.cmml" xref="S4.SS2.p1.3.m3.2.2.2"><apply id="S4.SS2.p1.3.m3.1.1.1.1.cmml" xref="S4.SS2.p1.3.m3.1.1.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.3.m3.1.1.1.1.1.cmml" xref="S4.SS2.p1.3.m3.1.1.1.1">subscript</csymbol><ci id="S4.SS2.p1.3.m3.1.1.1.1.2.cmml" xref="S4.SS2.p1.3.m3.1.1.1.1.2">𝑦</ci><ci id="S4.SS2.p1.3.m3.1.1.1.1.3.cmml" xref="S4.SS2.p1.3.m3.1.1.1.1.3">𝑙</ci></apply><apply id="S4.SS2.p1.3.m3.2.2.2.2.cmml" xref="S4.SS2.p1.3.m3.2.2.2.2"><csymbol cd="ambiguous" id="S4.SS2.p1.3.m3.2.2.2.2.1.cmml" xref="S4.SS2.p1.3.m3.2.2.2.2">subscript</csymbol><ci id="S4.SS2.p1.3.m3.2.2.2.2.2.cmml" xref="S4.SS2.p1.3.m3.2.2.2.2.2">𝑦</ci><ci id="S4.SS2.p1.3.m3.2.2.2.2.3.cmml" xref="S4.SS2.p1.3.m3.2.2.2.2.3">𝑢</ci></apply></set></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.3.m3.2c">\{y_{l},y_{u}\}</annotation></semantics></math>, with values <math id="S4.SS2.p1.4.m4.1" class="ltx_Math" alttext="v_{l}" display="inline"><semantics id="S4.SS2.p1.4.m4.1a"><msub id="S4.SS2.p1.4.m4.1.1" xref="S4.SS2.p1.4.m4.1.1.cmml"><mi id="S4.SS2.p1.4.m4.1.1.2" xref="S4.SS2.p1.4.m4.1.1.2.cmml">v</mi><mi id="S4.SS2.p1.4.m4.1.1.3" xref="S4.SS2.p1.4.m4.1.1.3.cmml">l</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.4.m4.1b"><apply id="S4.SS2.p1.4.m4.1.1.cmml" xref="S4.SS2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.4.m4.1.1.1.cmml" xref="S4.SS2.p1.4.m4.1.1">subscript</csymbol><ci id="S4.SS2.p1.4.m4.1.1.2.cmml" xref="S4.SS2.p1.4.m4.1.1.2">𝑣</ci><ci id="S4.SS2.p1.4.m4.1.1.3.cmml" xref="S4.SS2.p1.4.m4.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.4.m4.1c">v_{l}</annotation></semantics></math> and <math id="S4.SS2.p1.5.m5.1" class="ltx_Math" alttext="v_{u}" display="inline"><semantics id="S4.SS2.p1.5.m5.1a"><msub id="S4.SS2.p1.5.m5.1.1" xref="S4.SS2.p1.5.m5.1.1.cmml"><mi id="S4.SS2.p1.5.m5.1.1.2" xref="S4.SS2.p1.5.m5.1.1.2.cmml">v</mi><mi id="S4.SS2.p1.5.m5.1.1.3" xref="S4.SS2.p1.5.m5.1.1.3.cmml">u</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.5.m5.1b"><apply id="S4.SS2.p1.5.m5.1.1.cmml" xref="S4.SS2.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.5.m5.1.1.1.cmml" xref="S4.SS2.p1.5.m5.1.1">subscript</csymbol><ci id="S4.SS2.p1.5.m5.1.1.2.cmml" xref="S4.SS2.p1.5.m5.1.1.2">𝑣</ci><ci id="S4.SS2.p1.5.m5.1.1.3.cmml" xref="S4.SS2.p1.5.m5.1.1.3">𝑢</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.5.m5.1c">v_{u}</annotation></semantics></math> respectively.
For instance, we can formulate our previous sentiment analysis task as a regression problem in the range <math id="S4.SS2.p1.6.m6.2" class="ltx_Math" alttext="[0,1]" display="inline"><semantics id="S4.SS2.p1.6.m6.2a"><mrow id="S4.SS2.p1.6.m6.2.3.2" xref="S4.SS2.p1.6.m6.2.3.1.cmml"><mo stretchy="false" id="S4.SS2.p1.6.m6.2.3.2.1" xref="S4.SS2.p1.6.m6.2.3.1.cmml">[</mo><mn id="S4.SS2.p1.6.m6.1.1" xref="S4.SS2.p1.6.m6.1.1.cmml">0</mn><mo id="S4.SS2.p1.6.m6.2.3.2.2" xref="S4.SS2.p1.6.m6.2.3.1.cmml">,</mo><mn id="S4.SS2.p1.6.m6.2.2" xref="S4.SS2.p1.6.m6.2.2.cmml">1</mn><mo stretchy="false" id="S4.SS2.p1.6.m6.2.3.2.3" xref="S4.SS2.p1.6.m6.2.3.1.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.6.m6.2b"><interval closure="closed" id="S4.SS2.p1.6.m6.2.3.1.cmml" xref="S4.SS2.p1.6.m6.2.3.2"><cn type="integer" id="S4.SS2.p1.6.m6.1.1.cmml" xref="S4.SS2.p1.6.m6.1.1">0</cn><cn type="integer" id="S4.SS2.p1.6.m6.2.2.cmml" xref="S4.SS2.p1.6.m6.2.2">1</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.6.m6.2c">[0,1]</annotation></semantics></math>, where we slide between “<em id="S4.SS2.p1.9.1" class="ltx_emph ltx_font_italic">terrible</em>” (<math id="S4.SS2.p1.7.m7.1" class="ltx_Math" alttext="v_{l}=0" display="inline"><semantics id="S4.SS2.p1.7.m7.1a"><mrow id="S4.SS2.p1.7.m7.1.1" xref="S4.SS2.p1.7.m7.1.1.cmml"><msub id="S4.SS2.p1.7.m7.1.1.2" xref="S4.SS2.p1.7.m7.1.1.2.cmml"><mi id="S4.SS2.p1.7.m7.1.1.2.2" xref="S4.SS2.p1.7.m7.1.1.2.2.cmml">v</mi><mi id="S4.SS2.p1.7.m7.1.1.2.3" xref="S4.SS2.p1.7.m7.1.1.2.3.cmml">l</mi></msub><mo id="S4.SS2.p1.7.m7.1.1.1" xref="S4.SS2.p1.7.m7.1.1.1.cmml">=</mo><mn id="S4.SS2.p1.7.m7.1.1.3" xref="S4.SS2.p1.7.m7.1.1.3.cmml">0</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.7.m7.1b"><apply id="S4.SS2.p1.7.m7.1.1.cmml" xref="S4.SS2.p1.7.m7.1.1"><eq id="S4.SS2.p1.7.m7.1.1.1.cmml" xref="S4.SS2.p1.7.m7.1.1.1"></eq><apply id="S4.SS2.p1.7.m7.1.1.2.cmml" xref="S4.SS2.p1.7.m7.1.1.2"><csymbol cd="ambiguous" id="S4.SS2.p1.7.m7.1.1.2.1.cmml" xref="S4.SS2.p1.7.m7.1.1.2">subscript</csymbol><ci id="S4.SS2.p1.7.m7.1.1.2.2.cmml" xref="S4.SS2.p1.7.m7.1.1.2.2">𝑣</ci><ci id="S4.SS2.p1.7.m7.1.1.2.3.cmml" xref="S4.SS2.p1.7.m7.1.1.2.3">𝑙</ci></apply><cn type="integer" id="S4.SS2.p1.7.m7.1.1.3.cmml" xref="S4.SS2.p1.7.m7.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.7.m7.1c">v_{l}=0</annotation></semantics></math>) and “<em id="S4.SS2.p1.9.2" class="ltx_emph ltx_font_italic">great</em>” (<math id="S4.SS2.p1.8.m8.1" class="ltx_Math" alttext="v_{u}=1" display="inline"><semantics id="S4.SS2.p1.8.m8.1a"><mrow id="S4.SS2.p1.8.m8.1.1" xref="S4.SS2.p1.8.m8.1.1.cmml"><msub id="S4.SS2.p1.8.m8.1.1.2" xref="S4.SS2.p1.8.m8.1.1.2.cmml"><mi id="S4.SS2.p1.8.m8.1.1.2.2" xref="S4.SS2.p1.8.m8.1.1.2.2.cmml">v</mi><mi id="S4.SS2.p1.8.m8.1.1.2.3" xref="S4.SS2.p1.8.m8.1.1.2.3.cmml">u</mi></msub><mo id="S4.SS2.p1.8.m8.1.1.1" xref="S4.SS2.p1.8.m8.1.1.1.cmml">=</mo><mn id="S4.SS2.p1.8.m8.1.1.3" xref="S4.SS2.p1.8.m8.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.8.m8.1b"><apply id="S4.SS2.p1.8.m8.1.1.cmml" xref="S4.SS2.p1.8.m8.1.1"><eq id="S4.SS2.p1.8.m8.1.1.1.cmml" xref="S4.SS2.p1.8.m8.1.1.1"></eq><apply id="S4.SS2.p1.8.m8.1.1.2.cmml" xref="S4.SS2.p1.8.m8.1.1.2"><csymbol cd="ambiguous" id="S4.SS2.p1.8.m8.1.1.2.1.cmml" xref="S4.SS2.p1.8.m8.1.1.2">subscript</csymbol><ci id="S4.SS2.p1.8.m8.1.1.2.2.cmml" xref="S4.SS2.p1.8.m8.1.1.2.2">𝑣</ci><ci id="S4.SS2.p1.8.m8.1.1.2.3.cmml" xref="S4.SS2.p1.8.m8.1.1.2.3">𝑢</ci></apply><cn type="integer" id="S4.SS2.p1.8.m8.1.1.3.cmml" xref="S4.SS2.p1.8.m8.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.8.m8.1c">v_{u}=1</annotation></semantics></math>). In this way, we can express <math id="S4.SS2.p1.9.m9.1" class="ltx_Math" alttext="y" display="inline"><semantics id="S4.SS2.p1.9.m9.1a"><mi id="S4.SS2.p1.9.m9.1.1" xref="S4.SS2.p1.9.m9.1.1.cmml">y</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.9.m9.1b"><ci id="S4.SS2.p1.9.m9.1.1.cmml" xref="S4.SS2.p1.9.m9.1.1">𝑦</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.9.m9.1c">y</annotation></semantics></math> as a <em id="S4.SS2.p1.9.3" class="ltx_emph ltx_font_italic">mixture model</em>:</p>
<table id="S4.E2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S4.E2.m1.1" class="ltx_Math" alttext="y=v_{l}\cdot p(y_{l}\mid{x}_{\mathrm{in}})+v_{u}\cdot p(y_{u}\mid{x}_{\mathrm{in}})," display="block"><semantics id="S4.E2.m1.1a"><mrow id="S4.E2.m1.1.1.1" xref="S4.E2.m1.1.1.1.1.cmml"><mrow id="S4.E2.m1.1.1.1.1" xref="S4.E2.m1.1.1.1.1.cmml"><mi id="S4.E2.m1.1.1.1.1.4" xref="S4.E2.m1.1.1.1.1.4.cmml">y</mi><mo id="S4.E2.m1.1.1.1.1.3" xref="S4.E2.m1.1.1.1.1.3.cmml">=</mo><mrow id="S4.E2.m1.1.1.1.1.2" xref="S4.E2.m1.1.1.1.1.2.cmml"><mrow id="S4.E2.m1.1.1.1.1.1.1" xref="S4.E2.m1.1.1.1.1.1.1.cmml"><mrow id="S4.E2.m1.1.1.1.1.1.1.3" xref="S4.E2.m1.1.1.1.1.1.1.3.cmml"><msub id="S4.E2.m1.1.1.1.1.1.1.3.2" xref="S4.E2.m1.1.1.1.1.1.1.3.2.cmml"><mi id="S4.E2.m1.1.1.1.1.1.1.3.2.2" xref="S4.E2.m1.1.1.1.1.1.1.3.2.2.cmml">v</mi><mi id="S4.E2.m1.1.1.1.1.1.1.3.2.3" xref="S4.E2.m1.1.1.1.1.1.1.3.2.3.cmml">l</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S4.E2.m1.1.1.1.1.1.1.3.1" xref="S4.E2.m1.1.1.1.1.1.1.3.1.cmml">⋅</mo><mi id="S4.E2.m1.1.1.1.1.1.1.3.3" xref="S4.E2.m1.1.1.1.1.1.1.3.3.cmml">p</mi></mrow><mo lspace="0em" rspace="0em" id="S4.E2.m1.1.1.1.1.1.1.2" xref="S4.E2.m1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S4.E2.m1.1.1.1.1.1.1.1.1" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.E2.m1.1.1.1.1.1.1.1.1.2" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.E2.m1.1.1.1.1.1.1.1.1.1" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.cmml"><msub id="S4.E2.m1.1.1.1.1.1.1.1.1.1.2" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.2.cmml"><mi id="S4.E2.m1.1.1.1.1.1.1.1.1.1.2.2" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.2.2.cmml">y</mi><mi id="S4.E2.m1.1.1.1.1.1.1.1.1.1.2.3" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.2.3.cmml">l</mi></msub><mo id="S4.E2.m1.1.1.1.1.1.1.1.1.1.1" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.1.cmml">∣</mo><msub id="S4.E2.m1.1.1.1.1.1.1.1.1.1.3" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S4.E2.m1.1.1.1.1.1.1.1.1.1.3.2" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.3.2.cmml">x</mi><mi id="S4.E2.m1.1.1.1.1.1.1.1.1.1.3.3" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.3.3.cmml">in</mi></msub></mrow><mo stretchy="false" id="S4.E2.m1.1.1.1.1.1.1.1.1.3" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S4.E2.m1.1.1.1.1.2.3" xref="S4.E2.m1.1.1.1.1.2.3.cmml">+</mo><mrow id="S4.E2.m1.1.1.1.1.2.2" xref="S4.E2.m1.1.1.1.1.2.2.cmml"><mrow id="S4.E2.m1.1.1.1.1.2.2.3" xref="S4.E2.m1.1.1.1.1.2.2.3.cmml"><msub id="S4.E2.m1.1.1.1.1.2.2.3.2" xref="S4.E2.m1.1.1.1.1.2.2.3.2.cmml"><mi id="S4.E2.m1.1.1.1.1.2.2.3.2.2" xref="S4.E2.m1.1.1.1.1.2.2.3.2.2.cmml">v</mi><mi id="S4.E2.m1.1.1.1.1.2.2.3.2.3" xref="S4.E2.m1.1.1.1.1.2.2.3.2.3.cmml">u</mi></msub><mo lspace="0.222em" rspace="0.222em" id="S4.E2.m1.1.1.1.1.2.2.3.1" xref="S4.E2.m1.1.1.1.1.2.2.3.1.cmml">⋅</mo><mi id="S4.E2.m1.1.1.1.1.2.2.3.3" xref="S4.E2.m1.1.1.1.1.2.2.3.3.cmml">p</mi></mrow><mo lspace="0em" rspace="0em" id="S4.E2.m1.1.1.1.1.2.2.2" xref="S4.E2.m1.1.1.1.1.2.2.2.cmml">​</mo><mrow id="S4.E2.m1.1.1.1.1.2.2.1.1" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.cmml"><mo stretchy="false" id="S4.E2.m1.1.1.1.1.2.2.1.1.2" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.cmml">(</mo><mrow id="S4.E2.m1.1.1.1.1.2.2.1.1.1" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.cmml"><msub id="S4.E2.m1.1.1.1.1.2.2.1.1.1.2" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.2.cmml"><mi id="S4.E2.m1.1.1.1.1.2.2.1.1.1.2.2" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.2.2.cmml">y</mi><mi id="S4.E2.m1.1.1.1.1.2.2.1.1.1.2.3" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.2.3.cmml">u</mi></msub><mo id="S4.E2.m1.1.1.1.1.2.2.1.1.1.1" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.1.cmml">∣</mo><msub id="S4.E2.m1.1.1.1.1.2.2.1.1.1.3" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.3.cmml"><mi id="S4.E2.m1.1.1.1.1.2.2.1.1.1.3.2" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.3.2.cmml">x</mi><mi id="S4.E2.m1.1.1.1.1.2.2.1.1.1.3.3" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.3.3.cmml">in</mi></msub></mrow><mo stretchy="false" id="S4.E2.m1.1.1.1.1.2.2.1.1.3" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><mo id="S4.E2.m1.1.1.1.2" xref="S4.E2.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.E2.m1.1b"><apply id="S4.E2.m1.1.1.1.1.cmml" xref="S4.E2.m1.1.1.1"><eq id="S4.E2.m1.1.1.1.1.3.cmml" xref="S4.E2.m1.1.1.1.1.3"></eq><ci id="S4.E2.m1.1.1.1.1.4.cmml" xref="S4.E2.m1.1.1.1.1.4">𝑦</ci><apply id="S4.E2.m1.1.1.1.1.2.cmml" xref="S4.E2.m1.1.1.1.1.2"><plus id="S4.E2.m1.1.1.1.1.2.3.cmml" xref="S4.E2.m1.1.1.1.1.2.3"></plus><apply id="S4.E2.m1.1.1.1.1.1.1.cmml" xref="S4.E2.m1.1.1.1.1.1.1"><times id="S4.E2.m1.1.1.1.1.1.1.2.cmml" xref="S4.E2.m1.1.1.1.1.1.1.2"></times><apply id="S4.E2.m1.1.1.1.1.1.1.3.cmml" xref="S4.E2.m1.1.1.1.1.1.1.3"><ci id="S4.E2.m1.1.1.1.1.1.1.3.1.cmml" xref="S4.E2.m1.1.1.1.1.1.1.3.1">⋅</ci><apply id="S4.E2.m1.1.1.1.1.1.1.3.2.cmml" xref="S4.E2.m1.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S4.E2.m1.1.1.1.1.1.1.3.2.1.cmml" xref="S4.E2.m1.1.1.1.1.1.1.3.2">subscript</csymbol><ci id="S4.E2.m1.1.1.1.1.1.1.3.2.2.cmml" xref="S4.E2.m1.1.1.1.1.1.1.3.2.2">𝑣</ci><ci id="S4.E2.m1.1.1.1.1.1.1.3.2.3.cmml" xref="S4.E2.m1.1.1.1.1.1.1.3.2.3">𝑙</ci></apply><ci id="S4.E2.m1.1.1.1.1.1.1.3.3.cmml" xref="S4.E2.m1.1.1.1.1.1.1.3.3">𝑝</ci></apply><apply id="S4.E2.m1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E2.m1.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S4.E2.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.1">conditional</csymbol><apply id="S4.E2.m1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.E2.m1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.2">subscript</csymbol><ci id="S4.E2.m1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.2.2">𝑦</ci><ci id="S4.E2.m1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.2.3">𝑙</ci></apply><apply id="S4.E2.m1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.E2.m1.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S4.E2.m1.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.3.2">𝑥</ci><ci id="S4.E2.m1.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S4.E2.m1.1.1.1.1.1.1.1.1.1.3.3">in</ci></apply></apply></apply><apply id="S4.E2.m1.1.1.1.1.2.2.cmml" xref="S4.E2.m1.1.1.1.1.2.2"><times id="S4.E2.m1.1.1.1.1.2.2.2.cmml" xref="S4.E2.m1.1.1.1.1.2.2.2"></times><apply id="S4.E2.m1.1.1.1.1.2.2.3.cmml" xref="S4.E2.m1.1.1.1.1.2.2.3"><ci id="S4.E2.m1.1.1.1.1.2.2.3.1.cmml" xref="S4.E2.m1.1.1.1.1.2.2.3.1">⋅</ci><apply id="S4.E2.m1.1.1.1.1.2.2.3.2.cmml" xref="S4.E2.m1.1.1.1.1.2.2.3.2"><csymbol cd="ambiguous" id="S4.E2.m1.1.1.1.1.2.2.3.2.1.cmml" xref="S4.E2.m1.1.1.1.1.2.2.3.2">subscript</csymbol><ci id="S4.E2.m1.1.1.1.1.2.2.3.2.2.cmml" xref="S4.E2.m1.1.1.1.1.2.2.3.2.2">𝑣</ci><ci id="S4.E2.m1.1.1.1.1.2.2.3.2.3.cmml" xref="S4.E2.m1.1.1.1.1.2.2.3.2.3">𝑢</ci></apply><ci id="S4.E2.m1.1.1.1.1.2.2.3.3.cmml" xref="S4.E2.m1.1.1.1.1.2.2.3.3">𝑝</ci></apply><apply id="S4.E2.m1.1.1.1.1.2.2.1.1.1.cmml" xref="S4.E2.m1.1.1.1.1.2.2.1.1"><csymbol cd="latexml" id="S4.E2.m1.1.1.1.1.2.2.1.1.1.1.cmml" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.1">conditional</csymbol><apply id="S4.E2.m1.1.1.1.1.2.2.1.1.1.2.cmml" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.2"><csymbol cd="ambiguous" id="S4.E2.m1.1.1.1.1.2.2.1.1.1.2.1.cmml" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.2">subscript</csymbol><ci id="S4.E2.m1.1.1.1.1.2.2.1.1.1.2.2.cmml" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.2.2">𝑦</ci><ci id="S4.E2.m1.1.1.1.1.2.2.1.1.1.2.3.cmml" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.2.3">𝑢</ci></apply><apply id="S4.E2.m1.1.1.1.1.2.2.1.1.1.3.cmml" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.3"><csymbol cd="ambiguous" id="S4.E2.m1.1.1.1.1.2.2.1.1.1.3.1.cmml" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.3">subscript</csymbol><ci id="S4.E2.m1.1.1.1.1.2.2.1.1.1.3.2.cmml" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.3.2">𝑥</ci><ci id="S4.E2.m1.1.1.1.1.2.2.1.1.1.3.3.cmml" xref="S4.E2.m1.1.1.1.1.2.2.1.1.1.3.3">in</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.E2.m1.1c">y=v_{l}\cdot p(y_{l}\mid{x}_{\mathrm{in}})+v_{u}\cdot p(y_{u}\mid{x}_{\mathrm{in}}),</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(2)</span></td>
</tr></tbody>
</table>
<p id="S4.SS2.p1.17" class="ltx_p">where <math id="S4.SS2.p1.10.m1.1" class="ltx_Math" alttext="p(y_{u}\mid{x}_{\mathrm{in}})" display="inline"><semantics id="S4.SS2.p1.10.m1.1a"><mrow id="S4.SS2.p1.10.m1.1.1" xref="S4.SS2.p1.10.m1.1.1.cmml"><mi id="S4.SS2.p1.10.m1.1.1.3" xref="S4.SS2.p1.10.m1.1.1.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.10.m1.1.1.2" xref="S4.SS2.p1.10.m1.1.1.2.cmml">​</mo><mrow id="S4.SS2.p1.10.m1.1.1.1.1" xref="S4.SS2.p1.10.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS2.p1.10.m1.1.1.1.1.2" xref="S4.SS2.p1.10.m1.1.1.1.1.1.cmml">(</mo><mrow id="S4.SS2.p1.10.m1.1.1.1.1.1" xref="S4.SS2.p1.10.m1.1.1.1.1.1.cmml"><msub id="S4.SS2.p1.10.m1.1.1.1.1.1.2" xref="S4.SS2.p1.10.m1.1.1.1.1.1.2.cmml"><mi id="S4.SS2.p1.10.m1.1.1.1.1.1.2.2" xref="S4.SS2.p1.10.m1.1.1.1.1.1.2.2.cmml">y</mi><mi id="S4.SS2.p1.10.m1.1.1.1.1.1.2.3" xref="S4.SS2.p1.10.m1.1.1.1.1.1.2.3.cmml">u</mi></msub><mo id="S4.SS2.p1.10.m1.1.1.1.1.1.1" xref="S4.SS2.p1.10.m1.1.1.1.1.1.1.cmml">∣</mo><msub id="S4.SS2.p1.10.m1.1.1.1.1.1.3" xref="S4.SS2.p1.10.m1.1.1.1.1.1.3.cmml"><mi id="S4.SS2.p1.10.m1.1.1.1.1.1.3.2" xref="S4.SS2.p1.10.m1.1.1.1.1.1.3.2.cmml">x</mi><mi id="S4.SS2.p1.10.m1.1.1.1.1.1.3.3" xref="S4.SS2.p1.10.m1.1.1.1.1.1.3.3.cmml">in</mi></msub></mrow><mo stretchy="false" id="S4.SS2.p1.10.m1.1.1.1.1.3" xref="S4.SS2.p1.10.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.10.m1.1b"><apply id="S4.SS2.p1.10.m1.1.1.cmml" xref="S4.SS2.p1.10.m1.1.1"><times id="S4.SS2.p1.10.m1.1.1.2.cmml" xref="S4.SS2.p1.10.m1.1.1.2"></times><ci id="S4.SS2.p1.10.m1.1.1.3.cmml" xref="S4.SS2.p1.10.m1.1.1.3">𝑝</ci><apply id="S4.SS2.p1.10.m1.1.1.1.1.1.cmml" xref="S4.SS2.p1.10.m1.1.1.1.1"><csymbol cd="latexml" id="S4.SS2.p1.10.m1.1.1.1.1.1.1.cmml" xref="S4.SS2.p1.10.m1.1.1.1.1.1.1">conditional</csymbol><apply id="S4.SS2.p1.10.m1.1.1.1.1.1.2.cmml" xref="S4.SS2.p1.10.m1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.SS2.p1.10.m1.1.1.1.1.1.2.1.cmml" xref="S4.SS2.p1.10.m1.1.1.1.1.1.2">subscript</csymbol><ci id="S4.SS2.p1.10.m1.1.1.1.1.1.2.2.cmml" xref="S4.SS2.p1.10.m1.1.1.1.1.1.2.2">𝑦</ci><ci id="S4.SS2.p1.10.m1.1.1.1.1.1.2.3.cmml" xref="S4.SS2.p1.10.m1.1.1.1.1.1.2.3">𝑢</ci></apply><apply id="S4.SS2.p1.10.m1.1.1.1.1.1.3.cmml" xref="S4.SS2.p1.10.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.SS2.p1.10.m1.1.1.1.1.1.3.1.cmml" xref="S4.SS2.p1.10.m1.1.1.1.1.1.3">subscript</csymbol><ci id="S4.SS2.p1.10.m1.1.1.1.1.1.3.2.cmml" xref="S4.SS2.p1.10.m1.1.1.1.1.1.3.2">𝑥</ci><ci id="S4.SS2.p1.10.m1.1.1.1.1.1.3.3.cmml" xref="S4.SS2.p1.10.m1.1.1.1.1.1.3.3">in</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.10.m1.1c">p(y_{u}\mid{x}_{\mathrm{in}})</annotation></semantics></math> is the probability of <math id="S4.SS2.p1.11.m2.1" class="ltx_Math" alttext="y_{u}" display="inline"><semantics id="S4.SS2.p1.11.m2.1a"><msub id="S4.SS2.p1.11.m2.1.1" xref="S4.SS2.p1.11.m2.1.1.cmml"><mi id="S4.SS2.p1.11.m2.1.1.2" xref="S4.SS2.p1.11.m2.1.1.2.cmml">y</mi><mi id="S4.SS2.p1.11.m2.1.1.3" xref="S4.SS2.p1.11.m2.1.1.3.cmml">u</mi></msub><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.11.m2.1b"><apply id="S4.SS2.p1.11.m2.1.1.cmml" xref="S4.SS2.p1.11.m2.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.11.m2.1.1.1.cmml" xref="S4.SS2.p1.11.m2.1.1">subscript</csymbol><ci id="S4.SS2.p1.11.m2.1.1.2.cmml" xref="S4.SS2.p1.11.m2.1.1.2">𝑦</ci><ci id="S4.SS2.p1.11.m2.1.1.3.cmml" xref="S4.SS2.p1.11.m2.1.1.3">𝑢</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.11.m2.1c">y_{u}</annotation></semantics></math>, and <math id="S4.SS2.p1.12.m3.2" class="ltx_Math" alttext="p(y_{l}\mid{x}_{\mathrm{in}})=1-p(y_{u}\mid{x}_{\mathrm{in}})" display="inline"><semantics id="S4.SS2.p1.12.m3.2a"><mrow id="S4.SS2.p1.12.m3.2.2" xref="S4.SS2.p1.12.m3.2.2.cmml"><mrow id="S4.SS2.p1.12.m3.1.1.1" xref="S4.SS2.p1.12.m3.1.1.1.cmml"><mi id="S4.SS2.p1.12.m3.1.1.1.3" xref="S4.SS2.p1.12.m3.1.1.1.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.12.m3.1.1.1.2" xref="S4.SS2.p1.12.m3.1.1.1.2.cmml">​</mo><mrow id="S4.SS2.p1.12.m3.1.1.1.1.1" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS2.p1.12.m3.1.1.1.1.1.2" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.cmml">(</mo><mrow id="S4.SS2.p1.12.m3.1.1.1.1.1.1" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.cmml"><msub id="S4.SS2.p1.12.m3.1.1.1.1.1.1.2" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.2.cmml"><mi id="S4.SS2.p1.12.m3.1.1.1.1.1.1.2.2" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.2.2.cmml">y</mi><mi id="S4.SS2.p1.12.m3.1.1.1.1.1.1.2.3" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.2.3.cmml">l</mi></msub><mo id="S4.SS2.p1.12.m3.1.1.1.1.1.1.1" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.1.cmml">∣</mo><msub id="S4.SS2.p1.12.m3.1.1.1.1.1.1.3" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.3.cmml"><mi id="S4.SS2.p1.12.m3.1.1.1.1.1.1.3.2" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.3.2.cmml">x</mi><mi id="S4.SS2.p1.12.m3.1.1.1.1.1.1.3.3" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.3.3.cmml">in</mi></msub></mrow><mo stretchy="false" id="S4.SS2.p1.12.m3.1.1.1.1.1.3" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S4.SS2.p1.12.m3.2.2.3" xref="S4.SS2.p1.12.m3.2.2.3.cmml">=</mo><mrow id="S4.SS2.p1.12.m3.2.2.2" xref="S4.SS2.p1.12.m3.2.2.2.cmml"><mn id="S4.SS2.p1.12.m3.2.2.2.3" xref="S4.SS2.p1.12.m3.2.2.2.3.cmml">1</mn><mo id="S4.SS2.p1.12.m3.2.2.2.2" xref="S4.SS2.p1.12.m3.2.2.2.2.cmml">−</mo><mrow id="S4.SS2.p1.12.m3.2.2.2.1" xref="S4.SS2.p1.12.m3.2.2.2.1.cmml"><mi id="S4.SS2.p1.12.m3.2.2.2.1.3" xref="S4.SS2.p1.12.m3.2.2.2.1.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.12.m3.2.2.2.1.2" xref="S4.SS2.p1.12.m3.2.2.2.1.2.cmml">​</mo><mrow id="S4.SS2.p1.12.m3.2.2.2.1.1.1" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS2.p1.12.m3.2.2.2.1.1.1.2" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.cmml">(</mo><mrow id="S4.SS2.p1.12.m3.2.2.2.1.1.1.1" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.cmml"><msub id="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.2" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.2.cmml"><mi id="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.2.2" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.2.2.cmml">y</mi><mi id="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.2.3" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.2.3.cmml">u</mi></msub><mo id="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.1" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.1.cmml">∣</mo><msub id="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.3" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.3.cmml"><mi id="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.3.2" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.3.2.cmml">x</mi><mi id="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.3.3" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.3.3.cmml">in</mi></msub></mrow><mo stretchy="false" id="S4.SS2.p1.12.m3.2.2.2.1.1.1.3" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.12.m3.2b"><apply id="S4.SS2.p1.12.m3.2.2.cmml" xref="S4.SS2.p1.12.m3.2.2"><eq id="S4.SS2.p1.12.m3.2.2.3.cmml" xref="S4.SS2.p1.12.m3.2.2.3"></eq><apply id="S4.SS2.p1.12.m3.1.1.1.cmml" xref="S4.SS2.p1.12.m3.1.1.1"><times id="S4.SS2.p1.12.m3.1.1.1.2.cmml" xref="S4.SS2.p1.12.m3.1.1.1.2"></times><ci id="S4.SS2.p1.12.m3.1.1.1.3.cmml" xref="S4.SS2.p1.12.m3.1.1.1.3">𝑝</ci><apply id="S4.SS2.p1.12.m3.1.1.1.1.1.1.cmml" xref="S4.SS2.p1.12.m3.1.1.1.1.1"><csymbol cd="latexml" id="S4.SS2.p1.12.m3.1.1.1.1.1.1.1.cmml" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.1">conditional</csymbol><apply id="S4.SS2.p1.12.m3.1.1.1.1.1.1.2.cmml" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.SS2.p1.12.m3.1.1.1.1.1.1.2.1.cmml" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.2">subscript</csymbol><ci id="S4.SS2.p1.12.m3.1.1.1.1.1.1.2.2.cmml" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.2.2">𝑦</ci><ci id="S4.SS2.p1.12.m3.1.1.1.1.1.1.2.3.cmml" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.2.3">𝑙</ci></apply><apply id="S4.SS2.p1.12.m3.1.1.1.1.1.1.3.cmml" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.SS2.p1.12.m3.1.1.1.1.1.1.3.1.cmml" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.3">subscript</csymbol><ci id="S4.SS2.p1.12.m3.1.1.1.1.1.1.3.2.cmml" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.3.2">𝑥</ci><ci id="S4.SS2.p1.12.m3.1.1.1.1.1.1.3.3.cmml" xref="S4.SS2.p1.12.m3.1.1.1.1.1.1.3.3">in</ci></apply></apply></apply><apply id="S4.SS2.p1.12.m3.2.2.2.cmml" xref="S4.SS2.p1.12.m3.2.2.2"><minus id="S4.SS2.p1.12.m3.2.2.2.2.cmml" xref="S4.SS2.p1.12.m3.2.2.2.2"></minus><cn type="integer" id="S4.SS2.p1.12.m3.2.2.2.3.cmml" xref="S4.SS2.p1.12.m3.2.2.2.3">1</cn><apply id="S4.SS2.p1.12.m3.2.2.2.1.cmml" xref="S4.SS2.p1.12.m3.2.2.2.1"><times id="S4.SS2.p1.12.m3.2.2.2.1.2.cmml" xref="S4.SS2.p1.12.m3.2.2.2.1.2"></times><ci id="S4.SS2.p1.12.m3.2.2.2.1.3.cmml" xref="S4.SS2.p1.12.m3.2.2.2.1.3">𝑝</ci><apply id="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.cmml" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1"><csymbol cd="latexml" id="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.1.cmml" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.1">conditional</csymbol><apply id="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.2.cmml" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.2.1.cmml" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.2">subscript</csymbol><ci id="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.2.2.cmml" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.2.2">𝑦</ci><ci id="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.2.3.cmml" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.2.3">𝑢</ci></apply><apply id="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.3.cmml" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.3.1.cmml" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.3">subscript</csymbol><ci id="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.3.2.cmml" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.3.2">𝑥</ci><ci id="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.3.3.cmml" xref="S4.SS2.p1.12.m3.2.2.2.1.1.1.1.3.3">in</ci></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.12.m3.2c">p(y_{l}\mid{x}_{\mathrm{in}})=1-p(y_{u}\mid{x}_{\mathrm{in}})</annotation></semantics></math>.
Then we define <math id="S4.SS2.p1.13.m4.2" class="ltx_Math" alttext="\mathcal{M}\colon\{y_{l},y_{u}\}\rightarrow\mathcal{V}" display="inline"><semantics id="S4.SS2.p1.13.m4.2a"><mrow id="S4.SS2.p1.13.m4.2.2" xref="S4.SS2.p1.13.m4.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS2.p1.13.m4.2.2.4" xref="S4.SS2.p1.13.m4.2.2.4.cmml">ℳ</mi><mo lspace="0.278em" rspace="0.278em" id="S4.SS2.p1.13.m4.2.2.3" xref="S4.SS2.p1.13.m4.2.2.3.cmml">:</mo><mrow id="S4.SS2.p1.13.m4.2.2.2" xref="S4.SS2.p1.13.m4.2.2.2.cmml"><mrow id="S4.SS2.p1.13.m4.2.2.2.2.2" xref="S4.SS2.p1.13.m4.2.2.2.2.3.cmml"><mo stretchy="false" id="S4.SS2.p1.13.m4.2.2.2.2.2.3" xref="S4.SS2.p1.13.m4.2.2.2.2.3.cmml">{</mo><msub id="S4.SS2.p1.13.m4.1.1.1.1.1.1" xref="S4.SS2.p1.13.m4.1.1.1.1.1.1.cmml"><mi id="S4.SS2.p1.13.m4.1.1.1.1.1.1.2" xref="S4.SS2.p1.13.m4.1.1.1.1.1.1.2.cmml">y</mi><mi id="S4.SS2.p1.13.m4.1.1.1.1.1.1.3" xref="S4.SS2.p1.13.m4.1.1.1.1.1.1.3.cmml">l</mi></msub><mo id="S4.SS2.p1.13.m4.2.2.2.2.2.4" xref="S4.SS2.p1.13.m4.2.2.2.2.3.cmml">,</mo><msub id="S4.SS2.p1.13.m4.2.2.2.2.2.2" xref="S4.SS2.p1.13.m4.2.2.2.2.2.2.cmml"><mi id="S4.SS2.p1.13.m4.2.2.2.2.2.2.2" xref="S4.SS2.p1.13.m4.2.2.2.2.2.2.2.cmml">y</mi><mi id="S4.SS2.p1.13.m4.2.2.2.2.2.2.3" xref="S4.SS2.p1.13.m4.2.2.2.2.2.2.3.cmml">u</mi></msub><mo stretchy="false" id="S4.SS2.p1.13.m4.2.2.2.2.2.5" xref="S4.SS2.p1.13.m4.2.2.2.2.3.cmml">}</mo></mrow><mo stretchy="false" id="S4.SS2.p1.13.m4.2.2.2.3" xref="S4.SS2.p1.13.m4.2.2.2.3.cmml">→</mo><mi class="ltx_font_mathcaligraphic" id="S4.SS2.p1.13.m4.2.2.2.4" xref="S4.SS2.p1.13.m4.2.2.2.4.cmml">𝒱</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.13.m4.2b"><apply id="S4.SS2.p1.13.m4.2.2.cmml" xref="S4.SS2.p1.13.m4.2.2"><ci id="S4.SS2.p1.13.m4.2.2.3.cmml" xref="S4.SS2.p1.13.m4.2.2.3">:</ci><ci id="S4.SS2.p1.13.m4.2.2.4.cmml" xref="S4.SS2.p1.13.m4.2.2.4">ℳ</ci><apply id="S4.SS2.p1.13.m4.2.2.2.cmml" xref="S4.SS2.p1.13.m4.2.2.2"><ci id="S4.SS2.p1.13.m4.2.2.2.3.cmml" xref="S4.SS2.p1.13.m4.2.2.2.3">→</ci><set id="S4.SS2.p1.13.m4.2.2.2.2.3.cmml" xref="S4.SS2.p1.13.m4.2.2.2.2.2"><apply id="S4.SS2.p1.13.m4.1.1.1.1.1.1.cmml" xref="S4.SS2.p1.13.m4.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S4.SS2.p1.13.m4.1.1.1.1.1.1.1.cmml" xref="S4.SS2.p1.13.m4.1.1.1.1.1.1">subscript</csymbol><ci id="S4.SS2.p1.13.m4.1.1.1.1.1.1.2.cmml" xref="S4.SS2.p1.13.m4.1.1.1.1.1.1.2">𝑦</ci><ci id="S4.SS2.p1.13.m4.1.1.1.1.1.1.3.cmml" xref="S4.SS2.p1.13.m4.1.1.1.1.1.1.3">𝑙</ci></apply><apply id="S4.SS2.p1.13.m4.2.2.2.2.2.2.cmml" xref="S4.SS2.p1.13.m4.2.2.2.2.2.2"><csymbol cd="ambiguous" id="S4.SS2.p1.13.m4.2.2.2.2.2.2.1.cmml" xref="S4.SS2.p1.13.m4.2.2.2.2.2.2">subscript</csymbol><ci id="S4.SS2.p1.13.m4.2.2.2.2.2.2.2.cmml" xref="S4.SS2.p1.13.m4.2.2.2.2.2.2.2">𝑦</ci><ci id="S4.SS2.p1.13.m4.2.2.2.2.2.2.3.cmml" xref="S4.SS2.p1.13.m4.2.2.2.2.2.2.3">𝑢</ci></apply></set><ci id="S4.SS2.p1.13.m4.2.2.2.4.cmml" xref="S4.SS2.p1.13.m4.2.2.2.4">𝒱</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.13.m4.2c">\mathcal{M}\colon\{y_{l},y_{u}\}\rightarrow\mathcal{V}</annotation></semantics></math>,
and model <math id="S4.SS2.p1.14.m5.1" class="ltx_Math" alttext="p(y_{u}\mid{x}_{\mathrm{in}})" display="inline"><semantics id="S4.SS2.p1.14.m5.1a"><mrow id="S4.SS2.p1.14.m5.1.1" xref="S4.SS2.p1.14.m5.1.1.cmml"><mi id="S4.SS2.p1.14.m5.1.1.3" xref="S4.SS2.p1.14.m5.1.1.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.14.m5.1.1.2" xref="S4.SS2.p1.14.m5.1.1.2.cmml">​</mo><mrow id="S4.SS2.p1.14.m5.1.1.1.1" xref="S4.SS2.p1.14.m5.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS2.p1.14.m5.1.1.1.1.2" xref="S4.SS2.p1.14.m5.1.1.1.1.1.cmml">(</mo><mrow id="S4.SS2.p1.14.m5.1.1.1.1.1" xref="S4.SS2.p1.14.m5.1.1.1.1.1.cmml"><msub id="S4.SS2.p1.14.m5.1.1.1.1.1.2" xref="S4.SS2.p1.14.m5.1.1.1.1.1.2.cmml"><mi id="S4.SS2.p1.14.m5.1.1.1.1.1.2.2" xref="S4.SS2.p1.14.m5.1.1.1.1.1.2.2.cmml">y</mi><mi id="S4.SS2.p1.14.m5.1.1.1.1.1.2.3" xref="S4.SS2.p1.14.m5.1.1.1.1.1.2.3.cmml">u</mi></msub><mo id="S4.SS2.p1.14.m5.1.1.1.1.1.1" xref="S4.SS2.p1.14.m5.1.1.1.1.1.1.cmml">∣</mo><msub id="S4.SS2.p1.14.m5.1.1.1.1.1.3" xref="S4.SS2.p1.14.m5.1.1.1.1.1.3.cmml"><mi id="S4.SS2.p1.14.m5.1.1.1.1.1.3.2" xref="S4.SS2.p1.14.m5.1.1.1.1.1.3.2.cmml">x</mi><mi id="S4.SS2.p1.14.m5.1.1.1.1.1.3.3" xref="S4.SS2.p1.14.m5.1.1.1.1.1.3.3.cmml">in</mi></msub></mrow><mo stretchy="false" id="S4.SS2.p1.14.m5.1.1.1.1.3" xref="S4.SS2.p1.14.m5.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.14.m5.1b"><apply id="S4.SS2.p1.14.m5.1.1.cmml" xref="S4.SS2.p1.14.m5.1.1"><times id="S4.SS2.p1.14.m5.1.1.2.cmml" xref="S4.SS2.p1.14.m5.1.1.2"></times><ci id="S4.SS2.p1.14.m5.1.1.3.cmml" xref="S4.SS2.p1.14.m5.1.1.3">𝑝</ci><apply id="S4.SS2.p1.14.m5.1.1.1.1.1.cmml" xref="S4.SS2.p1.14.m5.1.1.1.1"><csymbol cd="latexml" id="S4.SS2.p1.14.m5.1.1.1.1.1.1.cmml" xref="S4.SS2.p1.14.m5.1.1.1.1.1.1">conditional</csymbol><apply id="S4.SS2.p1.14.m5.1.1.1.1.1.2.cmml" xref="S4.SS2.p1.14.m5.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.SS2.p1.14.m5.1.1.1.1.1.2.1.cmml" xref="S4.SS2.p1.14.m5.1.1.1.1.1.2">subscript</csymbol><ci id="S4.SS2.p1.14.m5.1.1.1.1.1.2.2.cmml" xref="S4.SS2.p1.14.m5.1.1.1.1.1.2.2">𝑦</ci><ci id="S4.SS2.p1.14.m5.1.1.1.1.1.2.3.cmml" xref="S4.SS2.p1.14.m5.1.1.1.1.1.2.3">𝑢</ci></apply><apply id="S4.SS2.p1.14.m5.1.1.1.1.1.3.cmml" xref="S4.SS2.p1.14.m5.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.SS2.p1.14.m5.1.1.1.1.1.3.1.cmml" xref="S4.SS2.p1.14.m5.1.1.1.1.1.3">subscript</csymbol><ci id="S4.SS2.p1.14.m5.1.1.1.1.1.3.2.cmml" xref="S4.SS2.p1.14.m5.1.1.1.1.1.3.2">𝑥</ci><ci id="S4.SS2.p1.14.m5.1.1.1.1.1.3.3.cmml" xref="S4.SS2.p1.14.m5.1.1.1.1.1.3.3">in</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.14.m5.1c">p(y_{u}\mid{x}_{\mathrm{in}})</annotation></semantics></math> the same as Eq. (<a href="#S4.E1" title="In 4.1 Classification ‣ 4 Prompt-based Fine-tuning ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>).
We fine-tune <math id="S4.SS2.p1.15.m6.1" class="ltx_Math" alttext="\mathcal{L}" display="inline"><semantics id="S4.SS2.p1.15.m6.1a"><mi class="ltx_font_mathcaligraphic" id="S4.SS2.p1.15.m6.1.1" xref="S4.SS2.p1.15.m6.1.1.cmml">ℒ</mi><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.15.m6.1b"><ci id="S4.SS2.p1.15.m6.1.1.cmml" xref="S4.SS2.p1.15.m6.1.1">ℒ</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.15.m6.1c">\mathcal{L}</annotation></semantics></math> to minimize the KL-divergence between the inferred <math id="S4.SS2.p1.16.m7.1" class="ltx_Math" alttext="p(y_{u}\mid{x}_{\mathrm{in}})" display="inline"><semantics id="S4.SS2.p1.16.m7.1a"><mrow id="S4.SS2.p1.16.m7.1.1" xref="S4.SS2.p1.16.m7.1.1.cmml"><mi id="S4.SS2.p1.16.m7.1.1.3" xref="S4.SS2.p1.16.m7.1.1.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S4.SS2.p1.16.m7.1.1.2" xref="S4.SS2.p1.16.m7.1.1.2.cmml">​</mo><mrow id="S4.SS2.p1.16.m7.1.1.1.1" xref="S4.SS2.p1.16.m7.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS2.p1.16.m7.1.1.1.1.2" xref="S4.SS2.p1.16.m7.1.1.1.1.1.cmml">(</mo><mrow id="S4.SS2.p1.16.m7.1.1.1.1.1" xref="S4.SS2.p1.16.m7.1.1.1.1.1.cmml"><msub id="S4.SS2.p1.16.m7.1.1.1.1.1.2" xref="S4.SS2.p1.16.m7.1.1.1.1.1.2.cmml"><mi id="S4.SS2.p1.16.m7.1.1.1.1.1.2.2" xref="S4.SS2.p1.16.m7.1.1.1.1.1.2.2.cmml">y</mi><mi id="S4.SS2.p1.16.m7.1.1.1.1.1.2.3" xref="S4.SS2.p1.16.m7.1.1.1.1.1.2.3.cmml">u</mi></msub><mo id="S4.SS2.p1.16.m7.1.1.1.1.1.1" xref="S4.SS2.p1.16.m7.1.1.1.1.1.1.cmml">∣</mo><msub id="S4.SS2.p1.16.m7.1.1.1.1.1.3" xref="S4.SS2.p1.16.m7.1.1.1.1.1.3.cmml"><mi id="S4.SS2.p1.16.m7.1.1.1.1.1.3.2" xref="S4.SS2.p1.16.m7.1.1.1.1.1.3.2.cmml">x</mi><mi id="S4.SS2.p1.16.m7.1.1.1.1.1.3.3" xref="S4.SS2.p1.16.m7.1.1.1.1.1.3.3.cmml">in</mi></msub></mrow><mo stretchy="false" id="S4.SS2.p1.16.m7.1.1.1.1.3" xref="S4.SS2.p1.16.m7.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.16.m7.1b"><apply id="S4.SS2.p1.16.m7.1.1.cmml" xref="S4.SS2.p1.16.m7.1.1"><times id="S4.SS2.p1.16.m7.1.1.2.cmml" xref="S4.SS2.p1.16.m7.1.1.2"></times><ci id="S4.SS2.p1.16.m7.1.1.3.cmml" xref="S4.SS2.p1.16.m7.1.1.3">𝑝</ci><apply id="S4.SS2.p1.16.m7.1.1.1.1.1.cmml" xref="S4.SS2.p1.16.m7.1.1.1.1"><csymbol cd="latexml" id="S4.SS2.p1.16.m7.1.1.1.1.1.1.cmml" xref="S4.SS2.p1.16.m7.1.1.1.1.1.1">conditional</csymbol><apply id="S4.SS2.p1.16.m7.1.1.1.1.1.2.cmml" xref="S4.SS2.p1.16.m7.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S4.SS2.p1.16.m7.1.1.1.1.1.2.1.cmml" xref="S4.SS2.p1.16.m7.1.1.1.1.1.2">subscript</csymbol><ci id="S4.SS2.p1.16.m7.1.1.1.1.1.2.2.cmml" xref="S4.SS2.p1.16.m7.1.1.1.1.1.2.2">𝑦</ci><ci id="S4.SS2.p1.16.m7.1.1.1.1.1.2.3.cmml" xref="S4.SS2.p1.16.m7.1.1.1.1.1.2.3">𝑢</ci></apply><apply id="S4.SS2.p1.16.m7.1.1.1.1.1.3.cmml" xref="S4.SS2.p1.16.m7.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.SS2.p1.16.m7.1.1.1.1.1.3.1.cmml" xref="S4.SS2.p1.16.m7.1.1.1.1.1.3">subscript</csymbol><ci id="S4.SS2.p1.16.m7.1.1.1.1.1.3.2.cmml" xref="S4.SS2.p1.16.m7.1.1.1.1.1.3.2">𝑥</ci><ci id="S4.SS2.p1.16.m7.1.1.1.1.1.3.3.cmml" xref="S4.SS2.p1.16.m7.1.1.1.1.1.3.3">in</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.16.m7.1c">p(y_{u}\mid{x}_{\mathrm{in}})</annotation></semantics></math>
and the observed mixture weight, <math id="S4.SS2.p1.17.m8.2" class="ltx_Math" alttext="(y-v_{l})/(v_{u}-v_{l})" display="inline"><semantics id="S4.SS2.p1.17.m8.2a"><mrow id="S4.SS2.p1.17.m8.2.2" xref="S4.SS2.p1.17.m8.2.2.cmml"><mrow id="S4.SS2.p1.17.m8.1.1.1.1" xref="S4.SS2.p1.17.m8.1.1.1.1.1.cmml"><mo stretchy="false" id="S4.SS2.p1.17.m8.1.1.1.1.2" xref="S4.SS2.p1.17.m8.1.1.1.1.1.cmml">(</mo><mrow id="S4.SS2.p1.17.m8.1.1.1.1.1" xref="S4.SS2.p1.17.m8.1.1.1.1.1.cmml"><mi id="S4.SS2.p1.17.m8.1.1.1.1.1.2" xref="S4.SS2.p1.17.m8.1.1.1.1.1.2.cmml">y</mi><mo id="S4.SS2.p1.17.m8.1.1.1.1.1.1" xref="S4.SS2.p1.17.m8.1.1.1.1.1.1.cmml">−</mo><msub id="S4.SS2.p1.17.m8.1.1.1.1.1.3" xref="S4.SS2.p1.17.m8.1.1.1.1.1.3.cmml"><mi id="S4.SS2.p1.17.m8.1.1.1.1.1.3.2" xref="S4.SS2.p1.17.m8.1.1.1.1.1.3.2.cmml">v</mi><mi id="S4.SS2.p1.17.m8.1.1.1.1.1.3.3" xref="S4.SS2.p1.17.m8.1.1.1.1.1.3.3.cmml">l</mi></msub></mrow><mo stretchy="false" id="S4.SS2.p1.17.m8.1.1.1.1.3" xref="S4.SS2.p1.17.m8.1.1.1.1.1.cmml">)</mo></mrow><mo id="S4.SS2.p1.17.m8.2.2.3" xref="S4.SS2.p1.17.m8.2.2.3.cmml">/</mo><mrow id="S4.SS2.p1.17.m8.2.2.2.1" xref="S4.SS2.p1.17.m8.2.2.2.1.1.cmml"><mo stretchy="false" id="S4.SS2.p1.17.m8.2.2.2.1.2" xref="S4.SS2.p1.17.m8.2.2.2.1.1.cmml">(</mo><mrow id="S4.SS2.p1.17.m8.2.2.2.1.1" xref="S4.SS2.p1.17.m8.2.2.2.1.1.cmml"><msub id="S4.SS2.p1.17.m8.2.2.2.1.1.2" xref="S4.SS2.p1.17.m8.2.2.2.1.1.2.cmml"><mi id="S4.SS2.p1.17.m8.2.2.2.1.1.2.2" xref="S4.SS2.p1.17.m8.2.2.2.1.1.2.2.cmml">v</mi><mi id="S4.SS2.p1.17.m8.2.2.2.1.1.2.3" xref="S4.SS2.p1.17.m8.2.2.2.1.1.2.3.cmml">u</mi></msub><mo id="S4.SS2.p1.17.m8.2.2.2.1.1.1" xref="S4.SS2.p1.17.m8.2.2.2.1.1.1.cmml">−</mo><msub id="S4.SS2.p1.17.m8.2.2.2.1.1.3" xref="S4.SS2.p1.17.m8.2.2.2.1.1.3.cmml"><mi id="S4.SS2.p1.17.m8.2.2.2.1.1.3.2" xref="S4.SS2.p1.17.m8.2.2.2.1.1.3.2.cmml">v</mi><mi id="S4.SS2.p1.17.m8.2.2.2.1.1.3.3" xref="S4.SS2.p1.17.m8.2.2.2.1.1.3.3.cmml">l</mi></msub></mrow><mo stretchy="false" id="S4.SS2.p1.17.m8.2.2.2.1.3" xref="S4.SS2.p1.17.m8.2.2.2.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.p1.17.m8.2b"><apply id="S4.SS2.p1.17.m8.2.2.cmml" xref="S4.SS2.p1.17.m8.2.2"><divide id="S4.SS2.p1.17.m8.2.2.3.cmml" xref="S4.SS2.p1.17.m8.2.2.3"></divide><apply id="S4.SS2.p1.17.m8.1.1.1.1.1.cmml" xref="S4.SS2.p1.17.m8.1.1.1.1"><minus id="S4.SS2.p1.17.m8.1.1.1.1.1.1.cmml" xref="S4.SS2.p1.17.m8.1.1.1.1.1.1"></minus><ci id="S4.SS2.p1.17.m8.1.1.1.1.1.2.cmml" xref="S4.SS2.p1.17.m8.1.1.1.1.1.2">𝑦</ci><apply id="S4.SS2.p1.17.m8.1.1.1.1.1.3.cmml" xref="S4.SS2.p1.17.m8.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S4.SS2.p1.17.m8.1.1.1.1.1.3.1.cmml" xref="S4.SS2.p1.17.m8.1.1.1.1.1.3">subscript</csymbol><ci id="S4.SS2.p1.17.m8.1.1.1.1.1.3.2.cmml" xref="S4.SS2.p1.17.m8.1.1.1.1.1.3.2">𝑣</ci><ci id="S4.SS2.p1.17.m8.1.1.1.1.1.3.3.cmml" xref="S4.SS2.p1.17.m8.1.1.1.1.1.3.3">𝑙</ci></apply></apply><apply id="S4.SS2.p1.17.m8.2.2.2.1.1.cmml" xref="S4.SS2.p1.17.m8.2.2.2.1"><minus id="S4.SS2.p1.17.m8.2.2.2.1.1.1.cmml" xref="S4.SS2.p1.17.m8.2.2.2.1.1.1"></minus><apply id="S4.SS2.p1.17.m8.2.2.2.1.1.2.cmml" xref="S4.SS2.p1.17.m8.2.2.2.1.1.2"><csymbol cd="ambiguous" id="S4.SS2.p1.17.m8.2.2.2.1.1.2.1.cmml" xref="S4.SS2.p1.17.m8.2.2.2.1.1.2">subscript</csymbol><ci id="S4.SS2.p1.17.m8.2.2.2.1.1.2.2.cmml" xref="S4.SS2.p1.17.m8.2.2.2.1.1.2.2">𝑣</ci><ci id="S4.SS2.p1.17.m8.2.2.2.1.1.2.3.cmml" xref="S4.SS2.p1.17.m8.2.2.2.1.1.2.3">𝑢</ci></apply><apply id="S4.SS2.p1.17.m8.2.2.2.1.1.3.cmml" xref="S4.SS2.p1.17.m8.2.2.2.1.1.3"><csymbol cd="ambiguous" id="S4.SS2.p1.17.m8.2.2.2.1.1.3.1.cmml" xref="S4.SS2.p1.17.m8.2.2.2.1.1.3">subscript</csymbol><ci id="S4.SS2.p1.17.m8.2.2.2.1.1.3.2.cmml" xref="S4.SS2.p1.17.m8.2.2.2.1.1.3.2">𝑣</ci><ci id="S4.SS2.p1.17.m8.2.2.2.1.1.3.3.cmml" xref="S4.SS2.p1.17.m8.2.2.2.1.1.3.3">𝑙</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.p1.17.m8.2c">(y-v_{l})/(v_{u}-v_{l})</annotation></semantics></math>.</p>
</div>
<figure id="S4.T2" class="ltx_table">
<div id="S4.T2.17" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:411.9pt;height:455.1pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(75.6pt,-83.6pt) scale(1.58022017812047,1.58022017812047) ;">
<table id="S4.T2.17.17" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T2.17.17.18.1" class="ltx_tr">
<th id="S4.T2.17.17.18.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt"><span id="S4.T2.17.17.18.1.1.1" class="ltx_text ltx_font_bold">Template</span></th>
<td id="S4.T2.17.17.18.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T2.17.17.18.1.2.1" class="ltx_text ltx_font_bold">Label words</span></td>
<td id="S4.T2.17.17.18.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="S4.T2.17.17.18.1.3.1" class="ltx_text ltx_font_bold">Accuracy</span></td>
</tr>
<tr id="S4.T2.17.17.19.2" class="ltx_tr">
<th id="S4.T2.17.17.19.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" colspan="2">SST-2 (positive/negative)</th>
<td id="S4.T2.17.17.19.2.2" class="ltx_td ltx_align_center ltx_border_t">mean (std)</td>
</tr>
<tr id="S4.T2.1.1.1" class="ltx_tr">
<th id="S4.T2.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">
<span id="S4.T2.1.1.1.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S4.T2.1.1.1.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S4.T2.1.1.1.1.1.m1.1a"><msub id="S4.T2.1.1.1.1.1.m1.1.1" xref="S4.T2.1.1.1.1.1.m1.1.1.cmml"><mi id="S4.T2.1.1.1.1.1.m1.1.1.2" xref="S4.T2.1.1.1.1.1.m1.1.1.2.cmml">S</mi><mn id="S4.T2.1.1.1.1.1.m1.1.1.3" xref="S4.T2.1.1.1.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T2.1.1.1.1.1.m1.1b"><apply id="S4.T2.1.1.1.1.1.m1.1.1.cmml" xref="S4.T2.1.1.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.1.1.1.1.1.m1.1.1.1.cmml" xref="S4.T2.1.1.1.1.1.m1.1.1">subscript</csymbol><ci id="S4.T2.1.1.1.1.1.m1.1.1.2.cmml" xref="S4.T2.1.1.1.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S4.T2.1.1.1.1.1.m1.1.1.3.cmml" xref="S4.T2.1.1.1.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.1.1.1.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> It was <span id="S4.T2.1.1.1.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</th>
<td id="S4.T2.1.1.1.2" class="ltx_td ltx_align_center ltx_border_t">great/terrible</td>
<td id="S4.T2.1.1.1.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T2.1.1.1.3.1" class="ltx_text ltx_font_bold">92.7 (0.9)</span></td>
</tr>
<tr id="S4.T2.2.2.2" class="ltx_tr">
<th id="S4.T2.2.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T2.2.2.2.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S4.T2.2.2.2.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S4.T2.2.2.2.1.1.m1.1a"><msub id="S4.T2.2.2.2.1.1.m1.1.1" xref="S4.T2.2.2.2.1.1.m1.1.1.cmml"><mi id="S4.T2.2.2.2.1.1.m1.1.1.2" xref="S4.T2.2.2.2.1.1.m1.1.1.2.cmml">S</mi><mn id="S4.T2.2.2.2.1.1.m1.1.1.3" xref="S4.T2.2.2.2.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T2.2.2.2.1.1.m1.1b"><apply id="S4.T2.2.2.2.1.1.m1.1.1.cmml" xref="S4.T2.2.2.2.1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.2.2.2.1.1.m1.1.1.1.cmml" xref="S4.T2.2.2.2.1.1.m1.1.1">subscript</csymbol><ci id="S4.T2.2.2.2.1.1.m1.1.1.2.cmml" xref="S4.T2.2.2.2.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S4.T2.2.2.2.1.1.m1.1.1.3.cmml" xref="S4.T2.2.2.2.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.2.2.2.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> It was <span id="S4.T2.2.2.2.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</th>
<td id="S4.T2.2.2.2.2" class="ltx_td ltx_align_center">good/bad</td>
<td id="S4.T2.2.2.2.3" class="ltx_td ltx_align_center">92.5 (1.0)</td>
</tr>
<tr id="S4.T2.3.3.3" class="ltx_tr">
<th id="S4.T2.3.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T2.3.3.3.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S4.T2.3.3.3.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S4.T2.3.3.3.1.1.m1.1a"><msub id="S4.T2.3.3.3.1.1.m1.1.1" xref="S4.T2.3.3.3.1.1.m1.1.1.cmml"><mi id="S4.T2.3.3.3.1.1.m1.1.1.2" xref="S4.T2.3.3.3.1.1.m1.1.1.2.cmml">S</mi><mn id="S4.T2.3.3.3.1.1.m1.1.1.3" xref="S4.T2.3.3.3.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T2.3.3.3.1.1.m1.1b"><apply id="S4.T2.3.3.3.1.1.m1.1.1.cmml" xref="S4.T2.3.3.3.1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.3.3.3.1.1.m1.1.1.1.cmml" xref="S4.T2.3.3.3.1.1.m1.1.1">subscript</csymbol><ci id="S4.T2.3.3.3.1.1.m1.1.1.2.cmml" xref="S4.T2.3.3.3.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S4.T2.3.3.3.1.1.m1.1.1.3.cmml" xref="S4.T2.3.3.3.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.3.3.3.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> It was <span id="S4.T2.3.3.3.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</th>
<td id="S4.T2.3.3.3.2" class="ltx_td ltx_align_center">cat/dog</td>
<td id="S4.T2.3.3.3.3" class="ltx_td ltx_align_center">91.5 (1.4)</td>
</tr>
<tr id="S4.T2.4.4.4" class="ltx_tr">
<th id="S4.T2.4.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T2.4.4.4.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S4.T2.4.4.4.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S4.T2.4.4.4.1.1.m1.1a"><msub id="S4.T2.4.4.4.1.1.m1.1.1" xref="S4.T2.4.4.4.1.1.m1.1.1.cmml"><mi id="S4.T2.4.4.4.1.1.m1.1.1.2" xref="S4.T2.4.4.4.1.1.m1.1.1.2.cmml">S</mi><mn id="S4.T2.4.4.4.1.1.m1.1.1.3" xref="S4.T2.4.4.4.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T2.4.4.4.1.1.m1.1b"><apply id="S4.T2.4.4.4.1.1.m1.1.1.cmml" xref="S4.T2.4.4.4.1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.4.4.4.1.1.m1.1.1.1.cmml" xref="S4.T2.4.4.4.1.1.m1.1.1">subscript</csymbol><ci id="S4.T2.4.4.4.1.1.m1.1.1.2.cmml" xref="S4.T2.4.4.4.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S4.T2.4.4.4.1.1.m1.1.1.3.cmml" xref="S4.T2.4.4.4.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.4.4.4.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> It was <span id="S4.T2.4.4.4.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</th>
<td id="S4.T2.4.4.4.2" class="ltx_td ltx_align_center">dog/cat</td>
<td id="S4.T2.4.4.4.3" class="ltx_td ltx_align_center">86.2 (5.4)</td>
</tr>
<tr id="S4.T2.5.5.5" class="ltx_tr">
<th id="S4.T2.5.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T2.5.5.5.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S4.T2.5.5.5.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S4.T2.5.5.5.1.1.m1.1a"><msub id="S4.T2.5.5.5.1.1.m1.1.1" xref="S4.T2.5.5.5.1.1.m1.1.1.cmml"><mi id="S4.T2.5.5.5.1.1.m1.1.1.2" xref="S4.T2.5.5.5.1.1.m1.1.1.2.cmml">S</mi><mn id="S4.T2.5.5.5.1.1.m1.1.1.3" xref="S4.T2.5.5.5.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T2.5.5.5.1.1.m1.1b"><apply id="S4.T2.5.5.5.1.1.m1.1.1.cmml" xref="S4.T2.5.5.5.1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.5.5.5.1.1.m1.1.1.1.cmml" xref="S4.T2.5.5.5.1.1.m1.1.1">subscript</csymbol><ci id="S4.T2.5.5.5.1.1.m1.1.1.2.cmml" xref="S4.T2.5.5.5.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S4.T2.5.5.5.1.1.m1.1.1.3.cmml" xref="S4.T2.5.5.5.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.5.5.5.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> It was <span id="S4.T2.5.5.5.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</th>
<td id="S4.T2.5.5.5.2" class="ltx_td ltx_align_center">terrible/great</td>
<td id="S4.T2.5.5.5.3" class="ltx_td ltx_align_center">83.2 (6.9)</td>
</tr>
<tr id="S4.T2.17.17.20.3" class="ltx_tr">
<th id="S4.T2.17.17.20.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Fine-tuning</th>
<td id="S4.T2.17.17.20.3.2" class="ltx_td ltx_align_center">-</td>
<td id="S4.T2.17.17.20.3.3" class="ltx_td ltx_align_center">81.4 (3.8)</td>
</tr>
<tr id="S4.T2.17.17.21.4" class="ltx_tr">
<th id="S4.T2.17.17.21.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t" colspan="2">SNLI (entailment/neutral/contradiction)</th>
<td id="S4.T2.17.17.21.4.2" class="ltx_td ltx_align_center ltx_border_t">mean (std)</td>
</tr>
<tr id="S4.T2.7.7.7" class="ltx_tr">
<th id="S4.T2.7.7.7.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">
<span id="S4.T2.6.6.6.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S4.T2.6.6.6.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S4.T2.6.6.6.1.1.m1.1a"><msub id="S4.T2.6.6.6.1.1.m1.1.1" xref="S4.T2.6.6.6.1.1.m1.1.1.cmml"><mi id="S4.T2.6.6.6.1.1.m1.1.1.2" xref="S4.T2.6.6.6.1.1.m1.1.1.2.cmml">S</mi><mn id="S4.T2.6.6.6.1.1.m1.1.1.3" xref="S4.T2.6.6.6.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T2.6.6.6.1.1.m1.1b"><apply id="S4.T2.6.6.6.1.1.m1.1.1.cmml" xref="S4.T2.6.6.6.1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.6.6.6.1.1.m1.1.1.1.cmml" xref="S4.T2.6.6.6.1.1.m1.1.1">subscript</csymbol><ci id="S4.T2.6.6.6.1.1.m1.1.1.2.cmml" xref="S4.T2.6.6.6.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S4.T2.6.6.6.1.1.m1.1.1.3.cmml" xref="S4.T2.6.6.6.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.6.6.6.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> ? <span id="S4.T2.7.7.7.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , <span id="S4.T2.7.7.7.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="S4.T2.7.7.7.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="S4.T2.7.7.7.2.2.m1.1a"><msub id="S4.T2.7.7.7.2.2.m1.1.1" xref="S4.T2.7.7.7.2.2.m1.1.1.cmml"><mi id="S4.T2.7.7.7.2.2.m1.1.1.2" xref="S4.T2.7.7.7.2.2.m1.1.1.2.cmml">S</mi><mn id="S4.T2.7.7.7.2.2.m1.1.1.3" xref="S4.T2.7.7.7.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T2.7.7.7.2.2.m1.1b"><apply id="S4.T2.7.7.7.2.2.m1.1.1.cmml" xref="S4.T2.7.7.7.2.2.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.7.7.7.2.2.m1.1.1.1.cmml" xref="S4.T2.7.7.7.2.2.m1.1.1">subscript</csymbol><ci id="S4.T2.7.7.7.2.2.m1.1.1.2.cmml" xref="S4.T2.7.7.7.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="S4.T2.7.7.7.2.2.m1.1.1.3.cmml" xref="S4.T2.7.7.7.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.7.7.7.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</th>
<td id="S4.T2.7.7.7.3" class="ltx_td ltx_align_center ltx_border_t">Yes/Maybe/No</td>
<td id="S4.T2.7.7.7.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S4.T2.7.7.7.4.1" class="ltx_text ltx_font_bold">77.2 (3.7)</span></td>
</tr>
<tr id="S4.T2.9.9.9" class="ltx_tr">
<th id="S4.T2.9.9.9.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T2.8.8.8.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S4.T2.8.8.8.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S4.T2.8.8.8.1.1.m1.1a"><msub id="S4.T2.8.8.8.1.1.m1.1.1" xref="S4.T2.8.8.8.1.1.m1.1.1.cmml"><mi id="S4.T2.8.8.8.1.1.m1.1.1.2" xref="S4.T2.8.8.8.1.1.m1.1.1.2.cmml">S</mi><mn id="S4.T2.8.8.8.1.1.m1.1.1.3" xref="S4.T2.8.8.8.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T2.8.8.8.1.1.m1.1b"><apply id="S4.T2.8.8.8.1.1.m1.1.1.cmml" xref="S4.T2.8.8.8.1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.8.8.8.1.1.m1.1.1.1.cmml" xref="S4.T2.8.8.8.1.1.m1.1.1">subscript</csymbol><ci id="S4.T2.8.8.8.1.1.m1.1.1.2.cmml" xref="S4.T2.8.8.8.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S4.T2.8.8.8.1.1.m1.1.1.3.cmml" xref="S4.T2.8.8.8.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.8.8.8.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="S4.T2.9.9.9.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , <span id="S4.T2.9.9.9.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="S4.T2.9.9.9.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="S4.T2.9.9.9.2.2.m1.1a"><msub id="S4.T2.9.9.9.2.2.m1.1.1" xref="S4.T2.9.9.9.2.2.m1.1.1.cmml"><mi id="S4.T2.9.9.9.2.2.m1.1.1.2" xref="S4.T2.9.9.9.2.2.m1.1.1.2.cmml">S</mi><mn id="S4.T2.9.9.9.2.2.m1.1.1.3" xref="S4.T2.9.9.9.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T2.9.9.9.2.2.m1.1b"><apply id="S4.T2.9.9.9.2.2.m1.1.1.cmml" xref="S4.T2.9.9.9.2.2.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.9.9.9.2.2.m1.1.1.1.cmml" xref="S4.T2.9.9.9.2.2.m1.1.1">subscript</csymbol><ci id="S4.T2.9.9.9.2.2.m1.1.1.2.cmml" xref="S4.T2.9.9.9.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="S4.T2.9.9.9.2.2.m1.1.1.3.cmml" xref="S4.T2.9.9.9.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.9.9.9.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</th>
<td id="S4.T2.9.9.9.3" class="ltx_td ltx_align_center">Yes/Maybe/No</td>
<td id="S4.T2.9.9.9.4" class="ltx_td ltx_align_center">76.2 (3.3)</td>
</tr>
<tr id="S4.T2.11.11.11" class="ltx_tr">
<th id="S4.T2.11.11.11.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T2.10.10.10.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S4.T2.10.10.10.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S4.T2.10.10.10.1.1.m1.1a"><msub id="S4.T2.10.10.10.1.1.m1.1.1" xref="S4.T2.10.10.10.1.1.m1.1.1.cmml"><mi id="S4.T2.10.10.10.1.1.m1.1.1.2" xref="S4.T2.10.10.10.1.1.m1.1.1.2.cmml">S</mi><mn id="S4.T2.10.10.10.1.1.m1.1.1.3" xref="S4.T2.10.10.10.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T2.10.10.10.1.1.m1.1b"><apply id="S4.T2.10.10.10.1.1.m1.1.1.cmml" xref="S4.T2.10.10.10.1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.10.10.10.1.1.m1.1.1.1.cmml" xref="S4.T2.10.10.10.1.1.m1.1.1">subscript</csymbol><ci id="S4.T2.10.10.10.1.1.m1.1.1.2.cmml" xref="S4.T2.10.10.10.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S4.T2.10.10.10.1.1.m1.1.1.3.cmml" xref="S4.T2.10.10.10.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.10.10.10.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> ? <span id="S4.T2.11.11.11.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> <span id="S4.T2.11.11.11.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="S4.T2.11.11.11.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="S4.T2.11.11.11.2.2.m1.1a"><msub id="S4.T2.11.11.11.2.2.m1.1.1" xref="S4.T2.11.11.11.2.2.m1.1.1.cmml"><mi id="S4.T2.11.11.11.2.2.m1.1.1.2" xref="S4.T2.11.11.11.2.2.m1.1.1.2.cmml">S</mi><mn id="S4.T2.11.11.11.2.2.m1.1.1.3" xref="S4.T2.11.11.11.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T2.11.11.11.2.2.m1.1b"><apply id="S4.T2.11.11.11.2.2.m1.1.1.cmml" xref="S4.T2.11.11.11.2.2.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.11.11.11.2.2.m1.1.1.1.cmml" xref="S4.T2.11.11.11.2.2.m1.1.1">subscript</csymbol><ci id="S4.T2.11.11.11.2.2.m1.1.1.2.cmml" xref="S4.T2.11.11.11.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="S4.T2.11.11.11.2.2.m1.1.1.3.cmml" xref="S4.T2.11.11.11.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.11.11.11.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</th>
<td id="S4.T2.11.11.11.3" class="ltx_td ltx_align_center">Yes/Maybe/No</td>
<td id="S4.T2.11.11.11.4" class="ltx_td ltx_align_center">74.9 (3.0)</td>
</tr>
<tr id="S4.T2.13.13.13" class="ltx_tr">
<th id="S4.T2.13.13.13.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T2.12.12.12.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S4.T2.12.12.12.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S4.T2.12.12.12.1.1.m1.1a"><msub id="S4.T2.12.12.12.1.1.m1.1.1" xref="S4.T2.12.12.12.1.1.m1.1.1.cmml"><mi id="S4.T2.12.12.12.1.1.m1.1.1.2" xref="S4.T2.12.12.12.1.1.m1.1.1.2.cmml">S</mi><mn id="S4.T2.12.12.12.1.1.m1.1.1.3" xref="S4.T2.12.12.12.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T2.12.12.12.1.1.m1.1b"><apply id="S4.T2.12.12.12.1.1.m1.1.1.cmml" xref="S4.T2.12.12.12.1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.12.12.12.1.1.m1.1.1.1.cmml" xref="S4.T2.12.12.12.1.1.m1.1.1">subscript</csymbol><ci id="S4.T2.12.12.12.1.1.m1.1.1.2.cmml" xref="S4.T2.12.12.12.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S4.T2.12.12.12.1.1.m1.1.1.3.cmml" xref="S4.T2.12.12.12.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.12.12.12.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> <span id="S4.T2.13.13.13.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="S4.T2.13.13.13.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="S4.T2.13.13.13.2.2.m1.1a"><msub id="S4.T2.13.13.13.2.2.m1.1.1" xref="S4.T2.13.13.13.2.2.m1.1.1.cmml"><mi id="S4.T2.13.13.13.2.2.m1.1.1.2" xref="S4.T2.13.13.13.2.2.m1.1.1.2.cmml">S</mi><mn id="S4.T2.13.13.13.2.2.m1.1.1.3" xref="S4.T2.13.13.13.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T2.13.13.13.2.2.m1.1b"><apply id="S4.T2.13.13.13.2.2.m1.1.1.cmml" xref="S4.T2.13.13.13.2.2.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.13.13.13.2.2.m1.1.1.1.cmml" xref="S4.T2.13.13.13.2.2.m1.1.1">subscript</csymbol><ci id="S4.T2.13.13.13.2.2.m1.1.1.2.cmml" xref="S4.T2.13.13.13.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="S4.T2.13.13.13.2.2.m1.1.1.3.cmml" xref="S4.T2.13.13.13.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.13.13.13.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span> <span id="S4.T2.13.13.13.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span>
</th>
<td id="S4.T2.13.13.13.3" class="ltx_td ltx_align_center">Yes/Maybe/No</td>
<td id="S4.T2.13.13.13.4" class="ltx_td ltx_align_center">65.8 (2.4)</td>
</tr>
<tr id="S4.T2.15.15.15" class="ltx_tr">
<th id="S4.T2.15.15.15.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T2.14.14.14.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S4.T2.14.14.14.1.1.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="S4.T2.14.14.14.1.1.m1.1a"><msub id="S4.T2.14.14.14.1.1.m1.1.1" xref="S4.T2.14.14.14.1.1.m1.1.1.cmml"><mi id="S4.T2.14.14.14.1.1.m1.1.1.2" xref="S4.T2.14.14.14.1.1.m1.1.1.2.cmml">S</mi><mn id="S4.T2.14.14.14.1.1.m1.1.1.3" xref="S4.T2.14.14.14.1.1.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T2.14.14.14.1.1.m1.1b"><apply id="S4.T2.14.14.14.1.1.m1.1.1.cmml" xref="S4.T2.14.14.14.1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.14.14.14.1.1.m1.1.1.1.cmml" xref="S4.T2.14.14.14.1.1.m1.1.1">subscript</csymbol><ci id="S4.T2.14.14.14.1.1.m1.1.1.2.cmml" xref="S4.T2.14.14.14.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S4.T2.14.14.14.1.1.m1.1.1.3.cmml" xref="S4.T2.14.14.14.1.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.14.14.14.1.1.m1.1c">S_{2}</annotation></semantics></math>&gt;</span> ? <span id="S4.T2.15.15.15.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , <span id="S4.T2.15.15.15.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="S4.T2.15.15.15.2.2.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S4.T2.15.15.15.2.2.m1.1a"><msub id="S4.T2.15.15.15.2.2.m1.1.1" xref="S4.T2.15.15.15.2.2.m1.1.1.cmml"><mi id="S4.T2.15.15.15.2.2.m1.1.1.2" xref="S4.T2.15.15.15.2.2.m1.1.1.2.cmml">S</mi><mn id="S4.T2.15.15.15.2.2.m1.1.1.3" xref="S4.T2.15.15.15.2.2.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T2.15.15.15.2.2.m1.1b"><apply id="S4.T2.15.15.15.2.2.m1.1.1.cmml" xref="S4.T2.15.15.15.2.2.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.15.15.15.2.2.m1.1.1.1.cmml" xref="S4.T2.15.15.15.2.2.m1.1.1">subscript</csymbol><ci id="S4.T2.15.15.15.2.2.m1.1.1.2.cmml" xref="S4.T2.15.15.15.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="S4.T2.15.15.15.2.2.m1.1.1.3.cmml" xref="S4.T2.15.15.15.2.2.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.15.15.15.2.2.m1.1c">S_{1}</annotation></semantics></math>&gt;</span>
</th>
<td id="S4.T2.15.15.15.3" class="ltx_td ltx_align_center">Yes/Maybe/No</td>
<td id="S4.T2.15.15.15.4" class="ltx_td ltx_align_center">62.9 (4.1)</td>
</tr>
<tr id="S4.T2.17.17.17" class="ltx_tr">
<th id="S4.T2.17.17.17.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<span id="S4.T2.16.16.16.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S4.T2.16.16.16.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S4.T2.16.16.16.1.1.m1.1a"><msub id="S4.T2.16.16.16.1.1.m1.1.1" xref="S4.T2.16.16.16.1.1.m1.1.1.cmml"><mi id="S4.T2.16.16.16.1.1.m1.1.1.2" xref="S4.T2.16.16.16.1.1.m1.1.1.2.cmml">S</mi><mn id="S4.T2.16.16.16.1.1.m1.1.1.3" xref="S4.T2.16.16.16.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T2.16.16.16.1.1.m1.1b"><apply id="S4.T2.16.16.16.1.1.m1.1.1.cmml" xref="S4.T2.16.16.16.1.1.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.16.16.16.1.1.m1.1.1.1.cmml" xref="S4.T2.16.16.16.1.1.m1.1.1">subscript</csymbol><ci id="S4.T2.16.16.16.1.1.m1.1.1.2.cmml" xref="S4.T2.16.16.16.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S4.T2.16.16.16.1.1.m1.1.1.3.cmml" xref="S4.T2.16.16.16.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.16.16.16.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> ? <span id="S4.T2.17.17.17.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , <span id="S4.T2.17.17.17.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="S4.T2.17.17.17.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="S4.T2.17.17.17.2.2.m1.1a"><msub id="S4.T2.17.17.17.2.2.m1.1.1" xref="S4.T2.17.17.17.2.2.m1.1.1.cmml"><mi id="S4.T2.17.17.17.2.2.m1.1.1.2" xref="S4.T2.17.17.17.2.2.m1.1.1.2.cmml">S</mi><mn id="S4.T2.17.17.17.2.2.m1.1.1.3" xref="S4.T2.17.17.17.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S4.T2.17.17.17.2.2.m1.1b"><apply id="S4.T2.17.17.17.2.2.m1.1.1.cmml" xref="S4.T2.17.17.17.2.2.m1.1.1"><csymbol cd="ambiguous" id="S4.T2.17.17.17.2.2.m1.1.1.1.cmml" xref="S4.T2.17.17.17.2.2.m1.1.1">subscript</csymbol><ci id="S4.T2.17.17.17.2.2.m1.1.1.2.cmml" xref="S4.T2.17.17.17.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="S4.T2.17.17.17.2.2.m1.1.1.3.cmml" xref="S4.T2.17.17.17.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.17.17.17.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</th>
<td id="S4.T2.17.17.17.3" class="ltx_td ltx_align_center">Maybe/No/Yes</td>
<td id="S4.T2.17.17.17.4" class="ltx_td ltx_align_center">60.6 (4.8)</td>
</tr>
<tr id="S4.T2.17.17.22.5" class="ltx_tr">
<th id="S4.T2.17.17.22.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Fine-tuning</th>
<td id="S4.T2.17.17.22.5.2" class="ltx_td ltx_align_center ltx_border_bb">-</td>
<td id="S4.T2.17.17.22.5.3" class="ltx_td ltx_align_center ltx_border_bb">48.4 (4.8)</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>The impact of templates and label words on prompt-based fine-tuning (<math id="S4.T2.19.m1.1" class="ltx_Math" alttext="K=16" display="inline"><semantics id="S4.T2.19.m1.1b"><mrow id="S4.T2.19.m1.1.1" xref="S4.T2.19.m1.1.1.cmml"><mi id="S4.T2.19.m1.1.1.2" xref="S4.T2.19.m1.1.1.2.cmml">K</mi><mo id="S4.T2.19.m1.1.1.1" xref="S4.T2.19.m1.1.1.1.cmml">=</mo><mn id="S4.T2.19.m1.1.1.3" xref="S4.T2.19.m1.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.T2.19.m1.1c"><apply id="S4.T2.19.m1.1.1.cmml" xref="S4.T2.19.m1.1.1"><eq id="S4.T2.19.m1.1.1.1.cmml" xref="S4.T2.19.m1.1.1.1"></eq><ci id="S4.T2.19.m1.1.1.2.cmml" xref="S4.T2.19.m1.1.1.2">𝐾</ci><cn type="integer" id="S4.T2.19.m1.1.1.3.cmml" xref="S4.T2.19.m1.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.T2.19.m1.1d">K=16</annotation></semantics></math>).
</figcaption>
</figure>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3 </span>Manual prompts: the good and the bad</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.3" class="ltx_p">The key challenge is to construct the template <math id="S4.SS3.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S4.SS3.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S4.SS3.p1.1.m1.1.1" xref="S4.SS3.p1.1.m1.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.1.m1.1b"><ci id="S4.SS3.p1.1.m1.1.1.cmml" xref="S4.SS3.p1.1.m1.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.1.m1.1c">\mathcal{T}</annotation></semantics></math> and label words <math id="S4.SS3.p1.2.m2.1" class="ltx_Math" alttext="\mathcal{M}(\mathcal{Y})" display="inline"><semantics id="S4.SS3.p1.2.m2.1a"><mrow id="S4.SS3.p1.2.m2.1.2" xref="S4.SS3.p1.2.m2.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S4.SS3.p1.2.m2.1.2.2" xref="S4.SS3.p1.2.m2.1.2.2.cmml">ℳ</mi><mo lspace="0em" rspace="0em" id="S4.SS3.p1.2.m2.1.2.1" xref="S4.SS3.p1.2.m2.1.2.1.cmml">​</mo><mrow id="S4.SS3.p1.2.m2.1.2.3.2" xref="S4.SS3.p1.2.m2.1.2.cmml"><mo stretchy="false" id="S4.SS3.p1.2.m2.1.2.3.2.1" xref="S4.SS3.p1.2.m2.1.2.cmml">(</mo><mi class="ltx_font_mathcaligraphic" id="S4.SS3.p1.2.m2.1.1" xref="S4.SS3.p1.2.m2.1.1.cmml">𝒴</mi><mo stretchy="false" id="S4.SS3.p1.2.m2.1.2.3.2.2" xref="S4.SS3.p1.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.2.m2.1b"><apply id="S4.SS3.p1.2.m2.1.2.cmml" xref="S4.SS3.p1.2.m2.1.2"><times id="S4.SS3.p1.2.m2.1.2.1.cmml" xref="S4.SS3.p1.2.m2.1.2.1"></times><ci id="S4.SS3.p1.2.m2.1.2.2.cmml" xref="S4.SS3.p1.2.m2.1.2.2">ℳ</ci><ci id="S4.SS3.p1.2.m2.1.1.cmml" xref="S4.SS3.p1.2.m2.1.1">𝒴</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.2.m2.1c">\mathcal{M}(\mathcal{Y})</annotation></semantics></math>—we refer to these two together as a <span id="S4.SS3.p1.3.1" class="ltx_text ltx_font_italic">prompt</span> <math id="S4.SS3.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{P}" display="inline"><semantics id="S4.SS3.p1.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="S4.SS3.p1.3.m3.1.1" xref="S4.SS3.p1.3.m3.1.1.cmml">𝒫</mi><annotation-xml encoding="MathML-Content" id="S4.SS3.p1.3.m3.1b"><ci id="S4.SS3.p1.3.m3.1.1.cmml" xref="S4.SS3.p1.3.m3.1.1">𝒫</ci></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p1.3.m3.1c">\mathcal{P}</annotation></semantics></math>.
Previous works <cite class="ltx_cite ltx_citemacro_cite">Schick and Schütze (<a href="#bib.bib35" title="" class="ltx_ref">2021a</a>, <a href="#bib.bib36" title="" class="ltx_ref">b</a>)</cite> hand-craft both the templates and label words, which usually requires domain expertise and trial-and-error.
Table <a href="#S2.T1" title="Table 1 ‣ Few-shot learning. ‣ 2 Related Work ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>
summarizes manual templates and label words chosen for each dataset in our experiments.
These templates and label words were designed by intuition, and by considering formats used in previous literature.</p>
</div>
<div id="S4.SS3.p2" class="ltx_para">
<p id="S4.SS3.p2.3" class="ltx_p">To better understand what constitutes a good template or label word, we conduct a pilot study on SST-2 and SNLI.
Table <a href="#S4.T2" title="Table 2 ‣ 4.2 Regression ‣ 4 Prompt-based Fine-tuning ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> shows that different prompts can lead to substantial differences in final accuracy.
Specifically, when a template is fixed, the better the label words match the “semantic classes”,
the better the final accuracy is
(<span id="S4.SS3.p2.3.1" class="ltx_text ltx_font_italic">great</span>/<span id="S4.SS3.p2.3.2" class="ltx_text ltx_font_italic">terrible</span> <math id="S4.SS3.p2.1.m1.1" class="ltx_Math" alttext="&gt;" display="inline"><semantics id="S4.SS3.p2.1.m1.1a"><mo id="S4.SS3.p2.1.m1.1.1" xref="S4.SS3.p2.1.m1.1.1.cmml">&gt;</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.1.m1.1b"><gt id="S4.SS3.p2.1.m1.1.1.cmml" xref="S4.SS3.p2.1.m1.1.1"></gt></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.1.m1.1c">&gt;</annotation></semantics></math> <span id="S4.SS3.p2.3.3" class="ltx_text ltx_font_italic">good</span>/<span id="S4.SS3.p2.3.4" class="ltx_text ltx_font_italic">bad</span> <math id="S4.SS3.p2.2.m2.1" class="ltx_Math" alttext="&gt;" display="inline"><semantics id="S4.SS3.p2.2.m2.1a"><mo id="S4.SS3.p2.2.m2.1.1" xref="S4.SS3.p2.2.m2.1.1.cmml">&gt;</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.2.m2.1b"><gt id="S4.SS3.p2.2.m2.1.1.cmml" xref="S4.SS3.p2.2.m2.1.1"></gt></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.2.m2.1c">&gt;</annotation></semantics></math> <span id="S4.SS3.p2.3.5" class="ltx_text ltx_font_italic">cat</span>/<span id="S4.SS3.p2.3.6" class="ltx_text ltx_font_italic">dog</span>).
In extreme cases where we swap plausible label words (e.g., <span id="S4.SS3.p2.3.7" class="ltx_text ltx_font_italic">terrible</span>/<span id="S4.SS3.p2.3.8" class="ltx_text ltx_font_italic">great</span>), we achieve the worst overall performance.<span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span>It is unclear, however, why RoBERTa thinks that “cat” is more positive than “dog”. The authors tend to disagree.</span></span></span>
Furthermore, with the same set of label words,
even a small change in the template
can make a difference.
For example, for SNLI, if we put <span id="S4.SS3.p2.3.9" class="ltx_text ltx_font_typewriter">[MASK]</span> at the end, or swap sentence order, we observe a <math id="S4.SS3.p2.3.m3.1" class="ltx_Math" alttext="&gt;" display="inline"><semantics id="S4.SS3.p2.3.m3.1a"><mo id="S4.SS3.p2.3.m3.1.1" xref="S4.SS3.p2.3.m3.1.1.cmml">&gt;</mo><annotation-xml encoding="MathML-Content" id="S4.SS3.p2.3.m3.1b"><gt id="S4.SS3.p2.3.m3.1.1.cmml" xref="S4.SS3.p2.3.m3.1.1"></gt></annotation-xml><annotation encoding="application/x-tex" id="S4.SS3.p2.3.m3.1c">&gt;</annotation></semantics></math>10% drop.
The above evidence clearly underlines the importance of selecting good templates and label words.
Searching for prompts, however, is hard, as the search space can be very large—especially for the template. Even worse, we only have a few examples to use to guide our search, which can easily overfit. We will address these issues next.</p>
</div>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Automatic Prompt Generation</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">We now explore principled ways of automating the search process for label words (§<a href="#S5.SS1" title="5.1 Automatic selection of label words ‣ 5 Automatic Prompt Generation ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.1</span></a>) and templates (§<a href="#S5.SS2" title="5.2 Automatic generation of templates ‣ 5 Automatic Prompt Generation ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.2</span></a>). Our goals are to reduce the human involvement required to design prompts, and to find more optimal settings than those that we manually choose.
Here, we assume a classification task, but the process for regression is analogous.</p>
</div>
<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Automatic selection of label words</h3>

<div id="S5.SS1.p1" class="ltx_para">
<p id="S5.SS1.p1.10" class="ltx_p">We first study how to construct a label word mapping <math id="S5.SS1.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{M}" display="inline"><semantics id="S5.SS1.p1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S5.SS1.p1.1.m1.1.1" xref="S5.SS1.p1.1.m1.1.1.cmml">ℳ</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.1.m1.1b"><ci id="S5.SS1.p1.1.m1.1.1.cmml" xref="S5.SS1.p1.1.m1.1.1">ℳ</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.1.m1.1c">\mathcal{M}</annotation></semantics></math> that maximizes accuracy on <math id="S5.SS1.p1.2.m2.1" class="ltx_Math" alttext="\mathcal{D}_{\text{dev}}" display="inline"><semantics id="S5.SS1.p1.2.m2.1a"><msub id="S5.SS1.p1.2.m2.1.1" xref="S5.SS1.p1.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS1.p1.2.m2.1.1.2" xref="S5.SS1.p1.2.m2.1.1.2.cmml">𝒟</mi><mtext id="S5.SS1.p1.2.m2.1.1.3" xref="S5.SS1.p1.2.m2.1.1.3a.cmml">dev</mtext></msub><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.2.m2.1b"><apply id="S5.SS1.p1.2.m2.1.1.cmml" xref="S5.SS1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S5.SS1.p1.2.m2.1.1.1.cmml" xref="S5.SS1.p1.2.m2.1.1">subscript</csymbol><ci id="S5.SS1.p1.2.m2.1.1.2.cmml" xref="S5.SS1.p1.2.m2.1.1.2">𝒟</ci><ci id="S5.SS1.p1.2.m2.1.1.3a.cmml" xref="S5.SS1.p1.2.m2.1.1.3"><mtext mathsize="70%" id="S5.SS1.p1.2.m2.1.1.3.cmml" xref="S5.SS1.p1.2.m2.1.1.3">dev</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.2.m2.1c">\mathcal{D}_{\text{dev}}</annotation></semantics></math> after fine-tuning, given a fixed template <math id="S5.SS1.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S5.SS1.p1.3.m3.1a"><mi class="ltx_font_mathcaligraphic" id="S5.SS1.p1.3.m3.1.1" xref="S5.SS1.p1.3.m3.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.3.m3.1b"><ci id="S5.SS1.p1.3.m3.1.1.cmml" xref="S5.SS1.p1.3.m3.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.3.m3.1c">\mathcal{T}</annotation></semantics></math>.
Naively searching all possible assignments, however, is (1) generally intractable, as the search space is exponential in the number of classes; and (2) prone to overfitting, as we will tend to uncover spurious correlations given only a few annotations. As a simple solution,
for each class <math id="S5.SS1.p1.4.m4.1" class="ltx_Math" alttext="c\in\mathcal{Y}" display="inline"><semantics id="S5.SS1.p1.4.m4.1a"><mrow id="S5.SS1.p1.4.m4.1.1" xref="S5.SS1.p1.4.m4.1.1.cmml"><mi id="S5.SS1.p1.4.m4.1.1.2" xref="S5.SS1.p1.4.m4.1.1.2.cmml">c</mi><mo id="S5.SS1.p1.4.m4.1.1.1" xref="S5.SS1.p1.4.m4.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S5.SS1.p1.4.m4.1.1.3" xref="S5.SS1.p1.4.m4.1.1.3.cmml">𝒴</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.4.m4.1b"><apply id="S5.SS1.p1.4.m4.1.1.cmml" xref="S5.SS1.p1.4.m4.1.1"><in id="S5.SS1.p1.4.m4.1.1.1.cmml" xref="S5.SS1.p1.4.m4.1.1.1"></in><ci id="S5.SS1.p1.4.m4.1.1.2.cmml" xref="S5.SS1.p1.4.m4.1.1.2">𝑐</ci><ci id="S5.SS1.p1.4.m4.1.1.3.cmml" xref="S5.SS1.p1.4.m4.1.1.3">𝒴</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.4.m4.1c">c\in\mathcal{Y}</annotation></semantics></math>, we construct a pruned set <math id="S5.SS1.p1.5.m5.1" class="ltx_Math" alttext="\mathcal{V}^{c}\subset\mathcal{V}" display="inline"><semantics id="S5.SS1.p1.5.m5.1a"><mrow id="S5.SS1.p1.5.m5.1.1" xref="S5.SS1.p1.5.m5.1.1.cmml"><msup id="S5.SS1.p1.5.m5.1.1.2" xref="S5.SS1.p1.5.m5.1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS1.p1.5.m5.1.1.2.2" xref="S5.SS1.p1.5.m5.1.1.2.2.cmml">𝒱</mi><mi id="S5.SS1.p1.5.m5.1.1.2.3" xref="S5.SS1.p1.5.m5.1.1.2.3.cmml">c</mi></msup><mo id="S5.SS1.p1.5.m5.1.1.1" xref="S5.SS1.p1.5.m5.1.1.1.cmml">⊂</mo><mi class="ltx_font_mathcaligraphic" id="S5.SS1.p1.5.m5.1.1.3" xref="S5.SS1.p1.5.m5.1.1.3.cmml">𝒱</mi></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.5.m5.1b"><apply id="S5.SS1.p1.5.m5.1.1.cmml" xref="S5.SS1.p1.5.m5.1.1"><subset id="S5.SS1.p1.5.m5.1.1.1.cmml" xref="S5.SS1.p1.5.m5.1.1.1"></subset><apply id="S5.SS1.p1.5.m5.1.1.2.cmml" xref="S5.SS1.p1.5.m5.1.1.2"><csymbol cd="ambiguous" id="S5.SS1.p1.5.m5.1.1.2.1.cmml" xref="S5.SS1.p1.5.m5.1.1.2">superscript</csymbol><ci id="S5.SS1.p1.5.m5.1.1.2.2.cmml" xref="S5.SS1.p1.5.m5.1.1.2.2">𝒱</ci><ci id="S5.SS1.p1.5.m5.1.1.2.3.cmml" xref="S5.SS1.p1.5.m5.1.1.2.3">𝑐</ci></apply><ci id="S5.SS1.p1.5.m5.1.1.3.cmml" xref="S5.SS1.p1.5.m5.1.1.3">𝒱</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.5.m5.1c">\mathcal{V}^{c}\subset\mathcal{V}</annotation></semantics></math> of the top <math id="S5.SS1.p1.6.m6.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.SS1.p1.6.m6.1a"><mi id="S5.SS1.p1.6.m6.1.1" xref="S5.SS1.p1.6.m6.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.6.m6.1b"><ci id="S5.SS1.p1.6.m6.1.1.cmml" xref="S5.SS1.p1.6.m6.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.6.m6.1c">k</annotation></semantics></math> vocabulary words based on their conditional likelihood using the initial <math id="S5.SS1.p1.7.m7.1" class="ltx_Math" alttext="\mathcal{L}" display="inline"><semantics id="S5.SS1.p1.7.m7.1a"><mi class="ltx_font_mathcaligraphic" id="S5.SS1.p1.7.m7.1.1" xref="S5.SS1.p1.7.m7.1.1.cmml">ℒ</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.7.m7.1b"><ci id="S5.SS1.p1.7.m7.1.1.cmml" xref="S5.SS1.p1.7.m7.1.1">ℒ</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.7.m7.1c">\mathcal{L}</annotation></semantics></math>. That is, let <math id="S5.SS1.p1.8.m8.1" class="ltx_Math" alttext="\mathcal{D}_{\text{train}}^{c}\subset\mathcal{D}_{\text{train}}" display="inline"><semantics id="S5.SS1.p1.8.m8.1a"><mrow id="S5.SS1.p1.8.m8.1.1" xref="S5.SS1.p1.8.m8.1.1.cmml"><msubsup id="S5.SS1.p1.8.m8.1.1.2" xref="S5.SS1.p1.8.m8.1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS1.p1.8.m8.1.1.2.2.2" xref="S5.SS1.p1.8.m8.1.1.2.2.2.cmml">𝒟</mi><mtext id="S5.SS1.p1.8.m8.1.1.2.2.3" xref="S5.SS1.p1.8.m8.1.1.2.2.3a.cmml">train</mtext><mi id="S5.SS1.p1.8.m8.1.1.2.3" xref="S5.SS1.p1.8.m8.1.1.2.3.cmml">c</mi></msubsup><mo id="S5.SS1.p1.8.m8.1.1.1" xref="S5.SS1.p1.8.m8.1.1.1.cmml">⊂</mo><msub id="S5.SS1.p1.8.m8.1.1.3" xref="S5.SS1.p1.8.m8.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS1.p1.8.m8.1.1.3.2" xref="S5.SS1.p1.8.m8.1.1.3.2.cmml">𝒟</mi><mtext id="S5.SS1.p1.8.m8.1.1.3.3" xref="S5.SS1.p1.8.m8.1.1.3.3a.cmml">train</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.8.m8.1b"><apply id="S5.SS1.p1.8.m8.1.1.cmml" xref="S5.SS1.p1.8.m8.1.1"><subset id="S5.SS1.p1.8.m8.1.1.1.cmml" xref="S5.SS1.p1.8.m8.1.1.1"></subset><apply id="S5.SS1.p1.8.m8.1.1.2.cmml" xref="S5.SS1.p1.8.m8.1.1.2"><csymbol cd="ambiguous" id="S5.SS1.p1.8.m8.1.1.2.1.cmml" xref="S5.SS1.p1.8.m8.1.1.2">superscript</csymbol><apply id="S5.SS1.p1.8.m8.1.1.2.2.cmml" xref="S5.SS1.p1.8.m8.1.1.2"><csymbol cd="ambiguous" id="S5.SS1.p1.8.m8.1.1.2.2.1.cmml" xref="S5.SS1.p1.8.m8.1.1.2">subscript</csymbol><ci id="S5.SS1.p1.8.m8.1.1.2.2.2.cmml" xref="S5.SS1.p1.8.m8.1.1.2.2.2">𝒟</ci><ci id="S5.SS1.p1.8.m8.1.1.2.2.3a.cmml" xref="S5.SS1.p1.8.m8.1.1.2.2.3"><mtext mathsize="70%" id="S5.SS1.p1.8.m8.1.1.2.2.3.cmml" xref="S5.SS1.p1.8.m8.1.1.2.2.3">train</mtext></ci></apply><ci id="S5.SS1.p1.8.m8.1.1.2.3.cmml" xref="S5.SS1.p1.8.m8.1.1.2.3">𝑐</ci></apply><apply id="S5.SS1.p1.8.m8.1.1.3.cmml" xref="S5.SS1.p1.8.m8.1.1.3"><csymbol cd="ambiguous" id="S5.SS1.p1.8.m8.1.1.3.1.cmml" xref="S5.SS1.p1.8.m8.1.1.3">subscript</csymbol><ci id="S5.SS1.p1.8.m8.1.1.3.2.cmml" xref="S5.SS1.p1.8.m8.1.1.3.2">𝒟</ci><ci id="S5.SS1.p1.8.m8.1.1.3.3a.cmml" xref="S5.SS1.p1.8.m8.1.1.3.3"><mtext mathsize="70%" id="S5.SS1.p1.8.m8.1.1.3.3.cmml" xref="S5.SS1.p1.8.m8.1.1.3.3">train</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.8.m8.1c">\mathcal{D}_{\text{train}}^{c}\subset\mathcal{D}_{\text{train}}</annotation></semantics></math> be the subset of all examples of class <math id="S5.SS1.p1.9.m9.1" class="ltx_Math" alttext="c" display="inline"><semantics id="S5.SS1.p1.9.m9.1a"><mi id="S5.SS1.p1.9.m9.1.1" xref="S5.SS1.p1.9.m9.1.1.cmml">c</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.9.m9.1b"><ci id="S5.SS1.p1.9.m9.1.1.cmml" xref="S5.SS1.p1.9.m9.1.1">𝑐</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.9.m9.1c">c</annotation></semantics></math>. We take <math id="S5.SS1.p1.10.m10.1" class="ltx_Math" alttext="\mathcal{V}^{c}" display="inline"><semantics id="S5.SS1.p1.10.m10.1a"><msup id="S5.SS1.p1.10.m10.1.1" xref="S5.SS1.p1.10.m10.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS1.p1.10.m10.1.1.2" xref="S5.SS1.p1.10.m10.1.1.2.cmml">𝒱</mi><mi id="S5.SS1.p1.10.m10.1.1.3" xref="S5.SS1.p1.10.m10.1.1.3.cmml">c</mi></msup><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.10.m10.1b"><apply id="S5.SS1.p1.10.m10.1.1.cmml" xref="S5.SS1.p1.10.m10.1.1"><csymbol cd="ambiguous" id="S5.SS1.p1.10.m10.1.1.1.cmml" xref="S5.SS1.p1.10.m10.1.1">superscript</csymbol><ci id="S5.SS1.p1.10.m10.1.1.2.cmml" xref="S5.SS1.p1.10.m10.1.1.2">𝒱</ci><ci id="S5.SS1.p1.10.m10.1.1.3.cmml" xref="S5.SS1.p1.10.m10.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.10.m10.1c">\mathcal{V}^{c}</annotation></semantics></math> as</p>
<table id="S5.E3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center">
<div id="S5.E3.m1.1.1.1" class="ltx_inline-block ltx_markedasmath ltx_transformed_outer" style="width:418.1pt;height:43.8pt;vertical-align:-19.9pt;"><span class="ltx_transformed_inner" style="transform:translate(104.2pt,-6.0pt) scale(1.99466299465906,1.99466299465906) ;">
<p id="S5.E3.m1.1.1.1.1" class="ltx_p"><math id="S5.E3.m1.1.1.1.1.m1.1" class="ltx_Math" alttext="\displaystyle\underset{v\in\mathcal{V}}{\mathrm{Top}\text{-}k}\left\{\sum_{{x}_{\mathrm{in}}\in\mathcal{D}_{\text{train}}^{c}}\log P_{\mathcal{L}}\Big{(}\texttt{[MASK]}=v\mid\mathcal{T}({x}_{\mathrm{in}})\Big{)}\right\}," display="inline"><semantics id="S5.E3.m1.1.1.1.1.m1.1a"><mrow id="S5.E3.m1.1.1.1.1.m1.1.1.1" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.cmml"><mrow id="S5.E3.m1.1.1.1.1.m1.1.1.1.1" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.cmml"><munder accentunder="true" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.cmml"><mrow id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.cmml"><mi id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.2.cmml">Top</mi><mo lspace="0em" rspace="0em" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.1" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.1.cmml">​</mo><mtext id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.3" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.3a.cmml">-</mtext><mo lspace="0em" rspace="0em" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.1a" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.1.cmml">​</mo><mi id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.4" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.4.cmml">k</mi></mrow><mrow id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.1" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.1.cmml"><mi id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.1.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.1.2.cmml">v</mi><mo id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.1.1" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.1.3" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.1.3.cmml">𝒱</mi></mrow></munder><mo lspace="0em" rspace="0em" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.2.cmml">​</mo><mrow id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.2.cmml"><mo id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.2.cmml">{</mo><mrow id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.cmml"><mstyle displaystyle="true" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.cmml"><munder id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2a" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.cmml"><mo movablelimits="false" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.2.cmml">∑</mo><mrow id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.cmml"><msub id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.2.cmml"><mi id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.2.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.2.2.cmml">x</mi><mi id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.2.3" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.2.3.cmml">in</mi></msub><mo id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.1" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.1.cmml">∈</mo><msubsup id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.2.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.2.2.cmml">𝒟</mi><mtext id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.2.3" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.2.3a.cmml">train</mtext><mi id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.3" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.3.cmml">c</mi></msubsup></mrow></munder></mstyle><mrow id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.cmml"><mrow id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.1" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3a" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.cmml">⁡</mo><msub id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.2.cmml"><mi id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.2.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.2.2.cmml">P</mi><mi class="ltx_font_mathcaligraphic" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.2.3" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.2.3.cmml">ℒ</mi></msub></mrow><mo lspace="0em" rspace="0em" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo maxsize="160%" minsize="160%" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.3a.cmml">[MASK]</mtext><mo id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">=</mo><mrow id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">v</mi><mo id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">∣</mo><mrow id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">𝒯</mi><mo lspace="0em" rspace="0em" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msub id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">in</mi></msub><mo stretchy="false" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><mo maxsize="160%" minsize="160%" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.3" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.3" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.2.cmml">}</mo></mrow></mrow><mo id="S5.E3.m1.1.1.1.1.m1.1.1.1.2" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S5.E3.m1.1.1.1.1.m1.1b"><apply id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1"><times id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.2.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.2"></times><apply id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3"><apply id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.1.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.1"><in id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.1.1.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.1.1"></in><ci id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.1.2.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.1.2">𝑣</ci><ci id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.1.3.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.1.3">𝒱</ci></apply><apply id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2"><times id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.1.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.1"></times><ci id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.2.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.2">Top</ci><ci id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.3a.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.3"><mtext id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.3.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.3">-</mtext></ci><ci id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.4.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.3.2.4">𝑘</ci></apply></apply><set id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.2.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1"><apply id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1"><apply id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2">subscript</csymbol><sum id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.2"></sum><apply id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3"><in id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.1.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.1"></in><apply id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.2.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.2"><csymbol cd="ambiguous" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.2.1.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.2">subscript</csymbol><ci id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.2.2.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.2.2">𝑥</ci><ci id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.2.3.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.2.3">in</ci></apply><apply id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3"><csymbol cd="ambiguous" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.1.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3">superscript</csymbol><apply id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.2.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3"><csymbol cd="ambiguous" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.2.1.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3">subscript</csymbol><ci id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.2.2.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.2.2">𝒟</ci><ci id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.2.3a.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.2.3"><mtext mathsize="50%" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.2.3.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.2.3">train</mtext></ci></apply><ci id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.3.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.2.3.3.3">𝑐</ci></apply></apply></apply><apply id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1"><times id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.2.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.2"></times><apply id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3"><log id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.1"></log><apply id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.2">subscript</csymbol><ci id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.2.2">𝑃</ci><ci id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.2.3.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.3.2.3">ℒ</ci></apply></apply><apply id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1"><eq id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2"></eq><ci id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.3a.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.3"><mtext class="ltx_mathvariant_monospace" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.3">[MASK]</mtext></ci><apply id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="latexml" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.2">conditional</csymbol><ci id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.3">𝑣</ci><apply id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1"><times id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.2"></times><ci id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.3">𝒯</ci><apply id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S5.E3.m1.1.1.1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.1.3">in</ci></apply></apply></apply></apply></apply></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.E3.m1.1.1.1.1.m1.1c">\displaystyle\underset{v\in\mathcal{V}}{\mathrm{Top}\text{-}k}\left\{\sum_{{x}_{\mathrm{in}}\in\mathcal{D}_{\text{train}}^{c}}\log P_{\mathcal{L}}\Big{(}\texttt{[MASK]}=v\mid\mathcal{T}({x}_{\mathrm{in}})\Big{)}\right\},</annotation></semantics></math></p>
</span></div>
</td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="0" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(3)</span></td>
</tr></tbody>
</table>
<p id="S5.SS1.p1.18" class="ltx_p">where <math id="S5.SS1.p1.11.m1.1" class="ltx_Math" alttext="{P}_{\mathcal{L}}" display="inline"><semantics id="S5.SS1.p1.11.m1.1a"><msub id="S5.SS1.p1.11.m1.1.1" xref="S5.SS1.p1.11.m1.1.1.cmml"><mi id="S5.SS1.p1.11.m1.1.1.2" xref="S5.SS1.p1.11.m1.1.1.2.cmml">P</mi><mi class="ltx_font_mathcaligraphic" id="S5.SS1.p1.11.m1.1.1.3" xref="S5.SS1.p1.11.m1.1.1.3.cmml">ℒ</mi></msub><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.11.m1.1b"><apply id="S5.SS1.p1.11.m1.1.1.cmml" xref="S5.SS1.p1.11.m1.1.1"><csymbol cd="ambiguous" id="S5.SS1.p1.11.m1.1.1.1.cmml" xref="S5.SS1.p1.11.m1.1.1">subscript</csymbol><ci id="S5.SS1.p1.11.m1.1.1.2.cmml" xref="S5.SS1.p1.11.m1.1.1.2">𝑃</ci><ci id="S5.SS1.p1.11.m1.1.1.3.cmml" xref="S5.SS1.p1.11.m1.1.1.3">ℒ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.11.m1.1c">{P}_{\mathcal{L}}</annotation></semantics></math> denotes the output probability distribution of <math id="S5.SS1.p1.12.m2.1" class="ltx_Math" alttext="\mathcal{L}" display="inline"><semantics id="S5.SS1.p1.12.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S5.SS1.p1.12.m2.1.1" xref="S5.SS1.p1.12.m2.1.1.cmml">ℒ</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.12.m2.1b"><ci id="S5.SS1.p1.12.m2.1.1.cmml" xref="S5.SS1.p1.12.m2.1.1">ℒ</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.12.m2.1c">\mathcal{L}</annotation></semantics></math>.
To further narrow down the search space, we find the top <math id="S5.SS1.p1.13.m3.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S5.SS1.p1.13.m3.1a"><mi id="S5.SS1.p1.13.m3.1.1" xref="S5.SS1.p1.13.m3.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.13.m3.1b"><ci id="S5.SS1.p1.13.m3.1.1.cmml" xref="S5.SS1.p1.13.m3.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.13.m3.1c">n</annotation></semantics></math> assignments over the pruned space that maximize zero-shot accuracy on <math id="S5.SS1.p1.14.m4.1" class="ltx_Math" alttext="\mathcal{D}_{\text{train}}" display="inline"><semantics id="S5.SS1.p1.14.m4.1a"><msub id="S5.SS1.p1.14.m4.1.1" xref="S5.SS1.p1.14.m4.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS1.p1.14.m4.1.1.2" xref="S5.SS1.p1.14.m4.1.1.2.cmml">𝒟</mi><mtext id="S5.SS1.p1.14.m4.1.1.3" xref="S5.SS1.p1.14.m4.1.1.3a.cmml">train</mtext></msub><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.14.m4.1b"><apply id="S5.SS1.p1.14.m4.1.1.cmml" xref="S5.SS1.p1.14.m4.1.1"><csymbol cd="ambiguous" id="S5.SS1.p1.14.m4.1.1.1.cmml" xref="S5.SS1.p1.14.m4.1.1">subscript</csymbol><ci id="S5.SS1.p1.14.m4.1.1.2.cmml" xref="S5.SS1.p1.14.m4.1.1.2">𝒟</ci><ci id="S5.SS1.p1.14.m4.1.1.3a.cmml" xref="S5.SS1.p1.14.m4.1.1.3"><mtext mathsize="70%" id="S5.SS1.p1.14.m4.1.1.3.cmml" xref="S5.SS1.p1.14.m4.1.1.3">train</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.14.m4.1c">\mathcal{D}_{\text{train}}</annotation></semantics></math> (both <math id="S5.SS1.p1.15.m5.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S5.SS1.p1.15.m5.1a"><mi id="S5.SS1.p1.15.m5.1.1" xref="S5.SS1.p1.15.m5.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.15.m5.1b"><ci id="S5.SS1.p1.15.m5.1.1.cmml" xref="S5.SS1.p1.15.m5.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.15.m5.1c">n</annotation></semantics></math> and <math id="S5.SS1.p1.16.m6.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.SS1.p1.16.m6.1a"><mi id="S5.SS1.p1.16.m6.1.1" xref="S5.SS1.p1.16.m6.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.16.m6.1b"><ci id="S5.SS1.p1.16.m6.1.1.cmml" xref="S5.SS1.p1.16.m6.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.16.m6.1c">k</annotation></semantics></math> are hyper-parameters, see
Appendix <a href="#A3.SS2" title="C.2 Prompt-based fine-tuning ‣ Appendix C Experimental Details ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C.2</span></a>).
Then we fine-tune all top <math id="S5.SS1.p1.17.m7.1" class="ltx_Math" alttext="n" display="inline"><semantics id="S5.SS1.p1.17.m7.1a"><mi id="S5.SS1.p1.17.m7.1.1" xref="S5.SS1.p1.17.m7.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.17.m7.1b"><ci id="S5.SS1.p1.17.m7.1.1.cmml" xref="S5.SS1.p1.17.m7.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.17.m7.1c">n</annotation></semantics></math> assignments, and re-rank to find the best one using <math id="S5.SS1.p1.18.m8.1" class="ltx_Math" alttext="\mathcal{D}_{\text{dev}}" display="inline"><semantics id="S5.SS1.p1.18.m8.1a"><msub id="S5.SS1.p1.18.m8.1.1" xref="S5.SS1.p1.18.m8.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS1.p1.18.m8.1.1.2" xref="S5.SS1.p1.18.m8.1.1.2.cmml">𝒟</mi><mtext id="S5.SS1.p1.18.m8.1.1.3" xref="S5.SS1.p1.18.m8.1.1.3a.cmml">dev</mtext></msub><annotation-xml encoding="MathML-Content" id="S5.SS1.p1.18.m8.1b"><apply id="S5.SS1.p1.18.m8.1.1.cmml" xref="S5.SS1.p1.18.m8.1.1"><csymbol cd="ambiguous" id="S5.SS1.p1.18.m8.1.1.1.cmml" xref="S5.SS1.p1.18.m8.1.1">subscript</csymbol><ci id="S5.SS1.p1.18.m8.1.1.2.cmml" xref="S5.SS1.p1.18.m8.1.1.2">𝒟</ci><ci id="S5.SS1.p1.18.m8.1.1.3a.cmml" xref="S5.SS1.p1.18.m8.1.1.3"><mtext mathsize="70%" id="S5.SS1.p1.18.m8.1.1.3.cmml" xref="S5.SS1.p1.18.m8.1.1.3">dev</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS1.p1.18.m8.1c">\mathcal{D}_{\text{dev}}</annotation></semantics></math>.
This approach is similar to the automatic verbalizer search methods in <cite class="ltx_cite ltx_citemacro_citet">Schick and Schütze (<a href="#bib.bib35" title="" class="ltx_ref">2021a</a>); Schick et al. (<a href="#bib.bib34" title="" class="ltx_ref">2020</a>)</cite>, except that we use a much simpler search process (brute-force) and also apply re-ranking—which we find to be quite helpful.</p>
</div>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Automatic generation of templates</h3>

<div id="S5.SS2.p1" class="ltx_para">
<p id="S5.SS2.p1.4" class="ltx_p">Next, we study how to generate a diverse set of templates <math id="S5.SS2.p1.1.m1.1" class="ltx_Math" alttext="\{\mathcal{T}\}" display="inline"><semantics id="S5.SS2.p1.1.m1.1a"><mrow id="S5.SS2.p1.1.m1.1.2.2" xref="S5.SS2.p1.1.m1.1.2.1.cmml"><mo stretchy="false" id="S5.SS2.p1.1.m1.1.2.2.1" xref="S5.SS2.p1.1.m1.1.2.1.cmml">{</mo><mi class="ltx_font_mathcaligraphic" id="S5.SS2.p1.1.m1.1.1" xref="S5.SS2.p1.1.m1.1.1.cmml">𝒯</mi><mo stretchy="false" id="S5.SS2.p1.1.m1.1.2.2.2" xref="S5.SS2.p1.1.m1.1.2.1.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.p1.1.m1.1b"><set id="S5.SS2.p1.1.m1.1.2.1.cmml" xref="S5.SS2.p1.1.m1.1.2.2"><ci id="S5.SS2.p1.1.m1.1.1.cmml" xref="S5.SS2.p1.1.m1.1.1">𝒯</ci></set></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p1.1.m1.1c">\{\mathcal{T}\}</annotation></semantics></math> automatically from a fixed set of label words <math id="S5.SS2.p1.2.m2.1" class="ltx_Math" alttext="\mathcal{M}(\mathcal{Y})" display="inline"><semantics id="S5.SS2.p1.2.m2.1a"><mrow id="S5.SS2.p1.2.m2.1.2" xref="S5.SS2.p1.2.m2.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS2.p1.2.m2.1.2.2" xref="S5.SS2.p1.2.m2.1.2.2.cmml">ℳ</mi><mo lspace="0em" rspace="0em" id="S5.SS2.p1.2.m2.1.2.1" xref="S5.SS2.p1.2.m2.1.2.1.cmml">​</mo><mrow id="S5.SS2.p1.2.m2.1.2.3.2" xref="S5.SS2.p1.2.m2.1.2.cmml"><mo stretchy="false" id="S5.SS2.p1.2.m2.1.2.3.2.1" xref="S5.SS2.p1.2.m2.1.2.cmml">(</mo><mi class="ltx_font_mathcaligraphic" id="S5.SS2.p1.2.m2.1.1" xref="S5.SS2.p1.2.m2.1.1.cmml">𝒴</mi><mo stretchy="false" id="S5.SS2.p1.2.m2.1.2.3.2.2" xref="S5.SS2.p1.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.p1.2.m2.1b"><apply id="S5.SS2.p1.2.m2.1.2.cmml" xref="S5.SS2.p1.2.m2.1.2"><times id="S5.SS2.p1.2.m2.1.2.1.cmml" xref="S5.SS2.p1.2.m2.1.2.1"></times><ci id="S5.SS2.p1.2.m2.1.2.2.cmml" xref="S5.SS2.p1.2.m2.1.2.2">ℳ</ci><ci id="S5.SS2.p1.2.m2.1.1.cmml" xref="S5.SS2.p1.2.m2.1.1">𝒴</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p1.2.m2.1c">\mathcal{M}(\mathcal{Y})</annotation></semantics></math>.
To address this challenging problem, we propose to use T5 <cite class="ltx_cite ltx_citemacro_cite">Raffel et al. (<a href="#bib.bib31" title="" class="ltx_ref">2020</a>)</cite>, a large pre-trained text-to-text Transformer.
T5 is pre-trained to fill in missing spans (replaced by T5 mask tokens, e.g., <span id="S5.SS2.p1.4.1" class="ltx_text ltx_font_typewriter">&lt;X&gt;</span> or <span id="S5.SS2.p1.4.2" class="ltx_text ltx_font_typewriter">&lt;Y&gt;</span>) in its input.
For example, given the input “<span id="S5.SS2.p1.4.3" class="ltx_text ltx_font_italic">Thank you <span id="S5.SS2.p1.4.3.1" class="ltx_text ltx_font_typewriter">&lt;X&gt;</span> me to your party <span id="S5.SS2.p1.4.3.2" class="ltx_text ltx_font_typewriter">&lt;Y&gt;</span> week</span>”, T5 is trained to generate “<span id="S5.SS2.p1.4.4" class="ltx_text ltx_font_typewriter ltx_font_italic">&lt;X&gt;<span id="S5.SS2.p1.4.4.1" class="ltx_text ltx_font_serif"> for inviting </span>&lt;Y&gt;<span id="S5.SS2.p1.4.4.2" class="ltx_text ltx_font_serif"> last </span>&lt;Z&gt;</span>”,
meaning that “<span id="S5.SS2.p1.4.5" class="ltx_text ltx_font_italic">for inviting</span>” is the replacement for <span id="S5.SS2.p1.4.6" class="ltx_text ltx_font_typewriter">&lt;X&gt;</span> and “<span id="S5.SS2.p1.4.7" class="ltx_text ltx_font_italic">last</span>” is the replacement for <span id="S5.SS2.p1.4.8" class="ltx_text ltx_font_typewriter">&lt;Y&gt;</span>.
This is well suited for prompt generation: we can simply take input sentences from <math id="S5.SS2.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{D}_{\text{train}}" display="inline"><semantics id="S5.SS2.p1.3.m3.1a"><msub id="S5.SS2.p1.3.m3.1.1" xref="S5.SS2.p1.3.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS2.p1.3.m3.1.1.2" xref="S5.SS2.p1.3.m3.1.1.2.cmml">𝒟</mi><mtext id="S5.SS2.p1.3.m3.1.1.3" xref="S5.SS2.p1.3.m3.1.1.3a.cmml">train</mtext></msub><annotation-xml encoding="MathML-Content" id="S5.SS2.p1.3.m3.1b"><apply id="S5.SS2.p1.3.m3.1.1.cmml" xref="S5.SS2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S5.SS2.p1.3.m3.1.1.1.cmml" xref="S5.SS2.p1.3.m3.1.1">subscript</csymbol><ci id="S5.SS2.p1.3.m3.1.1.2.cmml" xref="S5.SS2.p1.3.m3.1.1.2">𝒟</ci><ci id="S5.SS2.p1.3.m3.1.1.3a.cmml" xref="S5.SS2.p1.3.m3.1.1.3"><mtext mathsize="70%" id="S5.SS2.p1.3.m3.1.1.3.cmml" xref="S5.SS2.p1.3.m3.1.1.3">train</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p1.3.m3.1c">\mathcal{D}_{\text{train}}</annotation></semantics></math> and let the T5 model construct the template <math id="S5.SS2.p1.4.m4.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S5.SS2.p1.4.m4.1a"><mi class="ltx_font_mathcaligraphic" id="S5.SS2.p1.4.m4.1.1" xref="S5.SS2.p1.4.m4.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S5.SS2.p1.4.m4.1b"><ci id="S5.SS2.p1.4.m4.1.1.cmml" xref="S5.SS2.p1.4.m4.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p1.4.m4.1c">\mathcal{T}</annotation></semantics></math>, without having to specify a pre-defined number of tokens for it.</p>
</div>
<div id="S5.SS2.p2" class="ltx_para">
<p id="S5.SS2.p2.2" class="ltx_p">Given an input example <math id="S5.SS2.p2.1.m1.2" class="ltx_Math" alttext="({x}_{\mathrm{in}},y)\in\mathcal{D}_{\text{train}}" display="inline"><semantics id="S5.SS2.p2.1.m1.2a"><mrow id="S5.SS2.p2.1.m1.2.2" xref="S5.SS2.p2.1.m1.2.2.cmml"><mrow id="S5.SS2.p2.1.m1.2.2.1.1" xref="S5.SS2.p2.1.m1.2.2.1.2.cmml"><mo stretchy="false" id="S5.SS2.p2.1.m1.2.2.1.1.2" xref="S5.SS2.p2.1.m1.2.2.1.2.cmml">(</mo><msub id="S5.SS2.p2.1.m1.2.2.1.1.1" xref="S5.SS2.p2.1.m1.2.2.1.1.1.cmml"><mi id="S5.SS2.p2.1.m1.2.2.1.1.1.2" xref="S5.SS2.p2.1.m1.2.2.1.1.1.2.cmml">x</mi><mi id="S5.SS2.p2.1.m1.2.2.1.1.1.3" xref="S5.SS2.p2.1.m1.2.2.1.1.1.3.cmml">in</mi></msub><mo id="S5.SS2.p2.1.m1.2.2.1.1.3" xref="S5.SS2.p2.1.m1.2.2.1.2.cmml">,</mo><mi id="S5.SS2.p2.1.m1.1.1" xref="S5.SS2.p2.1.m1.1.1.cmml">y</mi><mo stretchy="false" id="S5.SS2.p2.1.m1.2.2.1.1.4" xref="S5.SS2.p2.1.m1.2.2.1.2.cmml">)</mo></mrow><mo id="S5.SS2.p2.1.m1.2.2.2" xref="S5.SS2.p2.1.m1.2.2.2.cmml">∈</mo><msub id="S5.SS2.p2.1.m1.2.2.3" xref="S5.SS2.p2.1.m1.2.2.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS2.p2.1.m1.2.2.3.2" xref="S5.SS2.p2.1.m1.2.2.3.2.cmml">𝒟</mi><mtext id="S5.SS2.p2.1.m1.2.2.3.3" xref="S5.SS2.p2.1.m1.2.2.3.3a.cmml">train</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.p2.1.m1.2b"><apply id="S5.SS2.p2.1.m1.2.2.cmml" xref="S5.SS2.p2.1.m1.2.2"><in id="S5.SS2.p2.1.m1.2.2.2.cmml" xref="S5.SS2.p2.1.m1.2.2.2"></in><interval closure="open" id="S5.SS2.p2.1.m1.2.2.1.2.cmml" xref="S5.SS2.p2.1.m1.2.2.1.1"><apply id="S5.SS2.p2.1.m1.2.2.1.1.1.cmml" xref="S5.SS2.p2.1.m1.2.2.1.1.1"><csymbol cd="ambiguous" id="S5.SS2.p2.1.m1.2.2.1.1.1.1.cmml" xref="S5.SS2.p2.1.m1.2.2.1.1.1">subscript</csymbol><ci id="S5.SS2.p2.1.m1.2.2.1.1.1.2.cmml" xref="S5.SS2.p2.1.m1.2.2.1.1.1.2">𝑥</ci><ci id="S5.SS2.p2.1.m1.2.2.1.1.1.3.cmml" xref="S5.SS2.p2.1.m1.2.2.1.1.1.3">in</ci></apply><ci id="S5.SS2.p2.1.m1.1.1.cmml" xref="S5.SS2.p2.1.m1.1.1">𝑦</ci></interval><apply id="S5.SS2.p2.1.m1.2.2.3.cmml" xref="S5.SS2.p2.1.m1.2.2.3"><csymbol cd="ambiguous" id="S5.SS2.p2.1.m1.2.2.3.1.cmml" xref="S5.SS2.p2.1.m1.2.2.3">subscript</csymbol><ci id="S5.SS2.p2.1.m1.2.2.3.2.cmml" xref="S5.SS2.p2.1.m1.2.2.3.2">𝒟</ci><ci id="S5.SS2.p2.1.m1.2.2.3.3a.cmml" xref="S5.SS2.p2.1.m1.2.2.3.3"><mtext mathsize="70%" id="S5.SS2.p2.1.m1.2.2.3.3.cmml" xref="S5.SS2.p2.1.m1.2.2.3.3">train</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p2.1.m1.2c">({x}_{\mathrm{in}},y)\in\mathcal{D}_{\text{train}}</annotation></semantics></math>, we consider the following simple conversions, denoted as <math id="S5.SS2.p2.2.m2.2" class="ltx_Math" alttext="\mathcal{T}_{\mathrm{g}}({x}_{\mathrm{in}},y)" display="inline"><semantics id="S5.SS2.p2.2.m2.2a"><mrow id="S5.SS2.p2.2.m2.2.2" xref="S5.SS2.p2.2.m2.2.2.cmml"><msub id="S5.SS2.p2.2.m2.2.2.3" xref="S5.SS2.p2.2.m2.2.2.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS2.p2.2.m2.2.2.3.2" xref="S5.SS2.p2.2.m2.2.2.3.2.cmml">𝒯</mi><mi mathvariant="normal" id="S5.SS2.p2.2.m2.2.2.3.3" xref="S5.SS2.p2.2.m2.2.2.3.3.cmml">g</mi></msub><mo lspace="0em" rspace="0em" id="S5.SS2.p2.2.m2.2.2.2" xref="S5.SS2.p2.2.m2.2.2.2.cmml">​</mo><mrow id="S5.SS2.p2.2.m2.2.2.1.1" xref="S5.SS2.p2.2.m2.2.2.1.2.cmml"><mo stretchy="false" id="S5.SS2.p2.2.m2.2.2.1.1.2" xref="S5.SS2.p2.2.m2.2.2.1.2.cmml">(</mo><msub id="S5.SS2.p2.2.m2.2.2.1.1.1" xref="S5.SS2.p2.2.m2.2.2.1.1.1.cmml"><mi id="S5.SS2.p2.2.m2.2.2.1.1.1.2" xref="S5.SS2.p2.2.m2.2.2.1.1.1.2.cmml">x</mi><mi id="S5.SS2.p2.2.m2.2.2.1.1.1.3" xref="S5.SS2.p2.2.m2.2.2.1.1.1.3.cmml">in</mi></msub><mo id="S5.SS2.p2.2.m2.2.2.1.1.3" xref="S5.SS2.p2.2.m2.2.2.1.2.cmml">,</mo><mi id="S5.SS2.p2.2.m2.1.1" xref="S5.SS2.p2.2.m2.1.1.cmml">y</mi><mo stretchy="false" id="S5.SS2.p2.2.m2.2.2.1.1.4" xref="S5.SS2.p2.2.m2.2.2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.p2.2.m2.2b"><apply id="S5.SS2.p2.2.m2.2.2.cmml" xref="S5.SS2.p2.2.m2.2.2"><times id="S5.SS2.p2.2.m2.2.2.2.cmml" xref="S5.SS2.p2.2.m2.2.2.2"></times><apply id="S5.SS2.p2.2.m2.2.2.3.cmml" xref="S5.SS2.p2.2.m2.2.2.3"><csymbol cd="ambiguous" id="S5.SS2.p2.2.m2.2.2.3.1.cmml" xref="S5.SS2.p2.2.m2.2.2.3">subscript</csymbol><ci id="S5.SS2.p2.2.m2.2.2.3.2.cmml" xref="S5.SS2.p2.2.m2.2.2.3.2">𝒯</ci><ci id="S5.SS2.p2.2.m2.2.2.3.3.cmml" xref="S5.SS2.p2.2.m2.2.2.3.3">g</ci></apply><interval closure="open" id="S5.SS2.p2.2.m2.2.2.1.2.cmml" xref="S5.SS2.p2.2.m2.2.2.1.1"><apply id="S5.SS2.p2.2.m2.2.2.1.1.1.cmml" xref="S5.SS2.p2.2.m2.2.2.1.1.1"><csymbol cd="ambiguous" id="S5.SS2.p2.2.m2.2.2.1.1.1.1.cmml" xref="S5.SS2.p2.2.m2.2.2.1.1.1">subscript</csymbol><ci id="S5.SS2.p2.2.m2.2.2.1.1.1.2.cmml" xref="S5.SS2.p2.2.m2.2.2.1.1.1.2">𝑥</ci><ci id="S5.SS2.p2.2.m2.2.2.1.1.1.3.cmml" xref="S5.SS2.p2.2.m2.2.2.1.1.1.3">in</ci></apply><ci id="S5.SS2.p2.2.m2.1.1.cmml" xref="S5.SS2.p2.2.m2.1.1">𝑦</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p2.2.m2.2c">\mathcal{T}_{\mathrm{g}}({x}_{\mathrm{in}},y)</annotation></semantics></math>, for formulating the T5 model inputs:<span id="footnote7" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">7</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">7</sup><span class="ltx_tag ltx_tag_note">7</span>We consider putting the label word both before and after the input sentence for single-sentence tasks. However, we find that it is always better to put the label words in the middle (between the two sentences) for sentence-pair tasks.</span></span></span></p>
<table id="S5.Ex2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center">
<div id="S5.Ex2.m1.1.1.1" class="ltx_inline-block ltx_markedasmath ltx_transformed_outer" style="width:399.3pt;height:61.4pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(102.2pt,-15.7pt) scale(2.04820794716397,2.04820794716397) ;">
<p id="S5.Ex2.m1.1.1.1.1" class="ltx_p"><math id="S5.Ex2.m1.1.1.1.1.m1.8" class="ltx_Math" alttext="\begin{aligned} \texttt{&lt;}S_{1}\texttt{&gt;}&amp;\longrightarrow~{}\texttt{&lt;X&gt;}~{}\mathcal{M}(y)~{}\texttt{&lt;Y&gt;}~{}\texttt{&lt;}S_{1}\texttt{&gt;},\\
\texttt{&lt;}S_{1}\texttt{&gt;}&amp;\longrightarrow~{}\texttt{&lt;}S_{1}\texttt{&gt;}~{}\texttt{&lt;X&gt;}~{}\mathcal{M}(y)~{}\texttt{&lt;Y&gt;},\\
\texttt{&lt;}S_{1}\texttt{&gt;},\texttt{&lt;}S_{2}\texttt{&gt;}&amp;\longrightarrow\texttt{&lt;}S_{1}\texttt{&gt;}~{}\texttt{&lt;X&gt;}~{}\mathcal{M}(y)~{}\texttt{&lt;Y&gt;}~{}\texttt{&lt;}S_{2}\texttt{&gt;}.\end{aligned}" display="inline"><semantics id="S5.Ex2.m1.1.1.1.1.m1.8a"><mtable columnspacing="0pt" rowspacing="0pt" id="S5.Ex2.m1.1.1.1.1.m1.8.8" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.cmml"><mtr id="S5.Ex2.m1.1.1.1.1.m1.8.8a" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.cmml"><mtd class="ltx_align_right" columnalign="right" id="S5.Ex2.m1.1.1.1.1.m1.8.8b" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.cmml"><mrow id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.cmml"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.2" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.2a.cmml">&lt;</mtext><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.1" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.1.cmml">​</mo><msub id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.3" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.3.cmml"><mi id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.3.2" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.3.2.cmml">S</mi><mn id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.3.3" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.3.3.cmml">1</mn></msub><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.1a" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.4" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.4a.cmml">&gt;</mtext></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S5.Ex2.m1.1.1.1.1.m1.8.8c" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.cmml"><mrow id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.cmml"><mrow id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.cmml"><mi id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.2" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.2.cmml"></mi><mo rspace="0.608em" stretchy="false" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.1" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.1.cmml">⟶</mo><mrow id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.cmml"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.2" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.2a.cmml">&lt;X&gt;</mtext><mo lspace="0.330em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.1" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.1.cmml">​</mo><mi class="ltx_font_mathcaligraphic" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.3" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.3.cmml">ℳ</mi><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.1a" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.1.cmml">​</mo><mrow id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.4.2" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.cmml"><mo stretchy="false" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.4.2.1" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.cmml">(</mo><mi id="S5.Ex2.m1.1.1.1.1.m1.1.1.1.1.1.1" xref="S5.Ex2.m1.1.1.1.1.m1.1.1.1.1.1.1.cmml">y</mi><mo stretchy="false" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.4.2.2" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.cmml">)</mo></mrow><mo lspace="0.330em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.1b" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.5" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.5a.cmml">&lt;Y&gt;</mtext><mo lspace="0.330em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.1c" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.6" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.6a.cmml">&lt;</mtext><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.1d" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.1.cmml">​</mo><msub id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.7" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.7.cmml"><mi id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.7.2" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.7.2.cmml">S</mi><mn id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.7.3" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.7.3.cmml">1</mn></msub><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.1e" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.8" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.8a.cmml">&gt;</mtext></mrow></mrow><mo id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.2" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.cmml">,</mo></mrow></mtd></mtr><mtr id="S5.Ex2.m1.1.1.1.1.m1.8.8d" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.cmml"><mtd class="ltx_align_right" columnalign="right" id="S5.Ex2.m1.1.1.1.1.m1.8.8e" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.cmml"><mrow id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.cmml"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.2" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.2a.cmml">&lt;</mtext><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.1" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.1.cmml">​</mo><msub id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.3" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.3.cmml"><mi id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.3.2" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.3.2.cmml">S</mi><mn id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.3.3" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.3.3.cmml">1</mn></msub><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.1a" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.4" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.4a.cmml">&gt;</mtext></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S5.Ex2.m1.1.1.1.1.m1.8.8f" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.cmml"><mrow id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.cmml"><mrow id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.cmml"><mi id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.2" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.2.cmml"></mi><mo rspace="0.608em" stretchy="false" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.1" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.1.cmml">⟶</mo><mrow id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.cmml"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.2" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.2a.cmml">&lt;</mtext><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.1" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.1.cmml">​</mo><msub id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.3" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.3.cmml"><mi id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.3.2" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.3.2.cmml">S</mi><mn id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.3.3" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.3.3.cmml">1</mn></msub><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.1a" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.4" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.4a.cmml">&gt;</mtext><mo lspace="0.330em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.1b" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.5" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.5a.cmml">&lt;X&gt;</mtext><mo lspace="0.330em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.1c" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.1.cmml">​</mo><mi class="ltx_font_mathcaligraphic" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.6" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.6.cmml">ℳ</mi><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.1d" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.1.cmml">​</mo><mrow id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.7.2" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.cmml"><mo stretchy="false" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.7.2.1" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.cmml">(</mo><mi id="S5.Ex2.m1.1.1.1.1.m1.3.3.3.1.1.1" xref="S5.Ex2.m1.1.1.1.1.m1.3.3.3.1.1.1.cmml">y</mi><mo stretchy="false" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.7.2.2" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.cmml">)</mo></mrow><mo lspace="0.330em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.1e" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.8" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.8a.cmml">&lt;Y&gt;</mtext></mrow></mrow><mo id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.2" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.cmml">,</mo></mrow></mtd></mtr><mtr id="S5.Ex2.m1.1.1.1.1.m1.8.8g" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.cmml"><mtd class="ltx_align_right" columnalign="right" id="S5.Ex2.m1.1.1.1.1.m1.8.8h" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.cmml"><mrow id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.3.cmml"><mrow id="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1" xref="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.2" xref="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.2a.cmml">&lt;</mtext><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.1" xref="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.1.cmml">​</mo><msub id="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.3" xref="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.3.cmml"><mi id="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.3.2" xref="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.3.2.cmml">S</mi><mn id="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.3.3" xref="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.3.3.cmml">1</mn></msub><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.1a" xref="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.4" xref="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.4a.cmml">&gt;</mtext></mrow><mo id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.3" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.3.cmml">,</mo><mrow id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.cmml"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.2" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.2a.cmml">&lt;</mtext><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.1" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.1.cmml">​</mo><msub id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.3" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.3.cmml"><mi id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.3.2" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.3.2.cmml">S</mi><mn id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.3.3" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.3.3.cmml">2</mn></msub><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.1a" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.4" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.4a.cmml">&gt;</mtext></mrow></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S5.Ex2.m1.1.1.1.1.m1.8.8i" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.cmml"><mrow id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.cmml"><mrow id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.cmml"><mi id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.2" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.2.cmml"></mi><mo stretchy="false" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.1" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.1.cmml">⟶</mo><mrow id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.cmml"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.2" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.2a.cmml">&lt;</mtext><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1.cmml">​</mo><msub id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.3" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.3.cmml"><mi id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.3.2" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.3.2.cmml">S</mi><mn id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.3.3" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.3.3.cmml">1</mn></msub><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1a" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.4" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.4a.cmml">&gt;</mtext><mo lspace="0.330em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1b" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.5" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.5a.cmml">&lt;X&gt;</mtext><mo lspace="0.330em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1c" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1.cmml">​</mo><mi class="ltx_font_mathcaligraphic" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.6" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.6.cmml">ℳ</mi><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1d" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1.cmml">​</mo><mrow id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.7.2" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.cmml"><mo stretchy="false" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.7.2.1" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.cmml">(</mo><mi id="S5.Ex2.m1.1.1.1.1.m1.7.7.7.3.1.1" xref="S5.Ex2.m1.1.1.1.1.m1.7.7.7.3.1.1.cmml">y</mi><mo stretchy="false" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.7.2.2" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.cmml">)</mo></mrow><mo lspace="0.330em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1e" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.8" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.8a.cmml">&lt;Y&gt;</mtext><mo lspace="0.330em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1f" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.9" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.9a.cmml">&lt;</mtext><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1g" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1.cmml">​</mo><msub id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.10" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.10.cmml"><mi id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.10.2" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.10.2.cmml">S</mi><mn id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.10.3" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.10.3.cmml">2</mn></msub><mo lspace="0em" rspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1h" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1.cmml">​</mo><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.11" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.11a.cmml">&gt;</mtext></mrow></mrow><mo lspace="0em" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.2" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.cmml">.</mo></mrow></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="S5.Ex2.m1.1.1.1.1.m1.8b"><matrix id="S5.Ex2.m1.1.1.1.1.m1.8.8.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8"><matrixrow id="S5.Ex2.m1.1.1.1.1.m1.8.8a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8"><apply id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1"><times id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.1"></times><ci id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.2a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.2"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.2">&lt;</mtext></ci><apply id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.3"><csymbol cd="ambiguous" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.3.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.3">subscript</csymbol><ci id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.3.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.3.2">𝑆</ci><cn type="integer" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.3.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.3.3">1</cn></apply><ci id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.4a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.4"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.4.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.3.1.4">&gt;</mtext></ci></apply><apply id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2"><ci id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.1">⟶</ci><csymbol cd="latexml" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.2">absent</csymbol><apply id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3"><times id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.1"></times><ci id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.2a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.2"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.2">&lt;X&gt;</mtext></ci><ci id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.3">ℳ</ci><ci id="S5.Ex2.m1.1.1.1.1.m1.1.1.1.1.1.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.1.1.1.1.1.1">𝑦</ci><ci id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.5a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.5"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.5.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.5">&lt;Y&gt;</mtext></ci><ci id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.6a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.6"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.6.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.6">&lt;</mtext></ci><apply id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.7.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.7"><csymbol cd="ambiguous" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.7.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.7">subscript</csymbol><ci id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.7.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.7.2">𝑆</ci><cn type="integer" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.7.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.7.3">1</cn></apply><ci id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.8a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.8"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.8.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.2.2.2.2.2.2.1.3.8">&gt;</mtext></ci></apply></apply></matrixrow><matrixrow id="S5.Ex2.m1.1.1.1.1.m1.8.8b.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8"><apply id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1"><times id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.1"></times><ci id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.2a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.2"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.2">&lt;</mtext></ci><apply id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.3"><csymbol cd="ambiguous" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.3.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.3">subscript</csymbol><ci id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.3.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.3.2">𝑆</ci><cn type="integer" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.3.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.3.3">1</cn></apply><ci id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.4a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.4"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.4.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.3.1.4">&gt;</mtext></ci></apply><apply id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2"><ci id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.1">⟶</ci><csymbol cd="latexml" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.2">absent</csymbol><apply id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3"><times id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.1"></times><ci id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.2a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.2"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.2">&lt;</mtext></ci><apply id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.3"><csymbol cd="ambiguous" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.3.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.3">subscript</csymbol><ci id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.3.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.3.2">𝑆</ci><cn type="integer" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.3.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.3.3">1</cn></apply><ci id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.4a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.4"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.4.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.4">&gt;</mtext></ci><ci id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.5a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.5"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.5.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.5">&lt;X&gt;</mtext></ci><ci id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.6.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.6">ℳ</ci><ci id="S5.Ex2.m1.1.1.1.1.m1.3.3.3.1.1.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.3.3.3.1.1.1">𝑦</ci><ci id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.8a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.8"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.8.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.4.4.4.2.2.2.1.3.8">&lt;Y&gt;</mtext></ci></apply></apply></matrixrow><matrixrow id="S5.Ex2.m1.1.1.1.1.m1.8.8c.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8"><list id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2"><apply id="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1"><times id="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.1"></times><ci id="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.2a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.2"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.2">&lt;</mtext></ci><apply id="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.3"><csymbol cd="ambiguous" id="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.3.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.3">subscript</csymbol><ci id="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.3.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.3.2">𝑆</ci><cn type="integer" id="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.3.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.3.3">1</cn></apply><ci id="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.4a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.4"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.4.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.5.5.5.1.1.1.1.4">&gt;</mtext></ci></apply><apply id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2"><times id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.1"></times><ci id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.2a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.2"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.2">&lt;</mtext></ci><apply id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.3"><csymbol cd="ambiguous" id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.3.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.3">subscript</csymbol><ci id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.3.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.3.2">𝑆</ci><cn type="integer" id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.3.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.3.3">2</cn></apply><ci id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.4a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.4"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.4.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.6.6.6.2.2.2.2.4">&gt;</mtext></ci></apply></list><apply id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2"><ci id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.1">⟶</ci><csymbol cd="latexml" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.2">absent</csymbol><apply id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3"><times id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.1"></times><ci id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.2a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.2"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.2">&lt;</mtext></ci><apply id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.3"><csymbol cd="ambiguous" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.3.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.3">subscript</csymbol><ci id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.3.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.3.2">𝑆</ci><cn type="integer" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.3.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.3.3">1</cn></apply><ci id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.4a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.4"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.4.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.4">&gt;</mtext></ci><ci id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.5a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.5"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.5.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.5">&lt;X&gt;</mtext></ci><ci id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.6.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.6">ℳ</ci><ci id="S5.Ex2.m1.1.1.1.1.m1.7.7.7.3.1.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.7.7.7.3.1.1">𝑦</ci><ci id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.8a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.8"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.8.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.8">&lt;Y&gt;</mtext></ci><ci id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.9a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.9"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.9.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.9">&lt;</mtext></ci><apply id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.10.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.10"><csymbol cd="ambiguous" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.10.1.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.10">subscript</csymbol><ci id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.10.2.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.10.2">𝑆</ci><cn type="integer" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.10.3.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.10.3">2</cn></apply><ci id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.11a.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.11"><mtext class="ltx_mathvariant_monospace" id="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.11.cmml" xref="S5.Ex2.m1.1.1.1.1.m1.8.8.8.4.2.2.1.3.11">&gt;</mtext></ci></apply></apply></matrixrow></matrix></annotation-xml><annotation encoding="application/x-tex" id="S5.Ex2.m1.1.1.1.1.m1.8c">\begin{aligned} \texttt{&lt;}S_{1}\texttt{&gt;}&amp;\longrightarrow~{}\texttt{&lt;X&gt;}~{}\mathcal{M}(y)~{}\texttt{&lt;Y&gt;}~{}\texttt{&lt;}S_{1}\texttt{&gt;},\\
\texttt{&lt;}S_{1}\texttt{&gt;}&amp;\longrightarrow~{}\texttt{&lt;}S_{1}\texttt{&gt;}~{}\texttt{&lt;X&gt;}~{}\mathcal{M}(y)~{}\texttt{&lt;Y&gt;},\\
\texttt{&lt;}S_{1}\texttt{&gt;},\texttt{&lt;}S_{2}\texttt{&gt;}&amp;\longrightarrow\texttt{&lt;}S_{1}\texttt{&gt;}~{}\texttt{&lt;X&gt;}~{}\mathcal{M}(y)~{}\texttt{&lt;Y&gt;}~{}\texttt{&lt;}S_{2}\texttt{&gt;}.\end{aligned}</annotation></semantics></math></p>
</span></div>
</td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="S5.SS2.p3" class="ltx_para">
<p id="S5.SS2.p3.4" class="ltx_p">As shown in Figure <a href="#S5.F2" title="Figure 2 ‣ 5.2 Automatic generation of templates ‣ 5 Automatic Prompt Generation ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, we rely on the T5 model to fill in the placeholders. When decoding, our goal here is to find an output that can work well for <em id="S5.SS2.p3.4.1" class="ltx_emph ltx_font_italic">all</em> examples in <math id="S5.SS2.p3.1.m1.1" class="ltx_Math" alttext="\mathcal{D}_{\text{train}}" display="inline"><semantics id="S5.SS2.p3.1.m1.1a"><msub id="S5.SS2.p3.1.m1.1.1" xref="S5.SS2.p3.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS2.p3.1.m1.1.1.2" xref="S5.SS2.p3.1.m1.1.1.2.cmml">𝒟</mi><mtext id="S5.SS2.p3.1.m1.1.1.3" xref="S5.SS2.p3.1.m1.1.1.3a.cmml">train</mtext></msub><annotation-xml encoding="MathML-Content" id="S5.SS2.p3.1.m1.1b"><apply id="S5.SS2.p3.1.m1.1.1.cmml" xref="S5.SS2.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S5.SS2.p3.1.m1.1.1.1.cmml" xref="S5.SS2.p3.1.m1.1.1">subscript</csymbol><ci id="S5.SS2.p3.1.m1.1.1.2.cmml" xref="S5.SS2.p3.1.m1.1.1.2">𝒟</ci><ci id="S5.SS2.p3.1.m1.1.1.3a.cmml" xref="S5.SS2.p3.1.m1.1.1.3"><mtext mathsize="70%" id="S5.SS2.p3.1.m1.1.1.3.cmml" xref="S5.SS2.p3.1.m1.1.1.3">train</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p3.1.m1.1c">\mathcal{D}_{\text{train}}</annotation></semantics></math>,
i.e.,
the output template <math id="S5.SS2.p3.2.m2.1" class="ltx_Math" alttext="\mathcal{T}" display="inline"><semantics id="S5.SS2.p3.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S5.SS2.p3.2.m2.1.1" xref="S5.SS2.p3.2.m2.1.1.cmml">𝒯</mi><annotation-xml encoding="MathML-Content" id="S5.SS2.p3.2.m2.1b"><ci id="S5.SS2.p3.2.m2.1.1.cmml" xref="S5.SS2.p3.2.m2.1.1">𝒯</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p3.2.m2.1c">\mathcal{T}</annotation></semantics></math> that maximizes <math id="S5.SS2.p3.3.m3.4" class="ltx_Math" alttext="\sum_{({x}_{\mathrm{in}},y)\in\mathcal{D}_{\text{train}}}{\log P_{\text{T5}}(\mathcal{T}\mid\mathcal{T}_{\mathrm{g}}({x}_{\mathrm{in}},y))}" display="inline"><semantics id="S5.SS2.p3.3.m3.4a"><mrow id="S5.SS2.p3.3.m3.4.4" xref="S5.SS2.p3.3.m3.4.4.cmml"><msub id="S5.SS2.p3.3.m3.4.4.2" xref="S5.SS2.p3.3.m3.4.4.2.cmml"><mo id="S5.SS2.p3.3.m3.4.4.2.2" xref="S5.SS2.p3.3.m3.4.4.2.2.cmml">∑</mo><mrow id="S5.SS2.p3.3.m3.2.2.2" xref="S5.SS2.p3.3.m3.2.2.2.cmml"><mrow id="S5.SS2.p3.3.m3.2.2.2.2.1" xref="S5.SS2.p3.3.m3.2.2.2.2.2.cmml"><mo stretchy="false" id="S5.SS2.p3.3.m3.2.2.2.2.1.2" xref="S5.SS2.p3.3.m3.2.2.2.2.2.cmml">(</mo><msub id="S5.SS2.p3.3.m3.2.2.2.2.1.1" xref="S5.SS2.p3.3.m3.2.2.2.2.1.1.cmml"><mi id="S5.SS2.p3.3.m3.2.2.2.2.1.1.2" xref="S5.SS2.p3.3.m3.2.2.2.2.1.1.2.cmml">x</mi><mi id="S5.SS2.p3.3.m3.2.2.2.2.1.1.3" xref="S5.SS2.p3.3.m3.2.2.2.2.1.1.3.cmml">in</mi></msub><mo id="S5.SS2.p3.3.m3.2.2.2.2.1.3" xref="S5.SS2.p3.3.m3.2.2.2.2.2.cmml">,</mo><mi id="S5.SS2.p3.3.m3.1.1.1.1" xref="S5.SS2.p3.3.m3.1.1.1.1.cmml">y</mi><mo stretchy="false" id="S5.SS2.p3.3.m3.2.2.2.2.1.4" xref="S5.SS2.p3.3.m3.2.2.2.2.2.cmml">)</mo></mrow><mo id="S5.SS2.p3.3.m3.2.2.2.3" xref="S5.SS2.p3.3.m3.2.2.2.3.cmml">∈</mo><msub id="S5.SS2.p3.3.m3.2.2.2.4" xref="S5.SS2.p3.3.m3.2.2.2.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS2.p3.3.m3.2.2.2.4.2" xref="S5.SS2.p3.3.m3.2.2.2.4.2.cmml">𝒟</mi><mtext id="S5.SS2.p3.3.m3.2.2.2.4.3" xref="S5.SS2.p3.3.m3.2.2.2.4.3a.cmml">train</mtext></msub></mrow></msub><mrow id="S5.SS2.p3.3.m3.4.4.1" xref="S5.SS2.p3.3.m3.4.4.1.cmml"><mrow id="S5.SS2.p3.3.m3.4.4.1.3" xref="S5.SS2.p3.3.m3.4.4.1.3.cmml"><mi id="S5.SS2.p3.3.m3.4.4.1.3.1" xref="S5.SS2.p3.3.m3.4.4.1.3.1.cmml">log</mi><mo lspace="0.167em" id="S5.SS2.p3.3.m3.4.4.1.3a" xref="S5.SS2.p3.3.m3.4.4.1.3.cmml">⁡</mo><msub id="S5.SS2.p3.3.m3.4.4.1.3.2" xref="S5.SS2.p3.3.m3.4.4.1.3.2.cmml"><mi id="S5.SS2.p3.3.m3.4.4.1.3.2.2" xref="S5.SS2.p3.3.m3.4.4.1.3.2.2.cmml">P</mi><mtext id="S5.SS2.p3.3.m3.4.4.1.3.2.3" xref="S5.SS2.p3.3.m3.4.4.1.3.2.3a.cmml">T5</mtext></msub></mrow><mo lspace="0em" rspace="0em" id="S5.SS2.p3.3.m3.4.4.1.2" xref="S5.SS2.p3.3.m3.4.4.1.2.cmml">​</mo><mrow id="S5.SS2.p3.3.m3.4.4.1.1.1" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.cmml"><mo stretchy="false" id="S5.SS2.p3.3.m3.4.4.1.1.1.2" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.cmml">(</mo><mrow id="S5.SS2.p3.3.m3.4.4.1.1.1.1" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS2.p3.3.m3.4.4.1.1.1.1.3" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.3.cmml">𝒯</mi><mo id="S5.SS2.p3.3.m3.4.4.1.1.1.1.2" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.2.cmml">∣</mo><mrow id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.cmml"><msub id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.3" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.3.2" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.3.2.cmml">𝒯</mi><mi mathvariant="normal" id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.3.3" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.3.3.cmml">g</mi></msub><mo lspace="0em" rspace="0em" id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.2" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.2.cmml">​</mo><mrow id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1.2" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.2.cmml">(</mo><msub id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1.1" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1.1.cmml"><mi id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1.1.2" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1.1.2.cmml">x</mi><mi id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1.1.3" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1.1.3.cmml">in</mi></msub><mo id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1.3" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.2.cmml">,</mo><mi id="S5.SS2.p3.3.m3.3.3" xref="S5.SS2.p3.3.m3.3.3.cmml">y</mi><mo stretchy="false" id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1.4" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.2.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S5.SS2.p3.3.m3.4.4.1.1.1.3" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.p3.3.m3.4b"><apply id="S5.SS2.p3.3.m3.4.4.cmml" xref="S5.SS2.p3.3.m3.4.4"><apply id="S5.SS2.p3.3.m3.4.4.2.cmml" xref="S5.SS2.p3.3.m3.4.4.2"><csymbol cd="ambiguous" id="S5.SS2.p3.3.m3.4.4.2.1.cmml" xref="S5.SS2.p3.3.m3.4.4.2">subscript</csymbol><sum id="S5.SS2.p3.3.m3.4.4.2.2.cmml" xref="S5.SS2.p3.3.m3.4.4.2.2"></sum><apply id="S5.SS2.p3.3.m3.2.2.2.cmml" xref="S5.SS2.p3.3.m3.2.2.2"><in id="S5.SS2.p3.3.m3.2.2.2.3.cmml" xref="S5.SS2.p3.3.m3.2.2.2.3"></in><interval closure="open" id="S5.SS2.p3.3.m3.2.2.2.2.2.cmml" xref="S5.SS2.p3.3.m3.2.2.2.2.1"><apply id="S5.SS2.p3.3.m3.2.2.2.2.1.1.cmml" xref="S5.SS2.p3.3.m3.2.2.2.2.1.1"><csymbol cd="ambiguous" id="S5.SS2.p3.3.m3.2.2.2.2.1.1.1.cmml" xref="S5.SS2.p3.3.m3.2.2.2.2.1.1">subscript</csymbol><ci id="S5.SS2.p3.3.m3.2.2.2.2.1.1.2.cmml" xref="S5.SS2.p3.3.m3.2.2.2.2.1.1.2">𝑥</ci><ci id="S5.SS2.p3.3.m3.2.2.2.2.1.1.3.cmml" xref="S5.SS2.p3.3.m3.2.2.2.2.1.1.3">in</ci></apply><ci id="S5.SS2.p3.3.m3.1.1.1.1.cmml" xref="S5.SS2.p3.3.m3.1.1.1.1">𝑦</ci></interval><apply id="S5.SS2.p3.3.m3.2.2.2.4.cmml" xref="S5.SS2.p3.3.m3.2.2.2.4"><csymbol cd="ambiguous" id="S5.SS2.p3.3.m3.2.2.2.4.1.cmml" xref="S5.SS2.p3.3.m3.2.2.2.4">subscript</csymbol><ci id="S5.SS2.p3.3.m3.2.2.2.4.2.cmml" xref="S5.SS2.p3.3.m3.2.2.2.4.2">𝒟</ci><ci id="S5.SS2.p3.3.m3.2.2.2.4.3a.cmml" xref="S5.SS2.p3.3.m3.2.2.2.4.3"><mtext mathsize="50%" id="S5.SS2.p3.3.m3.2.2.2.4.3.cmml" xref="S5.SS2.p3.3.m3.2.2.2.4.3">train</mtext></ci></apply></apply></apply><apply id="S5.SS2.p3.3.m3.4.4.1.cmml" xref="S5.SS2.p3.3.m3.4.4.1"><times id="S5.SS2.p3.3.m3.4.4.1.2.cmml" xref="S5.SS2.p3.3.m3.4.4.1.2"></times><apply id="S5.SS2.p3.3.m3.4.4.1.3.cmml" xref="S5.SS2.p3.3.m3.4.4.1.3"><log id="S5.SS2.p3.3.m3.4.4.1.3.1.cmml" xref="S5.SS2.p3.3.m3.4.4.1.3.1"></log><apply id="S5.SS2.p3.3.m3.4.4.1.3.2.cmml" xref="S5.SS2.p3.3.m3.4.4.1.3.2"><csymbol cd="ambiguous" id="S5.SS2.p3.3.m3.4.4.1.3.2.1.cmml" xref="S5.SS2.p3.3.m3.4.4.1.3.2">subscript</csymbol><ci id="S5.SS2.p3.3.m3.4.4.1.3.2.2.cmml" xref="S5.SS2.p3.3.m3.4.4.1.3.2.2">𝑃</ci><ci id="S5.SS2.p3.3.m3.4.4.1.3.2.3a.cmml" xref="S5.SS2.p3.3.m3.4.4.1.3.2.3"><mtext mathsize="70%" id="S5.SS2.p3.3.m3.4.4.1.3.2.3.cmml" xref="S5.SS2.p3.3.m3.4.4.1.3.2.3">T5</mtext></ci></apply></apply><apply id="S5.SS2.p3.3.m3.4.4.1.1.1.1.cmml" xref="S5.SS2.p3.3.m3.4.4.1.1.1"><csymbol cd="latexml" id="S5.SS2.p3.3.m3.4.4.1.1.1.1.2.cmml" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.2">conditional</csymbol><ci id="S5.SS2.p3.3.m3.4.4.1.1.1.1.3.cmml" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.3">𝒯</ci><apply id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.cmml" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1"><times id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.2.cmml" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.2"></times><apply id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.3.cmml" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.3.1.cmml" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.3">subscript</csymbol><ci id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.3.2.cmml" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.3.2">𝒯</ci><ci id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.3.3.cmml" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.3.3">g</ci></apply><interval closure="open" id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.2.cmml" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1"><apply id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1.1.cmml" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1.1.1.cmml" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1.1.2.cmml" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1.1.2">𝑥</ci><ci id="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1.1.3.cmml" xref="S5.SS2.p3.3.m3.4.4.1.1.1.1.1.1.1.1.3">in</ci></apply><ci id="S5.SS2.p3.3.m3.3.3.cmml" xref="S5.SS2.p3.3.m3.3.3">𝑦</ci></interval></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p3.3.m3.4c">\sum_{({x}_{\mathrm{in}},y)\in\mathcal{D}_{\text{train}}}{\log P_{\text{T5}}(\mathcal{T}\mid\mathcal{T}_{\mathrm{g}}({x}_{\mathrm{in}},y))}</annotation></semantics></math>,
where <math id="S5.SS2.p3.4.m4.1" class="ltx_Math" alttext="P_{\text{T5}}" display="inline"><semantics id="S5.SS2.p3.4.m4.1a"><msub id="S5.SS2.p3.4.m4.1.1" xref="S5.SS2.p3.4.m4.1.1.cmml"><mi id="S5.SS2.p3.4.m4.1.1.2" xref="S5.SS2.p3.4.m4.1.1.2.cmml">P</mi><mtext id="S5.SS2.p3.4.m4.1.1.3" xref="S5.SS2.p3.4.m4.1.1.3a.cmml">T5</mtext></msub><annotation-xml encoding="MathML-Content" id="S5.SS2.p3.4.m4.1b"><apply id="S5.SS2.p3.4.m4.1.1.cmml" xref="S5.SS2.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S5.SS2.p3.4.m4.1.1.1.cmml" xref="S5.SS2.p3.4.m4.1.1">subscript</csymbol><ci id="S5.SS2.p3.4.m4.1.1.2.cmml" xref="S5.SS2.p3.4.m4.1.1.2">𝑃</ci><ci id="S5.SS2.p3.4.m4.1.1.3a.cmml" xref="S5.SS2.p3.4.m4.1.1.3"><mtext mathsize="70%" id="S5.SS2.p3.4.m4.1.1.3.cmml" xref="S5.SS2.p3.4.m4.1.1.3">T5</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p3.4.m4.1c">P_{\text{T5}}</annotation></semantics></math> denotes the output probability distribution of T5.
It can be decomposed according to:</p>
<table id="S5.E4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center">
<div id="S5.E4.m1.1.1.1" class="ltx_inline-block ltx_markedasmath ltx_transformed_outer" style="width:408.7pt;height:38.2pt;vertical-align:-20.1pt;"><span class="ltx_transformed_inner" style="transform:translate(102.7pt,-4.5pt) scale(2.00958334658872,2.00958334658872) ;">
<p id="S5.E4.m1.1.1.1.1" class="ltx_p"><math id="S5.E4.m1.1.1.1.1.m1.6" class="ltx_Math" alttext="\displaystyle\sum_{j=1}^{|\mathcal{T}|}\sum_{~{}~{}~{}~{}~{}({x}_{\mathrm{in}},y)\in\mathcal{D}_{\text{train}}}{\log{P_{\text{T5}}\big{(}t_{j}\mid t_{1},...,t_{j-1},\mathcal{T}_{\mathrm{g}}\big{(}{x}_{\mathrm{in}},y\big{)}\big{)}}}," display="inline"><semantics id="S5.E4.m1.1.1.1.1.m1.6a"><mrow id="S5.E4.m1.1.1.1.1.m1.6.6.1" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.cmml"><mrow id="S5.E4.m1.1.1.1.1.m1.6.6.1.1" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.cmml"><mstyle displaystyle="true" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.cmml"><munderover id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2a" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.cmml"><mo movablelimits="false" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.2.cmml">∑</mo><mrow id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.3" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.3.cmml"><mi id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.3.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.3.2.cmml">j</mi><mo id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.3.1" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.3.1.cmml">=</mo><mn id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.3.3" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.3.3.cmml">1</mn></mrow><mrow id="S5.E4.m1.1.1.1.1.m1.1.1.1.3" xref="S5.E4.m1.1.1.1.1.m1.1.1.1.2.cmml"><mo stretchy="false" id="S5.E4.m1.1.1.1.1.m1.1.1.1.3.1" xref="S5.E4.m1.1.1.1.1.m1.1.1.1.2.1.cmml">|</mo><mi class="ltx_font_mathcaligraphic" id="S5.E4.m1.1.1.1.1.m1.1.1.1.1" xref="S5.E4.m1.1.1.1.1.m1.1.1.1.1.cmml">𝒯</mi><mo stretchy="false" id="S5.E4.m1.1.1.1.1.m1.1.1.1.3.2" xref="S5.E4.m1.1.1.1.1.m1.1.1.1.2.1.cmml">|</mo></mrow></munderover></mstyle><mrow id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.cmml"><mstyle displaystyle="true" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.2.cmml"><munder id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.2a" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.2.cmml"><mo movablelimits="false" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.2.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.2.2.cmml">∑</mo><mrow id="S5.E4.m1.1.1.1.1.m1.3.3.2" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.cmml"><mrow id="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.2.2.cmml"><mo stretchy="false" id="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1.2" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.2.2.cmml">(</mo><msub id="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1.1" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1.1.cmml"><mi id="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1.1.2" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1.1.2.cmml">x</mi><mi id="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1.1.3" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1.1.3.cmml">in</mi></msub><mo id="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1.3" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.2.2.cmml">,</mo><mi id="S5.E4.m1.1.1.1.1.m1.2.2.1.1" xref="S5.E4.m1.1.1.1.1.m1.2.2.1.1.cmml">y</mi><mo stretchy="false" id="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1.4" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.2.2.cmml">)</mo></mrow><mo id="S5.E4.m1.1.1.1.1.m1.3.3.2.3" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.3.cmml">∈</mo><msub id="S5.E4.m1.1.1.1.1.m1.3.3.2.4" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.E4.m1.1.1.1.1.m1.3.3.2.4.2" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.4.2.cmml">𝒟</mi><mtext id="S5.E4.m1.1.1.1.1.m1.3.3.2.4.3" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.4.3a.cmml">train</mtext></msub></mrow></munder></mstyle><mrow id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.cmml"><mrow id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.cmml"><mi id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.1" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.1.cmml">log</mi><mo lspace="0.167em" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3a" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.cmml">⁡</mo><msub id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.2.cmml"><mi id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.2.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.2.2.cmml">P</mi><mtext id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.2.3" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.2.3a.cmml">T5</mtext></msub></mrow><mo lspace="0em" rspace="0em" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.2.cmml">​</mo><mrow id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.cmml"><mo maxsize="120%" minsize="120%" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.cmml"><msub id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.5" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.5.cmml"><mi id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.5.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.5.2.cmml">t</mi><mi id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.5.3" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.5.3.cmml">j</mi></msub><mo id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.4" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.4.cmml">∣</mo><mrow id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.4.cmml"><msub id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.1.1.1" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.1.1.1.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.1.1.1.2.cmml">t</mi><mn id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.1.1.1.3" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.1.1.1.3.cmml">1</mn></msub><mo id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.4" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.4.cmml">,</mo><mi mathvariant="normal" id="S5.E4.m1.1.1.1.1.m1.5.5" xref="S5.E4.m1.1.1.1.1.m1.5.5.cmml">…</mi><mo id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.5" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.4.cmml">,</mo><msub id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.cmml"><mi id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.2.cmml">t</mi><mrow id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.3" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.3.cmml"><mi id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.3.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.3.2.cmml">j</mi><mo id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.3.1" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.3.1.cmml">−</mo><mn id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.3.3" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.3.3.cmml">1</mn></mrow></msub><mo id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.6" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.4.cmml">,</mo><mrow id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.cmml"><msub id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.3" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.3.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.3.2.cmml">𝒯</mi><mi mathvariant="normal" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.3.3" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.3.3.cmml">g</mi></msub><mo lspace="0em" rspace="0em" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.2.cmml">​</mo><mrow id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.2.cmml"><mo maxsize="120%" minsize="120%" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.2.cmml">(</mo><msub id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1.1" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1.1.cmml"><mi id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1.1.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1.1.2.cmml">x</mi><mi id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1.1.3" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1.1.3.cmml">in</mi></msub><mo id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1.3" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.2.cmml">,</mo><mi id="S5.E4.m1.1.1.1.1.m1.4.4" xref="S5.E4.m1.1.1.1.1.m1.4.4.cmml">y</mi><mo maxsize="120%" minsize="120%" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1.4" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.2.cmml">)</mo></mrow></mrow></mrow></mrow><mo maxsize="120%" minsize="120%" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.3" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><mo id="S5.E4.m1.1.1.1.1.m1.6.6.1.2" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.cmml">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S5.E4.m1.1.1.1.1.m1.6b"><apply id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1"><apply id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2"><csymbol cd="ambiguous" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2">superscript</csymbol><apply id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2"><csymbol cd="ambiguous" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2">subscript</csymbol><sum id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.2"></sum><apply id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.3.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.3"><eq id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.3.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.3.1"></eq><ci id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.3.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.3.2">𝑗</ci><cn type="integer" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.3.3.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.2.2.3.3">1</cn></apply></apply><apply id="S5.E4.m1.1.1.1.1.m1.1.1.1.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.1.1.1.3"><abs id="S5.E4.m1.1.1.1.1.m1.1.1.1.2.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.1.1.1.3.1"></abs><ci id="S5.E4.m1.1.1.1.1.m1.1.1.1.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.1.1.1.1">𝒯</ci></apply></apply><apply id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1"><apply id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.2"><csymbol cd="ambiguous" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.2.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.2">subscript</csymbol><sum id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.2.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.2.2"></sum><apply id="S5.E4.m1.1.1.1.1.m1.3.3.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.3.3.2"><in id="S5.E4.m1.1.1.1.1.m1.3.3.2.3.cmml" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.3"></in><interval closure="open" id="S5.E4.m1.1.1.1.1.m1.3.3.2.2.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1"><apply id="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1.1"><csymbol cd="ambiguous" id="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1.1.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1.1">subscript</csymbol><ci id="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1.1.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1.1.2">𝑥</ci><ci id="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1.1.3.cmml" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.2.1.1.3">in</ci></apply><ci id="S5.E4.m1.1.1.1.1.m1.2.2.1.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.2.2.1.1">𝑦</ci></interval><apply id="S5.E4.m1.1.1.1.1.m1.3.3.2.4.cmml" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.4"><csymbol cd="ambiguous" id="S5.E4.m1.1.1.1.1.m1.3.3.2.4.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.4">subscript</csymbol><ci id="S5.E4.m1.1.1.1.1.m1.3.3.2.4.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.4.2">𝒟</ci><ci id="S5.E4.m1.1.1.1.1.m1.3.3.2.4.3a.cmml" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.4.3"><mtext mathsize="50%" id="S5.E4.m1.1.1.1.1.m1.3.3.2.4.3.cmml" xref="S5.E4.m1.1.1.1.1.m1.3.3.2.4.3">train</mtext></ci></apply></apply></apply><apply id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1"><times id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.2"></times><apply id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3"><log id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.1"></log><apply id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.2"><csymbol cd="ambiguous" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.2.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.2">subscript</csymbol><ci id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.2.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.2.2">𝑃</ci><ci id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.2.3a.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.2.3"><mtext mathsize="70%" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.2.3.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.3.2.3">T5</mtext></ci></apply></apply><apply id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1"><csymbol cd="latexml" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.4.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.4">conditional</csymbol><apply id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.5.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.5"><csymbol cd="ambiguous" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.5.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.5">subscript</csymbol><ci id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.5.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.5.2">𝑡</ci><ci id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.5.3.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.5.3">𝑗</ci></apply><list id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.4.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3"><apply id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.1.1.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.1.1.1.2">𝑡</ci><cn type="integer" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.1.1.1.3">1</cn></apply><ci id="S5.E4.m1.1.1.1.1.m1.5.5.cmml" xref="S5.E4.m1.1.1.1.1.m1.5.5">…</ci><apply id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2">subscript</csymbol><ci id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.2">𝑡</ci><apply id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.3.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.3"><minus id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.3.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.3.1"></minus><ci id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.3.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.3.2">𝑗</ci><cn type="integer" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.3.3.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.2.2.2.3.3">1</cn></apply></apply><apply id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3"><times id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.2"></times><apply id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.3.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.3"><csymbol cd="ambiguous" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.3.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.3">subscript</csymbol><ci id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.3.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.3.2">𝒯</ci><ci id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.3.3.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.3.3">g</ci></apply><interval closure="open" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1"><apply id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1.1"><csymbol cd="ambiguous" id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1.1.1.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1.1">subscript</csymbol><ci id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1.1.2.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1.1.2">𝑥</ci><ci id="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1.1.3.cmml" xref="S5.E4.m1.1.1.1.1.m1.6.6.1.1.1.1.1.1.1.3.3.3.1.1.1.3">in</ci></apply><ci id="S5.E4.m1.1.1.1.1.m1.4.4.cmml" xref="S5.E4.m1.1.1.1.1.m1.4.4">𝑦</ci></interval></apply></list></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.E4.m1.1.1.1.1.m1.6c">\displaystyle\sum_{j=1}^{|\mathcal{T}|}\sum_{~{}~{}~{}~{}~{}({x}_{\mathrm{in}},y)\in\mathcal{D}_{\text{train}}}{\log{P_{\text{T5}}\big{(}t_{j}\mid t_{1},...,t_{j-1},\mathcal{T}_{\mathrm{g}}\big{(}{x}_{\mathrm{in}},y\big{)}\big{)}}},</annotation></semantics></math></p>
</span></div>
</td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="0" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(4)</span></td>
</tr></tbody>
</table>
<p id="S5.SS2.p3.5" class="ltx_p">where <math id="S5.SS2.p3.5.m1.4" class="ltx_Math" alttext="(t_{1},\ldots,t_{|\mathcal{T}|})" display="inline"><semantics id="S5.SS2.p3.5.m1.4a"><mrow id="S5.SS2.p3.5.m1.4.4.2" xref="S5.SS2.p3.5.m1.4.4.3.cmml"><mo stretchy="false" id="S5.SS2.p3.5.m1.4.4.2.3" xref="S5.SS2.p3.5.m1.4.4.3.cmml">(</mo><msub id="S5.SS2.p3.5.m1.3.3.1.1" xref="S5.SS2.p3.5.m1.3.3.1.1.cmml"><mi id="S5.SS2.p3.5.m1.3.3.1.1.2" xref="S5.SS2.p3.5.m1.3.3.1.1.2.cmml">t</mi><mn id="S5.SS2.p3.5.m1.3.3.1.1.3" xref="S5.SS2.p3.5.m1.3.3.1.1.3.cmml">1</mn></msub><mo id="S5.SS2.p3.5.m1.4.4.2.4" xref="S5.SS2.p3.5.m1.4.4.3.cmml">,</mo><mi mathvariant="normal" id="S5.SS2.p3.5.m1.2.2" xref="S5.SS2.p3.5.m1.2.2.cmml">…</mi><mo id="S5.SS2.p3.5.m1.4.4.2.5" xref="S5.SS2.p3.5.m1.4.4.3.cmml">,</mo><msub id="S5.SS2.p3.5.m1.4.4.2.2" xref="S5.SS2.p3.5.m1.4.4.2.2.cmml"><mi id="S5.SS2.p3.5.m1.4.4.2.2.2" xref="S5.SS2.p3.5.m1.4.4.2.2.2.cmml">t</mi><mrow id="S5.SS2.p3.5.m1.1.1.1.3" xref="S5.SS2.p3.5.m1.1.1.1.2.cmml"><mo stretchy="false" id="S5.SS2.p3.5.m1.1.1.1.3.1" xref="S5.SS2.p3.5.m1.1.1.1.2.1.cmml">|</mo><mi class="ltx_font_mathcaligraphic" id="S5.SS2.p3.5.m1.1.1.1.1" xref="S5.SS2.p3.5.m1.1.1.1.1.cmml">𝒯</mi><mo stretchy="false" id="S5.SS2.p3.5.m1.1.1.1.3.2" xref="S5.SS2.p3.5.m1.1.1.1.2.1.cmml">|</mo></mrow></msub><mo stretchy="false" id="S5.SS2.p3.5.m1.4.4.2.6" xref="S5.SS2.p3.5.m1.4.4.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S5.SS2.p3.5.m1.4b"><vector id="S5.SS2.p3.5.m1.4.4.3.cmml" xref="S5.SS2.p3.5.m1.4.4.2"><apply id="S5.SS2.p3.5.m1.3.3.1.1.cmml" xref="S5.SS2.p3.5.m1.3.3.1.1"><csymbol cd="ambiguous" id="S5.SS2.p3.5.m1.3.3.1.1.1.cmml" xref="S5.SS2.p3.5.m1.3.3.1.1">subscript</csymbol><ci id="S5.SS2.p3.5.m1.3.3.1.1.2.cmml" xref="S5.SS2.p3.5.m1.3.3.1.1.2">𝑡</ci><cn type="integer" id="S5.SS2.p3.5.m1.3.3.1.1.3.cmml" xref="S5.SS2.p3.5.m1.3.3.1.1.3">1</cn></apply><ci id="S5.SS2.p3.5.m1.2.2.cmml" xref="S5.SS2.p3.5.m1.2.2">…</ci><apply id="S5.SS2.p3.5.m1.4.4.2.2.cmml" xref="S5.SS2.p3.5.m1.4.4.2.2"><csymbol cd="ambiguous" id="S5.SS2.p3.5.m1.4.4.2.2.1.cmml" xref="S5.SS2.p3.5.m1.4.4.2.2">subscript</csymbol><ci id="S5.SS2.p3.5.m1.4.4.2.2.2.cmml" xref="S5.SS2.p3.5.m1.4.4.2.2.2">𝑡</ci><apply id="S5.SS2.p3.5.m1.1.1.1.2.cmml" xref="S5.SS2.p3.5.m1.1.1.1.3"><abs id="S5.SS2.p3.5.m1.1.1.1.2.1.cmml" xref="S5.SS2.p3.5.m1.1.1.1.3.1"></abs><ci id="S5.SS2.p3.5.m1.1.1.1.1.cmml" xref="S5.SS2.p3.5.m1.1.1.1.1">𝒯</ci></apply></apply></vector></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p3.5.m1.4c">(t_{1},\ldots,t_{|\mathcal{T}|})</annotation></semantics></math> are the template tokens.</p>
</div>
<figure id="S5.F2" class="ltx_figure"><img src="/html/2012.15723/assets/x2.png" id="S5.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="232" height="111" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Our approach for template generation.
</figcaption>
</figure>
<div id="S5.SS2.p4" class="ltx_para">
<p id="S5.SS2.p4.4" class="ltx_p">We use beam search to decode multiple template candidates. Concretely, we use a wide beam width (e.g., 100) to cheaply obtain a large set of diverse templates. We then fine-tune each generated template on <math id="S5.SS2.p4.1.m1.1" class="ltx_Math" alttext="\mathcal{D}_{\text{train}}" display="inline"><semantics id="S5.SS2.p4.1.m1.1a"><msub id="S5.SS2.p4.1.m1.1.1" xref="S5.SS2.p4.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS2.p4.1.m1.1.1.2" xref="S5.SS2.p4.1.m1.1.1.2.cmml">𝒟</mi><mtext id="S5.SS2.p4.1.m1.1.1.3" xref="S5.SS2.p4.1.m1.1.1.3a.cmml">train</mtext></msub><annotation-xml encoding="MathML-Content" id="S5.SS2.p4.1.m1.1b"><apply id="S5.SS2.p4.1.m1.1.1.cmml" xref="S5.SS2.p4.1.m1.1.1"><csymbol cd="ambiguous" id="S5.SS2.p4.1.m1.1.1.1.cmml" xref="S5.SS2.p4.1.m1.1.1">subscript</csymbol><ci id="S5.SS2.p4.1.m1.1.1.2.cmml" xref="S5.SS2.p4.1.m1.1.1.2">𝒟</ci><ci id="S5.SS2.p4.1.m1.1.1.3a.cmml" xref="S5.SS2.p4.1.m1.1.1.3"><mtext mathsize="70%" id="S5.SS2.p4.1.m1.1.1.3.cmml" xref="S5.SS2.p4.1.m1.1.1.3">train</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p4.1.m1.1c">\mathcal{D}_{\text{train}}</annotation></semantics></math> and use <math id="S5.SS2.p4.2.m2.1" class="ltx_Math" alttext="\mathcal{D}_{\text{dev}}" display="inline"><semantics id="S5.SS2.p4.2.m2.1a"><msub id="S5.SS2.p4.2.m2.1.1" xref="S5.SS2.p4.2.m2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS2.p4.2.m2.1.1.2" xref="S5.SS2.p4.2.m2.1.1.2.cmml">𝒟</mi><mtext id="S5.SS2.p4.2.m2.1.1.3" xref="S5.SS2.p4.2.m2.1.1.3a.cmml">dev</mtext></msub><annotation-xml encoding="MathML-Content" id="S5.SS2.p4.2.m2.1b"><apply id="S5.SS2.p4.2.m2.1.1.cmml" xref="S5.SS2.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S5.SS2.p4.2.m2.1.1.1.cmml" xref="S5.SS2.p4.2.m2.1.1">subscript</csymbol><ci id="S5.SS2.p4.2.m2.1.1.2.cmml" xref="S5.SS2.p4.2.m2.1.1.2">𝒟</ci><ci id="S5.SS2.p4.2.m2.1.1.3a.cmml" xref="S5.SS2.p4.2.m2.1.1.3"><mtext mathsize="70%" id="S5.SS2.p4.2.m2.1.1.3.cmml" xref="S5.SS2.p4.2.m2.1.1.3">dev</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p4.2.m2.1c">\mathcal{D}_{\text{dev}}</annotation></semantics></math> to either pick the single template with the best performance (Table <a href="#S6.T3" title="Table 3 ‣ 6 Fine-tuning with Demonstrations ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>), or the top <math id="S5.SS2.p4.3.m3.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S5.SS2.p4.3.m3.1a"><mi id="S5.SS2.p4.3.m3.1.1" xref="S5.SS2.p4.3.m3.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S5.SS2.p4.3.m3.1b"><ci id="S5.SS2.p4.3.m3.1.1.cmml" xref="S5.SS2.p4.3.m3.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p4.3.m3.1c">k</annotation></semantics></math> templates to use as an ensemble (Table <a href="#S7.T4" title="Table 4 ‣ 7.1 Main results ‣ 7 Experiments ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>). Though it might appear to be expensive to fine-tune the model on each individual template, this is fast in practice due to the small size of <math id="S5.SS2.p4.4.m4.1" class="ltx_Math" alttext="\mathcal{D}_{\text{train}}" display="inline"><semantics id="S5.SS2.p4.4.m4.1a"><msub id="S5.SS2.p4.4.m4.1.1" xref="S5.SS2.p4.4.m4.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S5.SS2.p4.4.m4.1.1.2" xref="S5.SS2.p4.4.m4.1.1.2.cmml">𝒟</mi><mtext id="S5.SS2.p4.4.m4.1.1.3" xref="S5.SS2.p4.4.m4.1.1.3a.cmml">train</mtext></msub><annotation-xml encoding="MathML-Content" id="S5.SS2.p4.4.m4.1b"><apply id="S5.SS2.p4.4.m4.1.1.cmml" xref="S5.SS2.p4.4.m4.1.1"><csymbol cd="ambiguous" id="S5.SS2.p4.4.m4.1.1.1.cmml" xref="S5.SS2.p4.4.m4.1.1">subscript</csymbol><ci id="S5.SS2.p4.4.m4.1.1.2.cmml" xref="S5.SS2.p4.4.m4.1.1.2">𝒟</ci><ci id="S5.SS2.p4.4.m4.1.1.3a.cmml" xref="S5.SS2.p4.4.m4.1.1.3"><mtext mathsize="70%" id="S5.SS2.p4.4.m4.1.1.3.cmml" xref="S5.SS2.p4.4.m4.1.1.3">train</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.SS2.p4.4.m4.1c">\mathcal{D}_{\text{train}}</annotation></semantics></math>, and is also fully automated: making it easy to use, compared to manually tuning prompts for each dataset.</p>
</div>
</section>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Fine-tuning with Demonstrations</h2>

<figure id="S6.T3" class="ltx_table">
<div id="S6.T3.6.6" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:455.2pt;height:308.3pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(-64.8pt,43.9pt) scale(0.778516782501365,0.778516782501365) ;">
<table id="S6.T3.6.6.6" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S6.T3.6.6.6.7.1" class="ltx_tr">
<th id="S6.T3.6.6.6.7.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<td id="S6.T3.6.6.6.7.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="S6.T3.6.6.6.7.1.2.1" class="ltx_text ltx_font_bold">SST-2</span></td>
<td id="S6.T3.6.6.6.7.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="S6.T3.6.6.6.7.1.3.1" class="ltx_text ltx_font_bold">SST-5</span></td>
<td id="S6.T3.6.6.6.7.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="S6.T3.6.6.6.7.1.4.1" class="ltx_text ltx_font_bold">MR</span></td>
<td id="S6.T3.6.6.6.7.1.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="S6.T3.6.6.6.7.1.5.1" class="ltx_text ltx_font_bold">CR</span></td>
<td id="S6.T3.6.6.6.7.1.6" class="ltx_td ltx_align_center ltx_border_tt"><span id="S6.T3.6.6.6.7.1.6.1" class="ltx_text ltx_font_bold">MPQA</span></td>
<td id="S6.T3.6.6.6.7.1.7" class="ltx_td ltx_align_center ltx_border_tt"><span id="S6.T3.6.6.6.7.1.7.1" class="ltx_text ltx_font_bold">Subj</span></td>
<td id="S6.T3.6.6.6.7.1.8" class="ltx_td ltx_align_center ltx_border_tt"><span id="S6.T3.6.6.6.7.1.8.1" class="ltx_text ltx_font_bold">TREC</span></td>
<td id="S6.T3.6.6.6.7.1.9" class="ltx_td ltx_align_center ltx_border_tt"><span id="S6.T3.6.6.6.7.1.9.1" class="ltx_text ltx_font_bold">CoLA</span></td>
</tr>
<tr id="S6.T3.6.6.6.8.2" class="ltx_tr">
<th id="S6.T3.6.6.6.8.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S6.T3.6.6.6.8.2.2" class="ltx_td ltx_align_center">(acc)</td>
<td id="S6.T3.6.6.6.8.2.3" class="ltx_td ltx_align_center">(acc)</td>
<td id="S6.T3.6.6.6.8.2.4" class="ltx_td ltx_align_center">(acc)</td>
<td id="S6.T3.6.6.6.8.2.5" class="ltx_td ltx_align_center">(acc)</td>
<td id="S6.T3.6.6.6.8.2.6" class="ltx_td ltx_align_center">(acc)</td>
<td id="S6.T3.6.6.6.8.2.7" class="ltx_td ltx_align_center">(acc)</td>
<td id="S6.T3.6.6.6.8.2.8" class="ltx_td ltx_align_center">(acc)</td>
<td id="S6.T3.6.6.6.8.2.9" class="ltx_td ltx_align_center">(Matt.)</td>
</tr>
<tr id="S6.T3.1.1.1.1" class="ltx_tr">
<th id="S6.T3.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Majority<sup id="S6.T3.1.1.1.1.1.1" class="ltx_sup">†</sup>
</th>
<td id="S6.T3.1.1.1.1.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.1.1.1.1.2.1" class="ltx_text ltx_font_italic">50.9</span></td>
<td id="S6.T3.1.1.1.1.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.1.1.1.1.3.1" class="ltx_text ltx_font_italic">23.1</span></td>
<td id="S6.T3.1.1.1.1.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.1.1.1.1.4.1" class="ltx_text ltx_font_italic">50.0</span></td>
<td id="S6.T3.1.1.1.1.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.1.1.1.1.5.1" class="ltx_text ltx_font_italic">50.0</span></td>
<td id="S6.T3.1.1.1.1.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.1.1.1.1.6.1" class="ltx_text ltx_font_italic">50.0</span></td>
<td id="S6.T3.1.1.1.1.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.1.1.1.1.7.1" class="ltx_text ltx_font_italic">50.0</span></td>
<td id="S6.T3.1.1.1.1.8" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.1.1.1.1.8.1" class="ltx_text ltx_font_italic">18.8</span></td>
<td id="S6.T3.1.1.1.1.9" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.1.1.1.1.9.1" class="ltx_text ltx_font_italic">0.0</span></td>
</tr>
<tr id="S6.T3.2.2.2.2" class="ltx_tr">
<th id="S6.T3.2.2.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Prompt-based zero-shot<sup id="S6.T3.2.2.2.2.1.1" class="ltx_sup">‡</sup>
</th>
<td id="S6.T3.2.2.2.2.2" class="ltx_td ltx_align_center">83.6</td>
<td id="S6.T3.2.2.2.2.3" class="ltx_td ltx_align_center">35.0</td>
<td id="S6.T3.2.2.2.2.4" class="ltx_td ltx_align_center">80.8</td>
<td id="S6.T3.2.2.2.2.5" class="ltx_td ltx_align_center">79.5</td>
<td id="S6.T3.2.2.2.2.6" class="ltx_td ltx_align_center">67.6</td>
<td id="S6.T3.2.2.2.2.7" class="ltx_td ltx_align_center">51.4</td>
<td id="S6.T3.2.2.2.2.8" class="ltx_td ltx_align_center">32.0</td>
<td id="S6.T3.2.2.2.2.9" class="ltx_td ltx_align_center">2.0</td>
</tr>
<tr id="S6.T3.6.6.6.9.3" class="ltx_tr">
<th id="S6.T3.6.6.6.9.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">“GPT-3” in-context learning</th>
<td id="S6.T3.6.6.6.9.3.2" class="ltx_td ltx_align_center">84.8 (1.3)</td>
<td id="S6.T3.6.6.6.9.3.3" class="ltx_td ltx_align_center">30.6 (0.9)</td>
<td id="S6.T3.6.6.6.9.3.4" class="ltx_td ltx_align_center">80.5 (1.7)</td>
<td id="S6.T3.6.6.6.9.3.5" class="ltx_td ltx_align_center">87.4 (0.8)</td>
<td id="S6.T3.6.6.6.9.3.6" class="ltx_td ltx_align_center">63.8 (2.1)</td>
<td id="S6.T3.6.6.6.9.3.7" class="ltx_td ltx_align_center">53.6 (1.0)</td>
<td id="S6.T3.6.6.6.9.3.8" class="ltx_td ltx_align_center">26.2 (2.4)</td>
<td id="S6.T3.6.6.6.9.3.9" class="ltx_td ltx_align_center">-1.5 (2.4)</td>
</tr>
<tr id="S6.T3.6.6.6.10.4" class="ltx_tr">
<th id="S6.T3.6.6.6.10.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Fine-tuning</th>
<td id="S6.T3.6.6.6.10.4.2" class="ltx_td ltx_align_center">81.4 (3.8)</td>
<td id="S6.T3.6.6.6.10.4.3" class="ltx_td ltx_align_center">43.9 (2.0)</td>
<td id="S6.T3.6.6.6.10.4.4" class="ltx_td ltx_align_center">76.9 (5.9)</td>
<td id="S6.T3.6.6.6.10.4.5" class="ltx_td ltx_align_center">75.8 (3.2)</td>
<td id="S6.T3.6.6.6.10.4.6" class="ltx_td ltx_align_center">72.0 (3.8)</td>
<td id="S6.T3.6.6.6.10.4.7" class="ltx_td ltx_align_center">90.8 (1.8)</td>
<td id="S6.T3.6.6.6.10.4.8" class="ltx_td ltx_align_center">88.8 (2.1)</td>
<td id="S6.T3.6.6.6.10.4.9" class="ltx_td ltx_align_center">
<span id="S6.T3.6.6.6.10.4.9.1" class="ltx_text ltx_font_bold">33.9</span> (14.3)</td>
</tr>
<tr id="S6.T3.6.6.6.11.5" class="ltx_tr">
<th id="S6.T3.6.6.6.11.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Prompt-based FT (man)</th>
<td id="S6.T3.6.6.6.11.5.2" class="ltx_td ltx_align_center ltx_border_t">92.7 (0.9)</td>
<td id="S6.T3.6.6.6.11.5.3" class="ltx_td ltx_align_center ltx_border_t">47.4 (2.5)</td>
<td id="S6.T3.6.6.6.11.5.4" class="ltx_td ltx_align_center ltx_border_t">87.0 (1.2)</td>
<td id="S6.T3.6.6.6.11.5.5" class="ltx_td ltx_align_center ltx_border_t">90.3 (1.0)</td>
<td id="S6.T3.6.6.6.11.5.6" class="ltx_td ltx_align_center ltx_border_t">84.7 (2.2)</td>
<td id="S6.T3.6.6.6.11.5.7" class="ltx_td ltx_align_center ltx_border_t">91.2 (1.1)</td>
<td id="S6.T3.6.6.6.11.5.8" class="ltx_td ltx_align_center ltx_border_t">84.8 (5.1)</td>
<td id="S6.T3.6.6.6.11.5.9" class="ltx_td ltx_align_center ltx_border_t">9.3 (7.3)</td>
</tr>
<tr id="S6.T3.6.6.6.12.6" class="ltx_tr">
<th id="S6.T3.6.6.6.12.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">+ demonstrations</th>
<td id="S6.T3.6.6.6.12.6.2" class="ltx_td ltx_align_center">92.6 (0.5)</td>
<td id="S6.T3.6.6.6.12.6.3" class="ltx_td ltx_align_center">
<span id="S6.T3.6.6.6.12.6.3.1" class="ltx_text ltx_font_bold">50.6</span> (1.4)</td>
<td id="S6.T3.6.6.6.12.6.4" class="ltx_td ltx_align_center">86.6 (2.2)</td>
<td id="S6.T3.6.6.6.12.6.5" class="ltx_td ltx_align_center">90.2 (1.2)</td>
<td id="S6.T3.6.6.6.12.6.6" class="ltx_td ltx_align_center">
<span id="S6.T3.6.6.6.12.6.6.1" class="ltx_text ltx_font_bold">87.0</span> (1.1)</td>
<td id="S6.T3.6.6.6.12.6.7" class="ltx_td ltx_align_center">
<span id="S6.T3.6.6.6.12.6.7.1" class="ltx_text ltx_font_bold">92.3</span> (0.8)</td>
<td id="S6.T3.6.6.6.12.6.8" class="ltx_td ltx_align_center">87.5 (3.2)</td>
<td id="S6.T3.6.6.6.12.6.9" class="ltx_td ltx_align_center">18.7 (8.8)</td>
</tr>
<tr id="S6.T3.6.6.6.13.7" class="ltx_tr">
<th id="S6.T3.6.6.6.13.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Prompt-based FT (auto)</th>
<td id="S6.T3.6.6.6.13.7.2" class="ltx_td ltx_align_center">92.3 (1.0)</td>
<td id="S6.T3.6.6.6.13.7.3" class="ltx_td ltx_align_center">49.2 (1.6)</td>
<td id="S6.T3.6.6.6.13.7.4" class="ltx_td ltx_align_center">85.5 (2.8)</td>
<td id="S6.T3.6.6.6.13.7.5" class="ltx_td ltx_align_center">89.0 (1.4)</td>
<td id="S6.T3.6.6.6.13.7.6" class="ltx_td ltx_align_center">85.8 (1.9)</td>
<td id="S6.T3.6.6.6.13.7.7" class="ltx_td ltx_align_center">91.2 (1.1)</td>
<td id="S6.T3.6.6.6.13.7.8" class="ltx_td ltx_align_center">88.2 (2.0)</td>
<td id="S6.T3.6.6.6.13.7.9" class="ltx_td ltx_align_center">14.0 (14.1)</td>
</tr>
<tr id="S6.T3.6.6.6.14.8" class="ltx_tr">
<th id="S6.T3.6.6.6.14.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">+ demonstrations</th>
<td id="S6.T3.6.6.6.14.8.2" class="ltx_td ltx_align_center">
<span id="S6.T3.6.6.6.14.8.2.1" class="ltx_text ltx_font_bold">93.0</span> (0.6)</td>
<td id="S6.T3.6.6.6.14.8.3" class="ltx_td ltx_align_center">49.5 (1.7)</td>
<td id="S6.T3.6.6.6.14.8.4" class="ltx_td ltx_align_center">
<span id="S6.T3.6.6.6.14.8.4.1" class="ltx_text ltx_font_bold">87.7</span> (1.4)</td>
<td id="S6.T3.6.6.6.14.8.5" class="ltx_td ltx_align_center">
<span id="S6.T3.6.6.6.14.8.5.1" class="ltx_text ltx_font_bold">91.0</span> (0.9)</td>
<td id="S6.T3.6.6.6.14.8.6" class="ltx_td ltx_align_center">86.5 (2.6)</td>
<td id="S6.T3.6.6.6.14.8.7" class="ltx_td ltx_align_center">91.4 (1.8)</td>
<td id="S6.T3.6.6.6.14.8.8" class="ltx_td ltx_align_center">
<span id="S6.T3.6.6.6.14.8.8.1" class="ltx_text ltx_font_bold">89.4</span> (1.7)</td>
<td id="S6.T3.6.6.6.14.8.9" class="ltx_td ltx_align_center">21.8 (15.9)</td>
</tr>
<tr id="S6.T3.3.3.3.3" class="ltx_tr">
<th id="S6.T3.3.3.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Fine-tuning (full)<sup id="S6.T3.3.3.3.3.1.1" class="ltx_sup">†</sup>
</th>
<td id="S6.T3.3.3.3.3.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.3.3.3.3.2.1" class="ltx_text ltx_font_italic">95.0</span></td>
<td id="S6.T3.3.3.3.3.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.3.3.3.3.3.1" class="ltx_text ltx_font_italic">58.7</span></td>
<td id="S6.T3.3.3.3.3.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.3.3.3.3.4.1" class="ltx_text ltx_font_italic">90.8</span></td>
<td id="S6.T3.3.3.3.3.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.3.3.3.3.5.1" class="ltx_text ltx_font_italic">89.4</span></td>
<td id="S6.T3.3.3.3.3.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.3.3.3.3.6.1" class="ltx_text ltx_font_italic">87.8</span></td>
<td id="S6.T3.3.3.3.3.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.3.3.3.3.7.1" class="ltx_text ltx_font_italic">97.0</span></td>
<td id="S6.T3.3.3.3.3.8" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.3.3.3.3.8.1" class="ltx_text ltx_font_italic">97.4</span></td>
<td id="S6.T3.3.3.3.3.9" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.3.3.3.3.9.1" class="ltx_text ltx_font_italic">62.6</span></td>
</tr>
<tr id="S6.T3.6.6.6.15.9" class="ltx_tr">
<th id="S6.T3.6.6.6.15.9.1" class="ltx_td ltx_th ltx_th_row ltx_border_t"></th>
<td id="S6.T3.6.6.6.15.9.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.6.6.6.15.9.2.1" class="ltx_text ltx_font_bold">MNLI</span></td>
<td id="S6.T3.6.6.6.15.9.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.6.6.6.15.9.3.1" class="ltx_text ltx_font_bold">MNLI-mm</span></td>
<td id="S6.T3.6.6.6.15.9.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.6.6.6.15.9.4.1" class="ltx_text ltx_font_bold">SNLI</span></td>
<td id="S6.T3.6.6.6.15.9.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.6.6.6.15.9.5.1" class="ltx_text ltx_font_bold">QNLI</span></td>
<td id="S6.T3.6.6.6.15.9.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.6.6.6.15.9.6.1" class="ltx_text ltx_font_bold">RTE</span></td>
<td id="S6.T3.6.6.6.15.9.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.6.6.6.15.9.7.1" class="ltx_text ltx_font_bold">MRPC</span></td>
<td id="S6.T3.6.6.6.15.9.8" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.6.6.6.15.9.8.1" class="ltx_text ltx_font_bold">QQP</span></td>
<td id="S6.T3.6.6.6.15.9.9" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.6.6.6.15.9.9.1" class="ltx_text ltx_font_bold">STS-B</span></td>
</tr>
<tr id="S6.T3.6.6.6.16.10" class="ltx_tr">
<th id="S6.T3.6.6.6.16.10.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S6.T3.6.6.6.16.10.2" class="ltx_td ltx_align_center">(acc)</td>
<td id="S6.T3.6.6.6.16.10.3" class="ltx_td ltx_align_center">(acc)</td>
<td id="S6.T3.6.6.6.16.10.4" class="ltx_td ltx_align_center">(acc)</td>
<td id="S6.T3.6.6.6.16.10.5" class="ltx_td ltx_align_center">(acc)</td>
<td id="S6.T3.6.6.6.16.10.6" class="ltx_td ltx_align_center">(acc)</td>
<td id="S6.T3.6.6.6.16.10.7" class="ltx_td ltx_align_center">(F1)</td>
<td id="S6.T3.6.6.6.16.10.8" class="ltx_td ltx_align_center">(F1)</td>
<td id="S6.T3.6.6.6.16.10.9" class="ltx_td ltx_align_center">(Pear.)</td>
</tr>
<tr id="S6.T3.4.4.4.4" class="ltx_tr">
<th id="S6.T3.4.4.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Majority<sup id="S6.T3.4.4.4.4.1.1" class="ltx_sup">†</sup>
</th>
<td id="S6.T3.4.4.4.4.2" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.4.4.4.4.2.1" class="ltx_text ltx_font_italic">32.7</span></td>
<td id="S6.T3.4.4.4.4.3" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.4.4.4.4.3.1" class="ltx_text ltx_font_italic">33.0</span></td>
<td id="S6.T3.4.4.4.4.4" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.4.4.4.4.4.1" class="ltx_text ltx_font_italic">33.8</span></td>
<td id="S6.T3.4.4.4.4.5" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.4.4.4.4.5.1" class="ltx_text ltx_font_italic">49.5</span></td>
<td id="S6.T3.4.4.4.4.6" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.4.4.4.4.6.1" class="ltx_text ltx_font_italic">52.7</span></td>
<td id="S6.T3.4.4.4.4.7" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.4.4.4.4.7.1" class="ltx_text ltx_font_italic">81.2</span></td>
<td id="S6.T3.4.4.4.4.8" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.4.4.4.4.8.1" class="ltx_text ltx_font_italic">0.0</span></td>
<td id="S6.T3.4.4.4.4.9" class="ltx_td ltx_align_center ltx_border_t"><span id="S6.T3.4.4.4.4.9.1" class="ltx_text ltx_font_italic">-</span></td>
</tr>
<tr id="S6.T3.5.5.5.5" class="ltx_tr">
<th id="S6.T3.5.5.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Prompt-based zero-shot<sup id="S6.T3.5.5.5.5.1.1" class="ltx_sup">‡</sup>
</th>
<td id="S6.T3.5.5.5.5.2" class="ltx_td ltx_align_center">50.8</td>
<td id="S6.T3.5.5.5.5.3" class="ltx_td ltx_align_center">51.7</td>
<td id="S6.T3.5.5.5.5.4" class="ltx_td ltx_align_center">49.5</td>
<td id="S6.T3.5.5.5.5.5" class="ltx_td ltx_align_center">50.8</td>
<td id="S6.T3.5.5.5.5.6" class="ltx_td ltx_align_center">51.3</td>
<td id="S6.T3.5.5.5.5.7" class="ltx_td ltx_align_center">61.9</td>
<td id="S6.T3.5.5.5.5.8" class="ltx_td ltx_align_center">49.7</td>
<td id="S6.T3.5.5.5.5.9" class="ltx_td ltx_align_center">-3.2</td>
</tr>
<tr id="S6.T3.6.6.6.17.11" class="ltx_tr">
<th id="S6.T3.6.6.6.17.11.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">“GPT-3” in-context learning</th>
<td id="S6.T3.6.6.6.17.11.2" class="ltx_td ltx_align_center">52.0 (0.7)</td>
<td id="S6.T3.6.6.6.17.11.3" class="ltx_td ltx_align_center">53.4 (0.6)</td>
<td id="S6.T3.6.6.6.17.11.4" class="ltx_td ltx_align_center">47.1 (0.6)</td>
<td id="S6.T3.6.6.6.17.11.5" class="ltx_td ltx_align_center">53.8 (0.4)</td>
<td id="S6.T3.6.6.6.17.11.6" class="ltx_td ltx_align_center">60.4 (1.4)</td>
<td id="S6.T3.6.6.6.17.11.7" class="ltx_td ltx_align_center">45.7 (6.0)</td>
<td id="S6.T3.6.6.6.17.11.8" class="ltx_td ltx_align_center">36.1 (5.2)</td>
<td id="S6.T3.6.6.6.17.11.9" class="ltx_td ltx_align_center">14.3 (2.8)</td>
</tr>
<tr id="S6.T3.6.6.6.18.12" class="ltx_tr">
<th id="S6.T3.6.6.6.18.12.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Fine-tuning</th>
<td id="S6.T3.6.6.6.18.12.2" class="ltx_td ltx_align_center">45.8 (6.4)</td>
<td id="S6.T3.6.6.6.18.12.3" class="ltx_td ltx_align_center">47.8 (6.8)</td>
<td id="S6.T3.6.6.6.18.12.4" class="ltx_td ltx_align_center">48.4 (4.8)</td>
<td id="S6.T3.6.6.6.18.12.5" class="ltx_td ltx_align_center">60.2 (6.5)</td>
<td id="S6.T3.6.6.6.18.12.6" class="ltx_td ltx_align_center">54.4 (3.9)</td>
<td id="S6.T3.6.6.6.18.12.7" class="ltx_td ltx_align_center">76.6 (2.5)</td>
<td id="S6.T3.6.6.6.18.12.8" class="ltx_td ltx_align_center">60.7 (4.3)</td>
<td id="S6.T3.6.6.6.18.12.9" class="ltx_td ltx_align_center">53.5 (8.5)</td>
</tr>
<tr id="S6.T3.6.6.6.19.13" class="ltx_tr">
<th id="S6.T3.6.6.6.19.13.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Prompt-based FT (man)</th>
<td id="S6.T3.6.6.6.19.13.2" class="ltx_td ltx_align_center ltx_border_t">68.3 (2.3)</td>
<td id="S6.T3.6.6.6.19.13.3" class="ltx_td ltx_align_center ltx_border_t">70.5 (1.9)</td>
<td id="S6.T3.6.6.6.19.13.4" class="ltx_td ltx_align_center ltx_border_t">77.2 (3.7)</td>
<td id="S6.T3.6.6.6.19.13.5" class="ltx_td ltx_align_center ltx_border_t">64.5 (4.2)</td>
<td id="S6.T3.6.6.6.19.13.6" class="ltx_td ltx_align_center ltx_border_t">69.1 (3.6)</td>
<td id="S6.T3.6.6.6.19.13.7" class="ltx_td ltx_align_center ltx_border_t">74.5 (5.3)</td>
<td id="S6.T3.6.6.6.19.13.8" class="ltx_td ltx_align_center ltx_border_t">65.5 (5.3)</td>
<td id="S6.T3.6.6.6.19.13.9" class="ltx_td ltx_align_center ltx_border_t">71.0 (7.0)</td>
</tr>
<tr id="S6.T3.6.6.6.20.14" class="ltx_tr">
<th id="S6.T3.6.6.6.20.14.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">+ demonstrations</th>
<td id="S6.T3.6.6.6.20.14.2" class="ltx_td ltx_align_center">
<span id="S6.T3.6.6.6.20.14.2.1" class="ltx_text ltx_font_bold">70.7</span> (1.3)</td>
<td id="S6.T3.6.6.6.20.14.3" class="ltx_td ltx_align_center">
<span id="S6.T3.6.6.6.20.14.3.1" class="ltx_text ltx_font_bold">72.0</span> (1.2)</td>
<td id="S6.T3.6.6.6.20.14.4" class="ltx_td ltx_align_center">
<span id="S6.T3.6.6.6.20.14.4.1" class="ltx_text ltx_font_bold">79.7</span> (1.5)</td>
<td id="S6.T3.6.6.6.20.14.5" class="ltx_td ltx_align_center">
<span id="S6.T3.6.6.6.20.14.5.1" class="ltx_text ltx_font_bold">69.2</span> (1.9)</td>
<td id="S6.T3.6.6.6.20.14.6" class="ltx_td ltx_align_center">68.7 (2.3)</td>
<td id="S6.T3.6.6.6.20.14.7" class="ltx_td ltx_align_center">77.8 (2.0)</td>
<td id="S6.T3.6.6.6.20.14.8" class="ltx_td ltx_align_center">
<span id="S6.T3.6.6.6.20.14.8.1" class="ltx_text ltx_font_bold">69.8</span> (1.8)</td>
<td id="S6.T3.6.6.6.20.14.9" class="ltx_td ltx_align_center">73.5 (5.1)</td>
</tr>
<tr id="S6.T3.6.6.6.21.15" class="ltx_tr">
<th id="S6.T3.6.6.6.21.15.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Prompt-based FT (auto)</th>
<td id="S6.T3.6.6.6.21.15.2" class="ltx_td ltx_align_center">68.3 (2.5)</td>
<td id="S6.T3.6.6.6.21.15.3" class="ltx_td ltx_align_center">70.1 (2.6)</td>
<td id="S6.T3.6.6.6.21.15.4" class="ltx_td ltx_align_center">77.1 (2.1)</td>
<td id="S6.T3.6.6.6.21.15.5" class="ltx_td ltx_align_center">68.3 (7.4)</td>
<td id="S6.T3.6.6.6.21.15.6" class="ltx_td ltx_align_center">
<span id="S6.T3.6.6.6.21.15.6.1" class="ltx_text ltx_font_bold">73.9</span> (2.2)</td>
<td id="S6.T3.6.6.6.21.15.7" class="ltx_td ltx_align_center">76.2 (2.3)</td>
<td id="S6.T3.6.6.6.21.15.8" class="ltx_td ltx_align_center">67.0 (3.0)</td>
<td id="S6.T3.6.6.6.21.15.9" class="ltx_td ltx_align_center">75.0 (3.3)</td>
</tr>
<tr id="S6.T3.6.6.6.22.16" class="ltx_tr">
<th id="S6.T3.6.6.6.22.16.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">+ demonstrations</th>
<td id="S6.T3.6.6.6.22.16.2" class="ltx_td ltx_align_center">70.0 (3.6)</td>
<td id="S6.T3.6.6.6.22.16.3" class="ltx_td ltx_align_center">
<span id="S6.T3.6.6.6.22.16.3.1" class="ltx_text ltx_font_bold">72.0</span> (3.1)</td>
<td id="S6.T3.6.6.6.22.16.4" class="ltx_td ltx_align_center">77.5 (3.5)</td>
<td id="S6.T3.6.6.6.22.16.5" class="ltx_td ltx_align_center">68.5 (5.4)</td>
<td id="S6.T3.6.6.6.22.16.6" class="ltx_td ltx_align_center">71.1 (5.3)</td>
<td id="S6.T3.6.6.6.22.16.7" class="ltx_td ltx_align_center">
<span id="S6.T3.6.6.6.22.16.7.1" class="ltx_text ltx_font_bold">78.1</span> (3.4)</td>
<td id="S6.T3.6.6.6.22.16.8" class="ltx_td ltx_align_center">67.7 (5.8)</td>
<td id="S6.T3.6.6.6.22.16.9" class="ltx_td ltx_align_center">
<span id="S6.T3.6.6.6.22.16.9.1" class="ltx_text ltx_font_bold">76.4</span> (6.2)</td>
</tr>
<tr id="S6.T3.6.6.6.6" class="ltx_tr">
<th id="S6.T3.6.6.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb ltx_border_t">Fine-tuning (full)<sup id="S6.T3.6.6.6.6.1.1" class="ltx_sup">†</sup>
</th>
<td id="S6.T3.6.6.6.6.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S6.T3.6.6.6.6.2.1" class="ltx_text ltx_font_italic">89.8</span></td>
<td id="S6.T3.6.6.6.6.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S6.T3.6.6.6.6.3.1" class="ltx_text ltx_font_italic">89.5</span></td>
<td id="S6.T3.6.6.6.6.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S6.T3.6.6.6.6.4.1" class="ltx_text ltx_font_italic">92.6</span></td>
<td id="S6.T3.6.6.6.6.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S6.T3.6.6.6.6.5.1" class="ltx_text ltx_font_italic">93.3</span></td>
<td id="S6.T3.6.6.6.6.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S6.T3.6.6.6.6.6.1" class="ltx_text ltx_font_italic">80.9</span></td>
<td id="S6.T3.6.6.6.6.7" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S6.T3.6.6.6.6.7.1" class="ltx_text ltx_font_italic">91.4</span></td>
<td id="S6.T3.6.6.6.6.8" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S6.T3.6.6.6.6.8.1" class="ltx_text ltx_font_italic">81.7</span></td>
<td id="S6.T3.6.6.6.6.9" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t"><span id="S6.T3.6.6.6.6.9.1" class="ltx_text ltx_font_italic">91.9</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 3: </span>
Our main results using RoBERTa-large.
<math id="S6.T3.10.m1.1" class="ltx_Math" alttext="\dagger" display="inline"><semantics id="S6.T3.10.m1.1b"><mo id="S6.T3.10.m1.1.1" xref="S6.T3.10.m1.1.1.cmml">†</mo><annotation-xml encoding="MathML-Content" id="S6.T3.10.m1.1c"><ci id="S6.T3.10.m1.1.1.cmml" xref="S6.T3.10.m1.1.1">†</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.T3.10.m1.1d">\dagger</annotation></semantics></math>: full training set is used (see dataset sizes in
Table <a href="#A2.T1" title="Table B.1 ‣ Appendix B Datasets ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B.1</span></a>);
<math id="S6.T3.11.m2.1" class="ltx_Math" alttext="\ddagger" display="inline"><semantics id="S6.T3.11.m2.1b"><mo id="S6.T3.11.m2.1.1" xref="S6.T3.11.m2.1.1.cmml">‡</mo><annotation-xml encoding="MathML-Content" id="S6.T3.11.m2.1c"><ci id="S6.T3.11.m2.1.1.cmml" xref="S6.T3.11.m2.1.1">‡</ci></annotation-xml><annotation encoding="application/x-tex" id="S6.T3.11.m2.1d">\ddagger</annotation></semantics></math>: no training examples are used; otherwise we use
<math id="S6.T3.12.m3.1" class="ltx_Math" alttext="K=16" display="inline"><semantics id="S6.T3.12.m3.1b"><mrow id="S6.T3.12.m3.1.1" xref="S6.T3.12.m3.1.1.cmml"><mi id="S6.T3.12.m3.1.1.2" xref="S6.T3.12.m3.1.1.2.cmml">K</mi><mo id="S6.T3.12.m3.1.1.1" xref="S6.T3.12.m3.1.1.1.cmml">=</mo><mn id="S6.T3.12.m3.1.1.3" xref="S6.T3.12.m3.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S6.T3.12.m3.1c"><apply id="S6.T3.12.m3.1.1.cmml" xref="S6.T3.12.m3.1.1"><eq id="S6.T3.12.m3.1.1.1.cmml" xref="S6.T3.12.m3.1.1.1"></eq><ci id="S6.T3.12.m3.1.1.2.cmml" xref="S6.T3.12.m3.1.1.2">𝐾</ci><cn type="integer" id="S6.T3.12.m3.1.1.3.cmml" xref="S6.T3.12.m3.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.T3.12.m3.1d">K=16</annotation></semantics></math> (per class)
for few-shot experiments.
We report mean (and standard deviation) performance over 5 different splits (§<a href="#S3" title="3 Problem Setup ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>).
Majority: majority class;
FT: fine-tuning;
man: manual prompt
(Table <a href="#S2.T1" title="Table 1 ‣ Few-shot learning. ‣ 2 Related Work ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>);
auto: automatically searched templates (§<a href="#S5.SS2" title="5.2 Automatic generation of templates ‣ 5 Automatic Prompt Generation ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5.2</span></a>);
“GPT-3” in-context learning: using the in-context learning proposed in <cite class="ltx_cite ltx_citemacro_citet">Brown et al. (<a href="#bib.bib8" title="" class="ltx_ref">2020</a>)</cite> with RoBERTa-large (no parameter updates).
</figcaption>
</figure>
<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">In this section, we study whether we can leverage demonstrations when <span id="S6.p1.1.1" class="ltx_text ltx_font_italic">fine-tuning</span> medium-sized LMs, and find better ways to exploit them.</p>
</div>
<section id="S6.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.1 </span>Training examples as demonstrations</h3>

<div id="S6.SS1.p1" class="ltx_para">
<p id="S6.SS1.p1.5" class="ltx_p">GPT-3’s naive approach to in-context learning simply involves concatenating the input with up to 32 examples randomly drawn from the training set.
This approach is suboptimal as
(1) the number of available demonstrations is bounded by the model’s maximum input length;<span id="footnote8" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">8</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">8</sup><span class="ltx_tag ltx_tag_note">8</span>GPT-3 uses a context size of 2,048 while most smaller language models (e.g., RoBERTa) have a context size of 512.</span></span></span>
and (2) mixing numerous random examples from different classes together creates extremely long contexts which can be hard to leverage, especially for a smaller model.
To address these issues,
we propose a simpler solution:
at each training step,
we randomly sample <span id="S6.SS1.p1.5.1" class="ltx_text ltx_font_italic">one<span id="footnote9" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">9</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">9</sup><span class="ltx_tag ltx_tag_note"><span id="footnote9.1.1.1" class="ltx_text ltx_font_upright">9</span></span><span id="footnote9.5" class="ltx_text ltx_font_upright">We also explored sampling multiple examples per class, but did not observe any improvements.</span></span></span></span></span> example <math id="S6.SS1.p1.1.m1.4" class="ltx_Math" alttext="\big{(}{x}_{\mathrm{in}}^{(c)},y_{\phantom{t}}^{(c)}\big{)}\in\mathcal{D}_{\text{train}}" display="inline"><semantics id="S6.SS1.p1.1.m1.4a"><mrow id="S6.SS1.p1.1.m1.4.4" xref="S6.SS1.p1.1.m1.4.4.cmml"><mrow id="S6.SS1.p1.1.m1.4.4.2.2" xref="S6.SS1.p1.1.m1.4.4.2.3.cmml"><mo maxsize="120%" minsize="120%" id="S6.SS1.p1.1.m1.4.4.2.2.3" xref="S6.SS1.p1.1.m1.4.4.2.3.cmml">(</mo><msubsup id="S6.SS1.p1.1.m1.3.3.1.1.1" xref="S6.SS1.p1.1.m1.3.3.1.1.1.cmml"><mi id="S6.SS1.p1.1.m1.3.3.1.1.1.2.2" xref="S6.SS1.p1.1.m1.3.3.1.1.1.2.2.cmml">x</mi><mi id="S6.SS1.p1.1.m1.3.3.1.1.1.2.3" xref="S6.SS1.p1.1.m1.3.3.1.1.1.2.3.cmml">in</mi><mrow id="S6.SS1.p1.1.m1.1.1.1.3" xref="S6.SS1.p1.1.m1.3.3.1.1.1.cmml"><mo stretchy="false" id="S6.SS1.p1.1.m1.1.1.1.3.1" xref="S6.SS1.p1.1.m1.3.3.1.1.1.cmml">(</mo><mi id="S6.SS1.p1.1.m1.1.1.1.1" xref="S6.SS1.p1.1.m1.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S6.SS1.p1.1.m1.1.1.1.3.2" xref="S6.SS1.p1.1.m1.3.3.1.1.1.cmml">)</mo></mrow></msubsup><mo id="S6.SS1.p1.1.m1.4.4.2.2.4" xref="S6.SS1.p1.1.m1.4.4.2.3.cmml">,</mo><msubsup id="S6.SS1.p1.1.m1.4.4.2.2.2" xref="S6.SS1.p1.1.m1.4.4.2.2.2.cmml"><mi id="S6.SS1.p1.1.m1.4.4.2.2.2.2.2" xref="S6.SS1.p1.1.m1.4.4.2.2.2.2.2.cmml">y</mi><mi id="S6.SS1.p1.1.m1.4.4.2.2.2.2.3" xref="S6.SS1.p1.1.m1.4.4.2.2.2.2.3.cmml"></mi><mrow id="S6.SS1.p1.1.m1.2.2.1.3" xref="S6.SS1.p1.1.m1.4.4.2.2.2.cmml"><mo stretchy="false" id="S6.SS1.p1.1.m1.2.2.1.3.1" xref="S6.SS1.p1.1.m1.4.4.2.2.2.cmml">(</mo><mi id="S6.SS1.p1.1.m1.2.2.1.1" xref="S6.SS1.p1.1.m1.2.2.1.1.cmml">c</mi><mo stretchy="false" id="S6.SS1.p1.1.m1.2.2.1.3.2" xref="S6.SS1.p1.1.m1.4.4.2.2.2.cmml">)</mo></mrow></msubsup><mo maxsize="120%" minsize="120%" id="S6.SS1.p1.1.m1.4.4.2.2.5" xref="S6.SS1.p1.1.m1.4.4.2.3.cmml">)</mo></mrow><mo id="S6.SS1.p1.1.m1.4.4.3" xref="S6.SS1.p1.1.m1.4.4.3.cmml">∈</mo><msub id="S6.SS1.p1.1.m1.4.4.4" xref="S6.SS1.p1.1.m1.4.4.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.SS1.p1.1.m1.4.4.4.2" xref="S6.SS1.p1.1.m1.4.4.4.2.cmml">𝒟</mi><mtext id="S6.SS1.p1.1.m1.4.4.4.3" xref="S6.SS1.p1.1.m1.4.4.4.3a.cmml">train</mtext></msub></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.1.m1.4b"><apply id="S6.SS1.p1.1.m1.4.4.cmml" xref="S6.SS1.p1.1.m1.4.4"><in id="S6.SS1.p1.1.m1.4.4.3.cmml" xref="S6.SS1.p1.1.m1.4.4.3"></in><interval closure="open" id="S6.SS1.p1.1.m1.4.4.2.3.cmml" xref="S6.SS1.p1.1.m1.4.4.2.2"><apply id="S6.SS1.p1.1.m1.3.3.1.1.1.cmml" xref="S6.SS1.p1.1.m1.3.3.1.1.1"><csymbol cd="ambiguous" id="S6.SS1.p1.1.m1.3.3.1.1.1.1.cmml" xref="S6.SS1.p1.1.m1.3.3.1.1.1">superscript</csymbol><apply id="S6.SS1.p1.1.m1.3.3.1.1.1.2.cmml" xref="S6.SS1.p1.1.m1.3.3.1.1.1"><csymbol cd="ambiguous" id="S6.SS1.p1.1.m1.3.3.1.1.1.2.1.cmml" xref="S6.SS1.p1.1.m1.3.3.1.1.1">subscript</csymbol><ci id="S6.SS1.p1.1.m1.3.3.1.1.1.2.2.cmml" xref="S6.SS1.p1.1.m1.3.3.1.1.1.2.2">𝑥</ci><ci id="S6.SS1.p1.1.m1.3.3.1.1.1.2.3.cmml" xref="S6.SS1.p1.1.m1.3.3.1.1.1.2.3">in</ci></apply><ci id="S6.SS1.p1.1.m1.1.1.1.1.cmml" xref="S6.SS1.p1.1.m1.1.1.1.1">𝑐</ci></apply><apply id="S6.SS1.p1.1.m1.4.4.2.2.2.cmml" xref="S6.SS1.p1.1.m1.4.4.2.2.2"><csymbol cd="ambiguous" id="S6.SS1.p1.1.m1.4.4.2.2.2.1.cmml" xref="S6.SS1.p1.1.m1.4.4.2.2.2">superscript</csymbol><apply id="S6.SS1.p1.1.m1.4.4.2.2.2.2.cmml" xref="S6.SS1.p1.1.m1.4.4.2.2.2"><csymbol cd="ambiguous" id="S6.SS1.p1.1.m1.4.4.2.2.2.2.1.cmml" xref="S6.SS1.p1.1.m1.4.4.2.2.2">subscript</csymbol><ci id="S6.SS1.p1.1.m1.4.4.2.2.2.2.2.cmml" xref="S6.SS1.p1.1.m1.4.4.2.2.2.2.2">𝑦</ci><csymbol cd="latexml" id="S6.SS1.p1.1.m1.4.4.2.2.2.2.3.cmml" xref="S6.SS1.p1.1.m1.4.4.2.2.2.2.3">absent</csymbol></apply><ci id="S6.SS1.p1.1.m1.2.2.1.1.cmml" xref="S6.SS1.p1.1.m1.2.2.1.1">𝑐</ci></apply></interval><apply id="S6.SS1.p1.1.m1.4.4.4.cmml" xref="S6.SS1.p1.1.m1.4.4.4"><csymbol cd="ambiguous" id="S6.SS1.p1.1.m1.4.4.4.1.cmml" xref="S6.SS1.p1.1.m1.4.4.4">subscript</csymbol><ci id="S6.SS1.p1.1.m1.4.4.4.2.cmml" xref="S6.SS1.p1.1.m1.4.4.4.2">𝒟</ci><ci id="S6.SS1.p1.1.m1.4.4.4.3a.cmml" xref="S6.SS1.p1.1.m1.4.4.4.3"><mtext mathsize="70%" id="S6.SS1.p1.1.m1.4.4.4.3.cmml" xref="S6.SS1.p1.1.m1.4.4.4.3">train</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.1.m1.4c">\big{(}{x}_{\mathrm{in}}^{(c)},y_{\phantom{t}}^{(c)}\big{)}\in\mathcal{D}_{\text{train}}</annotation></semantics></math> from each class,
convert it into <math id="S6.SS1.p1.2.m2.2" class="ltx_Math" alttext="\mathcal{T}\big{(}{x}_{\mathrm{in}}^{(c)}\big{)}" display="inline"><semantics id="S6.SS1.p1.2.m2.2a"><mrow id="S6.SS1.p1.2.m2.2.2" xref="S6.SS1.p1.2.m2.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.SS1.p1.2.m2.2.2.3" xref="S6.SS1.p1.2.m2.2.2.3.cmml">𝒯</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.2.m2.2.2.2" xref="S6.SS1.p1.2.m2.2.2.2.cmml">​</mo><mrow id="S6.SS1.p1.2.m2.2.2.1.1" xref="S6.SS1.p1.2.m2.2.2.1.1.1.cmml"><mo maxsize="120%" minsize="120%" id="S6.SS1.p1.2.m2.2.2.1.1.2" xref="S6.SS1.p1.2.m2.2.2.1.1.1.cmml">(</mo><msubsup id="S6.SS1.p1.2.m2.2.2.1.1.1" xref="S6.SS1.p1.2.m2.2.2.1.1.1.cmml"><mi id="S6.SS1.p1.2.m2.2.2.1.1.1.2.2" xref="S6.SS1.p1.2.m2.2.2.1.1.1.2.2.cmml">x</mi><mi id="S6.SS1.p1.2.m2.2.2.1.1.1.2.3" xref="S6.SS1.p1.2.m2.2.2.1.1.1.2.3.cmml">in</mi><mrow id="S6.SS1.p1.2.m2.1.1.1.3" xref="S6.SS1.p1.2.m2.2.2.1.1.1.cmml"><mo stretchy="false" id="S6.SS1.p1.2.m2.1.1.1.3.1" xref="S6.SS1.p1.2.m2.2.2.1.1.1.cmml">(</mo><mi id="S6.SS1.p1.2.m2.1.1.1.1" xref="S6.SS1.p1.2.m2.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S6.SS1.p1.2.m2.1.1.1.3.2" xref="S6.SS1.p1.2.m2.2.2.1.1.1.cmml">)</mo></mrow></msubsup><mo maxsize="120%" minsize="120%" id="S6.SS1.p1.2.m2.2.2.1.1.3" xref="S6.SS1.p1.2.m2.2.2.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.2.m2.2b"><apply id="S6.SS1.p1.2.m2.2.2.cmml" xref="S6.SS1.p1.2.m2.2.2"><times id="S6.SS1.p1.2.m2.2.2.2.cmml" xref="S6.SS1.p1.2.m2.2.2.2"></times><ci id="S6.SS1.p1.2.m2.2.2.3.cmml" xref="S6.SS1.p1.2.m2.2.2.3">𝒯</ci><apply id="S6.SS1.p1.2.m2.2.2.1.1.1.cmml" xref="S6.SS1.p1.2.m2.2.2.1.1"><csymbol cd="ambiguous" id="S6.SS1.p1.2.m2.2.2.1.1.1.1.cmml" xref="S6.SS1.p1.2.m2.2.2.1.1">superscript</csymbol><apply id="S6.SS1.p1.2.m2.2.2.1.1.1.2.cmml" xref="S6.SS1.p1.2.m2.2.2.1.1"><csymbol cd="ambiguous" id="S6.SS1.p1.2.m2.2.2.1.1.1.2.1.cmml" xref="S6.SS1.p1.2.m2.2.2.1.1">subscript</csymbol><ci id="S6.SS1.p1.2.m2.2.2.1.1.1.2.2.cmml" xref="S6.SS1.p1.2.m2.2.2.1.1.1.2.2">𝑥</ci><ci id="S6.SS1.p1.2.m2.2.2.1.1.1.2.3.cmml" xref="S6.SS1.p1.2.m2.2.2.1.1.1.2.3">in</ci></apply><ci id="S6.SS1.p1.2.m2.1.1.1.1.cmml" xref="S6.SS1.p1.2.m2.1.1.1.1">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.2.m2.2c">\mathcal{T}\big{(}{x}_{\mathrm{in}}^{(c)}\big{)}</annotation></semantics></math> with <span id="S6.SS1.p1.5.2" class="ltx_text ltx_font_typewriter">[MASK]</span> replaced by <math id="S6.SS1.p1.3.m3.2" class="ltx_Math" alttext="\mathcal{M}(y_{\phantom{t}}^{(c)})" display="inline"><semantics id="S6.SS1.p1.3.m3.2a"><mrow id="S6.SS1.p1.3.m3.2.2" xref="S6.SS1.p1.3.m3.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.SS1.p1.3.m3.2.2.3" xref="S6.SS1.p1.3.m3.2.2.3.cmml">ℳ</mi><mo lspace="0em" rspace="0em" id="S6.SS1.p1.3.m3.2.2.2" xref="S6.SS1.p1.3.m3.2.2.2.cmml">​</mo><mrow id="S6.SS1.p1.3.m3.2.2.1.1" xref="S6.SS1.p1.3.m3.2.2.1.1.1.cmml"><mo stretchy="false" id="S6.SS1.p1.3.m3.2.2.1.1.2" xref="S6.SS1.p1.3.m3.2.2.1.1.1.cmml">(</mo><msubsup id="S6.SS1.p1.3.m3.2.2.1.1.1" xref="S6.SS1.p1.3.m3.2.2.1.1.1.cmml"><mi id="S6.SS1.p1.3.m3.2.2.1.1.1.2.2" xref="S6.SS1.p1.3.m3.2.2.1.1.1.2.2.cmml">y</mi><mi id="S6.SS1.p1.3.m3.2.2.1.1.1.2.3" xref="S6.SS1.p1.3.m3.2.2.1.1.1.2.3.cmml"></mi><mrow id="S6.SS1.p1.3.m3.1.1.1.3" xref="S6.SS1.p1.3.m3.2.2.1.1.1.cmml"><mo stretchy="false" id="S6.SS1.p1.3.m3.1.1.1.3.1" xref="S6.SS1.p1.3.m3.2.2.1.1.1.cmml">(</mo><mi id="S6.SS1.p1.3.m3.1.1.1.1" xref="S6.SS1.p1.3.m3.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S6.SS1.p1.3.m3.1.1.1.3.2" xref="S6.SS1.p1.3.m3.2.2.1.1.1.cmml">)</mo></mrow></msubsup><mo stretchy="false" id="S6.SS1.p1.3.m3.2.2.1.1.3" xref="S6.SS1.p1.3.m3.2.2.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.3.m3.2b"><apply id="S6.SS1.p1.3.m3.2.2.cmml" xref="S6.SS1.p1.3.m3.2.2"><times id="S6.SS1.p1.3.m3.2.2.2.cmml" xref="S6.SS1.p1.3.m3.2.2.2"></times><ci id="S6.SS1.p1.3.m3.2.2.3.cmml" xref="S6.SS1.p1.3.m3.2.2.3">ℳ</ci><apply id="S6.SS1.p1.3.m3.2.2.1.1.1.cmml" xref="S6.SS1.p1.3.m3.2.2.1.1"><csymbol cd="ambiguous" id="S6.SS1.p1.3.m3.2.2.1.1.1.1.cmml" xref="S6.SS1.p1.3.m3.2.2.1.1">superscript</csymbol><apply id="S6.SS1.p1.3.m3.2.2.1.1.1.2.cmml" xref="S6.SS1.p1.3.m3.2.2.1.1"><csymbol cd="ambiguous" id="S6.SS1.p1.3.m3.2.2.1.1.1.2.1.cmml" xref="S6.SS1.p1.3.m3.2.2.1.1">subscript</csymbol><ci id="S6.SS1.p1.3.m3.2.2.1.1.1.2.2.cmml" xref="S6.SS1.p1.3.m3.2.2.1.1.1.2.2">𝑦</ci><csymbol cd="latexml" id="S6.SS1.p1.3.m3.2.2.1.1.1.2.3.cmml" xref="S6.SS1.p1.3.m3.2.2.1.1.1.2.3">absent</csymbol></apply><ci id="S6.SS1.p1.3.m3.1.1.1.1.cmml" xref="S6.SS1.p1.3.m3.1.1.1.1">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.3.m3.2c">\mathcal{M}(y_{\phantom{t}}^{(c)})</annotation></semantics></math>—we denote this as <math id="S6.SS1.p1.4.m4.4" class="ltx_Math" alttext="\tilde{\mathcal{T}}\big{(}{x}_{\mathrm{in}}^{(c)},y_{\phantom{t}}^{(c)}\big{)}" display="inline"><semantics id="S6.SS1.p1.4.m4.4a"><mrow id="S6.SS1.p1.4.m4.4.4" xref="S6.SS1.p1.4.m4.4.4.cmml"><mover accent="true" id="S6.SS1.p1.4.m4.4.4.4" xref="S6.SS1.p1.4.m4.4.4.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.SS1.p1.4.m4.4.4.4.2" xref="S6.SS1.p1.4.m4.4.4.4.2.cmml">𝒯</mi><mo id="S6.SS1.p1.4.m4.4.4.4.1" xref="S6.SS1.p1.4.m4.4.4.4.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S6.SS1.p1.4.m4.4.4.3" xref="S6.SS1.p1.4.m4.4.4.3.cmml">​</mo><mrow id="S6.SS1.p1.4.m4.4.4.2.2" xref="S6.SS1.p1.4.m4.4.4.2.3.cmml"><mo maxsize="120%" minsize="120%" id="S6.SS1.p1.4.m4.4.4.2.2.3" xref="S6.SS1.p1.4.m4.4.4.2.3.cmml">(</mo><msubsup id="S6.SS1.p1.4.m4.3.3.1.1.1" xref="S6.SS1.p1.4.m4.3.3.1.1.1.cmml"><mi id="S6.SS1.p1.4.m4.3.3.1.1.1.2.2" xref="S6.SS1.p1.4.m4.3.3.1.1.1.2.2.cmml">x</mi><mi id="S6.SS1.p1.4.m4.3.3.1.1.1.2.3" xref="S6.SS1.p1.4.m4.3.3.1.1.1.2.3.cmml">in</mi><mrow id="S6.SS1.p1.4.m4.1.1.1.3" xref="S6.SS1.p1.4.m4.3.3.1.1.1.cmml"><mo stretchy="false" id="S6.SS1.p1.4.m4.1.1.1.3.1" xref="S6.SS1.p1.4.m4.3.3.1.1.1.cmml">(</mo><mi id="S6.SS1.p1.4.m4.1.1.1.1" xref="S6.SS1.p1.4.m4.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S6.SS1.p1.4.m4.1.1.1.3.2" xref="S6.SS1.p1.4.m4.3.3.1.1.1.cmml">)</mo></mrow></msubsup><mo id="S6.SS1.p1.4.m4.4.4.2.2.4" xref="S6.SS1.p1.4.m4.4.4.2.3.cmml">,</mo><msubsup id="S6.SS1.p1.4.m4.4.4.2.2.2" xref="S6.SS1.p1.4.m4.4.4.2.2.2.cmml"><mi id="S6.SS1.p1.4.m4.4.4.2.2.2.2.2" xref="S6.SS1.p1.4.m4.4.4.2.2.2.2.2.cmml">y</mi><mi id="S6.SS1.p1.4.m4.4.4.2.2.2.2.3" xref="S6.SS1.p1.4.m4.4.4.2.2.2.2.3.cmml"></mi><mrow id="S6.SS1.p1.4.m4.2.2.1.3" xref="S6.SS1.p1.4.m4.4.4.2.2.2.cmml"><mo stretchy="false" id="S6.SS1.p1.4.m4.2.2.1.3.1" xref="S6.SS1.p1.4.m4.4.4.2.2.2.cmml">(</mo><mi id="S6.SS1.p1.4.m4.2.2.1.1" xref="S6.SS1.p1.4.m4.2.2.1.1.cmml">c</mi><mo stretchy="false" id="S6.SS1.p1.4.m4.2.2.1.3.2" xref="S6.SS1.p1.4.m4.4.4.2.2.2.cmml">)</mo></mrow></msubsup><mo maxsize="120%" minsize="120%" id="S6.SS1.p1.4.m4.4.4.2.2.5" xref="S6.SS1.p1.4.m4.4.4.2.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.4.m4.4b"><apply id="S6.SS1.p1.4.m4.4.4.cmml" xref="S6.SS1.p1.4.m4.4.4"><times id="S6.SS1.p1.4.m4.4.4.3.cmml" xref="S6.SS1.p1.4.m4.4.4.3"></times><apply id="S6.SS1.p1.4.m4.4.4.4.cmml" xref="S6.SS1.p1.4.m4.4.4.4"><ci id="S6.SS1.p1.4.m4.4.4.4.1.cmml" xref="S6.SS1.p1.4.m4.4.4.4.1">~</ci><ci id="S6.SS1.p1.4.m4.4.4.4.2.cmml" xref="S6.SS1.p1.4.m4.4.4.4.2">𝒯</ci></apply><interval closure="open" id="S6.SS1.p1.4.m4.4.4.2.3.cmml" xref="S6.SS1.p1.4.m4.4.4.2.2"><apply id="S6.SS1.p1.4.m4.3.3.1.1.1.cmml" xref="S6.SS1.p1.4.m4.3.3.1.1.1"><csymbol cd="ambiguous" id="S6.SS1.p1.4.m4.3.3.1.1.1.1.cmml" xref="S6.SS1.p1.4.m4.3.3.1.1.1">superscript</csymbol><apply id="S6.SS1.p1.4.m4.3.3.1.1.1.2.cmml" xref="S6.SS1.p1.4.m4.3.3.1.1.1"><csymbol cd="ambiguous" id="S6.SS1.p1.4.m4.3.3.1.1.1.2.1.cmml" xref="S6.SS1.p1.4.m4.3.3.1.1.1">subscript</csymbol><ci id="S6.SS1.p1.4.m4.3.3.1.1.1.2.2.cmml" xref="S6.SS1.p1.4.m4.3.3.1.1.1.2.2">𝑥</ci><ci id="S6.SS1.p1.4.m4.3.3.1.1.1.2.3.cmml" xref="S6.SS1.p1.4.m4.3.3.1.1.1.2.3">in</ci></apply><ci id="S6.SS1.p1.4.m4.1.1.1.1.cmml" xref="S6.SS1.p1.4.m4.1.1.1.1">𝑐</ci></apply><apply id="S6.SS1.p1.4.m4.4.4.2.2.2.cmml" xref="S6.SS1.p1.4.m4.4.4.2.2.2"><csymbol cd="ambiguous" id="S6.SS1.p1.4.m4.4.4.2.2.2.1.cmml" xref="S6.SS1.p1.4.m4.4.4.2.2.2">superscript</csymbol><apply id="S6.SS1.p1.4.m4.4.4.2.2.2.2.cmml" xref="S6.SS1.p1.4.m4.4.4.2.2.2"><csymbol cd="ambiguous" id="S6.SS1.p1.4.m4.4.4.2.2.2.2.1.cmml" xref="S6.SS1.p1.4.m4.4.4.2.2.2">subscript</csymbol><ci id="S6.SS1.p1.4.m4.4.4.2.2.2.2.2.cmml" xref="S6.SS1.p1.4.m4.4.4.2.2.2.2.2">𝑦</ci><csymbol cd="latexml" id="S6.SS1.p1.4.m4.4.4.2.2.2.2.3.cmml" xref="S6.SS1.p1.4.m4.4.4.2.2.2.2.3">absent</csymbol></apply><ci id="S6.SS1.p1.4.m4.2.2.1.1.cmml" xref="S6.SS1.p1.4.m4.2.2.1.1">𝑐</ci></apply></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.4.m4.4c">\tilde{\mathcal{T}}\big{(}{x}_{\mathrm{in}}^{(c)},y_{\phantom{t}}^{(c)}\big{)}</annotation></semantics></math>—and then concatenate them with <math id="S6.SS1.p1.5.m5.1" class="ltx_Math" alttext="{x}_{\mathrm{in}}" display="inline"><semantics id="S6.SS1.p1.5.m5.1a"><msub id="S6.SS1.p1.5.m5.1.1" xref="S6.SS1.p1.5.m5.1.1.cmml"><mi id="S6.SS1.p1.5.m5.1.1.2" xref="S6.SS1.p1.5.m5.1.1.2.cmml">x</mi><mi id="S6.SS1.p1.5.m5.1.1.3" xref="S6.SS1.p1.5.m5.1.1.3.cmml">in</mi></msub><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.5.m5.1b"><apply id="S6.SS1.p1.5.m5.1.1.cmml" xref="S6.SS1.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S6.SS1.p1.5.m5.1.1.1.cmml" xref="S6.SS1.p1.5.m5.1.1">subscript</csymbol><ci id="S6.SS1.p1.5.m5.1.1.2.cmml" xref="S6.SS1.p1.5.m5.1.1.2">𝑥</ci><ci id="S6.SS1.p1.5.m5.1.1.3.cmml" xref="S6.SS1.p1.5.m5.1.1.3">in</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.5.m5.1c">{x}_{\mathrm{in}}</annotation></semantics></math> (Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>(c)):</p>
<table id="S6.Ex3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center">
<div id="S6.Ex3.m1.1.1.1" class="ltx_inline-block ltx_markedasmath ltx_transformed_outer" style="width:418.1pt;height:36.1pt;vertical-align:-9.0pt;"><span class="ltx_transformed_inner" style="transform:translate(139.5pt,-9.0pt) scale(3.00797118871261,3.00797118871261) ;">
<p id="S6.Ex3.m1.1.1.1.1" class="ltx_p"><math id="S6.Ex3.m1.1.1.1.1.m1.7" class="ltx_Math" alttext="\displaystyle\mathcal{T}\big{(}{x}_{\mathrm{in}}\big{)}\oplus\tilde{\mathcal{T}}\big{(}{x}_{\mathrm{in}}^{(1)},y_{\phantom{t}}^{(1)}\big{)}\oplus\cdots\oplus\tilde{\mathcal{T}}\big{(}{x}_{\mathrm{in}}^{(|\mathcal{Y}|)},y_{\phantom{t}}^{(|\mathcal{Y}|)}\big{)}." display="inline"><semantics id="S6.Ex3.m1.1.1.1.1.m1.7a"><mrow id="S6.Ex3.m1.1.1.1.1.m1.7.7.1" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.cmml"><mrow id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.cmml"><mrow id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.3" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.3.cmml">𝒯</mi><mo lspace="0em" rspace="0em" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.2.cmml">​</mo><mrow id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1.1.cmml"><mo maxsize="120%" minsize="120%" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1.1.cmml">(</mo><msub id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1.1" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1.1.cmml"><mi id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1.1.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1.1.2.cmml">x</mi><mi id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1.1.3" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1.1.3.cmml">in</mi></msub><mo maxsize="120%" minsize="120%" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1.3" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.6" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.6.cmml">⊕</mo><mrow id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.cmml"><mover accent="true" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.4" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.4.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.4.2.cmml">𝒯</mi><mo id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.4.1" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.4.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.3" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.3.cmml">​</mo><mrow id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.3.cmml"><mo maxsize="120%" minsize="120%" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.3" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.3.cmml">(</mo><msubsup id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1.cmml"><mi id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1.2.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1.2.2.cmml">x</mi><mi id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1.2.3" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1.2.3.cmml">in</mi><mrow id="S6.Ex3.m1.1.1.1.1.m1.1.1.1.3" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1.cmml"><mo stretchy="false" id="S6.Ex3.m1.1.1.1.1.m1.1.1.1.3.1" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1.cmml">(</mo><mn id="S6.Ex3.m1.1.1.1.1.m1.1.1.1.1" xref="S6.Ex3.m1.1.1.1.1.m1.1.1.1.1.cmml">1</mn><mo stretchy="false" id="S6.Ex3.m1.1.1.1.1.m1.1.1.1.3.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1.cmml">)</mo></mrow></msubsup><mo id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.4" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.3.cmml">,</mo><msubsup id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2.cmml"><mi id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2.2.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2.2.2.cmml">y</mi><mi id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2.2.3" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2.2.3.cmml"></mi><mrow id="S6.Ex3.m1.1.1.1.1.m1.2.2.1.3" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2.cmml"><mo stretchy="false" id="S6.Ex3.m1.1.1.1.1.m1.2.2.1.3.1" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2.cmml">(</mo><mn id="S6.Ex3.m1.1.1.1.1.m1.2.2.1.1" xref="S6.Ex3.m1.1.1.1.1.m1.2.2.1.1.cmml">1</mn><mo stretchy="false" id="S6.Ex3.m1.1.1.1.1.m1.2.2.1.3.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2.cmml">)</mo></mrow></msubsup><mo maxsize="120%" minsize="120%" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.5" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.3.cmml">)</mo></mrow></mrow><mo id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.6a" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.6.cmml">⊕</mo><mi mathvariant="normal" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.7" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.7.cmml">⋯</mi><mo id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.6b" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.6.cmml">⊕</mo><mrow id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.cmml"><mover accent="true" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.4" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.4.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.4.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.4.2.cmml">𝒯</mi><mo id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.4.1" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.4.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.3" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.3.cmml">​</mo><mrow id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.3.cmml"><mo maxsize="120%" minsize="120%" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.3" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.3.cmml">(</mo><msubsup id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1.cmml"><mi id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1.2.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1.2.2.cmml">x</mi><mi id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1.2.3" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1.2.3.cmml">in</mi><mrow id="S6.Ex3.m1.1.1.1.1.m1.4.4.2.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1.cmml"><mo stretchy="false" id="S6.Ex3.m1.1.1.1.1.m1.4.4.2.2.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1.cmml">(</mo><mrow id="S6.Ex3.m1.1.1.1.1.m1.4.4.2.2.1.2" xref="S6.Ex3.m1.1.1.1.1.m1.4.4.2.2.1.1.cmml"><mo stretchy="false" id="S6.Ex3.m1.1.1.1.1.m1.4.4.2.2.1.2.1" xref="S6.Ex3.m1.1.1.1.1.m1.4.4.2.2.1.1.1.cmml">|</mo><mi class="ltx_font_mathcaligraphic" id="S6.Ex3.m1.1.1.1.1.m1.3.3.1.1" xref="S6.Ex3.m1.1.1.1.1.m1.3.3.1.1.cmml">𝒴</mi><mo stretchy="false" id="S6.Ex3.m1.1.1.1.1.m1.4.4.2.2.1.2.2" xref="S6.Ex3.m1.1.1.1.1.m1.4.4.2.2.1.1.1.cmml">|</mo></mrow><mo stretchy="false" id="S6.Ex3.m1.1.1.1.1.m1.4.4.2.2.3" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1.cmml">)</mo></mrow></msubsup><mo id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.4" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.3.cmml">,</mo><msubsup id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2.cmml"><mi id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2.2.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2.2.2.cmml">y</mi><mi id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2.2.3" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2.2.3.cmml"></mi><mrow id="S6.Ex3.m1.1.1.1.1.m1.6.6.2.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2.cmml"><mo stretchy="false" id="S6.Ex3.m1.1.1.1.1.m1.6.6.2.2.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2.cmml">(</mo><mrow id="S6.Ex3.m1.1.1.1.1.m1.6.6.2.2.1.2" xref="S6.Ex3.m1.1.1.1.1.m1.6.6.2.2.1.1.cmml"><mo stretchy="false" id="S6.Ex3.m1.1.1.1.1.m1.6.6.2.2.1.2.1" xref="S6.Ex3.m1.1.1.1.1.m1.6.6.2.2.1.1.1.cmml">|</mo><mi class="ltx_font_mathcaligraphic" id="S6.Ex3.m1.1.1.1.1.m1.5.5.1.1" xref="S6.Ex3.m1.1.1.1.1.m1.5.5.1.1.cmml">𝒴</mi><mo stretchy="false" id="S6.Ex3.m1.1.1.1.1.m1.6.6.2.2.1.2.2" xref="S6.Ex3.m1.1.1.1.1.m1.6.6.2.2.1.1.1.cmml">|</mo></mrow><mo stretchy="false" id="S6.Ex3.m1.1.1.1.1.m1.6.6.2.2.3" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2.cmml">)</mo></mrow></msubsup><mo maxsize="120%" minsize="120%" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.5" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.3.cmml">)</mo></mrow></mrow></mrow><mo lspace="0em" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.2" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.cmml">.</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.Ex3.m1.1.1.1.1.m1.7b"><apply id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1"><csymbol cd="latexml" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.6.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.6">direct-sum</csymbol><apply id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1"><times id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.2.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.2"></times><ci id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.3.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.3">𝒯</ci><apply id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1"><csymbol cd="ambiguous" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1.1.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1">subscript</csymbol><ci id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1.1.2.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1.1.2">𝑥</ci><ci id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1.1.3.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.1.1.1.1.3">in</ci></apply></apply><apply id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3"><times id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.3.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.3"></times><apply id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.4.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.4"><ci id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.4.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.4.1">~</ci><ci id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.4.2.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.4.2">𝒯</ci></apply><interval closure="open" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.3.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2"><apply id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1"><csymbol cd="ambiguous" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1">superscript</csymbol><apply id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1.2.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1"><csymbol cd="ambiguous" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1.2.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1">subscript</csymbol><ci id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1.2.2.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1.2.2">𝑥</ci><ci id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1.2.3.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.2.1.1.1.2.3">in</ci></apply><cn type="integer" id="S6.Ex3.m1.1.1.1.1.m1.1.1.1.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.1.1.1.1">1</cn></apply><apply id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2"><csymbol cd="ambiguous" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2">superscript</csymbol><apply id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2.2.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2"><csymbol cd="ambiguous" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2.2.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2">subscript</csymbol><ci id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2.2.2.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2.2.2">𝑦</ci><csymbol cd="latexml" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2.2.3.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.3.2.2.2.2.3">absent</csymbol></apply><cn type="integer" id="S6.Ex3.m1.1.1.1.1.m1.2.2.1.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.2.2.1.1">1</cn></apply></interval></apply><ci id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.7.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.7">⋯</ci><apply id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5"><times id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.3.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.3"></times><apply id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.4.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.4"><ci id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.4.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.4.1">~</ci><ci id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.4.2.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.4.2">𝒯</ci></apply><interval closure="open" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.3.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2"><apply id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1"><csymbol cd="ambiguous" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1">superscript</csymbol><apply id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1.2.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1"><csymbol cd="ambiguous" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1.2.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1">subscript</csymbol><ci id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1.2.2.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1.2.2">𝑥</ci><ci id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1.2.3.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.4.1.1.1.2.3">in</ci></apply><apply id="S6.Ex3.m1.1.1.1.1.m1.4.4.2.2.1.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.4.4.2.2.1.2"><abs id="S6.Ex3.m1.1.1.1.1.m1.4.4.2.2.1.1.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.4.4.2.2.1.2.1"></abs><ci id="S6.Ex3.m1.1.1.1.1.m1.3.3.1.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.3.3.1.1">𝒴</ci></apply></apply><apply id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2"><csymbol cd="ambiguous" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2">superscript</csymbol><apply id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2.2.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2"><csymbol cd="ambiguous" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2.2.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2">subscript</csymbol><ci id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2.2.2.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2.2.2">𝑦</ci><csymbol cd="latexml" id="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2.2.3.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.7.7.1.1.5.2.2.2.2.3">absent</csymbol></apply><apply id="S6.Ex3.m1.1.1.1.1.m1.6.6.2.2.1.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.6.6.2.2.1.2"><abs id="S6.Ex3.m1.1.1.1.1.m1.6.6.2.2.1.1.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.6.6.2.2.1.2.1"></abs><ci id="S6.Ex3.m1.1.1.1.1.m1.5.5.1.1.cmml" xref="S6.Ex3.m1.1.1.1.1.m1.5.5.1.1">𝒴</ci></apply></apply></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.Ex3.m1.1.1.1.1.m1.7c">\displaystyle\mathcal{T}\big{(}{x}_{\mathrm{in}}\big{)}\oplus\tilde{\mathcal{T}}\big{(}{x}_{\mathrm{in}}^{(1)},y_{\phantom{t}}^{(1)}\big{)}\oplus\cdots\oplus\tilde{\mathcal{T}}\big{(}{x}_{\mathrm{in}}^{(|\mathcal{Y}|)},y_{\phantom{t}}^{(|\mathcal{Y}|)}\big{)}.</annotation></semantics></math></p>
</span></div>
</td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S6.SS1.p1.10" class="ltx_p">Here <math id="S6.SS1.p1.6.m1.1" class="ltx_Math" alttext="\oplus" display="inline"><semantics id="S6.SS1.p1.6.m1.1a"><mo id="S6.SS1.p1.6.m1.1.1" xref="S6.SS1.p1.6.m1.1.1.cmml">⊕</mo><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.6.m1.1b"><csymbol cd="latexml" id="S6.SS1.p1.6.m1.1.1.cmml" xref="S6.SS1.p1.6.m1.1.1">direct-sum</csymbol></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.6.m1.1c">\oplus</annotation></semantics></math> denotes concatenation of input sequences.
During both training and inference we sample multiple demonstration sets for each <math id="S6.SS1.p1.7.m2.1" class="ltx_Math" alttext="{x}_{\mathrm{in}}" display="inline"><semantics id="S6.SS1.p1.7.m2.1a"><msub id="S6.SS1.p1.7.m2.1.1" xref="S6.SS1.p1.7.m2.1.1.cmml"><mi id="S6.SS1.p1.7.m2.1.1.2" xref="S6.SS1.p1.7.m2.1.1.2.cmml">x</mi><mi id="S6.SS1.p1.7.m2.1.1.3" xref="S6.SS1.p1.7.m2.1.1.3.cmml">in</mi></msub><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.7.m2.1b"><apply id="S6.SS1.p1.7.m2.1.1.cmml" xref="S6.SS1.p1.7.m2.1.1"><csymbol cd="ambiguous" id="S6.SS1.p1.7.m2.1.1.1.cmml" xref="S6.SS1.p1.7.m2.1.1">subscript</csymbol><ci id="S6.SS1.p1.7.m2.1.1.2.cmml" xref="S6.SS1.p1.7.m2.1.1.2">𝑥</ci><ci id="S6.SS1.p1.7.m2.1.1.3.cmml" xref="S6.SS1.p1.7.m2.1.1.3">in</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.7.m2.1c">{x}_{\mathrm{in}}</annotation></semantics></math>. Note that both <math id="S6.SS1.p1.8.m3.1" class="ltx_Math" alttext="{x}_{\mathrm{in}}" display="inline"><semantics id="S6.SS1.p1.8.m3.1a"><msub id="S6.SS1.p1.8.m3.1.1" xref="S6.SS1.p1.8.m3.1.1.cmml"><mi id="S6.SS1.p1.8.m3.1.1.2" xref="S6.SS1.p1.8.m3.1.1.2.cmml">x</mi><mi id="S6.SS1.p1.8.m3.1.1.3" xref="S6.SS1.p1.8.m3.1.1.3.cmml">in</mi></msub><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.8.m3.1b"><apply id="S6.SS1.p1.8.m3.1.1.cmml" xref="S6.SS1.p1.8.m3.1.1"><csymbol cd="ambiguous" id="S6.SS1.p1.8.m3.1.1.1.cmml" xref="S6.SS1.p1.8.m3.1.1">subscript</csymbol><ci id="S6.SS1.p1.8.m3.1.1.2.cmml" xref="S6.SS1.p1.8.m3.1.1.2">𝑥</ci><ci id="S6.SS1.p1.8.m3.1.1.3.cmml" xref="S6.SS1.p1.8.m3.1.1.3">in</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.8.m3.1c">{x}_{\mathrm{in}}</annotation></semantics></math> and demonstration examples are sampled from the same set <math id="S6.SS1.p1.9.m4.1" class="ltx_Math" alttext="\mathcal{D}_{\text{train}}" display="inline"><semantics id="S6.SS1.p1.9.m4.1a"><msub id="S6.SS1.p1.9.m4.1.1" xref="S6.SS1.p1.9.m4.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.SS1.p1.9.m4.1.1.2" xref="S6.SS1.p1.9.m4.1.1.2.cmml">𝒟</mi><mtext id="S6.SS1.p1.9.m4.1.1.3" xref="S6.SS1.p1.9.m4.1.1.3a.cmml">train</mtext></msub><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.9.m4.1b"><apply id="S6.SS1.p1.9.m4.1.1.cmml" xref="S6.SS1.p1.9.m4.1.1"><csymbol cd="ambiguous" id="S6.SS1.p1.9.m4.1.1.1.cmml" xref="S6.SS1.p1.9.m4.1.1">subscript</csymbol><ci id="S6.SS1.p1.9.m4.1.1.2.cmml" xref="S6.SS1.p1.9.m4.1.1.2">𝒟</ci><ci id="S6.SS1.p1.9.m4.1.1.3a.cmml" xref="S6.SS1.p1.9.m4.1.1.3"><mtext mathsize="70%" id="S6.SS1.p1.9.m4.1.1.3.cmml" xref="S6.SS1.p1.9.m4.1.1.3">train</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.9.m4.1c">\mathcal{D}_{\text{train}}</annotation></semantics></math> during training. At testing time, we still sample demonstration sets from <math id="S6.SS1.p1.10.m5.1" class="ltx_Math" alttext="\mathcal{D}_{\text{train}}" display="inline"><semantics id="S6.SS1.p1.10.m5.1a"><msub id="S6.SS1.p1.10.m5.1.1" xref="S6.SS1.p1.10.m5.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.SS1.p1.10.m5.1.1.2" xref="S6.SS1.p1.10.m5.1.1.2.cmml">𝒟</mi><mtext id="S6.SS1.p1.10.m5.1.1.3" xref="S6.SS1.p1.10.m5.1.1.3a.cmml">train</mtext></msub><annotation-xml encoding="MathML-Content" id="S6.SS1.p1.10.m5.1b"><apply id="S6.SS1.p1.10.m5.1.1.cmml" xref="S6.SS1.p1.10.m5.1.1"><csymbol cd="ambiguous" id="S6.SS1.p1.10.m5.1.1.1.cmml" xref="S6.SS1.p1.10.m5.1.1">subscript</csymbol><ci id="S6.SS1.p1.10.m5.1.1.2.cmml" xref="S6.SS1.p1.10.m5.1.1.2">𝒟</ci><ci id="S6.SS1.p1.10.m5.1.1.3a.cmml" xref="S6.SS1.p1.10.m5.1.1.3"><mtext mathsize="70%" id="S6.SS1.p1.10.m5.1.1.3.cmml" xref="S6.SS1.p1.10.m5.1.1.3">train</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS1.p1.10.m5.1c">\mathcal{D}_{\text{train}}</annotation></semantics></math> and ensemble predictions across all sets.</p>
</div>
</section>
<section id="S6.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.2 </span>Sampling similar demonstrations</h3>

<div id="S6.SS2.p1" class="ltx_para">
<p id="S6.SS2.p1.9" class="ltx_p">We observe that controlling the construction of the demonstration examples <math id="S6.SS2.p1.1.m1.3" class="ltx_Math" alttext="\{({x}_{\mathrm{in}}^{(c)},y_{\phantom{t}}^{(c)})\}" display="inline"><semantics id="S6.SS2.p1.1.m1.3a"><mrow id="S6.SS2.p1.1.m1.3.3.1" xref="S6.SS2.p1.1.m1.3.3.2.cmml"><mo stretchy="false" id="S6.SS2.p1.1.m1.3.3.1.2" xref="S6.SS2.p1.1.m1.3.3.2.cmml">{</mo><mrow id="S6.SS2.p1.1.m1.3.3.1.1.2" xref="S6.SS2.p1.1.m1.3.3.1.1.3.cmml"><mo stretchy="false" id="S6.SS2.p1.1.m1.3.3.1.1.2.3" xref="S6.SS2.p1.1.m1.3.3.1.1.3.cmml">(</mo><msubsup id="S6.SS2.p1.1.m1.3.3.1.1.1.1" xref="S6.SS2.p1.1.m1.3.3.1.1.1.1.cmml"><mi id="S6.SS2.p1.1.m1.3.3.1.1.1.1.2.2" xref="S6.SS2.p1.1.m1.3.3.1.1.1.1.2.2.cmml">x</mi><mi id="S6.SS2.p1.1.m1.3.3.1.1.1.1.2.3" xref="S6.SS2.p1.1.m1.3.3.1.1.1.1.2.3.cmml">in</mi><mrow id="S6.SS2.p1.1.m1.1.1.1.3" xref="S6.SS2.p1.1.m1.3.3.1.1.1.1.cmml"><mo stretchy="false" id="S6.SS2.p1.1.m1.1.1.1.3.1" xref="S6.SS2.p1.1.m1.3.3.1.1.1.1.cmml">(</mo><mi id="S6.SS2.p1.1.m1.1.1.1.1" xref="S6.SS2.p1.1.m1.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S6.SS2.p1.1.m1.1.1.1.3.2" xref="S6.SS2.p1.1.m1.3.3.1.1.1.1.cmml">)</mo></mrow></msubsup><mo id="S6.SS2.p1.1.m1.3.3.1.1.2.4" xref="S6.SS2.p1.1.m1.3.3.1.1.3.cmml">,</mo><msubsup id="S6.SS2.p1.1.m1.3.3.1.1.2.2" xref="S6.SS2.p1.1.m1.3.3.1.1.2.2.cmml"><mi id="S6.SS2.p1.1.m1.3.3.1.1.2.2.2.2" xref="S6.SS2.p1.1.m1.3.3.1.1.2.2.2.2.cmml">y</mi><mi id="S6.SS2.p1.1.m1.3.3.1.1.2.2.2.3" xref="S6.SS2.p1.1.m1.3.3.1.1.2.2.2.3.cmml"></mi><mrow id="S6.SS2.p1.1.m1.2.2.1.3" xref="S6.SS2.p1.1.m1.3.3.1.1.2.2.cmml"><mo stretchy="false" id="S6.SS2.p1.1.m1.2.2.1.3.1" xref="S6.SS2.p1.1.m1.3.3.1.1.2.2.cmml">(</mo><mi id="S6.SS2.p1.1.m1.2.2.1.1" xref="S6.SS2.p1.1.m1.2.2.1.1.cmml">c</mi><mo stretchy="false" id="S6.SS2.p1.1.m1.2.2.1.3.2" xref="S6.SS2.p1.1.m1.3.3.1.1.2.2.cmml">)</mo></mrow></msubsup><mo stretchy="false" id="S6.SS2.p1.1.m1.3.3.1.1.2.5" xref="S6.SS2.p1.1.m1.3.3.1.1.3.cmml">)</mo></mrow><mo stretchy="false" id="S6.SS2.p1.1.m1.3.3.1.3" xref="S6.SS2.p1.1.m1.3.3.2.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.1.m1.3b"><set id="S6.SS2.p1.1.m1.3.3.2.cmml" xref="S6.SS2.p1.1.m1.3.3.1"><interval closure="open" id="S6.SS2.p1.1.m1.3.3.1.1.3.cmml" xref="S6.SS2.p1.1.m1.3.3.1.1.2"><apply id="S6.SS2.p1.1.m1.3.3.1.1.1.1.cmml" xref="S6.SS2.p1.1.m1.3.3.1.1.1.1"><csymbol cd="ambiguous" id="S6.SS2.p1.1.m1.3.3.1.1.1.1.1.cmml" xref="S6.SS2.p1.1.m1.3.3.1.1.1.1">superscript</csymbol><apply id="S6.SS2.p1.1.m1.3.3.1.1.1.1.2.cmml" xref="S6.SS2.p1.1.m1.3.3.1.1.1.1"><csymbol cd="ambiguous" id="S6.SS2.p1.1.m1.3.3.1.1.1.1.2.1.cmml" xref="S6.SS2.p1.1.m1.3.3.1.1.1.1">subscript</csymbol><ci id="S6.SS2.p1.1.m1.3.3.1.1.1.1.2.2.cmml" xref="S6.SS2.p1.1.m1.3.3.1.1.1.1.2.2">𝑥</ci><ci id="S6.SS2.p1.1.m1.3.3.1.1.1.1.2.3.cmml" xref="S6.SS2.p1.1.m1.3.3.1.1.1.1.2.3">in</ci></apply><ci id="S6.SS2.p1.1.m1.1.1.1.1.cmml" xref="S6.SS2.p1.1.m1.1.1.1.1">𝑐</ci></apply><apply id="S6.SS2.p1.1.m1.3.3.1.1.2.2.cmml" xref="S6.SS2.p1.1.m1.3.3.1.1.2.2"><csymbol cd="ambiguous" id="S6.SS2.p1.1.m1.3.3.1.1.2.2.1.cmml" xref="S6.SS2.p1.1.m1.3.3.1.1.2.2">superscript</csymbol><apply id="S6.SS2.p1.1.m1.3.3.1.1.2.2.2.cmml" xref="S6.SS2.p1.1.m1.3.3.1.1.2.2"><csymbol cd="ambiguous" id="S6.SS2.p1.1.m1.3.3.1.1.2.2.2.1.cmml" xref="S6.SS2.p1.1.m1.3.3.1.1.2.2">subscript</csymbol><ci id="S6.SS2.p1.1.m1.3.3.1.1.2.2.2.2.cmml" xref="S6.SS2.p1.1.m1.3.3.1.1.2.2.2.2">𝑦</ci><csymbol cd="latexml" id="S6.SS2.p1.1.m1.3.3.1.1.2.2.2.3.cmml" xref="S6.SS2.p1.1.m1.3.3.1.1.2.2.2.3">absent</csymbol></apply><ci id="S6.SS2.p1.1.m1.2.2.1.1.cmml" xref="S6.SS2.p1.1.m1.2.2.1.1">𝑐</ci></apply></interval></set></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.1.m1.3c">\{({x}_{\mathrm{in}}^{(c)},y_{\phantom{t}}^{(c)})\}</annotation></semantics></math> is crucial for good final performance.
For example, if the set of contrastive demonstrations <math id="S6.SS2.p1.2.m2.1" class="ltx_Math" alttext="{x}_{\mathrm{in}}^{(c)}" display="inline"><semantics id="S6.SS2.p1.2.m2.1a"><msubsup id="S6.SS2.p1.2.m2.1.2" xref="S6.SS2.p1.2.m2.1.2.cmml"><mi id="S6.SS2.p1.2.m2.1.2.2.2" xref="S6.SS2.p1.2.m2.1.2.2.2.cmml">x</mi><mi id="S6.SS2.p1.2.m2.1.2.2.3" xref="S6.SS2.p1.2.m2.1.2.2.3.cmml">in</mi><mrow id="S6.SS2.p1.2.m2.1.1.1.3" xref="S6.SS2.p1.2.m2.1.2.cmml"><mo stretchy="false" id="S6.SS2.p1.2.m2.1.1.1.3.1" xref="S6.SS2.p1.2.m2.1.2.cmml">(</mo><mi id="S6.SS2.p1.2.m2.1.1.1.1" xref="S6.SS2.p1.2.m2.1.1.1.1.cmml">c</mi><mo stretchy="false" id="S6.SS2.p1.2.m2.1.1.1.3.2" xref="S6.SS2.p1.2.m2.1.2.cmml">)</mo></mrow></msubsup><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.2.m2.1b"><apply id="S6.SS2.p1.2.m2.1.2.cmml" xref="S6.SS2.p1.2.m2.1.2"><csymbol cd="ambiguous" id="S6.SS2.p1.2.m2.1.2.1.cmml" xref="S6.SS2.p1.2.m2.1.2">superscript</csymbol><apply id="S6.SS2.p1.2.m2.1.2.2.cmml" xref="S6.SS2.p1.2.m2.1.2"><csymbol cd="ambiguous" id="S6.SS2.p1.2.m2.1.2.2.1.cmml" xref="S6.SS2.p1.2.m2.1.2">subscript</csymbol><ci id="S6.SS2.p1.2.m2.1.2.2.2.cmml" xref="S6.SS2.p1.2.m2.1.2.2.2">𝑥</ci><ci id="S6.SS2.p1.2.m2.1.2.2.3.cmml" xref="S6.SS2.p1.2.m2.1.2.2.3">in</ci></apply><ci id="S6.SS2.p1.2.m2.1.1.1.1.cmml" xref="S6.SS2.p1.2.m2.1.1.1.1">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.2.m2.1c">{x}_{\mathrm{in}}^{(c)}</annotation></semantics></math> are all dramatically different—from each other, or from the query <math id="S6.SS2.p1.3.m3.1" class="ltx_Math" alttext="{x}_{\mathrm{in}}" display="inline"><semantics id="S6.SS2.p1.3.m3.1a"><msub id="S6.SS2.p1.3.m3.1.1" xref="S6.SS2.p1.3.m3.1.1.cmml"><mi id="S6.SS2.p1.3.m3.1.1.2" xref="S6.SS2.p1.3.m3.1.1.2.cmml">x</mi><mi id="S6.SS2.p1.3.m3.1.1.3" xref="S6.SS2.p1.3.m3.1.1.3.cmml">in</mi></msub><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.3.m3.1b"><apply id="S6.SS2.p1.3.m3.1.1.cmml" xref="S6.SS2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S6.SS2.p1.3.m3.1.1.1.cmml" xref="S6.SS2.p1.3.m3.1.1">subscript</csymbol><ci id="S6.SS2.p1.3.m3.1.1.2.cmml" xref="S6.SS2.p1.3.m3.1.1.2">𝑥</ci><ci id="S6.SS2.p1.3.m3.1.1.3.cmml" xref="S6.SS2.p1.3.m3.1.1.3">in</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.3.m3.1c">{x}_{\mathrm{in}}</annotation></semantics></math>—then it becomes challenging for the language model to decipher meaningful patterns. As a result, the model may simply ignore the context, or even get confused by the additional examples.
To address this issue, we devise a simple strategy in which we only sample examples that are semantically close to <math id="S6.SS2.p1.4.m4.1" class="ltx_Math" alttext="{x}_{\mathrm{in}}" display="inline"><semantics id="S6.SS2.p1.4.m4.1a"><msub id="S6.SS2.p1.4.m4.1.1" xref="S6.SS2.p1.4.m4.1.1.cmml"><mi id="S6.SS2.p1.4.m4.1.1.2" xref="S6.SS2.p1.4.m4.1.1.2.cmml">x</mi><mi id="S6.SS2.p1.4.m4.1.1.3" xref="S6.SS2.p1.4.m4.1.1.3.cmml">in</mi></msub><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.4.m4.1b"><apply id="S6.SS2.p1.4.m4.1.1.cmml" xref="S6.SS2.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S6.SS2.p1.4.m4.1.1.1.cmml" xref="S6.SS2.p1.4.m4.1.1">subscript</csymbol><ci id="S6.SS2.p1.4.m4.1.1.2.cmml" xref="S6.SS2.p1.4.m4.1.1.2">𝑥</ci><ci id="S6.SS2.p1.4.m4.1.1.3.cmml" xref="S6.SS2.p1.4.m4.1.1.3">in</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.4.m4.1c">{x}_{\mathrm{in}}</annotation></semantics></math>.
Specifically, we use a pre-trained SBERT <cite class="ltx_cite ltx_citemacro_cite">Reimers and Gurevych (<a href="#bib.bib33" title="" class="ltx_ref">2019</a>)</cite> model to obtain embeddings for all input sentences (for sentence-pair tasks, we use the concatenation of the two sentences).
Here we just feed the raw sentences without the templates into SBERT.
For each query <math id="S6.SS2.p1.5.m5.1" class="ltx_Math" alttext="{x}_{\mathrm{in}}" display="inline"><semantics id="S6.SS2.p1.5.m5.1a"><msub id="S6.SS2.p1.5.m5.1.1" xref="S6.SS2.p1.5.m5.1.1.cmml"><mi id="S6.SS2.p1.5.m5.1.1.2" xref="S6.SS2.p1.5.m5.1.1.2.cmml">x</mi><mi id="S6.SS2.p1.5.m5.1.1.3" xref="S6.SS2.p1.5.m5.1.1.3.cmml">in</mi></msub><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.5.m5.1b"><apply id="S6.SS2.p1.5.m5.1.1.cmml" xref="S6.SS2.p1.5.m5.1.1"><csymbol cd="ambiguous" id="S6.SS2.p1.5.m5.1.1.1.cmml" xref="S6.SS2.p1.5.m5.1.1">subscript</csymbol><ci id="S6.SS2.p1.5.m5.1.1.2.cmml" xref="S6.SS2.p1.5.m5.1.1.2">𝑥</ci><ci id="S6.SS2.p1.5.m5.1.1.3.cmml" xref="S6.SS2.p1.5.m5.1.1.3">in</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.5.m5.1c">{x}_{\mathrm{in}}</annotation></semantics></math> and each label <math id="S6.SS2.p1.6.m6.1" class="ltx_Math" alttext="c\in\mathcal{Y}" display="inline"><semantics id="S6.SS2.p1.6.m6.1a"><mrow id="S6.SS2.p1.6.m6.1.1" xref="S6.SS2.p1.6.m6.1.1.cmml"><mi id="S6.SS2.p1.6.m6.1.1.2" xref="S6.SS2.p1.6.m6.1.1.2.cmml">c</mi><mo id="S6.SS2.p1.6.m6.1.1.1" xref="S6.SS2.p1.6.m6.1.1.1.cmml">∈</mo><mi class="ltx_font_mathcaligraphic" id="S6.SS2.p1.6.m6.1.1.3" xref="S6.SS2.p1.6.m6.1.1.3.cmml">𝒴</mi></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.6.m6.1b"><apply id="S6.SS2.p1.6.m6.1.1.cmml" xref="S6.SS2.p1.6.m6.1.1"><in id="S6.SS2.p1.6.m6.1.1.1.cmml" xref="S6.SS2.p1.6.m6.1.1.1"></in><ci id="S6.SS2.p1.6.m6.1.1.2.cmml" xref="S6.SS2.p1.6.m6.1.1.2">𝑐</ci><ci id="S6.SS2.p1.6.m6.1.1.3.cmml" xref="S6.SS2.p1.6.m6.1.1.3">𝒴</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.6.m6.1c">c\in\mathcal{Y}</annotation></semantics></math>, we sort all training instances with the label <math id="S6.SS2.p1.7.m7.1" class="ltx_Math" alttext="x\in\mathcal{D}_{\text{train}}^{c}" display="inline"><semantics id="S6.SS2.p1.7.m7.1a"><mrow id="S6.SS2.p1.7.m7.1.1" xref="S6.SS2.p1.7.m7.1.1.cmml"><mi id="S6.SS2.p1.7.m7.1.1.2" xref="S6.SS2.p1.7.m7.1.1.2.cmml">x</mi><mo id="S6.SS2.p1.7.m7.1.1.1" xref="S6.SS2.p1.7.m7.1.1.1.cmml">∈</mo><msubsup id="S6.SS2.p1.7.m7.1.1.3" xref="S6.SS2.p1.7.m7.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="S6.SS2.p1.7.m7.1.1.3.2.2" xref="S6.SS2.p1.7.m7.1.1.3.2.2.cmml">𝒟</mi><mtext id="S6.SS2.p1.7.m7.1.1.3.2.3" xref="S6.SS2.p1.7.m7.1.1.3.2.3a.cmml">train</mtext><mi id="S6.SS2.p1.7.m7.1.1.3.3" xref="S6.SS2.p1.7.m7.1.1.3.3.cmml">c</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.7.m7.1b"><apply id="S6.SS2.p1.7.m7.1.1.cmml" xref="S6.SS2.p1.7.m7.1.1"><in id="S6.SS2.p1.7.m7.1.1.1.cmml" xref="S6.SS2.p1.7.m7.1.1.1"></in><ci id="S6.SS2.p1.7.m7.1.1.2.cmml" xref="S6.SS2.p1.7.m7.1.1.2">𝑥</ci><apply id="S6.SS2.p1.7.m7.1.1.3.cmml" xref="S6.SS2.p1.7.m7.1.1.3"><csymbol cd="ambiguous" id="S6.SS2.p1.7.m7.1.1.3.1.cmml" xref="S6.SS2.p1.7.m7.1.1.3">superscript</csymbol><apply id="S6.SS2.p1.7.m7.1.1.3.2.cmml" xref="S6.SS2.p1.7.m7.1.1.3"><csymbol cd="ambiguous" id="S6.SS2.p1.7.m7.1.1.3.2.1.cmml" xref="S6.SS2.p1.7.m7.1.1.3">subscript</csymbol><ci id="S6.SS2.p1.7.m7.1.1.3.2.2.cmml" xref="S6.SS2.p1.7.m7.1.1.3.2.2">𝒟</ci><ci id="S6.SS2.p1.7.m7.1.1.3.2.3a.cmml" xref="S6.SS2.p1.7.m7.1.1.3.2.3"><mtext mathsize="70%" id="S6.SS2.p1.7.m7.1.1.3.2.3.cmml" xref="S6.SS2.p1.7.m7.1.1.3.2.3">train</mtext></ci></apply><ci id="S6.SS2.p1.7.m7.1.1.3.3.cmml" xref="S6.SS2.p1.7.m7.1.1.3.3">𝑐</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.7.m7.1c">x\in\mathcal{D}_{\text{train}}^{c}</annotation></semantics></math> by their similarity score to the query <math id="S6.SS2.p1.8.m8.4" class="ltx_Math" alttext="\cos(\mathbf{e}({x}_{\mathrm{in}}),\mathbf{e}(x))" display="inline"><semantics id="S6.SS2.p1.8.m8.4a"><mrow id="S6.SS2.p1.8.m8.4.4.2" xref="S6.SS2.p1.8.m8.4.4.3.cmml"><mi id="S6.SS2.p1.8.m8.2.2" xref="S6.SS2.p1.8.m8.2.2.cmml">cos</mi><mo id="S6.SS2.p1.8.m8.4.4.2a" xref="S6.SS2.p1.8.m8.4.4.3.cmml">⁡</mo><mrow id="S6.SS2.p1.8.m8.4.4.2.2" xref="S6.SS2.p1.8.m8.4.4.3.cmml"><mo stretchy="false" id="S6.SS2.p1.8.m8.4.4.2.2.3" xref="S6.SS2.p1.8.m8.4.4.3.cmml">(</mo><mrow id="S6.SS2.p1.8.m8.3.3.1.1.1" xref="S6.SS2.p1.8.m8.3.3.1.1.1.cmml"><mi id="S6.SS2.p1.8.m8.3.3.1.1.1.3" xref="S6.SS2.p1.8.m8.3.3.1.1.1.3.cmml">𝐞</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p1.8.m8.3.3.1.1.1.2" xref="S6.SS2.p1.8.m8.3.3.1.1.1.2.cmml">​</mo><mrow id="S6.SS2.p1.8.m8.3.3.1.1.1.1.1" xref="S6.SS2.p1.8.m8.3.3.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S6.SS2.p1.8.m8.3.3.1.1.1.1.1.2" xref="S6.SS2.p1.8.m8.3.3.1.1.1.1.1.1.cmml">(</mo><msub id="S6.SS2.p1.8.m8.3.3.1.1.1.1.1.1" xref="S6.SS2.p1.8.m8.3.3.1.1.1.1.1.1.cmml"><mi id="S6.SS2.p1.8.m8.3.3.1.1.1.1.1.1.2" xref="S6.SS2.p1.8.m8.3.3.1.1.1.1.1.1.2.cmml">x</mi><mi id="S6.SS2.p1.8.m8.3.3.1.1.1.1.1.1.3" xref="S6.SS2.p1.8.m8.3.3.1.1.1.1.1.1.3.cmml">in</mi></msub><mo stretchy="false" id="S6.SS2.p1.8.m8.3.3.1.1.1.1.1.3" xref="S6.SS2.p1.8.m8.3.3.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S6.SS2.p1.8.m8.4.4.2.2.4" xref="S6.SS2.p1.8.m8.4.4.3.cmml">,</mo><mrow id="S6.SS2.p1.8.m8.4.4.2.2.2" xref="S6.SS2.p1.8.m8.4.4.2.2.2.cmml"><mi id="S6.SS2.p1.8.m8.4.4.2.2.2.2" xref="S6.SS2.p1.8.m8.4.4.2.2.2.2.cmml">𝐞</mi><mo lspace="0em" rspace="0em" id="S6.SS2.p1.8.m8.4.4.2.2.2.1" xref="S6.SS2.p1.8.m8.4.4.2.2.2.1.cmml">​</mo><mrow id="S6.SS2.p1.8.m8.4.4.2.2.2.3.2" xref="S6.SS2.p1.8.m8.4.4.2.2.2.cmml"><mo stretchy="false" id="S6.SS2.p1.8.m8.4.4.2.2.2.3.2.1" xref="S6.SS2.p1.8.m8.4.4.2.2.2.cmml">(</mo><mi id="S6.SS2.p1.8.m8.1.1" xref="S6.SS2.p1.8.m8.1.1.cmml">x</mi><mo stretchy="false" id="S6.SS2.p1.8.m8.4.4.2.2.2.3.2.2" xref="S6.SS2.p1.8.m8.4.4.2.2.2.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S6.SS2.p1.8.m8.4.4.2.2.5" xref="S6.SS2.p1.8.m8.4.4.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.8.m8.4b"><apply id="S6.SS2.p1.8.m8.4.4.3.cmml" xref="S6.SS2.p1.8.m8.4.4.2"><cos id="S6.SS2.p1.8.m8.2.2.cmml" xref="S6.SS2.p1.8.m8.2.2"></cos><apply id="S6.SS2.p1.8.m8.3.3.1.1.1.cmml" xref="S6.SS2.p1.8.m8.3.3.1.1.1"><times id="S6.SS2.p1.8.m8.3.3.1.1.1.2.cmml" xref="S6.SS2.p1.8.m8.3.3.1.1.1.2"></times><ci id="S6.SS2.p1.8.m8.3.3.1.1.1.3.cmml" xref="S6.SS2.p1.8.m8.3.3.1.1.1.3">𝐞</ci><apply id="S6.SS2.p1.8.m8.3.3.1.1.1.1.1.1.cmml" xref="S6.SS2.p1.8.m8.3.3.1.1.1.1.1"><csymbol cd="ambiguous" id="S6.SS2.p1.8.m8.3.3.1.1.1.1.1.1.1.cmml" xref="S6.SS2.p1.8.m8.3.3.1.1.1.1.1">subscript</csymbol><ci id="S6.SS2.p1.8.m8.3.3.1.1.1.1.1.1.2.cmml" xref="S6.SS2.p1.8.m8.3.3.1.1.1.1.1.1.2">𝑥</ci><ci id="S6.SS2.p1.8.m8.3.3.1.1.1.1.1.1.3.cmml" xref="S6.SS2.p1.8.m8.3.3.1.1.1.1.1.1.3">in</ci></apply></apply><apply id="S6.SS2.p1.8.m8.4.4.2.2.2.cmml" xref="S6.SS2.p1.8.m8.4.4.2.2.2"><times id="S6.SS2.p1.8.m8.4.4.2.2.2.1.cmml" xref="S6.SS2.p1.8.m8.4.4.2.2.2.1"></times><ci id="S6.SS2.p1.8.m8.4.4.2.2.2.2.cmml" xref="S6.SS2.p1.8.m8.4.4.2.2.2.2">𝐞</ci><ci id="S6.SS2.p1.8.m8.1.1.cmml" xref="S6.SS2.p1.8.m8.1.1">𝑥</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.8.m8.4c">\cos(\mathbf{e}({x}_{\mathrm{in}}),\mathbf{e}(x))</annotation></semantics></math>, and only sample from the top <math id="S6.SS2.p1.9.m9.1" class="ltx_Math" alttext="r=50\%" display="inline"><semantics id="S6.SS2.p1.9.m9.1a"><mrow id="S6.SS2.p1.9.m9.1.1" xref="S6.SS2.p1.9.m9.1.1.cmml"><mi id="S6.SS2.p1.9.m9.1.1.2" xref="S6.SS2.p1.9.m9.1.1.2.cmml">r</mi><mo id="S6.SS2.p1.9.m9.1.1.1" xref="S6.SS2.p1.9.m9.1.1.1.cmml">=</mo><mrow id="S6.SS2.p1.9.m9.1.1.3" xref="S6.SS2.p1.9.m9.1.1.3.cmml"><mn id="S6.SS2.p1.9.m9.1.1.3.2" xref="S6.SS2.p1.9.m9.1.1.3.2.cmml">50</mn><mo id="S6.SS2.p1.9.m9.1.1.3.1" xref="S6.SS2.p1.9.m9.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S6.SS2.p1.9.m9.1b"><apply id="S6.SS2.p1.9.m9.1.1.cmml" xref="S6.SS2.p1.9.m9.1.1"><eq id="S6.SS2.p1.9.m9.1.1.1.cmml" xref="S6.SS2.p1.9.m9.1.1.1"></eq><ci id="S6.SS2.p1.9.m9.1.1.2.cmml" xref="S6.SS2.p1.9.m9.1.1.2">𝑟</ci><apply id="S6.SS2.p1.9.m9.1.1.3.cmml" xref="S6.SS2.p1.9.m9.1.1.3"><csymbol cd="latexml" id="S6.SS2.p1.9.m9.1.1.3.1.cmml" xref="S6.SS2.p1.9.m9.1.1.3.1">percent</csymbol><cn type="integer" id="S6.SS2.p1.9.m9.1.1.3.2.cmml" xref="S6.SS2.p1.9.m9.1.1.3.2">50</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S6.SS2.p1.9.m9.1c">r=50\%</annotation></semantics></math> instances for each class to use as demonstrations.</p>
</div>
</section>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Experiments</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.1" class="ltx_p">We present our main results, and address several research questions pertaining to our LM-BFF approach.
Implementation details are in Appendix <a href="#A3" title="Appendix C Experimental Details ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">C</span></a>.</p>
</div>
<section id="S7.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">7.1 </span>Main results</h3>

<div id="S7.SS1.p1" class="ltx_para">
<p id="S7.SS1.p1.2" class="ltx_p">We use a RoBERTa-large model and set <math id="S7.SS1.p1.1.m1.1" class="ltx_Math" alttext="K=16" display="inline"><semantics id="S7.SS1.p1.1.m1.1a"><mrow id="S7.SS1.p1.1.m1.1.1" xref="S7.SS1.p1.1.m1.1.1.cmml"><mi id="S7.SS1.p1.1.m1.1.1.2" xref="S7.SS1.p1.1.m1.1.1.2.cmml">K</mi><mo id="S7.SS1.p1.1.m1.1.1.1" xref="S7.SS1.p1.1.m1.1.1.1.cmml">=</mo><mn id="S7.SS1.p1.1.m1.1.1.3" xref="S7.SS1.p1.1.m1.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.p1.1.m1.1b"><apply id="S7.SS1.p1.1.m1.1.1.cmml" xref="S7.SS1.p1.1.m1.1.1"><eq id="S7.SS1.p1.1.m1.1.1.1.cmml" xref="S7.SS1.p1.1.m1.1.1.1"></eq><ci id="S7.SS1.p1.1.m1.1.1.2.cmml" xref="S7.SS1.p1.1.m1.1.1.2">𝐾</ci><cn type="integer" id="S7.SS1.p1.1.m1.1.1.3.cmml" xref="S7.SS1.p1.1.m1.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.p1.1.m1.1c">K=16</annotation></semantics></math> in our experiments.
A comparison of using RoBERTa vs BERT can be found in
Appendix <a href="#A4" title="Appendix D Comparisons of BERT vs RoBERTa ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D</span></a>.
For automatic prompt search, in our main table we report automatic template search only (which consistently performs the best, see Table <a href="#S7.T5" title="Table 5 ‣ 7.1 Main results ‣ 7 Experiments ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>). To put our results in perspective, we compare to a number of baselines, namely
(1) standard fine-tuning in our few-shot setting;
(2) standard fine-tuning using the full training set;
(3) simply taking the most frequent class (measured on the full training set);
(4) prompt-based zero-shot prediction where we take our manual prompts and use <math id="S7.SS1.p1.2.m2.1" class="ltx_Math" alttext="\mathcal{L}" display="inline"><semantics id="S7.SS1.p1.2.m2.1a"><mi class="ltx_font_mathcaligraphic" id="S7.SS1.p1.2.m2.1.1" xref="S7.SS1.p1.2.m2.1.1.cmml">ℒ</mi><annotation-xml encoding="MathML-Content" id="S7.SS1.p1.2.m2.1b"><ci id="S7.SS1.p1.2.m2.1.1.cmml" xref="S7.SS1.p1.2.m2.1.1">ℒ</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.p1.2.m2.1c">\mathcal{L}</annotation></semantics></math> “out-of-the-box” without using any training examples; and
(5) “GPT-3” in-context learning, where we use the same prompt-based zero-shot setting, but augment the context with randomly sampled 32 demonstrations
(and still use RoBERTa-large, not GPT-3).</p>
</div>
<figure id="S7.T4" class="ltx_table">
<div id="S7.T4.6.6" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:403.3pt;height:207.4pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(79.1pt,-40.7pt) scale(1.64620236196596,1.64620236196596) ;">
<table id="S7.T4.6.6.6" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S7.T4.6.6.6.7.1" class="ltx_tr">
<th id="S7.T4.6.6.6.7.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt">Prompt-based Fine-tuning</th>
<td id="S7.T4.6.6.6.7.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="S7.T4.6.6.6.7.1.2.1" class="ltx_text ltx_font_bold">MNLI</span></td>
<td id="S7.T4.6.6.6.7.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="S7.T4.6.6.6.7.1.3.1" class="ltx_text ltx_font_bold">RTE</span></td>
</tr>
<tr id="S7.T4.1.1.1.1" class="ltx_tr">
<th id="S7.T4.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Our single manual <math id="S7.T4.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\mathcal{P}" display="inline"><semantics id="S7.T4.1.1.1.1.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S7.T4.1.1.1.1.1.m1.1.1" xref="S7.T4.1.1.1.1.1.m1.1.1.cmml">𝒫</mi><annotation-xml encoding="MathML-Content" id="S7.T4.1.1.1.1.1.m1.1b"><ci id="S7.T4.1.1.1.1.1.m1.1.1.cmml" xref="S7.T4.1.1.1.1.1.m1.1.1">𝒫</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.T4.1.1.1.1.1.m1.1c">\mathcal{P}</annotation></semantics></math>
</th>
<td id="S7.T4.1.1.1.1.2" class="ltx_td ltx_align_center ltx_border_t">68.3 (2.3)</td>
<td id="S7.T4.1.1.1.1.3" class="ltx_td ltx_align_center ltx_border_t">69.1 (3.6)</td>
</tr>
<tr id="S7.T4.2.2.2.2" class="ltx_tr">
<th id="S7.T4.2.2.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="S7.T4.2.2.2.2.1.m1.1" class="ltx_Math" alttext="\mathcal{P}_{\text{PET}}" display="inline"><semantics id="S7.T4.2.2.2.2.1.m1.1a"><msub id="S7.T4.2.2.2.2.1.m1.1.1" xref="S7.T4.2.2.2.2.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S7.T4.2.2.2.2.1.m1.1.1.2" xref="S7.T4.2.2.2.2.1.m1.1.1.2.cmml">𝒫</mi><mtext id="S7.T4.2.2.2.2.1.m1.1.1.3" xref="S7.T4.2.2.2.2.1.m1.1.1.3a.cmml">PET</mtext></msub><annotation-xml encoding="MathML-Content" id="S7.T4.2.2.2.2.1.m1.1b"><apply id="S7.T4.2.2.2.2.1.m1.1.1.cmml" xref="S7.T4.2.2.2.2.1.m1.1.1"><csymbol cd="ambiguous" id="S7.T4.2.2.2.2.1.m1.1.1.1.cmml" xref="S7.T4.2.2.2.2.1.m1.1.1">subscript</csymbol><ci id="S7.T4.2.2.2.2.1.m1.1.1.2.cmml" xref="S7.T4.2.2.2.2.1.m1.1.1.2">𝒫</ci><ci id="S7.T4.2.2.2.2.1.m1.1.1.3a.cmml" xref="S7.T4.2.2.2.2.1.m1.1.1.3"><mtext mathsize="70%" id="S7.T4.2.2.2.2.1.m1.1.1.3.cmml" xref="S7.T4.2.2.2.2.1.m1.1.1.3">PET</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T4.2.2.2.2.1.m1.1c">\mathcal{P}_{\text{PET}}</annotation></semantics></math></th>
<td id="S7.T4.2.2.2.2.2" class="ltx_td ltx_align_center">71.9 (1.5)</td>
<td id="S7.T4.2.2.2.2.3" class="ltx_td ltx_align_center">69.2 (4.0)</td>
</tr>
<tr id="S7.T4.4.4.4.4" class="ltx_tr">
<th id="S7.T4.4.4.4.4.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<math id="S7.T4.3.3.3.3.1.m1.1" class="ltx_Math" alttext="\mathcal{P}_{\text{ours}}" display="inline"><semantics id="S7.T4.3.3.3.3.1.m1.1a"><msub id="S7.T4.3.3.3.3.1.m1.1.1" xref="S7.T4.3.3.3.3.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S7.T4.3.3.3.3.1.m1.1.1.2" xref="S7.T4.3.3.3.3.1.m1.1.1.2.cmml">𝒫</mi><mtext id="S7.T4.3.3.3.3.1.m1.1.1.3" xref="S7.T4.3.3.3.3.1.m1.1.1.3a.cmml">ours</mtext></msub><annotation-xml encoding="MathML-Content" id="S7.T4.3.3.3.3.1.m1.1b"><apply id="S7.T4.3.3.3.3.1.m1.1.1.cmml" xref="S7.T4.3.3.3.3.1.m1.1.1"><csymbol cd="ambiguous" id="S7.T4.3.3.3.3.1.m1.1.1.1.cmml" xref="S7.T4.3.3.3.3.1.m1.1.1">subscript</csymbol><ci id="S7.T4.3.3.3.3.1.m1.1.1.2.cmml" xref="S7.T4.3.3.3.3.1.m1.1.1.2">𝒫</ci><ci id="S7.T4.3.3.3.3.1.m1.1.1.3a.cmml" xref="S7.T4.3.3.3.3.1.m1.1.1.3"><mtext mathsize="70%" id="S7.T4.3.3.3.3.1.m1.1.1.3.cmml" xref="S7.T4.3.3.3.3.1.m1.1.1.3">ours</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T4.3.3.3.3.1.m1.1c">\mathcal{P}_{\text{ours}}</annotation></semantics></math>, <math id="S7.T4.4.4.4.4.2.m2.2" class="ltx_Math" alttext="|\mathcal{P}_{\text{ours}}|=|\mathcal{P}_{\text{PET}}|" display="inline"><semantics id="S7.T4.4.4.4.4.2.m2.2a"><mrow id="S7.T4.4.4.4.4.2.m2.2.2" xref="S7.T4.4.4.4.4.2.m2.2.2.cmml"><mrow id="S7.T4.4.4.4.4.2.m2.1.1.1.1" xref="S7.T4.4.4.4.4.2.m2.1.1.1.2.cmml"><mo stretchy="false" id="S7.T4.4.4.4.4.2.m2.1.1.1.1.2" xref="S7.T4.4.4.4.4.2.m2.1.1.1.2.1.cmml">|</mo><msub id="S7.T4.4.4.4.4.2.m2.1.1.1.1.1" xref="S7.T4.4.4.4.4.2.m2.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S7.T4.4.4.4.4.2.m2.1.1.1.1.1.2" xref="S7.T4.4.4.4.4.2.m2.1.1.1.1.1.2.cmml">𝒫</mi><mtext id="S7.T4.4.4.4.4.2.m2.1.1.1.1.1.3" xref="S7.T4.4.4.4.4.2.m2.1.1.1.1.1.3a.cmml">ours</mtext></msub><mo stretchy="false" id="S7.T4.4.4.4.4.2.m2.1.1.1.1.3" xref="S7.T4.4.4.4.4.2.m2.1.1.1.2.1.cmml">|</mo></mrow><mo id="S7.T4.4.4.4.4.2.m2.2.2.3" xref="S7.T4.4.4.4.4.2.m2.2.2.3.cmml">=</mo><mrow id="S7.T4.4.4.4.4.2.m2.2.2.2.1" xref="S7.T4.4.4.4.4.2.m2.2.2.2.2.cmml"><mo stretchy="false" id="S7.T4.4.4.4.4.2.m2.2.2.2.1.2" xref="S7.T4.4.4.4.4.2.m2.2.2.2.2.1.cmml">|</mo><msub id="S7.T4.4.4.4.4.2.m2.2.2.2.1.1" xref="S7.T4.4.4.4.4.2.m2.2.2.2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S7.T4.4.4.4.4.2.m2.2.2.2.1.1.2" xref="S7.T4.4.4.4.4.2.m2.2.2.2.1.1.2.cmml">𝒫</mi><mtext id="S7.T4.4.4.4.4.2.m2.2.2.2.1.1.3" xref="S7.T4.4.4.4.4.2.m2.2.2.2.1.1.3a.cmml">PET</mtext></msub><mo stretchy="false" id="S7.T4.4.4.4.4.2.m2.2.2.2.1.3" xref="S7.T4.4.4.4.4.2.m2.2.2.2.2.1.cmml">|</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.T4.4.4.4.4.2.m2.2b"><apply id="S7.T4.4.4.4.4.2.m2.2.2.cmml" xref="S7.T4.4.4.4.4.2.m2.2.2"><eq id="S7.T4.4.4.4.4.2.m2.2.2.3.cmml" xref="S7.T4.4.4.4.4.2.m2.2.2.3"></eq><apply id="S7.T4.4.4.4.4.2.m2.1.1.1.2.cmml" xref="S7.T4.4.4.4.4.2.m2.1.1.1.1"><abs id="S7.T4.4.4.4.4.2.m2.1.1.1.2.1.cmml" xref="S7.T4.4.4.4.4.2.m2.1.1.1.1.2"></abs><apply id="S7.T4.4.4.4.4.2.m2.1.1.1.1.1.cmml" xref="S7.T4.4.4.4.4.2.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="S7.T4.4.4.4.4.2.m2.1.1.1.1.1.1.cmml" xref="S7.T4.4.4.4.4.2.m2.1.1.1.1.1">subscript</csymbol><ci id="S7.T4.4.4.4.4.2.m2.1.1.1.1.1.2.cmml" xref="S7.T4.4.4.4.4.2.m2.1.1.1.1.1.2">𝒫</ci><ci id="S7.T4.4.4.4.4.2.m2.1.1.1.1.1.3a.cmml" xref="S7.T4.4.4.4.4.2.m2.1.1.1.1.1.3"><mtext mathsize="70%" id="S7.T4.4.4.4.4.2.m2.1.1.1.1.1.3.cmml" xref="S7.T4.4.4.4.4.2.m2.1.1.1.1.1.3">ours</mtext></ci></apply></apply><apply id="S7.T4.4.4.4.4.2.m2.2.2.2.2.cmml" xref="S7.T4.4.4.4.4.2.m2.2.2.2.1"><abs id="S7.T4.4.4.4.4.2.m2.2.2.2.2.1.cmml" xref="S7.T4.4.4.4.4.2.m2.2.2.2.1.2"></abs><apply id="S7.T4.4.4.4.4.2.m2.2.2.2.1.1.cmml" xref="S7.T4.4.4.4.4.2.m2.2.2.2.1.1"><csymbol cd="ambiguous" id="S7.T4.4.4.4.4.2.m2.2.2.2.1.1.1.cmml" xref="S7.T4.4.4.4.4.2.m2.2.2.2.1.1">subscript</csymbol><ci id="S7.T4.4.4.4.4.2.m2.2.2.2.1.1.2.cmml" xref="S7.T4.4.4.4.4.2.m2.2.2.2.1.1.2">𝒫</ci><ci id="S7.T4.4.4.4.4.2.m2.2.2.2.1.1.3a.cmml" xref="S7.T4.4.4.4.4.2.m2.2.2.2.1.1.3"><mtext mathsize="70%" id="S7.T4.4.4.4.4.2.m2.2.2.2.1.1.3.cmml" xref="S7.T4.4.4.4.4.2.m2.2.2.2.1.1.3">PET</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T4.4.4.4.4.2.m2.2c">|\mathcal{P}_{\text{ours}}|=|\mathcal{P}_{\text{PET}}|</annotation></semantics></math>
</th>
<td id="S7.T4.4.4.4.4.3" class="ltx_td ltx_align_center">70.4 (3.1)</td>
<td id="S7.T4.4.4.4.4.4" class="ltx_td ltx_align_center">73.0 (3.2)</td>
</tr>
<tr id="S7.T4.6.6.6.8.2" class="ltx_tr">
<th id="S7.T4.6.6.6.8.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">+ demonstrations</th>
<td id="S7.T4.6.6.6.8.2.2" class="ltx_td ltx_align_center">74.0 (1.9)</td>
<td id="S7.T4.6.6.6.8.2.3" class="ltx_td ltx_align_center">71.9 (4.6)</td>
</tr>
<tr id="S7.T4.6.6.6.6" class="ltx_tr">
<th id="S7.T4.6.6.6.6.2" class="ltx_td ltx_align_left ltx_th ltx_th_row">
<math id="S7.T4.5.5.5.5.1.m1.1" class="ltx_Math" alttext="\mathcal{P}_{\text{ours}}" display="inline"><semantics id="S7.T4.5.5.5.5.1.m1.1a"><msub id="S7.T4.5.5.5.5.1.m1.1.1" xref="S7.T4.5.5.5.5.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S7.T4.5.5.5.5.1.m1.1.1.2" xref="S7.T4.5.5.5.5.1.m1.1.1.2.cmml">𝒫</mi><mtext id="S7.T4.5.5.5.5.1.m1.1.1.3" xref="S7.T4.5.5.5.5.1.m1.1.1.3a.cmml">ours</mtext></msub><annotation-xml encoding="MathML-Content" id="S7.T4.5.5.5.5.1.m1.1b"><apply id="S7.T4.5.5.5.5.1.m1.1.1.cmml" xref="S7.T4.5.5.5.5.1.m1.1.1"><csymbol cd="ambiguous" id="S7.T4.5.5.5.5.1.m1.1.1.1.cmml" xref="S7.T4.5.5.5.5.1.m1.1.1">subscript</csymbol><ci id="S7.T4.5.5.5.5.1.m1.1.1.2.cmml" xref="S7.T4.5.5.5.5.1.m1.1.1.2">𝒫</ci><ci id="S7.T4.5.5.5.5.1.m1.1.1.3a.cmml" xref="S7.T4.5.5.5.5.1.m1.1.1.3"><mtext mathsize="70%" id="S7.T4.5.5.5.5.1.m1.1.1.3.cmml" xref="S7.T4.5.5.5.5.1.m1.1.1.3">ours</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T4.5.5.5.5.1.m1.1c">\mathcal{P}_{\text{ours}}</annotation></semantics></math>, <math id="S7.T4.6.6.6.6.2.m2.1" class="ltx_Math" alttext="|\mathcal{P}_{\text{ours}}|=20" display="inline"><semantics id="S7.T4.6.6.6.6.2.m2.1a"><mrow id="S7.T4.6.6.6.6.2.m2.1.1" xref="S7.T4.6.6.6.6.2.m2.1.1.cmml"><mrow id="S7.T4.6.6.6.6.2.m2.1.1.1.1" xref="S7.T4.6.6.6.6.2.m2.1.1.1.2.cmml"><mo stretchy="false" id="S7.T4.6.6.6.6.2.m2.1.1.1.1.2" xref="S7.T4.6.6.6.6.2.m2.1.1.1.2.1.cmml">|</mo><msub id="S7.T4.6.6.6.6.2.m2.1.1.1.1.1" xref="S7.T4.6.6.6.6.2.m2.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S7.T4.6.6.6.6.2.m2.1.1.1.1.1.2" xref="S7.T4.6.6.6.6.2.m2.1.1.1.1.1.2.cmml">𝒫</mi><mtext id="S7.T4.6.6.6.6.2.m2.1.1.1.1.1.3" xref="S7.T4.6.6.6.6.2.m2.1.1.1.1.1.3a.cmml">ours</mtext></msub><mo stretchy="false" id="S7.T4.6.6.6.6.2.m2.1.1.1.1.3" xref="S7.T4.6.6.6.6.2.m2.1.1.1.2.1.cmml">|</mo></mrow><mo id="S7.T4.6.6.6.6.2.m2.1.1.2" xref="S7.T4.6.6.6.6.2.m2.1.1.2.cmml">=</mo><mn id="S7.T4.6.6.6.6.2.m2.1.1.3" xref="S7.T4.6.6.6.6.2.m2.1.1.3.cmml">20</mn></mrow><annotation-xml encoding="MathML-Content" id="S7.T4.6.6.6.6.2.m2.1b"><apply id="S7.T4.6.6.6.6.2.m2.1.1.cmml" xref="S7.T4.6.6.6.6.2.m2.1.1"><eq id="S7.T4.6.6.6.6.2.m2.1.1.2.cmml" xref="S7.T4.6.6.6.6.2.m2.1.1.2"></eq><apply id="S7.T4.6.6.6.6.2.m2.1.1.1.2.cmml" xref="S7.T4.6.6.6.6.2.m2.1.1.1.1"><abs id="S7.T4.6.6.6.6.2.m2.1.1.1.2.1.cmml" xref="S7.T4.6.6.6.6.2.m2.1.1.1.1.2"></abs><apply id="S7.T4.6.6.6.6.2.m2.1.1.1.1.1.cmml" xref="S7.T4.6.6.6.6.2.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="S7.T4.6.6.6.6.2.m2.1.1.1.1.1.1.cmml" xref="S7.T4.6.6.6.6.2.m2.1.1.1.1.1">subscript</csymbol><ci id="S7.T4.6.6.6.6.2.m2.1.1.1.1.1.2.cmml" xref="S7.T4.6.6.6.6.2.m2.1.1.1.1.1.2">𝒫</ci><ci id="S7.T4.6.6.6.6.2.m2.1.1.1.1.1.3a.cmml" xref="S7.T4.6.6.6.6.2.m2.1.1.1.1.1.3"><mtext mathsize="70%" id="S7.T4.6.6.6.6.2.m2.1.1.1.1.1.3.cmml" xref="S7.T4.6.6.6.6.2.m2.1.1.1.1.1.3">ours</mtext></ci></apply></apply><cn type="integer" id="S7.T4.6.6.6.6.2.m2.1.1.3.cmml" xref="S7.T4.6.6.6.6.2.m2.1.1.3">20</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T4.6.6.6.6.2.m2.1c">|\mathcal{P}_{\text{ours}}|=20</annotation></semantics></math>
</th>
<td id="S7.T4.6.6.6.6.3" class="ltx_td ltx_align_center">72.7 (2.5)</td>
<td id="S7.T4.6.6.6.6.4" class="ltx_td ltx_align_center">
<span id="S7.T4.6.6.6.6.4.1" class="ltx_text ltx_font_bold">73.1</span> (3.3)</td>
</tr>
<tr id="S7.T4.6.6.6.9.3" class="ltx_tr">
<th id="S7.T4.6.6.6.9.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">+ demonstrations</th>
<td id="S7.T4.6.6.6.9.3.2" class="ltx_td ltx_align_center ltx_border_bb">
<span id="S7.T4.6.6.6.9.3.2.1" class="ltx_text ltx_font_bold">75.4</span> (1.6)</td>
<td id="S7.T4.6.6.6.9.3.3" class="ltx_td ltx_align_center ltx_border_bb">72.3 (4.5)</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 4: </span>Ensemble models using manual prompts from PET <cite class="ltx_cite ltx_citemacro_cite">Schick and Schütze (<a href="#bib.bib35" title="" class="ltx_ref">2021a</a>, <a href="#bib.bib36" title="" class="ltx_ref">b</a>)</cite> and our automatic templates. PET uses 4 prompts for MNLI and 5 for RTE. We also use an equal number of templates
in <math id="S7.T4.8.m1.2" class="ltx_Math" alttext="|\mathcal{P}_{\text{ours}}|=|\mathcal{P}_{\text{PET}}|" display="inline"><semantics id="S7.T4.8.m1.2b"><mrow id="S7.T4.8.m1.2.2" xref="S7.T4.8.m1.2.2.cmml"><mrow id="S7.T4.8.m1.1.1.1.1" xref="S7.T4.8.m1.1.1.1.2.cmml"><mo stretchy="false" id="S7.T4.8.m1.1.1.1.1.2" xref="S7.T4.8.m1.1.1.1.2.1.cmml">|</mo><msub id="S7.T4.8.m1.1.1.1.1.1" xref="S7.T4.8.m1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S7.T4.8.m1.1.1.1.1.1.2" xref="S7.T4.8.m1.1.1.1.1.1.2.cmml">𝒫</mi><mtext id="S7.T4.8.m1.1.1.1.1.1.3" xref="S7.T4.8.m1.1.1.1.1.1.3a.cmml">ours</mtext></msub><mo stretchy="false" id="S7.T4.8.m1.1.1.1.1.3" xref="S7.T4.8.m1.1.1.1.2.1.cmml">|</mo></mrow><mo id="S7.T4.8.m1.2.2.3" xref="S7.T4.8.m1.2.2.3.cmml">=</mo><mrow id="S7.T4.8.m1.2.2.2.1" xref="S7.T4.8.m1.2.2.2.2.cmml"><mo stretchy="false" id="S7.T4.8.m1.2.2.2.1.2" xref="S7.T4.8.m1.2.2.2.2.1.cmml">|</mo><msub id="S7.T4.8.m1.2.2.2.1.1" xref="S7.T4.8.m1.2.2.2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S7.T4.8.m1.2.2.2.1.1.2" xref="S7.T4.8.m1.2.2.2.1.1.2.cmml">𝒫</mi><mtext id="S7.T4.8.m1.2.2.2.1.1.3" xref="S7.T4.8.m1.2.2.2.1.1.3a.cmml">PET</mtext></msub><mo stretchy="false" id="S7.T4.8.m1.2.2.2.1.3" xref="S7.T4.8.m1.2.2.2.2.1.cmml">|</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.T4.8.m1.2c"><apply id="S7.T4.8.m1.2.2.cmml" xref="S7.T4.8.m1.2.2"><eq id="S7.T4.8.m1.2.2.3.cmml" xref="S7.T4.8.m1.2.2.3"></eq><apply id="S7.T4.8.m1.1.1.1.2.cmml" xref="S7.T4.8.m1.1.1.1.1"><abs id="S7.T4.8.m1.1.1.1.2.1.cmml" xref="S7.T4.8.m1.1.1.1.1.2"></abs><apply id="S7.T4.8.m1.1.1.1.1.1.cmml" xref="S7.T4.8.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="S7.T4.8.m1.1.1.1.1.1.1.cmml" xref="S7.T4.8.m1.1.1.1.1.1">subscript</csymbol><ci id="S7.T4.8.m1.1.1.1.1.1.2.cmml" xref="S7.T4.8.m1.1.1.1.1.1.2">𝒫</ci><ci id="S7.T4.8.m1.1.1.1.1.1.3a.cmml" xref="S7.T4.8.m1.1.1.1.1.1.3"><mtext mathsize="70%" id="S7.T4.8.m1.1.1.1.1.1.3.cmml" xref="S7.T4.8.m1.1.1.1.1.1.3">ours</mtext></ci></apply></apply><apply id="S7.T4.8.m1.2.2.2.2.cmml" xref="S7.T4.8.m1.2.2.2.1"><abs id="S7.T4.8.m1.2.2.2.2.1.cmml" xref="S7.T4.8.m1.2.2.2.1.2"></abs><apply id="S7.T4.8.m1.2.2.2.1.1.cmml" xref="S7.T4.8.m1.2.2.2.1.1"><csymbol cd="ambiguous" id="S7.T4.8.m1.2.2.2.1.1.1.cmml" xref="S7.T4.8.m1.2.2.2.1.1">subscript</csymbol><ci id="S7.T4.8.m1.2.2.2.1.1.2.cmml" xref="S7.T4.8.m1.2.2.2.1.1.2">𝒫</ci><ci id="S7.T4.8.m1.2.2.2.1.1.3a.cmml" xref="S7.T4.8.m1.2.2.2.1.1.3"><mtext mathsize="70%" id="S7.T4.8.m1.2.2.2.1.1.3.cmml" xref="S7.T4.8.m1.2.2.2.1.1.3">PET</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T4.8.m1.2d">|\mathcal{P}_{\text{ours}}|=|\mathcal{P}_{\text{PET}}|</annotation></semantics></math>
for a fair comparison.</figcaption>
</figure>
<figure id="S7.T5" class="ltx_table">
<div id="S7.T5.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:390.3pt;height:155.9pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(82.5pt,-33.0pt) scale(1.73232744204041,1.73232744204041) ;">
<table id="S7.T5.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S7.T5.1.1.1.1" class="ltx_tr">
<th id="S7.T5.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="S7.T5.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S7.T5.1.1.1.1.2.1" class="ltx_text ltx_font_bold">SST-2</span></th>
<th id="S7.T5.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S7.T5.1.1.1.1.3.1" class="ltx_text ltx_font_bold">SNLI</span></th>
<th id="S7.T5.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S7.T5.1.1.1.1.4.1" class="ltx_text ltx_font_bold">TREC</span></th>
<th id="S7.T5.1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S7.T5.1.1.1.1.5.1" class="ltx_text ltx_font_bold">MRPC</span></th>
</tr>
<tr id="S7.T5.1.1.2.2" class="ltx_tr">
<th id="S7.T5.1.1.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_t">Manual</th>
<th id="S7.T5.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t"><span id="S7.T5.1.1.2.2.2.1" class="ltx_text ltx_font_bold">92.7</span></th>
<th id="S7.T5.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t"><span id="S7.T5.1.1.2.2.3.1" class="ltx_text ltx_font_bold">77.2</span></th>
<th id="S7.T5.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">84.8</th>
<th id="S7.T5.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">74.5</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S7.T5.1.1.3.1" class="ltx_tr">
<th id="S7.T5.1.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Auto T</th>
<td id="S7.T5.1.1.3.1.2" class="ltx_td ltx_align_center ltx_border_t">92.3</td>
<td id="S7.T5.1.1.3.1.3" class="ltx_td ltx_align_center ltx_border_t">77.1</td>
<td id="S7.T5.1.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t">88.2</td>
<td id="S7.T5.1.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t">76.2</td>
</tr>
<tr id="S7.T5.1.1.4.2" class="ltx_tr">
<th id="S7.T5.1.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">Auto L</th>
<td id="S7.T5.1.1.4.2.2" class="ltx_td ltx_align_center">91.5</td>
<td id="S7.T5.1.1.4.2.3" class="ltx_td ltx_align_center">75.6</td>
<td id="S7.T5.1.1.4.2.4" class="ltx_td ltx_align_center">87.0</td>
<td id="S7.T5.1.1.4.2.5" class="ltx_td ltx_align_center"><span id="S7.T5.1.1.4.2.5.1" class="ltx_text ltx_font_bold">77.2</span></td>
</tr>
<tr id="S7.T5.1.1.5.3" class="ltx_tr">
<th id="S7.T5.1.1.5.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">Auto T + L</th>
<td id="S7.T5.1.1.5.3.2" class="ltx_td ltx_align_center ltx_border_bb">92.1</td>
<td id="S7.T5.1.1.5.3.3" class="ltx_td ltx_align_center ltx_border_bb">77.0</td>
<td id="S7.T5.1.1.5.3.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T5.1.1.5.3.4.1" class="ltx_text ltx_font_bold">89.2</span></td>
<td id="S7.T5.1.1.5.3.5" class="ltx_td ltx_align_center ltx_border_bb">74.0</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 5: </span>
Comparison between manual prompts and different automatic prompt generation methods: auto-generated templates (Auto T), auto-generated label words (Auto L), and their combination (Auto T + L).
</figcaption>
</figure>
<section id="S7.SS1.SSS0.Px1" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Single-prompt results.</h4>

<div id="S7.SS1.SSS0.Px1.p1" class="ltx_para">
<p id="S7.SS1.SSS0.Px1.p1.1" class="ltx_p">Table <a href="#S6.T3" title="Table 3 ‣ 6 Fine-tuning with Demonstrations ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> shows our main results using a single prompt, either from our manually designed ones (Table <a href="#S2.T1" title="Table 1 ‣ Few-shot learning. ‣ 2 Related Work ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>) , or the best generated ones.
First, prompt-based zero-shot prediction achieves much better performance than the majority class, showing the pre-encoded knowledge in RoBERTa.
Also, “GPT-3” in-context learning
does not always improve over zero-shot prediction, likely because smaller language models are not expressive enough to use off-the-shelf like GPT-3.</p>
</div>
<div id="S7.SS1.SSS0.Px1.p2" class="ltx_para">
<p id="S7.SS1.SSS0.Px1.p2.1" class="ltx_p">Second, prompt-based fine-tuning can greatly outperform standard fine-tuning, both when using a manual prompt or a generated one.
CoLA is one interesting exception, as the input may be a non-grammatical sentence which is out of the distribution of <math id="S7.SS1.SSS0.Px1.p2.1.m1.1" class="ltx_Math" alttext="\mathcal{L}" display="inline"><semantics id="S7.SS1.SSS0.Px1.p2.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="S7.SS1.SSS0.Px1.p2.1.m1.1.1" xref="S7.SS1.SSS0.Px1.p2.1.m1.1.1.cmml">ℒ</mi><annotation-xml encoding="MathML-Content" id="S7.SS1.SSS0.Px1.p2.1.m1.1b"><ci id="S7.SS1.SSS0.Px1.p2.1.m1.1.1.cmml" xref="S7.SS1.SSS0.Px1.p2.1.m1.1.1">ℒ</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.SSS0.Px1.p2.1.m1.1c">\mathcal{L}</annotation></semantics></math>. Generally, our automatically searched templates can achieve comparable or even higher results than manual ones,
especially for tasks in which constructing strong manual templates is less intuitive (e.g., TREC, QNLI and MRPC).</p>
</div>
<figure id="S7.T6" class="ltx_table">
<div id="S7.T6.16.16" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:398.9pt;height:547.3pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(81.4pt,-111.6pt) scale(1.68906532538521,1.68906532538521) ;">
<table id="S7.T6.16.16.16" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S7.T6.16.16.16.17.1" class="ltx_tr">
<th id="S7.T6.16.16.16.17.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt"><span id="S7.T6.16.16.16.17.1.1.1" class="ltx_text ltx_font_bold">SST-2</span></th>
<td id="S7.T6.16.16.16.17.1.2" class="ltx_td ltx_align_left ltx_border_tt">(positive/negative)</td>
</tr>
<tr id="S7.T6.1.1.1.1" class="ltx_tr">
<th id="S7.T6.1.1.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Auto T</th>
<td id="S7.T6.1.1.1.1.1" class="ltx_td ltx_align_left ltx_border_t">
<math id="S7.T6.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\mathcal{M}(\mathcal{Y})" display="inline"><semantics id="S7.T6.1.1.1.1.1.m1.1a"><mrow id="S7.T6.1.1.1.1.1.m1.1.2" xref="S7.T6.1.1.1.1.1.m1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S7.T6.1.1.1.1.1.m1.1.2.2" xref="S7.T6.1.1.1.1.1.m1.1.2.2.cmml">ℳ</mi><mo lspace="0em" rspace="0em" id="S7.T6.1.1.1.1.1.m1.1.2.1" xref="S7.T6.1.1.1.1.1.m1.1.2.1.cmml">​</mo><mrow id="S7.T6.1.1.1.1.1.m1.1.2.3.2" xref="S7.T6.1.1.1.1.1.m1.1.2.cmml"><mo stretchy="false" id="S7.T6.1.1.1.1.1.m1.1.2.3.2.1" xref="S7.T6.1.1.1.1.1.m1.1.2.cmml">(</mo><mi class="ltx_font_mathcaligraphic" id="S7.T6.1.1.1.1.1.m1.1.1" xref="S7.T6.1.1.1.1.1.m1.1.1.cmml">𝒴</mi><mo stretchy="false" id="S7.T6.1.1.1.1.1.m1.1.2.3.2.2" xref="S7.T6.1.1.1.1.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.T6.1.1.1.1.1.m1.1b"><apply id="S7.T6.1.1.1.1.1.m1.1.2.cmml" xref="S7.T6.1.1.1.1.1.m1.1.2"><times id="S7.T6.1.1.1.1.1.m1.1.2.1.cmml" xref="S7.T6.1.1.1.1.1.m1.1.2.1"></times><ci id="S7.T6.1.1.1.1.1.m1.1.2.2.cmml" xref="S7.T6.1.1.1.1.1.m1.1.2.2">ℳ</ci><ci id="S7.T6.1.1.1.1.1.m1.1.1.cmml" xref="S7.T6.1.1.1.1.1.m1.1.1">𝒴</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T6.1.1.1.1.1.m1.1c">\mathcal{M}(\mathcal{Y})</annotation></semantics></math> = {great, terrible}</td>
</tr>
<tr id="S7.T6.2.2.2.2" class="ltx_tr">
<th id="S7.T6.2.2.2.2.2" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S7.T6.2.2.2.2.1" class="ltx_td ltx_align_left">#1. <span id="S7.T6.2.2.2.2.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S7.T6.2.2.2.2.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S7.T6.2.2.2.2.1.1.m1.1a"><msub id="S7.T6.2.2.2.2.1.1.m1.1.1" xref="S7.T6.2.2.2.2.1.1.m1.1.1.cmml"><mi id="S7.T6.2.2.2.2.1.1.m1.1.1.2" xref="S7.T6.2.2.2.2.1.1.m1.1.1.2.cmml">S</mi><mn id="S7.T6.2.2.2.2.1.1.m1.1.1.3" xref="S7.T6.2.2.2.2.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S7.T6.2.2.2.2.1.1.m1.1b"><apply id="S7.T6.2.2.2.2.1.1.m1.1.1.cmml" xref="S7.T6.2.2.2.2.1.1.m1.1.1"><csymbol cd="ambiguous" id="S7.T6.2.2.2.2.1.1.m1.1.1.1.cmml" xref="S7.T6.2.2.2.2.1.1.m1.1.1">subscript</csymbol><ci id="S7.T6.2.2.2.2.1.1.m1.1.1.2.cmml" xref="S7.T6.2.2.2.2.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S7.T6.2.2.2.2.1.1.m1.1.1.3.cmml" xref="S7.T6.2.2.2.2.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T6.2.2.2.2.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> A <span id="S7.T6.2.2.2.2.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> one .</td>
</tr>
<tr id="S7.T6.3.3.3.3" class="ltx_tr">
<th id="S7.T6.3.3.3.3.2" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S7.T6.3.3.3.3.1" class="ltx_td ltx_align_left">#2. <span id="S7.T6.3.3.3.3.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S7.T6.3.3.3.3.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S7.T6.3.3.3.3.1.1.m1.1a"><msub id="S7.T6.3.3.3.3.1.1.m1.1.1" xref="S7.T6.3.3.3.3.1.1.m1.1.1.cmml"><mi id="S7.T6.3.3.3.3.1.1.m1.1.1.2" xref="S7.T6.3.3.3.3.1.1.m1.1.1.2.cmml">S</mi><mn id="S7.T6.3.3.3.3.1.1.m1.1.1.3" xref="S7.T6.3.3.3.3.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S7.T6.3.3.3.3.1.1.m1.1b"><apply id="S7.T6.3.3.3.3.1.1.m1.1.1.cmml" xref="S7.T6.3.3.3.3.1.1.m1.1.1"><csymbol cd="ambiguous" id="S7.T6.3.3.3.3.1.1.m1.1.1.1.cmml" xref="S7.T6.3.3.3.3.1.1.m1.1.1">subscript</csymbol><ci id="S7.T6.3.3.3.3.1.1.m1.1.1.2.cmml" xref="S7.T6.3.3.3.3.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S7.T6.3.3.3.3.1.1.m1.1.1.3.cmml" xref="S7.T6.3.3.3.3.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T6.3.3.3.3.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> A <span id="S7.T6.3.3.3.3.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> piece .</td>
</tr>
<tr id="S7.T6.4.4.4.4" class="ltx_tr">
<th id="S7.T6.4.4.4.4.2" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S7.T6.4.4.4.4.1" class="ltx_td ltx_align_left">#3. <span id="S7.T6.4.4.4.4.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S7.T6.4.4.4.4.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S7.T6.4.4.4.4.1.1.m1.1a"><msub id="S7.T6.4.4.4.4.1.1.m1.1.1" xref="S7.T6.4.4.4.4.1.1.m1.1.1.cmml"><mi id="S7.T6.4.4.4.4.1.1.m1.1.1.2" xref="S7.T6.4.4.4.4.1.1.m1.1.1.2.cmml">S</mi><mn id="S7.T6.4.4.4.4.1.1.m1.1.1.3" xref="S7.T6.4.4.4.4.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S7.T6.4.4.4.4.1.1.m1.1b"><apply id="S7.T6.4.4.4.4.1.1.m1.1.1.cmml" xref="S7.T6.4.4.4.4.1.1.m1.1.1"><csymbol cd="ambiguous" id="S7.T6.4.4.4.4.1.1.m1.1.1.1.cmml" xref="S7.T6.4.4.4.4.1.1.m1.1.1">subscript</csymbol><ci id="S7.T6.4.4.4.4.1.1.m1.1.1.2.cmml" xref="S7.T6.4.4.4.4.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S7.T6.4.4.4.4.1.1.m1.1.1.3.cmml" xref="S7.T6.4.4.4.4.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T6.4.4.4.4.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> All in all <span id="S7.T6.4.4.4.4.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
</tr>
<tr id="S7.T6.6.6.6.6" class="ltx_tr">
<th id="S7.T6.6.6.6.6.3" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Auto L</th>
<td id="S7.T6.6.6.6.6.2" class="ltx_td ltx_align_left ltx_border_t">
<math id="S7.T6.5.5.5.5.1.m1.1" class="ltx_Math" alttext="\mathcal{T}({x}_{\mathrm{in}})" display="inline"><semantics id="S7.T6.5.5.5.5.1.m1.1a"><mrow id="S7.T6.5.5.5.5.1.m1.1.1" xref="S7.T6.5.5.5.5.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S7.T6.5.5.5.5.1.m1.1.1.3" xref="S7.T6.5.5.5.5.1.m1.1.1.3.cmml">𝒯</mi><mo lspace="0em" rspace="0em" id="S7.T6.5.5.5.5.1.m1.1.1.2" xref="S7.T6.5.5.5.5.1.m1.1.1.2.cmml">​</mo><mrow id="S7.T6.5.5.5.5.1.m1.1.1.1.1" xref="S7.T6.5.5.5.5.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S7.T6.5.5.5.5.1.m1.1.1.1.1.2" xref="S7.T6.5.5.5.5.1.m1.1.1.1.1.1.cmml">(</mo><msub id="S7.T6.5.5.5.5.1.m1.1.1.1.1.1" xref="S7.T6.5.5.5.5.1.m1.1.1.1.1.1.cmml"><mi id="S7.T6.5.5.5.5.1.m1.1.1.1.1.1.2" xref="S7.T6.5.5.5.5.1.m1.1.1.1.1.1.2.cmml">x</mi><mi id="S7.T6.5.5.5.5.1.m1.1.1.1.1.1.3" xref="S7.T6.5.5.5.5.1.m1.1.1.1.1.1.3.cmml">in</mi></msub><mo stretchy="false" id="S7.T6.5.5.5.5.1.m1.1.1.1.1.3" xref="S7.T6.5.5.5.5.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.T6.5.5.5.5.1.m1.1b"><apply id="S7.T6.5.5.5.5.1.m1.1.1.cmml" xref="S7.T6.5.5.5.5.1.m1.1.1"><times id="S7.T6.5.5.5.5.1.m1.1.1.2.cmml" xref="S7.T6.5.5.5.5.1.m1.1.1.2"></times><ci id="S7.T6.5.5.5.5.1.m1.1.1.3.cmml" xref="S7.T6.5.5.5.5.1.m1.1.1.3">𝒯</ci><apply id="S7.T6.5.5.5.5.1.m1.1.1.1.1.1.cmml" xref="S7.T6.5.5.5.5.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S7.T6.5.5.5.5.1.m1.1.1.1.1.1.1.cmml" xref="S7.T6.5.5.5.5.1.m1.1.1.1.1">subscript</csymbol><ci id="S7.T6.5.5.5.5.1.m1.1.1.1.1.1.2.cmml" xref="S7.T6.5.5.5.5.1.m1.1.1.1.1.1.2">𝑥</ci><ci id="S7.T6.5.5.5.5.1.m1.1.1.1.1.1.3.cmml" xref="S7.T6.5.5.5.5.1.m1.1.1.1.1.1.3">in</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T6.5.5.5.5.1.m1.1c">\mathcal{T}({x}_{\mathrm{in}})</annotation></semantics></math> = <span id="S7.T6.6.6.6.6.2.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S7.T6.6.6.6.6.2.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S7.T6.6.6.6.6.2.1.m1.1a"><msub id="S7.T6.6.6.6.6.2.1.m1.1.1" xref="S7.T6.6.6.6.6.2.1.m1.1.1.cmml"><mi id="S7.T6.6.6.6.6.2.1.m1.1.1.2" xref="S7.T6.6.6.6.6.2.1.m1.1.1.2.cmml">S</mi><mn id="S7.T6.6.6.6.6.2.1.m1.1.1.3" xref="S7.T6.6.6.6.6.2.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S7.T6.6.6.6.6.2.1.m1.1b"><apply id="S7.T6.6.6.6.6.2.1.m1.1.1.cmml" xref="S7.T6.6.6.6.6.2.1.m1.1.1"><csymbol cd="ambiguous" id="S7.T6.6.6.6.6.2.1.m1.1.1.1.cmml" xref="S7.T6.6.6.6.6.2.1.m1.1.1">subscript</csymbol><ci id="S7.T6.6.6.6.6.2.1.m1.1.1.2.cmml" xref="S7.T6.6.6.6.6.2.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S7.T6.6.6.6.6.2.1.m1.1.1.3.cmml" xref="S7.T6.6.6.6.6.2.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T6.6.6.6.6.2.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> It was <span id="S7.T6.6.6.6.6.2.2" class="ltx_text ltx_font_typewriter">[MASK]</span>.</td>
</tr>
<tr id="S7.T6.16.16.16.18.2" class="ltx_tr">
<th id="S7.T6.16.16.16.18.2.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S7.T6.16.16.16.18.2.2" class="ltx_td ltx_align_left">#1. irresistible/pathetic</td>
</tr>
<tr id="S7.T6.16.16.16.19.3" class="ltx_tr">
<th id="S7.T6.16.16.16.19.3.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S7.T6.16.16.16.19.3.2" class="ltx_td ltx_align_left">#2. wonderful/bad</td>
</tr>
<tr id="S7.T6.16.16.16.20.4" class="ltx_tr">
<th id="S7.T6.16.16.16.20.4.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S7.T6.16.16.16.20.4.2" class="ltx_td ltx_align_left">#3. delicious/bad</td>
</tr>
<tr id="S7.T6.16.16.16.21.5" class="ltx_tr">
<th id="S7.T6.16.16.16.21.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t"><span id="S7.T6.16.16.16.21.5.1.1" class="ltx_text ltx_font_bold">SNLI</span></th>
<td id="S7.T6.16.16.16.21.5.2" class="ltx_td ltx_align_left ltx_border_t">(entailment/neutral/contradiction)</td>
</tr>
<tr id="S7.T6.7.7.7.7" class="ltx_tr">
<th id="S7.T6.7.7.7.7.2" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Auto T</th>
<td id="S7.T6.7.7.7.7.1" class="ltx_td ltx_align_left ltx_border_t">
<math id="S7.T6.7.7.7.7.1.m1.1" class="ltx_Math" alttext="\mathcal{M}(\mathcal{Y})" display="inline"><semantics id="S7.T6.7.7.7.7.1.m1.1a"><mrow id="S7.T6.7.7.7.7.1.m1.1.2" xref="S7.T6.7.7.7.7.1.m1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="S7.T6.7.7.7.7.1.m1.1.2.2" xref="S7.T6.7.7.7.7.1.m1.1.2.2.cmml">ℳ</mi><mo lspace="0em" rspace="0em" id="S7.T6.7.7.7.7.1.m1.1.2.1" xref="S7.T6.7.7.7.7.1.m1.1.2.1.cmml">​</mo><mrow id="S7.T6.7.7.7.7.1.m1.1.2.3.2" xref="S7.T6.7.7.7.7.1.m1.1.2.cmml"><mo stretchy="false" id="S7.T6.7.7.7.7.1.m1.1.2.3.2.1" xref="S7.T6.7.7.7.7.1.m1.1.2.cmml">(</mo><mi class="ltx_font_mathcaligraphic" id="S7.T6.7.7.7.7.1.m1.1.1" xref="S7.T6.7.7.7.7.1.m1.1.1.cmml">𝒴</mi><mo stretchy="false" id="S7.T6.7.7.7.7.1.m1.1.2.3.2.2" xref="S7.T6.7.7.7.7.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.T6.7.7.7.7.1.m1.1b"><apply id="S7.T6.7.7.7.7.1.m1.1.2.cmml" xref="S7.T6.7.7.7.7.1.m1.1.2"><times id="S7.T6.7.7.7.7.1.m1.1.2.1.cmml" xref="S7.T6.7.7.7.7.1.m1.1.2.1"></times><ci id="S7.T6.7.7.7.7.1.m1.1.2.2.cmml" xref="S7.T6.7.7.7.7.1.m1.1.2.2">ℳ</ci><ci id="S7.T6.7.7.7.7.1.m1.1.1.cmml" xref="S7.T6.7.7.7.7.1.m1.1.1">𝒴</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T6.7.7.7.7.1.m1.1c">\mathcal{M}(\mathcal{Y})</annotation></semantics></math> = {Yes, Maybe, No}</td>
</tr>
<tr id="S7.T6.9.9.9.9" class="ltx_tr">
<th id="S7.T6.9.9.9.9.3" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S7.T6.9.9.9.9.2" class="ltx_td ltx_align_left">#1. <span id="S7.T6.8.8.8.8.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S7.T6.8.8.8.8.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S7.T6.8.8.8.8.1.1.m1.1a"><msub id="S7.T6.8.8.8.8.1.1.m1.1.1" xref="S7.T6.8.8.8.8.1.1.m1.1.1.cmml"><mi id="S7.T6.8.8.8.8.1.1.m1.1.1.2" xref="S7.T6.8.8.8.8.1.1.m1.1.1.2.cmml">S</mi><mn id="S7.T6.8.8.8.8.1.1.m1.1.1.3" xref="S7.T6.8.8.8.8.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S7.T6.8.8.8.8.1.1.m1.1b"><apply id="S7.T6.8.8.8.8.1.1.m1.1.1.cmml" xref="S7.T6.8.8.8.8.1.1.m1.1.1"><csymbol cd="ambiguous" id="S7.T6.8.8.8.8.1.1.m1.1.1.1.cmml" xref="S7.T6.8.8.8.8.1.1.m1.1.1">subscript</csymbol><ci id="S7.T6.8.8.8.8.1.1.m1.1.1.2.cmml" xref="S7.T6.8.8.8.8.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S7.T6.8.8.8.8.1.1.m1.1.1.3.cmml" xref="S7.T6.8.8.8.8.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T6.8.8.8.8.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="S7.T6.9.9.9.9.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , no , <span id="S7.T6.9.9.9.9.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="S7.T6.9.9.9.9.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="S7.T6.9.9.9.9.2.2.m1.1a"><msub id="S7.T6.9.9.9.9.2.2.m1.1.1" xref="S7.T6.9.9.9.9.2.2.m1.1.1.cmml"><mi id="S7.T6.9.9.9.9.2.2.m1.1.1.2" xref="S7.T6.9.9.9.9.2.2.m1.1.1.2.cmml">S</mi><mn id="S7.T6.9.9.9.9.2.2.m1.1.1.3" xref="S7.T6.9.9.9.9.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S7.T6.9.9.9.9.2.2.m1.1b"><apply id="S7.T6.9.9.9.9.2.2.m1.1.1.cmml" xref="S7.T6.9.9.9.9.2.2.m1.1.1"><csymbol cd="ambiguous" id="S7.T6.9.9.9.9.2.2.m1.1.1.1.cmml" xref="S7.T6.9.9.9.9.2.2.m1.1.1">subscript</csymbol><ci id="S7.T6.9.9.9.9.2.2.m1.1.1.2.cmml" xref="S7.T6.9.9.9.9.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="S7.T6.9.9.9.9.2.2.m1.1.1.3.cmml" xref="S7.T6.9.9.9.9.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T6.9.9.9.9.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
</tr>
<tr id="S7.T6.11.11.11.11" class="ltx_tr">
<th id="S7.T6.11.11.11.11.3" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S7.T6.11.11.11.11.2" class="ltx_td ltx_align_left">#2. <span id="S7.T6.10.10.10.10.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S7.T6.10.10.10.10.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S7.T6.10.10.10.10.1.1.m1.1a"><msub id="S7.T6.10.10.10.10.1.1.m1.1.1" xref="S7.T6.10.10.10.10.1.1.m1.1.1.cmml"><mi id="S7.T6.10.10.10.10.1.1.m1.1.1.2" xref="S7.T6.10.10.10.10.1.1.m1.1.1.2.cmml">S</mi><mn id="S7.T6.10.10.10.10.1.1.m1.1.1.3" xref="S7.T6.10.10.10.10.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S7.T6.10.10.10.10.1.1.m1.1b"><apply id="S7.T6.10.10.10.10.1.1.m1.1.1.cmml" xref="S7.T6.10.10.10.10.1.1.m1.1.1"><csymbol cd="ambiguous" id="S7.T6.10.10.10.10.1.1.m1.1.1.1.cmml" xref="S7.T6.10.10.10.10.1.1.m1.1.1">subscript</csymbol><ci id="S7.T6.10.10.10.10.1.1.m1.1.1.2.cmml" xref="S7.T6.10.10.10.10.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S7.T6.10.10.10.10.1.1.m1.1.1.3.cmml" xref="S7.T6.10.10.10.10.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T6.10.10.10.10.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="S7.T6.11.11.11.11.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , in this case <span id="S7.T6.11.11.11.11.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="S7.T6.11.11.11.11.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="S7.T6.11.11.11.11.2.2.m1.1a"><msub id="S7.T6.11.11.11.11.2.2.m1.1.1" xref="S7.T6.11.11.11.11.2.2.m1.1.1.cmml"><mi id="S7.T6.11.11.11.11.2.2.m1.1.1.2" xref="S7.T6.11.11.11.11.2.2.m1.1.1.2.cmml">S</mi><mn id="S7.T6.11.11.11.11.2.2.m1.1.1.3" xref="S7.T6.11.11.11.11.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S7.T6.11.11.11.11.2.2.m1.1b"><apply id="S7.T6.11.11.11.11.2.2.m1.1.1.cmml" xref="S7.T6.11.11.11.11.2.2.m1.1.1"><csymbol cd="ambiguous" id="S7.T6.11.11.11.11.2.2.m1.1.1.1.cmml" xref="S7.T6.11.11.11.11.2.2.m1.1.1">subscript</csymbol><ci id="S7.T6.11.11.11.11.2.2.m1.1.1.2.cmml" xref="S7.T6.11.11.11.11.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="S7.T6.11.11.11.11.2.2.m1.1.1.3.cmml" xref="S7.T6.11.11.11.11.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T6.11.11.11.11.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
</tr>
<tr id="S7.T6.13.13.13.13" class="ltx_tr">
<th id="S7.T6.13.13.13.13.3" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S7.T6.13.13.13.13.2" class="ltx_td ltx_align_left">#3. <span id="S7.T6.12.12.12.12.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S7.T6.12.12.12.12.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S7.T6.12.12.12.12.1.1.m1.1a"><msub id="S7.T6.12.12.12.12.1.1.m1.1.1" xref="S7.T6.12.12.12.12.1.1.m1.1.1.cmml"><mi id="S7.T6.12.12.12.12.1.1.m1.1.1.2" xref="S7.T6.12.12.12.12.1.1.m1.1.1.2.cmml">S</mi><mn id="S7.T6.12.12.12.12.1.1.m1.1.1.3" xref="S7.T6.12.12.12.12.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S7.T6.12.12.12.12.1.1.m1.1b"><apply id="S7.T6.12.12.12.12.1.1.m1.1.1.cmml" xref="S7.T6.12.12.12.12.1.1.m1.1.1"><csymbol cd="ambiguous" id="S7.T6.12.12.12.12.1.1.m1.1.1.1.cmml" xref="S7.T6.12.12.12.12.1.1.m1.1.1">subscript</csymbol><ci id="S7.T6.12.12.12.12.1.1.m1.1.1.2.cmml" xref="S7.T6.12.12.12.12.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S7.T6.12.12.12.12.1.1.m1.1.1.3.cmml" xref="S7.T6.12.12.12.12.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T6.12.12.12.12.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="S7.T6.13.13.13.13.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> this time <span id="S7.T6.13.13.13.13.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="S7.T6.13.13.13.13.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="S7.T6.13.13.13.13.2.2.m1.1a"><msub id="S7.T6.13.13.13.13.2.2.m1.1.1" xref="S7.T6.13.13.13.13.2.2.m1.1.1.cmml"><mi id="S7.T6.13.13.13.13.2.2.m1.1.1.2" xref="S7.T6.13.13.13.13.2.2.m1.1.1.2.cmml">S</mi><mn id="S7.T6.13.13.13.13.2.2.m1.1.1.3" xref="S7.T6.13.13.13.13.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S7.T6.13.13.13.13.2.2.m1.1b"><apply id="S7.T6.13.13.13.13.2.2.m1.1.1.cmml" xref="S7.T6.13.13.13.13.2.2.m1.1.1"><csymbol cd="ambiguous" id="S7.T6.13.13.13.13.2.2.m1.1.1.1.cmml" xref="S7.T6.13.13.13.13.2.2.m1.1.1">subscript</csymbol><ci id="S7.T6.13.13.13.13.2.2.m1.1.1.2.cmml" xref="S7.T6.13.13.13.13.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="S7.T6.13.13.13.13.2.2.m1.1.1.3.cmml" xref="S7.T6.13.13.13.13.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T6.13.13.13.13.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
</tr>
<tr id="S7.T6.16.16.16.16" class="ltx_tr">
<th id="S7.T6.16.16.16.16.4" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Auto L</th>
<td id="S7.T6.16.16.16.16.3" class="ltx_td ltx_align_left ltx_border_t">
<math id="S7.T6.14.14.14.14.1.m1.1" class="ltx_Math" alttext="\mathcal{T}({x}_{\mathrm{in}})" display="inline"><semantics id="S7.T6.14.14.14.14.1.m1.1a"><mrow id="S7.T6.14.14.14.14.1.m1.1.1" xref="S7.T6.14.14.14.14.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="S7.T6.14.14.14.14.1.m1.1.1.3" xref="S7.T6.14.14.14.14.1.m1.1.1.3.cmml">𝒯</mi><mo lspace="0em" rspace="0em" id="S7.T6.14.14.14.14.1.m1.1.1.2" xref="S7.T6.14.14.14.14.1.m1.1.1.2.cmml">​</mo><mrow id="S7.T6.14.14.14.14.1.m1.1.1.1.1" xref="S7.T6.14.14.14.14.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S7.T6.14.14.14.14.1.m1.1.1.1.1.2" xref="S7.T6.14.14.14.14.1.m1.1.1.1.1.1.cmml">(</mo><msub id="S7.T6.14.14.14.14.1.m1.1.1.1.1.1" xref="S7.T6.14.14.14.14.1.m1.1.1.1.1.1.cmml"><mi id="S7.T6.14.14.14.14.1.m1.1.1.1.1.1.2" xref="S7.T6.14.14.14.14.1.m1.1.1.1.1.1.2.cmml">x</mi><mi id="S7.T6.14.14.14.14.1.m1.1.1.1.1.1.3" xref="S7.T6.14.14.14.14.1.m1.1.1.1.1.1.3.cmml">in</mi></msub><mo stretchy="false" id="S7.T6.14.14.14.14.1.m1.1.1.1.1.3" xref="S7.T6.14.14.14.14.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S7.T6.14.14.14.14.1.m1.1b"><apply id="S7.T6.14.14.14.14.1.m1.1.1.cmml" xref="S7.T6.14.14.14.14.1.m1.1.1"><times id="S7.T6.14.14.14.14.1.m1.1.1.2.cmml" xref="S7.T6.14.14.14.14.1.m1.1.1.2"></times><ci id="S7.T6.14.14.14.14.1.m1.1.1.3.cmml" xref="S7.T6.14.14.14.14.1.m1.1.1.3">𝒯</ci><apply id="S7.T6.14.14.14.14.1.m1.1.1.1.1.1.cmml" xref="S7.T6.14.14.14.14.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S7.T6.14.14.14.14.1.m1.1.1.1.1.1.1.cmml" xref="S7.T6.14.14.14.14.1.m1.1.1.1.1">subscript</csymbol><ci id="S7.T6.14.14.14.14.1.m1.1.1.1.1.1.2.cmml" xref="S7.T6.14.14.14.14.1.m1.1.1.1.1.1.2">𝑥</ci><ci id="S7.T6.14.14.14.14.1.m1.1.1.1.1.1.3.cmml" xref="S7.T6.14.14.14.14.1.m1.1.1.1.1.1.3">in</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T6.14.14.14.14.1.m1.1c">\mathcal{T}({x}_{\mathrm{in}})</annotation></semantics></math> = <span id="S7.T6.15.15.15.15.2.1" class="ltx_text ltx_font_typewriter">&lt;<math id="S7.T6.15.15.15.15.2.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="S7.T6.15.15.15.15.2.1.m1.1a"><msub id="S7.T6.15.15.15.15.2.1.m1.1.1" xref="S7.T6.15.15.15.15.2.1.m1.1.1.cmml"><mi id="S7.T6.15.15.15.15.2.1.m1.1.1.2" xref="S7.T6.15.15.15.15.2.1.m1.1.1.2.cmml">S</mi><mn id="S7.T6.15.15.15.15.2.1.m1.1.1.3" xref="S7.T6.15.15.15.15.2.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="S7.T6.15.15.15.15.2.1.m1.1b"><apply id="S7.T6.15.15.15.15.2.1.m1.1.1.cmml" xref="S7.T6.15.15.15.15.2.1.m1.1.1"><csymbol cd="ambiguous" id="S7.T6.15.15.15.15.2.1.m1.1.1.1.cmml" xref="S7.T6.15.15.15.15.2.1.m1.1.1">subscript</csymbol><ci id="S7.T6.15.15.15.15.2.1.m1.1.1.2.cmml" xref="S7.T6.15.15.15.15.2.1.m1.1.1.2">𝑆</ci><cn type="integer" id="S7.T6.15.15.15.15.2.1.m1.1.1.3.cmml" xref="S7.T6.15.15.15.15.2.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T6.15.15.15.15.2.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> ? <span id="S7.T6.16.16.16.16.3.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , <span id="S7.T6.16.16.16.16.3.2" class="ltx_text ltx_font_typewriter">&lt;<math id="S7.T6.16.16.16.16.3.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="S7.T6.16.16.16.16.3.2.m1.1a"><msub id="S7.T6.16.16.16.16.3.2.m1.1.1" xref="S7.T6.16.16.16.16.3.2.m1.1.1.cmml"><mi id="S7.T6.16.16.16.16.3.2.m1.1.1.2" xref="S7.T6.16.16.16.16.3.2.m1.1.1.2.cmml">S</mi><mn id="S7.T6.16.16.16.16.3.2.m1.1.1.3" xref="S7.T6.16.16.16.16.3.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="S7.T6.16.16.16.16.3.2.m1.1b"><apply id="S7.T6.16.16.16.16.3.2.m1.1.1.cmml" xref="S7.T6.16.16.16.16.3.2.m1.1.1"><csymbol cd="ambiguous" id="S7.T6.16.16.16.16.3.2.m1.1.1.1.cmml" xref="S7.T6.16.16.16.16.3.2.m1.1.1">subscript</csymbol><ci id="S7.T6.16.16.16.16.3.2.m1.1.1.2.cmml" xref="S7.T6.16.16.16.16.3.2.m1.1.1.2">𝑆</ci><cn type="integer" id="S7.T6.16.16.16.16.3.2.m1.1.1.3.cmml" xref="S7.T6.16.16.16.16.3.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.T6.16.16.16.16.3.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
</tr>
<tr id="S7.T6.16.16.16.22.6" class="ltx_tr">
<th id="S7.T6.16.16.16.22.6.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S7.T6.16.16.16.22.6.2" class="ltx_td ltx_align_left">#1. Alright/Watch/Except</td>
</tr>
<tr id="S7.T6.16.16.16.23.7" class="ltx_tr">
<th id="S7.T6.16.16.16.23.7.1" class="ltx_td ltx_th ltx_th_row"></th>
<td id="S7.T6.16.16.16.23.7.2" class="ltx_td ltx_align_left">#2. Hi/Watch/Worse</td>
</tr>
<tr id="S7.T6.16.16.16.24.8" class="ltx_tr">
<th id="S7.T6.16.16.16.24.8.1" class="ltx_td ltx_th ltx_th_row ltx_border_bb"></th>
<td id="S7.T6.16.16.16.24.8.2" class="ltx_td ltx_align_left ltx_border_bb">#3. Regardless/Fortunately/Unless</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 6: </span>Examples of our automatically generated templates (Auto T) and label words (Auto L).</figcaption>
</figure>
<div id="S7.SS1.SSS0.Px1.p3" class="ltx_para">
<p id="S7.SS1.SSS0.Px1.p3.2" class="ltx_p">Finally, using demonstrations in context leads to consistent gains in a majority of tasks.
In summary, our combined solution—fine-tuning with automatically searched templates and sampled demonstration sets—achieves a <math id="S7.SS1.SSS0.Px1.p3.1.m1.1" class="ltx_Math" alttext="30\%" display="inline"><semantics id="S7.SS1.SSS0.Px1.p3.1.m1.1a"><mrow id="S7.SS1.SSS0.Px1.p3.1.m1.1.1" xref="S7.SS1.SSS0.Px1.p3.1.m1.1.1.cmml"><mn id="S7.SS1.SSS0.Px1.p3.1.m1.1.1.2" xref="S7.SS1.SSS0.Px1.p3.1.m1.1.1.2.cmml">30</mn><mo id="S7.SS1.SSS0.Px1.p3.1.m1.1.1.1" xref="S7.SS1.SSS0.Px1.p3.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.SSS0.Px1.p3.1.m1.1b"><apply id="S7.SS1.SSS0.Px1.p3.1.m1.1.1.cmml" xref="S7.SS1.SSS0.Px1.p3.1.m1.1.1"><csymbol cd="latexml" id="S7.SS1.SSS0.Px1.p3.1.m1.1.1.1.cmml" xref="S7.SS1.SSS0.Px1.p3.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S7.SS1.SSS0.Px1.p3.1.m1.1.1.2.cmml" xref="S7.SS1.SSS0.Px1.p3.1.m1.1.1.2">30</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.SSS0.Px1.p3.1.m1.1c">30\%</annotation></semantics></math> gain on SNLI compared to standard fine-tuning, and <math id="S7.SS1.SSS0.Px1.p3.2.m2.1" class="ltx_Math" alttext="11\%" display="inline"><semantics id="S7.SS1.SSS0.Px1.p3.2.m2.1a"><mrow id="S7.SS1.SSS0.Px1.p3.2.m2.1.1" xref="S7.SS1.SSS0.Px1.p3.2.m2.1.1.cmml"><mn id="S7.SS1.SSS0.Px1.p3.2.m2.1.1.2" xref="S7.SS1.SSS0.Px1.p3.2.m2.1.1.2.cmml">11</mn><mo id="S7.SS1.SSS0.Px1.p3.2.m2.1.1.1" xref="S7.SS1.SSS0.Px1.p3.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S7.SS1.SSS0.Px1.p3.2.m2.1b"><apply id="S7.SS1.SSS0.Px1.p3.2.m2.1.1.cmml" xref="S7.SS1.SSS0.Px1.p3.2.m2.1.1"><csymbol cd="latexml" id="S7.SS1.SSS0.Px1.p3.2.m2.1.1.1.cmml" xref="S7.SS1.SSS0.Px1.p3.2.m2.1.1.1">percent</csymbol><cn type="integer" id="S7.SS1.SSS0.Px1.p3.2.m2.1.1.2.cmml" xref="S7.SS1.SSS0.Px1.p3.2.m2.1.1.2">11</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS1.SSS0.Px1.p3.2.m2.1c">11\%</annotation></semantics></math> gain on average.</p>
</div>
</section>
<section id="S7.SS1.SSS0.Px2" class="ltx_paragraph">
<h4 class="ltx_title ltx_title_paragraph">Ensemble results.</h4>

<div id="S7.SS1.SSS0.Px2.p1" class="ltx_para">
<p id="S7.SS1.SSS0.Px2.p1.1" class="ltx_p">An advantage of automatic prompt search is that we can generate as many prompts as we want, train individual models, and create large ensembles. PET <cite class="ltx_cite ltx_citemacro_cite">Schick and Schütze (<a href="#bib.bib35" title="" class="ltx_ref">2021a</a>, <a href="#bib.bib36" title="" class="ltx_ref">b</a>)</cite> also ensembles multiple models trained with manual prompts.<span id="footnote10" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">10</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">10</sup><span class="ltx_tag ltx_tag_note">10</span>They then use unlabeled data and distillation to get a single model, which is outside of our scope.</span></span></span> In Table <a href="#S7.T4" title="Table 4 ‣ 7.1 Main results ‣ 7 Experiments ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>, we make a direct comparison of our searched prompts and PET’s manual prompts on MNLI and RTE (two datasets that we evaluate in common).<span id="footnote11" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">11</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">11</sup><span class="ltx_tag ltx_tag_note">11</span>In the PET NLI templates, the hypothesis is put before the premise, which we actually found to be suboptimal. In our experiments, we swap the two and get better results.</span></span></span>
As the results show, an ensemble with multiple templates always improves performance.
An ensemble of the same number of automatic templates achieves comparable or better performance than the ensemble of PET’s manual prompts. Increasing the number of automatic templates brings further gains.</p>
</div>
</section>
</section>
<section id="S7.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">7.2 </span>Analysis of generated prompts</h3>

<div id="S7.SS2.p1" class="ltx_para">
<p id="S7.SS2.p1.1" class="ltx_p">Table <a href="#S7.T5" title="Table 5 ‣ 7.1 Main results ‣ 7 Experiments ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> gives the results of using manual vs automatic prompts.
For automatic prompts, we compare
template search (Auto T),
label word search (Auto L),
and a joint variant (Auto T + L) in which we start from manual label words, apply Auto T, and then Auto L.
In most cases, Auto T achieves comparable or higher performance than manual ones, and is consistently the best variant. Auto L outperforms manual prompts on TREC and MRPC—but is considerably worse on SNLI. Auto T + L is often better than Auto L, but only sometimes better than Auto T.
Table <a href="#S7.T6" title="Table 6 ‣ Single-prompt results. ‣ 7.1 Main results ‣ 7 Experiments ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a> shows examples from Auto T and Auto L (A full list in Appendix <a href="#A5" title="Appendix E Generated Prompts ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E</span></a>).
Auto T templates generally fit the context and label words well, but can contain biased peculiarities (e.g., “<em id="S7.SS2.p1.1.1" class="ltx_emph ltx_font_italic"><span id="S7.SS2.p1.1.1.1" class="ltx_text ltx_font_upright">{</span>Yes/No<span id="S7.SS2.p1.1.1.2" class="ltx_text ltx_font_upright">}</span>, no</em>” in SNLI).
For Auto L words, things are mixed: while most look intuitively reasonable, there are also some mysterious abnormalities (e.g., “<em id="S7.SS2.p1.1.2" class="ltx_emph ltx_font_italic">Hi</em>” for the “entailment” class in SNLI).</p>
</div>
</section>
<section id="S7.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">7.3 </span>Analysis of demonstration sampling</h3>

<figure id="S7.T7" class="ltx_table">
<div id="S7.T7.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:151.8pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(88.2pt,-30.9pt) scale(1.68618604372699,1.68618604372699) ;">
<table id="S7.T7.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S7.T7.1.1.1.1" class="ltx_tr">
<th id="S7.T7.1.1.1.1.1" class="ltx_td ltx_th ltx_th_row ltx_border_tt"></th>
<th id="S7.T7.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S7.T7.1.1.1.1.2.1" class="ltx_text ltx_font_bold">SST-2</span></th>
<th id="S7.T7.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S7.T7.1.1.1.1.3.1" class="ltx_text ltx_font_bold">SNLI</span></th>
<th id="S7.T7.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S7.T7.1.1.1.1.4.1" class="ltx_text ltx_font_bold">TREC</span></th>
<th id="S7.T7.1.1.1.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S7.T7.1.1.1.1.5.1" class="ltx_text ltx_font_bold">MRPC</span></th>
</tr>
<tr id="S7.T7.1.1.2.2" class="ltx_tr">
<th id="S7.T7.1.1.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_t">Prompt-based FT</th>
<th id="S7.T7.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t"><span id="S7.T7.1.1.2.2.2.1" class="ltx_text ltx_font_bold">92.7</span></th>
<th id="S7.T7.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">77.2</th>
<th id="S7.T7.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">84.8</th>
<th id="S7.T7.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_t">74.5</th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S7.T7.1.1.3.1" class="ltx_tr">
<th id="S7.T7.1.1.3.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Uniform sampling</th>
<td id="S7.T7.1.1.3.1.2" class="ltx_td ltx_align_center ltx_border_t">92.3</td>
<td id="S7.T7.1.1.3.1.3" class="ltx_td ltx_align_center ltx_border_t">78.8</td>
<td id="S7.T7.1.1.3.1.4" class="ltx_td ltx_align_center ltx_border_t">85.6</td>
<td id="S7.T7.1.1.3.1.5" class="ltx_td ltx_align_center ltx_border_t">70.9</td>
</tr>
<tr id="S7.T7.1.1.4.2" class="ltx_tr">
<th id="S7.T7.1.1.4.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">+ RoBERTa sel.</th>
<td id="S7.T7.1.1.4.2.2" class="ltx_td ltx_align_center"><span id="S7.T7.1.1.4.2.2.1" class="ltx_text ltx_font_bold">92.7</span></td>
<td id="S7.T7.1.1.4.2.3" class="ltx_td ltx_align_center">79.5</td>
<td id="S7.T7.1.1.4.2.4" class="ltx_td ltx_align_center">83.4</td>
<td id="S7.T7.1.1.4.2.5" class="ltx_td ltx_align_center">76.6</td>
</tr>
<tr id="S7.T7.1.1.5.3" class="ltx_tr">
<th id="S7.T7.1.1.5.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">+ SBERT sel.</th>
<td id="S7.T7.1.1.5.3.2" class="ltx_td ltx_align_center ltx_border_bb">92.6</td>
<td id="S7.T7.1.1.5.3.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T7.1.1.5.3.3.1" class="ltx_text ltx_font_bold">79.7</span></td>
<td id="S7.T7.1.1.5.3.4" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T7.1.1.5.3.4.1" class="ltx_text ltx_font_bold">87.5</span></td>
<td id="S7.T7.1.1.5.3.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="S7.T7.1.1.5.3.5.1" class="ltx_text ltx_font_bold">77.8</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table 7: </span>
Impact of demonstration sampling strategies.
Uniform sampling randomly samples demonstrations, while selective (sel.) sampling only takes top sentences measured by the sentence encoders (§<a href="#S6" title="6 Fine-tuning with Demonstrations ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>).
</figcaption>
</figure>
<div id="S7.SS3.p1" class="ltx_para">
<p id="S7.SS3.p1.1" class="ltx_p">Table <a href="#S7.T7" title="Table 7 ‣ 7.3 Analysis of demonstration sampling ‣ 7 Experiments ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> compares the performance of demonstrations using uniform sampling to selective sampling by SBERT.
We acknowledge that SBERT is trained on SNLI and MNLI datasets, thus we also tried a simple sentence encoder using mean pooling of hidden representations from RoBERTa-large.
We find that in either case, using selective sampling outperforms uniform sampling, highlighting the importance of sampling similar examples for incorporating demonstrations in context.
</p>
</div>
</section>
<section id="S7.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">7.4 </span>Sample efficiency</h3>

<div id="S7.SS4.p1" class="ltx_para">
<p id="S7.SS4.p1.3" class="ltx_p">Figure <a href="#S8.F3" title="Figure 3 ‣ 8 Discussion ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> illustrates
how standard fine-tuning and our LM-BFF compare as <math id="S7.SS4.p1.1.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S7.SS4.p1.1.m1.1a"><mi id="S7.SS4.p1.1.m1.1.1" xref="S7.SS4.p1.1.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S7.SS4.p1.1.m1.1b"><ci id="S7.SS4.p1.1.m1.1.1.cmml" xref="S7.SS4.p1.1.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.SS4.p1.1.m1.1c">K</annotation></semantics></math> increases.
For a simple task such as SST-2 (also see MR, CR and MPQA in Table <a href="#S6.T3" title="Table 3 ‣ 6 Fine-tuning with Demonstrations ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>), despite using only 32 total examples, LM-BFF has already nearly saturated its performance and is comparable to standard fine-tuning over the entire dataset. On the harder task of SNLI, LM-BFF continues to improve as <math id="S7.SS4.p1.2.m2.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S7.SS4.p1.2.m2.1a"><mi id="S7.SS4.p1.2.m2.1.1" xref="S7.SS4.p1.2.m2.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S7.SS4.p1.2.m2.1b"><ci id="S7.SS4.p1.2.m2.1.1.cmml" xref="S7.SS4.p1.2.m2.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S7.SS4.p1.2.m2.1c">K</annotation></semantics></math> increases while still maintaining a performance gap over standard fine-tuning, until the two converge around <math id="S7.SS4.p1.3.m3.1" class="ltx_Math" alttext="K=256" display="inline"><semantics id="S7.SS4.p1.3.m3.1a"><mrow id="S7.SS4.p1.3.m3.1.1" xref="S7.SS4.p1.3.m3.1.1.cmml"><mi id="S7.SS4.p1.3.m3.1.1.2" xref="S7.SS4.p1.3.m3.1.1.2.cmml">K</mi><mo id="S7.SS4.p1.3.m3.1.1.1" xref="S7.SS4.p1.3.m3.1.1.1.cmml">=</mo><mn id="S7.SS4.p1.3.m3.1.1.3" xref="S7.SS4.p1.3.m3.1.1.3.cmml">256</mn></mrow><annotation-xml encoding="MathML-Content" id="S7.SS4.p1.3.m3.1b"><apply id="S7.SS4.p1.3.m3.1.1.cmml" xref="S7.SS4.p1.3.m3.1.1"><eq id="S7.SS4.p1.3.m3.1.1.1.cmml" xref="S7.SS4.p1.3.m3.1.1.1"></eq><ci id="S7.SS4.p1.3.m3.1.1.2.cmml" xref="S7.SS4.p1.3.m3.1.1.2">𝐾</ci><cn type="integer" id="S7.SS4.p1.3.m3.1.1.3.cmml" xref="S7.SS4.p1.3.m3.1.1.3">256</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S7.SS4.p1.3.m3.1c">K=256</annotation></semantics></math>.</p>
</div>
</section>
</section>
<section id="S8" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">8 </span>Discussion</h2>

<div id="S8.p1" class="ltx_para">
<p id="S8.p1.1" class="ltx_p">Reformulating NLP tasks as MLM has exciting implications for few-shot learning, but also has limitations.
First, while LM-BFF greatly outperforms standard fine-tuning, Table <a href="#S6.T3" title="Table 3 ‣ 6 Fine-tuning with Demonstrations ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> shows that, overall, the performance still substantially lags behind
fine-tuning with thousands of examples, especially for harder tasks.
Additionally, just like standard fine-tuning, our results also suffer from high variance.
As described in §<a href="#S2" title="2 Related Work ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>, several recent studies have tried to counter instability in few-shot fine-tuning
and we expect these methods to also help here.</p>
</div>
<div id="S8.p2" class="ltx_para">
<p id="S8.p2.1" class="ltx_p">With respect to automatic prompt generation, despite its effectiveness,
we still find it practically challenging to expand the search space,
or generalize well based on only approximately 32 examples.
This is partly due to our lingering reliance on <em id="S8.p2.1.1" class="ltx_emph ltx_font_italic">some</em> manual design—either manual templates (for label word search) or manual label words (for template search),
which allows us to get our search off the ground, but does also bias it towards areas of the search space that we might have already imagined.</p>
</div>
<div id="S8.p3" class="ltx_para">
<p id="S8.p3.1" class="ltx_p">Finally, it is important to clarify that LM-BFF favors certain tasks which (1) can be naturally posed as a “fill-in-the-blank” problem;
(2) have relatively short input sequences; and
(3) do not contain many output classes.
Issues (2) and (3) might be ameliorated with longer-context language models <cite class="ltx_cite ltx_citemacro_cite">(e.g., Beltagy et al., <a href="#bib.bib5" title="" class="ltx_ref">2020</a>)</cite>.
For tasks that are not straightforward to formulate in prompting, such as structured prediction, issue (1) is more fundamental.
We leave it as an open question for future work.</p>
</div>
<figure id="S8.F3" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2012.15723/assets/x3.png" id="S8.F3.g1" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_square" width="82" height="84" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2012.15723/assets/x4.png" id="S8.F3.g2" class="ltx_graphics ltx_centering ltx_figure_panel ltx_img_square" width="82" height="84" alt="Refer to caption"></div>
</div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Standard fine-tuning vs our LM-BFF as a function of <math id="S8.F3.3.m1.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S8.F3.3.m1.1b"><mi id="S8.F3.3.m1.1.1" xref="S8.F3.3.m1.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S8.F3.3.m1.1c"><ci id="S8.F3.3.m1.1.1.cmml" xref="S8.F3.3.m1.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S8.F3.3.m1.1d">K</annotation></semantics></math> (# instances per class). For lower <math id="S8.F3.4.m2.1" class="ltx_Math" alttext="K" display="inline"><semantics id="S8.F3.4.m2.1b"><mi id="S8.F3.4.m2.1.1" xref="S8.F3.4.m2.1.1.cmml">K</mi><annotation-xml encoding="MathML-Content" id="S8.F3.4.m2.1c"><ci id="S8.F3.4.m2.1.1.cmml" xref="S8.F3.4.m2.1.1">𝐾</ci></annotation-xml><annotation encoding="application/x-tex" id="S8.F3.4.m2.1d">K</annotation></semantics></math>, our method consistently outperforms standard fine-tuning.</figcaption>
</figure>
</section>
<section id="S9" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">9 </span>Conclusion</h2>

<div id="S9.p1" class="ltx_para">
<p id="S9.p1.2" class="ltx_p">In this paper we presented LM-BFF, a set of simple but effective techniques for fine-tuning language models using only a few examples. Our approach proposes to
(1) use prompt-based fine-tuning with automatically searched prompts;
and (2) include selected task demonstrations (training examples) as part of the input context.
We show that our method outperforms vanilla fine-tuning by up to <math id="S9.p1.1.m1.1" class="ltx_Math" alttext="30\%" display="inline"><semantics id="S9.p1.1.m1.1a"><mrow id="S9.p1.1.m1.1.1" xref="S9.p1.1.m1.1.1.cmml"><mn id="S9.p1.1.m1.1.1.2" xref="S9.p1.1.m1.1.1.2.cmml">30</mn><mo id="S9.p1.1.m1.1.1.1" xref="S9.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S9.p1.1.m1.1b"><apply id="S9.p1.1.m1.1.1.cmml" xref="S9.p1.1.m1.1.1"><csymbol cd="latexml" id="S9.p1.1.m1.1.1.1.cmml" xref="S9.p1.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S9.p1.1.m1.1.1.2.cmml" xref="S9.p1.1.m1.1.1.2">30</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S9.p1.1.m1.1c">30\%</annotation></semantics></math> (and <math id="S9.p1.2.m2.1" class="ltx_Math" alttext="11" display="inline"><semantics id="S9.p1.2.m2.1a"><mn id="S9.p1.2.m2.1.1" xref="S9.p1.2.m2.1.1.cmml">11</mn><annotation-xml encoding="MathML-Content" id="S9.p1.2.m2.1b"><cn type="integer" id="S9.p1.2.m2.1.1.cmml" xref="S9.p1.2.m2.1.1">11</cn></annotation-xml><annotation encoding="application/x-tex" id="S9.p1.2.m2.1c">11</annotation></semantics></math>% on average).
We concluded by discussing the limitations of our approach, and posed open questions for future study.</p>
</div>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Acknowledgements</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p">We thank the members of Princeton, MIT, Tsinghua NLP groups and the anonymous reviewers for their valuable feedback. TG is supported by a Graduate Fellowship at Princeton University and AF is supported by an NSF Graduate Research Fellowship. This research is also partly supported by a Google Research Scholar Award.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bansal et al. (2020a)</span>
<span class="ltx_bibblock">
Trapit Bansal, Rishikesh Jha, and Andrew McCallum. 2020a.

</span>
<span class="ltx_bibblock">Learning to few-shot learn across diverse natural language
classification tasks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">International Conference on Computational Linguistics
(COLING)</em>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bansal et al. (2020b)</span>
<span class="ltx_bibblock">
Trapit Bansal, Rishikesh Jha, Tsendsuren Munkhdalai, and Andrew McCallum.
2020b.

</span>
<span class="ltx_bibblock">Self-supervised meta-learning for few-shot natural language
classification tasks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bao et al. (2020)</span>
<span class="ltx_bibblock">
Yujia Bao, Menghua Wu, Shiyu Chang, and Regina Barzilay. 2020.

</span>
<span class="ltx_bibblock">Few-shot text classification with distributional signatures.

</span>
<span class="ltx_bibblock">In <em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations
(ICLR)</em>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bar Haim et al. (2006)</span>
<span class="ltx_bibblock">
Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo
Magnini, and Idan Szpektor. 2006.

</span>
<span class="ltx_bibblock">The second PASCAL recognising textual entailment challenge.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Beltagy et al. (2020)</span>
<span class="ltx_bibblock">
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.

</span>
<span class="ltx_bibblock">Longformer: The long-document Transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">arXiv:2004.05150</em>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bentivogli et al. (2009)</span>
<span class="ltx_bibblock">
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009.

</span>
<span class="ltx_bibblock">The fifth PASCAL recognizing textual entailment challenge.

</span>
<span class="ltx_bibblock">In <em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">TAC</em>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bowman et al. (2015)</span>
<span class="ltx_bibblock">
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning.
2015.

</span>
<span class="ltx_bibblock">A large annotated corpus for learning natural language inference.

</span>
<span class="ltx_bibblock">In <em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span>
<span class="ltx_bibblock">
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla
Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
et al. 2020.

</span>
<span class="ltx_bibblock">Language models are few-shot learners.

</span>
<span class="ltx_bibblock">In <em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems
(NeurIPS)</em>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cer et al. (2017)</span>
<span class="ltx_bibblock">
Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia.
2017.

</span>
<span class="ltx_bibblock">SemEval-2017 task 1: Semantic textual similarity multilingual and
crosslingual focused evaluation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">the 11th International Workshop on Semantic Evaluation
(SemEval-2017)</em>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al. (2020)</span>
<span class="ltx_bibblock">
Jiaao Chen, Zichao Yang, and Diyi Yang. 2020.

</span>
<span class="ltx_bibblock">MixText: Linguistically-informed interpolation of hidden space for
semi-supervised text classification.

</span>
<span class="ltx_bibblock">In <em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">Association for Computational Linguistics (ACL)</em>.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dagan et al. (2005)</span>
<span class="ltx_bibblock">
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005.

</span>
<span class="ltx_bibblock">The PASCAL recognising textual entailment challenge.

</span>
<span class="ltx_bibblock">In <em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">the First International Conference on Machine Learning
Challenges: Evaluating Predictive Uncertainty Visual Object Classification,
and Recognizing Textual Entailment</em>.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Davison et al. (2019)</span>
<span class="ltx_bibblock">
Joe Davison, Joshua Feldman, and Alexander M Rush. 2019.

</span>
<span class="ltx_bibblock">Commonsense knowledge mining from pretrained models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Devlin et al. (2019)</span>
<span class="ltx_bibblock">
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.

</span>
<span class="ltx_bibblock">BERT: Pre-training of deep bidirectional Transformers for
language understanding.

</span>
<span class="ltx_bibblock">In <em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">North American Chapter of the Association for Computational
Linguistics (NAACL)</em>.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dodge et al. (2020)</span>
<span class="ltx_bibblock">
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi,
and Noah Smith. 2020.

</span>
<span class="ltx_bibblock">Fine-tuning pretrained language models: Weight initializations, data
orders, and early stopping.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2002.06305</em>.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Dolan and Brockett (2005)</span>
<span class="ltx_bibblock">
William B. Dolan and Chris Brockett. 2005.

</span>
<span class="ltx_bibblock">Automatically constructing a corpus of sentential paraphrases.

</span>
<span class="ltx_bibblock">In <em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">the Third International Workshop on Paraphrasing
(IWP2005)</em>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Giampiccolo et al. (2007)</span>
<span class="ltx_bibblock">
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007.

</span>
<span class="ltx_bibblock">The third PASCAL recognizing textual entailment challenge.

</span>
<span class="ltx_bibblock">In <em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">the ACL-PASCAL Workshop on Textual Entailment and
Paraphrasing</em>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Han et al. (2018)</span>
<span class="ltx_bibblock">
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong
Sun. 2018.

</span>
<span class="ltx_bibblock">Fewrel: A large-scale supervised few-shot relation classification
dataset with state-of-the-art evaluation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Howard and Ruder (2018)</span>
<span class="ltx_bibblock">
Jeremy Howard and Sebastian Ruder. 2018.

</span>
<span class="ltx_bibblock">Universal language model fine-tuning for text classification.

</span>
<span class="ltx_bibblock">In <em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">Association for Computational Linguistics (ACL)</em>.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hu and Liu (2004)</span>
<span class="ltx_bibblock">
Minqing Hu and Bing Liu. 2004.

</span>
<span class="ltx_bibblock">Mining and summarizing customer reviews.

</span>
<span class="ltx_bibblock">In <em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">ACM SIGKDD international conference on Knowledge discovery
and data mining</em>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jiang et al. (2020)</span>
<span class="ltx_bibblock">
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020.

</span>
<span class="ltx_bibblock">How can we know what language models know?

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association of Computational Linguistics
(TACL)</em>.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lee et al. (2020)</span>
<span class="ltx_bibblock">
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. 2020.

</span>
<span class="ltx_bibblock">Mixout: Effective regularization to finetune large-scale pretrained
language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations
(ICLR)</em>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2019)</span>
<span class="ltx_bibblock">
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.

</span>
<span class="ltx_bibblock">RoBERTa: A robustly optimized BERT pretraining approach.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1907.11692</em>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mettes et al. (2019)</span>
<span class="ltx_bibblock">
Pascal Mettes, Elise van der Pol, and Cees Snoek. 2019.

</span>
<span class="ltx_bibblock">Hyperspherical prototype networks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems
(NeurIPS)</em>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Miyato et al. (2017)</span>
<span class="ltx_bibblock">
Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2017.

</span>
<span class="ltx_bibblock">Adversarial training methods for semi-supervised text classification.

</span>
<span class="ltx_bibblock">In <em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations
(ICLR)</em>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pang and Lee (2004)</span>
<span class="ltx_bibblock">
Bo Pang and Lillian Lee. 2004.

</span>
<span class="ltx_bibblock">A sentimental education: Sentiment analysis using subjectivity
summarization based on minimum cuts.

</span>
<span class="ltx_bibblock">In <em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">Association for Computational Linguistics (ACL)</em>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Pang and Lee (2005)</span>
<span class="ltx_bibblock">
Bo Pang and Lillian Lee. 2005.

</span>
<span class="ltx_bibblock">Seeing stars: Exploiting class relationships for sentiment
categorization with respect to rating scales.

</span>
<span class="ltx_bibblock">In <em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">Association for Computational Linguistics (ACL)</em>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Petroni et al. (2019)</span>
<span class="ltx_bibblock">
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton
Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.

</span>
<span class="ltx_bibblock">Language models as knowledge bases?

</span>
<span class="ltx_bibblock">In <em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Phang et al. (2018)</span>
<span class="ltx_bibblock">
Jason Phang, Thibault Févry, and Samuel R Bowman. 2018.

</span>
<span class="ltx_bibblock">Sentence encoders on STILTs: Supplementary training on intermediate
labeled-data tasks.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1811.01088</em>.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. (2018)</span>
<span class="ltx_bibblock">
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.

</span>
<span class="ltx_bibblock">Improving language understanding by generative pre-training.

</span>
<span class="ltx_bibblock">Technical report, OpenAI.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. (2019)</span>
<span class="ltx_bibblock">
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya
Sutskever. 2019.

</span>
<span class="ltx_bibblock">Language models are unsupervised multitask learners.

</span>
<span class="ltx_bibblock">Technical report, OpenAI.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raffel et al. (2020)</span>
<span class="ltx_bibblock">
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael
Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.

</span>
<span class="ltx_bibblock">Exploring the limits of transfer learning with a unified text-to-text
Transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">The Journal of Machine Learning Research (JMLR)</em>, 21(140).

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rajpurkar et al. (2016)</span>
<span class="ltx_bibblock">
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.

</span>
<span class="ltx_bibblock">SQuAD: 100,000+ questions for machine comprehension of text.

</span>
<span class="ltx_bibblock">In <em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Reimers and Gurevych (2019)</span>
<span class="ltx_bibblock">
Nils Reimers and Iryna Gurevych. 2019.

</span>
<span class="ltx_bibblock">Sentence-BERT: Sentence embeddings using Siamese BERT-networks.

</span>
<span class="ltx_bibblock">In <em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing and
International Joint Conference on Natural Language Processing
(EMNLP-IJCNLP)</em>.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schick et al. (2020)</span>
<span class="ltx_bibblock">
Timo Schick, Helmut Schmid, and Hinrich Schütze. 2020.

</span>
<span class="ltx_bibblock">Automatically identifying words that can serve as labels for few-shot
text classification.

</span>
<span class="ltx_bibblock">In <em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">International Conference on Computational Linguistics
(COLING)</em>.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schick and Schütze (2021a)</span>
<span class="ltx_bibblock">
Timo Schick and Hinrich Schütze. 2021a.

</span>
<span class="ltx_bibblock">Exploiting cloze questions for few-shot text classification and
natural language inference.

</span>
<span class="ltx_bibblock">In <em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">European Chapter of the Association for Computational
Linguistics (EACL)</em>.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Schick and Schütze (2021b)</span>
<span class="ltx_bibblock">
Timo Schick and Hinrich Schütze. 2021b.

</span>
<span class="ltx_bibblock">It’s not just size that matters: Small language models are also
few-shot learners.

</span>
<span class="ltx_bibblock">In <em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">North American Chapter of the Association for Computational
Linguistics (NAACL)</em>.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shin et al. (2020)</span>
<span class="ltx_bibblock">
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer
Singh. 2020.

</span>
<span class="ltx_bibblock">AutoPrompt: Automatic prompt construction for masked language
models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Socher et al. (2013)</span>
<span class="ltx_bibblock">
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning,
Andrew Ng, and Christopher Potts. 2013.

</span>
<span class="ltx_bibblock">Recursive deep models for semantic compositionality over a sentiment
treebank.

</span>
<span class="ltx_bibblock">In <em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Talmor et al. (2020)</span>
<span class="ltx_bibblock">
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020.

</span>
<span class="ltx_bibblock">oLMpics-on what language model pre-training captures.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association of Computational Linguistics
(TACL)</em>, 8.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Trinh and Le (2018)</span>
<span class="ltx_bibblock">
Trieu H Trinh and Quoc V Le. 2018.

</span>
<span class="ltx_bibblock">A simple method for commonsense reasoning.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1806.02847</em>.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Voorhees and Tice (2000)</span>
<span class="ltx_bibblock">
Ellen M Voorhees and Dawn M Tice. 2000.

</span>
<span class="ltx_bibblock">Building a question answering test collection.

</span>
<span class="ltx_bibblock">In <em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">the 23rd annual international ACM SIGIR conference on
Research and development in information retrieval</em>.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2019)</span>
<span class="ltx_bibblock">
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R
Bowman. 2019.

</span>
<span class="ltx_bibblock">GLUE: A multi-task benchmark and analysis platform for natural
language understanding.

</span>
<span class="ltx_bibblock">In <em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations
(ICLR)</em>.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Warstadt et al. (2019)</span>
<span class="ltx_bibblock">
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019.

</span>
<span class="ltx_bibblock">Neural network acceptability judgments.

</span>
<span class="ltx_bibblock"><em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association of Computational Linguistics
(TACL)</em>, 7.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wiebe et al. (2005)</span>
<span class="ltx_bibblock">
Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005.

</span>
<span class="ltx_bibblock">Annotating expressions of opinions and emotions in language.

</span>
<span class="ltx_bibblock"><em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">Language resources and evaluation</em>, 39(2-3).

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Williams et al. (2018)</span>
<span class="ltx_bibblock">
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.

</span>
<span class="ltx_bibblock">A broad-coverage challenge corpus for sentence understanding through
inference.

</span>
<span class="ltx_bibblock">In <em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">North American Chapter of the Association for Computational
Linguistics: Human Language Technologies (NAACL-HLT)</em>.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Xie et al. (2020)</span>
<span class="ltx_bibblock">
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020.

</span>
<span class="ltx_bibblock">Unsupervised data augmentation for consistency training.

</span>
<span class="ltx_bibblock"><em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">Advances in Neural Information Processing Systems (NeurIPS)</em>,
33.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yin et al. (2020)</span>
<span class="ltx_bibblock">
Wenpeng Yin, Nazneen Fatema Rajani, Dragomir Radev, Richard Socher, and Caiming
Xiong. 2020.

</span>
<span class="ltx_bibblock">Universal natural language processing with limited annotations: Try
few-shot textual entailment as a start.

</span>
<span class="ltx_bibblock">In <em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">Empirical Methods in Natural Language Processing (EMNLP)</em>.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yu et al. (2018)</span>
<span class="ltx_bibblock">
Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald
Tesauro, Haoyu Wang, and Bowen Zhou. 2018.

</span>
<span class="ltx_bibblock">Diverse few-shot text classification with multiple metrics.

</span>
<span class="ltx_bibblock">In <em id="bib.bib48.1.1" class="ltx_emph ltx_font_italic">North American Chapter of the Association for Computational
Linguistics (NAACL)</em>.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2021)</span>
<span class="ltx_bibblock">
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi.
2021.

</span>
<span class="ltx_bibblock">Revisiting few-sample BERT fine-tuning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">International Conference on Learning Representations
(ICLR)</em>.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhong et al. (2021)</span>
<span class="ltx_bibblock">
Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021.

</span>
<span class="ltx_bibblock">Factual probing is [MASK]: Learning vs. learning to recall.

</span>
<span class="ltx_bibblock">In <em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">North American Association for Computational Linguistics
(NAACL)</em>.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Impact of Development Sets</h2>

<div id="A1.p1" class="ltx_para">
<p id="A1.p1.2" class="ltx_p">Table <a href="#A1.T1" title="Table A.1 ‣ Appendix A Impact of Development Sets ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">A.1</span></a> shows how the size of the development sets can affect the final performance of the model.
For “No <math id="A1.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{D}_{\text{dev}}" display="inline"><semantics id="A1.p1.1.m1.1a"><msub id="A1.p1.1.m1.1.1" xref="A1.p1.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.p1.1.m1.1.1.2" xref="A1.p1.1.m1.1.1.2.cmml">𝒟</mi><mtext id="A1.p1.1.m1.1.1.3" xref="A1.p1.1.m1.1.1.3a.cmml">dev</mtext></msub><annotation-xml encoding="MathML-Content" id="A1.p1.1.m1.1b"><apply id="A1.p1.1.m1.1.1.cmml" xref="A1.p1.1.m1.1.1"><csymbol cd="ambiguous" id="A1.p1.1.m1.1.1.1.cmml" xref="A1.p1.1.m1.1.1">subscript</csymbol><ci id="A1.p1.1.m1.1.1.2.cmml" xref="A1.p1.1.m1.1.1.2">𝒟</ci><ci id="A1.p1.1.m1.1.1.3a.cmml" xref="A1.p1.1.m1.1.1.3"><mtext mathsize="70%" id="A1.p1.1.m1.1.1.3.cmml" xref="A1.p1.1.m1.1.1.3">dev</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.1.m1.1c">\mathcal{D}_{\text{dev}}</annotation></semantics></math>”, we take the same hyper-parameters from <cite class="ltx_cite ltx_citemacro_citet">Schick and Schütze (<a href="#bib.bib35" title="" class="ltx_ref">2021a</a>, <a href="#bib.bib36" title="" class="ltx_ref">b</a>)</cite>: batch size = 16, learning rate = 1e-5 and training steps = 250. We also experiment with a variant that we sample a development set of 10 times larger than the training set. We can see that using larger development sets leads to better performance, and this is why we stick to <math id="A1.p1.2.m2.2" class="ltx_Math" alttext="|\mathcal{D}_{\text{train}}|=|\mathcal{D}_{\text{dev}}|" display="inline"><semantics id="A1.p1.2.m2.2a"><mrow id="A1.p1.2.m2.2.2" xref="A1.p1.2.m2.2.2.cmml"><mrow id="A1.p1.2.m2.1.1.1.1" xref="A1.p1.2.m2.1.1.1.2.cmml"><mo stretchy="false" id="A1.p1.2.m2.1.1.1.1.2" xref="A1.p1.2.m2.1.1.1.2.1.cmml">|</mo><msub id="A1.p1.2.m2.1.1.1.1.1" xref="A1.p1.2.m2.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.p1.2.m2.1.1.1.1.1.2" xref="A1.p1.2.m2.1.1.1.1.1.2.cmml">𝒟</mi><mtext id="A1.p1.2.m2.1.1.1.1.1.3" xref="A1.p1.2.m2.1.1.1.1.1.3a.cmml">train</mtext></msub><mo stretchy="false" id="A1.p1.2.m2.1.1.1.1.3" xref="A1.p1.2.m2.1.1.1.2.1.cmml">|</mo></mrow><mo id="A1.p1.2.m2.2.2.3" xref="A1.p1.2.m2.2.2.3.cmml">=</mo><mrow id="A1.p1.2.m2.2.2.2.1" xref="A1.p1.2.m2.2.2.2.2.cmml"><mo stretchy="false" id="A1.p1.2.m2.2.2.2.1.2" xref="A1.p1.2.m2.2.2.2.2.1.cmml">|</mo><msub id="A1.p1.2.m2.2.2.2.1.1" xref="A1.p1.2.m2.2.2.2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.p1.2.m2.2.2.2.1.1.2" xref="A1.p1.2.m2.2.2.2.1.1.2.cmml">𝒟</mi><mtext id="A1.p1.2.m2.2.2.2.1.1.3" xref="A1.p1.2.m2.2.2.2.1.1.3a.cmml">dev</mtext></msub><mo stretchy="false" id="A1.p1.2.m2.2.2.2.1.3" xref="A1.p1.2.m2.2.2.2.2.1.cmml">|</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.2.m2.2b"><apply id="A1.p1.2.m2.2.2.cmml" xref="A1.p1.2.m2.2.2"><eq id="A1.p1.2.m2.2.2.3.cmml" xref="A1.p1.2.m2.2.2.3"></eq><apply id="A1.p1.2.m2.1.1.1.2.cmml" xref="A1.p1.2.m2.1.1.1.1"><abs id="A1.p1.2.m2.1.1.1.2.1.cmml" xref="A1.p1.2.m2.1.1.1.1.2"></abs><apply id="A1.p1.2.m2.1.1.1.1.1.cmml" xref="A1.p1.2.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.p1.2.m2.1.1.1.1.1.1.cmml" xref="A1.p1.2.m2.1.1.1.1.1">subscript</csymbol><ci id="A1.p1.2.m2.1.1.1.1.1.2.cmml" xref="A1.p1.2.m2.1.1.1.1.1.2">𝒟</ci><ci id="A1.p1.2.m2.1.1.1.1.1.3a.cmml" xref="A1.p1.2.m2.1.1.1.1.1.3"><mtext mathsize="70%" id="A1.p1.2.m2.1.1.1.1.1.3.cmml" xref="A1.p1.2.m2.1.1.1.1.1.3">train</mtext></ci></apply></apply><apply id="A1.p1.2.m2.2.2.2.2.cmml" xref="A1.p1.2.m2.2.2.2.1"><abs id="A1.p1.2.m2.2.2.2.2.1.cmml" xref="A1.p1.2.m2.2.2.2.1.2"></abs><apply id="A1.p1.2.m2.2.2.2.1.1.cmml" xref="A1.p1.2.m2.2.2.2.1.1"><csymbol cd="ambiguous" id="A1.p1.2.m2.2.2.2.1.1.1.cmml" xref="A1.p1.2.m2.2.2.2.1.1">subscript</csymbol><ci id="A1.p1.2.m2.2.2.2.1.1.2.cmml" xref="A1.p1.2.m2.2.2.2.1.1.2">𝒟</ci><ci id="A1.p1.2.m2.2.2.2.1.1.3a.cmml" xref="A1.p1.2.m2.2.2.2.1.1.3"><mtext mathsize="70%" id="A1.p1.2.m2.2.2.2.1.1.3.cmml" xref="A1.p1.2.m2.2.2.2.1.1.3">dev</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.2.m2.2c">|\mathcal{D}_{\text{train}}|=|\mathcal{D}_{\text{dev}}|</annotation></semantics></math> in our few-shot setting.</p>
</div>
<figure id="A1.T1" class="ltx_table">
<div id="A1.T1.6.6" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:249.3pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(91.6pt,-52.6pt) scale(1.73101562485723,1.73101562485723) ;">
<table id="A1.T1.6.6.6" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A1.T1.6.6.6.7.1" class="ltx_tr">
<th id="A1.T1.6.6.6.7.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt">Fine-tuning</th>
<th id="A1.T1.6.6.6.7.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T1.6.6.6.7.1.2.1" class="ltx_text ltx_font_bold">SST-2</span></th>
<th id="A1.T1.6.6.6.7.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T1.6.6.6.7.1.3.1" class="ltx_text ltx_font_bold">SNLI</span></th>
<th id="A1.T1.6.6.6.7.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T1.6.6.6.7.1.4.1" class="ltx_text ltx_font_bold">TREC</span></th>
<th id="A1.T1.6.6.6.7.1.5" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A1.T1.6.6.6.7.1.5.1" class="ltx_text ltx_font_bold">MRPC</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A1.T1.1.1.1.1" class="ltx_tr">
<th id="A1.T1.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">No <math id="A1.T1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="\mathcal{D}_{\text{dev}}" display="inline"><semantics id="A1.T1.1.1.1.1.1.m1.1a"><msub id="A1.T1.1.1.1.1.1.m1.1.1" xref="A1.T1.1.1.1.1.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.T1.1.1.1.1.1.m1.1.1.2" xref="A1.T1.1.1.1.1.1.m1.1.1.2.cmml">𝒟</mi><mtext id="A1.T1.1.1.1.1.1.m1.1.1.3" xref="A1.T1.1.1.1.1.1.m1.1.1.3a.cmml">dev</mtext></msub><annotation-xml encoding="MathML-Content" id="A1.T1.1.1.1.1.1.m1.1b"><apply id="A1.T1.1.1.1.1.1.m1.1.1.cmml" xref="A1.T1.1.1.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="A1.T1.1.1.1.1.1.m1.1.1.1.cmml" xref="A1.T1.1.1.1.1.1.m1.1.1">subscript</csymbol><ci id="A1.T1.1.1.1.1.1.m1.1.1.2.cmml" xref="A1.T1.1.1.1.1.1.m1.1.1.2">𝒟</ci><ci id="A1.T1.1.1.1.1.1.m1.1.1.3a.cmml" xref="A1.T1.1.1.1.1.1.m1.1.1.3"><mtext mathsize="70%" id="A1.T1.1.1.1.1.1.m1.1.1.3.cmml" xref="A1.T1.1.1.1.1.1.m1.1.1.3">dev</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.T1.1.1.1.1.1.m1.1c">\mathcal{D}_{\text{dev}}</annotation></semantics></math>
</th>
<td id="A1.T1.1.1.1.1.2" class="ltx_td ltx_align_center ltx_border_t">79.5</td>
<td id="A1.T1.1.1.1.1.3" class="ltx_td ltx_align_center ltx_border_t">49.2</td>
<td id="A1.T1.1.1.1.1.4" class="ltx_td ltx_align_center ltx_border_t">83.9</td>
<td id="A1.T1.1.1.1.1.5" class="ltx_td ltx_align_center ltx_border_t">77.8</td>
</tr>
<tr id="A1.T1.2.2.2.2" class="ltx_tr">
<th id="A1.T1.2.2.2.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A1.T1.2.2.2.2.1.m1.2" class="ltx_Math" alttext="|\mathcal{D}_{\text{dev}}|=|\mathcal{D}_{\text{train}}|" display="inline"><semantics id="A1.T1.2.2.2.2.1.m1.2a"><mrow id="A1.T1.2.2.2.2.1.m1.2.2" xref="A1.T1.2.2.2.2.1.m1.2.2.cmml"><mrow id="A1.T1.2.2.2.2.1.m1.1.1.1.1" xref="A1.T1.2.2.2.2.1.m1.1.1.1.2.cmml"><mo stretchy="false" id="A1.T1.2.2.2.2.1.m1.1.1.1.1.2" xref="A1.T1.2.2.2.2.1.m1.1.1.1.2.1.cmml">|</mo><msub id="A1.T1.2.2.2.2.1.m1.1.1.1.1.1" xref="A1.T1.2.2.2.2.1.m1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.T1.2.2.2.2.1.m1.1.1.1.1.1.2" xref="A1.T1.2.2.2.2.1.m1.1.1.1.1.1.2.cmml">𝒟</mi><mtext id="A1.T1.2.2.2.2.1.m1.1.1.1.1.1.3" xref="A1.T1.2.2.2.2.1.m1.1.1.1.1.1.3a.cmml">dev</mtext></msub><mo stretchy="false" id="A1.T1.2.2.2.2.1.m1.1.1.1.1.3" xref="A1.T1.2.2.2.2.1.m1.1.1.1.2.1.cmml">|</mo></mrow><mo id="A1.T1.2.2.2.2.1.m1.2.2.3" xref="A1.T1.2.2.2.2.1.m1.2.2.3.cmml">=</mo><mrow id="A1.T1.2.2.2.2.1.m1.2.2.2.1" xref="A1.T1.2.2.2.2.1.m1.2.2.2.2.cmml"><mo stretchy="false" id="A1.T1.2.2.2.2.1.m1.2.2.2.1.2" xref="A1.T1.2.2.2.2.1.m1.2.2.2.2.1.cmml">|</mo><msub id="A1.T1.2.2.2.2.1.m1.2.2.2.1.1" xref="A1.T1.2.2.2.2.1.m1.2.2.2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.T1.2.2.2.2.1.m1.2.2.2.1.1.2" xref="A1.T1.2.2.2.2.1.m1.2.2.2.1.1.2.cmml">𝒟</mi><mtext id="A1.T1.2.2.2.2.1.m1.2.2.2.1.1.3" xref="A1.T1.2.2.2.2.1.m1.2.2.2.1.1.3a.cmml">train</mtext></msub><mo stretchy="false" id="A1.T1.2.2.2.2.1.m1.2.2.2.1.3" xref="A1.T1.2.2.2.2.1.m1.2.2.2.2.1.cmml">|</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.T1.2.2.2.2.1.m1.2b"><apply id="A1.T1.2.2.2.2.1.m1.2.2.cmml" xref="A1.T1.2.2.2.2.1.m1.2.2"><eq id="A1.T1.2.2.2.2.1.m1.2.2.3.cmml" xref="A1.T1.2.2.2.2.1.m1.2.2.3"></eq><apply id="A1.T1.2.2.2.2.1.m1.1.1.1.2.cmml" xref="A1.T1.2.2.2.2.1.m1.1.1.1.1"><abs id="A1.T1.2.2.2.2.1.m1.1.1.1.2.1.cmml" xref="A1.T1.2.2.2.2.1.m1.1.1.1.1.2"></abs><apply id="A1.T1.2.2.2.2.1.m1.1.1.1.1.1.cmml" xref="A1.T1.2.2.2.2.1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.T1.2.2.2.2.1.m1.1.1.1.1.1.1.cmml" xref="A1.T1.2.2.2.2.1.m1.1.1.1.1.1">subscript</csymbol><ci id="A1.T1.2.2.2.2.1.m1.1.1.1.1.1.2.cmml" xref="A1.T1.2.2.2.2.1.m1.1.1.1.1.1.2">𝒟</ci><ci id="A1.T1.2.2.2.2.1.m1.1.1.1.1.1.3a.cmml" xref="A1.T1.2.2.2.2.1.m1.1.1.1.1.1.3"><mtext mathsize="70%" id="A1.T1.2.2.2.2.1.m1.1.1.1.1.1.3.cmml" xref="A1.T1.2.2.2.2.1.m1.1.1.1.1.1.3">dev</mtext></ci></apply></apply><apply id="A1.T1.2.2.2.2.1.m1.2.2.2.2.cmml" xref="A1.T1.2.2.2.2.1.m1.2.2.2.1"><abs id="A1.T1.2.2.2.2.1.m1.2.2.2.2.1.cmml" xref="A1.T1.2.2.2.2.1.m1.2.2.2.1.2"></abs><apply id="A1.T1.2.2.2.2.1.m1.2.2.2.1.1.cmml" xref="A1.T1.2.2.2.2.1.m1.2.2.2.1.1"><csymbol cd="ambiguous" id="A1.T1.2.2.2.2.1.m1.2.2.2.1.1.1.cmml" xref="A1.T1.2.2.2.2.1.m1.2.2.2.1.1">subscript</csymbol><ci id="A1.T1.2.2.2.2.1.m1.2.2.2.1.1.2.cmml" xref="A1.T1.2.2.2.2.1.m1.2.2.2.1.1.2">𝒟</ci><ci id="A1.T1.2.2.2.2.1.m1.2.2.2.1.1.3a.cmml" xref="A1.T1.2.2.2.2.1.m1.2.2.2.1.1.3"><mtext mathsize="70%" id="A1.T1.2.2.2.2.1.m1.2.2.2.1.1.3.cmml" xref="A1.T1.2.2.2.2.1.m1.2.2.2.1.1.3">train</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.T1.2.2.2.2.1.m1.2c">|\mathcal{D}_{\text{dev}}|=|\mathcal{D}_{\text{train}}|</annotation></semantics></math></th>
<td id="A1.T1.2.2.2.2.2" class="ltx_td ltx_align_center">81.4</td>
<td id="A1.T1.2.2.2.2.3" class="ltx_td ltx_align_center">48.4</td>
<td id="A1.T1.2.2.2.2.4" class="ltx_td ltx_align_center">88.8</td>
<td id="A1.T1.2.2.2.2.5" class="ltx_td ltx_align_center">76.6</td>
</tr>
<tr id="A1.T1.3.3.3.3" class="ltx_tr">
<th id="A1.T1.3.3.3.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A1.T1.3.3.3.3.1.m1.2" class="ltx_Math" alttext="|\mathcal{D}_{\text{dev}}|=10|\mathcal{D}_{\text{train}}|" display="inline"><semantics id="A1.T1.3.3.3.3.1.m1.2a"><mrow id="A1.T1.3.3.3.3.1.m1.2.2" xref="A1.T1.3.3.3.3.1.m1.2.2.cmml"><mrow id="A1.T1.3.3.3.3.1.m1.1.1.1.1" xref="A1.T1.3.3.3.3.1.m1.1.1.1.2.cmml"><mo stretchy="false" id="A1.T1.3.3.3.3.1.m1.1.1.1.1.2" xref="A1.T1.3.3.3.3.1.m1.1.1.1.2.1.cmml">|</mo><msub id="A1.T1.3.3.3.3.1.m1.1.1.1.1.1" xref="A1.T1.3.3.3.3.1.m1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.T1.3.3.3.3.1.m1.1.1.1.1.1.2" xref="A1.T1.3.3.3.3.1.m1.1.1.1.1.1.2.cmml">𝒟</mi><mtext id="A1.T1.3.3.3.3.1.m1.1.1.1.1.1.3" xref="A1.T1.3.3.3.3.1.m1.1.1.1.1.1.3a.cmml">dev</mtext></msub><mo stretchy="false" id="A1.T1.3.3.3.3.1.m1.1.1.1.1.3" xref="A1.T1.3.3.3.3.1.m1.1.1.1.2.1.cmml">|</mo></mrow><mo id="A1.T1.3.3.3.3.1.m1.2.2.3" xref="A1.T1.3.3.3.3.1.m1.2.2.3.cmml">=</mo><mrow id="A1.T1.3.3.3.3.1.m1.2.2.2" xref="A1.T1.3.3.3.3.1.m1.2.2.2.cmml"><mn id="A1.T1.3.3.3.3.1.m1.2.2.2.3" xref="A1.T1.3.3.3.3.1.m1.2.2.2.3.cmml">10</mn><mo lspace="0em" rspace="0em" id="A1.T1.3.3.3.3.1.m1.2.2.2.2" xref="A1.T1.3.3.3.3.1.m1.2.2.2.2.cmml">​</mo><mrow id="A1.T1.3.3.3.3.1.m1.2.2.2.1.1" xref="A1.T1.3.3.3.3.1.m1.2.2.2.1.2.cmml"><mo stretchy="false" id="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.2" xref="A1.T1.3.3.3.3.1.m1.2.2.2.1.2.1.cmml">|</mo><msub id="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.1" xref="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.1.2" xref="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.1.2.cmml">𝒟</mi><mtext id="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.1.3" xref="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.1.3a.cmml">train</mtext></msub><mo stretchy="false" id="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.3" xref="A1.T1.3.3.3.3.1.m1.2.2.2.1.2.1.cmml">|</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.T1.3.3.3.3.1.m1.2b"><apply id="A1.T1.3.3.3.3.1.m1.2.2.cmml" xref="A1.T1.3.3.3.3.1.m1.2.2"><eq id="A1.T1.3.3.3.3.1.m1.2.2.3.cmml" xref="A1.T1.3.3.3.3.1.m1.2.2.3"></eq><apply id="A1.T1.3.3.3.3.1.m1.1.1.1.2.cmml" xref="A1.T1.3.3.3.3.1.m1.1.1.1.1"><abs id="A1.T1.3.3.3.3.1.m1.1.1.1.2.1.cmml" xref="A1.T1.3.3.3.3.1.m1.1.1.1.1.2"></abs><apply id="A1.T1.3.3.3.3.1.m1.1.1.1.1.1.cmml" xref="A1.T1.3.3.3.3.1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.T1.3.3.3.3.1.m1.1.1.1.1.1.1.cmml" xref="A1.T1.3.3.3.3.1.m1.1.1.1.1.1">subscript</csymbol><ci id="A1.T1.3.3.3.3.1.m1.1.1.1.1.1.2.cmml" xref="A1.T1.3.3.3.3.1.m1.1.1.1.1.1.2">𝒟</ci><ci id="A1.T1.3.3.3.3.1.m1.1.1.1.1.1.3a.cmml" xref="A1.T1.3.3.3.3.1.m1.1.1.1.1.1.3"><mtext mathsize="70%" id="A1.T1.3.3.3.3.1.m1.1.1.1.1.1.3.cmml" xref="A1.T1.3.3.3.3.1.m1.1.1.1.1.1.3">dev</mtext></ci></apply></apply><apply id="A1.T1.3.3.3.3.1.m1.2.2.2.cmml" xref="A1.T1.3.3.3.3.1.m1.2.2.2"><times id="A1.T1.3.3.3.3.1.m1.2.2.2.2.cmml" xref="A1.T1.3.3.3.3.1.m1.2.2.2.2"></times><cn type="integer" id="A1.T1.3.3.3.3.1.m1.2.2.2.3.cmml" xref="A1.T1.3.3.3.3.1.m1.2.2.2.3">10</cn><apply id="A1.T1.3.3.3.3.1.m1.2.2.2.1.2.cmml" xref="A1.T1.3.3.3.3.1.m1.2.2.2.1.1"><abs id="A1.T1.3.3.3.3.1.m1.2.2.2.1.2.1.cmml" xref="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.2"></abs><apply id="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.1.cmml" xref="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.1"><csymbol cd="ambiguous" id="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.1.1.cmml" xref="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.1">subscript</csymbol><ci id="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.1.2.cmml" xref="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.1.2">𝒟</ci><ci id="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.1.3a.cmml" xref="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.1.3"><mtext mathsize="70%" id="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.1.3.cmml" xref="A1.T1.3.3.3.3.1.m1.2.2.2.1.1.1.3">train</mtext></ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.T1.3.3.3.3.1.m1.2c">|\mathcal{D}_{\text{dev}}|=10|\mathcal{D}_{\text{train}}|</annotation></semantics></math></th>
<td id="A1.T1.3.3.3.3.2" class="ltx_td ltx_align_center">83.5</td>
<td id="A1.T1.3.3.3.3.3" class="ltx_td ltx_align_center">52.0</td>
<td id="A1.T1.3.3.3.3.4" class="ltx_td ltx_align_center">89.4</td>
<td id="A1.T1.3.3.3.3.5" class="ltx_td ltx_align_center">79.6</td>
</tr>
<tr id="A1.T1.6.6.6.8.1" class="ltx_tr">
<th id="A1.T1.6.6.6.8.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Prompt-based FT</th>
<td id="A1.T1.6.6.6.8.1.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T1.6.6.6.8.1.2.1" class="ltx_text ltx_font_bold">SST-2</span></td>
<td id="A1.T1.6.6.6.8.1.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T1.6.6.6.8.1.3.1" class="ltx_text ltx_font_bold">SNLI</span></td>
<td id="A1.T1.6.6.6.8.1.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T1.6.6.6.8.1.4.1" class="ltx_text ltx_font_bold">TREC</span></td>
<td id="A1.T1.6.6.6.8.1.5" class="ltx_td ltx_align_center ltx_border_t"><span id="A1.T1.6.6.6.8.1.5.1" class="ltx_text ltx_font_bold">MRPC</span></td>
</tr>
<tr id="A1.T1.4.4.4.4" class="ltx_tr">
<th id="A1.T1.4.4.4.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">No <math id="A1.T1.4.4.4.4.1.m1.1" class="ltx_Math" alttext="\mathcal{D}_{\text{dev}}" display="inline"><semantics id="A1.T1.4.4.4.4.1.m1.1a"><msub id="A1.T1.4.4.4.4.1.m1.1.1" xref="A1.T1.4.4.4.4.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.T1.4.4.4.4.1.m1.1.1.2" xref="A1.T1.4.4.4.4.1.m1.1.1.2.cmml">𝒟</mi><mtext id="A1.T1.4.4.4.4.1.m1.1.1.3" xref="A1.T1.4.4.4.4.1.m1.1.1.3a.cmml">dev</mtext></msub><annotation-xml encoding="MathML-Content" id="A1.T1.4.4.4.4.1.m1.1b"><apply id="A1.T1.4.4.4.4.1.m1.1.1.cmml" xref="A1.T1.4.4.4.4.1.m1.1.1"><csymbol cd="ambiguous" id="A1.T1.4.4.4.4.1.m1.1.1.1.cmml" xref="A1.T1.4.4.4.4.1.m1.1.1">subscript</csymbol><ci id="A1.T1.4.4.4.4.1.m1.1.1.2.cmml" xref="A1.T1.4.4.4.4.1.m1.1.1.2">𝒟</ci><ci id="A1.T1.4.4.4.4.1.m1.1.1.3a.cmml" xref="A1.T1.4.4.4.4.1.m1.1.1.3"><mtext mathsize="70%" id="A1.T1.4.4.4.4.1.m1.1.1.3.cmml" xref="A1.T1.4.4.4.4.1.m1.1.1.3">dev</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.T1.4.4.4.4.1.m1.1c">\mathcal{D}_{\text{dev}}</annotation></semantics></math>
</th>
<td id="A1.T1.4.4.4.4.2" class="ltx_td ltx_align_center ltx_border_t">92.1</td>
<td id="A1.T1.4.4.4.4.3" class="ltx_td ltx_align_center ltx_border_t">75.3</td>
<td id="A1.T1.4.4.4.4.4" class="ltx_td ltx_align_center ltx_border_t">84.8</td>
<td id="A1.T1.4.4.4.4.5" class="ltx_td ltx_align_center ltx_border_t">70.2</td>
</tr>
<tr id="A1.T1.5.5.5.5" class="ltx_tr">
<th id="A1.T1.5.5.5.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><math id="A1.T1.5.5.5.5.1.m1.2" class="ltx_Math" alttext="|\mathcal{D}_{\text{dev}}|=|\mathcal{D}_{\text{train}}|" display="inline"><semantics id="A1.T1.5.5.5.5.1.m1.2a"><mrow id="A1.T1.5.5.5.5.1.m1.2.2" xref="A1.T1.5.5.5.5.1.m1.2.2.cmml"><mrow id="A1.T1.5.5.5.5.1.m1.1.1.1.1" xref="A1.T1.5.5.5.5.1.m1.1.1.1.2.cmml"><mo stretchy="false" id="A1.T1.5.5.5.5.1.m1.1.1.1.1.2" xref="A1.T1.5.5.5.5.1.m1.1.1.1.2.1.cmml">|</mo><msub id="A1.T1.5.5.5.5.1.m1.1.1.1.1.1" xref="A1.T1.5.5.5.5.1.m1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.T1.5.5.5.5.1.m1.1.1.1.1.1.2" xref="A1.T1.5.5.5.5.1.m1.1.1.1.1.1.2.cmml">𝒟</mi><mtext id="A1.T1.5.5.5.5.1.m1.1.1.1.1.1.3" xref="A1.T1.5.5.5.5.1.m1.1.1.1.1.1.3a.cmml">dev</mtext></msub><mo stretchy="false" id="A1.T1.5.5.5.5.1.m1.1.1.1.1.3" xref="A1.T1.5.5.5.5.1.m1.1.1.1.2.1.cmml">|</mo></mrow><mo id="A1.T1.5.5.5.5.1.m1.2.2.3" xref="A1.T1.5.5.5.5.1.m1.2.2.3.cmml">=</mo><mrow id="A1.T1.5.5.5.5.1.m1.2.2.2.1" xref="A1.T1.5.5.5.5.1.m1.2.2.2.2.cmml"><mo stretchy="false" id="A1.T1.5.5.5.5.1.m1.2.2.2.1.2" xref="A1.T1.5.5.5.5.1.m1.2.2.2.2.1.cmml">|</mo><msub id="A1.T1.5.5.5.5.1.m1.2.2.2.1.1" xref="A1.T1.5.5.5.5.1.m1.2.2.2.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.T1.5.5.5.5.1.m1.2.2.2.1.1.2" xref="A1.T1.5.5.5.5.1.m1.2.2.2.1.1.2.cmml">𝒟</mi><mtext id="A1.T1.5.5.5.5.1.m1.2.2.2.1.1.3" xref="A1.T1.5.5.5.5.1.m1.2.2.2.1.1.3a.cmml">train</mtext></msub><mo stretchy="false" id="A1.T1.5.5.5.5.1.m1.2.2.2.1.3" xref="A1.T1.5.5.5.5.1.m1.2.2.2.2.1.cmml">|</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.T1.5.5.5.5.1.m1.2b"><apply id="A1.T1.5.5.5.5.1.m1.2.2.cmml" xref="A1.T1.5.5.5.5.1.m1.2.2"><eq id="A1.T1.5.5.5.5.1.m1.2.2.3.cmml" xref="A1.T1.5.5.5.5.1.m1.2.2.3"></eq><apply id="A1.T1.5.5.5.5.1.m1.1.1.1.2.cmml" xref="A1.T1.5.5.5.5.1.m1.1.1.1.1"><abs id="A1.T1.5.5.5.5.1.m1.1.1.1.2.1.cmml" xref="A1.T1.5.5.5.5.1.m1.1.1.1.1.2"></abs><apply id="A1.T1.5.5.5.5.1.m1.1.1.1.1.1.cmml" xref="A1.T1.5.5.5.5.1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.T1.5.5.5.5.1.m1.1.1.1.1.1.1.cmml" xref="A1.T1.5.5.5.5.1.m1.1.1.1.1.1">subscript</csymbol><ci id="A1.T1.5.5.5.5.1.m1.1.1.1.1.1.2.cmml" xref="A1.T1.5.5.5.5.1.m1.1.1.1.1.1.2">𝒟</ci><ci id="A1.T1.5.5.5.5.1.m1.1.1.1.1.1.3a.cmml" xref="A1.T1.5.5.5.5.1.m1.1.1.1.1.1.3"><mtext mathsize="70%" id="A1.T1.5.5.5.5.1.m1.1.1.1.1.1.3.cmml" xref="A1.T1.5.5.5.5.1.m1.1.1.1.1.1.3">dev</mtext></ci></apply></apply><apply id="A1.T1.5.5.5.5.1.m1.2.2.2.2.cmml" xref="A1.T1.5.5.5.5.1.m1.2.2.2.1"><abs id="A1.T1.5.5.5.5.1.m1.2.2.2.2.1.cmml" xref="A1.T1.5.5.5.5.1.m1.2.2.2.1.2"></abs><apply id="A1.T1.5.5.5.5.1.m1.2.2.2.1.1.cmml" xref="A1.T1.5.5.5.5.1.m1.2.2.2.1.1"><csymbol cd="ambiguous" id="A1.T1.5.5.5.5.1.m1.2.2.2.1.1.1.cmml" xref="A1.T1.5.5.5.5.1.m1.2.2.2.1.1">subscript</csymbol><ci id="A1.T1.5.5.5.5.1.m1.2.2.2.1.1.2.cmml" xref="A1.T1.5.5.5.5.1.m1.2.2.2.1.1.2">𝒟</ci><ci id="A1.T1.5.5.5.5.1.m1.2.2.2.1.1.3a.cmml" xref="A1.T1.5.5.5.5.1.m1.2.2.2.1.1.3"><mtext mathsize="70%" id="A1.T1.5.5.5.5.1.m1.2.2.2.1.1.3.cmml" xref="A1.T1.5.5.5.5.1.m1.2.2.2.1.1.3">train</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.T1.5.5.5.5.1.m1.2c">|\mathcal{D}_{\text{dev}}|=|\mathcal{D}_{\text{train}}|</annotation></semantics></math></th>
<td id="A1.T1.5.5.5.5.2" class="ltx_td ltx_align_center">92.7</td>
<td id="A1.T1.5.5.5.5.3" class="ltx_td ltx_align_center">77.2</td>
<td id="A1.T1.5.5.5.5.4" class="ltx_td ltx_align_center">84.8</td>
<td id="A1.T1.5.5.5.5.5" class="ltx_td ltx_align_center">74.5</td>
</tr>
<tr id="A1.T1.6.6.6.6" class="ltx_tr">
<th id="A1.T1.6.6.6.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><math id="A1.T1.6.6.6.6.1.m1.2" class="ltx_Math" alttext="|\mathcal{D}_{\text{dev}}|=10|\mathcal{D}_{\text{train}}|" display="inline"><semantics id="A1.T1.6.6.6.6.1.m1.2a"><mrow id="A1.T1.6.6.6.6.1.m1.2.2" xref="A1.T1.6.6.6.6.1.m1.2.2.cmml"><mrow id="A1.T1.6.6.6.6.1.m1.1.1.1.1" xref="A1.T1.6.6.6.6.1.m1.1.1.1.2.cmml"><mo stretchy="false" id="A1.T1.6.6.6.6.1.m1.1.1.1.1.2" xref="A1.T1.6.6.6.6.1.m1.1.1.1.2.1.cmml">|</mo><msub id="A1.T1.6.6.6.6.1.m1.1.1.1.1.1" xref="A1.T1.6.6.6.6.1.m1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.T1.6.6.6.6.1.m1.1.1.1.1.1.2" xref="A1.T1.6.6.6.6.1.m1.1.1.1.1.1.2.cmml">𝒟</mi><mtext id="A1.T1.6.6.6.6.1.m1.1.1.1.1.1.3" xref="A1.T1.6.6.6.6.1.m1.1.1.1.1.1.3a.cmml">dev</mtext></msub><mo stretchy="false" id="A1.T1.6.6.6.6.1.m1.1.1.1.1.3" xref="A1.T1.6.6.6.6.1.m1.1.1.1.2.1.cmml">|</mo></mrow><mo id="A1.T1.6.6.6.6.1.m1.2.2.3" xref="A1.T1.6.6.6.6.1.m1.2.2.3.cmml">=</mo><mrow id="A1.T1.6.6.6.6.1.m1.2.2.2" xref="A1.T1.6.6.6.6.1.m1.2.2.2.cmml"><mn id="A1.T1.6.6.6.6.1.m1.2.2.2.3" xref="A1.T1.6.6.6.6.1.m1.2.2.2.3.cmml">10</mn><mo lspace="0em" rspace="0em" id="A1.T1.6.6.6.6.1.m1.2.2.2.2" xref="A1.T1.6.6.6.6.1.m1.2.2.2.2.cmml">​</mo><mrow id="A1.T1.6.6.6.6.1.m1.2.2.2.1.1" xref="A1.T1.6.6.6.6.1.m1.2.2.2.1.2.cmml"><mo stretchy="false" id="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.2" xref="A1.T1.6.6.6.6.1.m1.2.2.2.1.2.1.cmml">|</mo><msub id="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.1" xref="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.1.2" xref="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.1.2.cmml">𝒟</mi><mtext id="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.1.3" xref="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.1.3a.cmml">train</mtext></msub><mo stretchy="false" id="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.3" xref="A1.T1.6.6.6.6.1.m1.2.2.2.1.2.1.cmml">|</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A1.T1.6.6.6.6.1.m1.2b"><apply id="A1.T1.6.6.6.6.1.m1.2.2.cmml" xref="A1.T1.6.6.6.6.1.m1.2.2"><eq id="A1.T1.6.6.6.6.1.m1.2.2.3.cmml" xref="A1.T1.6.6.6.6.1.m1.2.2.3"></eq><apply id="A1.T1.6.6.6.6.1.m1.1.1.1.2.cmml" xref="A1.T1.6.6.6.6.1.m1.1.1.1.1"><abs id="A1.T1.6.6.6.6.1.m1.1.1.1.2.1.cmml" xref="A1.T1.6.6.6.6.1.m1.1.1.1.1.2"></abs><apply id="A1.T1.6.6.6.6.1.m1.1.1.1.1.1.cmml" xref="A1.T1.6.6.6.6.1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="A1.T1.6.6.6.6.1.m1.1.1.1.1.1.1.cmml" xref="A1.T1.6.6.6.6.1.m1.1.1.1.1.1">subscript</csymbol><ci id="A1.T1.6.6.6.6.1.m1.1.1.1.1.1.2.cmml" xref="A1.T1.6.6.6.6.1.m1.1.1.1.1.1.2">𝒟</ci><ci id="A1.T1.6.6.6.6.1.m1.1.1.1.1.1.3a.cmml" xref="A1.T1.6.6.6.6.1.m1.1.1.1.1.1.3"><mtext mathsize="70%" id="A1.T1.6.6.6.6.1.m1.1.1.1.1.1.3.cmml" xref="A1.T1.6.6.6.6.1.m1.1.1.1.1.1.3">dev</mtext></ci></apply></apply><apply id="A1.T1.6.6.6.6.1.m1.2.2.2.cmml" xref="A1.T1.6.6.6.6.1.m1.2.2.2"><times id="A1.T1.6.6.6.6.1.m1.2.2.2.2.cmml" xref="A1.T1.6.6.6.6.1.m1.2.2.2.2"></times><cn type="integer" id="A1.T1.6.6.6.6.1.m1.2.2.2.3.cmml" xref="A1.T1.6.6.6.6.1.m1.2.2.2.3">10</cn><apply id="A1.T1.6.6.6.6.1.m1.2.2.2.1.2.cmml" xref="A1.T1.6.6.6.6.1.m1.2.2.2.1.1"><abs id="A1.T1.6.6.6.6.1.m1.2.2.2.1.2.1.cmml" xref="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.2"></abs><apply id="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.1.cmml" xref="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.1"><csymbol cd="ambiguous" id="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.1.1.cmml" xref="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.1">subscript</csymbol><ci id="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.1.2.cmml" xref="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.1.2">𝒟</ci><ci id="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.1.3a.cmml" xref="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.1.3"><mtext mathsize="70%" id="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.1.3.cmml" xref="A1.T1.6.6.6.6.1.m1.2.2.2.1.1.1.3">train</mtext></ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.T1.6.6.6.6.1.m1.2c">|\mathcal{D}_{\text{dev}}|=10|\mathcal{D}_{\text{train}}|</annotation></semantics></math></th>
<td id="A1.T1.6.6.6.6.2" class="ltx_td ltx_align_center ltx_border_bb">93.0</td>
<td id="A1.T1.6.6.6.6.3" class="ltx_td ltx_align_center ltx_border_bb">79.7</td>
<td id="A1.T1.6.6.6.6.4" class="ltx_td ltx_align_center ltx_border_bb">89.3</td>
<td id="A1.T1.6.6.6.6.5" class="ltx_td ltx_align_center ltx_border_bb">80.9</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table A.1: </span>Impact of different sizes of development sets. Standard deviations are omitted here to save space. For No <math id="A1.T1.8.m1.1" class="ltx_Math" alttext="|\mathcal{D}_{\text{dev}}|" display="inline"><semantics id="A1.T1.8.m1.1b"><mrow id="A1.T1.8.m1.1.1.1" xref="A1.T1.8.m1.1.1.2.cmml"><mo stretchy="false" id="A1.T1.8.m1.1.1.1.2" xref="A1.T1.8.m1.1.1.2.1.cmml">|</mo><msub id="A1.T1.8.m1.1.1.1.1" xref="A1.T1.8.m1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A1.T1.8.m1.1.1.1.1.2" xref="A1.T1.8.m1.1.1.1.1.2.cmml">𝒟</mi><mtext id="A1.T1.8.m1.1.1.1.1.3" xref="A1.T1.8.m1.1.1.1.1.3a.cmml">dev</mtext></msub><mo stretchy="false" id="A1.T1.8.m1.1.1.1.3" xref="A1.T1.8.m1.1.1.2.1.cmml">|</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.T1.8.m1.1c"><apply id="A1.T1.8.m1.1.1.2.cmml" xref="A1.T1.8.m1.1.1.1"><abs id="A1.T1.8.m1.1.1.2.1.cmml" xref="A1.T1.8.m1.1.1.1.2"></abs><apply id="A1.T1.8.m1.1.1.1.1.cmml" xref="A1.T1.8.m1.1.1.1.1"><csymbol cd="ambiguous" id="A1.T1.8.m1.1.1.1.1.1.cmml" xref="A1.T1.8.m1.1.1.1.1">subscript</csymbol><ci id="A1.T1.8.m1.1.1.1.1.2.cmml" xref="A1.T1.8.m1.1.1.1.1.2">𝒟</ci><ci id="A1.T1.8.m1.1.1.1.1.3a.cmml" xref="A1.T1.8.m1.1.1.1.1.3"><mtext mathsize="70%" id="A1.T1.8.m1.1.1.1.1.3.cmml" xref="A1.T1.8.m1.1.1.1.1.3">dev</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.T1.8.m1.1d">|\mathcal{D}_{\text{dev}}|</annotation></semantics></math>, we use the same set of hyper-parameters as <cite class="ltx_cite ltx_citemacro_citet">Schick and Schütze (<a href="#bib.bib35" title="" class="ltx_ref">2021a</a>, <a href="#bib.bib36" title="" class="ltx_ref">b</a>)</cite>.</figcaption>
</figure>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Datasets</h2>

<div id="A2.p1" class="ltx_para">
<p id="A2.p1.1" class="ltx_p">For SNLI <cite class="ltx_cite ltx_citemacro_cite">Bowman et al. (<a href="#bib.bib7" title="" class="ltx_ref">2015</a>)</cite> and datasets from GLUE <cite class="ltx_cite ltx_citemacro_cite">Wang et al. (<a href="#bib.bib42" title="" class="ltx_ref">2019</a>)</cite>, including SST-2 <cite class="ltx_cite ltx_citemacro_cite">Socher et al. (<a href="#bib.bib38" title="" class="ltx_ref">2013</a>)</cite>, CoLA <cite class="ltx_cite ltx_citemacro_cite">Warstadt et al. (<a href="#bib.bib43" title="" class="ltx_ref">2019</a>)</cite>, MNLI <cite class="ltx_cite ltx_citemacro_cite">Williams et al. (<a href="#bib.bib45" title="" class="ltx_ref">2018</a>)</cite>, QNLI <cite class="ltx_cite ltx_citemacro_cite">Rajpurkar et al. (<a href="#bib.bib32" title="" class="ltx_ref">2016</a>)</cite>, RTE <cite class="ltx_cite ltx_citemacro_cite">Dagan et al. (<a href="#bib.bib11" title="" class="ltx_ref">2005</a>); Bar Haim et al. (<a href="#bib.bib4" title="" class="ltx_ref">2006</a>); Giampiccolo et al. (<a href="#bib.bib16" title="" class="ltx_ref">2007</a>); Bentivogli et al. (<a href="#bib.bib6" title="" class="ltx_ref">2009</a>)</cite>, MRPC <cite class="ltx_cite ltx_citemacro_cite">Dolan and Brockett (<a href="#bib.bib15" title="" class="ltx_ref">2005</a>)</cite>, QQP<span id="footnote12" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">12</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">12</sup><span class="ltx_tag ltx_tag_note">12</span><a target="_blank" href="https://www.quora.com/q/quoradata/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.quora.com/q/quoradata/</a></span></span></span> and STS-B <cite class="ltx_cite ltx_citemacro_cite">Cer et al. (<a href="#bib.bib9" title="" class="ltx_ref">2017</a>)</cite>, we follow <cite class="ltx_cite ltx_citemacro_citet">Zhang et al. (<a href="#bib.bib49" title="" class="ltx_ref">2021</a>)</cite> and use their original development sets for testing. For datasets which require a cross-validation evaluation—MR <cite class="ltx_cite ltx_citemacro_cite">Pang and Lee (<a href="#bib.bib26" title="" class="ltx_ref">2005</a>)</cite>, CR <cite class="ltx_cite ltx_citemacro_cite">Hu and Liu (<a href="#bib.bib19" title="" class="ltx_ref">2004</a>)</cite>, MPQA <cite class="ltx_cite ltx_citemacro_cite">Wiebe et al. (<a href="#bib.bib44" title="" class="ltx_ref">2005</a>)</cite>, Subj <cite class="ltx_cite ltx_citemacro_cite">Pang and Lee (<a href="#bib.bib25" title="" class="ltx_ref">2004</a>)</cite>—we simply randomly sample 2,000 examples as the testing set and leave them out from training. For SST-5 <cite class="ltx_cite ltx_citemacro_cite">Socher et al. (<a href="#bib.bib38" title="" class="ltx_ref">2013</a>)</cite> and TREC <cite class="ltx_cite ltx_citemacro_cite">Voorhees and Tice (<a href="#bib.bib41" title="" class="ltx_ref">2000</a>)</cite>, we use their official test sets. We show dataset statistics in Table <a href="#A2.T1" title="Table B.1 ‣ Appendix B Datasets ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">B.1</span></a>.</p>
</div>
<figure id="A2.T1" class="ltx_table">
<div id="A2.T1.3.3" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:858.6pt;height:467.7pt;vertical-align:-1.6pt;"><span class="ltx_transformed_inner" style="transform:translate(164.0pt,-89.0pt) scale(1.61823609765424,1.61823609765424) ;">
<table id="A2.T1.3.3.3" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="A2.T1.2.2.2.2" class="ltx_tr">
<th id="A2.T1.2.2.2.2.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="A2.T1.2.2.2.2.3.1" class="ltx_text ltx_font_bold">Category</span></th>
<th id="A2.T1.2.2.2.2.4" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="A2.T1.2.2.2.2.4.1" class="ltx_text ltx_font_bold">Dataset</span></th>
<th id="A2.T1.1.1.1.1.1" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><math id="A2.T1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="|\mathcal{Y}|" display="inline"><semantics id="A2.T1.1.1.1.1.1.m1.1a"><mrow id="A2.T1.1.1.1.1.1.m1.1.2.2" xref="A2.T1.1.1.1.1.1.m1.1.2.1.cmml"><mo stretchy="false" id="A2.T1.1.1.1.1.1.m1.1.2.2.1" xref="A2.T1.1.1.1.1.1.m1.1.2.1.1.cmml">|</mo><mi class="ltx_font_mathcaligraphic" id="A2.T1.1.1.1.1.1.m1.1.1" xref="A2.T1.1.1.1.1.1.m1.1.1.cmml">𝒴</mi><mo stretchy="false" id="A2.T1.1.1.1.1.1.m1.1.2.2.2" xref="A2.T1.1.1.1.1.1.m1.1.2.1.1.cmml">|</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.T1.1.1.1.1.1.m1.1b"><apply id="A2.T1.1.1.1.1.1.m1.1.2.1.cmml" xref="A2.T1.1.1.1.1.1.m1.1.2.2"><abs id="A2.T1.1.1.1.1.1.m1.1.2.1.1.cmml" xref="A2.T1.1.1.1.1.1.m1.1.2.2.1"></abs><ci id="A2.T1.1.1.1.1.1.m1.1.1.cmml" xref="A2.T1.1.1.1.1.1.m1.1.1">𝒴</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T1.1.1.1.1.1.m1.1c">|\mathcal{Y}|</annotation></semantics></math></th>
<th id="A2.T1.2.2.2.2.2" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt"><math id="A2.T1.2.2.2.2.2.m1.1" class="ltx_Math" alttext="L" display="inline"><semantics id="A2.T1.2.2.2.2.2.m1.1a"><mi id="A2.T1.2.2.2.2.2.m1.1.1" xref="A2.T1.2.2.2.2.2.m1.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="A2.T1.2.2.2.2.2.m1.1b"><ci id="A2.T1.2.2.2.2.2.m1.1.1.cmml" xref="A2.T1.2.2.2.2.2.m1.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.T1.2.2.2.2.2.m1.1c">L</annotation></semantics></math></th>
<th id="A2.T1.2.2.2.2.5" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">#Train</th>
<th id="A2.T1.2.2.2.2.6" class="ltx_td ltx_align_right ltx_th ltx_th_column ltx_border_tt">#Test</th>
<th id="A2.T1.2.2.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="A2.T1.2.2.2.2.7.1" class="ltx_text ltx_font_bold">Type</span></th>
<th id="A2.T1.2.2.2.2.8" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="A2.T1.2.2.2.2.8.1" class="ltx_text ltx_font_bold">Labels (classification tasks)</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="A2.T1.3.3.3.4.1" class="ltx_tr">
<td id="A2.T1.3.3.3.4.1.1" class="ltx_td ltx_border_tt"></td>
<td id="A2.T1.3.3.3.4.1.2" class="ltx_td ltx_align_left ltx_border_tt">SST-2</td>
<td id="A2.T1.3.3.3.4.1.3" class="ltx_td ltx_align_center ltx_border_tt">2</td>
<td id="A2.T1.3.3.3.4.1.4" class="ltx_td ltx_align_right ltx_border_tt">19</td>
<td id="A2.T1.3.3.3.4.1.5" class="ltx_td ltx_align_right ltx_border_tt">6,920</td>
<td id="A2.T1.3.3.3.4.1.6" class="ltx_td ltx_align_right ltx_border_tt">872</td>
<td id="A2.T1.3.3.3.4.1.7" class="ltx_td ltx_align_center ltx_border_tt">sentiment</td>
<td id="A2.T1.3.3.3.4.1.8" class="ltx_td ltx_align_left ltx_border_tt">positive, negative</td>
</tr>
<tr id="A2.T1.3.3.3.5.2" class="ltx_tr">
<td id="A2.T1.3.3.3.5.2.1" class="ltx_td"></td>
<td id="A2.T1.3.3.3.5.2.2" class="ltx_td ltx_align_left">SST-5</td>
<td id="A2.T1.3.3.3.5.2.3" class="ltx_td ltx_align_center">5</td>
<td id="A2.T1.3.3.3.5.2.4" class="ltx_td ltx_align_right">18</td>
<td id="A2.T1.3.3.3.5.2.5" class="ltx_td ltx_align_right">8,544</td>
<td id="A2.T1.3.3.3.5.2.6" class="ltx_td ltx_align_right">2,210</td>
<td id="A2.T1.3.3.3.5.2.7" class="ltx_td ltx_align_center">sentiment</td>
<td id="A2.T1.3.3.3.5.2.8" class="ltx_td ltx_align_left">v. pos., positive, neutral, negative, v. neg.</td>
</tr>
<tr id="A2.T1.3.3.3.6.3" class="ltx_tr">
<td id="A2.T1.3.3.3.6.3.1" class="ltx_td"></td>
<td id="A2.T1.3.3.3.6.3.2" class="ltx_td ltx_align_left">MR</td>
<td id="A2.T1.3.3.3.6.3.3" class="ltx_td ltx_align_center">2</td>
<td id="A2.T1.3.3.3.6.3.4" class="ltx_td ltx_align_right">20</td>
<td id="A2.T1.3.3.3.6.3.5" class="ltx_td ltx_align_right">8,662</td>
<td id="A2.T1.3.3.3.6.3.6" class="ltx_td ltx_align_right">2,000</td>
<td id="A2.T1.3.3.3.6.3.7" class="ltx_td ltx_align_center">sentiment</td>
<td id="A2.T1.3.3.3.6.3.8" class="ltx_td ltx_align_left">positive, negative</td>
</tr>
<tr id="A2.T1.3.3.3.7.4" class="ltx_tr">
<td id="A2.T1.3.3.3.7.4.1" class="ltx_td ltx_align_left">single-</td>
<td id="A2.T1.3.3.3.7.4.2" class="ltx_td ltx_align_left">CR</td>
<td id="A2.T1.3.3.3.7.4.3" class="ltx_td ltx_align_center">2</td>
<td id="A2.T1.3.3.3.7.4.4" class="ltx_td ltx_align_right">19</td>
<td id="A2.T1.3.3.3.7.4.5" class="ltx_td ltx_align_right">1,775</td>
<td id="A2.T1.3.3.3.7.4.6" class="ltx_td ltx_align_right">2,000</td>
<td id="A2.T1.3.3.3.7.4.7" class="ltx_td ltx_align_center">sentiment</td>
<td id="A2.T1.3.3.3.7.4.8" class="ltx_td ltx_align_left">positive, negative</td>
</tr>
<tr id="A2.T1.3.3.3.8.5" class="ltx_tr">
<td id="A2.T1.3.3.3.8.5.1" class="ltx_td ltx_align_left">sentence</td>
<td id="A2.T1.3.3.3.8.5.2" class="ltx_td ltx_align_left">MPQA</td>
<td id="A2.T1.3.3.3.8.5.3" class="ltx_td ltx_align_center">2</td>
<td id="A2.T1.3.3.3.8.5.4" class="ltx_td ltx_align_right">3</td>
<td id="A2.T1.3.3.3.8.5.5" class="ltx_td ltx_align_right">8,606</td>
<td id="A2.T1.3.3.3.8.5.6" class="ltx_td ltx_align_right">2,000</td>
<td id="A2.T1.3.3.3.8.5.7" class="ltx_td ltx_align_center">opinion polarity</td>
<td id="A2.T1.3.3.3.8.5.8" class="ltx_td ltx_align_left">positive, negative</td>
</tr>
<tr id="A2.T1.3.3.3.9.6" class="ltx_tr">
<td id="A2.T1.3.3.3.9.6.1" class="ltx_td"></td>
<td id="A2.T1.3.3.3.9.6.2" class="ltx_td ltx_align_left">Subj</td>
<td id="A2.T1.3.3.3.9.6.3" class="ltx_td ltx_align_center">2</td>
<td id="A2.T1.3.3.3.9.6.4" class="ltx_td ltx_align_right">23</td>
<td id="A2.T1.3.3.3.9.6.5" class="ltx_td ltx_align_right">8,000</td>
<td id="A2.T1.3.3.3.9.6.6" class="ltx_td ltx_align_right">2,000</td>
<td id="A2.T1.3.3.3.9.6.7" class="ltx_td ltx_align_center">subjectivity</td>
<td id="A2.T1.3.3.3.9.6.8" class="ltx_td ltx_align_left">subjective, objective</td>
</tr>
<tr id="A2.T1.3.3.3.10.7" class="ltx_tr">
<td id="A2.T1.3.3.3.10.7.1" class="ltx_td"></td>
<td id="A2.T1.3.3.3.10.7.2" class="ltx_td ltx_align_left">TREC</td>
<td id="A2.T1.3.3.3.10.7.3" class="ltx_td ltx_align_center">6</td>
<td id="A2.T1.3.3.3.10.7.4" class="ltx_td ltx_align_right">10</td>
<td id="A2.T1.3.3.3.10.7.5" class="ltx_td ltx_align_right">5,452</td>
<td id="A2.T1.3.3.3.10.7.6" class="ltx_td ltx_align_right">500</td>
<td id="A2.T1.3.3.3.10.7.7" class="ltx_td ltx_align_center">question cls.</td>
<td id="A2.T1.3.3.3.10.7.8" class="ltx_td ltx_align_left">abbr., entity, description, human, loc., num.</td>
</tr>
<tr id="A2.T1.3.3.3.11.8" class="ltx_tr">
<td id="A2.T1.3.3.3.11.8.1" class="ltx_td"></td>
<td id="A2.T1.3.3.3.11.8.2" class="ltx_td ltx_align_left">CoLA</td>
<td id="A2.T1.3.3.3.11.8.3" class="ltx_td ltx_align_center">2</td>
<td id="A2.T1.3.3.3.11.8.4" class="ltx_td ltx_align_right">8</td>
<td id="A2.T1.3.3.3.11.8.5" class="ltx_td ltx_align_right">8,551</td>
<td id="A2.T1.3.3.3.11.8.6" class="ltx_td ltx_align_right">1,042</td>
<td id="A2.T1.3.3.3.11.8.7" class="ltx_td ltx_align_center">acceptability</td>
<td id="A2.T1.3.3.3.11.8.8" class="ltx_td ltx_align_left">grammatical, not_grammatical</td>
</tr>
<tr id="A2.T1.3.3.3.12.9" class="ltx_tr">
<td id="A2.T1.3.3.3.12.9.1" class="ltx_td ltx_border_t"></td>
<td id="A2.T1.3.3.3.12.9.2" class="ltx_td ltx_align_left ltx_border_t">MNLI</td>
<td id="A2.T1.3.3.3.12.9.3" class="ltx_td ltx_align_center ltx_border_t">3</td>
<td id="A2.T1.3.3.3.12.9.4" class="ltx_td ltx_align_right ltx_border_t">22/11</td>
<td id="A2.T1.3.3.3.12.9.5" class="ltx_td ltx_align_right ltx_border_t">392,702</td>
<td id="A2.T1.3.3.3.12.9.6" class="ltx_td ltx_align_right ltx_border_t">9,815</td>
<td id="A2.T1.3.3.3.12.9.7" class="ltx_td ltx_align_center ltx_border_t">NLI</td>
<td id="A2.T1.3.3.3.12.9.8" class="ltx_td ltx_align_left ltx_border_t">entailment, neutral, contradiction</td>
</tr>
<tr id="A2.T1.3.3.3.13.10" class="ltx_tr">
<td id="A2.T1.3.3.3.13.10.1" class="ltx_td"></td>
<td id="A2.T1.3.3.3.13.10.2" class="ltx_td ltx_align_left">SNLI</td>
<td id="A2.T1.3.3.3.13.10.3" class="ltx_td ltx_align_center">3</td>
<td id="A2.T1.3.3.3.13.10.4" class="ltx_td ltx_align_right">14/8</td>
<td id="A2.T1.3.3.3.13.10.5" class="ltx_td ltx_align_right">549,367</td>
<td id="A2.T1.3.3.3.13.10.6" class="ltx_td ltx_align_right">9,842</td>
<td id="A2.T1.3.3.3.13.10.7" class="ltx_td ltx_align_center">NLI</td>
<td id="A2.T1.3.3.3.13.10.8" class="ltx_td ltx_align_left">entailment, neutral, contradiction</td>
</tr>
<tr id="A2.T1.3.3.3.14.11" class="ltx_tr">
<td id="A2.T1.3.3.3.14.11.1" class="ltx_td ltx_align_left">sentence-</td>
<td id="A2.T1.3.3.3.14.11.2" class="ltx_td ltx_align_left">QNLI</td>
<td id="A2.T1.3.3.3.14.11.3" class="ltx_td ltx_align_center">2</td>
<td id="A2.T1.3.3.3.14.11.4" class="ltx_td ltx_align_right">11/30</td>
<td id="A2.T1.3.3.3.14.11.5" class="ltx_td ltx_align_right">104,743</td>
<td id="A2.T1.3.3.3.14.11.6" class="ltx_td ltx_align_right">5,463</td>
<td id="A2.T1.3.3.3.14.11.7" class="ltx_td ltx_align_center">NLI</td>
<td id="A2.T1.3.3.3.14.11.8" class="ltx_td ltx_align_left">entailment, not_entailment</td>
</tr>
<tr id="A2.T1.3.3.3.15.12" class="ltx_tr">
<td id="A2.T1.3.3.3.15.12.1" class="ltx_td ltx_align_left">pair</td>
<td id="A2.T1.3.3.3.15.12.2" class="ltx_td ltx_align_left">RTE</td>
<td id="A2.T1.3.3.3.15.12.3" class="ltx_td ltx_align_center">2</td>
<td id="A2.T1.3.3.3.15.12.4" class="ltx_td ltx_align_right">49/10</td>
<td id="A2.T1.3.3.3.15.12.5" class="ltx_td ltx_align_right">2,490</td>
<td id="A2.T1.3.3.3.15.12.6" class="ltx_td ltx_align_right">277</td>
<td id="A2.T1.3.3.3.15.12.7" class="ltx_td ltx_align_center">NLI</td>
<td id="A2.T1.3.3.3.15.12.8" class="ltx_td ltx_align_left">entailment, not_entailment</td>
</tr>
<tr id="A2.T1.3.3.3.16.13" class="ltx_tr">
<td id="A2.T1.3.3.3.16.13.1" class="ltx_td"></td>
<td id="A2.T1.3.3.3.16.13.2" class="ltx_td ltx_align_left">MRPC</td>
<td id="A2.T1.3.3.3.16.13.3" class="ltx_td ltx_align_center">2</td>
<td id="A2.T1.3.3.3.16.13.4" class="ltx_td ltx_align_right">22/21</td>
<td id="A2.T1.3.3.3.16.13.5" class="ltx_td ltx_align_right">3,668</td>
<td id="A2.T1.3.3.3.16.13.6" class="ltx_td ltx_align_right">408</td>
<td id="A2.T1.3.3.3.16.13.7" class="ltx_td ltx_align_center">paraphrase</td>
<td id="A2.T1.3.3.3.16.13.8" class="ltx_td ltx_align_left">equivalent, not_equivalent</td>
</tr>
<tr id="A2.T1.3.3.3.17.14" class="ltx_tr">
<td id="A2.T1.3.3.3.17.14.1" class="ltx_td"></td>
<td id="A2.T1.3.3.3.17.14.2" class="ltx_td ltx_align_left">QQP</td>
<td id="A2.T1.3.3.3.17.14.3" class="ltx_td ltx_align_center">2</td>
<td id="A2.T1.3.3.3.17.14.4" class="ltx_td ltx_align_right">12/12</td>
<td id="A2.T1.3.3.3.17.14.5" class="ltx_td ltx_align_right">363,846</td>
<td id="A2.T1.3.3.3.17.14.6" class="ltx_td ltx_align_right">40,431</td>
<td id="A2.T1.3.3.3.17.14.7" class="ltx_td ltx_align_center">paraphrase</td>
<td id="A2.T1.3.3.3.17.14.8" class="ltx_td ltx_align_left">equivalent, not_equivalent</td>
</tr>
<tr id="A2.T1.3.3.3.3" class="ltx_tr">
<td id="A2.T1.3.3.3.3.2" class="ltx_td ltx_border_bb"></td>
<td id="A2.T1.3.3.3.3.3" class="ltx_td ltx_align_left ltx_border_bb">STS-B</td>
<td id="A2.T1.3.3.3.3.1" class="ltx_td ltx_align_center ltx_border_bb"><math id="A2.T1.3.3.3.3.1.m1.1" class="ltx_Math" alttext="\mathcal{R}" display="inline"><semantics id="A2.T1.3.3.3.3.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="A2.T1.3.3.3.3.1.m1.1.1" xref="A2.T1.3.3.3.3.1.m1.1.1.cmml">ℛ</mi><annotation-xml encoding="MathML-Content" id="A2.T1.3.3.3.3.1.m1.1b"><ci id="A2.T1.3.3.3.3.1.m1.1.1.cmml" xref="A2.T1.3.3.3.3.1.m1.1.1">ℛ</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.T1.3.3.3.3.1.m1.1c">\mathcal{R}</annotation></semantics></math></td>
<td id="A2.T1.3.3.3.3.4" class="ltx_td ltx_align_right ltx_border_bb">11/11</td>
<td id="A2.T1.3.3.3.3.5" class="ltx_td ltx_align_right ltx_border_bb">5,749</td>
<td id="A2.T1.3.3.3.3.6" class="ltx_td ltx_align_right ltx_border_bb">1,500</td>
<td id="A2.T1.3.3.3.3.7" class="ltx_td ltx_align_center ltx_border_bb">sent. similarity</td>
<td id="A2.T1.3.3.3.3.8" class="ltx_td ltx_align_left ltx_border_bb">-</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table B.1: </span>The datasets evaluated in this work. <math id="A2.T1.10.m1.1" class="ltx_Math" alttext="|\mathcal{Y}|" display="inline"><semantics id="A2.T1.10.m1.1b"><mrow id="A2.T1.10.m1.1.2.2" xref="A2.T1.10.m1.1.2.1.cmml"><mo stretchy="false" id="A2.T1.10.m1.1.2.2.1" xref="A2.T1.10.m1.1.2.1.1.cmml">|</mo><mi class="ltx_font_mathcaligraphic" id="A2.T1.10.m1.1.1" xref="A2.T1.10.m1.1.1.cmml">𝒴</mi><mo stretchy="false" id="A2.T1.10.m1.1.2.2.2" xref="A2.T1.10.m1.1.2.1.1.cmml">|</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.T1.10.m1.1c"><apply id="A2.T1.10.m1.1.2.1.cmml" xref="A2.T1.10.m1.1.2.2"><abs id="A2.T1.10.m1.1.2.1.1.cmml" xref="A2.T1.10.m1.1.2.2.1"></abs><ci id="A2.T1.10.m1.1.1.cmml" xref="A2.T1.10.m1.1.1">𝒴</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T1.10.m1.1d">|\mathcal{Y}|</annotation></semantics></math>: # of classes for classification tasks (with one exception: STS-B is a real-valued regression task over the interval <math id="A2.T1.11.m2.2" class="ltx_Math" alttext="[0,5]" display="inline"><semantics id="A2.T1.11.m2.2b"><mrow id="A2.T1.11.m2.2.3.2" xref="A2.T1.11.m2.2.3.1.cmml"><mo stretchy="false" id="A2.T1.11.m2.2.3.2.1" xref="A2.T1.11.m2.2.3.1.cmml">[</mo><mn id="A2.T1.11.m2.1.1" xref="A2.T1.11.m2.1.1.cmml">0</mn><mo id="A2.T1.11.m2.2.3.2.2" xref="A2.T1.11.m2.2.3.1.cmml">,</mo><mn id="A2.T1.11.m2.2.2" xref="A2.T1.11.m2.2.2.cmml">5</mn><mo stretchy="false" id="A2.T1.11.m2.2.3.2.3" xref="A2.T1.11.m2.2.3.1.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="A2.T1.11.m2.2c"><interval closure="closed" id="A2.T1.11.m2.2.3.1.cmml" xref="A2.T1.11.m2.2.3.2"><cn type="integer" id="A2.T1.11.m2.1.1.cmml" xref="A2.T1.11.m2.1.1">0</cn><cn type="integer" id="A2.T1.11.m2.2.2.cmml" xref="A2.T1.11.m2.2.2">5</cn></interval></annotation-xml><annotation encoding="application/x-tex" id="A2.T1.11.m2.2d">[0,5]</annotation></semantics></math>). <math id="A2.T1.12.m3.1" class="ltx_Math" alttext="L" display="inline"><semantics id="A2.T1.12.m3.1b"><mi id="A2.T1.12.m3.1.1" xref="A2.T1.12.m3.1.1.cmml">L</mi><annotation-xml encoding="MathML-Content" id="A2.T1.12.m3.1c"><ci id="A2.T1.12.m3.1.1.cmml" xref="A2.T1.12.m3.1.1">𝐿</ci></annotation-xml><annotation encoding="application/x-tex" id="A2.T1.12.m3.1d">L</annotation></semantics></math>: average # of words in input sentence(s). Note that we only sample <math id="A2.T1.13.m4.1" class="ltx_Math" alttext="\mathcal{D}_{\text{train}}" display="inline"><semantics id="A2.T1.13.m4.1b"><msub id="A2.T1.13.m4.1.1" xref="A2.T1.13.m4.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A2.T1.13.m4.1.1.2" xref="A2.T1.13.m4.1.1.2.cmml">𝒟</mi><mtext id="A2.T1.13.m4.1.1.3" xref="A2.T1.13.m4.1.1.3a.cmml">train</mtext></msub><annotation-xml encoding="MathML-Content" id="A2.T1.13.m4.1c"><apply id="A2.T1.13.m4.1.1.cmml" xref="A2.T1.13.m4.1.1"><csymbol cd="ambiguous" id="A2.T1.13.m4.1.1.1.cmml" xref="A2.T1.13.m4.1.1">subscript</csymbol><ci id="A2.T1.13.m4.1.1.2.cmml" xref="A2.T1.13.m4.1.1.2">𝒟</ci><ci id="A2.T1.13.m4.1.1.3a.cmml" xref="A2.T1.13.m4.1.1.3"><mtext mathsize="70%" id="A2.T1.13.m4.1.1.3.cmml" xref="A2.T1.13.m4.1.1.3">train</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T1.13.m4.1d">\mathcal{D}_{\text{train}}</annotation></semantics></math> and <math id="A2.T1.14.m5.1" class="ltx_Math" alttext="\mathcal{D}_{\text{dev}}" display="inline"><semantics id="A2.T1.14.m5.1b"><msub id="A2.T1.14.m5.1.1" xref="A2.T1.14.m5.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A2.T1.14.m5.1.1.2" xref="A2.T1.14.m5.1.1.2.cmml">𝒟</mi><mtext id="A2.T1.14.m5.1.1.3" xref="A2.T1.14.m5.1.1.3a.cmml">dev</mtext></msub><annotation-xml encoding="MathML-Content" id="A2.T1.14.m5.1c"><apply id="A2.T1.14.m5.1.1.cmml" xref="A2.T1.14.m5.1.1"><csymbol cd="ambiguous" id="A2.T1.14.m5.1.1.1.cmml" xref="A2.T1.14.m5.1.1">subscript</csymbol><ci id="A2.T1.14.m5.1.1.2.cmml" xref="A2.T1.14.m5.1.1.2">𝒟</ci><ci id="A2.T1.14.m5.1.1.3a.cmml" xref="A2.T1.14.m5.1.1.3"><mtext mathsize="70%" id="A2.T1.14.m5.1.1.3.cmml" xref="A2.T1.14.m5.1.1.3">dev</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T1.14.m5.1d">\mathcal{D}_{\text{dev}}</annotation></semantics></math> of <math id="A2.T1.15.m6.1" class="ltx_Math" alttext="K\times|\mathcal{Y}|" display="inline"><semantics id="A2.T1.15.m6.1b"><mrow id="A2.T1.15.m6.1.2" xref="A2.T1.15.m6.1.2.cmml"><mi id="A2.T1.15.m6.1.2.2" xref="A2.T1.15.m6.1.2.2.cmml">K</mi><mo lspace="0.222em" rspace="0.222em" id="A2.T1.15.m6.1.2.1" xref="A2.T1.15.m6.1.2.1.cmml">×</mo><mrow id="A2.T1.15.m6.1.2.3.2" xref="A2.T1.15.m6.1.2.3.1.cmml"><mo stretchy="false" id="A2.T1.15.m6.1.2.3.2.1" xref="A2.T1.15.m6.1.2.3.1.1.cmml">|</mo><mi class="ltx_font_mathcaligraphic" id="A2.T1.15.m6.1.1" xref="A2.T1.15.m6.1.1.cmml">𝒴</mi><mo stretchy="false" id="A2.T1.15.m6.1.2.3.2.2" xref="A2.T1.15.m6.1.2.3.1.1.cmml">|</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A2.T1.15.m6.1c"><apply id="A2.T1.15.m6.1.2.cmml" xref="A2.T1.15.m6.1.2"><times id="A2.T1.15.m6.1.2.1.cmml" xref="A2.T1.15.m6.1.2.1"></times><ci id="A2.T1.15.m6.1.2.2.cmml" xref="A2.T1.15.m6.1.2.2">𝐾</ci><apply id="A2.T1.15.m6.1.2.3.1.cmml" xref="A2.T1.15.m6.1.2.3.2"><abs id="A2.T1.15.m6.1.2.3.1.1.cmml" xref="A2.T1.15.m6.1.2.3.2.1"></abs><ci id="A2.T1.15.m6.1.1.cmml" xref="A2.T1.15.m6.1.1">𝒴</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A2.T1.15.m6.1d">K\times|\mathcal{Y}|</annotation></semantics></math> examples from the original training set in our few-shot experiments (§<a href="#S3" title="3 Problem Setup ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>).</figcaption>
</figure>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Experimental Details</h2>

<section id="A3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.1 </span>Hyper-parameter selection</h3>

<div id="A3.SS1.p1" class="ltx_para">
<p id="A3.SS1.p1.1" class="ltx_p">For grid search, we take learning rates from {1e-5, 2e-5, 5e-5} and batch sizes from {2, 4, 8}.
These numbers are picked by pilot experiments on the SST-2 and SNLI datasets.
We also use early stopping to avoid overfitting.
For each trial, we train the model for 1,000 steps, validate the performance every 100 steps, and take the best checkpoint.</p>
</div>
</section>
<section id="A3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.2 </span>Prompt-based fine-tuning</h3>

<div id="A3.SS2.p1" class="ltx_para">
<p id="A3.SS2.p1.6" class="ltx_p">Table <a href="#S2.T1" title="Table 1 ‣ Few-shot learning. ‣ 2 Related Work ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> shows all the manual templates and label words we use in experiment.
For automatically template generation, we take the T5-3B<span id="footnote13" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">13</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">13</sup><span class="ltx_tag ltx_tag_note">13</span>We take the T5 1.0 checkpoint, which is trained on both unsupervised and downstream task data. We compared it to T5 1.1 (without downstream task data) and did not find a significant difference in generated templates.</span></span></span> model, which is the largest publicly available one that can fit on a single GPU.
For automatically searching label words, we set <math id="A3.SS2.p1.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="A3.SS2.p1.1.m1.1a"><mi id="A3.SS2.p1.1.m1.1.1" xref="A3.SS2.p1.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="A3.SS2.p1.1.m1.1b"><ci id="A3.SS2.p1.1.m1.1.1.cmml" xref="A3.SS2.p1.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.SS2.p1.1.m1.1c">k</annotation></semantics></math> to 100 for all tasks except SST-5 and TREC. For SST-5 we set a smaller <math id="A3.SS2.p1.2.m2.1" class="ltx_Math" alttext="k=30" display="inline"><semantics id="A3.SS2.p1.2.m2.1a"><mrow id="A3.SS2.p1.2.m2.1.1" xref="A3.SS2.p1.2.m2.1.1.cmml"><mi id="A3.SS2.p1.2.m2.1.1.2" xref="A3.SS2.p1.2.m2.1.1.2.cmml">k</mi><mo id="A3.SS2.p1.2.m2.1.1.1" xref="A3.SS2.p1.2.m2.1.1.1.cmml">=</mo><mn id="A3.SS2.p1.2.m2.1.1.3" xref="A3.SS2.p1.2.m2.1.1.3.cmml">30</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.SS2.p1.2.m2.1b"><apply id="A3.SS2.p1.2.m2.1.1.cmml" xref="A3.SS2.p1.2.m2.1.1"><eq id="A3.SS2.p1.2.m2.1.1.1.cmml" xref="A3.SS2.p1.2.m2.1.1.1"></eq><ci id="A3.SS2.p1.2.m2.1.1.2.cmml" xref="A3.SS2.p1.2.m2.1.1.2">𝑘</ci><cn type="integer" id="A3.SS2.p1.2.m2.1.1.3.cmml" xref="A3.SS2.p1.2.m2.1.1.3">30</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS2.p1.2.m2.1c">k=30</annotation></semantics></math>, as it is a 5-way classification task. For TREC, we observe that filtering <math id="A3.SS2.p1.3.m3.1" class="ltx_Math" alttext="\mathcal{V}^{c}" display="inline"><semantics id="A3.SS2.p1.3.m3.1a"><msup id="A3.SS2.p1.3.m3.1.1" xref="A3.SS2.p1.3.m3.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A3.SS2.p1.3.m3.1.1.2" xref="A3.SS2.p1.3.m3.1.1.2.cmml">𝒱</mi><mi id="A3.SS2.p1.3.m3.1.1.3" xref="A3.SS2.p1.3.m3.1.1.3.cmml">c</mi></msup><annotation-xml encoding="MathML-Content" id="A3.SS2.p1.3.m3.1b"><apply id="A3.SS2.p1.3.m3.1.1.cmml" xref="A3.SS2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="A3.SS2.p1.3.m3.1.1.1.cmml" xref="A3.SS2.p1.3.m3.1.1">superscript</csymbol><ci id="A3.SS2.p1.3.m3.1.1.2.cmml" xref="A3.SS2.p1.3.m3.1.1.2">𝒱</ci><ci id="A3.SS2.p1.3.m3.1.1.3.cmml" xref="A3.SS2.p1.3.m3.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS2.p1.3.m3.1c">\mathcal{V}^{c}</annotation></semantics></math> using conditional likelihood alone is still noisy, thus we set <math id="A3.SS2.p1.4.m4.1" class="ltx_Math" alttext="k=1000" display="inline"><semantics id="A3.SS2.p1.4.m4.1a"><mrow id="A3.SS2.p1.4.m4.1.1" xref="A3.SS2.p1.4.m4.1.1.cmml"><mi id="A3.SS2.p1.4.m4.1.1.2" xref="A3.SS2.p1.4.m4.1.1.2.cmml">k</mi><mo id="A3.SS2.p1.4.m4.1.1.1" xref="A3.SS2.p1.4.m4.1.1.1.cmml">=</mo><mn id="A3.SS2.p1.4.m4.1.1.3" xref="A3.SS2.p1.4.m4.1.1.3.cmml">1000</mn></mrow><annotation-xml encoding="MathML-Content" id="A3.SS2.p1.4.m4.1b"><apply id="A3.SS2.p1.4.m4.1.1.cmml" xref="A3.SS2.p1.4.m4.1.1"><eq id="A3.SS2.p1.4.m4.1.1.1.cmml" xref="A3.SS2.p1.4.m4.1.1.1"></eq><ci id="A3.SS2.p1.4.m4.1.1.2.cmml" xref="A3.SS2.p1.4.m4.1.1.2">𝑘</ci><cn type="integer" id="A3.SS2.p1.4.m4.1.1.3.cmml" xref="A3.SS2.p1.4.m4.1.1.3">1000</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS2.p1.4.m4.1c">k=1000</annotation></semantics></math>, and then re-rank <math id="A3.SS2.p1.5.m5.1" class="ltx_Math" alttext="\mathcal{V}^{c}" display="inline"><semantics id="A3.SS2.p1.5.m5.1a"><msup id="A3.SS2.p1.5.m5.1.1" xref="A3.SS2.p1.5.m5.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A3.SS2.p1.5.m5.1.1.2" xref="A3.SS2.p1.5.m5.1.1.2.cmml">𝒱</mi><mi id="A3.SS2.p1.5.m5.1.1.3" xref="A3.SS2.p1.5.m5.1.1.3.cmml">c</mi></msup><annotation-xml encoding="MathML-Content" id="A3.SS2.p1.5.m5.1b"><apply id="A3.SS2.p1.5.m5.1.1.cmml" xref="A3.SS2.p1.5.m5.1.1"><csymbol cd="ambiguous" id="A3.SS2.p1.5.m5.1.1.1.cmml" xref="A3.SS2.p1.5.m5.1.1">superscript</csymbol><ci id="A3.SS2.p1.5.m5.1.1.2.cmml" xref="A3.SS2.p1.5.m5.1.1.2">𝒱</ci><ci id="A3.SS2.p1.5.m5.1.1.3.cmml" xref="A3.SS2.p1.5.m5.1.1.3">𝑐</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS2.p1.5.m5.1c">\mathcal{V}^{c}</annotation></semantics></math> by the nearest neighbors of the original manual label words and take the top 30 per class. We set <math id="A3.SS2.p1.6.m6.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A3.SS2.p1.6.m6.1a"><mi id="A3.SS2.p1.6.m6.1.1" xref="A3.SS2.p1.6.m6.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A3.SS2.p1.6.m6.1b"><ci id="A3.SS2.p1.6.m6.1.1.cmml" xref="A3.SS2.p1.6.m6.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A3.SS2.p1.6.m6.1c">n</annotation></semantics></math> to 100 in all experiments.
Due to the large number of trials in automatic search, we take a fixed set of hyper-parameters in this part: batch size of 8 and learning rate of 1e-5.</p>
</div>
<div id="A3.SS2.p2" class="ltx_para">
<p id="A3.SS2.p2.3" class="ltx_p">Since the idea of prompt-based fine-tuning is to make the input and output distribution close to the pre-training, the implementation details are crucial.
For templates, we put extra space before sentences if it is not at the beginning of the input.
Also, we lowercase the first letter of the sentence if it is concatenated with a prefix (e.g., <span id="A3.SS2.p2.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A3.SS2.p2.1.1.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A3.SS2.p2.1.1.m1.1a"><msub id="A3.SS2.p2.1.1.m1.1.1" xref="A3.SS2.p2.1.1.m1.1.1.cmml"><mi id="A3.SS2.p2.1.1.m1.1.1.2" xref="A3.SS2.p2.1.1.m1.1.1.2.cmml">S</mi><mn id="A3.SS2.p2.1.1.m1.1.1.3" xref="A3.SS2.p2.1.1.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A3.SS2.p2.1.1.m1.1b"><apply id="A3.SS2.p2.1.1.m1.1.1.cmml" xref="A3.SS2.p2.1.1.m1.1.1"><csymbol cd="ambiguous" id="A3.SS2.p2.1.1.m1.1.1.1.cmml" xref="A3.SS2.p2.1.1.m1.1.1">subscript</csymbol><ci id="A3.SS2.p2.1.1.m1.1.1.2.cmml" xref="A3.SS2.p2.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A3.SS2.p2.1.1.m1.1.1.3.cmml" xref="A3.SS2.p2.1.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS2.p2.1.1.m1.1c">S_{2}</annotation></semantics></math>&gt;</span> in Table <a href="#S2.T1" title="Table 1 ‣ Few-shot learning. ‣ 2 Related Work ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>).
Also if one sentence is appended any punctuation (e.g., <span id="A3.SS2.p2.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A3.SS2.p2.2.2.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A3.SS2.p2.2.2.m1.1a"><msub id="A3.SS2.p2.2.2.m1.1.1" xref="A3.SS2.p2.2.2.m1.1.1.cmml"><mi id="A3.SS2.p2.2.2.m1.1.1.2" xref="A3.SS2.p2.2.2.m1.1.1.2.cmml">S</mi><mn id="A3.SS2.p2.2.2.m1.1.1.3" xref="A3.SS2.p2.2.2.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A3.SS2.p2.2.2.m1.1b"><apply id="A3.SS2.p2.2.2.m1.1.1.cmml" xref="A3.SS2.p2.2.2.m1.1.1"><csymbol cd="ambiguous" id="A3.SS2.p2.2.2.m1.1.1.1.cmml" xref="A3.SS2.p2.2.2.m1.1.1">subscript</csymbol><ci id="A3.SS2.p2.2.2.m1.1.1.2.cmml" xref="A3.SS2.p2.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A3.SS2.p2.2.2.m1.1.1.3.cmml" xref="A3.SS2.p2.2.2.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS2.p2.2.2.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> in Table <a href="#S2.T1" title="Table 1 ‣ Few-shot learning. ‣ 2 Related Work ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>), then the last character of the original sentence is discarded.
Finally, we prepend a space for label words in <math id="A3.SS2.p2.3.m1.1" class="ltx_Math" alttext="\mathcal{M}(\mathcal{Y})" display="inline"><semantics id="A3.SS2.p2.3.m1.1a"><mrow id="A3.SS2.p2.3.m1.1.2" xref="A3.SS2.p2.3.m1.1.2.cmml"><mi class="ltx_font_mathcaligraphic" id="A3.SS2.p2.3.m1.1.2.2" xref="A3.SS2.p2.3.m1.1.2.2.cmml">ℳ</mi><mo lspace="0em" rspace="0em" id="A3.SS2.p2.3.m1.1.2.1" xref="A3.SS2.p2.3.m1.1.2.1.cmml">​</mo><mrow id="A3.SS2.p2.3.m1.1.2.3.2" xref="A3.SS2.p2.3.m1.1.2.cmml"><mo stretchy="false" id="A3.SS2.p2.3.m1.1.2.3.2.1" xref="A3.SS2.p2.3.m1.1.2.cmml">(</mo><mi class="ltx_font_mathcaligraphic" id="A3.SS2.p2.3.m1.1.1" xref="A3.SS2.p2.3.m1.1.1.cmml">𝒴</mi><mo stretchy="false" id="A3.SS2.p2.3.m1.1.2.3.2.2" xref="A3.SS2.p2.3.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.SS2.p2.3.m1.1b"><apply id="A3.SS2.p2.3.m1.1.2.cmml" xref="A3.SS2.p2.3.m1.1.2"><times id="A3.SS2.p2.3.m1.1.2.1.cmml" xref="A3.SS2.p2.3.m1.1.2.1"></times><ci id="A3.SS2.p2.3.m1.1.2.2.cmml" xref="A3.SS2.p2.3.m1.1.2.2">ℳ</ci><ci id="A3.SS2.p2.3.m1.1.1.cmml" xref="A3.SS2.p2.3.m1.1.1">𝒴</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.SS2.p2.3.m1.1c">\mathcal{M}(\mathcal{Y})</annotation></semantics></math>. For example, we use “_great” instead of “great” in the RoBERTa vocabulary, where “_” stands for space.</p>
</div>
</section>
<section id="A3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">C.3 </span>Fine-tuning with demonstrations</h3>

<div id="A3.SS3.p1" class="ltx_para">
<p id="A3.SS3.p1.1" class="ltx_p">When using demonstrations, we sample <math id="A3.SS3.p1.1.m1.1" class="ltx_Math" alttext="16" display="inline"><semantics id="A3.SS3.p1.1.m1.1a"><mn id="A3.SS3.p1.1.m1.1.1" xref="A3.SS3.p1.1.m1.1.1.cmml">16</mn><annotation-xml encoding="MathML-Content" id="A3.SS3.p1.1.m1.1b"><cn type="integer" id="A3.SS3.p1.1.m1.1.1.cmml" xref="A3.SS3.p1.1.m1.1.1">16</cn></annotation-xml><annotation encoding="application/x-tex" id="A3.SS3.p1.1.m1.1c">16</annotation></semantics></math> different sets of demonstrations for each input and average the predicted log probability for each class during inference.
We find that further increasing the number of samples does not bring substantial improvement.
Additional, we have tried different aggregation methods like taking the result with the maximum confidence and we did not find a meaningful improvement.
For selective demonstrations, we take <span id="A3.SS3.p1.1.1" class="ltx_text ltx_font_typewriter">roberta-large-nli-stsb- mean-tokens<span id="footnote14" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">14</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">14</sup><span class="ltx_tag ltx_tag_note"><span id="footnote14.1.1.1" class="ltx_text ltx_font_serif">14</span></span><a target="_blank" href="https://github.com/UKPLab/sentence-transformers" title="" class="ltx_ref ltx_url">https://github.com/UKPLab/sentence-transformers</a></span></span></span></span>
from <cite class="ltx_cite ltx_citemacro_citet">Reimers and Gurevych (<a href="#bib.bib33" title="" class="ltx_ref">2019</a>)</cite>
as our sentence embedding model.</p>
</div>
</section>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Comparisons of BERT vs RoBERTa</h2>

<figure id="A4.T1" class="ltx_table">
<div id="A4.T1.1.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:303.5pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(88.2pt,-61.8pt) scale(1.68618634388026,1.68618634388026) ;">
<table id="A4.T1.1.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A4.T1.1.1.1.2.1" class="ltx_tr">
<th id="A4.T1.1.1.1.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_tt"><span id="A4.T1.1.1.1.2.1.1.1" class="ltx_text ltx_font_bold">BERT-large</span></th>
<td id="A4.T1.1.1.1.2.1.2" class="ltx_td ltx_align_center ltx_border_tt"><span id="A4.T1.1.1.1.2.1.2.1" class="ltx_text ltx_font_bold">SST-2</span></td>
<td id="A4.T1.1.1.1.2.1.3" class="ltx_td ltx_align_center ltx_border_tt"><span id="A4.T1.1.1.1.2.1.3.1" class="ltx_text ltx_font_bold">SNLI</span></td>
<td id="A4.T1.1.1.1.2.1.4" class="ltx_td ltx_align_center ltx_border_tt"><span id="A4.T1.1.1.1.2.1.4.1" class="ltx_text ltx_font_bold">TREC</span></td>
<td id="A4.T1.1.1.1.2.1.5" class="ltx_td ltx_align_center ltx_border_tt"><span id="A4.T1.1.1.1.2.1.5.1" class="ltx_text ltx_font_bold">MRPC</span></td>
</tr>
<tr id="A4.T1.1.1.1.3.2" class="ltx_tr">
<th id="A4.T1.1.1.1.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Fine-tuning</th>
<td id="A4.T1.1.1.1.3.2.2" class="ltx_td ltx_align_center ltx_border_t">79.5</td>
<td id="A4.T1.1.1.1.3.2.3" class="ltx_td ltx_align_center ltx_border_t">51.4</td>
<td id="A4.T1.1.1.1.3.2.4" class="ltx_td ltx_align_center ltx_border_t">80.3</td>
<td id="A4.T1.1.1.1.3.2.5" class="ltx_td ltx_align_center ltx_border_t"><span id="A4.T1.1.1.1.3.2.5.1" class="ltx_text ltx_font_bold">74.4</span></td>
</tr>
<tr id="A4.T1.1.1.1.4.3" class="ltx_tr">
<th id="A4.T1.1.1.1.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Prompt-based FT</th>
<td id="A4.T1.1.1.1.4.3.2" class="ltx_td ltx_align_center ltx_border_t">85.6</td>
<td id="A4.T1.1.1.1.4.3.3" class="ltx_td ltx_align_center ltx_border_t">59.2</td>
<td id="A4.T1.1.1.1.4.3.4" class="ltx_td ltx_align_center ltx_border_t">79.0</td>
<td id="A4.T1.1.1.1.4.3.5" class="ltx_td ltx_align_center ltx_border_t">66.8</td>
</tr>
<tr id="A4.T1.1.1.1.5.4" class="ltx_tr">
<th id="A4.T1.1.1.1.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">+ demo (1-seg)</th>
<td id="A4.T1.1.1.1.5.4.2" class="ltx_td ltx_align_center"><span id="A4.T1.1.1.1.5.4.2.1" class="ltx_text ltx_font_bold">87.5</span></td>
<td id="A4.T1.1.1.1.5.4.3" class="ltx_td ltx_align_center">50.4</td>
<td id="A4.T1.1.1.1.5.4.4" class="ltx_td ltx_align_center">77.2</td>
<td id="A4.T1.1.1.1.5.4.5" class="ltx_td ltx_align_center">68.5</td>
</tr>
<tr id="A4.T1.1.1.1.6.5" class="ltx_tr">
<th id="A4.T1.1.1.1.6.5.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">+ demo (2-seg)</th>
<td id="A4.T1.1.1.1.6.5.2" class="ltx_td ltx_align_center">86.1</td>
<td id="A4.T1.1.1.1.6.5.3" class="ltx_td ltx_align_center"><span id="A4.T1.1.1.1.6.5.3.1" class="ltx_text ltx_font_bold">61.3</span></td>
<td id="A4.T1.1.1.1.6.5.4" class="ltx_td ltx_align_center">77.9</td>
<td id="A4.T1.1.1.1.6.5.5" class="ltx_td ltx_align_center">73.2</td>
</tr>
<tr id="A4.T1.1.1.1.1" class="ltx_tr">
<th id="A4.T1.1.1.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row">+ demo (<math id="A4.T1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A4.T1.1.1.1.1.1.m1.1a"><mi id="A4.T1.1.1.1.1.1.m1.1.1" xref="A4.T1.1.1.1.1.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A4.T1.1.1.1.1.1.m1.1b"><ci id="A4.T1.1.1.1.1.1.m1.1.1.cmml" xref="A4.T1.1.1.1.1.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.T1.1.1.1.1.1.m1.1c">n</annotation></semantics></math>-seg)</th>
<td id="A4.T1.1.1.1.1.2" class="ltx_td ltx_align_center">86.4</td>
<td id="A4.T1.1.1.1.1.3" class="ltx_td ltx_align_center">58.6</td>
<td id="A4.T1.1.1.1.1.4" class="ltx_td ltx_align_center"><span id="A4.T1.1.1.1.1.4.1" class="ltx_text ltx_font_bold">79.6</span></td>
<td id="A4.T1.1.1.1.1.5" class="ltx_td ltx_align_center">71.0</td>
</tr>
<tr id="A4.T1.1.1.1.7.6" class="ltx_tr">
<th id="A4.T1.1.1.1.7.6.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t"><span id="A4.T1.1.1.1.7.6.1.1" class="ltx_text ltx_font_bold">RoBERTa-large</span></th>
<td id="A4.T1.1.1.1.7.6.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A4.T1.1.1.1.7.6.2.1" class="ltx_text ltx_font_bold">SST-2</span></td>
<td id="A4.T1.1.1.1.7.6.3" class="ltx_td ltx_align_center ltx_border_t"><span id="A4.T1.1.1.1.7.6.3.1" class="ltx_text ltx_font_bold">SNLI</span></td>
<td id="A4.T1.1.1.1.7.6.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A4.T1.1.1.1.7.6.4.1" class="ltx_text ltx_font_bold">TREC</span></td>
<td id="A4.T1.1.1.1.7.6.5" class="ltx_td ltx_align_center ltx_border_t"><span id="A4.T1.1.1.1.7.6.5.1" class="ltx_text ltx_font_bold">MRPC</span></td>
</tr>
<tr id="A4.T1.1.1.1.8.7" class="ltx_tr">
<th id="A4.T1.1.1.1.8.7.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Fine-tuning</th>
<td id="A4.T1.1.1.1.8.7.2" class="ltx_td ltx_align_center ltx_border_t">81.4</td>
<td id="A4.T1.1.1.1.8.7.3" class="ltx_td ltx_align_center ltx_border_t">48.4</td>
<td id="A4.T1.1.1.1.8.7.4" class="ltx_td ltx_align_center ltx_border_t"><span id="A4.T1.1.1.1.8.7.4.1" class="ltx_text ltx_font_bold">88.8</span></td>
<td id="A4.T1.1.1.1.8.7.5" class="ltx_td ltx_align_center ltx_border_t">76.6</td>
</tr>
<tr id="A4.T1.1.1.1.9.8" class="ltx_tr">
<th id="A4.T1.1.1.1.9.8.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t">Prompt-based FT</th>
<td id="A4.T1.1.1.1.9.8.2" class="ltx_td ltx_align_center ltx_border_t"><span id="A4.T1.1.1.1.9.8.2.1" class="ltx_text ltx_font_bold">92.7</span></td>
<td id="A4.T1.1.1.1.9.8.3" class="ltx_td ltx_align_center ltx_border_t">77.2</td>
<td id="A4.T1.1.1.1.9.8.4" class="ltx_td ltx_align_center ltx_border_t">84.8</td>
<td id="A4.T1.1.1.1.9.8.5" class="ltx_td ltx_align_center ltx_border_t">74.5</td>
</tr>
<tr id="A4.T1.1.1.1.10.9" class="ltx_tr">
<th id="A4.T1.1.1.1.10.9.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb">+ demonstrations</th>
<td id="A4.T1.1.1.1.10.9.2" class="ltx_td ltx_align_center ltx_border_bb">92.6</td>
<td id="A4.T1.1.1.1.10.9.3" class="ltx_td ltx_align_center ltx_border_bb"><span id="A4.T1.1.1.1.10.9.3.1" class="ltx_text ltx_font_bold">79.7</span></td>
<td id="A4.T1.1.1.1.10.9.4" class="ltx_td ltx_align_center ltx_border_bb">87.5</td>
<td id="A4.T1.1.1.1.10.9.5" class="ltx_td ltx_align_center ltx_border_bb"><span id="A4.T1.1.1.1.10.9.5.1" class="ltx_text ltx_font_bold">77.8</span></td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table D.1: </span>A comparison of BERT-large vs RoBERTa-large. We use manual prompts in these experiments.</figcaption>
</figure>
<div id="A4.p1" class="ltx_para">
<p id="A4.p1.1" class="ltx_p">Table <a href="#A4.T1" title="Table D.1 ‣ Appendix D Comparisons of BERT vs RoBERTa ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D.1</span></a> compares the results of BERT-large (uncased) and RoBERTa-large in our settings.
Pre-trained BERT provides two segment embeddings (A/B) for different parts of input. The common practice, when fine-tuning BERT, is that using only segment A for single-sentence tasks, and using segment A/B for the two sentences in sentence-pair tasks.
In our case of incorporating demonstrations, however, we have more than two sentences. Thus we explore the following different strategies for segments:
(1) using the A segment for all sentences (1-seg);
(2) using the A segment for the original input and the B segment for the demonstrations (2-seg);
(3) using different segment embeddings for each sentence (<math id="A4.p1.1.m1.1" class="ltx_Math" alttext="n" display="inline"><semantics id="A4.p1.1.m1.1a"><mi id="A4.p1.1.m1.1.1" xref="A4.p1.1.m1.1.1.cmml">n</mi><annotation-xml encoding="MathML-Content" id="A4.p1.1.m1.1b"><ci id="A4.p1.1.m1.1.1.cmml" xref="A4.p1.1.m1.1.1">𝑛</ci></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.1.m1.1c">n</annotation></semantics></math>-seg), e.g., for SNLI, we use different segments for each premise and hypothesis in both the original input and the demonstrations, which leads to a total number of 8 segment embeddings. This introduces new segment embeddings (randomly initialized and learned during fine-tuning) as the pre-trained BERT only has two.</p>
</div>
<div id="A4.p2" class="ltx_para">
<p id="A4.p2.1" class="ltx_p">Table <a href="#A4.T1" title="Table D.1 ‣ Appendix D Comparisons of BERT vs RoBERTa ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">D.1</span></a> shows that
prompt-based fine-tuning with demonstrations also works for BERT, and 2-seg works the best when incorporating demonstrations.
Still, we take RoBERTa-large as our main model, for RoBERTa performs much better than BERT and RoBERTa saves the trouble to tune the usage of segment embeddings.</p>
</div>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Generated Prompts</h2>

<div id="A5.p1" class="ltx_para">
<p id="A5.p1.1" class="ltx_p">We demonstrate the top 3 automatically generated templates and label words for all tasks in Table <a href="#A5.T1" title="Table E.1 ‣ Appendix E Generated Prompts ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">E.1</span></a>. In general, most automatic templates are reasonable and grammatically correct.
For the label words, the generated results look intuitive for most single sentence tasks. For other tasks, the automatic ones can be counterintuitive in some cases.
It is still unclear why the language model picks these words and sometimes they actually work well.
We leave this for future study.</p>
</div>
<figure id="A5.T1" class="ltx_table">
<div id="A5.T1.68.68" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:828.2pt;height:1523pt;vertical-align:-1.4pt;"><span class="ltx_transformed_inner" style="transform:translate(115.3pt,-211.8pt) scale(1.3858100202242,1.3858100202242) ;">
<table id="A5.T1.68.68.68" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="A5.T1.68.68.68.69.1" class="ltx_tr">
<td id="A5.T1.68.68.68.69.1.1" class="ltx_td ltx_align_left ltx_border_tt"><span id="A5.T1.68.68.68.69.1.1.1" class="ltx_text ltx_font_bold">Task</span></td>
<td id="A5.T1.68.68.68.69.1.2" class="ltx_td ltx_align_left ltx_border_tt"><span id="A5.T1.68.68.68.69.1.2.1" class="ltx_text ltx_font_bold">Auto template</span></td>
<td id="A5.T1.68.68.68.69.1.3" class="ltx_td ltx_align_left ltx_border_tt"><span id="A5.T1.68.68.68.69.1.3.1" class="ltx_text ltx_font_bold">Auto label words</span></td>
</tr>
<tr id="A5.T1.68.68.68.70.2" class="ltx_tr">
<td id="A5.T1.68.68.68.70.2.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A5.T1.68.68.68.70.2.1.1" class="ltx_text ltx_font_bold">SST-2</span></td>
<td id="A5.T1.68.68.68.70.2.2" class="ltx_td ltx_align_left ltx_border_t">(positive/negative)</td>
<td id="A5.T1.68.68.68.70.2.3" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="A5.T1.1.1.1.1" class="ltx_tr">
<td id="A5.T1.1.1.1.1.2" class="ltx_td"></td>
<td id="A5.T1.1.1.1.1.1" class="ltx_td ltx_align_left">
<span id="A5.T1.1.1.1.1.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.1.1.1.1.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.1.1.1.1.1.1.m1.1a"><msub id="A5.T1.1.1.1.1.1.1.m1.1.1" xref="A5.T1.1.1.1.1.1.1.m1.1.1.cmml"><mi id="A5.T1.1.1.1.1.1.1.m1.1.1.2" xref="A5.T1.1.1.1.1.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.1.1.1.1.1.1.m1.1.1.3" xref="A5.T1.1.1.1.1.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.1.1.1.1.1.1.m1.1b"><apply id="A5.T1.1.1.1.1.1.1.m1.1.1.cmml" xref="A5.T1.1.1.1.1.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.1.1.1.1.1.1.m1.1.1.1.cmml" xref="A5.T1.1.1.1.1.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.1.1.1.1.1.1.m1.1.1.2.cmml" xref="A5.T1.1.1.1.1.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.1.1.1.1.1.1.m1.1.1.3.cmml" xref="A5.T1.1.1.1.1.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.1.1.1.1.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> A <span id="A5.T1.1.1.1.1.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> one .</td>
<td id="A5.T1.1.1.1.1.3" class="ltx_td ltx_align_left">irresistible/pathetic</td>
</tr>
<tr id="A5.T1.2.2.2.2" class="ltx_tr">
<td id="A5.T1.2.2.2.2.2" class="ltx_td"></td>
<td id="A5.T1.2.2.2.2.1" class="ltx_td ltx_align_left">
<span id="A5.T1.2.2.2.2.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.2.2.2.2.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.2.2.2.2.1.1.m1.1a"><msub id="A5.T1.2.2.2.2.1.1.m1.1.1" xref="A5.T1.2.2.2.2.1.1.m1.1.1.cmml"><mi id="A5.T1.2.2.2.2.1.1.m1.1.1.2" xref="A5.T1.2.2.2.2.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.2.2.2.2.1.1.m1.1.1.3" xref="A5.T1.2.2.2.2.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.2.2.2.2.1.1.m1.1b"><apply id="A5.T1.2.2.2.2.1.1.m1.1.1.cmml" xref="A5.T1.2.2.2.2.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.2.2.2.2.1.1.m1.1.1.1.cmml" xref="A5.T1.2.2.2.2.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.2.2.2.2.1.1.m1.1.1.2.cmml" xref="A5.T1.2.2.2.2.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.2.2.2.2.1.1.m1.1.1.3.cmml" xref="A5.T1.2.2.2.2.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.2.2.2.2.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> A <span id="A5.T1.2.2.2.2.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> piece .</td>
<td id="A5.T1.2.2.2.2.3" class="ltx_td ltx_align_left">wonderful/bad</td>
</tr>
<tr id="A5.T1.3.3.3.3" class="ltx_tr">
<td id="A5.T1.3.3.3.3.2" class="ltx_td"></td>
<td id="A5.T1.3.3.3.3.1" class="ltx_td ltx_align_left">
<span id="A5.T1.3.3.3.3.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.3.3.3.3.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.3.3.3.3.1.1.m1.1a"><msub id="A5.T1.3.3.3.3.1.1.m1.1.1" xref="A5.T1.3.3.3.3.1.1.m1.1.1.cmml"><mi id="A5.T1.3.3.3.3.1.1.m1.1.1.2" xref="A5.T1.3.3.3.3.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.3.3.3.3.1.1.m1.1.1.3" xref="A5.T1.3.3.3.3.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.3.3.3.3.1.1.m1.1b"><apply id="A5.T1.3.3.3.3.1.1.m1.1.1.cmml" xref="A5.T1.3.3.3.3.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.3.3.3.3.1.1.m1.1.1.1.cmml" xref="A5.T1.3.3.3.3.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.3.3.3.3.1.1.m1.1.1.2.cmml" xref="A5.T1.3.3.3.3.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.3.3.3.3.1.1.m1.1.1.3.cmml" xref="A5.T1.3.3.3.3.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.3.3.3.3.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> All in all <span id="A5.T1.3.3.3.3.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="A5.T1.3.3.3.3.3" class="ltx_td ltx_align_left">delicious/bad</td>
</tr>
<tr id="A5.T1.68.68.68.71.3" class="ltx_tr">
<td id="A5.T1.68.68.68.71.3.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A5.T1.68.68.68.71.3.1.1" class="ltx_text ltx_font_bold">SST-5</span></td>
<td id="A5.T1.68.68.68.71.3.2" class="ltx_td ltx_align_left ltx_border_t">(very positive/positive/neutral/negative/very negative)</td>
<td id="A5.T1.68.68.68.71.3.3" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="A5.T1.4.4.4.4" class="ltx_tr">
<td id="A5.T1.4.4.4.4.2" class="ltx_td"></td>
<td id="A5.T1.4.4.4.4.1" class="ltx_td ltx_align_left">
<span id="A5.T1.4.4.4.4.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.4.4.4.4.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.4.4.4.4.1.1.m1.1a"><msub id="A5.T1.4.4.4.4.1.1.m1.1.1" xref="A5.T1.4.4.4.4.1.1.m1.1.1.cmml"><mi id="A5.T1.4.4.4.4.1.1.m1.1.1.2" xref="A5.T1.4.4.4.4.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.4.4.4.4.1.1.m1.1.1.3" xref="A5.T1.4.4.4.4.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.4.4.4.4.1.1.m1.1b"><apply id="A5.T1.4.4.4.4.1.1.m1.1.1.cmml" xref="A5.T1.4.4.4.4.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.4.4.4.4.1.1.m1.1.1.1.cmml" xref="A5.T1.4.4.4.4.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.4.4.4.4.1.1.m1.1.1.2.cmml" xref="A5.T1.4.4.4.4.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.4.4.4.4.1.1.m1.1.1.3.cmml" xref="A5.T1.4.4.4.4.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.4.4.4.4.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> The movie is <span id="A5.T1.4.4.4.4.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="A5.T1.4.4.4.4.3" class="ltx_td ltx_align_left">wonderful/remarkable/hilarious/better/awful</td>
</tr>
<tr id="A5.T1.5.5.5.5" class="ltx_tr">
<td id="A5.T1.5.5.5.5.2" class="ltx_td"></td>
<td id="A5.T1.5.5.5.5.1" class="ltx_td ltx_align_left">
<span id="A5.T1.5.5.5.5.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.5.5.5.5.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.5.5.5.5.1.1.m1.1a"><msub id="A5.T1.5.5.5.5.1.1.m1.1.1" xref="A5.T1.5.5.5.5.1.1.m1.1.1.cmml"><mi id="A5.T1.5.5.5.5.1.1.m1.1.1.2" xref="A5.T1.5.5.5.5.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.5.5.5.5.1.1.m1.1.1.3" xref="A5.T1.5.5.5.5.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.5.5.5.5.1.1.m1.1b"><apply id="A5.T1.5.5.5.5.1.1.m1.1.1.cmml" xref="A5.T1.5.5.5.5.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.5.5.5.5.1.1.m1.1.1.1.cmml" xref="A5.T1.5.5.5.5.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.5.5.5.5.1.1.m1.1.1.2.cmml" xref="A5.T1.5.5.5.5.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.5.5.5.5.1.1.m1.1.1.3.cmml" xref="A5.T1.5.5.5.5.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.5.5.5.5.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> The music is <span id="A5.T1.5.5.5.5.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="A5.T1.5.5.5.5.3" class="ltx_td ltx_align_left">wonderful/perfect/hilarious/better/awful</td>
</tr>
<tr id="A5.T1.6.6.6.6" class="ltx_tr">
<td id="A5.T1.6.6.6.6.2" class="ltx_td"></td>
<td id="A5.T1.6.6.6.6.1" class="ltx_td ltx_align_left">
<span id="A5.T1.6.6.6.6.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.6.6.6.6.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.6.6.6.6.1.1.m1.1a"><msub id="A5.T1.6.6.6.6.1.1.m1.1.1" xref="A5.T1.6.6.6.6.1.1.m1.1.1.cmml"><mi id="A5.T1.6.6.6.6.1.1.m1.1.1.2" xref="A5.T1.6.6.6.6.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.6.6.6.6.1.1.m1.1.1.3" xref="A5.T1.6.6.6.6.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.6.6.6.6.1.1.m1.1b"><apply id="A5.T1.6.6.6.6.1.1.m1.1.1.cmml" xref="A5.T1.6.6.6.6.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.6.6.6.6.1.1.m1.1.1.1.cmml" xref="A5.T1.6.6.6.6.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.6.6.6.6.1.1.m1.1.1.2.cmml" xref="A5.T1.6.6.6.6.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.6.6.6.6.1.1.m1.1.1.3.cmml" xref="A5.T1.6.6.6.6.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.6.6.6.6.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> But it is <span id="A5.T1.6.6.6.6.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="A5.T1.6.6.6.6.3" class="ltx_td ltx_align_left">unforgettable/extraordinary/good/better/terrible</td>
</tr>
<tr id="A5.T1.68.68.68.72.4" class="ltx_tr">
<td id="A5.T1.68.68.68.72.4.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A5.T1.68.68.68.72.4.1.1" class="ltx_text ltx_font_bold">MR</span></td>
<td id="A5.T1.68.68.68.72.4.2" class="ltx_td ltx_align_left ltx_border_t">(positive/negative)</td>
<td id="A5.T1.68.68.68.72.4.3" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="A5.T1.7.7.7.7" class="ltx_tr">
<td id="A5.T1.7.7.7.7.2" class="ltx_td"></td>
<td id="A5.T1.7.7.7.7.1" class="ltx_td ltx_align_left">It was <span id="A5.T1.7.7.7.7.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> ! <span id="A5.T1.7.7.7.7.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.7.7.7.7.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.7.7.7.7.1.1.m1.1a"><msub id="A5.T1.7.7.7.7.1.1.m1.1.1" xref="A5.T1.7.7.7.7.1.1.m1.1.1.cmml"><mi id="A5.T1.7.7.7.7.1.1.m1.1.1.2" xref="A5.T1.7.7.7.7.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.7.7.7.7.1.1.m1.1.1.3" xref="A5.T1.7.7.7.7.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.7.7.7.7.1.1.m1.1b"><apply id="A5.T1.7.7.7.7.1.1.m1.1.1.cmml" xref="A5.T1.7.7.7.7.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.7.7.7.7.1.1.m1.1.1.1.cmml" xref="A5.T1.7.7.7.7.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.7.7.7.7.1.1.m1.1.1.2.cmml" xref="A5.T1.7.7.7.7.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.7.7.7.7.1.1.m1.1.1.3.cmml" xref="A5.T1.7.7.7.7.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.7.7.7.7.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.7.7.7.7.3" class="ltx_td ltx_align_left">epic/terrible</td>
</tr>
<tr id="A5.T1.8.8.8.8" class="ltx_tr">
<td id="A5.T1.8.8.8.8.2" class="ltx_td"></td>
<td id="A5.T1.8.8.8.8.1" class="ltx_td ltx_align_left">
<span id="A5.T1.8.8.8.8.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.8.8.8.8.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.8.8.8.8.1.1.m1.1a"><msub id="A5.T1.8.8.8.8.1.1.m1.1.1" xref="A5.T1.8.8.8.8.1.1.m1.1.1.cmml"><mi id="A5.T1.8.8.8.8.1.1.m1.1.1.2" xref="A5.T1.8.8.8.8.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.8.8.8.8.1.1.m1.1.1.3" xref="A5.T1.8.8.8.8.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.8.8.8.8.1.1.m1.1b"><apply id="A5.T1.8.8.8.8.1.1.m1.1.1.cmml" xref="A5.T1.8.8.8.8.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.8.8.8.8.1.1.m1.1.1.1.cmml" xref="A5.T1.8.8.8.8.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.8.8.8.8.1.1.m1.1.1.2.cmml" xref="A5.T1.8.8.8.8.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.8.8.8.8.1.1.m1.1.1.3.cmml" xref="A5.T1.8.8.8.8.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.8.8.8.8.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> It’s <span id="A5.T1.8.8.8.8.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="A5.T1.8.8.8.8.3" class="ltx_td ltx_align_left">epic/awful</td>
</tr>
<tr id="A5.T1.9.9.9.9" class="ltx_tr">
<td id="A5.T1.9.9.9.9.2" class="ltx_td"></td>
<td id="A5.T1.9.9.9.9.1" class="ltx_td ltx_align_left">
<span id="A5.T1.9.9.9.9.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.9.9.9.9.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.9.9.9.9.1.1.m1.1a"><msub id="A5.T1.9.9.9.9.1.1.m1.1.1" xref="A5.T1.9.9.9.9.1.1.m1.1.1.cmml"><mi id="A5.T1.9.9.9.9.1.1.m1.1.1.2" xref="A5.T1.9.9.9.9.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.9.9.9.9.1.1.m1.1.1.3" xref="A5.T1.9.9.9.9.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.9.9.9.9.1.1.m1.1b"><apply id="A5.T1.9.9.9.9.1.1.m1.1.1.cmml" xref="A5.T1.9.9.9.9.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.9.9.9.9.1.1.m1.1.1.1.cmml" xref="A5.T1.9.9.9.9.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.9.9.9.9.1.1.m1.1.1.2.cmml" xref="A5.T1.9.9.9.9.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.9.9.9.9.1.1.m1.1.1.3.cmml" xref="A5.T1.9.9.9.9.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.9.9.9.9.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> A <span id="A5.T1.9.9.9.9.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> piece of work .</td>
<td id="A5.T1.9.9.9.9.3" class="ltx_td ltx_align_left">exquisite/horrible</td>
</tr>
<tr id="A5.T1.68.68.68.73.5" class="ltx_tr">
<td id="A5.T1.68.68.68.73.5.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A5.T1.68.68.68.73.5.1.1" class="ltx_text ltx_font_bold">CR</span></td>
<td id="A5.T1.68.68.68.73.5.2" class="ltx_td ltx_align_left ltx_border_t">(positive/negative)</td>
<td id="A5.T1.68.68.68.73.5.3" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="A5.T1.10.10.10.10" class="ltx_tr">
<td id="A5.T1.10.10.10.10.2" class="ltx_td"></td>
<td id="A5.T1.10.10.10.10.1" class="ltx_td ltx_align_left">
<span id="A5.T1.10.10.10.10.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.10.10.10.10.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.10.10.10.10.1.1.m1.1a"><msub id="A5.T1.10.10.10.10.1.1.m1.1.1" xref="A5.T1.10.10.10.10.1.1.m1.1.1.cmml"><mi id="A5.T1.10.10.10.10.1.1.m1.1.1.2" xref="A5.T1.10.10.10.10.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.10.10.10.10.1.1.m1.1.1.3" xref="A5.T1.10.10.10.10.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.10.10.10.10.1.1.m1.1b"><apply id="A5.T1.10.10.10.10.1.1.m1.1.1.cmml" xref="A5.T1.10.10.10.10.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.10.10.10.10.1.1.m1.1.1.1.cmml" xref="A5.T1.10.10.10.10.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.10.10.10.10.1.1.m1.1.1.2.cmml" xref="A5.T1.10.10.10.10.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.10.10.10.10.1.1.m1.1.1.3.cmml" xref="A5.T1.10.10.10.10.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.10.10.10.10.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> It’s <span id="A5.T1.10.10.10.10.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> !</td>
<td id="A5.T1.10.10.10.10.3" class="ltx_td ltx_align_left">fantastic/horrible</td>
</tr>
<tr id="A5.T1.11.11.11.11" class="ltx_tr">
<td id="A5.T1.11.11.11.11.2" class="ltx_td"></td>
<td id="A5.T1.11.11.11.11.1" class="ltx_td ltx_align_left">
<span id="A5.T1.11.11.11.11.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.11.11.11.11.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.11.11.11.11.1.1.m1.1a"><msub id="A5.T1.11.11.11.11.1.1.m1.1.1" xref="A5.T1.11.11.11.11.1.1.m1.1.1.cmml"><mi id="A5.T1.11.11.11.11.1.1.m1.1.1.2" xref="A5.T1.11.11.11.11.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.11.11.11.11.1.1.m1.1.1.3" xref="A5.T1.11.11.11.11.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.11.11.11.11.1.1.m1.1b"><apply id="A5.T1.11.11.11.11.1.1.m1.1.1.cmml" xref="A5.T1.11.11.11.11.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.11.11.11.11.1.1.m1.1.1.1.cmml" xref="A5.T1.11.11.11.11.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.11.11.11.11.1.1.m1.1.1.2.cmml" xref="A5.T1.11.11.11.11.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.11.11.11.11.1.1.m1.1.1.3.cmml" xref="A5.T1.11.11.11.11.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.11.11.11.11.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> The quality is <span id="A5.T1.11.11.11.11.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="A5.T1.11.11.11.11.3" class="ltx_td ltx_align_left">neat/pointless</td>
</tr>
<tr id="A5.T1.12.12.12.12" class="ltx_tr">
<td id="A5.T1.12.12.12.12.2" class="ltx_td"></td>
<td id="A5.T1.12.12.12.12.1" class="ltx_td ltx_align_left">
<span id="A5.T1.12.12.12.12.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.12.12.12.12.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.12.12.12.12.1.1.m1.1a"><msub id="A5.T1.12.12.12.12.1.1.m1.1.1" xref="A5.T1.12.12.12.12.1.1.m1.1.1.cmml"><mi id="A5.T1.12.12.12.12.1.1.m1.1.1.2" xref="A5.T1.12.12.12.12.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.12.12.12.12.1.1.m1.1.1.3" xref="A5.T1.12.12.12.12.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.12.12.12.12.1.1.m1.1b"><apply id="A5.T1.12.12.12.12.1.1.m1.1.1.cmml" xref="A5.T1.12.12.12.12.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.12.12.12.12.1.1.m1.1.1.1.cmml" xref="A5.T1.12.12.12.12.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.12.12.12.12.1.1.m1.1.1.2.cmml" xref="A5.T1.12.12.12.12.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.12.12.12.12.1.1.m1.1.1.3.cmml" xref="A5.T1.12.12.12.12.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.12.12.12.12.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> That is <span id="A5.T1.12.12.12.12.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="A5.T1.12.12.12.12.3" class="ltx_td ltx_align_left">magnificent/unacceptable</td>
</tr>
<tr id="A5.T1.68.68.68.74.6" class="ltx_tr">
<td id="A5.T1.68.68.68.74.6.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A5.T1.68.68.68.74.6.1.1" class="ltx_text ltx_font_bold">MPQA</span></td>
<td id="A5.T1.68.68.68.74.6.2" class="ltx_td ltx_align_left ltx_border_t">(positive/negative)</td>
<td id="A5.T1.68.68.68.74.6.3" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="A5.T1.13.13.13.13" class="ltx_tr">
<td id="A5.T1.13.13.13.13.2" class="ltx_td"></td>
<td id="A5.T1.13.13.13.13.1" class="ltx_td ltx_align_left">
<span id="A5.T1.13.13.13.13.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.13.13.13.13.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.13.13.13.13.1.1.m1.1a"><msub id="A5.T1.13.13.13.13.1.1.m1.1.1" xref="A5.T1.13.13.13.13.1.1.m1.1.1.cmml"><mi id="A5.T1.13.13.13.13.1.1.m1.1.1.2" xref="A5.T1.13.13.13.13.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.13.13.13.13.1.1.m1.1.1.3" xref="A5.T1.13.13.13.13.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.13.13.13.13.1.1.m1.1b"><apply id="A5.T1.13.13.13.13.1.1.m1.1.1.cmml" xref="A5.T1.13.13.13.13.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.13.13.13.13.1.1.m1.1.1.1.cmml" xref="A5.T1.13.13.13.13.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.13.13.13.13.1.1.m1.1.1.2.cmml" xref="A5.T1.13.13.13.13.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.13.13.13.13.1.1.m1.1.1.3.cmml" xref="A5.T1.13.13.13.13.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.13.13.13.13.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> is <span id="A5.T1.13.13.13.13.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="A5.T1.13.13.13.13.3" class="ltx_td ltx_align_left">important/close</td>
</tr>
<tr id="A5.T1.14.14.14.14" class="ltx_tr">
<td id="A5.T1.14.14.14.14.2" class="ltx_td"></td>
<td id="A5.T1.14.14.14.14.1" class="ltx_td ltx_align_left">
<span id="A5.T1.14.14.14.14.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.14.14.14.14.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.14.14.14.14.1.1.m1.1a"><msub id="A5.T1.14.14.14.14.1.1.m1.1.1" xref="A5.T1.14.14.14.14.1.1.m1.1.1.cmml"><mi id="A5.T1.14.14.14.14.1.1.m1.1.1.2" xref="A5.T1.14.14.14.14.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.14.14.14.14.1.1.m1.1.1.3" xref="A5.T1.14.14.14.14.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.14.14.14.14.1.1.m1.1b"><apply id="A5.T1.14.14.14.14.1.1.m1.1.1.cmml" xref="A5.T1.14.14.14.14.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.14.14.14.14.1.1.m1.1.1.1.cmml" xref="A5.T1.14.14.14.14.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.14.14.14.14.1.1.m1.1.1.2.cmml" xref="A5.T1.14.14.14.14.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.14.14.14.14.1.1.m1.1.1.3.cmml" xref="A5.T1.14.14.14.14.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.14.14.14.14.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span>, <span id="A5.T1.14.14.14.14.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> !</td>
<td id="A5.T1.14.14.14.14.3" class="ltx_td ltx_align_left">needed/bad</td>
</tr>
<tr id="A5.T1.15.15.15.15" class="ltx_tr">
<td id="A5.T1.15.15.15.15.2" class="ltx_td"></td>
<td id="A5.T1.15.15.15.15.1" class="ltx_td ltx_align_left">
<span id="A5.T1.15.15.15.15.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.15.15.15.15.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.15.15.15.15.1.1.m1.1a"><msub id="A5.T1.15.15.15.15.1.1.m1.1.1" xref="A5.T1.15.15.15.15.1.1.m1.1.1.cmml"><mi id="A5.T1.15.15.15.15.1.1.m1.1.1.2" xref="A5.T1.15.15.15.15.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.15.15.15.15.1.1.m1.1.1.3" xref="A5.T1.15.15.15.15.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.15.15.15.15.1.1.m1.1b"><apply id="A5.T1.15.15.15.15.1.1.m1.1.1.cmml" xref="A5.T1.15.15.15.15.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.15.15.15.15.1.1.m1.1.1.1.cmml" xref="A5.T1.15.15.15.15.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.15.15.15.15.1.1.m1.1.1.2.cmml" xref="A5.T1.15.15.15.15.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.15.15.15.15.1.1.m1.1.1.3.cmml" xref="A5.T1.15.15.15.15.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.15.15.15.15.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span>. <span id="A5.T1.15.15.15.15.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="A5.T1.15.15.15.15.3" class="ltx_td ltx_align_left">unexpected/shocking</td>
</tr>
<tr id="A5.T1.68.68.68.75.7" class="ltx_tr">
<td id="A5.T1.68.68.68.75.7.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A5.T1.68.68.68.75.7.1.1" class="ltx_text ltx_font_bold">Subj</span></td>
<td id="A5.T1.68.68.68.75.7.2" class="ltx_td ltx_align_left ltx_border_t">(subjective/objective)</td>
<td id="A5.T1.68.68.68.75.7.3" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="A5.T1.16.16.16.16" class="ltx_tr">
<td id="A5.T1.16.16.16.16.2" class="ltx_td"></td>
<td id="A5.T1.16.16.16.16.1" class="ltx_td ltx_align_left">
<span id="A5.T1.16.16.16.16.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.16.16.16.16.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.16.16.16.16.1.1.m1.1a"><msub id="A5.T1.16.16.16.16.1.1.m1.1.1" xref="A5.T1.16.16.16.16.1.1.m1.1.1.cmml"><mi id="A5.T1.16.16.16.16.1.1.m1.1.1.2" xref="A5.T1.16.16.16.16.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.16.16.16.16.1.1.m1.1.1.3" xref="A5.T1.16.16.16.16.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.16.16.16.16.1.1.m1.1b"><apply id="A5.T1.16.16.16.16.1.1.m1.1.1.cmml" xref="A5.T1.16.16.16.16.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.16.16.16.16.1.1.m1.1.1.1.cmml" xref="A5.T1.16.16.16.16.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.16.16.16.16.1.1.m1.1.1.2.cmml" xref="A5.T1.16.16.16.16.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.16.16.16.16.1.1.m1.1.1.3.cmml" xref="A5.T1.16.16.16.16.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.16.16.16.16.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> It’s all <span id="A5.T1.16.16.16.16.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="A5.T1.16.16.16.16.3" class="ltx_td ltx_align_left">everywhere/tragic</td>
</tr>
<tr id="A5.T1.17.17.17.17" class="ltx_tr">
<td id="A5.T1.17.17.17.17.2" class="ltx_td"></td>
<td id="A5.T1.17.17.17.17.1" class="ltx_td ltx_align_left">
<span id="A5.T1.17.17.17.17.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.17.17.17.17.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.17.17.17.17.1.1.m1.1a"><msub id="A5.T1.17.17.17.17.1.1.m1.1.1" xref="A5.T1.17.17.17.17.1.1.m1.1.1.cmml"><mi id="A5.T1.17.17.17.17.1.1.m1.1.1.2" xref="A5.T1.17.17.17.17.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.17.17.17.17.1.1.m1.1.1.3" xref="A5.T1.17.17.17.17.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.17.17.17.17.1.1.m1.1b"><apply id="A5.T1.17.17.17.17.1.1.m1.1.1.cmml" xref="A5.T1.17.17.17.17.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.17.17.17.17.1.1.m1.1.1.1.cmml" xref="A5.T1.17.17.17.17.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.17.17.17.17.1.1.m1.1.1.2.cmml" xref="A5.T1.17.17.17.17.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.17.17.17.17.1.1.m1.1.1.3.cmml" xref="A5.T1.17.17.17.17.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.17.17.17.17.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> It’s <span id="A5.T1.17.17.17.17.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="A5.T1.17.17.17.17.3" class="ltx_td ltx_align_left">everywhere/horrifying</td>
</tr>
<tr id="A5.T1.18.18.18.18" class="ltx_tr">
<td id="A5.T1.18.18.18.18.2" class="ltx_td"></td>
<td id="A5.T1.18.18.18.18.1" class="ltx_td ltx_align_left">
<span id="A5.T1.18.18.18.18.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.18.18.18.18.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.18.18.18.18.1.1.m1.1a"><msub id="A5.T1.18.18.18.18.1.1.m1.1.1" xref="A5.T1.18.18.18.18.1.1.m1.1.1.cmml"><mi id="A5.T1.18.18.18.18.1.1.m1.1.1.2" xref="A5.T1.18.18.18.18.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.18.18.18.18.1.1.m1.1.1.3" xref="A5.T1.18.18.18.18.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.18.18.18.18.1.1.m1.1b"><apply id="A5.T1.18.18.18.18.1.1.m1.1.1.cmml" xref="A5.T1.18.18.18.18.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.18.18.18.18.1.1.m1.1.1.1.cmml" xref="A5.T1.18.18.18.18.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.18.18.18.18.1.1.m1.1.1.2.cmml" xref="A5.T1.18.18.18.18.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.18.18.18.18.1.1.m1.1.1.3.cmml" xref="A5.T1.18.18.18.18.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.18.18.18.18.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> Is it <span id="A5.T1.18.18.18.18.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> ?</td>
<td id="A5.T1.18.18.18.18.3" class="ltx_td ltx_align_left">something/surreal</td>
</tr>
<tr id="A5.T1.68.68.68.76.8" class="ltx_tr">
<td id="A5.T1.68.68.68.76.8.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A5.T1.68.68.68.76.8.1.1" class="ltx_text ltx_font_bold">TREC</span></td>
<td id="A5.T1.68.68.68.76.8.2" class="ltx_td ltx_align_left ltx_border_t">(abbreviation/entity/description/human/location/numeric)</td>
<td id="A5.T1.68.68.68.76.8.3" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="A5.T1.19.19.19.19" class="ltx_tr">
<td id="A5.T1.19.19.19.19.2" class="ltx_td"></td>
<td id="A5.T1.19.19.19.19.1" class="ltx_td ltx_align_left">Q: <span id="A5.T1.19.19.19.19.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> : <span id="A5.T1.19.19.19.19.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.19.19.19.19.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.19.19.19.19.1.1.m1.1a"><msub id="A5.T1.19.19.19.19.1.1.m1.1.1" xref="A5.T1.19.19.19.19.1.1.m1.1.1.cmml"><mi id="A5.T1.19.19.19.19.1.1.m1.1.1.2" xref="A5.T1.19.19.19.19.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.19.19.19.19.1.1.m1.1.1.3" xref="A5.T1.19.19.19.19.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.19.19.19.19.1.1.m1.1b"><apply id="A5.T1.19.19.19.19.1.1.m1.1.1.cmml" xref="A5.T1.19.19.19.19.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.19.19.19.19.1.1.m1.1.1.1.cmml" xref="A5.T1.19.19.19.19.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.19.19.19.19.1.1.m1.1.1.2.cmml" xref="A5.T1.19.19.19.19.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.19.19.19.19.1.1.m1.1.1.3.cmml" xref="A5.T1.19.19.19.19.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.19.19.19.19.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.19.19.19.19.3" class="ltx_td ltx_align_left">Application/Advisor/Discussion/Culture/Assignment/Minute</td>
</tr>
<tr id="A5.T1.20.20.20.20" class="ltx_tr">
<td id="A5.T1.20.20.20.20.2" class="ltx_td"></td>
<td id="A5.T1.20.20.20.20.1" class="ltx_td ltx_align_left">
<span id="A5.T1.20.20.20.20.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.20.20.20.20.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.20.20.20.20.1.1.m1.1a"><msub id="A5.T1.20.20.20.20.1.1.m1.1.1" xref="A5.T1.20.20.20.20.1.1.m1.1.1.cmml"><mi id="A5.T1.20.20.20.20.1.1.m1.1.1.2" xref="A5.T1.20.20.20.20.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.20.20.20.20.1.1.m1.1.1.3" xref="A5.T1.20.20.20.20.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.20.20.20.20.1.1.m1.1b"><apply id="A5.T1.20.20.20.20.1.1.m1.1.1.cmml" xref="A5.T1.20.20.20.20.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.20.20.20.20.1.1.m1.1.1.1.cmml" xref="A5.T1.20.20.20.20.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.20.20.20.20.1.1.m1.1.1.2.cmml" xref="A5.T1.20.20.20.20.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.20.20.20.20.1.1.m1.1.1.3.cmml" xref="A5.T1.20.20.20.20.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.20.20.20.20.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> Why <span id="A5.T1.20.20.20.20.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span>?</td>
<td id="A5.T1.20.20.20.20.3" class="ltx_td ltx_align_left">Production/AE/Context/Artist/Assignment/Minute</td>
</tr>
<tr id="A5.T1.21.21.21.21" class="ltx_tr">
<td id="A5.T1.21.21.21.21.2" class="ltx_td"></td>
<td id="A5.T1.21.21.21.21.1" class="ltx_td ltx_align_left">
<span id="A5.T1.21.21.21.21.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.21.21.21.21.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.21.21.21.21.1.1.m1.1a"><msub id="A5.T1.21.21.21.21.1.1.m1.1.1" xref="A5.T1.21.21.21.21.1.1.m1.1.1.cmml"><mi id="A5.T1.21.21.21.21.1.1.m1.1.1.2" xref="A5.T1.21.21.21.21.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.21.21.21.21.1.1.m1.1.1.3" xref="A5.T1.21.21.21.21.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.21.21.21.21.1.1.m1.1b"><apply id="A5.T1.21.21.21.21.1.1.m1.1.1.cmml" xref="A5.T1.21.21.21.21.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.21.21.21.21.1.1.m1.1.1.1.cmml" xref="A5.T1.21.21.21.21.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.21.21.21.21.1.1.m1.1.1.2.cmml" xref="A5.T1.21.21.21.21.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.21.21.21.21.1.1.m1.1.1.3.cmml" xref="A5.T1.21.21.21.21.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.21.21.21.21.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> Answer: <span id="A5.T1.21.21.21.21.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="A5.T1.21.21.21.21.3" class="ltx_td ltx_align_left">Personality/Advisor/Conclusion/Hum/Assignment/Minute</td>
</tr>
<tr id="A5.T1.68.68.68.77.9" class="ltx_tr">
<td id="A5.T1.68.68.68.77.9.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A5.T1.68.68.68.77.9.1.1" class="ltx_text ltx_font_bold">CoLA</span></td>
<td id="A5.T1.68.68.68.77.9.2" class="ltx_td ltx_align_left ltx_border_t">(grammatical/not_grammatical)</td>
<td id="A5.T1.68.68.68.77.9.3" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="A5.T1.22.22.22.22" class="ltx_tr">
<td id="A5.T1.22.22.22.22.2" class="ltx_td"></td>
<td id="A5.T1.22.22.22.22.1" class="ltx_td ltx_align_left">
<span id="A5.T1.22.22.22.22.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.22.22.22.22.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.22.22.22.22.1.1.m1.1a"><msub id="A5.T1.22.22.22.22.1.1.m1.1.1" xref="A5.T1.22.22.22.22.1.1.m1.1.1.cmml"><mi id="A5.T1.22.22.22.22.1.1.m1.1.1.2" xref="A5.T1.22.22.22.22.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.22.22.22.22.1.1.m1.1.1.3" xref="A5.T1.22.22.22.22.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.22.22.22.22.1.1.m1.1b"><apply id="A5.T1.22.22.22.22.1.1.m1.1.1.cmml" xref="A5.T1.22.22.22.22.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.22.22.22.22.1.1.m1.1.1.1.cmml" xref="A5.T1.22.22.22.22.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.22.22.22.22.1.1.m1.1.1.2.cmml" xref="A5.T1.22.22.22.22.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.22.22.22.22.1.1.m1.1.1.3.cmml" xref="A5.T1.22.22.22.22.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.22.22.22.22.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> You are <span id="A5.T1.22.22.22.22.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> .</td>
<td id="A5.T1.22.22.22.22.3" class="ltx_td ltx_align_left">one/proof</td>
</tr>
<tr id="A5.T1.23.23.23.23" class="ltx_tr">
<td id="A5.T1.23.23.23.23.2" class="ltx_td"></td>
<td id="A5.T1.23.23.23.23.1" class="ltx_td ltx_align_left">It is <span id="A5.T1.23.23.23.23.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> . <span id="A5.T1.23.23.23.23.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.23.23.23.23.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.23.23.23.23.1.1.m1.1a"><msub id="A5.T1.23.23.23.23.1.1.m1.1.1" xref="A5.T1.23.23.23.23.1.1.m1.1.1.cmml"><mi id="A5.T1.23.23.23.23.1.1.m1.1.1.2" xref="A5.T1.23.23.23.23.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.23.23.23.23.1.1.m1.1.1.3" xref="A5.T1.23.23.23.23.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.23.23.23.23.1.1.m1.1b"><apply id="A5.T1.23.23.23.23.1.1.m1.1.1.cmml" xref="A5.T1.23.23.23.23.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.23.23.23.23.1.1.m1.1.1.1.cmml" xref="A5.T1.23.23.23.23.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.23.23.23.23.1.1.m1.1.1.2.cmml" xref="A5.T1.23.23.23.23.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.23.23.23.23.1.1.m1.1.1.3.cmml" xref="A5.T1.23.23.23.23.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.23.23.23.23.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.23.23.23.23.3" class="ltx_td ltx_align_left">wrong/sad</td>
</tr>
<tr id="A5.T1.24.24.24.24" class="ltx_tr">
<td id="A5.T1.24.24.24.24.2" class="ltx_td"></td>
<td id="A5.T1.24.24.24.24.1" class="ltx_td ltx_align_left">I am <span id="A5.T1.24.24.24.24.1.2" class="ltx_text ltx_font_typewriter">[MASK]</span> . <span id="A5.T1.24.24.24.24.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.24.24.24.24.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.24.24.24.24.1.1.m1.1a"><msub id="A5.T1.24.24.24.24.1.1.m1.1.1" xref="A5.T1.24.24.24.24.1.1.m1.1.1.cmml"><mi id="A5.T1.24.24.24.24.1.1.m1.1.1.2" xref="A5.T1.24.24.24.24.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.24.24.24.24.1.1.m1.1.1.3" xref="A5.T1.24.24.24.24.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.24.24.24.24.1.1.m1.1b"><apply id="A5.T1.24.24.24.24.1.1.m1.1.1.cmml" xref="A5.T1.24.24.24.24.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.24.24.24.24.1.1.m1.1.1.1.cmml" xref="A5.T1.24.24.24.24.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.24.24.24.24.1.1.m1.1.1.2.cmml" xref="A5.T1.24.24.24.24.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.24.24.24.24.1.1.m1.1.1.3.cmml" xref="A5.T1.24.24.24.24.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.24.24.24.24.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.24.24.24.24.3" class="ltx_td ltx_align_left">misleading/disappointing</td>
</tr>
<tr id="A5.T1.68.68.68.78.10" class="ltx_tr">
<td id="A5.T1.68.68.68.78.10.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A5.T1.68.68.68.78.10.1.1" class="ltx_text ltx_font_bold">MNLI</span></td>
<td id="A5.T1.68.68.68.78.10.2" class="ltx_td ltx_align_left ltx_border_t">(entailment/neutral/contradiction)</td>
<td id="A5.T1.68.68.68.78.10.3" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="A5.T1.26.26.26.26" class="ltx_tr">
<td id="A5.T1.26.26.26.26.3" class="ltx_td"></td>
<td id="A5.T1.26.26.26.26.2" class="ltx_td ltx_align_left">
<span id="A5.T1.25.25.25.25.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.25.25.25.25.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.25.25.25.25.1.1.m1.1a"><msub id="A5.T1.25.25.25.25.1.1.m1.1.1" xref="A5.T1.25.25.25.25.1.1.m1.1.1.cmml"><mi id="A5.T1.25.25.25.25.1.1.m1.1.1.2" xref="A5.T1.25.25.25.25.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.25.25.25.25.1.1.m1.1.1.3" xref="A5.T1.25.25.25.25.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.25.25.25.25.1.1.m1.1b"><apply id="A5.T1.25.25.25.25.1.1.m1.1.1.cmml" xref="A5.T1.25.25.25.25.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.25.25.25.25.1.1.m1.1.1.1.cmml" xref="A5.T1.25.25.25.25.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.25.25.25.25.1.1.m1.1.1.2.cmml" xref="A5.T1.25.25.25.25.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.25.25.25.25.1.1.m1.1.1.3.cmml" xref="A5.T1.25.25.25.25.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.25.25.25.25.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="A5.T1.26.26.26.26.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , you are right , <span id="A5.T1.26.26.26.26.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.26.26.26.26.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.26.26.26.26.2.2.m1.1a"><msub id="A5.T1.26.26.26.26.2.2.m1.1.1" xref="A5.T1.26.26.26.26.2.2.m1.1.1.cmml"><mi id="A5.T1.26.26.26.26.2.2.m1.1.1.2" xref="A5.T1.26.26.26.26.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.26.26.26.26.2.2.m1.1.1.3" xref="A5.T1.26.26.26.26.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.26.26.26.26.2.2.m1.1b"><apply id="A5.T1.26.26.26.26.2.2.m1.1.1.cmml" xref="A5.T1.26.26.26.26.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.26.26.26.26.2.2.m1.1.1.1.cmml" xref="A5.T1.26.26.26.26.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.26.26.26.26.2.2.m1.1.1.2.cmml" xref="A5.T1.26.26.26.26.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.26.26.26.26.2.2.m1.1.1.3.cmml" xref="A5.T1.26.26.26.26.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.26.26.26.26.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.26.26.26.26.4" class="ltx_td ltx_align_left">Fine/Plus/Otherwise</td>
</tr>
<tr id="A5.T1.28.28.28.28" class="ltx_tr">
<td id="A5.T1.28.28.28.28.3" class="ltx_td"></td>
<td id="A5.T1.28.28.28.28.2" class="ltx_td ltx_align_left">
<span id="A5.T1.27.27.27.27.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.27.27.27.27.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.27.27.27.27.1.1.m1.1a"><msub id="A5.T1.27.27.27.27.1.1.m1.1.1" xref="A5.T1.27.27.27.27.1.1.m1.1.1.cmml"><mi id="A5.T1.27.27.27.27.1.1.m1.1.1.2" xref="A5.T1.27.27.27.27.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.27.27.27.27.1.1.m1.1.1.3" xref="A5.T1.27.27.27.27.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.27.27.27.27.1.1.m1.1b"><apply id="A5.T1.27.27.27.27.1.1.m1.1.1.cmml" xref="A5.T1.27.27.27.27.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.27.27.27.27.1.1.m1.1.1.1.cmml" xref="A5.T1.27.27.27.27.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.27.27.27.27.1.1.m1.1.1.2.cmml" xref="A5.T1.27.27.27.27.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.27.27.27.27.1.1.m1.1.1.3.cmml" xref="A5.T1.27.27.27.27.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.27.27.27.27.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="A5.T1.28.28.28.28.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> you’re right <span id="A5.T1.28.28.28.28.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.28.28.28.28.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.28.28.28.28.2.2.m1.1a"><msub id="A5.T1.28.28.28.28.2.2.m1.1.1" xref="A5.T1.28.28.28.28.2.2.m1.1.1.cmml"><mi id="A5.T1.28.28.28.28.2.2.m1.1.1.2" xref="A5.T1.28.28.28.28.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.28.28.28.28.2.2.m1.1.1.3" xref="A5.T1.28.28.28.28.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.28.28.28.28.2.2.m1.1b"><apply id="A5.T1.28.28.28.28.2.2.m1.1.1.cmml" xref="A5.T1.28.28.28.28.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.28.28.28.28.2.2.m1.1.1.1.cmml" xref="A5.T1.28.28.28.28.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.28.28.28.28.2.2.m1.1.1.2.cmml" xref="A5.T1.28.28.28.28.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.28.28.28.28.2.2.m1.1.1.3.cmml" xref="A5.T1.28.28.28.28.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.28.28.28.28.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.28.28.28.28.4" class="ltx_td ltx_align_left">There/Plus/Otherwise</td>
</tr>
<tr id="A5.T1.30.30.30.30" class="ltx_tr">
<td id="A5.T1.30.30.30.30.3" class="ltx_td"></td>
<td id="A5.T1.30.30.30.30.2" class="ltx_td ltx_align_left">
<span id="A5.T1.29.29.29.29.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.29.29.29.29.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.29.29.29.29.1.1.m1.1a"><msub id="A5.T1.29.29.29.29.1.1.m1.1.1" xref="A5.T1.29.29.29.29.1.1.m1.1.1.cmml"><mi id="A5.T1.29.29.29.29.1.1.m1.1.1.2" xref="A5.T1.29.29.29.29.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.29.29.29.29.1.1.m1.1.1.3" xref="A5.T1.29.29.29.29.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.29.29.29.29.1.1.m1.1b"><apply id="A5.T1.29.29.29.29.1.1.m1.1.1.cmml" xref="A5.T1.29.29.29.29.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.29.29.29.29.1.1.m1.1.1.1.cmml" xref="A5.T1.29.29.29.29.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.29.29.29.29.1.1.m1.1.1.2.cmml" xref="A5.T1.29.29.29.29.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.29.29.29.29.1.1.m1.1.1.3.cmml" xref="A5.T1.29.29.29.29.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.29.29.29.29.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="A5.T1.30.30.30.30.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> ! <span id="A5.T1.30.30.30.30.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.30.30.30.30.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.30.30.30.30.2.2.m1.1a"><msub id="A5.T1.30.30.30.30.2.2.m1.1.1" xref="A5.T1.30.30.30.30.2.2.m1.1.1.cmml"><mi id="A5.T1.30.30.30.30.2.2.m1.1.1.2" xref="A5.T1.30.30.30.30.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.30.30.30.30.2.2.m1.1.1.3" xref="A5.T1.30.30.30.30.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.30.30.30.30.2.2.m1.1b"><apply id="A5.T1.30.30.30.30.2.2.m1.1.1.cmml" xref="A5.T1.30.30.30.30.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.30.30.30.30.2.2.m1.1.1.1.cmml" xref="A5.T1.30.30.30.30.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.30.30.30.30.2.2.m1.1.1.2.cmml" xref="A5.T1.30.30.30.30.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.30.30.30.30.2.2.m1.1.1.3.cmml" xref="A5.T1.30.30.30.30.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.30.30.30.30.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.30.30.30.30.4" class="ltx_td ltx_align_left">Meaning/Plus/Otherwise</td>
</tr>
<tr id="A5.T1.68.68.68.79.11" class="ltx_tr">
<td id="A5.T1.68.68.68.79.11.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A5.T1.68.68.68.79.11.1.1" class="ltx_text ltx_font_bold">SNLI</span></td>
<td id="A5.T1.68.68.68.79.11.2" class="ltx_td ltx_align_left ltx_border_t">(entailment/neutral/contradiction)</td>
<td id="A5.T1.68.68.68.79.11.3" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="A5.T1.32.32.32.32" class="ltx_tr">
<td id="A5.T1.32.32.32.32.3" class="ltx_td"></td>
<td id="A5.T1.32.32.32.32.2" class="ltx_td ltx_align_left">
<span id="A5.T1.31.31.31.31.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.31.31.31.31.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.31.31.31.31.1.1.m1.1a"><msub id="A5.T1.31.31.31.31.1.1.m1.1.1" xref="A5.T1.31.31.31.31.1.1.m1.1.1.cmml"><mi id="A5.T1.31.31.31.31.1.1.m1.1.1.2" xref="A5.T1.31.31.31.31.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.31.31.31.31.1.1.m1.1.1.3" xref="A5.T1.31.31.31.31.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.31.31.31.31.1.1.m1.1b"><apply id="A5.T1.31.31.31.31.1.1.m1.1.1.cmml" xref="A5.T1.31.31.31.31.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.31.31.31.31.1.1.m1.1.1.1.cmml" xref="A5.T1.31.31.31.31.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.31.31.31.31.1.1.m1.1.1.2.cmml" xref="A5.T1.31.31.31.31.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.31.31.31.31.1.1.m1.1.1.3.cmml" xref="A5.T1.31.31.31.31.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.31.31.31.31.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="A5.T1.32.32.32.32.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , no , <span id="A5.T1.32.32.32.32.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.32.32.32.32.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.32.32.32.32.2.2.m1.1a"><msub id="A5.T1.32.32.32.32.2.2.m1.1.1" xref="A5.T1.32.32.32.32.2.2.m1.1.1.cmml"><mi id="A5.T1.32.32.32.32.2.2.m1.1.1.2" xref="A5.T1.32.32.32.32.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.32.32.32.32.2.2.m1.1.1.3" xref="A5.T1.32.32.32.32.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.32.32.32.32.2.2.m1.1b"><apply id="A5.T1.32.32.32.32.2.2.m1.1.1.cmml" xref="A5.T1.32.32.32.32.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.32.32.32.32.2.2.m1.1.1.1.cmml" xref="A5.T1.32.32.32.32.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.32.32.32.32.2.2.m1.1.1.2.cmml" xref="A5.T1.32.32.32.32.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.32.32.32.32.2.2.m1.1.1.3.cmml" xref="A5.T1.32.32.32.32.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.32.32.32.32.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.32.32.32.32.4" class="ltx_td ltx_align_left">Alright/Watch/Except</td>
</tr>
<tr id="A5.T1.34.34.34.34" class="ltx_tr">
<td id="A5.T1.34.34.34.34.3" class="ltx_td"></td>
<td id="A5.T1.34.34.34.34.2" class="ltx_td ltx_align_left">
<span id="A5.T1.33.33.33.33.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.33.33.33.33.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.33.33.33.33.1.1.m1.1a"><msub id="A5.T1.33.33.33.33.1.1.m1.1.1" xref="A5.T1.33.33.33.33.1.1.m1.1.1.cmml"><mi id="A5.T1.33.33.33.33.1.1.m1.1.1.2" xref="A5.T1.33.33.33.33.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.33.33.33.33.1.1.m1.1.1.3" xref="A5.T1.33.33.33.33.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.33.33.33.33.1.1.m1.1b"><apply id="A5.T1.33.33.33.33.1.1.m1.1.1.cmml" xref="A5.T1.33.33.33.33.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.33.33.33.33.1.1.m1.1.1.1.cmml" xref="A5.T1.33.33.33.33.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.33.33.33.33.1.1.m1.1.1.2.cmml" xref="A5.T1.33.33.33.33.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.33.33.33.33.1.1.m1.1.1.3.cmml" xref="A5.T1.33.33.33.33.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.33.33.33.33.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="A5.T1.34.34.34.34.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , in this case <span id="A5.T1.34.34.34.34.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.34.34.34.34.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.34.34.34.34.2.2.m1.1a"><msub id="A5.T1.34.34.34.34.2.2.m1.1.1" xref="A5.T1.34.34.34.34.2.2.m1.1.1.cmml"><mi id="A5.T1.34.34.34.34.2.2.m1.1.1.2" xref="A5.T1.34.34.34.34.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.34.34.34.34.2.2.m1.1.1.3" xref="A5.T1.34.34.34.34.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.34.34.34.34.2.2.m1.1b"><apply id="A5.T1.34.34.34.34.2.2.m1.1.1.cmml" xref="A5.T1.34.34.34.34.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.34.34.34.34.2.2.m1.1.1.1.cmml" xref="A5.T1.34.34.34.34.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.34.34.34.34.2.2.m1.1.1.2.cmml" xref="A5.T1.34.34.34.34.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.34.34.34.34.2.2.m1.1.1.3.cmml" xref="A5.T1.34.34.34.34.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.34.34.34.34.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.34.34.34.34.4" class="ltx_td ltx_align_left">Hi/Watch/Worse</td>
</tr>
<tr id="A5.T1.36.36.36.36" class="ltx_tr">
<td id="A5.T1.36.36.36.36.3" class="ltx_td"></td>
<td id="A5.T1.36.36.36.36.2" class="ltx_td ltx_align_left">
<span id="A5.T1.35.35.35.35.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.35.35.35.35.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.35.35.35.35.1.1.m1.1a"><msub id="A5.T1.35.35.35.35.1.1.m1.1.1" xref="A5.T1.35.35.35.35.1.1.m1.1.1.cmml"><mi id="A5.T1.35.35.35.35.1.1.m1.1.1.2" xref="A5.T1.35.35.35.35.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.35.35.35.35.1.1.m1.1.1.3" xref="A5.T1.35.35.35.35.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.35.35.35.35.1.1.m1.1b"><apply id="A5.T1.35.35.35.35.1.1.m1.1.1.cmml" xref="A5.T1.35.35.35.35.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.35.35.35.35.1.1.m1.1.1.1.cmml" xref="A5.T1.35.35.35.35.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.35.35.35.35.1.1.m1.1.1.2.cmml" xref="A5.T1.35.35.35.35.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.35.35.35.35.1.1.m1.1.1.3.cmml" xref="A5.T1.35.35.35.35.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.35.35.35.35.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="A5.T1.36.36.36.36.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> this time <span id="A5.T1.36.36.36.36.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.36.36.36.36.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.36.36.36.36.2.2.m1.1a"><msub id="A5.T1.36.36.36.36.2.2.m1.1.1" xref="A5.T1.36.36.36.36.2.2.m1.1.1.cmml"><mi id="A5.T1.36.36.36.36.2.2.m1.1.1.2" xref="A5.T1.36.36.36.36.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.36.36.36.36.2.2.m1.1.1.3" xref="A5.T1.36.36.36.36.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.36.36.36.36.2.2.m1.1b"><apply id="A5.T1.36.36.36.36.2.2.m1.1.1.cmml" xref="A5.T1.36.36.36.36.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.36.36.36.36.2.2.m1.1.1.1.cmml" xref="A5.T1.36.36.36.36.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.36.36.36.36.2.2.m1.1.1.2.cmml" xref="A5.T1.36.36.36.36.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.36.36.36.36.2.2.m1.1.1.3.cmml" xref="A5.T1.36.36.36.36.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.36.36.36.36.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.36.36.36.36.4" class="ltx_td ltx_align_left">Regardless/Fortunately/Unless</td>
</tr>
<tr id="A5.T1.68.68.68.80.12" class="ltx_tr">
<td id="A5.T1.68.68.68.80.12.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A5.T1.68.68.68.80.12.1.1" class="ltx_text ltx_font_bold">QNLI</span></td>
<td id="A5.T1.68.68.68.80.12.2" class="ltx_td ltx_align_left ltx_border_t">(entailment/not_entailment)</td>
<td id="A5.T1.68.68.68.80.12.3" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="A5.T1.38.38.38.38" class="ltx_tr">
<td id="A5.T1.38.38.38.38.3" class="ltx_td"></td>
<td id="A5.T1.38.38.38.38.2" class="ltx_td ltx_align_left">
<span id="A5.T1.37.37.37.37.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.37.37.37.37.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.37.37.37.37.1.1.m1.1a"><msub id="A5.T1.37.37.37.37.1.1.m1.1.1" xref="A5.T1.37.37.37.37.1.1.m1.1.1.cmml"><mi id="A5.T1.37.37.37.37.1.1.m1.1.1.2" xref="A5.T1.37.37.37.37.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.37.37.37.37.1.1.m1.1.1.3" xref="A5.T1.37.37.37.37.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.37.37.37.37.1.1.m1.1b"><apply id="A5.T1.37.37.37.37.1.1.m1.1.1.cmml" xref="A5.T1.37.37.37.37.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.37.37.37.37.1.1.m1.1.1.1.cmml" xref="A5.T1.37.37.37.37.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.37.37.37.37.1.1.m1.1.1.2.cmml" xref="A5.T1.37.37.37.37.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.37.37.37.37.1.1.m1.1.1.3.cmml" xref="A5.T1.37.37.37.37.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.37.37.37.37.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> ? <span id="A5.T1.38.38.38.38.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> . Yes , <span id="A5.T1.38.38.38.38.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.38.38.38.38.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.38.38.38.38.2.2.m1.1a"><msub id="A5.T1.38.38.38.38.2.2.m1.1.1" xref="A5.T1.38.38.38.38.2.2.m1.1.1.cmml"><mi id="A5.T1.38.38.38.38.2.2.m1.1.1.2" xref="A5.T1.38.38.38.38.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.38.38.38.38.2.2.m1.1.1.3" xref="A5.T1.38.38.38.38.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.38.38.38.38.2.2.m1.1b"><apply id="A5.T1.38.38.38.38.2.2.m1.1.1.cmml" xref="A5.T1.38.38.38.38.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.38.38.38.38.2.2.m1.1.1.1.cmml" xref="A5.T1.38.38.38.38.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.38.38.38.38.2.2.m1.1.1.2.cmml" xref="A5.T1.38.38.38.38.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.38.38.38.38.2.2.m1.1.1.3.cmml" xref="A5.T1.38.38.38.38.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.38.38.38.38.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.38.38.38.38.4" class="ltx_td ltx_align_left">Okay/Nonetheless</td>
</tr>
<tr id="A5.T1.40.40.40.40" class="ltx_tr">
<td id="A5.T1.40.40.40.40.3" class="ltx_td"></td>
<td id="A5.T1.40.40.40.40.2" class="ltx_td ltx_align_left">
<span id="A5.T1.39.39.39.39.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.39.39.39.39.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.39.39.39.39.1.1.m1.1a"><msub id="A5.T1.39.39.39.39.1.1.m1.1.1" xref="A5.T1.39.39.39.39.1.1.m1.1.1.cmml"><mi id="A5.T1.39.39.39.39.1.1.m1.1.1.2" xref="A5.T1.39.39.39.39.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.39.39.39.39.1.1.m1.1.1.3" xref="A5.T1.39.39.39.39.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.39.39.39.39.1.1.m1.1b"><apply id="A5.T1.39.39.39.39.1.1.m1.1.1.cmml" xref="A5.T1.39.39.39.39.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.39.39.39.39.1.1.m1.1.1.1.cmml" xref="A5.T1.39.39.39.39.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.39.39.39.39.1.1.m1.1.1.2.cmml" xref="A5.T1.39.39.39.39.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.39.39.39.39.1.1.m1.1.1.3.cmml" xref="A5.T1.39.39.39.39.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.39.39.39.39.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> ? <span id="A5.T1.40.40.40.40.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> . It is known that <span id="A5.T1.40.40.40.40.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.40.40.40.40.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.40.40.40.40.2.2.m1.1a"><msub id="A5.T1.40.40.40.40.2.2.m1.1.1" xref="A5.T1.40.40.40.40.2.2.m1.1.1.cmml"><mi id="A5.T1.40.40.40.40.2.2.m1.1.1.2" xref="A5.T1.40.40.40.40.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.40.40.40.40.2.2.m1.1.1.3" xref="A5.T1.40.40.40.40.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.40.40.40.40.2.2.m1.1b"><apply id="A5.T1.40.40.40.40.2.2.m1.1.1.cmml" xref="A5.T1.40.40.40.40.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.40.40.40.40.2.2.m1.1.1.1.cmml" xref="A5.T1.40.40.40.40.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.40.40.40.40.2.2.m1.1.1.2.cmml" xref="A5.T1.40.40.40.40.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.40.40.40.40.2.2.m1.1.1.3.cmml" xref="A5.T1.40.40.40.40.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.40.40.40.40.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.40.40.40.40.4" class="ltx_td ltx_align_left">Notably/Yet</td>
</tr>
<tr id="A5.T1.42.42.42.42" class="ltx_tr">
<td id="A5.T1.42.42.42.42.3" class="ltx_td"></td>
<td id="A5.T1.42.42.42.42.2" class="ltx_td ltx_align_left">
<span id="A5.T1.41.41.41.41.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.41.41.41.41.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.41.41.41.41.1.1.m1.1a"><msub id="A5.T1.41.41.41.41.1.1.m1.1.1" xref="A5.T1.41.41.41.41.1.1.m1.1.1.cmml"><mi id="A5.T1.41.41.41.41.1.1.m1.1.1.2" xref="A5.T1.41.41.41.41.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.41.41.41.41.1.1.m1.1.1.3" xref="A5.T1.41.41.41.41.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.41.41.41.41.1.1.m1.1b"><apply id="A5.T1.41.41.41.41.1.1.m1.1.1.cmml" xref="A5.T1.41.41.41.41.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.41.41.41.41.1.1.m1.1.1.1.cmml" xref="A5.T1.41.41.41.41.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.41.41.41.41.1.1.m1.1.1.2.cmml" xref="A5.T1.41.41.41.41.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.41.41.41.41.1.1.m1.1.1.3.cmml" xref="A5.T1.41.41.41.41.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.41.41.41.41.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> ? <span id="A5.T1.42.42.42.42.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , however , <span id="A5.T1.42.42.42.42.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.42.42.42.42.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.42.42.42.42.2.2.m1.1a"><msub id="A5.T1.42.42.42.42.2.2.m1.1.1" xref="A5.T1.42.42.42.42.2.2.m1.1.1.cmml"><mi id="A5.T1.42.42.42.42.2.2.m1.1.1.2" xref="A5.T1.42.42.42.42.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.42.42.42.42.2.2.m1.1.1.3" xref="A5.T1.42.42.42.42.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.42.42.42.42.2.2.m1.1b"><apply id="A5.T1.42.42.42.42.2.2.m1.1.1.cmml" xref="A5.T1.42.42.42.42.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.42.42.42.42.2.2.m1.1.1.1.cmml" xref="A5.T1.42.42.42.42.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.42.42.42.42.2.2.m1.1.1.2.cmml" xref="A5.T1.42.42.42.42.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.42.42.42.42.2.2.m1.1.1.3.cmml" xref="A5.T1.42.42.42.42.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.42.42.42.42.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.42.42.42.42.4" class="ltx_td ltx_align_left">Specifically/Notably</td>
</tr>
<tr id="A5.T1.68.68.68.81.13" class="ltx_tr">
<td id="A5.T1.68.68.68.81.13.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A5.T1.68.68.68.81.13.1.1" class="ltx_text ltx_font_bold">RTE</span></td>
<td id="A5.T1.68.68.68.81.13.2" class="ltx_td ltx_align_left ltx_border_t">(entailment/not_entailment)</td>
<td id="A5.T1.68.68.68.81.13.3" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="A5.T1.44.44.44.44" class="ltx_tr">
<td id="A5.T1.44.44.44.44.3" class="ltx_td"></td>
<td id="A5.T1.44.44.44.44.2" class="ltx_td ltx_align_left">
<span id="A5.T1.43.43.43.43.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.43.43.43.43.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.43.43.43.43.1.1.m1.1a"><msub id="A5.T1.43.43.43.43.1.1.m1.1.1" xref="A5.T1.43.43.43.43.1.1.m1.1.1.cmml"><mi id="A5.T1.43.43.43.43.1.1.m1.1.1.2" xref="A5.T1.43.43.43.43.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.43.43.43.43.1.1.m1.1.1.3" xref="A5.T1.43.43.43.43.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.43.43.43.43.1.1.m1.1b"><apply id="A5.T1.43.43.43.43.1.1.m1.1.1.cmml" xref="A5.T1.43.43.43.43.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.43.43.43.43.1.1.m1.1.1.1.cmml" xref="A5.T1.43.43.43.43.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.43.43.43.43.1.1.m1.1.1.2.cmml" xref="A5.T1.43.43.43.43.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.43.43.43.43.1.1.m1.1.1.3.cmml" xref="A5.T1.43.43.43.43.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.43.43.43.43.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="A5.T1.44.44.44.44.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , I believe <span id="A5.T1.44.44.44.44.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.44.44.44.44.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.44.44.44.44.2.2.m1.1a"><msub id="A5.T1.44.44.44.44.2.2.m1.1.1" xref="A5.T1.44.44.44.44.2.2.m1.1.1.cmml"><mi id="A5.T1.44.44.44.44.2.2.m1.1.1.2" xref="A5.T1.44.44.44.44.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.44.44.44.44.2.2.m1.1.1.3" xref="A5.T1.44.44.44.44.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.44.44.44.44.2.2.m1.1b"><apply id="A5.T1.44.44.44.44.2.2.m1.1.1.cmml" xref="A5.T1.44.44.44.44.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.44.44.44.44.2.2.m1.1.1.1.cmml" xref="A5.T1.44.44.44.44.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.44.44.44.44.2.2.m1.1.1.2.cmml" xref="A5.T1.44.44.44.44.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.44.44.44.44.2.2.m1.1.1.3.cmml" xref="A5.T1.44.44.44.44.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.44.44.44.44.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.44.44.44.44.4" class="ltx_td ltx_align_left">Clearly/Yet</td>
</tr>
<tr id="A5.T1.46.46.46.46" class="ltx_tr">
<td id="A5.T1.46.46.46.46.3" class="ltx_td"></td>
<td id="A5.T1.46.46.46.46.2" class="ltx_td ltx_align_left">
<span id="A5.T1.45.45.45.45.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.45.45.45.45.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.45.45.45.45.1.1.m1.1a"><msub id="A5.T1.45.45.45.45.1.1.m1.1.1" xref="A5.T1.45.45.45.45.1.1.m1.1.1.cmml"><mi id="A5.T1.45.45.45.45.1.1.m1.1.1.2" xref="A5.T1.45.45.45.45.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.45.45.45.45.1.1.m1.1.1.3" xref="A5.T1.45.45.45.45.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.45.45.45.45.1.1.m1.1b"><apply id="A5.T1.45.45.45.45.1.1.m1.1.1.cmml" xref="A5.T1.45.45.45.45.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.45.45.45.45.1.1.m1.1.1.1.cmml" xref="A5.T1.45.45.45.45.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.45.45.45.45.1.1.m1.1.1.2.cmml" xref="A5.T1.45.45.45.45.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.45.45.45.45.1.1.m1.1.1.3.cmml" xref="A5.T1.45.45.45.45.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.45.45.45.45.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="A5.T1.46.46.46.46.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , I think that <span id="A5.T1.46.46.46.46.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.46.46.46.46.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.46.46.46.46.2.2.m1.1a"><msub id="A5.T1.46.46.46.46.2.2.m1.1.1" xref="A5.T1.46.46.46.46.2.2.m1.1.1.cmml"><mi id="A5.T1.46.46.46.46.2.2.m1.1.1.2" xref="A5.T1.46.46.46.46.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.46.46.46.46.2.2.m1.1.1.3" xref="A5.T1.46.46.46.46.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.46.46.46.46.2.2.m1.1b"><apply id="A5.T1.46.46.46.46.2.2.m1.1.1.cmml" xref="A5.T1.46.46.46.46.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.46.46.46.46.2.2.m1.1.1.1.cmml" xref="A5.T1.46.46.46.46.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.46.46.46.46.2.2.m1.1.1.2.cmml" xref="A5.T1.46.46.46.46.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.46.46.46.46.2.2.m1.1.1.3.cmml" xref="A5.T1.46.46.46.46.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.46.46.46.46.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.46.46.46.46.4" class="ltx_td ltx_align_left">Accordingly/meanwhile</td>
</tr>
<tr id="A5.T1.48.48.48.48" class="ltx_tr">
<td id="A5.T1.48.48.48.48.3" class="ltx_td"></td>
<td id="A5.T1.48.48.48.48.2" class="ltx_td ltx_align_left">
<span id="A5.T1.47.47.47.47.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.47.47.47.47.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.47.47.47.47.1.1.m1.1a"><msub id="A5.T1.47.47.47.47.1.1.m1.1.1" xref="A5.T1.47.47.47.47.1.1.m1.1.1.cmml"><mi id="A5.T1.47.47.47.47.1.1.m1.1.1.2" xref="A5.T1.47.47.47.47.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.47.47.47.47.1.1.m1.1.1.3" xref="A5.T1.47.47.47.47.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.47.47.47.47.1.1.m1.1b"><apply id="A5.T1.47.47.47.47.1.1.m1.1.1.cmml" xref="A5.T1.47.47.47.47.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.47.47.47.47.1.1.m1.1.1.1.cmml" xref="A5.T1.47.47.47.47.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.47.47.47.47.1.1.m1.1.1.2.cmml" xref="A5.T1.47.47.47.47.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.47.47.47.47.1.1.m1.1.1.3.cmml" xref="A5.T1.47.47.47.47.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.47.47.47.47.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="A5.T1.48.48.48.48.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , I think <span id="A5.T1.48.48.48.48.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.48.48.48.48.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.48.48.48.48.2.2.m1.1a"><msub id="A5.T1.48.48.48.48.2.2.m1.1.1" xref="A5.T1.48.48.48.48.2.2.m1.1.1.cmml"><mi id="A5.T1.48.48.48.48.2.2.m1.1.1.2" xref="A5.T1.48.48.48.48.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.48.48.48.48.2.2.m1.1.1.3" xref="A5.T1.48.48.48.48.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.48.48.48.48.2.2.m1.1b"><apply id="A5.T1.48.48.48.48.2.2.m1.1.1.cmml" xref="A5.T1.48.48.48.48.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.48.48.48.48.2.2.m1.1.1.1.cmml" xref="A5.T1.48.48.48.48.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.48.48.48.48.2.2.m1.1.1.2.cmml" xref="A5.T1.48.48.48.48.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.48.48.48.48.2.2.m1.1.1.3.cmml" xref="A5.T1.48.48.48.48.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.48.48.48.48.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.48.48.48.48.4" class="ltx_td ltx_align_left">So/Meanwhile</td>
</tr>
<tr id="A5.T1.68.68.68.82.14" class="ltx_tr">
<td id="A5.T1.68.68.68.82.14.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A5.T1.68.68.68.82.14.1.1" class="ltx_text ltx_font_bold">MRPC</span></td>
<td id="A5.T1.68.68.68.82.14.2" class="ltx_td ltx_align_left ltx_border_t">(equivalent/not_equivalent)</td>
<td id="A5.T1.68.68.68.82.14.3" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="A5.T1.50.50.50.50" class="ltx_tr">
<td id="A5.T1.50.50.50.50.3" class="ltx_td"></td>
<td id="A5.T1.50.50.50.50.2" class="ltx_td ltx_align_left">
<span id="A5.T1.49.49.49.49.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.49.49.49.49.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.49.49.49.49.1.1.m1.1a"><msub id="A5.T1.49.49.49.49.1.1.m1.1.1" xref="A5.T1.49.49.49.49.1.1.m1.1.1.cmml"><mi id="A5.T1.49.49.49.49.1.1.m1.1.1.2" xref="A5.T1.49.49.49.49.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.49.49.49.49.1.1.m1.1.1.3" xref="A5.T1.49.49.49.49.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.49.49.49.49.1.1.m1.1b"><apply id="A5.T1.49.49.49.49.1.1.m1.1.1.cmml" xref="A5.T1.49.49.49.49.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.49.49.49.49.1.1.m1.1.1.1.cmml" xref="A5.T1.49.49.49.49.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.49.49.49.49.1.1.m1.1.1.2.cmml" xref="A5.T1.49.49.49.49.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.49.49.49.49.1.1.m1.1.1.3.cmml" xref="A5.T1.49.49.49.49.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.49.49.49.49.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="A5.T1.50.50.50.50.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> ! <span id="A5.T1.50.50.50.50.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.50.50.50.50.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.50.50.50.50.2.2.m1.1a"><msub id="A5.T1.50.50.50.50.2.2.m1.1.1" xref="A5.T1.50.50.50.50.2.2.m1.1.1.cmml"><mi id="A5.T1.50.50.50.50.2.2.m1.1.1.2" xref="A5.T1.50.50.50.50.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.50.50.50.50.2.2.m1.1.1.3" xref="A5.T1.50.50.50.50.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.50.50.50.50.2.2.m1.1b"><apply id="A5.T1.50.50.50.50.2.2.m1.1.1.cmml" xref="A5.T1.50.50.50.50.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.50.50.50.50.2.2.m1.1.1.1.cmml" xref="A5.T1.50.50.50.50.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.50.50.50.50.2.2.m1.1.1.2.cmml" xref="A5.T1.50.50.50.50.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.50.50.50.50.2.2.m1.1.1.3.cmml" xref="A5.T1.50.50.50.50.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.50.50.50.50.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.50.50.50.50.4" class="ltx_td ltx_align_left">Rather/Alas</td>
</tr>
<tr id="A5.T1.52.52.52.52" class="ltx_tr">
<td id="A5.T1.52.52.52.52.3" class="ltx_td"></td>
<td id="A5.T1.52.52.52.52.2" class="ltx_td ltx_align_left">
<span id="A5.T1.51.51.51.51.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.51.51.51.51.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.51.51.51.51.1.1.m1.1a"><msub id="A5.T1.51.51.51.51.1.1.m1.1.1" xref="A5.T1.51.51.51.51.1.1.m1.1.1.cmml"><mi id="A5.T1.51.51.51.51.1.1.m1.1.1.2" xref="A5.T1.51.51.51.51.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.51.51.51.51.1.1.m1.1.1.3" xref="A5.T1.51.51.51.51.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.51.51.51.51.1.1.m1.1b"><apply id="A5.T1.51.51.51.51.1.1.m1.1.1.cmml" xref="A5.T1.51.51.51.51.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.51.51.51.51.1.1.m1.1.1.1.cmml" xref="A5.T1.51.51.51.51.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.51.51.51.51.1.1.m1.1.1.2.cmml" xref="A5.T1.51.51.51.51.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.51.51.51.51.1.1.m1.1.1.3.cmml" xref="A5.T1.51.51.51.51.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.51.51.51.51.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="A5.T1.52.52.52.52.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> . This is the first time <span id="A5.T1.52.52.52.52.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.52.52.52.52.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.52.52.52.52.2.2.m1.1a"><msub id="A5.T1.52.52.52.52.2.2.m1.1.1" xref="A5.T1.52.52.52.52.2.2.m1.1.1.cmml"><mi id="A5.T1.52.52.52.52.2.2.m1.1.1.2" xref="A5.T1.52.52.52.52.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.52.52.52.52.2.2.m1.1.1.3" xref="A5.T1.52.52.52.52.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.52.52.52.52.2.2.m1.1b"><apply id="A5.T1.52.52.52.52.2.2.m1.1.1.cmml" xref="A5.T1.52.52.52.52.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.52.52.52.52.2.2.m1.1.1.1.cmml" xref="A5.T1.52.52.52.52.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.52.52.52.52.2.2.m1.1.1.2.cmml" xref="A5.T1.52.52.52.52.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.52.52.52.52.2.2.m1.1.1.3.cmml" xref="A5.T1.52.52.52.52.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.52.52.52.52.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.52.52.52.52.4" class="ltx_td ltx_align_left">At/Thus</td>
</tr>
<tr id="A5.T1.54.54.54.54" class="ltx_tr">
<td id="A5.T1.54.54.54.54.3" class="ltx_td"></td>
<td id="A5.T1.54.54.54.54.2" class="ltx_td ltx_align_left">
<span id="A5.T1.53.53.53.53.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.53.53.53.53.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.53.53.53.53.1.1.m1.1a"><msub id="A5.T1.53.53.53.53.1.1.m1.1.1" xref="A5.T1.53.53.53.53.1.1.m1.1.1.cmml"><mi id="A5.T1.53.53.53.53.1.1.m1.1.1.2" xref="A5.T1.53.53.53.53.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.53.53.53.53.1.1.m1.1.1.3" xref="A5.T1.53.53.53.53.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.53.53.53.53.1.1.m1.1b"><apply id="A5.T1.53.53.53.53.1.1.m1.1.1.cmml" xref="A5.T1.53.53.53.53.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.53.53.53.53.1.1.m1.1.1.1.cmml" xref="A5.T1.53.53.53.53.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.53.53.53.53.1.1.m1.1.1.2.cmml" xref="A5.T1.53.53.53.53.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.53.53.53.53.1.1.m1.1.1.3.cmml" xref="A5.T1.53.53.53.53.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.53.53.53.53.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="A5.T1.54.54.54.54.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> . That’s right . <span id="A5.T1.54.54.54.54.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.54.54.54.54.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.54.54.54.54.2.2.m1.1a"><msub id="A5.T1.54.54.54.54.2.2.m1.1.1" xref="A5.T1.54.54.54.54.2.2.m1.1.1.cmml"><mi id="A5.T1.54.54.54.54.2.2.m1.1.1.2" xref="A5.T1.54.54.54.54.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.54.54.54.54.2.2.m1.1.1.3" xref="A5.T1.54.54.54.54.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.54.54.54.54.2.2.m1.1b"><apply id="A5.T1.54.54.54.54.2.2.m1.1.1.cmml" xref="A5.T1.54.54.54.54.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.54.54.54.54.2.2.m1.1.1.1.cmml" xref="A5.T1.54.54.54.54.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.54.54.54.54.2.2.m1.1.1.2.cmml" xref="A5.T1.54.54.54.54.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.54.54.54.54.2.2.m1.1.1.3.cmml" xref="A5.T1.54.54.54.54.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.54.54.54.54.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.54.54.54.54.4" class="ltx_td ltx_align_left">Instead/Moreover</td>
</tr>
<tr id="A5.T1.68.68.68.83.15" class="ltx_tr">
<td id="A5.T1.68.68.68.83.15.1" class="ltx_td ltx_align_left ltx_border_t"><span id="A5.T1.68.68.68.83.15.1.1" class="ltx_text ltx_font_bold">QQP</span></td>
<td id="A5.T1.68.68.68.83.15.2" class="ltx_td ltx_align_left ltx_border_t">(equivalent/not_equivalent)</td>
<td id="A5.T1.68.68.68.83.15.3" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="A5.T1.56.56.56.56" class="ltx_tr">
<td id="A5.T1.56.56.56.56.3" class="ltx_td"></td>
<td id="A5.T1.56.56.56.56.2" class="ltx_td ltx_align_left">
<span id="A5.T1.55.55.55.55.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.55.55.55.55.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.55.55.55.55.1.1.m1.1a"><msub id="A5.T1.55.55.55.55.1.1.m1.1.1" xref="A5.T1.55.55.55.55.1.1.m1.1.1.cmml"><mi id="A5.T1.55.55.55.55.1.1.m1.1.1.2" xref="A5.T1.55.55.55.55.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.55.55.55.55.1.1.m1.1.1.3" xref="A5.T1.55.55.55.55.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.55.55.55.55.1.1.m1.1b"><apply id="A5.T1.55.55.55.55.1.1.m1.1.1.cmml" xref="A5.T1.55.55.55.55.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.55.55.55.55.1.1.m1.1.1.1.cmml" xref="A5.T1.55.55.55.55.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.55.55.55.55.1.1.m1.1.1.2.cmml" xref="A5.T1.55.55.55.55.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.55.55.55.55.1.1.m1.1.1.3.cmml" xref="A5.T1.55.55.55.55.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.55.55.55.55.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> ? <span id="A5.T1.56.56.56.56.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , but <span id="A5.T1.56.56.56.56.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.56.56.56.56.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.56.56.56.56.2.2.m1.1a"><msub id="A5.T1.56.56.56.56.2.2.m1.1.1" xref="A5.T1.56.56.56.56.2.2.m1.1.1.cmml"><mi id="A5.T1.56.56.56.56.2.2.m1.1.1.2" xref="A5.T1.56.56.56.56.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.56.56.56.56.2.2.m1.1.1.3" xref="A5.T1.56.56.56.56.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.56.56.56.56.2.2.m1.1b"><apply id="A5.T1.56.56.56.56.2.2.m1.1.1.cmml" xref="A5.T1.56.56.56.56.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.56.56.56.56.2.2.m1.1.1.1.cmml" xref="A5.T1.56.56.56.56.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.56.56.56.56.2.2.m1.1.1.2.cmml" xref="A5.T1.56.56.56.56.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.56.56.56.56.2.2.m1.1.1.3.cmml" xref="A5.T1.56.56.56.56.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.56.56.56.56.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.56.56.56.56.4" class="ltx_td ltx_align_left">Me/Since</td>
</tr>
<tr id="A5.T1.58.58.58.58" class="ltx_tr">
<td id="A5.T1.58.58.58.58.3" class="ltx_td"></td>
<td id="A5.T1.58.58.58.58.2" class="ltx_td ltx_align_left">
<span id="A5.T1.57.57.57.57.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.57.57.57.57.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.57.57.57.57.1.1.m1.1a"><msub id="A5.T1.57.57.57.57.1.1.m1.1.1" xref="A5.T1.57.57.57.57.1.1.m1.1.1.cmml"><mi id="A5.T1.57.57.57.57.1.1.m1.1.1.2" xref="A5.T1.57.57.57.57.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.57.57.57.57.1.1.m1.1.1.3" xref="A5.T1.57.57.57.57.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.57.57.57.57.1.1.m1.1b"><apply id="A5.T1.57.57.57.57.1.1.m1.1.1.cmml" xref="A5.T1.57.57.57.57.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.57.57.57.57.1.1.m1.1.1.1.cmml" xref="A5.T1.57.57.57.57.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.57.57.57.57.1.1.m1.1.1.2.cmml" xref="A5.T1.57.57.57.57.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.57.57.57.57.1.1.m1.1.1.3.cmml" xref="A5.T1.57.57.57.57.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.57.57.57.57.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> ? <span id="A5.T1.58.58.58.58.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , please , <span id="A5.T1.58.58.58.58.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.58.58.58.58.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.58.58.58.58.2.2.m1.1a"><msub id="A5.T1.58.58.58.58.2.2.m1.1.1" xref="A5.T1.58.58.58.58.2.2.m1.1.1.cmml"><mi id="A5.T1.58.58.58.58.2.2.m1.1.1.2" xref="A5.T1.58.58.58.58.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.58.58.58.58.2.2.m1.1.1.3" xref="A5.T1.58.58.58.58.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.58.58.58.58.2.2.m1.1b"><apply id="A5.T1.58.58.58.58.2.2.m1.1.1.cmml" xref="A5.T1.58.58.58.58.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.58.58.58.58.2.2.m1.1.1.1.cmml" xref="A5.T1.58.58.58.58.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.58.58.58.58.2.2.m1.1.1.2.cmml" xref="A5.T1.58.58.58.58.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.58.58.58.58.2.2.m1.1.1.3.cmml" xref="A5.T1.58.58.58.58.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.58.58.58.58.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.58.58.58.58.4" class="ltx_td ltx_align_left">Um/Best</td>
</tr>
<tr id="A5.T1.60.60.60.60" class="ltx_tr">
<td id="A5.T1.60.60.60.60.3" class="ltx_td"></td>
<td id="A5.T1.60.60.60.60.2" class="ltx_td ltx_align_left">
<span id="A5.T1.59.59.59.59.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.59.59.59.59.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.59.59.59.59.1.1.m1.1a"><msub id="A5.T1.59.59.59.59.1.1.m1.1.1" xref="A5.T1.59.59.59.59.1.1.m1.1.1.cmml"><mi id="A5.T1.59.59.59.59.1.1.m1.1.1.2" xref="A5.T1.59.59.59.59.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.59.59.59.59.1.1.m1.1.1.3" xref="A5.T1.59.59.59.59.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.59.59.59.59.1.1.m1.1b"><apply id="A5.T1.59.59.59.59.1.1.m1.1.1.cmml" xref="A5.T1.59.59.59.59.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.59.59.59.59.1.1.m1.1.1.1.cmml" xref="A5.T1.59.59.59.59.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.59.59.59.59.1.1.m1.1.1.2.cmml" xref="A5.T1.59.59.59.59.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.59.59.59.59.1.1.m1.1.1.3.cmml" xref="A5.T1.59.59.59.59.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.59.59.59.59.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> ? <span id="A5.T1.60.60.60.60.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , I want to know <span id="A5.T1.60.60.60.60.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.60.60.60.60.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.60.60.60.60.2.2.m1.1a"><msub id="A5.T1.60.60.60.60.2.2.m1.1.1" xref="A5.T1.60.60.60.60.2.2.m1.1.1.cmml"><mi id="A5.T1.60.60.60.60.2.2.m1.1.1.2" xref="A5.T1.60.60.60.60.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.60.60.60.60.2.2.m1.1.1.3" xref="A5.T1.60.60.60.60.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.60.60.60.60.2.2.m1.1b"><apply id="A5.T1.60.60.60.60.2.2.m1.1.1.cmml" xref="A5.T1.60.60.60.60.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.60.60.60.60.2.2.m1.1.1.1.cmml" xref="A5.T1.60.60.60.60.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.60.60.60.60.2.2.m1.1.1.2.cmml" xref="A5.T1.60.60.60.60.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.60.60.60.60.2.2.m1.1.1.3.cmml" xref="A5.T1.60.60.60.60.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.60.60.60.60.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.60.60.60.60.4" class="ltx_td ltx_align_left">Ironically/Beyond</td>
</tr>
<tr id="A5.T1.62.62.62.62" class="ltx_tr">
<td id="A5.T1.62.62.62.62.3" class="ltx_td ltx_align_left ltx_border_t"><span id="A5.T1.62.62.62.62.3.1" class="ltx_text ltx_font_bold">STS-B</span></td>
<td id="A5.T1.62.62.62.62.2" class="ltx_td ltx_align_left ltx_border_t">(<math id="A5.T1.61.61.61.61.1.m1.1" class="ltx_Math" alttext="y_{u}" display="inline"><semantics id="A5.T1.61.61.61.61.1.m1.1a"><msub id="A5.T1.61.61.61.61.1.m1.1.1" xref="A5.T1.61.61.61.61.1.m1.1.1.cmml"><mi id="A5.T1.61.61.61.61.1.m1.1.1.2" xref="A5.T1.61.61.61.61.1.m1.1.1.2.cmml">y</mi><mi id="A5.T1.61.61.61.61.1.m1.1.1.3" xref="A5.T1.61.61.61.61.1.m1.1.1.3.cmml">u</mi></msub><annotation-xml encoding="MathML-Content" id="A5.T1.61.61.61.61.1.m1.1b"><apply id="A5.T1.61.61.61.61.1.m1.1.1.cmml" xref="A5.T1.61.61.61.61.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.61.61.61.61.1.m1.1.1.1.cmml" xref="A5.T1.61.61.61.61.1.m1.1.1">subscript</csymbol><ci id="A5.T1.61.61.61.61.1.m1.1.1.2.cmml" xref="A5.T1.61.61.61.61.1.m1.1.1.2">𝑦</ci><ci id="A5.T1.61.61.61.61.1.m1.1.1.3.cmml" xref="A5.T1.61.61.61.61.1.m1.1.1.3">𝑢</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.61.61.61.61.1.m1.1c">y_{u}</annotation></semantics></math>/<math id="A5.T1.62.62.62.62.2.m2.1" class="ltx_Math" alttext="y_{l}" display="inline"><semantics id="A5.T1.62.62.62.62.2.m2.1a"><msub id="A5.T1.62.62.62.62.2.m2.1.1" xref="A5.T1.62.62.62.62.2.m2.1.1.cmml"><mi id="A5.T1.62.62.62.62.2.m2.1.1.2" xref="A5.T1.62.62.62.62.2.m2.1.1.2.cmml">y</mi><mi id="A5.T1.62.62.62.62.2.m2.1.1.3" xref="A5.T1.62.62.62.62.2.m2.1.1.3.cmml">l</mi></msub><annotation-xml encoding="MathML-Content" id="A5.T1.62.62.62.62.2.m2.1b"><apply id="A5.T1.62.62.62.62.2.m2.1.1.cmml" xref="A5.T1.62.62.62.62.2.m2.1.1"><csymbol cd="ambiguous" id="A5.T1.62.62.62.62.2.m2.1.1.1.cmml" xref="A5.T1.62.62.62.62.2.m2.1.1">subscript</csymbol><ci id="A5.T1.62.62.62.62.2.m2.1.1.2.cmml" xref="A5.T1.62.62.62.62.2.m2.1.1.2">𝑦</ci><ci id="A5.T1.62.62.62.62.2.m2.1.1.3.cmml" xref="A5.T1.62.62.62.62.2.m2.1.1.3">𝑙</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.62.62.62.62.2.m2.1c">y_{l}</annotation></semantics></math>)</td>
<td id="A5.T1.62.62.62.62.4" class="ltx_td ltx_border_t"></td>
</tr>
<tr id="A5.T1.64.64.64.64" class="ltx_tr">
<td id="A5.T1.64.64.64.64.3" class="ltx_td"></td>
<td id="A5.T1.64.64.64.64.2" class="ltx_td ltx_align_left">
<span id="A5.T1.63.63.63.63.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.63.63.63.63.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.63.63.63.63.1.1.m1.1a"><msub id="A5.T1.63.63.63.63.1.1.m1.1.1" xref="A5.T1.63.63.63.63.1.1.m1.1.1.cmml"><mi id="A5.T1.63.63.63.63.1.1.m1.1.1.2" xref="A5.T1.63.63.63.63.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.63.63.63.63.1.1.m1.1.1.3" xref="A5.T1.63.63.63.63.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.63.63.63.63.1.1.m1.1b"><apply id="A5.T1.63.63.63.63.1.1.m1.1.1.cmml" xref="A5.T1.63.63.63.63.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.63.63.63.63.1.1.m1.1.1.1.cmml" xref="A5.T1.63.63.63.63.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.63.63.63.63.1.1.m1.1.1.2.cmml" xref="A5.T1.63.63.63.63.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.63.63.63.63.1.1.m1.1.1.3.cmml" xref="A5.T1.63.63.63.63.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.63.63.63.63.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="A5.T1.64.64.64.64.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> sir <span id="A5.T1.64.64.64.64.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.64.64.64.64.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.64.64.64.64.2.2.m1.1a"><msub id="A5.T1.64.64.64.64.2.2.m1.1.1" xref="A5.T1.64.64.64.64.2.2.m1.1.1.cmml"><mi id="A5.T1.64.64.64.64.2.2.m1.1.1.2" xref="A5.T1.64.64.64.64.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.64.64.64.64.2.2.m1.1.1.3" xref="A5.T1.64.64.64.64.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.64.64.64.64.2.2.m1.1b"><apply id="A5.T1.64.64.64.64.2.2.m1.1.1.cmml" xref="A5.T1.64.64.64.64.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.64.64.64.64.2.2.m1.1.1.1.cmml" xref="A5.T1.64.64.64.64.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.64.64.64.64.2.2.m1.1.1.2.cmml" xref="A5.T1.64.64.64.64.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.64.64.64.64.2.2.m1.1.1.3.cmml" xref="A5.T1.64.64.64.64.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.64.64.64.64.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.64.64.64.64.4" class="ltx_td ltx_align_left">Note/Next</td>
</tr>
<tr id="A5.T1.66.66.66.66" class="ltx_tr">
<td id="A5.T1.66.66.66.66.3" class="ltx_td"></td>
<td id="A5.T1.66.66.66.66.2" class="ltx_td ltx_align_left">
<span id="A5.T1.65.65.65.65.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.65.65.65.65.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.65.65.65.65.1.1.m1.1a"><msub id="A5.T1.65.65.65.65.1.1.m1.1.1" xref="A5.T1.65.65.65.65.1.1.m1.1.1.cmml"><mi id="A5.T1.65.65.65.65.1.1.m1.1.1.2" xref="A5.T1.65.65.65.65.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.65.65.65.65.1.1.m1.1.1.3" xref="A5.T1.65.65.65.65.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.65.65.65.65.1.1.m1.1b"><apply id="A5.T1.65.65.65.65.1.1.m1.1.1.cmml" xref="A5.T1.65.65.65.65.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.65.65.65.65.1.1.m1.1.1.1.cmml" xref="A5.T1.65.65.65.65.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.65.65.65.65.1.1.m1.1.1.2.cmml" xref="A5.T1.65.65.65.65.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.65.65.65.65.1.1.m1.1.1.3.cmml" xref="A5.T1.65.65.65.65.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.65.65.65.65.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="A5.T1.66.66.66.66.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> , it is not . <span id="A5.T1.66.66.66.66.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.66.66.66.66.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.66.66.66.66.2.2.m1.1a"><msub id="A5.T1.66.66.66.66.2.2.m1.1.1" xref="A5.T1.66.66.66.66.2.2.m1.1.1.cmml"><mi id="A5.T1.66.66.66.66.2.2.m1.1.1.2" xref="A5.T1.66.66.66.66.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.66.66.66.66.2.2.m1.1.1.3" xref="A5.T1.66.66.66.66.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.66.66.66.66.2.2.m1.1b"><apply id="A5.T1.66.66.66.66.2.2.m1.1.1.cmml" xref="A5.T1.66.66.66.66.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.66.66.66.66.2.2.m1.1.1.1.cmml" xref="A5.T1.66.66.66.66.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.66.66.66.66.2.2.m1.1.1.2.cmml" xref="A5.T1.66.66.66.66.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.66.66.66.66.2.2.m1.1.1.3.cmml" xref="A5.T1.66.66.66.66.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.66.66.66.66.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.66.66.66.66.4" class="ltx_td ltx_align_left">Yesterday/meanwhile</td>
</tr>
<tr id="A5.T1.68.68.68.68" class="ltx_tr">
<td id="A5.T1.68.68.68.68.3" class="ltx_td ltx_border_bb"></td>
<td id="A5.T1.68.68.68.68.2" class="ltx_td ltx_align_left ltx_border_bb">
<span id="A5.T1.67.67.67.67.1.1" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.67.67.67.67.1.1.m1.1" class="ltx_Math" alttext="S_{1}" display="inline"><semantics id="A5.T1.67.67.67.67.1.1.m1.1a"><msub id="A5.T1.67.67.67.67.1.1.m1.1.1" xref="A5.T1.67.67.67.67.1.1.m1.1.1.cmml"><mi id="A5.T1.67.67.67.67.1.1.m1.1.1.2" xref="A5.T1.67.67.67.67.1.1.m1.1.1.2.cmml">S</mi><mn id="A5.T1.67.67.67.67.1.1.m1.1.1.3" xref="A5.T1.67.67.67.67.1.1.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.67.67.67.67.1.1.m1.1b"><apply id="A5.T1.67.67.67.67.1.1.m1.1.1.cmml" xref="A5.T1.67.67.67.67.1.1.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.67.67.67.67.1.1.m1.1.1.1.cmml" xref="A5.T1.67.67.67.67.1.1.m1.1.1">subscript</csymbol><ci id="A5.T1.67.67.67.67.1.1.m1.1.1.2.cmml" xref="A5.T1.67.67.67.67.1.1.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.67.67.67.67.1.1.m1.1.1.3.cmml" xref="A5.T1.67.67.67.67.1.1.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.67.67.67.67.1.1.m1.1c">S_{1}</annotation></semantics></math>&gt;</span> . <span id="A5.T1.68.68.68.68.2.3" class="ltx_text ltx_font_typewriter">[MASK]</span> . It is <span id="A5.T1.68.68.68.68.2.2" class="ltx_text ltx_font_typewriter">&lt;<math id="A5.T1.68.68.68.68.2.2.m1.1" class="ltx_Math" alttext="S_{2}" display="inline"><semantics id="A5.T1.68.68.68.68.2.2.m1.1a"><msub id="A5.T1.68.68.68.68.2.2.m1.1.1" xref="A5.T1.68.68.68.68.2.2.m1.1.1.cmml"><mi id="A5.T1.68.68.68.68.2.2.m1.1.1.2" xref="A5.T1.68.68.68.68.2.2.m1.1.1.2.cmml">S</mi><mn id="A5.T1.68.68.68.68.2.2.m1.1.1.3" xref="A5.T1.68.68.68.68.2.2.m1.1.1.3.cmml">2</mn></msub><annotation-xml encoding="MathML-Content" id="A5.T1.68.68.68.68.2.2.m1.1b"><apply id="A5.T1.68.68.68.68.2.2.m1.1.1.cmml" xref="A5.T1.68.68.68.68.2.2.m1.1.1"><csymbol cd="ambiguous" id="A5.T1.68.68.68.68.2.2.m1.1.1.1.cmml" xref="A5.T1.68.68.68.68.2.2.m1.1.1">subscript</csymbol><ci id="A5.T1.68.68.68.68.2.2.m1.1.1.2.cmml" xref="A5.T1.68.68.68.68.2.2.m1.1.1.2">𝑆</ci><cn type="integer" id="A5.T1.68.68.68.68.2.2.m1.1.1.3.cmml" xref="A5.T1.68.68.68.68.2.2.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.68.68.68.68.2.2.m1.1c">S_{2}</annotation></semantics></math>&gt;</span>
</td>
<td id="A5.T1.68.68.68.68.4" class="ltx_td ltx_align_left ltx_border_bb">Yeah/meanwhile</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table">Table E.1: </span>Top 3 automatically generated templates and label words for all tasks based on one split of <math id="A5.T1.70.m1.1" class="ltx_Math" alttext="K=16" display="inline"><semantics id="A5.T1.70.m1.1b"><mrow id="A5.T1.70.m1.1.1" xref="A5.T1.70.m1.1.1.cmml"><mi id="A5.T1.70.m1.1.1.2" xref="A5.T1.70.m1.1.1.2.cmml">K</mi><mo id="A5.T1.70.m1.1.1.1" xref="A5.T1.70.m1.1.1.1.cmml">=</mo><mn id="A5.T1.70.m1.1.1.3" xref="A5.T1.70.m1.1.1.3.cmml">16</mn></mrow><annotation-xml encoding="MathML-Content" id="A5.T1.70.m1.1c"><apply id="A5.T1.70.m1.1.1.cmml" xref="A5.T1.70.m1.1.1"><eq id="A5.T1.70.m1.1.1.1.cmml" xref="A5.T1.70.m1.1.1.1"></eq><ci id="A5.T1.70.m1.1.1.2.cmml" xref="A5.T1.70.m1.1.1.2">𝐾</ci><cn type="integer" id="A5.T1.70.m1.1.1.3.cmml" xref="A5.T1.70.m1.1.1.3">16</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A5.T1.70.m1.1d">K=16</annotation></semantics></math> training examples. Note that automatic template results are based on manual label words and automatic label word results are based on manual templates provided in Table <a href="#S2.T1" title="Table 1 ‣ Few-shot learning. ‣ 2 Related Work ‣ Making Pre-trained Language Models Better Few-shot Learners" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2012.15722" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2012.15723" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2012.15723">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2012.15723" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2012.15724" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Fri Mar  8 07:36:36 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2012.15723
The recent GPT-3 modelachieves remarkable few-shot performance solely by leveraging a natural-language prompt and a few task demonstrations as input context.
Inspired by their findings, we study few-shot learning in a more practical scenario, where we use smaller language models for which fine-tuning is computationally efficient.
We present LM-BFF—betterfew-shotfine-tuning oflanguagemodels111Alternatively, language models’bestfriendsforever.—a suite of simple and complementary techniques for fine-tuning language models on a small number of annotated examples. Our approach includes (1)  prompt-based fine-tuning together with a novel pipeline for automating prompt generation; and (2)  a refined strategy for dynamically and selectively incorporating demonstrations into each context.
Finally, we present a systematic evaluation for analyzing few-shot performance on a range of NLP tasks, including classification and regression. Our experiments demonstrate that our methods combine to dramatically outperform standard fine-tuning procedures in this low resource setting, achieving up to 30% absolute improvement, and 11% on average across all tasks. Our approach makes minimal assumptions on task resources and domain expertise, and hence constitutes a strong task-agnostic method for few-shot learning.222Our implementation is publicly available athttps://github.com/princeton-nlp/LM-BFF.
abstract
Making Pre-trained Language Models Better Few-shot Learners
Abstract
2012.15723
1IntroductionFigure 1:An illustration of (a)  masked language model (MLM)  pre-training, (b)  standard fine-tuning, and (c)  our proposed LM-BFF using prompt-based fine-tuning with demonstrations. The underlined text is the task-specifictemplate, and colored words arelabel words.The GPT-3 modelhas made waves in the NLP community by demonstrating astounding few-shot capabilities on myriad language understanding tasks.
Given only anatural language promptand a fewdemonstrationsof the task, GPT-3 is able to make accurate predictions without updating any of the weights of its underlying language model.
However, while remarkable, GPT-3 consists of 175B parameters, which makes it challenging to use in most real-wold applications.In this work, we study a more practical scenario in which we only assume access to a moderately-sized language model such as BERTor RoBERTa, and a small number of examples (i.e., afew-shotsetting) , which we can use to fine-tune the weights of
the language model.
This setting is appealing as
(1)  such models can be trained on typical research hardware;
(2)  few-shot settings are realistic, as it is generally both easy to acquire a few annotations (e.g., 32 examples)  and efficient to train on them; and
(3)  updating parameters typically leads to better performance.
Inspired by GPT-3’s findings, we propose several novel strategies for expanding its few-shot learning abilities to our setting, considering both classification and—for the first time—regression.First, we follow the route ofprompt-basedprediction, first developed by the GPT seriesfor zero-shot prediction and recently studied by PETfor fine-tuning.
Prompt-based prediction treats the downstream task as a (masked)  language modeling problem, where the model directly generates a textual response (referred to as alabel word)  to a given prompt defined by a task-specifictemplate(see Figure1(c) ) .
Finding the right prompts, however, is an art—requiring both domain expertise and an understanding of the language model’s inner workings.
Even if significant effort is invested, manual prompts are likely to be suboptimal.
We address this issue by introducing automatic prompt generation, including a pruned brute-force search to identify the best working label words, and a novel decoding objective to automatically generate templates using the generative T5 model—all of which only require the few-shot training data. This allows us to cheaply obtain effective prompts that match or outperform our manually chosen ones.Second, we adopt the idea of incorporating demonstrations as
additional context.
GPT-3’s naive “in-context learning” paradigm picks up to 32 randomly sampled examples, and concatenates them with the input. This method is not guaranteed to prioritize the most informative demonstrations, and mixing random examples from different classes together creates long contexts which can be hard to learn from. Additionally, the number of usable demonstrations is bounded by the model’s maximum input length.
We develop a more refined strategy, where, for each input, we randomly sample asingleexample at a time fromeachclass to create multiple, minimal demonstrationsets.
We also devise a novel sampling strategy that pairs inputs with similar examples, thereby providing the model with more discriminative comparisons.We present a systematic evaluation for analyzing few-shot performance on 8 single-sentence and 7 sentence-pair NLP tasks. We observe that given a small number of training examples, (1)  prompt-based fine-tuning largely outperforms standard fine-tuning; (2)  our automatic prompt search method matches or outperforms manual prompts; and (3)  incorporating demonstrations is effective for fine-tuning, and boosts few-shot performance.
Together, these simple-yet-effective methods contribute towards a dramatic improvement across the tasks we evaluate on,
and we obtain gains up to 30% absolute improvement (11% on average)  compared to standard fine-tuning.
For instance, we find that a RoBERTa-large model achieves around 90% accuracy on most binary sentence classification tasks, while only relying on 32 training examples. We refer to our approach as LM-BFF,betterfew-shotfine-tuning oflanguagemodels: a strong, task-agnostic method for few-shot learning.
introduction
Making Pre-trained Language Models Better Few-shot Learners
1Introduction
2012.15723
2Related WorkLanguage model prompting.The GPT seriesfueled the development of prompt-based learning,
and we follow many of its core concepts.
We are also greatly inspired by the recent PET work, although they mainly focus on a semi-supervised setting where a large set of unlabeled examples are provided.
We only use a few annotated examples as supervision, and also explore automatically generated prompts and fine-tuning with demonstrations.
Furthermore, we deviate from their evaluation by providing a more rigorous framework, as we will discuss in §3.
Finally, there is a large body of work on prompting for mining knowledge from pre-trained models. Different from these works, we focus on leveraging prompting for fine-tuning on downstream tasks.Automatic prompt search.andexplore ways of identifying label words automatically,
however, none of these results lead to better performance compared to hand-picked ones.
In contrast, our method searches over both templates and label words, and is able to match or outperform our manual prompts.
Several other attempts have been made in addition—yet these approaches either operate in limited domains,
such as finding patterns to express specific relations,
or require a large number of examples for gradient-guided search. Our approach aims to develop general-purpose search methods that rely only on a few annotations.Fine-tuning of language models.A number of recent studies have focused on better methods for fine-tuning language models. These works mainly focus on optimization and regularization techniques to stabilize fine-tuning. Here we use standard optimization techniques, and instead mainly focus our efforts on better prompt-based fine-tuning in a more extreme few-shot setting. We anticipate that results of these studies are largely complementary to ours.Few-shot learning.Broadly speaking, our setting is also connected to other few-shot learning paradigms in NLP, including
(1)  semi-supervised learning, where a set of unlabeled examples are given;
(2)  meta-learning, where a set of auxiliary tasks are given; and
(3)  intermediate training, where a related, intermediate task is given. We deviate from these settings by making minimal assumptions about available resources: we only assume a few annotated examples and a pre-trained language model. Our focus is on understanding how far we can push without any other advantages.TaskTemplateLabel wordsSST-2<>It was[MASK].positive: great, negative: terribleSST-5<>It was[MASK].v.positive: great, positive: good, neutral: okay, negative: bad, v.negative: terribleMR<>It was[MASK].positive: great, negative: terribleCR<>It was[MASK].positive: great, negative: terribleSubj<>This is[MASK].subjective: subjective, objective: objectiveTREC[MASK]:<>abbreviation: Expression, entity: Entity, description: Descriptionhuman: Human, location: Location, numeric: NumberCOLA<>This is[MASK].grammatical: correct, not_grammatical: incorrectMNLI<>?[MASK],<>entailment: Yes, netural: Maybe, contradiction: NoSNLI<>?[MASK],<>entailment: Yes, netural: Maybe, contradiction: NoQNLI<>?[MASK],<>entailment: Yes, not_entailment: NoRTE<>?[MASK],<>entailment: Yes, not_entailment: NoMRPC<>[MASK],<>equivalent: Yes, not_equivalent: NoQQP<>[MASK],<>equivalent: Yes, not_equivalent: NoSTS-B<>[MASK],<>: Yes,: NoTable 1:Manual templates and label words that we used in our experiments.
STS-B is a regression task (§4.2) .
body
Making Pre-trained Language Models Better Few-shot Learners
2Related Work
2012.15723
3Problem SetupTask formulation.In this work, we assume access to a pre-trained language modelthat we wish to fine-tune on a taskwith a label space. For the task, we only assumetraining examplesper class333For regression, we partition the data into two “classes” according to being above or below the median value.for the task’s training set, such that the total number of examples is, and.
Our goal is then to develop task-agnostic learning strategies that generalize well to an unseen test set. For model selection and hyper-parameter tuning, we assume a development set, of the same size as the few-shot training set, i.e.,. This distinction is important: using a larger development set confers a significant advantage
(see our experiments in AppendixA) ,
and subverts our initial goal of learning from limited data.444In contrast,do not use a development set, and adopt a set of hyper-parameters based on practical considerations.
This is akin to “shooting in the dark” on a setting that we show can have unintuitive outcomes.For all of the following experiments (unless specified otherwise) , we takeRoBERTa-large and.Evaluation datasets.We conduct a systematic study acrosssingle-sentence andsentence-pair English tasks, including
8 tasks from the GLUE benchmark,
SNLI, and 6 other popular sentence classification tasks (SST-5, MR, CR, MPQA, Subj, TREC) . All of the dataset details are provided in AppendixB. Forsingle-sentencetasks, the goal is to make a prediction based on an input sentence, such as whether a movie review is positive or not. Forsentence-pairtasks, the goal is to take a pair of input sentencesand predict the relationship between them. We also interchangeably refer to the inputs as<>or (<>,<>) .
Note that we mainly use SST-2 and SNLI for pilot experiments and model development, making it close to a true few-shot setting, at least for all the other datasets we evaluate on.Evaluation protocol.Systematically evaluating few-shot performance can be tricky.
It is well-known that
fine-tuning on small datasets can suffer from instability, and results may change dramatically given a new split of data.
To account for this, we measure average performance across 5 different randomly sampledandsplits.
This issue has also been discussed in—they suggest using a fixed set of training examples. We argue that sampling
multiple splits gives a
more robust measure of performance,
and a better estimate of the variance.
We also observe that hyper-parameters
can make a significant difference,
thus we sweep multiple hyper-parameters for each data sample, and take the best setting as measured on theof that sample (see AppendixC.1) .
body
Making Pre-trained Language Models Better Few-shot Learners
3Problem Setup
2012.15723
4Prompt-based Fine-tuningGiven a masked language model,
we first convert inputto a token sequence, and the language modelthen
mapsto a sequence of hidden vectors.
During standard fine-tuning, we usually takeor.
For downstream classification tasks with a label space, we train a task-specific head,, by maximizing the log-probability of the correct label, whereis the hidden vector of[CLS], andis a set of randomly initialized parameters introduced at the start of fine-tuning.
Similarly, for a regression task, we can introduceand optimize the mean squared error betweenand the gold label.
In either case, the number of new parameters can be substantial—for example, a simple binary classification task will introduce 2,048 new parameters for a RoBERTa-large model—making it challenging to learn from a small amount of annotated data (e.g., 32 examples) .An alternative approach to solving this problem isprompt-based fine-tuning, in whichis directly tasked with “auto-completing” natural language prompts. For instance, we can formulate a binary sentiment classification task using a prompt with
input(e.g., “No reason to watch it .”)  as:and letdecide whether it is more appropriate to fill in “great” (positive)  or “terrible” (negative)  for[MASK].
We now formalize this approach for classification and regression (§4.1and §4.2) , and discuss the importance of prompt selection (§4.3) .4.1ClassificationLetbe a mapping from the task label space to individual words555More generally, we can consider a one-to-many mappingin which we map labels to sets of words. However, we did not find significant gains in our experiments.in the vocabularyof.
Then for each, let the manipulationbe amasked language modeling(MLM)  input which contains one[MASK]token.
In this way, we can treat our task as an
MLM, and model the probability of predicting classas:(1) whereis the hidden vector of[MASK]anddenotes the pre-softmax vector corresponding to.
When supervised examplesare available,can be fine-tuned to minimize the cross-entropy loss.
It is important to note that this approach re-uses the pre-trained weightsand does not introduce any new parameters. It also reduces the gap between pre-training and fine-tuning, making it more effective in few-shot scenarios.4.2RegressionWe assume the same basic setup as in classification, but treat the label spaceas a bounded interval.
Inspired by,
we model the problem as an interpolation between two opposing poles,, with valuesandrespectively.
For instance, we can formulate our previous sentiment analysis task as a regression problem in the range, where we slide between “terrible” ()  and “great” () . In this way, we can expressas amixture model:(2) whereis the probability of, and.
Then we define,
and modelthe same as Eq. (1) .
We fine-tuneto minimize the KL-divergence between the inferredand the observed mixture weight,.TemplateLabel wordsAccuracySST-2 (positive/negative) mean (std) <>It was[MASK].great/terrible92.7 (0.9) <>It was[MASK].good/bad92.5 (1.0) <>It was[MASK].cat/dog91.5 (1.4) <>It was[MASK].dog/cat86.2 (5.4) <>It was[MASK].terrible/great83.2 (6.9) Fine-tuning-81.4 (3.8) SNLI (entailment/neutral/contradiction) mean (std) <>?[MASK],<>Yes/Maybe/No77.2 (3.7) <>.[MASK],<>Yes/Maybe/No76.2 (3.3) <>?[MASK]<>Yes/Maybe/No74.9 (3.0) <><>[MASK]Yes/Maybe/No65.8 (2.4) <>?[MASK],<>Yes/Maybe/No62.9 (4.1) <>?[MASK],<>Maybe/No/Yes60.6 (4.8) Fine-tuning-48.4 (4.8) Table 2:The impact of templates and label words on prompt-based fine-tuning () .4.3Manual prompts: the good and the badThe key challenge is to construct the templateand label words—we refer to these two together as aprompt.
Previous workshand-craft both the templates and label words, which usually requires domain expertise and trial-and-error.
Table1summarizes manual templates and label words chosen for each dataset in our experiments.
These templates and label words were designed by intuition, and by considering formats used in previous literature.To better understand what constitutes a good template or label word, we conduct a pilot study on SST-2 and SNLI.
Table2shows that different prompts can lead to substantial differences in final accuracy.
Specifically, when a template is fixed, the better the label words match the “semantic classes”,
the better the final accuracy is
(great/terriblegood/badcat/dog) .
In extreme cases where we swap plausible label words (e.g.,terrible/great) , we achieve the worst overall performance.666It is unclear, however, why RoBERTa thinks that “cat” is more positive than “dog”. The authors tend to disagree.Furthermore, with the same set of label words,
even a small change in the template
can make a difference.
For example, for SNLI, if we put[MASK]at the end, or swap sentence order, we observe a10% drop.
The above evidence clearly underlines the importance of selecting good templates and label words.
Searching for prompts, however, is hard, as the search space can be very large—especially for the template. Even worse, we only have a few examples to use to guide our search, which can easily overfit. We will address these issues next.
body
Making Pre-trained Language Models Better Few-shot Learners
4Prompt-based Fine-tuning
2012.15723
5Automatic Prompt GenerationWe now explore principled ways of automating the search process for label words (§5.1)  and templates (§5.2) . Our goals are to reduce the human involvement required to design prompts, and to find more optimal settings than those that we manually choose.
Here, we assume a classification task, but the process for regression is analogous.5.1Automatic selection of label wordsWe first study how to construct a label word mappingthat maximizes accuracy onafter fine-tuning, given a fixed template.
Naively searching all possible assignments, however, is (1)  generally intractable, as the search space is exponential in the number of classes; and (2)  prone to overfitting, as we will tend to uncover spurious correlations given only a few annotations. As a simple solution,
for each class, we construct a pruned setof the topvocabulary words based on their conditional likelihood using the initial. That is, letbe the subset of all examples of class. We takeas(3) wheredenotes the output probability distribution of.
To further narrow down the search space, we find the topassignments over the pruned space that maximize zero-shot accuracy on(bothandare hyper-parameters, see
AppendixC.2) .
Then we fine-tune all topassignments, and re-rank to find the best one using.
This approach is similar to the automatic verbalizer search methods in, except that we use a much simpler search process (brute-force)  and also apply re-ranking—which we find to be quite helpful.5.2Automatic generation of templatesNext, we study how to generate a diverse set of templatesautomatically from a fixed set of label words.
To address this challenging problem, we propose to use T5, a large pre-trained text-to-text Transformer.
T5 is pre-trained to fill in missing spans (replaced by T5 mask tokens, e.g.,<X>or<Y>)  in its input.
For example, given the input “Thank you<X>me to your party<Y>week”, T5 is trained to generate “<X>for inviting<Y>last<Z>”,
meaning that “for inviting” is the replacement for<X>and “last” is the replacement for<Y>.
This is well suited for prompt generation: we can simply take input sentences fromand let the T5 model construct the template, without having to specify a pre-defined number of tokens for it.Given an input example, we consider the following simple conversions, denoted as, for formulating the T5 model inputs:777We consider putting the label word both before and after the input sentence for single-sentence tasks. However, we find that it is always better to put the label words in the middle (between the two sentences)  for sentence-pair tasks.As shown in Figure2, we rely on the T5 model to fill in the placeholders. When decoding, our goal here is to find an output that can work well forallexamples in,
i.e.,
the output templatethat maximizes,
wheredenotes the output probability distribution of T5.
It can be decomposed according to:(4) whereare the template tokens.Figure 2:Our approach for template generation.We use beam search to decode multiple template candidates. Concretely, we use a wide beam width (e.g., 100)  to cheaply obtain a large set of diverse templates. We then fine-tune each generated template onand useto either pick the single template with the best performance (Table3) , or the toptemplates to use as an ensemble (Table4) . Though it might appear to be expensive to fine-tune the model on each individual template, this is fast in practice due to the small size of, and is also fully automated: making it easy to use, compared to manually tuning prompts for each dataset.
body
Making Pre-trained Language Models Better Few-shot Learners
5Automatic Prompt Generation
2012.15723
6Fine-tuning with DemonstrationsSST-2SST-5MRCRMPQASubjTRECCoLA(acc) (acc) (acc) (acc) (acc) (acc) (acc) (Matt.) Majority†50.923.150.050.050.050.018.80.0Prompt-based zero-shot‡83.635.080.879.567.651.432.02.0“GPT-3” in-context learning84.8 (1.3) 30.6 (0.9) 80.5 (1.7) 87.4 (0.8) 63.8 (2.1) 53.6 (1.0) 26.2 (2.4) -1.5 (2.4) Fine-tuning81.4 (3.8) 43.9 (2.0) 76.9 (5.9) 75.8 (3.2) 72.0 (3.8) 90.8 (1.8) 88.8 (2.1) 33.9(14.3) Prompt-based FT (man) 92.7 (0.9) 47.4 (2.5) 87.0 (1.2) 90.3 (1.0) 84.7 (2.2) 91.2 (1.1) 84.8 (5.1) 9.3 (7.3) + demonstrations92.6 (0.5) 50.6(1.4) 86.6 (2.2) 90.2 (1.2) 87.0(1.1) 92.3(0.8) 87.5 (3.2) 18.7 (8.8) Prompt-based FT (auto) 92.3 (1.0) 49.2 (1.6) 85.5 (2.8) 89.0 (1.4) 85.8 (1.9) 91.2 (1.1) 88.2 (2.0) 14.0 (14.1) + demonstrations93.0(0.6) 49.5 (1.7) 87.7(1.4) 91.0(0.9) 86.5 (2.6) 91.4 (1.8) 89.4(1.7) 21.8 (15.9) Fine-tuning (full) †95.058.790.889.487.897.097.462.6MNLIMNLI-mmSNLIQNLIRTEMRPCQQPSTS-B(acc) (acc) (acc) (acc) (acc) (F1) (F1) (Pear.) Majority†32.733.033.849.552.781.20.0-Prompt-based zero-shot‡50.851.749.550.851.361.949.7-3.2“GPT-3” in-context learning52.0 (0.7) 53.4 (0.6) 47.1 (0.6) 53.8 (0.4) 60.4 (1.4) 45.7 (6.0) 36.1 (5.2) 14.3 (2.8) Fine-tuning45.8 (6.4) 47.8 (6.8) 48.4 (4.8) 60.2 (6.5) 54.4 (3.9) 76.6 (2.5) 60.7 (4.3) 53.5 (8.5) Prompt-based FT (man) 68.3 (2.3) 70.5 (1.9) 77.2 (3.7) 64.5 (4.2) 69.1 (3.6) 74.5 (5.3) 65.5 (5.3) 71.0 (7.0) + demonstrations70.7(1.3) 72.0(1.2) 79.7(1.5) 69.2(1.9) 68.7 (2.3) 77.8 (2.0) 69.8(1.8) 73.5 (5.1) Prompt-based FT (auto) 68.3 (2.5) 70.1 (2.6) 77.1 (2.1) 68.3 (7.4) 73.9(2.2) 76.2 (2.3) 67.0 (3.0) 75.0 (3.3) + demonstrations70.0 (3.6) 72.0(3.1) 77.5 (3.5) 68.5 (5.4) 71.1 (5.3) 78.1(3.4) 67.7 (5.8) 76.4(6.2) Fine-tuning (full) †89.889.592.693.380.991.481.791.9Table 3:Our main results using RoBERTa-large.: full training set is used (see dataset sizes in
TableB.1) ;: no training examples are used; otherwise we use(per class) 
for few-shot experiments.
We report mean (and standard deviation)  performance over 5 different splits (§3) .
Majority: majority class;
FT: fine-tuning;
man: manual prompt
(Table1) ;
auto: automatically searched templates (§5.2) ;
“GPT-3” in-context learning: using the in-context learning proposed inwith RoBERTa-large (no parameter updates) .In this section, we study whether we can leverage demonstrations whenfine-tuningmedium-sized LMs, and find better ways to exploit them.6.1Training examples as demonstrationsGPT-3’s naive approach to in-context learning simply involves concatenating the input with up to 32 examples randomly drawn from the training set.
This approach is suboptimal as
(1)  the number of available demonstrations is bounded by the model’s maximum input length;888GPT-3 uses a context size of 2,048 while most smaller language models (e.g., RoBERTa)  have a context size of 512.and (2)  mixing numerous random examples from different classes together creates extremely long contexts which can be hard to leverage, especially for a smaller model.
To address these issues,
we propose a simpler solution:
at each training step,
we randomly sampleone999We also explored sampling multiple examples per class, but did not observe any improvements.examplefrom each class,
convert it intowith[MASK]replaced by—we denote this as—and then concatenate them with(Figure1(c) ) :Heredenotes concatenation of input sequences.
During both training and inference we sample multiple demonstration sets for each. Note that bothand demonstration examples are sampled from the same setduring training. At testing time, we still sample demonstration sets fromand ensemble predictions across all sets.6.2Sampling similar demonstrationsWe observe that controlling the construction of the demonstration examplesis crucial for good final performance.
For example, if the set of contrastive demonstrationsare all dramatically different—from each other, or from the query—then it becomes challenging for the language model to decipher meaningful patterns. As a result, the model may simply ignore the context, or even get confused by the additional examples.
To address this issue, we devise a simple strategy in which we only sample examples that are semantically close to.
Specifically, we use a pre-trained SBERTmodel to obtain embeddings for all input sentences (for sentence-pair tasks, we use the concatenation of the two sentences) .
Here we just feed the raw sentences without the templates into SBERT.
For each queryand each label, we sort all training instances with the labelby their similarity score to the query, and only sample from the topinstances for each class to use as demonstrations.
body
Making Pre-trained Language Models Better Few-shot Learners
6Fine-tuning with Demonstrations
2012.15723
7ExperimentsWe present our main results, and address several research questions pertaining to our LM-BFF approach.
Implementation details are in AppendixC.7.1Main resultsWe use a RoBERTa-large model and setin our experiments.
A comparison of using RoBERTa vs BERT can be found in
AppendixD.
For automatic prompt search, in our main table we report automatic template search only (which consistently performs the best, see Table5) . To put our results in perspective, we compare to a number of baselines, namely
(1)  standard fine-tuning in our few-shot setting;
(2)  standard fine-tuning using the full training set;
(3)  simply taking the most frequent class (measured on the full training set) ;
(4)  prompt-based zero-shot prediction where we take our manual prompts and use“out-of-the-box” without using any training examples; and
(5)  “GPT-3” in-context learning, where we use the same prompt-based zero-shot setting, but augment the context with randomly sampled 32 demonstrations
(and still use RoBERTa-large, not GPT-3) .Prompt-based Fine-tuningMNLIRTEOur single manual68.3 (2.3) 69.1 (3.6) 71.9 (1.5) 69.2 (4.0) ,70.4 (3.1) 73.0 (3.2) + demonstrations74.0 (1.9) 71.9 (4.6) ,72.7 (2.5) 73.1(3.3) + demonstrations75.4(1.6) 72.3 (4.5) Table 4:Ensemble models using manual prompts from PETand our automatic templates. PET uses 4 prompts for MNLI and 5 for RTE. We also use an equal number of templates
infor a fair comparison.SST-2SNLITRECMRPCManual92.777.284.874.5Auto T92.377.188.276.2Auto L91.575.687.077.2Auto T + L92.177.089.274.0Table 5:Comparison between manual prompts and different automatic prompt generation methods: auto-generated templates (Auto T) , auto-generated label words (Auto L) , and their combination (Auto T + L) .Single-prompt results.Table3shows our main results using a single prompt, either from our manually designed ones (Table1)  , or the best generated ones.
First, prompt-based zero-shot prediction achieves much better performance than the majority class, showing the pre-encoded knowledge in RoBERTa.
Also, “GPT-3” in-context learning
does not always improve over zero-shot prediction, likely because smaller language models are not expressive enough to use off-the-shelf like GPT-3.Second, prompt-based fine-tuning can greatly outperform standard fine-tuning, both when using a manual prompt or a generated one.
CoLA is one interesting exception, as the input may be a non-grammatical sentence which is out of the distribution of. Generally, our automatically searched templates can achieve comparable or even higher results than manual ones,
especially for tasks in which constructing strong manual templates is less intuitive (e.g., TREC, QNLI and MRPC) .SST-2(positive/negative) Auto T= {great, terrible}#1.<>A[MASK]one .#2.<>A[MASK]piece .#3.<>All in all[MASK].Auto L=<>It was[MASK].#1. irresistible/pathetic#2. wonderful/bad#3. delicious/badSNLI(entailment/neutral/contradiction) Auto T= {Yes, Maybe, No}#1.<>.[MASK], no ,<>#2.<>.[MASK], in this case<>#3.<>.[MASK]this time<>Auto L=<>?[MASK],<>#1. Alright/Watch/Except#2. Hi/Watch/Worse#3. Regardless/Fortunately/UnlessTable 6:Examples of our automatically generated templates (Auto T)  and label words (Auto L) .Finally, using demonstrations in context leads to consistent gains in a majority of tasks.
In summary, our combined solution—fine-tuning with automatically searched templates and sampled demonstration sets—achieves again on SNLI compared to standard fine-tuning, andgain on average.Ensemble results.An advantage of automatic prompt search is that we can generate as many prompts as we want, train individual models, and create large ensembles. PETalso ensembles multiple models trained with manual prompts.101010They then use unlabeled data and distillation to get a single model, which is outside of our scope.In Table4, we make a direct comparison of our searched prompts and PET’s manual prompts on MNLI and RTE (two datasets that we evaluate in common) .111111In the PET NLI templates, the hypothesis is put before the premise, which we actually found to be suboptimal. In our experiments, we swap the two and get better results.As the results show, an ensemble with multiple templates always improves performance.
An ensemble of the same number of automatic templates achieves comparable or better performance than the ensemble of PET’s manual prompts. Increasing the number of automatic templates brings further gains.7.2Analysis of generated promptsTable5gives the results of using manual vs automatic prompts.
For automatic prompts, we compare
template search (Auto T) ,
label word search (Auto L) ,
and a joint variant (Auto T + L)  in which we start from manual label words, apply Auto T, and then Auto L.
In most cases, Auto T achieves comparable or higher performance than manual ones, and is consistently the best variant. Auto L outperforms manual prompts on TREC and MRPC—but is considerably worse on SNLI. Auto T + L is often better than Auto L, but only sometimes better than Auto T.
Table6shows examples from Auto T and Auto L (A full list in AppendixE) .
Auto T templates generally fit the context and label words well, but can contain biased peculiarities (e.g., “{Yes/No}, no” in SNLI) .
For Auto L words, things are mixed: while most look intuitively reasonable, there are also some mysterious abnormalities (e.g., “Hi” for the “entailment” class in SNLI) .7.3Analysis of demonstration samplingSST-2SNLITRECMRPCPrompt-based FT92.777.284.874.5Uniform sampling92.378.885.670.9+ RoBERTa sel.92.779.583.476.6+ SBERT sel.92.679.787.577.8Table 7:Impact of demonstration sampling strategies.
Uniform sampling randomly samples demonstrations, while selective (sel.)  sampling only takes top sentences measured by the sentence encoders (§6) .Table7compares the performance of demonstrations using uniform sampling to selective sampling by SBERT.
We acknowledge that SBERT is trained on SNLI and MNLI datasets, thus we also tried a simple sentence encoder using mean pooling of hidden representations from RoBERTa-large.
We find that in either case, using selective sampling outperforms uniform sampling, highlighting the importance of sampling similar examples for incorporating demonstrations in context.7.4Sample efficiencyFigure3illustrates
how standard fine-tuning and our LM-BFF compare asincreases.
For a simple task such as SST-2 (also see MR, CR and MPQA in Table3) , despite using only 32 total examples, LM-BFF has already nearly saturated its performance and is comparable to standard fine-tuning over the entire dataset. On the harder task of SNLI, LM-BFF continues to improve asincreases while still maintaining a performance gap over standard fine-tuning, until the two converge around.
body
Making Pre-trained Language Models Better Few-shot Learners
7Experiments
2012.15723
8DiscussionReformulating NLP tasks as MLM has exciting implications for few-shot learning, but also has limitations.
First, while LM-BFF greatly outperforms standard fine-tuning, Table3shows that, overall, the performance still substantially lags behind
fine-tuning with thousands of examples, especially for harder tasks.
Additionally, just like standard fine-tuning, our results also suffer from high variance.
As described in §2, several recent studies have tried to counter instability in few-shot fine-tuning
and we expect these methods to also help here.With respect to automatic prompt generation, despite its effectiveness,
we still find it practically challenging to expand the search space,
or generalize well based on only approximately 32 examples.
This is partly due to our lingering reliance onsomemanual design—either manual templates (for label word search)  or manual label words (for template search) ,
which allows us to get our search off the ground, but does also bias it towards areas of the search space that we might have already imagined.Finally, it is important to clarify that LM-BFF favors certain tasks which (1)  can be naturally posed as a “fill-in-the-blank” problem;
(2)  have relatively short input sequences; and
(3)  do not contain many output classes.
Issues (2)  and (3)  might be ameliorated with longer-context language models.
For tasks that are not straightforward to formulate in prompting, such as structured prediction, issue (1)  is more fundamental.
We leave it as an open question for future work.Figure 3:Standard fine-tuning vs our LM-BFF as a function of(# instances per class) . For lower, our method consistently outperforms standard fine-tuning.
body
Making Pre-trained Language Models Better Few-shot Learners
8Discussion
2012.15723
9ConclusionIn this paper we presented LM-BFF, a set of simple but effective techniques for fine-tuning language models using only a few examples. Our approach proposes to
(1)  use prompt-based fine-tuning with automatically searched prompts;
and (2)  include selected task demonstrations (training examples)  as part of the input context.
We show that our method outperforms vanilla fine-tuning by up to(and% on average) .
We concluded by discussing the limitations of our approach, and posed open questions for future study.
body
Making Pre-trained Language Models Better Few-shot Learners
9Conclusion
2012.15723
AcknowledgementsWe thank the members of Princeton, MIT, Tsinghua NLP groups and the anonymous reviewers for their valuable feedback. TG is supported by a Graduate Fellowship at Princeton University and AF is supported by an NSF Graduate Research Fellowship. This research is also partly supported by a Google Research Scholar Award.
conclusion
Making Pre-trained Language Models Better Few-shot Learners
Acknowledgements