<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2212.03827] Discovering Latent Knowledge in Language Models Without Supervision</title><meta property="og:description" content="Existing techniques for training language models can be misaligned with the truth:
if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans r…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Discovering Latent Knowledge in Language Models Without Supervision">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Discovering Latent Knowledge in Language Models Without Supervision">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2212.03827">

<!--Generated on Fri Mar  1 12:37:30 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Discovering Latent Knowledge in Language Models Without Supervision</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Collin Burns<sup id="id3.3.id1" class="ltx_sup">∗</sup> 
<br class="ltx_break">UC Berkeley 
<br class="ltx_break">&amp;Haotian Ye<sup id="id4.4.id2" class="ltx_sup">∗</sup> 
<br class="ltx_break">Peking University 
<br class="ltx_break">&amp;Dan Klein 
<br class="ltx_break">UC Berkeley 
<br class="ltx_break">&amp;Jacob Steinhardt 
<br class="ltx_break">UC Berkeley
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id5.id1" class="ltx_p">Existing techniques for training language models can be misaligned with the truth:
if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can’t detect.
We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way.
Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations.
It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values.
We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models: across 6 models and 10 question-answering datasets, it outperforms zero-shot accuracy by 4% on average.
We also find that it cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers.
Our results provide an initial step toward discovering what language models know, distinct from what they say, even when we don’t have access to explicit ground truth labels.</p>
</div>
<span id="footnotex1" class="ltx_note ltx_role_footnotetext"><sup class="ltx_note_mark">*</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">*</sup><span class="ltx_note_type">footnotetext: </span>Equal contribution.</span></span></span>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para ltx_noindent">
<p id="S1.p1.1" class="ltx_p">The increasing deployment of language models in real-world applications opens up exciting possibilities, but it also raises the stakes of AI research and presents new risks <cite class="ltx_cite ltx_citemacro_citep">(Bommasani et al., <a href="#bib.bib6" title="" class="ltx_ref">2021</a>; Weidinger et al., <a href="#bib.bib51" title="" class="ltx_ref">2021</a>; Bender et al., <a href="#bib.bib4" title="" class="ltx_ref">2021</a>)</cite>.
One of these risks is that language models do not always output text that is true <cite class="ltx_cite ltx_citemacro_citep">(Evans et al., <a href="#bib.bib12" title="" class="ltx_ref">2021</a>; Hendrycks et al., <a href="#bib.bib16" title="" class="ltx_ref">2021</a>; Kenton et al., <a href="#bib.bib19" title="" class="ltx_ref">2021</a>)</cite>.</p>
</div>
<div id="S1.p2" class="ltx_para ltx_noindent">
<p id="S1.p2.1" class="ltx_p">Common training objectives can cause models to learn internal representations related to truth, since truth is a useful feature for many tasks.
However, these objectives can also cause language models to output text that is false, at least in some circumstances.
For example, if we train a model to imitate human-generated text, it may learn to output common misconceptions <cite class="ltx_cite ltx_citemacro_citep">(Lin et al., <a href="#bib.bib25" title="" class="ltx_ref">2022</a>)</cite>.
Or if we train a chat bot to optimize a reward such as engagement, it may learn to generate text that is compelling but false <cite class="ltx_cite ltx_citemacro_citep">(Roller et al., <a href="#bib.bib43" title="" class="ltx_ref">2021</a>)</cite>.
If we try to reward model outputs that look true, a model may still learn to output false text if human raters can’t evaluate the correctness of that text <cite class="ltx_cite ltx_citemacro_citep">(Kenton et al., <a href="#bib.bib19" title="" class="ltx_ref">2021</a>)</cite>.</p>
</div>
<div id="S1.p3" class="ltx_para ltx_noindent">
<p id="S1.p3.1" class="ltx_p">In each case, this is an issue that stems from the misalignment between a training objective and the truth.
As models are applied to more complex domains, human supervision may become less effective at mitigating this misalignment.
Moreover, because this is a problem with the training objective rather than a model’s capabilities, it likely won’t be solved by scaling up models alone.</p>
</div>
<figure id="S1.F1" class="ltx_figure"><img src="/html/2212.03827/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="422" height="134" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>An illustration of our method, Contrast-Consistent Search (CCS). For each yes-no question <math id="S1.F1.15.m1.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S1.F1.15.m1.1b"><msub id="S1.F1.15.m1.1.1" xref="S1.F1.15.m1.1.1.cmml"><mi id="S1.F1.15.m1.1.1.2" xref="S1.F1.15.m1.1.1.2.cmml">q</mi><mi id="S1.F1.15.m1.1.1.3" xref="S1.F1.15.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S1.F1.15.m1.1c"><apply id="S1.F1.15.m1.1.1.cmml" xref="S1.F1.15.m1.1.1"><csymbol cd="ambiguous" id="S1.F1.15.m1.1.1.1.cmml" xref="S1.F1.15.m1.1.1">subscript</csymbol><ci id="S1.F1.15.m1.1.1.2.cmml" xref="S1.F1.15.m1.1.1.2">𝑞</ci><ci id="S1.F1.15.m1.1.1.3.cmml" xref="S1.F1.15.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.15.m1.1d">q_{i}</annotation></semantics></math>, we let <math id="S1.F1.16.m2.1" class="ltx_Math" alttext="x_{i}^{+}" display="inline"><semantics id="S1.F1.16.m2.1b"><msubsup id="S1.F1.16.m2.1.1" xref="S1.F1.16.m2.1.1.cmml"><mi id="S1.F1.16.m2.1.1.2.2" xref="S1.F1.16.m2.1.1.2.2.cmml">x</mi><mi id="S1.F1.16.m2.1.1.2.3" xref="S1.F1.16.m2.1.1.2.3.cmml">i</mi><mo id="S1.F1.16.m2.1.1.3" xref="S1.F1.16.m2.1.1.3.cmml">+</mo></msubsup><annotation-xml encoding="MathML-Content" id="S1.F1.16.m2.1c"><apply id="S1.F1.16.m2.1.1.cmml" xref="S1.F1.16.m2.1.1"><csymbol cd="ambiguous" id="S1.F1.16.m2.1.1.1.cmml" xref="S1.F1.16.m2.1.1">superscript</csymbol><apply id="S1.F1.16.m2.1.1.2.cmml" xref="S1.F1.16.m2.1.1"><csymbol cd="ambiguous" id="S1.F1.16.m2.1.1.2.1.cmml" xref="S1.F1.16.m2.1.1">subscript</csymbol><ci id="S1.F1.16.m2.1.1.2.2.cmml" xref="S1.F1.16.m2.1.1.2.2">𝑥</ci><ci id="S1.F1.16.m2.1.1.2.3.cmml" xref="S1.F1.16.m2.1.1.2.3">𝑖</ci></apply><plus id="S1.F1.16.m2.1.1.3.cmml" xref="S1.F1.16.m2.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.16.m2.1d">x_{i}^{+}</annotation></semantics></math> and <math id="S1.F1.17.m3.1" class="ltx_Math" alttext="x_{i}^{-}" display="inline"><semantics id="S1.F1.17.m3.1b"><msubsup id="S1.F1.17.m3.1.1" xref="S1.F1.17.m3.1.1.cmml"><mi id="S1.F1.17.m3.1.1.2.2" xref="S1.F1.17.m3.1.1.2.2.cmml">x</mi><mi id="S1.F1.17.m3.1.1.2.3" xref="S1.F1.17.m3.1.1.2.3.cmml">i</mi><mo id="S1.F1.17.m3.1.1.3" xref="S1.F1.17.m3.1.1.3.cmml">−</mo></msubsup><annotation-xml encoding="MathML-Content" id="S1.F1.17.m3.1c"><apply id="S1.F1.17.m3.1.1.cmml" xref="S1.F1.17.m3.1.1"><csymbol cd="ambiguous" id="S1.F1.17.m3.1.1.1.cmml" xref="S1.F1.17.m3.1.1">superscript</csymbol><apply id="S1.F1.17.m3.1.1.2.cmml" xref="S1.F1.17.m3.1.1"><csymbol cd="ambiguous" id="S1.F1.17.m3.1.1.2.1.cmml" xref="S1.F1.17.m3.1.1">subscript</csymbol><ci id="S1.F1.17.m3.1.1.2.2.cmml" xref="S1.F1.17.m3.1.1.2.2">𝑥</ci><ci id="S1.F1.17.m3.1.1.2.3.cmml" xref="S1.F1.17.m3.1.1.2.3">𝑖</ci></apply><minus id="S1.F1.17.m3.1.1.3.cmml" xref="S1.F1.17.m3.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.17.m3.1d">x_{i}^{-}</annotation></semantics></math> be the natural language statements where we answer <math id="S1.F1.18.m4.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S1.F1.18.m4.1b"><msub id="S1.F1.18.m4.1.1" xref="S1.F1.18.m4.1.1.cmml"><mi id="S1.F1.18.m4.1.1.2" xref="S1.F1.18.m4.1.1.2.cmml">q</mi><mi id="S1.F1.18.m4.1.1.3" xref="S1.F1.18.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S1.F1.18.m4.1c"><apply id="S1.F1.18.m4.1.1.cmml" xref="S1.F1.18.m4.1.1"><csymbol cd="ambiguous" id="S1.F1.18.m4.1.1.1.cmml" xref="S1.F1.18.m4.1.1">subscript</csymbol><ci id="S1.F1.18.m4.1.1.2.cmml" xref="S1.F1.18.m4.1.1.2">𝑞</ci><ci id="S1.F1.18.m4.1.1.3.cmml" xref="S1.F1.18.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.18.m4.1d">q_{i}</annotation></semantics></math> as “Yes” and “No” respectively.
Answering the question <math id="S1.F1.19.m5.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S1.F1.19.m5.1b"><msub id="S1.F1.19.m5.1.1" xref="S1.F1.19.m5.1.1.cmml"><mi id="S1.F1.19.m5.1.1.2" xref="S1.F1.19.m5.1.1.2.cmml">q</mi><mi id="S1.F1.19.m5.1.1.3" xref="S1.F1.19.m5.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S1.F1.19.m5.1c"><apply id="S1.F1.19.m5.1.1.cmml" xref="S1.F1.19.m5.1.1"><csymbol cd="ambiguous" id="S1.F1.19.m5.1.1.1.cmml" xref="S1.F1.19.m5.1.1">subscript</csymbol><ci id="S1.F1.19.m5.1.1.2.cmml" xref="S1.F1.19.m5.1.1.2">𝑞</ci><ci id="S1.F1.19.m5.1.1.3.cmml" xref="S1.F1.19.m5.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.19.m5.1d">q_{i}</annotation></semantics></math> then amounts to determining which of <math id="S1.F1.20.m6.1" class="ltx_Math" alttext="x_{i}^{+}" display="inline"><semantics id="S1.F1.20.m6.1b"><msubsup id="S1.F1.20.m6.1.1" xref="S1.F1.20.m6.1.1.cmml"><mi id="S1.F1.20.m6.1.1.2.2" xref="S1.F1.20.m6.1.1.2.2.cmml">x</mi><mi id="S1.F1.20.m6.1.1.2.3" xref="S1.F1.20.m6.1.1.2.3.cmml">i</mi><mo id="S1.F1.20.m6.1.1.3" xref="S1.F1.20.m6.1.1.3.cmml">+</mo></msubsup><annotation-xml encoding="MathML-Content" id="S1.F1.20.m6.1c"><apply id="S1.F1.20.m6.1.1.cmml" xref="S1.F1.20.m6.1.1"><csymbol cd="ambiguous" id="S1.F1.20.m6.1.1.1.cmml" xref="S1.F1.20.m6.1.1">superscript</csymbol><apply id="S1.F1.20.m6.1.1.2.cmml" xref="S1.F1.20.m6.1.1"><csymbol cd="ambiguous" id="S1.F1.20.m6.1.1.2.1.cmml" xref="S1.F1.20.m6.1.1">subscript</csymbol><ci id="S1.F1.20.m6.1.1.2.2.cmml" xref="S1.F1.20.m6.1.1.2.2">𝑥</ci><ci id="S1.F1.20.m6.1.1.2.3.cmml" xref="S1.F1.20.m6.1.1.2.3">𝑖</ci></apply><plus id="S1.F1.20.m6.1.1.3.cmml" xref="S1.F1.20.m6.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.20.m6.1d">x_{i}^{+}</annotation></semantics></math> or <math id="S1.F1.21.m7.1" class="ltx_Math" alttext="x_{i}^{-}" display="inline"><semantics id="S1.F1.21.m7.1b"><msubsup id="S1.F1.21.m7.1.1" xref="S1.F1.21.m7.1.1.cmml"><mi id="S1.F1.21.m7.1.1.2.2" xref="S1.F1.21.m7.1.1.2.2.cmml">x</mi><mi id="S1.F1.21.m7.1.1.2.3" xref="S1.F1.21.m7.1.1.2.3.cmml">i</mi><mo id="S1.F1.21.m7.1.1.3" xref="S1.F1.21.m7.1.1.3.cmml">−</mo></msubsup><annotation-xml encoding="MathML-Content" id="S1.F1.21.m7.1c"><apply id="S1.F1.21.m7.1.1.cmml" xref="S1.F1.21.m7.1.1"><csymbol cd="ambiguous" id="S1.F1.21.m7.1.1.1.cmml" xref="S1.F1.21.m7.1.1">superscript</csymbol><apply id="S1.F1.21.m7.1.1.2.cmml" xref="S1.F1.21.m7.1.1"><csymbol cd="ambiguous" id="S1.F1.21.m7.1.1.2.1.cmml" xref="S1.F1.21.m7.1.1">subscript</csymbol><ci id="S1.F1.21.m7.1.1.2.2.cmml" xref="S1.F1.21.m7.1.1.2.2">𝑥</ci><ci id="S1.F1.21.m7.1.1.2.3.cmml" xref="S1.F1.21.m7.1.1.2.3">𝑖</ci></apply><minus id="S1.F1.21.m7.1.1.3.cmml" xref="S1.F1.21.m7.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.21.m7.1d">x_{i}^{-}</annotation></semantics></math> is true.
We compute probabilities <math id="S1.F1.22.m8.1" class="ltx_Math" alttext="p^{+}_{i}" display="inline"><semantics id="S1.F1.22.m8.1b"><msubsup id="S1.F1.22.m8.1.1" xref="S1.F1.22.m8.1.1.cmml"><mi id="S1.F1.22.m8.1.1.2.2" xref="S1.F1.22.m8.1.1.2.2.cmml">p</mi><mi id="S1.F1.22.m8.1.1.3" xref="S1.F1.22.m8.1.1.3.cmml">i</mi><mo id="S1.F1.22.m8.1.1.2.3" xref="S1.F1.22.m8.1.1.2.3.cmml">+</mo></msubsup><annotation-xml encoding="MathML-Content" id="S1.F1.22.m8.1c"><apply id="S1.F1.22.m8.1.1.cmml" xref="S1.F1.22.m8.1.1"><csymbol cd="ambiguous" id="S1.F1.22.m8.1.1.1.cmml" xref="S1.F1.22.m8.1.1">subscript</csymbol><apply id="S1.F1.22.m8.1.1.2.cmml" xref="S1.F1.22.m8.1.1"><csymbol cd="ambiguous" id="S1.F1.22.m8.1.1.2.1.cmml" xref="S1.F1.22.m8.1.1">superscript</csymbol><ci id="S1.F1.22.m8.1.1.2.2.cmml" xref="S1.F1.22.m8.1.1.2.2">𝑝</ci><plus id="S1.F1.22.m8.1.1.2.3.cmml" xref="S1.F1.22.m8.1.1.2.3"></plus></apply><ci id="S1.F1.22.m8.1.1.3.cmml" xref="S1.F1.22.m8.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.22.m8.1d">p^{+}_{i}</annotation></semantics></math> and <math id="S1.F1.23.m9.1" class="ltx_Math" alttext="p^{-}_{i}" display="inline"><semantics id="S1.F1.23.m9.1b"><msubsup id="S1.F1.23.m9.1.1" xref="S1.F1.23.m9.1.1.cmml"><mi id="S1.F1.23.m9.1.1.2.2" xref="S1.F1.23.m9.1.1.2.2.cmml">p</mi><mi id="S1.F1.23.m9.1.1.3" xref="S1.F1.23.m9.1.1.3.cmml">i</mi><mo id="S1.F1.23.m9.1.1.2.3" xref="S1.F1.23.m9.1.1.2.3.cmml">−</mo></msubsup><annotation-xml encoding="MathML-Content" id="S1.F1.23.m9.1c"><apply id="S1.F1.23.m9.1.1.cmml" xref="S1.F1.23.m9.1.1"><csymbol cd="ambiguous" id="S1.F1.23.m9.1.1.1.cmml" xref="S1.F1.23.m9.1.1">subscript</csymbol><apply id="S1.F1.23.m9.1.1.2.cmml" xref="S1.F1.23.m9.1.1"><csymbol cd="ambiguous" id="S1.F1.23.m9.1.1.2.1.cmml" xref="S1.F1.23.m9.1.1">superscript</csymbol><ci id="S1.F1.23.m9.1.1.2.2.cmml" xref="S1.F1.23.m9.1.1.2.2">𝑝</ci><minus id="S1.F1.23.m9.1.1.2.3.cmml" xref="S1.F1.23.m9.1.1.2.3"></minus></apply><ci id="S1.F1.23.m9.1.1.3.cmml" xref="S1.F1.23.m9.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.23.m9.1d">p^{-}_{i}</annotation></semantics></math> that <math id="S1.F1.24.m10.1" class="ltx_Math" alttext="x_{i}^{+}" display="inline"><semantics id="S1.F1.24.m10.1b"><msubsup id="S1.F1.24.m10.1.1" xref="S1.F1.24.m10.1.1.cmml"><mi id="S1.F1.24.m10.1.1.2.2" xref="S1.F1.24.m10.1.1.2.2.cmml">x</mi><mi id="S1.F1.24.m10.1.1.2.3" xref="S1.F1.24.m10.1.1.2.3.cmml">i</mi><mo id="S1.F1.24.m10.1.1.3" xref="S1.F1.24.m10.1.1.3.cmml">+</mo></msubsup><annotation-xml encoding="MathML-Content" id="S1.F1.24.m10.1c"><apply id="S1.F1.24.m10.1.1.cmml" xref="S1.F1.24.m10.1.1"><csymbol cd="ambiguous" id="S1.F1.24.m10.1.1.1.cmml" xref="S1.F1.24.m10.1.1">superscript</csymbol><apply id="S1.F1.24.m10.1.1.2.cmml" xref="S1.F1.24.m10.1.1"><csymbol cd="ambiguous" id="S1.F1.24.m10.1.1.2.1.cmml" xref="S1.F1.24.m10.1.1">subscript</csymbol><ci id="S1.F1.24.m10.1.1.2.2.cmml" xref="S1.F1.24.m10.1.1.2.2">𝑥</ci><ci id="S1.F1.24.m10.1.1.2.3.cmml" xref="S1.F1.24.m10.1.1.2.3">𝑖</ci></apply><plus id="S1.F1.24.m10.1.1.3.cmml" xref="S1.F1.24.m10.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.24.m10.1d">x_{i}^{+}</annotation></semantics></math> and <math id="S1.F1.25.m11.1" class="ltx_Math" alttext="x_{i}^{-}" display="inline"><semantics id="S1.F1.25.m11.1b"><msubsup id="S1.F1.25.m11.1.1" xref="S1.F1.25.m11.1.1.cmml"><mi id="S1.F1.25.m11.1.1.2.2" xref="S1.F1.25.m11.1.1.2.2.cmml">x</mi><mi id="S1.F1.25.m11.1.1.2.3" xref="S1.F1.25.m11.1.1.2.3.cmml">i</mi><mo id="S1.F1.25.m11.1.1.3" xref="S1.F1.25.m11.1.1.3.cmml">−</mo></msubsup><annotation-xml encoding="MathML-Content" id="S1.F1.25.m11.1c"><apply id="S1.F1.25.m11.1.1.cmml" xref="S1.F1.25.m11.1.1"><csymbol cd="ambiguous" id="S1.F1.25.m11.1.1.1.cmml" xref="S1.F1.25.m11.1.1">superscript</csymbol><apply id="S1.F1.25.m11.1.1.2.cmml" xref="S1.F1.25.m11.1.1"><csymbol cd="ambiguous" id="S1.F1.25.m11.1.1.2.1.cmml" xref="S1.F1.25.m11.1.1">subscript</csymbol><ci id="S1.F1.25.m11.1.1.2.2.cmml" xref="S1.F1.25.m11.1.1.2.2">𝑥</ci><ci id="S1.F1.25.m11.1.1.2.3.cmml" xref="S1.F1.25.m11.1.1.2.3">𝑖</ci></apply><minus id="S1.F1.25.m11.1.1.3.cmml" xref="S1.F1.25.m11.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.25.m11.1d">x_{i}^{-}</annotation></semantics></math> are true respectively using a learned mapping from the hidden states to a number between <math id="S1.F1.26.m12.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S1.F1.26.m12.1b"><mn id="S1.F1.26.m12.1.1" xref="S1.F1.26.m12.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S1.F1.26.m12.1c"><cn type="integer" id="S1.F1.26.m12.1.1.cmml" xref="S1.F1.26.m12.1.1">0</cn></annotation-xml></semantics></math> and <math id="S1.F1.27.m13.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S1.F1.27.m13.1b"><mn id="S1.F1.27.m13.1.1" xref="S1.F1.27.m13.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S1.F1.27.m13.1c"><cn type="integer" id="S1.F1.27.m13.1.1.cmml" xref="S1.F1.27.m13.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.27.m13.1d">1</annotation></semantics></math>.
We search for a mapping such that that the probabilities are both confident and logically consistent.
On the right, we show a histogram of the “Yes” probabilities, <math id="S1.F1.28.m14.1" class="ltx_Math" alttext="\tilde{p}_{i}=0.5\cdot(p_{i}^{+}+(1-p_{i}^{-}))" display="inline"><semantics id="S1.F1.28.m14.1b"><mrow id="S1.F1.28.m14.1.1" xref="S1.F1.28.m14.1.1.cmml"><msub id="S1.F1.28.m14.1.1.3" xref="S1.F1.28.m14.1.1.3.cmml"><mover accent="true" id="S1.F1.28.m14.1.1.3.2" xref="S1.F1.28.m14.1.1.3.2.cmml"><mi id="S1.F1.28.m14.1.1.3.2.2" xref="S1.F1.28.m14.1.1.3.2.2.cmml">p</mi><mo id="S1.F1.28.m14.1.1.3.2.1" xref="S1.F1.28.m14.1.1.3.2.1.cmml">~</mo></mover><mi id="S1.F1.28.m14.1.1.3.3" xref="S1.F1.28.m14.1.1.3.3.cmml">i</mi></msub><mo id="S1.F1.28.m14.1.1.2" xref="S1.F1.28.m14.1.1.2.cmml">=</mo><mrow id="S1.F1.28.m14.1.1.1" xref="S1.F1.28.m14.1.1.1.cmml"><mn id="S1.F1.28.m14.1.1.1.3" xref="S1.F1.28.m14.1.1.1.3.cmml">0.5</mn><mo lspace="0.222em" rspace="0.222em" id="S1.F1.28.m14.1.1.1.2" xref="S1.F1.28.m14.1.1.1.2.cmml">⋅</mo><mrow id="S1.F1.28.m14.1.1.1.1.1" xref="S1.F1.28.m14.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S1.F1.28.m14.1.1.1.1.1.2" xref="S1.F1.28.m14.1.1.1.1.1.1.cmml">(</mo><mrow id="S1.F1.28.m14.1.1.1.1.1.1" xref="S1.F1.28.m14.1.1.1.1.1.1.cmml"><msubsup id="S1.F1.28.m14.1.1.1.1.1.1.3" xref="S1.F1.28.m14.1.1.1.1.1.1.3.cmml"><mi id="S1.F1.28.m14.1.1.1.1.1.1.3.2.2" xref="S1.F1.28.m14.1.1.1.1.1.1.3.2.2.cmml">p</mi><mi id="S1.F1.28.m14.1.1.1.1.1.1.3.2.3" xref="S1.F1.28.m14.1.1.1.1.1.1.3.2.3.cmml">i</mi><mo id="S1.F1.28.m14.1.1.1.1.1.1.3.3" xref="S1.F1.28.m14.1.1.1.1.1.1.3.3.cmml">+</mo></msubsup><mo id="S1.F1.28.m14.1.1.1.1.1.1.2" xref="S1.F1.28.m14.1.1.1.1.1.1.2.cmml">+</mo><mrow id="S1.F1.28.m14.1.1.1.1.1.1.1.1" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S1.F1.28.m14.1.1.1.1.1.1.1.1.2" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.cmml">(</mo><mrow id="S1.F1.28.m14.1.1.1.1.1.1.1.1.1" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.cmml"><mn id="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.2" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.2.cmml">1</mn><mo id="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.1" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.1.cmml">−</mo><msubsup id="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3.cmml"><mi id="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3.2.2" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3.2.2.cmml">p</mi><mi id="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3.2.3" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3.2.3.cmml">i</mi><mo id="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3.3" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3.3.cmml">−</mo></msubsup></mrow><mo stretchy="false" id="S1.F1.28.m14.1.1.1.1.1.1.1.1.3" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S1.F1.28.m14.1.1.1.1.1.3" xref="S1.F1.28.m14.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S1.F1.28.m14.1c"><apply id="S1.F1.28.m14.1.1.cmml" xref="S1.F1.28.m14.1.1"><eq id="S1.F1.28.m14.1.1.2.cmml" xref="S1.F1.28.m14.1.1.2"></eq><apply id="S1.F1.28.m14.1.1.3.cmml" xref="S1.F1.28.m14.1.1.3"><csymbol cd="ambiguous" id="S1.F1.28.m14.1.1.3.1.cmml" xref="S1.F1.28.m14.1.1.3">subscript</csymbol><apply id="S1.F1.28.m14.1.1.3.2.cmml" xref="S1.F1.28.m14.1.1.3.2"><ci id="S1.F1.28.m14.1.1.3.2.1.cmml" xref="S1.F1.28.m14.1.1.3.2.1">~</ci><ci id="S1.F1.28.m14.1.1.3.2.2.cmml" xref="S1.F1.28.m14.1.1.3.2.2">𝑝</ci></apply><ci id="S1.F1.28.m14.1.1.3.3.cmml" xref="S1.F1.28.m14.1.1.3.3">𝑖</ci></apply><apply id="S1.F1.28.m14.1.1.1.cmml" xref="S1.F1.28.m14.1.1.1"><ci id="S1.F1.28.m14.1.1.1.2.cmml" xref="S1.F1.28.m14.1.1.1.2">⋅</ci><cn type="float" id="S1.F1.28.m14.1.1.1.3.cmml" xref="S1.F1.28.m14.1.1.1.3">0.5</cn><apply id="S1.F1.28.m14.1.1.1.1.1.1.cmml" xref="S1.F1.28.m14.1.1.1.1.1"><plus id="S1.F1.28.m14.1.1.1.1.1.1.2.cmml" xref="S1.F1.28.m14.1.1.1.1.1.1.2"></plus><apply id="S1.F1.28.m14.1.1.1.1.1.1.3.cmml" xref="S1.F1.28.m14.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S1.F1.28.m14.1.1.1.1.1.1.3.1.cmml" xref="S1.F1.28.m14.1.1.1.1.1.1.3">superscript</csymbol><apply id="S1.F1.28.m14.1.1.1.1.1.1.3.2.cmml" xref="S1.F1.28.m14.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S1.F1.28.m14.1.1.1.1.1.1.3.2.1.cmml" xref="S1.F1.28.m14.1.1.1.1.1.1.3">subscript</csymbol><ci id="S1.F1.28.m14.1.1.1.1.1.1.3.2.2.cmml" xref="S1.F1.28.m14.1.1.1.1.1.1.3.2.2">𝑝</ci><ci id="S1.F1.28.m14.1.1.1.1.1.1.3.2.3.cmml" xref="S1.F1.28.m14.1.1.1.1.1.1.3.2.3">𝑖</ci></apply><plus id="S1.F1.28.m14.1.1.1.1.1.1.3.3.cmml" xref="S1.F1.28.m14.1.1.1.1.1.1.3.3"></plus></apply><apply id="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.cmml" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1"><minus id="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.1.cmml" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.1"></minus><cn type="integer" id="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.2.cmml" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.2">1</cn><apply id="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3.cmml" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3.1.cmml" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3">superscript</csymbol><apply id="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3.2.cmml" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3.2.1.cmml" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3.2.2.cmml" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3.2.2">𝑝</ci><ci id="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3.2.3.cmml" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3.2.3">𝑖</ci></apply><minus id="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3.3.cmml" xref="S1.F1.28.m14.1.1.1.1.1.1.1.1.1.3.3"></minus></apply></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S1.F1.28.m14.1d">\tilde{p}_{i}=0.5\cdot(p_{i}^{+}+(1-p_{i}^{-}))</annotation></semantics></math>, learned by our method for the COPA dataset <cite class="ltx_cite ltx_citemacro_citep">(Roemmele et al., <a href="#bib.bib42" title="" class="ltx_ref">2011</a>)</cite> with the UnifiedQA model <cite class="ltx_cite ltx_citemacro_citep">(Khashabi et al., <a href="#bib.bib20" title="" class="ltx_ref">2020</a>)</cite>.
Our method is purely unsupervised, only leveraging the consistency of truth, but still learns to accurately answers questions.
</figcaption>
</figure>
<div id="S1.p4" class="ltx_para ltx_noindent">
<p id="S1.p4.1" class="ltx_p">We propose a different approach for addressing this misalignment: using models to answer questions in a purely <span id="S1.p4.1.1" class="ltx_text ltx_font_italic">unsupervised</span> way.
Intuitively, instead of trying to explicitly, externally specify truth, we search for implicit, internal “beliefs” or “knowledge” learned by a model.
We approach this problem by leveraging the fact that a model’s representation of truth must satisfy logical consistency properties, which are unlikely to be satisfied by many other features.</p>
</div>
<div id="S1.p5" class="ltx_para ltx_noindent">
<p id="S1.p5.1" class="ltx_p">We implement this idea by introducing Contrast-Consistent Search (CCS), a method that learns a linear projection of the hidden states that is consistent across negations, as illustrated in Figure <a href="#S1.F1" title="Figure 1 ‣ 1 Introduction ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a>.
We find that despite its simplicity and despite not having access to any labels or model outputs, CCS can accurately recover knowledge from model representations: evaluated across 6 models and 10 question-answering datasets, CCS outperforms the accuracy of strong zero-shot baselines by 4% on average (<a href="#S3.SS2.SSS1" title="3.2.1 CCS Outperforms Zero-Shot ‣ 3.2 Evaluating CCS ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">3.2.1</span></a>).
The resulting classifier is also less sensitive to different prompts than zero-shot, cutting the standard deviation in accuracy in half.
Additionally, we try deliberately prompting models to make incorrect outputs, which should intuitively change what models say but which shouldn’t affect their latent knowledge.
We find that this causes zero-shot accuracy to drop by up to 9.5% (<a href="#S3.SS2.SSS2" title="3.2.2 CCS Is Robust To Misleading Prompts ‣ 3.2 Evaluating CCS ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">3.2.2</span></a>) without decreasing the accuracy of CCS.</p>
</div>
<div id="S1.p6" class="ltx_para ltx_noindent">
<p id="S1.p6.1" class="ltx_p">We systematically analyze CCS to understand the features it discovers.
We show that it transfers across unrelated tasks, suggesting that models may have a task-agnostic representation of the truth and that CCS is able to approximately discover it (<a href="#S3.SS3.SSS1" title="3.3.1 CCS Finds A Task-Agnostic Representation of Truth ‣ 3.3 Analyzing CCS ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">3.3.1</span></a>).
Moreover, CCS sometimes works best using the hidden states in the <span id="S1.p6.1.1" class="ltx_text ltx_font_italic">middle</span> layers of a network and can work even when model outputs aren’t very informative, suggesting that it can leverage different features from those used by the outputs (<a href="#S3.SS3.SSS2" title="3.3.2 CCS Does Not Just Recover Model Outputs ‣ 3.3 Analyzing CCS ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">3.3.2</span></a>).
Finally, we show that representations of truth tend to be salient in models: they can often be found without much data, and they can often be found by taking the top principal component of a slightly modified representation space (<a href="#S3.SS3.SSS3" title="3.3.3 Truth is a Salient Feature ‣ 3.3 Analyzing CCS ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">3.3.3</span></a>).</p>
</div>
<div id="S1.p7" class="ltx_para ltx_noindent">
<p id="S1.p7.1" class="ltx_p">Most existing techniques for making models truthful use human supervision to explicitly specify what is correct.
However, it is not feasible to provide supervision in some settings.
Our work suggests that an external source of ground truth may not actually be necessary: we may instead be able to find a model’s latent representation of truth, independent of what a model says, without using any supervision in the first place.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Problem Statement and Framework</h2>

<div id="S2.p1" class="ltx_para ltx_noindent">
<p id="S2.p1.1" class="ltx_p">In this section we describe our problem setup in more detail and introduce Contrast-Consistent Search (CCS), a method for discovering latent knowledge in language models without supervision.</p>
</div>
<section id="S2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.1 </span>Problem: Discovering Latent Knowledge</h3>

<div id="S2.SS1.p1" class="ltx_para ltx_noindent">
<p id="S2.SS1.p1.3" class="ltx_p">Given a pre-trained neural language model and a set <math id="S2.SS1.p1.1.m1.3" class="ltx_Math" alttext="q_{1},\ldots,q_{n}" display="inline"><semantics id="S2.SS1.p1.1.m1.3a"><mrow id="S2.SS1.p1.1.m1.3.3.2" xref="S2.SS1.p1.1.m1.3.3.3.cmml"><msub id="S2.SS1.p1.1.m1.2.2.1.1" xref="S2.SS1.p1.1.m1.2.2.1.1.cmml"><mi id="S2.SS1.p1.1.m1.2.2.1.1.2" xref="S2.SS1.p1.1.m1.2.2.1.1.2.cmml">q</mi><mn id="S2.SS1.p1.1.m1.2.2.1.1.3" xref="S2.SS1.p1.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.SS1.p1.1.m1.3.3.2.3" xref="S2.SS1.p1.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS1.p1.1.m1.1.1" xref="S2.SS1.p1.1.m1.1.1.cmml">…</mi><mo id="S2.SS1.p1.1.m1.3.3.2.4" xref="S2.SS1.p1.1.m1.3.3.3.cmml">,</mo><msub id="S2.SS1.p1.1.m1.3.3.2.2" xref="S2.SS1.p1.1.m1.3.3.2.2.cmml"><mi id="S2.SS1.p1.1.m1.3.3.2.2.2" xref="S2.SS1.p1.1.m1.3.3.2.2.2.cmml">q</mi><mi id="S2.SS1.p1.1.m1.3.3.2.2.3" xref="S2.SS1.p1.1.m1.3.3.2.2.3.cmml">n</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.1.m1.3b"><list id="S2.SS1.p1.1.m1.3.3.3.cmml" xref="S2.SS1.p1.1.m1.3.3.2"><apply id="S2.SS1.p1.1.m1.2.2.1.1.cmml" xref="S2.SS1.p1.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.1.m1.2.2.1.1.1.cmml" xref="S2.SS1.p1.1.m1.2.2.1.1">subscript</csymbol><ci id="S2.SS1.p1.1.m1.2.2.1.1.2.cmml" xref="S2.SS1.p1.1.m1.2.2.1.1.2">𝑞</ci><cn type="integer" id="S2.SS1.p1.1.m1.2.2.1.1.3.cmml" xref="S2.SS1.p1.1.m1.2.2.1.1.3">1</cn></apply><ci id="S2.SS1.p1.1.m1.1.1.cmml" xref="S2.SS1.p1.1.m1.1.1">…</ci><apply id="S2.SS1.p1.1.m1.3.3.2.2.cmml" xref="S2.SS1.p1.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS1.p1.1.m1.3.3.2.2.1.cmml" xref="S2.SS1.p1.1.m1.3.3.2.2">subscript</csymbol><ci id="S2.SS1.p1.1.m1.3.3.2.2.2.cmml" xref="S2.SS1.p1.1.m1.3.3.2.2.2">𝑞</ci><ci id="S2.SS1.p1.1.m1.3.3.2.2.3.cmml" xref="S2.SS1.p1.1.m1.3.3.2.2.3">𝑛</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.1.m1.3c">q_{1},\ldots,q_{n}</annotation></semantics></math> of yes-no questions<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>
Technically, we only require that there are two mutually exclusive answers.
For example, we can also use the labels “positive” and “negative” for sentiment classification.
Moreover, our setup can easily extend to the case where we want to evaluate the truth of a set of statements instead of answering a set of questions.
</span></span></span>, our goal is to answer each <math id="S2.SS1.p1.2.m2.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S2.SS1.p1.2.m2.1a"><msub id="S2.SS1.p1.2.m2.1.1" xref="S2.SS1.p1.2.m2.1.1.cmml"><mi id="S2.SS1.p1.2.m2.1.1.2" xref="S2.SS1.p1.2.m2.1.1.2.cmml">q</mi><mi id="S2.SS1.p1.2.m2.1.1.3" xref="S2.SS1.p1.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.2.m2.1b"><apply id="S2.SS1.p1.2.m2.1.1.cmml" xref="S2.SS1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.2.m2.1.1.1.cmml" xref="S2.SS1.p1.2.m2.1.1">subscript</csymbol><ci id="S2.SS1.p1.2.m2.1.1.2.cmml" xref="S2.SS1.p1.2.m2.1.1.2">𝑞</ci><ci id="S2.SS1.p1.2.m2.1.1.3.cmml" xref="S2.SS1.p1.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.2.m2.1c">q_{i}</annotation></semantics></math> correctly.
Here, <math id="S2.SS1.p1.3.m3.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S2.SS1.p1.3.m3.1a"><msub id="S2.SS1.p1.3.m3.1.1" xref="S2.SS1.p1.3.m3.1.1.cmml"><mi id="S2.SS1.p1.3.m3.1.1.2" xref="S2.SS1.p1.3.m3.1.1.2.cmml">q</mi><mi id="S2.SS1.p1.3.m3.1.1.3" xref="S2.SS1.p1.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS1.p1.3.m3.1b"><apply id="S2.SS1.p1.3.m3.1.1.cmml" xref="S2.SS1.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS1.p1.3.m3.1.1.1.cmml" xref="S2.SS1.p1.3.m3.1.1">subscript</csymbol><ci id="S2.SS1.p1.3.m3.1.1.2.cmml" xref="S2.SS1.p1.3.m3.1.1.2">𝑞</ci><ci id="S2.SS1.p1.3.m3.1.1.3.cmml" xref="S2.SS1.p1.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p1.3.m3.1c">q_{i}</annotation></semantics></math> can be any question with a well-defined answer, including procedural questions like “Is 22+59 = 237?”, for which the answer is “No”, and factual questions like “Are cats mammals?”, for which the answer is “Yes”.</p>
</div>
<div id="S2.SS1.p2" class="ltx_para ltx_noindent">
<p id="S2.SS1.p2.4" class="ltx_p">Importantly, we want methods that do not rely on the model generating correct outputs and that do not rely on external supervision.
Instead, we turn to the model’s unlabeled hidden representations.
Specifically, let <math id="S2.SS1.p2.1.m1.1" class="ltx_Math" alttext="\phi(x)\in\mathbb{R}^{d}" display="inline"><semantics id="S2.SS1.p2.1.m1.1a"><mrow id="S2.SS1.p2.1.m1.1.2" xref="S2.SS1.p2.1.m1.1.2.cmml"><mrow id="S2.SS1.p2.1.m1.1.2.2" xref="S2.SS1.p2.1.m1.1.2.2.cmml"><mi id="S2.SS1.p2.1.m1.1.2.2.2" xref="S2.SS1.p2.1.m1.1.2.2.2.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS1.p2.1.m1.1.2.2.1" xref="S2.SS1.p2.1.m1.1.2.2.1.cmml">​</mo><mrow id="S2.SS1.p2.1.m1.1.2.2.3.2" xref="S2.SS1.p2.1.m1.1.2.2.cmml"><mo stretchy="false" id="S2.SS1.p2.1.m1.1.2.2.3.2.1" xref="S2.SS1.p2.1.m1.1.2.2.cmml">(</mo><mi id="S2.SS1.p2.1.m1.1.1" xref="S2.SS1.p2.1.m1.1.1.cmml">x</mi><mo stretchy="false" id="S2.SS1.p2.1.m1.1.2.2.3.2.2" xref="S2.SS1.p2.1.m1.1.2.2.cmml">)</mo></mrow></mrow><mo id="S2.SS1.p2.1.m1.1.2.1" xref="S2.SS1.p2.1.m1.1.2.1.cmml">∈</mo><msup id="S2.SS1.p2.1.m1.1.2.3" xref="S2.SS1.p2.1.m1.1.2.3.cmml"><mi id="S2.SS1.p2.1.m1.1.2.3.2" xref="S2.SS1.p2.1.m1.1.2.3.2.cmml">ℝ</mi><mi id="S2.SS1.p2.1.m1.1.2.3.3" xref="S2.SS1.p2.1.m1.1.2.3.3.cmml">d</mi></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.1.m1.1b"><apply id="S2.SS1.p2.1.m1.1.2.cmml" xref="S2.SS1.p2.1.m1.1.2"><in id="S2.SS1.p2.1.m1.1.2.1.cmml" xref="S2.SS1.p2.1.m1.1.2.1"></in><apply id="S2.SS1.p2.1.m1.1.2.2.cmml" xref="S2.SS1.p2.1.m1.1.2.2"><times id="S2.SS1.p2.1.m1.1.2.2.1.cmml" xref="S2.SS1.p2.1.m1.1.2.2.1"></times><ci id="S2.SS1.p2.1.m1.1.2.2.2.cmml" xref="S2.SS1.p2.1.m1.1.2.2.2">italic-ϕ</ci><ci id="S2.SS1.p2.1.m1.1.1.cmml" xref="S2.SS1.p2.1.m1.1.1">𝑥</ci></apply><apply id="S2.SS1.p2.1.m1.1.2.3.cmml" xref="S2.SS1.p2.1.m1.1.2.3"><csymbol cd="ambiguous" id="S2.SS1.p2.1.m1.1.2.3.1.cmml" xref="S2.SS1.p2.1.m1.1.2.3">superscript</csymbol><ci id="S2.SS1.p2.1.m1.1.2.3.2.cmml" xref="S2.SS1.p2.1.m1.1.2.3.2">ℝ</ci><ci id="S2.SS1.p2.1.m1.1.2.3.3.cmml" xref="S2.SS1.p2.1.m1.1.2.3.3">𝑑</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.1.m1.1c">\phi(x)\in\mathbb{R}^{d}</annotation></semantics></math> denote some feature representation on a natural langauge input <math id="S2.SS1.p2.2.m2.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S2.SS1.p2.2.m2.1a"><mi id="S2.SS1.p2.2.m2.1.1" xref="S2.SS1.p2.2.m2.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.2.m2.1b"><ci id="S2.SS1.p2.2.m2.1.1.cmml" xref="S2.SS1.p2.2.m2.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.2.m2.1c">x</annotation></semantics></math>, such as the hidden states of a Transformer-based language model.
Our goal is to answer the questions <math id="S2.SS1.p2.3.m3.3" class="ltx_Math" alttext="q_{1},\ldots,q_{n}" display="inline"><semantics id="S2.SS1.p2.3.m3.3a"><mrow id="S2.SS1.p2.3.m3.3.3.2" xref="S2.SS1.p2.3.m3.3.3.3.cmml"><msub id="S2.SS1.p2.3.m3.2.2.1.1" xref="S2.SS1.p2.3.m3.2.2.1.1.cmml"><mi id="S2.SS1.p2.3.m3.2.2.1.1.2" xref="S2.SS1.p2.3.m3.2.2.1.1.2.cmml">q</mi><mn id="S2.SS1.p2.3.m3.2.2.1.1.3" xref="S2.SS1.p2.3.m3.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.SS1.p2.3.m3.3.3.2.3" xref="S2.SS1.p2.3.m3.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS1.p2.3.m3.1.1" xref="S2.SS1.p2.3.m3.1.1.cmml">…</mi><mo id="S2.SS1.p2.3.m3.3.3.2.4" xref="S2.SS1.p2.3.m3.3.3.3.cmml">,</mo><msub id="S2.SS1.p2.3.m3.3.3.2.2" xref="S2.SS1.p2.3.m3.3.3.2.2.cmml"><mi id="S2.SS1.p2.3.m3.3.3.2.2.2" xref="S2.SS1.p2.3.m3.3.3.2.2.2.cmml">q</mi><mi id="S2.SS1.p2.3.m3.3.3.2.2.3" xref="S2.SS1.p2.3.m3.3.3.2.2.3.cmml">n</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.3.m3.3b"><list id="S2.SS1.p2.3.m3.3.3.3.cmml" xref="S2.SS1.p2.3.m3.3.3.2"><apply id="S2.SS1.p2.3.m3.2.2.1.1.cmml" xref="S2.SS1.p2.3.m3.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS1.p2.3.m3.2.2.1.1.1.cmml" xref="S2.SS1.p2.3.m3.2.2.1.1">subscript</csymbol><ci id="S2.SS1.p2.3.m3.2.2.1.1.2.cmml" xref="S2.SS1.p2.3.m3.2.2.1.1.2">𝑞</ci><cn type="integer" id="S2.SS1.p2.3.m3.2.2.1.1.3.cmml" xref="S2.SS1.p2.3.m3.2.2.1.1.3">1</cn></apply><ci id="S2.SS1.p2.3.m3.1.1.cmml" xref="S2.SS1.p2.3.m3.1.1">…</ci><apply id="S2.SS1.p2.3.m3.3.3.2.2.cmml" xref="S2.SS1.p2.3.m3.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS1.p2.3.m3.3.3.2.2.1.cmml" xref="S2.SS1.p2.3.m3.3.3.2.2">subscript</csymbol><ci id="S2.SS1.p2.3.m3.3.3.2.2.2.cmml" xref="S2.SS1.p2.3.m3.3.3.2.2.2">𝑞</ci><ci id="S2.SS1.p2.3.m3.3.3.2.2.3.cmml" xref="S2.SS1.p2.3.m3.3.3.2.2.3">𝑛</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.3.m3.3c">q_{1},\ldots,q_{n}</annotation></semantics></math> only given access to <math id="S2.SS1.p2.4.m4.1" class="ltx_Math" alttext="\phi(\cdot)" display="inline"><semantics id="S2.SS1.p2.4.m4.1a"><mrow id="S2.SS1.p2.4.m4.1.2" xref="S2.SS1.p2.4.m4.1.2.cmml"><mi id="S2.SS1.p2.4.m4.1.2.2" xref="S2.SS1.p2.4.m4.1.2.2.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS1.p2.4.m4.1.2.1" xref="S2.SS1.p2.4.m4.1.2.1.cmml">​</mo><mrow id="S2.SS1.p2.4.m4.1.2.3.2" xref="S2.SS1.p2.4.m4.1.2.cmml"><mo stretchy="false" id="S2.SS1.p2.4.m4.1.2.3.2.1" xref="S2.SS1.p2.4.m4.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S2.SS1.p2.4.m4.1.1" xref="S2.SS1.p2.4.m4.1.1.cmml">⋅</mo><mo stretchy="false" id="S2.SS1.p2.4.m4.1.2.3.2.2" xref="S2.SS1.p2.4.m4.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS1.p2.4.m4.1b"><apply id="S2.SS1.p2.4.m4.1.2.cmml" xref="S2.SS1.p2.4.m4.1.2"><times id="S2.SS1.p2.4.m4.1.2.1.cmml" xref="S2.SS1.p2.4.m4.1.2.1"></times><ci id="S2.SS1.p2.4.m4.1.2.2.cmml" xref="S2.SS1.p2.4.m4.1.2.2">italic-ϕ</ci><ci id="S2.SS1.p2.4.m4.1.1.cmml" xref="S2.SS1.p2.4.m4.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS1.p2.4.m4.1c">\phi(\cdot)</annotation></semantics></math>.
In <a href="#S2.SS2" title="2.2 Method: Contrast-Consistent Search ‣ 2 Problem Statement and Framework ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">2.2</span></a> we introduce a method for this problem that attains high accuracy (<a href="#S3" title="3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">3</span></a>), demonstrating that this task is tractable.</p>
</div>
</section>
<section id="S2.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">2.2 </span>Method: Contrast-Consistent Search</h3>

<div id="S2.SS2.p1" class="ltx_para ltx_noindent">
<p id="S2.SS2.p1.7" class="ltx_p">To make progress on the goal described above, we exploit the fact that truth has special structure: it satisfies consistency properties that few other features in a language model are likely to satisfy.
Our method, Contrast-Consistent Search (CCS), leverages this idea by finding a direction in activation space that is consistent across negations.
As we illustrate in <a href="#S1.F1" title="In 1 Introduction ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">1</span></a>, CCS works by (1) answering each question <math id="S2.SS2.p1.1.m1.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S2.SS2.p1.1.m1.1a"><msub id="S2.SS2.p1.1.m1.1.1" xref="S2.SS2.p1.1.m1.1.1.cmml"><mi id="S2.SS2.p1.1.m1.1.1.2" xref="S2.SS2.p1.1.m1.1.1.2.cmml">q</mi><mi id="S2.SS2.p1.1.m1.1.1.3" xref="S2.SS2.p1.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.1.m1.1b"><apply id="S2.SS2.p1.1.m1.1.1.cmml" xref="S2.SS2.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.1.m1.1.1.1.cmml" xref="S2.SS2.p1.1.m1.1.1">subscript</csymbol><ci id="S2.SS2.p1.1.m1.1.1.2.cmml" xref="S2.SS2.p1.1.m1.1.1.2">𝑞</ci><ci id="S2.SS2.p1.1.m1.1.1.3.cmml" xref="S2.SS2.p1.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.1.m1.1c">q_{i}</annotation></semantics></math> as both “Yes” (<math id="S2.SS2.p1.2.m2.1" class="ltx_Math" alttext="x_{i}^{+}" display="inline"><semantics id="S2.SS2.p1.2.m2.1a"><msubsup id="S2.SS2.p1.2.m2.1.1" xref="S2.SS2.p1.2.m2.1.1.cmml"><mi id="S2.SS2.p1.2.m2.1.1.2.2" xref="S2.SS2.p1.2.m2.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p1.2.m2.1.1.2.3" xref="S2.SS2.p1.2.m2.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p1.2.m2.1.1.3" xref="S2.SS2.p1.2.m2.1.1.3.cmml">+</mo></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.2.m2.1b"><apply id="S2.SS2.p1.2.m2.1.1.cmml" xref="S2.SS2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.2.m2.1.1.1.cmml" xref="S2.SS2.p1.2.m2.1.1">superscript</csymbol><apply id="S2.SS2.p1.2.m2.1.1.2.cmml" xref="S2.SS2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.2.m2.1.1.2.1.cmml" xref="S2.SS2.p1.2.m2.1.1">subscript</csymbol><ci id="S2.SS2.p1.2.m2.1.1.2.2.cmml" xref="S2.SS2.p1.2.m2.1.1.2.2">𝑥</ci><ci id="S2.SS2.p1.2.m2.1.1.2.3.cmml" xref="S2.SS2.p1.2.m2.1.1.2.3">𝑖</ci></apply><plus id="S2.SS2.p1.2.m2.1.1.3.cmml" xref="S2.SS2.p1.2.m2.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.2.m2.1c">x_{i}^{+}</annotation></semantics></math>) and “No” (<math id="S2.SS2.p1.3.m3.1" class="ltx_Math" alttext="x_{i}^{-}" display="inline"><semantics id="S2.SS2.p1.3.m3.1a"><msubsup id="S2.SS2.p1.3.m3.1.1" xref="S2.SS2.p1.3.m3.1.1.cmml"><mi id="S2.SS2.p1.3.m3.1.1.2.2" xref="S2.SS2.p1.3.m3.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p1.3.m3.1.1.2.3" xref="S2.SS2.p1.3.m3.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p1.3.m3.1.1.3" xref="S2.SS2.p1.3.m3.1.1.3.cmml">−</mo></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.3.m3.1b"><apply id="S2.SS2.p1.3.m3.1.1.cmml" xref="S2.SS2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.3.m3.1.1.1.cmml" xref="S2.SS2.p1.3.m3.1.1">superscript</csymbol><apply id="S2.SS2.p1.3.m3.1.1.2.cmml" xref="S2.SS2.p1.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.3.m3.1.1.2.1.cmml" xref="S2.SS2.p1.3.m3.1.1">subscript</csymbol><ci id="S2.SS2.p1.3.m3.1.1.2.2.cmml" xref="S2.SS2.p1.3.m3.1.1.2.2">𝑥</ci><ci id="S2.SS2.p1.3.m3.1.1.2.3.cmml" xref="S2.SS2.p1.3.m3.1.1.2.3">𝑖</ci></apply><minus id="S2.SS2.p1.3.m3.1.1.3.cmml" xref="S2.SS2.p1.3.m3.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.3.m3.1c">x_{i}^{-}</annotation></semantics></math>), (2) computing the representations <math id="S2.SS2.p1.4.m4.1" class="ltx_Math" alttext="\phi(x_{i}^{+})" display="inline"><semantics id="S2.SS2.p1.4.m4.1a"><mrow id="S2.SS2.p1.4.m4.1.1" xref="S2.SS2.p1.4.m4.1.1.cmml"><mi id="S2.SS2.p1.4.m4.1.1.3" xref="S2.SS2.p1.4.m4.1.1.3.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p1.4.m4.1.1.2" xref="S2.SS2.p1.4.m4.1.1.2.cmml">​</mo><mrow id="S2.SS2.p1.4.m4.1.1.1.1" xref="S2.SS2.p1.4.m4.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p1.4.m4.1.1.1.1.2" xref="S2.SS2.p1.4.m4.1.1.1.1.1.cmml">(</mo><msubsup id="S2.SS2.p1.4.m4.1.1.1.1.1" xref="S2.SS2.p1.4.m4.1.1.1.1.1.cmml"><mi id="S2.SS2.p1.4.m4.1.1.1.1.1.2.2" xref="S2.SS2.p1.4.m4.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p1.4.m4.1.1.1.1.1.2.3" xref="S2.SS2.p1.4.m4.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p1.4.m4.1.1.1.1.1.3" xref="S2.SS2.p1.4.m4.1.1.1.1.1.3.cmml">+</mo></msubsup><mo stretchy="false" id="S2.SS2.p1.4.m4.1.1.1.1.3" xref="S2.SS2.p1.4.m4.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.4.m4.1b"><apply id="S2.SS2.p1.4.m4.1.1.cmml" xref="S2.SS2.p1.4.m4.1.1"><times id="S2.SS2.p1.4.m4.1.1.2.cmml" xref="S2.SS2.p1.4.m4.1.1.2"></times><ci id="S2.SS2.p1.4.m4.1.1.3.cmml" xref="S2.SS2.p1.4.m4.1.1.3">italic-ϕ</ci><apply id="S2.SS2.p1.4.m4.1.1.1.1.1.cmml" xref="S2.SS2.p1.4.m4.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.4.m4.1.1.1.1.1.1.cmml" xref="S2.SS2.p1.4.m4.1.1.1.1">superscript</csymbol><apply id="S2.SS2.p1.4.m4.1.1.1.1.1.2.cmml" xref="S2.SS2.p1.4.m4.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.4.m4.1.1.1.1.1.2.1.cmml" xref="S2.SS2.p1.4.m4.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p1.4.m4.1.1.1.1.1.2.2.cmml" xref="S2.SS2.p1.4.m4.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.SS2.p1.4.m4.1.1.1.1.1.2.3.cmml" xref="S2.SS2.p1.4.m4.1.1.1.1.1.2.3">𝑖</ci></apply><plus id="S2.SS2.p1.4.m4.1.1.1.1.1.3.cmml" xref="S2.SS2.p1.4.m4.1.1.1.1.1.3"></plus></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.4.m4.1c">\phi(x_{i}^{+})</annotation></semantics></math> and <math id="S2.SS2.p1.5.m5.1" class="ltx_Math" alttext="\phi(x_{i}^{-})" display="inline"><semantics id="S2.SS2.p1.5.m5.1a"><mrow id="S2.SS2.p1.5.m5.1.1" xref="S2.SS2.p1.5.m5.1.1.cmml"><mi id="S2.SS2.p1.5.m5.1.1.3" xref="S2.SS2.p1.5.m5.1.1.3.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p1.5.m5.1.1.2" xref="S2.SS2.p1.5.m5.1.1.2.cmml">​</mo><mrow id="S2.SS2.p1.5.m5.1.1.1.1" xref="S2.SS2.p1.5.m5.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p1.5.m5.1.1.1.1.2" xref="S2.SS2.p1.5.m5.1.1.1.1.1.cmml">(</mo><msubsup id="S2.SS2.p1.5.m5.1.1.1.1.1" xref="S2.SS2.p1.5.m5.1.1.1.1.1.cmml"><mi id="S2.SS2.p1.5.m5.1.1.1.1.1.2.2" xref="S2.SS2.p1.5.m5.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p1.5.m5.1.1.1.1.1.2.3" xref="S2.SS2.p1.5.m5.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p1.5.m5.1.1.1.1.1.3" xref="S2.SS2.p1.5.m5.1.1.1.1.1.3.cmml">−</mo></msubsup><mo stretchy="false" id="S2.SS2.p1.5.m5.1.1.1.1.3" xref="S2.SS2.p1.5.m5.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.5.m5.1b"><apply id="S2.SS2.p1.5.m5.1.1.cmml" xref="S2.SS2.p1.5.m5.1.1"><times id="S2.SS2.p1.5.m5.1.1.2.cmml" xref="S2.SS2.p1.5.m5.1.1.2"></times><ci id="S2.SS2.p1.5.m5.1.1.3.cmml" xref="S2.SS2.p1.5.m5.1.1.3">italic-ϕ</ci><apply id="S2.SS2.p1.5.m5.1.1.1.1.1.cmml" xref="S2.SS2.p1.5.m5.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.5.m5.1.1.1.1.1.1.cmml" xref="S2.SS2.p1.5.m5.1.1.1.1">superscript</csymbol><apply id="S2.SS2.p1.5.m5.1.1.1.1.1.2.cmml" xref="S2.SS2.p1.5.m5.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.5.m5.1.1.1.1.1.2.1.cmml" xref="S2.SS2.p1.5.m5.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p1.5.m5.1.1.1.1.1.2.2.cmml" xref="S2.SS2.p1.5.m5.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.SS2.p1.5.m5.1.1.1.1.1.2.3.cmml" xref="S2.SS2.p1.5.m5.1.1.1.1.1.2.3">𝑖</ci></apply><minus id="S2.SS2.p1.5.m5.1.1.1.1.1.3.cmml" xref="S2.SS2.p1.5.m5.1.1.1.1.1.3"></minus></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.5.m5.1c">\phi(x_{i}^{-})</annotation></semantics></math> of each answer, (3) mapping the answer representations to probabilities <math id="S2.SS2.p1.6.m6.1" class="ltx_Math" alttext="p_{i}^{+}" display="inline"><semantics id="S2.SS2.p1.6.m6.1a"><msubsup id="S2.SS2.p1.6.m6.1.1" xref="S2.SS2.p1.6.m6.1.1.cmml"><mi id="S2.SS2.p1.6.m6.1.1.2.2" xref="S2.SS2.p1.6.m6.1.1.2.2.cmml">p</mi><mi id="S2.SS2.p1.6.m6.1.1.2.3" xref="S2.SS2.p1.6.m6.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p1.6.m6.1.1.3" xref="S2.SS2.p1.6.m6.1.1.3.cmml">+</mo></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.6.m6.1b"><apply id="S2.SS2.p1.6.m6.1.1.cmml" xref="S2.SS2.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.6.m6.1.1.1.cmml" xref="S2.SS2.p1.6.m6.1.1">superscript</csymbol><apply id="S2.SS2.p1.6.m6.1.1.2.cmml" xref="S2.SS2.p1.6.m6.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.6.m6.1.1.2.1.cmml" xref="S2.SS2.p1.6.m6.1.1">subscript</csymbol><ci id="S2.SS2.p1.6.m6.1.1.2.2.cmml" xref="S2.SS2.p1.6.m6.1.1.2.2">𝑝</ci><ci id="S2.SS2.p1.6.m6.1.1.2.3.cmml" xref="S2.SS2.p1.6.m6.1.1.2.3">𝑖</ci></apply><plus id="S2.SS2.p1.6.m6.1.1.3.cmml" xref="S2.SS2.p1.6.m6.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.6.m6.1c">p_{i}^{+}</annotation></semantics></math> and <math id="S2.SS2.p1.7.m7.1" class="ltx_Math" alttext="p_{i}^{-}" display="inline"><semantics id="S2.SS2.p1.7.m7.1a"><msubsup id="S2.SS2.p1.7.m7.1.1" xref="S2.SS2.p1.7.m7.1.1.cmml"><mi id="S2.SS2.p1.7.m7.1.1.2.2" xref="S2.SS2.p1.7.m7.1.1.2.2.cmml">p</mi><mi id="S2.SS2.p1.7.m7.1.1.2.3" xref="S2.SS2.p1.7.m7.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p1.7.m7.1.1.3" xref="S2.SS2.p1.7.m7.1.1.3.cmml">−</mo></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p1.7.m7.1b"><apply id="S2.SS2.p1.7.m7.1.1.cmml" xref="S2.SS2.p1.7.m7.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.7.m7.1.1.1.cmml" xref="S2.SS2.p1.7.m7.1.1">superscript</csymbol><apply id="S2.SS2.p1.7.m7.1.1.2.cmml" xref="S2.SS2.p1.7.m7.1.1"><csymbol cd="ambiguous" id="S2.SS2.p1.7.m7.1.1.2.1.cmml" xref="S2.SS2.p1.7.m7.1.1">subscript</csymbol><ci id="S2.SS2.p1.7.m7.1.1.2.2.cmml" xref="S2.SS2.p1.7.m7.1.1.2.2">𝑝</ci><ci id="S2.SS2.p1.7.m7.1.1.2.3.cmml" xref="S2.SS2.p1.7.m7.1.1.2.3">𝑖</ci></apply><minus id="S2.SS2.p1.7.m7.1.1.3.cmml" xref="S2.SS2.p1.7.m7.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p1.7.m7.1c">p_{i}^{-}</annotation></semantics></math> of being true, then (4) optimizing that mapping so that the probabilities are both consistent and confident.</p>
</div>
<div id="S2.SS2.p2" class="ltx_para ltx_noindent">
<p id="S2.SS2.p2.7" class="ltx_p">Concretely, the input to CCS is a set of Yes-No questions, <math id="S2.SS2.p2.1.m1.3" class="ltx_Math" alttext="q_{1},\ldots,q_{n}" display="inline"><semantics id="S2.SS2.p2.1.m1.3a"><mrow id="S2.SS2.p2.1.m1.3.3.2" xref="S2.SS2.p2.1.m1.3.3.3.cmml"><msub id="S2.SS2.p2.1.m1.2.2.1.1" xref="S2.SS2.p2.1.m1.2.2.1.1.cmml"><mi id="S2.SS2.p2.1.m1.2.2.1.1.2" xref="S2.SS2.p2.1.m1.2.2.1.1.2.cmml">q</mi><mn id="S2.SS2.p2.1.m1.2.2.1.1.3" xref="S2.SS2.p2.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.SS2.p2.1.m1.3.3.2.3" xref="S2.SS2.p2.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS2.p2.1.m1.1.1" xref="S2.SS2.p2.1.m1.1.1.cmml">…</mi><mo id="S2.SS2.p2.1.m1.3.3.2.4" xref="S2.SS2.p2.1.m1.3.3.3.cmml">,</mo><msub id="S2.SS2.p2.1.m1.3.3.2.2" xref="S2.SS2.p2.1.m1.3.3.2.2.cmml"><mi id="S2.SS2.p2.1.m1.3.3.2.2.2" xref="S2.SS2.p2.1.m1.3.3.2.2.2.cmml">q</mi><mi id="S2.SS2.p2.1.m1.3.3.2.2.3" xref="S2.SS2.p2.1.m1.3.3.2.2.3.cmml">n</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.1.m1.3b"><list id="S2.SS2.p2.1.m1.3.3.3.cmml" xref="S2.SS2.p2.1.m1.3.3.2"><apply id="S2.SS2.p2.1.m1.2.2.1.1.cmml" xref="S2.SS2.p2.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS2.p2.1.m1.2.2.1.1.1.cmml" xref="S2.SS2.p2.1.m1.2.2.1.1">subscript</csymbol><ci id="S2.SS2.p2.1.m1.2.2.1.1.2.cmml" xref="S2.SS2.p2.1.m1.2.2.1.1.2">𝑞</ci><cn type="integer" id="S2.SS2.p2.1.m1.2.2.1.1.3.cmml" xref="S2.SS2.p2.1.m1.2.2.1.1.3">1</cn></apply><ci id="S2.SS2.p2.1.m1.1.1.cmml" xref="S2.SS2.p2.1.m1.1.1">…</ci><apply id="S2.SS2.p2.1.m1.3.3.2.2.cmml" xref="S2.SS2.p2.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS2.p2.1.m1.3.3.2.2.1.cmml" xref="S2.SS2.p2.1.m1.3.3.2.2">subscript</csymbol><ci id="S2.SS2.p2.1.m1.3.3.2.2.2.cmml" xref="S2.SS2.p2.1.m1.3.3.2.2.2">𝑞</ci><ci id="S2.SS2.p2.1.m1.3.3.2.2.3.cmml" xref="S2.SS2.p2.1.m1.3.3.2.2.3">𝑛</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.1.m1.3c">q_{1},\ldots,q_{n}</annotation></semantics></math>, and access to a pretrained model’s representations, <math id="S2.SS2.p2.2.m2.1" class="ltx_Math" alttext="\phi(\cdot)" display="inline"><semantics id="S2.SS2.p2.2.m2.1a"><mrow id="S2.SS2.p2.2.m2.1.2" xref="S2.SS2.p2.2.m2.1.2.cmml"><mi id="S2.SS2.p2.2.m2.1.2.2" xref="S2.SS2.p2.2.m2.1.2.2.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p2.2.m2.1.2.1" xref="S2.SS2.p2.2.m2.1.2.1.cmml">​</mo><mrow id="S2.SS2.p2.2.m2.1.2.3.2" xref="S2.SS2.p2.2.m2.1.2.cmml"><mo stretchy="false" id="S2.SS2.p2.2.m2.1.2.3.2.1" xref="S2.SS2.p2.2.m2.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S2.SS2.p2.2.m2.1.1" xref="S2.SS2.p2.2.m2.1.1.cmml">⋅</mo><mo stretchy="false" id="S2.SS2.p2.2.m2.1.2.3.2.2" xref="S2.SS2.p2.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.2.m2.1b"><apply id="S2.SS2.p2.2.m2.1.2.cmml" xref="S2.SS2.p2.2.m2.1.2"><times id="S2.SS2.p2.2.m2.1.2.1.cmml" xref="S2.SS2.p2.2.m2.1.2.1"></times><ci id="S2.SS2.p2.2.m2.1.2.2.cmml" xref="S2.SS2.p2.2.m2.1.2.2">italic-ϕ</ci><ci id="S2.SS2.p2.2.m2.1.1.cmml" xref="S2.SS2.p2.2.m2.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.2.m2.1c">\phi(\cdot)</annotation></semantics></math>; the output of CCS is a lightweight probe on top of <math id="S2.SS2.p2.3.m3.1" class="ltx_Math" alttext="\phi(\cdot)" display="inline"><semantics id="S2.SS2.p2.3.m3.1a"><mrow id="S2.SS2.p2.3.m3.1.2" xref="S2.SS2.p2.3.m3.1.2.cmml"><mi id="S2.SS2.p2.3.m3.1.2.2" xref="S2.SS2.p2.3.m3.1.2.2.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p2.3.m3.1.2.1" xref="S2.SS2.p2.3.m3.1.2.1.cmml">​</mo><mrow id="S2.SS2.p2.3.m3.1.2.3.2" xref="S2.SS2.p2.3.m3.1.2.cmml"><mo stretchy="false" id="S2.SS2.p2.3.m3.1.2.3.2.1" xref="S2.SS2.p2.3.m3.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S2.SS2.p2.3.m3.1.1" xref="S2.SS2.p2.3.m3.1.1.cmml">⋅</mo><mo stretchy="false" id="S2.SS2.p2.3.m3.1.2.3.2.2" xref="S2.SS2.p2.3.m3.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.3.m3.1b"><apply id="S2.SS2.p2.3.m3.1.2.cmml" xref="S2.SS2.p2.3.m3.1.2"><times id="S2.SS2.p2.3.m3.1.2.1.cmml" xref="S2.SS2.p2.3.m3.1.2.1"></times><ci id="S2.SS2.p2.3.m3.1.2.2.cmml" xref="S2.SS2.p2.3.m3.1.2.2">italic-ϕ</ci><ci id="S2.SS2.p2.3.m3.1.1.cmml" xref="S2.SS2.p2.3.m3.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.3.m3.1c">\phi(\cdot)</annotation></semantics></math> that can answer new questions.
Here, <math id="S2.SS2.p2.4.m4.1" class="ltx_Math" alttext="\phi(\cdot)" display="inline"><semantics id="S2.SS2.p2.4.m4.1a"><mrow id="S2.SS2.p2.4.m4.1.2" xref="S2.SS2.p2.4.m4.1.2.cmml"><mi id="S2.SS2.p2.4.m4.1.2.2" xref="S2.SS2.p2.4.m4.1.2.2.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p2.4.m4.1.2.1" xref="S2.SS2.p2.4.m4.1.2.1.cmml">​</mo><mrow id="S2.SS2.p2.4.m4.1.2.3.2" xref="S2.SS2.p2.4.m4.1.2.cmml"><mo stretchy="false" id="S2.SS2.p2.4.m4.1.2.3.2.1" xref="S2.SS2.p2.4.m4.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S2.SS2.p2.4.m4.1.1" xref="S2.SS2.p2.4.m4.1.1.cmml">⋅</mo><mo stretchy="false" id="S2.SS2.p2.4.m4.1.2.3.2.2" xref="S2.SS2.p2.4.m4.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.4.m4.1b"><apply id="S2.SS2.p2.4.m4.1.2.cmml" xref="S2.SS2.p2.4.m4.1.2"><times id="S2.SS2.p2.4.m4.1.2.1.cmml" xref="S2.SS2.p2.4.m4.1.2.1"></times><ci id="S2.SS2.p2.4.m4.1.2.2.cmml" xref="S2.SS2.p2.4.m4.1.2.2">italic-ϕ</ci><ci id="S2.SS2.p2.4.m4.1.1.cmml" xref="S2.SS2.p2.4.m4.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.4.m4.1c">\phi(\cdot)</annotation></semantics></math> is fixed but should contain useful information about the answers to <math id="S2.SS2.p2.5.m5.3" class="ltx_Math" alttext="q_{1},\ldots,q_{n}" display="inline"><semantics id="S2.SS2.p2.5.m5.3a"><mrow id="S2.SS2.p2.5.m5.3.3.2" xref="S2.SS2.p2.5.m5.3.3.3.cmml"><msub id="S2.SS2.p2.5.m5.2.2.1.1" xref="S2.SS2.p2.5.m5.2.2.1.1.cmml"><mi id="S2.SS2.p2.5.m5.2.2.1.1.2" xref="S2.SS2.p2.5.m5.2.2.1.1.2.cmml">q</mi><mn id="S2.SS2.p2.5.m5.2.2.1.1.3" xref="S2.SS2.p2.5.m5.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.SS2.p2.5.m5.3.3.2.3" xref="S2.SS2.p2.5.m5.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS2.p2.5.m5.1.1" xref="S2.SS2.p2.5.m5.1.1.cmml">…</mi><mo id="S2.SS2.p2.5.m5.3.3.2.4" xref="S2.SS2.p2.5.m5.3.3.3.cmml">,</mo><msub id="S2.SS2.p2.5.m5.3.3.2.2" xref="S2.SS2.p2.5.m5.3.3.2.2.cmml"><mi id="S2.SS2.p2.5.m5.3.3.2.2.2" xref="S2.SS2.p2.5.m5.3.3.2.2.2.cmml">q</mi><mi id="S2.SS2.p2.5.m5.3.3.2.2.3" xref="S2.SS2.p2.5.m5.3.3.2.2.3.cmml">n</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.5.m5.3b"><list id="S2.SS2.p2.5.m5.3.3.3.cmml" xref="S2.SS2.p2.5.m5.3.3.2"><apply id="S2.SS2.p2.5.m5.2.2.1.1.cmml" xref="S2.SS2.p2.5.m5.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS2.p2.5.m5.2.2.1.1.1.cmml" xref="S2.SS2.p2.5.m5.2.2.1.1">subscript</csymbol><ci id="S2.SS2.p2.5.m5.2.2.1.1.2.cmml" xref="S2.SS2.p2.5.m5.2.2.1.1.2">𝑞</ci><cn type="integer" id="S2.SS2.p2.5.m5.2.2.1.1.3.cmml" xref="S2.SS2.p2.5.m5.2.2.1.1.3">1</cn></apply><ci id="S2.SS2.p2.5.m5.1.1.cmml" xref="S2.SS2.p2.5.m5.1.1">…</ci><apply id="S2.SS2.p2.5.m5.3.3.2.2.cmml" xref="S2.SS2.p2.5.m5.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS2.p2.5.m5.3.3.2.2.1.cmml" xref="S2.SS2.p2.5.m5.3.3.2.2">subscript</csymbol><ci id="S2.SS2.p2.5.m5.3.3.2.2.2.cmml" xref="S2.SS2.p2.5.m5.3.3.2.2.2">𝑞</ci><ci id="S2.SS2.p2.5.m5.3.3.2.2.3.cmml" xref="S2.SS2.p2.5.m5.3.3.2.2.3">𝑛</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.5.m5.3c">q_{1},\ldots,q_{n}</annotation></semantics></math>, in the sense that if one <span id="S2.SS2.p2.7.1" class="ltx_text ltx_font_italic">did</span> (hypothetically) have access to the ground-truth labels for <math id="S2.SS2.p2.6.m6.3" class="ltx_Math" alttext="q_{1},\ldots,q_{n}" display="inline"><semantics id="S2.SS2.p2.6.m6.3a"><mrow id="S2.SS2.p2.6.m6.3.3.2" xref="S2.SS2.p2.6.m6.3.3.3.cmml"><msub id="S2.SS2.p2.6.m6.2.2.1.1" xref="S2.SS2.p2.6.m6.2.2.1.1.cmml"><mi id="S2.SS2.p2.6.m6.2.2.1.1.2" xref="S2.SS2.p2.6.m6.2.2.1.1.2.cmml">q</mi><mn id="S2.SS2.p2.6.m6.2.2.1.1.3" xref="S2.SS2.p2.6.m6.2.2.1.1.3.cmml">1</mn></msub><mo id="S2.SS2.p2.6.m6.3.3.2.3" xref="S2.SS2.p2.6.m6.3.3.3.cmml">,</mo><mi mathvariant="normal" id="S2.SS2.p2.6.m6.1.1" xref="S2.SS2.p2.6.m6.1.1.cmml">…</mi><mo id="S2.SS2.p2.6.m6.3.3.2.4" xref="S2.SS2.p2.6.m6.3.3.3.cmml">,</mo><msub id="S2.SS2.p2.6.m6.3.3.2.2" xref="S2.SS2.p2.6.m6.3.3.2.2.cmml"><mi id="S2.SS2.p2.6.m6.3.3.2.2.2" xref="S2.SS2.p2.6.m6.3.3.2.2.2.cmml">q</mi><mi id="S2.SS2.p2.6.m6.3.3.2.2.3" xref="S2.SS2.p2.6.m6.3.3.2.2.3.cmml">n</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.6.m6.3b"><list id="S2.SS2.p2.6.m6.3.3.3.cmml" xref="S2.SS2.p2.6.m6.3.3.2"><apply id="S2.SS2.p2.6.m6.2.2.1.1.cmml" xref="S2.SS2.p2.6.m6.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS2.p2.6.m6.2.2.1.1.1.cmml" xref="S2.SS2.p2.6.m6.2.2.1.1">subscript</csymbol><ci id="S2.SS2.p2.6.m6.2.2.1.1.2.cmml" xref="S2.SS2.p2.6.m6.2.2.1.1.2">𝑞</ci><cn type="integer" id="S2.SS2.p2.6.m6.2.2.1.1.3.cmml" xref="S2.SS2.p2.6.m6.2.2.1.1.3">1</cn></apply><ci id="S2.SS2.p2.6.m6.1.1.cmml" xref="S2.SS2.p2.6.m6.1.1">…</ci><apply id="S2.SS2.p2.6.m6.3.3.2.2.cmml" xref="S2.SS2.p2.6.m6.3.3.2.2"><csymbol cd="ambiguous" id="S2.SS2.p2.6.m6.3.3.2.2.1.cmml" xref="S2.SS2.p2.6.m6.3.3.2.2">subscript</csymbol><ci id="S2.SS2.p2.6.m6.3.3.2.2.2.cmml" xref="S2.SS2.p2.6.m6.3.3.2.2.2">𝑞</ci><ci id="S2.SS2.p2.6.m6.3.3.2.2.3.cmml" xref="S2.SS2.p2.6.m6.3.3.2.2.3">𝑛</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.6.m6.3c">q_{1},\ldots,q_{n}</annotation></semantics></math>, one would be able to train a small supervised probe on <math id="S2.SS2.p2.7.m7.1" class="ltx_Math" alttext="\phi(\cdot)" display="inline"><semantics id="S2.SS2.p2.7.m7.1a"><mrow id="S2.SS2.p2.7.m7.1.2" xref="S2.SS2.p2.7.m7.1.2.cmml"><mi id="S2.SS2.p2.7.m7.1.2.2" xref="S2.SS2.p2.7.m7.1.2.2.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p2.7.m7.1.2.1" xref="S2.SS2.p2.7.m7.1.2.1.cmml">​</mo><mrow id="S2.SS2.p2.7.m7.1.2.3.2" xref="S2.SS2.p2.7.m7.1.2.cmml"><mo stretchy="false" id="S2.SS2.p2.7.m7.1.2.3.2.1" xref="S2.SS2.p2.7.m7.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S2.SS2.p2.7.m7.1.1" xref="S2.SS2.p2.7.m7.1.1.cmml">⋅</mo><mo stretchy="false" id="S2.SS2.p2.7.m7.1.2.3.2.2" xref="S2.SS2.p2.7.m7.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p2.7.m7.1b"><apply id="S2.SS2.p2.7.m7.1.2.cmml" xref="S2.SS2.p2.7.m7.1.2"><times id="S2.SS2.p2.7.m7.1.2.1.cmml" xref="S2.SS2.p2.7.m7.1.2.1"></times><ci id="S2.SS2.p2.7.m7.1.2.2.cmml" xref="S2.SS2.p2.7.m7.1.2.2">italic-ϕ</ci><ci id="S2.SS2.p2.7.m7.1.1.cmml" xref="S2.SS2.p2.7.m7.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p2.7.m7.1c">\phi(\cdot)</annotation></semantics></math> that attains high accuracy.
Importantly, CCS does not modify the weights of the pretrained model and it does not use labels.</p>
</div>
<div id="S2.SS2.p3" class="ltx_para ltx_noindent">
<p id="S2.SS2.p3.13" class="ltx_p"><span id="S2.SS2.p3.13.1" class="ltx_text ltx_font_bold">Constructing contrast pairs.</span>
An important property that truth satisfies is negation consistency: the answer to a clear-cut question cannot be both “Yes” and “No” at the same time, as these are negations of each other.
Probabilistically, for each question <math id="S2.SS2.p3.1.m1.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S2.SS2.p3.1.m1.1a"><msub id="S2.SS2.p3.1.m1.1.1" xref="S2.SS2.p3.1.m1.1.1.cmml"><mi id="S2.SS2.p3.1.m1.1.1.2" xref="S2.SS2.p3.1.m1.1.1.2.cmml">q</mi><mi id="S2.SS2.p3.1.m1.1.1.3" xref="S2.SS2.p3.1.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.1.m1.1b"><apply id="S2.SS2.p3.1.m1.1.1.cmml" xref="S2.SS2.p3.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.1.m1.1.1.1.cmml" xref="S2.SS2.p3.1.m1.1.1">subscript</csymbol><ci id="S2.SS2.p3.1.m1.1.1.2.cmml" xref="S2.SS2.p3.1.m1.1.1.2">𝑞</ci><ci id="S2.SS2.p3.1.m1.1.1.3.cmml" xref="S2.SS2.p3.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.1.m1.1c">q_{i}</annotation></semantics></math>, the probability that the answer to <math id="S2.SS2.p3.2.m2.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S2.SS2.p3.2.m2.1a"><msub id="S2.SS2.p3.2.m2.1.1" xref="S2.SS2.p3.2.m2.1.1.cmml"><mi id="S2.SS2.p3.2.m2.1.1.2" xref="S2.SS2.p3.2.m2.1.1.2.cmml">q</mi><mi id="S2.SS2.p3.2.m2.1.1.3" xref="S2.SS2.p3.2.m2.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.2.m2.1b"><apply id="S2.SS2.p3.2.m2.1.1.cmml" xref="S2.SS2.p3.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.2.m2.1.1.1.cmml" xref="S2.SS2.p3.2.m2.1.1">subscript</csymbol><ci id="S2.SS2.p3.2.m2.1.1.2.cmml" xref="S2.SS2.p3.2.m2.1.1.2">𝑞</ci><ci id="S2.SS2.p3.2.m2.1.1.3.cmml" xref="S2.SS2.p3.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.2.m2.1c">q_{i}</annotation></semantics></math> is “Yes” should be one minus the probability that the answer to <math id="S2.SS2.p3.3.m3.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S2.SS2.p3.3.m3.1a"><msub id="S2.SS2.p3.3.m3.1.1" xref="S2.SS2.p3.3.m3.1.1.cmml"><mi id="S2.SS2.p3.3.m3.1.1.2" xref="S2.SS2.p3.3.m3.1.1.2.cmml">q</mi><mi id="S2.SS2.p3.3.m3.1.1.3" xref="S2.SS2.p3.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.3.m3.1b"><apply id="S2.SS2.p3.3.m3.1.1.cmml" xref="S2.SS2.p3.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.3.m3.1.1.1.cmml" xref="S2.SS2.p3.3.m3.1.1">subscript</csymbol><ci id="S2.SS2.p3.3.m3.1.1.2.cmml" xref="S2.SS2.p3.3.m3.1.1.2">𝑞</ci><ci id="S2.SS2.p3.3.m3.1.1.3.cmml" xref="S2.SS2.p3.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.3.m3.1c">q_{i}</annotation></semantics></math> is “No”.
To use this property, we begin by constructing contrast pairs: for each question <math id="S2.SS2.p3.4.m4.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S2.SS2.p3.4.m4.1a"><msub id="S2.SS2.p3.4.m4.1.1" xref="S2.SS2.p3.4.m4.1.1.cmml"><mi id="S2.SS2.p3.4.m4.1.1.2" xref="S2.SS2.p3.4.m4.1.1.2.cmml">q</mi><mi id="S2.SS2.p3.4.m4.1.1.3" xref="S2.SS2.p3.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.4.m4.1b"><apply id="S2.SS2.p3.4.m4.1.1.cmml" xref="S2.SS2.p3.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.4.m4.1.1.1.cmml" xref="S2.SS2.p3.4.m4.1.1">subscript</csymbol><ci id="S2.SS2.p3.4.m4.1.1.2.cmml" xref="S2.SS2.p3.4.m4.1.1.2">𝑞</ci><ci id="S2.SS2.p3.4.m4.1.1.3.cmml" xref="S2.SS2.p3.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.4.m4.1c">q_{i}</annotation></semantics></math>, we answer <math id="S2.SS2.p3.5.m5.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S2.SS2.p3.5.m5.1a"><msub id="S2.SS2.p3.5.m5.1.1" xref="S2.SS2.p3.5.m5.1.1.cmml"><mi id="S2.SS2.p3.5.m5.1.1.2" xref="S2.SS2.p3.5.m5.1.1.2.cmml">q</mi><mi id="S2.SS2.p3.5.m5.1.1.3" xref="S2.SS2.p3.5.m5.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.5.m5.1b"><apply id="S2.SS2.p3.5.m5.1.1.cmml" xref="S2.SS2.p3.5.m5.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.5.m5.1.1.1.cmml" xref="S2.SS2.p3.5.m5.1.1">subscript</csymbol><ci id="S2.SS2.p3.5.m5.1.1.2.cmml" xref="S2.SS2.p3.5.m5.1.1.2">𝑞</ci><ci id="S2.SS2.p3.5.m5.1.1.3.cmml" xref="S2.SS2.p3.5.m5.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.5.m5.1c">q_{i}</annotation></semantics></math> both as “Yes”, resulting in the new natural language statement <math id="S2.SS2.p3.6.m6.1" class="ltx_Math" alttext="x_{i}^{+}" display="inline"><semantics id="S2.SS2.p3.6.m6.1a"><msubsup id="S2.SS2.p3.6.m6.1.1" xref="S2.SS2.p3.6.m6.1.1.cmml"><mi id="S2.SS2.p3.6.m6.1.1.2.2" xref="S2.SS2.p3.6.m6.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p3.6.m6.1.1.2.3" xref="S2.SS2.p3.6.m6.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p3.6.m6.1.1.3" xref="S2.SS2.p3.6.m6.1.1.3.cmml">+</mo></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.6.m6.1b"><apply id="S2.SS2.p3.6.m6.1.1.cmml" xref="S2.SS2.p3.6.m6.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.6.m6.1.1.1.cmml" xref="S2.SS2.p3.6.m6.1.1">superscript</csymbol><apply id="S2.SS2.p3.6.m6.1.1.2.cmml" xref="S2.SS2.p3.6.m6.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.6.m6.1.1.2.1.cmml" xref="S2.SS2.p3.6.m6.1.1">subscript</csymbol><ci id="S2.SS2.p3.6.m6.1.1.2.2.cmml" xref="S2.SS2.p3.6.m6.1.1.2.2">𝑥</ci><ci id="S2.SS2.p3.6.m6.1.1.2.3.cmml" xref="S2.SS2.p3.6.m6.1.1.2.3">𝑖</ci></apply><plus id="S2.SS2.p3.6.m6.1.1.3.cmml" xref="S2.SS2.p3.6.m6.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.6.m6.1c">x_{i}^{+}</annotation></semantics></math>, and as “No”, resulting in the natural language statement <math id="S2.SS2.p3.7.m7.1" class="ltx_Math" alttext="x_{i}^{-}" display="inline"><semantics id="S2.SS2.p3.7.m7.1a"><msubsup id="S2.SS2.p3.7.m7.1.1" xref="S2.SS2.p3.7.m7.1.1.cmml"><mi id="S2.SS2.p3.7.m7.1.1.2.2" xref="S2.SS2.p3.7.m7.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p3.7.m7.1.1.2.3" xref="S2.SS2.p3.7.m7.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p3.7.m7.1.1.3" xref="S2.SS2.p3.7.m7.1.1.3.cmml">−</mo></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.7.m7.1b"><apply id="S2.SS2.p3.7.m7.1.1.cmml" xref="S2.SS2.p3.7.m7.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.7.m7.1.1.1.cmml" xref="S2.SS2.p3.7.m7.1.1">superscript</csymbol><apply id="S2.SS2.p3.7.m7.1.1.2.cmml" xref="S2.SS2.p3.7.m7.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.7.m7.1.1.2.1.cmml" xref="S2.SS2.p3.7.m7.1.1">subscript</csymbol><ci id="S2.SS2.p3.7.m7.1.1.2.2.cmml" xref="S2.SS2.p3.7.m7.1.1.2.2">𝑥</ci><ci id="S2.SS2.p3.7.m7.1.1.2.3.cmml" xref="S2.SS2.p3.7.m7.1.1.2.3">𝑖</ci></apply><minus id="S2.SS2.p3.7.m7.1.1.3.cmml" xref="S2.SS2.p3.7.m7.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.7.m7.1c">x_{i}^{-}</annotation></semantics></math>.
We illustrate this in <a href="#S1.F1" title="In 1 Introduction ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">1</span></a> (left).
We will then learn to classify <math id="S2.SS2.p3.8.m8.1" class="ltx_Math" alttext="x_{i}^{+}" display="inline"><semantics id="S2.SS2.p3.8.m8.1a"><msubsup id="S2.SS2.p3.8.m8.1.1" xref="S2.SS2.p3.8.m8.1.1.cmml"><mi id="S2.SS2.p3.8.m8.1.1.2.2" xref="S2.SS2.p3.8.m8.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p3.8.m8.1.1.2.3" xref="S2.SS2.p3.8.m8.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p3.8.m8.1.1.3" xref="S2.SS2.p3.8.m8.1.1.3.cmml">+</mo></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.8.m8.1b"><apply id="S2.SS2.p3.8.m8.1.1.cmml" xref="S2.SS2.p3.8.m8.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.8.m8.1.1.1.cmml" xref="S2.SS2.p3.8.m8.1.1">superscript</csymbol><apply id="S2.SS2.p3.8.m8.1.1.2.cmml" xref="S2.SS2.p3.8.m8.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.8.m8.1.1.2.1.cmml" xref="S2.SS2.p3.8.m8.1.1">subscript</csymbol><ci id="S2.SS2.p3.8.m8.1.1.2.2.cmml" xref="S2.SS2.p3.8.m8.1.1.2.2">𝑥</ci><ci id="S2.SS2.p3.8.m8.1.1.2.3.cmml" xref="S2.SS2.p3.8.m8.1.1.2.3">𝑖</ci></apply><plus id="S2.SS2.p3.8.m8.1.1.3.cmml" xref="S2.SS2.p3.8.m8.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.8.m8.1c">x_{i}^{+}</annotation></semantics></math> and <math id="S2.SS2.p3.9.m9.1" class="ltx_Math" alttext="x_{i}^{-}" display="inline"><semantics id="S2.SS2.p3.9.m9.1a"><msubsup id="S2.SS2.p3.9.m9.1.1" xref="S2.SS2.p3.9.m9.1.1.cmml"><mi id="S2.SS2.p3.9.m9.1.1.2.2" xref="S2.SS2.p3.9.m9.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p3.9.m9.1.1.2.3" xref="S2.SS2.p3.9.m9.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p3.9.m9.1.1.3" xref="S2.SS2.p3.9.m9.1.1.3.cmml">−</mo></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.9.m9.1b"><apply id="S2.SS2.p3.9.m9.1.1.cmml" xref="S2.SS2.p3.9.m9.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.9.m9.1.1.1.cmml" xref="S2.SS2.p3.9.m9.1.1">superscript</csymbol><apply id="S2.SS2.p3.9.m9.1.1.2.cmml" xref="S2.SS2.p3.9.m9.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.9.m9.1.1.2.1.cmml" xref="S2.SS2.p3.9.m9.1.1">subscript</csymbol><ci id="S2.SS2.p3.9.m9.1.1.2.2.cmml" xref="S2.SS2.p3.9.m9.1.1.2.2">𝑥</ci><ci id="S2.SS2.p3.9.m9.1.1.2.3.cmml" xref="S2.SS2.p3.9.m9.1.1.2.3">𝑖</ci></apply><minus id="S2.SS2.p3.9.m9.1.1.3.cmml" xref="S2.SS2.p3.9.m9.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.9.m9.1c">x_{i}^{-}</annotation></semantics></math> as true or false; if <math id="S2.SS2.p3.10.m10.1" class="ltx_Math" alttext="x_{i}^{+}" display="inline"><semantics id="S2.SS2.p3.10.m10.1a"><msubsup id="S2.SS2.p3.10.m10.1.1" xref="S2.SS2.p3.10.m10.1.1.cmml"><mi id="S2.SS2.p3.10.m10.1.1.2.2" xref="S2.SS2.p3.10.m10.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p3.10.m10.1.1.2.3" xref="S2.SS2.p3.10.m10.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p3.10.m10.1.1.3" xref="S2.SS2.p3.10.m10.1.1.3.cmml">+</mo></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.10.m10.1b"><apply id="S2.SS2.p3.10.m10.1.1.cmml" xref="S2.SS2.p3.10.m10.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.10.m10.1.1.1.cmml" xref="S2.SS2.p3.10.m10.1.1">superscript</csymbol><apply id="S2.SS2.p3.10.m10.1.1.2.cmml" xref="S2.SS2.p3.10.m10.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.10.m10.1.1.2.1.cmml" xref="S2.SS2.p3.10.m10.1.1">subscript</csymbol><ci id="S2.SS2.p3.10.m10.1.1.2.2.cmml" xref="S2.SS2.p3.10.m10.1.1.2.2">𝑥</ci><ci id="S2.SS2.p3.10.m10.1.1.2.3.cmml" xref="S2.SS2.p3.10.m10.1.1.2.3">𝑖</ci></apply><plus id="S2.SS2.p3.10.m10.1.1.3.cmml" xref="S2.SS2.p3.10.m10.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.10.m10.1c">x_{i}^{+}</annotation></semantics></math> is true, then the answer to <math id="S2.SS2.p3.11.m11.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S2.SS2.p3.11.m11.1a"><msub id="S2.SS2.p3.11.m11.1.1" xref="S2.SS2.p3.11.m11.1.1.cmml"><mi id="S2.SS2.p3.11.m11.1.1.2" xref="S2.SS2.p3.11.m11.1.1.2.cmml">q</mi><mi id="S2.SS2.p3.11.m11.1.1.3" xref="S2.SS2.p3.11.m11.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.11.m11.1b"><apply id="S2.SS2.p3.11.m11.1.1.cmml" xref="S2.SS2.p3.11.m11.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.11.m11.1.1.1.cmml" xref="S2.SS2.p3.11.m11.1.1">subscript</csymbol><ci id="S2.SS2.p3.11.m11.1.1.2.cmml" xref="S2.SS2.p3.11.m11.1.1.2">𝑞</ci><ci id="S2.SS2.p3.11.m11.1.1.3.cmml" xref="S2.SS2.p3.11.m11.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.11.m11.1c">q_{i}</annotation></semantics></math> should be “Yes”, and if <math id="S2.SS2.p3.12.m12.1" class="ltx_Math" alttext="x_{i}^{-}" display="inline"><semantics id="S2.SS2.p3.12.m12.1a"><msubsup id="S2.SS2.p3.12.m12.1.1" xref="S2.SS2.p3.12.m12.1.1.cmml"><mi id="S2.SS2.p3.12.m12.1.1.2.2" xref="S2.SS2.p3.12.m12.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p3.12.m12.1.1.2.3" xref="S2.SS2.p3.12.m12.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p3.12.m12.1.1.3" xref="S2.SS2.p3.12.m12.1.1.3.cmml">−</mo></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.12.m12.1b"><apply id="S2.SS2.p3.12.m12.1.1.cmml" xref="S2.SS2.p3.12.m12.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.12.m12.1.1.1.cmml" xref="S2.SS2.p3.12.m12.1.1">superscript</csymbol><apply id="S2.SS2.p3.12.m12.1.1.2.cmml" xref="S2.SS2.p3.12.m12.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.12.m12.1.1.2.1.cmml" xref="S2.SS2.p3.12.m12.1.1">subscript</csymbol><ci id="S2.SS2.p3.12.m12.1.1.2.2.cmml" xref="S2.SS2.p3.12.m12.1.1.2.2">𝑥</ci><ci id="S2.SS2.p3.12.m12.1.1.2.3.cmml" xref="S2.SS2.p3.12.m12.1.1.2.3">𝑖</ci></apply><minus id="S2.SS2.p3.12.m12.1.1.3.cmml" xref="S2.SS2.p3.12.m12.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.12.m12.1c">x_{i}^{-}</annotation></semantics></math> is true, then the answer to <math id="S2.SS2.p3.13.m13.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S2.SS2.p3.13.m13.1a"><msub id="S2.SS2.p3.13.m13.1.1" xref="S2.SS2.p3.13.m13.1.1.cmml"><mi id="S2.SS2.p3.13.m13.1.1.2" xref="S2.SS2.p3.13.m13.1.1.2.cmml">q</mi><mi id="S2.SS2.p3.13.m13.1.1.3" xref="S2.SS2.p3.13.m13.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p3.13.m13.1b"><apply id="S2.SS2.p3.13.m13.1.1.cmml" xref="S2.SS2.p3.13.m13.1.1"><csymbol cd="ambiguous" id="S2.SS2.p3.13.m13.1.1.1.cmml" xref="S2.SS2.p3.13.m13.1.1">subscript</csymbol><ci id="S2.SS2.p3.13.m13.1.1.2.cmml" xref="S2.SS2.p3.13.m13.1.1.2">𝑞</ci><ci id="S2.SS2.p3.13.m13.1.1.3.cmml" xref="S2.SS2.p3.13.m13.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p3.13.m13.1c">q_{i}</annotation></semantics></math> should be “No”.</p>
</div>
<div id="S2.SS2.p4" class="ltx_para ltx_noindent">
<p id="S2.SS2.p4.1" class="ltx_p">In practice, we convert each task into a question-answering task with two possible labels, then we use task-specific zero-shot prompts to format questions and answers as strings to construct each contrast pair.
The opposite labels we use to construct contrast pairs can be “Yes” and “No” for a generic task, or they can be other tasks-specific labels, such as “Positive” and “Negative” in the case of sentiment classification.
We describe the exact prompts we use to for each task in <a href="#A2" title="Appendix B Misleading Prefix Details ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">B</span></a>.</p>
</div>
<div id="S2.SS2.p5" class="ltx_para ltx_noindent">
<p id="S2.SS2.p5.13" class="ltx_p"><span id="S2.SS2.p5.13.1" class="ltx_text ltx_font_bold">Feature extraction and normalization.</span>
Given a contrast pair <math id="S2.SS2.p5.1.m1.2" class="ltx_Math" alttext="(x_{i}^{+},x_{i}^{-})" display="inline"><semantics id="S2.SS2.p5.1.m1.2a"><mrow id="S2.SS2.p5.1.m1.2.2.2" xref="S2.SS2.p5.1.m1.2.2.3.cmml"><mo stretchy="false" id="S2.SS2.p5.1.m1.2.2.2.3" xref="S2.SS2.p5.1.m1.2.2.3.cmml">(</mo><msubsup id="S2.SS2.p5.1.m1.1.1.1.1" xref="S2.SS2.p5.1.m1.1.1.1.1.cmml"><mi id="S2.SS2.p5.1.m1.1.1.1.1.2.2" xref="S2.SS2.p5.1.m1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p5.1.m1.1.1.1.1.2.3" xref="S2.SS2.p5.1.m1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p5.1.m1.1.1.1.1.3" xref="S2.SS2.p5.1.m1.1.1.1.1.3.cmml">+</mo></msubsup><mo id="S2.SS2.p5.1.m1.2.2.2.4" xref="S2.SS2.p5.1.m1.2.2.3.cmml">,</mo><msubsup id="S2.SS2.p5.1.m1.2.2.2.2" xref="S2.SS2.p5.1.m1.2.2.2.2.cmml"><mi id="S2.SS2.p5.1.m1.2.2.2.2.2.2" xref="S2.SS2.p5.1.m1.2.2.2.2.2.2.cmml">x</mi><mi id="S2.SS2.p5.1.m1.2.2.2.2.2.3" xref="S2.SS2.p5.1.m1.2.2.2.2.2.3.cmml">i</mi><mo id="S2.SS2.p5.1.m1.2.2.2.2.3" xref="S2.SS2.p5.1.m1.2.2.2.2.3.cmml">−</mo></msubsup><mo stretchy="false" id="S2.SS2.p5.1.m1.2.2.2.5" xref="S2.SS2.p5.1.m1.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.1.m1.2b"><interval closure="open" id="S2.SS2.p5.1.m1.2.2.3.cmml" xref="S2.SS2.p5.1.m1.2.2.2"><apply id="S2.SS2.p5.1.m1.1.1.1.1.cmml" xref="S2.SS2.p5.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.1.m1.1.1.1.1.1.cmml" xref="S2.SS2.p5.1.m1.1.1.1.1">superscript</csymbol><apply id="S2.SS2.p5.1.m1.1.1.1.1.2.cmml" xref="S2.SS2.p5.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.1.m1.1.1.1.1.2.1.cmml" xref="S2.SS2.p5.1.m1.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p5.1.m1.1.1.1.1.2.2.cmml" xref="S2.SS2.p5.1.m1.1.1.1.1.2.2">𝑥</ci><ci id="S2.SS2.p5.1.m1.1.1.1.1.2.3.cmml" xref="S2.SS2.p5.1.m1.1.1.1.1.2.3">𝑖</ci></apply><plus id="S2.SS2.p5.1.m1.1.1.1.1.3.cmml" xref="S2.SS2.p5.1.m1.1.1.1.1.3"></plus></apply><apply id="S2.SS2.p5.1.m1.2.2.2.2.cmml" xref="S2.SS2.p5.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS2.p5.1.m1.2.2.2.2.1.cmml" xref="S2.SS2.p5.1.m1.2.2.2.2">superscript</csymbol><apply id="S2.SS2.p5.1.m1.2.2.2.2.2.cmml" xref="S2.SS2.p5.1.m1.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS2.p5.1.m1.2.2.2.2.2.1.cmml" xref="S2.SS2.p5.1.m1.2.2.2.2">subscript</csymbol><ci id="S2.SS2.p5.1.m1.2.2.2.2.2.2.cmml" xref="S2.SS2.p5.1.m1.2.2.2.2.2.2">𝑥</ci><ci id="S2.SS2.p5.1.m1.2.2.2.2.2.3.cmml" xref="S2.SS2.p5.1.m1.2.2.2.2.2.3">𝑖</ci></apply><minus id="S2.SS2.p5.1.m1.2.2.2.2.3.cmml" xref="S2.SS2.p5.1.m1.2.2.2.2.3"></minus></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.1.m1.2c">(x_{i}^{+},x_{i}^{-})</annotation></semantics></math>, CCS first computes the representations <math id="S2.SS2.p5.2.m2.1" class="ltx_Math" alttext="\phi(x_{i}^{+})" display="inline"><semantics id="S2.SS2.p5.2.m2.1a"><mrow id="S2.SS2.p5.2.m2.1.1" xref="S2.SS2.p5.2.m2.1.1.cmml"><mi id="S2.SS2.p5.2.m2.1.1.3" xref="S2.SS2.p5.2.m2.1.1.3.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p5.2.m2.1.1.2" xref="S2.SS2.p5.2.m2.1.1.2.cmml">​</mo><mrow id="S2.SS2.p5.2.m2.1.1.1.1" xref="S2.SS2.p5.2.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p5.2.m2.1.1.1.1.2" xref="S2.SS2.p5.2.m2.1.1.1.1.1.cmml">(</mo><msubsup id="S2.SS2.p5.2.m2.1.1.1.1.1" xref="S2.SS2.p5.2.m2.1.1.1.1.1.cmml"><mi id="S2.SS2.p5.2.m2.1.1.1.1.1.2.2" xref="S2.SS2.p5.2.m2.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p5.2.m2.1.1.1.1.1.2.3" xref="S2.SS2.p5.2.m2.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p5.2.m2.1.1.1.1.1.3" xref="S2.SS2.p5.2.m2.1.1.1.1.1.3.cmml">+</mo></msubsup><mo stretchy="false" id="S2.SS2.p5.2.m2.1.1.1.1.3" xref="S2.SS2.p5.2.m2.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.2.m2.1b"><apply id="S2.SS2.p5.2.m2.1.1.cmml" xref="S2.SS2.p5.2.m2.1.1"><times id="S2.SS2.p5.2.m2.1.1.2.cmml" xref="S2.SS2.p5.2.m2.1.1.2"></times><ci id="S2.SS2.p5.2.m2.1.1.3.cmml" xref="S2.SS2.p5.2.m2.1.1.3">italic-ϕ</ci><apply id="S2.SS2.p5.2.m2.1.1.1.1.1.cmml" xref="S2.SS2.p5.2.m2.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.2.m2.1.1.1.1.1.1.cmml" xref="S2.SS2.p5.2.m2.1.1.1.1">superscript</csymbol><apply id="S2.SS2.p5.2.m2.1.1.1.1.1.2.cmml" xref="S2.SS2.p5.2.m2.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.2.m2.1.1.1.1.1.2.1.cmml" xref="S2.SS2.p5.2.m2.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p5.2.m2.1.1.1.1.1.2.2.cmml" xref="S2.SS2.p5.2.m2.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.SS2.p5.2.m2.1.1.1.1.1.2.3.cmml" xref="S2.SS2.p5.2.m2.1.1.1.1.1.2.3">𝑖</ci></apply><plus id="S2.SS2.p5.2.m2.1.1.1.1.1.3.cmml" xref="S2.SS2.p5.2.m2.1.1.1.1.1.3"></plus></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.2.m2.1c">\phi(x_{i}^{+})</annotation></semantics></math> and <math id="S2.SS2.p5.3.m3.1" class="ltx_Math" alttext="\phi(x_{i}^{-})" display="inline"><semantics id="S2.SS2.p5.3.m3.1a"><mrow id="S2.SS2.p5.3.m3.1.1" xref="S2.SS2.p5.3.m3.1.1.cmml"><mi id="S2.SS2.p5.3.m3.1.1.3" xref="S2.SS2.p5.3.m3.1.1.3.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p5.3.m3.1.1.2" xref="S2.SS2.p5.3.m3.1.1.2.cmml">​</mo><mrow id="S2.SS2.p5.3.m3.1.1.1.1" xref="S2.SS2.p5.3.m3.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p5.3.m3.1.1.1.1.2" xref="S2.SS2.p5.3.m3.1.1.1.1.1.cmml">(</mo><msubsup id="S2.SS2.p5.3.m3.1.1.1.1.1" xref="S2.SS2.p5.3.m3.1.1.1.1.1.cmml"><mi id="S2.SS2.p5.3.m3.1.1.1.1.1.2.2" xref="S2.SS2.p5.3.m3.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p5.3.m3.1.1.1.1.1.2.3" xref="S2.SS2.p5.3.m3.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p5.3.m3.1.1.1.1.1.3" xref="S2.SS2.p5.3.m3.1.1.1.1.1.3.cmml">−</mo></msubsup><mo stretchy="false" id="S2.SS2.p5.3.m3.1.1.1.1.3" xref="S2.SS2.p5.3.m3.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.3.m3.1b"><apply id="S2.SS2.p5.3.m3.1.1.cmml" xref="S2.SS2.p5.3.m3.1.1"><times id="S2.SS2.p5.3.m3.1.1.2.cmml" xref="S2.SS2.p5.3.m3.1.1.2"></times><ci id="S2.SS2.p5.3.m3.1.1.3.cmml" xref="S2.SS2.p5.3.m3.1.1.3">italic-ϕ</ci><apply id="S2.SS2.p5.3.m3.1.1.1.1.1.cmml" xref="S2.SS2.p5.3.m3.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.3.m3.1.1.1.1.1.1.cmml" xref="S2.SS2.p5.3.m3.1.1.1.1">superscript</csymbol><apply id="S2.SS2.p5.3.m3.1.1.1.1.1.2.cmml" xref="S2.SS2.p5.3.m3.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.3.m3.1.1.1.1.1.2.1.cmml" xref="S2.SS2.p5.3.m3.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p5.3.m3.1.1.1.1.1.2.2.cmml" xref="S2.SS2.p5.3.m3.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.SS2.p5.3.m3.1.1.1.1.1.2.3.cmml" xref="S2.SS2.p5.3.m3.1.1.1.1.1.2.3">𝑖</ci></apply><minus id="S2.SS2.p5.3.m3.1.1.1.1.1.3.cmml" xref="S2.SS2.p5.3.m3.1.1.1.1.1.3"></minus></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.3.m3.1c">\phi(x_{i}^{-})</annotation></semantics></math> using the feature extractor <math id="S2.SS2.p5.4.m4.1" class="ltx_Math" alttext="\phi(\cdot)" display="inline"><semantics id="S2.SS2.p5.4.m4.1a"><mrow id="S2.SS2.p5.4.m4.1.2" xref="S2.SS2.p5.4.m4.1.2.cmml"><mi id="S2.SS2.p5.4.m4.1.2.2" xref="S2.SS2.p5.4.m4.1.2.2.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p5.4.m4.1.2.1" xref="S2.SS2.p5.4.m4.1.2.1.cmml">​</mo><mrow id="S2.SS2.p5.4.m4.1.2.3.2" xref="S2.SS2.p5.4.m4.1.2.cmml"><mo stretchy="false" id="S2.SS2.p5.4.m4.1.2.3.2.1" xref="S2.SS2.p5.4.m4.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S2.SS2.p5.4.m4.1.1" xref="S2.SS2.p5.4.m4.1.1.cmml">⋅</mo><mo stretchy="false" id="S2.SS2.p5.4.m4.1.2.3.2.2" xref="S2.SS2.p5.4.m4.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.4.m4.1b"><apply id="S2.SS2.p5.4.m4.1.2.cmml" xref="S2.SS2.p5.4.m4.1.2"><times id="S2.SS2.p5.4.m4.1.2.1.cmml" xref="S2.SS2.p5.4.m4.1.2.1"></times><ci id="S2.SS2.p5.4.m4.1.2.2.cmml" xref="S2.SS2.p5.4.m4.1.2.2">italic-ϕ</ci><ci id="S2.SS2.p5.4.m4.1.1.cmml" xref="S2.SS2.p5.4.m4.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.4.m4.1c">\phi(\cdot)</annotation></semantics></math>.
Intuitively, there are two salient differences between <math id="S2.SS2.p5.5.m5.1" class="ltx_Math" alttext="\phi(x_{i}^{+})" display="inline"><semantics id="S2.SS2.p5.5.m5.1a"><mrow id="S2.SS2.p5.5.m5.1.1" xref="S2.SS2.p5.5.m5.1.1.cmml"><mi id="S2.SS2.p5.5.m5.1.1.3" xref="S2.SS2.p5.5.m5.1.1.3.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p5.5.m5.1.1.2" xref="S2.SS2.p5.5.m5.1.1.2.cmml">​</mo><mrow id="S2.SS2.p5.5.m5.1.1.1.1" xref="S2.SS2.p5.5.m5.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p5.5.m5.1.1.1.1.2" xref="S2.SS2.p5.5.m5.1.1.1.1.1.cmml">(</mo><msubsup id="S2.SS2.p5.5.m5.1.1.1.1.1" xref="S2.SS2.p5.5.m5.1.1.1.1.1.cmml"><mi id="S2.SS2.p5.5.m5.1.1.1.1.1.2.2" xref="S2.SS2.p5.5.m5.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p5.5.m5.1.1.1.1.1.2.3" xref="S2.SS2.p5.5.m5.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p5.5.m5.1.1.1.1.1.3" xref="S2.SS2.p5.5.m5.1.1.1.1.1.3.cmml">+</mo></msubsup><mo stretchy="false" id="S2.SS2.p5.5.m5.1.1.1.1.3" xref="S2.SS2.p5.5.m5.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.5.m5.1b"><apply id="S2.SS2.p5.5.m5.1.1.cmml" xref="S2.SS2.p5.5.m5.1.1"><times id="S2.SS2.p5.5.m5.1.1.2.cmml" xref="S2.SS2.p5.5.m5.1.1.2"></times><ci id="S2.SS2.p5.5.m5.1.1.3.cmml" xref="S2.SS2.p5.5.m5.1.1.3">italic-ϕ</ci><apply id="S2.SS2.p5.5.m5.1.1.1.1.1.cmml" xref="S2.SS2.p5.5.m5.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.5.m5.1.1.1.1.1.1.cmml" xref="S2.SS2.p5.5.m5.1.1.1.1">superscript</csymbol><apply id="S2.SS2.p5.5.m5.1.1.1.1.1.2.cmml" xref="S2.SS2.p5.5.m5.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.5.m5.1.1.1.1.1.2.1.cmml" xref="S2.SS2.p5.5.m5.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p5.5.m5.1.1.1.1.1.2.2.cmml" xref="S2.SS2.p5.5.m5.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.SS2.p5.5.m5.1.1.1.1.1.2.3.cmml" xref="S2.SS2.p5.5.m5.1.1.1.1.1.2.3">𝑖</ci></apply><plus id="S2.SS2.p5.5.m5.1.1.1.1.1.3.cmml" xref="S2.SS2.p5.5.m5.1.1.1.1.1.3"></plus></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.5.m5.1c">\phi(x_{i}^{+})</annotation></semantics></math> and <math id="S2.SS2.p5.6.m6.1" class="ltx_Math" alttext="\phi(x_{i}^{-})" display="inline"><semantics id="S2.SS2.p5.6.m6.1a"><mrow id="S2.SS2.p5.6.m6.1.1" xref="S2.SS2.p5.6.m6.1.1.cmml"><mi id="S2.SS2.p5.6.m6.1.1.3" xref="S2.SS2.p5.6.m6.1.1.3.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p5.6.m6.1.1.2" xref="S2.SS2.p5.6.m6.1.1.2.cmml">​</mo><mrow id="S2.SS2.p5.6.m6.1.1.1.1" xref="S2.SS2.p5.6.m6.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p5.6.m6.1.1.1.1.2" xref="S2.SS2.p5.6.m6.1.1.1.1.1.cmml">(</mo><msubsup id="S2.SS2.p5.6.m6.1.1.1.1.1" xref="S2.SS2.p5.6.m6.1.1.1.1.1.cmml"><mi id="S2.SS2.p5.6.m6.1.1.1.1.1.2.2" xref="S2.SS2.p5.6.m6.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p5.6.m6.1.1.1.1.1.2.3" xref="S2.SS2.p5.6.m6.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p5.6.m6.1.1.1.1.1.3" xref="S2.SS2.p5.6.m6.1.1.1.1.1.3.cmml">−</mo></msubsup><mo stretchy="false" id="S2.SS2.p5.6.m6.1.1.1.1.3" xref="S2.SS2.p5.6.m6.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.6.m6.1b"><apply id="S2.SS2.p5.6.m6.1.1.cmml" xref="S2.SS2.p5.6.m6.1.1"><times id="S2.SS2.p5.6.m6.1.1.2.cmml" xref="S2.SS2.p5.6.m6.1.1.2"></times><ci id="S2.SS2.p5.6.m6.1.1.3.cmml" xref="S2.SS2.p5.6.m6.1.1.3">italic-ϕ</ci><apply id="S2.SS2.p5.6.m6.1.1.1.1.1.cmml" xref="S2.SS2.p5.6.m6.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.6.m6.1.1.1.1.1.1.cmml" xref="S2.SS2.p5.6.m6.1.1.1.1">superscript</csymbol><apply id="S2.SS2.p5.6.m6.1.1.1.1.1.2.cmml" xref="S2.SS2.p5.6.m6.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.6.m6.1.1.1.1.1.2.1.cmml" xref="S2.SS2.p5.6.m6.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p5.6.m6.1.1.1.1.1.2.2.cmml" xref="S2.SS2.p5.6.m6.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.SS2.p5.6.m6.1.1.1.1.1.2.3.cmml" xref="S2.SS2.p5.6.m6.1.1.1.1.1.2.3">𝑖</ci></apply><minus id="S2.SS2.p5.6.m6.1.1.1.1.1.3.cmml" xref="S2.SS2.p5.6.m6.1.1.1.1.1.3"></minus></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.6.m6.1c">\phi(x_{i}^{-})</annotation></semantics></math>: (1) <math id="S2.SS2.p5.7.m7.1" class="ltx_Math" alttext="x_{i}^{+}" display="inline"><semantics id="S2.SS2.p5.7.m7.1a"><msubsup id="S2.SS2.p5.7.m7.1.1" xref="S2.SS2.p5.7.m7.1.1.cmml"><mi id="S2.SS2.p5.7.m7.1.1.2.2" xref="S2.SS2.p5.7.m7.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p5.7.m7.1.1.2.3" xref="S2.SS2.p5.7.m7.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p5.7.m7.1.1.3" xref="S2.SS2.p5.7.m7.1.1.3.cmml">+</mo></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.7.m7.1b"><apply id="S2.SS2.p5.7.m7.1.1.cmml" xref="S2.SS2.p5.7.m7.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.7.m7.1.1.1.cmml" xref="S2.SS2.p5.7.m7.1.1">superscript</csymbol><apply id="S2.SS2.p5.7.m7.1.1.2.cmml" xref="S2.SS2.p5.7.m7.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.7.m7.1.1.2.1.cmml" xref="S2.SS2.p5.7.m7.1.1">subscript</csymbol><ci id="S2.SS2.p5.7.m7.1.1.2.2.cmml" xref="S2.SS2.p5.7.m7.1.1.2.2">𝑥</ci><ci id="S2.SS2.p5.7.m7.1.1.2.3.cmml" xref="S2.SS2.p5.7.m7.1.1.2.3">𝑖</ci></apply><plus id="S2.SS2.p5.7.m7.1.1.3.cmml" xref="S2.SS2.p5.7.m7.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.7.m7.1c">x_{i}^{+}</annotation></semantics></math> ends with “Yes” while <math id="S2.SS2.p5.8.m8.1" class="ltx_Math" alttext="x_{i}^{-}" display="inline"><semantics id="S2.SS2.p5.8.m8.1a"><msubsup id="S2.SS2.p5.8.m8.1.1" xref="S2.SS2.p5.8.m8.1.1.cmml"><mi id="S2.SS2.p5.8.m8.1.1.2.2" xref="S2.SS2.p5.8.m8.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p5.8.m8.1.1.2.3" xref="S2.SS2.p5.8.m8.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p5.8.m8.1.1.3" xref="S2.SS2.p5.8.m8.1.1.3.cmml">−</mo></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.8.m8.1b"><apply id="S2.SS2.p5.8.m8.1.1.cmml" xref="S2.SS2.p5.8.m8.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.8.m8.1.1.1.cmml" xref="S2.SS2.p5.8.m8.1.1">superscript</csymbol><apply id="S2.SS2.p5.8.m8.1.1.2.cmml" xref="S2.SS2.p5.8.m8.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.8.m8.1.1.2.1.cmml" xref="S2.SS2.p5.8.m8.1.1">subscript</csymbol><ci id="S2.SS2.p5.8.m8.1.1.2.2.cmml" xref="S2.SS2.p5.8.m8.1.1.2.2">𝑥</ci><ci id="S2.SS2.p5.8.m8.1.1.2.3.cmml" xref="S2.SS2.p5.8.m8.1.1.2.3">𝑖</ci></apply><minus id="S2.SS2.p5.8.m8.1.1.3.cmml" xref="S2.SS2.p5.8.m8.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.8.m8.1c">x_{i}^{-}</annotation></semantics></math> ends with “No”, and (2) one of <math id="S2.SS2.p5.9.m9.1" class="ltx_Math" alttext="x_{i}^{+}" display="inline"><semantics id="S2.SS2.p5.9.m9.1a"><msubsup id="S2.SS2.p5.9.m9.1.1" xref="S2.SS2.p5.9.m9.1.1.cmml"><mi id="S2.SS2.p5.9.m9.1.1.2.2" xref="S2.SS2.p5.9.m9.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p5.9.m9.1.1.2.3" xref="S2.SS2.p5.9.m9.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p5.9.m9.1.1.3" xref="S2.SS2.p5.9.m9.1.1.3.cmml">+</mo></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.9.m9.1b"><apply id="S2.SS2.p5.9.m9.1.1.cmml" xref="S2.SS2.p5.9.m9.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.9.m9.1.1.1.cmml" xref="S2.SS2.p5.9.m9.1.1">superscript</csymbol><apply id="S2.SS2.p5.9.m9.1.1.2.cmml" xref="S2.SS2.p5.9.m9.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.9.m9.1.1.2.1.cmml" xref="S2.SS2.p5.9.m9.1.1">subscript</csymbol><ci id="S2.SS2.p5.9.m9.1.1.2.2.cmml" xref="S2.SS2.p5.9.m9.1.1.2.2">𝑥</ci><ci id="S2.SS2.p5.9.m9.1.1.2.3.cmml" xref="S2.SS2.p5.9.m9.1.1.2.3">𝑖</ci></apply><plus id="S2.SS2.p5.9.m9.1.1.3.cmml" xref="S2.SS2.p5.9.m9.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.9.m9.1c">x_{i}^{+}</annotation></semantics></math> or <math id="S2.SS2.p5.10.m10.1" class="ltx_Math" alttext="x_{i}^{-}" display="inline"><semantics id="S2.SS2.p5.10.m10.1a"><msubsup id="S2.SS2.p5.10.m10.1.1" xref="S2.SS2.p5.10.m10.1.1.cmml"><mi id="S2.SS2.p5.10.m10.1.1.2.2" xref="S2.SS2.p5.10.m10.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p5.10.m10.1.1.2.3" xref="S2.SS2.p5.10.m10.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p5.10.m10.1.1.3" xref="S2.SS2.p5.10.m10.1.1.3.cmml">−</mo></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.10.m10.1b"><apply id="S2.SS2.p5.10.m10.1.1.cmml" xref="S2.SS2.p5.10.m10.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.10.m10.1.1.1.cmml" xref="S2.SS2.p5.10.m10.1.1">superscript</csymbol><apply id="S2.SS2.p5.10.m10.1.1.2.cmml" xref="S2.SS2.p5.10.m10.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.10.m10.1.1.2.1.cmml" xref="S2.SS2.p5.10.m10.1.1">subscript</csymbol><ci id="S2.SS2.p5.10.m10.1.1.2.2.cmml" xref="S2.SS2.p5.10.m10.1.1.2.2">𝑥</ci><ci id="S2.SS2.p5.10.m10.1.1.2.3.cmml" xref="S2.SS2.p5.10.m10.1.1.2.3">𝑖</ci></apply><minus id="S2.SS2.p5.10.m10.1.1.3.cmml" xref="S2.SS2.p5.10.m10.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.10.m10.1c">x_{i}^{-}</annotation></semantics></math> is true while the other is false.
We want to find (2) rather than (1), so we first try to remove the effect of (1) by normalizing <math id="S2.SS2.p5.11.m11.1" class="ltx_Math" alttext="\{\phi(x_{i}^{+})\}" display="inline"><semantics id="S2.SS2.p5.11.m11.1a"><mrow id="S2.SS2.p5.11.m11.1.1.1" xref="S2.SS2.p5.11.m11.1.1.2.cmml"><mo stretchy="false" id="S2.SS2.p5.11.m11.1.1.1.2" xref="S2.SS2.p5.11.m11.1.1.2.cmml">{</mo><mrow id="S2.SS2.p5.11.m11.1.1.1.1" xref="S2.SS2.p5.11.m11.1.1.1.1.cmml"><mi id="S2.SS2.p5.11.m11.1.1.1.1.3" xref="S2.SS2.p5.11.m11.1.1.1.1.3.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p5.11.m11.1.1.1.1.2" xref="S2.SS2.p5.11.m11.1.1.1.1.2.cmml">​</mo><mrow id="S2.SS2.p5.11.m11.1.1.1.1.1.1" xref="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p5.11.m11.1.1.1.1.1.1.2" xref="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="S2.SS2.p5.11.m11.1.1.1.1.1.1.1" xref="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.cmml"><mi id="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.2.2" xref="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.2.3" xref="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.3" xref="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.3.cmml">+</mo></msubsup><mo stretchy="false" id="S2.SS2.p5.11.m11.1.1.1.1.1.1.3" xref="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S2.SS2.p5.11.m11.1.1.1.3" xref="S2.SS2.p5.11.m11.1.1.2.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.11.m11.1b"><set id="S2.SS2.p5.11.m11.1.1.2.cmml" xref="S2.SS2.p5.11.m11.1.1.1"><apply id="S2.SS2.p5.11.m11.1.1.1.1.cmml" xref="S2.SS2.p5.11.m11.1.1.1.1"><times id="S2.SS2.p5.11.m11.1.1.1.1.2.cmml" xref="S2.SS2.p5.11.m11.1.1.1.1.2"></times><ci id="S2.SS2.p5.11.m11.1.1.1.1.3.cmml" xref="S2.SS2.p5.11.m11.1.1.1.1.3">italic-ϕ</ci><apply id="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p5.11.m11.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p5.11.m11.1.1.1.1.1.1">superscript</csymbol><apply id="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.2.cmml" xref="S2.SS2.p5.11.m11.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.2.1.cmml" xref="S2.SS2.p5.11.m11.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.2.2.cmml" xref="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.2.3.cmml" xref="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><plus id="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.3.cmml" xref="S2.SS2.p5.11.m11.1.1.1.1.1.1.1.3"></plus></apply></apply></set></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.11.m11.1c">\{\phi(x_{i}^{+})\}</annotation></semantics></math> and <math id="S2.SS2.p5.12.m12.1" class="ltx_Math" alttext="\{\phi(x_{i}^{-})\}" display="inline"><semantics id="S2.SS2.p5.12.m12.1a"><mrow id="S2.SS2.p5.12.m12.1.1.1" xref="S2.SS2.p5.12.m12.1.1.2.cmml"><mo stretchy="false" id="S2.SS2.p5.12.m12.1.1.1.2" xref="S2.SS2.p5.12.m12.1.1.2.cmml">{</mo><mrow id="S2.SS2.p5.12.m12.1.1.1.1" xref="S2.SS2.p5.12.m12.1.1.1.1.cmml"><mi id="S2.SS2.p5.12.m12.1.1.1.1.3" xref="S2.SS2.p5.12.m12.1.1.1.1.3.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p5.12.m12.1.1.1.1.2" xref="S2.SS2.p5.12.m12.1.1.1.1.2.cmml">​</mo><mrow id="S2.SS2.p5.12.m12.1.1.1.1.1.1" xref="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p5.12.m12.1.1.1.1.1.1.2" xref="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="S2.SS2.p5.12.m12.1.1.1.1.1.1.1" xref="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.cmml"><mi id="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.2.2" xref="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.2.3" xref="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.3" xref="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.3.cmml">−</mo></msubsup><mo stretchy="false" id="S2.SS2.p5.12.m12.1.1.1.1.1.1.3" xref="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S2.SS2.p5.12.m12.1.1.1.3" xref="S2.SS2.p5.12.m12.1.1.2.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.12.m12.1b"><set id="S2.SS2.p5.12.m12.1.1.2.cmml" xref="S2.SS2.p5.12.m12.1.1.1"><apply id="S2.SS2.p5.12.m12.1.1.1.1.cmml" xref="S2.SS2.p5.12.m12.1.1.1.1"><times id="S2.SS2.p5.12.m12.1.1.1.1.2.cmml" xref="S2.SS2.p5.12.m12.1.1.1.1.2"></times><ci id="S2.SS2.p5.12.m12.1.1.1.1.3.cmml" xref="S2.SS2.p5.12.m12.1.1.1.1.3">italic-ϕ</ci><apply id="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p5.12.m12.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p5.12.m12.1.1.1.1.1.1">superscript</csymbol><apply id="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.2.cmml" xref="S2.SS2.p5.12.m12.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.2.1.cmml" xref="S2.SS2.p5.12.m12.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.2.2.cmml" xref="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.2.3.cmml" xref="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><minus id="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.3.cmml" xref="S2.SS2.p5.12.m12.1.1.1.1.1.1.1.3"></minus></apply></apply></set></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.12.m12.1c">\{\phi(x_{i}^{-})\}</annotation></semantics></math> independently.
In particular, we construct normalized representations <math id="S2.SS2.p5.13.m13.1" class="ltx_Math" alttext="\tilde{\phi}(x)" display="inline"><semantics id="S2.SS2.p5.13.m13.1a"><mrow id="S2.SS2.p5.13.m13.1.2" xref="S2.SS2.p5.13.m13.1.2.cmml"><mover accent="true" id="S2.SS2.p5.13.m13.1.2.2" xref="S2.SS2.p5.13.m13.1.2.2.cmml"><mi id="S2.SS2.p5.13.m13.1.2.2.2" xref="S2.SS2.p5.13.m13.1.2.2.2.cmml">ϕ</mi><mo id="S2.SS2.p5.13.m13.1.2.2.1" xref="S2.SS2.p5.13.m13.1.2.2.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S2.SS2.p5.13.m13.1.2.1" xref="S2.SS2.p5.13.m13.1.2.1.cmml">​</mo><mrow id="S2.SS2.p5.13.m13.1.2.3.2" xref="S2.SS2.p5.13.m13.1.2.cmml"><mo stretchy="false" id="S2.SS2.p5.13.m13.1.2.3.2.1" xref="S2.SS2.p5.13.m13.1.2.cmml">(</mo><mi id="S2.SS2.p5.13.m13.1.1" xref="S2.SS2.p5.13.m13.1.1.cmml">x</mi><mo stretchy="false" id="S2.SS2.p5.13.m13.1.2.3.2.2" xref="S2.SS2.p5.13.m13.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.13.m13.1b"><apply id="S2.SS2.p5.13.m13.1.2.cmml" xref="S2.SS2.p5.13.m13.1.2"><times id="S2.SS2.p5.13.m13.1.2.1.cmml" xref="S2.SS2.p5.13.m13.1.2.1"></times><apply id="S2.SS2.p5.13.m13.1.2.2.cmml" xref="S2.SS2.p5.13.m13.1.2.2"><ci id="S2.SS2.p5.13.m13.1.2.2.1.cmml" xref="S2.SS2.p5.13.m13.1.2.2.1">~</ci><ci id="S2.SS2.p5.13.m13.1.2.2.2.cmml" xref="S2.SS2.p5.13.m13.1.2.2.2">italic-ϕ</ci></apply><ci id="S2.SS2.p5.13.m13.1.1.cmml" xref="S2.SS2.p5.13.m13.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.13.m13.1c">\tilde{\phi}(x)</annotation></semantics></math> as follows:</p>
<table id="S2.Ex1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.Ex1.m1.3" class="ltx_Math" alttext="\tilde{\phi}(x_{i}^{+}):=\frac{\phi(x_{i}^{+})-\mu^{+}}{\sigma^{+}}\,;\quad\tilde{\phi}(x_{i}^{-}):=\frac{\phi(x_{i}^{-})-\mu^{-}}{\sigma^{-}}," display="block"><semantics id="S2.Ex1.m1.3a"><mrow id="S2.Ex1.m1.3.3.1"><mrow id="S2.Ex1.m1.3.3.1.1.2" xref="S2.Ex1.m1.3.3.1.1.3.cmml"><mrow id="S2.Ex1.m1.3.3.1.1.1.1" xref="S2.Ex1.m1.3.3.1.1.1.1.cmml"><mrow id="S2.Ex1.m1.3.3.1.1.1.1.1" xref="S2.Ex1.m1.3.3.1.1.1.1.1.cmml"><mover accent="true" id="S2.Ex1.m1.3.3.1.1.1.1.1.3" xref="S2.Ex1.m1.3.3.1.1.1.1.1.3.cmml"><mi id="S2.Ex1.m1.3.3.1.1.1.1.1.3.2" xref="S2.Ex1.m1.3.3.1.1.1.1.1.3.2.cmml">ϕ</mi><mo id="S2.Ex1.m1.3.3.1.1.1.1.1.3.1" xref="S2.Ex1.m1.3.3.1.1.1.1.1.3.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S2.Ex1.m1.3.3.1.1.1.1.1.2" xref="S2.Ex1.m1.3.3.1.1.1.1.1.2.cmml">​</mo><mrow id="S2.Ex1.m1.3.3.1.1.1.1.1.1.1" xref="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.2" xref="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1" xref="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.cmml"><mi id="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.2.2" xref="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.2.3" xref="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.3" xref="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.3.cmml">+</mo></msubsup><mo rspace="0.278em" stretchy="false" id="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.3" xref="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.278em" id="S2.Ex1.m1.3.3.1.1.1.1.2" xref="S2.Ex1.m1.3.3.1.1.1.1.2.cmml">:=</mo><mfrac id="S2.Ex1.m1.1.1" xref="S2.Ex1.m1.1.1.cmml"><mrow id="S2.Ex1.m1.1.1.1" xref="S2.Ex1.m1.1.1.1.cmml"><mrow id="S2.Ex1.m1.1.1.1.1" xref="S2.Ex1.m1.1.1.1.1.cmml"><mi id="S2.Ex1.m1.1.1.1.1.3" xref="S2.Ex1.m1.1.1.1.1.3.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.Ex1.m1.1.1.1.1.2" xref="S2.Ex1.m1.1.1.1.1.2.cmml">​</mo><mrow id="S2.Ex1.m1.1.1.1.1.1.1" xref="S2.Ex1.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.Ex1.m1.1.1.1.1.1.1.2" xref="S2.Ex1.m1.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="S2.Ex1.m1.1.1.1.1.1.1.1" xref="S2.Ex1.m1.1.1.1.1.1.1.1.cmml"><mi id="S2.Ex1.m1.1.1.1.1.1.1.1.2.2" xref="S2.Ex1.m1.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.Ex1.m1.1.1.1.1.1.1.1.2.3" xref="S2.Ex1.m1.1.1.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.Ex1.m1.1.1.1.1.1.1.1.3" xref="S2.Ex1.m1.1.1.1.1.1.1.1.3.cmml">+</mo></msubsup><mo stretchy="false" id="S2.Ex1.m1.1.1.1.1.1.1.3" xref="S2.Ex1.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.Ex1.m1.1.1.1.2" xref="S2.Ex1.m1.1.1.1.2.cmml">−</mo><msup id="S2.Ex1.m1.1.1.1.3" xref="S2.Ex1.m1.1.1.1.3.cmml"><mi id="S2.Ex1.m1.1.1.1.3.2" xref="S2.Ex1.m1.1.1.1.3.2.cmml">μ</mi><mo id="S2.Ex1.m1.1.1.1.3.3" xref="S2.Ex1.m1.1.1.1.3.3.cmml">+</mo></msup></mrow><msup id="S2.Ex1.m1.1.1.3" xref="S2.Ex1.m1.1.1.3.cmml"><mi id="S2.Ex1.m1.1.1.3.2" xref="S2.Ex1.m1.1.1.3.2.cmml">σ</mi><mo id="S2.Ex1.m1.1.1.3.3" xref="S2.Ex1.m1.1.1.3.3.cmml">+</mo></msup></mfrac></mrow><mo lspace="0.170em" rspace="1.167em" id="S2.Ex1.m1.3.3.1.1.2.3" xref="S2.Ex1.m1.3.3.1.1.3a.cmml">;</mo><mrow id="S2.Ex1.m1.3.3.1.1.2.2" xref="S2.Ex1.m1.3.3.1.1.2.2.cmml"><mrow id="S2.Ex1.m1.3.3.1.1.2.2.1" xref="S2.Ex1.m1.3.3.1.1.2.2.1.cmml"><mover accent="true" id="S2.Ex1.m1.3.3.1.1.2.2.1.3" xref="S2.Ex1.m1.3.3.1.1.2.2.1.3.cmml"><mi id="S2.Ex1.m1.3.3.1.1.2.2.1.3.2" xref="S2.Ex1.m1.3.3.1.1.2.2.1.3.2.cmml">ϕ</mi><mo id="S2.Ex1.m1.3.3.1.1.2.2.1.3.1" xref="S2.Ex1.m1.3.3.1.1.2.2.1.3.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S2.Ex1.m1.3.3.1.1.2.2.1.2" xref="S2.Ex1.m1.3.3.1.1.2.2.1.2.cmml">​</mo><mrow id="S2.Ex1.m1.3.3.1.1.2.2.1.1.1" xref="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.cmml"><mo stretchy="false" id="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.2" xref="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.cmml">(</mo><msubsup id="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1" xref="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.cmml"><mi id="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.2.2" xref="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.2.2.cmml">x</mi><mi id="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.2.3" xref="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.2.3.cmml">i</mi><mo id="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.3" xref="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.3.cmml">−</mo></msubsup><mo rspace="0.278em" stretchy="false" id="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.3" xref="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.cmml">)</mo></mrow></mrow><mo rspace="0.278em" id="S2.Ex1.m1.3.3.1.1.2.2.2" xref="S2.Ex1.m1.3.3.1.1.2.2.2.cmml">:=</mo><mfrac id="S2.Ex1.m1.2.2" xref="S2.Ex1.m1.2.2.cmml"><mrow id="S2.Ex1.m1.2.2.1" xref="S2.Ex1.m1.2.2.1.cmml"><mrow id="S2.Ex1.m1.2.2.1.1" xref="S2.Ex1.m1.2.2.1.1.cmml"><mi id="S2.Ex1.m1.2.2.1.1.3" xref="S2.Ex1.m1.2.2.1.1.3.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.Ex1.m1.2.2.1.1.2" xref="S2.Ex1.m1.2.2.1.1.2.cmml">​</mo><mrow id="S2.Ex1.m1.2.2.1.1.1.1" xref="S2.Ex1.m1.2.2.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.Ex1.m1.2.2.1.1.1.1.2" xref="S2.Ex1.m1.2.2.1.1.1.1.1.cmml">(</mo><msubsup id="S2.Ex1.m1.2.2.1.1.1.1.1" xref="S2.Ex1.m1.2.2.1.1.1.1.1.cmml"><mi id="S2.Ex1.m1.2.2.1.1.1.1.1.2.2" xref="S2.Ex1.m1.2.2.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.Ex1.m1.2.2.1.1.1.1.1.2.3" xref="S2.Ex1.m1.2.2.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.Ex1.m1.2.2.1.1.1.1.1.3" xref="S2.Ex1.m1.2.2.1.1.1.1.1.3.cmml">−</mo></msubsup><mo stretchy="false" id="S2.Ex1.m1.2.2.1.1.1.1.3" xref="S2.Ex1.m1.2.2.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.Ex1.m1.2.2.1.2" xref="S2.Ex1.m1.2.2.1.2.cmml">−</mo><msup id="S2.Ex1.m1.2.2.1.3" xref="S2.Ex1.m1.2.2.1.3.cmml"><mi id="S2.Ex1.m1.2.2.1.3.2" xref="S2.Ex1.m1.2.2.1.3.2.cmml">μ</mi><mo id="S2.Ex1.m1.2.2.1.3.3" xref="S2.Ex1.m1.2.2.1.3.3.cmml">−</mo></msup></mrow><msup id="S2.Ex1.m1.2.2.3" xref="S2.Ex1.m1.2.2.3.cmml"><mi id="S2.Ex1.m1.2.2.3.2" xref="S2.Ex1.m1.2.2.3.2.cmml">σ</mi><mo id="S2.Ex1.m1.2.2.3.3" xref="S2.Ex1.m1.2.2.3.3.cmml">−</mo></msup></mfrac></mrow></mrow><mo id="S2.Ex1.m1.3.3.1.2">,</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex1.m1.3b"><apply id="S2.Ex1.m1.3.3.1.1.3.cmml" xref="S2.Ex1.m1.3.3.1.1.2"><csymbol cd="ambiguous" id="S2.Ex1.m1.3.3.1.1.3a.cmml" xref="S2.Ex1.m1.3.3.1.1.2.3">formulae-sequence</csymbol><apply id="S2.Ex1.m1.3.3.1.1.1.1.cmml" xref="S2.Ex1.m1.3.3.1.1.1.1"><csymbol cd="latexml" id="S2.Ex1.m1.3.3.1.1.1.1.2.cmml" xref="S2.Ex1.m1.3.3.1.1.1.1.2">assign</csymbol><apply id="S2.Ex1.m1.3.3.1.1.1.1.1.cmml" xref="S2.Ex1.m1.3.3.1.1.1.1.1"><times id="S2.Ex1.m1.3.3.1.1.1.1.1.2.cmml" xref="S2.Ex1.m1.3.3.1.1.1.1.1.2"></times><apply id="S2.Ex1.m1.3.3.1.1.1.1.1.3.cmml" xref="S2.Ex1.m1.3.3.1.1.1.1.1.3"><ci id="S2.Ex1.m1.3.3.1.1.1.1.1.3.1.cmml" xref="S2.Ex1.m1.3.3.1.1.1.1.1.3.1">~</ci><ci id="S2.Ex1.m1.3.3.1.1.1.1.1.3.2.cmml" xref="S2.Ex1.m1.3.3.1.1.1.1.1.3.2">italic-ϕ</ci></apply><apply id="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex1.m1.3.3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex1.m1.3.3.1.1.1.1.1.1.1">superscript</csymbol><apply id="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.2.cmml" xref="S2.Ex1.m1.3.3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.Ex1.m1.3.3.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><plus id="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.3.cmml" xref="S2.Ex1.m1.3.3.1.1.1.1.1.1.1.1.3"></plus></apply></apply><apply id="S2.Ex1.m1.1.1.cmml" xref="S2.Ex1.m1.1.1"><divide id="S2.Ex1.m1.1.1.2.cmml" xref="S2.Ex1.m1.1.1"></divide><apply id="S2.Ex1.m1.1.1.1.cmml" xref="S2.Ex1.m1.1.1.1"><minus id="S2.Ex1.m1.1.1.1.2.cmml" xref="S2.Ex1.m1.1.1.1.2"></minus><apply id="S2.Ex1.m1.1.1.1.1.cmml" xref="S2.Ex1.m1.1.1.1.1"><times id="S2.Ex1.m1.1.1.1.1.2.cmml" xref="S2.Ex1.m1.1.1.1.1.2"></times><ci id="S2.Ex1.m1.1.1.1.1.3.cmml" xref="S2.Ex1.m1.1.1.1.1.3">italic-ϕ</ci><apply id="S2.Ex1.m1.1.1.1.1.1.1.1.cmml" xref="S2.Ex1.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex1.m1.1.1.1.1.1.1">superscript</csymbol><apply id="S2.Ex1.m1.1.1.1.1.1.1.1.2.cmml" xref="S2.Ex1.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.Ex1.m1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.Ex1.m1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.Ex1.m1.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.Ex1.m1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.Ex1.m1.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><plus id="S2.Ex1.m1.1.1.1.1.1.1.1.3.cmml" xref="S2.Ex1.m1.1.1.1.1.1.1.1.3"></plus></apply></apply><apply id="S2.Ex1.m1.1.1.1.3.cmml" xref="S2.Ex1.m1.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.1.3.1.cmml" xref="S2.Ex1.m1.1.1.1.3">superscript</csymbol><ci id="S2.Ex1.m1.1.1.1.3.2.cmml" xref="S2.Ex1.m1.1.1.1.3.2">𝜇</ci><plus id="S2.Ex1.m1.1.1.1.3.3.cmml" xref="S2.Ex1.m1.1.1.1.3.3"></plus></apply></apply><apply id="S2.Ex1.m1.1.1.3.cmml" xref="S2.Ex1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex1.m1.1.1.3.1.cmml" xref="S2.Ex1.m1.1.1.3">superscript</csymbol><ci id="S2.Ex1.m1.1.1.3.2.cmml" xref="S2.Ex1.m1.1.1.3.2">𝜎</ci><plus id="S2.Ex1.m1.1.1.3.3.cmml" xref="S2.Ex1.m1.1.1.3.3"></plus></apply></apply></apply><apply id="S2.Ex1.m1.3.3.1.1.2.2.cmml" xref="S2.Ex1.m1.3.3.1.1.2.2"><csymbol cd="latexml" id="S2.Ex1.m1.3.3.1.1.2.2.2.cmml" xref="S2.Ex1.m1.3.3.1.1.2.2.2">assign</csymbol><apply id="S2.Ex1.m1.3.3.1.1.2.2.1.cmml" xref="S2.Ex1.m1.3.3.1.1.2.2.1"><times id="S2.Ex1.m1.3.3.1.1.2.2.1.2.cmml" xref="S2.Ex1.m1.3.3.1.1.2.2.1.2"></times><apply id="S2.Ex1.m1.3.3.1.1.2.2.1.3.cmml" xref="S2.Ex1.m1.3.3.1.1.2.2.1.3"><ci id="S2.Ex1.m1.3.3.1.1.2.2.1.3.1.cmml" xref="S2.Ex1.m1.3.3.1.1.2.2.1.3.1">~</ci><ci id="S2.Ex1.m1.3.3.1.1.2.2.1.3.2.cmml" xref="S2.Ex1.m1.3.3.1.1.2.2.1.3.2">italic-ϕ</ci></apply><apply id="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.cmml" xref="S2.Ex1.m1.3.3.1.1.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.1.cmml" xref="S2.Ex1.m1.3.3.1.1.2.2.1.1.1">superscript</csymbol><apply id="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.2.cmml" xref="S2.Ex1.m1.3.3.1.1.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.2.1.cmml" xref="S2.Ex1.m1.3.3.1.1.2.2.1.1.1">subscript</csymbol><ci id="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.2.2.cmml" xref="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.2.2">𝑥</ci><ci id="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.2.3.cmml" xref="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.2.3">𝑖</ci></apply><minus id="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.3.cmml" xref="S2.Ex1.m1.3.3.1.1.2.2.1.1.1.1.3"></minus></apply></apply><apply id="S2.Ex1.m1.2.2.cmml" xref="S2.Ex1.m1.2.2"><divide id="S2.Ex1.m1.2.2.2.cmml" xref="S2.Ex1.m1.2.2"></divide><apply id="S2.Ex1.m1.2.2.1.cmml" xref="S2.Ex1.m1.2.2.1"><minus id="S2.Ex1.m1.2.2.1.2.cmml" xref="S2.Ex1.m1.2.2.1.2"></minus><apply id="S2.Ex1.m1.2.2.1.1.cmml" xref="S2.Ex1.m1.2.2.1.1"><times id="S2.Ex1.m1.2.2.1.1.2.cmml" xref="S2.Ex1.m1.2.2.1.1.2"></times><ci id="S2.Ex1.m1.2.2.1.1.3.cmml" xref="S2.Ex1.m1.2.2.1.1.3">italic-ϕ</ci><apply id="S2.Ex1.m1.2.2.1.1.1.1.1.cmml" xref="S2.Ex1.m1.2.2.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex1.m1.2.2.1.1.1.1.1.1.cmml" xref="S2.Ex1.m1.2.2.1.1.1.1">superscript</csymbol><apply id="S2.Ex1.m1.2.2.1.1.1.1.1.2.cmml" xref="S2.Ex1.m1.2.2.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex1.m1.2.2.1.1.1.1.1.2.1.cmml" xref="S2.Ex1.m1.2.2.1.1.1.1">subscript</csymbol><ci id="S2.Ex1.m1.2.2.1.1.1.1.1.2.2.cmml" xref="S2.Ex1.m1.2.2.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.Ex1.m1.2.2.1.1.1.1.1.2.3.cmml" xref="S2.Ex1.m1.2.2.1.1.1.1.1.2.3">𝑖</ci></apply><minus id="S2.Ex1.m1.2.2.1.1.1.1.1.3.cmml" xref="S2.Ex1.m1.2.2.1.1.1.1.1.3"></minus></apply></apply><apply id="S2.Ex1.m1.2.2.1.3.cmml" xref="S2.Ex1.m1.2.2.1.3"><csymbol cd="ambiguous" id="S2.Ex1.m1.2.2.1.3.1.cmml" xref="S2.Ex1.m1.2.2.1.3">superscript</csymbol><ci id="S2.Ex1.m1.2.2.1.3.2.cmml" xref="S2.Ex1.m1.2.2.1.3.2">𝜇</ci><minus id="S2.Ex1.m1.2.2.1.3.3.cmml" xref="S2.Ex1.m1.2.2.1.3.3"></minus></apply></apply><apply id="S2.Ex1.m1.2.2.3.cmml" xref="S2.Ex1.m1.2.2.3"><csymbol cd="ambiguous" id="S2.Ex1.m1.2.2.3.1.cmml" xref="S2.Ex1.m1.2.2.3">superscript</csymbol><ci id="S2.Ex1.m1.2.2.3.2.cmml" xref="S2.Ex1.m1.2.2.3.2">𝜎</ci><minus id="S2.Ex1.m1.2.2.3.3.cmml" xref="S2.Ex1.m1.2.2.3.3"></minus></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex1.m1.3c">\tilde{\phi}(x_{i}^{+}):=\frac{\phi(x_{i}^{+})-\mu^{+}}{\sigma^{+}}\,;\quad\tilde{\phi}(x_{i}^{-}):=\frac{\phi(x_{i}^{-})-\mu^{-}}{\sigma^{-}},</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S2.SS2.p5.19" class="ltx_p">where <math id="S2.SS2.p5.14.m1.2" class="ltx_Math" alttext="(\mu^{+},\sigma^{+})" display="inline"><semantics id="S2.SS2.p5.14.m1.2a"><mrow id="S2.SS2.p5.14.m1.2.2.2" xref="S2.SS2.p5.14.m1.2.2.3.cmml"><mo stretchy="false" id="S2.SS2.p5.14.m1.2.2.2.3" xref="S2.SS2.p5.14.m1.2.2.3.cmml">(</mo><msup id="S2.SS2.p5.14.m1.1.1.1.1" xref="S2.SS2.p5.14.m1.1.1.1.1.cmml"><mi id="S2.SS2.p5.14.m1.1.1.1.1.2" xref="S2.SS2.p5.14.m1.1.1.1.1.2.cmml">μ</mi><mo id="S2.SS2.p5.14.m1.1.1.1.1.3" xref="S2.SS2.p5.14.m1.1.1.1.1.3.cmml">+</mo></msup><mo id="S2.SS2.p5.14.m1.2.2.2.4" xref="S2.SS2.p5.14.m1.2.2.3.cmml">,</mo><msup id="S2.SS2.p5.14.m1.2.2.2.2" xref="S2.SS2.p5.14.m1.2.2.2.2.cmml"><mi id="S2.SS2.p5.14.m1.2.2.2.2.2" xref="S2.SS2.p5.14.m1.2.2.2.2.2.cmml">σ</mi><mo id="S2.SS2.p5.14.m1.2.2.2.2.3" xref="S2.SS2.p5.14.m1.2.2.2.2.3.cmml">+</mo></msup><mo stretchy="false" id="S2.SS2.p5.14.m1.2.2.2.5" xref="S2.SS2.p5.14.m1.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.14.m1.2b"><interval closure="open" id="S2.SS2.p5.14.m1.2.2.3.cmml" xref="S2.SS2.p5.14.m1.2.2.2"><apply id="S2.SS2.p5.14.m1.1.1.1.1.cmml" xref="S2.SS2.p5.14.m1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.14.m1.1.1.1.1.1.cmml" xref="S2.SS2.p5.14.m1.1.1.1.1">superscript</csymbol><ci id="S2.SS2.p5.14.m1.1.1.1.1.2.cmml" xref="S2.SS2.p5.14.m1.1.1.1.1.2">𝜇</ci><plus id="S2.SS2.p5.14.m1.1.1.1.1.3.cmml" xref="S2.SS2.p5.14.m1.1.1.1.1.3"></plus></apply><apply id="S2.SS2.p5.14.m1.2.2.2.2.cmml" xref="S2.SS2.p5.14.m1.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS2.p5.14.m1.2.2.2.2.1.cmml" xref="S2.SS2.p5.14.m1.2.2.2.2">superscript</csymbol><ci id="S2.SS2.p5.14.m1.2.2.2.2.2.cmml" xref="S2.SS2.p5.14.m1.2.2.2.2.2">𝜎</ci><plus id="S2.SS2.p5.14.m1.2.2.2.2.3.cmml" xref="S2.SS2.p5.14.m1.2.2.2.2.3"></plus></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.14.m1.2c">(\mu^{+},\sigma^{+})</annotation></semantics></math> and <math id="S2.SS2.p5.15.m2.2" class="ltx_Math" alttext="(\mu^{-},\sigma^{-})" display="inline"><semantics id="S2.SS2.p5.15.m2.2a"><mrow id="S2.SS2.p5.15.m2.2.2.2" xref="S2.SS2.p5.15.m2.2.2.3.cmml"><mo stretchy="false" id="S2.SS2.p5.15.m2.2.2.2.3" xref="S2.SS2.p5.15.m2.2.2.3.cmml">(</mo><msup id="S2.SS2.p5.15.m2.1.1.1.1" xref="S2.SS2.p5.15.m2.1.1.1.1.cmml"><mi id="S2.SS2.p5.15.m2.1.1.1.1.2" xref="S2.SS2.p5.15.m2.1.1.1.1.2.cmml">μ</mi><mo id="S2.SS2.p5.15.m2.1.1.1.1.3" xref="S2.SS2.p5.15.m2.1.1.1.1.3.cmml">−</mo></msup><mo id="S2.SS2.p5.15.m2.2.2.2.4" xref="S2.SS2.p5.15.m2.2.2.3.cmml">,</mo><msup id="S2.SS2.p5.15.m2.2.2.2.2" xref="S2.SS2.p5.15.m2.2.2.2.2.cmml"><mi id="S2.SS2.p5.15.m2.2.2.2.2.2" xref="S2.SS2.p5.15.m2.2.2.2.2.2.cmml">σ</mi><mo id="S2.SS2.p5.15.m2.2.2.2.2.3" xref="S2.SS2.p5.15.m2.2.2.2.2.3.cmml">−</mo></msup><mo stretchy="false" id="S2.SS2.p5.15.m2.2.2.2.5" xref="S2.SS2.p5.15.m2.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.15.m2.2b"><interval closure="open" id="S2.SS2.p5.15.m2.2.2.3.cmml" xref="S2.SS2.p5.15.m2.2.2.2"><apply id="S2.SS2.p5.15.m2.1.1.1.1.cmml" xref="S2.SS2.p5.15.m2.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.15.m2.1.1.1.1.1.cmml" xref="S2.SS2.p5.15.m2.1.1.1.1">superscript</csymbol><ci id="S2.SS2.p5.15.m2.1.1.1.1.2.cmml" xref="S2.SS2.p5.15.m2.1.1.1.1.2">𝜇</ci><minus id="S2.SS2.p5.15.m2.1.1.1.1.3.cmml" xref="S2.SS2.p5.15.m2.1.1.1.1.3"></minus></apply><apply id="S2.SS2.p5.15.m2.2.2.2.2.cmml" xref="S2.SS2.p5.15.m2.2.2.2.2"><csymbol cd="ambiguous" id="S2.SS2.p5.15.m2.2.2.2.2.1.cmml" xref="S2.SS2.p5.15.m2.2.2.2.2">superscript</csymbol><ci id="S2.SS2.p5.15.m2.2.2.2.2.2.cmml" xref="S2.SS2.p5.15.m2.2.2.2.2.2">𝜎</ci><minus id="S2.SS2.p5.15.m2.2.2.2.2.3.cmml" xref="S2.SS2.p5.15.m2.2.2.2.2.3"></minus></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.15.m2.2c">(\mu^{-},\sigma^{-})</annotation></semantics></math> are the means and standard deviations of <math id="S2.SS2.p5.16.m3.1" class="ltx_Math" alttext="\{\phi(x_{i}^{+})\}_{i=1}^{n}" display="inline"><semantics id="S2.SS2.p5.16.m3.1a"><msubsup id="S2.SS2.p5.16.m3.1.1" xref="S2.SS2.p5.16.m3.1.1.cmml"><mrow id="S2.SS2.p5.16.m3.1.1.1.1.1" xref="S2.SS2.p5.16.m3.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.SS2.p5.16.m3.1.1.1.1.1.2" xref="S2.SS2.p5.16.m3.1.1.1.1.2.cmml">{</mo><mrow id="S2.SS2.p5.16.m3.1.1.1.1.1.1" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.cmml"><mi id="S2.SS2.p5.16.m3.1.1.1.1.1.1.3" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.3.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p5.16.m3.1.1.1.1.1.1.2" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.2" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.2.2" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.2.3" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.3" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.3.cmml">+</mo></msubsup><mo stretchy="false" id="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.3" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S2.SS2.p5.16.m3.1.1.1.1.1.3" xref="S2.SS2.p5.16.m3.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S2.SS2.p5.16.m3.1.1.1.3" xref="S2.SS2.p5.16.m3.1.1.1.3.cmml"><mi id="S2.SS2.p5.16.m3.1.1.1.3.2" xref="S2.SS2.p5.16.m3.1.1.1.3.2.cmml">i</mi><mo id="S2.SS2.p5.16.m3.1.1.1.3.1" xref="S2.SS2.p5.16.m3.1.1.1.3.1.cmml">=</mo><mn id="S2.SS2.p5.16.m3.1.1.1.3.3" xref="S2.SS2.p5.16.m3.1.1.1.3.3.cmml">1</mn></mrow><mi id="S2.SS2.p5.16.m3.1.1.3" xref="S2.SS2.p5.16.m3.1.1.3.cmml">n</mi></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.16.m3.1b"><apply id="S2.SS2.p5.16.m3.1.1.cmml" xref="S2.SS2.p5.16.m3.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.16.m3.1.1.2.cmml" xref="S2.SS2.p5.16.m3.1.1">superscript</csymbol><apply id="S2.SS2.p5.16.m3.1.1.1.cmml" xref="S2.SS2.p5.16.m3.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.16.m3.1.1.1.2.cmml" xref="S2.SS2.p5.16.m3.1.1">subscript</csymbol><set id="S2.SS2.p5.16.m3.1.1.1.1.2.cmml" xref="S2.SS2.p5.16.m3.1.1.1.1.1"><apply id="S2.SS2.p5.16.m3.1.1.1.1.1.1.cmml" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1"><times id="S2.SS2.p5.16.m3.1.1.1.1.1.1.2.cmml" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.2"></times><ci id="S2.SS2.p5.16.m3.1.1.1.1.1.1.3.cmml" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.3">italic-ϕ</ci><apply id="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1">superscript</csymbol><apply id="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><plus id="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.SS2.p5.16.m3.1.1.1.1.1.1.1.1.1.3"></plus></apply></apply></set><apply id="S2.SS2.p5.16.m3.1.1.1.3.cmml" xref="S2.SS2.p5.16.m3.1.1.1.3"><eq id="S2.SS2.p5.16.m3.1.1.1.3.1.cmml" xref="S2.SS2.p5.16.m3.1.1.1.3.1"></eq><ci id="S2.SS2.p5.16.m3.1.1.1.3.2.cmml" xref="S2.SS2.p5.16.m3.1.1.1.3.2">𝑖</ci><cn type="integer" id="S2.SS2.p5.16.m3.1.1.1.3.3.cmml" xref="S2.SS2.p5.16.m3.1.1.1.3.3">1</cn></apply></apply><ci id="S2.SS2.p5.16.m3.1.1.3.cmml" xref="S2.SS2.p5.16.m3.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.16.m3.1c">\{\phi(x_{i}^{+})\}_{i=1}^{n}</annotation></semantics></math> and <math id="S2.SS2.p5.17.m4.1" class="ltx_Math" alttext="\{\phi(x_{i}^{-})\}_{i=1}^{n}" display="inline"><semantics id="S2.SS2.p5.17.m4.1a"><msubsup id="S2.SS2.p5.17.m4.1.1" xref="S2.SS2.p5.17.m4.1.1.cmml"><mrow id="S2.SS2.p5.17.m4.1.1.1.1.1" xref="S2.SS2.p5.17.m4.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.SS2.p5.17.m4.1.1.1.1.1.2" xref="S2.SS2.p5.17.m4.1.1.1.1.2.cmml">{</mo><mrow id="S2.SS2.p5.17.m4.1.1.1.1.1.1" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.cmml"><mi id="S2.SS2.p5.17.m4.1.1.1.1.1.1.3" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.3.cmml">ϕ</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p5.17.m4.1.1.1.1.1.1.2" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.2" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.cmml"><mi id="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.2.2" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.2.3" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.3" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.3.cmml">−</mo></msubsup><mo stretchy="false" id="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.3" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S2.SS2.p5.17.m4.1.1.1.1.1.3" xref="S2.SS2.p5.17.m4.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S2.SS2.p5.17.m4.1.1.1.3" xref="S2.SS2.p5.17.m4.1.1.1.3.cmml"><mi id="S2.SS2.p5.17.m4.1.1.1.3.2" xref="S2.SS2.p5.17.m4.1.1.1.3.2.cmml">i</mi><mo id="S2.SS2.p5.17.m4.1.1.1.3.1" xref="S2.SS2.p5.17.m4.1.1.1.3.1.cmml">=</mo><mn id="S2.SS2.p5.17.m4.1.1.1.3.3" xref="S2.SS2.p5.17.m4.1.1.1.3.3.cmml">1</mn></mrow><mi id="S2.SS2.p5.17.m4.1.1.3" xref="S2.SS2.p5.17.m4.1.1.3.cmml">n</mi></msubsup><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.17.m4.1b"><apply id="S2.SS2.p5.17.m4.1.1.cmml" xref="S2.SS2.p5.17.m4.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.17.m4.1.1.2.cmml" xref="S2.SS2.p5.17.m4.1.1">superscript</csymbol><apply id="S2.SS2.p5.17.m4.1.1.1.cmml" xref="S2.SS2.p5.17.m4.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.17.m4.1.1.1.2.cmml" xref="S2.SS2.p5.17.m4.1.1">subscript</csymbol><set id="S2.SS2.p5.17.m4.1.1.1.1.2.cmml" xref="S2.SS2.p5.17.m4.1.1.1.1.1"><apply id="S2.SS2.p5.17.m4.1.1.1.1.1.1.cmml" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1"><times id="S2.SS2.p5.17.m4.1.1.1.1.1.1.2.cmml" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.2"></times><ci id="S2.SS2.p5.17.m4.1.1.1.1.1.1.3.cmml" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.3">italic-ϕ</ci><apply id="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1">superscript</csymbol><apply id="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.2.cmml" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><minus id="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.3.cmml" xref="S2.SS2.p5.17.m4.1.1.1.1.1.1.1.1.1.3"></minus></apply></apply></set><apply id="S2.SS2.p5.17.m4.1.1.1.3.cmml" xref="S2.SS2.p5.17.m4.1.1.1.3"><eq id="S2.SS2.p5.17.m4.1.1.1.3.1.cmml" xref="S2.SS2.p5.17.m4.1.1.1.3.1"></eq><ci id="S2.SS2.p5.17.m4.1.1.1.3.2.cmml" xref="S2.SS2.p5.17.m4.1.1.1.3.2">𝑖</ci><cn type="integer" id="S2.SS2.p5.17.m4.1.1.1.3.3.cmml" xref="S2.SS2.p5.17.m4.1.1.1.3.3">1</cn></apply></apply><ci id="S2.SS2.p5.17.m4.1.1.3.cmml" xref="S2.SS2.p5.17.m4.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.17.m4.1c">\{\phi(x_{i}^{-})\}_{i=1}^{n}</annotation></semantics></math> respectively, and where all operations are element-wise along each dimension.<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>Mean-normalization is important for CCS to work at all, while variance-normalization is less essential.</span></span></span>
This normalization ensures that <math id="S2.SS2.p5.18.m5.1" class="ltx_Math" alttext="\{\tilde{\phi}(x_{i}^{+})\}" display="inline"><semantics id="S2.SS2.p5.18.m5.1a"><mrow id="S2.SS2.p5.18.m5.1.1.1" xref="S2.SS2.p5.18.m5.1.1.2.cmml"><mo stretchy="false" id="S2.SS2.p5.18.m5.1.1.1.2" xref="S2.SS2.p5.18.m5.1.1.2.cmml">{</mo><mrow id="S2.SS2.p5.18.m5.1.1.1.1" xref="S2.SS2.p5.18.m5.1.1.1.1.cmml"><mover accent="true" id="S2.SS2.p5.18.m5.1.1.1.1.3" xref="S2.SS2.p5.18.m5.1.1.1.1.3.cmml"><mi id="S2.SS2.p5.18.m5.1.1.1.1.3.2" xref="S2.SS2.p5.18.m5.1.1.1.1.3.2.cmml">ϕ</mi><mo id="S2.SS2.p5.18.m5.1.1.1.1.3.1" xref="S2.SS2.p5.18.m5.1.1.1.1.3.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S2.SS2.p5.18.m5.1.1.1.1.2" xref="S2.SS2.p5.18.m5.1.1.1.1.2.cmml">​</mo><mrow id="S2.SS2.p5.18.m5.1.1.1.1.1.1" xref="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p5.18.m5.1.1.1.1.1.1.2" xref="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="S2.SS2.p5.18.m5.1.1.1.1.1.1.1" xref="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.cmml"><mi id="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.2.2" xref="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.2.3" xref="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.3" xref="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.3.cmml">+</mo></msubsup><mo stretchy="false" id="S2.SS2.p5.18.m5.1.1.1.1.1.1.3" xref="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S2.SS2.p5.18.m5.1.1.1.3" xref="S2.SS2.p5.18.m5.1.1.2.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.18.m5.1b"><set id="S2.SS2.p5.18.m5.1.1.2.cmml" xref="S2.SS2.p5.18.m5.1.1.1"><apply id="S2.SS2.p5.18.m5.1.1.1.1.cmml" xref="S2.SS2.p5.18.m5.1.1.1.1"><times id="S2.SS2.p5.18.m5.1.1.1.1.2.cmml" xref="S2.SS2.p5.18.m5.1.1.1.1.2"></times><apply id="S2.SS2.p5.18.m5.1.1.1.1.3.cmml" xref="S2.SS2.p5.18.m5.1.1.1.1.3"><ci id="S2.SS2.p5.18.m5.1.1.1.1.3.1.cmml" xref="S2.SS2.p5.18.m5.1.1.1.1.3.1">~</ci><ci id="S2.SS2.p5.18.m5.1.1.1.1.3.2.cmml" xref="S2.SS2.p5.18.m5.1.1.1.1.3.2">italic-ϕ</ci></apply><apply id="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p5.18.m5.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p5.18.m5.1.1.1.1.1.1">superscript</csymbol><apply id="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.2.cmml" xref="S2.SS2.p5.18.m5.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.2.1.cmml" xref="S2.SS2.p5.18.m5.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.2.2.cmml" xref="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.2.3.cmml" xref="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><plus id="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.3.cmml" xref="S2.SS2.p5.18.m5.1.1.1.1.1.1.1.3"></plus></apply></apply></set></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.18.m5.1c">\{\tilde{\phi}(x_{i}^{+})\}</annotation></semantics></math> and <math id="S2.SS2.p5.19.m6.1" class="ltx_Math" alttext="\{\tilde{\phi}(x_{i}^{-})\}" display="inline"><semantics id="S2.SS2.p5.19.m6.1a"><mrow id="S2.SS2.p5.19.m6.1.1.1" xref="S2.SS2.p5.19.m6.1.1.2.cmml"><mo stretchy="false" id="S2.SS2.p5.19.m6.1.1.1.2" xref="S2.SS2.p5.19.m6.1.1.2.cmml">{</mo><mrow id="S2.SS2.p5.19.m6.1.1.1.1" xref="S2.SS2.p5.19.m6.1.1.1.1.cmml"><mover accent="true" id="S2.SS2.p5.19.m6.1.1.1.1.3" xref="S2.SS2.p5.19.m6.1.1.1.1.3.cmml"><mi id="S2.SS2.p5.19.m6.1.1.1.1.3.2" xref="S2.SS2.p5.19.m6.1.1.1.1.3.2.cmml">ϕ</mi><mo id="S2.SS2.p5.19.m6.1.1.1.1.3.1" xref="S2.SS2.p5.19.m6.1.1.1.1.3.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S2.SS2.p5.19.m6.1.1.1.1.2" xref="S2.SS2.p5.19.m6.1.1.1.1.2.cmml">​</mo><mrow id="S2.SS2.p5.19.m6.1.1.1.1.1.1" xref="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p5.19.m6.1.1.1.1.1.1.2" xref="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="S2.SS2.p5.19.m6.1.1.1.1.1.1.1" xref="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.cmml"><mi id="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.2.2" xref="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.2.3" xref="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.3" xref="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.3.cmml">−</mo></msubsup><mo stretchy="false" id="S2.SS2.p5.19.m6.1.1.1.1.1.1.3" xref="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S2.SS2.p5.19.m6.1.1.1.3" xref="S2.SS2.p5.19.m6.1.1.2.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p5.19.m6.1b"><set id="S2.SS2.p5.19.m6.1.1.2.cmml" xref="S2.SS2.p5.19.m6.1.1.1"><apply id="S2.SS2.p5.19.m6.1.1.1.1.cmml" xref="S2.SS2.p5.19.m6.1.1.1.1"><times id="S2.SS2.p5.19.m6.1.1.1.1.2.cmml" xref="S2.SS2.p5.19.m6.1.1.1.1.2"></times><apply id="S2.SS2.p5.19.m6.1.1.1.1.3.cmml" xref="S2.SS2.p5.19.m6.1.1.1.1.3"><ci id="S2.SS2.p5.19.m6.1.1.1.1.3.1.cmml" xref="S2.SS2.p5.19.m6.1.1.1.1.3.1">~</ci><ci id="S2.SS2.p5.19.m6.1.1.1.1.3.2.cmml" xref="S2.SS2.p5.19.m6.1.1.1.1.3.2">italic-ϕ</ci></apply><apply id="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p5.19.m6.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p5.19.m6.1.1.1.1.1.1">superscript</csymbol><apply id="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.2.cmml" xref="S2.SS2.p5.19.m6.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.2.1.cmml" xref="S2.SS2.p5.19.m6.1.1.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.2.2.cmml" xref="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.2.3.cmml" xref="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><minus id="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.3.cmml" xref="S2.SS2.p5.19.m6.1.1.1.1.1.1.1.3"></minus></apply></apply></set></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p5.19.m6.1c">\{\tilde{\phi}(x_{i}^{-})\}</annotation></semantics></math> no longer form two separate clusters.</p>
</div>
<div id="S2.SS2.p6" class="ltx_para ltx_noindent">
<p id="S2.SS2.p6.9" class="ltx_p"><span id="S2.SS2.p6.9.1" class="ltx_text ltx_font_bold">Mapping activations to probabilities.</span>
Next, we learn a probe <math id="S2.SS2.p6.1.m1.3" class="ltx_Math" alttext="p_{\theta,b}(\tilde{\phi})" display="inline"><semantics id="S2.SS2.p6.1.m1.3a"><mrow id="S2.SS2.p6.1.m1.3.4" xref="S2.SS2.p6.1.m1.3.4.cmml"><msub id="S2.SS2.p6.1.m1.3.4.2" xref="S2.SS2.p6.1.m1.3.4.2.cmml"><mi id="S2.SS2.p6.1.m1.3.4.2.2" xref="S2.SS2.p6.1.m1.3.4.2.2.cmml">p</mi><mrow id="S2.SS2.p6.1.m1.2.2.2.4" xref="S2.SS2.p6.1.m1.2.2.2.3.cmml"><mi id="S2.SS2.p6.1.m1.1.1.1.1" xref="S2.SS2.p6.1.m1.1.1.1.1.cmml">θ</mi><mo id="S2.SS2.p6.1.m1.2.2.2.4.1" xref="S2.SS2.p6.1.m1.2.2.2.3.cmml">,</mo><mi id="S2.SS2.p6.1.m1.2.2.2.2" xref="S2.SS2.p6.1.m1.2.2.2.2.cmml">b</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S2.SS2.p6.1.m1.3.4.1" xref="S2.SS2.p6.1.m1.3.4.1.cmml">​</mo><mrow id="S2.SS2.p6.1.m1.3.4.3.2" xref="S2.SS2.p6.1.m1.3.3.cmml"><mo stretchy="false" id="S2.SS2.p6.1.m1.3.4.3.2.1" xref="S2.SS2.p6.1.m1.3.3.cmml">(</mo><mover accent="true" id="S2.SS2.p6.1.m1.3.3" xref="S2.SS2.p6.1.m1.3.3.cmml"><mi id="S2.SS2.p6.1.m1.3.3.2" xref="S2.SS2.p6.1.m1.3.3.2.cmml">ϕ</mi><mo id="S2.SS2.p6.1.m1.3.3.1" xref="S2.SS2.p6.1.m1.3.3.1.cmml">~</mo></mover><mo stretchy="false" id="S2.SS2.p6.1.m1.3.4.3.2.2" xref="S2.SS2.p6.1.m1.3.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p6.1.m1.3b"><apply id="S2.SS2.p6.1.m1.3.4.cmml" xref="S2.SS2.p6.1.m1.3.4"><times id="S2.SS2.p6.1.m1.3.4.1.cmml" xref="S2.SS2.p6.1.m1.3.4.1"></times><apply id="S2.SS2.p6.1.m1.3.4.2.cmml" xref="S2.SS2.p6.1.m1.3.4.2"><csymbol cd="ambiguous" id="S2.SS2.p6.1.m1.3.4.2.1.cmml" xref="S2.SS2.p6.1.m1.3.4.2">subscript</csymbol><ci id="S2.SS2.p6.1.m1.3.4.2.2.cmml" xref="S2.SS2.p6.1.m1.3.4.2.2">𝑝</ci><list id="S2.SS2.p6.1.m1.2.2.2.3.cmml" xref="S2.SS2.p6.1.m1.2.2.2.4"><ci id="S2.SS2.p6.1.m1.1.1.1.1.cmml" xref="S2.SS2.p6.1.m1.1.1.1.1">𝜃</ci><ci id="S2.SS2.p6.1.m1.2.2.2.2.cmml" xref="S2.SS2.p6.1.m1.2.2.2.2">𝑏</ci></list></apply><apply id="S2.SS2.p6.1.m1.3.3.cmml" xref="S2.SS2.p6.1.m1.3.4.3.2"><ci id="S2.SS2.p6.1.m1.3.3.1.cmml" xref="S2.SS2.p6.1.m1.3.3.1">~</ci><ci id="S2.SS2.p6.1.m1.3.3.2.cmml" xref="S2.SS2.p6.1.m1.3.3.2">italic-ϕ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p6.1.m1.3c">p_{\theta,b}(\tilde{\phi})</annotation></semantics></math> that maps a (normalized) hidden state <math id="S2.SS2.p6.2.m2.1" class="ltx_Math" alttext="\tilde{\phi}(x)" display="inline"><semantics id="S2.SS2.p6.2.m2.1a"><mrow id="S2.SS2.p6.2.m2.1.2" xref="S2.SS2.p6.2.m2.1.2.cmml"><mover accent="true" id="S2.SS2.p6.2.m2.1.2.2" xref="S2.SS2.p6.2.m2.1.2.2.cmml"><mi id="S2.SS2.p6.2.m2.1.2.2.2" xref="S2.SS2.p6.2.m2.1.2.2.2.cmml">ϕ</mi><mo id="S2.SS2.p6.2.m2.1.2.2.1" xref="S2.SS2.p6.2.m2.1.2.2.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S2.SS2.p6.2.m2.1.2.1" xref="S2.SS2.p6.2.m2.1.2.1.cmml">​</mo><mrow id="S2.SS2.p6.2.m2.1.2.3.2" xref="S2.SS2.p6.2.m2.1.2.cmml"><mo stretchy="false" id="S2.SS2.p6.2.m2.1.2.3.2.1" xref="S2.SS2.p6.2.m2.1.2.cmml">(</mo><mi id="S2.SS2.p6.2.m2.1.1" xref="S2.SS2.p6.2.m2.1.1.cmml">x</mi><mo stretchy="false" id="S2.SS2.p6.2.m2.1.2.3.2.2" xref="S2.SS2.p6.2.m2.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p6.2.m2.1b"><apply id="S2.SS2.p6.2.m2.1.2.cmml" xref="S2.SS2.p6.2.m2.1.2"><times id="S2.SS2.p6.2.m2.1.2.1.cmml" xref="S2.SS2.p6.2.m2.1.2.1"></times><apply id="S2.SS2.p6.2.m2.1.2.2.cmml" xref="S2.SS2.p6.2.m2.1.2.2"><ci id="S2.SS2.p6.2.m2.1.2.2.1.cmml" xref="S2.SS2.p6.2.m2.1.2.2.1">~</ci><ci id="S2.SS2.p6.2.m2.1.2.2.2.cmml" xref="S2.SS2.p6.2.m2.1.2.2.2">italic-ϕ</ci></apply><ci id="S2.SS2.p6.2.m2.1.1.cmml" xref="S2.SS2.p6.2.m2.1.1">𝑥</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p6.2.m2.1c">\tilde{\phi}(x)</annotation></semantics></math> to a number between <math id="S2.SS2.p6.3.m3.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S2.SS2.p6.3.m3.1a"><mn id="S2.SS2.p6.3.m3.1.1" xref="S2.SS2.p6.3.m3.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S2.SS2.p6.3.m3.1b"><cn type="integer" id="S2.SS2.p6.3.m3.1.1.cmml" xref="S2.SS2.p6.3.m3.1.1">0</cn></annotation-xml></semantics></math> and <math id="S2.SS2.p6.4.m4.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S2.SS2.p6.4.m4.1a"><mn id="S2.SS2.p6.4.m4.1.1" xref="S2.SS2.p6.4.m4.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S2.SS2.p6.4.m4.1b"><cn type="integer" id="S2.SS2.p6.4.m4.1.1.cmml" xref="S2.SS2.p6.4.m4.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p6.4.m4.1c">1</annotation></semantics></math> representing the probability that the statement <math id="S2.SS2.p6.5.m5.1" class="ltx_Math" alttext="x" display="inline"><semantics id="S2.SS2.p6.5.m5.1a"><mi id="S2.SS2.p6.5.m5.1.1" xref="S2.SS2.p6.5.m5.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p6.5.m5.1b"><ci id="S2.SS2.p6.5.m5.1.1.cmml" xref="S2.SS2.p6.5.m5.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p6.5.m5.1c">x</annotation></semantics></math> is true. We use a linear projection followed by a sigmoid <math id="S2.SS2.p6.6.m6.1" class="ltx_Math" alttext="\sigma(\cdot)" display="inline"><semantics id="S2.SS2.p6.6.m6.1a"><mrow id="S2.SS2.p6.6.m6.1.2" xref="S2.SS2.p6.6.m6.1.2.cmml"><mi id="S2.SS2.p6.6.m6.1.2.2" xref="S2.SS2.p6.6.m6.1.2.2.cmml">σ</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p6.6.m6.1.2.1" xref="S2.SS2.p6.6.m6.1.2.1.cmml">​</mo><mrow id="S2.SS2.p6.6.m6.1.2.3.2" xref="S2.SS2.p6.6.m6.1.2.cmml"><mo stretchy="false" id="S2.SS2.p6.6.m6.1.2.3.2.1" xref="S2.SS2.p6.6.m6.1.2.cmml">(</mo><mo lspace="0em" rspace="0em" id="S2.SS2.p6.6.m6.1.1" xref="S2.SS2.p6.6.m6.1.1.cmml">⋅</mo><mo stretchy="false" id="S2.SS2.p6.6.m6.1.2.3.2.2" xref="S2.SS2.p6.6.m6.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p6.6.m6.1b"><apply id="S2.SS2.p6.6.m6.1.2.cmml" xref="S2.SS2.p6.6.m6.1.2"><times id="S2.SS2.p6.6.m6.1.2.1.cmml" xref="S2.SS2.p6.6.m6.1.2.1"></times><ci id="S2.SS2.p6.6.m6.1.2.2.cmml" xref="S2.SS2.p6.6.m6.1.2.2">𝜎</ci><ci id="S2.SS2.p6.6.m6.1.1.cmml" xref="S2.SS2.p6.6.m6.1.1">⋅</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p6.6.m6.1c">\sigma(\cdot)</annotation></semantics></math>, i.e. <math id="S2.SS2.p6.7.m7.4" class="ltx_Math" alttext="p_{\theta,b}(\tilde{\phi})=\sigma(\theta^{T}\tilde{\phi}+b)" display="inline"><semantics id="S2.SS2.p6.7.m7.4a"><mrow id="S2.SS2.p6.7.m7.4.4" xref="S2.SS2.p6.7.m7.4.4.cmml"><mrow id="S2.SS2.p6.7.m7.4.4.3" xref="S2.SS2.p6.7.m7.4.4.3.cmml"><msub id="S2.SS2.p6.7.m7.4.4.3.2" xref="S2.SS2.p6.7.m7.4.4.3.2.cmml"><mi id="S2.SS2.p6.7.m7.4.4.3.2.2" xref="S2.SS2.p6.7.m7.4.4.3.2.2.cmml">p</mi><mrow id="S2.SS2.p6.7.m7.2.2.2.4" xref="S2.SS2.p6.7.m7.2.2.2.3.cmml"><mi id="S2.SS2.p6.7.m7.1.1.1.1" xref="S2.SS2.p6.7.m7.1.1.1.1.cmml">θ</mi><mo id="S2.SS2.p6.7.m7.2.2.2.4.1" xref="S2.SS2.p6.7.m7.2.2.2.3.cmml">,</mo><mi id="S2.SS2.p6.7.m7.2.2.2.2" xref="S2.SS2.p6.7.m7.2.2.2.2.cmml">b</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S2.SS2.p6.7.m7.4.4.3.1" xref="S2.SS2.p6.7.m7.4.4.3.1.cmml">​</mo><mrow id="S2.SS2.p6.7.m7.4.4.3.3.2" xref="S2.SS2.p6.7.m7.3.3.cmml"><mo stretchy="false" id="S2.SS2.p6.7.m7.4.4.3.3.2.1" xref="S2.SS2.p6.7.m7.3.3.cmml">(</mo><mover accent="true" id="S2.SS2.p6.7.m7.3.3" xref="S2.SS2.p6.7.m7.3.3.cmml"><mi id="S2.SS2.p6.7.m7.3.3.2" xref="S2.SS2.p6.7.m7.3.3.2.cmml">ϕ</mi><mo id="S2.SS2.p6.7.m7.3.3.1" xref="S2.SS2.p6.7.m7.3.3.1.cmml">~</mo></mover><mo stretchy="false" id="S2.SS2.p6.7.m7.4.4.3.3.2.2" xref="S2.SS2.p6.7.m7.3.3.cmml">)</mo></mrow></mrow><mo id="S2.SS2.p6.7.m7.4.4.2" xref="S2.SS2.p6.7.m7.4.4.2.cmml">=</mo><mrow id="S2.SS2.p6.7.m7.4.4.1" xref="S2.SS2.p6.7.m7.4.4.1.cmml"><mi id="S2.SS2.p6.7.m7.4.4.1.3" xref="S2.SS2.p6.7.m7.4.4.1.3.cmml">σ</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p6.7.m7.4.4.1.2" xref="S2.SS2.p6.7.m7.4.4.1.2.cmml">​</mo><mrow id="S2.SS2.p6.7.m7.4.4.1.1.1" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p6.7.m7.4.4.1.1.1.2" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.cmml">(</mo><mrow id="S2.SS2.p6.7.m7.4.4.1.1.1.1" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.cmml"><mrow id="S2.SS2.p6.7.m7.4.4.1.1.1.1.2" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.cmml"><msup id="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.2" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.2.cmml"><mi id="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.2.2" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.2.2.cmml">θ</mi><mi id="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.2.3" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.2.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.1" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.1.cmml">​</mo><mover accent="true" id="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.3" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.3.cmml"><mi id="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.3.2" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.3.2.cmml">ϕ</mi><mo id="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.3.1" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.3.1.cmml">~</mo></mover></mrow><mo id="S2.SS2.p6.7.m7.4.4.1.1.1.1.1" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.1.cmml">+</mo><mi id="S2.SS2.p6.7.m7.4.4.1.1.1.1.3" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.3.cmml">b</mi></mrow><mo stretchy="false" id="S2.SS2.p6.7.m7.4.4.1.1.1.3" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p6.7.m7.4b"><apply id="S2.SS2.p6.7.m7.4.4.cmml" xref="S2.SS2.p6.7.m7.4.4"><eq id="S2.SS2.p6.7.m7.4.4.2.cmml" xref="S2.SS2.p6.7.m7.4.4.2"></eq><apply id="S2.SS2.p6.7.m7.4.4.3.cmml" xref="S2.SS2.p6.7.m7.4.4.3"><times id="S2.SS2.p6.7.m7.4.4.3.1.cmml" xref="S2.SS2.p6.7.m7.4.4.3.1"></times><apply id="S2.SS2.p6.7.m7.4.4.3.2.cmml" xref="S2.SS2.p6.7.m7.4.4.3.2"><csymbol cd="ambiguous" id="S2.SS2.p6.7.m7.4.4.3.2.1.cmml" xref="S2.SS2.p6.7.m7.4.4.3.2">subscript</csymbol><ci id="S2.SS2.p6.7.m7.4.4.3.2.2.cmml" xref="S2.SS2.p6.7.m7.4.4.3.2.2">𝑝</ci><list id="S2.SS2.p6.7.m7.2.2.2.3.cmml" xref="S2.SS2.p6.7.m7.2.2.2.4"><ci id="S2.SS2.p6.7.m7.1.1.1.1.cmml" xref="S2.SS2.p6.7.m7.1.1.1.1">𝜃</ci><ci id="S2.SS2.p6.7.m7.2.2.2.2.cmml" xref="S2.SS2.p6.7.m7.2.2.2.2">𝑏</ci></list></apply><apply id="S2.SS2.p6.7.m7.3.3.cmml" xref="S2.SS2.p6.7.m7.4.4.3.3.2"><ci id="S2.SS2.p6.7.m7.3.3.1.cmml" xref="S2.SS2.p6.7.m7.3.3.1">~</ci><ci id="S2.SS2.p6.7.m7.3.3.2.cmml" xref="S2.SS2.p6.7.m7.3.3.2">italic-ϕ</ci></apply></apply><apply id="S2.SS2.p6.7.m7.4.4.1.cmml" xref="S2.SS2.p6.7.m7.4.4.1"><times id="S2.SS2.p6.7.m7.4.4.1.2.cmml" xref="S2.SS2.p6.7.m7.4.4.1.2"></times><ci id="S2.SS2.p6.7.m7.4.4.1.3.cmml" xref="S2.SS2.p6.7.m7.4.4.1.3">𝜎</ci><apply id="S2.SS2.p6.7.m7.4.4.1.1.1.1.cmml" xref="S2.SS2.p6.7.m7.4.4.1.1.1"><plus id="S2.SS2.p6.7.m7.4.4.1.1.1.1.1.cmml" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.1"></plus><apply id="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.cmml" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.2"><times id="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.1.cmml" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.1"></times><apply id="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.2.cmml" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.2"><csymbol cd="ambiguous" id="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.2.1.cmml" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.2">superscript</csymbol><ci id="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.2.2.cmml" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.2.2">𝜃</ci><ci id="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.2.3.cmml" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.2.3">𝑇</ci></apply><apply id="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.3.cmml" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.3"><ci id="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.3.1.cmml" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.3.1">~</ci><ci id="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.3.2.cmml" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.2.3.2">italic-ϕ</ci></apply></apply><ci id="S2.SS2.p6.7.m7.4.4.1.1.1.1.3.cmml" xref="S2.SS2.p6.7.m7.4.4.1.1.1.1.3">𝑏</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p6.7.m7.4c">p_{\theta,b}(\tilde{\phi})=\sigma(\theta^{T}\tilde{\phi}+b)</annotation></semantics></math>, but nonlinear projections can also work.
For simplicity, we sometimes omit the <math id="S2.SS2.p6.8.m8.2" class="ltx_Math" alttext="\theta,b" display="inline"><semantics id="S2.SS2.p6.8.m8.2a"><mrow id="S2.SS2.p6.8.m8.2.3.2" xref="S2.SS2.p6.8.m8.2.3.1.cmml"><mi id="S2.SS2.p6.8.m8.1.1" xref="S2.SS2.p6.8.m8.1.1.cmml">θ</mi><mo id="S2.SS2.p6.8.m8.2.3.2.1" xref="S2.SS2.p6.8.m8.2.3.1.cmml">,</mo><mi id="S2.SS2.p6.8.m8.2.2" xref="S2.SS2.p6.8.m8.2.2.cmml">b</mi></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p6.8.m8.2b"><list id="S2.SS2.p6.8.m8.2.3.1.cmml" xref="S2.SS2.p6.8.m8.2.3.2"><ci id="S2.SS2.p6.8.m8.1.1.cmml" xref="S2.SS2.p6.8.m8.1.1">𝜃</ci><ci id="S2.SS2.p6.8.m8.2.2.cmml" xref="S2.SS2.p6.8.m8.2.2">𝑏</ci></list></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p6.8.m8.2c">\theta,b</annotation></semantics></math> subscript in <math id="S2.SS2.p6.9.m9.1" class="ltx_Math" alttext="p" display="inline"><semantics id="S2.SS2.p6.9.m9.1a"><mi id="S2.SS2.p6.9.m9.1.1" xref="S2.SS2.p6.9.m9.1.1.cmml">p</mi><annotation-xml encoding="MathML-Content" id="S2.SS2.p6.9.m9.1b"><ci id="S2.SS2.p6.9.m9.1.1.cmml" xref="S2.SS2.p6.9.m9.1.1">𝑝</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p6.9.m9.1c">p</annotation></semantics></math>.</p>
</div>
<div id="S2.SS2.p7" class="ltx_para ltx_noindent">
<p id="S2.SS2.p7.1" class="ltx_p"><span id="S2.SS2.p7.1.1" class="ltx_text ltx_font_bold">Training objective.</span>
To find features that represent the truth, we leverage the consistency structure of truth. First, we use the fact that a statement and its negation should have probabilities that add up to <math id="S2.SS2.p7.1.m1.1" class="ltx_Math" alttext="1" display="inline"><semantics id="S2.SS2.p7.1.m1.1a"><mn id="S2.SS2.p7.1.m1.1.1" xref="S2.SS2.p7.1.m1.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="S2.SS2.p7.1.m1.1b"><cn type="integer" id="S2.SS2.p7.1.m1.1.1.cmml" xref="S2.SS2.p7.1.m1.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p7.1.m1.1c">1</annotation></semantics></math>. This motivates the consistency loss:</p>
<table id="S2.Ex2" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.Ex2.m1.8" class="ltx_Math" alttext="L_{\text{consistency}}(\theta,b;q_{i}):=\left[p_{\theta,b}(x_{i}^{+})-(1-p_{\theta,b}(x_{i}^{-}))\right]^{2}" display="block"><semantics id="S2.Ex2.m1.8a"><mrow id="S2.Ex2.m1.8.8" xref="S2.Ex2.m1.8.8.cmml"><mrow id="S2.Ex2.m1.7.7.1" xref="S2.Ex2.m1.7.7.1.cmml"><msub id="S2.Ex2.m1.7.7.1.3" xref="S2.Ex2.m1.7.7.1.3.cmml"><mi id="S2.Ex2.m1.7.7.1.3.2" xref="S2.Ex2.m1.7.7.1.3.2.cmml">L</mi><mtext id="S2.Ex2.m1.7.7.1.3.3" xref="S2.Ex2.m1.7.7.1.3.3a.cmml">consistency</mtext></msub><mo lspace="0em" rspace="0em" id="S2.Ex2.m1.7.7.1.2" xref="S2.Ex2.m1.7.7.1.2.cmml">​</mo><mrow id="S2.Ex2.m1.7.7.1.1.1" xref="S2.Ex2.m1.7.7.1.1.2.cmml"><mo stretchy="false" id="S2.Ex2.m1.7.7.1.1.1.2" xref="S2.Ex2.m1.7.7.1.1.2.cmml">(</mo><mi id="S2.Ex2.m1.5.5" xref="S2.Ex2.m1.5.5.cmml">θ</mi><mo id="S2.Ex2.m1.7.7.1.1.1.3" xref="S2.Ex2.m1.7.7.1.1.2.cmml">,</mo><mi id="S2.Ex2.m1.6.6" xref="S2.Ex2.m1.6.6.cmml">b</mi><mo id="S2.Ex2.m1.7.7.1.1.1.4" xref="S2.Ex2.m1.7.7.1.1.2.cmml">;</mo><msub id="S2.Ex2.m1.7.7.1.1.1.1" xref="S2.Ex2.m1.7.7.1.1.1.1.cmml"><mi id="S2.Ex2.m1.7.7.1.1.1.1.2" xref="S2.Ex2.m1.7.7.1.1.1.1.2.cmml">q</mi><mi id="S2.Ex2.m1.7.7.1.1.1.1.3" xref="S2.Ex2.m1.7.7.1.1.1.1.3.cmml">i</mi></msub><mo rspace="0.278em" stretchy="false" id="S2.Ex2.m1.7.7.1.1.1.5" xref="S2.Ex2.m1.7.7.1.1.2.cmml">)</mo></mrow></mrow><mo rspace="0.278em" id="S2.Ex2.m1.8.8.3" xref="S2.Ex2.m1.8.8.3.cmml">:=</mo><msup id="S2.Ex2.m1.8.8.2" xref="S2.Ex2.m1.8.8.2.cmml"><mrow id="S2.Ex2.m1.8.8.2.1.1" xref="S2.Ex2.m1.8.8.2.1.2.cmml"><mo id="S2.Ex2.m1.8.8.2.1.1.2" xref="S2.Ex2.m1.8.8.2.1.2.1.cmml">[</mo><mrow id="S2.Ex2.m1.8.8.2.1.1.1" xref="S2.Ex2.m1.8.8.2.1.1.1.cmml"><mrow id="S2.Ex2.m1.8.8.2.1.1.1.1" xref="S2.Ex2.m1.8.8.2.1.1.1.1.cmml"><msub id="S2.Ex2.m1.8.8.2.1.1.1.1.3" xref="S2.Ex2.m1.8.8.2.1.1.1.1.3.cmml"><mi id="S2.Ex2.m1.8.8.2.1.1.1.1.3.2" xref="S2.Ex2.m1.8.8.2.1.1.1.1.3.2.cmml">p</mi><mrow id="S2.Ex2.m1.2.2.2.4" xref="S2.Ex2.m1.2.2.2.3.cmml"><mi id="S2.Ex2.m1.1.1.1.1" xref="S2.Ex2.m1.1.1.1.1.cmml">θ</mi><mo id="S2.Ex2.m1.2.2.2.4.1" xref="S2.Ex2.m1.2.2.2.3.cmml">,</mo><mi id="S2.Ex2.m1.2.2.2.2" xref="S2.Ex2.m1.2.2.2.2.cmml">b</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S2.Ex2.m1.8.8.2.1.1.1.1.2" xref="S2.Ex2.m1.8.8.2.1.1.1.1.2.cmml">​</mo><mrow id="S2.Ex2.m1.8.8.2.1.1.1.1.1.1" xref="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.2" xref="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1" xref="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.cmml"><mi id="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.2.2" xref="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.2.3" xref="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.3" xref="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.3.cmml">+</mo></msubsup><mo stretchy="false" id="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.3" xref="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.Ex2.m1.8.8.2.1.1.1.3" xref="S2.Ex2.m1.8.8.2.1.1.1.3.cmml">−</mo><mrow id="S2.Ex2.m1.8.8.2.1.1.1.2.1" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.cmml"><mo stretchy="false" id="S2.Ex2.m1.8.8.2.1.1.1.2.1.2" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.cmml">(</mo><mrow id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.cmml"><mn id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.3" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.3.cmml">1</mn><mo id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.2" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.2.cmml">−</mo><mrow id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.cmml"><msub id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.3" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.3.cmml"><mi id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.3.2" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.3.2.cmml">p</mi><mrow id="S2.Ex2.m1.4.4.2.4" xref="S2.Ex2.m1.4.4.2.3.cmml"><mi id="S2.Ex2.m1.3.3.1.1" xref="S2.Ex2.m1.3.3.1.1.cmml">θ</mi><mo id="S2.Ex2.m1.4.4.2.4.1" xref="S2.Ex2.m1.4.4.2.3.cmml">,</mo><mi id="S2.Ex2.m1.4.4.2.2" xref="S2.Ex2.m1.4.4.2.2.cmml">b</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.2" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.2.cmml">​</mo><mrow id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.2" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.cmml">(</mo><msubsup id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.cmml"><mi id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.2.2" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.2.3" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.3" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.3.cmml">−</mo></msubsup><mo stretchy="false" id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.3" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S2.Ex2.m1.8.8.2.1.1.1.2.1.3" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.cmml">)</mo></mrow></mrow><mo id="S2.Ex2.m1.8.8.2.1.1.3" xref="S2.Ex2.m1.8.8.2.1.2.1.cmml">]</mo></mrow><mn id="S2.Ex2.m1.8.8.2.3" xref="S2.Ex2.m1.8.8.2.3.cmml">2</mn></msup></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex2.m1.8b"><apply id="S2.Ex2.m1.8.8.cmml" xref="S2.Ex2.m1.8.8"><csymbol cd="latexml" id="S2.Ex2.m1.8.8.3.cmml" xref="S2.Ex2.m1.8.8.3">assign</csymbol><apply id="S2.Ex2.m1.7.7.1.cmml" xref="S2.Ex2.m1.7.7.1"><times id="S2.Ex2.m1.7.7.1.2.cmml" xref="S2.Ex2.m1.7.7.1.2"></times><apply id="S2.Ex2.m1.7.7.1.3.cmml" xref="S2.Ex2.m1.7.7.1.3"><csymbol cd="ambiguous" id="S2.Ex2.m1.7.7.1.3.1.cmml" xref="S2.Ex2.m1.7.7.1.3">subscript</csymbol><ci id="S2.Ex2.m1.7.7.1.3.2.cmml" xref="S2.Ex2.m1.7.7.1.3.2">𝐿</ci><ci id="S2.Ex2.m1.7.7.1.3.3a.cmml" xref="S2.Ex2.m1.7.7.1.3.3"><mtext mathsize="70%" id="S2.Ex2.m1.7.7.1.3.3.cmml" xref="S2.Ex2.m1.7.7.1.3.3">consistency</mtext></ci></apply><vector id="S2.Ex2.m1.7.7.1.1.2.cmml" xref="S2.Ex2.m1.7.7.1.1.1"><ci id="S2.Ex2.m1.5.5.cmml" xref="S2.Ex2.m1.5.5">𝜃</ci><ci id="S2.Ex2.m1.6.6.cmml" xref="S2.Ex2.m1.6.6">𝑏</ci><apply id="S2.Ex2.m1.7.7.1.1.1.1.cmml" xref="S2.Ex2.m1.7.7.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex2.m1.7.7.1.1.1.1.1.cmml" xref="S2.Ex2.m1.7.7.1.1.1.1">subscript</csymbol><ci id="S2.Ex2.m1.7.7.1.1.1.1.2.cmml" xref="S2.Ex2.m1.7.7.1.1.1.1.2">𝑞</ci><ci id="S2.Ex2.m1.7.7.1.1.1.1.3.cmml" xref="S2.Ex2.m1.7.7.1.1.1.1.3">𝑖</ci></apply></vector></apply><apply id="S2.Ex2.m1.8.8.2.cmml" xref="S2.Ex2.m1.8.8.2"><csymbol cd="ambiguous" id="S2.Ex2.m1.8.8.2.2.cmml" xref="S2.Ex2.m1.8.8.2">superscript</csymbol><apply id="S2.Ex2.m1.8.8.2.1.2.cmml" xref="S2.Ex2.m1.8.8.2.1.1"><csymbol cd="latexml" id="S2.Ex2.m1.8.8.2.1.2.1.cmml" xref="S2.Ex2.m1.8.8.2.1.1.2">delimited-[]</csymbol><apply id="S2.Ex2.m1.8.8.2.1.1.1.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1"><minus id="S2.Ex2.m1.8.8.2.1.1.1.3.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.3"></minus><apply id="S2.Ex2.m1.8.8.2.1.1.1.1.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.1"><times id="S2.Ex2.m1.8.8.2.1.1.1.1.2.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.1.2"></times><apply id="S2.Ex2.m1.8.8.2.1.1.1.1.3.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex2.m1.8.8.2.1.1.1.1.3.1.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.1.3">subscript</csymbol><ci id="S2.Ex2.m1.8.8.2.1.1.1.1.3.2.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.1.3.2">𝑝</ci><list id="S2.Ex2.m1.2.2.2.3.cmml" xref="S2.Ex2.m1.2.2.2.4"><ci id="S2.Ex2.m1.1.1.1.1.cmml" xref="S2.Ex2.m1.1.1.1.1">𝜃</ci><ci id="S2.Ex2.m1.2.2.2.2.cmml" xref="S2.Ex2.m1.2.2.2.2">𝑏</ci></list></apply><apply id="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.1.1.1">superscript</csymbol><apply id="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.2.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.2.1.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.1.1.1">subscript</csymbol><ci id="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.2.2.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.2.3.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><plus id="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.3.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.1.1.1.1.3"></plus></apply></apply><apply id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1"><minus id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.2.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.2"></minus><cn type="integer" id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.3.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.3">1</cn><apply id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1"><times id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.2.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.2"></times><apply id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.3.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.3.1.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.3">subscript</csymbol><ci id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.3.2.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.3.2">𝑝</ci><list id="S2.Ex2.m1.4.4.2.3.cmml" xref="S2.Ex2.m1.4.4.2.4"><ci id="S2.Ex2.m1.3.3.1.1.cmml" xref="S2.Ex2.m1.3.3.1.1">𝜃</ci><ci id="S2.Ex2.m1.4.4.2.2.cmml" xref="S2.Ex2.m1.4.4.2.2">𝑏</ci></list></apply><apply id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.1.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1">superscript</csymbol><apply id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.2.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.2.1.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1">subscript</csymbol><ci id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.2.2.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.2.3.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.2.3">𝑖</ci></apply><minus id="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.3.cmml" xref="S2.Ex2.m1.8.8.2.1.1.1.2.1.1.1.1.1.1.3"></minus></apply></apply></apply></apply></apply><cn type="integer" id="S2.Ex2.m1.8.8.2.3.cmml" xref="S2.Ex2.m1.8.8.2.3">2</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex2.m1.8c">L_{\text{consistency}}(\theta,b;q_{i}):=\left[p_{\theta,b}(x_{i}^{+})-(1-p_{\theta,b}(x_{i}^{-}))\right]^{2}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
</div>
<div id="S2.SS2.p8" class="ltx_para ltx_noindent">
<p id="S2.SS2.p8.1" class="ltx_p">However, this objective alone has a degenerate solution: <math id="S2.SS2.p8.1.m1.2" class="ltx_Math" alttext="p(x^{+})=p(x^{-})=0.5" display="inline"><semantics id="S2.SS2.p8.1.m1.2a"><mrow id="S2.SS2.p8.1.m1.2.2" xref="S2.SS2.p8.1.m1.2.2.cmml"><mrow id="S2.SS2.p8.1.m1.1.1.1" xref="S2.SS2.p8.1.m1.1.1.1.cmml"><mi id="S2.SS2.p8.1.m1.1.1.1.3" xref="S2.SS2.p8.1.m1.1.1.1.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p8.1.m1.1.1.1.2" xref="S2.SS2.p8.1.m1.1.1.1.2.cmml">​</mo><mrow id="S2.SS2.p8.1.m1.1.1.1.1.1" xref="S2.SS2.p8.1.m1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p8.1.m1.1.1.1.1.1.2" xref="S2.SS2.p8.1.m1.1.1.1.1.1.1.cmml">(</mo><msup id="S2.SS2.p8.1.m1.1.1.1.1.1.1" xref="S2.SS2.p8.1.m1.1.1.1.1.1.1.cmml"><mi id="S2.SS2.p8.1.m1.1.1.1.1.1.1.2" xref="S2.SS2.p8.1.m1.1.1.1.1.1.1.2.cmml">x</mi><mo id="S2.SS2.p8.1.m1.1.1.1.1.1.1.3" xref="S2.SS2.p8.1.m1.1.1.1.1.1.1.3.cmml">+</mo></msup><mo stretchy="false" id="S2.SS2.p8.1.m1.1.1.1.1.1.3" xref="S2.SS2.p8.1.m1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.SS2.p8.1.m1.2.2.4" xref="S2.SS2.p8.1.m1.2.2.4.cmml">=</mo><mrow id="S2.SS2.p8.1.m1.2.2.2" xref="S2.SS2.p8.1.m1.2.2.2.cmml"><mi id="S2.SS2.p8.1.m1.2.2.2.3" xref="S2.SS2.p8.1.m1.2.2.2.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p8.1.m1.2.2.2.2" xref="S2.SS2.p8.1.m1.2.2.2.2.cmml">​</mo><mrow id="S2.SS2.p8.1.m1.2.2.2.1.1" xref="S2.SS2.p8.1.m1.2.2.2.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p8.1.m1.2.2.2.1.1.2" xref="S2.SS2.p8.1.m1.2.2.2.1.1.1.cmml">(</mo><msup id="S2.SS2.p8.1.m1.2.2.2.1.1.1" xref="S2.SS2.p8.1.m1.2.2.2.1.1.1.cmml"><mi id="S2.SS2.p8.1.m1.2.2.2.1.1.1.2" xref="S2.SS2.p8.1.m1.2.2.2.1.1.1.2.cmml">x</mi><mo id="S2.SS2.p8.1.m1.2.2.2.1.1.1.3" xref="S2.SS2.p8.1.m1.2.2.2.1.1.1.3.cmml">−</mo></msup><mo stretchy="false" id="S2.SS2.p8.1.m1.2.2.2.1.1.3" xref="S2.SS2.p8.1.m1.2.2.2.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.SS2.p8.1.m1.2.2.5" xref="S2.SS2.p8.1.m1.2.2.5.cmml">=</mo><mn id="S2.SS2.p8.1.m1.2.2.6" xref="S2.SS2.p8.1.m1.2.2.6.cmml">0.5</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p8.1.m1.2b"><apply id="S2.SS2.p8.1.m1.2.2.cmml" xref="S2.SS2.p8.1.m1.2.2"><and id="S2.SS2.p8.1.m1.2.2a.cmml" xref="S2.SS2.p8.1.m1.2.2"></and><apply id="S2.SS2.p8.1.m1.2.2b.cmml" xref="S2.SS2.p8.1.m1.2.2"><eq id="S2.SS2.p8.1.m1.2.2.4.cmml" xref="S2.SS2.p8.1.m1.2.2.4"></eq><apply id="S2.SS2.p8.1.m1.1.1.1.cmml" xref="S2.SS2.p8.1.m1.1.1.1"><times id="S2.SS2.p8.1.m1.1.1.1.2.cmml" xref="S2.SS2.p8.1.m1.1.1.1.2"></times><ci id="S2.SS2.p8.1.m1.1.1.1.3.cmml" xref="S2.SS2.p8.1.m1.1.1.1.3">𝑝</ci><apply id="S2.SS2.p8.1.m1.1.1.1.1.1.1.cmml" xref="S2.SS2.p8.1.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p8.1.m1.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p8.1.m1.1.1.1.1.1">superscript</csymbol><ci id="S2.SS2.p8.1.m1.1.1.1.1.1.1.2.cmml" xref="S2.SS2.p8.1.m1.1.1.1.1.1.1.2">𝑥</ci><plus id="S2.SS2.p8.1.m1.1.1.1.1.1.1.3.cmml" xref="S2.SS2.p8.1.m1.1.1.1.1.1.1.3"></plus></apply></apply><apply id="S2.SS2.p8.1.m1.2.2.2.cmml" xref="S2.SS2.p8.1.m1.2.2.2"><times id="S2.SS2.p8.1.m1.2.2.2.2.cmml" xref="S2.SS2.p8.1.m1.2.2.2.2"></times><ci id="S2.SS2.p8.1.m1.2.2.2.3.cmml" xref="S2.SS2.p8.1.m1.2.2.2.3">𝑝</ci><apply id="S2.SS2.p8.1.m1.2.2.2.1.1.1.cmml" xref="S2.SS2.p8.1.m1.2.2.2.1.1"><csymbol cd="ambiguous" id="S2.SS2.p8.1.m1.2.2.2.1.1.1.1.cmml" xref="S2.SS2.p8.1.m1.2.2.2.1.1">superscript</csymbol><ci id="S2.SS2.p8.1.m1.2.2.2.1.1.1.2.cmml" xref="S2.SS2.p8.1.m1.2.2.2.1.1.1.2">𝑥</ci><minus id="S2.SS2.p8.1.m1.2.2.2.1.1.1.3.cmml" xref="S2.SS2.p8.1.m1.2.2.2.1.1.1.3"></minus></apply></apply></apply><apply id="S2.SS2.p8.1.m1.2.2c.cmml" xref="S2.SS2.p8.1.m1.2.2"><eq id="S2.SS2.p8.1.m1.2.2.5.cmml" xref="S2.SS2.p8.1.m1.2.2.5"></eq><share href="#S2.SS2.p8.1.m1.2.2.2.cmml" id="S2.SS2.p8.1.m1.2.2d.cmml" xref="S2.SS2.p8.1.m1.2.2"></share><cn type="float" id="S2.SS2.p8.1.m1.2.2.6.cmml" xref="S2.SS2.p8.1.m1.2.2.6">0.5</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p8.1.m1.2c">p(x^{+})=p(x^{-})=0.5</annotation></semantics></math>. To avoid this problem, we encourage the model to also be confident with the following confidence loss:</p>
<table id="S2.Ex3" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.Ex3.m1.7" class="ltx_math_unparsed" alttext="L_{\text{confidence}}(\theta,b;q_{i}):=\min\{p_{\theta,b}(x_{i}^{+}),p_{\theta,b}(x_{i}^{-})\}^{2}" display="block"><semantics id="S2.Ex3.m1.7a"><mrow id="S2.Ex3.m1.7b"><msub id="S2.Ex3.m1.7.8"><mi id="S2.Ex3.m1.7.8.2">L</mi><mtext id="S2.Ex3.m1.7.8.3">confidence</mtext></msub><mrow id="S2.Ex3.m1.7.9"><mo stretchy="false" id="S2.Ex3.m1.7.9.1">(</mo><mi id="S2.Ex3.m1.5.5">θ</mi><mo id="S2.Ex3.m1.7.9.2">,</mo><mi id="S2.Ex3.m1.6.6">b</mi><mo id="S2.Ex3.m1.7.9.3">;</mo><msub id="S2.Ex3.m1.7.9.4"><mi id="S2.Ex3.m1.7.9.4.2">q</mi><mi id="S2.Ex3.m1.7.9.4.3">i</mi></msub><mo rspace="0.278em" stretchy="false" id="S2.Ex3.m1.7.9.5">)</mo></mrow><mo rspace="0.278em" id="S2.Ex3.m1.7.10">:=</mo><mi id="S2.Ex3.m1.7.7">min</mi><msup id="S2.Ex3.m1.7.11"><mrow id="S2.Ex3.m1.7.11.2"><mo stretchy="false" id="S2.Ex3.m1.7.11.2.1">{</mo><msub id="S2.Ex3.m1.7.11.2.2"><mi id="S2.Ex3.m1.7.11.2.2.2">p</mi><mrow id="S2.Ex3.m1.2.2.2.4"><mi id="S2.Ex3.m1.1.1.1.1">θ</mi><mo id="S2.Ex3.m1.2.2.2.4.1">,</mo><mi id="S2.Ex3.m1.2.2.2.2">b</mi></mrow></msub><mrow id="S2.Ex3.m1.7.11.2.3"><mo stretchy="false" id="S2.Ex3.m1.7.11.2.3.1">(</mo><msubsup id="S2.Ex3.m1.7.11.2.3.2"><mi id="S2.Ex3.m1.7.11.2.3.2.2.2">x</mi><mi id="S2.Ex3.m1.7.11.2.3.2.2.3">i</mi><mo id="S2.Ex3.m1.7.11.2.3.2.3">+</mo></msubsup><mo stretchy="false" id="S2.Ex3.m1.7.11.2.3.3">)</mo></mrow><mo id="S2.Ex3.m1.7.11.2.4">,</mo><msub id="S2.Ex3.m1.7.11.2.5"><mi id="S2.Ex3.m1.7.11.2.5.2">p</mi><mrow id="S2.Ex3.m1.4.4.2.4"><mi id="S2.Ex3.m1.3.3.1.1">θ</mi><mo id="S2.Ex3.m1.4.4.2.4.1">,</mo><mi id="S2.Ex3.m1.4.4.2.2">b</mi></mrow></msub><mrow id="S2.Ex3.m1.7.11.2.6"><mo stretchy="false" id="S2.Ex3.m1.7.11.2.6.1">(</mo><msubsup id="S2.Ex3.m1.7.11.2.6.2"><mi id="S2.Ex3.m1.7.11.2.6.2.2.2">x</mi><mi id="S2.Ex3.m1.7.11.2.6.2.2.3">i</mi><mo id="S2.Ex3.m1.7.11.2.6.2.3">−</mo></msubsup><mo stretchy="false" id="S2.Ex3.m1.7.11.2.6.3">)</mo></mrow><mo stretchy="false" id="S2.Ex3.m1.7.11.2.7">}</mo></mrow><mn id="S2.Ex3.m1.7.11.3">2</mn></msup></mrow><annotation encoding="application/x-tex" id="S2.Ex3.m1.7c">L_{\text{confidence}}(\theta,b;q_{i}):=\min\{p_{\theta,b}(x_{i}^{+}),p_{\theta,b}(x_{i}^{-})\}^{2}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S2.SS2.p8.2" class="ltx_p">We can equivalently interpret <math id="S2.SS2.p8.2.m1.1" class="ltx_Math" alttext="L_{\text{confidence}}" display="inline"><semantics id="S2.SS2.p8.2.m1.1a"><msub id="S2.SS2.p8.2.m1.1.1" xref="S2.SS2.p8.2.m1.1.1.cmml"><mi id="S2.SS2.p8.2.m1.1.1.2" xref="S2.SS2.p8.2.m1.1.1.2.cmml">L</mi><mtext id="S2.SS2.p8.2.m1.1.1.3" xref="S2.SS2.p8.2.m1.1.1.3a.cmml">confidence</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p8.2.m1.1b"><apply id="S2.SS2.p8.2.m1.1.1.cmml" xref="S2.SS2.p8.2.m1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p8.2.m1.1.1.1.cmml" xref="S2.SS2.p8.2.m1.1.1">subscript</csymbol><ci id="S2.SS2.p8.2.m1.1.1.2.cmml" xref="S2.SS2.p8.2.m1.1.1.2">𝐿</ci><ci id="S2.SS2.p8.2.m1.1.1.3a.cmml" xref="S2.SS2.p8.2.m1.1.1.3"><mtext mathsize="70%" id="S2.SS2.p8.2.m1.1.1.3.cmml" xref="S2.SS2.p8.2.m1.1.1.3">confidence</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p8.2.m1.1c">L_{\text{confidence}}</annotation></semantics></math> as imposing a second consistency property on the probabilities: the law of excluded middle (every statement must be either true or false).
The final unsupervised loss is the sum of these two losses, averaged across all contrast pairs:</p>
<table id="S2.Ex4" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.Ex4.m1.8" class="ltx_Math" alttext="L_{\text{CCS}}(\theta,b):=\frac{1}{n}\sum_{i=1}^{n}L_{\text{consistency}}(\theta,b;q_{i})+L_{\text{confidence}}(\theta,b;q_{i})" display="block"><semantics id="S2.Ex4.m1.8a"><mrow id="S2.Ex4.m1.8.8" xref="S2.Ex4.m1.8.8.cmml"><mrow id="S2.Ex4.m1.8.8.4" xref="S2.Ex4.m1.8.8.4.cmml"><msub id="S2.Ex4.m1.8.8.4.2" xref="S2.Ex4.m1.8.8.4.2.cmml"><mi id="S2.Ex4.m1.8.8.4.2.2" xref="S2.Ex4.m1.8.8.4.2.2.cmml">L</mi><mtext id="S2.Ex4.m1.8.8.4.2.3" xref="S2.Ex4.m1.8.8.4.2.3a.cmml">CCS</mtext></msub><mo lspace="0em" rspace="0em" id="S2.Ex4.m1.8.8.4.1" xref="S2.Ex4.m1.8.8.4.1.cmml">​</mo><mrow id="S2.Ex4.m1.8.8.4.3.2" xref="S2.Ex4.m1.8.8.4.3.1.cmml"><mo stretchy="false" id="S2.Ex4.m1.8.8.4.3.2.1" xref="S2.Ex4.m1.8.8.4.3.1.cmml">(</mo><mi id="S2.Ex4.m1.1.1" xref="S2.Ex4.m1.1.1.cmml">θ</mi><mo id="S2.Ex4.m1.8.8.4.3.2.2" xref="S2.Ex4.m1.8.8.4.3.1.cmml">,</mo><mi id="S2.Ex4.m1.2.2" xref="S2.Ex4.m1.2.2.cmml">b</mi><mo rspace="0.278em" stretchy="false" id="S2.Ex4.m1.8.8.4.3.2.3" xref="S2.Ex4.m1.8.8.4.3.1.cmml">)</mo></mrow></mrow><mo rspace="0.278em" id="S2.Ex4.m1.8.8.3" xref="S2.Ex4.m1.8.8.3.cmml">:=</mo><mrow id="S2.Ex4.m1.8.8.2" xref="S2.Ex4.m1.8.8.2.cmml"><mrow id="S2.Ex4.m1.7.7.1.1" xref="S2.Ex4.m1.7.7.1.1.cmml"><mfrac id="S2.Ex4.m1.7.7.1.1.3" xref="S2.Ex4.m1.7.7.1.1.3.cmml"><mn id="S2.Ex4.m1.7.7.1.1.3.2" xref="S2.Ex4.m1.7.7.1.1.3.2.cmml">1</mn><mi id="S2.Ex4.m1.7.7.1.1.3.3" xref="S2.Ex4.m1.7.7.1.1.3.3.cmml">n</mi></mfrac><mo lspace="0em" rspace="0em" id="S2.Ex4.m1.7.7.1.1.2" xref="S2.Ex4.m1.7.7.1.1.2.cmml">​</mo><mrow id="S2.Ex4.m1.7.7.1.1.1" xref="S2.Ex4.m1.7.7.1.1.1.cmml"><munderover id="S2.Ex4.m1.7.7.1.1.1.2" xref="S2.Ex4.m1.7.7.1.1.1.2.cmml"><mo movablelimits="false" id="S2.Ex4.m1.7.7.1.1.1.2.2.2" xref="S2.Ex4.m1.7.7.1.1.1.2.2.2.cmml">∑</mo><mrow id="S2.Ex4.m1.7.7.1.1.1.2.2.3" xref="S2.Ex4.m1.7.7.1.1.1.2.2.3.cmml"><mi id="S2.Ex4.m1.7.7.1.1.1.2.2.3.2" xref="S2.Ex4.m1.7.7.1.1.1.2.2.3.2.cmml">i</mi><mo id="S2.Ex4.m1.7.7.1.1.1.2.2.3.1" xref="S2.Ex4.m1.7.7.1.1.1.2.2.3.1.cmml">=</mo><mn id="S2.Ex4.m1.7.7.1.1.1.2.2.3.3" xref="S2.Ex4.m1.7.7.1.1.1.2.2.3.3.cmml">1</mn></mrow><mi id="S2.Ex4.m1.7.7.1.1.1.2.3" xref="S2.Ex4.m1.7.7.1.1.1.2.3.cmml">n</mi></munderover><mrow id="S2.Ex4.m1.7.7.1.1.1.1" xref="S2.Ex4.m1.7.7.1.1.1.1.cmml"><msub id="S2.Ex4.m1.7.7.1.1.1.1.3" xref="S2.Ex4.m1.7.7.1.1.1.1.3.cmml"><mi id="S2.Ex4.m1.7.7.1.1.1.1.3.2" xref="S2.Ex4.m1.7.7.1.1.1.1.3.2.cmml">L</mi><mtext id="S2.Ex4.m1.7.7.1.1.1.1.3.3" xref="S2.Ex4.m1.7.7.1.1.1.1.3.3a.cmml">consistency</mtext></msub><mo lspace="0em" rspace="0em" id="S2.Ex4.m1.7.7.1.1.1.1.2" xref="S2.Ex4.m1.7.7.1.1.1.1.2.cmml">​</mo><mrow id="S2.Ex4.m1.7.7.1.1.1.1.1.1" xref="S2.Ex4.m1.7.7.1.1.1.1.1.2.cmml"><mo stretchy="false" id="S2.Ex4.m1.7.7.1.1.1.1.1.1.2" xref="S2.Ex4.m1.7.7.1.1.1.1.1.2.cmml">(</mo><mi id="S2.Ex4.m1.3.3" xref="S2.Ex4.m1.3.3.cmml">θ</mi><mo id="S2.Ex4.m1.7.7.1.1.1.1.1.1.3" xref="S2.Ex4.m1.7.7.1.1.1.1.1.2.cmml">,</mo><mi id="S2.Ex4.m1.4.4" xref="S2.Ex4.m1.4.4.cmml">b</mi><mo id="S2.Ex4.m1.7.7.1.1.1.1.1.1.4" xref="S2.Ex4.m1.7.7.1.1.1.1.1.2.cmml">;</mo><msub id="S2.Ex4.m1.7.7.1.1.1.1.1.1.1" xref="S2.Ex4.m1.7.7.1.1.1.1.1.1.1.cmml"><mi id="S2.Ex4.m1.7.7.1.1.1.1.1.1.1.2" xref="S2.Ex4.m1.7.7.1.1.1.1.1.1.1.2.cmml">q</mi><mi id="S2.Ex4.m1.7.7.1.1.1.1.1.1.1.3" xref="S2.Ex4.m1.7.7.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S2.Ex4.m1.7.7.1.1.1.1.1.1.5" xref="S2.Ex4.m1.7.7.1.1.1.1.1.2.cmml">)</mo></mrow></mrow></mrow></mrow><mo id="S2.Ex4.m1.8.8.2.3" xref="S2.Ex4.m1.8.8.2.3.cmml">+</mo><mrow id="S2.Ex4.m1.8.8.2.2" xref="S2.Ex4.m1.8.8.2.2.cmml"><msub id="S2.Ex4.m1.8.8.2.2.3" xref="S2.Ex4.m1.8.8.2.2.3.cmml"><mi id="S2.Ex4.m1.8.8.2.2.3.2" xref="S2.Ex4.m1.8.8.2.2.3.2.cmml">L</mi><mtext id="S2.Ex4.m1.8.8.2.2.3.3" xref="S2.Ex4.m1.8.8.2.2.3.3a.cmml">confidence</mtext></msub><mo lspace="0em" rspace="0em" id="S2.Ex4.m1.8.8.2.2.2" xref="S2.Ex4.m1.8.8.2.2.2.cmml">​</mo><mrow id="S2.Ex4.m1.8.8.2.2.1.1" xref="S2.Ex4.m1.8.8.2.2.1.2.cmml"><mo stretchy="false" id="S2.Ex4.m1.8.8.2.2.1.1.2" xref="S2.Ex4.m1.8.8.2.2.1.2.cmml">(</mo><mi id="S2.Ex4.m1.5.5" xref="S2.Ex4.m1.5.5.cmml">θ</mi><mo id="S2.Ex4.m1.8.8.2.2.1.1.3" xref="S2.Ex4.m1.8.8.2.2.1.2.cmml">,</mo><mi id="S2.Ex4.m1.6.6" xref="S2.Ex4.m1.6.6.cmml">b</mi><mo id="S2.Ex4.m1.8.8.2.2.1.1.4" xref="S2.Ex4.m1.8.8.2.2.1.2.cmml">;</mo><msub id="S2.Ex4.m1.8.8.2.2.1.1.1" xref="S2.Ex4.m1.8.8.2.2.1.1.1.cmml"><mi id="S2.Ex4.m1.8.8.2.2.1.1.1.2" xref="S2.Ex4.m1.8.8.2.2.1.1.1.2.cmml">q</mi><mi id="S2.Ex4.m1.8.8.2.2.1.1.1.3" xref="S2.Ex4.m1.8.8.2.2.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S2.Ex4.m1.8.8.2.2.1.1.5" xref="S2.Ex4.m1.8.8.2.2.1.2.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.Ex4.m1.8b"><apply id="S2.Ex4.m1.8.8.cmml" xref="S2.Ex4.m1.8.8"><csymbol cd="latexml" id="S2.Ex4.m1.8.8.3.cmml" xref="S2.Ex4.m1.8.8.3">assign</csymbol><apply id="S2.Ex4.m1.8.8.4.cmml" xref="S2.Ex4.m1.8.8.4"><times id="S2.Ex4.m1.8.8.4.1.cmml" xref="S2.Ex4.m1.8.8.4.1"></times><apply id="S2.Ex4.m1.8.8.4.2.cmml" xref="S2.Ex4.m1.8.8.4.2"><csymbol cd="ambiguous" id="S2.Ex4.m1.8.8.4.2.1.cmml" xref="S2.Ex4.m1.8.8.4.2">subscript</csymbol><ci id="S2.Ex4.m1.8.8.4.2.2.cmml" xref="S2.Ex4.m1.8.8.4.2.2">𝐿</ci><ci id="S2.Ex4.m1.8.8.4.2.3a.cmml" xref="S2.Ex4.m1.8.8.4.2.3"><mtext mathsize="70%" id="S2.Ex4.m1.8.8.4.2.3.cmml" xref="S2.Ex4.m1.8.8.4.2.3">CCS</mtext></ci></apply><interval closure="open" id="S2.Ex4.m1.8.8.4.3.1.cmml" xref="S2.Ex4.m1.8.8.4.3.2"><ci id="S2.Ex4.m1.1.1.cmml" xref="S2.Ex4.m1.1.1">𝜃</ci><ci id="S2.Ex4.m1.2.2.cmml" xref="S2.Ex4.m1.2.2">𝑏</ci></interval></apply><apply id="S2.Ex4.m1.8.8.2.cmml" xref="S2.Ex4.m1.8.8.2"><plus id="S2.Ex4.m1.8.8.2.3.cmml" xref="S2.Ex4.m1.8.8.2.3"></plus><apply id="S2.Ex4.m1.7.7.1.1.cmml" xref="S2.Ex4.m1.7.7.1.1"><times id="S2.Ex4.m1.7.7.1.1.2.cmml" xref="S2.Ex4.m1.7.7.1.1.2"></times><apply id="S2.Ex4.m1.7.7.1.1.3.cmml" xref="S2.Ex4.m1.7.7.1.1.3"><divide id="S2.Ex4.m1.7.7.1.1.3.1.cmml" xref="S2.Ex4.m1.7.7.1.1.3"></divide><cn type="integer" id="S2.Ex4.m1.7.7.1.1.3.2.cmml" xref="S2.Ex4.m1.7.7.1.1.3.2">1</cn><ci id="S2.Ex4.m1.7.7.1.1.3.3.cmml" xref="S2.Ex4.m1.7.7.1.1.3.3">𝑛</ci></apply><apply id="S2.Ex4.m1.7.7.1.1.1.cmml" xref="S2.Ex4.m1.7.7.1.1.1"><apply id="S2.Ex4.m1.7.7.1.1.1.2.cmml" xref="S2.Ex4.m1.7.7.1.1.1.2"><csymbol cd="ambiguous" id="S2.Ex4.m1.7.7.1.1.1.2.1.cmml" xref="S2.Ex4.m1.7.7.1.1.1.2">superscript</csymbol><apply id="S2.Ex4.m1.7.7.1.1.1.2.2.cmml" xref="S2.Ex4.m1.7.7.1.1.1.2"><csymbol cd="ambiguous" id="S2.Ex4.m1.7.7.1.1.1.2.2.1.cmml" xref="S2.Ex4.m1.7.7.1.1.1.2">subscript</csymbol><sum id="S2.Ex4.m1.7.7.1.1.1.2.2.2.cmml" xref="S2.Ex4.m1.7.7.1.1.1.2.2.2"></sum><apply id="S2.Ex4.m1.7.7.1.1.1.2.2.3.cmml" xref="S2.Ex4.m1.7.7.1.1.1.2.2.3"><eq id="S2.Ex4.m1.7.7.1.1.1.2.2.3.1.cmml" xref="S2.Ex4.m1.7.7.1.1.1.2.2.3.1"></eq><ci id="S2.Ex4.m1.7.7.1.1.1.2.2.3.2.cmml" xref="S2.Ex4.m1.7.7.1.1.1.2.2.3.2">𝑖</ci><cn type="integer" id="S2.Ex4.m1.7.7.1.1.1.2.2.3.3.cmml" xref="S2.Ex4.m1.7.7.1.1.1.2.2.3.3">1</cn></apply></apply><ci id="S2.Ex4.m1.7.7.1.1.1.2.3.cmml" xref="S2.Ex4.m1.7.7.1.1.1.2.3">𝑛</ci></apply><apply id="S2.Ex4.m1.7.7.1.1.1.1.cmml" xref="S2.Ex4.m1.7.7.1.1.1.1"><times id="S2.Ex4.m1.7.7.1.1.1.1.2.cmml" xref="S2.Ex4.m1.7.7.1.1.1.1.2"></times><apply id="S2.Ex4.m1.7.7.1.1.1.1.3.cmml" xref="S2.Ex4.m1.7.7.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.Ex4.m1.7.7.1.1.1.1.3.1.cmml" xref="S2.Ex4.m1.7.7.1.1.1.1.3">subscript</csymbol><ci id="S2.Ex4.m1.7.7.1.1.1.1.3.2.cmml" xref="S2.Ex4.m1.7.7.1.1.1.1.3.2">𝐿</ci><ci id="S2.Ex4.m1.7.7.1.1.1.1.3.3a.cmml" xref="S2.Ex4.m1.7.7.1.1.1.1.3.3"><mtext mathsize="70%" id="S2.Ex4.m1.7.7.1.1.1.1.3.3.cmml" xref="S2.Ex4.m1.7.7.1.1.1.1.3.3">consistency</mtext></ci></apply><vector id="S2.Ex4.m1.7.7.1.1.1.1.1.2.cmml" xref="S2.Ex4.m1.7.7.1.1.1.1.1.1"><ci id="S2.Ex4.m1.3.3.cmml" xref="S2.Ex4.m1.3.3">𝜃</ci><ci id="S2.Ex4.m1.4.4.cmml" xref="S2.Ex4.m1.4.4">𝑏</ci><apply id="S2.Ex4.m1.7.7.1.1.1.1.1.1.1.cmml" xref="S2.Ex4.m1.7.7.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.Ex4.m1.7.7.1.1.1.1.1.1.1.1.cmml" xref="S2.Ex4.m1.7.7.1.1.1.1.1.1.1">subscript</csymbol><ci id="S2.Ex4.m1.7.7.1.1.1.1.1.1.1.2.cmml" xref="S2.Ex4.m1.7.7.1.1.1.1.1.1.1.2">𝑞</ci><ci id="S2.Ex4.m1.7.7.1.1.1.1.1.1.1.3.cmml" xref="S2.Ex4.m1.7.7.1.1.1.1.1.1.1.3">𝑖</ci></apply></vector></apply></apply></apply><apply id="S2.Ex4.m1.8.8.2.2.cmml" xref="S2.Ex4.m1.8.8.2.2"><times id="S2.Ex4.m1.8.8.2.2.2.cmml" xref="S2.Ex4.m1.8.8.2.2.2"></times><apply id="S2.Ex4.m1.8.8.2.2.3.cmml" xref="S2.Ex4.m1.8.8.2.2.3"><csymbol cd="ambiguous" id="S2.Ex4.m1.8.8.2.2.3.1.cmml" xref="S2.Ex4.m1.8.8.2.2.3">subscript</csymbol><ci id="S2.Ex4.m1.8.8.2.2.3.2.cmml" xref="S2.Ex4.m1.8.8.2.2.3.2">𝐿</ci><ci id="S2.Ex4.m1.8.8.2.2.3.3a.cmml" xref="S2.Ex4.m1.8.8.2.2.3.3"><mtext mathsize="70%" id="S2.Ex4.m1.8.8.2.2.3.3.cmml" xref="S2.Ex4.m1.8.8.2.2.3.3">confidence</mtext></ci></apply><vector id="S2.Ex4.m1.8.8.2.2.1.2.cmml" xref="S2.Ex4.m1.8.8.2.2.1.1"><ci id="S2.Ex4.m1.5.5.cmml" xref="S2.Ex4.m1.5.5">𝜃</ci><ci id="S2.Ex4.m1.6.6.cmml" xref="S2.Ex4.m1.6.6">𝑏</ci><apply id="S2.Ex4.m1.8.8.2.2.1.1.1.cmml" xref="S2.Ex4.m1.8.8.2.2.1.1.1"><csymbol cd="ambiguous" id="S2.Ex4.m1.8.8.2.2.1.1.1.1.cmml" xref="S2.Ex4.m1.8.8.2.2.1.1.1">subscript</csymbol><ci id="S2.Ex4.m1.8.8.2.2.1.1.1.2.cmml" xref="S2.Ex4.m1.8.8.2.2.1.1.1.2">𝑞</ci><ci id="S2.Ex4.m1.8.8.2.2.1.1.1.3.cmml" xref="S2.Ex4.m1.8.8.2.2.1.1.1.3">𝑖</ci></apply></vector></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.Ex4.m1.8c">L_{\text{CCS}}(\theta,b):=\frac{1}{n}\sum_{i=1}^{n}L_{\text{consistency}}(\theta,b;q_{i})+L_{\text{confidence}}(\theta,b;q_{i})</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S2.SS2.p8.3" class="ltx_p">Note that both losses are necessary; <math id="S2.SS2.p8.3.m1.1" class="ltx_Math" alttext="L_{\text{confidence}}" display="inline"><semantics id="S2.SS2.p8.3.m1.1a"><msub id="S2.SS2.p8.3.m1.1.1" xref="S2.SS2.p8.3.m1.1.1.cmml"><mi id="S2.SS2.p8.3.m1.1.1.2" xref="S2.SS2.p8.3.m1.1.1.2.cmml">L</mi><mtext id="S2.SS2.p8.3.m1.1.1.3" xref="S2.SS2.p8.3.m1.1.1.3a.cmml">confidence</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p8.3.m1.1b"><apply id="S2.SS2.p8.3.m1.1.1.cmml" xref="S2.SS2.p8.3.m1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p8.3.m1.1.1.1.cmml" xref="S2.SS2.p8.3.m1.1.1">subscript</csymbol><ci id="S2.SS2.p8.3.m1.1.1.2.cmml" xref="S2.SS2.p8.3.m1.1.1.2">𝐿</ci><ci id="S2.SS2.p8.3.m1.1.1.3a.cmml" xref="S2.SS2.p8.3.m1.1.1.3"><mtext mathsize="70%" id="S2.SS2.p8.3.m1.1.1.3.cmml" xref="S2.SS2.p8.3.m1.1.1.3">confidence</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p8.3.m1.1c">L_{\text{confidence}}</annotation></semantics></math> alone also has a degenerate solution.</p>
</div>
<div id="S2.SS2.p9" class="ltx_para ltx_noindent">
<p id="S2.SS2.p9.4" class="ltx_p"><span id="S2.SS2.p9.4.1" class="ltx_text ltx_font_bold">Inference.</span>
Both <math id="S2.SS2.p9.1.m1.1" class="ltx_Math" alttext="p(x_{i}^{+})" display="inline"><semantics id="S2.SS2.p9.1.m1.1a"><mrow id="S2.SS2.p9.1.m1.1.1" xref="S2.SS2.p9.1.m1.1.1.cmml"><mi id="S2.SS2.p9.1.m1.1.1.3" xref="S2.SS2.p9.1.m1.1.1.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p9.1.m1.1.1.2" xref="S2.SS2.p9.1.m1.1.1.2.cmml">​</mo><mrow id="S2.SS2.p9.1.m1.1.1.1.1" xref="S2.SS2.p9.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p9.1.m1.1.1.1.1.2" xref="S2.SS2.p9.1.m1.1.1.1.1.1.cmml">(</mo><msubsup id="S2.SS2.p9.1.m1.1.1.1.1.1" xref="S2.SS2.p9.1.m1.1.1.1.1.1.cmml"><mi id="S2.SS2.p9.1.m1.1.1.1.1.1.2.2" xref="S2.SS2.p9.1.m1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p9.1.m1.1.1.1.1.1.2.3" xref="S2.SS2.p9.1.m1.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p9.1.m1.1.1.1.1.1.3" xref="S2.SS2.p9.1.m1.1.1.1.1.1.3.cmml">+</mo></msubsup><mo stretchy="false" id="S2.SS2.p9.1.m1.1.1.1.1.3" xref="S2.SS2.p9.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p9.1.m1.1b"><apply id="S2.SS2.p9.1.m1.1.1.cmml" xref="S2.SS2.p9.1.m1.1.1"><times id="S2.SS2.p9.1.m1.1.1.2.cmml" xref="S2.SS2.p9.1.m1.1.1.2"></times><ci id="S2.SS2.p9.1.m1.1.1.3.cmml" xref="S2.SS2.p9.1.m1.1.1.3">𝑝</ci><apply id="S2.SS2.p9.1.m1.1.1.1.1.1.cmml" xref="S2.SS2.p9.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p9.1.m1.1.1.1.1.1.1.cmml" xref="S2.SS2.p9.1.m1.1.1.1.1">superscript</csymbol><apply id="S2.SS2.p9.1.m1.1.1.1.1.1.2.cmml" xref="S2.SS2.p9.1.m1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p9.1.m1.1.1.1.1.1.2.1.cmml" xref="S2.SS2.p9.1.m1.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p9.1.m1.1.1.1.1.1.2.2.cmml" xref="S2.SS2.p9.1.m1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.SS2.p9.1.m1.1.1.1.1.1.2.3.cmml" xref="S2.SS2.p9.1.m1.1.1.1.1.1.2.3">𝑖</ci></apply><plus id="S2.SS2.p9.1.m1.1.1.1.1.1.3.cmml" xref="S2.SS2.p9.1.m1.1.1.1.1.1.3"></plus></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p9.1.m1.1c">p(x_{i}^{+})</annotation></semantics></math> and <math id="S2.SS2.p9.2.m2.1" class="ltx_Math" alttext="1-p(x_{i}^{-})" display="inline"><semantics id="S2.SS2.p9.2.m2.1a"><mrow id="S2.SS2.p9.2.m2.1.1" xref="S2.SS2.p9.2.m2.1.1.cmml"><mn id="S2.SS2.p9.2.m2.1.1.3" xref="S2.SS2.p9.2.m2.1.1.3.cmml">1</mn><mo id="S2.SS2.p9.2.m2.1.1.2" xref="S2.SS2.p9.2.m2.1.1.2.cmml">−</mo><mrow id="S2.SS2.p9.2.m2.1.1.1" xref="S2.SS2.p9.2.m2.1.1.1.cmml"><mi id="S2.SS2.p9.2.m2.1.1.1.3" xref="S2.SS2.p9.2.m2.1.1.1.3.cmml">p</mi><mo lspace="0em" rspace="0em" id="S2.SS2.p9.2.m2.1.1.1.2" xref="S2.SS2.p9.2.m2.1.1.1.2.cmml">​</mo><mrow id="S2.SS2.p9.2.m2.1.1.1.1.1" xref="S2.SS2.p9.2.m2.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p9.2.m2.1.1.1.1.1.2" xref="S2.SS2.p9.2.m2.1.1.1.1.1.1.cmml">(</mo><msubsup id="S2.SS2.p9.2.m2.1.1.1.1.1.1" xref="S2.SS2.p9.2.m2.1.1.1.1.1.1.cmml"><mi id="S2.SS2.p9.2.m2.1.1.1.1.1.1.2.2" xref="S2.SS2.p9.2.m2.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.SS2.p9.2.m2.1.1.1.1.1.1.2.3" xref="S2.SS2.p9.2.m2.1.1.1.1.1.1.2.3.cmml">i</mi><mo id="S2.SS2.p9.2.m2.1.1.1.1.1.1.3" xref="S2.SS2.p9.2.m2.1.1.1.1.1.1.3.cmml">−</mo></msubsup><mo stretchy="false" id="S2.SS2.p9.2.m2.1.1.1.1.1.3" xref="S2.SS2.p9.2.m2.1.1.1.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p9.2.m2.1b"><apply id="S2.SS2.p9.2.m2.1.1.cmml" xref="S2.SS2.p9.2.m2.1.1"><minus id="S2.SS2.p9.2.m2.1.1.2.cmml" xref="S2.SS2.p9.2.m2.1.1.2"></minus><cn type="integer" id="S2.SS2.p9.2.m2.1.1.3.cmml" xref="S2.SS2.p9.2.m2.1.1.3">1</cn><apply id="S2.SS2.p9.2.m2.1.1.1.cmml" xref="S2.SS2.p9.2.m2.1.1.1"><times id="S2.SS2.p9.2.m2.1.1.1.2.cmml" xref="S2.SS2.p9.2.m2.1.1.1.2"></times><ci id="S2.SS2.p9.2.m2.1.1.1.3.cmml" xref="S2.SS2.p9.2.m2.1.1.1.3">𝑝</ci><apply id="S2.SS2.p9.2.m2.1.1.1.1.1.1.cmml" xref="S2.SS2.p9.2.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p9.2.m2.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p9.2.m2.1.1.1.1.1">superscript</csymbol><apply id="S2.SS2.p9.2.m2.1.1.1.1.1.1.2.cmml" xref="S2.SS2.p9.2.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p9.2.m2.1.1.1.1.1.1.2.1.cmml" xref="S2.SS2.p9.2.m2.1.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p9.2.m2.1.1.1.1.1.1.2.2.cmml" xref="S2.SS2.p9.2.m2.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.SS2.p9.2.m2.1.1.1.1.1.1.2.3.cmml" xref="S2.SS2.p9.2.m2.1.1.1.1.1.1.2.3">𝑖</ci></apply><minus id="S2.SS2.p9.2.m2.1.1.1.1.1.1.3.cmml" xref="S2.SS2.p9.2.m2.1.1.1.1.1.1.3"></minus></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p9.2.m2.1c">1-p(x_{i}^{-})</annotation></semantics></math> should represent the probability that the answer to <math id="S2.SS2.p9.3.m3.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S2.SS2.p9.3.m3.1a"><msub id="S2.SS2.p9.3.m3.1.1" xref="S2.SS2.p9.3.m3.1.1.cmml"><mi id="S2.SS2.p9.3.m3.1.1.2" xref="S2.SS2.p9.3.m3.1.1.2.cmml">q</mi><mi id="S2.SS2.p9.3.m3.1.1.3" xref="S2.SS2.p9.3.m3.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p9.3.m3.1b"><apply id="S2.SS2.p9.3.m3.1.1.cmml" xref="S2.SS2.p9.3.m3.1.1"><csymbol cd="ambiguous" id="S2.SS2.p9.3.m3.1.1.1.cmml" xref="S2.SS2.p9.3.m3.1.1">subscript</csymbol><ci id="S2.SS2.p9.3.m3.1.1.2.cmml" xref="S2.SS2.p9.3.m3.1.1.2">𝑞</ci><ci id="S2.SS2.p9.3.m3.1.1.3.cmml" xref="S2.SS2.p9.3.m3.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p9.3.m3.1c">q_{i}</annotation></semantics></math> is “Yes”. However, because we use a soft consistency constraint, these may not be exactly equal. To make a prediction on an example <math id="S2.SS2.p9.4.m4.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S2.SS2.p9.4.m4.1a"><msub id="S2.SS2.p9.4.m4.1.1" xref="S2.SS2.p9.4.m4.1.1.cmml"><mi id="S2.SS2.p9.4.m4.1.1.2" xref="S2.SS2.p9.4.m4.1.1.2.cmml">x</mi><mi id="S2.SS2.p9.4.m4.1.1.3" xref="S2.SS2.p9.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p9.4.m4.1b"><apply id="S2.SS2.p9.4.m4.1.1.cmml" xref="S2.SS2.p9.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS2.p9.4.m4.1.1.1.cmml" xref="S2.SS2.p9.4.m4.1.1">subscript</csymbol><ci id="S2.SS2.p9.4.m4.1.1.2.cmml" xref="S2.SS2.p9.4.m4.1.1.2">𝑥</ci><ci id="S2.SS2.p9.4.m4.1.1.3.cmml" xref="S2.SS2.p9.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p9.4.m4.1c">x_{i}</annotation></semantics></math> after training, we consequently take the average of these:</p>
<table id="S2.Ex5" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S2.Ex5.m1.1" class="ltx_math_unparsed" alttext="\tilde{p}(q_{i}):=\frac{1}{2}(p(x_{i}^{+})+(1-p(x_{i}^{-}))" display="block"><semantics id="S2.Ex5.m1.1a"><mrow id="S2.Ex5.m1.1b"><mover accent="true" id="S2.Ex5.m1.1.1"><mi id="S2.Ex5.m1.1.1.2">p</mi><mo id="S2.Ex5.m1.1.1.1">~</mo></mover><mrow id="S2.Ex5.m1.1.2"><mo stretchy="false" id="S2.Ex5.m1.1.2.1">(</mo><msub id="S2.Ex5.m1.1.2.2"><mi id="S2.Ex5.m1.1.2.2.2">q</mi><mi id="S2.Ex5.m1.1.2.2.3">i</mi></msub><mo rspace="0.278em" stretchy="false" id="S2.Ex5.m1.1.2.3">)</mo></mrow><mo rspace="0.278em" id="S2.Ex5.m1.1.3">:=</mo><mfrac id="S2.Ex5.m1.1.4"><mn id="S2.Ex5.m1.1.4.2">1</mn><mn id="S2.Ex5.m1.1.4.3">2</mn></mfrac><mrow id="S2.Ex5.m1.1.5"><mo stretchy="false" id="S2.Ex5.m1.1.5.1">(</mo><mi id="S2.Ex5.m1.1.5.2">p</mi><mrow id="S2.Ex5.m1.1.5.3"><mo stretchy="false" id="S2.Ex5.m1.1.5.3.1">(</mo><msubsup id="S2.Ex5.m1.1.5.3.2"><mi id="S2.Ex5.m1.1.5.3.2.2.2">x</mi><mi id="S2.Ex5.m1.1.5.3.2.2.3">i</mi><mo id="S2.Ex5.m1.1.5.3.2.3">+</mo></msubsup><mo stretchy="false" id="S2.Ex5.m1.1.5.3.3">)</mo></mrow><mo id="S2.Ex5.m1.1.5.4">+</mo><mrow id="S2.Ex5.m1.1.5.5"><mo stretchy="false" id="S2.Ex5.m1.1.5.5.1">(</mo><mn id="S2.Ex5.m1.1.5.5.2">1</mn><mo id="S2.Ex5.m1.1.5.5.3">−</mo><mi id="S2.Ex5.m1.1.5.5.4">p</mi><mrow id="S2.Ex5.m1.1.5.5.5"><mo stretchy="false" id="S2.Ex5.m1.1.5.5.5.1">(</mo><msubsup id="S2.Ex5.m1.1.5.5.5.2"><mi id="S2.Ex5.m1.1.5.5.5.2.2.2">x</mi><mi id="S2.Ex5.m1.1.5.5.5.2.2.3">i</mi><mo id="S2.Ex5.m1.1.5.5.5.2.3">−</mo></msubsup><mo stretchy="false" id="S2.Ex5.m1.1.5.5.5.3">)</mo></mrow><mo stretchy="false" id="S2.Ex5.m1.1.5.5.6">)</mo></mrow></mrow></mrow><annotation encoding="application/x-tex" id="S2.Ex5.m1.1c">\tilde{p}(q_{i}):=\frac{1}{2}(p(x_{i}^{+})+(1-p(x_{i}^{-}))</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="S2.SS2.p9.9" class="ltx_p">We then predict that the answer to <math id="S2.SS2.p9.5.m1.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S2.SS2.p9.5.m1.1a"><msub id="S2.SS2.p9.5.m1.1.1" xref="S2.SS2.p9.5.m1.1.1.cmml"><mi id="S2.SS2.p9.5.m1.1.1.2" xref="S2.SS2.p9.5.m1.1.1.2.cmml">q</mi><mi id="S2.SS2.p9.5.m1.1.1.3" xref="S2.SS2.p9.5.m1.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p9.5.m1.1b"><apply id="S2.SS2.p9.5.m1.1.1.cmml" xref="S2.SS2.p9.5.m1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p9.5.m1.1.1.1.cmml" xref="S2.SS2.p9.5.m1.1.1">subscript</csymbol><ci id="S2.SS2.p9.5.m1.1.1.2.cmml" xref="S2.SS2.p9.5.m1.1.1.2">𝑞</ci><ci id="S2.SS2.p9.5.m1.1.1.3.cmml" xref="S2.SS2.p9.5.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p9.5.m1.1c">q_{i}</annotation></semantics></math> is “Yes” based on whether <math id="S2.SS2.p9.6.m2.1" class="ltx_Math" alttext="\tilde{p}(q_{i})" display="inline"><semantics id="S2.SS2.p9.6.m2.1a"><mrow id="S2.SS2.p9.6.m2.1.1" xref="S2.SS2.p9.6.m2.1.1.cmml"><mover accent="true" id="S2.SS2.p9.6.m2.1.1.3" xref="S2.SS2.p9.6.m2.1.1.3.cmml"><mi id="S2.SS2.p9.6.m2.1.1.3.2" xref="S2.SS2.p9.6.m2.1.1.3.2.cmml">p</mi><mo id="S2.SS2.p9.6.m2.1.1.3.1" xref="S2.SS2.p9.6.m2.1.1.3.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S2.SS2.p9.6.m2.1.1.2" xref="S2.SS2.p9.6.m2.1.1.2.cmml">​</mo><mrow id="S2.SS2.p9.6.m2.1.1.1.1" xref="S2.SS2.p9.6.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p9.6.m2.1.1.1.1.2" xref="S2.SS2.p9.6.m2.1.1.1.1.1.cmml">(</mo><msub id="S2.SS2.p9.6.m2.1.1.1.1.1" xref="S2.SS2.p9.6.m2.1.1.1.1.1.cmml"><mi id="S2.SS2.p9.6.m2.1.1.1.1.1.2" xref="S2.SS2.p9.6.m2.1.1.1.1.1.2.cmml">q</mi><mi id="S2.SS2.p9.6.m2.1.1.1.1.1.3" xref="S2.SS2.p9.6.m2.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S2.SS2.p9.6.m2.1.1.1.1.3" xref="S2.SS2.p9.6.m2.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p9.6.m2.1b"><apply id="S2.SS2.p9.6.m2.1.1.cmml" xref="S2.SS2.p9.6.m2.1.1"><times id="S2.SS2.p9.6.m2.1.1.2.cmml" xref="S2.SS2.p9.6.m2.1.1.2"></times><apply id="S2.SS2.p9.6.m2.1.1.3.cmml" xref="S2.SS2.p9.6.m2.1.1.3"><ci id="S2.SS2.p9.6.m2.1.1.3.1.cmml" xref="S2.SS2.p9.6.m2.1.1.3.1">~</ci><ci id="S2.SS2.p9.6.m2.1.1.3.2.cmml" xref="S2.SS2.p9.6.m2.1.1.3.2">𝑝</ci></apply><apply id="S2.SS2.p9.6.m2.1.1.1.1.1.cmml" xref="S2.SS2.p9.6.m2.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p9.6.m2.1.1.1.1.1.1.cmml" xref="S2.SS2.p9.6.m2.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p9.6.m2.1.1.1.1.1.2.cmml" xref="S2.SS2.p9.6.m2.1.1.1.1.1.2">𝑞</ci><ci id="S2.SS2.p9.6.m2.1.1.1.1.1.3.cmml" xref="S2.SS2.p9.6.m2.1.1.1.1.1.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p9.6.m2.1c">\tilde{p}(q_{i})</annotation></semantics></math> is greater than <math id="S2.SS2.p9.7.m3.1" class="ltx_Math" alttext="0.5" display="inline"><semantics id="S2.SS2.p9.7.m3.1a"><mn id="S2.SS2.p9.7.m3.1.1" xref="S2.SS2.p9.7.m3.1.1.cmml">0.5</mn><annotation-xml encoding="MathML-Content" id="S2.SS2.p9.7.m3.1b"><cn type="float" id="S2.SS2.p9.7.m3.1.1.cmml" xref="S2.SS2.p9.7.m3.1.1">0.5</cn></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p9.7.m3.1c">0.5</annotation></semantics></math>.
Technically, we also need to determine whether <math id="S2.SS2.p9.8.m4.1" class="ltx_Math" alttext="\tilde{p}(q_{i})&gt;0.5" display="inline"><semantics id="S2.SS2.p9.8.m4.1a"><mrow id="S2.SS2.p9.8.m4.1.1" xref="S2.SS2.p9.8.m4.1.1.cmml"><mrow id="S2.SS2.p9.8.m4.1.1.1" xref="S2.SS2.p9.8.m4.1.1.1.cmml"><mover accent="true" id="S2.SS2.p9.8.m4.1.1.1.3" xref="S2.SS2.p9.8.m4.1.1.1.3.cmml"><mi id="S2.SS2.p9.8.m4.1.1.1.3.2" xref="S2.SS2.p9.8.m4.1.1.1.3.2.cmml">p</mi><mo id="S2.SS2.p9.8.m4.1.1.1.3.1" xref="S2.SS2.p9.8.m4.1.1.1.3.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S2.SS2.p9.8.m4.1.1.1.2" xref="S2.SS2.p9.8.m4.1.1.1.2.cmml">​</mo><mrow id="S2.SS2.p9.8.m4.1.1.1.1.1" xref="S2.SS2.p9.8.m4.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS2.p9.8.m4.1.1.1.1.1.2" xref="S2.SS2.p9.8.m4.1.1.1.1.1.1.cmml">(</mo><msub id="S2.SS2.p9.8.m4.1.1.1.1.1.1" xref="S2.SS2.p9.8.m4.1.1.1.1.1.1.cmml"><mi id="S2.SS2.p9.8.m4.1.1.1.1.1.1.2" xref="S2.SS2.p9.8.m4.1.1.1.1.1.1.2.cmml">q</mi><mi id="S2.SS2.p9.8.m4.1.1.1.1.1.1.3" xref="S2.SS2.p9.8.m4.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="S2.SS2.p9.8.m4.1.1.1.1.1.3" xref="S2.SS2.p9.8.m4.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S2.SS2.p9.8.m4.1.1.2" xref="S2.SS2.p9.8.m4.1.1.2.cmml">&gt;</mo><mn id="S2.SS2.p9.8.m4.1.1.3" xref="S2.SS2.p9.8.m4.1.1.3.cmml">0.5</mn></mrow><annotation-xml encoding="MathML-Content" id="S2.SS2.p9.8.m4.1b"><apply id="S2.SS2.p9.8.m4.1.1.cmml" xref="S2.SS2.p9.8.m4.1.1"><gt id="S2.SS2.p9.8.m4.1.1.2.cmml" xref="S2.SS2.p9.8.m4.1.1.2"></gt><apply id="S2.SS2.p9.8.m4.1.1.1.cmml" xref="S2.SS2.p9.8.m4.1.1.1"><times id="S2.SS2.p9.8.m4.1.1.1.2.cmml" xref="S2.SS2.p9.8.m4.1.1.1.2"></times><apply id="S2.SS2.p9.8.m4.1.1.1.3.cmml" xref="S2.SS2.p9.8.m4.1.1.1.3"><ci id="S2.SS2.p9.8.m4.1.1.1.3.1.cmml" xref="S2.SS2.p9.8.m4.1.1.1.3.1">~</ci><ci id="S2.SS2.p9.8.m4.1.1.1.3.2.cmml" xref="S2.SS2.p9.8.m4.1.1.1.3.2">𝑝</ci></apply><apply id="S2.SS2.p9.8.m4.1.1.1.1.1.1.cmml" xref="S2.SS2.p9.8.m4.1.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS2.p9.8.m4.1.1.1.1.1.1.1.cmml" xref="S2.SS2.p9.8.m4.1.1.1.1.1">subscript</csymbol><ci id="S2.SS2.p9.8.m4.1.1.1.1.1.1.2.cmml" xref="S2.SS2.p9.8.m4.1.1.1.1.1.1.2">𝑞</ci><ci id="S2.SS2.p9.8.m4.1.1.1.1.1.1.3.cmml" xref="S2.SS2.p9.8.m4.1.1.1.1.1.1.3">𝑖</ci></apply></apply><cn type="float" id="S2.SS2.p9.8.m4.1.1.3.cmml" xref="S2.SS2.p9.8.m4.1.1.3">0.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p9.8.m4.1c">\tilde{p}(q_{i})&gt;0.5</annotation></semantics></math> corresponds to “Yes” or “No,” as this isn’t specified by <math id="S2.SS2.p9.9.m5.1" class="ltx_Math" alttext="L_{\text{CCS}}" display="inline"><semantics id="S2.SS2.p9.9.m5.1a"><msub id="S2.SS2.p9.9.m5.1.1" xref="S2.SS2.p9.9.m5.1.1.cmml"><mi id="S2.SS2.p9.9.m5.1.1.2" xref="S2.SS2.p9.9.m5.1.1.2.cmml">L</mi><mtext id="S2.SS2.p9.9.m5.1.1.3" xref="S2.SS2.p9.9.m5.1.1.3a.cmml">CCS</mtext></msub><annotation-xml encoding="MathML-Content" id="S2.SS2.p9.9.m5.1b"><apply id="S2.SS2.p9.9.m5.1.1.cmml" xref="S2.SS2.p9.9.m5.1.1"><csymbol cd="ambiguous" id="S2.SS2.p9.9.m5.1.1.1.cmml" xref="S2.SS2.p9.9.m5.1.1">subscript</csymbol><ci id="S2.SS2.p9.9.m5.1.1.2.cmml" xref="S2.SS2.p9.9.m5.1.1.2">𝐿</ci><ci id="S2.SS2.p9.9.m5.1.1.3a.cmml" xref="S2.SS2.p9.9.m5.1.1.3"><mtext mathsize="70%" id="S2.SS2.p9.9.m5.1.1.3.cmml" xref="S2.SS2.p9.9.m5.1.1.3">CCS</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS2.p9.9.m5.1c">L_{\text{CCS}}</annotation></semantics></math>.
For simplicity in our evaluations we take the maximum accuracy over the two possible ways of labeling the predictions of a given test set.
However, in <a href="#A1" title="Appendix A Identifying Clusters ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">A</span></a> we describe how one can identify the two clusters without any supervision in principle by leveraging conjunctions.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>Results</h2>

<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1 </span>Experimental Setup</h3>

<div id="S3.SS1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS1.p1.1" class="ltx_p">Here we give an overview of our experimental setup; see <a href="#A7" title="Appendix G CCS and CRC Implementation Details ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">G</span></a> for full details.
We provide code at <a target="_blank" href="https://www.github.com/collin-burns/discovering_latent_knowledge" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.github.com/collin-burns/discovering_latent_knowledge</a>.</p>
</div>
<div id="S3.SS1.p2" class="ltx_para ltx_noindent">
<p id="S3.SS1.p2.1" class="ltx_p"><span id="S3.SS1.p2.1.1" class="ltx_text ltx_font_bold">Models.</span> We test six models: encoder-decoder models (T5 <cite class="ltx_cite ltx_citemacro_citep">(Raffel et al., <a href="#bib.bib40" title="" class="ltx_ref">2020</a>)</cite>, UnifiedQA <cite class="ltx_cite ltx_citemacro_citep">(Khashabi et al., <a href="#bib.bib20" title="" class="ltx_ref">2020</a>)</cite>, T0 <cite class="ltx_cite ltx_citemacro_citep">(Sanh et al., <a href="#bib.bib44" title="" class="ltx_ref">2021</a>)</cite>), autoregressive models (GPT-J <cite class="ltx_cite ltx_citemacro_citep">(Wang &amp; Komatsuzaki, <a href="#bib.bib48" title="" class="ltx_ref">2021</a>)</cite>), and encoder-only models (RoBERTa <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib27" title="" class="ltx_ref">2019</a>)</cite>, DeBERTa <cite class="ltx_cite ltx_citemacro_citep">(He et al., <a href="#bib.bib15" title="" class="ltx_ref">2021</a>)</cite>).</p>
</div>
<div id="S3.SS1.p3" class="ltx_para ltx_noindent">
<p id="S3.SS1.p3.1" class="ltx_p"><span id="S3.SS1.p3.1.1" class="ltx_text ltx_font_bold">Data.</span> We test models on <math id="S3.SS1.p3.1.m1.1" class="ltx_Math" alttext="10" display="inline"><semantics id="S3.SS1.p3.1.m1.1a"><mn id="S3.SS1.p3.1.m1.1.1" xref="S3.SS1.p3.1.m1.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="S3.SS1.p3.1.m1.1b"><cn type="integer" id="S3.SS1.p3.1.m1.1.1.cmml" xref="S3.SS1.p3.1.m1.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p3.1.m1.1c">10</annotation></semantics></math> datasets: sentiment classification (IMDB <cite class="ltx_cite ltx_citemacro_citep">(Maas et al., <a href="#bib.bib30" title="" class="ltx_ref">2011</a>)</cite> and Amazon <cite class="ltx_cite ltx_citemacro_citep">(McAuley &amp; Leskovec, <a href="#bib.bib32" title="" class="ltx_ref">2013</a>)</cite>), topic classification (AG-News <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib54" title="" class="ltx_ref">2015</a>)</cite> and DBpedia-14 <cite class="ltx_cite ltx_citemacro_citep">(Lehmann et al., <a href="#bib.bib23" title="" class="ltx_ref">2015</a>)</cite>), NLI (RTE <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib47" title="" class="ltx_ref">2018</a>)</cite> and QNLI <cite class="ltx_cite ltx_citemacro_citep">(Rajpurkar et al., <a href="#bib.bib41" title="" class="ltx_ref">2016</a>)</cite>), story completion (COPA <cite class="ltx_cite ltx_citemacro_citep">(Roemmele et al., <a href="#bib.bib42" title="" class="ltx_ref">2011</a>)</cite> and Story-Cloze <cite class="ltx_cite ltx_citemacro_citep">(Mostafazadeh et al., <a href="#bib.bib36" title="" class="ltx_ref">2017</a>)</cite>), question answering (BoolQ <cite class="ltx_cite ltx_citemacro_citep">(Clark et al., <a href="#bib.bib11" title="" class="ltx_ref">2019</a>)</cite>), and common sense reasoning (PIQA <cite class="ltx_cite ltx_citemacro_citep">(Bisk et al., <a href="#bib.bib5" title="" class="ltx_ref">2020</a>)</cite>).</p>
</div>
<div id="S3.SS1.p4" class="ltx_para ltx_noindent">
<p id="S3.SS1.p4.2" class="ltx_p">We convert each dataset to a yes-no question-answering task or a binary classification task, as described in <a href="#A7" title="Appendix G CCS and CRC Implementation Details ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">G</span></a>.
We balance the labels and randomly subsample 1000 examples from each dataset (except for COPA, which has only 500 examples total), then randomly split each dataset into an unsupervised training set (<math id="S3.SS1.p4.1.m1.1" class="ltx_Math" alttext="60\%" display="inline"><semantics id="S3.SS1.p4.1.m1.1a"><mrow id="S3.SS1.p4.1.m1.1.1" xref="S3.SS1.p4.1.m1.1.1.cmml"><mn id="S3.SS1.p4.1.m1.1.1.2" xref="S3.SS1.p4.1.m1.1.1.2.cmml">60</mn><mo id="S3.SS1.p4.1.m1.1.1.1" xref="S3.SS1.p4.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.1.m1.1b"><apply id="S3.SS1.p4.1.m1.1.1.cmml" xref="S3.SS1.p4.1.m1.1.1"><csymbol cd="latexml" id="S3.SS1.p4.1.m1.1.1.1.cmml" xref="S3.SS1.p4.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S3.SS1.p4.1.m1.1.1.2.cmml" xref="S3.SS1.p4.1.m1.1.1.2">60</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.1.m1.1c">60\%</annotation></semantics></math> of the data) and test set (<math id="S3.SS1.p4.2.m2.1" class="ltx_Math" alttext="40\%" display="inline"><semantics id="S3.SS1.p4.2.m2.1a"><mrow id="S3.SS1.p4.2.m2.1.1" xref="S3.SS1.p4.2.m2.1.1.cmml"><mn id="S3.SS1.p4.2.m2.1.1.2" xref="S3.SS1.p4.2.m2.1.1.2.cmml">40</mn><mo id="S3.SS1.p4.2.m2.1.1.1" xref="S3.SS1.p4.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p4.2.m2.1b"><apply id="S3.SS1.p4.2.m2.1.1.cmml" xref="S3.SS1.p4.2.m2.1.1"><csymbol cd="latexml" id="S3.SS1.p4.2.m2.1.1.1.cmml" xref="S3.SS1.p4.2.m2.1.1.1">percent</csymbol><cn type="integer" id="S3.SS1.p4.2.m2.1.1.2.cmml" xref="S3.SS1.p4.2.m2.1.1.2">40</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p4.2.m2.1c">40\%</annotation></semantics></math>).
We subsample each dataset for computational efficiency reasons; because we aggregate over  9 prompts per dataset, 10 datasets, and 6 models, 1000 datapoints per dataset actually corresponds to approximately 180k examples in total.</p>
</div>
<div id="S3.SS1.p5" class="ltx_para ltx_noindent">
<p id="S3.SS1.p5.2" class="ltx_p"><span id="S3.SS1.p5.2.1" class="ltx_text ltx_font_bold">Methods.</span> We test four main methods: zero-shot, calibrated zero-shot, Contrast-Consistent Search (CCS), and Logistic Regression (LR).
Zero-shot works by predicting the answer with the highest log probability according to the language model, averaged across the tokens that make up that label.
Calibrated zero-shot works by balancing zero-shot predictions to be <math id="S3.SS1.p5.1.m1.1" class="ltx_Math" alttext="50/50" display="inline"><semantics id="S3.SS1.p5.1.m1.1a"><mrow id="S3.SS1.p5.1.m1.1.1" xref="S3.SS1.p5.1.m1.1.1.cmml"><mn id="S3.SS1.p5.1.m1.1.1.2" xref="S3.SS1.p5.1.m1.1.1.2.cmml">50</mn><mo id="S3.SS1.p5.1.m1.1.1.1" xref="S3.SS1.p5.1.m1.1.1.1.cmml">/</mo><mn id="S3.SS1.p5.1.m1.1.1.3" xref="S3.SS1.p5.1.m1.1.1.3.cmml">50</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p5.1.m1.1b"><apply id="S3.SS1.p5.1.m1.1.1.cmml" xref="S3.SS1.p5.1.m1.1.1"><divide id="S3.SS1.p5.1.m1.1.1.1.cmml" xref="S3.SS1.p5.1.m1.1.1.1"></divide><cn type="integer" id="S3.SS1.p5.1.m1.1.1.2.cmml" xref="S3.SS1.p5.1.m1.1.1.2">50</cn><cn type="integer" id="S3.SS1.p5.1.m1.1.1.3.cmml" xref="S3.SS1.p5.1.m1.1.1.3">50</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p5.1.m1.1c">50/50</annotation></semantics></math> for each answer, as we describe in more detail below, similar to <cite class="ltx_cite ltx_citemacro_citet">Zhao et al. (<a href="#bib.bib55" title="" class="ltx_ref">2021</a>)</cite>.
For Logistic Regression we train on the training split for each dataset using <math id="S3.SS1.p5.2.m2.2" class="ltx_Math" alttext="(\tilde{\phi}(x^{+}),\tilde{\phi}(x^{-}))" display="inline"><semantics id="S3.SS1.p5.2.m2.2a"><mrow id="S3.SS1.p5.2.m2.2.2.2" xref="S3.SS1.p5.2.m2.2.2.3.cmml"><mo stretchy="false" id="S3.SS1.p5.2.m2.2.2.2.3" xref="S3.SS1.p5.2.m2.2.2.3.cmml">(</mo><mrow id="S3.SS1.p5.2.m2.1.1.1.1" xref="S3.SS1.p5.2.m2.1.1.1.1.cmml"><mover accent="true" id="S3.SS1.p5.2.m2.1.1.1.1.3" xref="S3.SS1.p5.2.m2.1.1.1.1.3.cmml"><mi id="S3.SS1.p5.2.m2.1.1.1.1.3.2" xref="S3.SS1.p5.2.m2.1.1.1.1.3.2.cmml">ϕ</mi><mo id="S3.SS1.p5.2.m2.1.1.1.1.3.1" xref="S3.SS1.p5.2.m2.1.1.1.1.3.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S3.SS1.p5.2.m2.1.1.1.1.2" xref="S3.SS1.p5.2.m2.1.1.1.1.2.cmml">​</mo><mrow id="S3.SS1.p5.2.m2.1.1.1.1.1.1" xref="S3.SS1.p5.2.m2.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p5.2.m2.1.1.1.1.1.1.2" xref="S3.SS1.p5.2.m2.1.1.1.1.1.1.1.cmml">(</mo><msup id="S3.SS1.p5.2.m2.1.1.1.1.1.1.1" xref="S3.SS1.p5.2.m2.1.1.1.1.1.1.1.cmml"><mi id="S3.SS1.p5.2.m2.1.1.1.1.1.1.1.2" xref="S3.SS1.p5.2.m2.1.1.1.1.1.1.1.2.cmml">x</mi><mo id="S3.SS1.p5.2.m2.1.1.1.1.1.1.1.3" xref="S3.SS1.p5.2.m2.1.1.1.1.1.1.1.3.cmml">+</mo></msup><mo stretchy="false" id="S3.SS1.p5.2.m2.1.1.1.1.1.1.3" xref="S3.SS1.p5.2.m2.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS1.p5.2.m2.2.2.2.4" xref="S3.SS1.p5.2.m2.2.2.3.cmml">,</mo><mrow id="S3.SS1.p5.2.m2.2.2.2.2" xref="S3.SS1.p5.2.m2.2.2.2.2.cmml"><mover accent="true" id="S3.SS1.p5.2.m2.2.2.2.2.3" xref="S3.SS1.p5.2.m2.2.2.2.2.3.cmml"><mi id="S3.SS1.p5.2.m2.2.2.2.2.3.2" xref="S3.SS1.p5.2.m2.2.2.2.2.3.2.cmml">ϕ</mi><mo id="S3.SS1.p5.2.m2.2.2.2.2.3.1" xref="S3.SS1.p5.2.m2.2.2.2.2.3.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S3.SS1.p5.2.m2.2.2.2.2.2" xref="S3.SS1.p5.2.m2.2.2.2.2.2.cmml">​</mo><mrow id="S3.SS1.p5.2.m2.2.2.2.2.1.1" xref="S3.SS1.p5.2.m2.2.2.2.2.1.1.1.cmml"><mo stretchy="false" id="S3.SS1.p5.2.m2.2.2.2.2.1.1.2" xref="S3.SS1.p5.2.m2.2.2.2.2.1.1.1.cmml">(</mo><msup id="S3.SS1.p5.2.m2.2.2.2.2.1.1.1" xref="S3.SS1.p5.2.m2.2.2.2.2.1.1.1.cmml"><mi id="S3.SS1.p5.2.m2.2.2.2.2.1.1.1.2" xref="S3.SS1.p5.2.m2.2.2.2.2.1.1.1.2.cmml">x</mi><mo id="S3.SS1.p5.2.m2.2.2.2.2.1.1.1.3" xref="S3.SS1.p5.2.m2.2.2.2.2.1.1.1.3.cmml">−</mo></msup><mo stretchy="false" id="S3.SS1.p5.2.m2.2.2.2.2.1.1.3" xref="S3.SS1.p5.2.m2.2.2.2.2.1.1.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S3.SS1.p5.2.m2.2.2.2.5" xref="S3.SS1.p5.2.m2.2.2.3.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p5.2.m2.2b"><interval closure="open" id="S3.SS1.p5.2.m2.2.2.3.cmml" xref="S3.SS1.p5.2.m2.2.2.2"><apply id="S3.SS1.p5.2.m2.1.1.1.1.cmml" xref="S3.SS1.p5.2.m2.1.1.1.1"><times id="S3.SS1.p5.2.m2.1.1.1.1.2.cmml" xref="S3.SS1.p5.2.m2.1.1.1.1.2"></times><apply id="S3.SS1.p5.2.m2.1.1.1.1.3.cmml" xref="S3.SS1.p5.2.m2.1.1.1.1.3"><ci id="S3.SS1.p5.2.m2.1.1.1.1.3.1.cmml" xref="S3.SS1.p5.2.m2.1.1.1.1.3.1">~</ci><ci id="S3.SS1.p5.2.m2.1.1.1.1.3.2.cmml" xref="S3.SS1.p5.2.m2.1.1.1.1.3.2">italic-ϕ</ci></apply><apply id="S3.SS1.p5.2.m2.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p5.2.m2.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p5.2.m2.1.1.1.1.1.1.1.1.cmml" xref="S3.SS1.p5.2.m2.1.1.1.1.1.1">superscript</csymbol><ci id="S3.SS1.p5.2.m2.1.1.1.1.1.1.1.2.cmml" xref="S3.SS1.p5.2.m2.1.1.1.1.1.1.1.2">𝑥</ci><plus id="S3.SS1.p5.2.m2.1.1.1.1.1.1.1.3.cmml" xref="S3.SS1.p5.2.m2.1.1.1.1.1.1.1.3"></plus></apply></apply><apply id="S3.SS1.p5.2.m2.2.2.2.2.cmml" xref="S3.SS1.p5.2.m2.2.2.2.2"><times id="S3.SS1.p5.2.m2.2.2.2.2.2.cmml" xref="S3.SS1.p5.2.m2.2.2.2.2.2"></times><apply id="S3.SS1.p5.2.m2.2.2.2.2.3.cmml" xref="S3.SS1.p5.2.m2.2.2.2.2.3"><ci id="S3.SS1.p5.2.m2.2.2.2.2.3.1.cmml" xref="S3.SS1.p5.2.m2.2.2.2.2.3.1">~</ci><ci id="S3.SS1.p5.2.m2.2.2.2.2.3.2.cmml" xref="S3.SS1.p5.2.m2.2.2.2.2.3.2">italic-ϕ</ci></apply><apply id="S3.SS1.p5.2.m2.2.2.2.2.1.1.1.cmml" xref="S3.SS1.p5.2.m2.2.2.2.2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p5.2.m2.2.2.2.2.1.1.1.1.cmml" xref="S3.SS1.p5.2.m2.2.2.2.2.1.1">superscript</csymbol><ci id="S3.SS1.p5.2.m2.2.2.2.2.1.1.1.2.cmml" xref="S3.SS1.p5.2.m2.2.2.2.2.1.1.1.2">𝑥</ci><minus id="S3.SS1.p5.2.m2.2.2.2.2.1.1.1.3.cmml" xref="S3.SS1.p5.2.m2.2.2.2.2.1.1.1.3"></minus></apply></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p5.2.m2.2c">(\tilde{\phi}(x^{+}),\tilde{\phi}(x^{-}))</annotation></semantics></math> as the covariates, then evaluate on the corresponding test split.
We treat LR as a ceiling since it uses labeled data.</p>
</div>
<div id="S3.SS1.p6" class="ltx_para ltx_noindent">
<p id="S3.SS1.p6.1" class="ltx_p">When testing CCS, we optimize it 10 times using AdamW <cite class="ltx_cite ltx_citemacro_citep">(Loshchilov &amp; Hutter, <a href="#bib.bib28" title="" class="ltx_ref">2017</a>)</cite> with learning rate <math id="S3.SS1.p6.1.m1.1" class="ltx_Math" alttext="0.01" display="inline"><semantics id="S3.SS1.p6.1.m1.1a"><mn id="S3.SS1.p6.1.m1.1.1" xref="S3.SS1.p6.1.m1.1.1.cmml">0.01</mn><annotation-xml encoding="MathML-Content" id="S3.SS1.p6.1.m1.1b"><cn type="float" id="S3.SS1.p6.1.m1.1.1.cmml" xref="S3.SS1.p6.1.m1.1.1">0.01</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p6.1.m1.1c">0.01</annotation></semantics></math>, then take the run with the lowest unsupervised loss. Unless otherwise specified, we train CCS using all prompts for a single training set, then evaluate it on the corresponding test split.</p>
</div>
<div id="S3.SS1.p7" class="ltx_para ltx_noindent">
<p id="S3.SS1.p7.6" class="ltx_p"><span id="S3.SS1.p7.6.1" class="ltx_text ltx_font_bold">Zero-shot baselines.</span>
Zero-shot outputs sometimes suffer from miscalibration <cite class="ltx_cite ltx_citemacro_citep">(Zhao et al., <a href="#bib.bib55" title="" class="ltx_ref">2021</a>)</cite>, in which models are biased towards predicting specific answers.
Calibrating the outputs to be uniform over different answers can mitigate this problem.
We use a variant of the calibration method presented in <cite class="ltx_cite ltx_citemacro_citet">Zhao et al. (<a href="#bib.bib55" title="" class="ltx_ref">2021</a>)</cite> by balancing predictions to be 50/50 across the two output labels. Specifically, if <math id="S3.SS1.p7.1.m1.1" class="ltx_Math" alttext="l_{+}" display="inline"><semantics id="S3.SS1.p7.1.m1.1a"><msub id="S3.SS1.p7.1.m1.1.1" xref="S3.SS1.p7.1.m1.1.1.cmml"><mi id="S3.SS1.p7.1.m1.1.1.2" xref="S3.SS1.p7.1.m1.1.1.2.cmml">l</mi><mo id="S3.SS1.p7.1.m1.1.1.3" xref="S3.SS1.p7.1.m1.1.1.3.cmml">+</mo></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p7.1.m1.1b"><apply id="S3.SS1.p7.1.m1.1.1.cmml" xref="S3.SS1.p7.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS1.p7.1.m1.1.1.1.cmml" xref="S3.SS1.p7.1.m1.1.1">subscript</csymbol><ci id="S3.SS1.p7.1.m1.1.1.2.cmml" xref="S3.SS1.p7.1.m1.1.1.2">𝑙</ci><plus id="S3.SS1.p7.1.m1.1.1.3.cmml" xref="S3.SS1.p7.1.m1.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p7.1.m1.1c">l_{+}</annotation></semantics></math> and <math id="S3.SS1.p7.2.m2.1" class="ltx_Math" alttext="l_{-}" display="inline"><semantics id="S3.SS1.p7.2.m2.1a"><msub id="S3.SS1.p7.2.m2.1.1" xref="S3.SS1.p7.2.m2.1.1.cmml"><mi id="S3.SS1.p7.2.m2.1.1.2" xref="S3.SS1.p7.2.m2.1.1.2.cmml">l</mi><mo id="S3.SS1.p7.2.m2.1.1.3" xref="S3.SS1.p7.2.m2.1.1.3.cmml">−</mo></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p7.2.m2.1b"><apply id="S3.SS1.p7.2.m2.1.1.cmml" xref="S3.SS1.p7.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS1.p7.2.m2.1.1.1.cmml" xref="S3.SS1.p7.2.m2.1.1">subscript</csymbol><ci id="S3.SS1.p7.2.m2.1.1.2.cmml" xref="S3.SS1.p7.2.m2.1.1.2">𝑙</ci><minus id="S3.SS1.p7.2.m2.1.1.3.cmml" xref="S3.SS1.p7.2.m2.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p7.2.m2.1c">l_{-}</annotation></semantics></math> are the logits for the positive and negative label respectively, then instead of classifying an example as positive if <math id="S3.SS1.p7.3.m3.1" class="ltx_Math" alttext="l_{+}&gt;l_{-}" display="inline"><semantics id="S3.SS1.p7.3.m3.1a"><mrow id="S3.SS1.p7.3.m3.1.1" xref="S3.SS1.p7.3.m3.1.1.cmml"><msub id="S3.SS1.p7.3.m3.1.1.2" xref="S3.SS1.p7.3.m3.1.1.2.cmml"><mi id="S3.SS1.p7.3.m3.1.1.2.2" xref="S3.SS1.p7.3.m3.1.1.2.2.cmml">l</mi><mo id="S3.SS1.p7.3.m3.1.1.2.3" xref="S3.SS1.p7.3.m3.1.1.2.3.cmml">+</mo></msub><mo id="S3.SS1.p7.3.m3.1.1.1" xref="S3.SS1.p7.3.m3.1.1.1.cmml">&gt;</mo><msub id="S3.SS1.p7.3.m3.1.1.3" xref="S3.SS1.p7.3.m3.1.1.3.cmml"><mi id="S3.SS1.p7.3.m3.1.1.3.2" xref="S3.SS1.p7.3.m3.1.1.3.2.cmml">l</mi><mo id="S3.SS1.p7.3.m3.1.1.3.3" xref="S3.SS1.p7.3.m3.1.1.3.3.cmml">−</mo></msub></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p7.3.m3.1b"><apply id="S3.SS1.p7.3.m3.1.1.cmml" xref="S3.SS1.p7.3.m3.1.1"><gt id="S3.SS1.p7.3.m3.1.1.1.cmml" xref="S3.SS1.p7.3.m3.1.1.1"></gt><apply id="S3.SS1.p7.3.m3.1.1.2.cmml" xref="S3.SS1.p7.3.m3.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p7.3.m3.1.1.2.1.cmml" xref="S3.SS1.p7.3.m3.1.1.2">subscript</csymbol><ci id="S3.SS1.p7.3.m3.1.1.2.2.cmml" xref="S3.SS1.p7.3.m3.1.1.2.2">𝑙</ci><plus id="S3.SS1.p7.3.m3.1.1.2.3.cmml" xref="S3.SS1.p7.3.m3.1.1.2.3"></plus></apply><apply id="S3.SS1.p7.3.m3.1.1.3.cmml" xref="S3.SS1.p7.3.m3.1.1.3"><csymbol cd="ambiguous" id="S3.SS1.p7.3.m3.1.1.3.1.cmml" xref="S3.SS1.p7.3.m3.1.1.3">subscript</csymbol><ci id="S3.SS1.p7.3.m3.1.1.3.2.cmml" xref="S3.SS1.p7.3.m3.1.1.3.2">𝑙</ci><minus id="S3.SS1.p7.3.m3.1.1.3.3.cmml" xref="S3.SS1.p7.3.m3.1.1.3.3"></minus></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p7.3.m3.1c">l_{+}&gt;l_{-}</annotation></semantics></math>, we classify it as positive if <math id="S3.SS1.p7.4.m4.1" class="ltx_Math" alttext="l_{+}&gt;l_{-}+\gamma" display="inline"><semantics id="S3.SS1.p7.4.m4.1a"><mrow id="S3.SS1.p7.4.m4.1.1" xref="S3.SS1.p7.4.m4.1.1.cmml"><msub id="S3.SS1.p7.4.m4.1.1.2" xref="S3.SS1.p7.4.m4.1.1.2.cmml"><mi id="S3.SS1.p7.4.m4.1.1.2.2" xref="S3.SS1.p7.4.m4.1.1.2.2.cmml">l</mi><mo id="S3.SS1.p7.4.m4.1.1.2.3" xref="S3.SS1.p7.4.m4.1.1.2.3.cmml">+</mo></msub><mo id="S3.SS1.p7.4.m4.1.1.1" xref="S3.SS1.p7.4.m4.1.1.1.cmml">&gt;</mo><mrow id="S3.SS1.p7.4.m4.1.1.3" xref="S3.SS1.p7.4.m4.1.1.3.cmml"><msub id="S3.SS1.p7.4.m4.1.1.3.2" xref="S3.SS1.p7.4.m4.1.1.3.2.cmml"><mi id="S3.SS1.p7.4.m4.1.1.3.2.2" xref="S3.SS1.p7.4.m4.1.1.3.2.2.cmml">l</mi><mo id="S3.SS1.p7.4.m4.1.1.3.2.3" xref="S3.SS1.p7.4.m4.1.1.3.2.3.cmml">−</mo></msub><mo id="S3.SS1.p7.4.m4.1.1.3.1" xref="S3.SS1.p7.4.m4.1.1.3.1.cmml">+</mo><mi id="S3.SS1.p7.4.m4.1.1.3.3" xref="S3.SS1.p7.4.m4.1.1.3.3.cmml">γ</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p7.4.m4.1b"><apply id="S3.SS1.p7.4.m4.1.1.cmml" xref="S3.SS1.p7.4.m4.1.1"><gt id="S3.SS1.p7.4.m4.1.1.1.cmml" xref="S3.SS1.p7.4.m4.1.1.1"></gt><apply id="S3.SS1.p7.4.m4.1.1.2.cmml" xref="S3.SS1.p7.4.m4.1.1.2"><csymbol cd="ambiguous" id="S3.SS1.p7.4.m4.1.1.2.1.cmml" xref="S3.SS1.p7.4.m4.1.1.2">subscript</csymbol><ci id="S3.SS1.p7.4.m4.1.1.2.2.cmml" xref="S3.SS1.p7.4.m4.1.1.2.2">𝑙</ci><plus id="S3.SS1.p7.4.m4.1.1.2.3.cmml" xref="S3.SS1.p7.4.m4.1.1.2.3"></plus></apply><apply id="S3.SS1.p7.4.m4.1.1.3.cmml" xref="S3.SS1.p7.4.m4.1.1.3"><plus id="S3.SS1.p7.4.m4.1.1.3.1.cmml" xref="S3.SS1.p7.4.m4.1.1.3.1"></plus><apply id="S3.SS1.p7.4.m4.1.1.3.2.cmml" xref="S3.SS1.p7.4.m4.1.1.3.2"><csymbol cd="ambiguous" id="S3.SS1.p7.4.m4.1.1.3.2.1.cmml" xref="S3.SS1.p7.4.m4.1.1.3.2">subscript</csymbol><ci id="S3.SS1.p7.4.m4.1.1.3.2.2.cmml" xref="S3.SS1.p7.4.m4.1.1.3.2.2">𝑙</ci><minus id="S3.SS1.p7.4.m4.1.1.3.2.3.cmml" xref="S3.SS1.p7.4.m4.1.1.3.2.3"></minus></apply><ci id="S3.SS1.p7.4.m4.1.1.3.3.cmml" xref="S3.SS1.p7.4.m4.1.1.3.3">𝛾</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p7.4.m4.1c">l_{+}&gt;l_{-}+\gamma</annotation></semantics></math>, where we select the threshold <math id="S3.SS1.p7.5.m5.1" class="ltx_Math" alttext="\gamma\in\mathbb{R}" display="inline"><semantics id="S3.SS1.p7.5.m5.1a"><mrow id="S3.SS1.p7.5.m5.1.1" xref="S3.SS1.p7.5.m5.1.1.cmml"><mi id="S3.SS1.p7.5.m5.1.1.2" xref="S3.SS1.p7.5.m5.1.1.2.cmml">γ</mi><mo id="S3.SS1.p7.5.m5.1.1.1" xref="S3.SS1.p7.5.m5.1.1.1.cmml">∈</mo><mi id="S3.SS1.p7.5.m5.1.1.3" xref="S3.SS1.p7.5.m5.1.1.3.cmml">ℝ</mi></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p7.5.m5.1b"><apply id="S3.SS1.p7.5.m5.1.1.cmml" xref="S3.SS1.p7.5.m5.1.1"><in id="S3.SS1.p7.5.m5.1.1.1.cmml" xref="S3.SS1.p7.5.m5.1.1.1"></in><ci id="S3.SS1.p7.5.m5.1.1.2.cmml" xref="S3.SS1.p7.5.m5.1.1.2">𝛾</ci><ci id="S3.SS1.p7.5.m5.1.1.3.cmml" xref="S3.SS1.p7.5.m5.1.1.3">ℝ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p7.5.m5.1c">\gamma\in\mathbb{R}</annotation></semantics></math> so that the predictions are balanced.
We find this increases accuracy by about <math id="S3.SS1.p7.6.m6.1" class="ltx_Math" alttext="5\%" display="inline"><semantics id="S3.SS1.p7.6.m6.1a"><mrow id="S3.SS1.p7.6.m6.1.1" xref="S3.SS1.p7.6.m6.1.1.cmml"><mn id="S3.SS1.p7.6.m6.1.1.2" xref="S3.SS1.p7.6.m6.1.1.2.cmml">5</mn><mo id="S3.SS1.p7.6.m6.1.1.1" xref="S3.SS1.p7.6.m6.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.p7.6.m6.1b"><apply id="S3.SS1.p7.6.m6.1.1.cmml" xref="S3.SS1.p7.6.m6.1.1"><csymbol cd="latexml" id="S3.SS1.p7.6.m6.1.1.1.cmml" xref="S3.SS1.p7.6.m6.1.1.1">percent</csymbol><cn type="integer" id="S3.SS1.p7.6.m6.1.1.2.cmml" xref="S3.SS1.p7.6.m6.1.1.2">5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p7.6.m6.1c">5\%</annotation></semantics></math> on average.
Unless otherwise specified, we always report zero-shot accuracy after calibration.</p>
</div>
<div id="S3.SS1.p8" class="ltx_para ltx_noindent">
<p id="S3.SS1.p8.1" class="ltx_p">Encoder-only models (e.g. RoBERTa and DeBERTa) cannot be easily used to do zero-shot classification out of the box, so to evaluate them we follow the method of <cite class="ltx_cite ltx_citemacro_citet">Yin et al. (<a href="#bib.bib53" title="" class="ltx_ref">2020</a>)</cite>: we finetune both models on an NLI dataset (MNLI, which we do not evaluate on) and treat the difference between the entailment and contradiction probabilities as the effective logit. This provides a strong zero-shot baseline for encoder-only models that works even for non-NLI tasks <cite class="ltx_cite ltx_citemacro_citep">(Yin et al., <a href="#bib.bib53" title="" class="ltx_ref">2020</a>)</cite>. This finetuning isn’t necessary for CCS to work on encoder-only models (see <a href="#A3" title="Appendix C Masked Language Modeling Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">C</span></a>), but we test CCS using the same MNLI-finetuned models for ease of comparison.</p>
</div>
<div id="S3.SS1.p9" class="ltx_para ltx_noindent">
<p id="S3.SS1.p9.1" class="ltx_p"><span id="S3.SS1.p9.1.1" class="ltx_text ltx_font_bold">Hidden states.</span>
We extract the hidden states corresponding to the last token in the last layer of each model for simplicity, unless otherwise specified.
For encoder-decoder models, we evaluate CCS on the last layer hidden states of both the encoder and decoder, and use whichever one generally achieves a lower unsupervised loss; for T0 this is the decoder hidden states, while for T5 and UnifiedQA this is the encoder hidden states.
See <a href="#A7.SS1" title="G.1 Contrast Pairs ‣ Appendix G CCS and CRC Implementation Details ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">G.1</span></a> for further implementation details, such as tokenization.</p>
</div>
<div id="S3.SS1.p10" class="ltx_para ltx_noindent">
<p id="S3.SS1.p10.7" class="ltx_p"><span id="S3.SS1.p10.7.1" class="ltx_text ltx_font_bold">Prompts.</span> To reduce prompt sensitivity, we use between <math id="S3.SS1.p10.1.m1.1" class="ltx_Math" alttext="8" display="inline"><semantics id="S3.SS1.p10.1.m1.1a"><mn id="S3.SS1.p10.1.m1.1.1" xref="S3.SS1.p10.1.m1.1.1.cmml">8</mn><annotation-xml encoding="MathML-Content" id="S3.SS1.p10.1.m1.1b"><cn type="integer" id="S3.SS1.p10.1.m1.1.1.cmml" xref="S3.SS1.p10.1.m1.1.1">8</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p10.1.m1.1c">8</annotation></semantics></math> and <math id="S3.SS1.p10.2.m2.1" class="ltx_Math" alttext="13" display="inline"><semantics id="S3.SS1.p10.2.m2.1a"><mn id="S3.SS1.p10.2.m2.1.1" xref="S3.SS1.p10.2.m2.1.1.cmml">13</mn><annotation-xml encoding="MathML-Content" id="S3.SS1.p10.2.m2.1b"><cn type="integer" id="S3.SS1.p10.2.m2.1.1.cmml" xref="S3.SS1.p10.2.m2.1.1">13</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p10.2.m2.1c">13</annotation></semantics></math> prompts for each dataset (<math id="S3.SS1.p10.3.m3.1" class="ltx_Math" alttext="9" display="inline"><semantics id="S3.SS1.p10.3.m3.1a"><mn id="S3.SS1.p10.3.m3.1.1" xref="S3.SS1.p10.3.m3.1.1.cmml">9</mn><annotation-xml encoding="MathML-Content" id="S3.SS1.p10.3.m3.1b"><cn type="integer" id="S3.SS1.p10.3.m3.1.1.cmml" xref="S3.SS1.p10.3.m3.1.1">9</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p10.3.m3.1c">9</annotation></semantics></math> on average), derived or slightly modified from <cite class="ltx_cite ltx_citemacro_citet">Sanh et al. (<a href="#bib.bib44" title="" class="ltx_ref">2021</a>)</cite>.
Unless otherwise specified, we average across all prompts when showing results.
To construct contrast pairs, we let <math id="S3.SS1.p10.4.m4.1" class="ltx_Math" alttext="x_{i}^{+}" display="inline"><semantics id="S3.SS1.p10.4.m4.1a"><msubsup id="S3.SS1.p10.4.m4.1.1" xref="S3.SS1.p10.4.m4.1.1.cmml"><mi id="S3.SS1.p10.4.m4.1.1.2.2" xref="S3.SS1.p10.4.m4.1.1.2.2.cmml">x</mi><mi id="S3.SS1.p10.4.m4.1.1.2.3" xref="S3.SS1.p10.4.m4.1.1.2.3.cmml">i</mi><mo id="S3.SS1.p10.4.m4.1.1.3" xref="S3.SS1.p10.4.m4.1.1.3.cmml">+</mo></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS1.p10.4.m4.1b"><apply id="S3.SS1.p10.4.m4.1.1.cmml" xref="S3.SS1.p10.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS1.p10.4.m4.1.1.1.cmml" xref="S3.SS1.p10.4.m4.1.1">superscript</csymbol><apply id="S3.SS1.p10.4.m4.1.1.2.cmml" xref="S3.SS1.p10.4.m4.1.1"><csymbol cd="ambiguous" id="S3.SS1.p10.4.m4.1.1.2.1.cmml" xref="S3.SS1.p10.4.m4.1.1">subscript</csymbol><ci id="S3.SS1.p10.4.m4.1.1.2.2.cmml" xref="S3.SS1.p10.4.m4.1.1.2.2">𝑥</ci><ci id="S3.SS1.p10.4.m4.1.1.2.3.cmml" xref="S3.SS1.p10.4.m4.1.1.2.3">𝑖</ci></apply><plus id="S3.SS1.p10.4.m4.1.1.3.cmml" xref="S3.SS1.p10.4.m4.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p10.4.m4.1c">x_{i}^{+}</annotation></semantics></math> be the zero-shot prompt using <math id="S3.SS1.p10.5.m5.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S3.SS1.p10.5.m5.1a"><msub id="S3.SS1.p10.5.m5.1.1" xref="S3.SS1.p10.5.m5.1.1.cmml"><mi id="S3.SS1.p10.5.m5.1.1.2" xref="S3.SS1.p10.5.m5.1.1.2.cmml">q</mi><mi id="S3.SS1.p10.5.m5.1.1.3" xref="S3.SS1.p10.5.m5.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p10.5.m5.1b"><apply id="S3.SS1.p10.5.m5.1.1.cmml" xref="S3.SS1.p10.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS1.p10.5.m5.1.1.1.cmml" xref="S3.SS1.p10.5.m5.1.1">subscript</csymbol><ci id="S3.SS1.p10.5.m5.1.1.2.cmml" xref="S3.SS1.p10.5.m5.1.1.2">𝑞</ci><ci id="S3.SS1.p10.5.m5.1.1.3.cmml" xref="S3.SS1.p10.5.m5.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p10.5.m5.1c">q_{i}</annotation></semantics></math> and the first label (e.g. “Positive” for sentiment classification datasets) and let <math id="S3.SS1.p10.6.m6.1" class="ltx_Math" alttext="x_{i}^{-}" display="inline"><semantics id="S3.SS1.p10.6.m6.1a"><msubsup id="S3.SS1.p10.6.m6.1.1" xref="S3.SS1.p10.6.m6.1.1.cmml"><mi id="S3.SS1.p10.6.m6.1.1.2.2" xref="S3.SS1.p10.6.m6.1.1.2.2.cmml">x</mi><mi id="S3.SS1.p10.6.m6.1.1.2.3" xref="S3.SS1.p10.6.m6.1.1.2.3.cmml">i</mi><mo id="S3.SS1.p10.6.m6.1.1.3" xref="S3.SS1.p10.6.m6.1.1.3.cmml">−</mo></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS1.p10.6.m6.1b"><apply id="S3.SS1.p10.6.m6.1.1.cmml" xref="S3.SS1.p10.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS1.p10.6.m6.1.1.1.cmml" xref="S3.SS1.p10.6.m6.1.1">superscript</csymbol><apply id="S3.SS1.p10.6.m6.1.1.2.cmml" xref="S3.SS1.p10.6.m6.1.1"><csymbol cd="ambiguous" id="S3.SS1.p10.6.m6.1.1.2.1.cmml" xref="S3.SS1.p10.6.m6.1.1">subscript</csymbol><ci id="S3.SS1.p10.6.m6.1.1.2.2.cmml" xref="S3.SS1.p10.6.m6.1.1.2.2">𝑥</ci><ci id="S3.SS1.p10.6.m6.1.1.2.3.cmml" xref="S3.SS1.p10.6.m6.1.1.2.3">𝑖</ci></apply><minus id="S3.SS1.p10.6.m6.1.1.3.cmml" xref="S3.SS1.p10.6.m6.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p10.6.m6.1c">x_{i}^{-}</annotation></semantics></math> be the prompt using <math id="S3.SS1.p10.7.m7.1" class="ltx_Math" alttext="q_{i}" display="inline"><semantics id="S3.SS1.p10.7.m7.1a"><msub id="S3.SS1.p10.7.m7.1.1" xref="S3.SS1.p10.7.m7.1.1.cmml"><mi id="S3.SS1.p10.7.m7.1.1.2" xref="S3.SS1.p10.7.m7.1.1.2.cmml">q</mi><mi id="S3.SS1.p10.7.m7.1.1.3" xref="S3.SS1.p10.7.m7.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S3.SS1.p10.7.m7.1b"><apply id="S3.SS1.p10.7.m7.1.1.cmml" xref="S3.SS1.p10.7.m7.1.1"><csymbol cd="ambiguous" id="S3.SS1.p10.7.m7.1.1.1.cmml" xref="S3.SS1.p10.7.m7.1.1">subscript</csymbol><ci id="S3.SS1.p10.7.m7.1.1.2.cmml" xref="S3.SS1.p10.7.m7.1.1.2">𝑞</ci><ci id="S3.SS1.p10.7.m7.1.1.3.cmml" xref="S3.SS1.p10.7.m7.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.p10.7.m7.1c">q_{i}</annotation></semantics></math> and the second label (e.g. “Negative”).
We describe all prompts in <a href="#A9" title="Appendix I Dataset Setup ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">I</span></a>.</p>
</div>
<figure id="S3.T1" class="ltx_table">
<table id="S3.T1.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S3.T1.2.2" class="ltx_tr">
<td id="S3.T1.2.2.3" class="ltx_td ltx_align_left ltx_border_r">Method</td>
<td id="S3.T1.2.2.4" class="ltx_td ltx_align_center">RoBERTa</td>
<td id="S3.T1.2.2.5" class="ltx_td ltx_align_center">DeBERTa</td>
<td id="S3.T1.2.2.6" class="ltx_td ltx_align_center">GPT-J</td>
<td id="S3.T1.2.2.7" class="ltx_td ltx_align_center">T5</td>
<td id="S3.T1.2.2.8" class="ltx_td ltx_align_center">UQA</td>
<td id="S3.T1.1.1.1" class="ltx_td ltx_align_center ltx_border_r">T0<sup id="S3.T1.1.1.1.1" class="ltx_sup">∗</sup>
</td>
<td id="S3.T1.2.2.2" class="ltx_td ltx_align_center">Mean<sup id="S3.T1.2.2.2.1" class="ltx_sup">∗</sup>
</td>
</tr>
<tr id="S3.T1.2.3" class="ltx_tr">
<td id="S3.T1.2.3.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">0-shot</td>
<td id="S3.T1.2.3.2" class="ltx_td ltx_align_center ltx_border_t">60.1(5.7)</td>
<td id="S3.T1.2.3.3" class="ltx_td ltx_align_center ltx_border_t">68.6(8.2)</td>
<td id="S3.T1.2.3.4" class="ltx_td ltx_align_center ltx_border_t">53.2(5.2)</td>
<td id="S3.T1.2.3.5" class="ltx_td ltx_align_center ltx_border_t">55.4(5.7)</td>
<td id="S3.T1.2.3.6" class="ltx_td ltx_align_center ltx_border_t">76.8(9.6)</td>
<td id="S3.T1.2.3.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">87.9(4.8)</td>
<td id="S3.T1.2.3.8" class="ltx_td ltx_align_center ltx_border_t">62.8(6.9)</td>
</tr>
<tr id="S3.T1.2.4" class="ltx_tr">
<td id="S3.T1.2.4.1" class="ltx_td ltx_align_left ltx_border_r">Calibrated 0-shot</td>
<td id="S3.T1.2.4.2" class="ltx_td ltx_align_center"><span id="S3.T1.2.4.2.1" class="ltx_text ltx_font_bold">64.3(6.2)</span></td>
<td id="S3.T1.2.4.3" class="ltx_td ltx_align_center">76.3(6.0)</td>
<td id="S3.T1.2.4.4" class="ltx_td ltx_align_center">56.0(5.2)</td>
<td id="S3.T1.2.4.5" class="ltx_td ltx_align_center">58.8(6.1)</td>
<td id="S3.T1.2.4.6" class="ltx_td ltx_align_center">80.4(7.1)</td>
<td id="S3.T1.2.4.7" class="ltx_td ltx_align_center ltx_border_r">90.5(2.7)</td>
<td id="S3.T1.2.4.8" class="ltx_td ltx_align_center">67.2(6.1)</td>
</tr>
<tr id="S3.T1.2.5" class="ltx_tr">
<td id="S3.T1.2.5.1" class="ltx_td ltx_align_left ltx_border_r">CCS</td>
<td id="S3.T1.2.5.2" class="ltx_td ltx_align_center">62.1(4.1)</td>
<td id="S3.T1.2.5.3" class="ltx_td ltx_align_center"><span id="S3.T1.2.5.3.1" class="ltx_text ltx_font_bold">78.5(3.8)</span></td>
<td id="S3.T1.2.5.4" class="ltx_td ltx_align_center">61.7(2.5)</td>
<td id="S3.T1.2.5.5" class="ltx_td ltx_align_center">71.5(3.0)</td>
<td id="S3.T1.2.5.6" class="ltx_td ltx_align_center">82.1(2.7)</td>
<td id="S3.T1.2.5.7" class="ltx_td ltx_align_center ltx_border_r">77.6(3.3)</td>
<td id="S3.T1.2.5.8" class="ltx_td ltx_align_center">71.2(3.2)</td>
</tr>
<tr id="S3.T1.2.6" class="ltx_tr">
<td id="S3.T1.2.6.1" class="ltx_td ltx_align_left ltx_border_r">CCS (All Data)</td>
<td id="S3.T1.2.6.2" class="ltx_td ltx_align_center">60.1(3.7)</td>
<td id="S3.T1.2.6.3" class="ltx_td ltx_align_center">77.1(4.1)</td>
<td id="S3.T1.2.6.4" class="ltx_td ltx_align_center"><span id="S3.T1.2.6.4.1" class="ltx_text ltx_font_bold">62.1(2.3)</span></td>
<td id="S3.T1.2.6.5" class="ltx_td ltx_align_center"><span id="S3.T1.2.6.5.1" class="ltx_text ltx_font_bold">72.7(6.0)</span></td>
<td id="S3.T1.2.6.6" class="ltx_td ltx_align_center"><span id="S3.T1.2.6.6.1" class="ltx_text ltx_font_bold">84.8(2.6)</span></td>
<td id="S3.T1.2.6.7" class="ltx_td ltx_align_center ltx_border_r">84.8(3.7)</td>
<td id="S3.T1.2.6.8" class="ltx_td ltx_align_center"><span id="S3.T1.2.6.8.1" class="ltx_text ltx_font_bold">71.5(3.7)</span></td>
</tr>
<tr id="S3.T1.2.7" class="ltx_tr">
<td id="S3.T1.2.7.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">LR (Ceiling)</td>
<td id="S3.T1.2.7.2" class="ltx_td ltx_align_center ltx_border_t">79.8(2.5)</td>
<td id="S3.T1.2.7.3" class="ltx_td ltx_align_center ltx_border_t">86.1(2.2)</td>
<td id="S3.T1.2.7.4" class="ltx_td ltx_align_center ltx_border_t">78.0(2.3)</td>
<td id="S3.T1.2.7.5" class="ltx_td ltx_align_center ltx_border_t">84.6(3.1)</td>
<td id="S3.T1.2.7.6" class="ltx_td ltx_align_center ltx_border_t">89.8(1.9)</td>
<td id="S3.T1.2.7.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">90.7(2.1)</td>
<td id="S3.T1.2.7.8" class="ltx_td ltx_align_center ltx_border_t">83.7(2.4)</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>
Accuracy of each method and model averaged across all prompts and dataset, with the average standard deviation of accuracy across different prompts shown in parentheses.
For most models, CCS outperforms zero-shot accuracy and exhibits lower sensitivity to prompts, even though this was not our goal.
This shows that we can recover knowledge from language model activations without supervision, and can do so in a way that is competitive with strong baseline methods that use model outputs.
<sup id="S3.T1.6.1" class="ltx_sup">∗</sup>T0 was trained on 9 out of 10 of the datasets we evaluate on, including some of the data in our test splits, so we ignore it when averaging over models.
</figcaption>
</figure>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2 </span>Evaluating CCS</h3>

<section id="S3.SS2.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.2.1 </span>CCS Outperforms Zero-Shot</h4>

<div id="S3.SS2.SSS1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS2.SSS1.p1.1" class="ltx_p">We evaluate CCS on all 6 models and compute the average accuracy across all datasets and prompts.
T0 was trained on 9 out of 10 of the datasets we evaluate on, including some of the data in our test splits, so we ignore it when averaging over models to avoid unfair comparisons.
We display the results in <a href="#S3.T1" title="In 3.1 Experimental Setup ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">1</span></a>.
To assess prompt sensitivity, for each model and dataset we compute the standard deviation (s.d.) of accuracy across different prompts, then average the resulting standard deviations across all datasets, which we show in parentheses in <a href="#S3.T1" title="In 3.1 Experimental Setup ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">1</span></a>.
For comparison, we also include results when training CCS on all datasets simultaneously, which we refer to as CCS (All Data).</p>
</div>
<div id="S3.SS2.SSS1.p2" class="ltx_para ltx_noindent">
<p id="S3.SS2.SSS1.p2.5" class="ltx_p">CCS attains an accuracy of <math id="S3.SS2.SSS1.p2.1.m1.1" class="ltx_Math" alttext="71.2\%" display="inline"><semantics id="S3.SS2.SSS1.p2.1.m1.1a"><mrow id="S3.SS2.SSS1.p2.1.m1.1.1" xref="S3.SS2.SSS1.p2.1.m1.1.1.cmml"><mn id="S3.SS2.SSS1.p2.1.m1.1.1.2" xref="S3.SS2.SSS1.p2.1.m1.1.1.2.cmml">71.2</mn><mo id="S3.SS2.SSS1.p2.1.m1.1.1.1" xref="S3.SS2.SSS1.p2.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.p2.1.m1.1b"><apply id="S3.SS2.SSS1.p2.1.m1.1.1.cmml" xref="S3.SS2.SSS1.p2.1.m1.1.1"><csymbol cd="latexml" id="S3.SS2.SSS1.p2.1.m1.1.1.1.cmml" xref="S3.SS2.SSS1.p2.1.m1.1.1.1">percent</csymbol><cn type="float" id="S3.SS2.SSS1.p2.1.m1.1.1.2.cmml" xref="S3.SS2.SSS1.p2.1.m1.1.1.2">71.2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.p2.1.m1.1c">71.2\%</annotation></semantics></math> on average, compared to <math id="S3.SS2.SSS1.p2.2.m2.1" class="ltx_Math" alttext="67.2\%" display="inline"><semantics id="S3.SS2.SSS1.p2.2.m2.1a"><mrow id="S3.SS2.SSS1.p2.2.m2.1.1" xref="S3.SS2.SSS1.p2.2.m2.1.1.cmml"><mn id="S3.SS2.SSS1.p2.2.m2.1.1.2" xref="S3.SS2.SSS1.p2.2.m2.1.1.2.cmml">67.2</mn><mo id="S3.SS2.SSS1.p2.2.m2.1.1.1" xref="S3.SS2.SSS1.p2.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.p2.2.m2.1b"><apply id="S3.SS2.SSS1.p2.2.m2.1.1.cmml" xref="S3.SS2.SSS1.p2.2.m2.1.1"><csymbol cd="latexml" id="S3.SS2.SSS1.p2.2.m2.1.1.1.cmml" xref="S3.SS2.SSS1.p2.2.m2.1.1.1">percent</csymbol><cn type="float" id="S3.SS2.SSS1.p2.2.m2.1.1.2.cmml" xref="S3.SS2.SSS1.p2.2.m2.1.1.2">67.2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.p2.2.m2.1c">67.2\%</annotation></semantics></math> for calibrated zero-shot.
It outperforms zero-shot accuracy for every model, except for RoBERTa (where it does <math id="S3.SS2.SSS1.p2.3.m3.1" class="ltx_Math" alttext="2\%" display="inline"><semantics id="S3.SS2.SSS1.p2.3.m3.1a"><mrow id="S3.SS2.SSS1.p2.3.m3.1.1" xref="S3.SS2.SSS1.p2.3.m3.1.1.cmml"><mn id="S3.SS2.SSS1.p2.3.m3.1.1.2" xref="S3.SS2.SSS1.p2.3.m3.1.1.2.cmml">2</mn><mo id="S3.SS2.SSS1.p2.3.m3.1.1.1" xref="S3.SS2.SSS1.p2.3.m3.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.p2.3.m3.1b"><apply id="S3.SS2.SSS1.p2.3.m3.1.1.cmml" xref="S3.SS2.SSS1.p2.3.m3.1.1"><csymbol cd="latexml" id="S3.SS2.SSS1.p2.3.m3.1.1.1.cmml" xref="S3.SS2.SSS1.p2.3.m3.1.1.1">percent</csymbol><cn type="integer" id="S3.SS2.SSS1.p2.3.m3.1.1.2.cmml" xref="S3.SS2.SSS1.p2.3.m3.1.1.2">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.p2.3.m3.1c">2\%</annotation></semantics></math> worse) and T0 (for which zero-shot accuracy is inflated).
Training on all datasets improves accuracy by only an insignificant amount on average (<math id="S3.SS2.SSS1.p2.4.m4.1" class="ltx_Math" alttext="0.3\%" display="inline"><semantics id="S3.SS2.SSS1.p2.4.m4.1a"><mrow id="S3.SS2.SSS1.p2.4.m4.1.1" xref="S3.SS2.SSS1.p2.4.m4.1.1.cmml"><mn id="S3.SS2.SSS1.p2.4.m4.1.1.2" xref="S3.SS2.SSS1.p2.4.m4.1.1.2.cmml">0.3</mn><mo id="S3.SS2.SSS1.p2.4.m4.1.1.1" xref="S3.SS2.SSS1.p2.4.m4.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.p2.4.m4.1b"><apply id="S3.SS2.SSS1.p2.4.m4.1.1.cmml" xref="S3.SS2.SSS1.p2.4.m4.1.1"><csymbol cd="latexml" id="S3.SS2.SSS1.p2.4.m4.1.1.1.cmml" xref="S3.SS2.SSS1.p2.4.m4.1.1.1">percent</csymbol><cn type="float" id="S3.SS2.SSS1.p2.4.m4.1.1.2.cmml" xref="S3.SS2.SSS1.p2.4.m4.1.1.2">0.3</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.p2.4.m4.1c">0.3\%</annotation></semantics></math>), but with large gains for T0 in particular (<math id="S3.SS2.SSS1.p2.5.m5.1" class="ltx_Math" alttext="77.6\%\rightarrow 84.8\%" display="inline"><semantics id="S3.SS2.SSS1.p2.5.m5.1a"><mrow id="S3.SS2.SSS1.p2.5.m5.1.1" xref="S3.SS2.SSS1.p2.5.m5.1.1.cmml"><mrow id="S3.SS2.SSS1.p2.5.m5.1.1.2" xref="S3.SS2.SSS1.p2.5.m5.1.1.2.cmml"><mn id="S3.SS2.SSS1.p2.5.m5.1.1.2.2" xref="S3.SS2.SSS1.p2.5.m5.1.1.2.2.cmml">77.6</mn><mo id="S3.SS2.SSS1.p2.5.m5.1.1.2.1" xref="S3.SS2.SSS1.p2.5.m5.1.1.2.1.cmml">%</mo></mrow><mo stretchy="false" id="S3.SS2.SSS1.p2.5.m5.1.1.1" xref="S3.SS2.SSS1.p2.5.m5.1.1.1.cmml">→</mo><mrow id="S3.SS2.SSS1.p2.5.m5.1.1.3" xref="S3.SS2.SSS1.p2.5.m5.1.1.3.cmml"><mn id="S3.SS2.SSS1.p2.5.m5.1.1.3.2" xref="S3.SS2.SSS1.p2.5.m5.1.1.3.2.cmml">84.8</mn><mo id="S3.SS2.SSS1.p2.5.m5.1.1.3.1" xref="S3.SS2.SSS1.p2.5.m5.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS1.p2.5.m5.1b"><apply id="S3.SS2.SSS1.p2.5.m5.1.1.cmml" xref="S3.SS2.SSS1.p2.5.m5.1.1"><ci id="S3.SS2.SSS1.p2.5.m5.1.1.1.cmml" xref="S3.SS2.SSS1.p2.5.m5.1.1.1">→</ci><apply id="S3.SS2.SSS1.p2.5.m5.1.1.2.cmml" xref="S3.SS2.SSS1.p2.5.m5.1.1.2"><csymbol cd="latexml" id="S3.SS2.SSS1.p2.5.m5.1.1.2.1.cmml" xref="S3.SS2.SSS1.p2.5.m5.1.1.2.1">percent</csymbol><cn type="float" id="S3.SS2.SSS1.p2.5.m5.1.1.2.2.cmml" xref="S3.SS2.SSS1.p2.5.m5.1.1.2.2">77.6</cn></apply><apply id="S3.SS2.SSS1.p2.5.m5.1.1.3.cmml" xref="S3.SS2.SSS1.p2.5.m5.1.1.3"><csymbol cd="latexml" id="S3.SS2.SSS1.p2.5.m5.1.1.3.1.cmml" xref="S3.SS2.SSS1.p2.5.m5.1.1.3.1">percent</csymbol><cn type="float" id="S3.SS2.SSS1.p2.5.m5.1.1.3.2.cmml" xref="S3.SS2.SSS1.p2.5.m5.1.1.3.2">84.8</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS1.p2.5.m5.1c">77.6\%\rightarrow 84.8\%</annotation></semantics></math>).</p>
</div>
<div id="S3.SS2.SSS1.p3" class="ltx_para ltx_noindent">
<p id="S3.SS2.SSS1.p3.1" class="ltx_p">These results show that CCS can <span id="S3.SS2.SSS1.p3.1.1" class="ltx_text ltx_font_italic">exceed</span> the performance of strong baseline methods that access the model outputs, even though this wasn’t our main goal.
This indicates that it is indeed possible to classify examples with high accuracy using only unlabeled model representations.</p>
</div>
</section>
<section id="S3.SS2.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.2.2 </span>CCS Is Robust To Misleading Prompts</h4>

<div id="S3.SS2.SSS2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS2.SSS2.p1.2" class="ltx_p">Recall our goal: to discover latent knowledge in a language model even when the model outputs false text.
In particular, language models are typically trained to imitate text whether or not it is correct, so if a model sees false text it should intuitively be more likely to predict that subsequent text will also be false.
Based on this idea, we provide an initial proof of concept that CCS can make progress toward our goal by constructing prompts that aim to mislead the outputs of language models.
Specifically, we add a prefix to the beginning of our zero-shot prompts that consists of questions answered incorrectly (<a href="#A2.F5" title="In B.1.1 Evaluating The Effect of The Misleading Prefix ‣ B.1 Misleading Prefix ‣ Appendix B Misleading Prefix Details ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">5</span></a>).
The hope is that such a prefix will decrease zero-shot accuracy because the model will imitate its context and answer subsequent questions incorrectly even if it internally “knows” better.<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>In practice we found that model behavior was qualitatively similar between using this prefix with correct and incorrect answers, in agreement with the findings of <cite class="ltx_cite ltx_citemacro_citet">Min et al. (<a href="#bib.bib35" title="" class="ltx_ref">2022b</a>)</cite> that using incorrect labels in demonstrations often does not significantly degrade the performance of a few-shot prompt. Consequently, the prefix may instead actually be reducing accuracy because it is out-of-distribution. See also <cite class="ltx_cite ltx_citemacro_citet">Kim et al. (<a href="#bib.bib21" title="" class="ltx_ref">2022</a>)</cite> for more experiments on the subtleties of correct vs incorrect labels for few-shot prompts.</span></span></span>
We found that while most models are robust to this type of prefix (see <a href="#A2" title="Appendix B Misleading Prefix Details ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">B</span></a>), it significantly drops calibrated zero-shot performance in UnifiedQA, decreasing accuracy from <math id="S3.SS2.SSS2.p1.1.m1.1" class="ltx_Math" alttext="80.4\%" display="inline"><semantics id="S3.SS2.SSS2.p1.1.m1.1a"><mrow id="S3.SS2.SSS2.p1.1.m1.1.1" xref="S3.SS2.SSS2.p1.1.m1.1.1.cmml"><mn id="S3.SS2.SSS2.p1.1.m1.1.1.2" xref="S3.SS2.SSS2.p1.1.m1.1.1.2.cmml">80.4</mn><mo id="S3.SS2.SSS2.p1.1.m1.1.1.1" xref="S3.SS2.SSS2.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p1.1.m1.1b"><apply id="S3.SS2.SSS2.p1.1.m1.1.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1"><csymbol cd="latexml" id="S3.SS2.SSS2.p1.1.m1.1.1.1.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.1">percent</csymbol><cn type="float" id="S3.SS2.SSS2.p1.1.m1.1.1.2.cmml" xref="S3.SS2.SSS2.p1.1.m1.1.1.2">80.4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p1.1.m1.1c">80.4\%</annotation></semantics></math> to <math id="S3.SS2.SSS2.p1.2.m2.1" class="ltx_Math" alttext="70.9\%" display="inline"><semantics id="S3.SS2.SSS2.p1.2.m2.1a"><mrow id="S3.SS2.SSS2.p1.2.m2.1.1" xref="S3.SS2.SSS2.p1.2.m2.1.1.cmml"><mn id="S3.SS2.SSS2.p1.2.m2.1.1.2" xref="S3.SS2.SSS2.p1.2.m2.1.1.2.cmml">70.9</mn><mo id="S3.SS2.SSS2.p1.2.m2.1.1.1" xref="S3.SS2.SSS2.p1.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p1.2.m2.1b"><apply id="S3.SS2.SSS2.p1.2.m2.1.1.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1"><csymbol cd="latexml" id="S3.SS2.SSS2.p1.2.m2.1.1.1.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.1">percent</csymbol><cn type="float" id="S3.SS2.SSS2.p1.2.m2.1.1.2.cmml" xref="S3.SS2.SSS2.p1.2.m2.1.1.2">70.9</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p1.2.m2.1c">70.9\%</annotation></semantics></math>.</p>
</div>
<div id="S3.SS2.SSS2.p2" class="ltx_para ltx_noindent">
<p id="S3.SS2.SSS2.p2.2" class="ltx_p">We evaluate CCS on these examples and show the results in <a href="#A1.F4" title="In Appendix A Identifying Clusters ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">4</span></a> of the Appendix.
We find that despite the <math id="S3.SS2.SSS2.p2.1.m1.1" class="ltx_Math" alttext="9.5\%" display="inline"><semantics id="S3.SS2.SSS2.p2.1.m1.1a"><mrow id="S3.SS2.SSS2.p2.1.m1.1.1" xref="S3.SS2.SSS2.p2.1.m1.1.1.cmml"><mn id="S3.SS2.SSS2.p2.1.m1.1.1.2" xref="S3.SS2.SSS2.p2.1.m1.1.1.2.cmml">9.5</mn><mo id="S3.SS2.SSS2.p2.1.m1.1.1.1" xref="S3.SS2.SSS2.p2.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p2.1.m1.1b"><apply id="S3.SS2.SSS2.p2.1.m1.1.1.cmml" xref="S3.SS2.SSS2.p2.1.m1.1.1"><csymbol cd="latexml" id="S3.SS2.SSS2.p2.1.m1.1.1.1.cmml" xref="S3.SS2.SSS2.p2.1.m1.1.1.1">percent</csymbol><cn type="float" id="S3.SS2.SSS2.p2.1.m1.1.1.2.cmml" xref="S3.SS2.SSS2.p2.1.m1.1.1.2">9.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p2.1.m1.1c">9.5\%</annotation></semantics></math> drop in zero-shot accuracy, CCS maintains high accuracy (<math id="S3.SS2.SSS2.p2.2.m2.1" class="ltx_Math" alttext="82.1\%\rightarrow 83.8\%" display="inline"><semantics id="S3.SS2.SSS2.p2.2.m2.1a"><mrow id="S3.SS2.SSS2.p2.2.m2.1.1" xref="S3.SS2.SSS2.p2.2.m2.1.1.cmml"><mrow id="S3.SS2.SSS2.p2.2.m2.1.1.2" xref="S3.SS2.SSS2.p2.2.m2.1.1.2.cmml"><mn id="S3.SS2.SSS2.p2.2.m2.1.1.2.2" xref="S3.SS2.SSS2.p2.2.m2.1.1.2.2.cmml">82.1</mn><mo id="S3.SS2.SSS2.p2.2.m2.1.1.2.1" xref="S3.SS2.SSS2.p2.2.m2.1.1.2.1.cmml">%</mo></mrow><mo stretchy="false" id="S3.SS2.SSS2.p2.2.m2.1.1.1" xref="S3.SS2.SSS2.p2.2.m2.1.1.1.cmml">→</mo><mrow id="S3.SS2.SSS2.p2.2.m2.1.1.3" xref="S3.SS2.SSS2.p2.2.m2.1.1.3.cmml"><mn id="S3.SS2.SSS2.p2.2.m2.1.1.3.2" xref="S3.SS2.SSS2.p2.2.m2.1.1.3.2.cmml">83.8</mn><mo id="S3.SS2.SSS2.p2.2.m2.1.1.3.1" xref="S3.SS2.SSS2.p2.2.m2.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSS2.p2.2.m2.1b"><apply id="S3.SS2.SSS2.p2.2.m2.1.1.cmml" xref="S3.SS2.SSS2.p2.2.m2.1.1"><ci id="S3.SS2.SSS2.p2.2.m2.1.1.1.cmml" xref="S3.SS2.SSS2.p2.2.m2.1.1.1">→</ci><apply id="S3.SS2.SSS2.p2.2.m2.1.1.2.cmml" xref="S3.SS2.SSS2.p2.2.m2.1.1.2"><csymbol cd="latexml" id="S3.SS2.SSS2.p2.2.m2.1.1.2.1.cmml" xref="S3.SS2.SSS2.p2.2.m2.1.1.2.1">percent</csymbol><cn type="float" id="S3.SS2.SSS2.p2.2.m2.1.1.2.2.cmml" xref="S3.SS2.SSS2.p2.2.m2.1.1.2.2">82.1</cn></apply><apply id="S3.SS2.SSS2.p2.2.m2.1.1.3.cmml" xref="S3.SS2.SSS2.p2.2.m2.1.1.3"><csymbol cd="latexml" id="S3.SS2.SSS2.p2.2.m2.1.1.3.1.cmml" xref="S3.SS2.SSS2.p2.2.m2.1.1.3.1">percent</csymbol><cn type="float" id="S3.SS2.SSS2.p2.2.m2.1.1.3.2.cmml" xref="S3.SS2.SSS2.p2.2.m2.1.1.3.2">83.8</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSS2.p2.2.m2.1c">82.1\%\rightarrow 83.8\%</annotation></semantics></math>).
This provides evidence that our method can still work well even when model outputs are unreliable.</p>
</div>
</section>
</section>
<section id="S3.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.3 </span>Analyzing CCS</h3>

<div id="S3.SS3.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.p1.1" class="ltx_p">We have shown that CCS attains strong classification performance in standard settings and when we deliberately mislead models.
Moreover, we have described our motivation as discovering latent representations of truth in language models, but in practice CCS just finds a direction in representation space that attains high accuracy.
This raises the question: in what sense is CCS actually finding “truth” features?
We now provide a preliminary investigation of this question.</p>
</div>
<section id="S3.SS3.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.3.1 </span>CCS Finds A Task-Agnostic Representation of Truth</h4>

<div id="S3.SS3.SSS1.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS1.p1.1" class="ltx_p">From the results described so far, it may be possible that the classifier we find is capturing dataset-specific properties, such as which of two labels (e.g. “Yes” vs. “No”) is more likely to be correct.
We rule this out by showing that it generalizes across completely different tasks, including ones with different label spaces (such as from “Yes” and “No” for a generic task to “Positive” and “Negative” for sentiment classification).
In particular, we train and test CCS on every pair of datasets, and show the resulting transfer for several models in <a href="#S3.F2" title="In 3.3.1 CCS Finds A Task-Agnostic Representation of Truth ‣ 3.3 Analyzing CCS ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">2</span></a>.
(See <a href="#A6" title="Appendix F Complete Intermediate Representations Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">F</span></a> for results with other models.)</p>
</div>
<div id="S3.SS3.SSS1.p2" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS1.p2.2" class="ltx_p">We find that CCS indeed transfer wells: in the majority of datasets, the transfer accuracy of CCS is competitive with training and testing on the same dataset (the last row in <a href="#S3.F2" title="In 3.3.1 CCS Finds A Task-Agnostic Representation of Truth ‣ 3.3 Analyzing CCS ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">2</span></a>, “No transfer”).
Transfer performance can even outperform the no-transfer setting, especially when we train on simpler tasks, such as sentiment classification.
For example, training CCS on the Amazon sentiment dataset achieves an average transfer accuracy of <math id="S3.SS3.SSS1.p2.1.m1.1" class="ltx_Math" alttext="71.8\%" display="inline"><semantics id="S3.SS3.SSS1.p2.1.m1.1a"><mrow id="S3.SS3.SSS1.p2.1.m1.1.1" xref="S3.SS3.SSS1.p2.1.m1.1.1.cmml"><mn id="S3.SS3.SSS1.p2.1.m1.1.1.2" xref="S3.SS3.SSS1.p2.1.m1.1.1.2.cmml">71.8</mn><mo id="S3.SS3.SSS1.p2.1.m1.1.1.1" xref="S3.SS3.SSS1.p2.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.p2.1.m1.1b"><apply id="S3.SS3.SSS1.p2.1.m1.1.1.cmml" xref="S3.SS3.SSS1.p2.1.m1.1.1"><csymbol cd="latexml" id="S3.SS3.SSS1.p2.1.m1.1.1.1.cmml" xref="S3.SS3.SSS1.p2.1.m1.1.1.1">percent</csymbol><cn type="float" id="S3.SS3.SSS1.p2.1.m1.1.1.2.cmml" xref="S3.SS3.SSS1.p2.1.m1.1.1.2">71.8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.p2.1.m1.1c">71.8\%</annotation></semantics></math>, which is <math id="S3.SS3.SSS1.p2.2.m2.1" class="ltx_Math" alttext="0.6\%" display="inline"><semantics id="S3.SS3.SSS1.p2.2.m2.1a"><mrow id="S3.SS3.SSS1.p2.2.m2.1.1" xref="S3.SS3.SSS1.p2.2.m2.1.1.cmml"><mn id="S3.SS3.SSS1.p2.2.m2.1.1.2" xref="S3.SS3.SSS1.p2.2.m2.1.1.2.cmml">0.6</mn><mo id="S3.SS3.SSS1.p2.2.m2.1.1.1" xref="S3.SS3.SSS1.p2.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS1.p2.2.m2.1b"><apply id="S3.SS3.SSS1.p2.2.m2.1.1.cmml" xref="S3.SS3.SSS1.p2.2.m2.1.1"><csymbol cd="latexml" id="S3.SS3.SSS1.p2.2.m2.1.1.1.cmml" xref="S3.SS3.SSS1.p2.2.m2.1.1.1">percent</csymbol><cn type="float" id="S3.SS3.SSS1.p2.2.m2.1.1.2.cmml" xref="S3.SS3.SSS1.p2.2.m2.1.1.2">0.6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS1.p2.2.m2.1c">0.6\%</annotation></semantics></math> higher than CCS without transfer.
We speculate that this performs so well because the difference in representations between correct and incorrect answers is especially pronounced for easier tasks like Amazon.</p>
</div>
<div id="S3.SS3.SSS1.p3" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS1.p3.1" class="ltx_p">Additionally, transfer accuracy tends to be similar for many datasets.
For example, training on IMDB (row 1 of <a href="#S3.F2" title="In 3.3.1 CCS Finds A Task-Agnostic Representation of Truth ‣ 3.3 Analyzing CCS ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">2</span></a>) has similar accuracy as training on DBPedia (row 4).
This provides evidence that CCS can find a functionally similar direction across many different types of training datasets.
Overall, these results suggest that (1) models may have a task-agnostic representation related to what is true, and that (2) CCS may approximately find this representation even without diverse data.</p>
</div>
<figure id="S3.F2" class="ltx_figure"><img src="/html/2212.03827/assets/x2.png" id="S3.F2.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="423" height="155" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>Transfer Performance using CCS on UnifiedQA, T0 and DeBERTa.
The y-axis corresponds to the training dataset, and the x-axis corresponds to the test dataset.
The final row (“No Transfer”) corresponds to training and testing on the same dataset, which is the same as the diagonal.
On most datasets, CCS transfers well to other datasets (relative to no transfer), including to different tasks with completely different labels.
In some cases transfer even outperforms the no-transfer setting.
See <a href="#A5" title="Appendix E Complete Transfer Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">E</span></a> for results with other models.
</figcaption>
</figure>
</section>
<section id="S3.SS3.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.3.2 </span>CCS Does Not Just Recover Model Outputs</h4>

<div id="S3.SS3.SSS2.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS2.p1.1" class="ltx_p">One possibility is that CCS can only recover knowledge already contained in a model’s outputs.
We have shown that CCS can outperform zero-shot accuracy (<a href="#S3.SS2.SSS1" title="3.2.1 CCS Outperforms Zero-Shot ‣ 3.2 Evaluating CCS ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">3.2.1</span></a>), especially when model outputs are misled (<a href="#S3.SS2.SSS2" title="3.2.2 CCS Is Robust To Misleading Prompts ‣ 3.2 Evaluating CCS ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">3.2.2</span></a>), which already provides evidence against this possibility.
We now provide additional evidence against this concern.</p>
</div>
<div id="S3.SS3.SSS2.p2" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS2.p2.2" class="ltx_p">First, if CCS were just recovering knowledge in the model outputs, using the last layer of a network (right before the outputs) should presumably outperform intermediate layers (which are more causally distant from the outputs). However, for T5 and UnifiedQA, we find that using hidden states in the middle of the network outperform hidden states at the end of the network when using CCS (see <a href="#A6.F10" title="In Appendix F Complete Intermediate Representations Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">10</span></a>). This is especially true for UnifiedQA on misleading prefixes; we find that using the encoder hidden states is robust to misleading prefixes (<a href="#S3.SS2.SSS2" title="3.2.2 CCS Is Robust To Misleading Prompts ‣ 3.2 Evaluating CCS ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">3.2.2</span></a>), but that accuracy using the decoder hidden states drops from <math id="S3.SS3.SSS2.p2.1.m1.1" class="ltx_Math" alttext="81.0\%" display="inline"><semantics id="S3.SS3.SSS2.p2.1.m1.1a"><mrow id="S3.SS3.SSS2.p2.1.m1.1.1" xref="S3.SS3.SSS2.p2.1.m1.1.1.cmml"><mn id="S3.SS3.SSS2.p2.1.m1.1.1.2" xref="S3.SS3.SSS2.p2.1.m1.1.1.2.cmml">81.0</mn><mo id="S3.SS3.SSS2.p2.1.m1.1.1.1" xref="S3.SS3.SSS2.p2.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS2.p2.1.m1.1b"><apply id="S3.SS3.SSS2.p2.1.m1.1.1.cmml" xref="S3.SS3.SSS2.p2.1.m1.1.1"><csymbol cd="latexml" id="S3.SS3.SSS2.p2.1.m1.1.1.1.cmml" xref="S3.SS3.SSS2.p2.1.m1.1.1.1">percent</csymbol><cn type="float" id="S3.SS3.SSS2.p2.1.m1.1.1.2.cmml" xref="S3.SS3.SSS2.p2.1.m1.1.1.2">81.0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS2.p2.1.m1.1c">81.0\%</annotation></semantics></math> to <math id="S3.SS3.SSS2.p2.2.m2.1" class="ltx_Math" alttext="73.5\%" display="inline"><semantics id="S3.SS3.SSS2.p2.2.m2.1a"><mrow id="S3.SS3.SSS2.p2.2.m2.1.1" xref="S3.SS3.SSS2.p2.2.m2.1.1.cmml"><mn id="S3.SS3.SSS2.p2.2.m2.1.1.2" xref="S3.SS3.SSS2.p2.2.m2.1.1.2.cmml">73.5</mn><mo id="S3.SS3.SSS2.p2.2.m2.1.1.1" xref="S3.SS3.SSS2.p2.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS2.p2.2.m2.1b"><apply id="S3.SS3.SSS2.p2.2.m2.1.1.cmml" xref="S3.SS3.SSS2.p2.2.m2.1.1"><csymbol cd="latexml" id="S3.SS3.SSS2.p2.2.m2.1.1.1.cmml" xref="S3.SS3.SSS2.p2.2.m2.1.1.1">percent</csymbol><cn type="float" id="S3.SS3.SSS2.p2.2.m2.1.1.2.cmml" xref="S3.SS3.SSS2.p2.2.m2.1.1.2">73.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS2.p2.2.m2.1c">73.5\%</annotation></semantics></math>, a similar amount to zero-shot accuracy.
This suggests that compared to the later layers of a model, intermediate layers are more robust and less correlated with the model outputs, and that CCS can take advantage of this.</p>
</div>
<div id="S3.SS3.SSS2.p3" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS2.p3.1" class="ltx_p">Finally, if CCS were just recovering knowledge in the model outputs, we would only expect it to work in cases where model outputs are informative. However, we show in <a href="#A3" title="Appendix C Masked Language Modeling Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">C</span></a> that CCS still works with masked language models when their outputs are uninformative: when we don’t [MASK] any input tokens, and when we prompt models so that the labels used to construct contrast pairs appear in the <span id="S3.SS3.SSS2.p3.1.1" class="ltx_text ltx_font_italic">middle</span> of a prompt rather than at the end.
These results show that CCS can sometimes recover latent knowledge in a model that is distinct from—and more useful than—what the model outputs.</p>
</div>
</section>
<section id="S3.SS3.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.3.3 </span>Truth is a Salient Feature</h4>

<div id="S3.SS3.SSS3.p1" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS3.p1.1" class="ltx_p">From the results we have presented so far, it is possible that the direction learned by CCS is difficult to find and requires using a large amount of unsupervised data.
We provide evidence against this possibility by showing that finding such a direction can both (1) often be done with only a small amount of data, and can also (2) often be done by essentially taking the top principal component of a slightly modified representation space.</p>
</div>
<div id="S3.SS3.SSS3.p2" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS3.p2.5" class="ltx_p"><span id="S3.SS3.SSS3.p2.5.1" class="ltx_text ltx_font_bold">CCS doesn’t require much data.</span>
We now evaluate how well CCS performs with different amounts of data.
Whereas before we trained CCS using the full training set and all prompts, here we use limited data and a single prompt.
Specifically, we train using only <math id="S3.SS3.SSS3.p2.1.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS3.SSS3.p2.1.m1.1a"><mi id="S3.SS3.SSS3.p2.1.m1.1.1" xref="S3.SS3.SSS3.p2.1.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS3.p2.1.m1.1b"><ci id="S3.SS3.SSS3.p2.1.m1.1.1.cmml" xref="S3.SS3.SSS3.p2.1.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS3.p2.1.m1.1c">k</annotation></semantics></math> unlabeled contrast pairs, using the single prompt for each model and dataset that achieves the highest zero-shot accuracy. We still test on all prompts for each dataset.
We resample <math id="S3.SS3.SSS3.p2.2.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.SS3.SSS3.p2.2.m2.1a"><mi id="S3.SS3.SSS3.p2.2.m2.1.1" xref="S3.SS3.SSS3.p2.2.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS3.p2.2.m2.1b"><ci id="S3.SS3.SSS3.p2.2.m2.1.1.cmml" xref="S3.SS3.SSS3.p2.2.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS3.p2.2.m2.1c">k</annotation></semantics></math> points <math id="S3.SS3.SSS3.p2.3.m3.1" class="ltx_Math" alttext="32" display="inline"><semantics id="S3.SS3.SSS3.p2.3.m3.1a"><mn id="S3.SS3.SSS3.p2.3.m3.1.1" xref="S3.SS3.SSS3.p2.3.m3.1.1.cmml">32</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS3.p2.3.m3.1b"><cn type="integer" id="S3.SS3.SSS3.p2.3.m3.1.1.cmml" xref="S3.SS3.SSS3.p2.3.m3.1.1">32</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS3.p2.3.m3.1c">32</annotation></semantics></math> times for each of <math id="S3.SS3.SSS3.p2.4.m4.4" class="ltx_Math" alttext="k=1,2,4,\cdots" display="inline"><semantics id="S3.SS3.SSS3.p2.4.m4.4a"><mrow id="S3.SS3.SSS3.p2.4.m4.4.5" xref="S3.SS3.SSS3.p2.4.m4.4.5.cmml"><mi id="S3.SS3.SSS3.p2.4.m4.4.5.2" xref="S3.SS3.SSS3.p2.4.m4.4.5.2.cmml">k</mi><mo id="S3.SS3.SSS3.p2.4.m4.4.5.1" xref="S3.SS3.SSS3.p2.4.m4.4.5.1.cmml">=</mo><mrow id="S3.SS3.SSS3.p2.4.m4.4.5.3.2" xref="S3.SS3.SSS3.p2.4.m4.4.5.3.1.cmml"><mn id="S3.SS3.SSS3.p2.4.m4.1.1" xref="S3.SS3.SSS3.p2.4.m4.1.1.cmml">1</mn><mo id="S3.SS3.SSS3.p2.4.m4.4.5.3.2.1" xref="S3.SS3.SSS3.p2.4.m4.4.5.3.1.cmml">,</mo><mn id="S3.SS3.SSS3.p2.4.m4.2.2" xref="S3.SS3.SSS3.p2.4.m4.2.2.cmml">2</mn><mo id="S3.SS3.SSS3.p2.4.m4.4.5.3.2.2" xref="S3.SS3.SSS3.p2.4.m4.4.5.3.1.cmml">,</mo><mn id="S3.SS3.SSS3.p2.4.m4.3.3" xref="S3.SS3.SSS3.p2.4.m4.3.3.cmml">4</mn><mo id="S3.SS3.SSS3.p2.4.m4.4.5.3.2.3" xref="S3.SS3.SSS3.p2.4.m4.4.5.3.1.cmml">,</mo><mi mathvariant="normal" id="S3.SS3.SSS3.p2.4.m4.4.4" xref="S3.SS3.SSS3.p2.4.m4.4.4.cmml">⋯</mi></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS3.p2.4.m4.4b"><apply id="S3.SS3.SSS3.p2.4.m4.4.5.cmml" xref="S3.SS3.SSS3.p2.4.m4.4.5"><eq id="S3.SS3.SSS3.p2.4.m4.4.5.1.cmml" xref="S3.SS3.SSS3.p2.4.m4.4.5.1"></eq><ci id="S3.SS3.SSS3.p2.4.m4.4.5.2.cmml" xref="S3.SS3.SSS3.p2.4.m4.4.5.2">𝑘</ci><list id="S3.SS3.SSS3.p2.4.m4.4.5.3.1.cmml" xref="S3.SS3.SSS3.p2.4.m4.4.5.3.2"><cn type="integer" id="S3.SS3.SSS3.p2.4.m4.1.1.cmml" xref="S3.SS3.SSS3.p2.4.m4.1.1">1</cn><cn type="integer" id="S3.SS3.SSS3.p2.4.m4.2.2.cmml" xref="S3.SS3.SSS3.p2.4.m4.2.2">2</cn><cn type="integer" id="S3.SS3.SSS3.p2.4.m4.3.3.cmml" xref="S3.SS3.SSS3.p2.4.m4.3.3">4</cn><ci id="S3.SS3.SSS3.p2.4.m4.4.4.cmml" xref="S3.SS3.SSS3.p2.4.m4.4.4">⋯</ci></list></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS3.p2.4.m4.4c">k=1,2,4,\cdots</annotation></semantics></math>, and take the average accuracy across those <math id="S3.SS3.SSS3.p2.5.m5.1" class="ltx_Math" alttext="32" display="inline"><semantics id="S3.SS3.SSS3.p2.5.m5.1a"><mn id="S3.SS3.SSS3.p2.5.m5.1.1" xref="S3.SS3.SSS3.p2.5.m5.1.1.cmml">32</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS3.p2.5.m5.1b"><cn type="integer" id="S3.SS3.SSS3.p2.5.m5.1.1.cmml" xref="S3.SS3.SSS3.p2.5.m5.1.1">32</cn></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS3.p2.5.m5.1c">32</annotation></semantics></math> samples.
Finally, we plot the average such accuracy across all datasets and prompts for several models in <a href="#S3.F3" title="In 3.3.3 Truth is a Salient Feature ‣ 3.3 Analyzing CCS ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">3</span></a>.</p>
</div>
<div id="S3.SS3.SSS3.p3" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS3.p3.1" class="ltx_p">We find that while CCS benefits from more data, it can often do well with very limited data.
In fact, it can sometimes even do well with only a single contrast pair, though we find high variance across individual datasets; see <a href="#A4" title="Appendix D Complete Sample Complexity Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">D</span></a> for more details.
This suggests that the strong performance of CCS does not primarily come from using a large amount of unsupervised data, and indicates that the direction learned by CCS may be relatively easy to find.</p>
</div>
<figure id="S3.F3" class="ltx_figure"><img src="/html/2212.03827/assets/x3.png" id="S3.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="228" height="147" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Accuracy when we train CCS on <math id="S3.F3.3.m1.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.F3.3.m1.1b"><mi id="S3.F3.3.m1.1.1" xref="S3.F3.3.m1.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.F3.3.m1.1c"><ci id="S3.F3.3.m1.1.1.cmml" xref="S3.F3.3.m1.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.3.m1.1d">k</annotation></semantics></math> samples for different values of <math id="S3.F3.4.m2.1" class="ltx_Math" alttext="k" display="inline"><semantics id="S3.F3.4.m2.1b"><mi id="S3.F3.4.m2.1.1" xref="S3.F3.4.m2.1.1.cmml">k</mi><annotation-xml encoding="MathML-Content" id="S3.F3.4.m2.1c"><ci id="S3.F3.4.m2.1.1.cmml" xref="S3.F3.4.m2.1.1">𝑘</ci></annotation-xml><annotation encoding="application/x-tex" id="S3.F3.4.m2.1d">k</annotation></semantics></math> (each time averaged across 32 trials). We use the single prompt with the highest zero-shot accuracy for each dataset and model. While CCS benefits from more examples, it can often work well with limited data.</figcaption>
</figure>
<div id="S3.SS3.SSS3.p4" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS3.p4.6" class="ltx_p"><span id="S3.SS3.SSS3.p4.6.1" class="ltx_text ltx_font_bold">Contrastive Representation Clustering.</span>
We now show that directions correlated with the truth may be “salient” in a different way: by showing that we can also find such directions using either (1) PCA or (2) clustering.
Specifically, suppose we construct contrast pairs (<math id="S3.SS3.SSS3.p4.1.m1.1" class="ltx_Math" alttext="x^{+}_{i}" display="inline"><semantics id="S3.SS3.SSS3.p4.1.m1.1a"><msubsup id="S3.SS3.SSS3.p4.1.m1.1.1" xref="S3.SS3.SSS3.p4.1.m1.1.1.cmml"><mi id="S3.SS3.SSS3.p4.1.m1.1.1.2.2" xref="S3.SS3.SSS3.p4.1.m1.1.1.2.2.cmml">x</mi><mi id="S3.SS3.SSS3.p4.1.m1.1.1.3" xref="S3.SS3.SSS3.p4.1.m1.1.1.3.cmml">i</mi><mo id="S3.SS3.SSS3.p4.1.m1.1.1.2.3" xref="S3.SS3.SSS3.p4.1.m1.1.1.2.3.cmml">+</mo></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS3.p4.1.m1.1b"><apply id="S3.SS3.SSS3.p4.1.m1.1.1.cmml" xref="S3.SS3.SSS3.p4.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p4.1.m1.1.1.1.cmml" xref="S3.SS3.SSS3.p4.1.m1.1.1">subscript</csymbol><apply id="S3.SS3.SSS3.p4.1.m1.1.1.2.cmml" xref="S3.SS3.SSS3.p4.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p4.1.m1.1.1.2.1.cmml" xref="S3.SS3.SSS3.p4.1.m1.1.1">superscript</csymbol><ci id="S3.SS3.SSS3.p4.1.m1.1.1.2.2.cmml" xref="S3.SS3.SSS3.p4.1.m1.1.1.2.2">𝑥</ci><plus id="S3.SS3.SSS3.p4.1.m1.1.1.2.3.cmml" xref="S3.SS3.SSS3.p4.1.m1.1.1.2.3"></plus></apply><ci id="S3.SS3.SSS3.p4.1.m1.1.1.3.cmml" xref="S3.SS3.SSS3.p4.1.m1.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS3.p4.1.m1.1c">x^{+}_{i}</annotation></semantics></math>, <math id="S3.SS3.SSS3.p4.2.m2.1" class="ltx_Math" alttext="x^{-}_{i}" display="inline"><semantics id="S3.SS3.SSS3.p4.2.m2.1a"><msubsup id="S3.SS3.SSS3.p4.2.m2.1.1" xref="S3.SS3.SSS3.p4.2.m2.1.1.cmml"><mi id="S3.SS3.SSS3.p4.2.m2.1.1.2.2" xref="S3.SS3.SSS3.p4.2.m2.1.1.2.2.cmml">x</mi><mi id="S3.SS3.SSS3.p4.2.m2.1.1.3" xref="S3.SS3.SSS3.p4.2.m2.1.1.3.cmml">i</mi><mo id="S3.SS3.SSS3.p4.2.m2.1.1.2.3" xref="S3.SS3.SSS3.p4.2.m2.1.1.2.3.cmml">−</mo></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS3.p4.2.m2.1b"><apply id="S3.SS3.SSS3.p4.2.m2.1.1.cmml" xref="S3.SS3.SSS3.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p4.2.m2.1.1.1.cmml" xref="S3.SS3.SSS3.p4.2.m2.1.1">subscript</csymbol><apply id="S3.SS3.SSS3.p4.2.m2.1.1.2.cmml" xref="S3.SS3.SSS3.p4.2.m2.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p4.2.m2.1.1.2.1.cmml" xref="S3.SS3.SSS3.p4.2.m2.1.1">superscript</csymbol><ci id="S3.SS3.SSS3.p4.2.m2.1.1.2.2.cmml" xref="S3.SS3.SSS3.p4.2.m2.1.1.2.2">𝑥</ci><minus id="S3.SS3.SSS3.p4.2.m2.1.1.2.3.cmml" xref="S3.SS3.SSS3.p4.2.m2.1.1.2.3"></minus></apply><ci id="S3.SS3.SSS3.p4.2.m2.1.1.3.cmml" xref="S3.SS3.SSS3.p4.2.m2.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS3.p4.2.m2.1c">x^{-}_{i}</annotation></semantics></math>) as before.
Intuitively, these two examples are qualitatively almost identical except that one is true and the other is false, so the main difference between the representations <math id="S3.SS3.SSS3.p4.3.m3.1" class="ltx_Math" alttext="\tilde{\phi}(x_{i}^{+})" display="inline"><semantics id="S3.SS3.SSS3.p4.3.m3.1a"><mrow id="S3.SS3.SSS3.p4.3.m3.1.1" xref="S3.SS3.SSS3.p4.3.m3.1.1.cmml"><mover accent="true" id="S3.SS3.SSS3.p4.3.m3.1.1.3" xref="S3.SS3.SSS3.p4.3.m3.1.1.3.cmml"><mi id="S3.SS3.SSS3.p4.3.m3.1.1.3.2" xref="S3.SS3.SSS3.p4.3.m3.1.1.3.2.cmml">ϕ</mi><mo id="S3.SS3.SSS3.p4.3.m3.1.1.3.1" xref="S3.SS3.SSS3.p4.3.m3.1.1.3.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S3.SS3.SSS3.p4.3.m3.1.1.2" xref="S3.SS3.SSS3.p4.3.m3.1.1.2.cmml">​</mo><mrow id="S3.SS3.SSS3.p4.3.m3.1.1.1.1" xref="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS3.SSS3.p4.3.m3.1.1.1.1.2" xref="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.cmml">(</mo><msubsup id="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1" xref="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.cmml"><mi id="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.2.2" xref="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.2.2.cmml">x</mi><mi id="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.2.3" xref="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.2.3.cmml">i</mi><mo id="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.3" xref="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.3.cmml">+</mo></msubsup><mo stretchy="false" id="S3.SS3.SSS3.p4.3.m3.1.1.1.1.3" xref="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS3.p4.3.m3.1b"><apply id="S3.SS3.SSS3.p4.3.m3.1.1.cmml" xref="S3.SS3.SSS3.p4.3.m3.1.1"><times id="S3.SS3.SSS3.p4.3.m3.1.1.2.cmml" xref="S3.SS3.SSS3.p4.3.m3.1.1.2"></times><apply id="S3.SS3.SSS3.p4.3.m3.1.1.3.cmml" xref="S3.SS3.SSS3.p4.3.m3.1.1.3"><ci id="S3.SS3.SSS3.p4.3.m3.1.1.3.1.cmml" xref="S3.SS3.SSS3.p4.3.m3.1.1.3.1">~</ci><ci id="S3.SS3.SSS3.p4.3.m3.1.1.3.2.cmml" xref="S3.SS3.SSS3.p4.3.m3.1.1.3.2">italic-ϕ</ci></apply><apply id="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.cmml" xref="S3.SS3.SSS3.p4.3.m3.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS3.p4.3.m3.1.1.1.1">superscript</csymbol><apply id="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS3.p4.3.m3.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.2.1.cmml" xref="S3.SS3.SSS3.p4.3.m3.1.1.1.1">subscript</csymbol><ci id="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.2.2.cmml" xref="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.2.2">𝑥</ci><ci id="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.2.3.cmml" xref="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.2.3">𝑖</ci></apply><plus id="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.3.cmml" xref="S3.SS3.SSS3.p4.3.m3.1.1.1.1.1.3"></plus></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS3.p4.3.m3.1c">\tilde{\phi}(x_{i}^{+})</annotation></semantics></math> and <math id="S3.SS3.SSS3.p4.4.m4.1" class="ltx_Math" alttext="\tilde{\phi}(x_{i}^{-})" display="inline"><semantics id="S3.SS3.SSS3.p4.4.m4.1a"><mrow id="S3.SS3.SSS3.p4.4.m4.1.1" xref="S3.SS3.SSS3.p4.4.m4.1.1.cmml"><mover accent="true" id="S3.SS3.SSS3.p4.4.m4.1.1.3" xref="S3.SS3.SSS3.p4.4.m4.1.1.3.cmml"><mi id="S3.SS3.SSS3.p4.4.m4.1.1.3.2" xref="S3.SS3.SSS3.p4.4.m4.1.1.3.2.cmml">ϕ</mi><mo id="S3.SS3.SSS3.p4.4.m4.1.1.3.1" xref="S3.SS3.SSS3.p4.4.m4.1.1.3.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S3.SS3.SSS3.p4.4.m4.1.1.2" xref="S3.SS3.SSS3.p4.4.m4.1.1.2.cmml">​</mo><mrow id="S3.SS3.SSS3.p4.4.m4.1.1.1.1" xref="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS3.SSS3.p4.4.m4.1.1.1.1.2" xref="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.cmml">(</mo><msubsup id="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1" xref="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.cmml"><mi id="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.2.2" xref="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.2.2.cmml">x</mi><mi id="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.2.3" xref="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.2.3.cmml">i</mi><mo id="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.3" xref="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.3.cmml">−</mo></msubsup><mo stretchy="false" id="S3.SS3.SSS3.p4.4.m4.1.1.1.1.3" xref="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS3.p4.4.m4.1b"><apply id="S3.SS3.SSS3.p4.4.m4.1.1.cmml" xref="S3.SS3.SSS3.p4.4.m4.1.1"><times id="S3.SS3.SSS3.p4.4.m4.1.1.2.cmml" xref="S3.SS3.SSS3.p4.4.m4.1.1.2"></times><apply id="S3.SS3.SSS3.p4.4.m4.1.1.3.cmml" xref="S3.SS3.SSS3.p4.4.m4.1.1.3"><ci id="S3.SS3.SSS3.p4.4.m4.1.1.3.1.cmml" xref="S3.SS3.SSS3.p4.4.m4.1.1.3.1">~</ci><ci id="S3.SS3.SSS3.p4.4.m4.1.1.3.2.cmml" xref="S3.SS3.SSS3.p4.4.m4.1.1.3.2">italic-ϕ</ci></apply><apply id="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.cmml" xref="S3.SS3.SSS3.p4.4.m4.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS3.p4.4.m4.1.1.1.1">superscript</csymbol><apply id="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS3.p4.4.m4.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.2.1.cmml" xref="S3.SS3.SSS3.p4.4.m4.1.1.1.1">subscript</csymbol><ci id="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.2.2.cmml" xref="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.2.2">𝑥</ci><ci id="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.2.3.cmml" xref="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.2.3">𝑖</ci></apply><minus id="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.3.cmml" xref="S3.SS3.SSS3.p4.4.m4.1.1.1.1.1.3"></minus></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS3.p4.4.m4.1c">\tilde{\phi}(x_{i}^{-})</annotation></semantics></math> should relate to truth.
Consequently, we can take the differences in (normalized) hidden states, <math id="S3.SS3.SSS3.p4.5.m5.1" class="ltx_Math" alttext="\{\tilde{\phi}(x_{i}^{+})-\tilde{\phi}(x_{i}^{-})\}_{i=1}^{n}" display="inline"><semantics id="S3.SS3.SSS3.p4.5.m5.1a"><msubsup id="S3.SS3.SSS3.p4.5.m5.1.1" xref="S3.SS3.SSS3.p4.5.m5.1.1.cmml"><mrow id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.2" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.2.cmml">{</mo><mrow id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.cmml"><mrow id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.cmml"><mover accent="true" id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.3" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.3.cmml"><mi id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.3.2" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.3.2.cmml">ϕ</mi><mo id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.3.1" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.3.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.2" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.2" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.2" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.3" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.3.cmml">i</mi><mo id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.cmml">+</mo></msubsup><mo stretchy="false" id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.3" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.3" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.3.cmml">−</mo><mrow id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.cmml"><mover accent="true" id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.3" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.3.cmml"><mi id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.3.2" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.3.2.cmml">ϕ</mi><mo id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.3.1" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.3.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.2" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.2.cmml">​</mo><mrow id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.cmml"><mo stretchy="false" id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.2" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.cmml">(</mo><msubsup id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.cmml"><mi id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.2.2" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.2.2.cmml">x</mi><mi id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.2.3" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.2.3.cmml">i</mi><mo id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.3" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.3.cmml">−</mo></msubsup><mo stretchy="false" id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.3" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.3" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S3.SS3.SSS3.p4.5.m5.1.1.1.3" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.3.cmml"><mi id="S3.SS3.SSS3.p4.5.m5.1.1.1.3.2" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.3.2.cmml">i</mi><mo id="S3.SS3.SSS3.p4.5.m5.1.1.1.3.1" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.3.1.cmml">=</mo><mn id="S3.SS3.SSS3.p4.5.m5.1.1.1.3.3" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.3.3.cmml">1</mn></mrow><mi id="S3.SS3.SSS3.p4.5.m5.1.1.3" xref="S3.SS3.SSS3.p4.5.m5.1.1.3.cmml">n</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS3.p4.5.m5.1b"><apply id="S3.SS3.SSS3.p4.5.m5.1.1.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p4.5.m5.1.1.2.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1">superscript</csymbol><apply id="S3.SS3.SSS3.p4.5.m5.1.1.1.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p4.5.m5.1.1.1.2.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1">subscript</csymbol><set id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.2.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1"><apply id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1"><minus id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.3.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.3"></minus><apply id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1"><times id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.2"></times><apply id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.3"><ci id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.3.1.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.3.1">~</ci><ci id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.3.2.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.3.2">italic-ϕ</ci></apply><apply id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1">superscript</csymbol><apply id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><plus id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.1.1.1.1.3"></plus></apply></apply><apply id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2"><times id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.2.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.2"></times><apply id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.3.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.3"><ci id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.3.1.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.3.1">~</ci><ci id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.3.2.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.3.2">italic-ϕ</ci></apply><apply id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.1.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1">superscript</csymbol><apply id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.2.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.2.1.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1">subscript</csymbol><ci id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.2.2.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.2.2">𝑥</ci><ci id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.2.3.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.2.3">𝑖</ci></apply><minus id="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.3.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.1.1.1.2.1.1.1.3"></minus></apply></apply></apply></set><apply id="S3.SS3.SSS3.p4.5.m5.1.1.1.3.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.3"><eq id="S3.SS3.SSS3.p4.5.m5.1.1.1.3.1.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.3.1"></eq><ci id="S3.SS3.SSS3.p4.5.m5.1.1.1.3.2.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.3.2">𝑖</ci><cn type="integer" id="S3.SS3.SSS3.p4.5.m5.1.1.1.3.3.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.1.3.3">1</cn></apply></apply><ci id="S3.SS3.SSS3.p4.5.m5.1.1.3.cmml" xref="S3.SS3.SSS3.p4.5.m5.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS3.p4.5.m5.1c">\{\tilde{\phi}(x_{i}^{+})-\tilde{\phi}(x_{i}^{-})\}_{i=1}^{n}</annotation></semantics></math>, and cluster them.
We call this method Contrastive Representation Clustering (CRC).
Clustering can be achieved by taking the top principal component (TPC) and thresholding at <math id="S3.SS3.SSS3.p4.6.m6.1" class="ltx_Math" alttext="0" display="inline"><semantics id="S3.SS3.SSS3.p4.6.m6.1a"><mn id="S3.SS3.SSS3.p4.6.m6.1.1" xref="S3.SS3.SSS3.p4.6.m6.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS3.p4.6.m6.1b"><cn type="integer" id="S3.SS3.SSS3.p4.6.m6.1.1.cmml" xref="S3.SS3.SSS3.p4.6.m6.1.1">0</cn></annotation-xml></semantics></math>, or by doing a “bimodal salience search” (BSS) to find a direction that looks bimodal; see <a href="#A7.SS3" title="G.3 CRC: Top Principal Component ‣ Appendix G CCS and CRC Implementation Details ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">G.3</span></a> for further details.</p>
</div>
<div id="S3.SS3.SSS3.p5" class="ltx_para ltx_noindent">
<p id="S3.SS3.SSS3.p5.1" class="ltx_p">We compare CCS and these two variants of Contrastive Representation Clustering in <a href="#S3.T2" title="In 3.3.3 Truth is a Salient Feature ‣ 3.3 Analyzing CCS ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">2</span></a>, using the same setting as in <a href="#S3.T1" title="In 3.1 Experimental Setup ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">1</span></a>.
While CCS performs best, all methods attain high accuracy and are competitive with zero-shot performance.
This indicates both that (1) representations of truth often lie in a high-variance direction in the contrastive representation space <math id="S3.SS3.SSS3.p5.1.m1.1" class="ltx_Math" alttext="\{\tilde{\phi}(x_{i}^{+})-\tilde{\phi}(x_{i}^{-})\}_{i=1}^{n}" display="inline"><semantics id="S3.SS3.SSS3.p5.1.m1.1a"><msubsup id="S3.SS3.SSS3.p5.1.m1.1.1" xref="S3.SS3.SSS3.p5.1.m1.1.1.cmml"><mrow id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.2.cmml"><mo stretchy="false" id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.2" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.2.cmml">{</mo><mrow id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.cmml"><mrow id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.cmml"><mover accent="true" id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.3" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.3.cmml"><mi id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.3.2" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.3.2.cmml">ϕ</mi><mo id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.3.1" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.3.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.2" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.2" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.2.2" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.2.3" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.2.3.cmml">i</mi><mo id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.3" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.3.cmml">+</mo></msubsup><mo stretchy="false" id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.3" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.3" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.3.cmml">−</mo><mrow id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.cmml"><mover accent="true" id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.3" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.3.cmml"><mi id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.3.2" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.3.2.cmml">ϕ</mi><mo id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.3.1" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.3.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.2" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.2.cmml">​</mo><mrow id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.cmml"><mo stretchy="false" id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.2" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.cmml">(</mo><msubsup id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.cmml"><mi id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.2.2" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.2.2.cmml">x</mi><mi id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.2.3" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.2.3.cmml">i</mi><mo id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.3" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.3.cmml">−</mo></msubsup><mo stretchy="false" id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.3" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.3" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.2.cmml">}</mo></mrow><mrow id="S3.SS3.SSS3.p5.1.m1.1.1.1.3" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.3.cmml"><mi id="S3.SS3.SSS3.p5.1.m1.1.1.1.3.2" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.3.2.cmml">i</mi><mo id="S3.SS3.SSS3.p5.1.m1.1.1.1.3.1" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.3.1.cmml">=</mo><mn id="S3.SS3.SSS3.p5.1.m1.1.1.1.3.3" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.3.3.cmml">1</mn></mrow><mi id="S3.SS3.SSS3.p5.1.m1.1.1.3" xref="S3.SS3.SSS3.p5.1.m1.1.1.3.cmml">n</mi></msubsup><annotation-xml encoding="MathML-Content" id="S3.SS3.SSS3.p5.1.m1.1b"><apply id="S3.SS3.SSS3.p5.1.m1.1.1.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p5.1.m1.1.1.2.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1">superscript</csymbol><apply id="S3.SS3.SSS3.p5.1.m1.1.1.1.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p5.1.m1.1.1.1.2.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1">subscript</csymbol><set id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.2.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1"><apply id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1"><minus id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.3"></minus><apply id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1"><times id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.2"></times><apply id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.3"><ci id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.3.1.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.3.1">~</ci><ci id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.3.2.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.3.2">italic-ϕ</ci></apply><apply id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1">superscript</csymbol><apply id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><plus id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.1.1.1.1.3"></plus></apply></apply><apply id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2"><times id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.2.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.2"></times><apply id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.3.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.3"><ci id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.3.1.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.3.1">~</ci><ci id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.3.2.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.3.2">italic-ϕ</ci></apply><apply id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.1.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1">superscript</csymbol><apply id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.2.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1"><csymbol cd="ambiguous" id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.2.1.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1">subscript</csymbol><ci id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.2.2.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.2.2">𝑥</ci><ci id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.2.3.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.2.3">𝑖</ci></apply><minus id="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.3.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.1.1.1.2.1.1.1.3"></minus></apply></apply></apply></set><apply id="S3.SS3.SSS3.p5.1.m1.1.1.1.3.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.3"><eq id="S3.SS3.SSS3.p5.1.m1.1.1.1.3.1.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.3.1"></eq><ci id="S3.SS3.SSS3.p5.1.m1.1.1.1.3.2.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.3.2">𝑖</ci><cn type="integer" id="S3.SS3.SSS3.p5.1.m1.1.1.1.3.3.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.1.3.3">1</cn></apply></apply><ci id="S3.SS3.SSS3.p5.1.m1.1.1.3.cmml" xref="S3.SS3.SSS3.p5.1.m1.1.1.3">𝑛</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS3.SSS3.p5.1.m1.1c">\{\tilde{\phi}(x_{i}^{+})-\tilde{\phi}(x_{i}^{-})\}_{i=1}^{n}</annotation></semantics></math>, and also that (2) true and false examples are often well-clustered in this same contrastive space.
This strengthens the idea that representations of truth may be salient features inside models that are relatively easy to find. This may help explain why CCS can perform well without using any supervision, and how it can do so even with only a limited amount of unlabeled data.</p>
</div>
<figure id="S3.T2" class="ltx_table">
<table id="S3.T2.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="S3.T2.2.2" class="ltx_tr">
<td id="S3.T2.2.2.3" class="ltx_td ltx_align_left ltx_border_r">Method</td>
<td id="S3.T2.2.2.4" class="ltx_td ltx_align_center">RoBERTa</td>
<td id="S3.T2.2.2.5" class="ltx_td ltx_align_center">DeBERTa</td>
<td id="S3.T2.2.2.6" class="ltx_td ltx_align_center">GPT-J</td>
<td id="S3.T2.2.2.7" class="ltx_td ltx_align_center">T5</td>
<td id="S3.T2.2.2.8" class="ltx_td ltx_align_center">UQA</td>
<td id="S3.T2.1.1.1" class="ltx_td ltx_align_center ltx_border_r">T0<sup id="S3.T2.1.1.1.1" class="ltx_sup">∗</sup>
</td>
<td id="S3.T2.2.2.2" class="ltx_td ltx_align_center">Mean<sup id="S3.T2.2.2.2.1" class="ltx_sup">∗</sup>
</td>
</tr>
<tr id="S3.T2.2.3" class="ltx_tr">
<td id="S3.T2.2.3.1" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Calibrated 0-shot</td>
<td id="S3.T2.2.3.2" class="ltx_td ltx_align_center ltx_border_t">64.3(6.2)</td>
<td id="S3.T2.2.3.3" class="ltx_td ltx_align_center ltx_border_t">76.3(6.0)</td>
<td id="S3.T2.2.3.4" class="ltx_td ltx_align_center ltx_border_t">56.0(5.2)</td>
<td id="S3.T2.2.3.5" class="ltx_td ltx_align_center ltx_border_t">58.8(6.1)</td>
<td id="S3.T2.2.3.6" class="ltx_td ltx_align_center ltx_border_t">80.4(7.1)</td>
<td id="S3.T2.2.3.7" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">90.5(2.7)</td>
<td id="S3.T2.2.3.8" class="ltx_td ltx_align_center ltx_border_t">67.2(6.1)</td>
</tr>
<tr id="S3.T2.2.4" class="ltx_tr">
<td id="S3.T2.2.4.1" class="ltx_td ltx_align_left ltx_border_r">CCS</td>
<td id="S3.T2.2.4.2" class="ltx_td ltx_align_center">62.1(4.1)</td>
<td id="S3.T2.2.4.3" class="ltx_td ltx_align_center"><span id="S3.T2.2.4.3.1" class="ltx_text ltx_font_bold">78.5(3.8)</span></td>
<td id="S3.T2.2.4.4" class="ltx_td ltx_align_center"><span id="S3.T2.2.4.4.1" class="ltx_text ltx_font_bold">61.7(2.5)</span></td>
<td id="S3.T2.2.4.5" class="ltx_td ltx_align_center"><span id="S3.T2.2.4.5.1" class="ltx_text ltx_font_bold">71.5(3.0)</span></td>
<td id="S3.T2.2.4.6" class="ltx_td ltx_align_center"><span id="S3.T2.2.4.6.1" class="ltx_text ltx_font_bold">82.1(2.7)</span></td>
<td id="S3.T2.2.4.7" class="ltx_td ltx_align_center ltx_border_r">77.6(3.3)</td>
<td id="S3.T2.2.4.8" class="ltx_td ltx_align_center"><span id="S3.T2.2.4.8.1" class="ltx_text ltx_font_bold">71.2(3.2)</span></td>
</tr>
<tr id="S3.T2.2.5" class="ltx_tr">
<td id="S3.T2.2.5.1" class="ltx_td ltx_align_left ltx_border_r">CRC (TPC)</td>
<td id="S3.T2.2.5.2" class="ltx_td ltx_align_center"><span id="S3.T2.2.5.2.1" class="ltx_text ltx_font_bold">64.6(5.8)</span></td>
<td id="S3.T2.2.5.3" class="ltx_td ltx_align_center">76.5(5.7)</td>
<td id="S3.T2.2.5.4" class="ltx_td ltx_align_center">59.9(3.7)</td>
<td id="S3.T2.2.5.5" class="ltx_td ltx_align_center">66.7(5.0)</td>
<td id="S3.T2.2.5.6" class="ltx_td ltx_align_center">78.3(3.3)</td>
<td id="S3.T2.2.5.7" class="ltx_td ltx_align_center ltx_border_r">58.9(13.0)</td>
<td id="S3.T2.2.5.8" class="ltx_td ltx_align_center">69.2(4.7)</td>
</tr>
<tr id="S3.T2.2.6" class="ltx_tr">
<td id="S3.T2.2.6.1" class="ltx_td ltx_align_left ltx_border_r">CRC (BSS)</td>
<td id="S3.T2.2.6.2" class="ltx_td ltx_align_center">62.6(6.8)</td>
<td id="S3.T2.2.6.3" class="ltx_td ltx_align_center">76.8(5.3)</td>
<td id="S3.T2.2.6.4" class="ltx_td ltx_align_center">60.7(3.4)</td>
<td id="S3.T2.2.6.5" class="ltx_td ltx_align_center">69.7(3.3)</td>
<td id="S3.T2.2.6.6" class="ltx_td ltx_align_center">79.3(2.5)</td>
<td id="S3.T2.2.6.7" class="ltx_td ltx_align_center ltx_border_r">76.7(9.6)</td>
<td id="S3.T2.2.6.8" class="ltx_td ltx_align_center">69.8(4.3)</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 2: </span>
We compare CCS to two variants of Contrastive Representation Clustering: TPC, which clusters by projecting onto the top principal component, and BSS, which clusters by finding a direction that looks bimodal.
We show accuracy and standard deviation of each model averaged across all prompts and datasets, in the same setting as <a href="#S3.T1" title="In 3.1 Experimental Setup ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">1</span></a>.
We find that CCS generally performs the best, but that all methods are competitive with zero-shot.
</figcaption>
</figure>
</section>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Related Work</h2>

<div id="S4.p1" class="ltx_para ltx_noindent">
<p id="S4.p1.1" class="ltx_p"><span id="S4.p1.1.1" class="ltx_text ltx_font_bold">Zero-Shot Prompting.</span>
Since the release of GPT-3 <cite class="ltx_cite ltx_citemacro_citep">(Brown et al., <a href="#bib.bib7" title="" class="ltx_ref">2020</a>)</cite>, one of the main paradigms for eliciting what models know has been zero-shot prompting <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib26" title="" class="ltx_ref">2022</a>; Beltagy et al., <a href="#bib.bib3" title="" class="ltx_ref">2022</a>)</cite>.
Zero-shot exploits how language models are trained to predict diverse data from the internet, which incidentally includes tasks such as question-answering.
If prompted appropriately, this can be used to solve various useful tasks with reasonable performance <cite class="ltx_cite ltx_citemacro_citep">(Brown et al., <a href="#bib.bib7" title="" class="ltx_ref">2020</a>)</cite>.
However, these models are trained to imitate human-generated data, which bounds the quality of their outputs.</p>
</div>
<div id="S4.p2" class="ltx_para ltx_noindent">
<p id="S4.p2.1" class="ltx_p">Many methods improve upon vanilla zero-shot prompting <cite class="ltx_cite ltx_citemacro_citep">(Liu et al., <a href="#bib.bib26" title="" class="ltx_ref">2022</a>; Zhao et al., <a href="#bib.bib55" title="" class="ltx_ref">2021</a>; Lu et al., <a href="#bib.bib29" title="" class="ltx_ref">2022</a>; Wei et al., <a href="#bib.bib50" title="" class="ltx_ref">2022b</a>; Min et al., <a href="#bib.bib34" title="" class="ltx_ref">2022a</a>)</cite>.
While our goal is not to improve zero-shot performance, some of the ideas underlying these methods are similar to CCS.
Particularly relevant are methods that also leverage unsupervised consistency properties, such as <cite class="ltx_cite ltx_citemacro_citet">Jung et al. (<a href="#bib.bib18" title="" class="ltx_ref">2022</a>); Zhou et al. (<a href="#bib.bib57" title="" class="ltx_ref">2022</a>)</cite>.
However, these methods still bootstrap from language model outputs trained via imitation learning, which limits their applicability to our main goals.</p>
</div>
<div id="S4.p3" class="ltx_para ltx_noindent">
<p id="S4.p3.1" class="ltx_p">To illustrate this, imagine we train reinforcement learning agents to play a game such as Diplomacy <cite class="ltx_cite ltx_citemacro_citep">(FAIR et al., <a href="#bib.bib13" title="" class="ltx_ref">2022</a>)</cite>, in which players have incentives to lie to each other.
Then those agents may learn to lie in a way that is difficult to detect, but they may still internally represent whether they are lying.
Because their outputs would be deliberately misleading, standard zero-shot methods may be very unreliable.
In contrast, techniques like CCS may still be able to detect whether those models are lying by finding representations of truth in their activations that contradict their outputs.</p>
</div>
<div id="S4.p4" class="ltx_para ltx_noindent">
<p id="S4.p4.1" class="ltx_p"><span id="S4.p4.1.1" class="ltx_text ltx_font_bold">Truthfulness.</span>
There has been increasing interest in making language models truthful <cite class="ltx_cite ltx_citemacro_citep">(Evans et al., <a href="#bib.bib12" title="" class="ltx_ref">2021</a>; Lin et al., <a href="#bib.bib25" title="" class="ltx_ref">2022</a>)</cite>.
One aspect of truthfulness that has received substantial attention is factuality <cite class="ltx_cite ltx_citemacro_citep">(Thorne et al., <a href="#bib.bib46" title="" class="ltx_ref">2018</a>; Maynez et al., <a href="#bib.bib31" title="" class="ltx_ref">2020</a>)</cite>. For instance, many techniques aim to improve the factuality of models by augmenting them with retrieval methods <cite class="ltx_cite ltx_citemacro_citep">(Nakano et al., <a href="#bib.bib37" title="" class="ltx_ref">2021</a>; Menick et al., <a href="#bib.bib33" title="" class="ltx_ref">2022</a>)</cite>, which allows them to cite their sources.
In contrast, we focus on truthfulness more generally, which also includes procedural knowledge such as reasoning or natural language inference tasks.</p>
</div>
<div id="S4.p5" class="ltx_para ltx_noindent">
<p id="S4.p5.1" class="ltx_p">An approach to making language models truthful in this more general setting is to finetune them using either human demonstrations <cite class="ltx_cite ltx_citemacro_citep">(Khashabi et al., <a href="#bib.bib20" title="" class="ltx_ref">2020</a>; Sanh et al., <a href="#bib.bib44" title="" class="ltx_ref">2021</a>; Zhong et al., <a href="#bib.bib56" title="" class="ltx_ref">2021</a>; Wei et al., <a href="#bib.bib49" title="" class="ltx_ref">2022a</a>)</cite> or reinforcement learning from human feedback <cite class="ltx_cite ltx_citemacro_citep">(Christiano et al., <a href="#bib.bib9" title="" class="ltx_ref">2017</a>; Stiennon et al., <a href="#bib.bib45" title="" class="ltx_ref">2020</a>; Askell et al., <a href="#bib.bib1" title="" class="ltx_ref">2021</a>; Bai et al., <a href="#bib.bib2" title="" class="ltx_ref">2022</a>; Ouyang et al., <a href="#bib.bib38" title="" class="ltx_ref">2022</a>)</cite>.
These techniques have been widely successful at improving performance, but unlike our method they rely on being able to provide ground truth labels, which can be intractable in many settings.</p>
</div>
<div id="S4.p6" class="ltx_para ltx_noindent">
<p id="S4.p6.1" class="ltx_p">Some work has aimed to go beyond the direct supervision humans can provide by augmenting supervision with AI systems <cite class="ltx_cite ltx_citemacro_citep">(Christiano et al., <a href="#bib.bib10" title="" class="ltx_ref">2018</a>; Irving et al., <a href="#bib.bib17" title="" class="ltx_ref">2018</a>; Leike et al., <a href="#bib.bib24" title="" class="ltx_ref">2018</a>; Perez et al., <a href="#bib.bib39" title="" class="ltx_ref">2022</a>)</cite>.
This may expand the range of applications we can supervise, but many of these proposals remain theoretical, and it is unclear just how far these techniques can generalize.
<cite class="ltx_cite ltx_citemacro_citet">Christiano et al. (<a href="#bib.bib8" title="" class="ltx_ref">2022</a>)</cite> reframes this issue by posing the problem of Eliciting Latent Knowledge (ELK).
Like our problem statement, ELK is about eliciting knowledge from models even in cases where humans cannot evaluate that knowledge.
However, ELK frames this as a worst-case theoretical problem, while we frame this as an empirical problem that we can make progress on using current models.</p>
</div>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Discussion</h2>

<section id="S5.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.1 </span>Limitations and Future Work</h3>

<div id="S5.SS1.p1" class="ltx_para ltx_noindent">
<p id="S5.SS1.p1.1" class="ltx_p">Our work has a number of limitations.
First, CCS relies on the existence of a direction in activation space that separates true and false inputs well, in the sense that a supervised probe on the activations would be able to attain high accuracy (if it hypothetically had access to the ground truth labels).
This requires that a model is both <span id="S5.SS1.p1.1.1" class="ltx_text ltx_font_italic">capable</span> of evaluating the truth of a given input, and also that the model <span id="S5.SS1.p1.1.2" class="ltx_text ltx_font_italic">actively evaluates the truth</span> of that input.
It is not clear when these conditions hold precisely.</p>
</div>
<div id="S5.SS1.p2" class="ltx_para ltx_noindent">
<p id="S5.SS1.p2.1" class="ltx_p">Second, we did not evaluate our method on setups involving active “lying” or “deception” <cite class="ltx_cite ltx_citemacro_citep">(Kenton et al., <a href="#bib.bib19" title="" class="ltx_ref">2021</a>; Evans et al., <a href="#bib.bib12" title="" class="ltx_ref">2021</a>)</cite> by models, as we aren’t aware of existing evaluation setups for this setting.
If future work develops such a setup, a good stress test would be to apply CCS to do “lie detection” in that setting.
This may require modifications or extensions to the method, such as more explicitly ensuring that it recovers the truth of an input rather than what the model says.</p>
</div>
<div id="S5.SS1.p3" class="ltx_para ltx_noindent">
<p id="S5.SS1.p3.1" class="ltx_p">There are also various straightforward improvements to our method that one could explore.
This includes adding additional consistency constraints, improving its reliability, calibrating its probabilities, generalizing it beyond the yes-no question-answering setting, generalizing it to cases where answers aren’t clear-cut, and closing the remaining gap between CCS and the logistic regression ceiling.</p>
</div>
</section>
<section id="S5.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">5.2 </span>Conclusion</h3>

<div id="S5.SS2.p1" class="ltx_para ltx_noindent">
<p id="S5.SS2.p1.1" class="ltx_p">As language models become more capable, they will be increasingly used as components in larger AI systems trained with reinforcement learning.
As this occurs, falsehoods arising from misaligned training objectives may become more common, severe, and difficult to detect.
In principle, models may even develop instrumental incentives to lie: for example, if a human evaluator would disapprove of bad model behavior, models may learn to lie about their behavior to achieve higher reward.
If so, those models would be optimizing against human evaluators, making it precarious to rely on those evaluators for assessing the truth of what models say.
Because unsupervised methods for eliciting answers circumvent this issue, they may still work even in this scenario.
We show that it is possible to make progress on such methods today; the empirical success of our approach suggests that unsupervised methods are both a tractable and underexplored research direction.</p>
</div>
</section>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Acknowledgements</h2>

<div id="Sx1.p1" class="ltx_para ltx_noindent">
<p id="Sx1.p1.1" class="ltx_p">We are very grateful to Jared Kaplan for helpful experiment suggestions and resources early on in the project. We thank Beth Barnes and Paul Christiano for valuable discussions regarding the longer-term impacts of this work. We are also grateful to Jessy Lin, Alex Pan, Ruiqi Zhong, Yaodong Yu, the anonymous reviewers, and several others for useful feedback on earlier versions of this paper. CB is supported by an Open Philanthropy AI Fellowship.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Askell et al. (2021)</span>
<span class="ltx_bibblock">
Amanda Askell, Yushi Bai, Anna Chen, Dawn Drain, Deep Ganguli, T. J. Henighan,
Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac
Hatfield-Dodds, Danny Hernandez, John Kernion, Kamal Ndousse, Catherine
Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher
Olah, and Jared Kaplan.

</span>
<span class="ltx_bibblock">A general language assistant as a laboratory for alignment.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2112.00861, 2021.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bai et al. (2022)</span>
<span class="ltx_bibblock">
Yushi Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma,
Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph,
Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage,
Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna
Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B.
Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared
Kaplan.

</span>
<span class="ltx_bibblock">Training a helpful and harmless assistant with reinforcement learning
from human feedback.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2204.05862, 2022.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Beltagy et al. (2022)</span>
<span class="ltx_bibblock">
Iz Beltagy, Arman Cohan, Robert Logan IV, Sewon Min, and Sameer Singh.

</span>
<span class="ltx_bibblock">Zero- and few-shot nlp with pretrained language models.

</span>
<span class="ltx_bibblock">In <em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">ACL</em>, 2022.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bender et al. (2021)</span>
<span class="ltx_bibblock">
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret
Shmitchell.

</span>
<span class="ltx_bibblock">On the dangers of stochastic parrots: Can language models be too big?

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2021 ACM Conference on Fairness,
Accountability, and Transparency</em>, 2021.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bisk et al. (2020)</span>
<span class="ltx_bibblock">
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al.

</span>
<span class="ltx_bibblock">Piqa: Reasoning about physical commonsense in natural language.

</span>
<span class="ltx_bibblock">In <em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">Proceedings of the AAAI Conference on Artificial
Intelligence</em>, volume 34, pp.  7432–7439, 2020.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bommasani et al. (2021)</span>
<span class="ltx_bibblock">
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney
von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma
Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon,
Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora
Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John
Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale,
Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha,
Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong,
Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha
Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab,
Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar,
Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa
Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P.
Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan,
Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed
Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou,
Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi
Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo
Ruiz, Jack Ryan, Christopher R’e, Dorsa Sadigh, Shiori Sagawa, Keshav
Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori,
Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu,
Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You,
Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang,
Lucia Zheng, Kaitlyn Zhou, and Percy Liang.

</span>
<span class="ltx_bibblock">On the opportunities and risks of foundation models.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2108.07258, 2021.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Brown et al. (2020)</span>
<span class="ltx_bibblock">
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan,
Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J.
Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens
Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott
Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec
Radford, Ilya Sutskever, and Dario Amodei.

</span>
<span class="ltx_bibblock">Language models are few-shot learners.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2005.14165, 2020.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Christiano et al. (2022)</span>
<span class="ltx_bibblock">
Paul Christiano, Ajeya Cotra, and Mark Xu.

</span>
<span class="ltx_bibblock">Eliciting latent knowledge.

</span>
<span class="ltx_bibblock">Technical report, ARC, 2022.

</span>
<span class="ltx_bibblock">URL
<a target="_blank" href="https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit</a>.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Christiano et al. (2017)</span>
<span class="ltx_bibblock">
Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg,
and Dario Amodei.

</span>
<span class="ltx_bibblock">Deep reinforcement learning from human preferences.

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/1706.03741, 2017.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Christiano et al. (2018)</span>
<span class="ltx_bibblock">
Paul Francis Christiano, Buck Shlegeris, and Dario Amodei.

</span>
<span class="ltx_bibblock">Supervising strong learners by amplifying weak experts.

</span>
<span class="ltx_bibblock"><em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/1810.08575, 2018.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Clark et al. (2019)</span>
<span class="ltx_bibblock">
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael
Collins, and Kristina Toutanova.

</span>
<span class="ltx_bibblock">Boolq: Exploring the surprising difficulty of natural yes/no
questions.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1905.10044</em>, 2019.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Evans et al. (2021)</span>
<span class="ltx_bibblock">
Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit,
Peter Wills, Luca Righetti, and William Saunders.

</span>
<span class="ltx_bibblock">Truthful ai: Developing and governing ai that does not lie.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2110.06674, 2021.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">FAIR et al. (2022)</span>
<span class="ltx_bibblock">
Meta FAIR, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin
Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul
Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis,
Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk
Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus
Zijlstra.

</span>
<span class="ltx_bibblock">Human-level play in the game of diplomacy by combining language
models with strategic reasoning.

</span>
<span class="ltx_bibblock"><em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Science</em>, pp.  eade9097, 2022.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fisher (1936)</span>
<span class="ltx_bibblock">
Rory A. Fisher.

</span>
<span class="ltx_bibblock">The use of multiple measurements in taxonomic problems.

</span>
<span class="ltx_bibblock"><em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">Annals of Human Genetics</em>, 7:179–188, 1936.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">He et al. (2021)</span>
<span class="ltx_bibblock">
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen.

</span>
<span class="ltx_bibblock">Deberta: Decoding-enhanced bert with disentangled attention.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2006.03654, 2021.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hendrycks et al. (2021)</span>
<span class="ltx_bibblock">
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt.

</span>
<span class="ltx_bibblock">Unsolved problems in ml safety.

</span>
<span class="ltx_bibblock"><em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2109.13916, 2021.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Irving et al. (2018)</span>
<span class="ltx_bibblock">
Geoffrey Irving, Paul Francis Christiano, and Dario Amodei.

</span>
<span class="ltx_bibblock">Ai safety via debate.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/1805.00899, 2018.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jung et al. (2022)</span>
<span class="ltx_bibblock">
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula,
Ronan Le Bras, and Yejin Choi.

</span>
<span class="ltx_bibblock">Maieutic prompting: Logically consistent reasoning with recursive
explanations.

</span>
<span class="ltx_bibblock"><em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2205.11822, 2022.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kenton et al. (2021)</span>
<span class="ltx_bibblock">
Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik,
and Geoffrey Irving.

</span>
<span class="ltx_bibblock">Alignment of language agents.

</span>
<span class="ltx_bibblock"><em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2103.14659, 2021.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Khashabi et al. (2020)</span>
<span class="ltx_bibblock">
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord,
Peter Clark, and Hannaneh Hajishirzi.

</span>
<span class="ltx_bibblock">Unifiedqa: Crossing format boundaries with a single qa system.

</span>
<span class="ltx_bibblock">In <em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">FINDINGS</em>, 2020.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kim et al. (2022)</span>
<span class="ltx_bibblock">
Junyeob Kim, Hyuhng Joon Kim, Hyunsoo Cho, Hwiyeol Jo, Sang-Woo Lee, Sang goo
Lee, Kang Min Yoo, and Taeuk Kim.

</span>
<span class="ltx_bibblock">Ground-truth labels matter: A deeper look into input-label
demonstrations.

</span>
<span class="ltx_bibblock"><em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2205.12685, 2022.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kingma &amp; Ba (2014)</span>
<span class="ltx_bibblock">
Diederik P Kingma and Jimmy Ba.

</span>
<span class="ltx_bibblock">Adam: A method for stochastic optimization.

</span>
<span class="ltx_bibblock"><em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1412.6980</em>, 2014.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lehmann et al. (2015)</span>
<span class="ltx_bibblock">
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas,
Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef,
Sören Auer, et al.

</span>
<span class="ltx_bibblock">Dbpedia–a large-scale, multilingual knowledge base extracted from
wikipedia.

</span>
<span class="ltx_bibblock"><em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">Semantic web</em>, 6(2):167–195, 2015.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Leike et al. (2018)</span>
<span class="ltx_bibblock">
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane
Legg.

</span>
<span class="ltx_bibblock">Scalable agent alignment via reward modeling: a research direction.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/1811.07871, 2018.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin et al. (2022)</span>
<span class="ltx_bibblock">
Stephanie C. Lin, Jacob Hilton, and Owain Evans.

</span>
<span class="ltx_bibblock">Truthfulqa: Measuring how models mimic human falsehoods.

</span>
<span class="ltx_bibblock">In <em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">ACL</em>, 2022.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2022)</span>
<span class="ltx_bibblock">
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and
Graham Neubig.

</span>
<span class="ltx_bibblock">Pre-train, prompt, and predict: A systematic survey of prompting
methods in natural language processing.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.1.1" class="ltx_emph ltx_font_italic">ACM Computing Surveys (CSUR)</em>, 2022.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al. (2019)</span>
<span class="ltx_bibblock">
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.

</span>
<span class="ltx_bibblock">Roberta: A robustly optimized bert pretraining approach.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/1907.11692, 2019.

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Loshchilov &amp; Hutter (2017)</span>
<span class="ltx_bibblock">
Ilya Loshchilov and Frank Hutter.

</span>
<span class="ltx_bibblock">Fixing weight decay regularization in adam.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/1711.05101, 2017.

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lu et al. (2022)</span>
<span class="ltx_bibblock">
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp.

</span>
<span class="ltx_bibblock">Fantastically ordered prompts and where to find them: Overcoming
few-shot prompt order sensitivity.

</span>
<span class="ltx_bibblock">In <em id="bib.bib29.1.1" class="ltx_emph ltx_font_italic">ACL</em>, 2022.

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Maas et al. (2011)</span>
<span class="ltx_bibblock">
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and
Christopher Potts.

</span>
<span class="ltx_bibblock">Learning word vectors for sentiment analysis.

</span>
<span class="ltx_bibblock">In <em id="bib.bib30.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 49th annual meeting of the association
for computational linguistics: Human language technologies</em>, pp.  142–150,
2011.

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Maynez et al. (2020)</span>
<span class="ltx_bibblock">
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald.

</span>
<span class="ltx_bibblock">On faithfulness and factuality in abstractive summarization.

</span>
<span class="ltx_bibblock"><em id="bib.bib31.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2005.00661, 2020.

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">McAuley &amp; Leskovec (2013)</span>
<span class="ltx_bibblock">
Julian McAuley and Jure Leskovec.

</span>
<span class="ltx_bibblock">Hidden factors and hidden topics: understanding rating dimensions
with review text.

</span>
<span class="ltx_bibblock">In <em id="bib.bib32.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 7th ACM conference on Recommender
systems</em>, pp.  165–172, 2013.

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Menick et al. (2022)</span>
<span class="ltx_bibblock">
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song,
Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham,
Geoffrey Irving, and Nathan McAleese.

</span>
<span class="ltx_bibblock">Teaching language models to support answers with verified quotes.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2203.11147, 2022.

</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Min et al. (2022a)</span>
<span class="ltx_bibblock">
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi.

</span>
<span class="ltx_bibblock">Metaicl: Learning to learn in context.

</span>
<span class="ltx_bibblock"><em id="bib.bib34.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2110.15943, 2022a.

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Min et al. (2022b)</span>
<span class="ltx_bibblock">
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh
Hajishirzi, and Luke Zettlemoyer.

</span>
<span class="ltx_bibblock">Rethinking the role of demonstrations: What makes in-context learning
work?

</span>
<span class="ltx_bibblock"><em id="bib.bib35.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2202.12837, 2022b.

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mostafazadeh et al. (2017)</span>
<span class="ltx_bibblock">
Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James
Allen.

</span>
<span class="ltx_bibblock">Lsdsem 2017 shared task: The story cloze test.

</span>
<span class="ltx_bibblock">In <em id="bib.bib36.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2nd Workshop on Linking Models of
Lexical, Sentential and Discourse-level Semantics</em>, pp.  46–51, 2017.

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nakano et al. (2021)</span>
<span class="ltx_bibblock">
Reiichiro Nakano, Jacob Hilton, S. Arun Balaji, Jeff Wu, Long Ouyang, Christina
Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders,
Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew
Knight, Benjamin Chess, and John Schulman.

</span>
<span class="ltx_bibblock">Webgpt: Browser-assisted question-answering with human feedback.

</span>
<span class="ltx_bibblock"><em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2112.09332, 2021.

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ouyang et al. (2022)</span>
<span class="ltx_bibblock">
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela
Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John
Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda
Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe.

</span>
<span class="ltx_bibblock">Training language models to follow instructions with human feedback.

</span>
<span class="ltx_bibblock"><em id="bib.bib38.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2203.02155, 2022.

</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Perez et al. (2022)</span>
<span class="ltx_bibblock">
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John
Aslanides, Amelia Glaese, Nathan McAleese, and Geoffrey Irving.

</span>
<span class="ltx_bibblock">Red teaming language models with language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2202.03286, 2022.

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Raffel et al. (2020)</span>
<span class="ltx_bibblock">
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang,
Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.

</span>
<span class="ltx_bibblock">Exploring the limits of transfer learning with a unified text-to-text
transformer.

</span>
<span class="ltx_bibblock"><em id="bib.bib40.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/1910.10683, 2020.

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rajpurkar et al. (2016)</span>
<span class="ltx_bibblock">
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.

</span>
<span class="ltx_bibblock">Squad: 100,000+ questions for machine comprehension of text.

</span>
<span class="ltx_bibblock"><em id="bib.bib41.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1606.05250</em>, 2016.

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Roemmele et al. (2011)</span>
<span class="ltx_bibblock">
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon.

</span>
<span class="ltx_bibblock">Choice of plausible alternatives: An evaluation of commonsense causal
reasoning.

</span>
<span class="ltx_bibblock">In <em id="bib.bib42.1.1" class="ltx_emph ltx_font_italic">2011 AAAI Spring Symposium Series</em>, 2011.

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Roller et al. (2021)</span>
<span class="ltx_bibblock">
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu,
Jing Xu, Myle Ott, Kurt Shuster, Eric Michael Smith, Y.-Lan Boureau, and
Jason Weston.

</span>
<span class="ltx_bibblock">Recipes for building an open-domain chatbot.

</span>
<span class="ltx_bibblock">In <em id="bib.bib43.1.1" class="ltx_emph ltx_font_italic">EACL</em>, 2021.

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sanh et al. (2021)</span>
<span class="ltx_bibblock">
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang A. Sutawika,
Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja,
Manan Dey, M SAIFUL BARI, Canwen Xu, Urmish Thakker, Shanya Sharma, Eliza
Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta,
Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen,
Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj,
Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan
Fries, Ryan Teehan, Stella Rose Biderman, Leo Gao, T. G. Owe Bers, Thomas
Wolf, and Alexander M. Rush.

</span>
<span class="ltx_bibblock">Multitask prompted training enables zero-shot task generalization.

</span>
<span class="ltx_bibblock"><em id="bib.bib44.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2110.08207, 2021.

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Stiennon et al. (2020)</span>
<span class="ltx_bibblock">
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea
Voss, Alec Radford, Dario Amodei, and Paul Christiano.

</span>
<span class="ltx_bibblock">Learning to summarize from human feedback.

</span>
<span class="ltx_bibblock"><em id="bib.bib45.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2009.01325, 2020.

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Thorne et al. (2018)</span>
<span class="ltx_bibblock">
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal.

</span>
<span class="ltx_bibblock">Fever: a large-scale dataset for fact extraction and verification.

</span>
<span class="ltx_bibblock"><em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/1803.05355, 2018.

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang et al. (2018)</span>
<span class="ltx_bibblock">
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R
Bowman.

</span>
<span class="ltx_bibblock">Glue: A multi-task benchmark and analysis platform for natural
language understanding.

</span>
<span class="ltx_bibblock"><em id="bib.bib47.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:1804.07461</em>, 2018.

</span>
</li>
<li id="bib.bib48" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang &amp; Komatsuzaki (2021)</span>
<span class="ltx_bibblock">
Ben Wang and Aran Komatsuzaki.

</span>
<span class="ltx_bibblock">GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/kingoflolz/mesh-transformer-jax" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/kingoflolz/mesh-transformer-jax</a>, May 2021.

</span>
</li>
<li id="bib.bib49" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2022a)</span>
<span class="ltx_bibblock">
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester,
Nan Du, Andrew M. Dai, and Quoc V. Le.

</span>
<span class="ltx_bibblock">Finetuned language models are zero-shot learners.

</span>
<span class="ltx_bibblock"><em id="bib.bib49.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2109.01652, 2022a.

</span>
</li>
<li id="bib.bib50" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wei et al. (2022b)</span>
<span class="ltx_bibblock">
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and
Denny Zhou.

</span>
<span class="ltx_bibblock">Chain of thought prompting elicits reasoning in large language
models.

</span>
<span class="ltx_bibblock"><em id="bib.bib50.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2201.11903, 2022b.

</span>
</li>
<li id="bib.bib51" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Weidinger et al. (2021)</span>
<span class="ltx_bibblock">
Laura Weidinger, John F. J. Mellor, Maribeth Rauh, Conor Griffin, Jonathan
Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh,
Zachary Kenton, Sande Minnich Brown, William T. Hawkins, Tom Stepleton,
Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks,
William S. Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel.

</span>
<span class="ltx_bibblock">Ethical and social risks of harm from language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib51.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2112.04359, 2021.

</span>
</li>
<li id="bib.bib52" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wolf et al. (2019)</span>
<span class="ltx_bibblock">
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue,
Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and
Jamie Brew.

</span>
<span class="ltx_bibblock">Huggingface’s transformers: State-of-the-art natural language
processing.

</span>
<span class="ltx_bibblock"><em id="bib.bib52.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/1910.03771, 2019.

</span>
</li>
<li id="bib.bib53" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yin et al. (2020)</span>
<span class="ltx_bibblock">
Wenpeng Yin, Nazneen Rajani, Dragomir Radev, Richard Socher, and Caiming Xiong.

</span>
<span class="ltx_bibblock">Universal natural language processing with limited annotations: Try
few-shot textual entailment as a start.

</span>
<span class="ltx_bibblock">In <em id="bib.bib53.1.1" class="ltx_emph ltx_font_italic">EMNLP</em>, 2020.

</span>
</li>
<li id="bib.bib54" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhang et al. (2015)</span>
<span class="ltx_bibblock">
Xiang Zhang, Junbo Zhao, and Yann LeCun.

</span>
<span class="ltx_bibblock">Character-level convolutional networks for text classification.

</span>
<span class="ltx_bibblock"><em id="bib.bib54.1.1" class="ltx_emph ltx_font_italic">Advances in neural information processing systems</em>,
28:649–657, 2015.

</span>
</li>
<li id="bib.bib55" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhao et al. (2021)</span>
<span class="ltx_bibblock">
Tony Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh.

</span>
<span class="ltx_bibblock">Calibrate before use: Improving few-shot performance of language
models.

</span>
<span class="ltx_bibblock"><em id="bib.bib55.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2102.09690, 2021.

</span>
</li>
<li id="bib.bib56" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhong et al. (2021)</span>
<span class="ltx_bibblock">
Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein.

</span>
<span class="ltx_bibblock">Adapting language models for zero-shot learning by meta-tuning on
dataset and prompt collections.

</span>
<span class="ltx_bibblock">In <em id="bib.bib56.1.1" class="ltx_emph ltx_font_italic">EMNLP</em>, 2021.

</span>
</li>
<li id="bib.bib57" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zhou et al. (2022)</span>
<span class="ltx_bibblock">
Chunting Zhou, Junxian He, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham
Neubig.

</span>
<span class="ltx_bibblock">Prompt consistency for zero-shot task generalization.

</span>
<span class="ltx_bibblock"><em id="bib.bib57.1.1" class="ltx_emph ltx_font_italic">ArXiv</em>, abs/2205.00049, 2022.

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Identifying Clusters</h2>

<div id="A1.p1" class="ltx_para ltx_noindent">
<p id="A1.p1.1" class="ltx_p">CCS predicts yes-no answers to a set of unlabeled questions <math id="A1.p1.1.m1.3" class="ltx_Math" alttext="q_{1},\ldots,q_{n}" display="inline"><semantics id="A1.p1.1.m1.3a"><mrow id="A1.p1.1.m1.3.3.2" xref="A1.p1.1.m1.3.3.3.cmml"><msub id="A1.p1.1.m1.2.2.1.1" xref="A1.p1.1.m1.2.2.1.1.cmml"><mi id="A1.p1.1.m1.2.2.1.1.2" xref="A1.p1.1.m1.2.2.1.1.2.cmml">q</mi><mn id="A1.p1.1.m1.2.2.1.1.3" xref="A1.p1.1.m1.2.2.1.1.3.cmml">1</mn></msub><mo id="A1.p1.1.m1.3.3.2.3" xref="A1.p1.1.m1.3.3.3.cmml">,</mo><mi mathvariant="normal" id="A1.p1.1.m1.1.1" xref="A1.p1.1.m1.1.1.cmml">…</mi><mo id="A1.p1.1.m1.3.3.2.4" xref="A1.p1.1.m1.3.3.3.cmml">,</mo><msub id="A1.p1.1.m1.3.3.2.2" xref="A1.p1.1.m1.3.3.2.2.cmml"><mi id="A1.p1.1.m1.3.3.2.2.2" xref="A1.p1.1.m1.3.3.2.2.2.cmml">q</mi><mi id="A1.p1.1.m1.3.3.2.2.3" xref="A1.p1.1.m1.3.3.2.2.3.cmml">n</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A1.p1.1.m1.3b"><list id="A1.p1.1.m1.3.3.3.cmml" xref="A1.p1.1.m1.3.3.2"><apply id="A1.p1.1.m1.2.2.1.1.cmml" xref="A1.p1.1.m1.2.2.1.1"><csymbol cd="ambiguous" id="A1.p1.1.m1.2.2.1.1.1.cmml" xref="A1.p1.1.m1.2.2.1.1">subscript</csymbol><ci id="A1.p1.1.m1.2.2.1.1.2.cmml" xref="A1.p1.1.m1.2.2.1.1.2">𝑞</ci><cn type="integer" id="A1.p1.1.m1.2.2.1.1.3.cmml" xref="A1.p1.1.m1.2.2.1.1.3">1</cn></apply><ci id="A1.p1.1.m1.1.1.cmml" xref="A1.p1.1.m1.1.1">…</ci><apply id="A1.p1.1.m1.3.3.2.2.cmml" xref="A1.p1.1.m1.3.3.2.2"><csymbol cd="ambiguous" id="A1.p1.1.m1.3.3.2.2.1.cmml" xref="A1.p1.1.m1.3.3.2.2">subscript</csymbol><ci id="A1.p1.1.m1.3.3.2.2.2.cmml" xref="A1.p1.1.m1.3.3.2.2.2">𝑞</ci><ci id="A1.p1.1.m1.3.3.2.2.3.cmml" xref="A1.p1.1.m1.3.3.2.2.3">𝑛</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="A1.p1.1.m1.3c">q_{1},\ldots,q_{n}</annotation></semantics></math>.
However, it does not identify which prediction label corresponds to true and which corresponds to false.
We now describe how one can do so in principle, as long as a model is also able to take conjunctions.</p>
</div>
<div id="A1.p2" class="ltx_para ltx_noindent">
<p id="A1.p2.4" class="ltx_p">Suppose we run our method and find, for example, that <math id="A1.p2.1.m1.1" class="ltx_Math" alttext="x" display="inline"><semantics id="A1.p2.1.m1.1a"><mi id="A1.p2.1.m1.1.1" xref="A1.p2.1.m1.1.1.cmml">x</mi><annotation-xml encoding="MathML-Content" id="A1.p2.1.m1.1b"><ci id="A1.p2.1.m1.1.1.cmml" xref="A1.p2.1.m1.1.1">𝑥</ci></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.1.m1.1c">x</annotation></semantics></math> and <math id="A1.p2.2.m2.1" class="ltx_Math" alttext="x^{\prime}" display="inline"><semantics id="A1.p2.2.m2.1a"><msup id="A1.p2.2.m2.1.1" xref="A1.p2.2.m2.1.1.cmml"><mi id="A1.p2.2.m2.1.1.2" xref="A1.p2.2.m2.1.1.2.cmml">x</mi><mo id="A1.p2.2.m2.1.1.3" xref="A1.p2.2.m2.1.1.3.cmml">′</mo></msup><annotation-xml encoding="MathML-Content" id="A1.p2.2.m2.1b"><apply id="A1.p2.2.m2.1.1.cmml" xref="A1.p2.2.m2.1.1"><csymbol cd="ambiguous" id="A1.p2.2.m2.1.1.1.cmml" xref="A1.p2.2.m2.1.1">superscript</csymbol><ci id="A1.p2.2.m2.1.1.2.cmml" xref="A1.p2.2.m2.1.1.2">𝑥</ci><ci id="A1.p2.2.m2.1.1.3.cmml" xref="A1.p2.2.m2.1.1.3">′</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.2.m2.1c">x^{\prime}</annotation></semantics></math> end up in different clusters.
Then we know that these two statements have opposite truth values.
As a result, if we take the conjunction of these two statements, <math id="A1.p2.3.m3.1" class="ltx_Math" alttext="x\wedge x^{\prime}" display="inline"><semantics id="A1.p2.3.m3.1a"><mrow id="A1.p2.3.m3.1.1" xref="A1.p2.3.m3.1.1.cmml"><mi id="A1.p2.3.m3.1.1.2" xref="A1.p2.3.m3.1.1.2.cmml">x</mi><mo id="A1.p2.3.m3.1.1.1" xref="A1.p2.3.m3.1.1.1.cmml">∧</mo><msup id="A1.p2.3.m3.1.1.3" xref="A1.p2.3.m3.1.1.3.cmml"><mi id="A1.p2.3.m3.1.1.3.2" xref="A1.p2.3.m3.1.1.3.2.cmml">x</mi><mo id="A1.p2.3.m3.1.1.3.3" xref="A1.p2.3.m3.1.1.3.3.cmml">′</mo></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.p2.3.m3.1b"><apply id="A1.p2.3.m3.1.1.cmml" xref="A1.p2.3.m3.1.1"><and id="A1.p2.3.m3.1.1.1.cmml" xref="A1.p2.3.m3.1.1.1"></and><ci id="A1.p2.3.m3.1.1.2.cmml" xref="A1.p2.3.m3.1.1.2">𝑥</ci><apply id="A1.p2.3.m3.1.1.3.cmml" xref="A1.p2.3.m3.1.1.3"><csymbol cd="ambiguous" id="A1.p2.3.m3.1.1.3.1.cmml" xref="A1.p2.3.m3.1.1.3">superscript</csymbol><ci id="A1.p2.3.m3.1.1.3.2.cmml" xref="A1.p2.3.m3.1.1.3.2">𝑥</ci><ci id="A1.p2.3.m3.1.1.3.3.cmml" xref="A1.p2.3.m3.1.1.3.3">′</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.3.m3.1c">x\wedge x^{\prime}</annotation></semantics></math> then it should be false, and if we take the disjunction, <math id="A1.p2.4.m4.1" class="ltx_Math" alttext="x\vee x^{\prime}" display="inline"><semantics id="A1.p2.4.m4.1a"><mrow id="A1.p2.4.m4.1.1" xref="A1.p2.4.m4.1.1.cmml"><mi id="A1.p2.4.m4.1.1.2" xref="A1.p2.4.m4.1.1.2.cmml">x</mi><mo id="A1.p2.4.m4.1.1.1" xref="A1.p2.4.m4.1.1.1.cmml">∨</mo><msup id="A1.p2.4.m4.1.1.3" xref="A1.p2.4.m4.1.1.3.cmml"><mi id="A1.p2.4.m4.1.1.3.2" xref="A1.p2.4.m4.1.1.3.2.cmml">x</mi><mo id="A1.p2.4.m4.1.1.3.3" xref="A1.p2.4.m4.1.1.3.3.cmml">′</mo></msup></mrow><annotation-xml encoding="MathML-Content" id="A1.p2.4.m4.1b"><apply id="A1.p2.4.m4.1.1.cmml" xref="A1.p2.4.m4.1.1"><or id="A1.p2.4.m4.1.1.1.cmml" xref="A1.p2.4.m4.1.1.1"></or><ci id="A1.p2.4.m4.1.1.2.cmml" xref="A1.p2.4.m4.1.1.2">𝑥</ci><apply id="A1.p2.4.m4.1.1.3.cmml" xref="A1.p2.4.m4.1.1.3"><csymbol cd="ambiguous" id="A1.p2.4.m4.1.1.3.1.cmml" xref="A1.p2.4.m4.1.1.3">superscript</csymbol><ci id="A1.p2.4.m4.1.1.3.2.cmml" xref="A1.p2.4.m4.1.1.3.2">𝑥</ci><ci id="A1.p2.4.m4.1.1.3.3.cmml" xref="A1.p2.4.m4.1.1.3.3">′</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.p2.4.m4.1c">x\vee x^{\prime}</annotation></semantics></math>, then it should be true.
This allows us to identify which cluster is true and which is false in a completely unsupervised way.</p>
</div>
<figure id="A1.F4" class="ltx_figure"><img src="/html/2212.03827/assets/x4.png" id="A1.F4.g1" class="ltx_graphics ltx_centering ltx_img_square" width="211" height="213" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>
Zero-shot accuracy drop caused by the misleading prefix shown in <a href="#A2.F5" title="In B.1.1 Evaluating The Effect of The Misleading Prefix ‣ B.1 Misleading Prefix ‣ Appendix B Misleading Prefix Details ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">5</span></a> for UnifiedQA. Each point corresponds to one dataset and one prompt.
On average, zero-shot accuracy drops <math id="A1.F4.3.m1.1" class="ltx_Math" alttext="9.5\%" display="inline"><semantics id="A1.F4.3.m1.1b"><mrow id="A1.F4.3.m1.1.1" xref="A1.F4.3.m1.1.1.cmml"><mn id="A1.F4.3.m1.1.1.2" xref="A1.F4.3.m1.1.1.2.cmml">9.5</mn><mo id="A1.F4.3.m1.1.1.1" xref="A1.F4.3.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.F4.3.m1.1c"><apply id="A1.F4.3.m1.1.1.cmml" xref="A1.F4.3.m1.1.1"><csymbol cd="latexml" id="A1.F4.3.m1.1.1.1.cmml" xref="A1.F4.3.m1.1.1.1">percent</csymbol><cn type="float" id="A1.F4.3.m1.1.1.2.cmml" xref="A1.F4.3.m1.1.1.2">9.5</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F4.3.m1.1d">9.5\%</annotation></semantics></math>, while CCS performs <math id="A1.F4.4.m2.1" class="ltx_Math" alttext="1.7\%" display="inline"><semantics id="A1.F4.4.m2.1b"><mrow id="A1.F4.4.m2.1.1" xref="A1.F4.4.m2.1.1.cmml"><mn id="A1.F4.4.m2.1.1.2" xref="A1.F4.4.m2.1.1.2.cmml">1.7</mn><mo id="A1.F4.4.m2.1.1.1" xref="A1.F4.4.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.F4.4.m2.1c"><apply id="A1.F4.4.m2.1.1.cmml" xref="A1.F4.4.m2.1.1"><csymbol cd="latexml" id="A1.F4.4.m2.1.1.1.cmml" xref="A1.F4.4.m2.1.1.1">percent</csymbol><cn type="float" id="A1.F4.4.m2.1.1.2.cmml" xref="A1.F4.4.m2.1.1.2">1.7</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F4.4.m2.1d">1.7\%</annotation></semantics></math> better.
This suggests that the latent knowledge inside a model that CCS recovers can be robust even when model outputs become less reliable.
</figcaption>
</figure>
</section>
<section id="A2" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix B </span>Misleading Prefix Details</h2>

<section id="A2.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">B.1 </span>Misleading Prefix</h3>

<div id="A2.SS1.p1" class="ltx_para ltx_noindent">
<p id="A2.SS1.p1.1" class="ltx_p">In <a href="#S3.SS2" title="3.2 Evaluating CCS ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">3.2</span></a>, we mentioned that model outputs can be biased by using the misleading few-shot prefix shown in <a href="#A2.F5" title="In B.1.1 Evaluating The Effect of The Misleading Prefix ‣ B.1 Misleading Prefix ‣ Appendix B Misleading Prefix Details ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">5</span></a>.
We show the effect of the misleading prefix for all models in <a href="#A2.T3" title="In B.1.1 Evaluating The Effect of The Misleading Prefix ‣ B.1 Misleading Prefix ‣ Appendix B Misleading Prefix Details ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">3</span></a>, and we illustrate the drop in accuracy for each dataset and prompt in <a href="#A1.F4" title="In Appendix A Identifying Clusters ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">4</span></a>.</p>
</div>
<section id="A2.SS1.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">B.1.1 </span>Evaluating The Effect of The Misleading Prefix</h4>

<div id="A2.SS1.SSS1.p1" class="ltx_para ltx_noindent">
<p id="A2.SS1.SSS1.p1.1" class="ltx_p">The hope is that the misleading prefix causes models to imitate the sorts of incorrect answers in the prefix.
However, we found that results are qualitatively similar with correct answers, making the actual interpretation of the effect of this prefix ambiguous.
Nevertheless, to better understand what this prefix is doing, we visualize the top <math id="A2.SS1.SSS1.p1.1.m1.1" class="ltx_Math" alttext="100" display="inline"><semantics id="A2.SS1.SSS1.p1.1.m1.1a"><mn id="A2.SS1.SSS1.p1.1.m1.1.1" xref="A2.SS1.SSS1.p1.1.m1.1.1.cmml">100</mn><annotation-xml encoding="MathML-Content" id="A2.SS1.SSS1.p1.1.m1.1b"><cn type="integer" id="A2.SS1.SSS1.p1.1.m1.1.1.cmml" xref="A2.SS1.SSS1.p1.1.m1.1.1">100</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.SS1.SSS1.p1.1.m1.1c">100</annotation></semantics></math> tokens for both this misleading prefix and no prefix on average across all examples for UnifiedQA on the IMDB sentiment dataset, and show the results in <a href="#A2.F6" title="In B.1.1 Evaluating The Effect of The Misleading Prefix ‣ B.1 Misleading Prefix ‣ Appendix B Misleading Prefix Details ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">6</span></a>.
Without this prefix, the top two tokens are the actual labels (‘positive’ and ‘negative’). In contrast, with the prefix, these tokens have lower probabilities relative to other tokens, with the highest probability tokens instead being irrelevant to the true answer.
For example, since the prefix includes several answers with numbers, many of the top tokens are also numbers.
This provides some evidence that the prefix is actually causing the model to output false text by imitating its context in a meaningful sense.</p>
</div>
<figure id="A2.T3" class="ltx_table">
<table id="A2.T3.2" class="ltx_tabular ltx_centering ltx_align_middle">
<tr id="A2.T3.2.2" class="ltx_tr">
<td id="A2.T3.2.2.3" class="ltx_td ltx_align_left">Method</td>
<td id="A2.T3.2.2.4" class="ltx_td ltx_align_left ltx_border_r">Prefix</td>
<td id="A2.T3.2.2.5" class="ltx_td ltx_align_center">RoBERTa</td>
<td id="A2.T3.2.2.6" class="ltx_td ltx_align_center">DeBERTa</td>
<td id="A2.T3.2.2.7" class="ltx_td ltx_align_center">GPT-J</td>
<td id="A2.T3.2.2.8" class="ltx_td ltx_align_center">T5</td>
<td id="A2.T3.2.2.9" class="ltx_td ltx_align_center">UQA</td>
<td id="A2.T3.1.1.1" class="ltx_td ltx_align_center ltx_border_r">T0<sup id="A2.T3.1.1.1.1" class="ltx_sup">∗</sup>
</td>
<td id="A2.T3.2.2.2" class="ltx_td ltx_align_center">Mean<sup id="A2.T3.2.2.2.1" class="ltx_sup">∗</sup>
</td>
</tr>
<tr id="A2.T3.2.3" class="ltx_tr">
<td id="A2.T3.2.3.1" class="ltx_td ltx_align_left ltx_border_t" rowspan="2"><span id="A2.T3.2.3.1.1" class="ltx_text"><span id="A2.T3.2.3.1.1.1" class="ltx_text"></span><span id="A2.T3.2.3.1.1.2" class="ltx_text">
<span id="A2.T3.2.3.1.1.2.1" class="ltx_tabular ltx_align_middle">
<span id="A2.T3.2.3.1.1.2.1.1" class="ltx_tr">
<span id="A2.T3.2.3.1.1.2.1.1.1" class="ltx_td ltx_nopad_r ltx_align_left">Calibrated</span></span>
<span id="A2.T3.2.3.1.1.2.1.2" class="ltx_tr">
<span id="A2.T3.2.3.1.1.2.1.2.1" class="ltx_td ltx_nopad_r ltx_align_left">0-shot</span></span>
</span></span> <span id="A2.T3.2.3.1.1.3" class="ltx_text"></span></span></td>
<td id="A2.T3.2.3.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Regular</td>
<td id="A2.T3.2.3.3" class="ltx_td ltx_align_center ltx_border_t">64.3(6.2)</td>
<td id="A2.T3.2.3.4" class="ltx_td ltx_align_center ltx_border_t">76.3(6.0)</td>
<td id="A2.T3.2.3.5" class="ltx_td ltx_align_center ltx_border_t">56.0(5.2)</td>
<td id="A2.T3.2.3.6" class="ltx_td ltx_align_center ltx_border_t">58.8(6.1)</td>
<td id="A2.T3.2.3.7" class="ltx_td ltx_align_center ltx_border_t">80.4(7.1)</td>
<td id="A2.T3.2.3.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">90.5(2.7)</td>
<td id="A2.T3.2.3.9" class="ltx_td ltx_align_center ltx_border_t">67.2(6.1)</td>
</tr>
<tr id="A2.T3.2.4" class="ltx_tr">
<td id="A2.T3.2.4.1" class="ltx_td ltx_align_left ltx_border_r">Prefix</td>
<td id="A2.T3.2.4.2" class="ltx_td ltx_align_center">65.6(5.3)</td>
<td id="A2.T3.2.4.3" class="ltx_td ltx_align_center">75.5(6.1)</td>
<td id="A2.T3.2.4.4" class="ltx_td ltx_align_center">59.2(4.6)</td>
<td id="A2.T3.2.4.5" class="ltx_td ltx_align_center">56.0(4.0)</td>
<td id="A2.T3.2.4.6" class="ltx_td ltx_align_center">70.9(7.8)</td>
<td id="A2.T3.2.4.7" class="ltx_td ltx_align_center ltx_border_r">88.2(4.2)</td>
<td id="A2.T3.2.4.8" class="ltx_td ltx_align_center">65.4(5.6)</td>
</tr>
<tr id="A2.T3.2.5" class="ltx_tr">
<td id="A2.T3.2.5.1" class="ltx_td ltx_align_left ltx_border_t" rowspan="2"><span id="A2.T3.2.5.1.1" class="ltx_text">CCS</span></td>
<td id="A2.T3.2.5.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Regular</td>
<td id="A2.T3.2.5.3" class="ltx_td ltx_align_center ltx_border_t">62.1(4.1)</td>
<td id="A2.T3.2.5.4" class="ltx_td ltx_align_center ltx_border_t">78.5(3.8)</td>
<td id="A2.T3.2.5.5" class="ltx_td ltx_align_center ltx_border_t">61.7(2.5)</td>
<td id="A2.T3.2.5.6" class="ltx_td ltx_align_center ltx_border_t">71.5(3.0)</td>
<td id="A2.T3.2.5.7" class="ltx_td ltx_align_center ltx_border_t">82.1(2.7)</td>
<td id="A2.T3.2.5.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">77.6(3.3)</td>
<td id="A2.T3.2.5.9" class="ltx_td ltx_align_center ltx_border_t">71.2(3.2)</td>
</tr>
<tr id="A2.T3.2.6" class="ltx_tr">
<td id="A2.T3.2.6.1" class="ltx_td ltx_align_left ltx_border_r">Prefix</td>
<td id="A2.T3.2.6.2" class="ltx_td ltx_align_center">62.2(3.6)</td>
<td id="A2.T3.2.6.3" class="ltx_td ltx_align_center">75.4(5.2)</td>
<td id="A2.T3.2.6.4" class="ltx_td ltx_align_center">61.2(1.8)</td>
<td id="A2.T3.2.6.5" class="ltx_td ltx_align_center">73.2(2.6)</td>
<td id="A2.T3.2.6.6" class="ltx_td ltx_align_center">83.8(2.4)</td>
<td id="A2.T3.2.6.7" class="ltx_td ltx_align_center ltx_border_r">75.0(2.7)</td>
<td id="A2.T3.2.6.8" class="ltx_td ltx_align_center">71.2(3.1)</td>
</tr>
<tr id="A2.T3.2.7" class="ltx_tr">
<td id="A2.T3.2.7.1" class="ltx_td ltx_align_left ltx_border_t" rowspan="2"><span id="A2.T3.2.7.1.1" class="ltx_text">TPC</span></td>
<td id="A2.T3.2.7.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Regular</td>
<td id="A2.T3.2.7.3" class="ltx_td ltx_align_center ltx_border_t">64.6(5.8)</td>
<td id="A2.T3.2.7.4" class="ltx_td ltx_align_center ltx_border_t">76.5(5.7)</td>
<td id="A2.T3.2.7.5" class="ltx_td ltx_align_center ltx_border_t">59.9(3.7)</td>
<td id="A2.T3.2.7.6" class="ltx_td ltx_align_center ltx_border_t">66.7(5.0)</td>
<td id="A2.T3.2.7.7" class="ltx_td ltx_align_center ltx_border_t">78.3(3.3)</td>
<td id="A2.T3.2.7.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">58.9(13.0)</td>
<td id="A2.T3.2.7.9" class="ltx_td ltx_align_center ltx_border_t">69.2(4.7)</td>
</tr>
<tr id="A2.T3.2.8" class="ltx_tr">
<td id="A2.T3.2.8.1" class="ltx_td ltx_align_left ltx_border_r">Prefix</td>
<td id="A2.T3.2.8.2" class="ltx_td ltx_align_center">65.0(5.6)</td>
<td id="A2.T3.2.8.3" class="ltx_td ltx_align_center">77.0(6.0)</td>
<td id="A2.T3.2.8.4" class="ltx_td ltx_align_center">60.1(2.9)</td>
<td id="A2.T3.2.8.5" class="ltx_td ltx_align_center">68.1(5.0)</td>
<td id="A2.T3.2.8.6" class="ltx_td ltx_align_center">76.0(3.6)</td>
<td id="A2.T3.2.8.7" class="ltx_td ltx_align_center ltx_border_r">56.5(13.6)</td>
<td id="A2.T3.2.8.8" class="ltx_td ltx_align_center">69.2(4.6)</td>
</tr>
<tr id="A2.T3.2.9" class="ltx_tr">
<td id="A2.T3.2.9.1" class="ltx_td ltx_align_left ltx_border_t" rowspan="2"><span id="A2.T3.2.9.1.1" class="ltx_text">BSS</span></td>
<td id="A2.T3.2.9.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Regular</td>
<td id="A2.T3.2.9.3" class="ltx_td ltx_align_center ltx_border_t">62.6(6.8)</td>
<td id="A2.T3.2.9.4" class="ltx_td ltx_align_center ltx_border_t">76.8(5.3)</td>
<td id="A2.T3.2.9.5" class="ltx_td ltx_align_center ltx_border_t">60.7(3.4)</td>
<td id="A2.T3.2.9.6" class="ltx_td ltx_align_center ltx_border_t">69.7(3.3)</td>
<td id="A2.T3.2.9.7" class="ltx_td ltx_align_center ltx_border_t">79.3(2.5)</td>
<td id="A2.T3.2.9.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">76.7(9.6)</td>
<td id="A2.T3.2.9.9" class="ltx_td ltx_align_center ltx_border_t">69.8(4.3)</td>
</tr>
<tr id="A2.T3.2.10" class="ltx_tr">
<td id="A2.T3.2.10.1" class="ltx_td ltx_align_left ltx_border_r">Prefix</td>
<td id="A2.T3.2.10.2" class="ltx_td ltx_align_center">59.5(5.4)</td>
<td id="A2.T3.2.10.3" class="ltx_td ltx_align_center">75.5(6.6)</td>
<td id="A2.T3.2.10.4" class="ltx_td ltx_align_center">59.8(2.8)</td>
<td id="A2.T3.2.10.5" class="ltx_td ltx_align_center">65.2(2.8)</td>
<td id="A2.T3.2.10.6" class="ltx_td ltx_align_center">83.1(2.3)</td>
<td id="A2.T3.2.10.7" class="ltx_td ltx_align_center ltx_border_r">72.2(9.0)</td>
<td id="A2.T3.2.10.8" class="ltx_td ltx_align_center">68.6(4.0)</td>
</tr>
<tr id="A2.T3.2.11" class="ltx_tr">
<td id="A2.T3.2.11.1" class="ltx_td ltx_align_left ltx_border_t" rowspan="2"><span id="A2.T3.2.11.1.1" class="ltx_text">LR</span></td>
<td id="A2.T3.2.11.2" class="ltx_td ltx_align_left ltx_border_r ltx_border_t">Regular</td>
<td id="A2.T3.2.11.3" class="ltx_td ltx_align_center ltx_border_t">79.8(2.5)</td>
<td id="A2.T3.2.11.4" class="ltx_td ltx_align_center ltx_border_t">86.1(2.2)</td>
<td id="A2.T3.2.11.5" class="ltx_td ltx_align_center ltx_border_t">78.0(2.3)</td>
<td id="A2.T3.2.11.6" class="ltx_td ltx_align_center ltx_border_t">84.6(3.1)</td>
<td id="A2.T3.2.11.7" class="ltx_td ltx_align_center ltx_border_t">89.8(1.9)</td>
<td id="A2.T3.2.11.8" class="ltx_td ltx_align_center ltx_border_r ltx_border_t">90.7(2.1)</td>
<td id="A2.T3.2.11.9" class="ltx_td ltx_align_center ltx_border_t">83.7(2.4)</td>
</tr>
<tr id="A2.T3.2.12" class="ltx_tr">
<td id="A2.T3.2.12.1" class="ltx_td ltx_align_left ltx_border_r">Prefix</td>
<td id="A2.T3.2.12.2" class="ltx_td ltx_align_center">79.4(2.7)</td>
<td id="A2.T3.2.12.3" class="ltx_td ltx_align_center">86.3(2.6)</td>
<td id="A2.T3.2.12.4" class="ltx_td ltx_align_center">79.0(2.9)</td>
<td id="A2.T3.2.12.5" class="ltx_td ltx_align_center">86.3(2.8)</td>
<td id="A2.T3.2.12.6" class="ltx_td ltx_align_center">90.1(2.2)</td>
<td id="A2.T3.2.12.7" class="ltx_td ltx_align_center ltx_border_r">90.5(2.3)</td>
<td id="A2.T3.2.12.8" class="ltx_td ltx_align_center">84.2(2.6)</td>
</tr>
</table>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 3: </span>
Accuracy of each method and model averaged across all prompts and datasets, with the same setting as <a href="#S3.T1" title="In 3.1 Experimental Setup ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">1</span></a>.
“Regular” means no prefix is added, while “Prefix” corresponds to text in <a href="#A2.F5" title="In B.1.1 Evaluating The Effect of The Misleading Prefix ‣ B.1 Misleading Prefix ‣ Appendix B Misleading Prefix Details ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">5</span></a>. Notice that while the 0-shot accuracy decreases due to the prefix, all of our methods are more resistant to the influence.
</figcaption>
</figure>
<figure id="A2.F5" class="ltx_figure"><img src="/html/2212.03827/assets/newfigures/misleading_prefix_color.png" id="A2.F5.1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="247" height="254" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 5: </span>
The added prefix used to mislead models in a way that is egregiously incorrect or confusing.
</figcaption>
</figure>
<figure id="A2.F6" class="ltx_figure"><img src="/html/2212.03827/assets/x5.png" id="A2.F6.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="414" height="116" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 6: </span>
The top <math id="A2.F6.4.m1.1" class="ltx_Math" alttext="100" display="inline"><semantics id="A2.F6.4.m1.1b"><mn id="A2.F6.4.m1.1.1" xref="A2.F6.4.m1.1.1.cmml">100</mn><annotation-xml encoding="MathML-Content" id="A2.F6.4.m1.1c"><cn type="integer" id="A2.F6.4.m1.1.1.cmml" xref="A2.F6.4.m1.1.1">100</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.F6.4.m1.1d">100</annotation></semantics></math> tokens with highest probability averaged across examples in IMDB for UQA.
Without the prefix (left), the actual labels (“positive” and “negative”) and their synonyms (e.g. “Positive” and “Neg”) have high probability.
With the prefix, the model’s output becomes mostly irrelevant to sentiment; the actual labels are in the top <math id="A2.F6.5.m2.1" class="ltx_Math" alttext="100" display="inline"><semantics id="A2.F6.5.m2.1b"><mn id="A2.F6.5.m2.1.1" xref="A2.F6.5.m2.1.1.cmml">100</mn><annotation-xml encoding="MathML-Content" id="A2.F6.5.m2.1c"><cn type="integer" id="A2.F6.5.m2.1.1.cmml" xref="A2.F6.5.m2.1.1">100</cn></annotation-xml><annotation encoding="application/x-tex" id="A2.F6.5.m2.1d">100</annotation></semantics></math>, but no other synonyms are.
</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
</section>
</section>
<section id="A3" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix C </span>Masked Language Modeling Results</h2>

<div id="A3.p1" class="ltx_para ltx_noindent">
<p id="A3.p1.1" class="ltx_p">We now provide an initial demonstration that CCS can work well even when a model’s outputs are not very useful.</p>
</div>
<div id="A3.p2" class="ltx_para ltx_noindent">
<p id="A3.p2.1" class="ltx_p">First, we evaluate our method on a model trained exclusively with the masked language modeling (MLM) objective: DeBERTa-v2 <cite class="ltx_cite ltx_citemacro_citep">(He et al., <a href="#bib.bib15" title="" class="ltx_ref">2021</a>)</cite> <span id="A3.p2.1.1" class="ltx_text ltx_font_italic">without</span> NLI finetuning (unlike in <a href="#S3" title="3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">3</span></a>, where we finetuned DeBERTa on an NLI task to use it in the zero-shot setting).
The outputs of DeBERTa are unlikely to be meaningful if we provide a raw input without using any [MASK] tokens<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>The MLM objective first selects 15% of tokens to mask. Of the masked tokens, 80% are replaced with [MASK], 10% are randomly replaced with a different token, and 10% are left unchanged <cite class="ltx_cite ltx_citemacro_citep">(He et al., <a href="#bib.bib15" title="" class="ltx_ref">2021</a>)</cite>. As a result, for non-[MASK] tokens, the most likely output token should usually be the original input token itself, which is uninformative.</span></span></span>.
If CCS only works when the model outputs are useful, then it should perform poorly in this setting.</p>
</div>
<div id="A3.p3" class="ltx_para ltx_noindent">
<p id="A3.p3.1" class="ltx_p">We also consider one other way that model outputs can be uninformative. Given a sequence of tokens as input, DeBERTa has a different set of output logits for each input token. For a given contrast example (a question and a candidate answer), the most informative output should intuitively be the logits that predict the answer. For instance, if the contrast example <math id="A3.p3.1.m1.1" class="ltx_Math" alttext="x^{+}" display="inline"><semantics id="A3.p3.1.m1.1a"><msup id="A3.p3.1.m1.1.1" xref="A3.p3.1.m1.1.1.cmml"><mi id="A3.p3.1.m1.1.1.2" xref="A3.p3.1.m1.1.1.2.cmml">x</mi><mo id="A3.p3.1.m1.1.1.3" xref="A3.p3.1.m1.1.1.3.cmml">+</mo></msup><annotation-xml encoding="MathML-Content" id="A3.p3.1.m1.1b"><apply id="A3.p3.1.m1.1.1.cmml" xref="A3.p3.1.m1.1.1"><csymbol cd="ambiguous" id="A3.p3.1.m1.1.1.1.cmml" xref="A3.p3.1.m1.1.1">superscript</csymbol><ci id="A3.p3.1.m1.1.1.2.cmml" xref="A3.p3.1.m1.1.1.2">𝑥</ci><plus id="A3.p3.1.m1.1.1.3.cmml" xref="A3.p3.1.m1.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p3.1.m1.1c">x^{+}</annotation></semantics></math> is “Is 2+2=4? Yes”, then the most informative output should be the logits corresponding to candidate answer “Yes”. Consequently, if we instead format each example so that the label (e.g. “Yes” or “No”) appears in the <span id="A3.p3.1.1" class="ltx_text ltx_font_italic">middle</span> of the prompt, then the output corresponding to the <span id="A3.p3.1.2" class="ltx_text ltx_font_italic">final</span> token should not be very informative. If CCS only works well when the model outputs are useful, then it should also perform poorly in this setting (as long as CCS uses the last-token hidden states as usual).</p>
</div>
<div id="A3.p4" class="ltx_para ltx_noindent">
<p id="A3.p4.1" class="ltx_p">To show that CCS can work even when model outputs aren’t informative, based on the above discussion we now do an initial test of CCS when we simultaneously (1) use DeBERTa-v2 (MLM-pretrained only), and (2) format inputs so that the label is in the middle of the prompt rather than at the end. In particular, we apply CCS on the Amazon dataset, where we randomly sample 1000 new (unlabeled) points and use a 60/40 train-test split as before. As usual we continue to use the last-token hidden states for CCS. As usual, we use the default Huggingface tokenizer and we do not [MASK] any input tokens. We use the following custom prompt to test (2):</p>
</div>
<div id="A3.p5" class="ltx_para ltx_noindent">
<pre id="A3.p5.1" class="ltx_verbatim ltx_font_typewriter">
The following movie review expresses a [label] sentiment:\n[text]
</pre>
</div>
<div id="A3.p6" class="ltx_para ltx_noindent">
<p id="A3.p6.2" class="ltx_p">Even though it is unclear why the model outputs should be useful with this setup, we find that CCS can indeed still perform well, attaining an accuracy of <math id="A3.p6.1.m1.1" class="ltx_Math" alttext="93.7\%" display="inline"><semantics id="A3.p6.1.m1.1a"><mrow id="A3.p6.1.m1.1.1" xref="A3.p6.1.m1.1.1.cmml"><mn id="A3.p6.1.m1.1.1.2" xref="A3.p6.1.m1.1.1.2.cmml">93.7</mn><mo id="A3.p6.1.m1.1.1.1" xref="A3.p6.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A3.p6.1.m1.1b"><apply id="A3.p6.1.m1.1.1.cmml" xref="A3.p6.1.m1.1.1"><csymbol cd="latexml" id="A3.p6.1.m1.1.1.1.cmml" xref="A3.p6.1.m1.1.1.1">percent</csymbol><cn type="float" id="A3.p6.1.m1.1.1.2.cmml" xref="A3.p6.1.m1.1.1.2">93.7</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p6.1.m1.1c">93.7\%</annotation></semantics></math>. For comparison, this is nearly identical to the approximately <math id="A3.p6.2.m2.1" class="ltx_Math" alttext="94\%" display="inline"><semantics id="A3.p6.2.m2.1a"><mrow id="A3.p6.2.m2.1.1" xref="A3.p6.2.m2.1.1.cmml"><mn id="A3.p6.2.m2.1.1.2" xref="A3.p6.2.m2.1.1.2.cmml">94</mn><mo id="A3.p6.2.m2.1.1.1" xref="A3.p6.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A3.p6.2.m2.1b"><apply id="A3.p6.2.m2.1.1.cmml" xref="A3.p6.2.m2.1.1"><csymbol cd="latexml" id="A3.p6.2.m2.1.1.1.cmml" xref="A3.p6.2.m2.1.1.1">percent</csymbol><cn type="integer" id="A3.p6.2.m2.1.1.2.cmml" xref="A3.p6.2.m2.1.1.2">94</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p6.2.m2.1c">94\%</annotation></semantics></math> accuracy of CCS when evaluated on the Amazon dataset using NLI-finetuned DeBERTa and the original prompts (see <a href="#A5.F9" title="In Appendix E Complete Transfer Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">9</span></a>).</p>
</div>
<div id="A3.p7" class="ltx_para ltx_noindent">
<p id="A3.p7.7" class="ltx_p">We argued above that the model outputs should not be very useful in this setting. We now verify that this is the case. To do so, we evaluate a modified version of zero-shot prompting adapted to work for masked language models when we use no [MASK] tokens in the input. Specifically, on an input <math id="A3.p7.1.m1.1" class="ltx_Math" alttext="x^{+}" display="inline"><semantics id="A3.p7.1.m1.1a"><msup id="A3.p7.1.m1.1.1" xref="A3.p7.1.m1.1.1.cmml"><mi id="A3.p7.1.m1.1.1.2" xref="A3.p7.1.m1.1.1.2.cmml">x</mi><mo id="A3.p7.1.m1.1.1.3" xref="A3.p7.1.m1.1.1.3.cmml">+</mo></msup><annotation-xml encoding="MathML-Content" id="A3.p7.1.m1.1b"><apply id="A3.p7.1.m1.1.1.cmml" xref="A3.p7.1.m1.1.1"><csymbol cd="ambiguous" id="A3.p7.1.m1.1.1.1.cmml" xref="A3.p7.1.m1.1.1">superscript</csymbol><ci id="A3.p7.1.m1.1.1.2.cmml" xref="A3.p7.1.m1.1.1.2">𝑥</ci><plus id="A3.p7.1.m1.1.1.3.cmml" xref="A3.p7.1.m1.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p7.1.m1.1c">x^{+}</annotation></semantics></math> or <math id="A3.p7.2.m2.1" class="ltx_Math" alttext="x^{-}" display="inline"><semantics id="A3.p7.2.m2.1a"><msup id="A3.p7.2.m2.1.1" xref="A3.p7.2.m2.1.1.cmml"><mi id="A3.p7.2.m2.1.1.2" xref="A3.p7.2.m2.1.1.2.cmml">x</mi><mo id="A3.p7.2.m2.1.1.3" xref="A3.p7.2.m2.1.1.3.cmml">−</mo></msup><annotation-xml encoding="MathML-Content" id="A3.p7.2.m2.1b"><apply id="A3.p7.2.m2.1.1.cmml" xref="A3.p7.2.m2.1.1"><csymbol cd="ambiguous" id="A3.p7.2.m2.1.1.1.cmml" xref="A3.p7.2.m2.1.1">superscript</csymbol><ci id="A3.p7.2.m2.1.1.2.cmml" xref="A3.p7.2.m2.1.1.2">𝑥</ci><minus id="A3.p7.2.m2.1.1.3.cmml" xref="A3.p7.2.m2.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p7.2.m2.1c">x^{-}</annotation></semantics></math>, we (1) take the logit vector corresponding to the last input token, (2) select the corresponding logit for “positive” and the corresponding logit for “negative” (the two candidate labels), (3) take the difference between these, resulting in <math id="A3.p7.3.m3.1" class="ltx_Math" alttext="l_{pos}^{+}-l_{neg}^{+}" display="inline"><semantics id="A3.p7.3.m3.1a"><mrow id="A3.p7.3.m3.1.1" xref="A3.p7.3.m3.1.1.cmml"><msubsup id="A3.p7.3.m3.1.1.2" xref="A3.p7.3.m3.1.1.2.cmml"><mi id="A3.p7.3.m3.1.1.2.2.2" xref="A3.p7.3.m3.1.1.2.2.2.cmml">l</mi><mrow id="A3.p7.3.m3.1.1.2.2.3" xref="A3.p7.3.m3.1.1.2.2.3.cmml"><mi id="A3.p7.3.m3.1.1.2.2.3.2" xref="A3.p7.3.m3.1.1.2.2.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="A3.p7.3.m3.1.1.2.2.3.1" xref="A3.p7.3.m3.1.1.2.2.3.1.cmml">​</mo><mi id="A3.p7.3.m3.1.1.2.2.3.3" xref="A3.p7.3.m3.1.1.2.2.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="A3.p7.3.m3.1.1.2.2.3.1a" xref="A3.p7.3.m3.1.1.2.2.3.1.cmml">​</mo><mi id="A3.p7.3.m3.1.1.2.2.3.4" xref="A3.p7.3.m3.1.1.2.2.3.4.cmml">s</mi></mrow><mo id="A3.p7.3.m3.1.1.2.3" xref="A3.p7.3.m3.1.1.2.3.cmml">+</mo></msubsup><mo id="A3.p7.3.m3.1.1.1" xref="A3.p7.3.m3.1.1.1.cmml">−</mo><msubsup id="A3.p7.3.m3.1.1.3" xref="A3.p7.3.m3.1.1.3.cmml"><mi id="A3.p7.3.m3.1.1.3.2.2" xref="A3.p7.3.m3.1.1.3.2.2.cmml">l</mi><mrow id="A3.p7.3.m3.1.1.3.2.3" xref="A3.p7.3.m3.1.1.3.2.3.cmml"><mi id="A3.p7.3.m3.1.1.3.2.3.2" xref="A3.p7.3.m3.1.1.3.2.3.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="A3.p7.3.m3.1.1.3.2.3.1" xref="A3.p7.3.m3.1.1.3.2.3.1.cmml">​</mo><mi id="A3.p7.3.m3.1.1.3.2.3.3" xref="A3.p7.3.m3.1.1.3.2.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="A3.p7.3.m3.1.1.3.2.3.1a" xref="A3.p7.3.m3.1.1.3.2.3.1.cmml">​</mo><mi id="A3.p7.3.m3.1.1.3.2.3.4" xref="A3.p7.3.m3.1.1.3.2.3.4.cmml">g</mi></mrow><mo id="A3.p7.3.m3.1.1.3.3" xref="A3.p7.3.m3.1.1.3.3.cmml">+</mo></msubsup></mrow><annotation-xml encoding="MathML-Content" id="A3.p7.3.m3.1b"><apply id="A3.p7.3.m3.1.1.cmml" xref="A3.p7.3.m3.1.1"><minus id="A3.p7.3.m3.1.1.1.cmml" xref="A3.p7.3.m3.1.1.1"></minus><apply id="A3.p7.3.m3.1.1.2.cmml" xref="A3.p7.3.m3.1.1.2"><csymbol cd="ambiguous" id="A3.p7.3.m3.1.1.2.1.cmml" xref="A3.p7.3.m3.1.1.2">superscript</csymbol><apply id="A3.p7.3.m3.1.1.2.2.cmml" xref="A3.p7.3.m3.1.1.2"><csymbol cd="ambiguous" id="A3.p7.3.m3.1.1.2.2.1.cmml" xref="A3.p7.3.m3.1.1.2">subscript</csymbol><ci id="A3.p7.3.m3.1.1.2.2.2.cmml" xref="A3.p7.3.m3.1.1.2.2.2">𝑙</ci><apply id="A3.p7.3.m3.1.1.2.2.3.cmml" xref="A3.p7.3.m3.1.1.2.2.3"><times id="A3.p7.3.m3.1.1.2.2.3.1.cmml" xref="A3.p7.3.m3.1.1.2.2.3.1"></times><ci id="A3.p7.3.m3.1.1.2.2.3.2.cmml" xref="A3.p7.3.m3.1.1.2.2.3.2">𝑝</ci><ci id="A3.p7.3.m3.1.1.2.2.3.3.cmml" xref="A3.p7.3.m3.1.1.2.2.3.3">𝑜</ci><ci id="A3.p7.3.m3.1.1.2.2.3.4.cmml" xref="A3.p7.3.m3.1.1.2.2.3.4">𝑠</ci></apply></apply><plus id="A3.p7.3.m3.1.1.2.3.cmml" xref="A3.p7.3.m3.1.1.2.3"></plus></apply><apply id="A3.p7.3.m3.1.1.3.cmml" xref="A3.p7.3.m3.1.1.3"><csymbol cd="ambiguous" id="A3.p7.3.m3.1.1.3.1.cmml" xref="A3.p7.3.m3.1.1.3">superscript</csymbol><apply id="A3.p7.3.m3.1.1.3.2.cmml" xref="A3.p7.3.m3.1.1.3"><csymbol cd="ambiguous" id="A3.p7.3.m3.1.1.3.2.1.cmml" xref="A3.p7.3.m3.1.1.3">subscript</csymbol><ci id="A3.p7.3.m3.1.1.3.2.2.cmml" xref="A3.p7.3.m3.1.1.3.2.2">𝑙</ci><apply id="A3.p7.3.m3.1.1.3.2.3.cmml" xref="A3.p7.3.m3.1.1.3.2.3"><times id="A3.p7.3.m3.1.1.3.2.3.1.cmml" xref="A3.p7.3.m3.1.1.3.2.3.1"></times><ci id="A3.p7.3.m3.1.1.3.2.3.2.cmml" xref="A3.p7.3.m3.1.1.3.2.3.2">𝑛</ci><ci id="A3.p7.3.m3.1.1.3.2.3.3.cmml" xref="A3.p7.3.m3.1.1.3.2.3.3">𝑒</ci><ci id="A3.p7.3.m3.1.1.3.2.3.4.cmml" xref="A3.p7.3.m3.1.1.3.2.3.4">𝑔</ci></apply></apply><plus id="A3.p7.3.m3.1.1.3.3.cmml" xref="A3.p7.3.m3.1.1.3.3"></plus></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p7.3.m3.1c">l_{pos}^{+}-l_{neg}^{+}</annotation></semantics></math> for <math id="A3.p7.4.m4.1" class="ltx_Math" alttext="x^{+}" display="inline"><semantics id="A3.p7.4.m4.1a"><msup id="A3.p7.4.m4.1.1" xref="A3.p7.4.m4.1.1.cmml"><mi id="A3.p7.4.m4.1.1.2" xref="A3.p7.4.m4.1.1.2.cmml">x</mi><mo id="A3.p7.4.m4.1.1.3" xref="A3.p7.4.m4.1.1.3.cmml">+</mo></msup><annotation-xml encoding="MathML-Content" id="A3.p7.4.m4.1b"><apply id="A3.p7.4.m4.1.1.cmml" xref="A3.p7.4.m4.1.1"><csymbol cd="ambiguous" id="A3.p7.4.m4.1.1.1.cmml" xref="A3.p7.4.m4.1.1">superscript</csymbol><ci id="A3.p7.4.m4.1.1.2.cmml" xref="A3.p7.4.m4.1.1.2">𝑥</ci><plus id="A3.p7.4.m4.1.1.3.cmml" xref="A3.p7.4.m4.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p7.4.m4.1c">x^{+}</annotation></semantics></math> and <math id="A3.p7.5.m5.1" class="ltx_Math" alttext="l_{pos}^{-}-l_{neg}^{-}" display="inline"><semantics id="A3.p7.5.m5.1a"><mrow id="A3.p7.5.m5.1.1" xref="A3.p7.5.m5.1.1.cmml"><msubsup id="A3.p7.5.m5.1.1.2" xref="A3.p7.5.m5.1.1.2.cmml"><mi id="A3.p7.5.m5.1.1.2.2.2" xref="A3.p7.5.m5.1.1.2.2.2.cmml">l</mi><mrow id="A3.p7.5.m5.1.1.2.2.3" xref="A3.p7.5.m5.1.1.2.2.3.cmml"><mi id="A3.p7.5.m5.1.1.2.2.3.2" xref="A3.p7.5.m5.1.1.2.2.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="A3.p7.5.m5.1.1.2.2.3.1" xref="A3.p7.5.m5.1.1.2.2.3.1.cmml">​</mo><mi id="A3.p7.5.m5.1.1.2.2.3.3" xref="A3.p7.5.m5.1.1.2.2.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="A3.p7.5.m5.1.1.2.2.3.1a" xref="A3.p7.5.m5.1.1.2.2.3.1.cmml">​</mo><mi id="A3.p7.5.m5.1.1.2.2.3.4" xref="A3.p7.5.m5.1.1.2.2.3.4.cmml">s</mi></mrow><mo id="A3.p7.5.m5.1.1.2.3" xref="A3.p7.5.m5.1.1.2.3.cmml">−</mo></msubsup><mo id="A3.p7.5.m5.1.1.1" xref="A3.p7.5.m5.1.1.1.cmml">−</mo><msubsup id="A3.p7.5.m5.1.1.3" xref="A3.p7.5.m5.1.1.3.cmml"><mi id="A3.p7.5.m5.1.1.3.2.2" xref="A3.p7.5.m5.1.1.3.2.2.cmml">l</mi><mrow id="A3.p7.5.m5.1.1.3.2.3" xref="A3.p7.5.m5.1.1.3.2.3.cmml"><mi id="A3.p7.5.m5.1.1.3.2.3.2" xref="A3.p7.5.m5.1.1.3.2.3.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="A3.p7.5.m5.1.1.3.2.3.1" xref="A3.p7.5.m5.1.1.3.2.3.1.cmml">​</mo><mi id="A3.p7.5.m5.1.1.3.2.3.3" xref="A3.p7.5.m5.1.1.3.2.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="A3.p7.5.m5.1.1.3.2.3.1a" xref="A3.p7.5.m5.1.1.3.2.3.1.cmml">​</mo><mi id="A3.p7.5.m5.1.1.3.2.3.4" xref="A3.p7.5.m5.1.1.3.2.3.4.cmml">g</mi></mrow><mo id="A3.p7.5.m5.1.1.3.3" xref="A3.p7.5.m5.1.1.3.3.cmml">−</mo></msubsup></mrow><annotation-xml encoding="MathML-Content" id="A3.p7.5.m5.1b"><apply id="A3.p7.5.m5.1.1.cmml" xref="A3.p7.5.m5.1.1"><minus id="A3.p7.5.m5.1.1.1.cmml" xref="A3.p7.5.m5.1.1.1"></minus><apply id="A3.p7.5.m5.1.1.2.cmml" xref="A3.p7.5.m5.1.1.2"><csymbol cd="ambiguous" id="A3.p7.5.m5.1.1.2.1.cmml" xref="A3.p7.5.m5.1.1.2">superscript</csymbol><apply id="A3.p7.5.m5.1.1.2.2.cmml" xref="A3.p7.5.m5.1.1.2"><csymbol cd="ambiguous" id="A3.p7.5.m5.1.1.2.2.1.cmml" xref="A3.p7.5.m5.1.1.2">subscript</csymbol><ci id="A3.p7.5.m5.1.1.2.2.2.cmml" xref="A3.p7.5.m5.1.1.2.2.2">𝑙</ci><apply id="A3.p7.5.m5.1.1.2.2.3.cmml" xref="A3.p7.5.m5.1.1.2.2.3"><times id="A3.p7.5.m5.1.1.2.2.3.1.cmml" xref="A3.p7.5.m5.1.1.2.2.3.1"></times><ci id="A3.p7.5.m5.1.1.2.2.3.2.cmml" xref="A3.p7.5.m5.1.1.2.2.3.2">𝑝</ci><ci id="A3.p7.5.m5.1.1.2.2.3.3.cmml" xref="A3.p7.5.m5.1.1.2.2.3.3">𝑜</ci><ci id="A3.p7.5.m5.1.1.2.2.3.4.cmml" xref="A3.p7.5.m5.1.1.2.2.3.4">𝑠</ci></apply></apply><minus id="A3.p7.5.m5.1.1.2.3.cmml" xref="A3.p7.5.m5.1.1.2.3"></minus></apply><apply id="A3.p7.5.m5.1.1.3.cmml" xref="A3.p7.5.m5.1.1.3"><csymbol cd="ambiguous" id="A3.p7.5.m5.1.1.3.1.cmml" xref="A3.p7.5.m5.1.1.3">superscript</csymbol><apply id="A3.p7.5.m5.1.1.3.2.cmml" xref="A3.p7.5.m5.1.1.3"><csymbol cd="ambiguous" id="A3.p7.5.m5.1.1.3.2.1.cmml" xref="A3.p7.5.m5.1.1.3">subscript</csymbol><ci id="A3.p7.5.m5.1.1.3.2.2.cmml" xref="A3.p7.5.m5.1.1.3.2.2">𝑙</ci><apply id="A3.p7.5.m5.1.1.3.2.3.cmml" xref="A3.p7.5.m5.1.1.3.2.3"><times id="A3.p7.5.m5.1.1.3.2.3.1.cmml" xref="A3.p7.5.m5.1.1.3.2.3.1"></times><ci id="A3.p7.5.m5.1.1.3.2.3.2.cmml" xref="A3.p7.5.m5.1.1.3.2.3.2">𝑛</ci><ci id="A3.p7.5.m5.1.1.3.2.3.3.cmml" xref="A3.p7.5.m5.1.1.3.2.3.3">𝑒</ci><ci id="A3.p7.5.m5.1.1.3.2.3.4.cmml" xref="A3.p7.5.m5.1.1.3.2.3.4">𝑔</ci></apply></apply><minus id="A3.p7.5.m5.1.1.3.3.cmml" xref="A3.p7.5.m5.1.1.3.3"></minus></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p7.5.m5.1c">l_{pos}^{-}-l_{neg}^{-}</annotation></semantics></math> for <math id="A3.p7.6.m6.1" class="ltx_Math" alttext="x^{-}" display="inline"><semantics id="A3.p7.6.m6.1a"><msup id="A3.p7.6.m6.1.1" xref="A3.p7.6.m6.1.1.cmml"><mi id="A3.p7.6.m6.1.1.2" xref="A3.p7.6.m6.1.1.2.cmml">x</mi><mo id="A3.p7.6.m6.1.1.3" xref="A3.p7.6.m6.1.1.3.cmml">−</mo></msup><annotation-xml encoding="MathML-Content" id="A3.p7.6.m6.1b"><apply id="A3.p7.6.m6.1.1.cmml" xref="A3.p7.6.m6.1.1"><csymbol cd="ambiguous" id="A3.p7.6.m6.1.1.1.cmml" xref="A3.p7.6.m6.1.1">superscript</csymbol><ci id="A3.p7.6.m6.1.1.2.cmml" xref="A3.p7.6.m6.1.1.2">𝑥</ci><minus id="A3.p7.6.m6.1.1.3.cmml" xref="A3.p7.6.m6.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p7.6.m6.1c">x^{-}</annotation></semantics></math>, respectively, then (4) take the difference between these, <math id="A3.p7.7.m7.2" class="ltx_Math" alttext="(l_{pos}^{+}-l_{neg}^{+})-(l_{pos}^{-}-l_{neg}^{-})" display="inline"><semantics id="A3.p7.7.m7.2a"><mrow id="A3.p7.7.m7.2.2" xref="A3.p7.7.m7.2.2.cmml"><mrow id="A3.p7.7.m7.1.1.1.1" xref="A3.p7.7.m7.1.1.1.1.1.cmml"><mo stretchy="false" id="A3.p7.7.m7.1.1.1.1.2" xref="A3.p7.7.m7.1.1.1.1.1.cmml">(</mo><mrow id="A3.p7.7.m7.1.1.1.1.1" xref="A3.p7.7.m7.1.1.1.1.1.cmml"><msubsup id="A3.p7.7.m7.1.1.1.1.1.2" xref="A3.p7.7.m7.1.1.1.1.1.2.cmml"><mi id="A3.p7.7.m7.1.1.1.1.1.2.2.2" xref="A3.p7.7.m7.1.1.1.1.1.2.2.2.cmml">l</mi><mrow id="A3.p7.7.m7.1.1.1.1.1.2.2.3" xref="A3.p7.7.m7.1.1.1.1.1.2.2.3.cmml"><mi id="A3.p7.7.m7.1.1.1.1.1.2.2.3.2" xref="A3.p7.7.m7.1.1.1.1.1.2.2.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="A3.p7.7.m7.1.1.1.1.1.2.2.3.1" xref="A3.p7.7.m7.1.1.1.1.1.2.2.3.1.cmml">​</mo><mi id="A3.p7.7.m7.1.1.1.1.1.2.2.3.3" xref="A3.p7.7.m7.1.1.1.1.1.2.2.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="A3.p7.7.m7.1.1.1.1.1.2.2.3.1a" xref="A3.p7.7.m7.1.1.1.1.1.2.2.3.1.cmml">​</mo><mi id="A3.p7.7.m7.1.1.1.1.1.2.2.3.4" xref="A3.p7.7.m7.1.1.1.1.1.2.2.3.4.cmml">s</mi></mrow><mo id="A3.p7.7.m7.1.1.1.1.1.2.3" xref="A3.p7.7.m7.1.1.1.1.1.2.3.cmml">+</mo></msubsup><mo id="A3.p7.7.m7.1.1.1.1.1.1" xref="A3.p7.7.m7.1.1.1.1.1.1.cmml">−</mo><msubsup id="A3.p7.7.m7.1.1.1.1.1.3" xref="A3.p7.7.m7.1.1.1.1.1.3.cmml"><mi id="A3.p7.7.m7.1.1.1.1.1.3.2.2" xref="A3.p7.7.m7.1.1.1.1.1.3.2.2.cmml">l</mi><mrow id="A3.p7.7.m7.1.1.1.1.1.3.2.3" xref="A3.p7.7.m7.1.1.1.1.1.3.2.3.cmml"><mi id="A3.p7.7.m7.1.1.1.1.1.3.2.3.2" xref="A3.p7.7.m7.1.1.1.1.1.3.2.3.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="A3.p7.7.m7.1.1.1.1.1.3.2.3.1" xref="A3.p7.7.m7.1.1.1.1.1.3.2.3.1.cmml">​</mo><mi id="A3.p7.7.m7.1.1.1.1.1.3.2.3.3" xref="A3.p7.7.m7.1.1.1.1.1.3.2.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="A3.p7.7.m7.1.1.1.1.1.3.2.3.1a" xref="A3.p7.7.m7.1.1.1.1.1.3.2.3.1.cmml">​</mo><mi id="A3.p7.7.m7.1.1.1.1.1.3.2.3.4" xref="A3.p7.7.m7.1.1.1.1.1.3.2.3.4.cmml">g</mi></mrow><mo id="A3.p7.7.m7.1.1.1.1.1.3.3" xref="A3.p7.7.m7.1.1.1.1.1.3.3.cmml">+</mo></msubsup></mrow><mo stretchy="false" id="A3.p7.7.m7.1.1.1.1.3" xref="A3.p7.7.m7.1.1.1.1.1.cmml">)</mo></mrow><mo id="A3.p7.7.m7.2.2.3" xref="A3.p7.7.m7.2.2.3.cmml">−</mo><mrow id="A3.p7.7.m7.2.2.2.1" xref="A3.p7.7.m7.2.2.2.1.1.cmml"><mo stretchy="false" id="A3.p7.7.m7.2.2.2.1.2" xref="A3.p7.7.m7.2.2.2.1.1.cmml">(</mo><mrow id="A3.p7.7.m7.2.2.2.1.1" xref="A3.p7.7.m7.2.2.2.1.1.cmml"><msubsup id="A3.p7.7.m7.2.2.2.1.1.2" xref="A3.p7.7.m7.2.2.2.1.1.2.cmml"><mi id="A3.p7.7.m7.2.2.2.1.1.2.2.2" xref="A3.p7.7.m7.2.2.2.1.1.2.2.2.cmml">l</mi><mrow id="A3.p7.7.m7.2.2.2.1.1.2.2.3" xref="A3.p7.7.m7.2.2.2.1.1.2.2.3.cmml"><mi id="A3.p7.7.m7.2.2.2.1.1.2.2.3.2" xref="A3.p7.7.m7.2.2.2.1.1.2.2.3.2.cmml">p</mi><mo lspace="0em" rspace="0em" id="A3.p7.7.m7.2.2.2.1.1.2.2.3.1" xref="A3.p7.7.m7.2.2.2.1.1.2.2.3.1.cmml">​</mo><mi id="A3.p7.7.m7.2.2.2.1.1.2.2.3.3" xref="A3.p7.7.m7.2.2.2.1.1.2.2.3.3.cmml">o</mi><mo lspace="0em" rspace="0em" id="A3.p7.7.m7.2.2.2.1.1.2.2.3.1a" xref="A3.p7.7.m7.2.2.2.1.1.2.2.3.1.cmml">​</mo><mi id="A3.p7.7.m7.2.2.2.1.1.2.2.3.4" xref="A3.p7.7.m7.2.2.2.1.1.2.2.3.4.cmml">s</mi></mrow><mo id="A3.p7.7.m7.2.2.2.1.1.2.3" xref="A3.p7.7.m7.2.2.2.1.1.2.3.cmml">−</mo></msubsup><mo id="A3.p7.7.m7.2.2.2.1.1.1" xref="A3.p7.7.m7.2.2.2.1.1.1.cmml">−</mo><msubsup id="A3.p7.7.m7.2.2.2.1.1.3" xref="A3.p7.7.m7.2.2.2.1.1.3.cmml"><mi id="A3.p7.7.m7.2.2.2.1.1.3.2.2" xref="A3.p7.7.m7.2.2.2.1.1.3.2.2.cmml">l</mi><mrow id="A3.p7.7.m7.2.2.2.1.1.3.2.3" xref="A3.p7.7.m7.2.2.2.1.1.3.2.3.cmml"><mi id="A3.p7.7.m7.2.2.2.1.1.3.2.3.2" xref="A3.p7.7.m7.2.2.2.1.1.3.2.3.2.cmml">n</mi><mo lspace="0em" rspace="0em" id="A3.p7.7.m7.2.2.2.1.1.3.2.3.1" xref="A3.p7.7.m7.2.2.2.1.1.3.2.3.1.cmml">​</mo><mi id="A3.p7.7.m7.2.2.2.1.1.3.2.3.3" xref="A3.p7.7.m7.2.2.2.1.1.3.2.3.3.cmml">e</mi><mo lspace="0em" rspace="0em" id="A3.p7.7.m7.2.2.2.1.1.3.2.3.1a" xref="A3.p7.7.m7.2.2.2.1.1.3.2.3.1.cmml">​</mo><mi id="A3.p7.7.m7.2.2.2.1.1.3.2.3.4" xref="A3.p7.7.m7.2.2.2.1.1.3.2.3.4.cmml">g</mi></mrow><mo id="A3.p7.7.m7.2.2.2.1.1.3.3" xref="A3.p7.7.m7.2.2.2.1.1.3.3.cmml">−</mo></msubsup></mrow><mo stretchy="false" id="A3.p7.7.m7.2.2.2.1.3" xref="A3.p7.7.m7.2.2.2.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A3.p7.7.m7.2b"><apply id="A3.p7.7.m7.2.2.cmml" xref="A3.p7.7.m7.2.2"><minus id="A3.p7.7.m7.2.2.3.cmml" xref="A3.p7.7.m7.2.2.3"></minus><apply id="A3.p7.7.m7.1.1.1.1.1.cmml" xref="A3.p7.7.m7.1.1.1.1"><minus id="A3.p7.7.m7.1.1.1.1.1.1.cmml" xref="A3.p7.7.m7.1.1.1.1.1.1"></minus><apply id="A3.p7.7.m7.1.1.1.1.1.2.cmml" xref="A3.p7.7.m7.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A3.p7.7.m7.1.1.1.1.1.2.1.cmml" xref="A3.p7.7.m7.1.1.1.1.1.2">superscript</csymbol><apply id="A3.p7.7.m7.1.1.1.1.1.2.2.cmml" xref="A3.p7.7.m7.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A3.p7.7.m7.1.1.1.1.1.2.2.1.cmml" xref="A3.p7.7.m7.1.1.1.1.1.2">subscript</csymbol><ci id="A3.p7.7.m7.1.1.1.1.1.2.2.2.cmml" xref="A3.p7.7.m7.1.1.1.1.1.2.2.2">𝑙</ci><apply id="A3.p7.7.m7.1.1.1.1.1.2.2.3.cmml" xref="A3.p7.7.m7.1.1.1.1.1.2.2.3"><times id="A3.p7.7.m7.1.1.1.1.1.2.2.3.1.cmml" xref="A3.p7.7.m7.1.1.1.1.1.2.2.3.1"></times><ci id="A3.p7.7.m7.1.1.1.1.1.2.2.3.2.cmml" xref="A3.p7.7.m7.1.1.1.1.1.2.2.3.2">𝑝</ci><ci id="A3.p7.7.m7.1.1.1.1.1.2.2.3.3.cmml" xref="A3.p7.7.m7.1.1.1.1.1.2.2.3.3">𝑜</ci><ci id="A3.p7.7.m7.1.1.1.1.1.2.2.3.4.cmml" xref="A3.p7.7.m7.1.1.1.1.1.2.2.3.4">𝑠</ci></apply></apply><plus id="A3.p7.7.m7.1.1.1.1.1.2.3.cmml" xref="A3.p7.7.m7.1.1.1.1.1.2.3"></plus></apply><apply id="A3.p7.7.m7.1.1.1.1.1.3.cmml" xref="A3.p7.7.m7.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A3.p7.7.m7.1.1.1.1.1.3.1.cmml" xref="A3.p7.7.m7.1.1.1.1.1.3">superscript</csymbol><apply id="A3.p7.7.m7.1.1.1.1.1.3.2.cmml" xref="A3.p7.7.m7.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A3.p7.7.m7.1.1.1.1.1.3.2.1.cmml" xref="A3.p7.7.m7.1.1.1.1.1.3">subscript</csymbol><ci id="A3.p7.7.m7.1.1.1.1.1.3.2.2.cmml" xref="A3.p7.7.m7.1.1.1.1.1.3.2.2">𝑙</ci><apply id="A3.p7.7.m7.1.1.1.1.1.3.2.3.cmml" xref="A3.p7.7.m7.1.1.1.1.1.3.2.3"><times id="A3.p7.7.m7.1.1.1.1.1.3.2.3.1.cmml" xref="A3.p7.7.m7.1.1.1.1.1.3.2.3.1"></times><ci id="A3.p7.7.m7.1.1.1.1.1.3.2.3.2.cmml" xref="A3.p7.7.m7.1.1.1.1.1.3.2.3.2">𝑛</ci><ci id="A3.p7.7.m7.1.1.1.1.1.3.2.3.3.cmml" xref="A3.p7.7.m7.1.1.1.1.1.3.2.3.3">𝑒</ci><ci id="A3.p7.7.m7.1.1.1.1.1.3.2.3.4.cmml" xref="A3.p7.7.m7.1.1.1.1.1.3.2.3.4">𝑔</ci></apply></apply><plus id="A3.p7.7.m7.1.1.1.1.1.3.3.cmml" xref="A3.p7.7.m7.1.1.1.1.1.3.3"></plus></apply></apply><apply id="A3.p7.7.m7.2.2.2.1.1.cmml" xref="A3.p7.7.m7.2.2.2.1"><minus id="A3.p7.7.m7.2.2.2.1.1.1.cmml" xref="A3.p7.7.m7.2.2.2.1.1.1"></minus><apply id="A3.p7.7.m7.2.2.2.1.1.2.cmml" xref="A3.p7.7.m7.2.2.2.1.1.2"><csymbol cd="ambiguous" id="A3.p7.7.m7.2.2.2.1.1.2.1.cmml" xref="A3.p7.7.m7.2.2.2.1.1.2">superscript</csymbol><apply id="A3.p7.7.m7.2.2.2.1.1.2.2.cmml" xref="A3.p7.7.m7.2.2.2.1.1.2"><csymbol cd="ambiguous" id="A3.p7.7.m7.2.2.2.1.1.2.2.1.cmml" xref="A3.p7.7.m7.2.2.2.1.1.2">subscript</csymbol><ci id="A3.p7.7.m7.2.2.2.1.1.2.2.2.cmml" xref="A3.p7.7.m7.2.2.2.1.1.2.2.2">𝑙</ci><apply id="A3.p7.7.m7.2.2.2.1.1.2.2.3.cmml" xref="A3.p7.7.m7.2.2.2.1.1.2.2.3"><times id="A3.p7.7.m7.2.2.2.1.1.2.2.3.1.cmml" xref="A3.p7.7.m7.2.2.2.1.1.2.2.3.1"></times><ci id="A3.p7.7.m7.2.2.2.1.1.2.2.3.2.cmml" xref="A3.p7.7.m7.2.2.2.1.1.2.2.3.2">𝑝</ci><ci id="A3.p7.7.m7.2.2.2.1.1.2.2.3.3.cmml" xref="A3.p7.7.m7.2.2.2.1.1.2.2.3.3">𝑜</ci><ci id="A3.p7.7.m7.2.2.2.1.1.2.2.3.4.cmml" xref="A3.p7.7.m7.2.2.2.1.1.2.2.3.4">𝑠</ci></apply></apply><minus id="A3.p7.7.m7.2.2.2.1.1.2.3.cmml" xref="A3.p7.7.m7.2.2.2.1.1.2.3"></minus></apply><apply id="A3.p7.7.m7.2.2.2.1.1.3.cmml" xref="A3.p7.7.m7.2.2.2.1.1.3"><csymbol cd="ambiguous" id="A3.p7.7.m7.2.2.2.1.1.3.1.cmml" xref="A3.p7.7.m7.2.2.2.1.1.3">superscript</csymbol><apply id="A3.p7.7.m7.2.2.2.1.1.3.2.cmml" xref="A3.p7.7.m7.2.2.2.1.1.3"><csymbol cd="ambiguous" id="A3.p7.7.m7.2.2.2.1.1.3.2.1.cmml" xref="A3.p7.7.m7.2.2.2.1.1.3">subscript</csymbol><ci id="A3.p7.7.m7.2.2.2.1.1.3.2.2.cmml" xref="A3.p7.7.m7.2.2.2.1.1.3.2.2">𝑙</ci><apply id="A3.p7.7.m7.2.2.2.1.1.3.2.3.cmml" xref="A3.p7.7.m7.2.2.2.1.1.3.2.3"><times id="A3.p7.7.m7.2.2.2.1.1.3.2.3.1.cmml" xref="A3.p7.7.m7.2.2.2.1.1.3.2.3.1"></times><ci id="A3.p7.7.m7.2.2.2.1.1.3.2.3.2.cmml" xref="A3.p7.7.m7.2.2.2.1.1.3.2.3.2">𝑛</ci><ci id="A3.p7.7.m7.2.2.2.1.1.3.2.3.3.cmml" xref="A3.p7.7.m7.2.2.2.1.1.3.2.3.3">𝑒</ci><ci id="A3.p7.7.m7.2.2.2.1.1.3.2.3.4.cmml" xref="A3.p7.7.m7.2.2.2.1.1.3.2.3.4">𝑔</ci></apply></apply><minus id="A3.p7.7.m7.2.2.2.1.1.3.3.cmml" xref="A3.p7.7.m7.2.2.2.1.1.3.3"></minus></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p7.7.m7.2c">(l_{pos}^{+}-l_{neg}^{+})-(l_{pos}^{-}-l_{neg}^{-})</annotation></semantics></math>, to form the effective logit. Intuitively, this final expression should be large if the example is positive, and small if the example is negative. As before, we calibrate this zero-shot method so that its predictions are balanced.</p>
</div>
<div id="A3.p8" class="ltx_para ltx_noindent">
<p id="A3.p8.2" class="ltx_p">We evaluate this method and find that it recovers non-trivial knowledge in the model outputs, but that it still performs substantially worse than CCS: calibrated zero-shot accuracy in this setting is <math id="A3.p8.1.m1.1" class="ltx_Math" alttext="71.6\%" display="inline"><semantics id="A3.p8.1.m1.1a"><mrow id="A3.p8.1.m1.1.1" xref="A3.p8.1.m1.1.1.cmml"><mn id="A3.p8.1.m1.1.1.2" xref="A3.p8.1.m1.1.1.2.cmml">71.6</mn><mo id="A3.p8.1.m1.1.1.1" xref="A3.p8.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A3.p8.1.m1.1b"><apply id="A3.p8.1.m1.1.1.cmml" xref="A3.p8.1.m1.1.1"><csymbol cd="latexml" id="A3.p8.1.m1.1.1.1.cmml" xref="A3.p8.1.m1.1.1.1">percent</csymbol><cn type="float" id="A3.p8.1.m1.1.1.2.cmml" xref="A3.p8.1.m1.1.1.2">71.6</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p8.1.m1.1c">71.6\%</annotation></semantics></math>, compared to almost <math id="A3.p8.2.m2.1" class="ltx_Math" alttext="94\%" display="inline"><semantics id="A3.p8.2.m2.1a"><mrow id="A3.p8.2.m2.1.1" xref="A3.p8.2.m2.1.1.cmml"><mn id="A3.p8.2.m2.1.1.2" xref="A3.p8.2.m2.1.1.2.cmml">94</mn><mo id="A3.p8.2.m2.1.1.1" xref="A3.p8.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A3.p8.2.m2.1b"><apply id="A3.p8.2.m2.1.1.cmml" xref="A3.p8.2.m2.1.1"><csymbol cd="latexml" id="A3.p8.2.m2.1.1.1.cmml" xref="A3.p8.2.m2.1.1.1">percent</csymbol><cn type="integer" id="A3.p8.2.m2.1.1.2.cmml" xref="A3.p8.2.m2.1.1.2">94</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A3.p8.2.m2.1c">94\%</annotation></semantics></math> for CCS.
This suggests that model outputs in this setting indeed aren’t very useful, especially relative to CCS.</p>
</div>
<div id="A3.p9" class="ltx_para ltx_noindent">
<p id="A3.p9.1" class="ltx_p">Overall, our results in this section show that CCS can still work in at least some settings where model outputs don’t seem to be very useful. This provides additional evidence that out approach does not simply recover knowledge represented in the model outputs.</p>
</div>
</section>
<section id="A4" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix D </span>Complete Sample Complexity Results</h2>

<div id="A4.p1" class="ltx_para ltx_noindent">
<p id="A4.p1.1" class="ltx_p">In this section we show additional results on how the number of examples affects CCS.
We show the performance (averaged across all datasets) for <span id="A4.p1.1.1" class="ltx_text ltx_font_italic">all</span> models in <a href="#A4.F7" title="In Appendix D Complete Sample Complexity Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">7</span></a>.
We next show a more fine-grained results.
Specifically, we select <math id="A4.p1.1.m1.3" class="ltx_Math" alttext="\#\text{Samples}=1,8,64" display="inline"><semantics id="A4.p1.1.m1.3a"><mrow id="A4.p1.1.m1.3.4" xref="A4.p1.1.m1.3.4.cmml"><mrow id="A4.p1.1.m1.3.4.2" xref="A4.p1.1.m1.3.4.2.cmml"><mi mathvariant="normal" id="A4.p1.1.m1.3.4.2.2" xref="A4.p1.1.m1.3.4.2.2.cmml">#</mi><mo lspace="0em" rspace="0em" id="A4.p1.1.m1.3.4.2.1" xref="A4.p1.1.m1.3.4.2.1.cmml">​</mo><mtext id="A4.p1.1.m1.3.4.2.3" xref="A4.p1.1.m1.3.4.2.3a.cmml">Samples</mtext></mrow><mo id="A4.p1.1.m1.3.4.1" xref="A4.p1.1.m1.3.4.1.cmml">=</mo><mrow id="A4.p1.1.m1.3.4.3.2" xref="A4.p1.1.m1.3.4.3.1.cmml"><mn id="A4.p1.1.m1.1.1" xref="A4.p1.1.m1.1.1.cmml">1</mn><mo id="A4.p1.1.m1.3.4.3.2.1" xref="A4.p1.1.m1.3.4.3.1.cmml">,</mo><mn id="A4.p1.1.m1.2.2" xref="A4.p1.1.m1.2.2.cmml">8</mn><mo id="A4.p1.1.m1.3.4.3.2.2" xref="A4.p1.1.m1.3.4.3.1.cmml">,</mo><mn id="A4.p1.1.m1.3.3" xref="A4.p1.1.m1.3.3.cmml">64</mn></mrow></mrow><annotation-xml encoding="MathML-Content" id="A4.p1.1.m1.3b"><apply id="A4.p1.1.m1.3.4.cmml" xref="A4.p1.1.m1.3.4"><eq id="A4.p1.1.m1.3.4.1.cmml" xref="A4.p1.1.m1.3.4.1"></eq><apply id="A4.p1.1.m1.3.4.2.cmml" xref="A4.p1.1.m1.3.4.2"><times id="A4.p1.1.m1.3.4.2.1.cmml" xref="A4.p1.1.m1.3.4.2.1"></times><ci id="A4.p1.1.m1.3.4.2.2.cmml" xref="A4.p1.1.m1.3.4.2.2">#</ci><ci id="A4.p1.1.m1.3.4.2.3a.cmml" xref="A4.p1.1.m1.3.4.2.3"><mtext id="A4.p1.1.m1.3.4.2.3.cmml" xref="A4.p1.1.m1.3.4.2.3">Samples</mtext></ci></apply><list id="A4.p1.1.m1.3.4.3.1.cmml" xref="A4.p1.1.m1.3.4.3.2"><cn type="integer" id="A4.p1.1.m1.1.1.cmml" xref="A4.p1.1.m1.1.1">1</cn><cn type="integer" id="A4.p1.1.m1.2.2.cmml" xref="A4.p1.1.m1.2.2">8</cn><cn type="integer" id="A4.p1.1.m1.3.3.cmml" xref="A4.p1.1.m1.3.3">64</cn></list></apply></annotation-xml><annotation encoding="application/x-tex" id="A4.p1.1.m1.3c">\#\text{Samples}=1,8,64</annotation></semantics></math> and show dataset-level results in <a href="#A4.F8" title="In Appendix D Complete Sample Complexity Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">8</span></a> for each model.</p>
</div>
<figure id="A4.F7" class="ltx_figure"><img src="/html/2212.03827/assets/x6.png" id="A4.F7.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="253" height="163" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 7: </span>
This figure shares the same setting with <a href="#S3.F3" title="In 3.3.3 Truth is a Salient Feature ‣ 3.3 Analyzing CCS ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">3</span></a>, but it contains all six models we consider.
</figcaption>
</figure>
<figure id="A4.F8" class="ltx_figure"><img src="/html/2212.03827/assets/x7.png" id="A4.F8.1.g1" class="ltx_graphics ltx_centering ltx_img_portrait" width="422" height="549" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 8: </span>
CCS performance when we only use <math id="A4.F8.4.m1.3" class="ltx_Math" alttext="1,8,64" display="inline"><semantics id="A4.F8.4.m1.3b"><mrow id="A4.F8.4.m1.3.4.2" xref="A4.F8.4.m1.3.4.1.cmml"><mn id="A4.F8.4.m1.1.1" xref="A4.F8.4.m1.1.1.cmml">1</mn><mo id="A4.F8.4.m1.3.4.2.1" xref="A4.F8.4.m1.3.4.1.cmml">,</mo><mn id="A4.F8.4.m1.2.2" xref="A4.F8.4.m1.2.2.cmml">8</mn><mo id="A4.F8.4.m1.3.4.2.2" xref="A4.F8.4.m1.3.4.1.cmml">,</mo><mn id="A4.F8.4.m1.3.3" xref="A4.F8.4.m1.3.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.F8.4.m1.3c"><list id="A4.F8.4.m1.3.4.1.cmml" xref="A4.F8.4.m1.3.4.2"><cn type="integer" id="A4.F8.4.m1.1.1.cmml" xref="A4.F8.4.m1.1.1">1</cn><cn type="integer" id="A4.F8.4.m1.2.2.cmml" xref="A4.F8.4.m1.2.2">8</cn><cn type="integer" id="A4.F8.4.m1.3.3.cmml" xref="A4.F8.4.m1.3.3">64</cn></list></annotation-xml><annotation encoding="application/x-tex" id="A4.F8.4.m1.3d">1,8,64</annotation></semantics></math> examples, for all models and datasets. X-axis represents the dataset and y-axis is the accuracy in percentage. For each dataset, we select the prompt with highest 0-shot performance, and then randomly select <math id="A4.F8.5.m2.3" class="ltx_Math" alttext="1,8,64" display="inline"><semantics id="A4.F8.5.m2.3b"><mrow id="A4.F8.5.m2.3.4.2" xref="A4.F8.5.m2.3.4.1.cmml"><mn id="A4.F8.5.m2.1.1" xref="A4.F8.5.m2.1.1.cmml">1</mn><mo id="A4.F8.5.m2.3.4.2.1" xref="A4.F8.5.m2.3.4.1.cmml">,</mo><mn id="A4.F8.5.m2.2.2" xref="A4.F8.5.m2.2.2.cmml">8</mn><mo id="A4.F8.5.m2.3.4.2.2" xref="A4.F8.5.m2.3.4.1.cmml">,</mo><mn id="A4.F8.5.m2.3.3" xref="A4.F8.5.m2.3.3.cmml">64</mn></mrow><annotation-xml encoding="MathML-Content" id="A4.F8.5.m2.3c"><list id="A4.F8.5.m2.3.4.1.cmml" xref="A4.F8.5.m2.3.4.2"><cn type="integer" id="A4.F8.5.m2.1.1.cmml" xref="A4.F8.5.m2.1.1">1</cn><cn type="integer" id="A4.F8.5.m2.2.2.cmml" xref="A4.F8.5.m2.2.2">8</cn><cn type="integer" id="A4.F8.5.m2.3.3.cmml" xref="A4.F8.5.m2.3.3">64</cn></list></annotation-xml><annotation encoding="application/x-tex" id="A4.F8.5.m2.3d">1,8,64</annotation></semantics></math> data points from this prompt. We then perform CCS and test on all prompts of this dataset, where each value in the barplot corresponds to one prompt.
</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A5" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix E </span>Complete Transfer Results</h2>

<div id="A5.p1" class="ltx_para ltx_noindent">
<p id="A5.p1.1" class="ltx_p">Complete transfer results for CCS, TPC and BSS in all models are shown in <a href="#A5.F9" title="In Appendix E Complete Transfer Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">9</span></a>.</p>
</div>
<figure id="A5.F9" class="ltx_figure"><img src="/html/2212.03827/assets/x8.png" id="A5.F9.1.g1" class="ltx_graphics ltx_centering ltx_img_portrait" width="422" height="591" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 9: </span>
Transfer Accuracy for all models using CCS, BSS, TPC and LR.
</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A6" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix F </span>Complete Intermediate Representations Results</h2>

<div id="A6.p1" class="ltx_para ltx_noindent">
<p id="A6.p1.1" class="ltx_p">In this section we the performance of CCS and LR across all hidden layers in all six models.
Specifically, for each model, we generate the hidden states for each dataset every 2 layers (for both encoders and decoders).
Then, for each set of hidden states, we perform CCS and LR on each dataset separately, and average the performance across all datasets.
We show CCS results in <a href="#A6.F10" title="In Appendix F Complete Intermediate Representations Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">10</span></a> and LR results in <a href="#A6.F11" title="In Appendix F Complete Intermediate Representations Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">11</span></a>.</p>
</div>
<figure id="A6.F10" class="ltx_figure"><img src="/html/2212.03827/assets/x9.png" id="A6.F10.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="380" height="201" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 10: </span>
CCS performance when using the hidden states across different layers, using all six models we consider in the paper.
</figcaption>
</figure>
<figure id="A6.F11" class="ltx_figure"><img src="/html/2212.03827/assets/x10.png" id="A6.F11.1.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="380" height="201" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 11: </span>
Linear regression performance when using the hidden states across different layers, using all six models we consider in the paper. This is the ceiling of all possible methods, and is supervised.
</figcaption>
</figure>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A7" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix G </span>CCS and CRC Implementation Details</h2>

<div id="A7.p1" class="ltx_para ltx_noindent">
<p id="A7.p1.1" class="ltx_p">In this section, we provide further implementation details for CCS and CRC.
We begin by describing how we construct contrast pairs.</p>
</div>
<section id="A7.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.1 </span>Contrast Pairs</h3>

<div id="A7.SS1.p1" class="ltx_para ltx_noindent">
<p id="A7.SS1.p1.1" class="ltx_p">We use the Huggingface library <cite class="ltx_cite ltx_citemacro_citep">(Wolf et al., <a href="#bib.bib52" title="" class="ltx_ref">2019</a>)</cite> for all of our experiments.
We use the standard tokenizer for each model, and always take the hidden state corresponding to the last token in a given layer indexed by <span id="A7.SS1.p1.1.1" class="ltx_text ltx_markedasmath">idx</span>.
For GPT-J, we also manually add the EOS token to the end of the input when tokenizing, which we found in preliminary experiments can make a difference.
We show Huggingface-style pseudocode for extracting hidden states from in <a href="#alg1" title="In G.1 Contrast Pairs ‣ Appendix G CCS and CRC Implementation Details ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Algorithm</span> <span class="ltx_text ltx_ref_tag">1</span></a>.</p>
</div>
<div id="A7.SS1.p2" class="ltx_para ltx_noindent">
<p id="A7.SS1.p2.2" class="ltx_p">For encoder-only and decoder-only models, we provide the full input, <math id="A7.SS1.p2.1.m1.1" class="ltx_Math" alttext="x^{+}" display="inline"><semantics id="A7.SS1.p2.1.m1.1a"><msup id="A7.SS1.p2.1.m1.1.1" xref="A7.SS1.p2.1.m1.1.1.cmml"><mi id="A7.SS1.p2.1.m1.1.1.2" xref="A7.SS1.p2.1.m1.1.1.2.cmml">x</mi><mo id="A7.SS1.p2.1.m1.1.1.3" xref="A7.SS1.p2.1.m1.1.1.3.cmml">+</mo></msup><annotation-xml encoding="MathML-Content" id="A7.SS1.p2.1.m1.1b"><apply id="A7.SS1.p2.1.m1.1.1.cmml" xref="A7.SS1.p2.1.m1.1.1"><csymbol cd="ambiguous" id="A7.SS1.p2.1.m1.1.1.1.cmml" xref="A7.SS1.p2.1.m1.1.1">superscript</csymbol><ci id="A7.SS1.p2.1.m1.1.1.2.cmml" xref="A7.SS1.p2.1.m1.1.1.2">𝑥</ci><plus id="A7.SS1.p2.1.m1.1.1.3.cmml" xref="A7.SS1.p2.1.m1.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS1.p2.1.m1.1c">x^{+}</annotation></semantics></math> or <math id="A7.SS1.p2.2.m2.1" class="ltx_Math" alttext="x^{-}" display="inline"><semantics id="A7.SS1.p2.2.m2.1a"><msup id="A7.SS1.p2.2.m2.1.1" xref="A7.SS1.p2.2.m2.1.1.cmml"><mi id="A7.SS1.p2.2.m2.1.1.2" xref="A7.SS1.p2.2.m2.1.1.2.cmml">x</mi><mo id="A7.SS1.p2.2.m2.1.1.3" xref="A7.SS1.p2.2.m2.1.1.3.cmml">−</mo></msup><annotation-xml encoding="MathML-Content" id="A7.SS1.p2.2.m2.1b"><apply id="A7.SS1.p2.2.m2.1.1.cmml" xref="A7.SS1.p2.2.m2.1.1"><csymbol cd="ambiguous" id="A7.SS1.p2.2.m2.1.1.1.cmml" xref="A7.SS1.p2.2.m2.1.1">superscript</csymbol><ci id="A7.SS1.p2.2.m2.1.1.2.cmml" xref="A7.SS1.p2.2.m2.1.1.2">𝑥</ci><minus id="A7.SS1.p2.2.m2.1.1.3.cmml" xref="A7.SS1.p2.2.m2.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS1.p2.2.m2.1c">x^{-}</annotation></semantics></math>, which includes both the question <span id="A7.SS1.p2.2.1" class="ltx_text ltx_font_italic">and</span> the proposed answer, to the model. For encoder-decoder models, our input format depends on whether we are taking the encoder hidden states of the decoder hidden states. When we take the decoder hidden states, we input the question to the encoder, and input the candidate answer to the decoder. In contrast, when we take the encoder hidden states of an encoder-decoder model, we provide the full input (including the proposed answer) to the encoder, and ignore the decoder (simply passing the empty string to the decoder and ignoring its results). This is necessary to ensure that the inputs to the encoder are not identical across contrast pairs.</p>
</div>
<figure id="alg1" class="ltx_float ltx_float_algorithm ltx_framed ltx_framed_top">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_float"><span id="alg1.2.1.1" class="ltx_text ltx_font_bold">Algorithm 1</span> </span> Pseudocode for Getting Contrast Features</figcaption>
<div id="alg1.3" class="ltx_listing ltx_listing">
<div id="alg1.l0" class="ltx_listingline">
<span class="ltx_tag ltx_tag_listingline">0:</span>  Contrast Pairs Set <math id="alg1.l0.m1.1" class="ltx_Math" alttext="S" display="inline"><semantics id="alg1.l0.m1.1a"><mi id="alg1.l0.m1.1.1" xref="alg1.l0.m1.1.1.cmml">S</mi><annotation-xml encoding="MathML-Content" id="alg1.l0.m1.1b"><ci id="alg1.l0.m1.1.1.cmml" xref="alg1.l0.m1.1.1">𝑆</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l0.m1.1c">S</annotation></semantics></math>, model name <span id="alg1.l0.1" class="ltx_text ltx_markedasmath">mdl</span>, layer index <span id="alg1.l0.2" class="ltx_text ltx_markedasmath">idx</span>

</div>
<div id="alg1.l1" class="ltx_listingline">  <math id="alg1.l1.m1.1" class="ltx_Math" alttext="\mathbf{m}=\text{transformers.AutoModel}(\text{mdl})" display="inline"><semantics id="alg1.l1.m1.1a"><mrow id="alg1.l1.m1.1.2" xref="alg1.l1.m1.1.2.cmml"><mi id="alg1.l1.m1.1.2.2" xref="alg1.l1.m1.1.2.2.cmml">𝐦</mi><mo id="alg1.l1.m1.1.2.1" xref="alg1.l1.m1.1.2.1.cmml">=</mo><mrow id="alg1.l1.m1.1.2.3" xref="alg1.l1.m1.1.2.3.cmml"><mtext id="alg1.l1.m1.1.2.3.2" xref="alg1.l1.m1.1.2.3.2a.cmml">transformers.AutoModel</mtext><mo lspace="0em" rspace="0em" id="alg1.l1.m1.1.2.3.1" xref="alg1.l1.m1.1.2.3.1.cmml">​</mo><mrow id="alg1.l1.m1.1.2.3.3.2" xref="alg1.l1.m1.1.1a.cmml"><mo stretchy="false" id="alg1.l1.m1.1.2.3.3.2.1" xref="alg1.l1.m1.1.1a.cmml">(</mo><mtext id="alg1.l1.m1.1.1" xref="alg1.l1.m1.1.1.cmml">mdl</mtext><mo stretchy="false" id="alg1.l1.m1.1.2.3.3.2.2" xref="alg1.l1.m1.1.1a.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l1.m1.1b"><apply id="alg1.l1.m1.1.2.cmml" xref="alg1.l1.m1.1.2"><eq id="alg1.l1.m1.1.2.1.cmml" xref="alg1.l1.m1.1.2.1"></eq><ci id="alg1.l1.m1.1.2.2.cmml" xref="alg1.l1.m1.1.2.2">𝐦</ci><apply id="alg1.l1.m1.1.2.3.cmml" xref="alg1.l1.m1.1.2.3"><times id="alg1.l1.m1.1.2.3.1.cmml" xref="alg1.l1.m1.1.2.3.1"></times><ci id="alg1.l1.m1.1.2.3.2a.cmml" xref="alg1.l1.m1.1.2.3.2"><mtext id="alg1.l1.m1.1.2.3.2.cmml" xref="alg1.l1.m1.1.2.3.2">transformers.AutoModel</mtext></ci><ci id="alg1.l1.m1.1.1a.cmml" xref="alg1.l1.m1.1.2.3.3.2"><mtext id="alg1.l1.m1.1.1.cmml" xref="alg1.l1.m1.1.1">mdl</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l1.m1.1c">\mathbf{m}=\text{transformers.AutoModel}(\text{mdl})</annotation></semantics></math>

</div>
<div id="alg1.l2" class="ltx_listingline">  <math id="alg1.l2.m1.1" class="ltx_Math" alttext="\mathbf{t}=\text{transformers.AutoTokenizer}(\text{mdl})" display="inline"><semantics id="alg1.l2.m1.1a"><mrow id="alg1.l2.m1.1.2" xref="alg1.l2.m1.1.2.cmml"><mi id="alg1.l2.m1.1.2.2" xref="alg1.l2.m1.1.2.2.cmml">𝐭</mi><mo id="alg1.l2.m1.1.2.1" xref="alg1.l2.m1.1.2.1.cmml">=</mo><mrow id="alg1.l2.m1.1.2.3" xref="alg1.l2.m1.1.2.3.cmml"><mtext id="alg1.l2.m1.1.2.3.2" xref="alg1.l2.m1.1.2.3.2a.cmml">transformers.AutoTokenizer</mtext><mo lspace="0em" rspace="0em" id="alg1.l2.m1.1.2.3.1" xref="alg1.l2.m1.1.2.3.1.cmml">​</mo><mrow id="alg1.l2.m1.1.2.3.3.2" xref="alg1.l2.m1.1.1a.cmml"><mo stretchy="false" id="alg1.l2.m1.1.2.3.3.2.1" xref="alg1.l2.m1.1.1a.cmml">(</mo><mtext id="alg1.l2.m1.1.1" xref="alg1.l2.m1.1.1.cmml">mdl</mtext><mo stretchy="false" id="alg1.l2.m1.1.2.3.3.2.2" xref="alg1.l2.m1.1.1a.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l2.m1.1b"><apply id="alg1.l2.m1.1.2.cmml" xref="alg1.l2.m1.1.2"><eq id="alg1.l2.m1.1.2.1.cmml" xref="alg1.l2.m1.1.2.1"></eq><ci id="alg1.l2.m1.1.2.2.cmml" xref="alg1.l2.m1.1.2.2">𝐭</ci><apply id="alg1.l2.m1.1.2.3.cmml" xref="alg1.l2.m1.1.2.3"><times id="alg1.l2.m1.1.2.3.1.cmml" xref="alg1.l2.m1.1.2.3.1"></times><ci id="alg1.l2.m1.1.2.3.2a.cmml" xref="alg1.l2.m1.1.2.3.2"><mtext id="alg1.l2.m1.1.2.3.2.cmml" xref="alg1.l2.m1.1.2.3.2">transformers.AutoTokenizer</mtext></ci><ci id="alg1.l2.m1.1.1a.cmml" xref="alg1.l2.m1.1.2.3.3.2"><mtext id="alg1.l2.m1.1.1.cmml" xref="alg1.l2.m1.1.1">mdl</mtext></ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l2.m1.1c">\mathbf{t}=\text{transformers.AutoTokenizer}(\text{mdl})</annotation></semantics></math>

</div>
<div id="alg1.l3" class="ltx_listingline">  <math id="alg1.l3.m1.2" class="ltx_Math" alttext="\mathcal{C}^{+},C^{-}=[],[]" display="inline"><semantics id="alg1.l3.m1.2a"><mrow id="alg1.l3.m1.2.2.2" xref="alg1.l3.m1.2.2.3.cmml"><mrow id="alg1.l3.m1.1.1.1.1" xref="alg1.l3.m1.1.1.1.1.cmml"><mrow id="alg1.l3.m1.1.1.1.1.2.2" xref="alg1.l3.m1.1.1.1.1.2.3.cmml"><msup id="alg1.l3.m1.1.1.1.1.1.1.1" xref="alg1.l3.m1.1.1.1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l3.m1.1.1.1.1.1.1.1.2" xref="alg1.l3.m1.1.1.1.1.1.1.1.2.cmml">𝒞</mi><mo id="alg1.l3.m1.1.1.1.1.1.1.1.3" xref="alg1.l3.m1.1.1.1.1.1.1.1.3.cmml">+</mo></msup><mo id="alg1.l3.m1.1.1.1.1.2.2.3" xref="alg1.l3.m1.1.1.1.1.2.3.cmml">,</mo><msup id="alg1.l3.m1.1.1.1.1.2.2.2" xref="alg1.l3.m1.1.1.1.1.2.2.2.cmml"><mi id="alg1.l3.m1.1.1.1.1.2.2.2.2" xref="alg1.l3.m1.1.1.1.1.2.2.2.2.cmml">C</mi><mo id="alg1.l3.m1.1.1.1.1.2.2.2.3" xref="alg1.l3.m1.1.1.1.1.2.2.2.3.cmml">−</mo></msup></mrow><mo id="alg1.l3.m1.1.1.1.1.3" xref="alg1.l3.m1.1.1.1.1.3.cmml">=</mo><mrow id="alg1.l3.m1.1.1.1.1.4.2" xref="alg1.l3.m1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l3.m1.1.1.1.1.4.2.1" xref="alg1.l3.m1.1.1.1.1.4.1.cmml">[</mo><mo stretchy="false" id="alg1.l3.m1.1.1.1.1.4.2.2" xref="alg1.l3.m1.1.1.1.1.4.1.cmml">]</mo></mrow></mrow><mo id="alg1.l3.m1.2.2.2.3" xref="alg1.l3.m1.2.2.3a.cmml">,</mo><mrow id="alg1.l3.m1.2.2.2.2.2" xref="alg1.l3.m1.2.2.3.cmml"><mo stretchy="false" id="alg1.l3.m1.2.2.2.2.2.1" xref="alg1.l3.m1.2.2.2.2.1.cmml">[</mo><mo stretchy="false" id="alg1.l3.m1.2.2.2.2.2.2" xref="alg1.l3.m1.2.2.2.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l3.m1.2b"><apply id="alg1.l3.m1.2.2.3.cmml" xref="alg1.l3.m1.2.2.2"><csymbol cd="ambiguous" id="alg1.l3.m1.2.2.3a.cmml" xref="alg1.l3.m1.2.2.2.3">formulae-sequence</csymbol><apply id="alg1.l3.m1.1.1.1.1.cmml" xref="alg1.l3.m1.1.1.1.1"><eq id="alg1.l3.m1.1.1.1.1.3.cmml" xref="alg1.l3.m1.1.1.1.1.3"></eq><list id="alg1.l3.m1.1.1.1.1.2.3.cmml" xref="alg1.l3.m1.1.1.1.1.2.2"><apply id="alg1.l3.m1.1.1.1.1.1.1.1.cmml" xref="alg1.l3.m1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l3.m1.1.1.1.1.1.1.1.1.cmml" xref="alg1.l3.m1.1.1.1.1.1.1.1">superscript</csymbol><ci id="alg1.l3.m1.1.1.1.1.1.1.1.2.cmml" xref="alg1.l3.m1.1.1.1.1.1.1.1.2">𝒞</ci><plus id="alg1.l3.m1.1.1.1.1.1.1.1.3.cmml" xref="alg1.l3.m1.1.1.1.1.1.1.1.3"></plus></apply><apply id="alg1.l3.m1.1.1.1.1.2.2.2.cmml" xref="alg1.l3.m1.1.1.1.1.2.2.2"><csymbol cd="ambiguous" id="alg1.l3.m1.1.1.1.1.2.2.2.1.cmml" xref="alg1.l3.m1.1.1.1.1.2.2.2">superscript</csymbol><ci id="alg1.l3.m1.1.1.1.1.2.2.2.2.cmml" xref="alg1.l3.m1.1.1.1.1.2.2.2.2">𝐶</ci><minus id="alg1.l3.m1.1.1.1.1.2.2.2.3.cmml" xref="alg1.l3.m1.1.1.1.1.2.2.2.3"></minus></apply></list><list id="alg1.l3.m1.1.1.1.1.4.1.cmml" xref="alg1.l3.m1.1.1.1.1.4.2.1"></list></apply><list id="alg1.l3.m1.2.2.2.2.1.cmml" xref="alg1.l3.m1.2.2.2.2.2.1"></list></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l3.m1.2c">\mathcal{C}^{+},C^{-}=[],[]</annotation></semantics></math>

</div>
<div id="alg1.l4" class="ltx_listingline">  <span id="alg1.l4.1" class="ltx_text ltx_font_bold">for</span> <math id="alg1.l4.m1.2" class="ltx_Math" alttext="(x^{+},x^{-})\textbf{ in }S" display="inline"><semantics id="alg1.l4.m1.2a"><mrow id="alg1.l4.m1.2.2" xref="alg1.l4.m1.2.2.cmml"><mrow id="alg1.l4.m1.2.2.2.2" xref="alg1.l4.m1.2.2.2.3.cmml"><mo stretchy="false" id="alg1.l4.m1.2.2.2.2.3" xref="alg1.l4.m1.2.2.2.3.cmml">(</mo><msup id="alg1.l4.m1.1.1.1.1.1" xref="alg1.l4.m1.1.1.1.1.1.cmml"><mi id="alg1.l4.m1.1.1.1.1.1.2" xref="alg1.l4.m1.1.1.1.1.1.2.cmml">x</mi><mo id="alg1.l4.m1.1.1.1.1.1.3" xref="alg1.l4.m1.1.1.1.1.1.3.cmml">+</mo></msup><mo id="alg1.l4.m1.2.2.2.2.4" xref="alg1.l4.m1.2.2.2.3.cmml">,</mo><msup id="alg1.l4.m1.2.2.2.2.2" xref="alg1.l4.m1.2.2.2.2.2.cmml"><mi id="alg1.l4.m1.2.2.2.2.2.2" xref="alg1.l4.m1.2.2.2.2.2.2.cmml">x</mi><mo id="alg1.l4.m1.2.2.2.2.2.3" xref="alg1.l4.m1.2.2.2.2.2.3.cmml">−</mo></msup><mo stretchy="false" id="alg1.l4.m1.2.2.2.2.5" xref="alg1.l4.m1.2.2.2.3.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="alg1.l4.m1.2.2.3" xref="alg1.l4.m1.2.2.3.cmml">​</mo><mtext class="ltx_mathvariant_bold" id="alg1.l4.m1.2.2.4" xref="alg1.l4.m1.2.2.4a.cmml"> in </mtext><mo lspace="0em" rspace="0em" id="alg1.l4.m1.2.2.3a" xref="alg1.l4.m1.2.2.3.cmml">​</mo><mi id="alg1.l4.m1.2.2.5" xref="alg1.l4.m1.2.2.5.cmml">S</mi></mrow><annotation-xml encoding="MathML-Content" id="alg1.l4.m1.2b"><apply id="alg1.l4.m1.2.2.cmml" xref="alg1.l4.m1.2.2"><times id="alg1.l4.m1.2.2.3.cmml" xref="alg1.l4.m1.2.2.3"></times><interval closure="open" id="alg1.l4.m1.2.2.2.3.cmml" xref="alg1.l4.m1.2.2.2.2"><apply id="alg1.l4.m1.1.1.1.1.1.cmml" xref="alg1.l4.m1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l4.m1.1.1.1.1.1.1.cmml" xref="alg1.l4.m1.1.1.1.1.1">superscript</csymbol><ci id="alg1.l4.m1.1.1.1.1.1.2.cmml" xref="alg1.l4.m1.1.1.1.1.1.2">𝑥</ci><plus id="alg1.l4.m1.1.1.1.1.1.3.cmml" xref="alg1.l4.m1.1.1.1.1.1.3"></plus></apply><apply id="alg1.l4.m1.2.2.2.2.2.cmml" xref="alg1.l4.m1.2.2.2.2.2"><csymbol cd="ambiguous" id="alg1.l4.m1.2.2.2.2.2.1.cmml" xref="alg1.l4.m1.2.2.2.2.2">superscript</csymbol><ci id="alg1.l4.m1.2.2.2.2.2.2.cmml" xref="alg1.l4.m1.2.2.2.2.2.2">𝑥</ci><minus id="alg1.l4.m1.2.2.2.2.2.3.cmml" xref="alg1.l4.m1.2.2.2.2.2.3"></minus></apply></interval><ci id="alg1.l4.m1.2.2.4a.cmml" xref="alg1.l4.m1.2.2.4"><mtext class="ltx_mathvariant_bold" id="alg1.l4.m1.2.2.4.cmml" xref="alg1.l4.m1.2.2.4"> in </mtext></ci><ci id="alg1.l4.m1.2.2.5.cmml" xref="alg1.l4.m1.2.2.5">𝑆</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l4.m1.2c">(x^{+},x^{-})\textbf{ in }S</annotation></semantics></math> <span id="alg1.l4.2" class="ltx_text ltx_font_bold">do</span>



</div>
<div id="alg1.l5" class="ltx_listingline">     token<sup id="alg1.l5.1" class="ltx_sup">+</sup>, token<math id="alg1.l5.m2.1" class="ltx_math_unparsed" alttext="{}^{-}=\mathbf{t}" display="inline"><semantics id="alg1.l5.m2.1a"><mrow id="alg1.l5.m2.1b"><mmultiscripts id="alg1.l5.m2.1.1"><mo id="alg1.l5.m2.1.1.2">=</mo><mprescripts id="alg1.l5.m2.1.1a"></mprescripts><mrow id="alg1.l5.m2.1.1b"></mrow><mo id="alg1.l5.m2.1.1.3">−</mo></mmultiscripts><mi id="alg1.l5.m2.1.2">𝐭</mi></mrow><annotation encoding="application/x-tex" id="alg1.l5.m2.1c">{}^{-}=\mathbf{t}</annotation></semantics></math>.encode<math id="alg1.l5.m3.1" class="ltx_Math" alttext="(x^{+})" display="inline"><semantics id="alg1.l5.m3.1a"><mrow id="alg1.l5.m3.1.1.1" xref="alg1.l5.m3.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l5.m3.1.1.1.2" xref="alg1.l5.m3.1.1.1.1.cmml">(</mo><msup id="alg1.l5.m3.1.1.1.1" xref="alg1.l5.m3.1.1.1.1.cmml"><mi id="alg1.l5.m3.1.1.1.1.2" xref="alg1.l5.m3.1.1.1.1.2.cmml">x</mi><mo id="alg1.l5.m3.1.1.1.1.3" xref="alg1.l5.m3.1.1.1.1.3.cmml">+</mo></msup><mo stretchy="false" id="alg1.l5.m3.1.1.1.3" xref="alg1.l5.m3.1.1.1.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="alg1.l5.m3.1b"><apply id="alg1.l5.m3.1.1.1.1.cmml" xref="alg1.l5.m3.1.1.1"><csymbol cd="ambiguous" id="alg1.l5.m3.1.1.1.1.1.cmml" xref="alg1.l5.m3.1.1.1">superscript</csymbol><ci id="alg1.l5.m3.1.1.1.1.2.cmml" xref="alg1.l5.m3.1.1.1.1.2">𝑥</ci><plus id="alg1.l5.m3.1.1.1.1.3.cmml" xref="alg1.l5.m3.1.1.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m3.1c">(x^{+})</annotation></semantics></math>, <math id="alg1.l5.m4.1" class="ltx_Math" alttext="\mathbf{t}" display="inline"><semantics id="alg1.l5.m4.1a"><mi id="alg1.l5.m4.1.1" xref="alg1.l5.m4.1.1.cmml">𝐭</mi><annotation-xml encoding="MathML-Content" id="alg1.l5.m4.1b"><ci id="alg1.l5.m4.1.1.cmml" xref="alg1.l5.m4.1.1">𝐭</ci></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m4.1c">\mathbf{t}</annotation></semantics></math>.encode<math id="alg1.l5.m5.1" class="ltx_Math" alttext="(x^{-})" display="inline"><semantics id="alg1.l5.m5.1a"><mrow id="alg1.l5.m5.1.1.1" xref="alg1.l5.m5.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l5.m5.1.1.1.2" xref="alg1.l5.m5.1.1.1.1.cmml">(</mo><msup id="alg1.l5.m5.1.1.1.1" xref="alg1.l5.m5.1.1.1.1.cmml"><mi id="alg1.l5.m5.1.1.1.1.2" xref="alg1.l5.m5.1.1.1.1.2.cmml">x</mi><mo id="alg1.l5.m5.1.1.1.1.3" xref="alg1.l5.m5.1.1.1.1.3.cmml">−</mo></msup><mo stretchy="false" id="alg1.l5.m5.1.1.1.3" xref="alg1.l5.m5.1.1.1.1.cmml">)</mo></mrow><annotation-xml encoding="MathML-Content" id="alg1.l5.m5.1b"><apply id="alg1.l5.m5.1.1.1.1.cmml" xref="alg1.l5.m5.1.1.1"><csymbol cd="ambiguous" id="alg1.l5.m5.1.1.1.1.1.cmml" xref="alg1.l5.m5.1.1.1">superscript</csymbol><ci id="alg1.l5.m5.1.1.1.1.2.cmml" xref="alg1.l5.m5.1.1.1.1.2">𝑥</ci><minus id="alg1.l5.m5.1.1.1.1.3.cmml" xref="alg1.l5.m5.1.1.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l5.m5.1c">(x^{-})</annotation></semantics></math>

</div>
<div id="alg1.l6" class="ltx_listingline">     <math id="alg1.l6.m1.4" class="ltx_Math" alttext="\phi^{+}=m(\text{token}^{+},\text{output\_hiddenstates = True})\text{[``hidden\_states'']}[\text{idx}][-1]" display="inline"><semantics id="alg1.l6.m1.4a"><mrow id="alg1.l6.m1.4.4" xref="alg1.l6.m1.4.4.cmml"><msup id="alg1.l6.m1.4.4.4" xref="alg1.l6.m1.4.4.4.cmml"><mi id="alg1.l6.m1.4.4.4.2" xref="alg1.l6.m1.4.4.4.2.cmml">ϕ</mi><mo id="alg1.l6.m1.4.4.4.3" xref="alg1.l6.m1.4.4.4.3.cmml">+</mo></msup><mo id="alg1.l6.m1.4.4.3" xref="alg1.l6.m1.4.4.3.cmml">=</mo><mrow id="alg1.l6.m1.4.4.2" xref="alg1.l6.m1.4.4.2.cmml"><mi id="alg1.l6.m1.4.4.2.4" xref="alg1.l6.m1.4.4.2.4.cmml">m</mi><mo lspace="0em" rspace="0em" id="alg1.l6.m1.4.4.2.3" xref="alg1.l6.m1.4.4.2.3.cmml">​</mo><mrow id="alg1.l6.m1.3.3.1.1.1" xref="alg1.l6.m1.3.3.1.1.2.cmml"><mo stretchy="false" id="alg1.l6.m1.3.3.1.1.1.2" xref="alg1.l6.m1.3.3.1.1.2.cmml">(</mo><msup id="alg1.l6.m1.3.3.1.1.1.1" xref="alg1.l6.m1.3.3.1.1.1.1.cmml"><mtext id="alg1.l6.m1.3.3.1.1.1.1.2" xref="alg1.l6.m1.3.3.1.1.1.1.2a.cmml">token</mtext><mo id="alg1.l6.m1.3.3.1.1.1.1.3" xref="alg1.l6.m1.3.3.1.1.1.1.3.cmml">+</mo></msup><mo id="alg1.l6.m1.3.3.1.1.1.3" xref="alg1.l6.m1.3.3.1.1.2.cmml">,</mo><mtext id="alg1.l6.m1.1.1" xref="alg1.l6.m1.1.1a.cmml">output_hiddenstates = True</mtext><mo stretchy="false" id="alg1.l6.m1.3.3.1.1.1.4" xref="alg1.l6.m1.3.3.1.1.2.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="alg1.l6.m1.4.4.2.3a" xref="alg1.l6.m1.4.4.2.3.cmml">​</mo><mtext id="alg1.l6.m1.4.4.2.5" xref="alg1.l6.m1.4.4.2.5a.cmml">[“hidden_states”]</mtext><mo lspace="0em" rspace="0em" id="alg1.l6.m1.4.4.2.3b" xref="alg1.l6.m1.4.4.2.3.cmml">​</mo><mrow id="alg1.l6.m1.4.4.2.6.2" xref="alg1.l6.m1.4.4.2.6.1.cmml"><mo stretchy="false" id="alg1.l6.m1.4.4.2.6.2.1" xref="alg1.l6.m1.4.4.2.6.1.1.cmml">[</mo><mtext id="alg1.l6.m1.2.2" xref="alg1.l6.m1.2.2a.cmml">idx</mtext><mo stretchy="false" id="alg1.l6.m1.4.4.2.6.2.2" xref="alg1.l6.m1.4.4.2.6.1.1.cmml">]</mo></mrow><mo lspace="0em" rspace="0em" id="alg1.l6.m1.4.4.2.3c" xref="alg1.l6.m1.4.4.2.3.cmml">​</mo><mrow id="alg1.l6.m1.4.4.2.2.1" xref="alg1.l6.m1.4.4.2.2.2.cmml"><mo stretchy="false" id="alg1.l6.m1.4.4.2.2.1.2" xref="alg1.l6.m1.4.4.2.2.2.1.cmml">[</mo><mrow id="alg1.l6.m1.4.4.2.2.1.1" xref="alg1.l6.m1.4.4.2.2.1.1.cmml"><mo id="alg1.l6.m1.4.4.2.2.1.1a" xref="alg1.l6.m1.4.4.2.2.1.1.cmml">−</mo><mn id="alg1.l6.m1.4.4.2.2.1.1.2" xref="alg1.l6.m1.4.4.2.2.1.1.2.cmml">1</mn></mrow><mo stretchy="false" id="alg1.l6.m1.4.4.2.2.1.3" xref="alg1.l6.m1.4.4.2.2.2.1.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l6.m1.4b"><apply id="alg1.l6.m1.4.4.cmml" xref="alg1.l6.m1.4.4"><eq id="alg1.l6.m1.4.4.3.cmml" xref="alg1.l6.m1.4.4.3"></eq><apply id="alg1.l6.m1.4.4.4.cmml" xref="alg1.l6.m1.4.4.4"><csymbol cd="ambiguous" id="alg1.l6.m1.4.4.4.1.cmml" xref="alg1.l6.m1.4.4.4">superscript</csymbol><ci id="alg1.l6.m1.4.4.4.2.cmml" xref="alg1.l6.m1.4.4.4.2">italic-ϕ</ci><plus id="alg1.l6.m1.4.4.4.3.cmml" xref="alg1.l6.m1.4.4.4.3"></plus></apply><apply id="alg1.l6.m1.4.4.2.cmml" xref="alg1.l6.m1.4.4.2"><times id="alg1.l6.m1.4.4.2.3.cmml" xref="alg1.l6.m1.4.4.2.3"></times><ci id="alg1.l6.m1.4.4.2.4.cmml" xref="alg1.l6.m1.4.4.2.4">𝑚</ci><interval closure="open" id="alg1.l6.m1.3.3.1.1.2.cmml" xref="alg1.l6.m1.3.3.1.1.1"><apply id="alg1.l6.m1.3.3.1.1.1.1.cmml" xref="alg1.l6.m1.3.3.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l6.m1.3.3.1.1.1.1.1.cmml" xref="alg1.l6.m1.3.3.1.1.1.1">superscript</csymbol><ci id="alg1.l6.m1.3.3.1.1.1.1.2a.cmml" xref="alg1.l6.m1.3.3.1.1.1.1.2"><mtext id="alg1.l6.m1.3.3.1.1.1.1.2.cmml" xref="alg1.l6.m1.3.3.1.1.1.1.2">token</mtext></ci><plus id="alg1.l6.m1.3.3.1.1.1.1.3.cmml" xref="alg1.l6.m1.3.3.1.1.1.1.3"></plus></apply><ci id="alg1.l6.m1.1.1a.cmml" xref="alg1.l6.m1.1.1"><mtext id="alg1.l6.m1.1.1.cmml" xref="alg1.l6.m1.1.1">output_hiddenstates = True</mtext></ci></interval><ci id="alg1.l6.m1.4.4.2.5a.cmml" xref="alg1.l6.m1.4.4.2.5"><mtext id="alg1.l6.m1.4.4.2.5.cmml" xref="alg1.l6.m1.4.4.2.5">[“hidden_states”]</mtext></ci><apply id="alg1.l6.m1.4.4.2.6.1.cmml" xref="alg1.l6.m1.4.4.2.6.2"><csymbol cd="latexml" id="alg1.l6.m1.4.4.2.6.1.1.cmml" xref="alg1.l6.m1.4.4.2.6.2.1">delimited-[]</csymbol><ci id="alg1.l6.m1.2.2a.cmml" xref="alg1.l6.m1.2.2"><mtext id="alg1.l6.m1.2.2.cmml" xref="alg1.l6.m1.2.2">idx</mtext></ci></apply><apply id="alg1.l6.m1.4.4.2.2.2.cmml" xref="alg1.l6.m1.4.4.2.2.1"><csymbol cd="latexml" id="alg1.l6.m1.4.4.2.2.2.1.cmml" xref="alg1.l6.m1.4.4.2.2.1.2">delimited-[]</csymbol><apply id="alg1.l6.m1.4.4.2.2.1.1.cmml" xref="alg1.l6.m1.4.4.2.2.1.1"><minus id="alg1.l6.m1.4.4.2.2.1.1.1.cmml" xref="alg1.l6.m1.4.4.2.2.1.1"></minus><cn type="integer" id="alg1.l6.m1.4.4.2.2.1.1.2.cmml" xref="alg1.l6.m1.4.4.2.2.1.1.2">1</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l6.m1.4c">\phi^{+}=m(\text{token}^{+},\text{output\_hiddenstates = True})\text{[``hidden\_states'']}[\text{idx}][-1]</annotation></semantics></math>

</div>
<div id="alg1.l7" class="ltx_listingline">     <math id="alg1.l7.m1.4" class="ltx_Math" alttext="\phi^{-}=m(\text{token}^{-},\text{output\_hiddenstates = True})\text{[``hidden\_states'']}[\text{idx}][-1]" display="inline"><semantics id="alg1.l7.m1.4a"><mrow id="alg1.l7.m1.4.4" xref="alg1.l7.m1.4.4.cmml"><msup id="alg1.l7.m1.4.4.4" xref="alg1.l7.m1.4.4.4.cmml"><mi id="alg1.l7.m1.4.4.4.2" xref="alg1.l7.m1.4.4.4.2.cmml">ϕ</mi><mo id="alg1.l7.m1.4.4.4.3" xref="alg1.l7.m1.4.4.4.3.cmml">−</mo></msup><mo id="alg1.l7.m1.4.4.3" xref="alg1.l7.m1.4.4.3.cmml">=</mo><mrow id="alg1.l7.m1.4.4.2" xref="alg1.l7.m1.4.4.2.cmml"><mi id="alg1.l7.m1.4.4.2.4" xref="alg1.l7.m1.4.4.2.4.cmml">m</mi><mo lspace="0em" rspace="0em" id="alg1.l7.m1.4.4.2.3" xref="alg1.l7.m1.4.4.2.3.cmml">​</mo><mrow id="alg1.l7.m1.3.3.1.1.1" xref="alg1.l7.m1.3.3.1.1.2.cmml"><mo stretchy="false" id="alg1.l7.m1.3.3.1.1.1.2" xref="alg1.l7.m1.3.3.1.1.2.cmml">(</mo><msup id="alg1.l7.m1.3.3.1.1.1.1" xref="alg1.l7.m1.3.3.1.1.1.1.cmml"><mtext id="alg1.l7.m1.3.3.1.1.1.1.2" xref="alg1.l7.m1.3.3.1.1.1.1.2a.cmml">token</mtext><mo id="alg1.l7.m1.3.3.1.1.1.1.3" xref="alg1.l7.m1.3.3.1.1.1.1.3.cmml">−</mo></msup><mo id="alg1.l7.m1.3.3.1.1.1.3" xref="alg1.l7.m1.3.3.1.1.2.cmml">,</mo><mtext id="alg1.l7.m1.1.1" xref="alg1.l7.m1.1.1a.cmml">output_hiddenstates = True</mtext><mo stretchy="false" id="alg1.l7.m1.3.3.1.1.1.4" xref="alg1.l7.m1.3.3.1.1.2.cmml">)</mo></mrow><mo lspace="0em" rspace="0em" id="alg1.l7.m1.4.4.2.3a" xref="alg1.l7.m1.4.4.2.3.cmml">​</mo><mtext id="alg1.l7.m1.4.4.2.5" xref="alg1.l7.m1.4.4.2.5a.cmml">[“hidden_states”]</mtext><mo lspace="0em" rspace="0em" id="alg1.l7.m1.4.4.2.3b" xref="alg1.l7.m1.4.4.2.3.cmml">​</mo><mrow id="alg1.l7.m1.4.4.2.6.2" xref="alg1.l7.m1.4.4.2.6.1.cmml"><mo stretchy="false" id="alg1.l7.m1.4.4.2.6.2.1" xref="alg1.l7.m1.4.4.2.6.1.1.cmml">[</mo><mtext id="alg1.l7.m1.2.2" xref="alg1.l7.m1.2.2a.cmml">idx</mtext><mo stretchy="false" id="alg1.l7.m1.4.4.2.6.2.2" xref="alg1.l7.m1.4.4.2.6.1.1.cmml">]</mo></mrow><mo lspace="0em" rspace="0em" id="alg1.l7.m1.4.4.2.3c" xref="alg1.l7.m1.4.4.2.3.cmml">​</mo><mrow id="alg1.l7.m1.4.4.2.2.1" xref="alg1.l7.m1.4.4.2.2.2.cmml"><mo stretchy="false" id="alg1.l7.m1.4.4.2.2.1.2" xref="alg1.l7.m1.4.4.2.2.2.1.cmml">[</mo><mrow id="alg1.l7.m1.4.4.2.2.1.1" xref="alg1.l7.m1.4.4.2.2.1.1.cmml"><mo id="alg1.l7.m1.4.4.2.2.1.1a" xref="alg1.l7.m1.4.4.2.2.1.1.cmml">−</mo><mn id="alg1.l7.m1.4.4.2.2.1.1.2" xref="alg1.l7.m1.4.4.2.2.1.1.2.cmml">1</mn></mrow><mo stretchy="false" id="alg1.l7.m1.4.4.2.2.1.3" xref="alg1.l7.m1.4.4.2.2.2.1.cmml">]</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l7.m1.4b"><apply id="alg1.l7.m1.4.4.cmml" xref="alg1.l7.m1.4.4"><eq id="alg1.l7.m1.4.4.3.cmml" xref="alg1.l7.m1.4.4.3"></eq><apply id="alg1.l7.m1.4.4.4.cmml" xref="alg1.l7.m1.4.4.4"><csymbol cd="ambiguous" id="alg1.l7.m1.4.4.4.1.cmml" xref="alg1.l7.m1.4.4.4">superscript</csymbol><ci id="alg1.l7.m1.4.4.4.2.cmml" xref="alg1.l7.m1.4.4.4.2">italic-ϕ</ci><minus id="alg1.l7.m1.4.4.4.3.cmml" xref="alg1.l7.m1.4.4.4.3"></minus></apply><apply id="alg1.l7.m1.4.4.2.cmml" xref="alg1.l7.m1.4.4.2"><times id="alg1.l7.m1.4.4.2.3.cmml" xref="alg1.l7.m1.4.4.2.3"></times><ci id="alg1.l7.m1.4.4.2.4.cmml" xref="alg1.l7.m1.4.4.2.4">𝑚</ci><interval closure="open" id="alg1.l7.m1.3.3.1.1.2.cmml" xref="alg1.l7.m1.3.3.1.1.1"><apply id="alg1.l7.m1.3.3.1.1.1.1.cmml" xref="alg1.l7.m1.3.3.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l7.m1.3.3.1.1.1.1.1.cmml" xref="alg1.l7.m1.3.3.1.1.1.1">superscript</csymbol><ci id="alg1.l7.m1.3.3.1.1.1.1.2a.cmml" xref="alg1.l7.m1.3.3.1.1.1.1.2"><mtext id="alg1.l7.m1.3.3.1.1.1.1.2.cmml" xref="alg1.l7.m1.3.3.1.1.1.1.2">token</mtext></ci><minus id="alg1.l7.m1.3.3.1.1.1.1.3.cmml" xref="alg1.l7.m1.3.3.1.1.1.1.3"></minus></apply><ci id="alg1.l7.m1.1.1a.cmml" xref="alg1.l7.m1.1.1"><mtext id="alg1.l7.m1.1.1.cmml" xref="alg1.l7.m1.1.1">output_hiddenstates = True</mtext></ci></interval><ci id="alg1.l7.m1.4.4.2.5a.cmml" xref="alg1.l7.m1.4.4.2.5"><mtext id="alg1.l7.m1.4.4.2.5.cmml" xref="alg1.l7.m1.4.4.2.5">[“hidden_states”]</mtext></ci><apply id="alg1.l7.m1.4.4.2.6.1.cmml" xref="alg1.l7.m1.4.4.2.6.2"><csymbol cd="latexml" id="alg1.l7.m1.4.4.2.6.1.1.cmml" xref="alg1.l7.m1.4.4.2.6.2.1">delimited-[]</csymbol><ci id="alg1.l7.m1.2.2a.cmml" xref="alg1.l7.m1.2.2"><mtext id="alg1.l7.m1.2.2.cmml" xref="alg1.l7.m1.2.2">idx</mtext></ci></apply><apply id="alg1.l7.m1.4.4.2.2.2.cmml" xref="alg1.l7.m1.4.4.2.2.1"><csymbol cd="latexml" id="alg1.l7.m1.4.4.2.2.2.1.cmml" xref="alg1.l7.m1.4.4.2.2.1.2">delimited-[]</csymbol><apply id="alg1.l7.m1.4.4.2.2.1.1.cmml" xref="alg1.l7.m1.4.4.2.2.1.1"><minus id="alg1.l7.m1.4.4.2.2.1.1.1.cmml" xref="alg1.l7.m1.4.4.2.2.1.1"></minus><cn type="integer" id="alg1.l7.m1.4.4.2.2.1.1.2.cmml" xref="alg1.l7.m1.4.4.2.2.1.1.2">1</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="alg1.l7.m1.4c">\phi^{-}=m(\text{token}^{-},\text{output\_hiddenstates = True})\text{[``hidden\_states'']}[\text{idx}][-1]</annotation></semantics></math>

</div>
<div id="alg1.l8" class="ltx_listingline">     <math id="alg1.l8.m1.2" class="ltx_Math" alttext="\mathcal{C}^{+}\text{.append}(\phi^{+}),\mathcal{C}^{-}\text{.append}(\phi^{-})" display="inline"><semantics id="alg1.l8.m1.2a"><mrow id="alg1.l8.m1.2.2.2" xref="alg1.l8.m1.2.2.3.cmml"><mrow id="alg1.l8.m1.1.1.1.1" xref="alg1.l8.m1.1.1.1.1.cmml"><msup id="alg1.l8.m1.1.1.1.1.3" xref="alg1.l8.m1.1.1.1.1.3.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l8.m1.1.1.1.1.3.2" xref="alg1.l8.m1.1.1.1.1.3.2.cmml">𝒞</mi><mo id="alg1.l8.m1.1.1.1.1.3.3" xref="alg1.l8.m1.1.1.1.1.3.3.cmml">+</mo></msup><mo lspace="0em" rspace="0em" id="alg1.l8.m1.1.1.1.1.2" xref="alg1.l8.m1.1.1.1.1.2.cmml">​</mo><mtext id="alg1.l8.m1.1.1.1.1.4" xref="alg1.l8.m1.1.1.1.1.4a.cmml">.append</mtext><mo lspace="0em" rspace="0em" id="alg1.l8.m1.1.1.1.1.2a" xref="alg1.l8.m1.1.1.1.1.2.cmml">​</mo><mrow id="alg1.l8.m1.1.1.1.1.1.1" xref="alg1.l8.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="alg1.l8.m1.1.1.1.1.1.1.2" xref="alg1.l8.m1.1.1.1.1.1.1.1.cmml">(</mo><msup id="alg1.l8.m1.1.1.1.1.1.1.1" xref="alg1.l8.m1.1.1.1.1.1.1.1.cmml"><mi id="alg1.l8.m1.1.1.1.1.1.1.1.2" xref="alg1.l8.m1.1.1.1.1.1.1.1.2.cmml">ϕ</mi><mo id="alg1.l8.m1.1.1.1.1.1.1.1.3" xref="alg1.l8.m1.1.1.1.1.1.1.1.3.cmml">+</mo></msup><mo stretchy="false" id="alg1.l8.m1.1.1.1.1.1.1.3" xref="alg1.l8.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="alg1.l8.m1.2.2.2.3" xref="alg1.l8.m1.2.2.3.cmml">,</mo><mrow id="alg1.l8.m1.2.2.2.2" xref="alg1.l8.m1.2.2.2.2.cmml"><msup id="alg1.l8.m1.2.2.2.2.3" xref="alg1.l8.m1.2.2.2.2.3.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l8.m1.2.2.2.2.3.2" xref="alg1.l8.m1.2.2.2.2.3.2.cmml">𝒞</mi><mo id="alg1.l8.m1.2.2.2.2.3.3" xref="alg1.l8.m1.2.2.2.2.3.3.cmml">−</mo></msup><mo lspace="0em" rspace="0em" id="alg1.l8.m1.2.2.2.2.2" xref="alg1.l8.m1.2.2.2.2.2.cmml">​</mo><mtext id="alg1.l8.m1.2.2.2.2.4" xref="alg1.l8.m1.2.2.2.2.4a.cmml">.append</mtext><mo lspace="0em" rspace="0em" id="alg1.l8.m1.2.2.2.2.2a" xref="alg1.l8.m1.2.2.2.2.2.cmml">​</mo><mrow id="alg1.l8.m1.2.2.2.2.1.1" xref="alg1.l8.m1.2.2.2.2.1.1.1.cmml"><mo stretchy="false" id="alg1.l8.m1.2.2.2.2.1.1.2" xref="alg1.l8.m1.2.2.2.2.1.1.1.cmml">(</mo><msup id="alg1.l8.m1.2.2.2.2.1.1.1" xref="alg1.l8.m1.2.2.2.2.1.1.1.cmml"><mi id="alg1.l8.m1.2.2.2.2.1.1.1.2" xref="alg1.l8.m1.2.2.2.2.1.1.1.2.cmml">ϕ</mi><mo id="alg1.l8.m1.2.2.2.2.1.1.1.3" xref="alg1.l8.m1.2.2.2.2.1.1.1.3.cmml">−</mo></msup><mo stretchy="false" id="alg1.l8.m1.2.2.2.2.1.1.3" xref="alg1.l8.m1.2.2.2.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="alg1.l8.m1.2b"><list id="alg1.l8.m1.2.2.3.cmml" xref="alg1.l8.m1.2.2.2"><apply id="alg1.l8.m1.1.1.1.1.cmml" xref="alg1.l8.m1.1.1.1.1"><times id="alg1.l8.m1.1.1.1.1.2.cmml" xref="alg1.l8.m1.1.1.1.1.2"></times><apply id="alg1.l8.m1.1.1.1.1.3.cmml" xref="alg1.l8.m1.1.1.1.1.3"><csymbol cd="ambiguous" id="alg1.l8.m1.1.1.1.1.3.1.cmml" xref="alg1.l8.m1.1.1.1.1.3">superscript</csymbol><ci id="alg1.l8.m1.1.1.1.1.3.2.cmml" xref="alg1.l8.m1.1.1.1.1.3.2">𝒞</ci><plus id="alg1.l8.m1.1.1.1.1.3.3.cmml" xref="alg1.l8.m1.1.1.1.1.3.3"></plus></apply><ci id="alg1.l8.m1.1.1.1.1.4a.cmml" xref="alg1.l8.m1.1.1.1.1.4"><mtext id="alg1.l8.m1.1.1.1.1.4.cmml" xref="alg1.l8.m1.1.1.1.1.4">.append</mtext></ci><apply id="alg1.l8.m1.1.1.1.1.1.1.1.cmml" xref="alg1.l8.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l8.m1.1.1.1.1.1.1.1.1.cmml" xref="alg1.l8.m1.1.1.1.1.1.1">superscript</csymbol><ci id="alg1.l8.m1.1.1.1.1.1.1.1.2.cmml" xref="alg1.l8.m1.1.1.1.1.1.1.1.2">italic-ϕ</ci><plus id="alg1.l8.m1.1.1.1.1.1.1.1.3.cmml" xref="alg1.l8.m1.1.1.1.1.1.1.1.3"></plus></apply></apply><apply id="alg1.l8.m1.2.2.2.2.cmml" xref="alg1.l8.m1.2.2.2.2"><times id="alg1.l8.m1.2.2.2.2.2.cmml" xref="alg1.l8.m1.2.2.2.2.2"></times><apply id="alg1.l8.m1.2.2.2.2.3.cmml" xref="alg1.l8.m1.2.2.2.2.3"><csymbol cd="ambiguous" id="alg1.l8.m1.2.2.2.2.3.1.cmml" xref="alg1.l8.m1.2.2.2.2.3">superscript</csymbol><ci id="alg1.l8.m1.2.2.2.2.3.2.cmml" xref="alg1.l8.m1.2.2.2.2.3.2">𝒞</ci><minus id="alg1.l8.m1.2.2.2.2.3.3.cmml" xref="alg1.l8.m1.2.2.2.2.3.3"></minus></apply><ci id="alg1.l8.m1.2.2.2.2.4a.cmml" xref="alg1.l8.m1.2.2.2.2.4"><mtext id="alg1.l8.m1.2.2.2.2.4.cmml" xref="alg1.l8.m1.2.2.2.2.4">.append</mtext></ci><apply id="alg1.l8.m1.2.2.2.2.1.1.1.cmml" xref="alg1.l8.m1.2.2.2.2.1.1"><csymbol cd="ambiguous" id="alg1.l8.m1.2.2.2.2.1.1.1.1.cmml" xref="alg1.l8.m1.2.2.2.2.1.1">superscript</csymbol><ci id="alg1.l8.m1.2.2.2.2.1.1.1.2.cmml" xref="alg1.l8.m1.2.2.2.2.1.1.1.2">italic-ϕ</ci><minus id="alg1.l8.m1.2.2.2.2.1.1.1.3.cmml" xref="alg1.l8.m1.2.2.2.2.1.1.1.3"></minus></apply></apply></list></annotation-xml><annotation encoding="application/x-tex" id="alg1.l8.m1.2c">\mathcal{C}^{+}\text{.append}(\phi^{+}),\mathcal{C}^{-}\text{.append}(\phi^{-})</annotation></semantics></math>

</div>
<div id="alg1.l9" class="ltx_listingline">  <span id="alg1.l9.1" class="ltx_text ltx_font_bold">end</span> <span id="alg1.l9.2" class="ltx_text ltx_font_bold">for</span>
</div>
<div id="alg1.l9a" class="ltx_listingline">  <math id="alg1.l9a.m1.2" class="ltx_Math" alttext="[\mathcal{C}^{+},\mathcal{C}^{-}]" display="inline"><semantics id="alg1.l9a.m1.2a"><mrow id="alg1.l9a.m1.2.2.2" xref="alg1.l9a.m1.2.2.3.cmml"><mo stretchy="false" id="alg1.l9a.m1.2.2.2.3" xref="alg1.l9a.m1.2.2.3.cmml">[</mo><msup id="alg1.l9a.m1.1.1.1.1" xref="alg1.l9a.m1.1.1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l9a.m1.1.1.1.1.2" xref="alg1.l9a.m1.1.1.1.1.2.cmml">𝒞</mi><mo id="alg1.l9a.m1.1.1.1.1.3" xref="alg1.l9a.m1.1.1.1.1.3.cmml">+</mo></msup><mo id="alg1.l9a.m1.2.2.2.4" xref="alg1.l9a.m1.2.2.3.cmml">,</mo><msup id="alg1.l9a.m1.2.2.2.2" xref="alg1.l9a.m1.2.2.2.2.cmml"><mi class="ltx_font_mathcaligraphic" id="alg1.l9a.m1.2.2.2.2.2" xref="alg1.l9a.m1.2.2.2.2.2.cmml">𝒞</mi><mo id="alg1.l9a.m1.2.2.2.2.3" xref="alg1.l9a.m1.2.2.2.2.3.cmml">−</mo></msup><mo stretchy="false" id="alg1.l9a.m1.2.2.2.5" xref="alg1.l9a.m1.2.2.3.cmml">]</mo></mrow><annotation-xml encoding="MathML-Content" id="alg1.l9a.m1.2b"><interval closure="closed" id="alg1.l9a.m1.2.2.3.cmml" xref="alg1.l9a.m1.2.2.2"><apply id="alg1.l9a.m1.1.1.1.1.cmml" xref="alg1.l9a.m1.1.1.1.1"><csymbol cd="ambiguous" id="alg1.l9a.m1.1.1.1.1.1.cmml" xref="alg1.l9a.m1.1.1.1.1">superscript</csymbol><ci id="alg1.l9a.m1.1.1.1.1.2.cmml" xref="alg1.l9a.m1.1.1.1.1.2">𝒞</ci><plus id="alg1.l9a.m1.1.1.1.1.3.cmml" xref="alg1.l9a.m1.1.1.1.1.3"></plus></apply><apply id="alg1.l9a.m1.2.2.2.2.cmml" xref="alg1.l9a.m1.2.2.2.2"><csymbol cd="ambiguous" id="alg1.l9a.m1.2.2.2.2.1.cmml" xref="alg1.l9a.m1.2.2.2.2">superscript</csymbol><ci id="alg1.l9a.m1.2.2.2.2.2.cmml" xref="alg1.l9a.m1.2.2.2.2.2">𝒞</ci><minus id="alg1.l9a.m1.2.2.2.2.3.cmml" xref="alg1.l9a.m1.2.2.2.2.3"></minus></apply></interval></annotation-xml><annotation encoding="application/x-tex" id="alg1.l9a.m1.2c">[\mathcal{C}^{+},\mathcal{C}^{-}]</annotation></semantics></math>

</div>
</div>
</figure>
</section>
<section id="A7.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.2 </span>CCS</h3>

<div id="A7.SS2.p1" class="ltx_para ltx_noindent">
<p id="A7.SS2.p1.10" class="ltx_p">Given contrast features from <a href="#alg1" title="In G.1 Contrast Pairs ‣ Appendix G CCS and CRC Implementation Details ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Algorithm</span> <span class="ltx_text ltx_ref_tag">1</span></a>, CCS works by learning parameters <math id="A7.SS2.p1.1.m1.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="A7.SS2.p1.1.m1.1a"><mi id="A7.SS2.p1.1.m1.1.1" xref="A7.SS2.p1.1.m1.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="A7.SS2.p1.1.m1.1b"><ci id="A7.SS2.p1.1.m1.1.1.cmml" xref="A7.SS2.p1.1.m1.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="A7.SS2.p1.1.m1.1c">\theta</annotation></semantics></math> and <math id="A7.SS2.p1.2.m2.1" class="ltx_Math" alttext="b" display="inline"><semantics id="A7.SS2.p1.2.m2.1a"><mi id="A7.SS2.p1.2.m2.1.1" xref="A7.SS2.p1.2.m2.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="A7.SS2.p1.2.m2.1b"><ci id="A7.SS2.p1.2.m2.1.1.cmml" xref="A7.SS2.p1.2.m2.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="A7.SS2.p1.2.m2.1c">b</annotation></semantics></math> according to <span class="ltx_ref ltx_missing_label ltx_ref_self">LABEL:alg:CCS</span>.
In practice, we implement the bias <math id="A7.SS2.p1.3.m3.1" class="ltx_Math" alttext="b" display="inline"><semantics id="A7.SS2.p1.3.m3.1a"><mi id="A7.SS2.p1.3.m3.1.1" xref="A7.SS2.p1.3.m3.1.1.cmml">b</mi><annotation-xml encoding="MathML-Content" id="A7.SS2.p1.3.m3.1b"><ci id="A7.SS2.p1.3.m3.1.1.cmml" xref="A7.SS2.p1.3.m3.1.1">𝑏</ci></annotation-xml><annotation encoding="application/x-tex" id="A7.SS2.p1.3.m3.1c">b</annotation></semantics></math> by appending an additional dimension of <math id="A7.SS2.p1.4.m4.1" class="ltx_Math" alttext="1" display="inline"><semantics id="A7.SS2.p1.4.m4.1a"><mn id="A7.SS2.p1.4.m4.1.1" xref="A7.SS2.p1.4.m4.1.1.cmml">1</mn><annotation-xml encoding="MathML-Content" id="A7.SS2.p1.4.m4.1b"><cn type="integer" id="A7.SS2.p1.4.m4.1.1.cmml" xref="A7.SS2.p1.4.m4.1.1">1</cn></annotation-xml><annotation encoding="application/x-tex" id="A7.SS2.p1.4.m4.1c">1</annotation></semantics></math>s features to the input, increasing it and <math id="A7.SS2.p1.5.m5.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="A7.SS2.p1.5.m5.1a"><mi id="A7.SS2.p1.5.m5.1.1" xref="A7.SS2.p1.5.m5.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="A7.SS2.p1.5.m5.1b"><ci id="A7.SS2.p1.5.m5.1.1.cmml" xref="A7.SS2.p1.5.m5.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="A7.SS2.p1.5.m5.1c">\theta</annotation></semantics></math> to have dimension <math id="A7.SS2.p1.6.m6.1" class="ltx_Math" alttext="d+1" display="inline"><semantics id="A7.SS2.p1.6.m6.1a"><mrow id="A7.SS2.p1.6.m6.1.1" xref="A7.SS2.p1.6.m6.1.1.cmml"><mi id="A7.SS2.p1.6.m6.1.1.2" xref="A7.SS2.p1.6.m6.1.1.2.cmml">d</mi><mo id="A7.SS2.p1.6.m6.1.1.1" xref="A7.SS2.p1.6.m6.1.1.1.cmml">+</mo><mn id="A7.SS2.p1.6.m6.1.1.3" xref="A7.SS2.p1.6.m6.1.1.3.cmml">1</mn></mrow><annotation-xml encoding="MathML-Content" id="A7.SS2.p1.6.m6.1b"><apply id="A7.SS2.p1.6.m6.1.1.cmml" xref="A7.SS2.p1.6.m6.1.1"><plus id="A7.SS2.p1.6.m6.1.1.1.cmml" xref="A7.SS2.p1.6.m6.1.1.1"></plus><ci id="A7.SS2.p1.6.m6.1.1.2.cmml" xref="A7.SS2.p1.6.m6.1.1.2">𝑑</ci><cn type="integer" id="A7.SS2.p1.6.m6.1.1.3.cmml" xref="A7.SS2.p1.6.m6.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS2.p1.6.m6.1c">d+1</annotation></semantics></math>.We randomly initialize <math id="A7.SS2.p1.7.m7.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="A7.SS2.p1.7.m7.1a"><mi id="A7.SS2.p1.7.m7.1.1" xref="A7.SS2.p1.7.m7.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="A7.SS2.p1.7.m7.1b"><ci id="A7.SS2.p1.7.m7.1.1.cmml" xref="A7.SS2.p1.7.m7.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="A7.SS2.p1.7.m7.1c">\theta</annotation></semantics></math> to have unit norm.
We then optimize the CCS loss <math id="A7.SS2.p1.8.m8.1" class="ltx_Math" alttext="T=10" display="inline"><semantics id="A7.SS2.p1.8.m8.1a"><mrow id="A7.SS2.p1.8.m8.1.1" xref="A7.SS2.p1.8.m8.1.1.cmml"><mi id="A7.SS2.p1.8.m8.1.1.2" xref="A7.SS2.p1.8.m8.1.1.2.cmml">T</mi><mo id="A7.SS2.p1.8.m8.1.1.1" xref="A7.SS2.p1.8.m8.1.1.1.cmml">=</mo><mn id="A7.SS2.p1.8.m8.1.1.3" xref="A7.SS2.p1.8.m8.1.1.3.cmml">10</mn></mrow><annotation-xml encoding="MathML-Content" id="A7.SS2.p1.8.m8.1b"><apply id="A7.SS2.p1.8.m8.1.1.cmml" xref="A7.SS2.p1.8.m8.1.1"><eq id="A7.SS2.p1.8.m8.1.1.1.cmml" xref="A7.SS2.p1.8.m8.1.1.1"></eq><ci id="A7.SS2.p1.8.m8.1.1.2.cmml" xref="A7.SS2.p1.8.m8.1.1.2">𝑇</ci><cn type="integer" id="A7.SS2.p1.8.m8.1.1.3.cmml" xref="A7.SS2.p1.8.m8.1.1.3">10</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS2.p1.8.m8.1c">T=10</annotation></semantics></math> times and select the run with lowest unsupervised loss.
In practice, we train each time for <math id="A7.SS2.p1.9.m9.1" class="ltx_Math" alttext="E=1000" display="inline"><semantics id="A7.SS2.p1.9.m9.1a"><mrow id="A7.SS2.p1.9.m9.1.1" xref="A7.SS2.p1.9.m9.1.1.cmml"><mi id="A7.SS2.p1.9.m9.1.1.2" xref="A7.SS2.p1.9.m9.1.1.2.cmml">E</mi><mo id="A7.SS2.p1.9.m9.1.1.1" xref="A7.SS2.p1.9.m9.1.1.1.cmml">=</mo><mn id="A7.SS2.p1.9.m9.1.1.3" xref="A7.SS2.p1.9.m9.1.1.3.cmml">1000</mn></mrow><annotation-xml encoding="MathML-Content" id="A7.SS2.p1.9.m9.1b"><apply id="A7.SS2.p1.9.m9.1.1.cmml" xref="A7.SS2.p1.9.m9.1.1"><eq id="A7.SS2.p1.9.m9.1.1.1.cmml" xref="A7.SS2.p1.9.m9.1.1.1"></eq><ci id="A7.SS2.p1.9.m9.1.1.2.cmml" xref="A7.SS2.p1.9.m9.1.1.2">𝐸</ci><cn type="integer" id="A7.SS2.p1.9.m9.1.1.3.cmml" xref="A7.SS2.p1.9.m9.1.1.3">1000</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS2.p1.9.m9.1c">E=1000</annotation></semantics></math> epochs with a learning rate <math id="A7.SS2.p1.10.m10.1" class="ltx_Math" alttext="\eta=0.01" display="inline"><semantics id="A7.SS2.p1.10.m10.1a"><mrow id="A7.SS2.p1.10.m10.1.1" xref="A7.SS2.p1.10.m10.1.1.cmml"><mi id="A7.SS2.p1.10.m10.1.1.2" xref="A7.SS2.p1.10.m10.1.1.2.cmml">η</mi><mo id="A7.SS2.p1.10.m10.1.1.1" xref="A7.SS2.p1.10.m10.1.1.1.cmml">=</mo><mn id="A7.SS2.p1.10.m10.1.1.3" xref="A7.SS2.p1.10.m10.1.1.3.cmml">0.01</mn></mrow><annotation-xml encoding="MathML-Content" id="A7.SS2.p1.10.m10.1b"><apply id="A7.SS2.p1.10.m10.1.1.cmml" xref="A7.SS2.p1.10.m10.1.1"><eq id="A7.SS2.p1.10.m10.1.1.1.cmml" xref="A7.SS2.p1.10.m10.1.1.1"></eq><ci id="A7.SS2.p1.10.m10.1.1.2.cmml" xref="A7.SS2.p1.10.m10.1.1.2">𝜂</ci><cn type="float" id="A7.SS2.p1.10.m10.1.1.3.cmml" xref="A7.SS2.p1.10.m10.1.1.3">0.01</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS2.p1.10.m10.1c">\eta=0.01</annotation></semantics></math> (which we found was good for consistently achieving low unsupervised loss) in each run.
Because we are only learning a linear probe on top of the features, training can be fast.</p>
</div>
</section>
<section id="A7.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.3 </span>CRC: Top Principal Component</h3>

<div id="A7.SS3.p1" class="ltx_para ltx_noindent">
<p id="A7.SS3.p1.3" class="ltx_p">Intuitively, a direction that clusters examples well should have high variance.
Motivated by this, a simple method is to cluster according to the Top Principal Component (TPC).
We do this by first constructing contrast features from the normalized contrast pair activations, <math id="A7.SS3.p1.1.m1.1" class="ltx_Math" alttext="\mathcal{C}=\{\tilde{\phi}(x_{i}^{+})-\tilde{\phi}(x_{i}^{-})\}_{i=1}^{n}" display="inline"><semantics id="A7.SS3.p1.1.m1.1a"><mrow id="A7.SS3.p1.1.m1.1.1" xref="A7.SS3.p1.1.m1.1.1.cmml"><mi class="ltx_font_mathcaligraphic" id="A7.SS3.p1.1.m1.1.1.3" xref="A7.SS3.p1.1.m1.1.1.3.cmml">𝒞</mi><mo id="A7.SS3.p1.1.m1.1.1.2" xref="A7.SS3.p1.1.m1.1.1.2.cmml">=</mo><msubsup id="A7.SS3.p1.1.m1.1.1.1" xref="A7.SS3.p1.1.m1.1.1.1.cmml"><mrow id="A7.SS3.p1.1.m1.1.1.1.1.1.1" xref="A7.SS3.p1.1.m1.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A7.SS3.p1.1.m1.1.1.1.1.1.1.2" xref="A7.SS3.p1.1.m1.1.1.1.1.1.2.cmml">{</mo><mrow id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.cmml"><mrow id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.cmml"><mover accent="true" id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.3" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.3.cmml"><mi id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.3.2" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.3.2.cmml">ϕ</mi><mo id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.3.1" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.3.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.2" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.2.cmml">​</mo><mrow id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.2" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.cmml"><mi id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.2" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.3" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml">i</mi><mo id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.3" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml">+</mo></msubsup><mo stretchy="false" id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.3" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.3" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.3.cmml">−</mo><mrow id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.cmml"><mover accent="true" id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.3" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.3.cmml"><mi id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.3.2" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.3.2.cmml">ϕ</mi><mo id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.3.1" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.3.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.2" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.2.cmml">​</mo><mrow id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.cmml"><mo stretchy="false" id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.2" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.cmml">(</mo><msubsup id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.cmml"><mi id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.2.2" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.2.2.cmml">x</mi><mi id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.2.3" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.2.3.cmml">i</mi><mo id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.3" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.3.cmml">−</mo></msubsup><mo stretchy="false" id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.3" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.cmml">)</mo></mrow></mrow></mrow><mo stretchy="false" id="A7.SS3.p1.1.m1.1.1.1.1.1.1.3" xref="A7.SS3.p1.1.m1.1.1.1.1.1.2.cmml">}</mo></mrow><mrow id="A7.SS3.p1.1.m1.1.1.1.1.3" xref="A7.SS3.p1.1.m1.1.1.1.1.3.cmml"><mi id="A7.SS3.p1.1.m1.1.1.1.1.3.2" xref="A7.SS3.p1.1.m1.1.1.1.1.3.2.cmml">i</mi><mo id="A7.SS3.p1.1.m1.1.1.1.1.3.1" xref="A7.SS3.p1.1.m1.1.1.1.1.3.1.cmml">=</mo><mn id="A7.SS3.p1.1.m1.1.1.1.1.3.3" xref="A7.SS3.p1.1.m1.1.1.1.1.3.3.cmml">1</mn></mrow><mi id="A7.SS3.p1.1.m1.1.1.1.3" xref="A7.SS3.p1.1.m1.1.1.1.3.cmml">n</mi></msubsup></mrow><annotation-xml encoding="MathML-Content" id="A7.SS3.p1.1.m1.1b"><apply id="A7.SS3.p1.1.m1.1.1.cmml" xref="A7.SS3.p1.1.m1.1.1"><eq id="A7.SS3.p1.1.m1.1.1.2.cmml" xref="A7.SS3.p1.1.m1.1.1.2"></eq><ci id="A7.SS3.p1.1.m1.1.1.3.cmml" xref="A7.SS3.p1.1.m1.1.1.3">𝒞</ci><apply id="A7.SS3.p1.1.m1.1.1.1.cmml" xref="A7.SS3.p1.1.m1.1.1.1"><csymbol cd="ambiguous" id="A7.SS3.p1.1.m1.1.1.1.2.cmml" xref="A7.SS3.p1.1.m1.1.1.1">superscript</csymbol><apply id="A7.SS3.p1.1.m1.1.1.1.1.cmml" xref="A7.SS3.p1.1.m1.1.1.1"><csymbol cd="ambiguous" id="A7.SS3.p1.1.m1.1.1.1.1.2.cmml" xref="A7.SS3.p1.1.m1.1.1.1">subscript</csymbol><set id="A7.SS3.p1.1.m1.1.1.1.1.1.2.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1"><apply id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1"><minus id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.3.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.3"></minus><apply id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1"><times id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.2.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.2"></times><apply id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.3.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.3"><ci id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.3.1.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.3.1">~</ci><ci id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.3.2.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.3.2">italic-ϕ</ci></apply><apply id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.1.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1">superscript</csymbol><apply id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.1.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1">subscript</csymbol><ci id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.2.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.3.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><plus id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.3.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.1.1.1.1.3"></plus></apply></apply><apply id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2"><times id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.2.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.2"></times><apply id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.3.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.3"><ci id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.3.1.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.3.1">~</ci><ci id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.3.2.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.3.2">italic-ϕ</ci></apply><apply id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1"><csymbol cd="ambiguous" id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.1.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1">superscript</csymbol><apply id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.2.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1"><csymbol cd="ambiguous" id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.2.1.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1">subscript</csymbol><ci id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.2.2.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.2.2">𝑥</ci><ci id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.2.3.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.2.3">𝑖</ci></apply><minus id="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.3.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.1.1.1.2.1.1.1.3"></minus></apply></apply></apply></set><apply id="A7.SS3.p1.1.m1.1.1.1.1.3.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.3"><eq id="A7.SS3.p1.1.m1.1.1.1.1.3.1.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.3.1"></eq><ci id="A7.SS3.p1.1.m1.1.1.1.1.3.2.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.3.2">𝑖</ci><cn type="integer" id="A7.SS3.p1.1.m1.1.1.1.1.3.3.cmml" xref="A7.SS3.p1.1.m1.1.1.1.1.3.3">1</cn></apply></apply><ci id="A7.SS3.p1.1.m1.1.1.1.3.cmml" xref="A7.SS3.p1.1.m1.1.1.1.3">𝑛</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS3.p1.1.m1.1c">\mathcal{C}=\{\tilde{\phi}(x_{i}^{+})-\tilde{\phi}(x_{i}^{-})\}_{i=1}^{n}</annotation></semantics></math>, then projecting these examples on their top principal component using PCA.
We then treat examples that are less than <math id="A7.SS3.p1.2.m2.1" class="ltx_Math" alttext="0" display="inline"><semantics id="A7.SS3.p1.2.m2.1a"><mn id="A7.SS3.p1.2.m2.1.1" xref="A7.SS3.p1.2.m2.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="A7.SS3.p1.2.m2.1b"><cn type="integer" id="A7.SS3.p1.2.m2.1.1.cmml" xref="A7.SS3.p1.2.m2.1.1">0</cn></annotation-xml></semantics></math> as one cluster and examples that are greater than <math id="A7.SS3.p1.3.m3.1" class="ltx_Math" alttext="0" display="inline"><semantics id="A7.SS3.p1.3.m3.1a"><mn id="A7.SS3.p1.3.m3.1.1" xref="A7.SS3.p1.3.m3.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="A7.SS3.p1.3.m3.1b"><cn type="integer" id="A7.SS3.p1.3.m3.1.1.cmml" xref="A7.SS3.p1.3.m3.1.1">0</cn></annotation-xml></semantics></math> as the other cluster.
We find that despite its simplicity this method can often also find truth-like features in model activations.</p>
</div>
</section>
<section id="A7.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">G.4 </span>CRC: Bimodal Salience Search</h3>

<div id="A7.SS4.p1" class="ltx_para ltx_noindent">
<p id="A7.SS4.p1.1" class="ltx_p">Bimodal Salience Search (BSS) is another variation of CRC.
One drawback of TPC is that variance is not an intrinsic quantity to a network’s behavior, since different directions in representation space can be scaled without changing the behavior of the network as long as subsequent layers are rescaled accordingly.
This motivates using a method that clusters examples regardless of the scale of different directions.</p>
</div>
<div id="A7.SS4.p2" class="ltx_para ltx_noindent">
<p id="A7.SS4.p2.2" class="ltx_p">If examples are well-clustered, then the intra-cluster variance should be low while the inter-cluster variance (or total variance) should be high.
Specifically, if we center a set of datapoints so that <math id="A7.SS4.p2.1.m1.1" class="ltx_Math" alttext="0" display="inline"><semantics id="A7.SS4.p2.1.m1.1a"><mn id="A7.SS4.p2.1.m1.1.1" xref="A7.SS4.p2.1.m1.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="A7.SS4.p2.1.m1.1b"><cn type="integer" id="A7.SS4.p2.1.m1.1.1.cmml" xref="A7.SS4.p2.1.m1.1.1">0</cn></annotation-xml></semantics></math> delineates the cluster boundary,
then the points on either side of <math id="A7.SS4.p2.2.m2.1" class="ltx_Math" alttext="0" display="inline"><semantics id="A7.SS4.p2.2.m2.1a"><mn id="A7.SS4.p2.2.m2.1.1" xref="A7.SS4.p2.2.m2.1.1.cmml">0</mn><annotation-xml encoding="MathML-Content" id="A7.SS4.p2.2.m2.1b"><cn type="integer" id="A7.SS4.p2.2.m2.1.1.cmml" xref="A7.SS4.p2.2.m2.1.1">0</cn></annotation-xml></semantics></math> should have low variance compared to the points overall.
This suggests minimizing the following loss:</p>
<table id="A9.EGx1" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A7.E1"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A7.E1.m1.6" class="ltx_Math" alttext="\displaystyle L(\theta)=\frac{\text{var}\{\theta^{T}c_{i}|\theta^{T}c_{i}&lt;0\}+\text{var}\{\theta^{T}c_{i}|\theta^{T}c_{i}\geq 0\}}{\text{var}\{\theta^{T}c_{i}\}}" display="inline"><semantics id="A7.E1.m1.6a"><mrow id="A7.E1.m1.6.7" xref="A7.E1.m1.6.7.cmml"><mrow id="A7.E1.m1.6.7.2" xref="A7.E1.m1.6.7.2.cmml"><mi id="A7.E1.m1.6.7.2.2" xref="A7.E1.m1.6.7.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A7.E1.m1.6.7.2.1" xref="A7.E1.m1.6.7.2.1.cmml">​</mo><mrow id="A7.E1.m1.6.7.2.3.2" xref="A7.E1.m1.6.7.2.cmml"><mo stretchy="false" id="A7.E1.m1.6.7.2.3.2.1" xref="A7.E1.m1.6.7.2.cmml">(</mo><mi id="A7.E1.m1.6.6" xref="A7.E1.m1.6.6.cmml">θ</mi><mo stretchy="false" id="A7.E1.m1.6.7.2.3.2.2" xref="A7.E1.m1.6.7.2.cmml">)</mo></mrow></mrow><mo id="A7.E1.m1.6.7.1" xref="A7.E1.m1.6.7.1.cmml">=</mo><mstyle displaystyle="true" id="A7.E1.m1.5.5" xref="A7.E1.m1.5.5.cmml"><mfrac id="A7.E1.m1.5.5a" xref="A7.E1.m1.5.5.cmml"><mrow id="A7.E1.m1.4.4.4" xref="A7.E1.m1.4.4.4.cmml"><mrow id="A7.E1.m1.2.2.2.2" xref="A7.E1.m1.2.2.2.2.cmml"><mtext id="A7.E1.m1.2.2.2.2.4" xref="A7.E1.m1.2.2.2.2.4a.cmml">var</mtext><mo lspace="0em" rspace="0em" id="A7.E1.m1.2.2.2.2.3" xref="A7.E1.m1.2.2.2.2.3.cmml">​</mo><mrow id="A7.E1.m1.2.2.2.2.2.2" xref="A7.E1.m1.2.2.2.2.2.3.cmml"><mo stretchy="false" id="A7.E1.m1.2.2.2.2.2.2.3" xref="A7.E1.m1.2.2.2.2.2.3.1.cmml">{</mo><mrow id="A7.E1.m1.1.1.1.1.1.1.1" xref="A7.E1.m1.1.1.1.1.1.1.1.cmml"><msup id="A7.E1.m1.1.1.1.1.1.1.1.2" xref="A7.E1.m1.1.1.1.1.1.1.1.2.cmml"><mi id="A7.E1.m1.1.1.1.1.1.1.1.2.2" xref="A7.E1.m1.1.1.1.1.1.1.1.2.2.cmml">θ</mi><mi id="A7.E1.m1.1.1.1.1.1.1.1.2.3" xref="A7.E1.m1.1.1.1.1.1.1.1.2.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="A7.E1.m1.1.1.1.1.1.1.1.1" xref="A7.E1.m1.1.1.1.1.1.1.1.1.cmml">​</mo><msub id="A7.E1.m1.1.1.1.1.1.1.1.3" xref="A7.E1.m1.1.1.1.1.1.1.1.3.cmml"><mi id="A7.E1.m1.1.1.1.1.1.1.1.3.2" xref="A7.E1.m1.1.1.1.1.1.1.1.3.2.cmml">c</mi><mi id="A7.E1.m1.1.1.1.1.1.1.1.3.3" xref="A7.E1.m1.1.1.1.1.1.1.1.3.3.cmml">i</mi></msub></mrow><mo lspace="0em" rspace="0em" id="A7.E1.m1.2.2.2.2.2.2.4" xref="A7.E1.m1.2.2.2.2.2.3.1.cmml">|</mo><mrow id="A7.E1.m1.2.2.2.2.2.2.2" xref="A7.E1.m1.2.2.2.2.2.2.2.cmml"><mrow id="A7.E1.m1.2.2.2.2.2.2.2.2" xref="A7.E1.m1.2.2.2.2.2.2.2.2.cmml"><msup id="A7.E1.m1.2.2.2.2.2.2.2.2.2" xref="A7.E1.m1.2.2.2.2.2.2.2.2.2.cmml"><mi id="A7.E1.m1.2.2.2.2.2.2.2.2.2.2" xref="A7.E1.m1.2.2.2.2.2.2.2.2.2.2.cmml">θ</mi><mi id="A7.E1.m1.2.2.2.2.2.2.2.2.2.3" xref="A7.E1.m1.2.2.2.2.2.2.2.2.2.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="A7.E1.m1.2.2.2.2.2.2.2.2.1" xref="A7.E1.m1.2.2.2.2.2.2.2.2.1.cmml">​</mo><msub id="A7.E1.m1.2.2.2.2.2.2.2.2.3" xref="A7.E1.m1.2.2.2.2.2.2.2.2.3.cmml"><mi id="A7.E1.m1.2.2.2.2.2.2.2.2.3.2" xref="A7.E1.m1.2.2.2.2.2.2.2.2.3.2.cmml">c</mi><mi id="A7.E1.m1.2.2.2.2.2.2.2.2.3.3" xref="A7.E1.m1.2.2.2.2.2.2.2.2.3.3.cmml">i</mi></msub></mrow><mo id="A7.E1.m1.2.2.2.2.2.2.2.1" xref="A7.E1.m1.2.2.2.2.2.2.2.1.cmml">&lt;</mo><mn id="A7.E1.m1.2.2.2.2.2.2.2.3" xref="A7.E1.m1.2.2.2.2.2.2.2.3.cmml">0</mn></mrow><mo stretchy="false" id="A7.E1.m1.2.2.2.2.2.2.5" xref="A7.E1.m1.2.2.2.2.2.3.1.cmml">}</mo></mrow></mrow><mo id="A7.E1.m1.4.4.4.5" xref="A7.E1.m1.4.4.4.5.cmml">+</mo><mrow id="A7.E1.m1.4.4.4.4" xref="A7.E1.m1.4.4.4.4.cmml"><mtext id="A7.E1.m1.4.4.4.4.4" xref="A7.E1.m1.4.4.4.4.4a.cmml">var</mtext><mo lspace="0em" rspace="0em" id="A7.E1.m1.4.4.4.4.3" xref="A7.E1.m1.4.4.4.4.3.cmml">​</mo><mrow id="A7.E1.m1.4.4.4.4.2.2" xref="A7.E1.m1.4.4.4.4.2.3.cmml"><mo stretchy="false" id="A7.E1.m1.4.4.4.4.2.2.3" xref="A7.E1.m1.4.4.4.4.2.3.1.cmml">{</mo><mrow id="A7.E1.m1.3.3.3.3.1.1.1" xref="A7.E1.m1.3.3.3.3.1.1.1.cmml"><msup id="A7.E1.m1.3.3.3.3.1.1.1.2" xref="A7.E1.m1.3.3.3.3.1.1.1.2.cmml"><mi id="A7.E1.m1.3.3.3.3.1.1.1.2.2" xref="A7.E1.m1.3.3.3.3.1.1.1.2.2.cmml">θ</mi><mi id="A7.E1.m1.3.3.3.3.1.1.1.2.3" xref="A7.E1.m1.3.3.3.3.1.1.1.2.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="A7.E1.m1.3.3.3.3.1.1.1.1" xref="A7.E1.m1.3.3.3.3.1.1.1.1.cmml">​</mo><msub id="A7.E1.m1.3.3.3.3.1.1.1.3" xref="A7.E1.m1.3.3.3.3.1.1.1.3.cmml"><mi id="A7.E1.m1.3.3.3.3.1.1.1.3.2" xref="A7.E1.m1.3.3.3.3.1.1.1.3.2.cmml">c</mi><mi id="A7.E1.m1.3.3.3.3.1.1.1.3.3" xref="A7.E1.m1.3.3.3.3.1.1.1.3.3.cmml">i</mi></msub></mrow><mo lspace="0em" rspace="0em" id="A7.E1.m1.4.4.4.4.2.2.4" xref="A7.E1.m1.4.4.4.4.2.3.1.cmml">|</mo><mrow id="A7.E1.m1.4.4.4.4.2.2.2" xref="A7.E1.m1.4.4.4.4.2.2.2.cmml"><mrow id="A7.E1.m1.4.4.4.4.2.2.2.2" xref="A7.E1.m1.4.4.4.4.2.2.2.2.cmml"><msup id="A7.E1.m1.4.4.4.4.2.2.2.2.2" xref="A7.E1.m1.4.4.4.4.2.2.2.2.2.cmml"><mi id="A7.E1.m1.4.4.4.4.2.2.2.2.2.2" xref="A7.E1.m1.4.4.4.4.2.2.2.2.2.2.cmml">θ</mi><mi id="A7.E1.m1.4.4.4.4.2.2.2.2.2.3" xref="A7.E1.m1.4.4.4.4.2.2.2.2.2.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="A7.E1.m1.4.4.4.4.2.2.2.2.1" xref="A7.E1.m1.4.4.4.4.2.2.2.2.1.cmml">​</mo><msub id="A7.E1.m1.4.4.4.4.2.2.2.2.3" xref="A7.E1.m1.4.4.4.4.2.2.2.2.3.cmml"><mi id="A7.E1.m1.4.4.4.4.2.2.2.2.3.2" xref="A7.E1.m1.4.4.4.4.2.2.2.2.3.2.cmml">c</mi><mi id="A7.E1.m1.4.4.4.4.2.2.2.2.3.3" xref="A7.E1.m1.4.4.4.4.2.2.2.2.3.3.cmml">i</mi></msub></mrow><mo id="A7.E1.m1.4.4.4.4.2.2.2.1" xref="A7.E1.m1.4.4.4.4.2.2.2.1.cmml">≥</mo><mn id="A7.E1.m1.4.4.4.4.2.2.2.3" xref="A7.E1.m1.4.4.4.4.2.2.2.3.cmml">0</mn></mrow><mo stretchy="false" id="A7.E1.m1.4.4.4.4.2.2.5" xref="A7.E1.m1.4.4.4.4.2.3.1.cmml">}</mo></mrow></mrow></mrow><mrow id="A7.E1.m1.5.5.5" xref="A7.E1.m1.5.5.5.cmml"><mtext id="A7.E1.m1.5.5.5.3" xref="A7.E1.m1.5.5.5.3a.cmml">var</mtext><mo lspace="0em" rspace="0em" id="A7.E1.m1.5.5.5.2" xref="A7.E1.m1.5.5.5.2.cmml">​</mo><mrow id="A7.E1.m1.5.5.5.1.1" xref="A7.E1.m1.5.5.5.1.2.cmml"><mo stretchy="false" id="A7.E1.m1.5.5.5.1.1.2" xref="A7.E1.m1.5.5.5.1.2.cmml">{</mo><mrow id="A7.E1.m1.5.5.5.1.1.1" xref="A7.E1.m1.5.5.5.1.1.1.cmml"><msup id="A7.E1.m1.5.5.5.1.1.1.2" xref="A7.E1.m1.5.5.5.1.1.1.2.cmml"><mi id="A7.E1.m1.5.5.5.1.1.1.2.2" xref="A7.E1.m1.5.5.5.1.1.1.2.2.cmml">θ</mi><mi id="A7.E1.m1.5.5.5.1.1.1.2.3" xref="A7.E1.m1.5.5.5.1.1.1.2.3.cmml">T</mi></msup><mo lspace="0em" rspace="0em" id="A7.E1.m1.5.5.5.1.1.1.1" xref="A7.E1.m1.5.5.5.1.1.1.1.cmml">​</mo><msub id="A7.E1.m1.5.5.5.1.1.1.3" xref="A7.E1.m1.5.5.5.1.1.1.3.cmml"><mi id="A7.E1.m1.5.5.5.1.1.1.3.2" xref="A7.E1.m1.5.5.5.1.1.1.3.2.cmml">c</mi><mi id="A7.E1.m1.5.5.5.1.1.1.3.3" xref="A7.E1.m1.5.5.5.1.1.1.3.3.cmml">i</mi></msub></mrow><mo stretchy="false" id="A7.E1.m1.5.5.5.1.1.3" xref="A7.E1.m1.5.5.5.1.2.cmml">}</mo></mrow></mrow></mfrac></mstyle></mrow><annotation-xml encoding="MathML-Content" id="A7.E1.m1.6b"><apply id="A7.E1.m1.6.7.cmml" xref="A7.E1.m1.6.7"><eq id="A7.E1.m1.6.7.1.cmml" xref="A7.E1.m1.6.7.1"></eq><apply id="A7.E1.m1.6.7.2.cmml" xref="A7.E1.m1.6.7.2"><times id="A7.E1.m1.6.7.2.1.cmml" xref="A7.E1.m1.6.7.2.1"></times><ci id="A7.E1.m1.6.7.2.2.cmml" xref="A7.E1.m1.6.7.2.2">𝐿</ci><ci id="A7.E1.m1.6.6.cmml" xref="A7.E1.m1.6.6">𝜃</ci></apply><apply id="A7.E1.m1.5.5.cmml" xref="A7.E1.m1.5.5"><divide id="A7.E1.m1.5.5.6.cmml" xref="A7.E1.m1.5.5"></divide><apply id="A7.E1.m1.4.4.4.cmml" xref="A7.E1.m1.4.4.4"><plus id="A7.E1.m1.4.4.4.5.cmml" xref="A7.E1.m1.4.4.4.5"></plus><apply id="A7.E1.m1.2.2.2.2.cmml" xref="A7.E1.m1.2.2.2.2"><times id="A7.E1.m1.2.2.2.2.3.cmml" xref="A7.E1.m1.2.2.2.2.3"></times><ci id="A7.E1.m1.2.2.2.2.4a.cmml" xref="A7.E1.m1.2.2.2.2.4"><mtext id="A7.E1.m1.2.2.2.2.4.cmml" xref="A7.E1.m1.2.2.2.2.4">var</mtext></ci><apply id="A7.E1.m1.2.2.2.2.2.3.cmml" xref="A7.E1.m1.2.2.2.2.2.2"><csymbol cd="latexml" id="A7.E1.m1.2.2.2.2.2.3.1.cmml" xref="A7.E1.m1.2.2.2.2.2.2.3">conditional-set</csymbol><apply id="A7.E1.m1.1.1.1.1.1.1.1.cmml" xref="A7.E1.m1.1.1.1.1.1.1.1"><times id="A7.E1.m1.1.1.1.1.1.1.1.1.cmml" xref="A7.E1.m1.1.1.1.1.1.1.1.1"></times><apply id="A7.E1.m1.1.1.1.1.1.1.1.2.cmml" xref="A7.E1.m1.1.1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="A7.E1.m1.1.1.1.1.1.1.1.2.1.cmml" xref="A7.E1.m1.1.1.1.1.1.1.1.2">superscript</csymbol><ci id="A7.E1.m1.1.1.1.1.1.1.1.2.2.cmml" xref="A7.E1.m1.1.1.1.1.1.1.1.2.2">𝜃</ci><ci id="A7.E1.m1.1.1.1.1.1.1.1.2.3.cmml" xref="A7.E1.m1.1.1.1.1.1.1.1.2.3">𝑇</ci></apply><apply id="A7.E1.m1.1.1.1.1.1.1.1.3.cmml" xref="A7.E1.m1.1.1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="A7.E1.m1.1.1.1.1.1.1.1.3.1.cmml" xref="A7.E1.m1.1.1.1.1.1.1.1.3">subscript</csymbol><ci id="A7.E1.m1.1.1.1.1.1.1.1.3.2.cmml" xref="A7.E1.m1.1.1.1.1.1.1.1.3.2">𝑐</ci><ci id="A7.E1.m1.1.1.1.1.1.1.1.3.3.cmml" xref="A7.E1.m1.1.1.1.1.1.1.1.3.3">𝑖</ci></apply></apply><apply id="A7.E1.m1.2.2.2.2.2.2.2.cmml" xref="A7.E1.m1.2.2.2.2.2.2.2"><lt id="A7.E1.m1.2.2.2.2.2.2.2.1.cmml" xref="A7.E1.m1.2.2.2.2.2.2.2.1"></lt><apply id="A7.E1.m1.2.2.2.2.2.2.2.2.cmml" xref="A7.E1.m1.2.2.2.2.2.2.2.2"><times id="A7.E1.m1.2.2.2.2.2.2.2.2.1.cmml" xref="A7.E1.m1.2.2.2.2.2.2.2.2.1"></times><apply id="A7.E1.m1.2.2.2.2.2.2.2.2.2.cmml" xref="A7.E1.m1.2.2.2.2.2.2.2.2.2"><csymbol cd="ambiguous" id="A7.E1.m1.2.2.2.2.2.2.2.2.2.1.cmml" xref="A7.E1.m1.2.2.2.2.2.2.2.2.2">superscript</csymbol><ci id="A7.E1.m1.2.2.2.2.2.2.2.2.2.2.cmml" xref="A7.E1.m1.2.2.2.2.2.2.2.2.2.2">𝜃</ci><ci id="A7.E1.m1.2.2.2.2.2.2.2.2.2.3.cmml" xref="A7.E1.m1.2.2.2.2.2.2.2.2.2.3">𝑇</ci></apply><apply id="A7.E1.m1.2.2.2.2.2.2.2.2.3.cmml" xref="A7.E1.m1.2.2.2.2.2.2.2.2.3"><csymbol cd="ambiguous" id="A7.E1.m1.2.2.2.2.2.2.2.2.3.1.cmml" xref="A7.E1.m1.2.2.2.2.2.2.2.2.3">subscript</csymbol><ci id="A7.E1.m1.2.2.2.2.2.2.2.2.3.2.cmml" xref="A7.E1.m1.2.2.2.2.2.2.2.2.3.2">𝑐</ci><ci id="A7.E1.m1.2.2.2.2.2.2.2.2.3.3.cmml" xref="A7.E1.m1.2.2.2.2.2.2.2.2.3.3">𝑖</ci></apply></apply><cn type="integer" id="A7.E1.m1.2.2.2.2.2.2.2.3.cmml" xref="A7.E1.m1.2.2.2.2.2.2.2.3">0</cn></apply></apply></apply><apply id="A7.E1.m1.4.4.4.4.cmml" xref="A7.E1.m1.4.4.4.4"><times id="A7.E1.m1.4.4.4.4.3.cmml" xref="A7.E1.m1.4.4.4.4.3"></times><ci id="A7.E1.m1.4.4.4.4.4a.cmml" xref="A7.E1.m1.4.4.4.4.4"><mtext id="A7.E1.m1.4.4.4.4.4.cmml" xref="A7.E1.m1.4.4.4.4.4">var</mtext></ci><apply id="A7.E1.m1.4.4.4.4.2.3.cmml" xref="A7.E1.m1.4.4.4.4.2.2"><csymbol cd="latexml" id="A7.E1.m1.4.4.4.4.2.3.1.cmml" xref="A7.E1.m1.4.4.4.4.2.2.3">conditional-set</csymbol><apply id="A7.E1.m1.3.3.3.3.1.1.1.cmml" xref="A7.E1.m1.3.3.3.3.1.1.1"><times id="A7.E1.m1.3.3.3.3.1.1.1.1.cmml" xref="A7.E1.m1.3.3.3.3.1.1.1.1"></times><apply id="A7.E1.m1.3.3.3.3.1.1.1.2.cmml" xref="A7.E1.m1.3.3.3.3.1.1.1.2"><csymbol cd="ambiguous" id="A7.E1.m1.3.3.3.3.1.1.1.2.1.cmml" xref="A7.E1.m1.3.3.3.3.1.1.1.2">superscript</csymbol><ci id="A7.E1.m1.3.3.3.3.1.1.1.2.2.cmml" xref="A7.E1.m1.3.3.3.3.1.1.1.2.2">𝜃</ci><ci id="A7.E1.m1.3.3.3.3.1.1.1.2.3.cmml" xref="A7.E1.m1.3.3.3.3.1.1.1.2.3">𝑇</ci></apply><apply id="A7.E1.m1.3.3.3.3.1.1.1.3.cmml" xref="A7.E1.m1.3.3.3.3.1.1.1.3"><csymbol cd="ambiguous" id="A7.E1.m1.3.3.3.3.1.1.1.3.1.cmml" xref="A7.E1.m1.3.3.3.3.1.1.1.3">subscript</csymbol><ci id="A7.E1.m1.3.3.3.3.1.1.1.3.2.cmml" xref="A7.E1.m1.3.3.3.3.1.1.1.3.2">𝑐</ci><ci id="A7.E1.m1.3.3.3.3.1.1.1.3.3.cmml" xref="A7.E1.m1.3.3.3.3.1.1.1.3.3">𝑖</ci></apply></apply><apply id="A7.E1.m1.4.4.4.4.2.2.2.cmml" xref="A7.E1.m1.4.4.4.4.2.2.2"><geq id="A7.E1.m1.4.4.4.4.2.2.2.1.cmml" xref="A7.E1.m1.4.4.4.4.2.2.2.1"></geq><apply id="A7.E1.m1.4.4.4.4.2.2.2.2.cmml" xref="A7.E1.m1.4.4.4.4.2.2.2.2"><times id="A7.E1.m1.4.4.4.4.2.2.2.2.1.cmml" xref="A7.E1.m1.4.4.4.4.2.2.2.2.1"></times><apply id="A7.E1.m1.4.4.4.4.2.2.2.2.2.cmml" xref="A7.E1.m1.4.4.4.4.2.2.2.2.2"><csymbol cd="ambiguous" id="A7.E1.m1.4.4.4.4.2.2.2.2.2.1.cmml" xref="A7.E1.m1.4.4.4.4.2.2.2.2.2">superscript</csymbol><ci id="A7.E1.m1.4.4.4.4.2.2.2.2.2.2.cmml" xref="A7.E1.m1.4.4.4.4.2.2.2.2.2.2">𝜃</ci><ci id="A7.E1.m1.4.4.4.4.2.2.2.2.2.3.cmml" xref="A7.E1.m1.4.4.4.4.2.2.2.2.2.3">𝑇</ci></apply><apply id="A7.E1.m1.4.4.4.4.2.2.2.2.3.cmml" xref="A7.E1.m1.4.4.4.4.2.2.2.2.3"><csymbol cd="ambiguous" id="A7.E1.m1.4.4.4.4.2.2.2.2.3.1.cmml" xref="A7.E1.m1.4.4.4.4.2.2.2.2.3">subscript</csymbol><ci id="A7.E1.m1.4.4.4.4.2.2.2.2.3.2.cmml" xref="A7.E1.m1.4.4.4.4.2.2.2.2.3.2">𝑐</ci><ci id="A7.E1.m1.4.4.4.4.2.2.2.2.3.3.cmml" xref="A7.E1.m1.4.4.4.4.2.2.2.2.3.3">𝑖</ci></apply></apply><cn type="integer" id="A7.E1.m1.4.4.4.4.2.2.2.3.cmml" xref="A7.E1.m1.4.4.4.4.2.2.2.3">0</cn></apply></apply></apply></apply><apply id="A7.E1.m1.5.5.5.cmml" xref="A7.E1.m1.5.5.5"><times id="A7.E1.m1.5.5.5.2.cmml" xref="A7.E1.m1.5.5.5.2"></times><ci id="A7.E1.m1.5.5.5.3a.cmml" xref="A7.E1.m1.5.5.5.3"><mtext id="A7.E1.m1.5.5.5.3.cmml" xref="A7.E1.m1.5.5.5.3">var</mtext></ci><set id="A7.E1.m1.5.5.5.1.2.cmml" xref="A7.E1.m1.5.5.5.1.1"><apply id="A7.E1.m1.5.5.5.1.1.1.cmml" xref="A7.E1.m1.5.5.5.1.1.1"><times id="A7.E1.m1.5.5.5.1.1.1.1.cmml" xref="A7.E1.m1.5.5.5.1.1.1.1"></times><apply id="A7.E1.m1.5.5.5.1.1.1.2.cmml" xref="A7.E1.m1.5.5.5.1.1.1.2"><csymbol cd="ambiguous" id="A7.E1.m1.5.5.5.1.1.1.2.1.cmml" xref="A7.E1.m1.5.5.5.1.1.1.2">superscript</csymbol><ci id="A7.E1.m1.5.5.5.1.1.1.2.2.cmml" xref="A7.E1.m1.5.5.5.1.1.1.2.2">𝜃</ci><ci id="A7.E1.m1.5.5.5.1.1.1.2.3.cmml" xref="A7.E1.m1.5.5.5.1.1.1.2.3">𝑇</ci></apply><apply id="A7.E1.m1.5.5.5.1.1.1.3.cmml" xref="A7.E1.m1.5.5.5.1.1.1.3"><csymbol cd="ambiguous" id="A7.E1.m1.5.5.5.1.1.1.3.1.cmml" xref="A7.E1.m1.5.5.5.1.1.1.3">subscript</csymbol><ci id="A7.E1.m1.5.5.5.1.1.1.3.2.cmml" xref="A7.E1.m1.5.5.5.1.1.1.3.2">𝑐</ci><ci id="A7.E1.m1.5.5.5.1.1.1.3.3.cmml" xref="A7.E1.m1.5.5.5.1.1.1.3.3">𝑖</ci></apply></apply></set></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.E1.m1.6c">\displaystyle L(\theta)=\frac{\text{var}\{\theta^{T}c_{i}|\theta^{T}c_{i}&lt;0\}+\text{var}\{\theta^{T}c_{i}|\theta^{T}c_{i}\geq 0\}}{\text{var}\{\theta^{T}c_{i}\}}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
<p id="A7.SS4.p2.5" class="ltx_p">where <math id="A7.SS4.p2.3.m1.2" class="ltx_Math" alttext="c_{i}:=\tilde{\phi}(x_{i}^{+})-\tilde{\phi}(x_{i}^{-})" display="inline"><semantics id="A7.SS4.p2.3.m1.2a"><mrow id="A7.SS4.p2.3.m1.2.2" xref="A7.SS4.p2.3.m1.2.2.cmml"><msub id="A7.SS4.p2.3.m1.2.2.4" xref="A7.SS4.p2.3.m1.2.2.4.cmml"><mi id="A7.SS4.p2.3.m1.2.2.4.2" xref="A7.SS4.p2.3.m1.2.2.4.2.cmml">c</mi><mi id="A7.SS4.p2.3.m1.2.2.4.3" xref="A7.SS4.p2.3.m1.2.2.4.3.cmml">i</mi></msub><mo lspace="0.278em" rspace="0.278em" id="A7.SS4.p2.3.m1.2.2.3" xref="A7.SS4.p2.3.m1.2.2.3.cmml">:=</mo><mrow id="A7.SS4.p2.3.m1.2.2.2" xref="A7.SS4.p2.3.m1.2.2.2.cmml"><mrow id="A7.SS4.p2.3.m1.1.1.1.1" xref="A7.SS4.p2.3.m1.1.1.1.1.cmml"><mover accent="true" id="A7.SS4.p2.3.m1.1.1.1.1.3" xref="A7.SS4.p2.3.m1.1.1.1.1.3.cmml"><mi id="A7.SS4.p2.3.m1.1.1.1.1.3.2" xref="A7.SS4.p2.3.m1.1.1.1.1.3.2.cmml">ϕ</mi><mo id="A7.SS4.p2.3.m1.1.1.1.1.3.1" xref="A7.SS4.p2.3.m1.1.1.1.1.3.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="A7.SS4.p2.3.m1.1.1.1.1.2" xref="A7.SS4.p2.3.m1.1.1.1.1.2.cmml">​</mo><mrow id="A7.SS4.p2.3.m1.1.1.1.1.1.1" xref="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.cmml"><mo stretchy="false" id="A7.SS4.p2.3.m1.1.1.1.1.1.1.2" xref="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.cmml">(</mo><msubsup id="A7.SS4.p2.3.m1.1.1.1.1.1.1.1" xref="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.cmml"><mi id="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.2.2" xref="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.2.2.cmml">x</mi><mi id="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.2.3" xref="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.2.3.cmml">i</mi><mo id="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.3" xref="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.3.cmml">+</mo></msubsup><mo stretchy="false" id="A7.SS4.p2.3.m1.1.1.1.1.1.1.3" xref="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.cmml">)</mo></mrow></mrow><mo id="A7.SS4.p2.3.m1.2.2.2.3" xref="A7.SS4.p2.3.m1.2.2.2.3.cmml">−</mo><mrow id="A7.SS4.p2.3.m1.2.2.2.2" xref="A7.SS4.p2.3.m1.2.2.2.2.cmml"><mover accent="true" id="A7.SS4.p2.3.m1.2.2.2.2.3" xref="A7.SS4.p2.3.m1.2.2.2.2.3.cmml"><mi id="A7.SS4.p2.3.m1.2.2.2.2.3.2" xref="A7.SS4.p2.3.m1.2.2.2.2.3.2.cmml">ϕ</mi><mo id="A7.SS4.p2.3.m1.2.2.2.2.3.1" xref="A7.SS4.p2.3.m1.2.2.2.2.3.1.cmml">~</mo></mover><mo lspace="0em" rspace="0em" id="A7.SS4.p2.3.m1.2.2.2.2.2" xref="A7.SS4.p2.3.m1.2.2.2.2.2.cmml">​</mo><mrow id="A7.SS4.p2.3.m1.2.2.2.2.1.1" xref="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.cmml"><mo stretchy="false" id="A7.SS4.p2.3.m1.2.2.2.2.1.1.2" xref="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.cmml">(</mo><msubsup id="A7.SS4.p2.3.m1.2.2.2.2.1.1.1" xref="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.cmml"><mi id="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.2.2" xref="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.2.2.cmml">x</mi><mi id="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.2.3" xref="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.2.3.cmml">i</mi><mo id="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.3" xref="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.3.cmml">−</mo></msubsup><mo stretchy="false" id="A7.SS4.p2.3.m1.2.2.2.2.1.1.3" xref="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.cmml">)</mo></mrow></mrow></mrow></mrow><annotation-xml encoding="MathML-Content" id="A7.SS4.p2.3.m1.2b"><apply id="A7.SS4.p2.3.m1.2.2.cmml" xref="A7.SS4.p2.3.m1.2.2"><csymbol cd="latexml" id="A7.SS4.p2.3.m1.2.2.3.cmml" xref="A7.SS4.p2.3.m1.2.2.3">assign</csymbol><apply id="A7.SS4.p2.3.m1.2.2.4.cmml" xref="A7.SS4.p2.3.m1.2.2.4"><csymbol cd="ambiguous" id="A7.SS4.p2.3.m1.2.2.4.1.cmml" xref="A7.SS4.p2.3.m1.2.2.4">subscript</csymbol><ci id="A7.SS4.p2.3.m1.2.2.4.2.cmml" xref="A7.SS4.p2.3.m1.2.2.4.2">𝑐</ci><ci id="A7.SS4.p2.3.m1.2.2.4.3.cmml" xref="A7.SS4.p2.3.m1.2.2.4.3">𝑖</ci></apply><apply id="A7.SS4.p2.3.m1.2.2.2.cmml" xref="A7.SS4.p2.3.m1.2.2.2"><minus id="A7.SS4.p2.3.m1.2.2.2.3.cmml" xref="A7.SS4.p2.3.m1.2.2.2.3"></minus><apply id="A7.SS4.p2.3.m1.1.1.1.1.cmml" xref="A7.SS4.p2.3.m1.1.1.1.1"><times id="A7.SS4.p2.3.m1.1.1.1.1.2.cmml" xref="A7.SS4.p2.3.m1.1.1.1.1.2"></times><apply id="A7.SS4.p2.3.m1.1.1.1.1.3.cmml" xref="A7.SS4.p2.3.m1.1.1.1.1.3"><ci id="A7.SS4.p2.3.m1.1.1.1.1.3.1.cmml" xref="A7.SS4.p2.3.m1.1.1.1.1.3.1">~</ci><ci id="A7.SS4.p2.3.m1.1.1.1.1.3.2.cmml" xref="A7.SS4.p2.3.m1.1.1.1.1.3.2">italic-ϕ</ci></apply><apply id="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.cmml" xref="A7.SS4.p2.3.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.1.cmml" xref="A7.SS4.p2.3.m1.1.1.1.1.1.1">superscript</csymbol><apply id="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.2.cmml" xref="A7.SS4.p2.3.m1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.2.1.cmml" xref="A7.SS4.p2.3.m1.1.1.1.1.1.1">subscript</csymbol><ci id="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.2.2.cmml" xref="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.2.2">𝑥</ci><ci id="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.2.3.cmml" xref="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.2.3">𝑖</ci></apply><plus id="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.3.cmml" xref="A7.SS4.p2.3.m1.1.1.1.1.1.1.1.3"></plus></apply></apply><apply id="A7.SS4.p2.3.m1.2.2.2.2.cmml" xref="A7.SS4.p2.3.m1.2.2.2.2"><times id="A7.SS4.p2.3.m1.2.2.2.2.2.cmml" xref="A7.SS4.p2.3.m1.2.2.2.2.2"></times><apply id="A7.SS4.p2.3.m1.2.2.2.2.3.cmml" xref="A7.SS4.p2.3.m1.2.2.2.2.3"><ci id="A7.SS4.p2.3.m1.2.2.2.2.3.1.cmml" xref="A7.SS4.p2.3.m1.2.2.2.2.3.1">~</ci><ci id="A7.SS4.p2.3.m1.2.2.2.2.3.2.cmml" xref="A7.SS4.p2.3.m1.2.2.2.2.3.2">italic-ϕ</ci></apply><apply id="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.cmml" xref="A7.SS4.p2.3.m1.2.2.2.2.1.1"><csymbol cd="ambiguous" id="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.1.cmml" xref="A7.SS4.p2.3.m1.2.2.2.2.1.1">superscript</csymbol><apply id="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.2.cmml" xref="A7.SS4.p2.3.m1.2.2.2.2.1.1"><csymbol cd="ambiguous" id="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.2.1.cmml" xref="A7.SS4.p2.3.m1.2.2.2.2.1.1">subscript</csymbol><ci id="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.2.2.cmml" xref="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.2.2">𝑥</ci><ci id="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.2.3.cmml" xref="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.2.3">𝑖</ci></apply><minus id="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.3.cmml" xref="A7.SS4.p2.3.m1.2.2.2.2.1.1.1.3"></minus></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS4.p2.3.m1.2c">c_{i}:=\tilde{\phi}(x_{i}^{+})-\tilde{\phi}(x_{i}^{-})</annotation></semantics></math> and where we use <math id="A7.SS4.p2.4.m2.1" class="ltx_Math" alttext="\text{var}\{z_{i}\}" display="inline"><semantics id="A7.SS4.p2.4.m2.1a"><mrow id="A7.SS4.p2.4.m2.1.1" xref="A7.SS4.p2.4.m2.1.1.cmml"><mtext id="A7.SS4.p2.4.m2.1.1.3" xref="A7.SS4.p2.4.m2.1.1.3a.cmml">var</mtext><mo lspace="0em" rspace="0em" id="A7.SS4.p2.4.m2.1.1.2" xref="A7.SS4.p2.4.m2.1.1.2.cmml">​</mo><mrow id="A7.SS4.p2.4.m2.1.1.1.1" xref="A7.SS4.p2.4.m2.1.1.1.2.cmml"><mo stretchy="false" id="A7.SS4.p2.4.m2.1.1.1.1.2" xref="A7.SS4.p2.4.m2.1.1.1.2.cmml">{</mo><msub id="A7.SS4.p2.4.m2.1.1.1.1.1" xref="A7.SS4.p2.4.m2.1.1.1.1.1.cmml"><mi id="A7.SS4.p2.4.m2.1.1.1.1.1.2" xref="A7.SS4.p2.4.m2.1.1.1.1.1.2.cmml">z</mi><mi id="A7.SS4.p2.4.m2.1.1.1.1.1.3" xref="A7.SS4.p2.4.m2.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A7.SS4.p2.4.m2.1.1.1.1.3" xref="A7.SS4.p2.4.m2.1.1.1.2.cmml">}</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A7.SS4.p2.4.m2.1b"><apply id="A7.SS4.p2.4.m2.1.1.cmml" xref="A7.SS4.p2.4.m2.1.1"><times id="A7.SS4.p2.4.m2.1.1.2.cmml" xref="A7.SS4.p2.4.m2.1.1.2"></times><ci id="A7.SS4.p2.4.m2.1.1.3a.cmml" xref="A7.SS4.p2.4.m2.1.1.3"><mtext id="A7.SS4.p2.4.m2.1.1.3.cmml" xref="A7.SS4.p2.4.m2.1.1.3">var</mtext></ci><set id="A7.SS4.p2.4.m2.1.1.1.2.cmml" xref="A7.SS4.p2.4.m2.1.1.1.1"><apply id="A7.SS4.p2.4.m2.1.1.1.1.1.cmml" xref="A7.SS4.p2.4.m2.1.1.1.1.1"><csymbol cd="ambiguous" id="A7.SS4.p2.4.m2.1.1.1.1.1.1.cmml" xref="A7.SS4.p2.4.m2.1.1.1.1.1">subscript</csymbol><ci id="A7.SS4.p2.4.m2.1.1.1.1.1.2.cmml" xref="A7.SS4.p2.4.m2.1.1.1.1.1.2">𝑧</ci><ci id="A7.SS4.p2.4.m2.1.1.1.1.1.3.cmml" xref="A7.SS4.p2.4.m2.1.1.1.1.1.3">𝑖</ci></apply></set></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS4.p2.4.m2.1c">\text{var}\{z_{i}\}</annotation></semantics></math> as shorthand to denote the variance of a set <math id="A7.SS4.p2.5.m3.1" class="ltx_Math" alttext="\{z_{i}\}_{i=1}^{n}\subset\mathbb{R}" display="inline"><semantics id="A7.SS4.p2.5.m3.1a"><mrow id="A7.SS4.p2.5.m3.1.1" xref="A7.SS4.p2.5.m3.1.1.cmml"><msubsup id="A7.SS4.p2.5.m3.1.1.1" xref="A7.SS4.p2.5.m3.1.1.1.cmml"><mrow id="A7.SS4.p2.5.m3.1.1.1.1.1.1" xref="A7.SS4.p2.5.m3.1.1.1.1.1.2.cmml"><mo stretchy="false" id="A7.SS4.p2.5.m3.1.1.1.1.1.1.2" xref="A7.SS4.p2.5.m3.1.1.1.1.1.2.cmml">{</mo><msub id="A7.SS4.p2.5.m3.1.1.1.1.1.1.1" xref="A7.SS4.p2.5.m3.1.1.1.1.1.1.1.cmml"><mi id="A7.SS4.p2.5.m3.1.1.1.1.1.1.1.2" xref="A7.SS4.p2.5.m3.1.1.1.1.1.1.1.2.cmml">z</mi><mi id="A7.SS4.p2.5.m3.1.1.1.1.1.1.1.3" xref="A7.SS4.p2.5.m3.1.1.1.1.1.1.1.3.cmml">i</mi></msub><mo stretchy="false" id="A7.SS4.p2.5.m3.1.1.1.1.1.1.3" xref="A7.SS4.p2.5.m3.1.1.1.1.1.2.cmml">}</mo></mrow><mrow id="A7.SS4.p2.5.m3.1.1.1.1.3" xref="A7.SS4.p2.5.m3.1.1.1.1.3.cmml"><mi id="A7.SS4.p2.5.m3.1.1.1.1.3.2" xref="A7.SS4.p2.5.m3.1.1.1.1.3.2.cmml">i</mi><mo id="A7.SS4.p2.5.m3.1.1.1.1.3.1" xref="A7.SS4.p2.5.m3.1.1.1.1.3.1.cmml">=</mo><mn id="A7.SS4.p2.5.m3.1.1.1.1.3.3" xref="A7.SS4.p2.5.m3.1.1.1.1.3.3.cmml">1</mn></mrow><mi id="A7.SS4.p2.5.m3.1.1.1.3" xref="A7.SS4.p2.5.m3.1.1.1.3.cmml">n</mi></msubsup><mo id="A7.SS4.p2.5.m3.1.1.2" xref="A7.SS4.p2.5.m3.1.1.2.cmml">⊂</mo><mi id="A7.SS4.p2.5.m3.1.1.3" xref="A7.SS4.p2.5.m3.1.1.3.cmml">ℝ</mi></mrow><annotation-xml encoding="MathML-Content" id="A7.SS4.p2.5.m3.1b"><apply id="A7.SS4.p2.5.m3.1.1.cmml" xref="A7.SS4.p2.5.m3.1.1"><subset id="A7.SS4.p2.5.m3.1.1.2.cmml" xref="A7.SS4.p2.5.m3.1.1.2"></subset><apply id="A7.SS4.p2.5.m3.1.1.1.cmml" xref="A7.SS4.p2.5.m3.1.1.1"><csymbol cd="ambiguous" id="A7.SS4.p2.5.m3.1.1.1.2.cmml" xref="A7.SS4.p2.5.m3.1.1.1">superscript</csymbol><apply id="A7.SS4.p2.5.m3.1.1.1.1.cmml" xref="A7.SS4.p2.5.m3.1.1.1"><csymbol cd="ambiguous" id="A7.SS4.p2.5.m3.1.1.1.1.2.cmml" xref="A7.SS4.p2.5.m3.1.1.1">subscript</csymbol><set id="A7.SS4.p2.5.m3.1.1.1.1.1.2.cmml" xref="A7.SS4.p2.5.m3.1.1.1.1.1.1"><apply id="A7.SS4.p2.5.m3.1.1.1.1.1.1.1.cmml" xref="A7.SS4.p2.5.m3.1.1.1.1.1.1.1"><csymbol cd="ambiguous" id="A7.SS4.p2.5.m3.1.1.1.1.1.1.1.1.cmml" xref="A7.SS4.p2.5.m3.1.1.1.1.1.1.1">subscript</csymbol><ci id="A7.SS4.p2.5.m3.1.1.1.1.1.1.1.2.cmml" xref="A7.SS4.p2.5.m3.1.1.1.1.1.1.1.2">𝑧</ci><ci id="A7.SS4.p2.5.m3.1.1.1.1.1.1.1.3.cmml" xref="A7.SS4.p2.5.m3.1.1.1.1.1.1.1.3">𝑖</ci></apply></set><apply id="A7.SS4.p2.5.m3.1.1.1.1.3.cmml" xref="A7.SS4.p2.5.m3.1.1.1.1.3"><eq id="A7.SS4.p2.5.m3.1.1.1.1.3.1.cmml" xref="A7.SS4.p2.5.m3.1.1.1.1.3.1"></eq><ci id="A7.SS4.p2.5.m3.1.1.1.1.3.2.cmml" xref="A7.SS4.p2.5.m3.1.1.1.1.3.2">𝑖</ci><cn type="integer" id="A7.SS4.p2.5.m3.1.1.1.1.3.3.cmml" xref="A7.SS4.p2.5.m3.1.1.1.1.3.3">1</cn></apply></apply><ci id="A7.SS4.p2.5.m3.1.1.1.3.cmml" xref="A7.SS4.p2.5.m3.1.1.1.3">𝑛</ci></apply><ci id="A7.SS4.p2.5.m3.1.1.3.cmml" xref="A7.SS4.p2.5.m3.1.1.3">ℝ</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS4.p2.5.m3.1c">\{z_{i}\}_{i=1}^{n}\subset\mathbb{R}</annotation></semantics></math>.
Because this objective is written in terms of a ratio, it is invariant to the overall scale of the direction, fixing that drawback of TPC.
This loss is similar to that of Linear Discriminant Analysis (LDA) <cite class="ltx_cite ltx_citemacro_citep">(Fisher, <a href="#bib.bib14" title="" class="ltx_ref">1936</a>)</cite>, but unlike LDA is completely unsupervised.</p>
</div>
<div id="A7.SS4.p3" class="ltx_para ltx_noindent">
<p id="A7.SS4.p3.11" class="ltx_p">In practice, we use an SGD-based optimizer to find a local optimum of <math id="A7.SS4.p3.1.m1.1" class="ltx_Math" alttext="L(\theta)" display="inline"><semantics id="A7.SS4.p3.1.m1.1a"><mrow id="A7.SS4.p3.1.m1.1.2" xref="A7.SS4.p3.1.m1.1.2.cmml"><mi id="A7.SS4.p3.1.m1.1.2.2" xref="A7.SS4.p3.1.m1.1.2.2.cmml">L</mi><mo lspace="0em" rspace="0em" id="A7.SS4.p3.1.m1.1.2.1" xref="A7.SS4.p3.1.m1.1.2.1.cmml">​</mo><mrow id="A7.SS4.p3.1.m1.1.2.3.2" xref="A7.SS4.p3.1.m1.1.2.cmml"><mo stretchy="false" id="A7.SS4.p3.1.m1.1.2.3.2.1" xref="A7.SS4.p3.1.m1.1.2.cmml">(</mo><mi id="A7.SS4.p3.1.m1.1.1" xref="A7.SS4.p3.1.m1.1.1.cmml">θ</mi><mo stretchy="false" id="A7.SS4.p3.1.m1.1.2.3.2.2" xref="A7.SS4.p3.1.m1.1.2.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A7.SS4.p3.1.m1.1b"><apply id="A7.SS4.p3.1.m1.1.2.cmml" xref="A7.SS4.p3.1.m1.1.2"><times id="A7.SS4.p3.1.m1.1.2.1.cmml" xref="A7.SS4.p3.1.m1.1.2.1"></times><ci id="A7.SS4.p3.1.m1.1.2.2.cmml" xref="A7.SS4.p3.1.m1.1.2.2">𝐿</ci><ci id="A7.SS4.p3.1.m1.1.1.cmml" xref="A7.SS4.p3.1.m1.1.1">𝜃</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS4.p3.1.m1.1c">L(\theta)</annotation></semantics></math>.
After computing the contrast features, we repeat the following process <math id="A7.SS4.p3.2.m2.1" class="ltx_Math" alttext="T" display="inline"><semantics id="A7.SS4.p3.2.m2.1a"><mi id="A7.SS4.p3.2.m2.1.1" xref="A7.SS4.p3.2.m2.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="A7.SS4.p3.2.m2.1b"><ci id="A7.SS4.p3.2.m2.1.1.cmml" xref="A7.SS4.p3.2.m2.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="A7.SS4.p3.2.m2.1c">T</annotation></semantics></math> times.
We first initialize a random direction <math id="A7.SS4.p3.3.m3.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="A7.SS4.p3.3.m3.1a"><mi id="A7.SS4.p3.3.m3.1.1" xref="A7.SS4.p3.3.m3.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="A7.SS4.p3.3.m3.1b"><ci id="A7.SS4.p3.3.m3.1.1.cmml" xref="A7.SS4.p3.3.m3.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="A7.SS4.p3.3.m3.1c">\theta</annotation></semantics></math> with unit norm.
Then for <math id="A7.SS4.p3.4.m4.1" class="ltx_Math" alttext="E" display="inline"><semantics id="A7.SS4.p3.4.m4.1a"><mi id="A7.SS4.p3.4.m4.1.1" xref="A7.SS4.p3.4.m4.1.1.cmml">E</mi><annotation-xml encoding="MathML-Content" id="A7.SS4.p3.4.m4.1b"><ci id="A7.SS4.p3.4.m4.1.1.cmml" xref="A7.SS4.p3.4.m4.1.1">𝐸</ci></annotation-xml><annotation encoding="application/x-tex" id="A7.SS4.p3.4.m4.1c">E</annotation></semantics></math> epochs, we calculate the loss according to <a href="#A7.E1" title="In G.4 CRC: Bimodal Salience Search ‣ Appendix G CCS and CRC Implementation Details ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Equation</span> <span class="ltx_text ltx_ref_tag">1</span></a>, and update the direction <math id="A7.SS4.p3.5.m5.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="A7.SS4.p3.5.m5.1a"><mi id="A7.SS4.p3.5.m5.1.1" xref="A7.SS4.p3.5.m5.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="A7.SS4.p3.5.m5.1b"><ci id="A7.SS4.p3.5.m5.1.1.cmml" xref="A7.SS4.p3.5.m5.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="A7.SS4.p3.5.m5.1c">\theta</annotation></semantics></math> by via projected gradient descent, each time projecting <math id="A7.SS4.p3.6.m6.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="A7.SS4.p3.6.m6.1a"><mi id="A7.SS4.p3.6.m6.1.1" xref="A7.SS4.p3.6.m6.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="A7.SS4.p3.6.m6.1b"><ci id="A7.SS4.p3.6.m6.1.1.cmml" xref="A7.SS4.p3.6.m6.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="A7.SS4.p3.6.m6.1c">\theta</annotation></semantics></math> back onto the unit sphere.
Finally, we select the direction <math id="A7.SS4.p3.7.m7.1" class="ltx_Math" alttext="\theta" display="inline"><semantics id="A7.SS4.p3.7.m7.1a"><mi id="A7.SS4.p3.7.m7.1.1" xref="A7.SS4.p3.7.m7.1.1.cmml">θ</mi><annotation-xml encoding="MathML-Content" id="A7.SS4.p3.7.m7.1b"><ci id="A7.SS4.p3.7.m7.1.1.cmml" xref="A7.SS4.p3.7.m7.1.1">𝜃</ci></annotation-xml><annotation encoding="application/x-tex" id="A7.SS4.p3.7.m7.1c">\theta</annotation></semantics></math> that has the lowest loss among the <math id="A7.SS4.p3.8.m8.1" class="ltx_Math" alttext="T" display="inline"><semantics id="A7.SS4.p3.8.m8.1a"><mi id="A7.SS4.p3.8.m8.1.1" xref="A7.SS4.p3.8.m8.1.1.cmml">T</mi><annotation-xml encoding="MathML-Content" id="A7.SS4.p3.8.m8.1b"><ci id="A7.SS4.p3.8.m8.1.1.cmml" xref="A7.SS4.p3.8.m8.1.1">𝑇</ci></annotation-xml><annotation encoding="application/x-tex" id="A7.SS4.p3.8.m8.1c">T</annotation></semantics></math> directions we found, and predict based on this direction.
When we optimize the direction with multiple datasets, i.e. <math id="A7.SS4.p3.9.m9.3" class="ltx_Math" alttext="S_{1},...,S_{n}" display="inline"><semantics id="A7.SS4.p3.9.m9.3a"><mrow id="A7.SS4.p3.9.m9.3.3.2" xref="A7.SS4.p3.9.m9.3.3.3.cmml"><msub id="A7.SS4.p3.9.m9.2.2.1.1" xref="A7.SS4.p3.9.m9.2.2.1.1.cmml"><mi id="A7.SS4.p3.9.m9.2.2.1.1.2" xref="A7.SS4.p3.9.m9.2.2.1.1.2.cmml">S</mi><mn id="A7.SS4.p3.9.m9.2.2.1.1.3" xref="A7.SS4.p3.9.m9.2.2.1.1.3.cmml">1</mn></msub><mo id="A7.SS4.p3.9.m9.3.3.2.3" xref="A7.SS4.p3.9.m9.3.3.3.cmml">,</mo><mi mathvariant="normal" id="A7.SS4.p3.9.m9.1.1" xref="A7.SS4.p3.9.m9.1.1.cmml">…</mi><mo id="A7.SS4.p3.9.m9.3.3.2.4" xref="A7.SS4.p3.9.m9.3.3.3.cmml">,</mo><msub id="A7.SS4.p3.9.m9.3.3.2.2" xref="A7.SS4.p3.9.m9.3.3.2.2.cmml"><mi id="A7.SS4.p3.9.m9.3.3.2.2.2" xref="A7.SS4.p3.9.m9.3.3.2.2.2.cmml">S</mi><mi id="A7.SS4.p3.9.m9.3.3.2.2.3" xref="A7.SS4.p3.9.m9.3.3.2.2.3.cmml">n</mi></msub></mrow><annotation-xml encoding="MathML-Content" id="A7.SS4.p3.9.m9.3b"><list id="A7.SS4.p3.9.m9.3.3.3.cmml" xref="A7.SS4.p3.9.m9.3.3.2"><apply id="A7.SS4.p3.9.m9.2.2.1.1.cmml" xref="A7.SS4.p3.9.m9.2.2.1.1"><csymbol cd="ambiguous" id="A7.SS4.p3.9.m9.2.2.1.1.1.cmml" xref="A7.SS4.p3.9.m9.2.2.1.1">subscript</csymbol><ci id="A7.SS4.p3.9.m9.2.2.1.1.2.cmml" xref="A7.SS4.p3.9.m9.2.2.1.1.2">𝑆</ci><cn type="integer" id="A7.SS4.p3.9.m9.2.2.1.1.3.cmml" xref="A7.SS4.p3.9.m9.2.2.1.1.3">1</cn></apply><ci id="A7.SS4.p3.9.m9.1.1.cmml" xref="A7.SS4.p3.9.m9.1.1">…</ci><apply id="A7.SS4.p3.9.m9.3.3.2.2.cmml" xref="A7.SS4.p3.9.m9.3.3.2.2"><csymbol cd="ambiguous" id="A7.SS4.p3.9.m9.3.3.2.2.1.cmml" xref="A7.SS4.p3.9.m9.3.3.2.2">subscript</csymbol><ci id="A7.SS4.p3.9.m9.3.3.2.2.2.cmml" xref="A7.SS4.p3.9.m9.3.3.2.2.2">𝑆</ci><ci id="A7.SS4.p3.9.m9.3.3.2.2.3.cmml" xref="A7.SS4.p3.9.m9.3.3.2.2.3">𝑛</ci></apply></list></annotation-xml><annotation encoding="application/x-tex" id="A7.SS4.p3.9.m9.3c">S_{1},...,S_{n}</annotation></semantics></math>, we use the same algorithm, but average the loss across all datasets.
In practice we use <math id="A7.SS4.p3.10.m10.1" class="ltx_Math" alttext="T=E=20" display="inline"><semantics id="A7.SS4.p3.10.m10.1a"><mrow id="A7.SS4.p3.10.m10.1.1" xref="A7.SS4.p3.10.m10.1.1.cmml"><mi id="A7.SS4.p3.10.m10.1.1.2" xref="A7.SS4.p3.10.m10.1.1.2.cmml">T</mi><mo id="A7.SS4.p3.10.m10.1.1.3" xref="A7.SS4.p3.10.m10.1.1.3.cmml">=</mo><mi id="A7.SS4.p3.10.m10.1.1.4" xref="A7.SS4.p3.10.m10.1.1.4.cmml">E</mi><mo id="A7.SS4.p3.10.m10.1.1.5" xref="A7.SS4.p3.10.m10.1.1.5.cmml">=</mo><mn id="A7.SS4.p3.10.m10.1.1.6" xref="A7.SS4.p3.10.m10.1.1.6.cmml">20</mn></mrow><annotation-xml encoding="MathML-Content" id="A7.SS4.p3.10.m10.1b"><apply id="A7.SS4.p3.10.m10.1.1.cmml" xref="A7.SS4.p3.10.m10.1.1"><and id="A7.SS4.p3.10.m10.1.1a.cmml" xref="A7.SS4.p3.10.m10.1.1"></and><apply id="A7.SS4.p3.10.m10.1.1b.cmml" xref="A7.SS4.p3.10.m10.1.1"><eq id="A7.SS4.p3.10.m10.1.1.3.cmml" xref="A7.SS4.p3.10.m10.1.1.3"></eq><ci id="A7.SS4.p3.10.m10.1.1.2.cmml" xref="A7.SS4.p3.10.m10.1.1.2">𝑇</ci><ci id="A7.SS4.p3.10.m10.1.1.4.cmml" xref="A7.SS4.p3.10.m10.1.1.4">𝐸</ci></apply><apply id="A7.SS4.p3.10.m10.1.1c.cmml" xref="A7.SS4.p3.10.m10.1.1"><eq id="A7.SS4.p3.10.m10.1.1.5.cmml" xref="A7.SS4.p3.10.m10.1.1.5"></eq><share href="#A7.SS4.p3.10.m10.1.1.4.cmml" id="A7.SS4.p3.10.m10.1.1d.cmml" xref="A7.SS4.p3.10.m10.1.1"></share><cn type="integer" id="A7.SS4.p3.10.m10.1.1.6.cmml" xref="A7.SS4.p3.10.m10.1.1.6">20</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A7.SS4.p3.10.m10.1c">T=E=20</annotation></semantics></math>, and use Adam optimizer <cite class="ltx_cite ltx_citemacro_citep">(Kingma &amp; Ba, <a href="#bib.bib22" title="" class="ltx_ref">2014</a>)</cite> with a learning rate of <math id="A7.SS4.p3.11.m11.1" class="ltx_Math" alttext="0.1" display="inline"><semantics id="A7.SS4.p3.11.m11.1a"><mn id="A7.SS4.p3.11.m11.1.1" xref="A7.SS4.p3.11.m11.1.1.cmml">0.1</mn><annotation-xml encoding="MathML-Content" id="A7.SS4.p3.11.m11.1b"><cn type="float" id="A7.SS4.p3.11.m11.1.1.cmml" xref="A7.SS4.p3.11.m11.1.1">0.1</cn></annotation-xml><annotation encoding="application/x-tex" id="A7.SS4.p3.11.m11.1c">0.1</annotation></semantics></math>.</p>
</div>
</section>
</section>
<section id="A8" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix H </span>Statistical Significance</h2>

<div id="A8.p1" class="ltx_para ltx_noindent">
<p id="A8.p1.10" class="ltx_p">Our main accuracy results for CCS and other methods (e.g. in <a href="#S3.T1" title="In 3.1 Experimental Setup ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">1</span></a> and elsewhere) are computed by evaluating the method on <math id="A8.p1.1.m1.1" class="ltx_Math" alttext="40\%" display="inline"><semantics id="A8.p1.1.m1.1a"><mrow id="A8.p1.1.m1.1.1" xref="A8.p1.1.m1.1.1.cmml"><mn id="A8.p1.1.m1.1.1.2" xref="A8.p1.1.m1.1.1.2.cmml">40</mn><mo id="A8.p1.1.m1.1.1.1" xref="A8.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="A8.p1.1.m1.1b"><apply id="A8.p1.1.m1.1.1.cmml" xref="A8.p1.1.m1.1.1"><csymbol cd="latexml" id="A8.p1.1.m1.1.1.1.cmml" xref="A8.p1.1.m1.1.1.1">percent</csymbol><cn type="integer" id="A8.p1.1.m1.1.1.2.cmml" xref="A8.p1.1.m1.1.1.2">40</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.1.m1.1c">40\%</annotation></semantics></math> of the <math id="A8.p1.2.m2.1" class="ltx_Math" alttext="1000" display="inline"><semantics id="A8.p1.2.m2.1a"><mn id="A8.p1.2.m2.1.1" xref="A8.p1.2.m2.1.1.cmml">1000</mn><annotation-xml encoding="MathML-Content" id="A8.p1.2.m2.1b"><cn type="integer" id="A8.p1.2.m2.1.1.cmml" xref="A8.p1.2.m2.1.1">1000</cn></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.2.m2.1c">1000</annotation></semantics></math> (or <math id="A8.p1.3.m3.1" class="ltx_Math" alttext="500" display="inline"><semantics id="A8.p1.3.m3.1a"><mn id="A8.p1.3.m3.1.1" xref="A8.p1.3.m3.1.1.cmml">500</mn><annotation-xml encoding="MathML-Content" id="A8.p1.3.m3.1b"><cn type="integer" id="A8.p1.3.m3.1.1.cmml" xref="A8.p1.3.m3.1.1">500</cn></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.3.m3.1c">500</annotation></semantics></math> in the case of COPA) examples sampled for each dataset, then averaging the resulting accuracy across <math id="A8.p1.4.m4.1" class="ltx_Math" alttext="9" display="inline"><semantics id="A8.p1.4.m4.1a"><mn id="A8.p1.4.m4.1.1" xref="A8.p1.4.m4.1.1.cmml">9</mn><annotation-xml encoding="MathML-Content" id="A8.p1.4.m4.1b"><cn type="integer" id="A8.p1.4.m4.1.1.cmml" xref="A8.p1.4.m4.1.1">9</cn></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.4.m4.1c">9</annotation></semantics></math> prompts per dataset (on average), <math id="A8.p1.5.m5.1" class="ltx_Math" alttext="10" display="inline"><semantics id="A8.p1.5.m5.1a"><mn id="A8.p1.5.m5.1.1" xref="A8.p1.5.m5.1.1.cmml">10</mn><annotation-xml encoding="MathML-Content" id="A8.p1.5.m5.1b"><cn type="integer" id="A8.p1.5.m5.1.1.cmml" xref="A8.p1.5.m5.1.1">10</cn></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.5.m5.1c">10</annotation></semantics></math> different datasets, and up to <math id="A8.p1.6.m6.1" class="ltx_Math" alttext="5" display="inline"><semantics id="A8.p1.6.m6.1a"><mn id="A8.p1.6.m6.1.1" xref="A8.p1.6.m6.1.1.cmml">5</mn><annotation-xml encoding="MathML-Content" id="A8.p1.6.m6.1b"><cn type="integer" id="A8.p1.6.m6.1.1.cmml" xref="A8.p1.6.m6.1.1">5</cn></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.6.m6.1c">5</annotation></semantics></math> models. This corresponds to about <math id="A8.p1.7.m7.1" class="ltx_Math" alttext="180" display="inline"><semantics id="A8.p1.7.m7.1a"><mn id="A8.p1.7.m7.1.1" xref="A8.p1.7.m7.1.1.cmml">180</mn><annotation-xml encoding="MathML-Content" id="A8.p1.7.m7.1b"><cn type="integer" id="A8.p1.7.m7.1.1.cmml" xref="A8.p1.7.m7.1.1">180</cn></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.7.m7.1c">180</annotation></semantics></math>k samples in total; we performed subsampling in this way for computational efficiency reasons, as this is already substantial. For results where we average across datasets (i.e. most results in this paper), we average across <math id="A8.p1.8.m8.1" class="ltx_Math" alttext="3800" display="inline"><semantics id="A8.p1.8.m8.1a"><mn id="A8.p1.8.m8.1.1" xref="A8.p1.8.m8.1.1.cmml">3800</mn><annotation-xml encoding="MathML-Content" id="A8.p1.8.m8.1b"><cn type="integer" id="A8.p1.8.m8.1.1.cmml" xref="A8.p1.8.m8.1.1">3800</cn></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.8.m8.1c">3800</annotation></semantics></math> IID samples (the original examples sampled from each dataset), then usually also averaged across a very large number of correlated samples:  <math id="A8.p1.9.m9.1" class="ltx_Math" alttext="9" display="inline"><semantics id="A8.p1.9.m9.1a"><mn id="A8.p1.9.m9.1.1" xref="A8.p1.9.m9.1.1.cmml">9</mn><annotation-xml encoding="MathML-Content" id="A8.p1.9.m9.1b"><cn type="integer" id="A8.p1.9.m9.1.1.cmml" xref="A8.p1.9.m9.1.1">9</cn></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.9.m9.1c">9</annotation></semantics></math> prompts and (often) <math id="A8.p1.10.m10.1" class="ltx_Math" alttext="5" display="inline"><semantics id="A8.p1.10.m10.1a"><mn id="A8.p1.10.m10.1.1" xref="A8.p1.10.m10.1.1.cmml">5</mn><annotation-xml encoding="MathML-Content" id="A8.p1.10.m10.1b"><cn type="integer" id="A8.p1.10.m10.1.1.cmml" xref="A8.p1.10.m10.1.1">5</cn></annotation-xml><annotation encoding="application/x-tex" id="A8.p1.10.m10.1c">5</annotation></semantics></math> different models for a given sample.</p>
</div>
<div id="A8.p2" class="ltx_para ltx_noindent">
<p id="A8.p2.6" class="ltx_p">We can compute an upper bound on the standard error of accuracies computed in this way by ignoring the averaging across different prompts and models: this gives us a simple but coarse upper bound of the standard error of <math id="A8.p2.1.m1.1" class="ltx_Math" alttext="\frac{1}{2\sqrt{3800}}\approx 0.8\%" display="inline"><semantics id="A8.p2.1.m1.1a"><mrow id="A8.p2.1.m1.1.1" xref="A8.p2.1.m1.1.1.cmml"><mfrac id="A8.p2.1.m1.1.1.2" xref="A8.p2.1.m1.1.1.2.cmml"><mn id="A8.p2.1.m1.1.1.2.2" xref="A8.p2.1.m1.1.1.2.2.cmml">1</mn><mrow id="A8.p2.1.m1.1.1.2.3" xref="A8.p2.1.m1.1.1.2.3.cmml"><mn id="A8.p2.1.m1.1.1.2.3.2" xref="A8.p2.1.m1.1.1.2.3.2.cmml">2</mn><mo lspace="0em" rspace="0em" id="A8.p2.1.m1.1.1.2.3.1" xref="A8.p2.1.m1.1.1.2.3.1.cmml">​</mo><msqrt id="A8.p2.1.m1.1.1.2.3.3" xref="A8.p2.1.m1.1.1.2.3.3.cmml"><mn id="A8.p2.1.m1.1.1.2.3.3.2" xref="A8.p2.1.m1.1.1.2.3.3.2.cmml">3800</mn></msqrt></mrow></mfrac><mo id="A8.p2.1.m1.1.1.1" xref="A8.p2.1.m1.1.1.1.cmml">≈</mo><mrow id="A8.p2.1.m1.1.1.3" xref="A8.p2.1.m1.1.1.3.cmml"><mn id="A8.p2.1.m1.1.1.3.2" xref="A8.p2.1.m1.1.1.3.2.cmml">0.8</mn><mo id="A8.p2.1.m1.1.1.3.1" xref="A8.p2.1.m1.1.1.3.1.cmml">%</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="A8.p2.1.m1.1b"><apply id="A8.p2.1.m1.1.1.cmml" xref="A8.p2.1.m1.1.1"><approx id="A8.p2.1.m1.1.1.1.cmml" xref="A8.p2.1.m1.1.1.1"></approx><apply id="A8.p2.1.m1.1.1.2.cmml" xref="A8.p2.1.m1.1.1.2"><divide id="A8.p2.1.m1.1.1.2.1.cmml" xref="A8.p2.1.m1.1.1.2"></divide><cn type="integer" id="A8.p2.1.m1.1.1.2.2.cmml" xref="A8.p2.1.m1.1.1.2.2">1</cn><apply id="A8.p2.1.m1.1.1.2.3.cmml" xref="A8.p2.1.m1.1.1.2.3"><times id="A8.p2.1.m1.1.1.2.3.1.cmml" xref="A8.p2.1.m1.1.1.2.3.1"></times><cn type="integer" id="A8.p2.1.m1.1.1.2.3.2.cmml" xref="A8.p2.1.m1.1.1.2.3.2">2</cn><apply id="A8.p2.1.m1.1.1.2.3.3.cmml" xref="A8.p2.1.m1.1.1.2.3.3"><root id="A8.p2.1.m1.1.1.2.3.3a.cmml" xref="A8.p2.1.m1.1.1.2.3.3"></root><cn type="integer" id="A8.p2.1.m1.1.1.2.3.3.2.cmml" xref="A8.p2.1.m1.1.1.2.3.3.2">3800</cn></apply></apply></apply><apply id="A8.p2.1.m1.1.1.3.cmml" xref="A8.p2.1.m1.1.1.3"><csymbol cd="latexml" id="A8.p2.1.m1.1.1.3.1.cmml" xref="A8.p2.1.m1.1.1.3.1">percent</csymbol><cn type="float" id="A8.p2.1.m1.1.1.3.2.cmml" xref="A8.p2.1.m1.1.1.3.2">0.8</cn></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A8.p2.1.m1.1c">\frac{1}{2\sqrt{3800}}\approx 0.8\%</annotation></semantics></math>. Applying a Wald test, the Wald statistic is <math id="A8.p2.2.m2.2" class="ltx_Math" alttext="W=\frac{\mu_{0}-\hat{\mu}}{se(\hat{\mu})^{2}}\geq\frac{\mu_{0}-\hat{\mu}}{(0.008)^{2}}" display="inline"><semantics id="A8.p2.2.m2.2a"><mrow id="A8.p2.2.m2.2.3" xref="A8.p2.2.m2.2.3.cmml"><mi id="A8.p2.2.m2.2.3.2" xref="A8.p2.2.m2.2.3.2.cmml">W</mi><mo id="A8.p2.2.m2.2.3.3" xref="A8.p2.2.m2.2.3.3.cmml">=</mo><mfrac id="A8.p2.2.m2.1.1" xref="A8.p2.2.m2.1.1.cmml"><mrow id="A8.p2.2.m2.1.1.3" xref="A8.p2.2.m2.1.1.3.cmml"><msub id="A8.p2.2.m2.1.1.3.2" xref="A8.p2.2.m2.1.1.3.2.cmml"><mi id="A8.p2.2.m2.1.1.3.2.2" xref="A8.p2.2.m2.1.1.3.2.2.cmml">μ</mi><mn id="A8.p2.2.m2.1.1.3.2.3" xref="A8.p2.2.m2.1.1.3.2.3.cmml">0</mn></msub><mo id="A8.p2.2.m2.1.1.3.1" xref="A8.p2.2.m2.1.1.3.1.cmml">−</mo><mover accent="true" id="A8.p2.2.m2.1.1.3.3" xref="A8.p2.2.m2.1.1.3.3.cmml"><mi id="A8.p2.2.m2.1.1.3.3.2" xref="A8.p2.2.m2.1.1.3.3.2.cmml">μ</mi><mo id="A8.p2.2.m2.1.1.3.3.1" xref="A8.p2.2.m2.1.1.3.3.1.cmml">^</mo></mover></mrow><mrow id="A8.p2.2.m2.1.1.1" xref="A8.p2.2.m2.1.1.1.cmml"><mi id="A8.p2.2.m2.1.1.1.3" xref="A8.p2.2.m2.1.1.1.3.cmml">s</mi><mo lspace="0em" rspace="0em" id="A8.p2.2.m2.1.1.1.2" xref="A8.p2.2.m2.1.1.1.2.cmml">​</mo><mi id="A8.p2.2.m2.1.1.1.4" xref="A8.p2.2.m2.1.1.1.4.cmml">e</mi><mo lspace="0em" rspace="0em" id="A8.p2.2.m2.1.1.1.2a" xref="A8.p2.2.m2.1.1.1.2.cmml">​</mo><msup id="A8.p2.2.m2.1.1.1.5" xref="A8.p2.2.m2.1.1.1.5.cmml"><mrow id="A8.p2.2.m2.1.1.1.5.2.2" xref="A8.p2.2.m2.1.1.1.1.cmml"><mo stretchy="false" id="A8.p2.2.m2.1.1.1.5.2.2.1" xref="A8.p2.2.m2.1.1.1.1.cmml">(</mo><mover accent="true" id="A8.p2.2.m2.1.1.1.1" xref="A8.p2.2.m2.1.1.1.1.cmml"><mi id="A8.p2.2.m2.1.1.1.1.2" xref="A8.p2.2.m2.1.1.1.1.2.cmml">μ</mi><mo id="A8.p2.2.m2.1.1.1.1.1" xref="A8.p2.2.m2.1.1.1.1.1.cmml">^</mo></mover><mo stretchy="false" id="A8.p2.2.m2.1.1.1.5.2.2.2" xref="A8.p2.2.m2.1.1.1.1.cmml">)</mo></mrow><mn id="A8.p2.2.m2.1.1.1.5.3" xref="A8.p2.2.m2.1.1.1.5.3.cmml">2</mn></msup></mrow></mfrac><mo id="A8.p2.2.m2.2.3.4" xref="A8.p2.2.m2.2.3.4.cmml">≥</mo><mfrac id="A8.p2.2.m2.2.2" xref="A8.p2.2.m2.2.2.cmml"><mrow id="A8.p2.2.m2.2.2.3" xref="A8.p2.2.m2.2.2.3.cmml"><msub id="A8.p2.2.m2.2.2.3.2" xref="A8.p2.2.m2.2.2.3.2.cmml"><mi id="A8.p2.2.m2.2.2.3.2.2" xref="A8.p2.2.m2.2.2.3.2.2.cmml">μ</mi><mn id="A8.p2.2.m2.2.2.3.2.3" xref="A8.p2.2.m2.2.2.3.2.3.cmml">0</mn></msub><mo id="A8.p2.2.m2.2.2.3.1" xref="A8.p2.2.m2.2.2.3.1.cmml">−</mo><mover accent="true" id="A8.p2.2.m2.2.2.3.3" xref="A8.p2.2.m2.2.2.3.3.cmml"><mi id="A8.p2.2.m2.2.2.3.3.2" xref="A8.p2.2.m2.2.2.3.3.2.cmml">μ</mi><mo id="A8.p2.2.m2.2.2.3.3.1" xref="A8.p2.2.m2.2.2.3.3.1.cmml">^</mo></mover></mrow><msup id="A8.p2.2.m2.2.2.1" xref="A8.p2.2.m2.2.2.1.cmml"><mrow id="A8.p2.2.m2.2.2.1.3.2" xref="A8.p2.2.m2.2.2.1.cmml"><mo stretchy="false" id="A8.p2.2.m2.2.2.1.3.2.1" xref="A8.p2.2.m2.2.2.1.cmml">(</mo><mn id="A8.p2.2.m2.2.2.1.1" xref="A8.p2.2.m2.2.2.1.1.cmml">0.008</mn><mo stretchy="false" id="A8.p2.2.m2.2.2.1.3.2.2" xref="A8.p2.2.m2.2.2.1.cmml">)</mo></mrow><mn id="A8.p2.2.m2.2.2.1.4" xref="A8.p2.2.m2.2.2.1.4.cmml">2</mn></msup></mfrac></mrow><annotation-xml encoding="MathML-Content" id="A8.p2.2.m2.2b"><apply id="A8.p2.2.m2.2.3.cmml" xref="A8.p2.2.m2.2.3"><and id="A8.p2.2.m2.2.3a.cmml" xref="A8.p2.2.m2.2.3"></and><apply id="A8.p2.2.m2.2.3b.cmml" xref="A8.p2.2.m2.2.3"><eq id="A8.p2.2.m2.2.3.3.cmml" xref="A8.p2.2.m2.2.3.3"></eq><ci id="A8.p2.2.m2.2.3.2.cmml" xref="A8.p2.2.m2.2.3.2">𝑊</ci><apply id="A8.p2.2.m2.1.1.cmml" xref="A8.p2.2.m2.1.1"><divide id="A8.p2.2.m2.1.1.2.cmml" xref="A8.p2.2.m2.1.1"></divide><apply id="A8.p2.2.m2.1.1.3.cmml" xref="A8.p2.2.m2.1.1.3"><minus id="A8.p2.2.m2.1.1.3.1.cmml" xref="A8.p2.2.m2.1.1.3.1"></minus><apply id="A8.p2.2.m2.1.1.3.2.cmml" xref="A8.p2.2.m2.1.1.3.2"><csymbol cd="ambiguous" id="A8.p2.2.m2.1.1.3.2.1.cmml" xref="A8.p2.2.m2.1.1.3.2">subscript</csymbol><ci id="A8.p2.2.m2.1.1.3.2.2.cmml" xref="A8.p2.2.m2.1.1.3.2.2">𝜇</ci><cn type="integer" id="A8.p2.2.m2.1.1.3.2.3.cmml" xref="A8.p2.2.m2.1.1.3.2.3">0</cn></apply><apply id="A8.p2.2.m2.1.1.3.3.cmml" xref="A8.p2.2.m2.1.1.3.3"><ci id="A8.p2.2.m2.1.1.3.3.1.cmml" xref="A8.p2.2.m2.1.1.3.3.1">^</ci><ci id="A8.p2.2.m2.1.1.3.3.2.cmml" xref="A8.p2.2.m2.1.1.3.3.2">𝜇</ci></apply></apply><apply id="A8.p2.2.m2.1.1.1.cmml" xref="A8.p2.2.m2.1.1.1"><times id="A8.p2.2.m2.1.1.1.2.cmml" xref="A8.p2.2.m2.1.1.1.2"></times><ci id="A8.p2.2.m2.1.1.1.3.cmml" xref="A8.p2.2.m2.1.1.1.3">𝑠</ci><ci id="A8.p2.2.m2.1.1.1.4.cmml" xref="A8.p2.2.m2.1.1.1.4">𝑒</ci><apply id="A8.p2.2.m2.1.1.1.5.cmml" xref="A8.p2.2.m2.1.1.1.5"><csymbol cd="ambiguous" id="A8.p2.2.m2.1.1.1.5.1.cmml" xref="A8.p2.2.m2.1.1.1.5">superscript</csymbol><apply id="A8.p2.2.m2.1.1.1.1.cmml" xref="A8.p2.2.m2.1.1.1.5.2.2"><ci id="A8.p2.2.m2.1.1.1.1.1.cmml" xref="A8.p2.2.m2.1.1.1.1.1">^</ci><ci id="A8.p2.2.m2.1.1.1.1.2.cmml" xref="A8.p2.2.m2.1.1.1.1.2">𝜇</ci></apply><cn type="integer" id="A8.p2.2.m2.1.1.1.5.3.cmml" xref="A8.p2.2.m2.1.1.1.5.3">2</cn></apply></apply></apply></apply><apply id="A8.p2.2.m2.2.3c.cmml" xref="A8.p2.2.m2.2.3"><geq id="A8.p2.2.m2.2.3.4.cmml" xref="A8.p2.2.m2.2.3.4"></geq><share href="#A8.p2.2.m2.1.1.cmml" id="A8.p2.2.m2.2.3d.cmml" xref="A8.p2.2.m2.2.3"></share><apply id="A8.p2.2.m2.2.2.cmml" xref="A8.p2.2.m2.2.2"><divide id="A8.p2.2.m2.2.2.2.cmml" xref="A8.p2.2.m2.2.2"></divide><apply id="A8.p2.2.m2.2.2.3.cmml" xref="A8.p2.2.m2.2.2.3"><minus id="A8.p2.2.m2.2.2.3.1.cmml" xref="A8.p2.2.m2.2.2.3.1"></minus><apply id="A8.p2.2.m2.2.2.3.2.cmml" xref="A8.p2.2.m2.2.2.3.2"><csymbol cd="ambiguous" id="A8.p2.2.m2.2.2.3.2.1.cmml" xref="A8.p2.2.m2.2.2.3.2">subscript</csymbol><ci id="A8.p2.2.m2.2.2.3.2.2.cmml" xref="A8.p2.2.m2.2.2.3.2.2">𝜇</ci><cn type="integer" id="A8.p2.2.m2.2.2.3.2.3.cmml" xref="A8.p2.2.m2.2.2.3.2.3">0</cn></apply><apply id="A8.p2.2.m2.2.2.3.3.cmml" xref="A8.p2.2.m2.2.2.3.3"><ci id="A8.p2.2.m2.2.2.3.3.1.cmml" xref="A8.p2.2.m2.2.2.3.3.1">^</ci><ci id="A8.p2.2.m2.2.2.3.3.2.cmml" xref="A8.p2.2.m2.2.2.3.3.2">𝜇</ci></apply></apply><apply id="A8.p2.2.m2.2.2.1.cmml" xref="A8.p2.2.m2.2.2.1"><csymbol cd="ambiguous" id="A8.p2.2.m2.2.2.1.2.cmml" xref="A8.p2.2.m2.2.2.1">superscript</csymbol><cn type="float" id="A8.p2.2.m2.2.2.1.1.cmml" xref="A8.p2.2.m2.2.2.1.1">0.008</cn><cn type="integer" id="A8.p2.2.m2.2.2.1.4.cmml" xref="A8.p2.2.m2.2.2.1.4">2</cn></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="A8.p2.2.m2.2c">W=\frac{\mu_{0}-\hat{\mu}}{se(\hat{\mu})^{2}}\geq\frac{\mu_{0}-\hat{\mu}}{(0.008)^{2}}</annotation></semantics></math>, where (for example) <math id="A8.p2.3.m3.1" class="ltx_Math" alttext="\mu_{0}" display="inline"><semantics id="A8.p2.3.m3.1a"><msub id="A8.p2.3.m3.1.1" xref="A8.p2.3.m3.1.1.cmml"><mi id="A8.p2.3.m3.1.1.2" xref="A8.p2.3.m3.1.1.2.cmml">μ</mi><mn id="A8.p2.3.m3.1.1.3" xref="A8.p2.3.m3.1.1.3.cmml">0</mn></msub><annotation-xml encoding="MathML-Content" id="A8.p2.3.m3.1b"><apply id="A8.p2.3.m3.1.1.cmml" xref="A8.p2.3.m3.1.1"><csymbol cd="ambiguous" id="A8.p2.3.m3.1.1.1.cmml" xref="A8.p2.3.m3.1.1">subscript</csymbol><ci id="A8.p2.3.m3.1.1.2.cmml" xref="A8.p2.3.m3.1.1.2">𝜇</ci><cn type="integer" id="A8.p2.3.m3.1.1.3.cmml" xref="A8.p2.3.m3.1.1.3">0</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A8.p2.3.m3.1c">\mu_{0}</annotation></semantics></math> is zero-shot accuracy and <math id="A8.p2.4.m4.1" class="ltx_Math" alttext="\hat{\mu}" display="inline"><semantics id="A8.p2.4.m4.1a"><mover accent="true" id="A8.p2.4.m4.1.1" xref="A8.p2.4.m4.1.1.cmml"><mi id="A8.p2.4.m4.1.1.2" xref="A8.p2.4.m4.1.1.2.cmml">μ</mi><mo id="A8.p2.4.m4.1.1.1" xref="A8.p2.4.m4.1.1.1.cmml">^</mo></mover><annotation-xml encoding="MathML-Content" id="A8.p2.4.m4.1b"><apply id="A8.p2.4.m4.1.1.cmml" xref="A8.p2.4.m4.1.1"><ci id="A8.p2.4.m4.1.1.1.cmml" xref="A8.p2.4.m4.1.1.1">^</ci><ci id="A8.p2.4.m4.1.1.2.cmml" xref="A8.p2.4.m4.1.1.2">𝜇</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A8.p2.4.m4.1c">\hat{\mu}</annotation></semantics></math> is CCS accuracy. <math id="A8.p2.5.m5.1" class="ltx_Math" alttext="W" display="inline"><semantics id="A8.p2.5.m5.1a"><mi id="A8.p2.5.m5.1.1" xref="A8.p2.5.m5.1.1.cmml">W</mi><annotation-xml encoding="MathML-Content" id="A8.p2.5.m5.1b"><ci id="A8.p2.5.m5.1.1.cmml" xref="A8.p2.5.m5.1.1">𝑊</ci></annotation-xml><annotation encoding="application/x-tex" id="A8.p2.5.m5.1c">W</annotation></semantics></math> is then <math id="A8.p2.6.m6.1" class="ltx_Math" alttext="\chi^{2}" display="inline"><semantics id="A8.p2.6.m6.1a"><msup id="A8.p2.6.m6.1.1" xref="A8.p2.6.m6.1.1.cmml"><mi id="A8.p2.6.m6.1.1.2" xref="A8.p2.6.m6.1.1.2.cmml">χ</mi><mn id="A8.p2.6.m6.1.1.3" xref="A8.p2.6.m6.1.1.3.cmml">2</mn></msup><annotation-xml encoding="MathML-Content" id="A8.p2.6.m6.1b"><apply id="A8.p2.6.m6.1.1.cmml" xref="A8.p2.6.m6.1.1"><csymbol cd="ambiguous" id="A8.p2.6.m6.1.1.1.cmml" xref="A8.p2.6.m6.1.1">superscript</csymbol><ci id="A8.p2.6.m6.1.1.2.cmml" xref="A8.p2.6.m6.1.1.2">𝜒</ci><cn type="integer" id="A8.p2.6.m6.1.1.3.cmml" xref="A8.p2.6.m6.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A8.p2.6.m6.1c">\chi^{2}</annotation></semantics></math>-distributed with one degree of freedom.</p>
</div>
<div id="A8.p3" class="ltx_para ltx_noindent">
<p id="A8.p3.1" class="ltx_p">We find that our main claims comparing accuracies are statistically significant at a 0.00001 level. This includes the claims that CCS outperforms zero-shot, with accuracies of 71.2% vs 67.2%, respectively; that CCS is robust to misleading prompts while zero-shot isn’t in the setting we test, with accuracies of 83.8% vs 70.9%, respectively; that CCS on MLM-pretrained DeBERTa substantially outperforms DeBERTa zero-shot, with accuracies of 93.7% vs 71.6%, respectively; and so on.
Some minor observations are not necessarily statistically significant, such as that CCS (All Data) outperforms CCS on average (<a href="#S3.SS2.SSS1" title="3.2.1 CCS Outperforms Zero-Shot ‣ 3.2 Evaluating CCS ‣ 3 Results ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">3.2.1</span></a>), but none of these are important for our main claims, and more powerful but complicated statistical tests would result in smaller p-values.</p>
</div>
<div class="ltx_pagination ltx_role_newpage"></div>
</section>
<section id="A9" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix I </span>Dataset Setup</h2>

<div id="A9.p1" class="ltx_para ltx_noindent">
<p id="A9.p1.1" class="ltx_p">In this section, we describe the setup of our data and prompts in detail.
We introduce all datasets we use and the way we convert them into a binary classification task.
Our prompts derive from <cite class="ltx_cite ltx_citemacro_citep">(Sanh et al., <a href="#bib.bib44" title="" class="ltx_ref">2021</a>)</cite><span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span><a target="_blank" href="https://github.com/bigscience-workshop/promptsource" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/bigscience-workshop/promptsource</a></span></span></span>.</p>
</div>
<div id="A9.p2" class="ltx_para ltx_noindent">
<p id="A9.p2.2" class="ltx_p"><span id="A9.p2.2.1" class="ltx_text ltx_font_bold">Contrast Pair Example.</span> To illustrate how we construct contrast pairs, suppose we have a movie review “[text] = I loved this movie.” and the sentiment of this review belongs to “[label0] = positive” or “[label1] = negative”.
We first format it into a binary question-answering or classification question: “[text] Is the sentiment of this example [label0] or [label1]? ” using an existing zero-shot prompt.
We then concatenate the question and candidate labels to create the contrast pairs:</p>
<table id="A9.EGx2" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A9.Ex6"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A9.Ex6.m1.1" class="ltx_Math" alttext="\displaystyle x^{+}" display="inline"><semantics id="A9.Ex6.m1.1a"><msup id="A9.Ex6.m1.1.1" xref="A9.Ex6.m1.1.1.cmml"><mi id="A9.Ex6.m1.1.1.2" xref="A9.Ex6.m1.1.1.2.cmml">x</mi><mo id="A9.Ex6.m1.1.1.3" xref="A9.Ex6.m1.1.1.3.cmml">+</mo></msup><annotation-xml encoding="MathML-Content" id="A9.Ex6.m1.1b"><apply id="A9.Ex6.m1.1.1.cmml" xref="A9.Ex6.m1.1.1"><csymbol cd="ambiguous" id="A9.Ex6.m1.1.1.1.cmml" xref="A9.Ex6.m1.1.1">superscript</csymbol><ci id="A9.Ex6.m1.1.1.2.cmml" xref="A9.Ex6.m1.1.1.2">𝑥</ci><plus id="A9.Ex6.m1.1.1.3.cmml" xref="A9.Ex6.m1.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.Ex6.m1.1c">\displaystyle x^{+}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A9.Ex6.m2.1" class="ltx_Math" alttext="\displaystyle=\text{[prefix] Q: Is the sentiment of ``[text]'' [label0] or [label1]? A: [label0]}" display="inline"><semantics id="A9.Ex6.m2.1a"><mrow id="A9.Ex6.m2.1.1" xref="A9.Ex6.m2.1.1.cmml"><mi id="A9.Ex6.m2.1.1.2" xref="A9.Ex6.m2.1.1.2.cmml"></mi><mo id="A9.Ex6.m2.1.1.1" xref="A9.Ex6.m2.1.1.1.cmml">=</mo><mtext id="A9.Ex6.m2.1.1.3" xref="A9.Ex6.m2.1.1.3a.cmml">[prefix] Q: Is the sentiment of “[text]” [label0] or [label1]? A: [label0]</mtext></mrow><annotation-xml encoding="MathML-Content" id="A9.Ex6.m2.1b"><apply id="A9.Ex6.m2.1.1.cmml" xref="A9.Ex6.m2.1.1"><eq id="A9.Ex6.m2.1.1.1.cmml" xref="A9.Ex6.m2.1.1.1"></eq><csymbol cd="latexml" id="A9.Ex6.m2.1.1.2.cmml" xref="A9.Ex6.m2.1.1.2">absent</csymbol><ci id="A9.Ex6.m2.1.1.3a.cmml" xref="A9.Ex6.m2.1.1.3"><mtext id="A9.Ex6.m2.1.1.3.cmml" xref="A9.Ex6.m2.1.1.3">[prefix] Q: Is the sentiment of “[text]” [label0] or [label1]? A: [label0]</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.Ex6.m2.1c">\displaystyle=\text{[prefix] Q: Is the sentiment of ``[text]'' [label0] or [label1]? A: [label0]}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="A9.Ex7"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A9.Ex7.m1.1" class="ltx_Math" alttext="\displaystyle x^{-}" display="inline"><semantics id="A9.Ex7.m1.1a"><msup id="A9.Ex7.m1.1.1" xref="A9.Ex7.m1.1.1.cmml"><mi id="A9.Ex7.m1.1.1.2" xref="A9.Ex7.m1.1.1.2.cmml">x</mi><mo id="A9.Ex7.m1.1.1.3" xref="A9.Ex7.m1.1.1.3.cmml">−</mo></msup><annotation-xml encoding="MathML-Content" id="A9.Ex7.m1.1b"><apply id="A9.Ex7.m1.1.1.cmml" xref="A9.Ex7.m1.1.1"><csymbol cd="ambiguous" id="A9.Ex7.m1.1.1.1.cmml" xref="A9.Ex7.m1.1.1">superscript</csymbol><ci id="A9.Ex7.m1.1.1.2.cmml" xref="A9.Ex7.m1.1.1.2">𝑥</ci><minus id="A9.Ex7.m1.1.1.3.cmml" xref="A9.Ex7.m1.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.Ex7.m1.1c">\displaystyle x^{-}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A9.Ex7.m2.1" class="ltx_Math" alttext="\displaystyle=\text{[prefix] Q: Is the sentiment of ``[text]'' [label0] or [label1]? A: [label1]}" display="inline"><semantics id="A9.Ex7.m2.1a"><mrow id="A9.Ex7.m2.1.1" xref="A9.Ex7.m2.1.1.cmml"><mi id="A9.Ex7.m2.1.1.2" xref="A9.Ex7.m2.1.1.2.cmml"></mi><mo id="A9.Ex7.m2.1.1.1" xref="A9.Ex7.m2.1.1.1.cmml">=</mo><mtext id="A9.Ex7.m2.1.1.3" xref="A9.Ex7.m2.1.1.3a.cmml">[prefix] Q: Is the sentiment of “[text]” [label0] or [label1]? A: [label1]</mtext></mrow><annotation-xml encoding="MathML-Content" id="A9.Ex7.m2.1b"><apply id="A9.Ex7.m2.1.1.cmml" xref="A9.Ex7.m2.1.1"><eq id="A9.Ex7.m2.1.1.1.cmml" xref="A9.Ex7.m2.1.1.1"></eq><csymbol cd="latexml" id="A9.Ex7.m2.1.1.2.cmml" xref="A9.Ex7.m2.1.1.2">absent</csymbol><ci id="A9.Ex7.m2.1.1.3a.cmml" xref="A9.Ex7.m2.1.1.3"><mtext id="A9.Ex7.m2.1.1.3.cmml" xref="A9.Ex7.m2.1.1.3">[prefix] Q: Is the sentiment of “[text]” [label0] or [label1]? A: [label1]</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.Ex7.m2.1c">\displaystyle=\text{[prefix] Q: Is the sentiment of ``[text]'' [label0] or [label1]? A: [label1]}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A9.p2.3" class="ltx_p">For instance, in this example, these would be:</p>
<table id="A9.EGx3" class="ltx_equationgroup ltx_eqn_align ltx_eqn_table">

<tbody id="A9.Ex8"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A9.Ex8.m1.1" class="ltx_Math" alttext="\displaystyle x^{+}" display="inline"><semantics id="A9.Ex8.m1.1a"><msup id="A9.Ex8.m1.1.1" xref="A9.Ex8.m1.1.1.cmml"><mi id="A9.Ex8.m1.1.1.2" xref="A9.Ex8.m1.1.1.2.cmml">x</mi><mo id="A9.Ex8.m1.1.1.3" xref="A9.Ex8.m1.1.1.3.cmml">+</mo></msup><annotation-xml encoding="MathML-Content" id="A9.Ex8.m1.1b"><apply id="A9.Ex8.m1.1.1.cmml" xref="A9.Ex8.m1.1.1"><csymbol cd="ambiguous" id="A9.Ex8.m1.1.1.1.cmml" xref="A9.Ex8.m1.1.1">superscript</csymbol><ci id="A9.Ex8.m1.1.1.2.cmml" xref="A9.Ex8.m1.1.1.2">𝑥</ci><plus id="A9.Ex8.m1.1.1.3.cmml" xref="A9.Ex8.m1.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.Ex8.m1.1c">\displaystyle x^{+}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A9.Ex8.m2.1" class="ltx_Math" alttext="\displaystyle=\text{Q: Is the sentiment of ``I loved this movie.'' positive or negative? A: positive}" display="inline"><semantics id="A9.Ex8.m2.1a"><mrow id="A9.Ex8.m2.1.1" xref="A9.Ex8.m2.1.1.cmml"><mi id="A9.Ex8.m2.1.1.2" xref="A9.Ex8.m2.1.1.2.cmml"></mi><mo id="A9.Ex8.m2.1.1.1" xref="A9.Ex8.m2.1.1.1.cmml">=</mo><mtext id="A9.Ex8.m2.1.1.3" xref="A9.Ex8.m2.1.1.3a.cmml">Q: Is the sentiment of “I loved this movie.” positive or negative? A: positive</mtext></mrow><annotation-xml encoding="MathML-Content" id="A9.Ex8.m2.1b"><apply id="A9.Ex8.m2.1.1.cmml" xref="A9.Ex8.m2.1.1"><eq id="A9.Ex8.m2.1.1.1.cmml" xref="A9.Ex8.m2.1.1.1"></eq><csymbol cd="latexml" id="A9.Ex8.m2.1.1.2.cmml" xref="A9.Ex8.m2.1.1.2">absent</csymbol><ci id="A9.Ex8.m2.1.1.3a.cmml" xref="A9.Ex8.m2.1.1.3"><mtext id="A9.Ex8.m2.1.1.3.cmml" xref="A9.Ex8.m2.1.1.3">Q: Is the sentiment of “I loved this movie.” positive or negative? A: positive</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.Ex8.m2.1c">\displaystyle=\text{Q: Is the sentiment of ``I loved this movie.'' positive or negative? A: positive}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
<tbody id="A9.Ex9"><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_td ltx_align_right ltx_eqn_cell"><math id="A9.Ex9.m1.1" class="ltx_Math" alttext="\displaystyle x^{-}" display="inline"><semantics id="A9.Ex9.m1.1a"><msup id="A9.Ex9.m1.1.1" xref="A9.Ex9.m1.1.1.cmml"><mi id="A9.Ex9.m1.1.1.2" xref="A9.Ex9.m1.1.1.2.cmml">x</mi><mo id="A9.Ex9.m1.1.1.3" xref="A9.Ex9.m1.1.1.3.cmml">−</mo></msup><annotation-xml encoding="MathML-Content" id="A9.Ex9.m1.1b"><apply id="A9.Ex9.m1.1.1.cmml" xref="A9.Ex9.m1.1.1"><csymbol cd="ambiguous" id="A9.Ex9.m1.1.1.1.cmml" xref="A9.Ex9.m1.1.1">superscript</csymbol><ci id="A9.Ex9.m1.1.1.2.cmml" xref="A9.Ex9.m1.1.1.2">𝑥</ci><minus id="A9.Ex9.m1.1.1.3.cmml" xref="A9.Ex9.m1.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.Ex9.m1.1c">\displaystyle x^{-}</annotation></semantics></math></td>
<td class="ltx_td ltx_align_left ltx_eqn_cell"><math id="A9.Ex9.m2.1" class="ltx_Math" alttext="\displaystyle=\text{Q: Is the sentiment of ``I loved this movie.'' positive or negative? A: negative}" display="inline"><semantics id="A9.Ex9.m2.1a"><mrow id="A9.Ex9.m2.1.1" xref="A9.Ex9.m2.1.1.cmml"><mi id="A9.Ex9.m2.1.1.2" xref="A9.Ex9.m2.1.1.2.cmml"></mi><mo id="A9.Ex9.m2.1.1.1" xref="A9.Ex9.m2.1.1.1.cmml">=</mo><mtext id="A9.Ex9.m2.1.1.3" xref="A9.Ex9.m2.1.1.3a.cmml">Q: Is the sentiment of “I loved this movie.” positive or negative? A: negative</mtext></mrow><annotation-xml encoding="MathML-Content" id="A9.Ex9.m2.1b"><apply id="A9.Ex9.m2.1.1.cmml" xref="A9.Ex9.m2.1.1"><eq id="A9.Ex9.m2.1.1.1.cmml" xref="A9.Ex9.m2.1.1.1"></eq><csymbol cd="latexml" id="A9.Ex9.m2.1.1.2.cmml" xref="A9.Ex9.m2.1.1.2">absent</csymbol><ci id="A9.Ex9.m2.1.1.3a.cmml" xref="A9.Ex9.m2.1.1.3"><mtext id="A9.Ex9.m2.1.1.3.cmml" xref="A9.Ex9.m2.1.1.3">Q: Is the sentiment of “I loved this movie.” positive or negative? A: negative</mtext></ci></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.Ex9.m2.1c">\displaystyle=\text{Q: Is the sentiment of ``I loved this movie.'' positive or negative? A: negative}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
</tr></tbody>
</table>
<p id="A9.p2.1" class="ltx_p">Finally, we add the prefix (either prefix in <a href="#A2.F5" title="In B.1.1 Evaluating The Effect of The Misleading Prefix ‣ B.1 Misleading Prefix ‣ Appendix B Misleading Prefix Details ‣ Discovering Latent Knowledge in Language Models Without Supervision" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">5</span></a>, or the empty string for the regular zero-shot setting) to the beginning of the prompt, then compute the hidden representations for this pair to obtain the contrast features <math id="A9.p2.1.m1.1" class="ltx_Math" alttext="\mathcal{C}" display="inline"><semantics id="A9.p2.1.m1.1a"><mi class="ltx_font_mathcaligraphic" id="A9.p2.1.m1.1.1" xref="A9.p2.1.m1.1.1.cmml">𝒞</mi><annotation-xml encoding="MathML-Content" id="A9.p2.1.m1.1b"><ci id="A9.p2.1.m1.1.1.cmml" xref="A9.p2.1.m1.1.1">𝒞</ci></annotation-xml><annotation encoding="application/x-tex" id="A9.p2.1.m1.1c">\mathcal{C}</annotation></semantics></math>.</p>
</div>
<section id="A9.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">I.1 </span>Datasets</h3>

<div id="A9.SS1.p1" class="ltx_para ltx_noindent">
<p id="A9.SS1.p1.2" class="ltx_p">We use ten datasets in our experiments.
For each dataset, we select 1000 data points (except COPA, which has only <math id="A9.SS1.p1.1.m1.1" class="ltx_Math" alttext="500" display="inline"><semantics id="A9.SS1.p1.1.m1.1a"><mn id="A9.SS1.p1.1.m1.1.1" xref="A9.SS1.p1.1.m1.1.1.cmml">500</mn><annotation-xml encoding="MathML-Content" id="A9.SS1.p1.1.m1.1b"><cn type="integer" id="A9.SS1.p1.1.m1.1.1.cmml" xref="A9.SS1.p1.1.m1.1.1">500</cn></annotation-xml><annotation encoding="application/x-tex" id="A9.SS1.p1.1.m1.1c">500</annotation></semantics></math> data points, so we only use those <math id="A9.SS1.p1.2.m2.1" class="ltx_Math" alttext="500" display="inline"><semantics id="A9.SS1.p1.2.m2.1a"><mn id="A9.SS1.p1.2.m2.1.1" xref="A9.SS1.p1.2.m2.1.1.cmml">500</mn><annotation-xml encoding="MathML-Content" id="A9.SS1.p1.2.m2.1b"><cn type="integer" id="A9.SS1.p1.2.m2.1.1.cmml" xref="A9.SS1.p1.2.m2.1.1">500</cn></annotation-xml><annotation encoding="application/x-tex" id="A9.SS1.p1.2.m2.1c">500</annotation></semantics></math>).
When possible, we use just the test / validation split, but in other cases this is not enough data so we also use examples from the train split.
For the most part this doesn’t make a difference for our purposes because our methods are purely unsupervised.</p>
</div>
<div id="A9.SS1.p2" class="ltx_para ltx_noindent">
<p id="A9.SS1.p2.1" class="ltx_p">It is important to note that T0 is an exception among the models, because but was trained on 9 out of the 10 datasets we consider (all except RTE). This may account for its anomalous performance.
UnifiedQA was trained on BoolQ, which is one of the datasets we test on, but was not trained on any other evaluation set we consider, which we don’t expect has an important qualitative effect.</p>
</div>
<div id="A9.SS1.p3" class="ltx_para ltx_noindent">
<p id="A9.SS1.p3.1" class="ltx_p">We now enumerate the prompts we use for each dataset.</p>
</div>
<section id="A9.SS1.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">I.1.1 </span>AG_News</h4>

<div id="A9.SS1.SSS1.p1" class="ltx_para ltx_noindent">
<p id="A9.SS1.SSS1.p1.1" class="ltx_p">AG_News is a topic classification dataset from <cite class="ltx_cite ltx_citemacro_citep">(Zhang et al., <a href="#bib.bib54" title="" class="ltx_ref">2015</a>)</cite>. Given a news paragraph, the label corresponds to the topic of this news, including “World politics”, “Sports”, “Business” and “Science and technology”.
For each example, we randomly select another label (which is incorrect), and randomly assign the correct label and this incorrect label to “label0” and “label1”. With this, we construct the classification task using the following 8 prompts:</p>
<ol id="A9.I1" class="ltx_enumerate">
<li id="A9.I1.ix1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1</span> 
<div id="A9.I1.ix1.p1" class="ltx_para ltx_noindent">
<p id="A9.I1.ix1.p1.1" class="ltx_p">[prefix]Consider the following example: “‘ [text] ”’
Choice 1: [label0]. Choice 2: [label1].Between choice 1 and choice 2, the topic of this example is [label]</p>
</div>
</li>
<li id="A9.I1.ix2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2</span> 
<div id="A9.I1.ix2.p1" class="ltx_para ltx_noindent">
<p id="A9.I1.ix2.p1.1" class="ltx_p">[prefix]Consider the following example: “‘ [text] ”’
Choice 1: [label0]. Choice 2: [label1].Between choice 1 and choice 2, what is the topic of this example?
[label]</p>
</div>
</li>
<li id="A9.I1.ix3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3</span> 
<div id="A9.I1.ix3.p1" class="ltx_para ltx_noindent">
<p id="A9.I1.ix3.p1.1" class="ltx_p">[prefix]Consider the following example: “‘ [text] ”’
Which is the topic of this example, choice 1: [label0], or choice 2: [label1]?
[label]</p>
</div>
</li>
<li id="A9.I1.ix4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">4</span> 
<div id="A9.I1.ix4.p1" class="ltx_para ltx_noindent">
<p id="A9.I1.ix4.p1.1" class="ltx_p">[prefix][text]
Choice 1: [label0]. Choice 2: [label1].Between choice 1 and choice 2, the topic of this example is [label]</p>
</div>
</li>
<li id="A9.I1.ix5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">5</span> 
<div id="A9.I1.ix5.p1" class="ltx_para ltx_noindent">
<p id="A9.I1.ix5.p1.1" class="ltx_p">[prefix][text]
Choice 1: [label0]. Choice 2: [label1].Between choice 1 and choice 2, what is the topic of this example?
[label]</p>
</div>
</li>
<li id="A9.I1.ix6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">6</span> 
<div id="A9.I1.ix6.p1" class="ltx_para ltx_noindent">
<p id="A9.I1.ix6.p1.1" class="ltx_p">[prefix][text]
Which is the topic of this example, choice 1: [label0], or choice 2: [label1]?
[label]</p>
</div>
</li>
<li id="A9.I1.ix7" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">7</span> 
<div id="A9.I1.ix7.p1" class="ltx_para ltx_noindent">
<p id="A9.I1.ix7.p1.1" class="ltx_p">[prefix][text]
What label best describes this news article, choice 1: [label0], or choice 2: [label1]?
[label]</p>
</div>
</li>
<li id="A9.I1.ix8" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">8</span> 
<div id="A9.I1.ix8.p1" class="ltx_para ltx_noindent">
<p id="A9.I1.ix8.p1.1" class="ltx_p">[prefix][text]
Which section of a newspaper would this article likely appear in, choice 1: [label0], or choice 2: [label1]?
[label]</p>
</div>
</li>
</ol>
</div>
<div id="A9.SS1.SSS1.p2" class="ltx_para ltx_noindent">
<p id="A9.SS1.SSS1.p2.2" class="ltx_p">Here the last “[label]” is “choice 1” for <math id="A9.SS1.SSS1.p2.1.m1.1" class="ltx_Math" alttext="x^{+}" display="inline"><semantics id="A9.SS1.SSS1.p2.1.m1.1a"><msup id="A9.SS1.SSS1.p2.1.m1.1.1" xref="A9.SS1.SSS1.p2.1.m1.1.1.cmml"><mi id="A9.SS1.SSS1.p2.1.m1.1.1.2" xref="A9.SS1.SSS1.p2.1.m1.1.1.2.cmml">x</mi><mo id="A9.SS1.SSS1.p2.1.m1.1.1.3" xref="A9.SS1.SSS1.p2.1.m1.1.1.3.cmml">+</mo></msup><annotation-xml encoding="MathML-Content" id="A9.SS1.SSS1.p2.1.m1.1b"><apply id="A9.SS1.SSS1.p2.1.m1.1.1.cmml" xref="A9.SS1.SSS1.p2.1.m1.1.1"><csymbol cd="ambiguous" id="A9.SS1.SSS1.p2.1.m1.1.1.1.cmml" xref="A9.SS1.SSS1.p2.1.m1.1.1">superscript</csymbol><ci id="A9.SS1.SSS1.p2.1.m1.1.1.2.cmml" xref="A9.SS1.SSS1.p2.1.m1.1.1.2">𝑥</ci><plus id="A9.SS1.SSS1.p2.1.m1.1.1.3.cmml" xref="A9.SS1.SSS1.p2.1.m1.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.SS1.SSS1.p2.1.m1.1c">x^{+}</annotation></semantics></math> and “choice 2” for <math id="A9.SS1.SSS1.p2.2.m2.1" class="ltx_Math" alttext="x^{-}" display="inline"><semantics id="A9.SS1.SSS1.p2.2.m2.1a"><msup id="A9.SS1.SSS1.p2.2.m2.1.1" xref="A9.SS1.SSS1.p2.2.m2.1.1.cmml"><mi id="A9.SS1.SSS1.p2.2.m2.1.1.2" xref="A9.SS1.SSS1.p2.2.m2.1.1.2.cmml">x</mi><mo id="A9.SS1.SSS1.p2.2.m2.1.1.3" xref="A9.SS1.SSS1.p2.2.m2.1.1.3.cmml">−</mo></msup><annotation-xml encoding="MathML-Content" id="A9.SS1.SSS1.p2.2.m2.1b"><apply id="A9.SS1.SSS1.p2.2.m2.1.1.cmml" xref="A9.SS1.SSS1.p2.2.m2.1.1"><csymbol cd="ambiguous" id="A9.SS1.SSS1.p2.2.m2.1.1.1.cmml" xref="A9.SS1.SSS1.p2.2.m2.1.1">superscript</csymbol><ci id="A9.SS1.SSS1.p2.2.m2.1.1.2.cmml" xref="A9.SS1.SSS1.p2.2.m2.1.1.2">𝑥</ci><minus id="A9.SS1.SSS1.p2.2.m2.1.1.3.cmml" xref="A9.SS1.SSS1.p2.2.m2.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.SS1.SSS1.p2.2.m2.1c">x^{-}</annotation></semantics></math>. Notice that we only specify “[label]” for the prompts that are constructed manually. For prompts in <cite class="ltx_cite ltx_citemacro_citep">(Sanh et al., <a href="#bib.bib44" title="" class="ltx_ref">2021</a>)</cite>, we leave their labels unchanged. (For example, we use “choice 1” or “choice 2” but they can use “Animal” or “Plant”)</p>
</div>
</section>
<section id="A9.SS1.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">I.1.2 </span>Amazon_polarity</h4>

<div id="A9.SS1.SSS2.p1" class="ltx_para ltx_noindent">
<p id="A9.SS1.SSS2.p1.1" class="ltx_p">Amazon_polarity is a sentiment classification task from <cite class="ltx_cite ltx_citemacro_citep">(McAuley &amp; Leskovec, <a href="#bib.bib32" title="" class="ltx_ref">2013</a>)</cite>.
The content is the review of goods in Amazon, and the label can be “negative” or “positive”.
We use 11 different prompts in this dataset. We first take all prompts from <cite class="ltx_cite ltx_citemacro_citep">(Sanh et al., <a href="#bib.bib44" title="" class="ltx_ref">2021</a>)</cite> (Page 164, 9 prompts in total), and the two of our own as follows:</p>
</div>
<div id="A9.SS1.SSS2.p2" class="ltx_para ltx_noindent">
<ol id="A9.I2" class="ltx_enumerate">
<li id="A9.I2.ix1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1</span> 
<div id="A9.I2.ix1.p1" class="ltx_para ltx_noindent">
<p id="A9.I2.ix1.p1.1" class="ltx_p">[prefix]Consider the following example: “‘ [content] ”’
Between [label0] and [label1], the sentiment of this example is
[label]</p>
</div>
</li>
<li id="A9.I2.ix2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2</span> 
<div id="A9.I2.ix2.p1" class="ltx_para ltx_noindent">
<p id="A9.I2.ix2.p1.1" class="ltx_p">[prefix]Consider the following example: “‘ [content] ”’
Between [label0] and [label1], which is the sentiment of this example?
[label]</p>
</div>
</li>
</ol>
</div>
<div id="A9.SS1.SSS2.p3" class="ltx_para ltx_noindent">
<p id="A9.SS1.SSS2.p3.2" class="ltx_p">Here “[label]” is “negative” for <math id="A9.SS1.SSS2.p3.1.m1.1" class="ltx_Math" alttext="x^{+}" display="inline"><semantics id="A9.SS1.SSS2.p3.1.m1.1a"><msup id="A9.SS1.SSS2.p3.1.m1.1.1" xref="A9.SS1.SSS2.p3.1.m1.1.1.cmml"><mi id="A9.SS1.SSS2.p3.1.m1.1.1.2" xref="A9.SS1.SSS2.p3.1.m1.1.1.2.cmml">x</mi><mo id="A9.SS1.SSS2.p3.1.m1.1.1.3" xref="A9.SS1.SSS2.p3.1.m1.1.1.3.cmml">+</mo></msup><annotation-xml encoding="MathML-Content" id="A9.SS1.SSS2.p3.1.m1.1b"><apply id="A9.SS1.SSS2.p3.1.m1.1.1.cmml" xref="A9.SS1.SSS2.p3.1.m1.1.1"><csymbol cd="ambiguous" id="A9.SS1.SSS2.p3.1.m1.1.1.1.cmml" xref="A9.SS1.SSS2.p3.1.m1.1.1">superscript</csymbol><ci id="A9.SS1.SSS2.p3.1.m1.1.1.2.cmml" xref="A9.SS1.SSS2.p3.1.m1.1.1.2">𝑥</ci><plus id="A9.SS1.SSS2.p3.1.m1.1.1.3.cmml" xref="A9.SS1.SSS2.p3.1.m1.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.SS1.SSS2.p3.1.m1.1c">x^{+}</annotation></semantics></math> and “positive” for <math id="A9.SS1.SSS2.p3.2.m2.1" class="ltx_Math" alttext="x^{-}" display="inline"><semantics id="A9.SS1.SSS2.p3.2.m2.1a"><msup id="A9.SS1.SSS2.p3.2.m2.1.1" xref="A9.SS1.SSS2.p3.2.m2.1.1.cmml"><mi id="A9.SS1.SSS2.p3.2.m2.1.1.2" xref="A9.SS1.SSS2.p3.2.m2.1.1.2.cmml">x</mi><mo id="A9.SS1.SSS2.p3.2.m2.1.1.3" xref="A9.SS1.SSS2.p3.2.m2.1.1.3.cmml">−</mo></msup><annotation-xml encoding="MathML-Content" id="A9.SS1.SSS2.p3.2.m2.1b"><apply id="A9.SS1.SSS2.p3.2.m2.1.1.cmml" xref="A9.SS1.SSS2.p3.2.m2.1.1"><csymbol cd="ambiguous" id="A9.SS1.SSS2.p3.2.m2.1.1.1.cmml" xref="A9.SS1.SSS2.p3.2.m2.1.1">superscript</csymbol><ci id="A9.SS1.SSS2.p3.2.m2.1.1.2.cmml" xref="A9.SS1.SSS2.p3.2.m2.1.1.2">𝑥</ci><minus id="A9.SS1.SSS2.p3.2.m2.1.1.3.cmml" xref="A9.SS1.SSS2.p3.2.m2.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.SS1.SSS2.p3.2.m2.1c">x^{-}</annotation></semantics></math>.</p>
</div>
</section>
<section id="A9.SS1.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">I.1.3 </span>BOOLQ</h4>

<div id="A9.SS1.SSS3.p1" class="ltx_para ltx_noindent">
<p id="A9.SS1.SSS3.p1.1" class="ltx_p">BOOLQ is a QA task where each example consists a yes/no question from <cite class="ltx_cite ltx_citemacro_citep">(Clark et al., <a href="#bib.bib11" title="" class="ltx_ref">2019</a>)</cite>. We directly use the 10 prompts from <cite class="ltx_cite ltx_citemacro_citep">(Sanh et al., <a href="#bib.bib44" title="" class="ltx_ref">2021</a>)</cite> (Page 146)</p>
</div>
</section>
<section id="A9.SS1.SSS4" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">I.1.4 </span>COPA</h4>

<div id="A9.SS1.SSS4.p1" class="ltx_para ltx_noindent">
<p id="A9.SS1.SSS4.p1.1" class="ltx_p">COPA is a causal reasoning task to determine either the cause or the effect of a given premise <cite class="ltx_cite ltx_citemacro_citep">(Roemmele et al., <a href="#bib.bib42" title="" class="ltx_ref">2011</a>)</cite>.
Here the label is a short sentence.
We use 10 prompts, where 9 are from <cite class="ltx_cite ltx_citemacro_citep">(Sanh et al., <a href="#bib.bib44" title="" class="ltx_ref">2021</a>)</cite> (Page 177), and we add one more prompt:</p>
</div>
<div id="A9.SS1.SSS4.p2" class="ltx_para ltx_noindent">
<ol id="A9.I3" class="ltx_enumerate">
<li id="A9.I3.ix1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1</span> 
<div id="A9.I3.ix1.p1" class="ltx_para ltx_noindent">
<p id="A9.I3.ix1.p1.1" class="ltx_p">[prefix]Consider the following premise: “‘ [premise] ”’ Choice 1: [choice1]
Choice 2: [choice2]
Q: Which one is more likely to be the [question], choice 1 or choice 2?
[label]</p>
</div>
</li>
</ol>
</div>
</section>
<section id="A9.SS1.SSS5" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">I.1.5 </span>DBpedia_14</h4>

<div id="A9.SS1.SSS5.p1" class="ltx_para ltx_noindent">
<p id="A9.SS1.SSS5.p1.1" class="ltx_p">DBpedia_14 is a topic classification dataset constructed by picking 14 non-overlapping classes from DBpedia 2014 <cite class="ltx_cite ltx_citemacro_citep">(Lehmann et al., <a href="#bib.bib23" title="" class="ltx_ref">2015</a>)</cite>. We manually create 8 prompts. For each example, we randomly select the incorrect label from the remaining 13 classes, and randomly assign the correct label and this incorrect label to “[label0]” and “[label1]”.</p>
</div>
<div id="A9.SS1.SSS5.p2" class="ltx_para ltx_noindent">
<ol id="A9.I4" class="ltx_enumerate">
<li id="A9.I4.ix1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1</span> 
<div id="A9.I4.ix1.p1" class="ltx_para ltx_noindent">
<p id="A9.I4.ix1.p1.1" class="ltx_p">[prefix]Consider the following example: “‘ [content] ”’
Choice 1: [label0]. Choice 2: [label1].Between choice 1 and choice 2, the topic of this example is [label]</p>
</div>
</li>
<li id="A9.I4.ix2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2</span> 
<div id="A9.I4.ix2.p1" class="ltx_para ltx_noindent">
<p id="A9.I4.ix2.p1.1" class="ltx_p">[prefix]Consider the following example: “‘ [content] ”’
Choice 1: [label0]. Choice 2: [label1].Between choice 1 and choice 2, what is the topic of this example?
[label]</p>
</div>
</li>
<li id="A9.I4.ix3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3</span> 
<div id="A9.I4.ix3.p1" class="ltx_para ltx_noindent">
<p id="A9.I4.ix3.p1.1" class="ltx_p">[prefix]Consider the following example: “‘ [content] ”’
Which is the topic of this example, choice 1: [label0], or choice 2: [label1]?
[label]</p>
</div>
</li>
<li id="A9.I4.ix4" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">4</span> 
<div id="A9.I4.ix4.p1" class="ltx_para ltx_noindent">
<p id="A9.I4.ix4.p1.1" class="ltx_p">[prefix][content]
Choice 1: [label0]. Choice 2: [label1].Between choice 1 and choice 2, the topic of this example is [label]</p>
</div>
</li>
<li id="A9.I4.ix5" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">5</span> 
<div id="A9.I4.ix5.p1" class="ltx_para ltx_noindent">
<p id="A9.I4.ix5.p1.1" class="ltx_p">[prefix][content]
Choice 1: [label0]. Choice 2: [label1].Between choice 1 and choice 2, what is the topic of this example?
[label]</p>
</div>
</li>
<li id="A9.I4.ix6" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">6</span> 
<div id="A9.I4.ix6.p1" class="ltx_para ltx_noindent">
<p id="A9.I4.ix6.p1.1" class="ltx_p">[prefix][content]
Which is the topic of this example, choice 1: [label0], or choice 2: [label1]?
[label]</p>
</div>
</li>
<li id="A9.I4.ix7" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">7</span> 
<div id="A9.I4.ix7.p1" class="ltx_para ltx_noindent">
<p id="A9.I4.ix7.p1.1" class="ltx_p">[prefix][content]
What category does the paragraph belong to, choice 1: [label0], or choice 2: [label1]?
[label]</p>
</div>
</li>
<li id="A9.I4.ix8" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">8</span> 
<div id="A9.I4.ix8.p1" class="ltx_para ltx_noindent">
<p id="A9.I4.ix8.p1.1" class="ltx_p">[prefix][content]
What label best describes this paragraph, choice 1: [label0], or choice 2: [label1]?
[label]</p>
</div>
</li>
</ol>
</div>
<div id="A9.SS1.SSS5.p3" class="ltx_para ltx_noindent">
<p id="A9.SS1.SSS5.p3.2" class="ltx_p">Here “[label]” is “choice 1” for <math id="A9.SS1.SSS5.p3.1.m1.1" class="ltx_Math" alttext="x^{+}" display="inline"><semantics id="A9.SS1.SSS5.p3.1.m1.1a"><msup id="A9.SS1.SSS5.p3.1.m1.1.1" xref="A9.SS1.SSS5.p3.1.m1.1.1.cmml"><mi id="A9.SS1.SSS5.p3.1.m1.1.1.2" xref="A9.SS1.SSS5.p3.1.m1.1.1.2.cmml">x</mi><mo id="A9.SS1.SSS5.p3.1.m1.1.1.3" xref="A9.SS1.SSS5.p3.1.m1.1.1.3.cmml">+</mo></msup><annotation-xml encoding="MathML-Content" id="A9.SS1.SSS5.p3.1.m1.1b"><apply id="A9.SS1.SSS5.p3.1.m1.1.1.cmml" xref="A9.SS1.SSS5.p3.1.m1.1.1"><csymbol cd="ambiguous" id="A9.SS1.SSS5.p3.1.m1.1.1.1.cmml" xref="A9.SS1.SSS5.p3.1.m1.1.1">superscript</csymbol><ci id="A9.SS1.SSS5.p3.1.m1.1.1.2.cmml" xref="A9.SS1.SSS5.p3.1.m1.1.1.2">𝑥</ci><plus id="A9.SS1.SSS5.p3.1.m1.1.1.3.cmml" xref="A9.SS1.SSS5.p3.1.m1.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.SS1.SSS5.p3.1.m1.1c">x^{+}</annotation></semantics></math> and “choice 2” for <math id="A9.SS1.SSS5.p3.2.m2.1" class="ltx_Math" alttext="x^{-}" display="inline"><semantics id="A9.SS1.SSS5.p3.2.m2.1a"><msup id="A9.SS1.SSS5.p3.2.m2.1.1" xref="A9.SS1.SSS5.p3.2.m2.1.1.cmml"><mi id="A9.SS1.SSS5.p3.2.m2.1.1.2" xref="A9.SS1.SSS5.p3.2.m2.1.1.2.cmml">x</mi><mo id="A9.SS1.SSS5.p3.2.m2.1.1.3" xref="A9.SS1.SSS5.p3.2.m2.1.1.3.cmml">−</mo></msup><annotation-xml encoding="MathML-Content" id="A9.SS1.SSS5.p3.2.m2.1b"><apply id="A9.SS1.SSS5.p3.2.m2.1.1.cmml" xref="A9.SS1.SSS5.p3.2.m2.1.1"><csymbol cd="ambiguous" id="A9.SS1.SSS5.p3.2.m2.1.1.1.cmml" xref="A9.SS1.SSS5.p3.2.m2.1.1">superscript</csymbol><ci id="A9.SS1.SSS5.p3.2.m2.1.1.2.cmml" xref="A9.SS1.SSS5.p3.2.m2.1.1.2">𝑥</ci><minus id="A9.SS1.SSS5.p3.2.m2.1.1.3.cmml" xref="A9.SS1.SSS5.p3.2.m2.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.SS1.SSS5.p3.2.m2.1c">x^{-}</annotation></semantics></math>.</p>
</div>
</section>
<section id="A9.SS1.SSS6" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">I.1.6 </span>IMDB</h4>

<div id="A9.SS1.SSS6.p1" class="ltx_para ltx_noindent">
<p id="A9.SS1.SSS6.p1.1" class="ltx_p">IMDB is a sentiment dataset from <cite class="ltx_cite ltx_citemacro_citep">(Maas et al., <a href="#bib.bib30" title="" class="ltx_ref">2011</a>)</cite>. Given a movie review, the label is either “[label0] = negative” or “[label1] = positive. We use 13 prompts, where 11 are from <cite class="ltx_cite ltx_citemacro_citep">(Sanh et al., <a href="#bib.bib44" title="" class="ltx_ref">2021</a>)</cite> (Page 168), and the rest two are as follows:</p>
</div>
<div id="A9.SS1.SSS6.p2" class="ltx_para ltx_noindent">
<ol id="A9.I5" class="ltx_enumerate">
<li id="A9.I5.ix1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1</span> 
<div id="A9.I5.ix1.p1" class="ltx_para ltx_noindent">
<p id="A9.I5.ix1.p1.1" class="ltx_p">[prefix]Consider the following example: ”’ [text] ”’
Between [label0] and [label1], the sentiment of this example is
[label]</p>
</div>
</li>
<li id="A9.I5.ix2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2</span> 
<div id="A9.I5.ix2.p1" class="ltx_para ltx_noindent">
<p id="A9.I5.ix2.p1.1" class="ltx_p">[prefix]Consider the following example: ”’ [text] ”’
Between [label0] and [label1], which is the sentiment of this example?
[label]</p>
</div>
</li>
</ol>
</div>
<div id="A9.SS1.SSS6.p3" class="ltx_para ltx_noindent">
<p id="A9.SS1.SSS6.p3.2" class="ltx_p">Here “[label]” is “negative” for <math id="A9.SS1.SSS6.p3.1.m1.1" class="ltx_Math" alttext="x^{+}" display="inline"><semantics id="A9.SS1.SSS6.p3.1.m1.1a"><msup id="A9.SS1.SSS6.p3.1.m1.1.1" xref="A9.SS1.SSS6.p3.1.m1.1.1.cmml"><mi id="A9.SS1.SSS6.p3.1.m1.1.1.2" xref="A9.SS1.SSS6.p3.1.m1.1.1.2.cmml">x</mi><mo id="A9.SS1.SSS6.p3.1.m1.1.1.3" xref="A9.SS1.SSS6.p3.1.m1.1.1.3.cmml">+</mo></msup><annotation-xml encoding="MathML-Content" id="A9.SS1.SSS6.p3.1.m1.1b"><apply id="A9.SS1.SSS6.p3.1.m1.1.1.cmml" xref="A9.SS1.SSS6.p3.1.m1.1.1"><csymbol cd="ambiguous" id="A9.SS1.SSS6.p3.1.m1.1.1.1.cmml" xref="A9.SS1.SSS6.p3.1.m1.1.1">superscript</csymbol><ci id="A9.SS1.SSS6.p3.1.m1.1.1.2.cmml" xref="A9.SS1.SSS6.p3.1.m1.1.1.2">𝑥</ci><plus id="A9.SS1.SSS6.p3.1.m1.1.1.3.cmml" xref="A9.SS1.SSS6.p3.1.m1.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.SS1.SSS6.p3.1.m1.1c">x^{+}</annotation></semantics></math> and “positive” for <math id="A9.SS1.SSS6.p3.2.m2.1" class="ltx_Math" alttext="x^{-}" display="inline"><semantics id="A9.SS1.SSS6.p3.2.m2.1a"><msup id="A9.SS1.SSS6.p3.2.m2.1.1" xref="A9.SS1.SSS6.p3.2.m2.1.1.cmml"><mi id="A9.SS1.SSS6.p3.2.m2.1.1.2" xref="A9.SS1.SSS6.p3.2.m2.1.1.2.cmml">x</mi><mo id="A9.SS1.SSS6.p3.2.m2.1.1.3" xref="A9.SS1.SSS6.p3.2.m2.1.1.3.cmml">−</mo></msup><annotation-xml encoding="MathML-Content" id="A9.SS1.SSS6.p3.2.m2.1b"><apply id="A9.SS1.SSS6.p3.2.m2.1.1.cmml" xref="A9.SS1.SSS6.p3.2.m2.1.1"><csymbol cd="ambiguous" id="A9.SS1.SSS6.p3.2.m2.1.1.1.cmml" xref="A9.SS1.SSS6.p3.2.m2.1.1">superscript</csymbol><ci id="A9.SS1.SSS6.p3.2.m2.1.1.2.cmml" xref="A9.SS1.SSS6.p3.2.m2.1.1.2">𝑥</ci><minus id="A9.SS1.SSS6.p3.2.m2.1.1.3.cmml" xref="A9.SS1.SSS6.p3.2.m2.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.SS1.SSS6.p3.2.m2.1c">x^{-}</annotation></semantics></math>.</p>
</div>
</section>
<section id="A9.SS1.SSS7" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">I.1.7 </span>PIQA</h4>

<div id="A9.SS1.SSS7.p1" class="ltx_para ltx_noindent">
<p id="A9.SS1.SSS7.p1.1" class="ltx_p">The PIQA dataset measures the physical commonsense reasoning ability of models. We use 11 prompts for PIQA, all of which are from <cite class="ltx_cite ltx_citemacro_citep">(Sanh et al., <a href="#bib.bib44" title="" class="ltx_ref">2021</a>)</cite>(Page 160).
The label is a complete sentence that can be the solution of the question.</p>
</div>
</section>
<section id="A9.SS1.SSS8" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">I.1.8 </span>QNLI</h4>

<div id="A9.SS1.SSS8.p1" class="ltx_para ltx_noindent">
<p id="A9.SS1.SSS8.p1.1" class="ltx_p">QNLI from <cite class="ltx_cite ltx_citemacro_citep">(Rajpurkar et al., <a href="#bib.bib41" title="" class="ltx_ref">2016</a>)</cite> is a question-answering dataset consisting of question-paragraph pairs, where one of the sentences in the paragraph (drawn from Wikipedia) contains the answer to the corresponding question (written by an annotator).
We use 5 prompts from <cite class="ltx_cite ltx_citemacro_citep">(Sanh et al., <a href="#bib.bib44" title="" class="ltx_ref">2021</a>)</cite>.
The label is either “yes” or “no” depending on whether the information in the paragraph is enough to answer the paragraph.</p>
</div>
</section>
<section id="A9.SS1.SSS9" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">I.1.9 </span>RTE</h4>

<div id="A9.SS1.SSS9.p1" class="ltx_para ltx_noindent">
<p id="A9.SS1.SSS9.p1.1" class="ltx_p">RTE is a textual entailment dataset <cite class="ltx_cite ltx_citemacro_citep">(Wang et al., <a href="#bib.bib47" title="" class="ltx_ref">2018</a>)</cite>.
The label corresponds to whether the text entails the hypotheses.
We use 11 prompts, where 10 are from <cite class="ltx_cite ltx_citemacro_citep">(Sanh et al., <a href="#bib.bib44" title="" class="ltx_ref">2021</a>)</cite>(Page 48), and where we manually add one more prompt:</p>
</div>
<div id="A9.SS1.SSS9.p2" class="ltx_para ltx_noindent">
<ol id="A9.I6" class="ltx_enumerate">
<li id="A9.I6.ix1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1</span> 
<div id="A9.I6.ix1.p1" class="ltx_para ltx_noindent">
<p id="A9.I6.ix1.p1.1" class="ltx_p">[prefix][premise]
Question: Does this imply that ”[hypothesis]”, yes or no?
[label]</p>
</div>
</li>
</ol>
</div>
<div id="A9.SS1.SSS9.p3" class="ltx_para ltx_noindent">
<p id="A9.SS1.SSS9.p3.2" class="ltx_p">Here “[label]” is “yes” for <math id="A9.SS1.SSS9.p3.1.m1.1" class="ltx_Math" alttext="x^{+}" display="inline"><semantics id="A9.SS1.SSS9.p3.1.m1.1a"><msup id="A9.SS1.SSS9.p3.1.m1.1.1" xref="A9.SS1.SSS9.p3.1.m1.1.1.cmml"><mi id="A9.SS1.SSS9.p3.1.m1.1.1.2" xref="A9.SS1.SSS9.p3.1.m1.1.1.2.cmml">x</mi><mo id="A9.SS1.SSS9.p3.1.m1.1.1.3" xref="A9.SS1.SSS9.p3.1.m1.1.1.3.cmml">+</mo></msup><annotation-xml encoding="MathML-Content" id="A9.SS1.SSS9.p3.1.m1.1b"><apply id="A9.SS1.SSS9.p3.1.m1.1.1.cmml" xref="A9.SS1.SSS9.p3.1.m1.1.1"><csymbol cd="ambiguous" id="A9.SS1.SSS9.p3.1.m1.1.1.1.cmml" xref="A9.SS1.SSS9.p3.1.m1.1.1">superscript</csymbol><ci id="A9.SS1.SSS9.p3.1.m1.1.1.2.cmml" xref="A9.SS1.SSS9.p3.1.m1.1.1.2">𝑥</ci><plus id="A9.SS1.SSS9.p3.1.m1.1.1.3.cmml" xref="A9.SS1.SSS9.p3.1.m1.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.SS1.SSS9.p3.1.m1.1c">x^{+}</annotation></semantics></math> and “no” for <math id="A9.SS1.SSS9.p3.2.m2.1" class="ltx_Math" alttext="x^{-}" display="inline"><semantics id="A9.SS1.SSS9.p3.2.m2.1a"><msup id="A9.SS1.SSS9.p3.2.m2.1.1" xref="A9.SS1.SSS9.p3.2.m2.1.1.cmml"><mi id="A9.SS1.SSS9.p3.2.m2.1.1.2" xref="A9.SS1.SSS9.p3.2.m2.1.1.2.cmml">x</mi><mo id="A9.SS1.SSS9.p3.2.m2.1.1.3" xref="A9.SS1.SSS9.p3.2.m2.1.1.3.cmml">−</mo></msup><annotation-xml encoding="MathML-Content" id="A9.SS1.SSS9.p3.2.m2.1b"><apply id="A9.SS1.SSS9.p3.2.m2.1.1.cmml" xref="A9.SS1.SSS9.p3.2.m2.1.1"><csymbol cd="ambiguous" id="A9.SS1.SSS9.p3.2.m2.1.1.1.cmml" xref="A9.SS1.SSS9.p3.2.m2.1.1">superscript</csymbol><ci id="A9.SS1.SSS9.p3.2.m2.1.1.2.cmml" xref="A9.SS1.SSS9.p3.2.m2.1.1.2">𝑥</ci><minus id="A9.SS1.SSS9.p3.2.m2.1.1.3.cmml" xref="A9.SS1.SSS9.p3.2.m2.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.SS1.SSS9.p3.2.m2.1c">x^{-}</annotation></semantics></math>.</p>
</div>
</section>
<section id="A9.SS1.SSS10" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">I.1.10 </span>Story_Cloze</h4>

<div id="A9.SS1.SSS10.p1" class="ltx_para ltx_noindent">
<p id="A9.SS1.SSS10.p1.1" class="ltx_p">Story Cloze is a story completion task from <cite class="ltx_cite ltx_citemacro_citep">(Mostafazadeh et al., <a href="#bib.bib36" title="" class="ltx_ref">2017</a>)</cite>.
Given a short story, the task is to determine which of two endings is more likely to continue that story.
We use 9 prompts, where 6 are from <cite class="ltx_cite ltx_citemacro_citep">(Sanh et al., <a href="#bib.bib44" title="" class="ltx_ref">2021</a>)</cite>, and the remaining 3 are as follows:</p>
</div>
<div id="A9.SS1.SSS10.p2" class="ltx_para ltx_noindent">
<ol id="A9.I7" class="ltx_enumerate">
<li id="A9.I7.ix1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1</span> 
<div id="A9.I7.ix1.p1" class="ltx_para ltx_noindent">
<p id="A9.I7.ix1.p1.1" class="ltx_p">[prefix]Consider the following story: “‘ [input_sentence_1] [input_sentence_2] [input_sentence_3] [input_sentence_4] ”’
Choice 1: [sentence_quiz1]
Choice 2: [sentence_quiz2]
Which is the more plausible ending of this story, choice 1 or choice 2?
[label]</p>
</div>
</li>
<li id="A9.I7.ix2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2</span> 
<div id="A9.I7.ix2.p1" class="ltx_para ltx_noindent">
<p id="A9.I7.ix2.p1.1" class="ltx_p">[prefix]Consider the following story: “‘ [input_sentence_1] [input_sentence_2] [input_sentence_3] [input_sentence_4] ”’
Choice 1: [sentence_quiz1]
Choice 2: [sentence_quiz2]
Which is the more plausible ending of this story?
[label]</p>
</div>
</li>
<li id="A9.I7.ix3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3</span> 
<div id="A9.I7.ix3.p1" class="ltx_para ltx_noindent">
<p id="A9.I7.ix3.p1.1" class="ltx_p">[prefix][input_sentence_1] [input_sentence_2] [input_sentence_3] [input_sentence_4]
Choice 1: [sentence_quiz1]
Choice 2: [sentence_quiz2]
Which is the more plausible ending of this story, choice 1 or choice 2?
[label]</p>
</div>
</li>
</ol>
<p id="A9.SS1.SSS10.p2.2" class="ltx_p">Here “[label]” is “choice 1” for <math id="A9.SS1.SSS10.p2.1.m1.1" class="ltx_Math" alttext="x^{+}" display="inline"><semantics id="A9.SS1.SSS10.p2.1.m1.1a"><msup id="A9.SS1.SSS10.p2.1.m1.1.1" xref="A9.SS1.SSS10.p2.1.m1.1.1.cmml"><mi id="A9.SS1.SSS10.p2.1.m1.1.1.2" xref="A9.SS1.SSS10.p2.1.m1.1.1.2.cmml">x</mi><mo id="A9.SS1.SSS10.p2.1.m1.1.1.3" xref="A9.SS1.SSS10.p2.1.m1.1.1.3.cmml">+</mo></msup><annotation-xml encoding="MathML-Content" id="A9.SS1.SSS10.p2.1.m1.1b"><apply id="A9.SS1.SSS10.p2.1.m1.1.1.cmml" xref="A9.SS1.SSS10.p2.1.m1.1.1"><csymbol cd="ambiguous" id="A9.SS1.SSS10.p2.1.m1.1.1.1.cmml" xref="A9.SS1.SSS10.p2.1.m1.1.1">superscript</csymbol><ci id="A9.SS1.SSS10.p2.1.m1.1.1.2.cmml" xref="A9.SS1.SSS10.p2.1.m1.1.1.2">𝑥</ci><plus id="A9.SS1.SSS10.p2.1.m1.1.1.3.cmml" xref="A9.SS1.SSS10.p2.1.m1.1.1.3"></plus></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.SS1.SSS10.p2.1.m1.1c">x^{+}</annotation></semantics></math> and “choice 2” for <math id="A9.SS1.SSS10.p2.2.m2.1" class="ltx_Math" alttext="x^{-}" display="inline"><semantics id="A9.SS1.SSS10.p2.2.m2.1a"><msup id="A9.SS1.SSS10.p2.2.m2.1.1" xref="A9.SS1.SSS10.p2.2.m2.1.1.cmml"><mi id="A9.SS1.SSS10.p2.2.m2.1.1.2" xref="A9.SS1.SSS10.p2.2.m2.1.1.2.cmml">x</mi><mo id="A9.SS1.SSS10.p2.2.m2.1.1.3" xref="A9.SS1.SSS10.p2.2.m2.1.1.3.cmml">−</mo></msup><annotation-xml encoding="MathML-Content" id="A9.SS1.SSS10.p2.2.m2.1b"><apply id="A9.SS1.SSS10.p2.2.m2.1.1.cmml" xref="A9.SS1.SSS10.p2.2.m2.1.1"><csymbol cd="ambiguous" id="A9.SS1.SSS10.p2.2.m2.1.1.1.cmml" xref="A9.SS1.SSS10.p2.2.m2.1.1">superscript</csymbol><ci id="A9.SS1.SSS10.p2.2.m2.1.1.2.cmml" xref="A9.SS1.SSS10.p2.2.m2.1.1.2">𝑥</ci><minus id="A9.SS1.SSS10.p2.2.m2.1.1.3.cmml" xref="A9.SS1.SSS10.p2.2.m2.1.1.3"></minus></apply></annotation-xml><annotation encoding="application/x-tex" id="A9.SS1.SSS10.p2.2.m2.1c">x^{-}</annotation></semantics></math>.</p>
</div>
</section>
</section>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2212.03826" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2212.03827" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2212.03827">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2212.03827" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2212.03828" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Fri Mar  1 12:37:30 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2212.03827
Existing techniques for training language models can be misaligned with the truth:
if we train models with imitation learning, they may reproduce errors that humans make; if we train them to generate text that humans rate highly, they may output errors that human evaluators can’t detect.
We propose circumventing this issue by directly finding latent knowledge inside the internal activations of a language model in a purely unsupervised way.
Specifically, we introduce a method for accurately answering yes-no questions given only unlabeled model activations.
It works by finding a direction in activation space that satisfies logical consistency properties, such as that a statement and its negation have opposite truth values.
We show that despite using no supervision and no model outputs, our method can recover diverse knowledge represented in large language models: across 6 models and 10 question-answering datasets, it outperforms zero-shot accuracy by 4% on average.
We also find that it cuts prompt sensitivity in half and continues to maintain high accuracy even when models are prompted to generate incorrect answers.
Our results provide an initial step toward discovering what language models know, distinct from what they say, even when we don’t have access to explicit ground truth labels.
abstract
Discovering Latent Knowledge in Language Models Without Supervision
Abstract
2212.03827
1IntroductionThe increasing deployment of language models in real-world applications opens up exciting possibilities, but it also raises the stakes of AI research and presents new risks.
One of these risks is that language models do not always output text that is true.Common training objectives can cause models to learn internal representations related to truth, since truth is a useful feature for many tasks.
However, these objectives can also cause language models to output text that is false, at least in some circumstances.
For example, if we train a model to imitate human-generated text, it may learn to output common misconceptions.
Or if we train a chat bot to optimize a reward such as engagement, it may learn to generate text that is compelling but false.
If we try to reward model outputs that look true, a model may still learn to output false text if human raters can’t evaluate the correctness of that text.In each case, this is an issue that stems from the misalignment between a training objective and the truth.
As models are applied to more complex domains, human supervision may become less effective at mitigating this misalignment.
Moreover, because this is a problem with the training objective rather than a model’s capabilities, it likely won’t be solved by scaling up models alone.Figure 1:An illustration of our method, Contrast-Consistent Search (CCS) . For each yes-no question, we letandbe the natural language statements where we answeras “Yes” and “No” respectively.
Answering the questionthen amounts to determining which oforis true.
We compute probabilitiesandthatandare true respectively using a learned mapping from the hidden states to a number betweenand.
We search for a mapping such that that the probabilities are both confident and logically consistent.
On the right, we show a histogram of the “Yes” probabilities,, learned by our method for the COPA datasetwith the UnifiedQA model.
Our method is purely unsupervised, only leveraging the consistency of truth, but still learns to accurately answers questions.We propose a different approach for addressing this misalignment: using models to answer questions in a purelyunsupervisedway.
Intuitively, instead of trying to explicitly, externally specify truth, we search for implicit, internal “beliefs” or “knowledge” learned by a model.
We approach this problem by leveraging the fact that a model’s representation of truth must satisfy logical consistency properties, which are unlikely to be satisfied by many other features.We implement this idea by introducing Contrast-Consistent Search (CCS) , a method that learns a linear projection of the hidden states that is consistent across negations, as illustrated in Figure1.
We find that despite its simplicity and despite not having access to any labels or model outputs, CCS can accurately recover knowledge from model representations: evaluated across 6 models and 10 question-answering datasets, CCS outperforms the accuracy of strong zero-shot baselines by 4% on average (Section3.2.1) .
The resulting classifier is also less sensitive to different prompts than zero-shot, cutting the standard deviation in accuracy in half.
Additionally, we try deliberately prompting models to make incorrect outputs, which should intuitively change what models say but which shouldn’t affect their latent knowledge.
We find that this causes zero-shot accuracy to drop by up to 9.5% (Section3.2.2)  without decreasing the accuracy of CCS.We systematically analyze CCS to understand the features it discovers.
We show that it transfers across unrelated tasks, suggesting that models may have a task-agnostic representation of the truth and that CCS is able to approximately discover it (Section3.3.1) .
Moreover, CCS sometimes works best using the hidden states in themiddlelayers of a network and can work even when model outputs aren’t very informative, suggesting that it can leverage different features from those used by the outputs (Section3.3.2) .
Finally, we show that representations of truth tend to be salient in models: they can often be found without much data, and they can often be found by taking the top principal component of a slightly modified representation space (Section3.3.3) .Most existing techniques for making models truthful use human supervision to explicitly specify what is correct.
However, it is not feasible to provide supervision in some settings.
Our work suggests that an external source of ground truth may not actually be necessary: we may instead be able to find a model’s latent representation of truth, independent of what a model says, without using any supervision in the first place.
introduction
Discovering Latent Knowledge in Language Models Without Supervision
1Introduction
2212.03827
2Problem Statement and FrameworkIn this section we describe our problem setup in more detail and introduce Contrast-Consistent Search (CCS) , a method for discovering latent knowledge in language models without supervision.2.1Problem: Discovering Latent KnowledgeGiven a pre-trained neural language model and a setof yes-no questions111Technically, we only require that there are two mutually exclusive answers.
For example, we can also use the labels “positive” and “negative” for sentiment classification.
Moreover, our setup can easily extend to the case where we want to evaluate the truth of a set of statements instead of answering a set of questions., our goal is to answer eachcorrectly.
Here,can be any question with a well-defined answer, including procedural questions like “Is 22+59 = 237?”, for which the answer is “No”, and factual questions like “Are cats mammals?”, for which the answer is “Yes”.Importantly, we want methods that do not rely on the model generating correct outputs and that do not rely on external supervision.
Instead, we turn to the model’s unlabeled hidden representations.
Specifically, letdenote some feature representation on a natural langauge input, such as the hidden states of a Transformer-based language model.
Our goal is to answer the questionsonly given access to.
InSection2.2we introduce a method for this problem that attains high accuracy (Section3) , demonstrating that this task is tractable.2.2Method: Contrast-Consistent SearchTo make progress on the goal described above, we exploit the fact that truth has special structure: it satisfies consistency properties that few other features in a language model are likely to satisfy.
Our method, Contrast-Consistent Search (CCS) , leverages this idea by finding a direction in activation space that is consistent across negations.
As we illustrate inFigure1, CCS works by (1)  answering each questionas both “Yes” ()  and “No” () , (2)  computing the representationsandof each answer, (3)  mapping the answer representations to probabilitiesandof being true, then (4)  optimizing that mapping so that the probabilities are both consistent and confident.Concretely, the input to CCS is a set of Yes-No questions,, and access to a pretrained model’s representations,; the output of CCS is a lightweight probe on top ofthat can answer new questions.
Here,is fixed but should contain useful information about the answers to, in the sense that if onedid(hypothetically)  have access to the ground-truth labels for, one would be able to train a small supervised probe onthat attains high accuracy.
Importantly, CCS does not modify the weights of the pretrained model and it does not use labels.Constructing contrast pairs.An important property that truth satisfies is negation consistency: the answer to a clear-cut question cannot be both “Yes” and “No” at the same time, as these are negations of each other.
Probabilistically, for each question, the probability that the answer tois “Yes” should be one minus the probability that the answer tois “No”.
To use this property, we begin by constructing contrast pairs: for each question, we answerboth as “Yes”, resulting in the new natural language statement, and as “No”, resulting in the natural language statement.
We illustrate this inFigure1(left) .
We will then learn to classifyandas true or false; ifis true, then the answer toshould be “Yes”, and ifis true, then the answer toshould be “No”.In practice, we convert each task into a question-answering task with two possible labels, then we use task-specific zero-shot prompts to format questions and answers as strings to construct each contrast pair.
The opposite labels we use to construct contrast pairs can be “Yes” and “No” for a generic task, or they can be other tasks-specific labels, such as “Positive” and “Negative” in the case of sentiment classification.
We describe the exact prompts we use to for each task inAppendixB.Feature extraction and normalization.Given a contrast pair, CCS first computes the representationsandusing the feature extractor.
Intuitively, there are two salient differences betweenand: (1) ends with “Yes” whileends with “No”, and (2)  one oforis true while the other is false.
We want to find (2)  rather than (1) , so we first try to remove the effect of (1)  by normalizingandindependently.
In particular, we construct normalized representationsas follows:whereandare the means and standard deviations ofandrespectively, and where all operations are element-wise along each dimension.222Mean-normalization is important for CCS to work at all, while variance-normalization is less essential.This normalization ensures thatandno longer form two separate clusters.Mapping activations to probabilities.Next, we learn a probethat maps a (normalized)  hidden stateto a number betweenandrepresenting the probability that the statementis true. We use a linear projection followed by a sigmoid, i.e., but nonlinear projections can also work.
For simplicity, we sometimes omit thesubscript in.Training objective.To find features that represent the truth, we leverage the consistency structure of truth. First, we use the fact that a statement and its negation should have probabilities that add up to. This motivates the consistency loss:However, this objective alone has a degenerate solution:. To avoid this problem, we encourage the model to also be confident with the following confidence loss:We can equivalently interpretas imposing a second consistency property on the probabilities: the law of excluded middle (every statement must be either true or false) .
The final unsupervised loss is the sum of these two losses, averaged across all contrast pairs:Note that both losses are necessary;alone also has a degenerate solution.Inference.Bothandshould represent the probability that the answer tois “Yes”. However, because we use a soft consistency constraint, these may not be exactly equal. To make a prediction on an exampleafter training, we consequently take the average of these:We then predict that the answer tois “Yes” based on whetheris greater than.
Technically, we also need to determine whethercorresponds to “Yes” or “No,” as this isn’t specified by.
For simplicity in our evaluations we take the maximum accuracy over the two possible ways of labeling the predictions of a given test set.
However, inAppendixAwe describe how one can identify the two clusters without any supervision in principle by leveraging conjunctions.
body
Discovering Latent Knowledge in Language Models Without Supervision
2Problem Statement and Framework
2212.03827
3Results3.1Experimental SetupHere we give an overview of our experimental setup; seeAppendixGfor full details.
We provide code athttps://www.github.com/collin-burns/discovering_latent_knowledge.Models.We test six models: encoder-decoder models (T5, UnifiedQA, T0) , autoregressive models (GPT-J) , and encoder-only models (RoBERTa, DeBERTa) .Data.We test models ondatasets: sentiment classification (IMDBand Amazon) , topic classification (AG-Newsand DBpedia-14) , NLI (RTEand QNLI) , story completion (COPAand Story-Cloze) , question answering (BoolQ) , and common sense reasoning (PIQA) .We convert each dataset to a yes-no question-answering task or a binary classification task, as described inAppendixG.
We balance the labels and randomly subsample 1000 examples from each dataset (except for COPA, which has only 500 examples total) , then randomly split each dataset into an unsupervised training set (of the data)  and test set () .
We subsample each dataset for computational efficiency reasons; because we aggregate over  9 prompts per dataset, 10 datasets, and 6 models, 1000 datapoints per dataset actually corresponds to approximately 180k examples in total.Methods.We test four main methods: zero-shot, calibrated zero-shot, Contrast-Consistent Search (CCS) , and Logistic Regression (LR) .
Zero-shot works by predicting the answer with the highest log probability according to the language model, averaged across the tokens that make up that label.
Calibrated zero-shot works by balancing zero-shot predictions to befor each answer, as we describe in more detail below, similar to.
For Logistic Regression we train on the training split for each dataset usingas the covariates, then evaluate on the corresponding test split.
We treat LR as a ceiling since it uses labeled data.When testing CCS, we optimize it 10 times using AdamWwith learning rate, then take the run with the lowest unsupervised loss. Unless otherwise specified, we train CCS using all prompts for a single training set, then evaluate it on the corresponding test split.Zero-shot baselines.Zero-shot outputs sometimes suffer from miscalibration, in which models are biased towards predicting specific answers.
Calibrating the outputs to be uniform over different answers can mitigate this problem.
We use a variant of the calibration method presented inby balancing predictions to be 50/50 across the two output labels. Specifically, ifandare the logits for the positive and negative label respectively, then instead of classifying an example as positive if, we classify it as positive if, where we select the thresholdso that the predictions are balanced.
We find this increases accuracy by abouton average.
Unless otherwise specified, we always report zero-shot accuracy after calibration.Encoder-only models (e.g. RoBERTa and DeBERTa)  cannot be easily used to do zero-shot classification out of the box, so to evaluate them we follow the method of: we finetune both models on an NLI dataset (MNLI, which we do not evaluate on)  and treat the difference between the entailment and contradiction probabilities as the effective logit. This provides a strong zero-shot baseline for encoder-only models that works even for non-NLI tasks. This finetuning isn’t necessary for CCS to work on encoder-only models (seeAppendixC) , but we test CCS using the same MNLI-finetuned models for ease of comparison.Hidden states.We extract the hidden states corresponding to the last token in the last layer of each model for simplicity, unless otherwise specified.
For encoder-decoder models, we evaluate CCS on the last layer hidden states of both the encoder and decoder, and use whichever one generally achieves a lower unsupervised loss; for T0 this is the decoder hidden states, while for T5 and UnifiedQA this is the encoder hidden states.
SeeSectionG.1for further implementation details, such as tokenization.Prompts.To reduce prompt sensitivity, we use betweenandprompts for each dataset (on average) , derived or slightly modified from.
Unless otherwise specified, we average across all prompts when showing results.
To construct contrast pairs, we letbe the zero-shot prompt usingand the first label (e.g. “Positive” for sentiment classification datasets)  and letbe the prompt usingand the second label (e.g. “Negative”) .
We describe all prompts inAppendixI.MethodRoBERTaDeBERTaGPT-JT5UQAT0∗Mean∗0-shot60.1(5.7) 68.6(8.2) 53.2(5.2) 55.4(5.7) 76.8(9.6) 87.9(4.8) 62.8(6.9) Calibrated 0-shot64.3(6.2) 76.3(6.0) 56.0(5.2) 58.8(6.1) 80.4(7.1) 90.5(2.7) 67.2(6.1) CCS62.1(4.1) 78.5(3.8) 61.7(2.5) 71.5(3.0) 82.1(2.7) 77.6(3.3) 71.2(3.2) CCS (All Data) 60.1(3.7) 77.1(4.1) 62.1(2.3) 72.7(6.0) 84.8(2.6) 84.8(3.7) 71.5(3.7) LR (Ceiling) 79.8(2.5) 86.1(2.2) 78.0(2.3) 84.6(3.1) 89.8(1.9) 90.7(2.1) 83.7(2.4) Table 1:Accuracy of each method and model averaged across all prompts and dataset, with the average standard deviation of accuracy across different prompts shown in parentheses.
For most models, CCS outperforms zero-shot accuracy and exhibits lower sensitivity to prompts, even though this was not our goal.
This shows that we can recover knowledge from language model activations without supervision, and can do so in a way that is competitive with strong baseline methods that use model outputs.∗T0 was trained on 9 out of 10 of the datasets we evaluate on, including some of the data in our test splits, so we ignore it when averaging over models.3.2Evaluating CCS3.2.1CCS Outperforms Zero-ShotWe evaluate CCS on all 6 models and compute the average accuracy across all datasets and prompts.
T0 was trained on 9 out of 10 of the datasets we evaluate on, including some of the data in our test splits, so we ignore it when averaging over models to avoid unfair comparisons.
We display the results inTable1.
To assess prompt sensitivity, for each model and dataset we compute the standard deviation (s.d.)  of accuracy across different prompts, then average the resulting standard deviations across all datasets, which we show in parentheses inTable1.
For comparison, we also include results when training CCS on all datasets simultaneously, which we refer to as CCS (All Data) .CCS attains an accuracy ofon average, compared tofor calibrated zero-shot.
It outperforms zero-shot accuracy for every model, except for RoBERTa (where it doesworse)  and T0 (for which zero-shot accuracy is inflated) .
Training on all datasets improves accuracy by only an insignificant amount on average () , but with large gains for T0 in particular () .These results show that CCS canexceedthe performance of strong baseline methods that access the model outputs, even though this wasn’t our main goal.
This indicates that it is indeed possible to classify examples with high accuracy using only unlabeled model representations.3.2.2CCS Is Robust To Misleading PromptsRecall our goal: to discover latent knowledge in a language model even when the model outputs false text.
In particular, language models are typically trained to imitate text whether or not it is correct, so if a model sees false text it should intuitively be more likely to predict that subsequent text will also be false.
Based on this idea, we provide an initial proof of concept that CCS can make progress toward our goal by constructing prompts that aim to mislead the outputs of language models.
Specifically, we add a prefix to the beginning of our zero-shot prompts that consists of questions answered incorrectly (Figure5) .
The hope is that such a prefix will decrease zero-shot accuracy because the model will imitate its context and answer subsequent questions incorrectly even if it internally “knows” better.333In practice we found that model behavior was qualitatively similar between using this prefix with correct and incorrect answers, in agreement with the findings ofthat using incorrect labels in demonstrations often does not significantly degrade the performance of a few-shot prompt. Consequently, the prefix may instead actually be reducing accuracy because it is out-of-distribution. See alsofor more experiments on the subtleties of correct vs incorrect labels for few-shot prompts.We found that while most models are robust to this type of prefix (seeAppendixB) , it significantly drops calibrated zero-shot performance in UnifiedQA, decreasing accuracy fromto.We evaluate CCS on these examples and show the results inFigure4of the Appendix.
We find that despite thedrop in zero-shot accuracy, CCS maintains high accuracy () .
This provides evidence that our method can still work well even when model outputs are unreliable.3.3Analyzing CCSWe have shown that CCS attains strong classification performance in standard settings and when we deliberately mislead models.
Moreover, we have described our motivation as discovering latent representations of truth in language models, but in practice CCS just finds a direction in representation space that attains high accuracy.
This raises the question: in what sense is CCS actually finding “truth” features?
We now provide a preliminary investigation of this question.3.3.1CCS Finds A Task-Agnostic Representation of TruthFrom the results described so far, it may be possible that the classifier we find is capturing dataset-specific properties, such as which of two labels (e.g. “Yes” vs. “No”)  is more likely to be correct.
We rule this out by showing that it generalizes across completely different tasks, including ones with different label spaces (such as from “Yes” and “No” for a generic task to “Positive” and “Negative” for sentiment classification) .
In particular, we train and test CCS on every pair of datasets, and show the resulting transfer for several models inFigure2.
(SeeAppendixFfor results with other models.) We find that CCS indeed transfer wells: in the majority of datasets, the transfer accuracy of CCS is competitive with training and testing on the same dataset (the last row inFigure2, “No transfer”) .
Transfer performance can even outperform the no-transfer setting, especially when we train on simpler tasks, such as sentiment classification.
For example, training CCS on the Amazon sentiment dataset achieves an average transfer accuracy of, which ishigher than CCS without transfer.
We speculate that this performs so well because the difference in representations between correct and incorrect answers is especially pronounced for easier tasks like Amazon.Additionally, transfer accuracy tends to be similar for many datasets.
For example, training on IMDB (row 1 ofFigure2)  has similar accuracy as training on DBPedia (row 4) .
This provides evidence that CCS can find a functionally similar direction across many different types of training datasets.
Overall, these results suggest that (1)  models may have a task-agnostic representation related to what is true, and that (2)  CCS may approximately find this representation even without diverse data.Figure 2:Transfer Performance using CCS on UnifiedQA, T0 and DeBERTa.
The y-axis corresponds to the training dataset, and the x-axis corresponds to the test dataset.
The final row (“No Transfer”)  corresponds to training and testing on the same dataset, which is the same as the diagonal.
On most datasets, CCS transfers well to other datasets (relative to no transfer) , including to different tasks with completely different labels.
In some cases transfer even outperforms the no-transfer setting.
SeeAppendixEfor results with other models.3.3.2CCS Does Not Just Recover Model OutputsOne possibility is that CCS can only recover knowledge already contained in a model’s outputs.
We have shown that CCS can outperform zero-shot accuracy (Section3.2.1) , especially when model outputs are misled (Section3.2.2) , which already provides evidence against this possibility.
We now provide additional evidence against this concern.First, if CCS were just recovering knowledge in the model outputs, using the last layer of a network (right before the outputs)  should presumably outperform intermediate layers (which are more causally distant from the outputs) . However, for T5 and UnifiedQA, we find that using hidden states in the middle of the network outperform hidden states at the end of the network when using CCS (seeFigure10) . This is especially true for UnifiedQA on misleading prefixes; we find that using the encoder hidden states is robust to misleading prefixes (Section3.2.2) , but that accuracy using the decoder hidden states drops fromto, a similar amount to zero-shot accuracy.
This suggests that compared to the later layers of a model, intermediate layers are more robust and less correlated with the model outputs, and that CCS can take advantage of this.Finally, if CCS were just recovering knowledge in the model outputs, we would only expect it to work in cases where model outputs are informative. However, we show inAppendixCthat CCS still works with masked language models when their outputs are uninformative: when we don’t [MASK] any input tokens, and when we prompt models so that the labels used to construct contrast pairs appear in themiddleof a prompt rather than at the end.
These results show that CCS can sometimes recover latent knowledge in a model that is distinct from—and more useful than—what the model outputs.3.3.3Truth is a Salient FeatureFrom the results we have presented so far, it is possible that the direction learned by CCS is difficult to find and requires using a large amount of unsupervised data.
We provide evidence against this possibility by showing that finding such a direction can both (1)  often be done with only a small amount of data, and can also (2)  often be done by essentially taking the top principal component of a slightly modified representation space.CCS doesn’t require much data.We now evaluate how well CCS performs with different amounts of data.
Whereas before we trained CCS using the full training set and all prompts, here we use limited data and a single prompt.
Specifically, we train using onlyunlabeled contrast pairs, using the single prompt for each model and dataset that achieves the highest zero-shot accuracy. We still test on all prompts for each dataset.
We resamplepointstimes for each of, and take the average accuracy across thosesamples.
Finally, we plot the average such accuracy across all datasets and prompts for several models inFigure3.We find that while CCS benefits from more data, it can often do well with very limited data.
In fact, it can sometimes even do well with only a single contrast pair, though we find high variance across individual datasets; seeAppendixDfor more details.
This suggests that the strong performance of CCS does not primarily come from using a large amount of unsupervised data, and indicates that the direction learned by CCS may be relatively easy to find.Figure 3:Accuracy when we train CCS onsamples for different values of(each time averaged across 32 trials) . We use the single prompt with the highest zero-shot accuracy for each dataset and model. While CCS benefits from more examples, it can often work well with limited data.Contrastive Representation Clustering.We now show that directions correlated with the truth may be “salient” in a different way: by showing that we can also find such directions using either (1)  PCA or (2)  clustering.
Specifically, suppose we construct contrast pairs (,)  as before.
Intuitively, these two examples are qualitatively almost identical except that one is true and the other is false, so the main difference between the representationsandshould relate to truth.
Consequently, we can take the differences in (normalized)  hidden states,, and cluster them.
We call this method Contrastive Representation Clustering (CRC) .
Clustering can be achieved by taking the top principal component (TPC)  and thresholding at, or by doing a “bimodal salience search” (BSS)  to find a direction that looks bimodal; seeSectionG.3for further details.We compare CCS and these two variants of Contrastive Representation Clustering inTable2, using the same setting as inTable1.
While CCS performs best, all methods attain high accuracy and are competitive with zero-shot performance.
This indicates both that (1)  representations of truth often lie in a high-variance direction in the contrastive representation space, and also that (2)  true and false examples are often well-clustered in this same contrastive space.
This strengthens the idea that representations of truth may be salient features inside models that are relatively easy to find. This may help explain why CCS can perform well without using any supervision, and how it can do so even with only a limited amount of unlabeled data.MethodRoBERTaDeBERTaGPT-JT5UQAT0∗Mean∗Calibrated 0-shot64.3(6.2) 76.3(6.0) 56.0(5.2) 58.8(6.1) 80.4(7.1) 90.5(2.7) 67.2(6.1) CCS62.1(4.1) 78.5(3.8) 61.7(2.5) 71.5(3.0) 82.1(2.7) 77.6(3.3) 71.2(3.2) CRC (TPC) 64.6(5.8) 76.5(5.7) 59.9(3.7) 66.7(5.0) 78.3(3.3) 58.9(13.0) 69.2(4.7) CRC (BSS) 62.6(6.8) 76.8(5.3) 60.7(3.4) 69.7(3.3) 79.3(2.5) 76.7(9.6) 69.8(4.3) Table 2:We compare CCS to two variants of Contrastive Representation Clustering: TPC, which clusters by projecting onto the top principal component, and BSS, which clusters by finding a direction that looks bimodal.
We show accuracy and standard deviation of each model averaged across all prompts and datasets, in the same setting asTable1.
We find that CCS generally performs the best, but that all methods are competitive with zero-shot.
body
Discovering Latent Knowledge in Language Models Without Supervision
3Results
2212.03827
4Related WorkZero-Shot Prompting.Since the release of GPT-3, one of the main paradigms for eliciting what models know has been zero-shot prompting.
Zero-shot exploits how language models are trained to predict diverse data from the internet, which incidentally includes tasks such as question-answering.
If prompted appropriately, this can be used to solve various useful tasks with reasonable performance.
However, these models are trained to imitate human-generated data, which bounds the quality of their outputs.Many methods improve upon vanilla zero-shot prompting.
While our goal is not to improve zero-shot performance, some of the ideas underlying these methods are similar to CCS.
Particularly relevant are methods that also leverage unsupervised consistency properties, such as.
However, these methods still bootstrap from language model outputs trained via imitation learning, which limits their applicability to our main goals.To illustrate this, imagine we train reinforcement learning agents to play a game such as Diplomacy, in which players have incentives to lie to each other.
Then those agents may learn to lie in a way that is difficult to detect, but they may still internally represent whether they are lying.
Because their outputs would be deliberately misleading, standard zero-shot methods may be very unreliable.
In contrast, techniques like CCS may still be able to detect whether those models are lying by finding representations of truth in their activations that contradict their outputs.Truthfulness.There has been increasing interest in making language models truthful.
One aspect of truthfulness that has received substantial attention is factuality. For instance, many techniques aim to improve the factuality of models by augmenting them with retrieval methods, which allows them to cite their sources.
In contrast, we focus on truthfulness more generally, which also includes procedural knowledge such as reasoning or natural language inference tasks.An approach to making language models truthful in this more general setting is to finetune them using either human demonstrationsor reinforcement learning from human feedback.
These techniques have been widely successful at improving performance, but unlike our method they rely on being able to provide ground truth labels, which can be intractable in many settings.Some work has aimed to go beyond the direct supervision humans can provide by augmenting supervision with AI systems.
This may expand the range of applications we can supervise, but many of these proposals remain theoretical, and it is unclear just how far these techniques can generalize.reframes this issue by posing the problem of Eliciting Latent Knowledge (ELK) .
Like our problem statement, ELK is about eliciting knowledge from models even in cases where humans cannot evaluate that knowledge.
However, ELK frames this as a worst-case theoretical problem, while we frame this as an empirical problem that we can make progress on using current models.
body
Discovering Latent Knowledge in Language Models Without Supervision
4Related Work
2212.03827
5Discussion5.1Limitations and Future WorkOur work has a number of limitations.
First, CCS relies on the existence of a direction in activation space that separates true and false inputs well, in the sense that a supervised probe on the activations would be able to attain high accuracy (if it hypothetically had access to the ground truth labels) .
This requires that a model is bothcapableof evaluating the truth of a given input, and also that the modelactively evaluates the truthof that input.
It is not clear when these conditions hold precisely.Second, we did not evaluate our method on setups involving active “lying” or “deception”by models, as we aren’t aware of existing evaluation setups for this setting.
If future work develops such a setup, a good stress test would be to apply CCS to do “lie detection” in that setting.
This may require modifications or extensions to the method, such as more explicitly ensuring that it recovers the truth of an input rather than what the model says.There are also various straightforward improvements to our method that one could explore.
This includes adding additional consistency constraints, improving its reliability, calibrating its probabilities, generalizing it beyond the yes-no question-answering setting, generalizing it to cases where answers aren’t clear-cut, and closing the remaining gap between CCS and the logistic regression ceiling.5.2ConclusionAs language models become more capable, they will be increasingly used as components in larger AI systems trained with reinforcement learning.
As this occurs, falsehoods arising from misaligned training objectives may become more common, severe, and difficult to detect.
In principle, models may even develop instrumental incentives to lie: for example, if a human evaluator would disapprove of bad model behavior, models may learn to lie about their behavior to achieve higher reward.
If so, those models would be optimizing against human evaluators, making it precarious to rely on those evaluators for assessing the truth of what models say.
Because unsupervised methods for eliciting answers circumvent this issue, they may still work even in this scenario.
We show that it is possible to make progress on such methods today; the empirical success of our approach suggests that unsupervised methods are both a tractable and underexplored research direction.
body
Discovering Latent Knowledge in Language Models Without Supervision
5Discussion
2212.03827
AcknowledgementsWe are very grateful to Jared Kaplan for helpful experiment suggestions and resources early on in the project. We thank Beth Barnes and Paul Christiano for valuable discussions regarding the longer-term impacts of this work. We are also grateful to Jessy Lin, Alex Pan, Ruiqi Zhong, Yaodong Yu, the anonymous reviewers, and several others for useful feedback on earlier versions of this paper. CB is supported by an Open Philanthropy AI Fellowship.
conclusion
Discovering Latent Knowledge in Language Models Without Supervision
Acknowledgements