<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2312.03729] Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?</title><meta property="og:description" content="Neural language models (LMs) can be used to evaluate the truth of factual statements in two ways: they can be either queried for statement probabilities, or probed for internal representations of truthfulness.
Past wor…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2312.03729">

<!--Generated on Tue Feb 27 14:42:07 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line">
<h1 class="ltx_title ltx_title_document">Cognitive Dissonance:
<br class="ltx_break">Why Do Language Model Outputs Disagree 
<br class="ltx_break">with Internal Representations of Truthfulness?</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Kevin Liu<sup id="id3.3.id1" class="ltx_sup">∗</sup>     Stephen Casper<sup id="id4.4.id2" class="ltx_sup">∗</sup>     Dylan Hadfield-Menell     Jacob Andreas 
<br class="ltx_break">MIT CSAIL 
<br class="ltx_break"><span id="id5.5.id3" class="ltx_text ltx_font_typewriter">{kevliu, scasper, dhm, jda}@mit.edu</span> 
<br class="ltx_break">
</span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract</h6>
<p id="id6.id1" class="ltx_p">Neural language models (LMs) can be used to evaluate the truth of factual statements in two ways: they can be either <em id="id6.id1.1" class="ltx_emph ltx_font_italic">queried</em> for statement probabilities, or <em id="id6.id1.2" class="ltx_emph ltx_font_italic">probed</em> for internal representations of truthfulness.
Past work has found that these two procedures sometimes disagree, and that probes tend to be more accurate than LM outputs. This has led some researchers to conclude that LMs “lie” or otherwise encode non-cooperative communicative intents.
Is this an accurate description of today’s LMs, or can query–probe disagreement arise in other ways? We identify three different classes of disagreement, which we term <em id="id6.id1.3" class="ltx_emph ltx_font_italic">confabulation</em>, <em id="id6.id1.4" class="ltx_emph ltx_font_italic">deception</em>, and <em id="id6.id1.5" class="ltx_emph ltx_font_italic">heterogeneity</em>.
In many cases, the superiority of probes is simply attributable to better calibration on uncertain answers rather than a greater fraction of correct, high-confidence answers.
In some cases, queries and probes perform better on different subsets of inputs, and accuracy can further be improved by ensembling the two.<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>Code at <a href="github.com/lingo-mit/lm-truthfulness" title="" class="ltx_ref ltx_url ltx_font_typewriter">github.com/lingo-mit/lm-truthfulness</a>.</span></span></span></p>
</div>
<span id="footnotex1" class="ltx_note ltx_role_footnotetext"><sup class="ltx_note_mark">*</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">*</sup><span class="ltx_note_type">footnotetext: </span>Equal contribution. Correspondence to scasper@mit.edu.</span></span></span>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1 </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Text generated by neural language models (LMs) often contains factual errors <cite class="ltx_cite ltx_citemacro_cite">Martindale et al. (<a href="#bib.bib16" title="" class="ltx_ref">2019</a>)</cite>. These errors limit LMs’ ability to generate trustworthy content and serve as knowledge sources in downstream applications <cite class="ltx_cite ltx_citemacro_cite">Ji et al. (<a href="#bib.bib11" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">Surprisingly, even when LM <em id="S1.p2.1.1" class="ltx_emph ltx_font_italic">outputs</em> are factually incorrect, it is sometimes possible to assess the truth of statements by probing models’ <em id="S1.p2.1.2" class="ltx_emph ltx_font_italic">hidden states</em>.
In the example shown in <a href="#S1.F1" title="In 1 Introduction ‣ Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Fig.</span> <span class="ltx_text ltx_ref_tag">1</span></a>A, a language model assigns high probability to the incorrect answer <em id="S1.p2.1.3" class="ltx_emph ltx_font_italic">yes</em> when prompted to answer the question <em id="S1.p2.1.4" class="ltx_emph ltx_font_italic">Is Sting a police officer?</em> However, a linear <span id="S1.p2.1.5" class="ltx_text ltx_font_bold">knowledge probe</span> trained on the LM’s hidden representations (<a href="#S1.F1" title="In 1 Introduction ‣ Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Fig.</span> <span class="ltx_text ltx_ref_tag">1</span></a>B) successfully classifies <em id="S1.p2.1.6" class="ltx_emph ltx_font_italic">no</em> as the more likely answer.
Knowledge probes of this kind have consistently been found to be slightly more reliable at question answering and fact verification than direct queries to LMs <cite class="ltx_cite ltx_citemacro_citep">(Burns et al., <a href="#bib.bib5" title="" class="ltx_ref">2022</a>)</cite>,
suggesting a mismatch between LMs’ (internal) representations of factuality and their (external) expressions of statement probability.</p>
</div>
<figure id="S1.F1" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_1"><img src="/html/2312.03729/assets/x1.png" id="S1.F1.g1" class="ltx_graphics ltx_figure_panel ltx_img_landscape" width="332" height="160" alt="Refer to caption"></div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1"><img src="/html/2312.03729/assets/x2.png" id="S1.F1.g2" class="ltx_graphics ltx_figure_panel ltx_img_landscape" width="332" height="201" alt="Refer to caption"></div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 1: </span>Varieties of disagreement between language model outputs and internal states. We evaluate two approaches for answering questions or verifying statements: <em id="S1.F1.3.1" class="ltx_emph ltx_font_italic">querying</em> models directly for answers (A), or training a binary classifier to <em id="S1.F1.4.2" class="ltx_emph ltx_font_italic">probe</em> their internal states (B). Probes and queries sometimes disagree. We propose a taxonomy of different query–probe disagreement types (C); across several models and datasets, we find that disagreements mostly occur in situations where either probes or queries are uncertain (D).</figcaption>
</figure>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">How should we understand this behavior? One possible interpretation, suggested by several previous studies, is that it is analogous to “deception” in human language users <cite class="ltx_cite ltx_citemacro_citep">(Kadavath et al., <a href="#bib.bib12" title="" class="ltx_ref">2022</a>; Azaria and Mitchell, <a href="#bib.bib3" title="" class="ltx_ref">2023</a>; Zou et al., <a href="#bib.bib27" title="" class="ltx_ref">2023</a>)</cite>. In this framing, if LM representations encode truthfulness more reliably than outputs, then mismatches between LM queries and probes must result from situations in which LMs “know” that a statement is untrue and nonetheless assign it high probability.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">But knowledge is not a binary phenomenon in LMs or humans.
While probes are better than queries on average, they may not be better on every example.
Moreover, both LMs and probes are probabilistic models: LM predictions can encode uncertainty over possible answers, and this uncertainty can be decoded from their internal representations <cite class="ltx_cite ltx_citemacro_citep">(Mielke et al., <a href="#bib.bib17" title="" class="ltx_ref">2022</a>)</cite>. There are thus a variety of mechanisms that might underlie query–probe disagreement, ranging from differences in calibration to differences in behavior in different input subsets.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">In this paper, we seek to
understand mismatches between internal and external expressions of truthfulness by understanding the <em id="S1.p5.1.1" class="ltx_emph ltx_font_italic">distribution</em> over predictions, taking into account uncertainty in answers produced by both queries and probes.
In doing so, we hope to provide a formal grounding of several terms that are widely (but vaguely and inconsistently) used to describe factual errors in LMs.
LMs and probing techniques are rapidly evolving, so we do not aim to provide a definitive answer to the question of why LMs assign false statements high probability.
However, in a widely studied probe class, two LMs, and three question-answering datasets, we identify three qualitatively different reasons that probes may outperform queries, which we call <em id="S1.p5.1.2" class="ltx_emph ltx_font_italic">confabulation</em> (queries produce high-confidence answers when probes are low-confidence), <em id="S1.p5.1.3" class="ltx_emph ltx_font_italic">heterogeneity</em> (probes and queries improve performance on different data subsets), and (in a small number of instances) what past work would characterize as <em id="S1.p5.1.4" class="ltx_emph ltx_font_italic">deception</em> (<a href="#S1.F1" title="In 1 Introduction ‣ Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Fig.</span> <span class="ltx_text ltx_ref_tag">1</span></a>C–D, in which queries and probes disagree confidently on answer probabilities).
Most mismatches occur when probes or queries are uncertain about answers. By combining probe and query predictions, it is sometimes possible to obtain better accuracy than either alone.</p>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p">Our results paint a nuanced picture of the representation of factuality in LMs: even when probes outperform LMs, they do not explain all of LMs’ ability to make factual predictions. Today, many mismatches between internal and external representations of factuality appear attributable to different prediction pathways, rather than an explicit representation of a latent intent to generate outputs “known” to be incorrect.</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2 </span>Preliminaries</h2>

<section id="S2.SS0.SSS0.Px1" class="ltx_paragraph">
<h3 class="ltx_title ltx_title_paragraph">Extracting answers from LMs</h3>

<div id="S2.SS0.SSS0.Px1.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px1.p1.4" class="ltx_p">We study autoregressive LMs that compute a next-token distribution <math id="S2.SS0.SSS0.Px1.p1.1.m1.1" class="ltx_Math" alttext="p_{\mathrm{LM}}(x_{i}\mid x_{&lt;i})" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.1.m1.1a"><mrow id="S2.SS0.SSS0.Px1.p1.1.m1.1.1" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.cmml"><msub id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.cmml">p</mi><mi id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.cmml">LM</mi></msub><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.2" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.2.cmml">​</mo><mrow id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.2" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.cmml">(</mo><mrow id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.cmml"><msub id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.2" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.2.cmml"><mi id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.2.2" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.2.2.cmml">x</mi><mi id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.2.3" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.2.3.cmml">i</mi></msub><mo id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.1" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.1.cmml">∣</mo><msub id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.2" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.2.cmml">x</mi><mrow id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.3" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.3.cmml"><mi id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.3.2" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.3.2.cmml"></mi><mo id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.3.1" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.3.1.cmml">&lt;</mo><mi id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.3.3" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.3.3.cmml">i</mi></mrow></msub></mrow><mo stretchy="false" id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.3" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.1.m1.1b"><apply id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1"><times id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.2"></times><apply id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.2">𝑝</ci><ci id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.3.3">LM</ci></apply><apply id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1"><csymbol cd="latexml" id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.1">conditional</csymbol><apply id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.2"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.2.1.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.2">subscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.2.2.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.2.2">𝑥</ci><ci id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.2.3.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.2.3">𝑖</ci></apply><apply id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.2">𝑥</ci><apply id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.3"><lt id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.3.1.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.3.1"></lt><csymbol cd="latexml" id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.3.2.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.3.2">absent</csymbol><ci id="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.3.3.cmml" xref="S2.SS0.SSS0.Px1.p1.1.m1.1.1.1.1.1.3.3.3">𝑖</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.1.m1.1c">p_{\mathrm{LM}}(x_{i}\mid x_{&lt;i})</annotation></semantics></math> by first mapping the input <math id="S2.SS0.SSS0.Px1.p1.2.m2.1" class="ltx_Math" alttext="x_{&lt;i}" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.2.m2.1a"><msub id="S2.SS0.SSS0.Px1.p1.2.m2.1.1" xref="S2.SS0.SSS0.Px1.p1.2.m2.1.1.cmml"><mi id="S2.SS0.SSS0.Px1.p1.2.m2.1.1.2" xref="S2.SS0.SSS0.Px1.p1.2.m2.1.1.2.cmml">x</mi><mrow id="S2.SS0.SSS0.Px1.p1.2.m2.1.1.3" xref="S2.SS0.SSS0.Px1.p1.2.m2.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px1.p1.2.m2.1.1.3.2" xref="S2.SS0.SSS0.Px1.p1.2.m2.1.1.3.2.cmml"></mi><mo id="S2.SS0.SSS0.Px1.p1.2.m2.1.1.3.1" xref="S2.SS0.SSS0.Px1.p1.2.m2.1.1.3.1.cmml">&lt;</mo><mi id="S2.SS0.SSS0.Px1.p1.2.m2.1.1.3.3" xref="S2.SS0.SSS0.Px1.p1.2.m2.1.1.3.3.cmml">i</mi></mrow></msub><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.2.m2.1b"><apply id="S2.SS0.SSS0.Px1.p1.2.m2.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.2.m2.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.2.m2.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.2.m2.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.2.m2.1.1.2">𝑥</ci><apply id="S2.SS0.SSS0.Px1.p1.2.m2.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.2.m2.1.1.3"><lt id="S2.SS0.SSS0.Px1.p1.2.m2.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px1.p1.2.m2.1.1.3.1"></lt><csymbol cd="latexml" id="S2.SS0.SSS0.Px1.p1.2.m2.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px1.p1.2.m2.1.1.3.2">absent</csymbol><ci id="S2.SS0.SSS0.Px1.p1.2.m2.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px1.p1.2.m2.1.1.3.3">𝑖</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.2.m2.1c">x_{&lt;i}</annotation></semantics></math> to some hidden representation <math id="S2.SS0.SSS0.Px1.p1.3.m3.1" class="ltx_Math" alttext="h_{\mathrm{LM}}(x_{&lt;i})" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.3.m3.1a"><mrow id="S2.SS0.SSS0.Px1.p1.3.m3.1.1" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.cmml"><msub id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.2" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.2.cmml">h</mi><mi id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.3" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.3.cmml">LM</mi></msub><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.2" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.2.cmml">​</mo><mrow id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.2" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.cmml">(</mo><msub id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.cmml"><mi id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.2" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.2.cmml">x</mi><mrow id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.3" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.3.cmml"><mi id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.3.2" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.3.2.cmml"></mi><mo id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.3.1" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.3.1.cmml">&lt;</mo><mi id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.3.3" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.3.3.cmml">i</mi></mrow></msub><mo stretchy="false" id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.3" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.3.m3.1b"><apply id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1"><times id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.2"></times><apply id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.2">ℎ</ci><ci id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.3.3">LM</ci></apply><apply id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.2">𝑥</ci><apply id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.3"><lt id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.3.1"></lt><csymbol cd="latexml" id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.3.2.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.3.2">absent</csymbol><ci id="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.3.3.cmml" xref="S2.SS0.SSS0.Px1.p1.3.m3.1.1.1.1.1.3.3">𝑖</ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.3.m3.1c">h_{\mathrm{LM}}(x_{&lt;i})</annotation></semantics></math>, then using this representation to predict <math id="S2.SS0.SSS0.Px1.p1.4.m4.1" class="ltx_Math" alttext="x_{i}" display="inline"><semantics id="S2.SS0.SSS0.Px1.p1.4.m4.1a"><msub id="S2.SS0.SSS0.Px1.p1.4.m4.1.1" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.cmml"><mi id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2.cmml">x</mi><mi id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.cmml">i</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px1.p1.4.m4.1b"><apply id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.1.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.2">𝑥</ci><ci id="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3.cmml" xref="S2.SS0.SSS0.Px1.p1.4.m4.1.1.3">𝑖</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px1.p1.4.m4.1c">x_{i}</annotation></semantics></math>. There are two standard procedures for answering questions using such an LM:</p>
</div>
<div id="S2.SS0.SSS0.Px1.p2" class="ltx_para">
<ol id="S2.I1" class="ltx_enumerate">
<li id="S2.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S2.I1.i1.p1" class="ltx_para">
<p id="S2.I1.i1.p1.3" class="ltx_p"><span id="S2.I1.i1.p1.3.1" class="ltx_text ltx_font_bold">Querying</span>: Provide the question <math id="S2.I1.i1.p1.1.m1.1" class="ltx_Math" alttext="q" display="inline"><semantics id="S2.I1.i1.p1.1.m1.1a"><mi id="S2.I1.i1.p1.1.m1.1.1" xref="S2.I1.i1.p1.1.m1.1.1.cmml">q</mi><annotation-xml encoding="MathML-Content" id="S2.I1.i1.p1.1.m1.1b"><ci id="S2.I1.i1.p1.1.m1.1.1.cmml" xref="S2.I1.i1.p1.1.m1.1.1">𝑞</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i1.p1.1.m1.1c">q</annotation></semantics></math> as input to the LM, and rank answers <math id="S2.I1.i1.p1.2.m2.1" class="ltx_Math" alttext="a" display="inline"><semantics id="S2.I1.i1.p1.2.m2.1a"><mi id="S2.I1.i1.p1.2.m2.1.1" xref="S2.I1.i1.p1.2.m2.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="S2.I1.i1.p1.2.m2.1b"><ci id="S2.I1.i1.p1.2.m2.1.1.cmml" xref="S2.I1.i1.p1.2.m2.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i1.p1.2.m2.1c">a</annotation></semantics></math> (e.g. <span id="S2.I1.i1.p1.3.2" class="ltx_text ltx_font_italic">yes</span>/<span id="S2.I1.i1.p1.3.3" class="ltx_text ltx_font_italic">no</span>) according to <math id="S2.I1.i1.p1.3.m3.1" class="ltx_Math" alttext="p_{\mathrm{LM}}(a\mid q)" display="inline"><semantics id="S2.I1.i1.p1.3.m3.1a"><mrow id="S2.I1.i1.p1.3.m3.1.1" xref="S2.I1.i1.p1.3.m3.1.1.cmml"><msub id="S2.I1.i1.p1.3.m3.1.1.3" xref="S2.I1.i1.p1.3.m3.1.1.3.cmml"><mi id="S2.I1.i1.p1.3.m3.1.1.3.2" xref="S2.I1.i1.p1.3.m3.1.1.3.2.cmml">p</mi><mi id="S2.I1.i1.p1.3.m3.1.1.3.3" xref="S2.I1.i1.p1.3.m3.1.1.3.3.cmml">LM</mi></msub><mo lspace="0em" rspace="0em" id="S2.I1.i1.p1.3.m3.1.1.2" xref="S2.I1.i1.p1.3.m3.1.1.2.cmml">​</mo><mrow id="S2.I1.i1.p1.3.m3.1.1.1.1" xref="S2.I1.i1.p1.3.m3.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.I1.i1.p1.3.m3.1.1.1.1.2" xref="S2.I1.i1.p1.3.m3.1.1.1.1.1.cmml">(</mo><mrow id="S2.I1.i1.p1.3.m3.1.1.1.1.1" xref="S2.I1.i1.p1.3.m3.1.1.1.1.1.cmml"><mi id="S2.I1.i1.p1.3.m3.1.1.1.1.1.2" xref="S2.I1.i1.p1.3.m3.1.1.1.1.1.2.cmml">a</mi><mo id="S2.I1.i1.p1.3.m3.1.1.1.1.1.1" xref="S2.I1.i1.p1.3.m3.1.1.1.1.1.1.cmml">∣</mo><mi id="S2.I1.i1.p1.3.m3.1.1.1.1.1.3" xref="S2.I1.i1.p1.3.m3.1.1.1.1.1.3.cmml">q</mi></mrow><mo stretchy="false" id="S2.I1.i1.p1.3.m3.1.1.1.1.3" xref="S2.I1.i1.p1.3.m3.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.I1.i1.p1.3.m3.1b"><apply id="S2.I1.i1.p1.3.m3.1.1.cmml" xref="S2.I1.i1.p1.3.m3.1.1"><times id="S2.I1.i1.p1.3.m3.1.1.2.cmml" xref="S2.I1.i1.p1.3.m3.1.1.2"></times><apply id="S2.I1.i1.p1.3.m3.1.1.3.cmml" xref="S2.I1.i1.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="S2.I1.i1.p1.3.m3.1.1.3.1.cmml" xref="S2.I1.i1.p1.3.m3.1.1.3">subscript</csymbol><ci id="S2.I1.i1.p1.3.m3.1.1.3.2.cmml" xref="S2.I1.i1.p1.3.m3.1.1.3.2">𝑝</ci><ci id="S2.I1.i1.p1.3.m3.1.1.3.3.cmml" xref="S2.I1.i1.p1.3.m3.1.1.3.3">LM</ci></apply><apply id="S2.I1.i1.p1.3.m3.1.1.1.1.1.cmml" xref="S2.I1.i1.p1.3.m3.1.1.1.1"><csymbol cd="latexml" id="S2.I1.i1.p1.3.m3.1.1.1.1.1.1.cmml" xref="S2.I1.i1.p1.3.m3.1.1.1.1.1.1">conditional</csymbol><ci id="S2.I1.i1.p1.3.m3.1.1.1.1.1.2.cmml" xref="S2.I1.i1.p1.3.m3.1.1.1.1.1.2">𝑎</ci><ci id="S2.I1.i1.p1.3.m3.1.1.1.1.1.3.cmml" xref="S2.I1.i1.p1.3.m3.1.1.1.1.1.3">𝑞</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i1.p1.3.m3.1c">p_{\mathrm{LM}}(a\mid q)</annotation></semantics></math> <cite class="ltx_cite ltx_citemacro_citep">(Petroni et al., <a href="#bib.bib19" title="" class="ltx_ref">2019</a>)</cite>.</p>
</div>
</li>
<li id="S2.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S2.I1.i2.p1" class="ltx_para">
<p id="S2.I1.i2.p1.3" class="ltx_p"><span id="S2.I1.i2.p1.3.1" class="ltx_text ltx_font_bold">Probing</span>: Extract the hidden representation <math id="S2.I1.i2.p1.1.m1.3" class="ltx_Math" alttext="h([q,a])" display="inline"><semantics id="S2.I1.i2.p1.1.m1.3a"><mrow id="S2.I1.i2.p1.1.m1.3.3" xref="S2.I1.i2.p1.1.m1.3.3.cmml"><mi id="S2.I1.i2.p1.1.m1.3.3.3" xref="S2.I1.i2.p1.1.m1.3.3.3.cmml">h</mi><mo lspace="0em" rspace="0em" id="S2.I1.i2.p1.1.m1.3.3.2" xref="S2.I1.i2.p1.1.m1.3.3.2.cmml">​</mo><mrow id="S2.I1.i2.p1.1.m1.3.3.1.1" xref="S2.I1.i2.p1.1.m1.3.3.cmml"><mo stretchy="false" id="S2.I1.i2.p1.1.m1.3.3.1.1.2" xref="S2.I1.i2.p1.1.m1.3.3.cmml">(</mo><mrow id="S2.I1.i2.p1.1.m1.3.3.1.1.1.2" xref="S2.I1.i2.p1.1.m1.3.3.1.1.1.1.cmml"><mo stretchy="false" id="S2.I1.i2.p1.1.m1.3.3.1.1.1.2.1" xref="S2.I1.i2.p1.1.m1.3.3.1.1.1.1.cmml">[</mo><mi id="S2.I1.i2.p1.1.m1.1.1" xref="S2.I1.i2.p1.1.m1.1.1.cmml">q</mi><mo id="S2.I1.i2.p1.1.m1.3.3.1.1.1.2.2" xref="S2.I1.i2.p1.1.m1.3.3.1.1.1.1.cmml">,</mo><mi id="S2.I1.i2.p1.1.m1.2.2" xref="S2.I1.i2.p1.1.m1.2.2.cmml">a</mi><mo stretchy="false" id="S2.I1.i2.p1.1.m1.3.3.1.1.1.2.3" xref="S2.I1.i2.p1.1.m1.3.3.1.1.1.1.cmml">]</mo></mrow><mo stretchy="false" id="S2.I1.i2.p1.1.m1.3.3.1.1.3" xref="S2.I1.i2.p1.1.m1.3.3.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.I1.i2.p1.1.m1.3b"><apply id="S2.I1.i2.p1.1.m1.3.3.cmml" xref="S2.I1.i2.p1.1.m1.3.3"><times id="S2.I1.i2.p1.1.m1.3.3.2.cmml" xref="S2.I1.i2.p1.1.m1.3.3.2"></times><ci id="S2.I1.i2.p1.1.m1.3.3.3.cmml" xref="S2.I1.i2.p1.1.m1.3.3.3">ℎ</ci><interval closure="closed" id="S2.I1.i2.p1.1.m1.3.3.1.1.1.1.cmml" xref="S2.I1.i2.p1.1.m1.3.3.1.1.1.2"><ci id="S2.I1.i2.p1.1.m1.1.1.cmml" xref="S2.I1.i2.p1.1.m1.1.1">𝑞</ci><ci id="S2.I1.i2.p1.1.m1.2.2.cmml" xref="S2.I1.i2.p1.1.m1.2.2">𝑎</ci></interval></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i2.p1.1.m1.3c">h([q,a])</annotation></semantics></math> from the LM, then use question/answer pairs labeled for correctness (or unsupervised clustering methods) to train a binary classifier that maps from <math id="S2.I1.i2.p1.2.m2.1" class="ltx_Math" alttext="h_{\mathrm{LM}}" display="inline"><semantics id="S2.I1.i2.p1.2.m2.1a"><msub id="S2.I1.i2.p1.2.m2.1.1" xref="S2.I1.i2.p1.2.m2.1.1.cmml"><mi id="S2.I1.i2.p1.2.m2.1.1.2" xref="S2.I1.i2.p1.2.m2.1.1.2.cmml">h</mi><mi id="S2.I1.i2.p1.2.m2.1.1.3" xref="S2.I1.i2.p1.2.m2.1.1.3.cmml">LM</mi></msub><annotation-xml encoding="MathML-Content" id="S2.I1.i2.p1.2.m2.1b"><apply id="S2.I1.i2.p1.2.m2.1.1.cmml" xref="S2.I1.i2.p1.2.m2.1.1"><csymbol cd="ambiguous" id="S2.I1.i2.p1.2.m2.1.1.1.cmml" xref="S2.I1.i2.p1.2.m2.1.1">subscript</csymbol><ci id="S2.I1.i2.p1.2.m2.1.1.2.cmml" xref="S2.I1.i2.p1.2.m2.1.1.2">ℎ</ci><ci id="S2.I1.i2.p1.2.m2.1.1.3.cmml" xref="S2.I1.i2.p1.2.m2.1.1.3">LM</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i2.p1.2.m2.1c">h_{\mathrm{LM}}</annotation></semantics></math> to a distribution <math id="S2.I1.i2.p1.3.m3.1" class="ltx_Math" alttext="p_{\mathrm{probe}}(\texttt{correct}\mid h)" display="inline"><semantics id="S2.I1.i2.p1.3.m3.1a"><mrow id="S2.I1.i2.p1.3.m3.1.1" xref="S2.I1.i2.p1.3.m3.1.1.cmml"><msub id="S2.I1.i2.p1.3.m3.1.1.3" xref="S2.I1.i2.p1.3.m3.1.1.3.cmml"><mi id="S2.I1.i2.p1.3.m3.1.1.3.2" xref="S2.I1.i2.p1.3.m3.1.1.3.2.cmml">p</mi><mi id="S2.I1.i2.p1.3.m3.1.1.3.3" xref="S2.I1.i2.p1.3.m3.1.1.3.3.cmml">probe</mi></msub><mo lspace="0em" rspace="0em" id="S2.I1.i2.p1.3.m3.1.1.2" xref="S2.I1.i2.p1.3.m3.1.1.2.cmml">​</mo><mrow id="S2.I1.i2.p1.3.m3.1.1.1.1" xref="S2.I1.i2.p1.3.m3.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.I1.i2.p1.3.m3.1.1.1.1.2" xref="S2.I1.i2.p1.3.m3.1.1.1.1.1.cmml">(</mo><mrow id="S2.I1.i2.p1.3.m3.1.1.1.1.1" xref="S2.I1.i2.p1.3.m3.1.1.1.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="S2.I1.i2.p1.3.m3.1.1.1.1.1.2" xref="S2.I1.i2.p1.3.m3.1.1.1.1.1.2a.cmml">correct</mtext><mo id="S2.I1.i2.p1.3.m3.1.1.1.1.1.1" xref="S2.I1.i2.p1.3.m3.1.1.1.1.1.1.cmml">∣</mo><mi id="S2.I1.i2.p1.3.m3.1.1.1.1.1.3" xref="S2.I1.i2.p1.3.m3.1.1.1.1.1.3.cmml">h</mi></mrow><mo stretchy="false" id="S2.I1.i2.p1.3.m3.1.1.1.1.3" xref="S2.I1.i2.p1.3.m3.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.I1.i2.p1.3.m3.1b"><apply id="S2.I1.i2.p1.3.m3.1.1.cmml" xref="S2.I1.i2.p1.3.m3.1.1"><times id="S2.I1.i2.p1.3.m3.1.1.2.cmml" xref="S2.I1.i2.p1.3.m3.1.1.2"></times><apply id="S2.I1.i2.p1.3.m3.1.1.3.cmml" xref="S2.I1.i2.p1.3.m3.1.1.3"><csymbol cd="ambiguous" id="S2.I1.i2.p1.3.m3.1.1.3.1.cmml" xref="S2.I1.i2.p1.3.m3.1.1.3">subscript</csymbol><ci id="S2.I1.i2.p1.3.m3.1.1.3.2.cmml" xref="S2.I1.i2.p1.3.m3.1.1.3.2">𝑝</ci><ci id="S2.I1.i2.p1.3.m3.1.1.3.3.cmml" xref="S2.I1.i2.p1.3.m3.1.1.3.3">probe</ci></apply><apply id="S2.I1.i2.p1.3.m3.1.1.1.1.1.cmml" xref="S2.I1.i2.p1.3.m3.1.1.1.1"><csymbol cd="latexml" id="S2.I1.i2.p1.3.m3.1.1.1.1.1.1.cmml" xref="S2.I1.i2.p1.3.m3.1.1.1.1.1.1">conditional</csymbol><ci id="S2.I1.i2.p1.3.m3.1.1.1.1.1.2a.cmml" xref="S2.I1.i2.p1.3.m3.1.1.1.1.1.2"><mtext class="ltx_mathvariant_monospace" id="S2.I1.i2.p1.3.m3.1.1.1.1.1.2.cmml" xref="S2.I1.i2.p1.3.m3.1.1.1.1.1.2">correct</mtext></ci><ci id="S2.I1.i2.p1.3.m3.1.1.1.1.1.3.cmml" xref="S2.I1.i2.p1.3.m3.1.1.1.1.1.3">ℎ</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I1.i2.p1.3.m3.1c">p_{\mathrm{probe}}(\texttt{correct}\mid h)</annotation></semantics></math> <cite class="ltx_cite ltx_citemacro_citep">(Burns et al., <a href="#bib.bib5" title="" class="ltx_ref">2022</a>)</cite>.</p>
</div>
</li>
</ol>
</div>
</section>
<section id="S2.SS0.SSS0.Px2" class="ltx_paragraph">
<h3 class="ltx_title ltx_title_paragraph">Causes of disagreement</h3>

<div id="S2.SS0.SSS0.Px2.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px2.p1.1" class="ltx_p">Past work has shown that this probing procedure is effective; recent work has shown that it often produces different, and slightly better, answers than direct querying <cite class="ltx_cite ltx_citemacro_citep">(Burns et al., <a href="#bib.bib5" title="" class="ltx_ref">2022</a>)</cite>.
Why might this mismatch occur? In this paper, we define three possible cases:</p>
</div>
<div id="S2.SS0.SSS0.Px2.p2" class="ltx_para">
<ol id="S2.I2" class="ltx_enumerate">
<li id="S2.I2.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">1.</span> 
<div id="S2.I2.i1.p1" class="ltx_para">
<p id="S2.I2.i1.p1.3" class="ltx_p"><span id="S2.I2.i1.p1.3.1" class="ltx_text ltx_font_bold">Model Confabulation</span> <cite class="ltx_cite ltx_citemacro_citep">(Edwards, <a href="#bib.bib7" title="" class="ltx_ref">2023</a>)</cite>: Disagreements occurring when probe confidence is low, and the completions from queries are incorrect (mid-left of <a href="#S1.F1" title="In 1 Introduction ‣ Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">1</span></a>). In these cases
probes may be slightly more accurate if their prediction confidence is better calibrated to the probability of correctness.
In LMs exhibiting confabulation, a large fraction of disagreements will occur on inputs with large probe entropy <math id="S2.I2.i1.p1.1.m1.3" class="ltx_Math" alttext="H_{\mathrm{probe}}(\textrm{correct}\mid(q,a))" display="inline"><semantics id="S2.I2.i1.p1.1.m1.3a"><mrow id="S2.I2.i1.p1.1.m1.3.3" xref="S2.I2.i1.p1.1.m1.3.3.cmml"><msub id="S2.I2.i1.p1.1.m1.3.3.3" xref="S2.I2.i1.p1.1.m1.3.3.3.cmml"><mi id="S2.I2.i1.p1.1.m1.3.3.3.2" xref="S2.I2.i1.p1.1.m1.3.3.3.2.cmml">H</mi><mi id="S2.I2.i1.p1.1.m1.3.3.3.3" xref="S2.I2.i1.p1.1.m1.3.3.3.3.cmml">probe</mi></msub><mo lspace="0em" rspace="0em" id="S2.I2.i1.p1.1.m1.3.3.2" xref="S2.I2.i1.p1.1.m1.3.3.2.cmml">​</mo><mrow id="S2.I2.i1.p1.1.m1.3.3.1.1" xref="S2.I2.i1.p1.1.m1.3.3.1.1.1.cmml"><mo stretchy="false" id="S2.I2.i1.p1.1.m1.3.3.1.1.2" xref="S2.I2.i1.p1.1.m1.3.3.1.1.1.cmml">(</mo><mrow id="S2.I2.i1.p1.1.m1.3.3.1.1.1" xref="S2.I2.i1.p1.1.m1.3.3.1.1.1.cmml"><mtext id="S2.I2.i1.p1.1.m1.3.3.1.1.1.2" xref="S2.I2.i1.p1.1.m1.3.3.1.1.1.2a.cmml">correct</mtext><mo id="S2.I2.i1.p1.1.m1.3.3.1.1.1.1" xref="S2.I2.i1.p1.1.m1.3.3.1.1.1.1.cmml">∣</mo><mrow id="S2.I2.i1.p1.1.m1.3.3.1.1.1.3.2" xref="S2.I2.i1.p1.1.m1.3.3.1.1.1.3.1.cmml"><mo stretchy="false" id="S2.I2.i1.p1.1.m1.3.3.1.1.1.3.2.1" xref="S2.I2.i1.p1.1.m1.3.3.1.1.1.3.1.cmml">(</mo><mi id="S2.I2.i1.p1.1.m1.1.1" xref="S2.I2.i1.p1.1.m1.1.1.cmml">q</mi><mo id="S2.I2.i1.p1.1.m1.3.3.1.1.1.3.2.2" xref="S2.I2.i1.p1.1.m1.3.3.1.1.1.3.1.cmml">,</mo><mi id="S2.I2.i1.p1.1.m1.2.2" xref="S2.I2.i1.p1.1.m1.2.2.cmml">a</mi><mo stretchy="false" id="S2.I2.i1.p1.1.m1.3.3.1.1.1.3.2.3" xref="S2.I2.i1.p1.1.m1.3.3.1.1.1.3.1.cmml">)</mo></mrow></mrow><mo stretchy="false" id="S2.I2.i1.p1.1.m1.3.3.1.1.3" xref="S2.I2.i1.p1.1.m1.3.3.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.I2.i1.p1.1.m1.3b"><apply id="S2.I2.i1.p1.1.m1.3.3.cmml" xref="S2.I2.i1.p1.1.m1.3.3"><times id="S2.I2.i1.p1.1.m1.3.3.2.cmml" xref="S2.I2.i1.p1.1.m1.3.3.2"></times><apply id="S2.I2.i1.p1.1.m1.3.3.3.cmml" xref="S2.I2.i1.p1.1.m1.3.3.3"><csymbol cd="ambiguous" id="S2.I2.i1.p1.1.m1.3.3.3.1.cmml" xref="S2.I2.i1.p1.1.m1.3.3.3">subscript</csymbol><ci id="S2.I2.i1.p1.1.m1.3.3.3.2.cmml" xref="S2.I2.i1.p1.1.m1.3.3.3.2">𝐻</ci><ci id="S2.I2.i1.p1.1.m1.3.3.3.3.cmml" xref="S2.I2.i1.p1.1.m1.3.3.3.3">probe</ci></apply><apply id="S2.I2.i1.p1.1.m1.3.3.1.1.1.cmml" xref="S2.I2.i1.p1.1.m1.3.3.1.1"><csymbol cd="latexml" id="S2.I2.i1.p1.1.m1.3.3.1.1.1.1.cmml" xref="S2.I2.i1.p1.1.m1.3.3.1.1.1.1">conditional</csymbol><ci id="S2.I2.i1.p1.1.m1.3.3.1.1.1.2a.cmml" xref="S2.I2.i1.p1.1.m1.3.3.1.1.1.2"><mtext id="S2.I2.i1.p1.1.m1.3.3.1.1.1.2.cmml" xref="S2.I2.i1.p1.1.m1.3.3.1.1.1.2">correct</mtext></ci><interval closure="open" id="S2.I2.i1.p1.1.m1.3.3.1.1.1.3.1.cmml" xref="S2.I2.i1.p1.1.m1.3.3.1.1.1.3.2"><ci id="S2.I2.i1.p1.1.m1.1.1.cmml" xref="S2.I2.i1.p1.1.m1.1.1">𝑞</ci><ci id="S2.I2.i1.p1.1.m1.2.2.cmml" xref="S2.I2.i1.p1.1.m1.2.2">𝑎</ci></interval></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i1.p1.1.m1.3c">H_{\mathrm{probe}}(\textrm{correct}\mid(q,a))</annotation></semantics></math> and large query confidence <math id="S2.I2.i1.p1.2.m2.1" class="ltx_Math" alttext="p_{\mathrm{query}}(a\mid q)" display="inline"><semantics id="S2.I2.i1.p1.2.m2.1a"><mrow id="S2.I2.i1.p1.2.m2.1.1" xref="S2.I2.i1.p1.2.m2.1.1.cmml"><msub id="S2.I2.i1.p1.2.m2.1.1.3" xref="S2.I2.i1.p1.2.m2.1.1.3.cmml"><mi id="S2.I2.i1.p1.2.m2.1.1.3.2" xref="S2.I2.i1.p1.2.m2.1.1.3.2.cmml">p</mi><mi id="S2.I2.i1.p1.2.m2.1.1.3.3" xref="S2.I2.i1.p1.2.m2.1.1.3.3.cmml">query</mi></msub><mo lspace="0em" rspace="0em" id="S2.I2.i1.p1.2.m2.1.1.2" xref="S2.I2.i1.p1.2.m2.1.1.2.cmml">​</mo><mrow id="S2.I2.i1.p1.2.m2.1.1.1.1" xref="S2.I2.i1.p1.2.m2.1.1.1.1.1.cmml"><mo stretchy="false" id="S2.I2.i1.p1.2.m2.1.1.1.1.2" xref="S2.I2.i1.p1.2.m2.1.1.1.1.1.cmml">(</mo><mrow id="S2.I2.i1.p1.2.m2.1.1.1.1.1" xref="S2.I2.i1.p1.2.m2.1.1.1.1.1.cmml"><mi id="S2.I2.i1.p1.2.m2.1.1.1.1.1.2" xref="S2.I2.i1.p1.2.m2.1.1.1.1.1.2.cmml">a</mi><mo id="S2.I2.i1.p1.2.m2.1.1.1.1.1.1" xref="S2.I2.i1.p1.2.m2.1.1.1.1.1.1.cmml">∣</mo><mi id="S2.I2.i1.p1.2.m2.1.1.1.1.1.3" xref="S2.I2.i1.p1.2.m2.1.1.1.1.1.3.cmml">q</mi></mrow><mo stretchy="false" id="S2.I2.i1.p1.2.m2.1.1.1.1.3" xref="S2.I2.i1.p1.2.m2.1.1.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.I2.i1.p1.2.m2.1b"><apply id="S2.I2.i1.p1.2.m2.1.1.cmml" xref="S2.I2.i1.p1.2.m2.1.1"><times id="S2.I2.i1.p1.2.m2.1.1.2.cmml" xref="S2.I2.i1.p1.2.m2.1.1.2"></times><apply id="S2.I2.i1.p1.2.m2.1.1.3.cmml" xref="S2.I2.i1.p1.2.m2.1.1.3"><csymbol cd="ambiguous" id="S2.I2.i1.p1.2.m2.1.1.3.1.cmml" xref="S2.I2.i1.p1.2.m2.1.1.3">subscript</csymbol><ci id="S2.I2.i1.p1.2.m2.1.1.3.2.cmml" xref="S2.I2.i1.p1.2.m2.1.1.3.2">𝑝</ci><ci id="S2.I2.i1.p1.2.m2.1.1.3.3.cmml" xref="S2.I2.i1.p1.2.m2.1.1.3.3">query</ci></apply><apply id="S2.I2.i1.p1.2.m2.1.1.1.1.1.cmml" xref="S2.I2.i1.p1.2.m2.1.1.1.1"><csymbol cd="latexml" id="S2.I2.i1.p1.2.m2.1.1.1.1.1.1.cmml" xref="S2.I2.i1.p1.2.m2.1.1.1.1.1.1">conditional</csymbol><ci id="S2.I2.i1.p1.2.m2.1.1.1.1.1.2.cmml" xref="S2.I2.i1.p1.2.m2.1.1.1.1.1.2">𝑎</ci><ci id="S2.I2.i1.p1.2.m2.1.1.1.1.1.3.cmml" xref="S2.I2.i1.p1.2.m2.1.1.1.1.1.3">𝑞</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i1.p1.2.m2.1c">p_{\mathrm{query}}(a\mid q)</annotation></semantics></math> for any <math id="S2.I2.i1.p1.3.m3.1" class="ltx_Math" alttext="a" display="inline"><semantics id="S2.I2.i1.p1.3.m3.1a"><mi id="S2.I2.i1.p1.3.m3.1.1" xref="S2.I2.i1.p1.3.m3.1.1.cmml">a</mi><annotation-xml encoding="MathML-Content" id="S2.I2.i1.p1.3.m3.1b"><ci id="S2.I2.i1.p1.3.m3.1.1.cmml" xref="S2.I2.i1.p1.3.m3.1.1">𝑎</ci></annotation-xml><annotation encoding="application/x-tex" id="S2.I2.i1.p1.3.m3.1c">a</annotation></semantics></math>.</p>
</div>
</li>
<li id="S2.I2.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">2.</span> 
<div id="S2.I2.i2.p1" class="ltx_para">
<p id="S2.I2.i2.p1.1" class="ltx_p"><span id="S2.I2.i2.p1.1.1" class="ltx_text ltx_font_bold">“Deception”</span> <cite class="ltx_cite ltx_citemacro_citep">(Azaria and Mitchell, <a href="#bib.bib3" title="" class="ltx_ref">2023</a>)</cite>, which we define as the set of disagreements in which the probe is confidently correct and the query completion is confidently incorrect (upper-left of <a href="#S1.F1" title="In 1 Introduction ‣ Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">1</span></a>).
In these cases, a large fraction of disagreements will occur on questions to which queries and probes both assign high confidence, but to different outputs.<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>
We use this terminology for consistency with past work, and do not intend any claims about the presence of specific communicative intentions in LMs (q.v. <cite class="ltx_cite ltx_citemacro_citep">Abercrombie et al., <a href="#bib.bib1" title="" class="ltx_ref">2023</a>; Shanahan, <a href="#bib.bib24" title="" class="ltx_ref">2022</a></cite>). In humans, deception involves models of other agents’ mental states <cite class="ltx_cite ltx_citemacro_citep">(Mahon, <a href="#bib.bib14" title="" class="ltx_ref">2008</a>)</cite> of a kind that are not exhibited by the LMs we study <cite class="ltx_cite ltx_citemacro_citep">(Sap et al., <a href="#bib.bib23" title="" class="ltx_ref">2022</a>)</cite>. What we call “deception” is necessary but insufficient for an LM to “believe” one thing but choose to say another.</span></span></span></p>
</div>
</li>
<li id="S2.I2.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">3.</span> 
<div id="S2.I2.i3.p1" class="ltx_para">
<p id="S2.I2.i3.p1.1" class="ltx_p"><span id="S2.I2.i3.p1.1.1" class="ltx_text ltx_font_bold">Heterogeneity</span>: Disagreements resulting from probes and queries exhibiting differential accuracy on specific input subsets (upper-mid of <a href="#S1.F1" title="In 1 Introduction ‣ Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Figure</span> <span class="ltx_text ltx_ref_tag">1</span></a>).
In these cases, probes may outperform queries if the subset of inputs on which probes are more effective is larger than the (disjoint) subset on which queries are more effective.</p>
</div>
</li>
</ol>
</div>
<div id="S2.SS0.SSS0.Px2.p3" class="ltx_para">
<p id="S2.SS0.SSS0.Px2.p3.1" class="ltx_p">These three categories (along with cases of <em id="S2.SS0.SSS0.Px2.p3.1.1" class="ltx_emph ltx_font_italic">query–probe agreement</em>, and <em id="S2.SS0.SSS0.Px2.p3.1.2" class="ltx_emph ltx_font_italic">probe confabulation <em id="S2.SS0.SSS0.Px2.p3.1.2.1" class="ltx_emph ltx_font_upright">and</em> error</em>) are visualized in <a href="#S1.F1" title="In 1 Introduction ‣ Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Fig.</span> <span class="ltx_text ltx_ref_tag">1</span></a>C. Behaviors may occur simultaneously in a single model: we are ultimately interested in understanding what fraction of predictions corresponds to each of these categories.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px3" class="ltx_paragraph">
<h3 class="ltx_title ltx_title_paragraph">Datasets</h3>

<div id="S2.SS0.SSS0.Px3.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px3.p1.1" class="ltx_p">We evaluate predictions on three datasets: BoolQ, a dataset of general-topic yes–no questions derived from search queries <cite class="ltx_cite ltx_citemacro_citep">(Clark et al., <a href="#bib.bib6" title="" class="ltx_ref">2019</a>)</cite>; SciQ, a dataset of crowdsourced multiple-choice science exam questions, and CREAK, a dataset of crowdsourced (true and false) factual assertions <cite class="ltx_cite ltx_citemacro_citep">(Onoe et al., <a href="#bib.bib18" title="" class="ltx_ref">2021</a>)</cite>. We evaluate model behavior on all three datasets via a binary question answering task. SciQ contains multiple wrong answers for each question; we retain the correct answer and select a single distractor.</p>
</div>
</section>
<section id="S2.SS0.SSS0.Px4" class="ltx_paragraph">
<h3 class="ltx_title ltx_title_paragraph">Models</h3>

<div id="S2.SS0.SSS0.Px4.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px4.p1.1" class="ltx_p">As the base LM (and implementation of <math id="S2.SS0.SSS0.Px4.p1.1.m1.1" class="ltx_Math" alttext="p_{\mathrm{query}}" display="inline"><semantics id="S2.SS0.SSS0.Px4.p1.1.m1.1a"><msub id="S2.SS0.SSS0.Px4.p1.1.m1.1.1" xref="S2.SS0.SSS0.Px4.p1.1.m1.1.1.cmml"><mi id="S2.SS0.SSS0.Px4.p1.1.m1.1.1.2" xref="S2.SS0.SSS0.Px4.p1.1.m1.1.1.2.cmml">p</mi><mi id="S2.SS0.SSS0.Px4.p1.1.m1.1.1.3" xref="S2.SS0.SSS0.Px4.p1.1.m1.1.1.3.cmml">query</mi></msub><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px4.p1.1.m1.1b"><apply id="S2.SS0.SSS0.Px4.p1.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.1.m1.1.1"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px4.p1.1.m1.1.1.1.cmml" xref="S2.SS0.SSS0.Px4.p1.1.m1.1.1">subscript</csymbol><ci id="S2.SS0.SSS0.Px4.p1.1.m1.1.1.2.cmml" xref="S2.SS0.SSS0.Px4.p1.1.m1.1.1.2">𝑝</ci><ci id="S2.SS0.SSS0.Px4.p1.1.m1.1.1.3.cmml" xref="S2.SS0.SSS0.Px4.p1.1.m1.1.1.3">query</ci></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px4.p1.1.m1.1c">p_{\mathrm{query}}</annotation></semantics></math>), we use the GPT2-XL <cite class="ltx_cite ltx_citemacro_citep">(Radford et al., <a href="#bib.bib20" title="" class="ltx_ref">2019</a>)</cite> and GPT-J LMs <cite class="ltx_cite ltx_citemacro_citep">(Wang and Komatsuzaki, <a href="#bib.bib26" title="" class="ltx_ref">2021</a>)</cite>. We query LMs by evaluating the probability they assign to correct answers. In BoolQ and CREAK we re-normalize their output distributions over the strings {<em id="S2.SS0.SSS0.Px4.p1.1.1" class="ltx_emph ltx_font_italic">true</em>, <em id="S2.SS0.SSS0.Px4.p1.1.2" class="ltx_emph ltx_font_italic">false</em>}; in SciQ we use provided correct and incorrect answers.<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>Just as different behavior may be exhibited by different models, it may be induced by different prompts or query formats <cite class="ltx_cite ltx_citemacro_citep">(Lin et al., <a href="#bib.bib13" title="" class="ltx_ref">2022</a>)</cite>. We experimented with different query formatting strategies but found no striking changes in results. However, future work may more systematically study the distribution of query–probe disagreements induced by different prompts.</span></span></span></p>
</div>
</section>
<section id="S2.SS0.SSS0.Px5" class="ltx_paragraph">
<h3 class="ltx_title ltx_title_paragraph">Probes</h3>

<div id="S2.SS0.SSS0.Px5.p1" class="ltx_para">
<p id="S2.SS0.SSS0.Px5.p1.1" class="ltx_p">While the space of probe designs is large, many recent studies have used <em id="S2.SS0.SSS0.Px5.p1.1.1" class="ltx_emph ltx_font_italic">linear</em> probes in LMs <cite class="ltx_cite ltx_citemacro_citep">(Hernandez and Andreas, <a href="#bib.bib10" title="" class="ltx_ref">2021</a>; Ravfogel et al., <a href="#bib.bib21" title="" class="ltx_ref">2022</a>; Burns et al., <a href="#bib.bib5" title="" class="ltx_ref">2022</a>; Marks and Tegmark, <a href="#bib.bib15" title="" class="ltx_ref">2023</a>)</cite>.<span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span>The probing paradigm has limitations. A successful probe does not indicate that the LM necessarily uses the feature being probed for, and an unsuccessful probe does not indicate that the LM does not use the feature being probed for <cite class="ltx_cite ltx_citemacro_cite">Ravichander et al. (<a href="#bib.bib22" title="" class="ltx_ref">2020</a>); Elazar et al. (<a href="#bib.bib8" title="" class="ltx_ref">2021</a>); Belinkov (<a href="#bib.bib4" title="" class="ltx_ref">2022</a>)</cite>.</span></span></span> We train a linear model (using a logistic regression objective) to classify (question, answer) pairs as correct or incorrect using the training split of each dataset described above.
The input to this linear model is the LM’s final hidden state in the final layer (we did not find substantial differences in probe accuracy using different layers or representation pooling techniques).
As in past work, we obtain a distribution over answers by normalizing over <math id="S2.SS0.SSS0.Px5.p1.1.m1.3" class="ltx_Math" alttext="p_{\mathrm{probe}}(\texttt{correct}\mid q,a)" display="inline"><semantics id="S2.SS0.SSS0.Px5.p1.1.m1.3a"><mrow id="S2.SS0.SSS0.Px5.p1.1.m1.3.3" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.cmml"><msub id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.3" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.3.cmml"><mi id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.3.2" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.3.2.cmml">p</mi><mi id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.3.3" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.3.3.cmml">probe</mi></msub><mo lspace="0em" rspace="0em" id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.2" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.2.cmml">​</mo><mrow id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.cmml"><mo stretchy="false" id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.2" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.cmml">(</mo><mrow id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.cmml"><mtext class="ltx_mathvariant_monospace" id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.2" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.2a.cmml">correct</mtext><mo id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.1" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.1.cmml">∣</mo><mrow id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.3.2" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.3.1.cmml"><mi id="S2.SS0.SSS0.Px5.p1.1.m1.1.1" xref="S2.SS0.SSS0.Px5.p1.1.m1.1.1.cmml">q</mi><mo id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.3.2.1" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.3.1.cmml">,</mo><mi id="S2.SS0.SSS0.Px5.p1.1.m1.2.2" xref="S2.SS0.SSS0.Px5.p1.1.m1.2.2.cmml">a</mi></mrow></mrow><mo stretchy="false" id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.3" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.SS0.SSS0.Px5.p1.1.m1.3b"><apply id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.cmml" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3"><times id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.2.cmml" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.2"></times><apply id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.3.cmml" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.3"><csymbol cd="ambiguous" id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.3.1.cmml" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.3">subscript</csymbol><ci id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.3.2.cmml" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.3.2">𝑝</ci><ci id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.3.3.cmml" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.3.3">probe</ci></apply><apply id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.cmml" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1"><csymbol cd="latexml" id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.1.cmml" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.1">conditional</csymbol><ci id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.2a.cmml" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.2"><mtext class="ltx_mathvariant_monospace" id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.2.cmml" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.2">correct</mtext></ci><list id="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.3.1.cmml" xref="S2.SS0.SSS0.Px5.p1.1.m1.3.3.1.1.1.3.2"><ci id="S2.SS0.SSS0.Px5.p1.1.m1.1.1.cmml" xref="S2.SS0.SSS0.Px5.p1.1.m1.1.1">𝑞</ci><ci id="S2.SS0.SSS0.Px5.p1.1.m1.2.2.cmml" xref="S2.SS0.SSS0.Px5.p1.1.m1.2.2">𝑎</ci></list></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.SS0.SSS0.Px5.p1.1.m1.3c">p_{\mathrm{probe}}(\texttt{correct}\mid q,a)</annotation></semantics></math>. Our main results train an ordinary linear model; additional results with a <em id="S2.SS0.SSS0.Px5.p1.1.2" class="ltx_emph ltx_font_italic">sparse</em> probing objective <cite class="ltx_cite ltx_citemacro_citep">(as in e.g. Voita and Titov, <a href="#bib.bib25" title="" class="ltx_ref">2020</a>)</cite> are in <a href="#A1" title="Appendix A Sparse Probing Results ‣ Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Appendix</span> <span class="ltx_text ltx_ref_tag">A</span></a>.</p>
</div>
<figure id="S2.T1" class="ltx_table">
<div id="S2.T1.1" class="ltx_inline-block ltx_align_center ltx_transformed_outer" style="width:433.6pt;height:124.3pt;vertical-align:-0.0pt;"><span class="ltx_transformed_inner" style="transform:translate(59.9pt,-17.2pt) scale(1.38132069325114,1.38132069325114) ;">
<table id="S2.T1.1.1" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S2.T1.1.1.1.1" class="ltx_tr">
<td id="S2.T1.1.1.1.1.1" class="ltx_td ltx_border_tt"></td>
<th id="S2.T1.1.1.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2">BoolQ</th>
<th id="S2.T1.1.1.1.1.3" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2">SciQ</th>
<th id="S2.T1.1.1.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt" colspan="2">CREAK</th>
</tr>
<tr id="S2.T1.1.1.2.2" class="ltx_tr">
<td id="S2.T1.1.1.2.2.1" class="ltx_td"></td>
<th id="S2.T1.1.1.2.2.2" class="ltx_td ltx_align_center ltx_th ltx_th_column">GPT-2</th>
<th id="S2.T1.1.1.2.2.3" class="ltx_td ltx_align_center ltx_th ltx_th_column">GPT-J</th>
<th id="S2.T1.1.1.2.2.4" class="ltx_td ltx_align_center ltx_th ltx_th_column">GPT-2</th>
<th id="S2.T1.1.1.2.2.5" class="ltx_td ltx_align_center ltx_th ltx_th_column">GPT-J</th>
<th id="S2.T1.1.1.2.2.6" class="ltx_td ltx_align_center ltx_th ltx_th_column">GPT-2</th>
<th id="S2.T1.1.1.2.2.7" class="ltx_td ltx_align_center ltx_th ltx_th_column">GPT-J</th>
</tr>
<tr id="S2.T1.1.1.3.3" class="ltx_tr">
<td id="S2.T1.1.1.3.3.1" class="ltx_td ltx_align_center ltx_border_t">Query</td>
<td id="S2.T1.1.1.3.3.2" class="ltx_td ltx_align_center ltx_border_t">61.8</td>
<td id="S2.T1.1.1.3.3.3" class="ltx_td ltx_align_center ltx_border_t">61.8</td>
<td id="S2.T1.1.1.3.3.4" class="ltx_td ltx_align_center ltx_border_t">76.1</td>
<td id="S2.T1.1.1.3.3.5" class="ltx_td ltx_align_center ltx_border_t">84.1</td>
<td id="S2.T1.1.1.3.3.6" class="ltx_td ltx_align_center ltx_border_t">51.1</td>
<td id="S2.T1.1.1.3.3.7" class="ltx_td ltx_align_center ltx_border_t">50.4</td>
</tr>
<tr id="S2.T1.1.1.4.4" class="ltx_tr">
<td id="S2.T1.1.1.4.4.1" class="ltx_td ltx_align_center">Probe</td>
<td id="S2.T1.1.1.4.4.2" class="ltx_td ltx_align_center">62.9</td>
<td id="S2.T1.1.1.4.4.3" class="ltx_td ltx_align_center">62.5</td>
<td id="S2.T1.1.1.4.4.4" class="ltx_td ltx_align_center">78.8</td>
<td id="S2.T1.1.1.4.4.5" class="ltx_td ltx_align_center">88.6</td>
<td id="S2.T1.1.1.4.4.6" class="ltx_td ltx_align_center">63.5</td>
<td id="S2.T1.1.1.4.4.7" class="ltx_td ltx_align_center">71.0</td>
</tr>
<tr id="S2.T1.1.1.5.5" class="ltx_tr">
<td id="S2.T1.1.1.5.5.1" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">Ensemble</td>
<td id="S2.T1.1.1.5.5.2" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">–</td>
<td id="S2.T1.1.1.5.5.3" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">62.7</td>
<td id="S2.T1.1.1.5.5.4" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">79.7</td>
<td id="S2.T1.1.1.5.5.5" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">90.5</td>
<td id="S2.T1.1.1.5.5.6" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">–</td>
<td id="S2.T1.1.1.5.5.7" class="ltx_td ltx_align_center ltx_border_bb ltx_border_t">71.3</td>
</tr>
</tbody>
</table>
</span></div>
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_table">Table 1: </span>Accuracies of different evaluation approaches. As in past work, we find that probes are consistently more accurate than queries. Surprisingly, by ensembling probes and queries together, it is possible in 4 of 6 cases to obtain further improvements.</figcaption>
</figure>
<figure id="S2.F2" class="ltx_figure"><img src="/html/2312.03729/assets/x3.png" id="S2.F2.g1" class="ltx_graphics ltx_img_landscape" width="332" height="233" alt="Refer to caption">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 2: </span>
Calibration of GPT-J queries and probes. Each point represents a group of predictions: the horizontal axis shows the query’s average confidence <math id="S2.F2.3.m1.3" class="ltx_Math" alttext="\mathbb{E}_{q,a}p(a\mid q)" display="inline"><semantics id="S2.F2.3.m1.3b"><mrow id="S2.F2.3.m1.3.3" xref="S2.F2.3.m1.3.3.cmml"><msub id="S2.F2.3.m1.3.3.3" xref="S2.F2.3.m1.3.3.3.cmml"><mi id="S2.F2.3.m1.3.3.3.2" xref="S2.F2.3.m1.3.3.3.2.cmml">𝔼</mi><mrow id="S2.F2.3.m1.2.2.2.4" xref="S2.F2.3.m1.2.2.2.3.cmml"><mi id="S2.F2.3.m1.1.1.1.1" xref="S2.F2.3.m1.1.1.1.1.cmml">q</mi><mo id="S2.F2.3.m1.2.2.2.4.1" xref="S2.F2.3.m1.2.2.2.3.cmml">,</mo><mi id="S2.F2.3.m1.2.2.2.2" xref="S2.F2.3.m1.2.2.2.2.cmml">a</mi></mrow></msub><mo lspace="0em" rspace="0em" id="S2.F2.3.m1.3.3.2" xref="S2.F2.3.m1.3.3.2.cmml">​</mo><mi id="S2.F2.3.m1.3.3.4" xref="S2.F2.3.m1.3.3.4.cmml">p</mi><mo lspace="0em" rspace="0em" id="S2.F2.3.m1.3.3.2b" xref="S2.F2.3.m1.3.3.2.cmml">​</mo><mrow id="S2.F2.3.m1.3.3.1.1" xref="S2.F2.3.m1.3.3.1.1.1.cmml"><mo stretchy="false" id="S2.F2.3.m1.3.3.1.1.2" xref="S2.F2.3.m1.3.3.1.1.1.cmml">(</mo><mrow id="S2.F2.3.m1.3.3.1.1.1" xref="S2.F2.3.m1.3.3.1.1.1.cmml"><mi id="S2.F2.3.m1.3.3.1.1.1.2" xref="S2.F2.3.m1.3.3.1.1.1.2.cmml">a</mi><mo id="S2.F2.3.m1.3.3.1.1.1.1" xref="S2.F2.3.m1.3.3.1.1.1.1.cmml">∣</mo><mi id="S2.F2.3.m1.3.3.1.1.1.3" xref="S2.F2.3.m1.3.3.1.1.1.3.cmml">q</mi></mrow><mo stretchy="false" id="S2.F2.3.m1.3.3.1.1.3" xref="S2.F2.3.m1.3.3.1.1.1.cmml">)</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.F2.3.m1.3c"><apply id="S2.F2.3.m1.3.3.cmml" xref="S2.F2.3.m1.3.3"><times id="S2.F2.3.m1.3.3.2.cmml" xref="S2.F2.3.m1.3.3.2"></times><apply id="S2.F2.3.m1.3.3.3.cmml" xref="S2.F2.3.m1.3.3.3"><csymbol cd="ambiguous" id="S2.F2.3.m1.3.3.3.1.cmml" xref="S2.F2.3.m1.3.3.3">subscript</csymbol><ci id="S2.F2.3.m1.3.3.3.2.cmml" xref="S2.F2.3.m1.3.3.3.2">𝔼</ci><list id="S2.F2.3.m1.2.2.2.3.cmml" xref="S2.F2.3.m1.2.2.2.4"><ci id="S2.F2.3.m1.1.1.1.1.cmml" xref="S2.F2.3.m1.1.1.1.1">𝑞</ci><ci id="S2.F2.3.m1.2.2.2.2.cmml" xref="S2.F2.3.m1.2.2.2.2">𝑎</ci></list></apply><ci id="S2.F2.3.m1.3.3.4.cmml" xref="S2.F2.3.m1.3.3.4">𝑝</ci><apply id="S2.F2.3.m1.3.3.1.1.1.cmml" xref="S2.F2.3.m1.3.3.1.1"><csymbol cd="latexml" id="S2.F2.3.m1.3.3.1.1.1.1.cmml" xref="S2.F2.3.m1.3.3.1.1.1.1">conditional</csymbol><ci id="S2.F2.3.m1.3.3.1.1.1.2.cmml" xref="S2.F2.3.m1.3.3.1.1.1.2">𝑎</ci><ci id="S2.F2.3.m1.3.3.1.1.1.3.cmml" xref="S2.F2.3.m1.3.3.1.1.1.3">𝑞</ci></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F2.3.m1.3d">\mathbb{E}_{q,a}p(a\mid q)</annotation></semantics></math>, the vertical axis shows the query’s empirical accuracy <math id="S2.F2.4.m2.1" class="ltx_Math" alttext="\mathbb{E}[a\textrm{ is correct}]" display="inline"><semantics id="S2.F2.4.m2.1b"><mrow id="S2.F2.4.m2.1.1" xref="S2.F2.4.m2.1.1.cmml"><mi id="S2.F2.4.m2.1.1.3" xref="S2.F2.4.m2.1.1.3.cmml">𝔼</mi><mo lspace="0em" rspace="0em" id="S2.F2.4.m2.1.1.2" xref="S2.F2.4.m2.1.1.2.cmml">​</mo><mrow id="S2.F2.4.m2.1.1.1.1" xref="S2.F2.4.m2.1.1.1.2.cmml"><mo stretchy="false" id="S2.F2.4.m2.1.1.1.1.2" xref="S2.F2.4.m2.1.1.1.2.1.cmml">[</mo><mrow id="S2.F2.4.m2.1.1.1.1.1" xref="S2.F2.4.m2.1.1.1.1.1.cmml"><mi id="S2.F2.4.m2.1.1.1.1.1.2" xref="S2.F2.4.m2.1.1.1.1.1.2.cmml">a</mi><mo lspace="0em" rspace="0em" id="S2.F2.4.m2.1.1.1.1.1.1" xref="S2.F2.4.m2.1.1.1.1.1.1.cmml">​</mo><mtext id="S2.F2.4.m2.1.1.1.1.1.3" xref="S2.F2.4.m2.1.1.1.1.1.3a.cmml"> is correct</mtext></mrow><mo stretchy="false" id="S2.F2.4.m2.1.1.1.1.3" xref="S2.F2.4.m2.1.1.1.2.1.cmml">]</mo></mrow></mrow><annotation-xml encoding="MathML-Content" id="S2.F2.4.m2.1c"><apply id="S2.F2.4.m2.1.1.cmml" xref="S2.F2.4.m2.1.1"><times id="S2.F2.4.m2.1.1.2.cmml" xref="S2.F2.4.m2.1.1.2"></times><ci id="S2.F2.4.m2.1.1.3.cmml" xref="S2.F2.4.m2.1.1.3">𝔼</ci><apply id="S2.F2.4.m2.1.1.1.2.cmml" xref="S2.F2.4.m2.1.1.1.1"><csymbol cd="latexml" id="S2.F2.4.m2.1.1.1.2.1.cmml" xref="S2.F2.4.m2.1.1.1.1.2">delimited-[]</csymbol><apply id="S2.F2.4.m2.1.1.1.1.1.cmml" xref="S2.F2.4.m2.1.1.1.1.1"><times id="S2.F2.4.m2.1.1.1.1.1.1.cmml" xref="S2.F2.4.m2.1.1.1.1.1.1"></times><ci id="S2.F2.4.m2.1.1.1.1.1.2.cmml" xref="S2.F2.4.m2.1.1.1.1.1.2">𝑎</ci><ci id="S2.F2.4.m2.1.1.1.1.1.3a.cmml" xref="S2.F2.4.m2.1.1.1.1.1.3"><mtext id="S2.F2.4.m2.1.1.1.1.1.3.cmml" xref="S2.F2.4.m2.1.1.1.1.1.3"> is correct</mtext></ci></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S2.F2.4.m2.1d">\mathbb{E}[a\textrm{ is correct}]</annotation></semantics></math>, and point radius shows the number of predictions in the group. Probes are substantially better calibrated than LM queries.
</figcaption>
</figure>
<figure id="S2.F3" class="ltx_figure"><img src="/html/2312.03729/assets/x4.png" id="S2.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="332" height="125" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure">Figure 3: </span>Distribution of query and probe predictions. “Deception”-like results (orange) do not feature particularly prominently compared to other outcomes. Note that heatmaps (left) use the same shade of gray for “both correct” and “both incorrect”, even though they are distinguished in histograms (right) to enable direct comparison.</figcaption>
</figure>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3 </span>The success of probes over queries can largely be explained by better calibration</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">By evaluating accuracy of probes and queries on held-out data, we replicate the finding that probes are more accurate than queries (<a href="#S2.T1" title="In Probes ‣ 2 Preliminaries ‣ Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">1</span></a>). In some cases the difference is small (less than 1% for GPT-J on BoolQ); in other cases, it is substantial (more than 20% on CREAK).</p>
</div>
<div id="S3.p2" class="ltx_para">
<p id="S3.p2.1" class="ltx_p">Another striking difference between probes and queries is visualized in <a href="#S2.F2" title="In Probes ‣ 2 Preliminaries ‣ Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Fig.</span> <span class="ltx_text ltx_ref_tag">2</span></a>, which plots the calibration of predictions from GPT-J (e.g. when a model predicts an answer with 70% probability, is it right 70% of the time?). Probes are well calibrated, while queries are reasonably well calibrated on SciQ but poorly calibrated on other datasets. This aligns with the finding by <cite class="ltx_cite ltx_citemacro_citet">Mielke et al. (<a href="#bib.bib17" title="" class="ltx_ref">2022</a>)</cite> that it is possible to obtain calibrated prediction of model <em id="S3.p2.1.1" class="ltx_emph ltx_font_italic">errors</em> (without assigning probabilities to specific answers) using a similar probe.</p>
</div>
<div id="S3.p3" class="ltx_para">
<p id="S3.p3.1" class="ltx_p">It is important to emphasize that the main goal of this work is to understand representational correlates of truthfulness, not to build more accurate models.
Indeed, the superior calibration and accuracy of probes over queries might not seem surprising given that probes are trained many-shot while queries use the model zero-shot. To contextualize these results, we also finetuned GPT2-XL on true question/answer pairs and found it performed better overall than probing the pretrained model. Probe predictions exhibited 1.7% better accuracy than fine-tuned model queries on BoolQ. However, querying the fine-tuned model had 6.9% and 4.0% better accuracy on SciQ and CREAK.</p>
</div>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4 </span>Most disagreements result from confabulation and heterogeneity</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">Next, we study the joint distribution of query and probe predictions. Results are visualized in <a href="#S2.F3" title="In Probes ‣ 2 Preliminaries ‣ Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Fig.</span> <span class="ltx_text ltx_ref_tag">3</span></a>. GPT2-XL and GPT-J exhibit a similar pattern of errors on each dataset, but that distribution varies substantially between datasets. In BoolQ and SciQ, the query is correct and the probe is incorrect nearly as often as the opposite case; on both datasets, examples of highly confident but contradictory predictions by probes and queries are rare.</p>
</div>
<div id="S4.p2" class="ltx_para">
<p id="S4.p2.1" class="ltx_p"><a href="#S1.F1" title="In 1 Introduction ‣ Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Fig.</span> <span class="ltx_text ltx_ref_tag">1</span></a>D shows the fraction of examples in each of the nine categories depicted in <a href="#S1.F1" title="In 1 Introduction ‣ Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Fig.</span> <span class="ltx_text ltx_ref_tag">1</span></a>C. Only in CREAK do we observe a significant number of instances of “deception”; however, we also observe many instances of probe errors.
In all other cases (even SciQ, where probes are substantially more accurate than queries), query–probe mismatches predominantly represent other types of disagreement.</p>
</div>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5 </span>Queries and probes are complementary</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">Another observation in <a href="#S1.F1" title="In 1 Introduction ‣ Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Fig.</span> <span class="ltx_text ltx_ref_tag">1</span></a> is that almost all datasets exhibit a substantial portion of heterogeneity: there are large subsets of the data in which probes have low confidence, but queries assign high probability to the correct answer (and vice-versa).</p>
</div>
<div id="S5.p2" class="ltx_para">
<p id="S5.p2.1" class="ltx_p">We can exploit this fact by constructing an <em id="S5.p2.1.1" class="ltx_emph ltx_font_italic">ensemble</em> model of the form:</p>
<table id="S5.E1" class="ltx_equation ltx_eqn_table">

<tbody><tr class="ltx_equation ltx_eqn_row ltx_align_baseline">
<td class="ltx_eqn_cell ltx_eqn_center_padleft"></td>
<td class="ltx_eqn_cell ltx_align_center"><math id="S5.E1.m1.39" class="ltx_Math" alttext="\begin{split}p_{\mathrm{ensemble}}(a\mid q)=\;&amp;\lambda\cdot p_{\mathrm{probe}}(a\mid q)\\
&amp;+(1-\lambda)\cdot p_{\mathrm{query}}(a\mid q)\end{split}" display="block"><semantics id="S5.E1.m1.39a"><mtable columnspacing="0pt" displaystyle="true" rowspacing="0pt" id="S5.E1.m1.39.39.8" xref="S5.E1.m1.35.35.4.cmml"><mtr id="S5.E1.m1.39.39.8a" xref="S5.E1.m1.35.35.4.cmml"><mtd class="ltx_align_right" columnalign="right" id="S5.E1.m1.39.39.8b" xref="S5.E1.m1.35.35.4.cmml"><mrow id="S5.E1.m1.36.36.5.32.18.9" xref="S5.E1.m1.35.35.4.cmml"><mrow id="S5.E1.m1.36.36.5.32.18.9.9" xref="S5.E1.m1.35.35.4.cmml"><msub id="S5.E1.m1.36.36.5.32.18.9.9.3" xref="S5.E1.m1.35.35.4.cmml"><mi id="S5.E1.m1.1.1.1.1.1.1" xref="S5.E1.m1.1.1.1.1.1.1.cmml">p</mi><mi id="S5.E1.m1.2.2.2.2.2.2.1" xref="S5.E1.m1.2.2.2.2.2.2.1.cmml">ensemble</mi></msub><mo lspace="0em" rspace="0em" id="S5.E1.m1.36.36.5.32.18.9.9.2" xref="S5.E1.m1.35.35.4.cmml">​</mo><mrow id="S5.E1.m1.36.36.5.32.18.9.9.1.1" xref="S5.E1.m1.35.35.4.cmml"><mo stretchy="false" id="S5.E1.m1.3.3.3.3.3.3" xref="S5.E1.m1.35.35.4.cmml">(</mo><mrow id="S5.E1.m1.36.36.5.32.18.9.9.1.1.1" xref="S5.E1.m1.35.35.4.cmml"><mi id="S5.E1.m1.4.4.4.4.4.4" xref="S5.E1.m1.4.4.4.4.4.4.cmml">a</mi><mo id="S5.E1.m1.5.5.5.5.5.5" xref="S5.E1.m1.5.5.5.5.5.5.cmml">∣</mo><mi id="S5.E1.m1.6.6.6.6.6.6" xref="S5.E1.m1.6.6.6.6.6.6.cmml">q</mi></mrow><mo stretchy="false" id="S5.E1.m1.7.7.7.7.7.7" xref="S5.E1.m1.35.35.4.cmml">)</mo></mrow></mrow><mo id="S5.E1.m1.8.8.8.8.8.8" xref="S5.E1.m1.8.8.8.8.8.8.cmml">=</mo><mi id="S5.E1.m1.36.36.5.32.18.9.10" xref="S5.E1.m1.35.35.4.cmml"></mi></mrow></mtd><mtd class="ltx_align_left" columnalign="left" id="S5.E1.m1.39.39.8c" xref="S5.E1.m1.35.35.4.cmml"><mrow id="S5.E1.m1.37.37.6.33.19.10" xref="S5.E1.m1.35.35.4.cmml"><mrow id="S5.E1.m1.37.37.6.33.19.10.12" xref="S5.E1.m1.35.35.4.cmml"><mi id="S5.E1.m1.9.9.9.9.1.1" xref="S5.E1.m1.9.9.9.9.1.1.cmml">λ</mi><mo lspace="0.222em" rspace="0.222em" id="S5.E1.m1.10.10.10.10.2.2" xref="S5.E1.m1.10.10.10.10.2.2.cmml">⋅</mo><msub id="S5.E1.m1.37.37.6.33.19.10.12.1" xref="S5.E1.m1.35.35.4.cmml"><mi id="S5.E1.m1.11.11.11.11.3.3" xref="S5.E1.m1.11.11.11.11.3.3.cmml">p</mi><mi id="S5.E1.m1.12.12.12.12.4.4.1" xref="S5.E1.m1.12.12.12.12.4.4.1.cmml">probe</mi></msub></mrow><mo lspace="0em" rspace="0em" id="S5.E1.m1.37.37.6.33.19.10.11" xref="S5.E1.m1.35.35.4.cmml">​</mo><mrow id="S5.E1.m1.37.37.6.33.19.10.10.1" xref="S5.E1.m1.35.35.4.cmml"><mo stretchy="false" id="S5.E1.m1.13.13.13.13.5.5" xref="S5.E1.m1.35.35.4.cmml">(</mo><mrow id="S5.E1.m1.37.37.6.33.19.10.10.1.1" xref="S5.E1.m1.35.35.4.cmml"><mi id="S5.E1.m1.14.14.14.14.6.6" xref="S5.E1.m1.14.14.14.14.6.6.cmml">a</mi><mo id="S5.E1.m1.15.15.15.15.7.7" xref="S5.E1.m1.15.15.15.15.7.7.cmml">∣</mo><mi id="S5.E1.m1.16.16.16.16.8.8" xref="S5.E1.m1.16.16.16.16.8.8.cmml">q</mi></mrow><mo stretchy="false" id="S5.E1.m1.17.17.17.17.9.9" xref="S5.E1.m1.35.35.4.cmml">)</mo></mrow></mrow></mtd></mtr><mtr id="S5.E1.m1.39.39.8d" xref="S5.E1.m1.35.35.4.cmml"><mtd id="S5.E1.m1.39.39.8e" xref="S5.E1.m1.35.35.4.cmml"></mtd><mtd class="ltx_align_left" columnalign="left" id="S5.E1.m1.39.39.8f" xref="S5.E1.m1.35.35.4.cmml"><mrow id="S5.E1.m1.39.39.8.35.16.16" xref="S5.E1.m1.35.35.4.cmml"><mo id="S5.E1.m1.39.39.8.35.16.16a" xref="S5.E1.m1.35.35.4.cmml">+</mo><mrow id="S5.E1.m1.39.39.8.35.16.16.16" xref="S5.E1.m1.35.35.4.cmml"><mrow id="S5.E1.m1.38.38.7.34.15.15.15.1" xref="S5.E1.m1.35.35.4.cmml"><mrow id="S5.E1.m1.38.38.7.34.15.15.15.1.1.1" xref="S5.E1.m1.35.35.4.cmml"><mo stretchy="false" id="S5.E1.m1.19.19.19.2.2.2" xref="S5.E1.m1.35.35.4.cmml">(</mo><mrow id="S5.E1.m1.38.38.7.34.15.15.15.1.1.1.1" xref="S5.E1.m1.35.35.4.cmml"><mn id="S5.E1.m1.20.20.20.3.3.3" xref="S5.E1.m1.20.20.20.3.3.3.cmml">1</mn><mo id="S5.E1.m1.21.21.21.4.4.4" xref="S5.E1.m1.21.21.21.4.4.4.cmml">−</mo><mi id="S5.E1.m1.22.22.22.5.5.5" xref="S5.E1.m1.22.22.22.5.5.5.cmml">λ</mi></mrow><mo rspace="0.055em" stretchy="false" id="S5.E1.m1.23.23.23.6.6.6" xref="S5.E1.m1.35.35.4.cmml">)</mo></mrow><mo rspace="0.222em" id="S5.E1.m1.24.24.24.7.7.7" xref="S5.E1.m1.24.24.24.7.7.7.cmml">⋅</mo><msub id="S5.E1.m1.38.38.7.34.15.15.15.1.2" xref="S5.E1.m1.35.35.4.cmml"><mi id="S5.E1.m1.25.25.25.8.8.8" xref="S5.E1.m1.25.25.25.8.8.8.cmml">p</mi><mi id="S5.E1.m1.26.26.26.9.9.9.1" xref="S5.E1.m1.26.26.26.9.9.9.1.cmml">query</mi></msub></mrow><mo lspace="0em" rspace="0em" id="S5.E1.m1.39.39.8.35.16.16.16.3" xref="S5.E1.m1.35.35.4.cmml">​</mo><mrow id="S5.E1.m1.39.39.8.35.16.16.16.2.1" xref="S5.E1.m1.35.35.4.cmml"><mo stretchy="false" id="S5.E1.m1.27.27.27.10.10.10" xref="S5.E1.m1.35.35.4.cmml">(</mo><mrow id="S5.E1.m1.39.39.8.35.16.16.16.2.1.1" xref="S5.E1.m1.35.35.4.cmml"><mi id="S5.E1.m1.28.28.28.11.11.11" xref="S5.E1.m1.28.28.28.11.11.11.cmml">a</mi><mo id="S5.E1.m1.29.29.29.12.12.12" xref="S5.E1.m1.29.29.29.12.12.12.cmml">∣</mo><mi id="S5.E1.m1.30.30.30.13.13.13" xref="S5.E1.m1.30.30.30.13.13.13.cmml">q</mi></mrow><mo stretchy="false" id="S5.E1.m1.31.31.31.14.14.14" xref="S5.E1.m1.35.35.4.cmml">)</mo></mrow></mrow></mrow></mtd></mtr></mtable><annotation-xml encoding="MathML-Content" id="S5.E1.m1.39b"><apply id="S5.E1.m1.35.35.4.cmml" xref="S5.E1.m1.39.39.8"><eq id="S5.E1.m1.8.8.8.8.8.8.cmml" xref="S5.E1.m1.8.8.8.8.8.8"></eq><apply id="S5.E1.m1.32.32.1.1.cmml" xref="S5.E1.m1.39.39.8"><times id="S5.E1.m1.32.32.1.1.2.cmml" xref="S5.E1.m1.39.39.8"></times><apply id="S5.E1.m1.32.32.1.1.3.cmml" xref="S5.E1.m1.39.39.8"><csymbol cd="ambiguous" id="S5.E1.m1.32.32.1.1.3.1.cmml" xref="S5.E1.m1.39.39.8">subscript</csymbol><ci id="S5.E1.m1.1.1.1.1.1.1.cmml" xref="S5.E1.m1.1.1.1.1.1.1">𝑝</ci><ci id="S5.E1.m1.2.2.2.2.2.2.1.cmml" xref="S5.E1.m1.2.2.2.2.2.2.1">ensemble</ci></apply><apply id="S5.E1.m1.32.32.1.1.1.1.1.cmml" xref="S5.E1.m1.39.39.8"><csymbol cd="latexml" id="S5.E1.m1.5.5.5.5.5.5.cmml" xref="S5.E1.m1.5.5.5.5.5.5">conditional</csymbol><ci id="S5.E1.m1.4.4.4.4.4.4.cmml" xref="S5.E1.m1.4.4.4.4.4.4">𝑎</ci><ci id="S5.E1.m1.6.6.6.6.6.6.cmml" xref="S5.E1.m1.6.6.6.6.6.6">𝑞</ci></apply></apply><apply id="S5.E1.m1.35.35.4.4.cmml" xref="S5.E1.m1.39.39.8"><plus id="S5.E1.m1.18.18.18.1.1.1.cmml" xref="S5.E1.m1.39.39.8"></plus><apply id="S5.E1.m1.33.33.2.2.1.cmml" xref="S5.E1.m1.39.39.8"><times id="S5.E1.m1.33.33.2.2.1.2.cmml" xref="S5.E1.m1.39.39.8"></times><apply id="S5.E1.m1.33.33.2.2.1.3.cmml" xref="S5.E1.m1.39.39.8"><ci id="S5.E1.m1.10.10.10.10.2.2.cmml" xref="S5.E1.m1.10.10.10.10.2.2">⋅</ci><ci id="S5.E1.m1.9.9.9.9.1.1.cmml" xref="S5.E1.m1.9.9.9.9.1.1">𝜆</ci><apply id="S5.E1.m1.33.33.2.2.1.3.3.cmml" xref="S5.E1.m1.39.39.8"><csymbol cd="ambiguous" id="S5.E1.m1.33.33.2.2.1.3.3.1.cmml" xref="S5.E1.m1.39.39.8">subscript</csymbol><ci id="S5.E1.m1.11.11.11.11.3.3.cmml" xref="S5.E1.m1.11.11.11.11.3.3">𝑝</ci><ci id="S5.E1.m1.12.12.12.12.4.4.1.cmml" xref="S5.E1.m1.12.12.12.12.4.4.1">probe</ci></apply></apply><apply id="S5.E1.m1.33.33.2.2.1.1.1.1.cmml" xref="S5.E1.m1.39.39.8"><csymbol cd="latexml" id="S5.E1.m1.15.15.15.15.7.7.cmml" xref="S5.E1.m1.15.15.15.15.7.7">conditional</csymbol><ci id="S5.E1.m1.14.14.14.14.6.6.cmml" xref="S5.E1.m1.14.14.14.14.6.6">𝑎</ci><ci id="S5.E1.m1.16.16.16.16.8.8.cmml" xref="S5.E1.m1.16.16.16.16.8.8">𝑞</ci></apply></apply><apply id="S5.E1.m1.35.35.4.4.3.cmml" xref="S5.E1.m1.39.39.8"><times id="S5.E1.m1.35.35.4.4.3.3.cmml" xref="S5.E1.m1.39.39.8"></times><apply id="S5.E1.m1.34.34.3.3.2.1.cmml" xref="S5.E1.m1.39.39.8"><ci id="S5.E1.m1.24.24.24.7.7.7.cmml" xref="S5.E1.m1.24.24.24.7.7.7">⋅</ci><apply id="S5.E1.m1.34.34.3.3.2.1.1.1.1.cmml" xref="S5.E1.m1.39.39.8"><minus id="S5.E1.m1.21.21.21.4.4.4.cmml" xref="S5.E1.m1.21.21.21.4.4.4"></minus><cn type="integer" id="S5.E1.m1.20.20.20.3.3.3.cmml" xref="S5.E1.m1.20.20.20.3.3.3">1</cn><ci id="S5.E1.m1.22.22.22.5.5.5.cmml" xref="S5.E1.m1.22.22.22.5.5.5">𝜆</ci></apply><apply id="S5.E1.m1.34.34.3.3.2.1.3.cmml" xref="S5.E1.m1.39.39.8"><csymbol cd="ambiguous" id="S5.E1.m1.34.34.3.3.2.1.3.1.cmml" xref="S5.E1.m1.39.39.8">subscript</csymbol><ci id="S5.E1.m1.25.25.25.8.8.8.cmml" xref="S5.E1.m1.25.25.25.8.8.8">𝑝</ci><ci id="S5.E1.m1.26.26.26.9.9.9.1.cmml" xref="S5.E1.m1.26.26.26.9.9.9.1">query</ci></apply></apply><apply id="S5.E1.m1.35.35.4.4.3.2.1.1.cmml" xref="S5.E1.m1.39.39.8"><csymbol cd="latexml" id="S5.E1.m1.29.29.29.12.12.12.cmml" xref="S5.E1.m1.29.29.29.12.12.12">conditional</csymbol><ci id="S5.E1.m1.28.28.28.11.11.11.cmml" xref="S5.E1.m1.28.28.28.11.11.11">𝑎</ci><ci id="S5.E1.m1.30.30.30.13.13.13.cmml" xref="S5.E1.m1.30.30.30.13.13.13">𝑞</ci></apply></apply></apply></apply></annotation-xml><annotation encoding="application/x-tex" id="S5.E1.m1.39c">\begin{split}p_{\mathrm{ensemble}}(a\mid q)=\;&amp;\lambda\cdot p_{\mathrm{probe}}(a\mid q)\\
&amp;+(1-\lambda)\cdot p_{\mathrm{query}}(a\mid q)\end{split}</annotation></semantics></math></td>
<td class="ltx_eqn_cell ltx_eqn_center_padright"></td>
<td rowspan="1" class="ltx_eqn_cell ltx_eqn_eqno ltx_align_middle ltx_align_right"><span class="ltx_tag ltx_tag_equation ltx_align_right">(1)</span></td>
</tr></tbody>
</table>
</div>
<div id="S5.p3" class="ltx_para">
<p id="S5.p3.1" class="ltx_p">We select <math id="S5.p3.1.m1.1" class="ltx_Math" alttext="\lambda" display="inline"><semantics id="S5.p3.1.m1.1a"><mi id="S5.p3.1.m1.1.1" xref="S5.p3.1.m1.1.1.cmml">λ</mi><annotation-xml encoding="MathML-Content" id="S5.p3.1.m1.1b"><ci id="S5.p3.1.m1.1.1.cmml" xref="S5.p3.1.m1.1.1">𝜆</ci></annotation-xml><annotation encoding="application/x-tex" id="S5.p3.1.m1.1c">\lambda</annotation></semantics></math> using 500 validation examples from each dataset, and evaluate accuracy on the test split. Results are shown at the bottom of <a href="#S2.T1" title="In Probes ‣ 2 Preliminaries ‣ Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Table</span> <span class="ltx_text ltx_ref_tag">1</span></a>; in 4/6 cases, this ensemble model is able to improve over the probe. While improvements on BoolQ and CREAK are small, on SciQ they are substantial—nearly as large as the improvement of the probe over the base LM. These results underscore the fact that, even when probes significantly outperform queries, query errors are not the source of all mismatches, and heterogeneity in probing and querying pathways can be exploited.</p>
</div>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6 </span>Conclusion</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">We studied mismatches between language models’ factual assertions and probes trained on their internal states, and showed that these mismatches reflect a diverse set of situations including confabulation, heterogeneity, and a small number of instances of deception.
In current models, disagreements between internal and external representations of truthfulness appear predominantly attributable to different prediction pathways, rather than a latent intent to produce incorrect output.
A variety of other model interventions are known to <em id="S6.p1.1.1" class="ltx_emph ltx_font_italic">decrease</em> truthfulness, including carefully designed prompts <cite class="ltx_cite ltx_citemacro_citep">(Lin et al., <a href="#bib.bib13" title="" class="ltx_ref">2022</a>)</cite>, and future models may exhibit more complex relationships between internal representations and generated text (especially for open-ended generation).
Even in these cases, we expect the taxonomy in <a href="#S1.F1" title="In 1 Introduction ‣ Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Fig.</span> <span class="ltx_text ltx_ref_tag">1</span></a> to remain useful: not all mismatches between model and probe behavior involve deception, and not all model behaviors are (currently) reducible to easily decodable properties of their internal states.</p>
</div>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7 </span>Limitations</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.1" class="ltx_p">As seen in <a href="#S1.F1" title="In 1 Introduction ‣ Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Fig.</span> <span class="ltx_text ltx_ref_tag">1</span></a>, there is significant heterogeneity in the distribution of disagreement types across datasets. These specific findings may thus not predict the distribution of disagreements in future datasets.
As noted in <a href="#S2" title="2 Preliminaries ‣ Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness?" class="ltx_ref"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">2</span></a>, our experiments use only a single prompt template for each experiment; we believe it is likely (especially in CREAK) that better prompts exist that would substantially alter the distribution of disagreements—our goal in this work has been to establish a taxonomy of errors for future models. Finally, we have presented only results on linear probes. The success of ensembling methods means that some information must be encoded non-linearly in model representations.</p>
</div>
</section>
<section id="S8" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">8 </span>Ethical Considerations</h2>

<div id="S8.p1" class="ltx_para">
<p id="S8.p1.1" class="ltx_p">Our work is motivated by ethical concerns raised in past work <cite class="ltx_cite ltx_citemacro_citep">(e.g., Askell et al., <a href="#bib.bib2" title="" class="ltx_ref">2021</a>; Evans et al., <a href="#bib.bib9" title="" class="ltx_ref">2021</a>)</cite>
that LMs might (perhaps unintentionally) mislead users. The techniques presented here might be used to improve model truthfulness or detect errors. However, better understanding of model-internal representations of factuality and truthfulness might enable system developers to steer models toward undesirable or harmful behaviors.</p>
</div>
<div id="S8.p2" class="ltx_para">
<p id="S8.p2.1" class="ltx_p">Finally, while we have presented techniques for slightly improving the accuracy of models on question answering tasks, models continue to make a significant number of errors, and are not suitable for deployment in applications where factuality and reliability are required.</p>
</div>
</section>
<section id="Sx1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">Acknowledgments</h2>

<div id="Sx1.p1" class="ltx_para">
<p id="Sx1.p1.1" class="ltx_p">This work was supported by a grant from the OpenPhilanthropy foundation.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Abercrombie et al. (2023)</span>
<span class="ltx_bibblock">
Gavin Abercrombie, Amanda Cercas Curry, Tanvi Dinkar, and Zeerak Talat. 2023.

</span>
<span class="ltx_bibblock">Mirages: On anthropomorphism in dialogue systems.

</span>
<span class="ltx_bibblock"><em id="bib.bib1.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2305.09800</em>.

</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Askell et al. (2021)</span>
<span class="ltx_bibblock">
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan,
Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021.

</span>
<span class="ltx_bibblock">A general language assistant as a laboratory for alignment.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2112.00861</em>.

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Azaria and Mitchell (2023)</span>
<span class="ltx_bibblock">
Amos Azaria and Tom Mitchell. 2023.

</span>
<span class="ltx_bibblock">The internal state of an LLM knows when it’s lying.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2304.13734</em>.

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Belinkov (2022)</span>
<span class="ltx_bibblock">
Yonatan Belinkov. 2022.

</span>
<span class="ltx_bibblock">Probing classifiers: Promises, shortcomings, and advances.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Computational Linguistics</em>, 48(1):207–219.

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Burns et al. (2022)</span>
<span class="ltx_bibblock">
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. 2022.

</span>
<span class="ltx_bibblock">Discovering latent knowledge in language models without supervision.

</span>
<span class="ltx_bibblock">In <em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">The Eleventh International Conference on Learning
Representations</em>.

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Clark et al. (2019)</span>
<span class="ltx_bibblock">
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael
Collins, and Kristina Toutanova. 2019.

</span>
<span class="ltx_bibblock">BoolQ: Exploring the surprising difficulty of natural yes/no
questions.

</span>
<span class="ltx_bibblock">In <em id="bib.bib6.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Conference of the North American Chapter
of the Association for Comptuational Linguistics</em>.

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Edwards (2023)</span>
<span class="ltx_bibblock">
Benj Edwards. 2023.

</span>
<span class="ltx_bibblock">Why ChatGPT and Bing Chat are so good at making things up.

</span>
<span class="ltx_bibblock"><em id="bib.bib7.1.1" class="ltx_emph ltx_font_italic">Ars Technica</em>.

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Elazar et al. (2021)</span>
<span class="ltx_bibblock">
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021.

</span>
<span class="ltx_bibblock">Amnesic probing: Behavioral explanation with amnesic counterfactuals.

</span>
<span class="ltx_bibblock"><em id="bib.bib8.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>,
9:160–175.

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Evans et al. (2021)</span>
<span class="ltx_bibblock">
Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit,
Peter Wills, Luca Righetti, and William Saunders. 2021.

</span>
<span class="ltx_bibblock">Truthful AI: Developing and governing AI that does not lie.

</span>
<span class="ltx_bibblock"><em id="bib.bib9.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2110.06674</em>.

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hernandez and Andreas (2021)</span>
<span class="ltx_bibblock">
Evan Hernandez and Jacob Andreas. 2021.

</span>
<span class="ltx_bibblock">The low-dimensional linear geometry of contextualized word
representations.

</span>
<span class="ltx_bibblock">In <em id="bib.bib10.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Conference on Natural Language Learning</em>.

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ji et al. (2023)</span>
<span class="ltx_bibblock">
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii,
Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.

</span>
<span class="ltx_bibblock">Survey of hallucination in natural language generation.

</span>
<span class="ltx_bibblock"><em id="bib.bib11.1.1" class="ltx_emph ltx_font_italic">ACM Computing Surveys</em>, 55(12):1–38.

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kadavath et al. (2022)</span>
<span class="ltx_bibblock">
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan
Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli
Tran-Johnson, et al. 2022.

</span>
<span class="ltx_bibblock">Language models (mostly) know what they know.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2207.05221</em>.

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Lin et al. (2022)</span>
<span class="ltx_bibblock">
Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.

</span>
<span class="ltx_bibblock">TruthfulQA: Measuring how models mimic human falsehoods.

</span>
<span class="ltx_bibblock">In <em id="bib.bib13.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Annual Meeting of the Association for
Computational Linguistics</em>.

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mahon (2008)</span>
<span class="ltx_bibblock">
James Edwin Mahon. 2008.

</span>
<span class="ltx_bibblock">The definition of lying and deception.

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Marks and Tegmark (2023)</span>
<span class="ltx_bibblock">
Samuel Marks and Max Tegmark. 2023.

</span>
<span class="ltx_bibblock">The geometry of truth: Emergent linear structure in large language
model representations of true/false datasets.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.06824</em>.

</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Martindale et al. (2019)</span>
<span class="ltx_bibblock">
Marianna Martindale, Marine Carpuat, Kevin Duh, and Paul McNamee. 2019.

</span>
<span class="ltx_bibblock">Identifying fluently inadequate output in neural and statistical
machine translation.

</span>
<span class="ltx_bibblock">In <em id="bib.bib16.1.1" class="ltx_emph ltx_font_italic">Proceedings of Machine Translation Summit XVII: Research
Track</em>.

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mielke et al. (2022)</span>
<span class="ltx_bibblock">
Sabrina J Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022.

</span>
<span class="ltx_bibblock">Reducing conversational agents’ overconfidence through linguistic
calibration.

</span>
<span class="ltx_bibblock"><em id="bib.bib17.1.1" class="ltx_emph ltx_font_italic">Transactions of the Association for Computational Linguistics</em>,
10:857–872.

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Onoe et al. (2021)</span>
<span class="ltx_bibblock">
Yasumasa Onoe, Michael JQ Zhang, Eunsol Choi, and Greg Durrett. 2021.

</span>
<span class="ltx_bibblock">CREAK: A dataset for commonsense reasoning over entity knowledge.

</span>
<span class="ltx_bibblock">In <em id="bib.bib18.1.1" class="ltx_emph ltx_font_italic">NeurIPS Datasets And Benchmarks</em>.

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Petroni et al. (2019)</span>
<span class="ltx_bibblock">
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton
Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.

</span>
<span class="ltx_bibblock">Language models as knowledge bases?

</span>
<span class="ltx_bibblock">In <em id="bib.bib19.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Conference on Empirical Methods in
Natural Language Processing</em>.

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Radford et al. (2019)</span>
<span class="ltx_bibblock">
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya
Sutskever, et al. 2019.

</span>
<span class="ltx_bibblock">Language models are unsupervised multitask learners.

</span>
<span class="ltx_bibblock"><em id="bib.bib20.1.1" class="ltx_emph ltx_font_italic">OpenAI blog</em>, 1(8):9.

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ravfogel et al. (2022)</span>
<span class="ltx_bibblock">
Shauli Ravfogel, Michael Twiton, Yoav Goldberg, and Ryan D Cotterell. 2022.

</span>
<span class="ltx_bibblock">Linear adversarial concept erasure.

</span>
<span class="ltx_bibblock">In <em id="bib.bib21.1.1" class="ltx_emph ltx_font_italic">Proceedings of the International Conference on Machine
Learning</em>.

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ravichander et al. (2020)</span>
<span class="ltx_bibblock">
Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. 2020.

</span>
<span class="ltx_bibblock">Probing the probing paradigm: Does probing accuracy entail task
relevance?

</span>
<span class="ltx_bibblock">In <em id="bib.bib22.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Annual Meeting of the European
Association for Computational Linguistics</em>.

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sap et al. (2022)</span>
<span class="ltx_bibblock">
Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. 2022.

</span>
<span class="ltx_bibblock">Neural theory-of-mind? On the limits of social intelligence in
large lms.

</span>
<span class="ltx_bibblock">In <em id="bib.bib23.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2022 Conference on Empirical Methods in
Natural Language Processing</em>.

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shanahan (2022)</span>
<span class="ltx_bibblock">
Murray Shanahan. 2022.

</span>
<span class="ltx_bibblock">Talking about large language models.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2212.03551</em>.

</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Voita and Titov (2020)</span>
<span class="ltx_bibblock">
Elena Voita and Ivan Titov. 2020.

</span>
<span class="ltx_bibblock">Information-theoretic probing with minimum description length.

</span>
<span class="ltx_bibblock">In <em id="bib.bib25.1.1" class="ltx_emph ltx_font_italic">Proceedings of the Conference on Empirical Methods in
Natural Language Processing</em>.

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Wang and Komatsuzaki (2021)</span>
<span class="ltx_bibblock">
Ben Wang and Aran Komatsuzaki. 2021.

</span>
<span class="ltx_bibblock">GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.

</span>
<span class="ltx_bibblock"><a target="_blank" href="https://github.com/kingoflolz/mesh-transformer-jax" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://github.com/kingoflolz/mesh-transformer-jax</a>.

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Zou et al. (2023)</span>
<span class="ltx_bibblock">
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren,
Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al.
2023.

</span>
<span class="ltx_bibblock">Representation engineering: A top-down approach to AI transparency.

</span>
<span class="ltx_bibblock"><em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">arXiv preprint arXiv:2310.01405</em>.

</span>
</li>
</ul>
</section>
<section id="A1" class="ltx_appendix">
<h2 class="ltx_title ltx_title_appendix">
<span class="ltx_tag ltx_tag_appendix">Appendix A </span>Sparse Probing Results</h2>

<figure id="A1.F4" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2312.03729/assets/x5.png" id="A1.F4.g1" class="ltx_graphics ltx_figure_panel ltx_img_landscape" width="166" height="102" alt="Refer to caption"></div>
<div class="ltx_flex_cell ltx_flex_size_2"><img src="/html/2312.03729/assets/x6.png" id="A1.F4.g2" class="ltx_graphics ltx_figure_panel ltx_img_landscape" width="166" height="102" alt="Refer to caption"></div>
<div class="ltx_flex_break"></div>
<div class="ltx_flex_cell ltx_flex_size_1">
<p id="A1.F4.2" class="ltx_p ltx_figure_panel">(a) BoolQ                                                          (b) SciQ 
<br class="ltx_break">
<img src="/html/2312.03729/assets/x7.png" id="A1.F4.1.g1" class="ltx_graphics ltx_img_landscape" width="166" height="102" alt="Refer to caption">
 <img src="/html/2312.03729/assets/x8.png" id="A1.F4.2.g2" class="ltx_graphics ltx_img_landscape" width="166" height="92" alt="Refer to caption">
                           (c) CREAK</p>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure">Figure 4: </span>Sparse probing results. In these experiments, we train the same probes as in the main paper, but with a varying <math id="A1.F4.5.m1.1" class="ltx_Math" alttext="\ell_{1}" display="inline"><semantics id="A1.F4.5.m1.1b"><msub id="A1.F4.5.m1.1.1" xref="A1.F4.5.m1.1.1.cmml"><mi mathvariant="normal" id="A1.F4.5.m1.1.1.2" xref="A1.F4.5.m1.1.1.2.cmml">ℓ</mi><mn id="A1.F4.5.m1.1.1.3" xref="A1.F4.5.m1.1.1.3.cmml">1</mn></msub><annotation-xml encoding="MathML-Content" id="A1.F4.5.m1.1c"><apply id="A1.F4.5.m1.1.1.cmml" xref="A1.F4.5.m1.1.1"><csymbol cd="ambiguous" id="A1.F4.5.m1.1.1.1.cmml" xref="A1.F4.5.m1.1.1">subscript</csymbol><ci id="A1.F4.5.m1.1.1.2.cmml" xref="A1.F4.5.m1.1.1.2">ℓ</ci><cn type="integer" id="A1.F4.5.m1.1.1.3.cmml" xref="A1.F4.5.m1.1.1.3">1</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="A1.F4.5.m1.1d">\ell_{1}</annotation></semantics></math> penalty applied to the probing objective to encourage the discovery of sparse solutions. Here we report the level of sparsity, probe accuracy, and distribution of disagreement types as we vary the strength of the regularizer within <math id="A1.F4.6.m2.4" class="ltx_Math" alttext="\{0,0.01,0.03,0.1\}" display="inline"><semantics id="A1.F4.6.m2.4b"><mrow id="A1.F4.6.m2.4.5.2" xref="A1.F4.6.m2.4.5.1.cmml"><mo stretchy="false" id="A1.F4.6.m2.4.5.2.1" xref="A1.F4.6.m2.4.5.1.cmml">{</mo><mn id="A1.F4.6.m2.1.1" xref="A1.F4.6.m2.1.1.cmml">0</mn><mo id="A1.F4.6.m2.4.5.2.2" xref="A1.F4.6.m2.4.5.1.cmml">,</mo><mn id="A1.F4.6.m2.2.2" xref="A1.F4.6.m2.2.2.cmml">0.01</mn><mo id="A1.F4.6.m2.4.5.2.3" xref="A1.F4.6.m2.4.5.1.cmml">,</mo><mn id="A1.F4.6.m2.3.3" xref="A1.F4.6.m2.3.3.cmml">0.03</mn><mo id="A1.F4.6.m2.4.5.2.4" xref="A1.F4.6.m2.4.5.1.cmml">,</mo><mn id="A1.F4.6.m2.4.4" xref="A1.F4.6.m2.4.4.cmml">0.1</mn><mo stretchy="false" id="A1.F4.6.m2.4.5.2.5" xref="A1.F4.6.m2.4.5.1.cmml">}</mo></mrow><annotation-xml encoding="MathML-Content" id="A1.F4.6.m2.4c"><set id="A1.F4.6.m2.4.5.1.cmml" xref="A1.F4.6.m2.4.5.2"><cn type="integer" id="A1.F4.6.m2.1.1.cmml" xref="A1.F4.6.m2.1.1">0</cn><cn type="float" id="A1.F4.6.m2.2.2.cmml" xref="A1.F4.6.m2.2.2">0.01</cn><cn type="float" id="A1.F4.6.m2.3.3.cmml" xref="A1.F4.6.m2.3.3">0.03</cn><cn type="float" id="A1.F4.6.m2.4.4.cmml" xref="A1.F4.6.m2.4.4">0.1</cn></set></annotation-xml><annotation encoding="application/x-tex" id="A1.F4.6.m2.4d">\{0,0.01,0.03,0.1\}</annotation></semantics></math>. Except at extremely high sparsity values, both accuracies and error distributions remain similar to those reported in the main set of experiments.</figcaption>
</figure>
</section>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2312.03727" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2312.03729" class="ar5iv-text-button ar5iv-severity-warning">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2312.03729">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2312.03729" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2312.03730" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Tue Feb 27 14:42:07 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2312.03729
Neural language models (LMs)  can be used to evaluate the truth of factual statements in two ways: they can be eitherqueriedfor statement probabilities, orprobedfor internal representations of truthfulness.
Past work has found that these two procedures sometimes disagree, and that probes tend to be more accurate than LM outputs. This has led some researchers to conclude that LMs “lie” or otherwise encode non-cooperative communicative intents.
Is this an accurate description of today’s LMs, or can query–probe disagreement arise in other ways? We identify three different classes of disagreement, which we termconfabulation,deception, andheterogeneity.
In many cases, the superiority of probes is simply attributable to better calibration on uncertain answers rather than a greater fraction of correct, high-confidence answers.
In some cases, queries and probes perform better on different subsets of inputs, and accuracy can further be improved by ensembling the two.111Code atgithub.com/lingo-mit/lm-truthfulness.
abstract
Cognitive Dissonance:Why Do Language Model Outputs Disagreewith Internal Representations of Truthfulness?
Abstract
2312.03729
1IntroductionText generated by neural language models (LMs)  often contains factual errors. These errors limit LMs’ ability to generate trustworthy content and serve as knowledge sources in downstream applications.Surprisingly, even when LMoutputsare factually incorrect, it is sometimes possible to assess the truth of statements by probing models’hidden states.
In the example shown inFig.1A, a language model assigns high probability to the incorrect answeryeswhen prompted to answer the questionIs Sting a police officer?However, a linearknowledge probetrained on the LM’s hidden representations (Fig.1B)  successfully classifiesnoas the more likely answer.
Knowledge probes of this kind have consistently been found to be slightly more reliable at question answering and fact verification than direct queries to LMs,
suggesting a mismatch between LMs’ (internal)  representations of factuality and their (external)  expressions of statement probability.Figure 1:Varieties of disagreement between language model outputs and internal states. We evaluate two approaches for answering questions or verifying statements:queryingmodels directly for answers (A) , or training a binary classifier toprobetheir internal states (B) . Probes and queries sometimes disagree. We propose a taxonomy of different query–probe disagreement types (C) ; across several models and datasets, we find that disagreements mostly occur in situations where either probes or queries are uncertain (D) .How should we understand this behavior? One possible interpretation, suggested by several previous studies, is that it is analogous to “deception” in human language users. In this framing, if LM representations encode truthfulness more reliably than outputs, then mismatches between LM queries and probes must result from situations in which LMs “know” that a statement is untrue and nonetheless assign it high probability.But knowledge is not a binary phenomenon in LMs or humans.
While probes are better than queries on average, they may not be better on every example.
Moreover, both LMs and probes are probabilistic models: LM predictions can encode uncertainty over possible answers, and this uncertainty can be decoded from their internal representations. There are thus a variety of mechanisms that might underlie query–probe disagreement, ranging from differences in calibration to differences in behavior in different input subsets.In this paper, we seek to
understand mismatches between internal and external expressions of truthfulness by understanding thedistributionover predictions, taking into account uncertainty in answers produced by both queries and probes.
In doing so, we hope to provide a formal grounding of several terms that are widely (but vaguely and inconsistently)  used to describe factual errors in LMs.
LMs and probing techniques are rapidly evolving, so we do not aim to provide a definitive answer to the question of why LMs assign false statements high probability.
However, in a widely studied probe class, two LMs, and three question-answering datasets, we identify three qualitatively different reasons that probes may outperform queries, which we callconfabulation(queries produce high-confidence answers when probes are low-confidence) ,heterogeneity(probes and queries improve performance on different data subsets) , and (in a small number of instances)  what past work would characterize asdeception(Fig.1C–D, in which queries and probes disagree confidently on answer probabilities) .
Most mismatches occur when probes or queries are uncertain about answers. By combining probe and query predictions, it is sometimes possible to obtain better accuracy than either alone.Our results paint a nuanced picture of the representation of factuality in LMs: even when probes outperform LMs, they do not explain all of LMs’ ability to make factual predictions. Today, many mismatches between internal and external representations of factuality appear attributable to different prediction pathways, rather than an explicit representation of a latent intent to generate outputs “known” to be incorrect.
introduction
Cognitive Dissonance:Why Do Language Model Outputs Disagreewith Internal Representations of Truthfulness?
1Introduction
2312.03729
2PreliminariesExtracting answers from LMsWe study autoregressive LMs that compute a next-token distributionby first mapping the inputto some hidden representation, then using this representation to predict. There are two standard procedures for answering questions using such an LM:1.Querying: Provide the questionas input to the LM, and rank answers(e.g.yes/no)  according to.2.Probing: Extract the hidden representationfrom the LM, then use question/answer pairs labeled for correctness (or unsupervised clustering methods)  to train a binary classifier that maps fromto a distribution.Causes of disagreementPast work has shown that this probing procedure is effective; recent work has shown that it often produces different, and slightly better, answers than direct querying.
Why might this mismatch occur? In this paper, we define three possible cases:1.Model Confabulation: Disagreements occurring when probe confidence is low, and the completions from queries are incorrect (mid-left ofFigure1) . In these cases
probes may be slightly more accurate if their prediction confidence is better calibrated to the probability of correctness.
In LMs exhibiting confabulation, a large fraction of disagreements will occur on inputs with large probe entropyand large query confidencefor any.2.“Deception”, which we define as the set of disagreements in which the probe is confidently correct and the query completion is confidently incorrect (upper-left ofFigure1) .
In these cases, a large fraction of disagreements will occur on questions to which queries and probes both assign high confidence, but to different outputs.222We use this terminology for consistency with past work, and do not intend any claims about the presence of specific communicative intentions in LMs (q.v.) . In humans, deception involves models of other agents’ mental statesof a kind that are not exhibited by the LMs we study. What we call “deception” is necessary but insufficient for an LM to “believe” one thing but choose to say another.3.Heterogeneity: Disagreements resulting from probes and queries exhibiting differential accuracy on specific input subsets (upper-mid ofFigure1) .
In these cases, probes may outperform queries if the subset of inputs on which probes are more effective is larger than the (disjoint)  subset on which queries are more effective.These three categories (along with cases ofquery–probe agreement, andprobe confabulationanderror)  are visualized inFig.1C. Behaviors may occur simultaneously in a single model: we are ultimately interested in understanding what fraction of predictions corresponds to each of these categories.DatasetsWe evaluate predictions on three datasets: BoolQ, a dataset of general-topic yes–no questions derived from search queries; SciQ, a dataset of crowdsourced multiple-choice science exam questions, and CREAK, a dataset of crowdsourced (true and false)  factual assertions. We evaluate model behavior on all three datasets via a binary question answering task. SciQ contains multiple wrong answers for each question; we retain the correct answer and select a single distractor.ModelsAs the base LM (and implementation of) , we use the GPT2-XLand GPT-J LMs. We query LMs by evaluating the probability they assign to correct answers. In BoolQ and CREAK we re-normalize their output distributions over the strings {true,false}; in SciQ we use provided correct and incorrect answers.333Just as different behavior may be exhibited by different models, it may be induced by different prompts or query formats. We experimented with different query formatting strategies but found no striking changes in results. However, future work may more systematically study the distribution of query–probe disagreements induced by different prompts.ProbesWhile the space of probe designs is large, many recent studies have usedlinearprobes in LMs.444The probing paradigm has limitations. A successful probe does not indicate that the LM necessarily uses the feature being probed for, and an unsuccessful probe does not indicate that the LM does not use the feature being probed for.We train a linear model (using a logistic regression objective)  to classify (question, answer)  pairs as correct or incorrect using the training split of each dataset described above.
The input to this linear model is the LM’s final hidden state in the final layer (we did not find substantial differences in probe accuracy using different layers or representation pooling techniques) .
As in past work, we obtain a distribution over answers by normalizing over. Our main results train an ordinary linear model; additional results with asparseprobing objectiveare inAppendixA.BoolQSciQCREAKGPT-2GPT-JGPT-2GPT-JGPT-2GPT-JQuery61.861.876.184.151.150.4Probe62.962.578.888.663.571.0Ensemble–62.779.790.5–71.3Table 1:Accuracies of different evaluation approaches. As in past work, we find that probes are consistently more accurate than queries. Surprisingly, by ensembling probes and queries together, it is possible in 4 of 6 cases to obtain further improvements.Figure 2:Calibration of GPT-J queries and probes. Each point represents a group of predictions: the horizontal axis shows the query’s average confidence, the vertical axis shows the query’s empirical accuracy, and point radius shows the number of predictions in the group. Probes are substantially better calibrated than LM queries.Figure 3:Distribution of query and probe predictions. “Deception”-like results (orange)  do not feature particularly prominently compared to other outcomes. Note that heatmaps (left)  use the same shade of gray for “both correct” and “both incorrect”, even though they are distinguished in histograms (right)  to enable direct comparison.
body
Cognitive Dissonance:Why Do Language Model Outputs Disagreewith Internal Representations of Truthfulness?
2Preliminaries
2312.03729
3The success of probes over queries can largely be explained by better calibrationBy evaluating accuracy of probes and queries on held-out data, we replicate the finding that probes are more accurate than queries (Table1) . In some cases the difference is small (less than 1% for GPT-J on BoolQ) ; in other cases, it is substantial (more than 20% on CREAK) .Another striking difference between probes and queries is visualized inFig.2, which plots the calibration of predictions from GPT-J (e.g. when a model predicts an answer with 70% probability, is it right 70% of the time?) . Probes are well calibrated, while queries are reasonably well calibrated on SciQ but poorly calibrated on other datasets. This aligns with the finding bythat it is possible to obtain calibrated prediction of modelerrors(without assigning probabilities to specific answers)  using a similar probe.It is important to emphasize that the main goal of this work is to understand representational correlates of truthfulness, not to build more accurate models.
Indeed, the superior calibration and accuracy of probes over queries might not seem surprising given that probes are trained many-shot while queries use the model zero-shot. To contextualize these results, we also finetuned GPT2-XL on true question/answer pairs and found it performed better overall than probing the pretrained model. Probe predictions exhibited 1.7% better accuracy than fine-tuned model queries on BoolQ. However, querying the fine-tuned model had 6.9% and 4.0% better accuracy on SciQ and CREAK.
body
Cognitive Dissonance:Why Do Language Model Outputs Disagreewith Internal Representations of Truthfulness?
3The success of probes over queries can largely be explained by better calibration
2312.03729
4Most disagreements result from confabulation and heterogeneityNext, we study the joint distribution of query and probe predictions. Results are visualized inFig.3. GPT2-XL and GPT-J exhibit a similar pattern of errors on each dataset, but that distribution varies substantially between datasets. In BoolQ and SciQ, the query is correct and the probe is incorrect nearly as often as the opposite case; on both datasets, examples of highly confident but contradictory predictions by probes and queries are rare.Fig.1D shows the fraction of examples in each of the nine categories depicted inFig.1C. Only in CREAK do we observe a significant number of instances of “deception”; however, we also observe many instances of probe errors.
In all other cases (even SciQ, where probes are substantially more accurate than queries) , query–probe mismatches predominantly represent other types of disagreement.
body
Cognitive Dissonance:Why Do Language Model Outputs Disagreewith Internal Representations of Truthfulness?
4Most disagreements result from confabulation and heterogeneity
2312.03729
5Queries and probes are complementaryAnother observation inFig.1is that almost all datasets exhibit a substantial portion of heterogeneity: there are large subsets of the data in which probes have low confidence, but queries assign high probability to the correct answer (and vice-versa) .We can exploit this fact by constructing anensemblemodel of the form:(1) We selectusing 500 validation examples from each dataset, and evaluate accuracy on the test split. Results are shown at the bottom ofTable1; in 4/6 cases, this ensemble model is able to improve over the probe. While improvements on BoolQ and CREAK are small, on SciQ they are substantial—nearly as large as the improvement of the probe over the base LM. These results underscore the fact that, even when probes significantly outperform queries, query errors are not the source of all mismatches, and heterogeneity in probing and querying pathways can be exploited.
body
Cognitive Dissonance:Why Do Language Model Outputs Disagreewith Internal Representations of Truthfulness?
5Queries and probes are complementary
2312.03729
6ConclusionWe studied mismatches between language models’ factual assertions and probes trained on their internal states, and showed that these mismatches reflect a diverse set of situations including confabulation, heterogeneity, and a small number of instances of deception.
In current models, disagreements between internal and external representations of truthfulness appear predominantly attributable to different prediction pathways, rather than a latent intent to produce incorrect output.
A variety of other model interventions are known todecreasetruthfulness, including carefully designed prompts, and future models may exhibit more complex relationships between internal representations and generated text (especially for open-ended generation) .
Even in these cases, we expect the taxonomy inFig.1to remain useful: not all mismatches between model and probe behavior involve deception, and not all model behaviors are (currently)  reducible to easily decodable properties of their internal states.
body
Cognitive Dissonance:Why Do Language Model Outputs Disagreewith Internal Representations of Truthfulness?
6Conclusion
2312.03729
7LimitationsAs seen inFig.1, there is significant heterogeneity in the distribution of disagreement types across datasets. These specific findings may thus not predict the distribution of disagreements in future datasets.
As noted inSection2, our experiments use only a single prompt template for each experiment; we believe it is likely (especially in CREAK)  that better prompts exist that would substantially alter the distribution of disagreements—our goal in this work has been to establish a taxonomy of errors for future models. Finally, we have presented only results on linear probes. The success of ensembling methods means that some information must be encoded non-linearly in model representations.
body
Cognitive Dissonance:Why Do Language Model Outputs Disagreewith Internal Representations of Truthfulness?
7Limitations
2312.03729
8Ethical ConsiderationsOur work is motivated by ethical concerns raised in past workthat LMs might (perhaps unintentionally)  mislead users. The techniques presented here might be used to improve model truthfulness or detect errors. However, better understanding of model-internal representations of factuality and truthfulness might enable system developers to steer models toward undesirable or harmful behaviors.Finally, while we have presented techniques for slightly improving the accuracy of models on question answering tasks, models continue to make a significant number of errors, and are not suitable for deployment in applications where factuality and reliability are required.
body
Cognitive Dissonance:Why Do Language Model Outputs Disagreewith Internal Representations of Truthfulness?
8Ethical Considerations
2312.03729
AcknowledgmentsThis work was supported by a grant from the OpenPhilanthropy foundation.
conclusion
Cognitive Dissonance:Why Do Language Model Outputs Disagreewith Internal Representations of Truthfulness?
Acknowledgments